These two startups want to compete with Google in internet indexing

Google's dominance in the search market is a consolidated fact. But a key element of that power is less well known: the vast index of pages the company has built and maintains. This index is so large that even alternative search engines often pay to use it.

Now, two small startups are launching their own indexing project. And current conditions may give them a better chance of success compared to past initiatives.

THE European Search Perspective (ESP), a partnership between Ecosia, from Berlin, and Qwant, from Paris, intends to start by focusing on indexing pages in French and German. The objective is to reduce the dependence of its search engines on Google and Bing (Microsoft) indexes, which are expensive and have usage restrictions.

A combination of factors favor ESP: reduced server costs, increased demand for search indexes for generative AI projects, and the recent increase in Bing index access fees. “A little pressure, a little opportunity”, summarizes Christian Kroll, CEO of Ecosia.

The initial objective of ESP is not to completely eliminate Google and Microsoft costs, but to reduce them. At the same time, the two startups aim to generate revenue by offering search data to AI developers and other technology companies.

“We pay per query and I hope that, in a year from now, a significant part of our searches will be answered by our own index. Reaching 50% is perfectly viable”, says Kroll. Even so, Ecosia and Qwant would remain dependent on the two giants. “For the last 10% to 20%, we will still need to access Google and Bing,” he admits.

As for less common languages, such as Portuguese, Kroll is more skeptical. He believes that the lack of sufficient data to train the index would hamper progress in this case.

The goal is to reduce your search engines' dependence on Google and Bing's indexes.

Additionally, ESP must address a common hurdle for new indexing initiatives: many sites only allow Google, Bing, and sometimes one or two other crawlers to access their content. This forces startups to decide whether to respect these rules or risk being blocked.

Kroll warns that relying on just two large North American companies to index the web is a serious problem for Europe – especially given President-elect Donald Trump's nationalist agenda, which, according to Ecosia's CEO, could harm European companies in discussions on technology and commerce.

Tim Berners-Lee, inventor of the web, shares this concern. “When there is such a strong monopoly, innovation is hampered,” he says. For him, the monopoly in search indexing may be a case that market forces alone cannot resolve. “Some monopolies may be natural, but in Europe there are those who argue that search should be treated as a public good.”

Google's defeat in the antitrust case brought by the United States government opens up new possibilities. While the main focus is to demand that Google sell the Chrome browser and end preferential deals with companies like Apple, one part of the trial could have a more direct impact on competition.

This clause requires Google to offer its search index “at marginal cost” to competitors, with the same quality and speed as its own search engine. Furthermore, it must make anonymous data about how users interact with search results freely available.

Kroll has yet to comment on how these changes in the US could help his cause, but he recognizes how much the search market has changed since Google was just a startup challenging the tech giants.

“The technological challenge can be overcome, and I believe there is enough money to do so,” he says. “But we need the right regulations to make it all work.”

---

ABOUT THE AUTHOR

Rob Pegoraro is a journalist who writes about computers, gadgets, social media, apps, and other things that blink or beep. find out more

---