While I agree that it's very difficult to compete on generic search, it's certainly not hard to compete in niche domains. And you don't need anything like 15 billion pages for those.
It’s easy to make a decent small search engine: nobody is working to game your algorithm. Vastly harder to do this as you scale to google size and popularity.
I get that it's not easy to build a good search engine, but on the surface it doesn't seem to be that hard a technical problem to solve either. Is it simply that the R&D required to build something competitive is too high for most companies?
I see opportunity for niche search engines, there are several areas that google does not do well in on purpose it seems.
I think the truly hard part is that so many people accept whatever default is already there. If they get an android phone - they use the search box there. If they are using chrome browser, whatever input box is there on the first screen is obviously the url bar and use that (you and I may know the difference, the average user doesn't care, it's one less click to just type 'google' into the url box in the center of the page, of fbook or whatever, then google brings up the url you were going to (not searching).
This is why I think there is much less hype about competing in this space. Unless there is a thing forcing companies to put other browsers and search boxes on phones, tablets and chromebooks like the microsoft IE debacle so long ago.. then trying to be the next google is impossible, even if you had better results, better tech, etc.
Regardless of that, I think it's quite possible to make much better niche search engines and get them used. If ten micro engines could make 1% of googles revenues each, that would be a decent amount of money in my neck of the woods.
I'd like to see other people post more sources about search tech in general, several searches last year only brought a few info bits on what it may cost to create an index of the net - someone posted some numbers using servers bought off ebay and a rack at hurricane I think - had some numbers for the cost of servers to pull a new index every month or so?
Certainly the tech and costs have changed since that was published, but not much I've seen.
I'm pretty excited at this project posted recently: https://news.ycombinator.com/item?id=16976941
( Show HN: A search engine that doesn't track you, where users vote for results (github.com) )
I am hoping to get some people together to make a less persnickety and fussbudgety search option for people who don't want to be babysit with censoring kids gloves when looking for fun things.
If anyone wants to make a couple adults only engines, or ones that are more fun, let me know.
Average people talk in slang and cut up about less high brow things, the big G gives rank to the college papers and deranks for so many things, it's on the road to being the next yellow pages and sciences journal, but not the place to go when you want fun things anymore.
Yea I really wonder why they don't have competition. You'd think a search engine built with java in the mid 90s would be easy to replicate at least to it's 2005 era standard using modern tech.
What is lacking are easy ways to make vertical specific search engines. You are never going to make a better valuing algorithm than Google for general purpose search. A system designed by domain experts should be able to beat Google most of the time within their own tiny niche.
Great write up and break down of why search is hard and how unique a position Google is in. That said searching the entire web as a business isn't very feasible for many companies now and hence most of the other giants competing with Google instead use the asymmetric approach of bringing content into their own garden and building a specialized search engine around it. Since the data in their own garden is structured the amount of work to build an acceptable quality search engine is relatively less. See apple's app store, facebook search, twitter's search, amazon's a9 etc.
What's so ridiculously hard about indexing and searching? Google does it well, but if I had to use MSN or Yahoo I'd hardly notice.
It's very hard for a startup to beat Google at general search, but by choosing a subset (cheap prices, reviews of consumer electronics, vacations, flights, symptoms of illnesses, mailinglists and forums, blogs, movies, music lyrics, torrents, shareware, etc etc etc) startups will slowly encroach on Google's territory.
Finally, it may be very difficult to build a better Google, but I find it extremely easy to imagine one.
You can't build something that can compete with Google without some very very deep pockets and lot's of data.
Same problem with niche search engines, unless they have some unique features/properties that Google doesn't have you are plainly better off with Google.
Google has a Monopoly on search, which looking at the market will hold up for the foreseeable future.
So unless you can offer a specific feature(set) for a niche or you just want to build it for the hell of it I wouldn't reccomend anyone to go into search engines.
I would probably go with what duckduckgo does but offer unique features that are usefull for one or more niches.
A few years ago, I investigated if there is a business opportunity in quality search.
The biggest hurdle I found no way around is the content. Not all but a lot of high-quality information is paid. You get access to it as a user but not if you want to index it as a platform. And you need a lot of different providers to have good coverage. That might get easier if you have lots of users, but you do not get users without content. The platform chicken-egg problem.
Profitable niches like Bloomberg’s business information definitively exist but this would not be the high-quality generic search engine you described.
It's not there because at this point, it's self-evident. There may be a path to profitability for niche engines, or engines which do something quite different from the big players, but it seems ever more unlikely that a new direct rival to Google/Microsoft/Yahoo will arise.
I mean, I get your point. But there's a difference between the pre-Google world and now: before Google, search sucked, and everyone knew it. Now, search is awesome. It's not really a pain point.
There are still other search engines and were many more prior to "google" becoming a verb. They won over those because their search results were faster/better. Yet there is no barrier to entry in websearch. As seen with DDG, some people value privacy higher so they have a niche.
I think the real reason why nobody really tries to compete is not upfront money or techinical issues, but that nobody can think of a way to monetize successfully while attractively differentiating themselves from google search.
Just as a side note: not every search engine is a direct competitor to Google. You can have vertical search engines serving a specific purpose. These can do very well by executing on a narrow use case. Something the Googles can’t do.
There are more than enough users aware of search to support a fledgling competitor that managed to deliver higher quality results.
That competitor simply doesn’t exist yet, and I think that’s because no one has figured out how to beat Google at search (which is why I think real ingenuity is required).
Yes the Internet needs several viable search competitors but since the cost of maintaining all the data and creating a good algorithm are so high the barrier to entry seems almost insurmountable without some really clever innovations.
Sure, but the comment I replied to did not talk about a "niche" search engine. If you find a specific area of search that there is no good search engine for, or if existing products are really bad, of course you may succeed in that space, probably because you're at that point targeting the enterprise market -- but expecting normal consumers to pay for a search engine is naive at best.
reply