Google search used to be one of the wonders of the modern digital world, but (and I admit this is subjective) it feels like the quality of results has decreased drastically over the years. I just can't find stuff as easily as I used to. It still comes out on top for e.g. querying for technical information but for other things (image search) I don't even bother with it anymore. It also feels like certain kinds of queries are subject to malicious SEO practices where it becomes impossible to find results that aren't from the same group of tabloids/media outlets.
What happened? Did they let their golden goose starve to death? Or is this all in my head and Google is actually fine for everyone else?
For one, like you said, there are so many spam websites out there that it's extremely difficult for Google (or any search engine for that matter) to gather genuine results. I also think that it's a matter of incentive. As long as traffic doesn't decline, does Google really have any incentive to fix or improve things?
Also, I presume the search algorithm is not how it used to be too, what with ML models being used left and right. I wonder if anyone really knows how we get the results anymore. There are a few initiatives like kagi but I know too few about them to say anything at this point. For now, I am quite content with appending "reddit" at the end of every non-technical query nowadays... at least until they start gaming this as well.
Pretty much the core of the issue. Shouldn't it be possible to, somewhat reliably, filter out all commercial content? That would probably increase the average result quality by a lot.
Google search has been in decline for at least the past decade, and been surpassed easily for at least the past 5 years. Try running a local instance of a meta-search engine like https://github.com/searx/searx - you'll never want to go back.
This greatly appeals to me, but I don't quite understand how these work. I looked into searx and friends, and it seems too good to be true. Like, if it's turning around and making requests to Google, don't they detect that as bot-like traffic? Seems like a good way to get your IP blocked or worse.
Is it feasibly priced? Often enough APIs from big walled services don't seem to be any more than something to point towards when regulators raise concerns about market dominance.
It serve its users, you are probably in the niche category that's declining, or rather the casual group is growing in number and as a result your category is being diluted even more, tech stuff usually are old stuff, users prefer new and up to date topics
I know that's the big money-maker for them, but it irritates me that almost the entire first page of results is adverts made to look like search results.
It's great at showing explicit ads or "content marketing" ads-that-land-in-it-without-paying-Google-directly.
On one hand you have Google, trying to keep patching an algorithm while also maintaining an ad revenue stream.
On the other hand you have the whole of humanity's ingenuity trying to make money for themselves.
Why do we think one system is going to be able to defeat the whole universe of creative hacks to try to beat that system at any reasonable rate?
(It's a wonder that it's even still adequate... but as long as the main competitors are using the same machine-and-math-driven approach, just with fewer resources to throw at it, it's unlikely that there will be a ubiquitous replacement making a major improvement any time soon.)
It's absolutely declined, no question. Further, the advanced search tools have been all but eviscerated, so it's even more difficult to find what you want even if you know what you're looking for.
OTOH adding 'forum' (or sometimes 'discussion') has been quite effective recently, unlike a couple of years ago as I remember. I even get the occasional forum result without having to do so. Possibly intentional in response to feedback?
You have to use "site:reddit.com" which selects that specific domain, otherwise you get any page that contains the word reddit, and spammers are certainly catching on to that trick.
Isn’t it mostly that the structure of content on the web has changed? It's mostly centralized social media now, so that's where the most relevant results are going to be. Most of the attention is there, anyway.
I read through the search you linked, and it was an interesting journey.
I'd say that I'm not sure the search you linked supports the idea that people have been asking about a decline in quality in Google for 13 years continuously. Most of the threads about that topic seem to start about 4 years ago and pick up in frequency until the now.
Though this could just be a result of the search engine's limited ability to index old threads or some other sampling bias (like there being more users on the site recently).
All of that said here's roughly what I saw in the threads you linked:
2009: People notice that advanced settings in search seem to not work as well as before.
2011: It seems like Google now will start to ignore some of the words in your query if they're uncommon.
2011: Google starts giving preferential treatment to its own products like Google Reviews, Shopping, and Youtube.
2012: Users note Google Plus's "+1"s seem to have an outsized influence on search results.
2013: Google Hummingbird launches - it now appears that your query always gets passed through an interpretation layer. Making precise queries becomes significantly more difficult. Note this was probably (?) to make voice search return better results since those queries were more conversational.
2015: The importance of site performance in general and mobile specifically seems to rankings seemingly increased significantly. Likely influenced by Android One project.
2016: Google increases the importance of following their "webmaster guidelines" and W3C validation.
2016: Amit Singhal who oversaw Google Search for 15 years leaves the company to join Uber.
2017: Google starts to try and identify and deprioritize "offensive", "upsetting", and "inappropriate" content, as crowdsourced from users and paid evaluators.
2018: Google launches an update aimed at websites with low "expertise, authority, or trustworthiness." They state that they punish sites which are "not in line with the general scientific consensus" or which have negative sentiment associated with them on review sites.
- It's about this point that threads about Google's decreasing search quality show a strong uptick.
2019: Google starts using BERT - a machine learning model for interpreting search queries.
2020: Google releases updates aimed at further reducing the rankings of "misinformation" or "biased" content.
- At this point there appears to be strong consensus, even outside of technical circles that Google's search results are deteriorating. To the point where there are memes and popular Youtube videos about it.
2021: Google releases MUM - an AI/ML model that aims to reduce the number of queries a user needs to make by including results for related queries that it thinks you'll make next.
2022 (May): An update to Google's search algorithm deprioritizes reference websites like dictionaries, lyric websites, and wikis while promoting video content.
2022 (June): After years denying that a pages 'freshness' affects its ranking, Google increases its importance in the algorithm dramatically - reportedly affecting the ranking of 35% of search results.
----
I'm not a search engine guy and don't have any particular knowledge about Google's search engine. But reading through the history of HN discussion about it does paint an interesting picture.
I'll leave it to others to speculate about any correlations between these changes and the end product.
I appreciate the analysis! This is definitely interesting to see laid out.
You're right, it was definitely hyperbolic to say it's been posted monthly on a continuous basis dating back that far. As you've pointed out, it would have been more accurate to say it was first posted 14 years ago and has been posted (close to) monthly for the last few years.
There’s been some debate about this here on HN and someone made a point that resonated with me:
The quality of _the web_ has been in decline lately.
With ML’s capacity to paraphrase original content and to generate plausible rubbish content from scratch, it’s very difficult for Google’s pagerank (or whatever they call their algo these days) to fight back.
That been said, there does seem to be a fair bit of scraping and paste going on. I’m surprised G is not looking at published dates and lowering the the scammers ranking.
Is it the quality of the web that has been declining, or is our ability to find quality information on the web (due to Google and friends) affecting our perception of the web?
I'm on the fence on this.
Contributors and commenters on HN manage to surface many interesting sites that would be difficult to find using a modern search engine. There is also a large number of old sites that have continued to exist, even if they are otherwise unmaintained, which are also difficult to find using a modern search engine. On the other hand, these sites may be less common than they used to be.
Likewise, scraped and pasted websites are nothing new. It has always been a relatively low effort way to post content. What seems to be new is how often nearly identical pages appear in the top search results. This could be because it is more common, but it may also be because the algorithms are favouring very particular types of content.
Also the quality of hyperlinks has declined precipitously. Google's big insight was that what people say about a page matters more than what the page says itself. Hyperlinks were fundamental to the (pre-2013) ranking algorithm, in ways that were far more fundamental than just pagerank. This worked really well when people authored HTML pages by hand (or in FrontPage or DreamWeaver) and would promiscuously link out whenever another site was relevant.
It works really poorly when all the links are paid for, or bargained for, or part of social media sites that are overrun with spammers and use rel=nofollow anyway, or are internal-only because every site wants to be its own walled garden. That's the web we've got now.
As well as the fact fewer ppl are maintaining their own websites or blog. The content is now in places that cant be crawled as easily like Instagram, Tiktok and Twitter as well as FB groups.
The loss of blogs is so huge in all sorts of ways.
Some random person's travel blog in 2008 would be full of genuinely useful information. That's gone now, replaced by regular people just posting pics to social media and then affiliate-link filled seo-optimized travel sites, half the time written by someone who hasn't even gone to the place but is just copying info and adding sponsored content.
The same is true for essentially everything- gardening, video games, books, bikes, bird watching, baking. The genuine amateur enthusiast content isn't published on the open web in accessible text. It's locked in walled gardens or just never created in the first place, with people choosing to post a couple pictures or videos rather than write a blog post about it.
Look at python documentation. There is a metric ton of spam sites that copy and paste python documentation with ads inserted. Google ranks these higher than the primary source.
I suspect google is boosting ad supported content over non ad supported content. Directly incentivizing paraphrased/copy pasta content.
I see this with questions and answers from reddit or other forums which get syndicated into various other 'developer' sites and get high rankings on Google.
Search engines should let us configure a whitelist of sites for certain categories/context of search.
Which can be so frustrating when a bad answer gets propagated this way.
I've had times where the same "bad" information (whether completely wrong, incomplete, misleading, not best-practice, confusing, whatever manner of "bad") showed up on multiple different sites all on the first page of Google, often clearly copied from one another or the same original quora/stackoverflow/whatever.
I'm pretty sure that Google search page rank, ranks sites with Google ads HIGHER than the same site without any ads. Of course they would, it gives them better metrics to show their Google ads are effective and they can charge more.
It's not just you. Google is now what Altavista and the other search engines it replaced became. The difference is that those went to crap in only a couple years where Google held out for decades and then suddenly has spiraled to where it is now. And it's the same reasons those search engines in the 90s went down the tubes. Everyone who wasn't there will tell you Google's superior technology (PageRank) is why it won. But that's not really why. It's because AltaVista, Lycos, etc. got awful to use, which cause us to look for an alternative. There was this nice clean upstart that said "Don't be evil." is their motto.
With all the Googlers here and this question being asked again and again including by myself I wonder if this feedback has already reached the right folks at google but there just isn't much that could be done given the business incentives at play.
Honestly I don’t think anyone at Google or Facebook really cares that much. They are old companies, the big money has been made, and it’s pretty much impossible to launch a new successful product. Everything big has been acquisitions for a decade. If you climb up to executive level it’s just twiddling thumbs until the cash cow starts to die, then you ride off into the sunset because nothing is going to replace the cash cow.
And then the products - does anyone wake up every day feeling super excited to help push ads and shrill partisan hacks to boomers? It’s like to work there you just accept “this is how it is and I like money” and you punch a card and leave. It must be a miserable existence for anyone who truly ponders it.
I dont know about miserable. The money, benefits and work/life balance can be very good. And while the work is uninteresting, its probably a lot more tolerable than working elsewhere.
It is a philosophical thing for me personally. I did a stretch at a meaningless company with good pay and perks and I was so unbelievably bored. If you have mentally accepted your productive output is without personal meaning and you are only in it for a check, then I agree it’s a good gig. I just personally can’t do it. And I think the general no-fucks-given by the workforce is why a lot of stuff is just stagnating and declining.
Gotcha. I also share the same sentiment but my sentiment is even darker. I think most products dont need to exist - not just Google or BigTech. The same applies to project mgt tools, that new fintech product, or that new SaaS tool. Almost everything has been done. Given that i cant go back to school to learn to do genomics or rocket engineering, im just gonna resign myself to go thru the motions and get a paycheck, and thats it.
I am a perpetual optimist and I disagree that everything has been done. We have 500 kinds of deodorant because the market needs it - each one is a little different and resonates with a different consumer for some reason. Believing "we only need one thing" of anything is authoritarian.
My path out of the grind was entrepreneurialism. Much more rewarding personally and financially. Good luck to you
I don't know, I've been there and it's like I have an opportunity to learn and do something wildly different or new and that's something to look forward to even if the only person that will appreciate what I did is myself. Imagine working at google if you found a flaw in their page rank algorithm or monarch db implementation/design? That would feel gratifying. I personally don't care much if I contribute to a society that gives -1 shits about me or appreciated by coworkers that will throw me under the bus if it meant a brownie point for them (been there, done that).
1. Search has to be hard because filtering spam is a cat and mouse game inherently.
2. Useful information has moved off of searchable sources. Instead of forums, we have discord, facebook groups, slack channels, twitter...
3. There is simply more shit, exponentially, than when I was young, and therefore there's more garbage lying around. The ratio of shit:not shit hasn't gotten better.
Text search on the web will slowly die. No one trusts the results of random text. Google is in the adversarial position of wanting to censor certain answers as well as present answers that maximize their own revenue.
People will search video based content instead, and use the fact that a human spoke the information, as well as comments/upvotes to vet it as trustworthy material (like on TikTok).
Google search as we know it will slowly die, and then will decline like Facebook. TikTok will steal search marketshare as their video clips span all of human life.
I don't understand this argument. "Votes" and Comments are almost completely gamed on every popular website already, I personally don't trust them in 95% of cases. Video content is usually very much a lowest common denominator type of information, and I don't understand why I should trust "a human said this" over "a human wrote this".
TikTok basically eviscerated facebook over night. Gen Z already makes run of the mill searches on that platform, and is increasingly doing so. Older generations cannot see the massive changes occuring. Facebook stock tanking is not random. Neither is Netflix. TikTok is completely underestimated. All of the new content is being created there. People are not creating text based content on facebook or on blogs. Reddit is the last bastion
I agree. When Alex Jones looks me in the eye and tells me the military is putting chemicals in the water to turn the frogs gay, I know he's telling me the truth.
What would happen to Alphabet/Google if over 50% of people were blocking their ads?
Stop using Google products. Block their ads. Don't use their web browser.
I recommend Mozilla Firefox and uBlock Origin but any other web browser reduces their tracking and allows ad blocking. It looks like Google Chrome will prevent ad blockers soon.
In the last couple of months Google's results have improved for me, I'm no longer seeing stack overflow clones, there are fewer content farm results too.
1) I can search for a handful of specific terms I remember were on a webpage I didn't bookmark, and I can type them in and find what I'm looking for instead of them being "interpreted" by something trying to be too smart
2) Nobody bothers to send them the same DMCA takedowns that they send to Google, so I can find torrents and things I couldn't otherwise
But on most days, Google is easier to find useful stuff when I don't know specifically what I'm looking for or the exact terms to use.
In an ideal world, I'd have Google default to more of how DuckDuckGo seems to work, with a "fuzzy search" option you can use if needed
If you want high quality materials, you need a University library or a journal, whitepapers or an academic textbook. Or if you want practical information and experience, you need to be an employee or apprentice or business owner.
The web is the lowest common denominator. The high quality information is elsewhere and Google isn't allowed to reveal it due to rights holders, such as Google Scholar and Google Books.
I tend to spend my time on Infinity Family https://o2oo.li and https://halfbakery.com a community of inventors and creators. We have an ontology to solve the world's problems and work towards goals and projects as a community. We relax and share opinionated perspectives to problems in the world.
No, Google search remains awesome for me. I’d switch if there was something that gave better results, but that doesn’t exist for me. I gave DDG a fair shake (about 3 months) and it was horrid.
I’m also not concerned with privacy/whatever so I won’t switch to something equally as good or worse to make a statement.
I suspect, but can’t prove that the entire internet is rotting. I get the feeling that organic user contributions are plummeting and being swamped by bots. HN is a small life raft. Many of the niche subreddits I’ve been following for years are seeing their post frequency drop, from dozens a days to a few a week. All the other forums are gone. Maybe there’s something going on in the apps, but they aren’t searchable. The internet is becoming a cold dead place. There’s not much for Google to find these days.
Case in point: I was just trying to find some information about TGV rail routes. Standard Google searched yielded nothing but spam. Appending “Reddit” game me posts more than 5 years old with out of date information.
It feels as if internet is sharded just like MMORPG servers. Nowadays there are more humans than ever with access to internet but organic creations are getting harder to find and everything is swamped with SEO spam.
I still can’t exactly point my finger on what’s wrong.
The impact of walled gardens and "use our app for a better experience" has accumulated.
The constant moving of communities is also a problem. We dropped irc, join us on slack. Crickets.. er we moved to discord. Each move must impact engagement negatively.
I feel like so much has moved to Discord. That’s great for immediacy and back-and-fourth interaction but it puts all of that information that used to be Google searchable into a black hole of Discord searches.
You might be interested in Linen (https://www.linen.dev/) which aims to make Slack and Discord communities Google-Searchable. I cannot attest its effectiveness, but I figure it seems relevant.
Could you elaborate a bit more about everything moving to Discord? Is this a thing generally or just a side effects of being in a team that uses discord for work?
Many communities have moved to discord. Where once there might have been a web site on geocities and an irc channel or mail list, now there is just a discord server.
geocities and all those little sites are gone, and what replaced them are a sequence of less-googlable less-preservable things like myspace pages, facebook pages and groups, blogspot/blogger pages, and now discord servers and youtube channels.
There isn't a good platform for individuals and small communities that aren't profitable businesses to host their resources and documents and communications. There are only companies trying to capitalize on people's desire for it, providing something that only purports to provide that platform but really doesn't.
In some ways, there is nothing stopping people from making niche topic sites, but something seems to have changed that makes it just less likely. Maybe there used to be free tiers (was tripod and geocities free?) and now there is only cheap ($5 for a vps), or maybe it's that now you have to do more work yourself setting up a vps vs a premade site host, and the current equivalents of free and fully managed are discord & facebook?
Something is definitely different and somehow the aggregate result is a lot less of the old plethora of niche topic sites. Maybe just that everyone got jobs and has no time to make those sites any more, and now all we have are developers personal blog sites which are really just promo for themselves not really of any value to you or me.
Maybe also the passing of the innocent initial window of time where an individual or small group could run a phpBB on the public net without it getting hacked or overrun by spam or sued or in legal trouble for the member-written content etc, without an income stream to pay moderators and lawyers.
Probably it's a combination of those and even more other factors I'm not thinking of.
I have also noticed a kind of implosion happening - an emptying-out of real human-generated thoughts and content. I have hypotheses but they would be considered highly "ideological" in this forum.
Eh, I'm left wing, that's all, and let me say that I come here to get points of view that are not my own as a way of not existing purely in echo chambers. So I'm not saying this place is bad in any way, in fact it's one of the few places I still come to find links to read and so forth. I barely touch Reddit even, anymore, cause it seems like every subreddit has become captive of one viewpoint or another. Matters purely technical soothe my mind at this point, because there's really no way to cram politics into a manual on network programming, for instance. I was probably feeling gloomy when I posted that last comment.
That said, in the comments here I see a lot of people doing great systemic analysis of the problems of the current internet, which nonetheless (as a Marxist in the real sense of the word) fails to probe deep enough to identify the fact that the internet with these problems is the internet dominated by sites with a profit motive first and foremost.
I got here in the early 90s and hung out on irc and usenet, and I watched the internet become how it is. Way I see it, we had a set of community rules that worked fine as long as we were all people enjoying our spare time here. Once people came wanting to make money, things started going bad and they've only gotten worse and worse. We always had shitty trolls, but now you can actually get rich by being a shitty troll, and that is a much worse situation.
But I think change is coming nonetheless at this point - back in the 90s there was still a lot of middle class affluence here in North America, and GenX was either doing well, or else believing the hard work mythology, the "hustle culture" thing, such that along with the collapse of communism (which was never my jam, though I have tankies in my family) the 90s and aughts were basically Capitalism's victory laps.
But these days American streets are not safe, my own quiet Canadian prairie hometown has shantytowns that spring up all over the place and get cleared out by the cops only to reappear down the street. We call stabbings "The Winnipeg Handshake" now. I grew up there, it was not like this even twenty years ago, let alone when I was a kid and rode my bike all over town, even through the bad neighbourhoods, and rarely got hassled by anyone.
Basically they've got this unsustainable economic system that requires constant steroid shots to keep going on the one hand, which they are simultaneously claiming to be the strongest and best system we can have, and you know how it goes with steroids - at first you get big and strong, but pretty soon you're randomly breaking things in public fits of rage, and eventually your heart explodes and you die.
Capitalism is in its "public fits of rage" stage of its abuse, and the surveillance capitalism internet is reflecting the weakness in that their endlessly gameable algorithms have snuffed out the old internet and simultaneously been seized by bad actors, which is also a good description of politics around these parts (we have fascists up here too, it's not as bad as it is for Americans but we've got our Trump wannabe taking over the Conservative party as I type this).
My remedies would be socialist in nature, and seem like complete nonstarters to me in the current sociopolitical environment: The Internet was conceived by governments, military and universities as a public good. My first Internet access was through my local university, not as a student, just as a public yahoo using their (not actually) paid service - in theory I owe them $1/hr for thousands and thousands of hours, but they never attempted to collect a single one of them, from me or anyone else. The old internet was not about money, and money coming to the internet is what has caused the near-complete societal collapse we are currently facing in just a few years, because the internet is indeed an amazing way to connect lots of people together and democratize things.
So, kick the capitalists off the Internet, I say. Nationalize the fibre system and bring it to everyone free of charge, it's just a bunch of wires and switches in the final analysis. Hire public service workers who do things like build and moderate a site that does the same (good) purposes of facebook (Mastadon, for instance - we already have the solutions available, we just need server capacity and code maintenance)... but with no algorithmic manipulation of users' feeds. Show them what they subscribe to in chronological order. Privacy vis-a-vis government is not an issue when you're talking about public posts, as I see it. Social media is what you do for everyone, government included, to see.
At this point, I've probably lost most of the forum. Pay for free internet with my taxes? I mean, we're paying to drone attack weddings right now, is cheap, free internet really so obscene in comparison to the US military budget?
I'm also not saying that removing the algorithmic leeches from our social media lives is going to solve everything, but it will certainly prevent another Cambridge Analytica. I have no idea how big a problem bots are currently, but I'm positive they're gonna be a much bigger problem in years to come, with Machine Learning well in play. As long as people with profits to worry about are deciding what happens on the internet, we're screwed, man.
The internet, like medicine, needs the profit motive completely outlawed. That's the bit that I'm pretty sure is a non-seller to a lot of the folks around here. But that's ok, like I said, I'm here as one part of my attempt to not live in an echo chamber. Mostly I just try to read and soak up what gets said, because I am aware of my bias. But I do love the internet, and even before the internet I was a BBS kid, I have used modems to socialize basically my entire life, and this place is just no fun at all, anymore.
I just get grumpy sometimes, I guess, when I watch people get 99% of the way there, but won't take the name of Capital in vain. Profit must not be interfered with. This is the Ferengi timeline.
“Profit” isn’t always measured in money. Influence and control are also profit, even if do not increase your bank account size or improve your geopolitical strategic position.
I dunno man. Like I said, back then, we were all just occupying our spare time on MUDs and whatnot. Pretty sure everyone I interacted with could pass a turing test, back then. Bots were things we used to maintain our irc channels and get ops from.
Cheers. I actually think that we all agree a lot more than not on most things, but the algorithms make line go up when we're fighting. Line must go up.
1. You are the product, not the customer. The customer is advertisers.
2. What you see is what advertisers want, not what you want.
3. All ad-supported media converges on this no matter where it started.
No, it is not just you, the quality of Google results has gotten dramatically worse. BTW, you can ask Google 'is Google in decline' and see for yourself (DISCLAIMER: my answer on Quora is highly ranked where you can go into more details).
In short, Internet as we have known it has fallen apart because there are hardly any true organic links left. They used to be the lifeblood of the Web, the links based SOLELY on the relevance and quality of the linked page. Such a notion is quaint these days.
The Web used to grow exponentially in the early days, from say 1994-2008. Google launched with 100M webpages in their index, then they grew it to 1B in 2000 but they were NOT the first to reach that milestone, they were beaten to it by a Norwegian search company FAST that launched alltheweb.com with 1B before Google in 2000.
FAST was acquired by Yahoo in 2003 and you can guess what happened to them. For a few years Google and Yahoo were playing off against each other in terms of the size of their indexes, with Google always in the lead. But around 2003 the game of numbers stopped as they both announced they would stop publishing numbers. The Web continued to grow, still basically exponentially, and the next big milestone was announcement of Cuil in 2008. Cuil was a competing search engine created by a top team of Anna Patterson and Tom Costello, also with Luis Monier from Altavista. Their claim was that they would launch with an index of 120B pages, with an index bigger than Google.
That was widely considered an outrageous claim, as the notion that Google knows practically everything was already firmly entrenched. But they did manage to stir things up a bit, to the point of Google issuing a vague release in their official blog claiming they knew about 1 trillion urls. Of course they did not mention anything about indexing all that but the damage was done.
Shortly upon launch, the quality of Cuil results turned out to be far worse than expected which is really a shame as their basic premises were spot on, apart from index quality. Cuil then promptly fizzled out.
Note that projecting exponential growth (doubling every 18 mo i.e. quadrupling every three years) since 2008 we would expect 4^4*120B or more than 3 quadrillion(!) for the size of the Web index, with Google knowing 8 times that.
Such an expectation is plain silly, especially having simple queries such as e.g. 'Novak Djokovic', or 'Roger Federer' on Google returning less than 100 results.
But all this is only a (smaller) part of the story. Indexing is now a LEGACY technology, more than 20 years old. Users expect much more than returning bunch of blue links with matched keywords in response to their queries. They want much of the time direct answers to their questions.
The technology to do it has been known for 10 years now, in terms of dense vectors also known as word, sentence and other types of embeddings. Direct answers would be then found by nearest neighbor search. The scale of the system would of course have to be in the billions. BTW, it is a very interesting open question how many direct answers Google can provide now, in terms of infoboxes/featured snippets. Google has been coy about the issue but in my professional opinion, as a founder of multiple search engines, the answer is no more than around 20B. Feel free to shed more light on the subject and challenge this number.
In summary, the time has come to have a system based on vectors and nearest neighbor search with billions of vectors, giving direct answers to queries, with no ads nor tracking,and hopefully with API too.
One more DISCLAIMER, such systems are online for all to try and play with, at https://qaagi.com (for causal queries about causes and effects of things with billions of ranked answers) and https://yottaanswers.com (for factoid and and general questions what/how/where etc. with billions of answers). Both of the projects are led and principally funded by me, Borislav Agapiev.
> Users expect much more than returning bunch of blue links with matched keywords in response to their queries. They want much of the time direct answers to their questions.
Well, yes (and Google does that remarkably well sometimes) ... but also no. I often just want good results, damn it.
I do suspect that this type of technology would be helpful in judging whether a combination of words might actually be relevant to the query rather than just containing some of the same words - so the underlying challenge might end up being the same.
Yes, there are cases users want links and there is no problem displaying them in the results with direct answers e.g as sources. It is really about the change of focus, to providing direct answers, in both short and longer form, as opposed to just the snippet. Snippets have been around from the very beginning and they served us and Google well but it is high time to move on from line and a half with bunch of dots in between words.
Yes, it's getting worse, but somehow no one can beat their very low bar. I always start with Brave or DDG, and a high percentage of the time I need to resort to `!g`.
Recently, I couldn't remember the domain name for the web game, https://plurality.fun. Searching "plurality game" in DDG, Brave, and even Bing, the results are quite bad¹. However, the site I wanted was the #1 result on Google.
I've been using alternative search engines like DDG since ~2015, and I'm quite disappointed at the lack of innovation so far. It feels like a $20 bill on the sidewalk, so I assume I'm missing some reason why they all suck, but for the life of me I don't have an explanation.
¹To be fair, the site I wanted is pretty crappy, and not necessarily what everyone wants with those search terms. Nevertheless, it was #14 on Brave's search, where 3 of the top 5 results are links to the same academic paper, "When are plurality rule voting games dominance-solvable?" on three separate domains.
I get amazing results while programming if I use DuckDuckGo.
I think google dorking make the suggestions better. Also, using bangs on duckduckgo has vastly improved my search results. If you know where to find stuff, you will get good results. For example if I am looking for review or recommendations, I use site:"reddit.com" .
For easy queries DDG seems to be better at showing the primary source. For more complex stuff I still find Google much better - often enough the right result, which is ranked highly on Google, can't be found at all with DDG.
I don't think Google Search got worse, as in, I don't think changes they made directly made search a worse experience than not adding the change. I think that spam posters, scammers, et al., have gotten much better at gaming SEO, that Google got stuck with two bad options: (1) do nothing and have all common search terms loaded with spam, scams, and viruses, or (2) change how their search works to lessen the amount of bad actors, but also reduce the good search outcomes showing up.
Maybe I'm wrong. This is just a hunch based on looking at how big online communities fail, and how services that rely on user generated content fail. At a certain scale, you can no longer rely on people being good actors with light moderation. You need to tighten up moderation, even if it means making the community overall worse than it's peak. As, the alternative is letting the bad actors make it even worse than that.
More and more lately I've been coming across spam, reposted content from sites like StackOverflow, w3schools, Reddit, etc., but posted on different domains, on a page plastered with ads. To me, this is a peak into the arms race between Google's Search trying to return good results, and bad actors trying to get their spam to the front results.
They want to make as much money as possible so they can go into space, or fund their extropian fantasies, and can afford to pay lots of unimaginative people like lawyers, marketing and bean-counting people lots of money to achieve it. Advertising is the crutch.
Meanwhile I have the same question about DDG. Lately I've had to explicitly use google to get sensible results for direct by-name queries that ddg doesn't show even after 20 pages.
I'm talking innocuous stuff not political or porn or warez etc.
I know the domain name of a vintage computer wiki site. I search the name by name, I don't even need a search engine to find me the name, I did the main work for them already and showed up already knowing a domain name, and yet even so, all I get are 20 pages of links to mail list posts archived by narkive, and never the actual site. Google, of course, first link and most links on the first and all other pages of results.
It's bizarre and makes you wonder about all those unknown unknowns.
Just use more than one search engine. SearX mix results from multiple search engines. Kagi.com (now payed) has "lenses" features that do similar things.
Or just search on other search providers like you.com, qwant.com, startpage.com or DDG.
What happened? Did they let their golden goose starve to death? Or is this all in my head and Google is actually fine for everyone else?