> they’ve demonstrated gross incompetence in privacy
Not sure I buy the example that is given here.
1. It's an issue in their browser app, not their search service.
2. It's not completely indefensible: it allows fetching favicons (potentially) much faster, since they're cached, and they promise that the favicon service is 100% anonymous anyway.
> The search results suck! The authoritative sources for anything I want to find are almost always buried beneath 2-5 results from content scrapers and blogspam. This is also true of other search engines like Google.
This part is kinda funny because "DuckDuckGo sucks, it's just as bad as Google" is ... not the sort of complaint you normally hear about an alternative search engine, nor does it really connect with any of the normal reasons people consider alternative search engines.
That said, I agree with this point. Both DDG and Google seem to be losing the spam war, from what I can tell. And the diagnosis is a good one too: the problem with modern search engines is that they're not opinionated / biased enough!
> Crucially, I would not have it crawling the entire web from the outset. Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Pages that these sites link to would be crawled as well, and given tier 2 status, recursively up to an arbitrary N tiers.
This is, obviously, very different from the modern search engine paradigm where domains are treated neutrally at the outset, and then they "learn" weights from how often they get linked and so on. (I'm not sure whether it's possible to make these opinionated decisions in an open source way, but it seems like obviously the right way to go for higher quality results.) Some kind of logic like "For Python programming queries, docs.python.org and then StackExchange are the tier 1 sources" seems to be the kind of hard-coded information that would vastly improve my experience trying to look things up on DuckDuckGo.
> This is also a somewhat misleading title since it's not about our search engine, which people would assume
I understand the controversy (or at least I think I do), and I also respect what you're trying to explain.
But if DuckDuckGo has to have a browser (instead of simply recommending to its users to change their default search engine on their current browser), then that browser needs to live up to its promises, and also to very high standards.
On the current page for the DDG browser [1] it says at the top:
> Seamlessly take control of your personal information, no matter where the Internet takes you.
If this means anything, it means the DDG browser blocks all trackers, always, in all places, no?
> Sure, but just lately DDG started deteriorating for me as well.
Might be an issue with one or more of their backends censoring certain phrases in a sporadic fashion. While they do have their own crawler,[1] I don't think it has a significant effect on the breadth or accuracy of their results.
> First, it is misleading to say our results just come from Bing. That's far from the case in actuality.
That's just corpo-speak. For the most part, Duckduckgo is Bing with some additional features. That's true to the extent that when Bing decided to censor the Tank Man image, it was removed from your results too. [1] Not that you guys refrain from censorship yourselves. [2]
The crawler (DuckDuckBot) doesn't have much of an impact on the search results, it's mainly used to provide instant answers. [3]
> Luckily, there is duckduckgo. However, I have to admit, the search results of duckduckgo are by far inferior to what Google used to give.
For cases where DuckDuckGo isn't quite enough, I usually rely on StartPage [1]. It uses results from Google, but like DuckDuckGo it doesn't track its users.
Startpage, like DuckDuckGo, also works well in Lynx (I've just tested in Lynx 2.8.9 on Ubuntu 16.04).
Additionally, you can get StartPage results right from within DuckDuckGo by just appending the !sp shortcut to your search query [2]
Edit: you may want to keep in mind, however, that StartPage is now owned by an advertising company. Some users took issue with that. Personally, I'm OK with that as long as users are not being tracked. Relevant HN thread: https://news.ycombinator.com/item?id=21371577
> Okay, but how do you know Duck Duck Go is privacy orientated?
My privacy results are not different than yours (assuming we have the same region set up). Ads that are shown are purely context-based. There's no account to which my search data is tied to. My device makes no third-party connections when opening the results page.
That's better than 95% of the web. Perfect is the enemy of good.
> Maybe because google have spent years compiling data about me, but I had to switch to google search yesterday to get a result that wasn't showing on DDG.
> 1) Wait, does DuckDuckGo use Bing’s search index?
They do what a company called metacrawler did back in the 90's (and I see still does now) but w/o the privacy angle and whatever other bells and whistles DDG adds. It also crawls but I don't know what percentage of that makes up the index. Generally I don't believe it's a very large part from memory.
> How do you save a search in a non-personally identifiable way?
Save a sha256 hash of every search for 24 hours. If you see the same hash from >10 distinct IP addresses in a 24 hour period, save the search terms.
That's just off the top of my head, I have no reason to think they're doing it exactly like that. The point is that you're claiming that we shouldn't trust DuckDuckGo because you can't think of a way that they could securely and privately do what they do -- but that's just your intuitions, for whatever they may be worth.
I also don't really buy the worries you have with the last two questions, e.g.:
> How do you verify that DuckDuckGo does not secretly cooperate with more powerful coercive actors?
How would you verify that for any centralized service, open source or not? I think your security concerns go a bit beyond what most people interested in critiquing / improving DDG can reasonably expect to achieve.
> If I want to prefetch results, I unerstand it means people know I'm looking at those results.
Why do that though when you can have your cake and eat it? Google's proposed solution allows you to prefetch results _and_ not allow the third-party to know you're looking at those results.
> the counter consideration is that your implicitly saying you trust Google with that info more than the actual service providers
If I'm using Google search, then yes, obviously I'm fine with Google knowing what I searched for. If I wasn't okay with that, I most certainly would not be using Google search. In contrast, I'm less likely to be okay with a random site in the search results page that I haven't clicked on knowing that I saw a link to their site in a search results page.
Note that if I were using DuckDuckGo instead and DuckDuckGo supported AMP, then my browser would prefetch from DuckDuckGo's AMP cache, not Google's. No additional information is being shared with any party who doesn't already possess that information. (DuckDuckGo already knows what I searched for. Me loading an AMP page from them related to that query reveals no additional information.)
> Indeed. That's the point of HTTPS. As a user I don't expect that contract undermined even by a claimed benevolent Google
Could you explain how Google's proposed solution here undermines HTTPS? Note that the OP talks about using the upcoming [Web Package standard][1] to distribute AMP pages. This standard would allow the integrity guarantees of HTTPS to be preserved even when the page in question is being served by Google's AMP cache rather than the original server.
An irritation for me with DDG is them formatting result URLs incorrectly, and then ignoring any feedback about it. Two examples are them adding spurious www. prefixes to Google code results, and leaving out slashes when creating Apple developer URLs.
DDG could distinguish themselves by playing bazaar to Google's cathedral, but they don't appear to. A search engine that uses crowd sourcing and feedback would be disruptive IMO.
The examples. https://duckduckgo.com/?q=apsw - note the second link is on google hosting and shows code.google.com/p/apsw/ but clicking on it gives page not found because it goes to WWW.code.google.com/p/apsw/. https://duckduckgo.com/?q=NSString and note infobox at top which goes to Apple developer but clicking gives an error. The link should have a slash between Reference and NSString at the end.
Google is highly influential in how information is consumed and distributed. People use chrome to browse web, google search for searching information etc. If they have major control over how information is distributed, they have major control over the web. Ofcourse there are other similar services that are not provided by Google, but they are very less influential due to lack of users.
> You should be scared of walled gardens displacing the web.
What if web becomes a walled garden ? Look at what Google did with DRM[1], even Mozilla had to give up their efforts to fight against it[2]. How could that be possible without having a major influence over the web ?
> But it's already set up for you in DDG, so why not use it?
Because if I wanna search Reddit, I'd prefer to have '!r' search directly within Reddit and not litter my browser history with tons of duckduckgo entries like this:
Are they? Google could manually remove the worst SEO sites from the index and like reduce the spam you get for dev queries with like 95%.
It takes time to build up rank and like 10s to remove the site.
There is an extension that does this client side for Firefox. Try it. (I don't remember the name. I only have it on desktop).
The only explaintions are that Google is too dysfunctional to do anything or they are doing it on purpose to increase revenue. Otherwise there is no way the SEO would suddenly become this bad.
> I don't understand how it's related to this particular issue though.
Because when it first started I think there was a mutual, non-spoken agreement between web content-owners and Google, sort of "I give you our content for free to use it on your search engine and you bring me back visitors by trying to be impartial in your SERPs; but I don't expect you to compete against me, not now, not ever".
That didn't last long, to say the least, but said content owners are now prisoners of Google (yes, it's a big word, but that's the fact). If my personal experience counts, I let Googlebot crawl my website like crazy, at speed rates for which I would certainly ban any other potential search engine bots, and that's because Google brings in 98% of my search engine traffic. And I'm sure I'm not the only one banning strange-looking bots, which could belong to potential new, wonderful search engines, but which at this rate will never see the light of day.
And, unfortunately, things like DuckDuckGo are sub-optimal outside of the English sphere of influence, at least that's what I could see in my case.
Not sure I buy the example that is given here.
1. It's an issue in their browser app, not their search service.
2. It's not completely indefensible: it allows fetching favicons (potentially) much faster, since they're cached, and they promise that the favicon service is 100% anonymous anyway.
3. They responded to user feedback and switched to fetching favicons locally, so this is no longer an issue. https://github.com/duckduckgo/Android/issues/527#issuecommen...
> The search results suck! The authoritative sources for anything I want to find are almost always buried beneath 2-5 results from content scrapers and blogspam. This is also true of other search engines like Google.
This part is kinda funny because "DuckDuckGo sucks, it's just as bad as Google" is ... not the sort of complaint you normally hear about an alternative search engine, nor does it really connect with any of the normal reasons people consider alternative search engines.
That said, I agree with this point. Both DDG and Google seem to be losing the spam war, from what I can tell. And the diagnosis is a good one too: the problem with modern search engines is that they're not opinionated / biased enough!
> Crucially, I would not have it crawling the entire web from the outset. Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Pages that these sites link to would be crawled as well, and given tier 2 status, recursively up to an arbitrary N tiers.
This is, obviously, very different from the modern search engine paradigm where domains are treated neutrally at the outset, and then they "learn" weights from how often they get linked and so on. (I'm not sure whether it's possible to make these opinionated decisions in an open source way, but it seems like obviously the right way to go for higher quality results.) Some kind of logic like "For Python programming queries, docs.python.org and then StackExchange are the tier 1 sources" seems to be the kind of hard-coded information that would vastly improve my experience trying to look things up on DuckDuckGo.
reply