As someone who used to do a decent amount of web and search engine results scraping in a past life, one thing I’ve never understood about DDG is how on earth this setup is economical. If they’re hitting (presumably) bing’s api a few billion times a month, surely they’re paying a lot of cash to do that, even at a heavily negotiated rate.
Also, so many folks here are saying they use the bang option to search Google, through DDG. So either they’re using google’s search api or scraping on the background, both of which are expensive in their own ways.
Beyond technical, at a business level, how are they actually pulling this off?
I’ve always wondered how the economics of DDG work out, considering they’re basically a proxy for Bing with a bunch of other scraped/API-based data sources they layer on top. Are they paying Bing per query? How can that net out positively?
I suppose at their volume they’ve negotiated great API rates, but their own ads can’t possibly bring in enough revenue to cover the expenses of the API calls…
DDG also just uses the bing search API for all their results seemingly. They claim to use other sources as well, but in the tests I conducted building Kagi last year, it was always identical to bing. They must have a special deal with microsoft because the API fees at that volume are completely unsustainable, as in the operating costs are 5-10x the theoretical profits from advertising. If they were actually scraping, this wouldn't be a thing: https://help.duckduckgo.com/duckduckgo-help-pages/company/ad...
Does anyone know how ddg and co. pay for that? Do they get special rates? Because the normal price for using the Search API of Google or bing seems too expensive to be viable.
"DuckDuckGo, which aggregates information obtained from 400 other non-Google sources, including its own modest crawler.)"
I looked into it, and it seems DDG is using Bing and Yahoo search API and lots of other sources.
I looked into the pricing of Yahoo's search API / Bing search API,it ranges from $0.80 / 1000 queries to several dollars per thousand queries.
It seems to expensive to be economically viable with ads, what am I missing ?
I mean the !g bang - DDG has the option of getting results from Google and Bing. Bing offers an API for $6/10000 queries, but Google does not - I'm curious how they have access to Google search results.
DDG is Bing, they use its API to get the search results. They augment it with other sources to provide the "value added" part, but that's a tiny part. DDG doesn't want you to know that it is Bing, but Bing is what it is.
Likely to be monetary reasons. DDG is basically (not entirely accurate but close enough) in the business of reselling Bing Search API and monetizing it through Bing ad network revenue (+some affiliate revenue through Amazon and Ebay).
If you also added Google results, your input costs essentially double. Also if so much of your marketing is based on bashing Google, it would be harder to justify such move from a branding perspective.
You're trivializing search. DDG only works because for the hard stuff it's grandfathered into an old Bing API. DDG is small and scrappy but it's built on the shoulders of a giant investing billions of dollars.
I cannot explain it, hopefully someone from DDG actually chimes in :-) If they just use the Bing API, how can their results be better than Bing? I suppose they also use other sources.
btw it is literally just bing. The only reason they can afford it is bing gives them free API access. Would be impossible for DDG to afford the usual API fees without something like a $5/month subscription for normal usage. If they didn't give it away for free to DDG, bing itself would lose double digit usage numbers, since they report DDG usage as their own usage. Source: worked for a competitor.
My interpretation of this is they pay for API access and sends anonymised queries to Google and/or Bing.
I'm still puzzled how they get so extremely much better results compared to using any of thise directly or using DDG which does the same only with Bing.
Also a bit worried because I think there was a time when DDG felt the same before they suddenly gave in and started serving answers for queries I didn't ask just like Google but I'm not really sure.
Can you provide source of the claim that DDG search is 'outsourcing the search infrastructure Bing'? This appears to be a regular claim, but as far as I know, DDG buys search data from multiple parties, including Bing, Yandex, and many others. They have their own web crawler as well.
What efforts has DDG actually taken to build a decentralized search index or anything to that degree? Do they make profit, and is their business model literally just acting as a proxy for Bing and what not?
As far as I know, DDG uses both data from other search engines (Google, Bing) as well as its own crawling bots. If they only used their own crawlers, with the current size of the internet it would take a long time for them to get a good amount of data.
Isn't that just mostly using major search engine indexes? I know DDG some of its own indexing, but how much do they really do on their own? I don't see the benefit of getting DDG for them when they'll have to deal with the major search engines providing most of the data anyway.
Also, so many folks here are saying they use the bang option to search Google, through DDG. So either they’re using google’s search api or scraping on the background, both of which are expensive in their own ways.
Beyond technical, at a business level, how are they actually pulling this off?
reply