I keep wondering all the time why torrent search is based on websites (centralized), which can be taken down, etc., while once you have a torrent file or a magnet/hash everything is distributed.
Is there a main reason why there isn't (AFAIK, even though I haven't really researched) a distributed search that wouldn't have these problems? Is it a tech problem that literally can't be solved? Or it just hasn't been done? It seems like search is the obvious weak link, since the websites keep disappearing or taken down or blocked by governments and ISPs, etc.
On a somewhat related note: are there good options in terms of P2P/distributed torrent search engines, which would make it so that websites like the Pirate Bay don't even need to exist? Instead, that distributed index of torrents could be searched directly from the torrent clients.
It doesn't (strictly) need to have centralized search; you can passively collect info from the DHT swarm and build up an index over time of torrents people are sharing.
Some of this stuff, like peer exchange and magnet links, are not new and are in most torrent clients for years. Pirate Bay has started promoting magnet links and rumour is that they'll stop doing torrents all together.
It's the search part that's hard to do, and I wonder how tribler do it? Anyone analysed their code? I wonder how you'd do a true decentralised p2p search system.... (i.e. do to p2p search what google did to web search, come up with a new algorithmic invention)
Distributed discovery is possible, but you quickly run into spam/noise problems. Centralized communities generally have features such as curation and deduplication (duplication is bad for torrents, as it causes peers to be spread across multiple slightly-different packages of files, reducing the available peers for both).
The only alternative would be some kind of distributed torrent index. But would need to work in a browser to be accessible to people. So, unless browsers add first class support for this, this means a web site with ads, probably and then you have the same problems again.
Unfortunately BitTorrent is still heavily centralized when it comes to discoverability. Without a website hosting .torrent files or magnet links, it's very limited.
There's not really a coherent distributed system for the torrent files, but there's not really a need, the ad-hoc system works pretty well.
The real problem is tracking the active clients -- without widespread DHT trackers like TPB would have collapsed under their own weight years ago. As it is there's frequent timeouts and massive response latency. It doesn't help that the software running the trackers is almost universally awful PHP garbage.
Torrent links are centralized in trackers and you'd see a single tracker site dominate if the legal atmosphere allowed for it.
And honestly, it doesn't work as well. If you don't care about decentralization for decentralization sake its not ideal. Streaming websites don't use it. Netflix doesn't torrent a video to you.
If you are thinking about decentralized search, it already exists in some P2P networks (ex: Kademlia)
However, it is nowhere as effective as commercial services. So much that the most effective way of finding pirate stuff is often to Google them despite all the copyright takedowns.
edonkey has had distributed search for a long time. It's possible to maintain an keyword index of magnet links in a dht, and then you remove the need for the torrent site completely.
I think that requiring torrent files and trackers was a policy decision to deflect liability away from the client implementer to multiple third parties. That's why bit torrent is still around and Grokster isn't. There's no technical need for them.
Decentralized search is a complex problem and there exist different approaches to it with different degree of centralization and resource requirements.
DC++ is a bit more decentralized than BitTorrent. There still are central servers ("hubs"), but they don't even host any metadata. Search works by the hub broadcasting all search queries to all online peers and them replying with results if they have any. The file transfers themselves are p2p.
I have an idea that's kind of more decentralized. Initially envisioned as a missing global search feature for the fediverse, but can be adapter for anything that has a similar network structure. A server has a number of peers already established because of the ActivityPub federation. Each server would send to its peers some kind of bloom filter that determines the tags or keywords that this server has results for. Then, when searching, your server would find the peers who are likely to have what you want, and only send your search query to them. If there aren't any, then it would send your query to the peers that have most users (with some random bias for load balancing purposes) because they're likely to have more connectivity, and they would point you where to look based on their own peers and their bloom filters. There would also need to be some kind of reputation system (centralized server lists? p2p exchange of scores/reports?) so that servers that return spam or intentionally wrong results would get punished.
This could probably be made to work in a fully-decentralized p2p network, but I imagine it would be too easy to abuse. Getting a new domain costs money, yet getting a new IP or public key is free and easy.
I honestly think that the idea of letting people search inside a torrent before fully downloading it has been overlooked and might be the next big step in P2P file-sharing: https://github.com/lmatteis/torrent-net
The idea is that you can actually query stuff inside a torrent; like actually perform an SQL query and only the pieces relevant to the query will be downloaded.
This sort of allows for distributed querying if you think about it, since your query could be satisfied by many different peers that are seeding/leeching.
Imagine a Web built this way. Where sites are served by people that visit them and are not just static sites, but fully queryable as you'd expect in the normal HTTP web.
Is there a main reason why there isn't (AFAIK, even though I haven't really researched) a distributed search that wouldn't have these problems? Is it a tech problem that literally can't be solved? Or it just hasn't been done? It seems like search is the obvious weak link, since the websites keep disappearing or taken down or blocked by governments and ISPs, etc.
reply