As someone who hasn't used usenet for downloading content, why is it that it seems that a few central indexing services are the way people access this medium? Wouldn't it be possible - and preferable - to distribute the index through p2p? That way, there would be no single point of failure.
I'll not jump ship until my downloads are frequently disrupted at the block level, across multiple providers. Pretty difficult to achieve, given the architecture.
Indices come and go... Much of the Usenet crowd know how to set up an indexer if need be. Perhaps a hidden service is warranted?
What if it was only a directory of torrents? Not hosting the content itself, but indexing it.
(I've never developed anything with blockchain or torrents, so my apologies if there's a technical reason why this sucks.)
If I understand Pirate Bay and other torrent sites correctly, they don't host any content. They just index what's available P2P.
I'm not sure how large that index would become, but I could imagine chunking the index itself out so that it doesn't take much room on any one user's machine.
The chunking of the index could theoretically be done in such a way that no individual chunk would have any useable information, so no individual user could be accused of linking to illegal content. (I think that's conceivable? Again, not sure.)
(Please note: This is pure hypotheticals. I have zero interest in seeing such a system come about.)
On a somewhat related note: are there good options in terms of P2P/distributed torrent search engines, which would make it so that websites like the Pirate Bay don't even need to exist? Instead, that distributed index of torrents could be searched directly from the torrent clients.
The only alternative would be some kind of distributed torrent index. But would need to work in a browser to be accessible to people. So, unless browsers add first class support for this, this means a web site with ads, probably and then you have the same problems again.
Indeed. But nothing nothing stops them from storing torrents of torrents. The search index would be just that, the text index. Which would point to another torrent storing the actual content. Assets would point to yet other torrents.
Would be interesting to learn how it's currently partitioned. I would mimic the same portioning system but use torrent instead so users can help with hosting. And use sqltorrent to serve queries efficiently.
Sounds like setting up a torrenting system is in my future, since my beloved newsgroups are dying a slow death.
The big killer for newsgroups is indexers. Posts of content to the newsgroups have to be obsfucated to avoid near instant DMCA takedowns, which means that only a handful of indexers are given the 'keys' by the pirate release teams. That centralization makes the indexers a big target, and prevents just starting new ones quickly. To add to that, running an indexer still requires fairly decent hardware, especially with all the automated (*arr) tools that hammer APIs for searching. The ones that are left charge pretty high prices for access, especially considering they could (and have) just shut down without warning after you've paid some large yearly/lifetime fee.
I didn't rule out private trackers, I just failed to mention it.
In any case, I personally find Usenet superior to private trackers but I can see it from the other side. They both have pros and cons.
My main gripe with private trackers is that it's not as fast and I have to constantly worry about my ratio and seed. Granted, I pay for Usenet so it's a trade-off.
It bugs me that it has to be in this stupid base64 encoded format, which is wasteful. Also, the binaries have to be cut up chunks of a certain size; it's all rather hacky. Instead of providing Usenet servers it would be better to offer something like bittorrent caches.
I'm continually surprised that Bittorrent is still relying on content indexing websites. The edonkey community solved this problem years ago with fully distributed search that only relies on the peers themselves[0].
Can anyone with insight into the BT comunity/technology offer any ideas why they never moved in this direction?
Or: what would it take for everyone to have part of a copy of the index, according to their capacity / needs / etc., and for them to be able to easily search the parts of the index they don't have?
To pre-empt the "you've just described BitTorrent" comment - only in the vaguest sense; you'd need search functionality on the chunks themselves, and ideally you wouldn't have to copy (or even stream) your peers' search index chunks to search them.
I guess there's a trust / security issue here around "search index poisoning"; to resolve that, you'd probably have to lean on SSL verification and all of its attendant infrastructure for now.
Something like Freenet would be better for this I think, since the "what" you share is limited by how much space you give the datastore, unlike bittorrent where you consciously decide what and what not to keep seeding.
It still has a problem with non-popular content being hard to find, but not nearly as much.
In contrast to the other responses, I actually think this is not a bad idea. Doesn't it exist already in the form of private torrent trackers? I do not pay for usenet, but it sounds similar.
So why not share all the downloaded content through a p2p torrent network, rather than to keep them in our own servers to ourselves, so that everyone else benefits?
I keep wondering all the time why torrent search is based on websites (centralized), which can be taken down, etc., while once you have a torrent file or a magnet/hash everything is distributed.
Is there a main reason why there isn't (AFAIK, even though I haven't really researched) a distributed search that wouldn't have these problems? Is it a tech problem that literally can't be solved? Or it just hasn't been done? It seems like search is the obvious weak link, since the websites keep disappearing or taken down or blocked by governments and ISPs, etc.
A simple solution that works now tends to beat a complex solution still on the drawing board. I'd say go ahead with the torrents plus distributed off-line index.
So Bittorrent is decentralized but you need a centralized index of the available torrents? Is this correct? Can you find anything without a site like KAT? TIA.
I'm advocating it as the content distribution layer, not the social interaction layer. You don't even need all of the metadata, as there are BEP extensions to distribute that as an index [0] or an RSS-style feed [1]. Images are particularly painful, like the previews in Mastodon, because they hammer the origin server rather than looking around at what other peers have. This problem will only get worse as the federation grows in size, both in users and in servers.
I say irony because we had all these scaling problems two decades ago during the peak P2P years, including federation and implicit DDoSing, and BitTorrent emerged as the lasting protocol because it could handle the scaling issues.
I say tragedy because for this to work you do need some openness, which puts people at risk of content companies trolling for you, ISPs dropping you, and is completely antithetical to the control Silicon Valley demands over its users and data.
reply