erdgeist

Well at least for the near future it would probably work like how people have to find torrents: different sites index meta data in their own data bases.

Maybe with something like opentracker there could be an endpoint like /search that translates search phrases into a sparse column vector and looks up cosine similarity against that instances low dimensional space representation of the tf-idf against torrent hashes, those low dimensional space representation files can also be distributed in torrent file and as well as way to constantly update/compute new low dimensional space representation files as hashes are added.

For example it would probably look something like what I do in [0] but as a compiled module linked into and accessed from [1] as well constant update/rebuild of ldsr files like [2] linked and kicked off from flags passed in [3] as new hashes come in.

[0]: https://github.com/cinquemb/hackedteam-email-index-mining/blob/master/search_emails/search_emails.cpp

[1]: https://erdgeist.org/gitweb/opentracker/tree/ot_http.c

[2]: https://github.com/cinquemb/hackedteam-email-index-mining/blob/master/mine_emails.cpp

[3]: https://erdgeist.org/gitweb/opentracker/tree/opentracker.c

One part that really makes The Pirate Bay very efficient is their torrent tracker software. Unlike Python and PHP implementations, theirs is written in C, with very little dependencies. It's called OpenTracker — http://erdgeist.org/arts/software/opentracker/. It's intended to run on a WLAN router; it's that efficient. Especially see the "Philosophy" section :)

Of course, other projects can also use this for very legal purposes. E.g. my http://driverpacks.net project uses torrents for distributing (legal!) downloads to avoid expensive server bills. The infrastructure is simple: a central OpenTracker instance, plus a few seedboxes (±$15/month) that get new files via rsync (to guarantee each file always has seeders), and voila: terabytes per day of traffic at ±$1/day.
Live stats at http://driverpacks.net/stats — powered by http://drupal.org/project/opentracker.

Hopefully this is useful for somebody :)

Interesting story. Especially how often they tried to develop a new tracker. Any more insights why they failed so often?

Another already opensourced torrent tracker is opentracker http://erdgeist.org/arts/software/opentracker/. Its written in c and was developed because other trackers were slow or needed much resources. After some time piratebay started to use opentracker because it was much faster then the other trackers.

There was also a great talk[1] (German) at the 24C3 about the development of opentracker and some nice insider stories about running a big torrent tacker like getting a call from the provider "you are under a ddos attack.." "aeh no, thats normal and the load on the server is really low" (they had to change the hosting company ;0)

[1] http://dewy.fem.tu-ilmenau.de/CCC/24C3/mp4/24c3-2355-de-tracker_fahrn-COMPATIBLE.mp4

That you don't need to upload a torrent file somewhere in order for it to be made available by the tracker. That is, it's not running in the typical "whitelist mode", but instead, by simply announcing yourself and a torrent hash, that torrent is made available in the swarm for anyone to get in on.

This mode of operation is one of the fundamental modes of opentracker - http://erdgeist.org/arts/software/opentracker/ - which is the software pretty much all of the many open BitTorrent trackers run.

Thanks :). If I'll find the time, I'll have to dig them up.

Or maybe I'll attempt to contribute the important parts to an established project, like ezjail [0][1], which seems to be quite competent at handling jails, but is specialized on running services.

Refs:

[0]: http://erdgeist.org/arts/software/ezjail/

[1]: https://www-legacy.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails-ezjail.html