libtorrent Blog

The .torrent file is separate to the torrent metadata which you need to fetch for magnet links. This is mentioned a bit later in that section:

> The .torrent file size is not smaller for a v2 torrent, since it still contains the piece hashes, but the info-dictionary is, which is the part needed for magnet links to start downloading.

Also it seems that it'll be easier to detect peers which are sending bad data and correct it. I guess this is because previously you could fetch a piece from more than one peer but because the hashing was done at the piece level you didn't know which part of the block was wrong (meaning you'd have to redownload the whole thing). But now you can -- with a bit of tracking -- tell which block of a piece was sent from which peer and then redownload it from another (likely blocking the original peer), because you know which block hash failed to validate. (EDIT: This is correct -- [1] explains the issue and how torrent clients have worked around this problem in v1 torrents.)

Merkle trees have some other benefits (you can cache parts of the tree in such a way that repeatedly checksumming data where lots of leaves are unchanged is cheaper, and you can efficiently prove that a given leaf was part of the hash), but I don't know if that's going to be useful for BitTorrent.

[1]: https://blog.libtorrent.org/2011/11/smart-ban/

>I did this for two reasons, the first being that I cannot, for the life of me, figure out how to use asyncio to do local file IO and not a network request, but maybe I'm just an idiot.

I don't think you are an idiot I just think you didn't search well enough. There is a reason there is no local file IO with asyncio.

Check these links for more info:

* https://stackoverflow.com/questions/87892/what-is-the-status-of-posix-asynchronous-i-o-aio

* http://blog.libtorrent.org/2012/10/asynchronous-disk-io/

From what I understand the way libuv (what node.js uses) gets around OS limits is with a thread pool.

Also, this documentation might be helpful:

* https://docs.python.org/3.5/library/asyncio-dev.html#handle-blocking-functions-correctly

My experiences don't seem to mirror your frustration. I found asyncio quite useful for tinkering with a simple web scraper that hit multiple sites at once (each with a different response time) and munged all the data together into one data set.

Thanks for the write-up!

Is there a detailed overview about which functions in libuv (Nodejs) rely on blocking primitives and thus are using the thread pool to work async?

From [here](http://docs.libuv.org/en/latest/design.html), it sounds like all file IO is always based on blocking primitives, and native async file IO primitives are not used, although such async file IO primitives do exists and were tried out in libtorrent (http://blog.libtorrent.org/2012/10/asynchronous-disk-io/). The result of that experiment however was mostly that the thread pool solution was simpler to code (I guess).

From the libuv design doc, the overview is:

* Filesystem operations

* DNS functions (getaddrinfo and getnameinfo)

* User specified code via uv_queue_work()

I wonder whether this is really the best solution of if some combination of a thread pool and native async disk IO primitives could perform better.

No, I don't have a demonstration since nothing like this was ever widely adopted, but it is an opinion shared among the bittorrent community:
"One of the key algorithms in bittorrent is the rarest-first piece picker. It is vital to bittorrent’s performance that the piece picker fulfills both of these requirements:

The rarest piece is picked (from the client’s point of view of the swarm)
If two or more pieces have the same rarity, pick one of them at random

The reason to pick a random rarest piece is to always strive towards evening out the piece distribution in the swarm. Having an even piece distribution improves peers’ ability to trade pieces and improves the swarm’s tolerance to peers leaving."
http://blog.libtorrent.org/2011/11/writing-a-fast-piece-picker/

BEP46 aka Dynamic Content Updates and Decentralization is, if I recall correctly, rolled into BEP-52 , aka The BitTorrent Protocol Specification v2 which has been the standard since 2020 (or so) and on paper since 2017 (ish).

( again, IIRC )

Question is, of course, how many release groups, public and private trackers are supporting this feature?

https://medium.com/@kyodo-tech/bittorrent-protocol-v2-and-dynamic-content-updates-ee2d8cbd05df

https://www.bittorrent.org/beps/bep_0046.html

https://www.bittorrent.org/beps/bep_0052.html

https://blog.libtorrent.org/2020/09/bittorrent-v2/

Node.js uses libuv for async file operations -- And libuv has its own internal thread pool to make file ops appear fully async to a node program.

From the libuv docs [0]:

Unlike network I/O, there are no platform-specific file I/O primitives libuv could rely on,
so the current approach is to run blocking file I/O operations in a thread pool.
For a thorough explanation of the cross-platform file I/O landscape, checkout this post [1].

[0] http://docs.libuv.org/en/v1.x/design.html#file-i-o

[1] https://blog.libtorrent.org/2012/10/asynchronous-disk-io/