Signals and Threads

This had been my impression of PTP too, and I think in many circumstances it's true, but I recently listened to this episode from Jane Street's Signals and Threads podcast that gave some interesting explanation of why they chose to base their synchronization system on NTP: https://signalsandthreads.com/clock-synchronization/

Preview: "Yeah. I think that’s roughly the conclusion I came to, that that’s what makes PTP more accurate than NTP, which was surprising to me. And then I did a bunch of research and was talking to various people in the industry, and at various conferences and stuff, and there was some agreement that you can make NTP also very accurate you just have to control some of these things, so there are… in addition to being able to do hardware timestamping with PTP packets some cards, these days, support the ability to hardware timestamp all the packets, and if your machine is just acting as an NTP server and most of the packets it receives are NTP packets, well then you’re effectively timestamping NTP packets. Some cards also will timestamp just NTP packets. They can sort of recognize them and timestamp only those, but it was sort of like “Okay if we have the right hardware, we can get the timestamping bit of it. That’s kind of an interesting thing. With the different NTPD implementation, chrony being the other implementation I’m talking about as opposed to the reference one, you can turn that knob for how frequently you should poll your server, I think as much as like 16 times a second. There’s a bit of like diminishing returns there, it’s not always better to go lower… point being, you can tune it to at least match sort of what PTP’s default of once a second.

And the more I dug, and the more I talked to people, the more people told me, “Hey, you definitely do not want to involve your switches in your time distribution. If you can figure out a way to leave them out of it, you should do so.” I was happy to hear that in some ways, because right now the reliability. or the sort of, the responsibility of the time distribution kind of lies with one group, and that’s fine. When you then have this responsibility shared across multiple groups, right, it becomes a lot more complicated. Every switch upgrade, suddenly, you’re concerned. “Well, Is it possible that this new version of the firmware you’re putting on that version of that particular switch has a bug related to this PTP stuff and is causing problems?”

Given all of that, I started to believe that it was possible that we could solve this problem of getting within 100 microseconds using NTP and I sort of set out to try and see if I could actually do that."

People who aren't PL theorists understand the multicore benefits even if the retrofitting paper might be over their heads. On the other hand, the bounding data races in time and space paper/presentation is also a fairly significant change that I don't think the average dev has an easy to understand relationship with.

Anil covers it in a bit more plain English in this Signals and Threads episode. Here's a bit of the transcript, starting at about 50 minutes in: https://signalsandthreads.com/what-is-an-operating-system/

> Ron: Do you have a pithy example of a pitfall in multicore Java that doesn’t exist in multicore OCaml?

> Anil: There’s something called a data race. And when you have a data race, this means that two threads of parallel execution are accessing the same memory at the same time. At this point, the program has to decide what the semantics are. In C++, for example, when you have a data race, it results in undefined behavior for the rest of the program, the program can do anything. Conventionally, daemons could fly out of your nose is an example of just what the compiler can do.

> In Java, you can have data races that are bounded in time so the fact that you change a value can mean later on in execution, because of the workings of the JVM, you can then have some kind of undefined behavior. It’s very hard to debug because it is happening temporally across executions of multiple threads.

> In OCaml, we guarantee that the program is consistent and sequentially consistent between data races. It’s hard to explain any more without showing you fragments of code. But conceptually, if there’s a data race in OCaml code, it will not spread in either space or time. In C++, if there’s a data race, it’ll spread to the rest of the codebase. In Java, if there’s a data race, it’ll spread through potentially multiple executions of that bit of code in the future.

> In OCaml, none of those things happen. The data race happens, some consequence exists in that particular part of the code but it doesn’t spread through the program. So if you’re debugging it, you can spot your data race because it happens in a very constrained part of the application and that modularity is obviously essential for any kind of semantic reasoning about the program because you can’t be looking in your logging library for undefined behavior when you’re working on a trading strategy or something else. It’s got to be in your face, at the point.

(and so on)

> … [T]he notion of this sort of finer level of control is definitely accurate. But Rust gives up something really big to do what they do. They give up garbage collection. Within Jane Street and generally in programming, I think a lot of the time garbage collection is fantastic, right? It allows us to program without worrying about these low-level details that for most programs, we don’t need to.

> So, for instance, when I’m writing the OCaml compiler—which is written in OCaml, of course—I do want it to be performant, but I don’t really care about microsecond latency. And so, I’m happy to write my compiler in a way that allocates a bunch of memory and then when it’s done with that memory, the garbage collector comes and gets rid of it. This is really, really convenient. In Rust, we don’t have that. In Rust, we are forced to think about memory, allocation, and deallocation at every point in time. And this gives us this fine level of control, but it comes at a real cost, and it means now we’re sort of fighting against this ‘pay-as-you-go principle’ that we were talking about earlier. In Rust, for memory, you don’t pay as you go. Everyone has to pay all the time for Rust fine memory control. So we wanna keep the high-level, garbage-collected OCaml for most applications, and then when we really want to have this fine level of control, we wanna have that, too.

> And so, some of the work that we’re doing is around adding that finer level of control, but just where we need it.

Link to 25:34: https://signalsandthreads.com/future-of-programming/#2534

There are also prefetch instructions. I listened to https://www.youtube.com/watch?v=SetLtBH43_U recently (transcript: https://signalsandthreads.com/memory-management/), part of it talked about some work in OCaml's GC.

> ...each individual memory request that’s not in cache still has to wait the full 300 cycles. But if we can get 10 of them going at the same time, then we can be hitting a new cache line every 30 cycles. I mean, that’s not as good as getting one every 16 cycles, but it’s close. You’re actually able to get to a reasonable fraction of the raw memory bandwidth of the machine while just traversing randomly over this huge one gigabyte heap

https://github.com/ocaml/ocaml/pull/10195 shows the change adding prefetching to the marking phase (https://github.com/ocaml/ocaml/pull/9934 was done earlier for sweeping). There are some benchmarks in the thread/linked from the thread.

Jane Street has been publishing some interesting projects recently. See Signals and Threads podcast State Machine Replication, and Why You Should Care [0] posted 2 days ago.

I came across the incr_dom library [1], which efficiently calculates the diff on the projected DOM based on a diff of the model data, sorta like React but more... mathematically grounded (?). incr_dom was then reformulated in Bonsai [2] which refactors and generalizes the idea to work with more than DOM. There was a recent Signals and Threads podcast about it a few months ago [3].

[0]: https://news.ycombinator.com/item?id=31100023

[1]: https://opensource.janestreet.com/incr_dom/

[2]: https://opensource.janestreet.com/bonsai/

[3]: https://signalsandthreads.com/building-a-ui-framework/

I found Headland's blog post about quant trading useful [0] and as well as their other blog posts [1].

Jane Street's podcast and blog are insightful too [2]. I read somewhere that Dr. Simons (one of the founds of RenTech) has this book in his office [3].

0. https://blog.headlandstech.com/2017/08/03/quantitative-tradi...

1. https://blog.headlandstech.com/

2. https://blog.janestreet.com/ and https://signalsandthreads.com/

3. The Econometrics of Financial Markets, https://www.amazon.com/gp/product/0691043019/