The Pernosco team writes about their omniscient debugger for x86, which records program execution and builds a queryable database of everything that happened.
HN links it in debugging discussions as an alternative to rr's interface and as an example of omniscient debugging; the authors themselves describe how it works.
"Time travel debugging" seems to be taking off as the term - "reverse debugger" was around for a long time but doesn't seem to have taken root. It has fixed the conversation of "We make a reversible debugger", "Ah so you put the bugs in", though!
> rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines.
Hopefully we'll see cloud environments find a way to expose more of the hardware performance counter stuff by default. The alternative is to use an instrumentation-based system that doesn't need access to performance counters. That said, as with machine-level virtualisation, it'd be nice to see the CPU vendors provide more explicit hardware support over time to make implementation easier / higher performance than current approaches.
> Valuable, but not as valuable as some other options. Plus, using GDB is. Ugh.
Looks like there's some support / guidance for using an IDE interface to rr's GDB - https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE
There's also Pernosco https://pernos.co/ which is a cloud service (hosted versions possible) from the original developers of rr. It post-processes uploaded rr traces into a database. This post-processing is compute-intensive but happens on their beefy worker machines. Once it's done, they can retrieve program state across time almost instantly - their website is worth checking out for…
Author here: I honestly expected it to be much worse. The tracing integration in particular is fantastic - being able to just slap a `#[instrument(...)]` attribute on any function and then be able to filter which spans you want to see is kind of a superpower.
That said I think much could be improved, still. I only breezed through the perf/nperf stuff because I've used them before, and through the self-profiling stuff because rustc devs helped me with it.
I would've killed for a REPL while I was working on this (or a server architecture so I could write my _own_ rustc queries), and step through them, etc. I like Kate's approach to this[1] - I'd just like something a little more... interactive.
What do _you_ think it should look like? I feel like a ton of good ideas come from folks who "simply didn't know it was impossible" (more accurately: haven't been trained to accept to work with subpar tools). I'm excited to try out Pernosco[2] for example, it seems like a much-needed rethink of debuggers.
[1]: https://twitter.com/thingskatedid/status/1386077675211526148
[2]: https://pernos.co/
I haven't personally used Replay, but from my experience using rr (a native debugger that also provides time-traveling features) being able to replay execution both backwards and forwards in time on a whim is amazing. These tools excel at diagnosing bugs that are hard to reproduce, because you only have to reproduce the bug once under the debugger and then you can endlessly replay that execution until you figure it out. As Jason said above, you can retroactively add print statements in places that would be useful, without having to waste time trying to reproduce the bug again!
roc (the original author of rr) founded a company to build an even more compelling product on top of rr called Pernosco. They have some mind-blowing demos I'd recommend you check out: https://pernos.co/ .
Being able to easily answer questions like "where did the value in this variable come from, and when did it get set?" makes debugging a wildly different experience.
Someone needs to design a programming language and an IDE (and possibly a new OS too) with great debugging as the primary goal. Debugger and IDE support is always just thrown on later in every new language these days.
"Omniscient debugging" as seen in https://pernos.co/ is the holy grail. Time-travel debugging would also be great.
People are far too obsessed with static type checking. A lot of this time would be better spent invested in live debugging tools.
When I'm editing my code I want to see exactly what _value_ each variable contains, the type really doesn't matter so much. Wallaby.js/Console.ninja is a great example of this.
Good debugging, especially deterministic record/replay is usually complicated by the OS. I often wonder what an OS would look like if designed with debugging as a top priority.
For execution, a stack is a continuation. For debugging, we pretend like it's a historical record, and mostly get away with it. Various things break the correspondence slightly. TCO breaks it a lot more.
Debugging is important. It doesn't get enough respect. Stacks are a pretty critical component of debugging, for better or worse.
It would be great if we didn't depend on this fiction quite so much. With native code, there are definitely alternative options now, such as rr[1] and Pernosco[2] where if you want to look back in time—well, you just go back in time. For JavaScript, that's becoming more and more possible with things like Replay[3]. Perhaps before long, the debugging argument will just go away.
[1] https://rr-project.org/
[2] https://pernos.co/
[3] https://www.replay.io/
See also https://pernos.co/ which is based on rr but adds a queryable database of the whole program execution, which allows you to do things like this:
> [...] just click on the incorrect value. With full program history Pernosco can immediately explain where this value came from. The value is tracked backwards through events such as memcpys or moves in and out of registers until we reach a point where the value "originated" from, and each step in this process is displayed in the "Dataflow" panel that opens automatically. There is no need to read or understand the code and think about what might have happened, we can simply ask the debugger what did happen.
https://pernos.co/examples/race-condition