QuestDB's company blog covers its time series database, with posts on benchmarks, SIMD, Java and Rust performance.
HN links it for QuestDB benchmarks against TimescaleDB, InfluxDB and ClickHouse. One commenter asks why Rust was added, and another questions the comparison set.
Yes and no. Statically typed languages only know that data stored in some piece of memory was conforming to some kind of shape/ interface when it was first stored there. That's why tricks like SIMD Within A Register (SWAR) work at all. E.g. when you need to parse temperatures from string input very fast like in the 1BRC: https://questdb.com/blog/billion-row-challenge-step-by-step/ How does your type system help there?
With static typing, you are doing specification and optimization at the same time, which is maybe necessary because compilers and languages are not sufficiently smart but also because of this mix it complicates reasoning about correctness and performance. Also static typing introduces a whole universe of problems with itself. That's why we have reflection or stuff like memory inefficient IP address objects in Java:
For a simple IPv4 address normally representable using 4 bytes/ 32 bits Java uses 56 bytes. The reason for it is Inet4Address object takes 24 B and the InetAddressHolder object takes another 32 B. The InetAddressHolder can contain not only the address but also the address family and original hostname that was possibly resolved to the address.
For an IPv6 address normally representable using 16 bytes/ 128 bits Java uses 120 bytes. An Inet6Address contains the InetAddressHolder inherited from InetAddress and adds an Inet6AddressHolder that has additional information such as the scope of the address and a byte array containing the actual address.…
I edit a database newsletter – https://dbweekly.com/ – so tend to always have my eyes out for new releases, what's coming along, and what not. And I thought I'd share a few more things that have jumped out at me recently in case anyone's in the mood for spelunking.
1. QuestDB – https://questdb.io/ – is a performance-focused, open-source time-series database that uses SQL. It makes heavy use of SIMD and vectorization for the performance end of things.
2. GridDB - https://griddb.net/en/ - is an in-memory NoSQL time-series database (there's a theme lately with these!) out of Toshiba that was boasting doing 5m writes per second and 60m reads per second on a 20 node cluster recently.
3. MeiliSearch - https://github.com/meilisearch/MeiliSearch – not exactly a database but basically an Elastic-esque search server written in Rust. Seems to have really taken off.
4. Dolt – https://github.com/liquidata-inc/dolt – bills itself as a 'Git for data'. It's relational, speaks SQL, but has version control on everything.
TerminusDB, KVRocks, and ImmuDB also get honorable mentions.
InfoWorld also had an article recently about 9 'offbeat' databases to check out if you want to go even further: https://www.infoworld.com/article/3533410/9-offbeat-databases-worth-a-look.html
Exciting times in the database space!
> Unless I'm missing something, hashing function is fast compared to random bouncing around inside ram – very much faster then random memory accesses. So I can't see how it make a difference.
In a GROUP BY, you may have a few hundred million rows, but only a few hundred groups within them. A slow function would slow down things dramatically in that case since the hash table remain small and data access is potentially linear.
> Then you've got sorted data, in which case use a merge join instead of a hash join surely.
This property is beneficial for GROUP BY which includes a timestamp or a function over timestamp. QuestDB organizes data sorted by time, so relying on insertion order may help to avoid redundant sorting if there is an ORDER BY clause with the timestamp column.
As for merge join, we also use it in ASOF join: https://questdb.io/docs/reference/sql/join/#asof-join
Looks like they have an interesting range of customers[0] so my take on "why even add the Rust" is because their customers are already using Java, a total rewrite might be considered irresponsible as it would be incompatible with their existing customer base. I do have to wonder though, if some serious Java refactoring in any way, would have helped at all. How many code smells do they have going on in their codebase? Or is it to the best of their knowledge a really clean Java codebase?
Note Discord themselves has used Rust for bottlenecks with Erlang/Elixir/Beam[1].
[0]: https://questdb.io/customers/
[1]: https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users
I'm writing docs for https://questdb.io/ which offers SQL-based analytics for time series. We have quite a few users already who use us for ML applications that rely on performance.
Instead of writing an MQTT -> PostgreSQL bridge, you could use Telegraf to listen to MQTT topics and write data to QuestDB over InfluxDB line protocol when it meets certain criteria. One of our users shared the tooling they use in industrial IoT and a tutorial here for exactly this use case: https://questdb.io/tutorial/2020/08/25/questitto/#stack
Implement some none-trivial algorithm and then try to optimize the heck out of it. Focus on the concrete CPU, architecture, cache, and memory in the hardware you have in front of you. Forget portability for a moment.
This comes to mind: https://www.morling.dev/blog/one-billion-row-challenge/
Read how others have done it. Here's an example in Java that goes relatively low-level to squeeze out performance: https://questdb.io/blog/billion-row-challenge-step-by-step/