SingleStore's company blog covers its distributed SQL database, with posts on benchmarks, indexes, vector search, and engineering practices.
HN uses it for job posts and comparisons to other HTAP databases, and one commenter notes its license prohibits benchmarking.
> it's a bit difficult to wrap my mind around how Arrow will help in distributed systems
Comparing with the role of Protobuf is perhaps easiest, there's a good FAQ entry [0] which concludes: "Arrow and Protobuf complement each other well. For example, Arrow Flight uses gRPC and Protobuf to serialize its commands, while data is serialized using the binary Arrow IPC protocol".
This will be increasingly significant due to the hardware trends in network & memory (and ultimately storage too) compared with CPUs. I posted about that in a comment a few days ago [1], but it's worth sharing again:
> here’s a chart comparing the throughputs of typical memory, I/O and networking technologies used in servers in 2020 against those technologies in 2023
> Everything got faster, but the relative ratios also completely flipped
> memory located remotely across a network link can now be accessed with no penalty in throughput
The graphs demonstrate it very clearly: https://blog.enfabrica.net/the-next-step-in-high-performance-distributed-computing-systems-4f98f13064ac
> would be great to read a bit more about some optimizations that would help improving Postgres leaving out pure analytical use cases
Unfortunately I don't have a good reference on that to hand but I'll take a look around and reply again soon.
[0] https://arrow.apache.org/faq/#how-does-arrow-relate-to-protobuf
[1] https://news.ycombinator.com/item?id=37365816
[2] https://www.singlestore.com/comparisons/postgresql/
SingleStoreDB (formerly MemSQL) (https://singlestore.com) | India, Portugal | Full Time | Remote SingleStoreDB is a database focused on high performance and hybrid workloads (HTAP). Our customers include half of the top 10 US banks, 2 of the top 3 US telcos, and 12% of the Fortune 100.
Our product is a distributed, relational database that handles both transactions and real-time analytics at scale. Querying is done through standard SQL drivers and syntax, leveraging a broad ecosystem of drivers and applications.
We have been building SingleStore since 2011, and building a database-as-a-service product involves a bunch of interesting technical challenges such as: * Compilers and code generation
* Distributed computing and fault tolerance
* UI and User Experience
* Integrations with connectors and third-party tools (e.g., Business Intelligence tools like Tableau)
* Infrastructure that scales to dozens of thousands of database clusters across GCP, Azure and AWS
We have the following positions open:
* Partner Integration Engineer (Portugal, India)
* Software Engineer, Managed Services (Portugal, India)
Careers page with individual links for each open position: https://www.singlestore.com/careers/jobs/.
If you want to learn more about the engineering work we do, check out https://www.singlestore.com/blog/category/engineering/.
Feel free to email me directly via david at singlestore dot com.
When we talk about how SingleStore unifies transactions and analytics into a single database, and how fast it is at both, people understandably find it hard to believe at first - most database systems today do one or the other well, but not both.
Here are numbers to back up our claims, on the industry-standard benchmarks TPC-H (analytics) and TPC-C (transactions). SingleStore delivers state-of-the-art performance on both, compared to leading cloud data warehouses and a leading cloud operational database - SingleStore matches the data warehouses on TPC-H and matches the operational dbs on TPC-C, and does it with excellent scale-out performance in the cloud. On the other hand, the data warehouses couldn't even run TPC-C, and the operational database didn't finish TPC-H within 24 hours.
Not every workload needs the full spectrum of transactions and analytics, but modern applications increasingly need at least some of both. We think SingleStore is the general-purpose database system of the future, capable of running a breadth of workloads and unifying multiple disparate domain-specific database systems into one.
Check out the results at https://www.singlestore.com/blog/tpc-benchmarking-results/ and if you're interested, try out SingleStore for yourself - we published the benchmark materials in our github repo https://github.com/memsql/benchmarks-tpc
Clickhouse is great a single table analytical queries (Group-by + aggregates + filters over a single table). It can match any top tier columnstore database at this, maybe even better at smaller scales, and if that is all you need it will do the job well
Beyond this, it has a naive query optimizer and limited ability to run distributed joins. This is why you won't find well known analytical benchmark results like TPC-H and TPC-DS for clickhouse vs other SQL data warehouses.
SinglestoreDB has been doing real time analytics much longer then clickhouse and has a much more mature feature set at this point[1][2]. Our go to market is more big enterprise driven so we are definitely much less well known among developers. Revenue share-wise I suspect we are the leader in real time analytics (see disclaimer below - I'm biased but our revenue is reaching thresholds for IPO readiness).
Also, if you follow Jacks later posts you'll see he replaces DynamoDB and Redis with SinglestoreDB as well. Now were getting into places Clickhouse doesn't tread at all...
Disclaimer: I'm one of the cofounders of MemSQL/SingleStoreDB (still working away on making it better all these years later...).
[1] https://www.singlestore.com/blog/the-technical-capabilities-your-database-needs-for-real-time-analytics/
[2] https://dl.acm.org/doi/abs/10.1145/3514221.3526055
(I used to work at SingleStore, and now work at Antithesis)
SingleStore (f.k.a. MemSQL) used lock-free skiplists extensively as the backing storage of their rowstore tables and indexes. Adam Prout (ex CTO) wrote about it here: https://www.singlestore.com/blog/what-is-skiplist-why-skiplist-index-for-memsql/
When SingleStore added a Columnar storage option (LSM tree), L0 was simply a rowstore table. Since rowstore was already a highly optimized, durable, and large-scale storage engine, it allowed L0 to absorb a highly concurrent transactional write workload. This capability was a key part of SingleStore's ability to handle HTAP workloads. If you want to learn more, take a look at this paper which documents the entire system end-to-end: https://dl.acm.org/doi/10.1145/3514221.3526055
Beyond being closed source, SingleStoreDB's License explicitly prohibits benchmarking:
https://www.singlestore.com/assets/contracts/singlestore-free-tier-software-license.pdf
2. Restrictions. You acknowledge that the Software, and its structure, organization, and source code, constitute SingleStore’s and its suppliers’ valuable trade secrets, and the Software is subject to the following restrictions. Specifically, Customer shall not, and shall not
...
conduct any competitive analysis, publish or share with any third party any results any results of any technical evaluation or benchmark tests performed on the Software, or disclose Software features, errors or bugs to a third party without SingleStore’s prior written consent (“Benchmarking”); or