Citus Data
4 HN recommenders · recommended in 6 months, 2016–2020
54:54 · 14K views · Nov 24, 2015
Dan Robinson, then CTO of Heap, explains how Heap runs its product analytics backend on PostgreSQL and CitusDB. It covers conversion funnels, retention and cohort queries across tens of billions of events and 50 TB of data, plus Postgres and Kafka together and the indexing strategy.
HN commenters point to it as a first-hand overview of Heap's Postgres/Citus architecture and its distinctive indexing approach.
8 comments from Hacker News
I was just watching this[0] excellent talk from one of the Heap Analytics engineers on their use of CitusDB + Postgres to do event processing. What use-case would TrailDB be the obvious, hands-down way to go (vs maybe Postgres + Citus)? [0]: this video
You need a queue in front of your database irregardless of write latency. Otherwise you tie your availability to database availability as downtime (even for upgrades) is often unavoidable and network problems are common. Dan gave a pretty good talk about the high-level details of how/why postgres a couple of years back: this video . One reason of why pg is that SQL is really powerful for building complex queries.
Regarding a), a sharded postgresql (i.e. with citus data) can easily accomodate that workflow with just a tiny bit of extra overhead. Re c) this strongly depends on the usecase, I've seen companies use a) to avoid split-brain problems and having to manage two data pipelines to great success at similar scales. You might find this video interesting.
Keep in mind that Heap collects a huge amount of data and has a huge dataset. I like this[1] talk, it gives you an overview of their architecture. Great post though, thank you for sharing. [1] this video
If you haven't already seen it, Dan from Heap Analytics has a great talk about how they use Postgres and Citus. this video