The engineering blog of Heap, a product analytics company, with many posts on PostgreSQL performance, indexing, and JSONB.
HN commenters mention Heap as an analytics tool that automatically captures all user actions, and recommend it as an alternative to Google Analytics.
Hi. I'm one of the database engineers at Heap. This is a good question. There are several reasons why we use EC2. First of all, I will say I love RDS as a product. We actually do use RDS for a number of our services. We use Postgres on EC2 only for our primary data store. As for reasons why we use EC2:
Cost - Our primary data store has >1 Petabyte of raw data stored across dozens of Postgres instances. The amount of data we store is at the point where RDS is too expensive for us. The cost of an instance on RDS is more than twice the cost on EC2. For example, an on-demand r4.8xl on EC2 instance costs $2.13 an hour, while an RDS r4.8xl costs $4.80 an hour.
Performance - The only kind of disk available on RDS is EBS. EBS is slow compared to the NVMe the i3s provide. We used to use r3s with EBS and got a major speedup when we switched to i3s. As a side note, the cost of an i3 is also less than the cost of an r3 with an equivalent amount of EBS.
Configuration - By using EC2 we can configure our machines in ways we wouldn't be able to if we used RDS. For example, we run ZFS on our EC2 instances which compresses our data by 2x. By compressing our data, we get a major cost saving and a major performance boost at the same time! There isn't an easy way to compress your data if you use RDS.
Introspection - There are times where we've needed to debug performance problems with Postgres and EXPLAIN ANALYZE won't suffice. A good example is we used flame graphs to see what Postgres was…
Congrats to Mixpanel on the raise. I have heard many great things from them and used them at a previous company. They've done a great job of making it a no-brainer for startups to start using more advanced analytics tools (not just Google Analytics) from day 1. I think we're all better off for the ability it gives product developers to improve their product based on actual usage--since it means the overall quality of products is far better than if vanity metrics drove all product decisions.
I noticed they mentioned Heap Analytics (https://heapanalytics.com/) as one of their competitors. We've been using Heap for over a year and it seems like the logical and magical next step in analytics. Mixpanel gave you smarter analytics on things you had the foresight to track, but Heap automatically tracks everything from the day you integrate it. That means you can get smart analytics even on things you didn't have the clairvoyance to start tracking 6 months ago, or didn't have the resources to insert tracking code in.
For startups, Heap's automatic and retroactive tracking is huge. It means we can iterate on product features and marketing/outreach schemes way more quickly while still getting insight into what's successful and what's not. It's not perfect--a couple times we've added special class names to our HTML elements so Heap can distinguish them, but that's still easier than adding manual tracking code--but it's a huge improvement over the old way.
I noticed Heap has a page…
While you do need to create tables for your stats, you are able to ALTER TABLE just as you can for normal Postgres without problem. It may not be as pain-free as NoSQL in that regard, but building tooling for this or automating this is certainly possible.
Alternatively if you do want pure blog storage, Postgres's JSONB datatype works as well, although with some performance trade-offs. This can work quite well if you have some structured data that lives alongside other unstructured data. [1][2]
Building or putting a REST front-end in front of PostgreSQL should not really be an issue (we built one for our hosted version), and there are already a fair number of PostgreSQL clients/libs. That said, for ease of use we are already thinking of adding an HTTP interface.
For Grafana: We were actually working on our own connector to support PostgreSQL, but found out that one is already in the works by the Grafana team (which will work out of the box with Timescale, because each of our nodes look like PostgreSQL to the outside world). This doesn't seem like it'll be an issue for very long.
We've mentioned retention policies elsewhere, but you do bring up a good point. Instead of dropping data, support for aggregating data in older chunks to a more coarse-grain resolution is something we are already looking into.
[1] https://www.citusdata.com/blog/2016/07/14/choosing-nosql-hstore-json-jsonb/
[2] https://blog.heapanalytics.com/when-to-avoid-jsonb-in-a-postgresql-schema/
I know that applying on jobs pages is not popular here on HN but, well, sometimes it works. In my opinion, having some projects on GitHub certainly helps, even if they are just hobby projects with no use in real world. So it doesn't mean that you must spend months working on open source projects on the evenings and weekends, hoping to get noticed. Same for technical blog posts. It definitely provides more signal to the person who is reviewing your application.
Also, very few companies are 100% remote, and making remote team work is not trivial, so it's important to ask the right questions when interviewing. Don't forget that's it is a two way street! Personally, I was looking for these two key things:
- at least ~30% engineers should be remote;
- there should be a strong culture of written async communication.
Once I discovered a company which I found interesting ( https://heapanalytics.com ), checked the jobs page, noticed that they are hiring remotely and just applied a few months later. Had several interviews over video chat / slack, no white board coding or CS trivia. In the end it worked out and I got an offer. Totally recommend applying at Heap - we are doing cool and challenging technical things (querying hundreds of TBs of data in seconds and making it reliable), the interview process is great and our small distributed remote team is made up of engineers from 4 continents, from North America to Australia!
For 1 TB of data and 1 TB of data growth yoy, you might be able to get away with vanilla Postgres with reasonable sharding/partitioning/data structure.
Without too much details, I am working on a project that handles automated reports/metrics and our only problem with Postgres has been our write-heavy work load (554 M transactions a day with ~250 GB a week). This is a still very early goings and a fraction of our "production" target scale, but we haven't had any read issues.
Our problem is that our constant writes to tables mean that our checkpoints [http://dba.stackexchange.com/questions/61822/what-happens-in-postgresql-checkpoint] started to take significant periods of time and happen more and more frequently. Postgres also has some write amplification [http://blog.heapanalytics.com/speeding-up-postgresql-queries-with-partial-indexes/] and VACUUMING challenges [https://www.postgresql.org/docs/current/static/routine-vacuuming.html].
But again these issues are specifically due to our write-heavy, timeseries data. For now we mitigate the effects with sharding and partitioning as we transition to Cassandra...but it sounds like you don't have a similarly write-heavy workload. So I think you might be able to get away with just Postgres.
https://www.postgresql.org/about/
http://stackoverflow.com/questions/21866113/how-big-is-too-big-for-a-postgresql-table
Heap (YC W13) | Software Engineer | San Francisco, CA | Fulltime | https://heapanalytics.com
Help us bring data science to the masses. We're a small team building the analytics engine that powers 6,000+ companies and helps them make smarter decisions.
Heap automates away the pain of data. Other analytics tools require you to define events upfront and manually instrument code. Instead, Heap automatically captures everything: clicks, submissions, taps, etc. This lets our users analyze data instantly and retroactively, without writing code.
We’d particularly like to get to know you if:
- you communicate with clarity and precision. We care about this almost as much as your technical ability.
- you're passionate, even if it's not about analytics. There should be something you know more about than anyone on our team.
- you’ve written 10,000 lines of code that look embarrassing in retrospect. ... and you've written 10,000 lines of good code since then.
Our app-layer stack is TypeScript, React, Node.js, Redis, and PostgreSQL.
Under the hood, Heap is powered by CoffeeScript, Scala, ZooKeeper, Kafka, and CitusDB.
If you are interested, please email at anojh@heapanalytics.com or visit https://heapanalytics.com/jobs