Citus Data Blog

Citus Data (https://www.citusdata.com/) | Technical Solutions Engineer | Full-time | San Francisco

Citus is worry-free Postgres. Built to scale out, Citus is an extension to Postgres that is available as open source, as on-prem enterprise software, and as a fully-managed database as a service.

We are a fast-growing database startup in San Francisco. We founded Citus Data with a vision to combine the best aspects of relational databases with the elastic scale of NoSQL. And we’re building a team with some of the best minds in distributed systems, databases, and Postgres—along with engineers with extensive operational experience managing millions of Postgres databases in the cloud.

As a solutions engineer, you will be responsible for analyzing customer requirements, understanding their use-case, and building an end-to-end solution using Citus technologies. You will need to quickly prototype solutions, run proof-of-concepts to demonstrate value, and also help customers move to production. Every customer is different, and thus adaptability and being a quick learner is crucial. Your day to day role will involve talking to customers, writing code, performance and query tuning and debugging and solving problems.

You can learn more about the position and apply here: https://www.citusdata.com/jobs/solutionsengineer

Postgres.

* PostgreSQL supports most of the major features of SQL:2016. Out of 177 mandatory features required for full Core conformance, PostgreSQL conforms to at least 170. In addition, there is a long list of supported optional features. It might be worth noting that at the time of writing, no current version of any database management system claims full conformance to Core SQL:2016.

* Scales Vertically and Horizontally with tools such as citus (https://www.citusdata.com/)

* Postgres supports RLS (Row Level Security) allowing you to implement authorization at the database level.

* It can support many 1000's of transaction per second running on commodity hardware.

* NoSQL support. Postgres can store and search JSON and other types of unstructured data.

* Postgres has earned a strong reputation for its proven architecture, reliability, data integrity, robust feature set, extensibility, and the dedication of the open source community behind the software to consistently deliver performant and innovative solutions.

This depends what you mean by "scaling". Postgres can easily handle 10s of Terabytes on a single server (though this is dependent upon your use case), but when it comes to multiple host sharded databases at the moment MySQL has some more mature options like Vitess [1], where your options in Postgres are things like Citus [2] which just recently open sourced their shard mover, but still have the HA and replication feature behind expensive enterprise pricing.

For most people not dealing with the multiple terabyte database size, Postgres is what I would definitely recommend. But once you get above that, then there are no easy answers and you would need to look at what YouTube did with Vitess [3]

[1] https://vitess.io/

[2] https://www.citusdata.com/

[3] https://vitess.io/docs/overview/history/

I remember building a system about 15 years ago which automatically failed over in under a second - the application connected to the most recent one and set up replication to the other one. If the master went down then the slave was promoted to master and when the original master came back it was configured as slave.

Looks like https://www.citusdata.com/blog/2019/05/30/introducing-pg-auto-failover/ will sort out a Postgres master master for you.

I certainly don’t have auto updates on all my servers (wouldn’t be good if a tv program going out to millions of people suddenly vanished — there’s always a glitch when it fails over), I do have it on public facing servers though, so in your case I’d have it on the web servers but not on the database.

It is worth noting that Citus was acquired by Microsoft a few years ago (not a secret, but may not be obvious to everyone), and they have since shifted heavily towards the Azure side of the world.

Back when it happened we tried to buy their product and they were not sure if the Citus standalone product was even going to exist, and they refused to demo it even. Odd timing possibly, but it's a data point.

On the positive side of things, the shard rebalancing was not open source back then iirc, which made the open source version pretty useless. Now it seems to be open source: https://www.citusdata.com/product/comparison -- pretty cool.

I'd still be careful to bank on it as a Citus only customer or open source user.

If you'd like to give Postgres a serious shot, I'd recommend taking a look at Citus: http://citusdata.com/. It's a distributed database forked from Postgres, and can easily handle billions of rows in realtime for certain use-cases. We use it at MixRank to store and run analytics on for ads we crawl in the internet (a respectable amount of data), and it handles extremely well. (Full disclosure: I know the Citus team well).

I've also pushed Postgres to its limits and suspect you'd be able to make it work with that much data, though I wouldn't consider it a trivial exercise executing that transition on a production system. If you're seriously considering it though, feel free to reach out via the email in my profile.