Crunchy Data Blog

(Author here.) I think you're confusing a couple different major classes of languages — declarative versus procedural. I said specifically "most expressive ever for querying and manipulating data", and I'd stand by that.

The nice part about SQL compared to the languages you've cited is that you just need to specify the "what" and not the "how". I tell it what data to fetch, where I want it from, how I wanted it grouped/sorted (etc.), and the engine figures out the best (and probably a very efficient) way to do that. This is different from a procedural language like Python or Rust where it'd be on me to also define every implementation detail — get data from here, join by doing this and with that algorithm, sort what comes out, ...

This gives SQL a number of very nice properties:

* It's very fast and succinct to write.

* Even very complex queries are likely to be executed as optimally as they can be (because database engines tend to be very good at this).

* It's so high-level that it can actually get more performant over time. See for example Postgres 12, which eliminated an optimization fence for CTEs (common table expressions) [1], which means that a lot of queries got faster automatically, without the user having to do _anything_.

I love a lot the languages you listed, but SQL is a different animal, and amazing in a way that's quite unique.

---

[1] https://info.crunchydata.com/blog/with-queries-present-future-common-table-expressions

We do, although we're in the middle of moving our entire Heroku Postgres spend over to Crunchy Data [1].

We were getting close to one of the big jumps on the standard pricing of Heroku Postgres, and we would have had to basically double our monthly cost to lift the max data we could store from 1.5TB to 2.0TB. On Crunchy Data, that additional disk space will be like 1% more rather than 100% more.

While investigating Crunchy, I ran some benchmarks, and I found Crunchy Bridge Postgres to be running 3X faster than Heroku Postgres.

Heroku seems to be working on some interesting new things, but I feel burned by the subpar performance and lack of basically any new features over many years. I don't know if the new Aurora-based database will be faster than Crunchy, but the benchmarks they're talking about sound like they're finally about to catch them. But we also have better features on Crunchy, too, such as logical replication. Logical replication is still not available on Heroku.

The experience for deploying apps and having add-ons is still pretty easy, but we'll see how that improves. HTTP2 support is still in beta.

1. https://www.crunchydata.com

Both of the kubernetes operators are extremely good. Zalando's uses Patroni for HA, and CrunchyData's has their own HA impl. Both have great backup systems built in.

> Any way the Postgres DB can be setup in a fire and forget setup and scaled just by adding new nodes for read and write?

Generally I'd reply yes and no, yes, adding readers is pretty easy with these operators. No, postgres is generally single writer.

But the CrunchyData k8s operator (pgo) has active-active mode. I dont know much scalability this really buys, since ultimately everyone needs to soak the write load. There is sharding too, built into postgres, but I count myself as blessed that so far straight up verticle scaling has tackled every problem I've ever had and sharding hasnt beem required. Anyways here's a post on setting up active active in pgo, https://www.crunchydata.com/blog/multi-cluster-postgres-federation-on-kubernetes-with-postgres-14-using-gitops

One thing missing from your post is any information on workload. What kind of data are you storing, what kind of use are you expecting,... it's hard to make recommendatuons in the abstract.

I think you're projecting an implementation of a queue in Postgres, which isn't how most people implement these things. [0] We're not doing table level locks, or creating contention with multiple queue producers or consumers, and they're not "one typo away from disaster in prod".

To do this right, you're using row level locking e.g. SELECT FOR UPDATE/SKIP LOCKED [1], and hopefully you're already using idle_in_transaction_session_timeout to deal with total consumer failures. A properly designed queue in Postgres runs more-or-less in parallel, and supports (really fantastic features like) atomic row locks across all resources needed to serve the queue request.

If you need extremely long consumer timeouts, it's also totally fine to use RLLs in addition to state on the job itself.

[0] - https://www.crunchydata.com/blog/message-queuing-using-native-postgresql
[1] - https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/

Postgresql can do everything I need, but not everything I want, and apologies to mysql fans, but I refuse to touch it.

And, I think over the years they have been such an awesome community that has helped me irregularly when I popped into IRC every year or two for a problem I was having. https://github.com/davidfetter ; https://github.com/RhodiumToad ; https://github.com/bmomjian - you are great peeps!

What I would like to see are more security usability enhancements over and above what you find in https://www.crunchydata.com/products/hardened-postgres/

Security Enhancements List:

Attack Resistance

Better protections for the DBMS itself in assume compromised scenarios, or malicious users

Customized implementations per compliance/security regime

Accelerators for various security configurations

Self-audit, self-secure

Functionality Enhancements List:

Continuous Sync Postgres to other serious DB (not a one-way sync, not a DB move)

That requires data to already be in Postgres, otherwise you have to ETL data into it first.

DuckDB on the other hand works with data as-is (Parquet, TSV, sqlite, postgres... whether on disk, S3, etc.) with requiring an ETL step (though if the data isn't already in a columnar format, things are gonna be slow... but it will still work).

I work with Parquet data directly with no ETL step. I can literally drop into Jupyter or a Python REPL and duckdb.query("from '*.parquet'")

Correct me if I'm wrong, but I don't think that's possible with Postgis. (even pg_parquet requires copying? [1])

[1] https://www.crunchydata.com/blog/pg_parquet-an-extension-to-connect-postgres-and-parquet