Red Planet Labs describes Rama, its Clojure-based platform for building scalable backends, with posts on a Twitter-scale Mastodon and testing.
HN commenters point to it for Rama's Mastodon implementation and case studies, and one enjoyed the preemptive replies to critics in "Make Worse Software, Slower".
I think the persons behind Rust are working on a particularly hard and complicated problems, so I don't want to say that async Rust work is bad. I would instead say it is complicated and hard to understand and use casually, instead.
Async Rust is many language features and behaviour all interacting with each other at the same time, to create something more complicatedly than how you would describe the problem you're actually trying to solve (I want to do X when Y happens, and I want to do X when Y happens × the number of hardware threads). When you're using async rust, you are having to think more carefully about:
* memory management (Arc) and safety and performance
* concurrency
* parallelism, arbitrary interleavings
* thread safety
* lifetimes
* function colouring
All interacting together to create a high cognitive load.
Is the assembly equivalent of multithreading and async complicated?
Multithreading, async, coroutines, concurrency and parallelism is my hobby of research I enjoy. I journal about it all the time.
* I think there's a way to design systems to be data intensive (Kleppmann) and data orientated (Mike Acton) with known-to-scale practices.
* I want programming languages to adopt these known-to-scale practices and make them easy.
* I want programs written in the model of the language to scale (linearly) by default. Rama from Red Planet Labs is an example of a model that scales.
* HN user mgaunard [0] told me about "algorithmic skeletons" which might…
> ... its also reasonable to believe that I wasn't a strong enough clojure developer to fix those issues...
Technically it can likely be done but in practice pushing Clojure code for performance (eg, [0]) doesn't follow the usual language idioms and relies on a sophisticated understanding of the interactions between program design, Clojure, the Clojure compiler and the JVM. I'd suggest pointing more at the language than the programmer. In a situation where direct memory management is required Clojure is a weaker choice.
Although in Clojure's defence it leverages immutability to squeeze out some surprising performance benefits without much effort. I would expect Clojure code to be fast by default but challenging to hand-optimise further if that extra level of control is required. Clojure makes it easy to write code that performs well, it is a stronger language for situations where ordinary business logic changes are the productivity bottleneck.
[0] https://blog.redplanetlabs.com/2020/09/02/clojure-faster/
https://blog.redplanetlabs.com/2024/01/09/everything-wrong-with-databases-and-why-their-complexity-is-now-unnecessary/#Restrictive_schemas
> It’s common to instead use adapter libraries that map a domain representation to a database representation, such as ORMs. However, such an abstraction frequently leaks and causes issues. ...
FWIW, I'm creating a tool (strategy) that is neither an ORM or an abstraction layer (eg JOOQ) or template-based (eg myBatis). Just type safe adapters for normal SQL statements.
Will be announcing an alpha release "Any Week Now".
If anyone has an idea for how to monetize yet another database client library, I'm all ears. I just need to eat, pay rent, and buy dog kibble.
> Running on a real cluster
> Until Rama is publicly available, the only way to get access to the full Rama release is by applying to our private beta.
I thought it was coming out last August? What happened?
(As per your link, https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/)
Edit: @nathanmarz that's cool you have enough interest to keep things moving with it being private. I'm only interested in open solutions for this domain, though. IMO, accepting lock-in is a lot to ask. Best wishes ~ MD
The database I'm working with, Rama, has an interesting way to do "instant migrations" at scale. Your migration function runs on every read until data is durably migrated in the background. So you get migrated data immediately after deploy, while Rama quietly persists the changes during normal processing. No downtime.
https://blog.redplanetlabs.com/2024/09/30/migrating-terabytes-of-data-instantly-can-your-alter-table-do-this/
They _did_ serve the data. From TFA:
> To demonstrate its scale, we operated the instance with 100M bots posting 3,500 times per second at 403 average fanout.
The linked post about their demo [1] has more details about the challenges involved, including rendering home timelines.
[1]: https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/