Nathan Marz

DISCLAIMER: I'm one of PMC member of Apache Storm.

Even though Clojure has been Nathan's first-class language, not everyone in contributors and even committers couldn't become native on Clojure. (Though I guess part of committers could feel Clojure as native.) It might not be a problem when Nathan builds most of things in Storm (I'll link his blog post which describes lesson learned from Storm on his perspective http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned.html), but when Storm becomes popular and also no longer his toy project, Clojure matters for scalibility of the project.

Personally, I had a hard time understanding plenty of macros and even chained macros - that might not be a problem if I were allowed to have plenty of times to study Clojure and get used to it, but possible contributors don't get a chance when they're ready. In many cases, the first time they read the code on what they're using is when they encounter issues on stage/production and have to dig with stack trace. For me, I had to investigate on Storm because my team suffered from lack of documentation and also lack of knowledge on internal which ended up just taking workarounds. It was pretty hard time learning language to just read and understand the details to track down the issue. I even couldn't think of writing code on Clojure.

When I decided to contribute Apache Storm, I started from connectors and CI build which wasn't written on Clojure. After some months I was able to…

It's hard to compare Clojure and Scala. Scala is a multi-paradigm programming language with strong OOP support and functional support. It's arguably more verbose than Clojure but looks much more similar to Java.

Clojure is a Lisp. Lisp stands aside all other programming languages, first of all, because it supports syntactic abstraction (a.k.a. "code is data"). Hardcode addicts (I'm not one of them) say there are only two programming languages - Lisp and non-Lisp.

Here's a good comparison of Scala and Clojure: http://stackoverflow.com/questions/1314732/scala-vs-groovy-vs-clojure

When we made the decision to switch to Clojure, several things affected it, in no particular order:
- we had some people who were already very proficient in Lisp
- we liked how expressive and compact it was
- Lisp is considered to possess immense expressive power (see http://www.paulgraham.com/lisp.html)
- we were enamoured by Cascalog (http://nathanmarz.com/blog/introducing-cascalog-a-clojure-based-query-language-for-hado.html), and it's written in and for Clojure. This one payed off very well.
- Lisp has a reputation of being great at manipulating data: lists, graphs, etc.

Here's a good answer from one of our engineers: http://www.quora.com/Clojure/Why-would-someone-learn-Clojure

As for libraries, both Clojure and Scala are JVM-based, and Clojure has a very good syntax for Java interop, so all Java libraries are available to us. But, of course, Clojure community also spits out libraries…

At my company, we're actually doing that as well, but we took it further: no tiers at all.

We do load-balancing at the client, every server has a public IP address, we use DNS round-robin only to distribute the login phase; after that, it's location-based with re-connect logic in the app.

We don't store the entire database on every node, but we do the Cassandra-style equivalent: each datacenter has all of the data. We can store all of the data in a single datacenter.

We never have to worry if the "database" is down, because the app server is also down whenever it is.

We even have global visibility on all machines in a Cassandra-like design -- something that took Google's Spanner team custom GPS and atomic timekeeping hardware to achieve. We're doing it with a lightweight consensus algorithm that achieves a reliable commit timestamp globally, and NTP. We can take a global snapshot, or return a globally-consistent query across all data at any time.

I think we'll see a lot of architectures built like this in the (near) future. Collapsing the tiers is great for maintainability and performance.

----

FWIW I found http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html to be...lacking. He ignores all problems related to multi-update completely. Nathan's solution does nothing to solve consistency problems except in the trivial situation where every piece of data can be "updated" (really appended) completely in isolation from any other update.

You actually do need…

1) Startups/business. If you haven't, I would read all of Paul Grahams essays. I think startups are a great option (http://www.collegeanswerz.com/risks-and-rewards-of-startups), but even if you don't end up starting one, you should understand the business. I would also highly recommend reading Rework.

2) Your site. The top part (good design is good business) and the footer (contact me) are solid, but the rest could use a lot of work. The periods in the 'Hi there' paragraph are on the next line. Could just be my browser, but something you should look into. That paragraph definitely should be "nicer". Maybe something like this: http://twitter.github.io/bootstrap/2.3.2/examples/carousel.html. The design of the case study is solid.

The main problem with your site I think is the lack of information. Tell us more about yourself. What you do? What are your aspirations? Hobbies? Thoughts/opinions? Show us that you know design and programming by telling us your philosophies. Also, have a blog (http://nathanmarz.com/blog/break-into-silicon-valley-with-a-blog-1.html).

3) Design resources. I'm new to programming and design too (~ 1 year of on and off experience), but there are some resources that have helped me a lot.
- Definitely learn bootstrap (http://twitter.github.io/bootstrap/index.html).
- Best resource on typography (http://practicaltypography.com/)
- Best resource on color (https://www.helpscout.net/blog/psychology-of-color/)
- Amazing…

It is interesting to read the history of Storm from the original author (Nathan Marz) himself: http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned.html

Clojure allowed him to build the initial Storm release by himself in only 5 months.

> I made all of Storm's APIs in Java, but implemented Storm in Clojure. By keeping Storm's APIs 100% Java, Storm was ensured to have a very large amount of potential users. By doing the implementation in Clojure, I was able to be a lot more productive and get the project working sooner.

Also interesting is that Nathan has now founded a startup working on something which sounds like the next evolution of Storm, and he's building it in Clojure again:

https://redplanetlabs.com/

They recently got 5M in funding and are hiring. Building a Clojure team for it.

Personally, I feel the Storm 2.0 release rewrite to Java is really just a case of the new maintainers not knowing Clojure very well, and being primarily Java devs, looking for contributions from people using Storm, which tend itself to be mostly Java shops. Clojure is not quick to pick up, and very different from Java, so unlike Scala, Kotlin, C#, Go, etc. Someone with a Java background and no Clojure or Lisp experience won't be able to contribute as easily. And that's fine, and possibly best for Storm's future now that Nathan has moved on. Unless a Clojure dev had stepped up and adopted Storm, this change isn't at all surprising.

"Datomic is not an update-in-place system. All data is retained by default."

I'm becoming more and more convinced that your canonical data store should be append-only whenever possible (see eg [1][2] for detailed arguments). It's nice to see first class support for this.

[1] http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

[2] http://martinfowler.com/articles/lmax.html

EDIT: Just read through the whitepaper. Looks like the indexes / storage engine form an MVCC (http://en.wikipedia.org/wiki/Multiversion_concurrency_control) key-value store, similar to Clojure's STM. Peers cache data and run datalog queries locally.

This could be either an available or consistent system, depending on how cache invalidation in peers works. In the available, eventually-consistent case you have the added benefit that all queries see a consistent snapshot of the system, even if that snapshot is not totally current.

Like most of Hickey's work, the whole thing seems really obvious in hindsight. It also bears a lot of similarity to Nathan Marz' recommendations for data processing and schema design.