Hortonworks Blog

To put this into context I would recommend reading 'MonetDB/X100: Hyper-Pipelining Query Execution' [0]. Vectorized execution has been sort of an open secret in database industry for quite some time now.

For me, is particularly interesting reading the Spark achievements. I was part of the similar Hive effort (the Stinger initiative [1]) and I contributed some parts of the Hive vectorized execution [2]. I see the same solution that applied to Hive now applies to Spark:

- move to a columnar, highly compressed storage format (Parquet, for Hive it was ORC)

- implement a vectorized execution engine

- code generation instead of plan interpretation. This is particularly interesting for me because for Hive this was discussed then and actually not adopted (ORC and vectorized execution had, justifiably, bigger priority).

Looking at the numbers presented in OP, it looks very nice. Aggregates, Filters, Sort, Scan ('decoding') show big improvement (I would expected these, is exactly what vectorized execution is best at). I like that Hash-Join also shows significant improvement, is obvious their implementation is better than the HIVE-4850 I did, of which I'm not too proud. The SM/SMB join is not affected, no surprise there.

I would like to see a separation of how much of the improvement comes from vectorization vs. how much from code generation. I get the feeling that the way they did it these cannot be separated. I think there is no vectorized plan/operators to compare against the…

Well, they could require that companies raising money get an audit done. That way, the crowdfunding platform isn't doing the due diligence - they're simply requiring that the company gets an accountants firm to do it. Auditing companies is a fairly common and well-understood process. I would imagine that the market for such services is fairly competitive, so the cost shouldn't be unreasonable.

I think that a problem with GAAP is that, while it's designed to represent a standardised view of a company's accounts, it can be quite complex and isn't necessarily particularly easy to understand, which can end up being counterproductive.

For example, out of curiosity, I went and looked up Hortonworks' Q4 2014 results[1] to see if they had, indeed, reached a $100m run rate by the end of 2014. Two revenue numbers are listed. GAAP revenue was $12.7m and "non-GAAP" revenue was $16.7m. The $4m difference turns out to be a "contra-revenue" booking relating to a Yahoo! warrant (effectively options that were issued to Yahoo!) that became exercisable when the company IPO'd (which was during that quarter). It appears that the Yahoo! warrant was granted as part of a deal in which Yahoo! committed to being a client Hortonworks and, hence, GAAP required that the $4m in cumulative revenue that Hortonworks had received from Yahoo! as a client, be deducted from the revenue for that quarter (even thought that revenue was generated over the course of several years), and that the rest of the $52m…

Similar to your point about mixed workloads, I have a hunch that Mark's benchmarks are not comprehensive enough to correlate well to real-world usage across a lot of different scenarios, even on pure OLAP workloads. It's great that a billion rows can be aggregated in 0.02 seconds, but there's a reason TPC-H uses 9(-ish?) different queries with varying aggregations and joins, vs. these benchmarks on a single table. (Of course, if your use case is heavy on a specific type of aggregation, it probably makes sense to optimize for that at the expense of other query performance.)

And - perhaps I missed it, but his benchmarks don't seem to utilize rollup/materialization unless the DB does it automatically (or at least easily) on the backend.

As is, it's almost certain that Citus would underperform most of the leaders here. The PG9.5 benchmark actually uses the Citus-developed cstore_fdw extension, and it shows up towards the bottom, albeit running on a single node with hardware a few CPU generations old. (Same as used for the Clickhouse benchmark.) I am curious how Citus/Postgres might perform using the HLL / TopN extensions, though.

Also of note is his Redshift benchmark was run on magnetic drives on ds2 instances, not SSDs. Using those would almost certainly bump performance up a bit.

Druid is optimized for aggregation and filtering, and is somewhat similar to BigQuery on the backend, as I understand it. The Cloudflare blog posted elsewhere in the thread covers it briefly.…

Don't worry about needing to catch up. Stuff is moving so fast these days, you're always working with something new. Everyone is in a continual update mode so it's not like you have 10 years of catching up to do. Tech has turned over a 10 times since then. You could say 10 years and 2 years are functionally equivalent from a new tech point of view.

And don't worry about corps and recruiters. Focus on a problem you want to solve, and update your skills in the context of learning what you need to know to solve that problem. If you can leverage your industry experience in the problem domain, even better.

Data is driving everything so developing a data analysis/machine learning skillset will put you into any industry you want. Professor Yaser Abu-Mostafa's "Learning From Data" is a gem of a course that helps you see the physics underpinning the learning (metaphorically of course -- ML is mostly vectors, matrices, linear algebra and such). The course videos are online for free (http://work.caltech.edu/telecourse.html), and you can get the corresponding book on Amazon -- it's short (http://www.amazon.com/Learning-From-Data-Yaser-Abu-Mostafa/dp/1600490069/).

Python is a good general purpose language for getting back in the groove. It's used for everything, from server-side scripting to Web dev to machine learning, and everywhere in between. "Coding the Matrix" (https://www.coursera.org/course/matrix, http://codingthematrix.com/) is an online course by Prof Philip Klein that…

I've had a similar question before. I have heard of as few as 60,000 observations was considered "big data" [1], yet at my company, we generate about 60 million pharmacy claims every 3 months, and no one here calls it big data. In terms of storage, it's on the order of a couple hundred terabytes for all our data. This is considered small enough that we can query it with traditional SQL.

Big Data, the experts say, is more about the novel way you analyze data, relative to the difficulty of the problem. A speaker at PyCon who was talking about algorithms and data structures for handling genetic data had a term that I like quite a bit better: "Data of Unusual Size" (C. Titus Brown at MSU was the speaker).

"Big Data" is a really big buzzword right now, but the term is overused and often does not convey the meaning it's supposed to. "Big" is a relative term. The novelty of how much data is being used as opposed to how much used to be used (in the sumo case, they had never handled so much data before) is what makes it big.

As for Hadoop, you'd want to use it when it's no longer feasible to keep your data stored in an RDBMS, or when speed becomes an issue, or when you want your schema to be more flexible than an RDBMS. If you are not concerned with the reliability of your data (RDBMS make the safety of the data a paramount priority--read the wikipedia page on ACID to see these guarantees), there's plenty of reasons for choosing Hadoop.

[1]:…

In the critical path many things can matter that ordinarily don't. When working on perf improvements for Storm 1.0 release, profiling revealed hotspots in Clojure code. Adding type hints to avoid dynamic typing fixed some of these hotspots. In other places, Clojure primitives showed up as a bottleneck. These required rewriting some code in Java and invoking it from Clojure.

This was mentioned briefly here:
https://hortonworks.com/blog/microbenchmarking-storm-1-0-performance/

The Clojure based core has served the community well for some time. It was still powered by Disruptor (a fast Java messaging library). So there was always some back and forth between Java and Clojure.

For Storm 2.0, the Java rewrite happened first. The re-architecture work might have not happened without it (at least for the foreseeable near future)... primarily due to Java expertise being more easily available.