The AMPLab Blog is from UC Berkeley's Algorithms, Machines and People Lab, covering big data systems like Spark, Shark and the Berkeley Data Analytics Stack.
HN links it for the big data query engine benchmark; commenters note its limits, as it uses few tables and simple queries, and credit AMPLab for Spark and Mesos.
My list is somewhat of the same but also different as well. It is focused on specialization within those trends rather than just a broad overview. The difference is specialization or niche in these markets can mean the difference from surviving as a business or failing.
Something else to be aware of is if you develop something now you are most likely to see it's production use within 3-5 years of development and then you are most likely to see it's market either growing or not in 10 years. This is an interesting dichotomy because it means picking things that may be hard but boring, or fun depending on how you look at them.
I am not affiliated with any of these companies. I just think some of them are well-placed in the market.
-Devops (mostly containers and virtualization software, but also fast/easy build software, customizable configs that adapt better and searchable/informative apis)
-Devops massive ssl/managing keys/passwords deployment software (there are a few automated tools that may help with this but there is in general not too many companies currently handling it)
-Devops massive worker model software or software that helps automation in newer/interesting ways like stackengine (http://stackengine.com/blog/) just got $1 million in funding and provides a way to convert vagrant images to docker images and back again in a fast fashion
-Devops with companies that provide continuous integration at scale or preconfigured devops things that require too much learning…
Thank you for responding. I indeed spent time/effort in way of my crafted reply because it caught me eye. I haven't fully parsed the paper but covered a number of pages that colored the nature of my inquiry. I in no way intended to take anything away from the author of the paper but wanted to get at what you yourself declared :
"was accumulated in the AMPLab (the predecessor of the RISELab, see https://amplab.cs.berkeley.edu/)" as The backstory behind the paper as I clearly surmised there was one its historical nature. I also wanted to understand how long this was being worked due to the nature of the language used in the paper and how the concepts and language familiarly fit in with other things I've seen. And this right here :
"The lab's sponsors are also helpful, some of them have been experimenting with the system internally and giving us feedback." Yes, I understand the nature of this is moreso for corporate use cases than it is for academic and furthering therein. I was in search of names but already have a number of them I can surmise and a handful more that I will derive. I think its interesting what is being done here but there were choice words that were stated in the paper that limit it. At this juncture and time in the state of AI development, I will reserve any other commentary beyond stating that there are an incredible amount of fundamental limits in approaching things this way that fall on deaf ears due to the shut off nature/sponsorship of such…
In addition to Google...
DataBricks is a multibillion dollar company and was based on research at the AMPLab UC Berkeley, which was funded by an NSF Expeditions grant. https://amplab.cs.berkeley.edu/news/amp-awarded-10m-nsf-expeditions-grant/
Duolingo is a multibillion dollar company and was based on NSF funded research. https://www.nsf.gov/science-matters/nsf-gave-duolingo-its-wings
My previous startup, Wombat Security Technologies, was based on NSF funded research of just $1.2M. It led to over 200 jobs, and we helped protect millions of people around the world through our cybersecurity training.
I'm sure there are dozens of other startups that I don't know of also based on US Federal funding from NSF and NIH.
Strong science leads to a strong economy, and a strong economy is essential to national security.
We're also in an AI arms race with other countries around the world. Cutting science funding right now is a massive self-inflicted wound.
For everyone who is a US citizen, please write your Senators and House representatives pushing back against the chaos and the proposed science cuts. It only takes a few minutes, and the future of science in the US needs every bit of help it can get right now.
Redshift is an especially limited SQL engine considering it doesn't support UDFs. It is wicked fast, but what you get in speed you lose in flexibility. Current (well, February, but fairly current) benchmarks[0] place Impala and Shark (SQL on top of Spark) within grasp of Redshift while pulling data from disk and, for certain workloads, on par or faster than Redshift. This is without using a columnar file format.
Impala is impressive technology, but it does require you to run dedicated Impala daemons as it doesn't use map reduce under the hood. Shark is especially interesting, however, because it is fast AND build on top of spark, so you can run raw Spark jobs, SQL queries, graph processing and ML all on the same cluster. Shark currently uses Hive to generate it's query plans, but the Spark project is working on implementing it's own SQL engine called Catalyst[1] that promises to be a significant improvement.
[0] https://amplab.cs.berkeley.edu/benchmark/
[1] https://spark-summit.org/talk/armbrust-catalyst-a-query-optimization-framework-for-spark-and-shark/
I agree with your findings on Hadoop. It is a very complicated ecosystem. The core is tied to JVM and abstractions of map-reduce are heavy and difficult to use. There are multiple domain specific languages/api to do the same thing : pig, cascading. Yet if you pick one you cant use other. Hadoop feels like tower of babel.
But I am not in the favour of creating another stack as Hadoop replacement. Spark fills the role of distributed computation framework very well. It uses scala a functional language which provides neat abstractions for map,reduce,filter and other operations. It introduces RDD to abstract data. All the components graphx, sparksql, blinkdb use scala.
Also building a new map-reduce engine is a difficult task. The computation engine will lie in the middle of stack not on top as shown. Libraries will sit on top of that, so if you build an engine you will have to build everything on top of that.
See the Spark ecosystem : https://amplab.cs.berkeley.edu/software/
CAP theorem can apply to any clustered system, it doesn't have to be multi-site. What happens if 6 of your 12 machines die? What if they get cut off from the other 6?
edit: There's a bit of discussion further down about the SQL implementation. That's something I was very curious about as well. The projects linked below spend a lot of time working on supporting full ANSI SQL, and reducing latency by pushing down as many operations as possible. The Overview page doesn't appear to mention how filtering, aggregation, windowing, etc. work in your system.
Also, I noticed on your website that you compare InfiniSQL to Hadoop. How do you feel it compares to Impala (http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/) and Shark (https://amplab.cs.berkeley.edu/projects/shark-making-apache-hive-run-at-interactive-speeds/)?