Cloudera's company blog covers Hadoop, Impala, Kudu and data science techniques, written by its engineers.
HN links it for Hadoop setup references, Namenode high availability details, and a Kudu benchmark that commenters cite when discussing InfluxDB at scale.
The use cases for such random access was from our experiences to maintain large-scale training data, with the needs to debugging model performance against to data, which requires fast slicing and dicing over the dataset, filtering into subset of dataset that satisfy certain distributions, and visualize them on internal debugging UI interactively.
A few of users use random access to do shuffling and training on subsets as well.
Many of them migrated from a system where you can store asset URL in your format of choice (say parquet/tfrecord), to Lance, partially because putting asset (i.e., image, large tensors, lidar point cloud) physically together can lead to better scan perf than loading a lot of small files from S3 or on-prem object store / file systems, due to much less metadata server load over the directory / key&value structures. (i.e. Some of our users see similar issues like this decade old article https://blog.cloudera.com/the-small-files-problem/)
One motivation to design lance is to avoid creating a new copy of training dataset just for one model / training iteration. This is one copy of data for maintenance (update/schema evolution/deletion), analytics & visualization, and training.
For scanning, Lance has been proved slightly faster than Tfrecord and parquet over object store (S3 and GCS). I'd contribute the scan performance just to Rust and tokio async I/O. It is not necessary a better design in scan as we just try to scan "no more data than parquet" when…
Cloudera | Location: San Francisco or Palo Alto, CA | Full-time| Onsite | Software Engineers in Cloud, Test, Site Reliability, Hadoop, Test, Automation, Spark, QA, Security and more | www.cloudera.com
Picture yourself at Cloudera in 2016!
Our mission: Founded by leading experts on big data from Facebook, Google, Oracle and Yahoo, Cloudera's mission is to bring the power of Hadoop, MapReduce, and distributed storage to companies of all sizes in the enterprise, Internet and government sectors.
Over the past few years, Cloudera has experienced tremendous growth, making us the leading contributor to the Hadoop ecosystem and a leading provider of enterprise solutions for Big Data.
Cloudera is revolutionizing enterprise data management by offering the first unified Platform for Big Data, an enterprise data hub built on Apache Hadoop™. Cloudera offers enterprises one place to store, process and analyze all their data, empowering them to extend the value of existing investments while enabling fundamental new ways to derive value from their data. Only Cloudera offers everything needed on a journey to an enterprise data hub, including software for business critical data challenges such as storage, access, management, analysis, security and search.
Check out all our roles and apply here: http://bit.ly/2016EngCareers
Glassdoor's Employees' Choice Awards 2016 http://bit.ly/1ORttqv
The Best Startup CEOs To Work For http://read.bi/1KR6zvN
20 Big Data Companies Leading the Way…
Database optimization posts are always interesting, but it's really hard to do any apples to apples comparison. Your performance is going to depend mostly on your hardware, internal database settings and tunings, and OS level tunings. I'm glad this one included some insight into the SQLite settings disabled, but there's always going to be too many factors to easily compare this to your own setup.
For most SQL systems, the fastest way to do inserts is always just going to batched inserts. There's maybe some extra tricks to reduce network costs/optimize batches [0], but at it's core you are still essentially inserting into the table through the normal insert path. You can basically then only try and reduce the amount of work done on the DB side per insert, or optimize your OS for your workload.
Some other DB systems (more common in NoSQL) let you actually do real bulk loads [1] where you are writing direct(ish) database files and actually bypassing much of the normal write path.
[0] https://dev.mysql.com/doc/refman/5.7/en/insert-optimization.html
[1] https://blog.cloudera.com/how-to-use-hbase-bulk-loading-and-why/
We were just joking at work today how Cloudera's new Kudu "fast storage system"[1] compares itself to "fast" HBase. At least there they were talking about random access (something that HBase is reasonably ok at), and put HDFS on the "slow" end of the spectrum when talking about storage.
This system claims:
Experiments show that WTF can qualitatively outperform the industry-standard HDFS distributed filesystem, up to a factor of four in a sorting benchmark, by reducing I/O costs
They benchmark a sort 300GB of data. You know what's fast at sorting 300GB data? My laptop.
Sort a PB[2] much faster than Spark+HDFS and then it might be worth looking at. Even 100TB[3] is an interesting size.
Maybe I'm being too harsh? What am I missing here?
[1] http://blog.cloudera.com/blog/2015/09/kudu-new-apache-hadoop-storage-for-fast-analytics-on-fast-data/
[2] https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html (note that this was on disk with HDFS, not Spark in memory).
[3] http://sortbenchmark.org/
Although I have a lot of respect for the amplab, they did not do their due diligence with that benchmark. Mainly for a few reasons, they didn't test using columnar storage in Hadoop (ORC / Parquet), which is what Redshift is using underneath (a proprietary columnar store). Also, the most complicated query they ran was a two table join, and from what I can tell, there wasn't any concurrent workload testing.
(disclaimer: I'm a Cloudera employee):
I recommend checking out the following blog, not because my employer wrote it, but because the guys behind the benchmark did an incredible job making the benchmark competitive. They also show metrics that a lot of the other people are not showing, for example concurrent workload capabilities, CPU efficiency, etc.
Impala, Hive (on Tez), Shark, Presto
http://blog.cloudera.com/blog/2014/05/new-sql-choices-in-the-apache-hadoop-ecosystem-why-impala-continues-to-lead/
Most of gcc is still written in C. Only a tiny amount of new code is in C++. In my opinion, that's the worst of both worlds-- inconsistent code, plus C++ overheads.
I work with some fairly recent version of both compilers-- gcc 4.7.2 and clang 3.2. I also know a developer on the LLVM team. clang is way ahead of gcc in many ways, and the gcc devs know it.
We still ship software built with gcc, rather than clang, because gcc is still better at optimization. (gcc also supports more obscure targets, though that's not our biggest concern).
We also make use of llvm's ability to dynamically recompile code on the fly in Cloudera Impala. More information here: http://blog.cloudera.com/blog/2013/02/inside-cloudera-impala-runtime-code-generation/
This would not be possible with gcc no matter how much work we did (license issues-- Impala is Apache 2.0, gcc is GPLv3).