Michael Noll writes tutorials on Hadoop, Kafka, Storm and large-scale data processing, including setup guides and stream processing concepts.
Commenters recommend his introductory Big Data articles, especially the Hadoop single-node setup and Python MapReduce tutorials, as good starting points for newcomers.
Hadoop itself consists of different parts: HDFS is a distributed filesystem that can span across lot's of machines and stores data in blobs of varying sizes.
MapReduce is the Google idea from before 2004 how to do calculations on lot's of data. Now there is also YARN that could be described as a general job scheduler.
At the moment a lot of people use software on YARK like Spark (does more in memory, is faster, can use the GPU on the cluster machines).
So if you have biological data you could feed that into a HDFS and have Spark or MapReduce jobs that process the data. The clue about Hadoop is that don't need to care about getting the clustering and distributed setup right. This is done by Hadoop for you. You program like you would program a single thread algorithm (at least in the simple cases).
E.g. this Python code counts words over as much data as you want in parallel: http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
If you google each of these projects you'll find a lot of information.
Here are the original papers that should give a good idea:
- HFDS (Google Filesystem was the original idea - Hadoop was a free implementation by Yahoo - http://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf
- MapReduce -
http://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf
- Spark: http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf
If you want a good…
Java's not actually a bad choice, considering some of the tools that are available. Hadoop[1], HBase[2], etc. Depending on what you are trying to accomplish of course, you could also include Lucene[3], Solr[4], Mahout[5] and Weka[6] in that list.
Of course cool Java libraries don't mean you have to work in Java. Scala[7] seems to be gaining quite a bit of momentum, as does Clojure[8].
From your reading list, it sounds like you may be interested in predictive analysis, machine learning, etc. If so, Python[9] is pretty popular in those circles and has some nice libraries available as well[10]. And you can do neat things like use a Hadoop cluster for map/reduce processing, even if you have a client written in Python (or whatever)[11].
Of course Map/Reduce[12] isn't the only distributed processing model out there. You might want to learn MPI[13] then explore the MPI bindings that are available for many (most?) languages, and look at other models of distributed / parallel computing.
Anyway, to summarize, I'd say that if you already know one or more of Java, Python, C, C++ or Scala, you're in good shape. If not, some combination of the above should fit the bill. And yes, R might be handy as well.
Not to take anything away from Erlang, Haskell, Forth, Factor, Scheme, Lisp, OCaml, Perl, or any other language of course. I'm just not as familiar with those so I can't say much about them.
1. http://hadoop.apache.org/core/
2. http://hbase.apache.org/
3.…
What If We Could Rebuild Kafka From Scratch? - Gunnar Morling (https://news.ycombinator.com/item?id=43790420)
1. Five Advantages of Log-Based Change Data Capture (https://debezium.io/blog/2018/07/19/advantages-of-log-based-change-data-capture/)
2. Why Data Architecture Should Guide Your Query Engine Decision (https://www.onehouse.ai/blog/why-data-architecture-should-guide-your-query-engine-decision)
3. Reimagine Apache Kafka (https://streamnative.io/blog/ursa-reimagine-apache-kafka-for-the-cost-conscious-data-streaming)
4. Book Review: Designing Data-Intensive Applications | Henrik Warne's blog (https://henrikwarne.com/2019/07/27/book-review-designing-data-intensive-applications/)
5. Of Streams and Tables in Kafka and Stream Processing, Part 1 (https://www.michael-noll.com/blog/2018/04/05/of-stream-and-tables-in-kafka-and-stream-processing-part1/)
There're many things you can do with a spare linux box. I do keep a few 'project ideas' links just in case I got some spare time on hand.
Home file server with ZFS - http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/
Proxy Server - http://devcentral.f5.com/weblogs/macvittie/archive/2008/10/08/8-things-you-can-do-with-a-proxy.aspx
Map-Reduce fun with Hadoop - http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)
Donate bandwidth with a Tor Server - http://ubuntu-tutorials.com/2007/08/31/how-to-run-a-tor-server-donate-bandwidth-to-the-tor-network/
More good ideas - http://www.dailycupoftech.com/2007/09/17/put-that-old-computer-to-good-use/
A Dial-In server - http://www.howtoforge.com/linux_dialin_server
There isn't any pain. Best of all it's easy to setup hadoop on a single machine, or two machines here and there and then them into a large cluster:
Here's a good tutorial that may help:
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
(If you're using a Linux machine just follow that; otherwise get VMWare running Ubuntu and set it up there).
Michael Noll has some really good introductory articles to Big Data, and I highly recommend his posts for anyone starting in this field.
His post on writing Python streaming programs in Hadoop - http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/ - got me started in this area two years ago, and that's what I'm still doing these days! :)