Metamarkets, an analytics company, writes about real-time data systems like Druid and Samza and about large-scale data work.
HN links it for posts on Druid, Samza, and its Hacker News headline analysis. Many comments are the company's own hiring posts.
I know no solution doing all we need, only some interesting components. Here are some more links:
storing data
* couchdb http://couchdb.apache.org/. Good if you already know the analysis you want to do. drawback is that indexes you create (= materialized views) are monodimensional. a multidimensional indexing on top of it would simply be great. Not great if you need real real time reporting and your data change a lot.
* mongodb: more flexible than couchdb for ad-hoc querying
* graph databases: neo4j, orientdb, ... might be a good alternative to OLAP
* ElasticSearch http://www.elasticsearch.org/ lets you have faceted indexes (= index and retrieve facts according to dimensions values). Pluggable to a many storage solutions
reporting / visualizations
* d3.js http://mbostock.github.com/d3/ far above other charting libraries
* Simile Exhibit http://simile-widgets.org/exhibit3/ Great for faceted browsing (facet = dimension)
* Cube http://square.github.com/cube/ Time series visualizations
* mind gapper https://github.com/RandomEtc/mind-gapper-js visualizations la Hans Rosling's Gapminder
* dejavis http://beta.dejavis.org/ looks offline for the moment
Big data
* Hadoop + data-warehousing / Querying solutions Apache Pig and/or hive
ETL (nothing exceptional here)
* Yahoo Pipes
* Python flow
* Google Refine
Saas
* Bime http://bimeanalytics.com/
* Druid http://metamarkets.com/2011/druid-part-i-real-time-analytics-at-a-billion-rows-per-second/
Here's a list of relevant links I've compiled over the past year or two:
- http://nathanael.hevenet.com/the-best-time-to-post-on-hacker-news-a-comprehensive-answer/
- http://minimaxir.com/2014/02/hacking-hacker-news/
- http://minimaxir.com/2014/10/hn-comments-about-comments/
- http://karpathy.ca/myblog/2013/11/27/quantifying-hacker-news-with-50-days-of-data/ https://cs.stanford.edu/people/karpathy/hn_analysis.html
- http://metamarkets.com/2011/hacking-hacker-news-headlines/
- http://danluu.com/randomize-hn/
- http://www.bayesianwitch.com/blog/2013/why_hn_shouldnt_use_randomized_algorithms.html
- http://camdp.com/blogs/multi-armed-bandits
- http://gkosev.blogspot.com/2012/08/fixing-hacker-news-mathematical-approach.html
- http://blog.rjmetrics.com/2012/10/17/surprising-hacker-news-data-analysis/
- http://blog.rjmetrics.com/2012/10/24/how-to-get-on-the-front-page-of-hacker-news/
- http://blog.datadive.net/which-topics-get-the-upvote-on-hacker-news/
- https://news.ycombinator.com/item?id=8790134
- https://porter.io/blog/hackernews-cheaters-catch-me-if-you-can/ https://news.ycombinator.com/item?id=9332889
- http://www.righto.com/2013/11/how-hacker-news-ranking-really-works.html
- https://news.ycombinator.com/item?id=9336759 https://news.ycombinator.com/item?id=9333611
- http://www.almostinfinite.com/other/hacker-news-new-page-scroll-of-death.html?
- From the event page -
Tech Talk: Eric Tschetter on Druid - Distributed Exploration of High Dimensional Data
Druid is a distributed system in use at Metamarkets (http://www.metamarkets.com) to facilitate rapid exploration of high dimensional data. They use Druid to expose impression monetization data to internet publishers along any arbitrary combination of demographic, content and sales-based dimensions. One Druid cluster currently exposes a data set of >15 billion rows of data representing >500 billion impressions in hypercubes of varying dimensionality (largest is 28 dimensions) while allowing for exploration using top lists and timeseries in sub-second latencies. The tech talk will be a discussion of the design considerations and architecture of the system.
http://metamarkets.com/2011/druid-part-i-real-time-analytics-at-a-billion-rows-per-second/
About Eric Tschetter
Eric Tschetter is the lead architect of Druid, Metamarkets' distributed, in-memory database. He held senior engineering positions at Ning and LinkedIn before joining Metamarkets. At LinkedIn, Eric productized LinkedIn's PYMK with Hadoop. He holds bachelors degrees in Computer Science and Japanese from the University of Texas at Austin, and a M.S. from the University of Tokyo in Computer Science.
I believe that the note on the bottom, about machine learning, is not germane to the graph displayed. The graph is a straight visualization of hacker news post rank.
The machine learning discusses this earlier blog post: http://metamarkets.com/2011/hacking-hacker-news-headlines/
I don't understand why the authors combined boosting with stochastic-gradient descent.
Boosting means you do greedy, forward-selection of features. You add one feature at a time, the one that has the largest gradient with respect to the loss. i.e. you pick the feature that looks like it will fit the data the best, given what the model currently knows, and then add the feature to the model. You can then choose the optimal weight for the feature using a line search.
Why is SGD necessary in this scenario? After you picked your features, did you go back and re-adjust the weights? I'm not sure why this is necessary. If you are going to do SGD over many features with strong normalization, why not do it over the entire feature vector and see which features disappear?
Metamarkets | https://metamarkets.com/ | SF, NYC | On-site | Fulltime
HIRING: Sr Backend Software Engineers (Scala/Java), Sr Full Stack Software Engineer, Front End Architect, DevOps, Interaction Designer, Technical Account Manager
Metamarkets is the leading provider of interactive analytics for the $30B programmatic marketing industry, and inventor of the Druid open-source data store. We're a 40 person startup scaling out a next-generation streaming analytics stack to handle 1 trillion events daily, leveraging the leading open-source frameworks Kafka + Samza + Druid (developed in house) + Spark.
What we offer: Comprehensive medical/dental/vision, free lunches, quarterly offsites & annual Tahoe trip, WFH Wednesdays.
Apply at https://goo.gl/He15hU or mike.driscoll [at] metamarkets [dot] com.
Metamarkets - http://www.metamarkets.com/ - San Francisco, CA
Looking for: Interns, Full-Time, H1B transfer-friendly
Metamarkets is looking for people who are excited by data.
We have a number of different engineering positions open both on the front-end and the back-end.
On the front-end we're running entirely in JavaScript/CoffeeScript with Node.js for our server-side along with a framework called DVL that builds on top of D3.js.
On the back-end we're using everything from Java, Scala, R, Hadoop, Pig, ZooKeeper, to Memcached. This is all to help run our own distributed, column-oriented analytical datastore called Druid.
Please see http://metamarkets.com/jobs/ for more information about the jobs we have open. Questions please send to hiring@metamarkets.com