BigML Blog

most people don't really think logically or quantitatively. whatever math people learned in high school and college was forgotten when trying to make adult decisions. and they didn't even get to the good stuff.

data science is successful now because they are applying quantitative methods to the social sciences -- problems most people think about. the average educated person can understand by using the magic of "machine learning" we can solve problems in politics, health, social issues, etc.

Nate Silver's lack of a math background works to his advantage. he realizes the bar for quantitative understanding in the public sphere is very low. whatever little bit he does, he has to explain to a room full of drunk people in 30 seconds or less. he is very good at this.

neither employers nor the public want to admit that at the end of the day SOMEONE has to be responsible for these calculations. i have a master's degree in math -- which nobody gave a hoot about (see item 5 in this article http://gigaom.com/2013/04/16/how-to-hire-data-scientists-and-get-hired-as-one/ ) we weren't taught how to communicate our ideas and results to the public -- in fact we were discouraged from it. that is difficult to unlearn.

i've heard of attempts to "democratize" machine learning.
http://blog.bigml.com/2013/03/06/democratizing-machine-learning-with-c/
articles like these are usually written by companies with a product or book to sell or distribute. what needs to be democratized is…

A very powerful command line tool for Machine Learning: http://bigmler.readthedocs.org/en/latest/

Create a predictive model is as simple as:
bigmler --train < data.csv

or create a predictive model for Bitcoin volume taking online data from Quandl and parsing it with jq in just one line.

curl --silent "http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" | jq -c ".data" | bigmler --train --field-attributes <(count=0; for i in `curl --silent "http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" | jq -c ".column_names[]"`; do echo "$count, $i"; count=$[$count+1]; done) --name bitcoin

More info here: http://blog.bigml.com/2013/01/31/fly-your-ml-cloud-like-a-kite-with-bigmler-the-command-line-tool-for-machine-learning/

http://bitly.com/bundles/hmason/1

http://commondatastorage.googleapis.com/books/icdar2007/README.txt

http://datahub.io/

http://blog.bigml.com/2013/02/28/data-data-data-thousands-of-public-data-sources/

http://iatiregistry.org/

http://open.undp.org/

http://data.worldbank.org/

https://explore.data.gov/catalog/raw/

http://www.data.gov/opendatasites

http://data.gov.be/datasets

http://opencorporates.com/

http://glasspockets.org/work/reportingcommitment/api.html

http://thedata.harvard.edu/dvn/

http://www.reddit.com/r/datasets

http://archive.ics.uci.edu/ml/

http://cleandatahub.org/

http://datacatalogs.org/

http://archive.org/details/oxford-2005-facebook-matrix

here's some other data hubs/search engines, endless lists:

http://datahub.io/

http://blog.bigml.com/2013/02/28/data-data-data-thousands-of-public-data-sources/

http://tm.durusau.net/?p=39312

http://dvn.iq.harvard.edu/dvn/

_____________

this subreddit seems like a decent place to ask questions

http://www.reddit.com/r/datasets