Gustavo Niemeyer writes about his open source work, including Python's dateutil, the mgo MongoDB driver for Go, gopkg.in, and godeb.
HN commenters link it for his tools: dateutil for Python dates, mgo for MongoDB, gopkg.in for versioned Go imports, and godeb for installing Go.
I've been working to get Haskell approved at my place of employment for 1.5 years but getting the US Gov't to change is rather hard; I've had to sit the "tech" people down and explain that javascript != java... with that baseline, explaining 'Why Haskell' is non-trivial. Come September, after 1.5 years of effort, I should have 'all the FOSS'. Until then I have to wait.
It's worth noting that iPython Notebook + pandas vs. cassava + conduits (even with iHaskell Notebook) serve very different ends. If I need to explore how to do something, I'd use Haskell. But I'm still in phase 2 (phase 1: collect underpants) and I've yet to find anything as powerful and flexible as the iPython Notebook + pandas + hdf5 stack that also just works. I can just move faster with that stack than anything else I've ever seen. That being said, I'm knowingly deferring bugs to the runtime -- 'tis the cost of python.
If you're unfamiliar with pandas, the "quick vignette" here[0] is decent enough. The reason pandas is awesome, IMO, isn't actually because pandas is awesome (which it is) but because it's embedded in a full language. Julia, R, etc... can do the same stuff (maybe faster), but I wouldn't also want to program, say, a production web-app in them (though I have high hopes for Julia).
Fuzzy parsing on dates: pandas by default uses dateutil[1] which is both awesome and slow.
10GB CSV... : yeah... it's "fast for python" but pandas is admittedly doing a lot in that time, namely putting it into a…
We did prototype for a while with fluentd as well as Cassandra and a few other technologies which we decided not to use in the end. The answer to why we chose a lot of what we did was to satisfy our requirements. The "Minimal Hardware with Redundancy And Scale" and "Minimal Software Layers" requirements came into play in regards to your question. We found that most off the shelf technologies are great at scaling, but they have a cost that would have required us to employ a lot of hardware. We were able to reach the speed we needed by writing our own Go client to ship off the logs to our cluster, and that also gave us the customizability and control to be able to make sure we weren't consuming all of the resources our servers needed during peak times to deliver content.
As for why we chose the database we did, while TokuMX is basically a drop in replacement for MongoDB, it has features that make it much more usable for our situation. Specifically, a compression rate of over 80% without negatively impacting our insert speed and document level locking. And because TokuMX works as a drop in replacement for MongoDB, we were able to use the MongoDB driver (mgo http://labix.org/v2/mgo) for Go which is just a really great tool. Additionally, we are now able to leverage the MongoDB aggregation framework which allows us to use the data in a way that builds very helpful aggregations of the data very quickly.
I like what the go authors think about dependencies being backwards compatible, but not everyone thinks that way, and let's face it, people make mistakes and make incompatible changes unintentionally.
I started a project last weekend (still a WIP), go get version[1], that basically proxies the `go get` commands, but fools it into thinking the master branch is either a tag or a version to get around some of these problems, while aiming to still work with the go tool.
For example, if you have a tag named v1.0.3, you would change your import path to be something like this:
gogetver.com/github.com/user/project.v1.0.3
There's also gopkg.in[2] which does something similar, although it appears to only work with github.
[1]: https://github.com/rsenk330/gogetver
[2]: http://labix.org/gopkg.in
> Why are you not able to easily version git dependencies? Go's solution to this problem is to tell you to create an entirely new git repository for each major version. Really? If they didn't want to go full blown dependency versioning with something like CocoaPods, they could at least let you specify a git branch or tag.
Take a look at gopkg.in[1]. This is a service that makes branches in other repositories go-gettable. Eg. go getting gopkg.in/foo/bar/v5 will download branch v5 from github.com/foo/bar. This seems to me to be a way of dealing reasonaly well with the issue -- the go get tool itself doesn't need to understand anything at all here.
[1] http://labix.org/gopkg.in
http://labix.org/gopkg.in is a rather good solution for distributing libraries with transitive dependencies. It requires critical mass for it to really take off, but the level of effort is extremely low for implementation (make a branch on github, don't break it... done!). I think this low level of effort is really important, as I have been able to simply poke authors of my favorite packages for support, and had it in a day or so.
As for an internal project (non-distributed) I think vendoring is the solution everyone who is sane has always been using, why would you risk you project on a random guy deciding he wants to delete his github account.
golang.org offers two types of downloads for Linux: a source tarball and a pre-compiled tarball. If you download the pre-compiled tarball, you just have to extract it somewhere and set some environment variables (all explained in detail on golang.org).
If you really want to install Go via your package manager, you can install the golang package from the Ubuntu repositories. However, this package is naturally on an outdated version of Go. Also worth mentioning is godeb[1] which can generate and install a .deb of any version of Go for you.
[1]: http://blog.labix.org/2013/06/15/in-flight-deb-packages-of-go