Data Science at the Command Line

What Hacker News says

1. ignurantJan 2024
   Some of the most enlightening books I’ve read when I was first learning Ruby were Text Processing in Ruby, and Building Awesome Command Line Apps in Ruby 2. They each reveal certain features and perspectives that work towards this end, such as text parsing moves, Ruby flags to help you build shell 1-liners you can pipe against, and features with stdio beyond just printing to stdout.

   Then add in something like Pry or Irb, where you are able to build castles in your sandbox.

   Most of my data exploration happens in Pry.

   A final book I’ll toss out is Data Science at the Command Line, in particular the first 40 or so pages. They highlight the amount of tooling that exists that’s just python shell scripts posing as bins. (Ruby of course has every bit of the same potential.) I had always been aware of this, but I found the way it was presented to be very inspirational, and largely transformed how I work with data.

   A good practical example I use regularly is: I have a project set up that keeps connection strings for ten or so SQL Server DBs that I regularly interact with. I have constants defined to expedite connections. The [Sequel library](https://sequel.jeremyevans.net/) is absolutely delightful to use. I have a `bin/console` file that sets up a pry session hooking up the default environment and tools I like to work with. Now it’s very easy to find tables with certain names, schemas, containing certain data, certain sprocs, mass update definitions across our entire system.

   ``` # Something failed, and not everything loaded as you expected # explore explore explore…

   db::CompetitorPricing.tables

   db::CompetitorPricing.tables.grep(/^pricing_/)

   latest = DB::CompetitorPricing.tables.grep(/^pricing_/).map {|table| [table, table.max(:create_ts)}

   latest = latest.sort_by(&:last)

   # serialize to csv, or json, or perhaps copy the data into excel:

   Clipboard.copy latest.map{ _1.join("\t")}.join("\n") ```

   Just like with work in the shell, you have a really easy time iteratively exploring the problem and building up your answer. The ability to serialize your data you’ve found, and keep your favorite tools in your pocket feels extremely productive. And of course, all of this can be written in ruby 1-liner shell scripts, or more complex shell scripts to pipe in and out of other tools if desired.

   Show morethread
2. a_bonoboNov 2023
   I usually recommend Data Science At The Command Line https://jeroenjanssens.com/dsatcl/

   Not because it explains bash well, but because it fits with my data science/bioinformatics niche. It's all about these funky sort/uniq/rev/cut/uniq/awk/wc/sed patterns I use everyday fiddling with csv files.

   thread
3. asicspJun 2023
   For field extraction (like the first example: `awk -F, '{print $1}'`), you can also use `cut -d, -f1`

   GNU datamash (https://www.gnu.org/software/datamash/) provides features like groupby, statistical operations, etc.

   See also this free ebook: Data Science at the Command Line (https://jeroenjanssens.com/dsatcl/)

   thread
4. a_bonoboMay 2023
   There's a similar O'Reilly book, Data Science at the Command Line, which I can also recommend: https://datascienceatthecommandline.com/

   Buffalo's book is great and I generally recommend it in my bioinformatics career, but it does go into some speciifc niches (especially the IRanges/GenomicRanges parts!). Jannsen's book is a bit more 'generic', for lack of a better word

   thread
5. coeneedellApr 2023

     Location: Philadelphia, PA
     Remote: Yes
     Willing to relocate: Yes but only to New York City
     Technologies: Data Science
     Resume/CV: https://coen.needell.org/files/cv.pdf
     email: coeneedell@gmail.com
   

   I'm currently a staff researcher at Penn. I was trying to get a PhD but failed to secure offers, so I'm moving to industry. Historically I've been the sole data scientist and technologist on all projects, and as such have had to learn and become proficient in every step of the data science and analysis pipeline. I'm quick to learn, and I love learning how to use new tools, understanding their philosophy and what they're good for (or not good for!) I generally follow the "Data Science at the Command Line" approach to my work, and I like to build web systems for sharing my work from the lower level (d3.js over full-fat systems like observable, tableau, etc).
   Show morethread
6. gullywhumperNov 2022
   See also Jeroen Janssens' Data Science at the Command Line:

   https://datascienceatthecommandline.com/2e/

   thread