Ted Dziuba, a software engineer, writes blunt opinion pieces on web scaling, Node.js, NoSQL, devops, and software process.
HN commenters bring up his older posts, such as Taco Bell Programming and the Node.js piece, as classic and funny critiques.
The Way(tm) that I was taught/experienced is as follows:
o Logs are there for ignoring. You need them precisely twice: once when you are developing, and once when the thing's gone to shit, but they are never verbose enough when you need them.
o Using logs to derive metrics is an expensive fools errand pushed by splunk and the cloud equivalents(ie cloudwatch and the like). Its slow, inaccurate and horrendously expensive.
o using logs for monitoring is a fools errand. Its always too slow, and really really fucking brittle.
o metrics are king.
o pull model metrics is an antipattern
o Graphite + grafana is still actually quite good, although time resolution isn't there.
o You need to raid your metrics stores
o We had a bunch of metrics servers in a raid 1, which were then in a raid 0 for performance, all behind loadbalancers and DNS Cnames with a really low TTL.
o Cloudwatch metrics are utterly shite
o Cloudwatch is actually entirely shit.
o tracing is great, and brilliant for performance monitoring.
o Xray from AWS is good, but only really for lambdas.
o tracing is fragile and doesn't really plug and play end to end, unless you have the engineering discipline to enforce "the one true" tracing system everywhere
but what do you monitor?
http://widgetsandshit.com/teddziuba/2011/03/monitoring-theory.html this still is canonical.
In short, everything should have a minimum set of graphs, CPU, Memory, connections, upstream service response times hits per second and…
Same experience here. As a small organization, the quotes we got from cloud providers have always been prohibitively expensive compared to running things locally, even when we accounted for geographical redundancy, generous labor costs, etc. Plus, we get to keep know how and avoid lock-in, which are extremely important things in the long term.
Besides, running things locally can be refreshingly simple if you are just starting something and you don't need tons of extra stuff, which becomes accidental complexity between you, the problem, and a solution. This old post described that point quite well by comparing Unix to Taco Bell: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html. See HN discussion: https://news.ycombinator.com/item?id=10829512.
I am sure for some use-cases cloud services might be worth it, especially if you are a large organization and you get huge discounts. But I see lots of business types blindly advocating for clouds, without understanding costs and technical tradeoffs. Fortunately, the trend seems to be plateauing. I see an increasing demand for people with HPC, DB administration, and sysadmin skills.
I'll leave this here:
https://www.slideshare.net/iammutex/what-every-data-programmer-needs-to-know-about-disks
https://www.safaribooksonline.com/library/view/the-data-sessions/9781449315054/oreillyvideos886685.html
TL;DR Disk IO is pretty much a black box in virtualized environments. All the cool stuff you can do with disks (and memory, and tmpfs) to improve performance seems worth hosting your own hardware.
Personally I'd just read everything Ted Dziuba wrote and rethink how much "Ops" is dying.
http://widgetsandshit.com/teddziuba/archives.html
Maybe it's just being replaced by a bunch of code to fix problems caused by "cloud" hosting performance bottlenecks.
I'm not saying this is necessarily bad, but you should be aware that you're probably trading time managing and picking hardware that suits your needs for additional coding time.
>> http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html
> Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ.
> The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A "distributed crawler" is really only like 10 lines of shell script.
I generally agree, but it's probably only 10 lines if you assume you never have to deal with any errors.
Spreadsheets are awesome, and I think that another thing people are dismissing is grep, awk and friends.
I get it that both spreadsheets and unix tools might not be trendy, but they fulfill the Taco Bell Programming idea [1]; they provide simple, scalable, and efficient solutions.
Summing data from column B for each unique value in column A would be as easy as
`awk -F ',' '{a[$1] += $2} END{for (i in a) print i, a[i]}' my-input-file.csv'`
Sure, it would take a few tries to get right, but could for example R do something this concise and dynamic that can run in parallel?
[1] http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html
heh I've noticed this, too. There are a group of us at work that are known as "git surgeons", maybe that's the modifying noun you're looking for!
In general, claiming status as a <technology> <noun> will get you positive attention, as long as it doesn't come off too arrogant (e.g. expert, master, w/e)
e.g. git guru, nix nerd.
It reminds me of this excellent article [0] that I like to share with all the engineers I work with, every year or so.
In that case, you'd be a "taco bell programmer".
But it's maybe too self-effacing depending on your opinion of taco bell :)
[0]: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html