Code as Craft

A few of my favoriates from my RSS reader (in no particular order) include:

- http://www.kalzumeus.com/blog/ I love reading anything by Patrick McKenzie because he writes about incredibly valuable things, usually focusing on how to make sales, lifecycle emails, running his companies, and other general start-up related information. You might recognize him on HN as patio11

- https://blog.cloudflare.com/ CloudFlare has really excellent content and often writes about SSL/TLS related topics, which personally interest me. They have good technical explanations of SSL/TLS vulnerabilities and discuss how they provide SSL to all of their customers. Also, they discuss how they handle massive scale and other interesting things.

- https://aws.amazon.com/blogs/aws/category/week-in-review/ I often find really good articles linked from the AWS blog. I like to stay current on the latest/greatest features and improvements which AWS puts out there and the Week in Review posts are often easy to skim and see if a linked article is worth reading in its entirety.

- https://codeascraft.com/ Etsy's engineering blog is fantastic. I am always blown away at the passion that comes through from their posts. They seem to have a really sharp set of devs over there and I just enjoy reading their stuff. I especially enjoy the quarterly performance review posts, which show graphs about latencies throughout their stack.

- https://strongloop.com/ The StrongLoop blog has some really great content for NodeJS…

Another good package to look at is Etsy's Skyline - https://github.com/etsy/skyline

Their introduction of the Kale stack (which includes Skyline) is a great read - https://codeascraft.com/2013/06/11/introducing-kale/

I spent a month or so evaluating anomaly detection systems and I can tell you a few things the twitter post fails to mention

1. You can get a long way with an ensemble of simple techniques. And it's always better than any single technique.

I wouldn't recommend trying to install skyline but re-implementing the ensemble of anomaly classifiers they have might take you a day or two and it will get you 90% of the way there.

2. The False Positive rate is probably your most important metric.

Detecting anomalies is good, but your ops team already has plenty of alerts to deal with. If you throw false positives at them from a new system they will hate you and ignore the new system. Most papers report ROC curve data as the metric for classification, that can be ok too.

3. Don't build something complex when a threshold will do.

If a point anomaly is obvious to a human, you absolutely should not build a complex system to detect it, just use thresholds. It's only when you want to detect anomalies before they cross a threshold that you should start on this kind of task. That leads me to

4. Almost all anomalies have a temporal component.

If your detector isn't ultimately looking at multiple sources of data and finding patterns that initially look like normal behavior…

This reminds me of what we've been doing for a while, the "blameless postmortem." The technique is championed by Etsy, and you can read more about it in an article that introduces their debriefing guide:

https://codeascraft.com/2016/11/17/debriefing-facilitation-guide/

From the linked PDF from the article:

> “Adaptability and learning. We learn through honest, blameless reflection on lessons and surprises. We believe that traditional root cause analysis makes learning from mistakes difficult. Our blameless post-mortem process is a widely-cited technique that we believe is becoming best practice among organizations that value innovation. Blameless postmortems drive a significant percentage of our development as we analyze what about our production environment was less than optimal and rapidly make corresponding adjustments.”
(Etsy, Inc., 2015)

The idea, boiled down, is to inspect timelines, procedures, and actions and develop a narrative of how an incident came to be. The goal is for everyone to walk away with a (better) understanding of everything. With this, people are better armed to put into place solutions.

One example from the text is where an engineer pushes out a change because they thought the build system had zero failures. The push breaks the system and causes a regression that should have been caught in the tests. During the postmortem, the engineer says, "I thought the tests had zero failures. I guess I need to be more diligent in the future." Upon further…

Etsy | http://etsy.com | Brooklyn, NY ONSITE FULL-TIME

I'm a product manager here at Etsy and love it here. More than 20 million people buy from Etsy each year and we have 1.5 million sellers offering 36 million unique handmade, vintage, and craft items.

The people are really smart, friendly, a little quirky, and they care a lot about the product, the infrastructure, and the community. If you have read our engineering blog, https://codeascraft.com/, you know these folks know their stuff.

Feel free to reach out or comment with questions.

Jason

Full-Stack Software Engineer

You'll be helping us build new and awesome experiences for people shopping on Etsy. We're looking for full-stack engineers - you'll be dealing with the MySQL, Memcache, PHP, and front-end stuff (HTML/CSS/JS). The teams working on each feature are pretty small (a few engineers, a designer, and a product manager), so there's a lot of meaningful collaboration.

Because we're making user-facing features, we believe strongly in collecting and analyzing numerical data to help guide our decisions. To that end, you'll be able to use our in-house analytics tools to gather information and gain insights. You'll also be able to use Etsy's Continuous Deployment architecture to push code to production whenever you please (and you'll probably push on your first day here).

https://www.etsy.com/careers/job/oJoaWfwX

>"AFAIK no one at Etsy has said that they run their own servers because it's more authentic, part of their "craft", etc."

I'm not sure I would agree with that based on a recent-ish post on “code as craft” the Etsy engineering blog:

"At Etsy, the vast majority of our computing happens on physical servers that live in our own data centers. Since we don’t do much in the cloud, we’ve developed tools to automate away some of the most tedious aspects of managing physical infrastructure. This tooling helps us take new hardware from initial power on to being production-ready in a manner of minutes, saving time and energy for both data center technicians racking hardware and engineers who need to bring up new servers. It was only recently, however, that this toolset started getting the love and attention that really exemplifies the idea of code as craft."

Source:
https://codeascraft.com/2016/02/22/putting-the-dev-in-devops-bringing-software-engineering-to-operations-infrastructure-tooling/

Tangentially related to the hardware discussion - a bit of dated hardware post:

https://codeascraft.com/2012/08/31/what-hardware-powers-etsy-com/

I completely disagree.

If you work on a small engineering team and want to broaden your knowledge of technologies you might want to use, reading about approaches that larger companies use is a great source of information. Most cutting edge software engineering (not computer science), occurs in technology companies. So if you ignore "corporate" engineering blogs, then you are hugely limiting your exposure to how software engineering works in the real world.

Of course the companies are not blogging about everything, but that is irrelevant. Your choices are either a) not learn anything or b) learn stuff that might be dated (but probably also battle tested and rock solid).

And I think you are way off with the advertising thing. The reason most companies make these engineering blogs is to attract engineering talent, thus their incentive is to post cool stuff which is interesting and exciting. Look at Etsy's great engineering blog [1] which I've personally learned quite a bit from. Read a few posts... they aren't trying to sell arts and crafts.

----

1: https://codeascraft.com/, in the OP's list