Honeybadger Blog

My recent use case is needing an alert if a certain metric of my system exceeds a threshold or is not present. The simplest viable alerting tool might take these parameters:

* alert threshold

* comparison operator

* expected datapoint interval (in units of time)

* whether to consider missing data breaching (boolean)

* if missing data is breaching, margin of skew allowed (in units of time)

(assume the tool magically has access to metric source, that’s not important)

I’m not sure whether the concept was named prior to this tool [0], but this tool’s name has become a colloquial term: a “Dead Man’s Snitch” style of alert. Once familiar with the idea, I found an exception-monitoring vendor [1] that I was already using at the time had built this feature in. I quickly found use cases and was broadly able to eliminate large volumes of “cronspam” and email filters in favor of the much more elegant construct — because it was so easy to implement and maintain, it was viable for a whole long tail of second class system metrics that otherwise didn’t deserve proper instrumentation.

CloudWatch on the other hand is lower level and more flexible (naturally with higher learning curve). It fundamentally provides similar value by offloading alerting state and logic to a trusted managed system, but is distinctly inferior for my use case:

* It treats missing metrics as literally breaching the threshold and makes no semantic distinction, which spikes the whole abstraction. You can (only…

I've been assembling a list of these lately for a book that I'm working on. (http://startingandsustaining.com) Some of the categories are fairly loose as some apps don't fit nicely into categorical buckets, but hopefully this is a helpful list.

--Browser/Email Testing

BrowserStack (http://www.browserstack.com)

Litmus (http://litmus.com)

--Bug/Issue Tracking

BugHerd (http://bugherd.com)

Lighthouse (http://lighthouseapp.com)

Sifter (http://sifterapp.com) (Disclaimer: I built this.)

--Planning & Project Management

Sprintly (http://sprint.ly)

Podio (https://podio.com)

Flow (http://www.getflow.com)

Interstate (http://interstateapp.com)

Basecamp (http://basecamp.com)

Apollo (http://www.apollohq.com)

Pivotal (http://www.pivotaltracker.com)

Asana (http://www.asana.com)

Trello (https://trello.com)

Blossom (https://www.blossom.io)

Trajectory (https://www.apptrajectory.com)

--Business & Traffic Analytics

KissMetrics (http://kissmetrics.com)

MixPanel (http://mixpanel.com)

DigMyData (http://digmydata.com)

--Continuous Integration / Code Quality

Travis (https://travis-ci.org)

Circle (http://circleci.com)

CodeClimate (http://codeclimate.com)

Sempaphore (https://semaphoreapp.com)

--Dashboards

Ducksboard (http://ducksboard.com)

Geckoboard (http://www.geckoboard.com)

Instrumental (https://instrumentalapp.com)

--Error/Exception Handling

Sentry (https://getsentry.com)

Coalmine (https://www.getcoalmine.com)

HoneyBadger (https://www.honeybadger.io)

BugSnag…

Yes, I was highlighting my view.
To respond to the advantages mentioned:

- Roda (or Sinatra) gives routing. Actually it's very advanced routing that can be composed

- JSON serialization can be achieved more easily with Hash + Array and a subset of basic types (boolean, numeric, string, nil), which are easier to generate than with jbuilder, and it's also more performant. That being said, nothing stops developers from using tilt-jbuilder (gem) to get immediate support from Tilt for jbuilder, which can be used from Sinatra and Roda seamlessly, and can also be called everywhere, not just views

- Roda has built-in security too (I don't know about Sinatra, but I presume it does), there is a section in the README: https://github.com/jeremyevans/roda#security- as well as multiple plugins http://roda.jeremyevans.net/documentation.html#external

- Rails testing is actually _harder_ than normal Ruby testing. On top of my mind, a glaring example of this is the impossibility to instantiate a Controller and do dependency injection on them, which should be normal for any object. This is not a problem if dealing with just Ruby. Rack-test gives the same "request specs" that are present in Rails (actually it's built on top of that). As usual, the option of choosing different testing frameworks is available, given that it's normal Ruby. There isn't a need for special testing framework integration, so it's possible to use the beautiful Test Bench: http://test-bench.software/

- Devise is…

Sure, it can be accomplished in a variety of ways: swapping a migrated replica; breaking big schema changes into multiple incremental migrations that minimize the amount of time the database is locked and back-filling data; or spinning up an entirely new cluster, migrating everything, and then switching traffic over from the old cluster to the new cluster at the load balancer. Any number of these techniques can be mixed and matched to accomplish minimal and/or zero downtime depending on your constraints. [1] The most important thing is making sure your application code is able to operate with multiple schema versions.

I used to have a presentation from Facebook where they spoke about their zero-downtime migration process, but I can't find it, so this SO post [2] will do. Of course, there's a bunch more information out there to read up on.

[1] https://www.honeybadger.io/blog/2013/08/06/zero-downtime-migrations-of-large-databases-using-rails-postgres-and-redis

[2] http://stackoverflow.com/questions/6740856/how-do-big-companies-like-say-facebook-do-migrations-without-having-downtime

Not Java or Python, but here’s some Ruby: https://www.honeybadger.io/blog/how-to-try-again-when-exceptions-happen-in-ruby/

The above article talks about exploiting Ruby's support for call/cc to do something similar.

A noteworthy difference: Ruby's support for call/cc is baked into it's runtime, while in Haskell you can implement call/cc as a normal library. This is done by leaning on Haskell's monadic "do" syntax and a suitable implementation of the Monad class.

http://hackage.haskell.org/package/mtl-2.2.2/docs/Control-Monad-Cont.html

I'm rooting from my phone, so I can't type up a this-vs-that. But maybe that piques your interest and you can read up on Haskell and programming with monads.

We are using Elixir/Cowboy at https://www.honeybadger.io to receive log drain data from our customers' Heroku apps. It's reliable and fast, but deployment has been a sort point (we're spoiled by Capistrano). I recently changed the deployment to elasticbeanstalk with the multiple docker image approach, and that has been much nicer.

The ecosystem is still young -- a lot like Rails was in the early days -- so there are rough edges, but I'm betting they will get sorted out, just like they did in Rails land.