Honeycomb Blog

Hear hear.

At CoreOS I set up a number of dashboards specifically to socialize the normative behavior of our systems. Think of it as the difference between the person who just drives their car versus the person who knows what feels and sounds "normal".

The latter can tell when they need an oil change because of different vibrations in the engine and how the car sounds pulling up to a stop light (because of the engine sounds being reflected back into the window by parked cars).

Big surprise, it had the desired effect.

I'm especially with you on the notion of disparate metrics. While correlation is not causation, it's still a useful diagnostic tool.

Let's say someone in marketing walks a dashboard and and sees the following:

1) a new version has been pushed
2) customer tickets are up 20% over the number they're used to seeing

Does that mean that the new version caused the tickets? No. Will that allow them to ask? Absolutely. Will that urge behavior to reach out to support and release management to see if there's an interesting story to share internally (or with the world)? Hopefully.

You hit the nail on the head by calling out the absolutist / "I'm the authority on this matter" / "there is a single correct perspective" tone.

Your lack of shock at the author selling something can be remedied. They have a dog in this fight: https://www.honeycomb.io/teammember/charity-majors/

The real takeaway here: Honeycomb is a great tool for log and trace analysis, and by using Honeycomb's minimal data model [1] for traces you can expand tracing throughout much more of your stack. Now Honeycomb itself makes you pay by the event so Slack's internal buffer and collection system really lets them use it effectively for just the data they need. The low adoption of tracing is also due to the high cost of application level integration, which they are deferring by using the existing tracing systems service layer integration.

So if you come at this as a Honeycomb user Slack has not really re-invented any wheel so much built their own backend to funnel data to Honeycomb and use it's excellent trace viewing and light weight analytics UI [0]. In fact API and model which this article goes to great lengths to explain and justify is basically the Honeycomb tracing API and data model [1]. Which itself is an evolution of the system Facebook uses [2]. It is very cool that they have plugged the gap in Honeycomb which is the lack of historical analysis capability by also funneling all data into Presto.

I would love to see the system they use for keeping field/tag names under control. Honeycomb really strains under large numbers of field names.

0 - https://www.honeycomb.io/trace/

1 - https://docs.honeycomb.io/getting-data-in/tracing/send-trace-data/#manual-tracing

2 -…

Great article! I always love hearing Stripe talking about their internals.

I've been using this practice and I agree that it's incredibly useful. I think because people tend to think in terms of "logs", they end up overlooking the much more useful construct of "canonical logs". Many fine-grained logs themselves are almost always less useful than the fewer fully-described canonical logs. Other observability tools often call these "events" instead of "logs" for that reason.

There's a tool call Honeycomb [1] that gives you exactly what this article's talking about in a really nicely designed package out of the box. And since it handles all of the ingestion and visualization, you don't have to worry about setting up Kafka, or the performance of logplexes, or teaching everyone SQL, or how to get nice graphs. I was a little skeptical at first, but after using it for over a year now I'm completely converted.

If you record fully-described "events" for each request, and you use sub-spans for the smaller segments of requests, you also get a waterfall-style trace visualization. Which eliminates the last need for fine-grained logs completely.

If this article seems interesting to you, I'd highly, highly recommend Honeycomb. (Completely unaffiliated, I just think it's a great product.)

[1]: https://www.honeycomb.io/

> I find it a huge step backwards. As far as I know, there's no full text search and the types of reports and aggregation you can do are extremely primitive.

Oof. Honeycomb is for fast, realtime analytics: starting with a high-level question in your mind ("why did our throughput drop by 50%?") and rapidly iterating on a hypothesis (examples in [0]). ELK can... be used for that, but is optimized for another (as you said, full-text search and generating static reports).

Being able to flip from a funny-looking graph directly into "raw data" mode is intended to be a bonus in Honeycomb, not the primary way you interact with your data.

While we believe that fulltext search has its place, beyond a certain point (most production systems, these days), sifting through log lines is a brute-force method of answering questions about your systems — especially if you're not sure what the proverbial needle you're searching for looks like. [1]

(But mherdeg's answer is great, go back and read theirs while you're here :))

0: https://www.honeycomb.io/blog/troubleshooting-in-honeycomb-choose-your-own-adventure/
1: https://www.honeycomb.io/blog/the-true-cost-of-search-first-problem-solving-on-your-production-systems/

I read though half of the article thinking "This is like Honeycomb[1] way of thinking" without realizing it was written by Charity Majors[2].

So, keep in mind that the article is biased towards their business - they are in the Observability realm - and the long-term goal is making you think that you have to achieve super-deployment speed, and that involves deploying to prod often, and have a good system (Honeycomb) to know at any moment what's going on in production with real data. And if it breaks, you just have to quickly rollback.

Aaaand, I buy that. I mean, I think she has totally a point on this: having a safe and quick way to production that leads you to be confident on rolling out things quickly because you have a safe and fast path back, it's a good thing for any software engineering team/organization. It's not easy to go there and you still obviously need tests while developing/integrating, you need canary tests, you need to program in a defensive manner etc etc, but not being scared of sending code to prod and "test" how it will really hold the production pressure it something I'm personally trying to achieve at $DAYJOB.

[1] https://www.honeycomb.io/
[2] https://nitter.net/mipsytipsy

By creating events that contain both the duration of the request and whether it succeeded, you can create graphs that show you the detail you need. Unless you include those data together at the beginning, it will be impossible to tease them apart later on. Combining them into one graph will likely conceal the difference in the two cases, as you describe, unless you feed them in to a system that an natively tease them apart as easily as show them together (such as http://honeycomb.io). So it seems like the disagreement is more about visualization than collection (the section of the blog in which that quote appears).

The originally quoted advice, to show "the duration it took to serve a response to a request, also labelled by successes or errors" remains good advice, so long as the visualization of that data makes clear the separation.

I absolutely agree that careful consideration is required when choosing what to put on dashboards to avoid confusion. That seems to be a separate issue.

(bias alert - I work on Honeycomb, and care deeply about collecting data in a way that lets you pull away the irrelevant data to illuminate the real problems.)