Engineers at incident.io write about building their incident management product, covering Go backend performance, monolith design, on-call practice and learning from incidents.
HN commenters mention it as an incident management tool alongside PagerDuty and FireHydrant, and recommend its engineering blog. Several commenters are employees.
I think the main problem is that measuring a lot of these things is incredibly hard. The post mentioned laptops and build time and it reminded me of a blog post[1] trying to figure out if it's worth upgrading to m3 macbooks. I encorage you to read the post yourself, but my impression is that the whole project was a huge waste of time. I suspect most attempts to measure impact will end this way. I also think a lot of the recomendations on fighting frupidity are seriously flawd. For example letting developers decide isn't any garentee that you've made the right choice, but it is a garentee that you've made an expensive one. Unless the developers are basically paying for their own tools like in a startup, this is definetly a way to piss away a companiese funds on developers that are more concerend with their own career than the profitibility of the company.
[1] https://incident.io/blog/festive-macbooks
I've experienced all these problems. There are solutions trying to address them. E.g. https://incident.io/ (which I'm not affiliated with in any way). It's not easy though. I think they all come from the root cause of teams not investing enough into making oncall processes and solutions good, and in particular not keeping things up-to-date. As you say, runbooks are often outdated. The same happens with lists mapping component ownership to teams.
There's another problem (#8 to add to the list) I also felt pain from: how you're scheduled to work oncall. We had ad-hoc manual scheduling of who would work oncall when. A tool for solving that is https://oncallscheduler.com (which I am affiliated with). It automates the oncall scheduling, while making it fair, predictable, and gives all engineers self-service control over when and how they're scheduled. I'd love some feedback on it.
I'm one of the engineers at incident.io, so thanks for the mention! But yes, when we first started the majority of our customers bought us for incident management and used PagerDuty as their on-call provider (phones them, receives the alerts, hands-off to us when an incident starts).
We've since built on-call directly into our product and we've had loads of those customers migrate entirely into us, dropping PagerDuty. The biggest customers we're onboarding now tend to buy us as their sole on-call and incident management tool, too.
We have a bunch of case studies of people who've moved if that's useful to anyone. My favourite is probably Intercom who migrated from PagerDuty into our on-call in a few weeks (the Intercom team are great fun to work with!)
https://incident.io/customers/intercom
It's great to see more competition in this space. Generally speaking, what I miss in these "incident management" products is also an integrated, flawless way to handle incidents when they're happening. I'm talking about:
1. Quickly creating a proper chat
2. Quickly creating an incident document where you can pin chat messages and use it in the post-mortem. Ideally, pinning some graphs that you'd extract from your observability solutions
3. Having a status page to put a small description for non-technical stakeholders.
PagerDuty covers some of this. Monzo's Response [1] and now incident.io [2] try to cover it too. I'd like to have this experience end-to-end.
1 - https://github.com/monzo/response
2 - https://incident.io/
Cool idea. I thought I’d try to make a Kagi Lens to accomplish the same thing:
https://kagi.com/lenses/LdYine8hZtYmrt8yTMngOUtvTM9rmkRy
Kagi Lenses can be defined in many ways, one of which is specifying URLs to search. Unfortunately you can only provide 10 URLs per lens. Here are the ones I chose:
https://stripe.com/blog/engineering, https://engineering.fb.com/, https://www.uber.com/en-US/blog/engineering/, https://netflixtechblog.com/, https://research.google/blog/, https://technology.riotgames.com/, https://incident.io/blog, https://www.anthropic.com/engineering, https://openai.com/news/, https://shopify.engineering/
> At incident.io, our engineering team runs on-call Friday-to-Friday, with a 5pm handover. This works great for us: as an engineer coming off the pager you get to finish work nice and early on a Friday and enjoy your evening. [0]
[0]: https://incident.io/guide/on-call/compassionate-on-call/
Is this an American thing? 5pm is not “nice and early on a Friday”! 4pm is about what time you should be finishing, and 3pm is “nice and early”.
Love, Australia x