PagerDuty Blog

I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here.

Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to gain root access. We were alerted to this activity and fully revoked the attacker's access within 60 minutes of the first node being compromised. Working with Linode support, we discovered which user account was being used and completely deactivated the user. We also isolated the VMs, and performed forensics on read-only copies of their disk images.

In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side.

We also have evidence from access logs provided by Linode…

Disclaimer: I work at http://www.pagerduty.com so feel free to tar and feather accordingly.

Carrying the duty pager is a painful experience for some fraction of companies, BUT the long term trends are promising. Here's what I'd keep an eye out for (I've been on call for ~5 years):

* Does being on call affect your other commitments? At PD we scale back the number of predicted story points by ~50 for the devs that are on call.

* Are you empowered to permanently fix the root cause of whatever woke you up? (that's where that 50% of time goes) If you aren't, that's a big red flag. Not all developers take advantage of it, but the ones that do are much happier once they kill the root cause with fire.

* Are you compensated for on call? Among our customers, we have a few that pay $500/week for on call duty, that seems to be the rate at which you can easily find people to swap shifts with.

* Make sure you are off call sometimes. Seriously.

* Who owns the pain report? Someone needs to track how often (and when) people are disturbed and make sure that you are making progress as a team (Github's Ops team is amazingly good at this). If the house is always on fire, you're not a firefighter, you're a person who lives in a flaming house.

* Is it a NOC model, where you can write down common things to try to solve a type of problem (and then you're only paged as an exception) or are you paged for everything? (That's a severe over simplification)

* What is the expected response…

This might work at low traffic levels, but there are better ways. In our internal error catcher, we've seen deploys that caused 20,000+ errors in under a minute. Even when we used similar techniques to mail ourselves errors on a low traffic site, it was important to introduce dead time, not alerting again for (say) 5 mins after the first alert so as not to flood our inbox.

A useful technique btw is to use a ring buffer to collect debug logging, and when an error occurs dump this buffer into the message along with the stacktrace. It gives you more context when you're not logging debug to disk, and is fast. (see eg http://www.exampler.com/writing/ring-buffer.pdf, https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/SMTPAppender.html)

Services like https://www.pagerduty.com/ can contact you on more channels when problems happen, and also deal with dead time etc not /re/ alerting you constantly. The Slack mobile client would have got every one of those messages as a push.

Pagerduty won't help with aggregating/exploring stacktraces though, for that, there's eg https://airbrake.io or for mobile apps, https://try.crashlytics.com/.

Finally there's also http://www.splunk.com/ for aggregating logs; you can build some quite complex queries from it and do alerting on the results (not quite as fancy as pagerduty, but functional).

There are many other tools in this space, worth looking around to see if there's any SaaS you can use or crib ideas from.

For those of us unfamiliar, this quote is from a TV show, not a real CEO.

Don't get me wrong, it's a damned good parody, but let's not mistake fiction for reality.

Some real ones:

> None of this would be possible without you, our leadership, and our board — thank you for your grit and resilience, your commitment to our customers and your support of our values and people. I am reminded in moments like this, of something Martin Luther King said, that “the ultimate measure of a [leader] is not where [they] stand in the moments of comfort and convenience, but where [they] stand in times of challenge and controversy.”

https://www.pagerduty.com/blog/letter-to-employees/

> We had to lay off 90% of our support team because of this AI chatbot. Tough? Yes. Necessary? Absolutely.

https://www.boredpanda.com/ceo-fires-support-staff-after-ai-outperforms-them/

And because a picture is worth 1000 words, Braden Wallake's cryface selfie

https://www.reddit.com/r/technology/comments/wlg92h/ceos_linkedin_crying_selfie_about_layoffs_met/

PagerDuty | San Francisco | Toronto | Atlanta

PagerDuty is the leading digital operations management platform for organizations. Over 10,000 enterprises and small to mid-size organizations globally trust PagerDuty to improve digital operations, drive revenue, mitigate threats, protect assets, and delight customers. We were included in the 2017 Deloitte Technology Fast 500 for the second year in a row, Inc. 500 and Forbes Cloud 100 lists as well as the 2018 Best Places to Work in the Bay Area.

We're hiring for a number of roles:

Senior Software Engineer | San Francisco, Toronto, Atlanta
https://jobs.lever.co/pagerduty/6f501ae4-a5ef-46d0-955a-d3417b147b9b

Senior Database Automation Engineer
San Francisco, Toronto, Atlanta
https://www.pagerduty.com/careers/jobs/?jobid=a0K37000008p93CEAQ

Senior Data Engineer, Analytics | San Francisco, Toronto, Atlanta
https://jobs.lever.co/pagerduty/73c0c2cf-e786-4438-8c94-5a20e3b88585

And many more:
https://jobs.lever.co/pagerduty?lever-origin=applied&lever-source%5B%5D=HackerNews

PagerDuty | San Francisco | Toronto | Remote

PagerDuty is the leading digital operations management platform for organizations. Over 10,000 enterprises and small to mid-size organizations globally trust PagerDuty to improve digital operations, drive revenue, mitigate threats, protect assets, and delight customers. We were included in the 2017 Deloitte Technology Fast 500 for the second year in a row, Inc. 500 and Forbes Cloud 100 lists as well as the 2018 Best Places to Work in the Bay Area.

We're hiring for a number of roles:
Senior Software Engineer | San Francisco, Toronto, Remote
https://jobs.lever.co/pagerduty/6f501ae4-a5ef-46d0-955a-d3417b147b9b

Senior Database Automation Engineer | San Francisco, Toronto, Remote
https://jobs.lever.co/pagerduty/1c2ac0da-1499-4e17-9de9-08995687ab3c

Senior iOS Engineer | Toronto
https://jobs.lever.co/pagerduty/e8eb05ff-adf0-4f10-a26d-d5637234ed92

And many more: https://www.pagerduty.com/careers/