DigitalOcean Blog

Last week ended on a real low note for many of us at DO. We took a perfectly good customer and gave them an experience no one should have to go through (all while he was trying to leave on vacation no less). We can and must do better. To do better we need to learn from our mistakes. To that end, we also think sharing the information about this incident openly is the best way to help all our customers understand what happened and what we are doing to prevent it in the future.

Yesterday we completed our postmortem analysis of the incident involving Nicolas (@w3Nicolas) and his company Raisup (@raisupcom). With their permission we are sharing the full report on our blog here:

https://blog.digitalocean.com/an-update-on-last-weeks-customer-shutdown-incident/

Digital Ocean's statement on the events:

https://blog.digitalocean.com/an-update-on-last-weeks-customer-shutdown-incident/

Lots of support related issues here.

As someone who worked in support for a long time it isn't surprising to see this play out and the whole "Support and Security Operations leadership will create new workflows to allow abuse-related events to leverage the 24/7 structure of Support." while probably a good action to take, is one of those things that you see in support time and again and so rarely do you see an appropriate response as much as a patch for "this one thing won't happen again".

Nobody cares to staff, fund, and support the support teams until something goes wrong, and then it is usually "new workflows" for support.

Another speculative execution side-channel cache timing vulnerability, a la https://meltdownattack.com

Related disclosure from DigitalOcean: https://blog.digitalocean.com/a-message-about-l1tf/

Intel is evasive about the precise attack scenario, but DigitalOcean reports "an attacker could theoretically use one Droplet to view another Droplet’s memory," implying that the side-channel is invasive (that is, theft of data is possible) rather than cooperative (that is, data can be intentionally leaked between processes, a less harmful sort of side-channel).

But then there's Outline, from Alphabet's Jigsaw.[0] It utterly simplifies the creation of private VPN servers on Digital Ocean droplets.[1] They use Shadowsocks obfuscation. It's apparently effective against China's Great Firewall, and seems popular among Chinese dissidents. So the main intended application is pretty obvious, no?

So are they playing both sides? Or trying to con the Chinese government? Or just a large and loosely coordinated organizarion?

0) https://jigsaw.google.com/

1) https://blog.digitalocean.com/digitalocean-outline-jigsaw-vpn/

I'm glad they plugged their outbound network transfer fees compared to the others[1]. I was shocked and horrified when my AWS bill (which I pay myself) quadrupled due to outgoing network transfer fees. It's truly outrageous what they charge. I use Digital Ocean a lot now simply to avoid nasty surprises like that. I hope AWS and Google change that.

[1] https://blog.digitalocean.com/its-all-about-the-bandwidth-why-many-network-intensive-services-select-digitalocean-as-their-cloud/

Cross posting a colleague's reply on the blog post [0].

> We’ve had a WordPress stack at DigitalOcean for a long time and it’s tremendously popular – and now we've opened our platform up so partners can bring their 1-Click Apps to all our users through the Marketplace. Openlitespeed has a new WordPress stack too, and we’re also happy to have ideas for changes and new things on ideas.digitalocean.com

0: https://blog.digitalocean.com/introducing-digitalocean-marketplace