Server Fault Blog

First of all, massive kudos for the Stack Exchange-like technical transparency. Definitely consider a massive upgrade album like http://blog.serverfault.com/2015/03/05/how-we-upgrade-a-live-data-center/ and http://imgur.com/a/X1HoY!

GitLab is awesome. I'm really sad that in the past two or three months I've only found one GitLab link on HN to click. There really needs to be more. (I'm not sure if this is because I'm browsing in AEDT or if GitLab isn't used a lot on here.)

I wondered about how you guys might do advertising to get more mindshare, and then I realized one possible explanation about why you're doing this: getting technical advice from the community means everyone's had a part to play, and they're likely to remember that. Good move ;)

---

In my case, I have little (okay, 0) practical experience; a lot of the following is mentioned experimentally, to see how these ideas would handle the described environment. It's pretty much all stuff I've read online.

I welcome replies that shoot down any of these ideas.

> Disk

Disks can be slow so we looked at improving latency. Higher RPM hard drives typically come in GB instead of TB sizes. Going all SSD is too expensive. To improve latency we plan to fit every server with an SSD card. On the fileservers this will be used as a cache. We're thinking about using Bcache for this.

There's already been another brief comment (https://news.ycombinator.com/item?id=13153317) about ZFS.

So, I'll ask. Why not ZFS? You don't have…

You only pay for licenses over 4 OS and 2 SQL Server (or outright new ones) after graduating BizSpark. 2 SQL Servers (OS licenses are chump change by comparison) go a loooong way* too, as you're presumably scaling up rather than out.

http://www.microsoft.com/bizspark/about/Graduation.aspx

Through the grapevine I hear the "review" process (for more free licenses at graduation) is extraordinarily accommodating, though I'd guess if you're spinning up a box per customer or something insane you'd get denied.

Lots of people get this wrong, I'm surprised Microsoft hasn't carved this correction into a mountain somewhere...

*Stack Exchange is running 2 + failover ( http://blog.serverfault.com/2011/09/30/the-stack-exchange-architecture-2011-edition-episode-1/ ), and was running on 1 not too terribly long ago.

Not at all. The Windows OS now has command line accessible management points that are similar to Linux. There is still a great deal of difference in their management models (I blogged about this a while back - http://blog.serverfault.com/2013/06/03/cross-platform-configuration-management-is-hard/)

That I was hired as a Windows specialist was so that we can go deeper on the OS side and the PowerShell side, just as we have a Linux expert to go deeper on the Linux side. Our sysadmin team was just more tilted with experience on the Linux side (though you wouldn't know it - as almost everyone I work with would qualify as a senior admin in any Windows shop in the world).

Those drives still cost about 20k or so to purchase, for only about a 25% increase for reads and 10% for writes (http://blog.serverfault.com/2011/02/09/our-storage-decision/) why would we pay that extra for so little gain?

Also, with FusionIO you are putting all of your eggs in one basket - drive wise. If that card dies you are done. In short we don't _need_ the FusionIO so why would we put out that kind of money for it?

Also we went to 96GB about a year or so ago when we setup those boxes it was a reasonable amount of memory to put in now - we are maxing the thing out at 288GB now.

It turns out that I haven't been the only person surprised by the TCP incast pattern, including folks who don't use Erlang at all (e.g. Hadoop users). I was happy to write about it, to help spread the word. Even if I'd stumbled upon Kyle Brandt's posting http://blog.serverfault.com/2011/06/27/per-second-measurements-dont-cut-it/ ("Per Second Measurements Don’t Cut It"), I don't think I'd've realized the full consequences of packets dropped by microbursts of traffic when there's "obviously plenty of bandwidth available".