Backblaze Blog

There are pretty much three major costs to consider when it comes to video online. They are storage, delivery and encoding.

Storage is pretty cheap. S3 starts at $0.14 per GB for the first TB and then drops to $0.125 for the next 50TB and so on. There might be some cheaper cloud storage options out there. If you don't need the durability that S3 provides, you can use their reduced redundancy option which is like 40% cheaper. There are other cloud storage provides out there worth considering like Rackspace's Cloud Files, although I think S3 is the cheapest. CDNs also often include their own storage so that might also be something to consider. You might be able to get a good rate if you committed to a minimum storage and bandwidth package. If you wanted, you could even go the route of building your own storage units (http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/) which presents it's own pros and cons.

Delivery also isn't that expensive. CloudFront, for instance, charges $0.12/GB for the first 10TB and $0.08/Gb for the next 40TB. If you're streaming over 10TB per month, you can start to get discounts by committing to monthly minimums. Again, there are other providers out there, such as limelight and akamai, but as far as I can tell, they don't publish their prices. That usually means you probably need upfront commitments with them. Because of that, there is probably a lot of room to negotiate the price of bandwidth. Delivery will most…

This isn't my area of expertise, but when you factor in redundancy, you're looking at a petabyte of data just for what you have now. You'll need it to failover plus have at least one full backup at a separate location. Commodity hardware almost seems prohibitively expensive. You'll have the hardware, 600+ 2tb drives just to start, space, electricity, cooling, maintenance (with that many drives, you'll probably be replacing them often).

If you went with the solution that BackBlaze rolled for themselves, you'd be looking at 15 of their servers (for redundancy), without even adding any more data in the future. Using the solution they came up with, that would run almost $120,000. I have no idea what the monthly maintenance expenses would be.

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

You are undertaking a nearly herculean feat, here. I don't think any of the professional data storage solutions out there will be any cheaper, either. In fact, a petabyte of storage (especially if you need doubles of all hardware to place at another location) would probably run nearer a million bucks or more.

It might even be worth, as someone else linked, considering BackBlaze themselves. At least find out what they could do for you. They offer a business plan at something like $50/computer for unlimited storage. Obviously they're not going to store 0.5-1.0 petabytes of data for that price, but on the scale that you need, you might be able to…

Rack space is an ongoing cost while the hardware is a fixed cost. Presumably they've decided it's worth having a slightly higher fixed cost per GB in exchange for a reduced ongoing cost per GB. (There's probably other benefits as well, such as reduced workload and just getting the space online now.)

Let's take a look...

They do have an oldish post on hosting costs:
http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/

A rack in 2011 cost them $2,100 per month (it's more now, guaranteed). Their racks look to hold about 11 machines.
Each pod is then about $191/mo.

Each rack thus needs 1.375 extra pods at "low density". That's $262.50 a month. Since you now need about 10% more pods, you're paying about $29 more in hosting costs per month per pod.

Your savings are not $688, since you also need to construct 1/8 pod later. A low-density pod will cost about $2719, with is $340 split 8 ways. So real hardware savings is $348. Hosting costs eat that in exactly a year, and they are certainly expecting a longer life than that. (That's 2011 hosting costs, which are certainly lower.)

So it's worthwhile to spend a little extra for the additional drives. (Of course, if they could get the extra five in with more modest hardware, as seems quite possible, that would be a sure win. But sometimes ease of development/assembly/spare parts wins over hardware cost.)

You're right, my math was flawed, though possibly not in the way you were implying, because:

I added a T1 ($250/month) to a building rental ($2000/month), and then said that they'd have $10,000/month left over.

Why would I do such a thing? Why, so that:

They could upgrade to redundant DS3s (or better!)

They could buy extra drives!

They could pay someone to live within a few miles and carry a cell phone that accepts text messages!

...etc.

Doubly funny, to me, is that "disk failure" is being mentioned in the comments on an article about how all of their disks failed all at once. So, no, I certainly did not forget about disk failure. Indeed it's one of the reasons that I'd rather not host certain things as EC2 instances.

As far as scalability: Well, how far would 67TB carry you? Because you could deploy 67TB every single month and still have money left over. [1]

To get to the meat of my argument: I think that what's both great and horrible about EC2-like services is that they allow programmers to deploy Really Big Things without having to employ or deal with system administrators.

As a result, it's a solution that's being misapplied in some cases.

[1]: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ -- it even has pretty pictures

Check your math. You're talking about recording the keystrokes and URLs people visit on smartphones. That adds up to such a phenomenally-small fraction of the data they actually download, storage for all 141+ million users could probably be accomplished by your average geek with a checkbook.

For scale, lets assume that each user sends, oh, 100 megabytes of data over a year. Roughly equivalent to 100 million characters, or (going with a 5mb ascii version of the bible I just googled) 20 bibles worth of text, or 1 million URLs at 100 characters long (that's roughly one URL every 30 seconds), so this is likely an overestimate, possibly substantially. And they have all 141 million users for the full year. Punch in the numbers, and you get 141 terabytes of data. Without compression.

That's microscopic. I can buy hard drives for that off the shelf for not too much money. Here, Backblaze sells that much for $7.4k in a single box, which is absolute pennies to a company with 141 million customers: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/

Meanwhile, Amazon has pricing tiers going into the petabytes for S3, and very likely receives far more than our theoretical 141TB in a single day. And they're not the size of the moon.

Backblaze wrote about it here (2011): http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ and here (2013): http://blog.backblaze.com/2013/02/20/180tb-of-good-vibrations-storage-pod-3-0/

We are constantly looking at new hard drives, evaluating them for reliability and power consumption. The Hitachi 3TB drive (Hitachi Deskstar 5K3000 HDS5C3030ALA630) is our current favorite for both its low power demand and astounding reliability. The Western Digital and Seagate equivalents we tested saw much higher rates of popping out of RAID arrays and drive failure. Even the Western Digital Enterprise Hard Drives had the same high failure rates. The Hitachi drives, on the other hand, perform wonderfully.

In the second article they say that WD-RED are 2nd in reliability (WD-RED did not exist 2011). I'm happy that I've got a cheap Hitachi Ultrastar. But who knows.

As a personal anecdote: WD-Green failure rates are huge here. 24/7 Desktop machines, 240 drives. I've replaced in last 12 month at least 20 drives.