Eric Hammond

What I like most about developing on EC2 is the ability to save an development environment as an Amazon Machine Image (AMI), and then spin up instances of it as needed. I can fine-tune the environment over time (just spin up a new instance, make changes, and save a new AMI), as well as have different environments for different types of development.

I think you could do something similar with Slicehost/Linode using disk images and configuration profiles, but it doesn't seem they offer anything like Amazon's machine images. Correct me if I'm wrong, though.

Of course, the flip-side of all this is that any individual instance of an AMI is inherently ephemeral, and you need to treat it as such. Even the reliability claims of EBS are fairly weak. [1] That means EC2 instance storage, even EBS storage attached to an instance, isn't truly persistent, and I haven't quite ironed out my system for dealing with this.

You'll probably want to run your own Git server [2] and commit code as it's completed. That still leaves the possibility of data loss if your instance dies after you've written code, but before you've made a commit. You could just take periodic snapshots of the EBS volume where your in-progress code resides, but whatever period you choose for the frequency (like every 30 minutes) is equal to the amount of work you could lose. And that sort of constant backup isn't really what snapshots are meant for anyway. [3] I think the ideal solution would be to run Dropbox to get…

I know this is a little late for your case but you can migrate snapshots from 1 region to another. We spend sunday moving from US-East 1 to US-West 1 and took 2 hours to setup our entire infrastructure consisting of 20 servers & about 180 GB database

There are several options

1. Manually - http://alestic.com/2010/10/ec2-ami-copy

2. Automatically via Scripts
a) migrate-ebs-image.pl - http://search.cpan.org/~lds/VM-EC2/bin/migrate-ebs-image.pl
b) CloudyScript Migrate SnapShot - https://cloudyscripts.com/tool/show/4
c) CloudyScript Migrate AMI - https://cloudyscripts.com/tool/show/5

3. Commercial Services
a) http://ylastic.com/
b) RightScale

* ylastic is really cheap at $25 / month
* I used the perl script migrate-ebs-image.pl, super simple to install and use.
* CloudyScripts has open sourced their Ruby gem so you can build on top of it. They also have a free web form you can just use (but may not be secure enough). But you can launch their AMI in your own instance which should be secure

Hope that helps for next time!

I've recently been looking at it too, and trying to find clarification of the retrieval fee. It seems there's a lot of (potential) misinformation, though I'm still trying to verify that.

My guess is the best idea is to upload to S3, using whatever tools you can for that, and then archive from S3 to Glacier using the AWS lifecycle rules.

See previous discussion at http://news.ycombinator.com/item?id=4411536 and http://liangzan.net/aws-glacier-calculator/ - my concern is that they don't seem to match Amazon's Glacier calculator at http://calculator.s3.amazonaws.com/calc5.html

http://alestic.com/2012/12/s3-glacier-costs also seems relevant.

So be sure you know what you're getting yourself into, otherwise you could end up with thousands of USD in billing.

If you get clarity on the retrieval fee, please let me/us know.

Two reasons come to mind:

* the failure rate of an EBS volume (assuming a bug-free EBS software stack!) should be less than any one spinning disk

* these machines don't have disk space set aside for copying the root volumes to them from S3 to boot. (There's a 10GB limit, and to allow for this to be somewhere, you'd have to have another SSD, or partition the existing ones.)

There's a good blog post about it on Rightscale: http://blog.rightscale.com/2012/11/02/new-ec2-instance-types-and-coordinated-failures-in-the-cloud/

esh's "You Should Use EBS Boot Instances" also lays out a number of good reasons to use EBS for boot: http://alestic.com/2012/01/ec2-ebs-boot-recommended

You at least have two 120GB ephemeral SSDs to play with when you're running - a luxury you don't have on the new M3 classes.

This article includes a script that will perform a consistent snapshot of an XFS-formatted EBS drive (in linux).

http://alestic.com/2009/09/ec2-consistent-snapshot

Consistent snapshots are needed when the disk contains multiple files that must be kept in sync with each other - such as a database's data files and the journal. If your DB gets its disk snapshotted while these files are out of sync, the DB will need to recover the missing data by replaying the journal. Not all DB's can do this reliably for all types of changes (e.g. MySQL's MyISAM engine is fragile in this regard). So you need to make sure that the files in the snapshot all reflect a consistent state of the disk.

Very cool. Out of curiosity, are there any plans to make it easy to provision/create AWS boxes securely via chroot via Eric Hammond's approaches at Alestic? Currently it is a very tedious process to make a secure image on Amazon so that no credentials leak into history and whatnot, it is a tedious process. Making this easy would go a long way to making it trivial to use vagrant to safely publish public boxes for EC2.

More details on this problem here:
http://news.ycombinator.com/item?id=4457336

Examples of Eric Hammond's process at Alestic:
http://alestic.com/alestic-git/
https://github.com/alestic/alestic-git