Creating With Code

If you’ve been following all of the latest buzz about data and applications “in the cloud”, then you’re probably already familiar with Amazon’s Simple Storage Service (S3). If you’re not, or if you’ve never had the chance to play with it, then this post should help you get up, running, and able to easily store your files online, where they can be accessed from anywhere, and won’t be lost of your computer breaks or goes missing.

What is Amazon’s S3?

To put it simply, S3 is a pay-as-you-go service for storing data on Amazon’s servers. Roughly speaking, you pay for bandwidth in and out of the service as well as for data stored on the servers. For light use, bandwidth costs are about $0.10 per Gigabyte uploaded to S3, $0.17 per Gigabyte downloaded, and storage is $0.15 per Gigabyte per month. So, for example, if you uploaded 5 Gigabytes worth of pictures to S3 on May 1st and did nothing for the rest of the month, you would owe about $1.25 come June 1st (that’s $0.50 for the upload, $0.75 for the storage). Data stored on S3 is actually stored across many different servers all along Amazon’s infrastructure, making it highly unlikely that anything put there will become lost or corrupted. You can learn more at Amazon’s info page for S3.

One of the great things about S3 (and Amazon’s other web services) is that they have a well-documented API for allowing people to write programs to interact with it. In fact, many great tools have already been written to allow you to manage data on S3. The rest of this post will help you get signed up for S3 and introduce you to an excellent tool for using S3: the S3 Firefox Organizer (S3Fox).

Getting an S3 Account

If you have an Amazon account (and who doesn’t?) it’s easy to sign up.

1. If you don’t have an Amazon Web Services (AWS) developer account, sign up for one by visiting http://aws.amazon.com/ and clicking “Sign Up for AWS”. All you need is your Amazon account information and a credit card to use for billing at the beginning of every month.
2. Once you’ve created your AWS developer account, you can sign up for S3 by visiting http://aws.amazon.com/s3/ and clicking the big shiny “Sign Up For This Web Service” button. After agreeing to the terms of service, your S3 account will be all set up and ready to store your data.

Your S3 Credentials

Before you can actually use any software with your S3 account, you’ll need to get your “AWS Access Identifiers”. These identifiers (called your “Access Key ID” and your “Secret Access Key”) are long strings that verify you (or a piece of software acting on your behalf) as the owner of your S3 account. You can retrieve these by visiting http://aws.amazon.com/, clicking on “Your Web Services Account”, and selecting “AWS Access Identifiers”. Copy your identifiers somewhere safe, as you will need them for the next section.

The S3 Firefox Organizer (S3Fox)

If you use Firefox (and you should!), you can use the excellent S3Fox plugin to browse and manage data stored on one or more S3 accounts. Once you’ve set up your account in S3Fox (by entering the identifiers described above), you can view your S3 account almost like a remote drive or FTP client.

1. Install the S3Fox plugin either from the S3Fox page on the Mozilla add-ons site or from the S3Fox home page.
2. Once you’ve got the plugin installed, open it up in Firefox by opening the Tools menu and selecting S3 Organizer.
3. With the organizer open, select the Manage Accounts link in the upper-left-hand corner of the interface.
4. Give your account a name (for convenience), then enter the identifiers for your S3 account.
5. You’re ready to use S3Fox to manage files on S3!

Making your first “bucket”

Now that you’ve set up your account with S3Fox, you can begin storing data there. One of the first things to do before uploading files to S3 is to create at least one “bucket” in which to store your data. While S3Fox gives us a nice file-system-like or FTP-like interface, the fact remains that the root directories shown in S3Fox (those that appear in the right-hand pane for the “/” directory) are actually “buckets” to S3. I’ll explain more about those in a later blog post, but for now just keep in mind that an S3 bucket must be globally unique. That means that only one S3 user anywhere can have a bucket with a given name. This is similar to the concept of domain names.

You can create your first bucket by clicking the Create Directory button (the little shiny blue folder) on the right-hand pane, and giving the new directory a unique name. Some examples of a unique name could be your own domain name, or a subdomain like backups.mydomain.com, or even something random like awesome-files-for-jane-smith. You can read more about working with buckets from the S3 technical docs here.

Enter this new directory by double-clicking on it. You can now use S3Fox to browse around your local files in the left-hand pane, and select any files or directories for upload. Similarly, you can set up S3Fox on other machines you might have (like a laptop or a desktop) and download files or directories to those machines from S3.

Another nice feature of S3Fox is the ability to synchronize local folders to S3. Setting up synchronized folders allows you to easily backup directories by uploading only the files that have changed. This saves both time and money, which is a good thing.

Limitations of S3Fox

S3Fox is excellent, but has a couple of major downsides. The most painful downside is that S3Fox is not written to handle very large numbers of files at one time. If you try to upload or synchronize a directory with a huge number of files, for instance, you will notice that S3Fox basically freezes up your browser for several seconds (maybe minutes!) while it figures out which files need updating. S3Fox is also not the fastest tool for transferring files, as it pauses for a fraction of a second between each upload or download. For these reasons, I think S3Fox should only be used for dealing with a few files at a time, or for browsing to understand what is in your S3 account.

In a future post I’ll talk about S3Sync, an excellent command-line tool for uploading and downloading large numbers of files to and from S3 which has neither of these limitations (though it doesn’t have the pretty interface).

Support the Developer!

If you like S3Fox, why not support the developer by donating via the PayPal link on his page? Perhaps, with your donations and feedback, we will see S3Fox continue to improve!