Sunday January 16, 2011

Storing backups is a long-standing S3 use case but, until the release of IAM the only way to use S3 for backups was to use the same credentials you use for everything else (launching EC2 instances, deploying artifacts from S3, etc). Now with IAM, we can create users with individual credentials, create user groups, create access policies and assign policies to users and groups. Here's how we can use boto to create granular access controls for S3 backups.

So let's create a user and a group within our AWS account to handle backups. Start by pulling and installing the latest boto release from github. Let's say you have an S3 bucket called "backup-bucket" and you want to have a user whose rights within your AWS infrastructure is confined to putting, getting and deleting backup files from that bucket. This is what you need to do:

1. Create a connection to the AWS IAM service:

   import boto
   iam = boto.connect_iam()
   

2. Create a user that will be responsible for the backup storage. When the credentials are created, the access_key_id and access_key_id components of the response will be necessary for the user to use them, save those values:

   backup_user = iam.create_user('backup_user')
   backup_credentials = iam.create_access_key('backup_user')
   print backup_credentials['create_access_key_response']['create_access_key_result']['access_key']['access_key_id']
   print backup_credentials['create_access_key_response']['create_access_key_result']['access_key']['secret_access_key']
   

3. Create a group that will be assigned permissions and put the user in that group:

   iam.create_group('backup_group')
   iam.add_user_to_group('backup_group', 'backup_user')
   

4. Define a backup policy and assign it to the group:

   backup_policy_json="""{
     "Statement":[{
         "Action":["s3:DeleteObject",
           "s3:GetObject",
           "s3:PutObject"
         ],
         "Effect":"Allow",
         "Resource":"arn:aws:s3:::backup-bucket/*"
       }
     ]
   }"""
   created_backup_policy_resp = iam.put_group_policy('backup_group', 'S3BackupPolicy', backup_policy_json)
   

Note: the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables need to be set or they need to be provided as arguments to the boto.connect_iam() call (which wraps the boto.iam.IAMConnection ctor).

Altogether Now:

import boto
iam = boto.connect_iam()
backup_user = iam.create_user('backup_user')
backup_credentials = iam.create_access_key('backup_user')
print backup_credentials['create_access_key_response']['create_access_key_result']['access_key']['access_key_id']
print backup_credentials['create_access_key_response']['create_access_key_result']['access_key']['secret_access_key']
iam.create_group('backup_group')
iam.add_user_to_group('backup_group', 'backup_user')
backup_policy_json="""{
  "Statement":[{
      "Action":["s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Effect":"Allow",
      "Resource":"arn:aws:s3:::backup-bucket/*"
    }
  ]
}"""
created_backup_policy_resp = iam.put_group_policy('backup_group', 'S3BackupPolicy', backup_policy_json)

While the command line tools for IAM are OK (Eric Hammond wrote a good post about them in Improving Security on EC2 With AWS Identity and Access Management (IAM)), I really prefer interacting with AWS programmatically over remembering all of the command line options for the tools that AWS distributes. I thought using boto and IAM was blog-worthy because I had a hard time discerning the correct policy JSON from the AWS docs. There's a vocabulary for the "Action" component and a format for the "Resource" component that wasn't readily apparent but after some digging around and trying some things, I arrived at the above policy incantation. Aside from my production server uses, I think IAM will make using 3rd party services that require AWS credentials to do desktop backups much more attractive; creating multiple AWS accounts and setting up cross-account permissions is a pain but handing over keys for all of your AWS resources to a third party is just untenable.

In the meantime, if you're using your master credentials for everything on AWS, stop. Adopt IAM instead! To read more about boto and IAM, check out

• IAM documentation
• boto.iam. IAMConnection
• Using Identity & Access Management (IAM) Service in boto