Nov 19, 2017
TIL: About AWS Organizations, trust relationships, and bastion accounts
AWS Organizations allow you to create multiple, isolated AWS accounts, centrally control access to AWS services, and consolidate billing between teams and projects. Each AWS account can be considered its own “bucket” of AWS resources (S3 buckets, EC2 instances, RDS instances, etc.).
Having multiple AWS accounts is crucial to your organization’s security. From the security perspective, a developer who has been added to the “development” AWS account would be able to create an EC2 instance in the “development” VPC, but she wouldn’t be able to terminate an EC2 instance in the “production” VPC. Similarly, the developer would not be able to delete objects in a production S3 bucket, even though she could create S3 buckets of her own.
(Note that you still need to mind your S3 bucket policies. A poorly crafted bucket policy can allow unintended team members, or even anyone, to delete objects in the bucket through its public endpoint.)
Before you create an AWS organization, you should decide how you want to split up your accounts. Some examples include:
- Each stage in your software’s development lifecycle (dev, testing, QA, production) resides in its own discrete environment
- You have teams that need limited access to all of your environments, like security auditors, or support engineers
Most importantly, each account should have an associated role e-mail address. Note that I said, “role.” You should not just assign Shelly to the “development” account, and Kurt to the “QA” account. You should create “awsdev@mycompany.com” and “awsqa@mycompany.com” e-mail addresses, and make teams or individuals leading teams principally responsible for each. Check that these e-mail addresses work before getting started.
Once you get started, the path is painfully obvious:
AWS really, really, really wants you to do this. When creating an organization, you will be prompted to create either a basic organization, or a full-featured one. Full featured organizations allow you to specify account-wide limits on service usage through Service Control Policies, or SCPs.
Once the organization is created, you can either invite existing AWS accounts into your organization, or you can create new accounts. If you create new accounts, you will need to supply the role e-mail addresses covered earlier.
Note that you can (and should) move accounts into organizational units (OUs). OUs are groups of AWS accounts (not to be confused with IAM accounts within an individual AWS account). You can apply Service Control Policies to OUs as well as to individual accounts in your organization.
Your new organization comes with one Service Control Policy (SCP): “FullAWSAccess.” This policy allows users within an AWS account to use all AWS services, within their own account. The most important thing to know about Service Control Policies is that they *override* local IAM policies set by the administrators of an account. I may create a bastion account that only has read access to each “real” account’s AWS resources. I may also want to enable the manager of this account, who also happens to be the support team lead, to add and remove users and groups within his AWS account on his own.
To help out our support team lead, I allow him full access to IAM resources, but only to IAM resources. Everything else is denied. Denying everything else through Service Control Policies prevents support team members from spinning up their own rogue EC2 instances, creating personal S3 buckets, or worse yet, circumventing organizational access policies.
Let’s examine how typical software companies are partitioned:
- Operations is responsible for production
- QA folks are responsible for staging and testing
- Developers are responsible for writing code
- Support engineers are awesome because they deflect the masses from the first three groups
Often, operations people fill in and patch holes in the staging, testing and development environments (whether they should, or not, is up to debate). Therefore, when designing my AWS accounts, I would make operations people the most powerful members of my organization and give them global access. The QA team, whose job it is to maintain a somewhat stable (but not critical) environment for software testing, are given access to staging, test, and development environments.
Developers write code and break stuff. Therefore, they should be given access only to the development environment (or AWS account). Of course, we’re assuming that developers aren’t using Docker on their own machines, but that’s a different day’s topic.
Last, but certainly not least, is support. For support, visibility into your environment is crucial. Your operations folks already have carte blanche, but your security folks need global read access.
In this case, I will create four AWS accounts: company-prod, company-test, company-dev, and company-support. I will add all of these accounts to my organization.
Recall that when I create the organization, it automatically creates a “FullAWSAccess” Service Control Policy. The first thing I’m going to do is create two Organizational Units: “operations” and “support.” Here’s where Service Control Policies and trust relationships come in.
I’ll add three AWS accounts to the “operations” OU: company-prod, company-test, and company-dev. To the “support” OU, I’ll add the company-support account. Then I will create two Service Control Policies:
- An “IAMAccess” policy, allowing members of an AWS account to take full control of IAM objects in their own account (e.g. users, groups, roles):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1511029209000",
"Effect": "Allow",
"Action": [
"iam:*"
],
"Resource": [
"*"
]
}
]
}- An “AssumeRole” policy, allowing members of an AWS account to use the sts:AssumeRole action. Note that “sts” stands for Secure Transport Services:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1511031936000",
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"*"
]
}
]
}Next, I will attach the “FullAWSAccess” policy to the “operations” OU. Attaching this policy allows members of each AWS account to control all of the AWS resources within. I can inform the team leads that they can begin working with the security team to craft their own IAM policies that restrict team members to their job duties (blast areas).
The support team is special. I’ll add the “IAMAccess” policy and the “AssumeRole” policy to the “support” OU, but I will *not* add the “FullAWSAccess” policy. Here, I’ve successfully locked support team members out of creating their own resources (or elevating themselves to root).
By employing Service Control Policies, I have successfully allowed each team into its own environment, but recall that we need to allow operations folks and support people into everything, whether they’re reading or writing. Enter trust policies.
In AWS, permissions are granted through IAM roles. IAM roles can apply to groups, users, EC2 instances, and a lot of other AWS identities, but for the purposes of this post they can apply to different AWS accounts. We’ll create IAM roles in each of our four accounts that allow different, but common, levels of access to each account’s team.
In the production AWS account, I’ll create one IAM role:
- I’ll associate it with the company-support AWS account.
- I’ll attach the “SupportUser” IAM profile (which is pre-canned!), which provides read-only access to AWS resources in the production account.
This IAM role and its associated IAM profile look like this:
NOTE: while slapping this post together, I noticed the handy link in the two screenshots, above, that I can supply to members of the support team. I am absolutely going to use these links in my real $JOB.
The “SupportUser” IAM policy is pre-canned from AWS. You can make a copy of it, if you wish, and edit your copy. Or you can just use the pre-canned policy.
Until now, I’ve demonstrated how to grant the support team full-blown, read-only access to my production AWS account. In order to give the support team all of the access that it needs, I would rinse and repeat for the company-test and company-dev accounts.
Finally, in the test and dev accounts, I would create additional IAM roles that allow “AdministratorAccess” privileges to operations folks. This allows ops folks to administer staging, test, and development environments in a pinch, and allows access between test and dev accounts as you see fit:
To set up all of my environments, I simply rinse and repeat.
Bonus time: the AWS CLI supports AWS organizations, too. This makes using tools like packer much easier. With a simple query like this:
aws organizations list-accounts --query 'Accounts[].Id' --output text223039870235 689568198667 XXXXXXXXXXXX
I can automatically share any AMIs I create with all of my accounts using packer’s ami-users statement, in my packer.json file. I can use a dumbed down AWS account to create custom AMI’s, automatically share them, and keep close tabs on waste that packer occasionally leaves around.
That’s just one use case, but if you’re like me, full access to the Organizations API means that handling your AWS accounts and permissions is a matter of scripting accounts, OUs, and Service Control Policies through your language of choice.
