Thursday January 9th, 2020


Terraform: Hands on Amazon EKS IAM Role support

Please for more info on how all this works on EKS and IAM, please refers to the post IAM Roles for EKS Service Account where eksctl has been used to demonstrate a pod using an IAM role to perform AWS API calls. And in this tutorial, I will do the same demo but using Terraform. In this demo hands on, I will create the following:

  • Kubernetes cluster and one worker node via managed node group
  • IAM OIDC provider
  • Kubernetes Service Account annotated with an IAM policy
  • And Finally a pod to test accessing some AWS services

Terraform Icon

I will be using terraform version greater or equal to 0.12.0 (in my case I am using 0.12.18 to be precise) in this tutorial to provision the necessary AWS resources and please make sure kubergrunt is installed which will be used to retrieve the OIDC thumbprint using terraform external data source.

kubergrunt: https://github.com/gruntwork-io/gruntwork-installer

Terraform: https://www.terraform.io/

Git repository located containing all files: https://github.com/kalilou/eks-iam-tutorial and the structure look the following:

├── OIDCAssumeRole.json.tmpl   # Contains the role assume policy 
├── eks_cluster.tf             # EKS cluster resources 
├── eks_worker_node.tf         # EKS managed node groups resources 
├── iam.tf                     # IAM Roles/Policies resources 
├── main.tf                    # Terraform AWS provider configuration 
├── security_group.tf          # Security group resources 
└── thumprint.sh               # Script to retrieve the OIDC thumbprint

Minor note, please replace REDACTED with the right value for your use case.

Create a kubernetes cluster and managed eks worker node group

I will assume that you have your VPC setup already. And now here the content of the terraform file for creating the EKS cluster (eks_cluster.tf).

resource "aws_eks_cluster" "this" {
  name     = "eks_iam_example"
  role_arn = aws_iam_role.cluster_role.arn

  vpc_config {
    subnet_ids = local.subnet_ids
    security_group_ids = [aws_security_group.control_plane_sg.id]
  }
  
  // You can also remove it and let terraform use the default values
  timeouts {
    create = "35m" // Default value is set to 20 minutes
    delete = "20m" // Default value is set to 15 minutes
  }

  enabled_cluster_log_types = [
    "api",
    "audit",
    "scheduler",
    "authenticator",
    "controllerManager"
  ]

  depends_on = [
    "aws_cloudwatch_log_group.this",
    "aws_iam_role_policy_attachment.AmazonEKSClusterPolicy",
    "aws_iam_role_policy_attachment.AmazonEKSServicePolicy",
  ]
}

resource "aws_cloudwatch_log_group" "this" {
  name              = "/aws/eks/eks_iam_example/cluster"
  retention_in_days = 7
  tags = {
    Name = "eks-cluster-example"
  }
}
data "external" "thumb" {
  program = ["kubergrunt", "eks", "oidc-thumbprint", "--issuer-url", aws_eks_cluster.this.identity.0.oidc.0.issuer]
}

// Resource for creating the OIDC provider
resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumb.result.thumbprint]
  url             = aws_eks_cluster.this.identity.0.oidc.0.issuer
}
resource "aws_security_group" "control_plane_sg" {
  name        = "control-plane-sg"
  description = "EKS control plane security group"
  vpc_id      = "REDACTED"

  // Allow worker node to communicate with eks cluster
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    self = true
  }

  // Allow eks to access the internet
  egress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    cidr_blocks     = ["0.0.0.0/0"]
  }
}

The resource awsiamopenidconnectprovider is the OIDC provider. This will support the federation access hence allowing to assume an IAM role via the Secure Token Service (STS).

Create a Managed Node Group

Now create the managed node group file (eksworkernode.tf) with the following content. 

resource "aws_eks_node_group" "example" {
  cluster_name    = aws_eks_cluster.this.name
  node_group_name = "eks-iam-workernode-example"
  node_role_arn   = aws_iam_role.worker_node_role.arn
  subnet_ids      = local.subnet_ids

  scaling_config {
    desired_size = 1
    max_size     = 1
    min_size     = 1
  }

  instance_types = [
    "t3.small"
  ]

  depends_on = [
    aws_iam_role_policy_attachment.AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.AmazonEC2ContainerRegistryReadOnly,
  ]
}

Create the IAM Roles/Policies

Now create the IAM roles/policies file (iam.tf) with the following contains. 

resource "aws_iam_role" "cluster_role" {
  name = "eks-iam-cluster-role"

  assume_role_policy = jsonencode({
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = {
        Service = "eks.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })

tags = {
    Name = "eks-cluster-example"
  }
}

resource "aws_iam_role_policy_attachment" "AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.cluster_role.name
}

resource "aws_iam_role_policy_attachment" "AmazonEKSServicePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.cluster_role.name
}

resource "aws_iam_role" "worker_node_role" {
  name = "eks-node-group-example"

  assume_role_policy = jsonencode({
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })

  tags = {
      eksCluster = aws_eks_cluster.this.id
      Name = "eks-cluster-example"
  }
}

resource "aws_iam_role_policy_attachment" "AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.worker_node_role.name
}

resource "aws_iam_role_policy_attachment" "AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.worker_node_role.name
}

resource "aws_iam_role_policy_attachment" "AmazonEC2ContainerRegistryReadOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.worker_node_role.name
}

Create the main Terraform file

Create now the main file containing AWS provider and terraform configuration blocks. 

provider "aws" {
  region              = "REDACTED"
  version             = "2.43.0"
}

terraform {
  required_version =  "> 0.12.0"

  backend "s3" {
    bucket   = "eks-iam-example-revolight"
    key      = "eks-iam.state"
    region   = "REGION_REDACTED"
  }
}

locals {
  subnet_ids = [
    "subnet-a1REDACTED",
    "subnet-b9REDACTED",
    "subnet-d7REDACTED"
  ]
}

Provision the resources

Now run terraform to create the specified resources. 

$ terraform init 
Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.43.0...

Terraform has been successfully initialized!
.........

$ terraform apply 

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:
................

Verify cluster is properly created

Now let's verify the cluster has been properly created, prior to the verification, I will setup my kubeconfig in order to run kubectl CLI. 

# Set up the kubeconfig by running this command below
$ aws eks --region REDACTED update-kubeconfig --name eks_iam_example --alias eks-iam
Added new context eks-iam to /Users/REDACTED/.kube/config</code>

Verify the cluster has been created.

$ aws eks list-clusters  --region REDACTED
{
  "clusters": [
      "eks_iam_example"
  ]
}

Verify the worker node has been also created.

$ kubectl get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-172-31-9-73.REDACTED.compute.internal   Ready    <none>   31m   v1.14.7-eks-1861c5

Verify the OIDC provider has been created for the cluster.

$ aws eks describe-cluster --name eks_iam_example --query cluster.identity.oidc.issuer --region REDACTED
"https://oidc.eks.REDACTED.amazonaws.com/id/296C***REDACTED***EBC3"

Create a role and a kubernetes service account

In this section, I will create a Kubernetes Service Account and an IAM role which will associated the service account and later on used by a pod to perform AWS API calls.

Create a policy template file which will be used in the IAM role creation. 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "${OIDC_ARN}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_URL}:sub": "system:serviceaccount:${NAMESPACE}:${SA_NAME}"
        }
      }
    }
  ]
}

Add these lines to the "iam.tf" file to create the IAM role.

resource "aws_iam_role" "read_only" {
  name = "AWSReadonly"
  assume_role_policy =  templatefile("OIDCAssumeRole.json.tmpl", {
    OIDC_ARN = aws_iam_openid_connect_provider.this.arn,
    OIDC_URL = replace(aws_iam_openid_connect_provider.this.url, "https://", ""),
    NAMESPACE = "default", SA_NAME = "eks-iam-example"
  })
  depends_on = [aws_iam_openid_connect_provider.this]
}

resource "aws_iam_role_policy_attachment" "read_only" {
  role       = aws_iam_role.read_only.name
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}

Apply terraform to create the role.

$ terraform apply 
aws_cloudwatch_log_group.this: Refreshing state... [id=/aws/eks/eks_iam_example/cluster]
aws_iam_role.cluster_role: Refreshing state... [id=eks-iam-cluster-role]
.............

Create Kubernetes service account yaml file which will use the IAM Role created above via annotation (service_account.yaml).

apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-iam-example
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::REDACTED:role/AWSReadonly

Create the Service Account.

$ kubectl apply -f service_account.yaml -n default
serviceaccount/eks-iam-example created

List Service Accounts.

$ kubectl get sa -n default
NAME              SECRETS   AGE
default           1         65m
eks-iam-example   1         2s

Describe Service Account eks-iam-example.

$ kubectl describe sa eks-iam-example -n default
Name:                eks-iam-example
Namespace:           default
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::943700067695:role/AWSReadonly
                      kubectl.kubernetes.io/last-applied-configuration:
                        {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{"eks.amazonaws.com/role-arn":"arn:aws:iam::REDACTED:role/AWSRead...
Image pull secrets:  <none>
Mountable secrets:   eks-iam-example-token-dk5n4
Tokens:              eks-iam-example-token-dk5n4
Events:              <none>

Create a Kubernetes deployment to assume the IAM role **via the Service Account

Create a deployment file (eksiamexample.yaml).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: eks-iam-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: eks-iam-example
  template:
    metadata:
      labels:
        app: eks-iam-example
    spec:
      serviceAccountName: eks-iam-example
      containers:
      - name: eks-iam-example
        image: sdscello/awscli:latest
        ports:
        - containerPort: 80

Create the eks-iam-example deployment.

$ kubectl apply -f kubernetes/eks_iam_example.yaml -n default
deployment.apps/eks-iam-example created

List all pods.

$ kubectl get pods -n default
NAME                               READY   STATUS    RESTARTS   AGE
eks-iam-example-854b4f5ddc-pnt64   1/1     Running   0          32s

List all environment variables to verify that AWSROLEARN and AWSWEBIDENTITYTOKENFILE have been injected. 

$ kubectl exec eks-iam-example-854b4f5ddc-pnt64 -n default env|grep AWS
AWS_ROLE_ARN=arn:aws:iam::REDACTED:role/AWSReadonly
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Run AWS CLI to list all buckets which should work as expected. 

$ kubectl exec eks-iam-example-854b4f5ddc-pnt64 -n default aws s3 ls
2019-09-16 11:08:18 REDACTED-*********************** 
2017-11-01 09:37:38 REDACTED-***********************
2018-06-27 09:06:16 thesis.kaliloudiaby.com

Run AWS CLI to create a bucket which should be denied as expected.

$ kubectl exec -it  eks-iam-example-854b4f5ddc-pnt64 -n default  bash
root@eks-iam-example-854b4f5ddc-pnt64:/  aws s3api create-bucket --bucket  eks-iam-test-revolight

An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied</code>

Cleanup

$ terraform destroy

Same tutorial using eksctl

This tutorial has been also done using eks, refer to: IAM Roles for EKS Service Account

© 2020 Revolight AB