Amazon VPC was introduced in 2016 to provide customers with network-level isolation for their critical workloads in the cloud. With AWS’ market share in the public cloud, the need for clients to interact privately with third-party providers to consume features instead of building them in-house has become even more critical. Though the VPC was designed for client isolation, it became a crucial means for further isolation and segmentation between different business units, departments, workloads, development teams, and even microservices. Connecting these domains privately with no exposure to the internet in a secure, scalable, and resilient manner can be achieved with AWS PrivateLink.

In this step-by-step tutorial, we will use Terraform to build and deploy a third party service and use PrivateLink to expose and consume the service from a consumer’s AWS account.

Prerequisites

To proceed with this tutorial make sure you have the following software installed:

• AWS Accounts: You will need two AWS accounts, one for the service producer and the other for the service consumer.
• AWS CLI & profile: AWS CLI is required to deploy the project to the service producer and service consumer accounts. Set up different AWS CLI profiles for each AWS account.
• Terraform: We will use Terraform to deploy the project.

To verify if all of the prerequisites are installed, you can run the following commands:

# check if the correct AWS credentials are set up for each account by specifying the profile
aws sts get-caller-identity --profile <name_of_awscli_profile>

# check if you have Terraform installed
terraform -v

# check if you have set up multiple profiles for your AWS accounts
aws configure list-profiles

‍
Architecture Overview

High-Level Architecture Diagram:

‍
PrivateLink Concepts

Service Providers:

The owner of a service is the service provider. Service providers include AWS, AWS Partners, and other AWS accounts. Service providers can host their services using AWS resources, such as EC2 instances, or using on-premises servers.

Endpoint services:

A service provider creates an endpoint service to make their service available in a Region. A service provider must specify a load balancer when creating an endpoint service. The load balancer receives requests from service consumers and routes them to your service.

By default, your endpoint service is not available to service consumers. You must add permissions that allow specific AWS principals to connect to your endpoint service.

Service names:

Each endpoint service is identified by a service name. A service consumer must specify the name of the service when creating a VPC endpoint. Service consumers can query the service names for AWS services. Service providers must share the names of their services with service consumers.

Service consumers:

The user of a service is a service consumer. Service consumers can access endpoint services from AWS resources, such as EC2 instances, or from on-premises servers.

VPC endpoints:

A service consumer creates a VPC endpoint to connect their VPC to an endpoint service. A service consumer must specify the service name of the endpoint service when creating a VPC endpoint. There are multiple types of VPC endpoints. You must create the type of VPC endpoint that's required by the endpoint service.

Endpoint policies:

A VPC endpoint policy is an IAM resource policy that you attach to a VPC endpoint. It determines which principals can use the VPC endpoint to access the endpoint service. The default VPC endpoint policy allows all actions by all principals on all resources over the VPC endpoint.

‍

Service Provider VPC

The service provider VPC is the VPC that hosts the third-party service. This VPC has the following components:

Core Infrastructure Components:

• A VPC with public subnets in two Availability Zones (AZ).
• An Internet Gateway (IGW) to provide internet access to the public subnet.
• A public route table with a default route pointing to the internet gateway.
• A NAT Gateway (NATGW) deployed in a public subnet with access to the internet.
• Two private subnets in two AZs.
• A private route table with a default route to the NATGW. This will provide outbound internet access for our application running on EC2 instances to download updates but at the same time block inbound traffic originating from the internet.

PrivateLink Components:

• EC2 instances in an Auto Scaling group that hosts the service. The instances will reside in the private subnets
• A private Network Load Balancer (NLB) which will distribute traffic to the EC2 application instances
• An Endpoint Service tied to the NLB to expose the service within the Region.

Service Consumer VPC

This VPC resides in the client’s AWS account and has the following components that would permit clients to consume the Endpoint Service:

Core Infrastructure Components:

• A VPC with private subnets in two Availability Zones (AZ).
• An EC2 instance representing a consumer.
• A VPC Interface Endpoint for Systems Manager’s Session Manager to provide private access to the EC2 instance in the private subnet.

PrivateLink Components:

• A VPC Interface Endpoint to expose the Endpoint Service.

Implementation: Setting Up the Service Provider

Terraform provider

We declare the AWS Terraform provider in the provider.tf file and specify the AWS CLI profile in the profile field for the Service Producer AWS account.

terraform {
  required_providers {
    aws = {
        source = "hashicorp/aws"
        # version = "5.14.0"
    }
  }
}

provider "aws" {
    alias = "service_provider"
    region = "eu-west-1"
    shared_credentials_files = ["~/.aws/credentials"]
    profile = "default"
    default_tags {
      tags = {
        use_case = "tutorial"
      }
    }
}

‍

The Service’s Infrastructure

We use the terraform-aws-modules/vpc/aws open source module to quickly set up a VPC with 2 public and 2 private subnets. We enable a NATGW so the service instances in the private subnet can download updates from the internet.

module "service_provider_vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = local.name
  cidr = var.vpc_cidr

  azs             = local.azs
  public_subnets  = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k)]
  private_subnets = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k + 10)]

  enable_nat_gateway = true
  single_nat_gateway = true

  enable_dns_hostnames = true
  enable_dns_support =  true 

  tags = merge(local.common_tags, {
    Name = "${local.name}-vpc"
  })
}

‍‍

The Application Security Group

The security group for the service providers application filters only HTTP and HTTPS traffic from the NLB.

resource "aws_security_group" "application_http" {
  name        = "application-http"
  description = "Allow HTTP/HTTPS traffic from consumers"
  vpc_id      = module.service_provider_vpc.vpc_id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  
    security_groups = [aws_security_group.endpoint_service.id]
  }
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]  
  }

  tags = merge (local.common_tags, {
    Name = "application-http"
  })
}

‍

The Application

The infrastructure for the application is defined in a launch template resource with an auto scaling group.

resource "aws_launch_template" "application" {
  name_prefix   = "application-product"
  image_id      = data.aws_ami.amazon_linux_2.id
  instance_type = "t2.micro"
  user_data = filebase64("${path.module}/webserver.sh")
  network_interfaces {
    associate_public_ip_address = false 
    subnet_id = element(module.service_provider_vpc.private_subnets, 0)
    security_groups             = [aws_security_group.application_http.id]
  }
  tag_specifications {
    resource_type = "instance"
    tags = merge(local.common_tags, {
      Name = local.instance_name
    })
  }

  lifecycle {
    create_before_destroy = true
  }

}
resource "aws_autoscaling_group" "application" {
  desired_capacity   = 2
  max_size           = 3
  min_size           = 2
  vpc_zone_identifier = module.service_provider_vpc.private_subnets
  launch_template {
    id      = aws_launch_template.application.id
    version = "$Latest"
  }
}

‍

The NLB Security Group

The NLB security group filters our HTTP traffic on port 80 from consumers.

 
resource "aws_security_group" "endpoint_service" {
  name        = "endpoint-service"
  description = "Allow HTTP/HTTPS traffic from consumers"
  vpc_id      = module.service_provider_vpc.vpc_id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  
  }
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]  
  }

  tags = merge (local.common_tags, {
    Name = "endpoint-service"
  })
}

‍

‍The Target Group & Listener

We create a target group for the application instances and a listener that listens to requests on TCP port 80 and forwards all the traffic to the application instances in the auto scaling group.

resource "aws_lb_target_group" "private_nlb_tg" {
  name        = "application-private-nlb-tg"
  port        = 80
  protocol    = "TCP"
  vpc_id      = module.service_provider_vpc.vpc_id
  target_type = "instance"

  health_check {
    path                = "/"
    healthy_threshold   = 2
    unhealthy_threshold = 2
    timeout             = 5
    interval            = 10
    port                = 80
    protocol            = "HTTP"
  }
  tags = local.common_tags
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_lb_listener" "private_nlb_listener" {
  load_balancer_arn = aws_lb.private_nlb.arn
  port              = 80
  protocol          = "TCP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.private_nlb_tg.arn
  }
  tags = local.common_tags
}

‍

The Network Load Balancer

We create the NLB which will receive traffic from clients and load balance to all registered instances in the autoscaling group.

resource "aws_lb" "private_nlb" {
  name               = "application-private-nlb"
  internal           = false
  load_balancer_type = "network"
  subnets            = module.service_provider_vpc.private_subnets
  security_groups    = [aws_security_group.endpoint_service.id]

  enable_deletion_protection = false
  enable_cross_zone_load_balancing = true 

  tags = local.common_tags

  lifecycle {
    create_before_destroy = true
  }
}

‍

‍Attach the Target Group to the NLB

We attach the autoscaling group with the NLB target group.

resource "aws_autoscaling_attachment" "private_nlb" {
  autoscaling_group_name = aws_autoscaling_group.application.id
  lb_target_group_arn   = aws_lb_target_group.private_nlb_tg.arn
}

‍

The PrivateLink Endpoint Service

Finally, we use the aws_vpc_endpoint_service resource to create our AWS PrivateLink Endpoint service.

We set the acceptance_required filed to false so our consumers automatically have access to the endpoint service without requiring explicit approval. This is just for demonstration purposes.

resource "aws_vpc_endpoint_service" "this" {
    acceptance_required = false # should be true in real life
    network_load_balancer_arns = [aws_lb.private_nlb.arn]
    allowed_principals = ["arn:aws:iam::${var.cross_account_id}:root"]

    tags = local.common_tags

}

‍
Setting Up the Consumer Account

Terraform Provider

We declare the AWS Terraform provider in the provider.tf file and specify the AWS CLI profile in the profile field for the service consumer AWS account.

terraform {
  required_providers {
    aws = {
        source = "hashicorp/aws"
        # version = "5.14.0"
    }
  }
}

 provider "aws" {
  region                   = var.region
  shared_credentials_files = ["~/.aws/credentials"]
  profile                  = "aai-admin"
  default_tags {
    tags = {
      use_case = "tutorial"
    }
  }

}

‍

The Consumer’s Infrastructure

We  set up a simple VPC with a private subnet. Resources in this VPC do not have any access to the internet.

resource "aws_vpc" "this" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = merge(local.common_tags, {
    Name = "${local.name}-vpc"
  })
}


resource "aws_subnet" "pri_sn1_az1" {
  vpc_id     = aws_vpc.this.id
  cidr_block = cidrsubnet(var.vpc_cidr, 8, 0)
  availability_zone       = local.az1
  tags = merge(local.common_tags, {
    Name = "pri-sn1-az1"
  })
    lifecycle {
    create_before_destroy = true
  }
}

resource "aws_route_table" "pri_rt1_az1" {
  vpc_id = aws_vpc.this.id

  tags = merge(local.common_tags ,{
    Name    = "pri-rt1-az1"
  })
}

resource "aws_route_table_association" "pri_rta1_az2" {
  subnet_id      = aws_subnet.pri_sn1_az1.id
  route_table_id = aws_route_table.pri_rt1_az1.id
}

‍

‍Security Group for Session Manager Interface Endpoint

We use security groups to filter only HTTPS traffic on port 443 required by Session Manager to privately connect to the EC2 instance we will use to test the PrivateLink Endpoint Service.

resource "aws_security_group" "ssm" {
  name        = "allow-ssm"
  description = "Allow traffic to SSM endpoint"
  vpc_id      = aws_vpc.this.id
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [aws_subnet.pri_sn1_az1.cidr_block]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
    lifecycle {
    create_before_destroy = true
  }
}

‍

Session Manager Interface Endpoints

To privately connect to the EC2 instance using Session Manager, we need to set up Interface endpoints to ec2messages, ssm and ssmmessages services. Since all the services will be used as one, we declare their endpoint addresses as local variables and use a single aws_vpc_endpoint Terraform resource to configure them.

  ssm_services = {
  "ec2messages" : {
    "name" : "com.amazonaws.${var.region}.ec2messages"
  },
  "ssm" : {
    "name" : "com.amazonaws.${var.region}.ssm"
  },
  "ssmmessages" : {
    "name" : "com.amazonaws.${var.region}.ssmmessages"
  }
}

resource "aws_vpc_endpoint" "ssm_ep" {
  for_each = local.ssm_services
  vpc_id   = aws_vpc.this.id
  ip_address_type     = "ipv4"
  vpc_endpoint_type   = "Interface"

  service_name        = each.value.name
  security_group_ids  = [aws_security_group.ssm.id]
  private_dns_enabled = true
  subnet_ids          = [aws_subnet.pri_sn1_az1.id]
}

‍

‍Security Group for PrivateLink Endpoint Service’ Interface Endpoint

We set up another security group to filter only HTTP traffic on port 80 to the VPC Interface Endpoint which our EC2 instance will use to access the Endpoint Service.

resource "aws_security_group" "privateLink_service" {
  name        = "privateLink-service"
  description = "Security group for privateLink Interface Endpoint"
  vpc_id      = aws_vpc.this.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = merge(local.common_tags,{
    Name = "privateLink-service"
  })

  lifecycle {
    create_before_destroy = true
  }
}

‍

‍PrivateLink Endpoint Service’ Interface Endpoint

We create the VPC Interface Endpoint. We use the service_name filed to pass a variable which holds the name of the Endpoint Service.

resource "aws_vpc_endpoint" "privateLink_service" {
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = false
  vpc_id              = aws_vpc.this.id
  service_name        = var.privateLink_service_name
  security_group_ids  = [aws_security_group.privateLink_service.id]
  subnet_ids          = [aws_subnet.pri_sn1_az1.id]

  tags = merge(local.common_tags,{
    Name = "privateLink-service"
  })
}

‍

‍Security Group for the Consumer’s Client

The security group for our EC2 test instance allows only HTTPS traffic on port 443 so Session Manager can access it. This is the only traffic that is allowed inbound. With AWS PrivateLink, only consumers can initiate a request to the provider.

resource "aws_security_group" "ssm_client" {
  name        = "ssm-client"
  description = "allow traffic from SSM Session maanger"
  vpc_id      = aws_vpc.this.id
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "TCP"
    cidr_blocks = [aws_subnet.pri_sn1_az1.cidr_block]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  tags = merge(local.common_tags,{
    Name = "ssm-client"
  })
    lifecycle {
    create_before_destroy = true
  }
}

‍

‍Consumer’s Client

We  create an Instance profile for the test EC2 instance and attach the AmazonSSMManagedInstanceCore AWS Managed Policy which has the necessary permissions to enable AWS Systems Manager service core functionality.

resource "aws_iam_role" "ec2_exec_role" {
  name = "ec2-exec-role"

  assume_role_policy = jsonencode(
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
})

  tags = merge(local.common_tags, {
    tag-key = "ec2-exec-role"
  })
}


resource "aws_iam_policy_attachment" "ssm_manager_attachment" {
  name       = "ec2-exec-attachement"
  roles      = [aws_iam_role.ec2_exec_role.name]
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

resource "aws_iam_instance_profile" "ec2_instance_profile" {
  name = "ec2-instance-profile"
  role = aws_iam_role.ec2_exec_role.name
}

‍We define the necessary configurations for the EC2 instance in a private subnet.

resource "aws_instance" "privateLink_consumer" {
  ami                    = local.ami
  instance_type          = "t2.micro"
  subnet_id              = aws_subnet.pri_sn1_az1.id
  associate_public_ip_address = false
  key_name = "default-euw1"
  vpc_security_group_ids = [aws_security_group.ssm_client.id]
  iam_instance_profile = aws_iam_instance_profile.ec2_instance_profile.name

  tags = merge(local.common_tags, {
    Name = local.instance_name
  })
}


‍Deployment

Clone the Repository

git clone https://github.com/FonNkwenti/tf-cross-account-privateLink.git
cd tf-cross-account-privateLink/

‍

‍Deploy the Service Provider

Go to the service-provider directory.

tf-cross-account-privateLink git:(main) ✗ cd service-producer/

Update the variables.tf with your variable preferences or leave the defaults.

Update the locals.tf with your own local variables or work with the defaults.

Create a terraform.tfvars file and pass in values for some of the variables.

region                      = "eu-west-1"
account_id                  = 123456789012
cross_account_id            = 345678901234
environment                 = "dev"
project_name                = "tf-cross-account-privateLink"
service_name                = "privateLink-service"
cost_center                 = "237"

Initialize Terraform.

service-producer git:(main) ✗ terraform init

Deploy the service-provider stack.

service-producer git:(main) ✗ terraform apply --auto-approve

Copy the value of the privateLink_service_name output after the deployment is completed. We will pass the Endpoint Service name as a variable when deploying our Consumer VPC.

privateLink_service_name = "com.amazonaws.vpce.eu-west-1.vpce-svc-058a2bf106bf77968"

‍

Deploy the Service Consumer

Go to the cross-account-service-consumer directory.

service-producer git:(main) ✗ cd ../cross-account-service-consumer/

Update the variables.tf with your variable preferences or leave the defaults

Update the locals.tf with your own local variable preferences or work with the defaults

Create a terraform.tfvars  file and pass in values for some of the variables. Make sure you copy and pass the value of the privateLink_service_name variable from the Terraform output in the Service Provider deployment.

region                      = "eu-west-1"
account_id                  = 123456789012
cross_account_id            = 345678901234
privateLink_service_name    = "com.amazonaws.vpce.eu-west-1.vpce-svc-058a2bf106bf77968"
environment                 = "dev"
project_name                = "tf-cross-account-privateLink"
service_name                = "privateLink-service"
cost_center                 = "237"

‍Initialize Terraform.

cross-account-service-consumer git:(main) ✗ terrraform init

Deploy the service-provider stack.

cross-account-service-consumer git:(main) ✗ terraform apply --auto-approve

After the deployment is completed, copy the value of the session_manager_link and interface_endpoint_dns_name from the Terraform output. We will use the session_manager_link to open up a session to our consumer’s EC2 client to test connectivity to the Endpoint Service in the service provider’s account.

session_manager_link = "https://console.aws.amazon.com/systems-manager/session-manager/i-079d5d8918970a57a"
interface_endpoint_dns_name = "vpce-02623c0267accd034-0p68ymlz.vpce-svc-058a2bf106bf77968.eu-west-1.vpce.amazonaws.com"

‍Testing the Setup

We will open up a session to the EC2 instance in the consumer’s account and test connectivity to the service provider using CURL.

The rest will be run in the service consumers account so you must be logged into it to avoid confusion. Follow the steps below.

        1. Copy the session manager link gotten from the Terraform output after deploying the cross account service consumer. Paste in in your web browser to open up a session to the instance.

    2. Test connectivity using CURL to the interface endpoint copied from the Terraform output of the service consumer deployment.

‍

Troubleshooting Tips

• Service Provider and Consumer VPCs are deploying to the same AWS Account:
  
  • Make sure you have two AWS accounts.
  • Make sure you create IAM users with sufficient permissions.
  • Make sure you download the Secret Access Key and Access Key ID for the IAM users.
  • Make sure you used the AWS configure —profile <<profile_name_of_account_1>> in your terminal to set up profiles for each AWS Account.
  • Make sure you updated the provider.tf file with the AWS profile names accordingly.
  • Make sure that you create a terraform.tf vars file for each folder and update the values of the account_id with the AWS account ID of the service provider and the cross_account_id with that of the service consumer account.
• I am prompted to provide values for variables when I run terraform apply
  • This happens because you forgot to create the terraform.tfvars file. Make sure you create the file and update the values.

‍

Clean Up Resources

Remove all resources created by Terraform in the Service Consumer's account

1. Navigate to the cross-account-service-consumer directory:

cd cross-account-service-consumer/

        2. Destroy all Terraform resources:

terraform destroy --auto-apply

Remove all resources created by Terraform in the Service Producers's account

1. Navigate to the service-producer directory:

cd service-producer/

        2. Destroy all Terraform resources:

terraform destroy --auto-apply

‍

Conclusion

We’ve shown the steps required for a third-party service provider to expose their services to consumers in different AWS accounts in a safe manager through AWS PrivateLink. We used Terraform as our IaC tool but the same steps can be applied with any other IaC tools. As usual with any project in the cloud, you must think about security and provide only least privilege access and implement security in depth. Monitoring and logging as well as cost optimization are important to keep in mind as well.

‍

Additional Resources

https://github.com/FonNkwenti/tf-cross-account-privateLink

Integrating third-party services in the AWS Cloud - AWS Prescriptive Guidance

What is AWS PrivateLink? - Amazon Virtual Private Cloud

AWS PrivateLink Pricing – AWS

‍