In networked environments, Domain Name System (DNS) plays an essential role by translating user-friendly domain names into IP addresses that systems use to communicate. This makes interactions smoother, as services can be accessed through recognizable and consistent names rather than numeric IP addresses, which may change over time or across deployments. For businesses and developers using AWS, Amazon Route53 offers a robust DNS service that supports high availability, fault tolerance, and ease of management.

However, enabling private DNS communication between Amazon Virtual Private Clouds (VPCs) poses unique challenges, particularly when resources such as EC2 instances or load balancers are in private subnets with no internet access. By default, each VPC operates in an isolated manner, meaning instances within one VPC cannot communicate directly with those in another VPC unless specific configurations are made. This isolation can complicate scenarios where resources need to communicate using custom DNS names, rather than relying on AWS-assigned private DNS or IP addresses that may not meet application-specific naming or security requirements.

This tutorial explores how Amazon Route53 Private Hosted Zones (PHZ) can be configured to allow private DNS resolution across VPCs.

In the first part, we will look at how to create a PHZ for an EC2 instance using an “A” DNS record and in the second part, we will create a PHZ for an “Alias” record for an internal Application Load Balancer (ALB).

Let’s dive into how you can implement this configuration step-by-step using Amazon Route 53!

Key Services and Concepts

Amazon Route53 Hosted Zones

In Amazon Route53, hosted zones are containers for DNS records, that contain information about how to route traffic for specific domains and subdomains.

Amazon Route53 Private Hosted Zones (PHZs) are hosted zones in AWS Route53 that are specifically designed for managing private DNS within a Virtual Private Cloud (VPC) environment. Unlike public hosted zones, which allow DNS queries to be resolved over the internet, PHZs are strictly accessible within the VPCs associated. This makes them ideal for use cases where private resources need to communicate using DNS names but shouldn’t be exposed to the public internet.

A Record

A records are used to route traffic to a single resource like a web server or database using IPv4 address in dotted decimal notation. We will use this an A record to map a friendly DNS name to the IPv4 address of an EC2 instance.

AWS Transit Gateway

AWS Transit Gateway connects your Amazon Virtual Private Clouds (VPCs) and on-premises networks through a central hub. This connection simplifies your network and puts an end to complex peering relationships. Transit Gateway acts as a highly scalable cloud router—each new connection is made only once.

Architecture Overview

In the first part, we have two VPCs: Client and Services VPCs. The Service VPC has a service running on an EC2 instance and the Client VPC has an EC2 instance client. Both VPCs are attached to, and interconnected via an AWS VPC Transit Gateway (TGW). The traffic flow is “east-west” via the TGW. A PHZ is created for the EC2-based service in the Services VPC with an “A” record mapping the IPv4 of the instance, 10.15.0.100 with a human-readable DNS name, app.myservice.internal.  This PHZ is then shared with the Client VPC so the client EC2 instance can use the human-friendly name to reach the EC2-based service.

High-Level Architecture Diagram

Walkthrough

An EC2 instance client that needs to connect to a service in the Service VPC via a human friendly DNS name, (app.myservice.internal)

1. The EC2 instance first needs to resolve the custom domain name by querying the VPC + 2 resolver. The PHZ for app.myservice.internal is associated with the Client VPC so the custom domain name will be resolved to the layer 3 static IPv4 address.
2. Once the EC2 instance obtains the IP address from the DNS resolution, it sends the traffic to the TGW’s Elastic Network Interface (ENI) as per its route table.
3. The TGW ENI forwards the packet to the TGW.
4. As per the TGW Shared Services route table, the packet is forwarded to the Service VPC.
5. The TGW ENI in the Service VPC forwards the packet to the ENI of the EC2 instance-based service.
6. The response packet from the EC2 instance-based service is sent back to the TGW ENI.
7. The packet is sent to the TGW.
8. As per the Shared Services route table associated with the Service VPC attachment, the traffic is forwarded to the Client VPC.
9. The response packet is sent by the TGW ENI in the Client VPC to the client EC2 instance’s ENI.

Implementation

Prerequisites

To proceed with this tutorial make sure you have the following software installed:

• AWS Account
• AWS CLI
• Terraform

To verify if all of the prerequisites are installed, you can run the following commands:

# check if the correct profile and access key is setup
aws configure 

# Easy obtain your AWS Account ID
aws sts get-caller-identity 

# check if you have Terraform installed
terraform -v

‍
Step 1: Layer 3 Reachability Between Client & Service VPCs

The purpose of DNS resolution is to enable client resources to use human-friendly names to reach services but there must IP reachability between those resources. In our scenario, IP reachability between the the VPCs is achieved via an AWS Transit Gateway. The important points to note about the VPC interconnectivity are;

• The client EC2 instance must belong to a private subnet with an associated route table which must have a route with a target of the TGW’s ENI. To be precise, the route points the Service VPC’s CIDR block to the TGW for effective packet forwarding. We achieve this by installing static routes in both VPCs.

resource "aws_route" "services_to_client" {
  count                  = length(module.services_vpc.private_route_table_ids)
  route_table_id         = element(module.services_vpc.private_route_table_ids, count.index)
  destination_cidr_block = local.client_vpc_cidr
  transit_gateway_id     = aws_ec2_transit_gateway.main_tgw.id
}

resource "aws_route" "client_to_services" {
  count                  = length(module.client_vpc.private_route_table_ids)
  route_table_id         = element(module.client_vpc.private_route_table_ids, count.index)
  destination_cidr_block = local.services_vpc_cidr
  transit_gateway_id     = aws_ec2_transit_gateway.main_tgw.id
}

‍


• At the level of the TGW, both Client and Service VPCs attachments must be in the same route table. We also install static routes for effective packet forwarding.

resource "aws_ec2_transit_gateway_route_table_association" "service_vpc_association" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.service_vpc_attachment.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.services_rt.id
}

resource "aws_ec2_transit_gateway_route_table_association" "client_vpc_association" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.client_vpc_attachment.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.services_rt.id
}

resource "aws_ec2_transit_gateway_route" "service_to_client" {
  destination_cidr_block         = local.client_vpc_cidr
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.services_rt.id
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.client_vpc_attachment.id
}

resource "aws_ec2_transit_gateway_route" "client_to_service" {
  destination_cidr_block         = local.services_vpc_cidr
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.services_rt.id
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.service_vpc_attachment.id
}

Step 2: Define the Custom Domain

The next step is to create a Route 53 Private Hosted Zone (PHZ) with a custom domain name, such as myservice.internal. This PHZ will allow us to set up DNS names that are only accessible within the associated VPCs, ensuring secure and isolated name resolution for internal services.

• Create and associate a PHZ with the Service VPC.

resource "aws_route53_zone" "my_service" {
  name     = var.app_name
  vpc {
    vpc_id = module.services_vpc.vpc_id
  }
  comment = "Private hosted zone for ${var.app_name}"

  tags = {
    Name = "${var.app_name}-phz"
  }

  lifecycle {
    create_before_destroy = true
    ignore_changes = [ vpc ]
  }
}

AWS automatically creates a set of name servers for the hosted zone, but unlike public hosted zones, these name servers aren’t reachable outside the associated VPCs. With the private hosted zone established and associated with the server VPC, we now have a custom DNS space (service.internal) in which we can create private DNS records for the server resources.

Step 3: Create the DNS Record

Create a DNS A record to map the EC2-based service instance’s private IP address to a custom domain name, app.myservice.internal.

resource "aws_route53_record" "my_app" {
  zone_id = aws_route53_zone.my_service.id
  name    = "app.${var.app_name}"
  type    = "A"
  ttl     = 300
  records = [ module.services_instance.private_ip ]

  lifecycle {
    create_before_destroy = true
  }
}

‍


In the next step, we’ll associate the hosted zone with the client VPC to enable cross-VPC access to these DNS records.

Step 4: Share the Private Hosted Zone with the Client VPC

To enable DNS-based communication between instances in the Client VPC and Server VPC, we’ll need to associate the Route 53 (PHZ) created in Step 2 with the Client VPC. By doing this, DNS queries originating in the Client VPC will be able to resolve records in the private hosted zone, allowing seamless communication using the custom DNS names defined for resources in the Server VPC.

resource "aws_route53_zone_association" "client_vpc_association" {
  zone_id = aws_route53_zone.my_service.id
  vpc_id  = module.client_vpc.vpc_id

  lifecycle {
    create_before_destroy = true
  }
}

Once the Client VPC is associated, any DNS query from an instance in the Client VPC for a record in the private hosted zone (such as app.myservice.internal) will be resolvable, as long as the corresponding record is created in Route 53.

Deployment

Clone the Repository

git clone https://github.com/FonNkwenti/tf-route53-phz-cross-vpc.git
cd tf-route53-phz-cross-vpc/

Open the Directory to the A_Record Project

Go to the A_Record directory.

tf-route53-phz-cross-vpc git:(main) ✗ cd A_Record/

• Update the variables.tf with your variable preferences or leave the defaults.
• Update the locals.tf with your own local variables or work with the defaults.
• Create a terraform.tfvars file and pass in values for some of the variables.

main_region              = "eu-west-1"
account_id               = 123456789123
environment              = "dev"
project_name             = "tf-route53-phz-cross-vpc"
service_name             = "services.internal"
cost_center              = "237"
ssh_key_pair             = "my_key_pair"

Initialize Terraform.

A_record git:(main) ✗ terraform init

Deploy the project.

A_record git:(main) ✗ terraform apply --auto-approve

Copy the output values after the deployment is completed. We will use them to test the cross VPC DNS resolution.

‍


Testing DNS Resolution from the Client VPC

We will use EC2 Instance connect Endpoint to private connect to the Client EC2 instance in the Client VPC to run our tests.

To ensure everything is set up correctly, we’ll test DNS resolution from an instance within the Client VPC. This step verifies that the instances in the Client VPC can access the custom DNS records created in the PHZ and can connect to the servers in the Server VPC using those DNS names.

     1. Log In to an Instance in the Client VPC:

• SSH into an instance within the Client VPC. Make sure the instance is in a subnet with routing configured to allow traffic to the Server VPC.

     2. Test DNS Resolution:

• Use DNS commands like ping,  nslookup, or curl to verify that the custom DNS name resolves correctly to the server’s private IP.
• Using ping:

Successful output should show that server.myapp.internal resolves to the server’s private IP.

• Using nslookup (for more detailed DNS resolution info):

The output should display the IP address mapped to server1.myapp.internal.

• Using curl:
• This command will display the server’s IP address and confirm that the name resolution is happening correctly through the private hosted zone.

Troubleshooting Tips

• DNS Resolution Fails:
  • Ensure the private hosted zone is associated with both the Server and Client VPCs.
  • Check that DNS resolution is enabled in both VPCs. Since we are using the terraform-aws-modules/vpc/aws open source modules to create the VPCs, ensure that the values for enable_dns_hostnames and enable_dns_hostnames are set to true.
• Connectivity Issues:
  • Use the AWS Network Manager’s Reachability Analyzer to pinpoint where there is a connectivity failure along the network path between the client instance and the service instance.
  • Verify that the security groups and network ACLs on both the client and server instances allow traffic between the VPCs.
  • Make sure VPC peering connections are properly set up, and relevant routes are added to route tables.
• Terraform Deployment Issues:
  • Make sure you are using a valid SSH Key pair in the region where you are deploying the project as this step is not explicitly described in this tutorial.

Clean Up Resources

Remove all resources created by Terraform in the Service Consumer's account

1. Navigate to the  A_Record directory:

cd tf-Route 53-phz-cross-vpc/A_Record/

        2. Destroy all Terraform resources:

terraform destroy --auto-apply

Summary and Key Takeaways

In this tutorial, we walked through how to set up cross-VPC DNS resolution using Amazon Route53 Private Hosted Zones (PHZ). By creating a PHZ, associating it with both the Client and Server VPCs, and adding custom DNS records, we enabled instances in a Client VPC to access resources in the Server VPC using easy-to-remember custom DNS names. Here’s a quick recap of the process:

1. VPC-to-VPC Connectivity: Established layer 3 connectivity with a Transit Gateway.
2. Private Hosted Zone Setup: Created a Route53 PHZ in the Server VPC and associated it with the Client VPC.
3. Custom DNS Records: Added an A record for the private hosted zone, mapping custom DNS names to the private IP of our Service EC2 instance.
4. Testing: Verified the setup by testing DNS resolution from the Client VPC using standard networking commands like ping, and nslookup, and curl.

Next Steps

In the next part, we will focus on a typical scenario with an internal Application load balancer and show how to setup a an Alias Record for its automatically assigned DNS name.

Additional Resources

• AWS Documentation on Route 53 Private Hosted Zones
• AWS Transit Gateway
• DNS Settings in AWS VPCs
• GitHub Repo

These resources provide further insights into DNS management, VPC networking, and Route 53 configurations.