Page Comparison

...

Prior to switching to a VPC, each EC2 and Database in the Synapse stack was issued a public IP address and was visible to the public internet.  This means the While each machine was still protected by a firewall (one or more Security Groups), SSH keys and passwords, were the only protections from hackersit was theoretically possible for hackers to probe the public visible machines for weaknesses.  By deploying machines into switching to a VPC, it becomes is possible to completely hide the EC2s and Databases instances from the public internet.

...

At a high level, a VPC is a named container for virtual network.  For example, the Synapse production VPC was assigned a CIDR of 10.22.0.0/16.  This means Synapse networks contains 65,536 IP address between 10.22.0.0 and 10.22.255.255.  The VPC is then further divided into subnets which can be declared as either Private or Public.  Instances contained in Public subnets can be assigned public IP addresses and therefore can be seen on the public internet.  Conversely, instances deployed in private subnets do not have public IP address and cannot be seen by the public internet.  In fact, instance Instance in private subnets are only visible to machines deployed within the containing VPC.  We will cover how internal developers can access to machines in private subnets in the next a later section.

Figure 1. Synapse VPC

...

The Synapse VPC is divided into two public and six private subnets that span two Availability Zone (AZ): us-east-1a and us-east-b (see Figure 1).   The deployment of all instances (EC2s and Databases) across two zones ensure redundancy should an outage occur in a single zone (such as the event in April 2011). For details on each subnet see Table 1.

Table 1.  Synapse VPC 10.22.0.0/16 subnets

...

Given that subnet address cannot overlap and must be allocated from a fix range of IP addresses (defined by the VPC), it would be awkward to dynamical allocate new subnets for each new stack.  Instead, we decided to create fixed/permanent subnets to deploy new stacks into each week, thus returning to the blue green naming scheme.  Since we occasionally , need to have three concurrently production-capable stacks running at a time, we included a red subnets.  While we We will continue to give each new stack a numeric designation, but we will now also assign a color to each new stack.  Each color will be assigned in a round-robin manner.  For example, stack 226 will be deployed to Red, 227 to Blue, 228 to Green, 229 to Red etc..

...

Any machine that must be publicly visible must be deployed to a public subnet.  Since it is not possible to isolate public subnets from each other, there was little value in creating a public subnet for each color.  One Instead, one public subnet per availability zone is the minimums number possiblewas created.  Each public subnet will contain the Elastic Beanstalk loadbalancers for each environment (portal, repo, works) of each stack.  There is also a NAT Gateway deployed to each public subnet (need one NAT per AZ).  We will cover the details of the NAT Gateways in the next a later section.

Each subnet has a Route Table that answers the question of how network traffic flows out of the subnet.  Table 2 shows the route table used by both public subnets.  Note: The most selective route will always be used.

Table 2. Public subnet route table entries

...

Each loadbalancer deployed in the public subnets is protected by the firewall called a Security Groups that blocks access to all ports except 80 (HTTP) and 443 (HTTPS).  See Table 3 for the details of the loadbalancer security groups.

Table 3. Security Groups for Elastic Beanstalk Loadbalancers deployed to public subnets

...

As stated above, any machine deployed to a private subnet is not visible from outside the VPC.  All Synapse Each EC2 and MySQL database instances are deployed within private subnets.  More specifically, when of a stack will be deployed to the permanent two private subnets matching its configured of a single color.  For example, if stack 226 is configured to be Red,  the EC2s and databases for that stack will be deployed to both of the Red private subnetsRedPrivateUsEast1a and RedPrivateUsEast1b.  The private subnets for each color are assigned CIDRs such that all of the address within both private subnets fall within a single CIDR for each color (see Table 4). This means the CIDR of each color can be used to isolate each stack.  For example, instances in red subnets cannot see instances in blue subnets, or vise versa.  Table 5 shows an example of the Security group applied to the EC2 instances within the red subnet using the color group CIDR.  Table 6 shows an example of the Security Group applied to the MySQL databases deployed in the red private subnets.

Table 4. CIDR for each Color Group

Table 5.  Security Group applied to EC2 instances in Red Private Subnets

Table 6. Security Group applied to MySQL RDS instances in Red Private Subnets

...

Just like with public subnets, each private subnet has a Route Table that describes how network traffic flows out of the subnet.  Table 7 shows an example of the Route Table on the RedPrivateUsEast1a subnet.

Table 7.  The Route Table on RedPrivateUsEast1a subnet

...

Internet Access from Within a Private Subnet

All of the EC2 instances deployed as part of Synapse Synpase need to make calls to AWS, Google, Jira, etc..  However, each instances is deployed in a private subnet without a public IP address so how does the response to and outside request make it back to the private machine?  The answer is the Network Address Translator (NAT) Gateway.  Figure 1 shows that each public subnet has a NAT Gateway, and Table 7 shows each private subnets has a route to  0.0.0.0/0 that points to the NAT Gateway within the same AZ.  Each NAT has a public IP address and acts a proxy for outgoing requests from instances in the private subnets.  When NAT receives an outgoing request from a private instance, it will replace the response address of the request with its own public address before forwarding the request.  That wayAs a result, the response goes back to the NAT, which then forwards the response to the original caller within the private subnet.  Note: The NAT will not respond to requests that originate from the outside world so it cannot be used by gain access to the private machines.

...

While setting up the Synapse VPC it was our plan to utilizes Network Access Control Lists (Network ACLs) to further isolate each private subnet.   In fact, we originally setup Network ACL similar to the security groups in Tables 3, 5, and 6.  With the Network ACLs in place the EC2 instances were unable to communicate with the outside world (for example, 'ping amazon.com failed' would hang).  The following document outlines the important differences between Security Groups and Network ACLs: Comparison of Security Groups and Network ACLs.  The explanation of the observed communication failures is summarized in Table 8..  In short, if a private subnet needs outgoing access on to 0.0.0.0/0, then you must also allow incoming access to 0.0.0.0/0 in the Network ACL.

Table 8. Important difference between Security Groups and Network ACLs

...

Earlier we stated that instances deployed to private subnets are only visible to machines within the same VPC.  This is a true statement.  So how does a developer connect to an instances deployed in a private subnet?  The short answer: the developer must tunnel into the target VPC.  Assume Figure 2. shows an example of an of how a developer needs connects to connect to the MySQL database at 10.22.47.107 as shown in Figure 2contained within a private subnet.

Figure 2. VPN tunnel to private subnet of a VPC

...

1. The call to 10.22.47.107 will be directed to the Sage Firewall
2. If the VPN users belongs to the 'Synapse Developers' group the request will be forwarded to the SNAT
3. The SNAT (a type of NAT) will replace the caller's real address with its own address: 10.1.33.107 and forward the request
4. A route table entry directs all traffic with a destination of 10.22.0.0/16 to the Synapse VPC Peering connection.
5. The peering connection has been granted access to allow all traffic between the two VPC (10.1.0.0/16 and 10.22.0.0/16) so allows the request to proceed can tunnel to the Synapse VPC
6. The request is now Once in the target Synapse VPC, with a requester address of requestor's address will be 10.1.33.107.  Since the database security group allows 10.1.0.0/16 to connect to on port 3306, the connection is established (assuming allowed if the correct database username/password was provided).is provided.

Cloudformation

The entire Synapse stack is built using the Java code from the Synapse-Stack-Builder project.  In the past, the stack builder would create AWS resources by directly calling AWS services via the AWS Java client.  One of the stated goals for switching Synapse to use a VPC included switching the stack building process to Cloudeformation templates.  While cloudformation templates can be awkward to create, using them to build/manage a stack has several advantages:

• A template can be read to discover exactly how a stack was created.
• The creation or update of a template acts as a quasi-transaction, so either all of the changes are either applied successfully or rolled-back completely.
• Cleaning up a stack is now as simple as deleting the stack.  Deleting the resources of a stack was previously a manual process.

The resources of the Synapse stack are manged as five separate cloud formation stacks templates shown in Table 9..  All template JSON files are created by merging actual templates with input parameters using Apache's Velocity.

Table 9.  The cloudformation templates of Synapse