...
At a high level, a VPC is a named container for virtual network. For example, the Synapse production VPC was assigned a CIDR of 10.2220.0.0/16. This means Synapse networks contains 65,536 IP address between 10.2220.0.0 and 10.2220.255.255. The VPC is then further divided into subnets which can be declared as either Private or Public. Instances contained in Public subnets can be assigned public IP addresses and therefore can be seen on the public internet. Conversely, instances deployed in private subnets do not have public IP address and cannot be seen by the public internet. Instance in private subnets are only visible to machines deployed within the containing VPC. We will cover how internal developers can access to machines in private subnets in a later section.
Figure 1. Synapse VPC
The Synapse VPC is divided into two six public and six twenty-four private subnets that span two six Availability Zone (AZ): us-east-1a and , us-east-b, us-east-c, us-east-d, us-east-e, us-east-f (see Figure 1). The deployment of all instances (EC2s and Databases) across two six zones ensure redundancy should an outage occur in a single zone (such as the event in April 2011), and broadens the availablity of instance types. For details on each subnet see Table 1.
...
Subnet Name | Type | CIDR | First | Last | Total | ||
---|---|---|---|---|---|---|---|
PublicUsEast1a | Public | 10.2220.0.0/2021 | 10.2220.0.0 | 10.2220.157.255 | 40962048 | ||
PublicUsEast1b | Public | 10.2220.168.0/2021 | 10.2220.168.0 | 10.2220.3115.255 | 40962048 | ||
RedPrivateUsEast1aPublicUsEast1c | Private | 10.2220.3216.0/21 | 10.2220.3216.0 | 10.2220.3923.255 | 2048 | ||
RedPrivateUsEast1bPublicUsEast1d | Private | 10.2220.4024.0/21 | 10.2220.4024.0 | 10.2220.4731.255 | 2048 | ||
BluePrivateUsEast1aPublicUsEast1e | Private | 10.2220.4832.0/21 | 10.2220.48.0 | 10.2220.5539.255 | 2048 | ||
BluePrivateUsEast1bPublicUsEast1f | Private | 10.2220.5640.0/21 | 10.20.40.0 | 10.20.47.255 | 2048 | ||
RedPrivateUsEast1a | Private | 10.22.56.0.20.48.0/24 | 10.20.48.0 | 10.20.48.255 | 256 | ||
RedPrivateUsEast1b | Private | 10.20.49.0/24 | 10.22.63.255 | 2048 | |||
GreenPrivateUsEast1a | Private | 10.22.64.0/21 | 10.22.64.0 | 10.22.71.255 | 2048 | ||
GreenPrivateUsEast1b | Private | 10.22.72.0/21 | 10.22.72.0 | 10.22.79.255 | 204820.49.0 | 10.20.49.255 | 256 |
RedPrivateUsEast1c | Private | 10.20.50.0/24 | 10.20.50.0 | 10.20.50.255 | 256 | ||
RedPrivateUsEast1d | Private | 10.20.51.0/24 | 10.20.51.0 | 10.20.51.255 | 256 | ||
RedPrivateUsEast1e | Private | 10.20.52.0/24 | 10.20.52.0 | 10.20.52.255 | 256 | ||
RedPrivateUsEast1f | Private | 10.20.53.0/24 | 10.20.53.0 | 10.20.53.255 | 256 | ||
BluePrivateUsEast1a | Private | 10.20.56.0/24 | 10.20.56.0 | 10.20.56.255 | 256 | ||
BluePrivateUsEast1b | Private | 10.20.57.0/24 | 10.20.57.0 | 10.20.57.255 | 256 | ||
BluePrivateUsEast1c | Private | 10.20.58.0/24 | 10.20.58.0 | 10.20.58.255 | 256 | ||
BluePrivateUsEast1d | Private | 10.20.59.0/24 | 10.20.59.0 | 10.20.59.255 | 256 | ||
BluePrivateUsEast1e | Private | 10.20.60.0/24 | 10.20.60.0 | 10.20.60.255 | 256 | ||
BluePrivateUsEast1f | Private | 10.20.61.0/24 | 10.20.61.0 | 10.20.61.255 | 256 | ||
GreenPrivateUsEast1a | Private | 10.20.64.0/24 | 10.20.64.0 | 10.20.64.255 | 256 | ||
GreenPrivateUsEast1b | Private | 10.20.65.0/24 | 10.20.65.0 | 10.20.65.255 | 256 | ||
GreenPrivateUsEast1c | Private | 10.20.66.0/24 | 10.20.66.0 | 10.20.66.255 | 256 | ||
GreenPrivateUsEast1d | Private | 10.20.67.0/24 | 10.20.67.0 | 10.20.67.255 | 256 | ||
GreenPrivateUsEast1e | Private | 10.20.68.0/24 | 10.20.68.0 | 10.20.68.255 | 256 | ||
GreenPrivateUsEast1f | Private | 10.20.69.0/24 | 10.20.69.0 | 10.20.69.255 | 256 | ||
OrangePrivateUsEast1a | Private | 10.20.72.0/24 | 10.20.72.0 | 10.20.72.255 | 256 | ||
OrangePrivateUsEast1b | Private | 10.20.73.0/24 | 10.20.73.0 | 10.20.73.255 | 256 | ||
OrangePrivateUsEast1c | Private | 10.20.74.0/24 | 10.20.74.0 | 10.20.74.255 | 256 | ||
OrangePrivateUsEast1d | Private | 10.20.75.0/24 | 10.20.75.0 | 10.20.75.255 | 256 | ||
OrangePrivateUsEast1e | Private | 10.20.76.0/24 | 10.20.76.0 | 10.20.76.255 | 256 | ||
OrangePrivateUsEast1f | Private | 10.20.77.0/24 | 10.20.77.0 | 10.20.77.255 | 256 |
Table 1. Synapse VPC 10.22.0.0/16 subnets
...
Given that subnet address cannot overlap and must be allocated from a fix range of IP addresses (defined by the VPC), it would be awkward to dynamical allocate new subnets for each new stack. Instead, we decided to create fixed/permanent subnets to deploy new stacks into each week, thus returning to the blue green naming scheme. Since we occasionally need to have three production-capable stacks running at a time, we included a red subnets. We will continue to give each new stack a numeric designation, but we will also assign a color to each new stack. Each color will be assigned in a round-robin manner. For example, stack 226 will be deployed to Red, 227 to Blue, 228 to Green, 229 to Red etc.. The Orange subnets are reserved for shared resources such as the ID generator database.
Public subnets
Any machine that must be publicly visible must be deployed to a public subnet. Since it is not possible to isolate public subnets from each other, there was little value in creating a public subnet for each color. Instead, one public subnet per availability zone was created. Each public subnet will contain the Elastic Beanstalk loadbalancers for each environment (portal, repo, works) of each stack. There is also a NAT Gateway deployed to each public subnet (need one NAT per AZ). We will cover the details of the NAT Gateways in a later section.
...
Destination | Target | Description |
---|---|---|
10.2220.0.0/16 | local | The default entry that identifies all address within this VPC (10.2220.0.0/16) can be found within this VPC (local) |
10.1.0.0/16 | VPN Peering Connection | If the destination IP address in the VPN VPC (10.1.0.0/16), then use the VPC peering connection that connects the two VPCs |
0.0.0.0/0 | Internet Gateway | If the destination is any other IP address (0.0.0.0/0), then use the VPC's Internet Gateway. |
...
Color Group | CIDR | First | Last | Total |
---|---|---|---|---|
Red | 10.2220.3248.0/2021 | 10.2220.3248.0 | 10.2220.4753.255 | 40962048 |
Blue | 10.2220.4856.0/2021 | 10.2220.4856.0 | 10.22.63.255 | 40962048 |
Green | 10.2220.64.0/2021 | 10.2220.64.0 | 10.22.7969.255 | 40962048 |
Table 4. CIDR for each Color Group
...
Type | CIDR/SG | Port Rang | Description |
---|---|---|---|
HTTP | 10.2220.3248.0/2021 | 80 | Allow machines within either Red private subnet to connect with HTTP |
SSH | 10.1.0.0/16 | 22 | Allows traffic from the Sage VPN (10.1.0.0/16) to connect with SSH |
HTTP | Loadbalancer Security Group | 80 | Allow the loadbalancers from this stack to connect with HTTP |
...
Type | CIDR | Port Range | Description |
---|---|---|---|
MySQL | 10.2220.3248.0/2021 | 3306 | Allows machines within either Red private subnet to connect on port 3306 |
MySQL | 10.1.0.0/16 | 3306 | Allows traffic from the Sage VPN (10.1.0.0/16) to connect on port 3306 |
...
Destination | Target | Description |
---|---|---|
10.2220.0.0/16 | local | The default entry that identifies all address within this VPC (10.2220.0.0/16) can be found within this VPC (local) |
10.1.0.0/16 | VPN Peering Connection | If the destination IP address in the VPN VPC (10.1.0.0/16), then use the VPC peering connection that connects the two VPCs |
0.0.0.0/0 | NAT Gateway us-east-1a | If the destination is any other IP address (0.0.0.0/0), then use the NAT Gateway that is within the same availability zone. |
...
Earlier we stated that instances deployed to private subnets are only visible to machines within the same VPC. This is a true statement. So how does a developer connect to an instances deployed in a private subnet? The short answer: the developer must tunnel into the target VPC. Figure 2. shows an example of an of how a developer connects to the MySQL database at 10.2220.47.107 contained within a private subnet.
...
When a developer makes a VPN connection to vpn.sagebase.org, the VPN client will add a route for 10.2220.0.0/16 on the developer's machine. This route will direct all network traffic with a destination within 10.2220.0.0/16 to Sage-Admin-Central firewall. Once connected to the VPN, an attempt to connect to 10.2220.47.107 will proceed as follows:
- The call to 10.2220.47.107 will be directed to the Sage Firewall
- If the VPN users belongs to the 'Synapse Developers' group the request will be forwarded to the SNAT
- The SNAT (a type of NAT) will replace the caller's real address with its own address: 10.1.33.107 and forward the request
- A route table entry directs all traffic with a destination of 10.2220.0.0/16 to the Synapse VPC Peering connection.
- The peering connection has been granted access to allow all traffic between the two VPC (10.1.0.0/16 and 10.2220.0.0/16) so the request can tunnel to the Synapse VPC
- Once in the Synapse VPC, requestor's address will be 10.1.33.107. Since the database security group allows 10.1.0.0/16 to connect on port 3306, the connection is allowed if the correct database username/password is provided.
...
The resources of the Synapse stack are manged as five separate cloud formation stacks templates shown in Table 9.. All template JSON files are created by merging actual templates with input parameters using Apache's Velocity.
Stack Name | Frequency | Description | Example | ||||||
---|---|---|---|---|---|---|---|---|---|
synapse-prod-vpc | one time | A one-time stack to create the Synapse VPC and associated subnets, route tables, network ACLs, security groups, and VPC peering |
| ||||||
prod-<instance>-shared-resources | once per week | Created once each week (i.e. prod-226-shared-resources) to build all of the of the shared resources of a given stack including MySQL databases and the Elastic Beanstalk Application. |
| ||||||
repo-prod-<instance>-<number> | one per version per week | Created for each version of the repository-services deployed per week. For example, if we needed two separate instances of repository services for prod-226, then the first would be repo-prod-226-0 and the second would be repo-prod-226-1. Creates the Elastic Beanstalk Environment and CNAME for the repository-services. |
| ||||||
workers-prod-<instance>-<number> | one per version per week | Created for each version of the workers deployed per week. Similar to repo-prod. Creates the Elastic Beanstalk Environment and CNAME for the workers. |
| ||||||
portal-prod-<instance>-<number> | one per version per week | Created for each version of the portal deployed per week. Similar to repo-prod. Creates the Elastic Beanstalk Environment and CNAME for the portal. |
|
Table 9. The cloudformation templates of Synapse
...