The proposed solution to PLFM-6526 was to was to limit the public’s access to our staging stacks. We created PLFM-6786 as a project to explore our technical options for limiting access to staging. The following document is the result of that exploration.

Existing Beanstalk Stacks

Each week, we deploy a new instance of Synapse that is composed of four separate cloud formation stacks with each part identified with an incremental instance number. For example, a stack with an instance number of 101 would have the following cloud formation stacks:

• prod-101-shared-resource - This stack defines all of the back-end shared resources that supports Synapse, including databases, search indexes, sqs queues, sns topics, etc..

• portal-prod-101-0 – The Elastic Beanstalk stack for the main Synapse UI for www.synapse.org.

• repo-prod-101-0 – The Elastic Beanstalk stack for the Synapse REST API for: repo-prod.prod.sagebase.org.

• workers-prod-101-0 – The Elastic Beanstalk stack for the worker machines that support the Synapse API. Note: Even though this is deployed as an EB, it does not respond to web-service requests.

In this document we will be focusing on the two EB stacks: portal & repo, both of which provide web-services for Synapse.

Open

Figure 1.

Figure 1 shows two instances: 101 (prod) & 102 (staging) elastic beanstalk stacks that can exist at the same time. The 101 stack is accessible via the www.synapse.org CNAME, while the 102 stack is accessible via the staging.synapse.org CNAME. Each EB stack is composed of a self-contained Application Load Balancer (ALB) and a scaling group of all of the EC2s machines for that stack.

Notice that both the prod and staging ALBs are deployed in a public subnet, while the EC2s are deployed in private subnet. This means the ALBs are publicly accessible while the EC2s are not. With this architecture the public can access staging.

Deployment

With the existing architecture, deploying a new stack is as simple as running a stack-builder script providing a new instance number. Once the stack is built, the staging CNAME is changed to point to the new stack, thereby making that stack accessible to the public.

Promoting Staging to Production

With the existing architecture, promoting the staging stack (102) to become production is as simple as assigning the production CNAME to stack 102’s ALB. All new web-services requests will then start going to the 102 stack. At that point, the old production stack (101) can be deleted.

Option A

Option A is a slight variation on the existing architecture. See Figure 2.

Open

Figure 2.

Deployment

A new staging stack would be deployed by running a stack-builder script with a new instance number (102) and parameter of ‘isPublic=false’. The script would deploy the new EB’s ALB into the private subnet since it was passed ‘isPublic=false’. The staging CNAME would then be assigned to 102’s ALB. Since this ALB is in the private subnet, it will not be publicly visible. In fact, a user would need to connect to the Sage VPN in order to access 102.

Promoting Staging to Production

With Option A, the promotion of 102 to be production is not as simple as changing a CNAME. Unfortunately, Elastic Beanstalk does not allow an existing ALB to be moved from the private subnet to the public subnet. To make stack 102 publicly visible we would need to deploy a new 102 stack instance with the parameter ‘isPublic=true’. After the new public 102 is validated, we would then be able to change the production CNAME to point to the new 102 public ALB.

Option B

Option B involves building two new permanent ALBs, one public, and the other private. These ALBs will then be shared with various elastic beanstalk stacks depending on how a stack is configured. For this option each Elastic Beanstalk shares an existing ALB rather than creating its own ALB internally. See Figure 3.

Deployement

Deployment under Option B would be similar to Option A, where an ‘isPublic=false’ parameter would be provided when stack 102 is built the first time. However, staging CNAME will not change as it is permanently bound to the shared private ALB.

Promoting Staging to Production

Unfortunately, Elastic Beanstalk does not support changing its shared ALB after it has been created. Therefore, just like with Option A, we will need to deploy a new stack with the parameter ‘isPublic=true’ to create a new EB that is shared with the public ALB. See Figure 4.

Figure 4 shows a single public shared ALB that is pointing to two different stack 101 & 102. Both stacks are configured with the same path information. When the ALB encounters duplicate paths that for the same request, it will send all traffic to the path with the lowest priority. Therefore, all traffic as depicted in Figure 4, will continue to go to portal-prod-101 (since it has a lower priority). To promote 102 to be the new production, the ALB must be changed to assign 102 a lower priority than 101. At that point all new traffic will go to stack 102.

This option has the benefit of not needing a CNAME change to promote a new stack. However, we were unable to figure out how to validate the new stack (102) before the final priority change. For example, if something was wrong with the new 102 stack we would not know about until after we promoted to be production.

Pros & Cons