Design Goals

We want our Stack Builder to solve the following problems:

Running integration tests on laptops can take a very long time and use a lot of CPU, memory, and I/O. Running integration tests on the cloud allows us to run integration tests faster and we can multitask and get other stuff done while we’re waiting.
Local developer environments sometimes go out of sync with our the development stage in AWS. We need a way to get them back in sync.
New developers and interns have to go through a lot of steps to set up their environment. The Stack Builder can offload a lot of that work.
Integration tests or manual tests can sometimes leave behind cruft in our developer environment, which may cause other integ tests to fail. We need a way to wipe the our developer environments clean.
Similarly, the MySQL database might have a merge error or otherwise have problems with the schemas. We’ll need a way to wipe the database, as well as any corresponding entries in DynamoDB.

How It Works

The StackBuilder will create a developer environment in AWS, including a MySQL database using AWS Aurora, and Elastic Beanstalk environments for the Server and the Worker, as well as any necessary AWS infrastructure needed. The DynamoDB, S3, SQS, and SNS Initializers in BridgeServer2 will then create the resources for those services.

This developer environment will be keyed off the env (usually local) and user (developer name) to namespace each developer’s environment, so that developers won’t conflict with each other. Each developer will have a full-fledged Bridge Server and Worker stack in AWS, and Jenkins tests pointed at these stacks, so each developer can run integ tests against their cloud environments, leaving their laptops free for doing other tasks.

It’s pretty straightforward to build a Server or Worker version from code changes on your laptop and deploy them to your own personal Elastic Beanstalk environment. Instructions will be written on how to do this, so that integ tests can be run in the cloud before pushing code to GitHub.

Components

CloudFormation Templates - The backbone of our Stack Builder will be our CloudFormation templates, as defined in BridgeServer2-Infra and BridgeWorkerPlatform-infra. We won’t need all the templates, since some are shared for setting up network resources, and some are just irrelevant for developer environments (like dashboards). In particular, the ones we need are: bridgeserver2-db, bridgeserver2, and bridgeworker.

StackBuilder Java Code - Some Java code to upload and instantiate the CloudFormation templates. This will also need to call the Jenkins Remote API to create server and worker integ tests in Jenkins, pointed at the developer environment in AWS. (Note: This will require VPN access and Jenkins credentials.)

The StackBuilder will read configs from BridgeServer2.conf and BridgeWorkerPlatform.conf (including overriding from the local configs) and use these to populate the parameters for the CloudFormation stacks. In particular, it will get the AWS Keys from ~/BridgeServer2.conf and use these keys to deploy the CloudFormation Templates to AWS.

We will also need to create a StackBuilder config, so that developers can override the CloudFormation templates with local files. This will allow us to test (most) CloudFormation changes.

StackCleaner Java Code - Some Java code to clean a developer’s cloud environment. Specifically, this cleanup code will need to drop all tables in the MySQL database and all DynamoDB tables. (Note that connecting to the MySQL database will require VPN access.)

Open Questions

Should the local laptop environment share the same SQL database, DynamoDB tables, S3 buckets, etc? The main pro to sharing the same database, etc, is that the developer’s local environment and cloud environments will have the exact same data and same versions, making it easy to have consistent tests across both local and cloud environments. The main con is that if the local environment uses the cloud SQL database, this will require VPN access, which adds extra pain to development on the laptop. I can reach out to IT and ask them if developer databases need to be on the VPN. If not, we could very easily share the SQL database between the local and cloud environments. If not, it would be best for the local environment to talk to a local SQL database.

If the local environment has its own separate local SQL database, should we still share DynamoDB tables and S3 buckets? In particular, some DynamoDB tables (such as StudyConsent) will throw startup errors if they don’t match the database. With this in mind, we’ll need to completely wipe SQL and DynamoDB every time we switch between the local and cloud environments (which would add a lot of friction to the development process).

Alternatively, we could have cloud environments have their own separate set of DynamoDB tables, which would double the DynamoDB tables in our AWS account. This is fine, but we would need to ask for a table limit increase from AWS.

Alternate Design

Instead of writing custom Java code for the Stack Builder, we can use the AWS Service Catalog.

Pros: This provides us with a fancy UI for creating stacks from CloudFormation templates.

Cons: Developers would need to manually create stacks from CloudFormation templates and manually fill in input parameters.