Elastic Container Registry with Synapse

Synapse has a Docker registry. The registry is implemented using Docker’s open source registry. Two copies run, on each of a pair of EC2 instances behind an ALB. On each EC2 we run the registry as a Docker container, obtained from DockerHub. There are several open issues with our current implementation:

1. Site Reliability Engineering: The infrastructure was deployed manually and should be scripted using CloudFormation for reproducibility. (See PLFM-7259 - Getting issue details... STATUS );

2. Upgrading the registry: There is no process for updating the running version of the open source registry;

3. Scanning the registry server: FISMA/FedRAMP requires that we can both the EC2 instances and the registry container instances, the latter prior to deployment as a part of an automated CI/CD process (i.e., as part of addressing (1), above).

4. Scanning the registry contents: Today we do not scan images loaded into the Synapse registry for vulnerabilities. We have open issues to evaluate and implement a registry scanner. See: PLFM-7429 - Getting issue details... STATUS

All of the above could be addressed by using AWS Elastic Container Registry (ECR) rather than the open source registry. ECR is a hosted solution which would eliminate the infrastructure we have deployed and the first three issues listed above. Further, ECR has a scanning capability, providing an immediate solution to the last issue.

The open source registry allows delegating authorization and we use the feature to let Synapse permissions control which Docker repositories a user can access (push/pull). We have avoided ECR because it doesn’t have this feature, it requires authorization be done using IAM policies. However it is possible to map Synapse permissions to fine grained (repository-level) permissions using AWS STS with in-line policies (much as we do when issuing STS tokens to access Synapse files). To understand how, it’s important to first understand how registry authentication works, as depicted here: https://docs.docker.com/registry/spec/auth/token/. With this understanding, the flow of web requests would be:

1. (Optional) The Docker client makes a login request, passing a PAT or username/password pair. Synapse validates the credentials and returns.

2. The Docker client makes a request (push/pull) to the registry endpoint, which we would map to a Synapse request.

3. Since the request has no authentication header, a 401 response is returned, telling the client where to authenticate.

4. The Docker client follows the WWW-authenticate directive to a second Synapse endpoint, passing a user name + password or personal access token along with the scopes requested. Synapse evaluates the request against the user’s permissions, creates an STS token scoped to the requested repositories and with a short lifetime, and creates a Docker authorization token.

5. Synapse issues a request to ECR’s authorization endpoint, authenticating with the token it created in the previous step.

6. Synapse returns the ECR’s response to the client. (This is a simple pass through.)

7. The Docker client repeats the request in (1), this time with the authorization token from (6).

8. Seeing the auth' token, the registry endpoint then redirects (using a 307 response) the client to the actual ECR endpoint. In the case of a docker push operation, Synapse first calls the ECR Create Repository service before returning the redirect.

9. ECR completes the push or pull operation.

10. ECR emits an event which we can connect to a chosen Synapse API using Event Bridge. For PUSH and DELETE events, Synapse will create, update or delete the corresponding Synapse entity. (Synapse already has a web hook for events: https://rest-docs.synapse.org/rest/POST/events.html. This API would be similar.)

The implementation is simple, the most complex step being 4, above, which works as follows:

1. Synapse parses the incoming authorization request, much as it does for its /bearerToken endpoint, today. The request includes the repositories and the kind of access (push/pull) requested.

2. Synapse determines the subset of requested authorization granted by the user’s permissions (and intersected with the PAT), again much as it does today.

3. Synapse invokes the STS Assume Role service for a role which is scoped to ECR, and applies an inline policy of the form:

   {
      "Version":"2012-10-17",
      "Statement":[
         {
            "Sid":"GetAuthorizationToken",
            "Effect":"Allow",
            "Action":[
               "ecr:GetAuthorizationToken"
            ],
            "Resource":"*"
         },
         {
            "Sid":"ManageRepositoryContents",
            "Effect":"Allow",
            "Action":[
                   "ecr:BatchCheckLayerAvailability",
                   "ecr:GetDownloadUrlForLayer",
                   "ecr:GetRepositoryPolicy",
                   "ecr:DescribeRepositories",
                   "ecr:ListImages",
                   "ecr:DescribeImages",
                   "ecr:BatchGetImage",
                   "ecr:InitiateLayerUpload",
                   "ecr:UploadLayerPart",
                   "ecr:CompleteLayerUpload",
                   "ecr:PutImage",
                   "ecr:ListImages"
            ],
            "Resource":"arn:aws:ecr:us-east-1:325565585839:syn123456/my/repo"
         }
      ]
   }

4. Synapse then calls the ECR GetAuthorizationToken service, authenticated by the STS token returned by the previous Assume Role request.

5. Synapse forwards the client’s authentication request to ECR however it replaces the client’s credentials with the token created in step 4.

6. Synapse returns the response to the client.

Content Migration

We have to migrate the content of the current Docker registry to ECR. A tool for doing it is here: https://hub.docker.com/r/docker/migrator. Note that there is a discrepancy between the list of repo’s returned by the registry and the list of repo’s indexed in Synapse, the former being much larger. We should investigate the discrepancy and see if we can eliminate the ‘orphaned’ repositories.