...

We are using a fork of an existing Github Project created by Upside Travel.

How it works

A brief overview of the system is listed below (diagram created using https://cloudcraft.co/):

...

There is currently nothing checking the Dead Letter SQS. Its purpose is to catch SNS messages that were unable to be delivered to the Scanner Lambda so that undelivered messages do not fail silently.

Building/installing the scanner

1. git clone https://github.com/Sage-Bionetworks/bucket-antivirus-function
   On Windows make sure you have cloned it with Unix-style line endings (LF) instead of Windows-style line endings (CRLF). You may do this by changing your git config or just run the dos2unix utility on the folder.
2. Download Docker Community Edition: https://www.docker.com/community-edition#/download

3. On Mac/Linux enter the following commands your cloned bucket-antivirus-function folder

   Code Block
   language bash
   make

   On Windows using Command Prompt while inside the bucket-antivirus-function folder(In File Explorer hold down Shift key and right-click then select "Open command window here"). This is just the same command that is run by make but pwd does not work on Windows and must be substituted with %CD%.

   Code Block
   docker run --rm -ti -v %CD%:/opt/app amazonlinux:latest /bin/bash -c "cd /opt/app && ./build_lambda.sh"

   If you did not check out the git repository with Unix-style line endings on Windows you may get an error: ": No such file or directory"

4. The build should have created a file called /build/lambda.zip
5. Upload the zip file to an S3 Bucket that you own. This is how the CloudFormation stack will access your built lambda.zip
6. Create a CloudFormation Stack with using virus-scan-cloudformation.json located in the git repository.

Triggering the scanner manually

...

Normally, files will be automatically scanned upload. However, it may be beneficial to rescan files when new virus definitions are added. To trigger the Scanner Lambda manually, a JSON message that mimics the S3 Event Notification JSON format must be written to the Scanner Trigger SNS. Below is a stripped down JSON example that contains all the information that the scanner will need from the JSON:

{
  "Records": [
    {
      "eventSource": "aws:s3",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "bucket": {
          "name": "test.scan.bucket.sagebase.org"
        },
        "object": {
          "key": "eicar.com",
        }
      }
    }
  ]
}

Limitations

• We can only scan files up to a certain size (25MB). It is very time/resource consuming to scan very large files (if we scan every file uploaded) and most files containing virus are small in size.
• The synapse production bucket is setup to notify only multipart uploads (See 

  Jira Legacy
  server System JIRA serverId ba6fb084-9827-3160-8067-8ac7470f78b2 key PLFM-7065

  ). We do not scan simple uploads since synapse always uses multipart uploads and each part is uploaded as a simple upload that would create a lot of overhead

Deployment

The lambda function is built using as jenkins job that builds a zip package containing the python code for the lambda(s) and uploads it to our jfrog repository as a generic artifact. The stack builder creates a dedicated stack for the lambda(s) downloading the zip artifact and uploading to an S3 bucket so that it can be referenced by the function(s), additionally each bucket that need scanning is configured to send notifications after a multipart upload to the SNS topic that the scanner function is triggered by.

Error Messages

This section documents common error messages, what they mean, and what action to take (if any).

...