Page Comparison

...

1. A user applies a cost allocation to the project. If the cost allocation does not exist, a new bucket is created. If the bucket is successfully created or if the cost allocation exists, the call succeeds for the user. A successful call triggers the next step.
2. A worker collects the file handles in that project that are stored in Synapse storage and are not in the cost allocation. These file handles are sent to another worker to accomplish the next step.
3. A worker is given a file handle and a destination bucket. If the file is not already in that bucket, it is copied to the destination bucket. If the copy is successful, the file handle is modified to point to the new location. The old file is archived and marked for later deletion.

Detailed

...

Requirements 

...

...

• Creation of a new bootstrapped "Cost Allocation Team" to have access to these APIs.
• Retrieval of file handle metadata, particularly project association and file size 
  • File Replication Table
    • Columns from File Table: ID, ETAG, PREVIEW_ID, CREATED_ON, CREATED_BY, METADATA_TYPE, CONTENT_TYPE, CONTENT_SIZE, CONTENT_MD5, BUCKET_NAME, NAME, KEY, STORAGE_LOCATION_ID, ENDPOINT
    • Primary key ID
  • Creation of this table allows retrieval of file metadata by joining it with the entity replication table. This allows us to find all of the file handles and their metadata in one database call. Without this table, we must query the tables database to find the entities in a project, and then separately query the repo database to retrieve the metadata of those files.

  • Jira Legacy
    server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId ba6fb084-9827-3160-8067-8ac7470f78b2 key PLFM-4148

     is another issue that may benefit from this
• Enumeration of cost allocations
  • Cost allocation table: ID, NAME, BUCKET, CREATED_BY, CREATED_ON
• Associate cost allocations and projects
  • Cost Allocation association table
    • Columns: COST_ALLOCATION_ID, PROJECT_ID
    • Primary key: PROJECT_ID (a project may have no more than one cost allocation)
• Overriding uploads to default Synapse storage to redirect to the cost allocation bucket.
• An eventually consistent algorithm to move files to a new bucket, and then modifying the file handle to reflect the move
  
  • Asynchronous worker scans file replication table for files that are in the project with the cost allocation AND are in Synapse storage AND are not in the correct cost allocation bucket
    • This worker creates bundles of files that should be updated and sends these bundles to another asynchronous worker
  • Asynchronous worker finds each file in the bundle in the files table and verifies that it is not in the correct cost allocation bucket
    • The file is copied to the new S3 bucket and updates the file handle, archiving/deleting the old underlying file.
    • The actual copying may be fairly trivial with AWS Batch Operations for S3 Buckets and Copy Objects Between S3 Buckets With Lambda 
      • We still must somehow mark the old file for deletion, perhaps by moving it to low-cost storage with a lifecycle deletion policy
      • If we can track the batch operations, we can modify file handles one-by-one. Otherwise we may have to find another solution, or wait for the batch operation to complete

...