Jira tickets:
PLFM-5314 - Getting issue details... STATUS
PLFM-5315 - Getting issue details... STATUS
Background Jiras:
Background
See Use Case 1 in Service to migrate content at the S3/Storage Level: Use Cases for use case notes.
In Synapse, the top 10 projects by file size account for about 3/4 of our total S3 bucket. These projects can be very expensive, so there is a need to determine the costs of projects. Using file metadata, we can determine approximate size very easily. Egress is more difficult to determine, but per the analysis in PLFM-5009, storage counts for approximately 80% of our bill. There is not currently a need to be incredibly precise/accurate, so we may simply ignore egress for now. At the moment, cost of egress can be assumed to be distributed proportionally to costs of storage.
API
We can create a new Cost Allocation service in Synapse that can be used by members of a "Synapse Cost Allocation Team", a bootstrapped team with users that have the authority to manage these cost allocations.
Cost allocations in Synapse are groupings of projects that should be pooled for cost services. Projects can only have 0 or 1 cost allocation.
Cost allocations are uniquely identified by their name and can be created by a member of the Cost Allocation Team by assigning one to a project.
When a project is assigned to a cost allocation, the underlying files associated with that project (that is, the file handles used by all versions of all file entities in a project) will be included in the total cost of that cost allocation.
A member of the cost allocation team can make an asynchronous query to retrieve a CSV report with the sizes of their choice of all cost allocations, all allocated projects, or all unallocated projects.
All requests can only be made members of the Synapse Cost Allocation Team
| Verb | URI | Request body | Response body | Notes |
|---|---|---|---|---|
| POST | /entity/{entityId}/costAllocation | name: String | CostAllocation id: String name: String bucket: String projects: Array<String> eTag: String createdBy: Long createdOn: Date | Associates a project with a cost allocation. If the cost allocation doesn't exist, it creates a new one. If the project is currently associated with a different cost allocation, it will be associated with the new one. Name is case-insensitive (will be coerced to lowercase) and can include alphanumeric, "-", ".", and "_". |
| GET | /entity/{entityId}/costAllocation | None | CostAllocation | Gets the cost allocation for a specific project. This request can only be made by a member of the Synapse Cost Allocation Team |
| DELETE | /entity/{entityId}/costAllocation | None | None | Removes the cost allocation tied to a project. The contents of the project that are in the cost allocation storage location will be moved to the default Synapse storage. After all of the contents have been moved, the project is removed from the cost allocation. |
| GET | /costAllocation/report/csv/async/get/{token} | None | CostAllocationReportResult: resultsFileHandleId: String timestamp: Date | Get an object containing a file handle that points to a Cost Allocation Report CSV |
| POST | /costAllocation/report/csv/async/start | CostAllocationReportRequest type: Enum (COST_ALLOCATIONS, ALLOCATED_PROJECTS, UNALLOCATED_PROJECTS) | AsyncJobId | Initiates a job to create a CSV report for the sizes of cost allocations or unallocated projects in Synapse. The type describes the type of report will be generated. COST_ALLOCATIONS will generate a report of all cost allocations, ALLOCATED_PROJECTS will generate a report of the largest projects that are assigned to cost allocations, and UNALLOCATED_PROJECTS will generate a report of the largest projects that are not assigned to cost allocations. See examples below. |
Sample Reports
Type: COST_ALLOCATION
| Cost Allocation ID | Name | Size (B) | Proportion of Synapse Storage |
|---|---|---|---|
| 18 | amp-ad | 424483013985391 | 0.6534 |
| 1 | grant123 | 53579813875383 | 0.0824 |
| 4 | smith_projects | 31482472428417 | 0.0414 |
| ... | ... | ... | ... |
| 0 | unallocated | 89573285798719 | 0.2139 |
Type: ALLOCATED_PROJECTS
| Project ID | Name | Size (B) | Proportion of Synapse Storage |
|---|---|---|---|
| syn154314 | Project Onc-RNA | 124483013985391 | 0.1244 |
| syn428582 | Super Cool Data | 74579813875383 | 0.0644 |
| syn523913 | Smith Lab Repository 1 | 31482472428417 | 0.0311 |
| ... | ... | ... | ... |
| 0 | unallocated | 424483013985391 | 0.8538 |
Type: UNALLOCATED_PROJECTS
| Project ID | Name | Size (B) | Proportion of Synapse Storage |
|---|---|---|---|
| syn123456 | Cool Project 123 | 14483013985391 | 0.1829 |
| syn999999 | Research Group Data | 8579813875383 | 0.1041 |
| syn583725 | Smith Lab Repository 2 | 5148247242841 | 0.0612 |
| ... | ... | ... | ... |
| 0 | allocated | 424483013985391 | 0.7538 |
Implementation Details
This section is unrelated to the API. Feel free to ignore it if it is not within your scope of concern.
Detailed Requirements
- Creation of a new bootstrapped "Cost Allocation Team" to have access to these APIs.
- Retrieval of file handle metadata, particularly project association and file size
- File Replication Table
- Columns from File Table: ID, ETAG, PREVIEW_ID, CREATED_ON, CREATED_BY, METADATA_TYPE, CONTENT_TYPE, CONTENT_SIZE, CONTENT_MD5, BUCKET_NAME, NAME, KEY, STORAGE_LOCATION_ID, ENDPOINT
- Primary key ID
- Creation of this table allows retrieval of file metadata by joining it with the entity replication table. This allows us to find all of the file handles and metadata for a particular project in one database call. Without this table, we must query the tables database to find the entities in a project, and then separately query the repo database to retrieve the metadata of those files.
- Entity Replication Table
- New column COST_ALLOCATION_ID
- PLFM-4148 - Getting issue details... STATUS is another issue that may benefit from this
- File Replication Table
- Enumeration of cost allocations
- Cost allocation table: ID, NAME, CREATED_BY, CREATED_ON
- Associate cost allocations and projects
- Cost Allocation association table
- Columns: COST_ALLOCATION_ID, PROJECT_ID
- Primary key: PROJECT_ID (a project may have no more than one cost allocation)
- Cost Allocation association table
Concerns
- This method will not accurately capture egress. It simply calculates proportions of cost based on storage.