...
| Table of Content Zone | |
|---|---|
|
Jira tickettickets:
| Jira Legacy | ||||||||
|---|---|---|---|---|---|---|---|---|
|
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Background Jiras:
| Jira Legacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Background
See Use Case 1 in Service to migrate content at the S3/Storage Level: Use Cases for use case notes.
In Synapse, the top 10 projects by file size account for about 3/4 of our total S3 bucket. These projects can be very expensive, so there is a need to determine the costs of projects. Using file metadata, we can determine approximate size very easily. Egress is more difficult to determine, but per the analysis in PLFM-5009, storage counts for approximately 80% of our bill. There is not currently a need to be incredibly precise/accurate, so we may simply ignore egress for now. At the moment, cost of egress can be assumed to be distributed proportionally to costs of storage.
API
We can create Create a new Cost Allocation Storage Report asynchronous service in Synapse that can be used by members of a "Synapse Cost Allocation Reports Team", a bootstrapped team with users that have the authority to manage create these cost allocationsreports.The members
A member of the Cost Allocation Team can create new cost allocations with a descriptive name (e.g. ampad, nih-r01-12345) that matches how costs should be broken down in Synapse. When a new cost allocation being created, project sizes can be calculated by group to determine that cost allocation's total impact.After a cost allocation is created, a project can be assigned to it. Cost allocations can have many projects, but a project can be assigned to at most one cost allocation. When a project is assigned to a cost allocation, the underlying files associated with that project (that is, the file handles pointed to by all versions of all file entities in a project) will be included in the total cost of that cost allocation.Reports team can make an asynchronous query to retrieve a CSV report about the usage of the Synapse S3 bucket with project-level resolution across all projects.
| Verb | URI | Request body | Response body | Notes |
|---|---|---|---|---|
| GET | / |
None
CostAllocationPage
body: Array<CostAllocation>
nextPageToken: String
| storageReport/csv/async/ |
CostAllocationReportRequest
numberOfResults: Long
allocated: Boolean
AsyncJobId
Initiates a job to create a CSV report for the sizes of cost allocations or unallocated projects in Synapse.
The results will contain the top <numberOfResults> cost allocations/unallocated projects by descending size.
If allocated is true, the report will include the largest cost allocations. If allocated is false, the report will include the largest projects that are not currently assigned to a cost allocation.
This request can only be made by a member of the Synapse Cost Allocation Team
| get/{token} | None |
DownloadStorageReportResponse: resultsFileHandleId: String timestamp: Date | Get an object containing a file handle that points to a |
Storage Report CSV The caller can download a CSV with the file handle ID | |
| POST | / |
name: String
CostAllocation
id: String
name: String
bucket: String
projects: Array<String>
eTag: String
createdBy: Long
createdOn: Date
Associates a project with a cost allocation. If the cost allocation doesn't exist, it creates a new one. If the project is currently associated with a different cost allocation, it will be replaced with a new one.
Name is case-insensitive (will be coerced to lowercase) and can include alphanumeric, "-", ".", and "_".
CostAllocation
Removes the cost allocation tied to a project. The contents of the project that are in the cost allocation storage location will be moved to the default Synapse storage.
After all of the contents have been moved, the project is removed from the cost allocation.
Sample Reports
CostAllocation report (default, allocated=TRUE)
...
Unallocated projects report (allocated=FALSE)
| storageReport/csv/async/start | DownloadStorageReportRequest type: Enum (ALL_PROJECTS) | AsyncJobId | Initiates a job to create a CSV report for the sizes of projects in Synapse (where size is usage of the Synapse S3 bucket). The request will create a report about all projects when specifying ALL_PROJECTS. The enum allows requests for different types of reports (for example, project groups, if that gets implemented) |
Sample Report
Type: ALL_PROJECTS
| Project ID | Name | Size (B) |
|---|
| syn5382532 | Cool Project |
| 1 |
| 424483013985391 |
| syn635535 | NIH-Grant 53532 Public Data Repository | 53579813875383 |
| syn9359135 | Dr. Smith's Private FASTQ files | 31482472428417 |
| ... | ... |
| .. |
| . |
Implementation Details
| Note |
|---|
This section is unrelated to the API. Feel free to ignore it if it is not within your scope of concern. |
Detailed Requirements
- Creation of a new bootstrapped "Cost Allocation Reports Team" to have access to these APIs.
- Retrieval of file handle metadata, particularly project association and file size
- File Replication Table
- Columns from File Table: ID, ETAG, PREVIEW_ID, CREATED_ON, CREATED_BY, METADATA_TYPE, CONTENT_TYPE, CONTENT_SIZE, CONTENT_MD5, BUCKET_NAME, NAME, KEY, STORAGE_LOCATION_ID, ENDPOINT
- Primary key ID
- Creation of this table allows retrieval of file metadata by joining it with the entity replication table. This allows us to find all of the file handles and metadata for a particular project in one database call. Without this table, we must query the tables database to find the entities in a project, and then separately query the repo database to retrieve the metadata of those files.
- Entity Replication Table
- New column COST_ALLOCATION_ID
is another issue that may benefit from thisJira Legacy server System JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId ba6fb084-9827-3160-8067-8ac7470f78b2 key PLFM-4148
- File Replication Table
- Enumeration of cost allocations
- Cost allocation table: ID, NAME, CREATED_BY, CREATED_ON
- Associate cost allocations and projects
- Cost Allocation association table
- Columns: COST_ALLOCATION_ID, PROJECT_ID Primary key: PROJECT_ID (a project may have no more than one cost allocation)
Concerns
- This method will not accurately capture egress. It simply calculates proportions of cost based on storage.
...