Versions Compared

Old Version 6

changes.mady.by.user Sandhra Sokhal

Saved on

compared with

New Version Current

changes.mady.by.user Sandhra Sokhal

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

In Synapse we record all the file upload and download events and monthly file upload and download is calculated for each project by distinct users.

In Synapse, we track all the file upload and download events, and calculate monthly file uploads and downloads for each project based on distinct users.

  1. Each file upload and download event is captured by FileEventRecordWorker and sent to a kinesis stream, which stores the data in S3 in queryable Parquet format. The file event can be queried using <Env><Stack>firehoselogs.fileuploadsrecords and <Stack>firehoselogs.filedownloadsrecords Glue tables in Athena. As we changed the data warehouse architecture the same data is also available in warehouse.filedownloadrecords and warehouse.fileuploadrecords.

  2. StatisticsMonthlyStatusWatcherWorker dentifies identifies unprocessed months for project statistics and initiates a processing request by sending a message to a queue, allowing the processing to begin for the specified object type and month.

  3. StatisticsMonthlyWorker retrieve retrieves the message from queue and processes it. This worker executes the Athena query for file upload and file download statistics and store stores the results in the STATISTICS_MONTHLY_PROJECT_FILES table of the Synapse main database.

  4. The Synapse users who are admins or have read permissions on a project can access the statistics with https://rest-docs.synapse.org/rest/POST/statistics.html

...

Synapse offers integration support to third-party services. The basic idea here is that the calculation and management of statistics and making them available to users should be implemented by third party. Third party services can follow below the following processes to achieve this.

  1. Pre-calculate the project statistics and store the statistics in S3, database table or any other suitable storage.

  2. The third party App will connect to Synapse via single singsign-on(OAuth) to authenticate user. Once authenticated, the Synapse API will provide access token in response.

  3. Using the access token retrieved in Step 2, the third party service will connect to Synapse API https://rest-docs.synapse.org/rest/GET/entity/id/permissions.html to get UserEntityPermissions.

  4. Using the UserEntityPermissions response, the third party app can verify user permission on entity and provide monthly project statistics to the user, if eligible.

...