| Table of Contents | ||||||
|---|---|---|---|---|---|---|
|
...
- The process is relatively lengthy and requires non trivial technical skills
- The data is limited to a 6 months window, the current solution to this problem is to store incremental updates on an external source (cvs file on S3)
- The data cannot be easily integrated in the synapse portal or other components (in some cases the files are manually annotated with the number of downloads)
- The system has an all or nothing policy for accessing the data, that is (for good reason) only accessible to a specific subset of synapse employees, this does not allow the users of the synapse platform to access this kind of data without asking a synapse engineer
An example of usage report generated using the Synapse Usage Report written by Kenny Daily using the data warehouse:
Files and Downloads
Files in synapse are referenced through an abstraction (FileHandle) that maintain the information about the link to the content of the file itself (e.g. an S3 bucket). A file handle is then referenced in many places (such as FileEntity and WikiPage, see FileHandleAssociateType) as pointers to the actual file content. In order to actually download the content the synapse platform allows to generated a pre-signed url (according to the location where the file is stored) that can be used to directly download the file. Note that the platform has no way to guarantee that the pre-signed url is actually used by the client in order to download a file. Every single pre-signed url request in the codebase comes down to a single method getURLForFileHandle.
...
/asynchronous/job API (See Asynchronous Job API) so that a client can send a request to compute certain statistics and wait for the final response polling the API for the result. This allows to offload the computation to a background worker keeping the API servers free of heavy computation. The two main objects that the statistics API will need to extend are: AsynchronousRequestBody and AsynchronousResponseBody that represent respectively the request for a background job and the response from the job (returned as part of the AsynchronousJobStatus).Request Objects
In the following we provide a list of object representations for requests of entity statistics:
...
DownloadStatisticsRequest <extends> EntityStatisticsRequest
The actual job request for that extends the EntityStatisticsRequest to get the download statistics of an entity, this will be used to get download and users counts only.
| Property | Required | Type | Default Value | Description |
|---|---|---|---|---|
| fromTimestamp | No | Date | A timestamp used to limit the considered time frame from the given timestamp | |
| toTimestamp | No | Date | A timestamp used to limit the considered time frame until the given timestamp |
...
Extends the DownloadStatisticsRequest in order to compute the download counts (along with the unique users count) bucketed into a fixed amount of time, e.g. grouping by a given time frame, which allows to build an histogram and have the download trend over the specified time frame.
| Property | Required | Type | Default Value | Description |
|---|---|---|---|---|
| aggregationType | Yes | STRING | 'daily' | Specifies the type of aggregation in order to group the count of downloads, allowed values are: 'daily', 'weekly', 'monthly', 'yearly'. Note: For this type of job a limit is imposed on the time frame allowed for each aggregationType, in particular the maximum amount of buckets within the time frame according to the aggregation type should be less or equals than 100: Considering a resolution of 1 day, for each aggregation type we have max 100 days, 100 weeks (52 days * 100), 100 months (30 days * 100) or 100 years (365 days * 100)). |
Response Objects
In the following we provide a list of object representation for the statistics API:
<<abstract>> EntityStatisticsResponse <extends> AsynchronousResponseBody
Common parent object for the response regarding a request for statistics on an entity. This object is an AsynchronousResponseBody that will be included as part of the AsynchronousJobStatus when requesting the status of the job.
...
...
DownloadStatisticsResponse <extends> EntityStatisticsResponse
Represents the response for a job request for computing the download statistics for an entity (In response to a DownloadStatisticsRequest).
...
DownloadBucketsStatisticsResponse <extends> DownloadStatisticsResponse
Represents the response for a job request for computing the bucketed download statistics for an entity (in response to a DownloadBucketsStatisticsRequest). Note that the response extends the DownloadStatisticsResponse so that it includes the total counts.
...
StatisticsCountBucket
The purpose of this object is to include information about the count of a certain metric within a specific time frame:
...
Endpoints
...
| limit | No | Integer | 30 | Used for pagination, limits the number of buckets to the given value. This value has a max allowed value of 100. |
| offset | No | Integer | 0 | Used for pagination, specifies the offset index for the page of buckets. This value is a function of the given limit: the first page has an offset of 0, the second page offset is the limit value itself (e.g. 30 by default). |
Response Objects
In the following we provide a list of object representation for the statistics API:
<<abstract>> EntityStatisticsResponse <extends> AsynchronousResponseBody
Common parent object for the response regarding a request for statistics on an entity. This object is an AsynchronousResponseBody that will be included as part of the AsynchronousJobStatus when requesting the status of the job.
| Property | Type | Description |
|---|---|---|
lastUpdateTimestamp | Date | A timestamp representing the freshness of the statistic (last update date time of the requested statistic) |
DownloadStatisticsResponse <extends> EntityStatisticsResponse
Represents the response for a job request for computing the download statistics for an entity (In response to a DownloadStatisticsRequest).
| Property | Type | Description |
|---|---|---|
| totalCount | INTEGER | The total number of downloads |
| usersCount | INTEGER | The total number of unique users that performed a download |
DownloadBucketsStatisticsResponse <extends> DownloadStatisticsResponse
Represents the response for a job request for computing the bucketed download statistics for an entity (in response to a DownloadBucketsStatisticsRequest). Note that the response extends the DownloadStatisticsResponse so that it includes the total counts.
| Property | Type | Description |
|---|---|---|
| downloadCountBuckets | ARRAY<StatisticCountBucket> | An array of StatisticCountBucket each containing the download count for one bucket, according to the aggregationType specified in the request |
| userCountBuckets | ARRAY<StatisticCountBucket> | An array of StatisticCountBucket each containing the unique users count for one bucket, according to the aggregationType specified in the request |
| limit | Integer | Will include the limit value used in the initial request for pagination purposes |
| offset | Integer | Will include the offset value used in the initial request for pagination purposes |
StatisticsCountBucket
The purpose of this object is to include information about the count of a certain metric within a specific time frame:
| Property | Type | Description |
|---|---|---|
| total | INTEGER | The count in the time frame |
| fromTimestamp | STRING | The starting timestamp of the time frame represented by the bucket |
| toTimestamp | STRING | The ending timestamp of the time frame represented by the bucket |
Endpoints
The API reuses the endpoints from the Asynchronous Job API (We could potentially add a dedicated /statistics endpoint just for clarity).
| Method | Endpoint | Request Body | Response Body | Description | Restrictions |
|---|---|---|---|---|---|
| POST | /asynchronous/job |
| AsynchronousJobStatus: the responseBody property in the status will contain either the
| Allows to submit a job to gather the download statistics for an entity. The id returned by the request is used in order to get the job status. |
|
| GET | /asynchronous/job/{id} | N/A | AsynchronousJobStatus | Allows to get the current status of the statistics job with the given id. |
Additionally we propose to have dedicated statistics endpoints to be consistent with the current API design that accepts the above requests:
| Method | Endpoint | Request Body | Response Body | Description |
|---|
| POST | / |
- DownloadStatisticsRequest
- DownloadBucketsStatisticsRequest
- DownloadStatisticsResponse
or the - DownloadBucketsStatisticsResponse
- The entity specified in the request must be a project or a file entity (if not 400)
- The user should have view permission on the entity specified in the body (if not 403)
- The limit on the number of buckets for aggregations specified by the time frame in the request should be less or equal than 100. This limit applies only when the request is of DownloadBucketsStatisticsRequest type (if limit exceeds 400)
| statistics/entity/download/async/start |
| AsynchronousJobStatus | Allows to submit the job to gather the download statistics for an entity. The id returned by the request is used in order to the the job status. | |
| GET | /statistics/entity/download/async/get/{asyncToken} | N/A | AsynchronousJobStatus | Allows to get the current status of the statistics job with the given id |
Web Client Integration
The web client should have a way to show the statistics for a project or a specific file, some initial ideas:
...