We're updating the issue view to help you get more done. 

Integrate downloads/uploads events with kinesis firehose streams

Description

In order to collect statistics for downloads/uploads we send the events to kinesis firehose streams. The stream will funnel the data to S3 that is later analyzed with AWS Athena.

For this task we need to refactor the stack builder so that the streams are created with potentially support for Apache Parquet so we can query them easily with AWS Athena. We also need to integrate with Glue so that the table schema is setup for the stream.

Some refactoring is needed in the synapse backend so that sending the download/uploads (and in general other type of records) to kinesis firehose streams is robust enough.

Environment

None

Status

Assignee

Marco Marasca

Reporter

Marco Marasca

Labels

None

Validator

None

Release Version History

None

Epic Link

Sprint

Priority

Major