Document toolboxDocument toolbox

Upload metrics to Redshift

This is done manually with the following steps:

  1. Download the logs from the logentries S3 bucket.  I like to use the AWS CLI: aws s3 sync s3://<bucket> <local/directory>
  2. Extracts the metrics JSON clobs from the logs. See metrics.sh below for an example.

  3. Convert date time values and make them proper for Redshift import.  See metrics.py below.

  4. Upload the metrics file to the org-sagebridge-metrics-prod bucket.

  5. Launch a Redshift client and run a COPY command to dump the file into the Redshift metrics table.

The code mentioned above is in the S3 bucket: org-sagebridge-metrics-prod.

  1. metrics.sh  This is the grep that extracts the metrics JSON clobs from the log files and aggregates them into a single file.
  2. metrics.py  Python scripts that formats the date time values to make them proper for Redshift import.
  3. redshift.cmd  The Redshift COPY command.