Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

...

...

...

...

Heroku logs

Heroku logs are a stream of events which are collated and routed by Logplex.  There are 3 types: API logs, system logs, and app logs.

Actions about the stack via the Heroku API are tracked by the API logs.  Example:

Code Block
Sep 02 14:59:03 bridge-prod heroku/api: Deploy f158691 by xyz@abc.org

System logs are events of the Heroku stack.  Of particular interest is the router log:

Code Block
Sep 03 14:24:20 bridge-prod heroku/router: at=info method=GET path="/?study=api" host=webservices.sagebridge.org
 request_id=4f839190-0943-8a41-9081-5b9faa862f31 fwd="173.194.33.105" dyno=web.5 connect=4ms service=7ms status=200 bytes=509

...

Notice that the request ID is recorded across different types of logs.  It can used to trace events of a particular request.

Consumers of the Heroku logs

Logentries

Heroku logs are drained to the Heroku add-on Logentries.  Logentries archives the logs permanently to S3 and alerts us on high error rates.  For details on alert setup, see the page Monitoring policies.

Librato

Librato monitors stack performance by analyzing the Heroku API logs and system logs.

...

(Note Librato was introduced as a partial alternative to New Relic.  New Relic has since been disabled as it does not support Play 2.4.x.)

Redshift

Currently a manual process is carried out regularly to export the structured metrics to Redshift.  There the logs can join other tables for the purpose of answering business questions, performing audits, and to a lesser extent, monitoring the system.

TODO: 

Jira Legacy
serverJIRA (sagebionetworks.jira.com)
serverIdba6fb084-9827-3160-8067-8ac7470f78b2
keyBRIDGE-500

Logging and monitoring on AWS resources

We have CloudWatch alerts set up on DynamoDB metrics.  For details, see the page Monitoring policies.

...