Overview

• We capture the application logs and store them in AWS Cloudwatch.
• This document will walk you through
  • AWS Cloudwatch important terminologies that the user needs to know about before exploring the logs
  • How you can explore logs for different components of the system

Terminologies

Terminology	Description
Cloudwatch Log Stream	A log stream is generally intended to represent the sequence of events coming from the application instance or resource being monitored.
Cloudwatch Log Group	Log groups define groups of log streams that share the same retention, monitoring settings.
Structured Logging	In structured logging, instead of printing the log as a string, we print it as a JSON with different attributes. This allows easy filtering of logs based on different attributes, greatly simplifying log exploration and debugging, regardless of the viewing platform.

How do we map logs of our system to log groups and streams?

• The log groups and log streams are created on the following basis
  • Log Group: /dynamofl/application-logs/${CLUSTER_NAME}/${NAMESPACE}
  • Log Stream: /${CONTAINER_NAME}.${POD_NAME}
• Log Group -> We’ve got different components deployed in different namespaces. Let’s say the cluster’s name is eks-cluster
  • API, DynamoEval Scaled Jobs -> default
    • Log Group -> /dynamofl/application-logs/eks-cluster/default
  • MongoDB -> mongodb
    • Log Group -> /dynamofl/application-logs/eks-cluster/mongodb
• Log Stream -> Whenever a new pod gets created a log stream for that pod is created
  • API -> api.api-deployment-85c44459b9-94m8q

Structured Logging

• We’ve started adding structured logs into different components. The logs now have certain attributes using which we can easily filter the logs
• Structured logging enables simple filtering based on some metadata attached to the logs instead of complex search approaches over the text of the log

Component Wise Log Exploration

• DynamoEval
• Generic
  • Covers the logs for API, UI, MongoDB, PostgreSQL, Monitoring Components, etc