CloudWatch Logs, metric filters and alarms: from a log line to a red light

Amazon CloudWatch Logs with CloudWatch alarms, in plain words

Amazon CloudWatch Logs is where AWS services and your own code send their log lines. Lambda, API Gateway, ECS and many others write there by default, and anything else can send events with one API call. Logs live in log groups (one per app or function, usually) made of log streams (one per source). You choose how long each group keeps its data; the default is forever, which is the most common reason log bills creep up.

Reading raw logs only goes so far, so two tools sit on top. Logs Insights is a query language for asking questions after the fact: how many errors per route in the last hour, what was the average latency, show me the last ten failures. If your logs are JSON, it finds the fields by name with no setup. A metric filter works in the other direction: it watches events as they arrive and turns matches into a CloudWatch metric, such as "errors per minute".

Once something is a metric, a CloudWatch alarm can watch it and change state (OK, ALARM, INSUFFICIENT_DATA) when it crosses a threshold, and optionally notify an SNS topic, scale a group or run another action. Today's lab wires all three together: JSON logs in, a metric filter counting errors, an alarm that turns red at three errors in a minute, and two Logs Insights queries to see what happened.

The AI angle

AI apps are hard to debug without logs: which prompt timed out, how many tokens a request used, when a guardrail stepped in. Amazon Bedrock's model invocation logging (off by default, set up per Region) can send each call's model ID, token counts and request and response bodies to a CloudWatch Logs group. From there the same tools apply: a Logs Insights query for average output tokens per model, a metric filter that counts failed calls, an alarm when timeouts spike like today's made-up model timeout lines. Because those logs can contain full prompts and answers, give the log group short retention and tight access.

Hands-on lab: Log JSON events, count errors with a metric filter and trip an alarm

From AWS CloudShell, create a log group with 1-day retention, add a metric filter that counts JSON ERROR events, create an alarm on that metric, send eight log lines, query them with Logs Insights, watch the alarm go to ALARM, then delete everything and confirm it's gone.

1. Open CloudShell and create a log group with short retention

   Choose the CloudShell icon in the console's top bar. A log group holds related logs, and a log stream is one source writing into it (one app instance, one Lambda container). Setting retention to 1 day means CloudWatch Logs expires the events on its own even if you forget cleanup; the default is to keep them forever. If CloudShell restarts during the lab, run the variable lines again.

   LG=/sandbox/lab-app
   STREAM=web-1
   FILTER=sandbox-error-count
   NS=SandboxLab
   METRIC=ErrorCount
   ALARM=sandbox-error-alarm
   echo "Region: $AWS_REGION"
   aws logs create-log-group --log-group-name "$LG"
   aws logs put-retention-policy --log-group-name "$LG" --retention-in-days 1
   aws logs create-log-stream --log-group-name "$LG" --log-stream-name "$STREAM"
   aws logs describe-log-groups --log-group-name-prefix "$LG" \
     --query 'logGroups[].[logGroupName,retentionInDays]' --output text

   Done

2. Turn ERROR lines into a metric with a metric filter

   A metric filter watches events as they arrive and publishes a CloudWatch metric. The pattern { $.level = "ERROR" } matches JSON events whose level field is ERROR; each match publishes 1, and defaultValue=0 publishes 0 for events that don't match, so the metric shows a real zero instead of a gap. Create the filter before sending logs: filters never look back at events that already arrived.

   aws logs put-metric-filter --log-group-name "$LG" --filter-name "$FILTER" \
     --filter-pattern '{ $.level = "ERROR" }' \
     --metric-transformations metricName=$METRIC,metricNamespace=$NS,metricValue=1,defaultValue=0,unit=Count
   aws logs describe-metric-filters --log-group-name "$LG" \
     --query 'metricFilters[].[filterName,filterPattern]' --output text

   Done

3. Create an alarm on the new metric

   The alarm adds up (Sum) the error metric over one 60-second period and goes to ALARM at 3 or more errors. notBreaching tells it to treat minutes with no data as healthy, which suits an error counter. There is no --alarm-actions, so nothing gets emailed or triggered; in real use you would point it at an SNS topic or another action. A brand-new alarm starts in INSUFFICIENT_DATA.

   aws cloudwatch put-metric-alarm --alarm-name "$ALARM" \
     --namespace "$NS" --metric-name "$METRIC" \
     --statistic Sum --period 60 --evaluation-periods 1 \
     --threshold 3 --comparison-operator GreaterThanOrEqualToThreshold \
     --treat-missing-data notBreaching
   aws cloudwatch describe-alarms --alarm-names "$ALARM" \
     --query 'MetricAlarms[0].[AlarmName,StateValue]' --output text

   Done

4. Send eight structured log lines

   Each event needs a timestamp in milliseconds and a message. The messages are JSON, the way many apps log, so the filter and Logs Insights can read fields by name. Events in one call must be in time order and no older than 14 days. The batch below is a made-up chat app: five INFO lines and three ERROR lines, sent a millisecond apart so they land in the same minute. A response with no rejectedLogEventsInfo means all eight were accepted.

   NOW=$(date +%s%3N)
   cat > events.json <<EOF
   [
    {"timestamp": $((NOW+1)), "message": "{\"level\":\"INFO\",\"route\":\"/chat\",\"latency\":120}"},
    {"timestamp": $((NOW+2)), "message": "{\"level\":\"INFO\",\"route\":\"/chat\",\"latency\":140}"},
    {"timestamp": $((NOW+3)), "message": "{\"level\":\"ERROR\",\"route\":\"/chat\",\"latency\":2300,\"error\":\"model timeout\"}"},
    {"timestamp": $((NOW+4)), "message": "{\"level\":\"INFO\",\"route\":\"/search\",\"latency\":80}"},
    {"timestamp": $((NOW+5)), "message": "{\"level\":\"ERROR\",\"route\":\"/chat\",\"latency\":1900,\"error\":\"model timeout\"}"},
    {"timestamp": $((NOW+6)), "message": "{\"level\":\"INFO\",\"route\":\"/search\",\"latency\":95}"},
    {"timestamp": $((NOW+7)), "message": "{\"level\":\"ERROR\",\"route\":\"/upload\",\"latency\":450,\"error\":\"file too large\"}"},
    {"timestamp": $((NOW+8)), "message": "{\"level\":\"INFO\",\"route\":\"/chat\",\"latency\":130}"}
   ]
   EOF
   aws logs put-log-events --log-group-name "$LG" --log-stream-name "$STREAM" \
     --log-events file://events.json

   Done

5. Ask questions with Logs Insights

   Logs Insights discovers the JSON fields (level, route, latency, error) on its own. start-query takes its time range in seconds, not milliseconds, and returns a query ID; get-query-results returns rows once the status is Complete. Wait about 30 seconds after step 4 so the events are searchable. The first query counts events and averages latency per level; the second lists only the errors, newest first. If a query returns nothing, wait a little and run it again.

   run_query() {
     QID=$(aws logs start-query --log-group-name "$LG" \
       --start-time $(( $(date +%s) - 900 )) --end-time $(( $(date +%s) + 60 )) \
       --query-string "$1" --query queryId --output text)
     for i in 1 2 3 4 5 6 7 8 9 10; do
       [ "$(aws logs get-query-results --query-id "$QID" --query status --output text)" = Complete ] && break
       sleep 2
     done
     aws logs get-query-results --query-id "$QID" --query 'results[*][*].value' --output text
   }
   run_query 'stats count(*) as events, avg(latency) as avg_ms by level'
   run_query 'filter level = "ERROR" | fields @timestamp, route, error | sort @timestamp desc'

   Done

6. Watch the metric and the alarm react

   Give CloudWatch two or three minutes to publish the filter's data points and evaluate the alarm. The metric should show a Sum of 3 for the minute your logs arrived, and the alarm should move to ALARM with a reason that mentions the threshold. A few minutes later, with no new data and notBreaching, it drifts back to OK. If the metric is still empty, wait a minute and run the commands again.

   aws cloudwatch get-metric-statistics --namespace "$NS" --metric-name "$METRIC" \
     --start-time $(date -u -d '-15 minutes' +%FT%TZ) --end-time $(date -u -d '+1 minute' +%FT%TZ) \
     --period 60 --statistics Sum \
     --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text
   aws cloudwatch describe-alarms --alarm-names "$ALARM" \
     --query 'MetricAlarms[0].[StateValue,StateReason]' --output text

   Done

aws cloudwatch delete-alarms --alarm-names "$ALARM"
aws logs delete-metric-filter --log-group-name "$LG" --filter-name "$FILTER"
aws logs delete-log-group --log-group-name "$LG"
rm -f events.json
aws cloudwatch describe-alarms --alarm-names "$ALARM" --query 'MetricAlarms[].AlarmName' --output text
aws logs describe-log-groups --log-group-name-prefix "$LG" --query 'logGroups[].logGroupName' --output text

Gotchas

• Metric filters only count events that arrive after the filter exists. Adding one today won't tell you how many errors you had yesterday; use a Logs Insights query for that.
• Time units differ between calls: put-log-events timestamps are milliseconds, while start-query takes start and end times in seconds. Mixing them up gives rejected events or empty results.
• Log groups keep data forever unless you set retention. Put a retention policy on every group you create, including the ones Lambda creates for you on first run.
• Don't add high-cardinality dimensions, such as request IDs or IP addresses, to a metric filter. Each distinct value becomes a separately billed custom metric, and CloudWatch Logs may disable a filter that creates too many.
• News: since October 21, 2026, Amazon S3 server access logs can be delivered to CloudWatch Logs in every AWS Region, including GovCloud (US), the European Sovereign Cloud and China, so the query-and-alarm pattern from today's lab works for bucket access logs everywhere.