AWS API Gateway Custom JSON Logging Tutorial with CloudFormation Examples

When working with AWS API Gateway at scale, having structured logs in JSON format makes it easier to query, analyze, and monitor your API traffic. This guide walks through implementing a custom JSON logging format for API Gateway access logs, including how to manage the configuration through AWS Systems Manager Parameter Store and deploy it via CloudFormation.

Prerequisites

This article assumes you already have setup API Gateway to have permission to send logs to CloudWatch Logs. Setup Permissions for CloudWatch logging. This step needs to be done in each AWS account and each region.

Custom JSON Log Format Configuration

{
  "apiId": "$context.apiId",
  "authorizerIntegrationLatency": "$context.authorizer.integrationLatency",
  "awsRequestId": "$context.awsEndpointRequestId",
  "caller": "$context.identity.caller",
  "errorMessage": "$context.error.message",
  "errorResponseType": "$context.error.responseType",
  "httpMethod": "$context.httpMethod",
  "integrationLatency": "$context.integrationLatency",
  "ip": "$context.identity.sourceIp",
  "path": "$context.path",
  "principalId": "$context.authorizer.principalId",
  "protocol": "$context.protocol",
  "requestId": "$context.requestId",
  "requestTime": "$context.requestTime",
  "resourcePath": "$context.resourcePath",
  "responseLatency": "$context.responseLatency",
  "responseLength": "$context.responseLength",
  "status": "$context.status",
  "user": "$context.identity.user",
  "userAgent": "$context.identity.userAgent"
}

Understanding the Log Parameters

apiId: The unique identifier for your API Gateway REST API
authorizerIntegrationLatency: Time in milliseconds between API Gateway calling the authorizer and receiving a response
awsRequestId: AWS endpoint request ID (useful for AWS support tickets)
caller: The principal identifier of the caller making the request
errorMessage: Error message string in case of request failure
errorResponseType: Error response type (e.g., DEFAULT_4XX, DEFAULT_5XX)
httpMethod: HTTP method used (GET, POST, PUT, DELETE, etc.)
integrationLatency: Time in milliseconds between API Gateway relaying a request to the backend and receiving a response
ip: Source IP address of the TCP connection making the request
path: The complete path for the request
principalId: The principal user identification associated with the token
protocol: Request protocol (e.g., HTTP/1.1)
requestId: Unique identifier for the request
requestTime: CLF-formatted request time (dd/MMM/yyyy:HH:mm:ss ±hhmm)
resourcePath: The resource path as defined in API Gateway
responseLatency: Total time in milliseconds from request received to response sent
responseLength: Response payload length in bytes
status: HTTP response status code
user: The user identity making the request
userAgent: User-Agent header value from the request

Implementation with CloudFormation and Parameter Store

First, store the JSON format in SSM Parameter Store under a namespace such as /cloudformation/apigw-access-logs-format. This allows you to maintain consistency across multiple APIs and update the format centrally.

CloudFormation Template Example

---
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: Pet Store - API
Parameters:
  AppName:
    Description: Name of application
    Type: String
  Environment:
    Type: String

  ApiGatewayAccessLogFormat:
    Type: AWS::SSM::Parameter::Value<String>
    Description: API Gateway access log format
    Default: /cloudformation/apigw-access-logs-format

Resources:
  RestApi:
    Type: AWS::Serverless::Api
    Properties:
      Name: !Sub ${AppName}-${Environment}
      StageName: live
      EndpointConfiguration:
        Type: REGIONAL
      TracingEnabled: true

      MethodSettings:
        - DataTraceEnabled: true
          MetricsEnabled: true
          HttpMethod: '*'
          LoggingLevel: INFO
          ResourcePath: /*

      AccessLogSetting:
        DestinationArn: !GetAtt RestApiLogGroup.Arn
        Format: !Ref ApiGatewayAccessLogFormat

  RestApiLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub API-Gateway-Execution-Logs_${RestApi}/${RestApi.Stage}
      RetentionInDays: 7

Sample Log Output in CloudWatch Logs

Here’s what the structured logs look like in CloudWatch Logs:

{
  "apiId": "41ov1oq4qg",
  "authorizerIntegrationLatency": "-",
  "awsRequestId": "75458d31-48b4-49a3-8f91-a499250195c8",
  "caller": "-",
  "errorMessage": "-",
  "errorResponseType": "-",
  "httpMethod": "GET",
  "integrationLatency": "12",
  "ip": "15.177.10.91",
  "path": "/petstore/v1/health",
  "principalId": "-",
  "protocol": "HTTP/1.1",
  "requestId": "a7f20309-8014-45ad-8083-1f4f64ca38a5",
  "requestTime": "04/Apr/2025:12:26:33 +0000",
  "resourcePath": "/v1/health",
  "responseLatency": "21",
  "responseLength": "15",
  "status": "200",
  "user": "-",
  "userAgent": "Amazon-Route53-Health-Check-Service (ref 4ca88a3b-fd4b-4651-b166-02cf181a42f4; report http://amzn.to/1vsZADi)"
}

Note that fields with no applicable value show as "-" in the logs.

Benefits of This Approach

Structured Querying: JSON format enables easy querying with CloudWatch Insights
Centralized Management: Using Parameter Store allows you to update log formats across all APIs from a single location
Comprehensive Monitoring: Capture all essential metrics including latency, errors, and user information