PL
· Kubernetes · 5 min read

Full HTTP Request/Response Logging in Istio with Envoy Lua Filters

Istio’s Envoy sidecars give you free mTLS, traffic routing, and observability. However, by default, Envoy only logs metadata like method, path, status code, and duration. Request and response bodies are not logged because they’re not easily accessible at the HTTP filter level.

I want to show you how to use Envoy’s Lua filter extension to capture full headers and bodies, then emit them as structured JSON logs.

How Envoy Sidecars Process Requests

%%{ init: { 'look': 'handDrawn' } }%%
graph LR
    A[Inbound request] --> B[Listener<br/>:8000]
    B --> C[HTTP Connection Manager<br/>decoder filters]
    C --> D[Lua filter<br/>envoy_on_request]
    D --> E[Router filter<br/>upstream connection]
    F[Backend pod]
    F --> G[Router filter<br/>decoder]
    G --> H[Lua filter<br/>envoy_on_response]
    H --> I[Access log<br/>JSON to stdout]

The Lua filter runs at two points: once on the request path (before forwarding to the upstream) and once on the response path (before the access log is written). We use dynamicMetadata to store the captured data between these two phases.

Request Flow Through Envoy Filters

%%{ init: { 'look': 'handDrawn' } }%%
sequenceDiagram
    participant C as Client
    participant EP as Envoy Proxy (Sidecar)
    participant BE as Backend Pod
    
    C->>EP: POST /api with body {"user": "alice"}
    EP->>EP: envoy_on_request (Lua)<br/>Capture headers + body<br/>Store in dynamicMetadata
    EP->>BE: Forward request
    BE-->>EP: Response 200<br/>body {"status": "ok"}
    EP->>EP: envoy_on_response (Lua)<br/>Capture response headers + body<br/>Store in dynamicMetadata
    EP-->>C: Response forwarded
    EP->>EP: Access log (JSON)<br/>Includes DYNAMIC_METADATA<br/>from Lua filter

The Lua Filter

-- envoy_on_request: runs on every inbound request
function envoy_on_request(request_handle)
  -- Capture all headers
  local headers = request_handle:headers()
  local headersMap = {}
  for key, value in pairs(headers) do
    headersMap[key] = value
  end
  request_handle:streamInfo():dynamicMetadata():set(
    "envoy.lua", "request_headers", headersMap
  )

  -- Capture request body in chunks
  local requestBody = ""
  for chunk in request_handle:bodyChunks() do
    if chunk:length() > 0 then
      requestBody = requestBody .. chunk:getBytes(0, chunk:length())
    end
  end
  request_handle:streamInfo():dynamicMetadata():set(
    "envoy.lua", "request_body", requestBody
  )
end

-- envoy_on_response: runs on every outbound response
function envoy_on_response(response_handle)
  local headers = response_handle:headers()
  local headersMap = {}
  for key, value in pairs(headers) do
    headersMap[key] = value
  end
  response_handle:streamInfo():dynamicMetadata():set(
    "envoy.lua", "response_headers", headersMap
  )

  local responseBody = ""
  for chunk in response_handle:bodyChunks() do
    if chunk:length() > 0 then
      responseBody = responseBody .. chunk:getBytes(0, chunk:length())
    end
  end
  response_handle:streamInfo():dynamicMetadata():set(
    "envoy.lua", "response_body", responseBody
  )
end

The key is dynamicMetadata:set("envoy.lua", key, value) — this stores data in the stream’s metadata context, which persists through the request/response lifecycle and is readable in the access log format.

The EnvoyFilter Resource

The EnvoyFilter patches Istio sidecar proxies to inject the Lua filter. The inlineCode contains the same logic shown above:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: request-response-logging-filter
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: ANY  # Sidecars AND ingress gateways
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |
            -- See full implementation in "The Lua Filter" section above
            function envoy_on_request(request_handle)
              local headers = request_handle:headers()
              local headersMap = {}
              for key, value in pairs(headers) do
                headersMap[key] = value
              end
              request_handle:streamInfo():dynamicMetadata():set("envoy.lua", "request_headers", headersMap)
              local requestBody = ""
              for chunk in request_handle:bodyChunks() do
                if chunk:length() > 0 then
                  requestBody = requestBody .. chunk:getBytes(0, chunk:length())
                end
              end
              request_handle:streamInfo():dynamicMetadata():set("envoy.lua", "request_body", requestBody)
            end
            function envoy_on_response(response_handle)
              local headers = response_handle:headers()
              local headersMap = {}
              for key, value in pairs(headers) do
                headersMap[key] = value
              end
              response_handle:streamInfo():dynamicMetadata():set("envoy.lua", "response_headers", headersMap)
              local responseBody = ""
              for chunk in response_handle:bodyChunks() do
                if chunk:length() > 0 then
                  responseBody = responseBody .. chunk:getBytes(0, chunk:length())
                end
              end
              response_handle:streamInfo():dynamicMetadata():set("envoy.lua", "response_body", responseBody)
            end

context: ANY applies this to all sidecars (and the ingress gateway). The Lua filter is inserted before the Router filter, which is the standard Envoy HTTP filter chain.

The Access Log Format

# meshConfig in IstioOperator or values file
meshConfig:
  accessLogFile: /dev/stdout
  accessLogEncoding: JSON
  accessLogFormat: |
    {
      "protocol": "%PROTOCOL%",
      "method": "%REQ(:METHOD)%",
      "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
      "responseCode": "%RESPONSE_CODE%",
      "clientDuration": "%DURATION%",
      "responseCodeDetails": "%RESPONSE_CODE_DETAILS%",
      "upstreamCluster": "%UPSTREAM_CLUSTER%",
      "traceId": "%REQ(X-B3-TRACEID)%",
      "responseFlags": "%RESPONSE_FLAGS%",
      "upstreamHost": "%UPSTREAM_HOST%",
      "requestHeaders": "%DYNAMIC_METADATA(envoy.lua:request_headers)%",
      "requestBody": "%DYNAMIC_METADATA(envoy.lua:request_body)%",
      "responseHeaders": "%DYNAMIC_METADATA(envoy.lua:response_headers)%",
      "responseBody": "%DYNAMIC_METADATA(envoy.lua:response_body)%"
    }

The %DYNAMIC_METADATA(envoy.lua:request_body)% operator reads the value we stored in the Lua filter.

Sample Output

{
  "method": "POST",
  "path": "/post",
  "responseCode": 200,
  "clientDuration": 1,
  "upstreamHost": "10.1.0.1:80",
  "traceId": "njain3jbr231kkn",
  "requestBody": "{ \"request\": \"hi there\" }",
  "responseBody": "{ \"response\": \"you are welcome\" }",
  "requestHeaders": {
    "content-type": "application/json",
    "accept": "*/*",
    ":method": "POST",
    ":path": "/post",
    "user-agent": "PostmanRuntime/10.0.0"
  },
  "responseHeaders": {
    "content-type": "application/json",
    "date": "Sat, 28 Jan 2025 21:36:59 GMT",
    "content-length": "1252"
  }
}

Shipping Logs to Elasticsearch

%%{ init: { 'look': 'handDrawn' } }%%
graph LR
    A[Pod sidecar<br/>Envoy logs JSON] --> B[Fluentd / Fluent Bit<br/>tail /dev/stdout]
    B --> C[Elasticsearch<br/>index: istio-logs]
    C --> D[Kibana<br/>request explorer]
    D --> E[Filter by path,<br/>status, traceId]

Since logs go to /dev/stdout as JSON, Fluentd or Fluent Bit can tail them and forward to Elasticsearch. The structured JSON fields become searchable in Kibana.

Caveats

  • Performance: Capturing and logging full bodies adds latency and storage. Use this in non-production environments or with sampling.
  • PII: Request/response bodies may contain sensitive data. Apply filtering or masking before shipping logs.
  • Large bodies: Very large bodies can exhaust memory in the filter. Set appropriate limits.

For troubleshooting API issues, debugging request/response mismatches, and understanding latency at the HTTP level, full body logging is invaluable — and with Envoy’s Lua filter, it’s surprisingly straightforward to implement.

KubernetesObservability

Related reading

· Kubernetes When Your k3s Cluster Goes NotReady Because the Clock is Too Fast · Containers When Bitnami Changed Their Docker Repository: A Vendor Lock-In Postmortem · Homelab Homelab Infrastructure as Code: Pi-hole + DNS + VPN on k3s