saahityaedams

13 Mar 2026

Querying ALB Access Logs Via Duckdb

  • Ensure that you have duckdb installed on your local machine.
  • Ensure you have the enabled appropriate settings to ensure that access logs are logged to a s3 bucket
  • In your duckdb console run the following to let duckdb know how to pickup the AWS creds. (ensure you have aws creds that let you access this bucket and its contents)
INSTALL aws;
LOAD aws;
CALL load_aws_credentials();
SET s3_region='ap-south-1';
  • The following points alb_logs table in local to the folder at s3 path and indicates the rows.
CREATE VIEW alb_logs AS
    SELECT * FROM read_csv(
        's3://abc-ace-alb-new-access-logs-prod/access-logs/AWSLogs/ABC/elasticloadbalancing/ap-south-1/2026/03/11/*.log.gz',
    delim=' ',
    quote='"',
    header=false,
    columns={
        'type': 'VARCHAR',
        'time': 'TIMESTAMP',
        'elb': 'VARCHAR',
        'client': 'VARCHAR',
        'target': 'VARCHAR',
        'request_processing_time': 'DOUBLE',
        'target_processing_time': 'DOUBLE',
        'response_processing_time': 'DOUBLE',
        'elb_status_code': 'INTEGER',
        'target_status_code': 'VARCHAR',
        'received_bytes': 'BIGINT',
        'sent_bytes': 'BIGINT',
        'request': 'VARCHAR',
        'user_agent': 'VARCHAR',
        'ssl_cipher': 'VARCHAR',
        'ssl_protocol': 'VARCHAR',
        'target_group_arn': 'VARCHAR',
        'trace_id': 'VARCHAR',
        'domain_name': 'VARCHAR',
        'chosen_cert_arn': 'VARCHAR',
        'matched_rule_priority': 'VARCHAR',
        'request_creation_time': 'VARCHAR',
        'actions_executed': 'VARCHAR',
        'redirect_url': 'VARCHAR',
        'error_reason': 'VARCHAR',
        'target_port_list': 'VARCHAR',
        'target_status_code_list': 'VARCHAR',
        'classification': 'VARCHAR',
        'classification_reason': 'VARCHAR',
        'conn_trace_id': 'VARCHAR',
        'alpn_fe_protocol': 'VARCHAR',
        'alpn_be_protocol': 'VARCHAR',
        'alpn_client_preference_list': 'VARCHAR'
    }
);
  • You can run queries in the duckdb console like this
SELECT time, request, target_processing_time, elb_status_code
    FROM alb_logs
    WHERE target_processing_time > 5
        AND request LIKE '%/payment/%'
    ORDER BY target_processing_time DESC LIMIT 5;
    
100% ▕████████████████████████████████████████████████████████████▏
┌────────────────────────────┬───────────────────────────────────────────────────────────────────────────────┬────────────────────────┬─────────────────┐
│            time            │                                    request                                    │ target_processing_time │ elb_status_code │
│         timestamp          │                                    varchar                                    │         double         │      int32      │
├────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────┼─────────────────┤
│ 2026-03-11 09:50:42.346488 │ POST https://api.abc.com:443/payment/v2/authenticate HTTP/2.0             │                 39.629 │             200 │
│ 2026-03-11 04:20:04.4695   │ GET https://api.abc.com:443/payment/v3/getPendingCollectRequests HTTP/2.0 │                 12.489 │             200 │
│ 2026-03-11 09:42:41.942846 │ POST https://api.abc.com:443/payment/v4/secure/authenticate HTTP/2.0      │                 11.219 │             200 │
│ 2026-03-11 04:20:04.804197 │ GET https://api.abc.com:443/payment/v3/getPendingCollectRequests HTTP/2.0 │                 10.906 │             200 │
│ 2026-03-11 04:20:03.692783 │ GET https://api.abc.com:443/payment/v3/getPendingCollectRequests HTTP/2.0 │                 10.895 │             200 │
└────────────────────────────┴───────────────────────────────────────────────────────────────────────────────┴────────────────────────┴─────────────────┘
  • AND the greatest thing is copying this into claude code, asking it to make a skill and then asking that skill to get answers to high level questions (source ip for spikes in traffic, suspicious user agents, etc, weird apis being hit)