Query Parquet Files on S3 Without Downloading Them

The 40 GB problem

The dataset lives in a bucket, you want twenty rows, and the reflex is aws s3 cp. That downloads gigabytes to answer a question the file's own metadata could have answered. Parquet is designed for the opposite: its footer lists row groups and column chunks with exact byte offsets, so a reader that can issue HTTP range requests fetches only what a query touches. DuckDB's httpfs extension does exactly that.

Setup

INSTALL httpfs;
LOAD httpfs;

SELECT * FROM 's3://my-bucket/events/2026-09-01/part-000.parquet' LIMIT 20;

Any public HTTPS URL works with no credentials at all:

SELECT count(*) FROM 'https://data.example.com/exports/events.parquet';

Credentials without hardcoding them

Use the credential chain so DuckDB picks up the same profile, environment variables or instance role the AWS CLI uses. No keys in the SQL, nothing to leak into a notebook checked into git:

CREATE SECRET aws_default (TYPE s3, PROVIDER credential_chain);

The chain covers the usual sources in order, including the standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables, so a CI job needs no keys in the SQL either — only the region, whose absence is the single most common cause of an opaque failure:

CREATE SECRET aws_ci (
  TYPE s3,
  PROVIDER credential_chain,
  CHAIN 'env;config;instance',
  REGION 'ap-northeast-1'
);

The same mechanism covers other object stores: TYPE gcs for Google Cloud Storage with HMAC keys, TYPE r2 for Cloudflare R2, TYPE azure for Azure Blob Storage.

Globs and partition pruning

A prefix full of daily exports is one query:

SELECT dt, event_type, count(*) AS n
FROM read_parquet('s3://my-bucket/events/**/*.parquet', hive_partitioning = true)
WHERE dt BETWEEN '2026-09-01' AND '2026-09-07'
GROUP BY dt, event_type
ORDER BY dt;

With hive_partitioning = true, path segments like dt=2026-09-01/ become a real column, and the WHERE clause eliminates non-matching files before a single byte of their data is requested. Within the files that survive, row-group statistics in the footer prune further.

Two habits keep the transfer small, and therefore the query fast and the egress bill low:

• Name the columns. SELECT user_id, amount reads two column chunks; SELECT * reads all of them. On wide tables this is a 10x difference.
• Filter on partition keys first. A predicate on a regular column still requires opening every file's footer; a predicate on a partition key skips the files outright.

Gotchas

• s3a:// and s3n:// are Hadoop-specific schemes. DuckDB wants s3://.
• A glob needs s3:ListBucket on the prefix. Credentials that happily read one explicit key will fail on *.parquet with an access error.
• S3-compatible stores (MinIO, Ceph) need ENDPOINT and usually URL_STYLE 'path' in the secret, or every request goes to AWS instead.
• Repeated interactive queries re-request ranges. If you are iterating on the same slice, materialize it once: CREATE TABLE sample AS SELECT ....

Once the file is local

When you do pull a file down — a sample, a single partition, an export from a colleague — the browser tools take it from there with no install: the Parquet Viewer shows the schema and row count from the footer, and the SQL Workbench runs the same DuckDB SQL locally, with nothing uploaded. For the CLI and desktop options, see the DuckDB viewer comparison.