Combine Multiple CSV Files into One Parquet File

The situation

A system exports one CSV per day, store or batch — 2026-08-01.csv, 2026-08-02.csv, dozens more — and you want them as a single fast, compressed file. Concatenating with cat breaks immediately: every file repeats its header row, and any file whose columns are ordered differently silently corrupts the result. The right move is to union them properly and write Parquet once.

One command with DuckDB

The DuckDB CLI↗ does the whole job with a glob:

duckdb -c "COPY (
  SELECT * FROM read_csv('exports/*.csv', union_by_name=true)
) TO 'combined.parquet' (FORMAT PARQUET, COMPRESSION zstd);"

union_by_name=true is the important option. Without it, files are matched column-by-position, so an export whose columns were reordered in March puts values into the wrong fields. With it, columns are matched by header name, files missing a column get NULL there, and schema drift — a field added halfway through the year — merges cleanly.

To record where each row came from, let DuckDB add the source path as a column:

duckdb -c "COPY (
  SELECT * FROM read_csv('exports/*.csv', union_by_name=true, filename=true)
) TO 'combined.parquet' (FORMAT PARQUET, COMPRESSION zstd);"

When type inference bites

CSV has no types, so DuckDB infers them from a sample of each file. Across many files this can disagree: a ref_code column that happens to be all digits in January reads as BIGINT, then February contains A-1023 and the union fails with a conversion error. Two fixes, in order of preference:

-- Scan entire files instead of a sample before deciding types
read_csv('exports/*.csv', union_by_name=true, sample_size=-1)

-- Nuclear option: read everything as text, cast the columns you trust
read_csv('exports/*.csv', union_by_name=true, all_varchar=true)

With all_varchar=true, cast deliberately in the SELECT (CAST(amount AS DECIMAL(12,2))) so the Parquet file ends up with real types rather than strings — the compression and query-speed benefits of Parquet largely come from typed columns.

Doing it in the browser instead

No terminal available, or the data too sensitive to move around? Drop all the CSVs into the SQL Workbench — it runs DuckDB as WebAssembly locally, nothing is uploaded. Globs cannot expand in a browser, so list the files explicitly:

SELECT * FROM 'day1.csv'
UNION ALL BY NAME
SELECT * FROM 'day2.csv'
UNION ALL BY NAME
SELECT * FROM 'day3.csv';

Download the result as CSV, then feed it to the CSV to Parquet converter, which re-infers types and writes a compressed Parquet file. Fine for a handful of files; for fifty, the CLI glob above is the better tool.

Verify the output

Check the result before deleting anything: drop combined.parquet into the Parquet Viewer and confirm the row count matches the sum of the inputs and the schema shows the types you expect — not VARCHAR everywhere. If your inputs are already Parquet rather than CSV, the mechanics differ slightly; see How to Merge Multiple Parquet Files into One.