How to Compare Two Parquet Files and See What Changed

The situation

You have two versions of the same dataset — yesterday's export and today's, the output of a pipeline before and after a code change, or a file a colleague sent back "with a few fixes" — and you need to know exactly what changed. Renaming one of them _final does not answer that, and a byte comparison is useless with Parquet: row-group layout, compression and encodings change the bytes without changing a single value.

What you actually want is a semantic diff: join the two files on a key and classify every row as added, removed, changed or unchanged.

Option 1: diff in the browser (no install)

The Parquet Diff tool on this site does the join locally:

1. Drop the two files (old and new) onto the page.
2. Schema changes — added, removed or re-typed columns — appear immediately, read from the file footers before any rows are touched.
3. Pick the join key (an id-like column is preselected) and hit Compare rows.
4. You get counts of added / removed / changed / unchanged rows, and each changed row shows exactly which cells differ as old → new.

The comparison runs in an in-browser DuckDB database via WebAssembly, so nothing is uploaded — fine for confidential data, and multi-hundred-MB files work because the counting happens inside the database rather than in page memory.

Option 2: DuckDB SQL

The same idea expressed directly in SQL, if you already have DuckDB installed (or want to run it in the SQL Workbench):

-- Rows in new but not old (added)
SELECT * FROM read_parquet('new.parquet')
EXCEPT
SELECT * FROM read_parquet('old.parquet');

-- Changed rows for a key column `id`, showing both sides
SELECT o.id, o.amount AS amount_old, n.amount AS amount_new
FROM read_parquet('old.parquet') o
JOIN read_parquet('new.parquet') n USING (id)
WHERE o.amount IS DISTINCT FROM n.amount;

EXCEPT treats entire rows as the unit of comparison, so it cannot tell a "changed" row from a removed-plus-added pair. The join version can — that is why keyed comparison is the more useful shape, and it is what the browser tool automates for every column at once.

Option 3: pandas

import pandas as pd

old = pd.read_parquet("old.parquet").set_index("id")
new = pd.read_parquet("new.parquet").set_index("id")

added = new.loc[new.index.difference(old.index)]
removed = old.loc[old.index.difference(new.index)]

both = old.index.intersection(new.index)
changed_mask = (old.loc[both] != new.loc[both]).any(axis=1)
changed = new.loc[both][changed_mask]

This works, and it is what most people end up writing — but it loads both files fully into memory, needs care around NaN comparisons (NaN != NaN is True, so untouched null cells count as "changed" unless you handle them), and the script gets rewritten from scratch every time the schema is different.

Watch out for

• Non-unique keys. If the key repeats, the join multiplies rows and every count inflates. Verify uniqueness first (SELECT count(*), count(DISTINCT id) FROM ...).
• Type drift. A column that changed from DOUBLE to INT64 can make the join itself fail or silently compare unequal values. Compare schemas before comparing rows.
• Floats and timestamps. Exact equality on floating-point columns can flag rows that differ only by representation; decide whether a tolerance matters for your data before trusting a large "changed" count.