Check If Two Parquet Files Contain Identical Data

The question is yes or no

Sometimes you don't need a full diff. You rewrote a file with a different codec, compacted a partition, moved a pipeline to a new engine, or got a "fixed" file from a colleague, and you only need to know one thing: is the data still exactly the same? If it is, you're done. If it isn't, then you go hunting for what changed.

The obvious check gets it wrong:

shasum -a 256 old.parquet new.parquet

Matching checksums do mean the data is identical. Different checksums mean nothing, though. Changing the codec from Snappy to Zstd, writing different row-group sizes or even upgrading the writer library changes every byte while every value stays put. You have to compare the data instead.

Step 1: compare the schemas

Column names, order and types must match before comparing values means anything. Run this for both files in the SQL Workbench, or drop them into the Parquet Viewer:

DESCRIBE SELECT * FROM 'old.parquet';
DESCRIBE SELECT * FROM 'new.parquet';

A column that changed from INTEGER to BIGINT holds the same numbers but hashes differently. Settle whether that counts as a change before going on.

Step 2: a fingerprint per file

This query reduces a whole file to two numbers:

SELECT count(*) AS n, sum(hash(t)) AS fingerprint
FROM read_parquet('old.parquet') t;

hash(t) hashes the entire row (in DuckDB, t refers to the row as a struct). sum adds those hashes up, and since addition ignores order, a file sorted differently gives the same fingerprint. Run the same query on new.parquet. If both n and fingerprint match, the files hold the same multiset of rows.

Use sum, not bit_xor. XOR cancels pairs, so a file where one row got duplicated twice would wrongly match the original.

This is a single streaming scan, so it works on files much larger than memory, and it also works across the files of a dataset:

SELECT count(*), sum(hash(t))
FROM read_parquet('events/*/*.parquet', hive_partitioning = true) t;

Step 3: an exact check, if you need proof

A fingerprint is probabilistic. When you want an exact answer, take the difference in both directions with EXCEPT ALL, which respects duplicates:

SELECT count(*) AS differing_rows FROM (
  (SELECT * FROM 'old.parquet' EXCEPT ALL SELECT * FROM 'new.parquet')
  UNION ALL
  (SELECT * FROM 'new.parquet' EXCEPT ALL SELECT * FROM 'old.parquet')
);

Zero means identical. EXCEPT matches columns by position, so if only the column order differs, list the columns explicitly in the same order on both sides.

Comparing Parquet with its CSV export

The same fingerprint checks whether a CSV conversion lost anything. Cast the CSV back to the Parquet types first:

SELECT count(*), sum(hash(t)) FROM (
  SELECT id::BIGINT AS id, amount::DECIMAL(18,2) AS amount, created_at::TIMESTAMP AS created_at
  FROM read_csv('export.csv')
) t;

If this matches the Parquet fingerprint, the round trip kept every value. If it doesn't, the usual suspects are timestamps that lost their time zone or sub-second precision, and floats printed with fewer digits.

When the answer is "no"

A mismatch tells you that something changed, not what. To see which rows were added, removed or edited, use the Parquet Diff tool or the keyed SQL in comparing two Parquet files. Floating-point columns deserve a look first: -0.0 versus 0.0, or a value recomputed with a different summation order, changes the hash even though nobody would call it a real change.