kaal:claim:5541658-032

Benchmark results for legal LLMs may overstate capability because of data contamination: if a model saw a benchmark's ground truth answers during training, its measured performance reflects memorization rather than genuine generalization.

Source quote, verbatim

If a model has already seen a benchmark's ground-truth answers during training, its performance may reflect memorization rather than genuine generalization, making it hard to assess its ability on truly unseen tasks.

From

Wulf A. Kaal, Morgan A. Gray, The Evolving Role of Artificial Intelligence in Law (2025), A. Improved Algorithms for Complex Legal Datasets, p. 35
https://ssrn.com/abstract=5541658 · source PDF

Cite as

Wulf A. Kaal, Morgan A. Gray, The Evolving Role of Artificial Intelligence in Law (2025). SSRN: https://ssrn.com/abstract=5541658

Holds when

closed source LLMs evaluated on public benchmarks

Classification

failuresupport: evidencedfailure: benchmark data contaminationfamily: measurement-and-metric-failureeconomics

Verify

The quote above is an exact substring of the source PDF, whose sha256 is e543a2d698fcd522d4d02e034cc9ee1344d0015d2c824b40b9e05ab7c0728c60. Extraction method: pdf-text-layer.
Attestation record: colloquium/attestations/e4da1d77cc559c12...json
Verify the binding yourself: curl -s https://wulfkaal.github.io/claims/5541658-032.md | sha256sum