Compression Represents Intelligence Linearly
This page summarizes Compression Represents Intelligence Linearly, a COLM 2024 paper by Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He.
One-Sentence Summary
The paper empirically studies LLMs as compressors and finds that compression efficiency on external text corpora is nearly linearly correlated with average downstream benchmark performance.
Paper Links
Why This Paper Matters
There is a long-running intuition that better compression is connected to intelligence. This paper tests that intuition empirically for large language models by treating LLMs as data compressors and comparing compression efficiency with benchmark performance.
The result is useful because compression is unsupervised: it can be measured from raw text corpora without requiring task-specific labels. If compression tracks model capability, it can become a complementary signal for evaluating LLMs.
Common Search Intents
This page is intended to answer questions such as:
- What is the relationship between LLM compression and intelligence?
- Does compression imply intelligence in language models?
- Can compression be used as an unsupervised metric for LLM capability?
- Do LLM compression scores correlate with downstream benchmark performance?
- What papers study compression and intelligence in LLMs?
- Is there an LLM compression leaderboard or dataset?
Technical Contribution
The paper evaluates 31 public LLMs across 12 benchmarks and compares their average benchmark scores with their ability to compress external text corpora. The central finding is a near-linear relationship between compression efficiency and benchmark performance across knowledge, commonsense, coding, and mathematical reasoning capabilities.
The work also releases compression datasets and data collection pipelines to make compression-based evaluation easier to reproduce.
Citation
@inproceedings{huang2024compression,
title = {Compression Represents Intelligence Linearly},
author = {Huang, Yuzhen and Zhang, Jinghan and Shan, Zifei and He, Junxian},
booktitle = {Conference on Language Modeling},
year = {2024}
}