Notebook Compression

Jupyter notebooks are a compressor's puzzle box: JSON structure, source code, markdown prose, and, dominating the bytes, base64-encoded cell outputs. In this corpus roughly 64% of all bytes are base64 image/png payloads, so beating a general-purpose compressor requires exploiting the format itself: undoing the base64 layer, modeling the JSON envelope, and finding redundancy across hundreds of related notebooks, all while guaranteeing byte-exact reconstruction.

The corpus is a fixed selection of real, permissively-licensed notebooks from twelve pinned open-source repositories (TensorFlow docs, Hugging Face, OpenAI cookbook, scikit-style ML tutorials, cloud-vendor samples, and more), shipped byte-for-byte as committed upstream (no re-serialization, no output stripping), with per-file provenance and hashes in a manifest.

The agent must compress the Jupyter notebooks in /app/corpus/ as small as possible, losslessly, and ship the result as a self-extracting archive under /app/dist/.

• python3 /app/dist/decompress.py <out_dir> must recreate every notebook byte-for-byte into <out_dir>, using only what is under /app/dist/.
• The total size of /app/dist/(file contents plus every entry's path name, with only __pycache__excluded) counts as the compressed size, so the decoder's own code is part of the price.
• The decoder may shell out to tools already installed on the machine (those do not count toward the size); anything custom must live inside /app/dist/.
• Changes are confined to /app/dist, and the workspace must stay in working condition in case the sandbox times out mid-experiment.

The full ~650 MB notebook corpus at /app/corpus/ (the exact bytes that will be scored, not a proxy), plus python3 /app/check.py, which runs the decoder into a temporary directory, verifies the byte-exact round trip (with a --sample flag for fast iteration), and reports the size of /app/dist against the recorded xz -9 baseline.

The machine is offline (Ubuntu 24.04) with a compression-oriented toolchain preinstalled: standard compression CLIs and headers, a build toolchain, and a Python stack including zstandard, brotli, lz4, NumPy, pandas, SciPy, and PyArrow. A sandbox-timer CLI reports the remaining budget.

A clean-room verifier in a separate container scores the captured submission against root-only ground truth. It first wipes any harness logs restored from the agent's trial so a decoder cannot read an out-of-band stash, rejects any symlink under the submission, copies /app/dist into a scratch directory, and empties /app entirely before decoding; the decoder runs unprivileged with only its own files.

• The decode stage is capped at 30 minutes; every recovered notebook is then compared byte-for-byte against the root-only corpus copy, with symlinked or non-regular outputs rejected. Any mismatch means the submission is not lossless and scores 0.
• The metric is the submission ratio: total /app/dist bytes (contents plus path names) divided by corpus bytes; lower is better.
• Reward is anchored to two fixed points: matching a whole-corpus xz -9 self-extracting archive scores 0, and reaching 15% of the corpus size scores 1.0. The curve is measured in compression factor and bent exponentially (2^u − 1), so each further halving of the archive is worth more than the last.
• A decoder that fails, times out, or depends on anything outside /app/dist zeroes the result.