The author of the blog post investigates whether the general‑purpose compressor gzip can be repurposed as a language model without any neural network or learned parameters. According to the post, the experiment relies on the well‑known equivalence between compression and prediction: a model that assigns high probability to a sequence will encode it in few bits, and any compressor implicitly contains such a probability model. The post explains that gzip’s DEFLATE algorithm works by searching for matches of the incoming bytes in a 32 KiB sliding window; when a continuation matches something already in the window it is encoded as a cheap back‑reference, resulting in a small increase in compressed size. This property lets the compressor score candidate continuations: the shorter the gzip output of context + candidate, the more the continuation is expected by the model. To turn scoring into generation, the author describes a beam search procedure implemented in a tool called gzipt. The process begins with a user‑supplied prompt, which becomes part of the context that gzip sees alongside a priming corpus loaded into its window. At each step the algorithm keeps the beam_width most compressible partial continuations, tries every byte that appears in the corpus as a possible next token, scores each by measuring the length of gzip(context + candidate), and prunes back to the best beam_width candidates. After a predefined horizon the most compressible full span is selected (or sampled if a temperature parameter is set) and appended to the generated text, after which the loop repeats. The post notes that only the most recent tail of the generated output is retained in the scoring context to avoid degenerate verbatim loops that would arise if the entire history were visible to DEFLATE. The author provides unedited samples obtained after priming gzipt on a tiny Shakespeare corpus. The generated text, while not perfectly coherent, contains recognizable fragments and follows stylistic patterns of the source material, suggesting that the compressor has captured some statistical regularities. The post also mentions that a naïve approach of picking the single best‑scoring next byte fails because gzip reports only integer byte lengths, causing many ties; the beam search look‑ahead resolves this quantization noise. Finally, the author acknowledges that the original paper Language Modeling is Compression reported poor performance with this method, but that adding beam search—an idea noted in that paper—substantially improves generation quality.
