RESEARCH

The Price of Token Boundaries: Compression Certificates and Prediction

ArXiv cs.AI · Wed, 30 Sep 2026 04:00:00 GMT

arXiv:2609.35869v1 Announce Type: new Abstract: Pre-tokenisation restricts which text fragments can become prediction units, but its compression cost is obscured when tokenisers are compared only under the same boundaries. We measure this cost by bounding the minimum token count

Read original source Discuss with SiiMON