The Spectrum Dispatch News

technology

UTF-8000 Proposes Unlimited-Length UTF‑8 Encoding

The project describes a variable‑length encoding that extends UTF‑8 to arbitrarily many bytes while preserving ASCII compatibility and self‑synchronization.

UTF-8000 Proposes Unlimited-Length UTF‑8 Encoding

UTF-8000 is presented as an unlimited UTF‑8 encoding that keeps ASCII as a subset and introduces no special cases, preserving all existing UTF‑8 properties. The source notes that a reference implementation can be tried with the command $ pipx install UTF-8000 and clarifies that the project is not endorsed by or representative of the Unicode Consortium, describing it as a fun standalone project or proposal. The TLDR section shows how UTF‑8000 builds on the familiar UTF‑8 patterns: ASCII uses one byte, 2‑byte UTF‑8 uses the pattern 110xxxxx 10xxxxxx, 3‑byte uses 1110xxxx 10xxxxxx 10xxxxxx, and 4‑byte uses 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx. UTF‑8000 continues this progression with 5‑byte units beginning 111110xx followed by four continuation bytes, 6‑byte units beginning 1111110x followed by five continuation bytes, and so on, up to a demonstrated 22‑byte unit that starts with eight leading 1 bits in the first byte, then a series of 10xxxxxx bytes, and ends with a final byte that carries the remaining content bits. The anatomy explanation highlights that UTF‑8000’s main contribution is clarity on splitting the highest bits of the first byte into self‑synchronization bits and start bits, then striping those start bits across the continuation bytes when needed to encode arbitrarily large code units. The glossary defines key terms: a codepoint is a non‑negative integer; a code unit is the sequence of UTF‑8000 bytes that encode a single codepoint; the first byte begins a unit and necessarily is a start byte, distinguished by its self‑synchronization prefix; continuation bytes follow the first byte and share the same self‑synchronization prefix; the self‑synchronization prefix consists of the highest bits of each byte and allows instant identification of whether a byte is a first or continuation byte, enabling random access and error recovery. Start bytes contain the unary‑coded start bits that indicate the length of the unit in bytes; content bytes hold the content bits where the codepoint’s binary value is stored. Only two special cases are retained from UTF‑8: ASCII remains unchanged, and the original 2‑byte UTF‑8 overlong check uses four mandatory content bits while longer units use five. The 22‑byte example is offered as a prototypical illustration, not a special case, showing how multiple start bytes can be combined to reach arbitrary lengths. Overall, UTF‑8000 is offered as a clear, experimental proposal for extending UTF‑8’s self‑synchronizing, self‑punctuating design to unlimited code unit sizes, accessible for testing via the pipx‑installable reference implementation.

UTF-8000 Proposes Unlimited-Length UTF‑8 Encoding

Key facts

Sources

← All posts