How to Stream Base64 into UTF-8 Text in Python

Keep incomplete Base64 quartets between reads, and keep one incremental UTF-8 decoder across the decoded byte batches. Flush that decoder at the actual end of the stream; a complete Base64 group can still end inside a UTF-8 character.

Where do the bytes wait when a quartet splits a character?

The fixture that crosses both boundaries

We generated the fixtures on 2026-10-21 with Python 3.12.14. The Unicode input A€🚀Z occupies 9 bytes: 41e282acf09f9a805a. Its Base64 value is QeKCrPCfmoBa, exactly 12 ASCII characters. Our ASCII control ABCDEFGHI also occupies 9 bytes and encodes to 12 characters, QUJDREVGR0hJ. We held input length constant while changing the character boundaries.

The first group produced 3 bytes but emitted only A; e282 remained buffered. The second group completed the euro sign and left f09f buffered for the rocket. The third group completed that rocket and emitted Z, leaving an empty buffer. The three decoded batches total 3 + 3 + 3 = 9 bytes. The final flush added no text and raised no error.

The distinction is not a new Base64 alphabet or a different character set. For an entire pasted value, use the Base64 decoder. This experiment addresses a separate operation: converting arriving fragments into text before the complete value has arrived.

Changing the read size does not remove the second state

We tested read sizes 1, 2, 3, 4, 5, 7 and 12 against the unchanged Unicode file. Both paths used the same Base64 remainder handling. The stateful path recovered all 9 bytes and A€🚀Z in every row. The stateless text path failed in the first six rows; only the 12-character read supplied the whole text in one decoded batch. A four-character transport read was therefore not sufficient for this input.

Every possible partition, with an ASCII control

We then enumerated every way to cut each supplied Base64 value into nonempty contiguous fragments. A 12-character value has 11 internal cut positions. Each position is either cut or uncut, giving 2 ** 11 = 2048 partitions. The CSV records each cut mask and both outcomes. This is exhaustive for these short values; it is not a random sample of files or a prediction about production traffic.

The Unicode fixture completed in all 2,048 stateful runs, compared with 8 stateless runs; 2,048 minus 8 leaves 2,040 rejections. The same-length ASCII control completed in all 2,048 runs on both paths. Our successful ASCII tests would have hidden the Unicode boundary fault. For each successful Unicode run, the script also compared the recovered hex with 41e282acf09f9a805a and the text with A€🚀Z, rather than accepting the absence of an exception.

A waiting character is different from a truncated stream

For the truncated UTF-8 control, we removed the final two bytes, 80 5a, from our 9-byte Unicode fixture. The remaining file measures 7 bytes and still has a valid 12-character Base64 spelling, QeKCrPCfmg==. All 7 bytes survive strict Base64 decoding. After the three groups, the text prefix is A€ and the UTF-8 buffer contains f09f9a: 4 emitted-text bytes plus 3 waiting bytes equals the 7-byte input. Finalization rejects that incomplete rocket.

'utf-8' codec can't decode bytes in position 0-2: unexpected end of data

We also removed only the last encoded character from the complete fixture. That different control contains 11 Base64 characters. Our strict fixture policy consumes 8 characters into 6 bytes and leaves moB, a three-character Base64 remainder. It rejects at the Base64 end check before attempting the UTF-8 final flush. Both controls expose the text prefix A€, but the recorded failure phases differ. The report retains each remainder so that the visible prefix cannot be mistaken for a complete result.

The invalid-byte control is a third case: 41ff5a, 3 bytes encoded as Qf9a. Base64 round-trips it exactly, then strict UTF-8 fails at byte position 1 during the feed operation, with reason invalid start byte. It has only 4 encoded characters and 3 internal cut positions, hence 8 partitions; none completed as UTF-8. This control did not need an end-of-stream flush to reveal its error.

Reproduce the measurements

Download reproduce.py and run python reproduce.py. It creates a generated directory beside the script containing the fixtures, measurements.json and partitions.csv. No packages or network access are required. This shorter check isolates the three complete quartets; the full script also handles transport remainders and the negative controls.

import base64, codecs



groups = ["QeKC", "rPCf", "moBa"]

decoder = codecs.getincrementaldecoder("utf-8")("strict")

text = ""

for group in groups:

    raw = base64.b64decode(group, validate=True)

    text += decoder.decode(raw, final=False)

    print(raw.hex(), repr(text), decoder.getstate()[0].hex())

text += decoder.decode(b"", final=True)

assert text == "A\u20ac\U0001f680Z"

print(len(text.encode("utf-8")), repr(text))

Exact fixture downloads: Unicode bytes, Unicode Base64, ASCII control, truncated UTF-8 control, invalid-byte control. The Unicode binary SHA-256 is 7263630b1422e120fbe21f9a8689aeb946d60af1928c533e320427dc8dbcbafa.

What to keep, and what this example does not handle

Use a separate accumulator for encoded characters and a persistent decoder for text. Python documents the behavior of incremental decoding and finalization; b64decode returns bytes. Buffering and incremental UTF-8 are established techniques. The measured fixture traces, partition outcomes and paired controls above are our contribution, not a claim to have invented those techniques.

The script is a small-fixture experiment, not a production parser. It expects one continuous, unwrapped standard Base64 value with complete terminal quartets and any required padding. It deliberately rejects an incomplete terminal quartet rather than guessing whether transport lost characters. JSON framing, MIME whitespace, Base64url, concatenated separately padded messages and encoding detection require their own explicit rules. Keep output provisional until finalization succeeds; set input and output limits when adapting this to larger streams. The demo uses fixed public samples and sends no sample text to a server.

Can I add padding to every arriving fragment?

No. An arriving fragment is not necessarily the end of the encoded value. Carry an unfinished group to the next read. If the protocol permits omitted padding, handle that only at the confirmed end under that protocol's rules.

Does a successful Base64 decode prove that the text is intact?

No. Check the bytes, then the intended text encoding and its final state. For compressed data, there is another boundary: the gzip member experiment tests whether decompression consumes all members.

For intact XML bytes that become the wrong characters before parsing, compare the XML declaration and predecoding experiment; that concerns encoding selection, not split multibyte sequences.