How (and why) to implement streaming in your LLM application
1–5 of 5 posts
Re: How (and why) to implement streaming in your LLM application
#2No mention of Unicode or UTF-8. Another incorrect implementation proliferates.
You need to be handling LLM output as partial UTF-8 encoded sequences, not as complete UTF-8 encoded strings.
In this case, this means you create one TextDecoder for the entire response, then call `decode` for each streamed token, with the `stream` option set to `true` [0]. That way, incomplete UTF-8 codepoints can be completed by future tokens, and not discarded.
If you do not do this, most languages other than English will not output properly. They will be interspersed with garbage, characters will be missing. Certainly anything using Chinese, Japanese, or Korean characters. It can become entirely unreadable and unusable.
This issue is present in the vast majority of scripts and interfaces that handle streaming responses from LLMs, and it's a huge pet peeve of mine.
I've also forwarded this concern to the author over email.
[0]: https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder...
Re: How (and why) to implement streaming in your LLM application
#3No mention of Unicode or UTF-8. Another incorrect implementation proliferates. You need to be handling LLM output as partial UTF-8 encoded sequences, not as complete UTF-8 encoded strings. In this case, this means you create one TextDecoder for the entire response, then call `decode` for each streamed token, with the `stream` option set to `true` [0]. That way, incomplete UTF-8 codepoints can be completed by future t…
hey, thanks for pointing this out! we work on english only applications so missed this. will add a note to the blog
Re: How (and why) to implement streaming in your LLM application
#4No mention of Unicode or UTF-8. Another incorrect implementation proliferates. You need to be handling LLM output as partial UTF-8 encoded sequences, not as complete UTF-8 encoded strings. In this case, this means you create one TextDecoder for the entire response, then call `decode` for each streamed token, with the `stream` option set to `true` [0]. That way, incomplete UTF-8 codepoints can be completed by future t…
hey, thanks for pointing this out! we work on english only applications so missed this. will add a note to the blog
ty! feel free to excuse the email if it sounds a bit agitated, this just bites me all over the place so it's really annoying to see it perpetuated
Re: How (and why) to implement streaming in your LLM application
#5No mention of Unicode or UTF-8. Another incorrect implementation proliferates. You need to be handling LLM output as partial UTF-8 encoded sequences, not as complete UTF-8 encoded strings. In this case, this means you create one TextDecoder for the entire response, then call `decode` for each streamed token, with the `stream` option set to `true` [0]. That way, incomplete UTF-8 codepoints can be completed by future t…
hey, thanks for pointing this out! we work on english only applications so missed this. will add a note to the blog
There doesn't seem to be any note or correction yet, do you have an ETA on this?