Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preventing Flash of Incomplete Markdown when streaming AI responses

engineering.streak.com

Post reply on HN
Print view View original

11–14 of 14 posts

Re: Preventing Flash of Incomplete Markdown when streaming AI responses

#11

Post by kristopolous »

kherud wrote: post #3

Is there a general solution to this problem? I assume you can only start buffering tokens once you see a construct, for which there are continuations, that once completed, would lead to the previous text being rendered differently. Of course you don't want to keep buffering for too long, since this would defeat the purpose of streaming. And you never know if the potential construct will actually be generated. Also, t…

This exact problem is why I wrote Streamdown https://github.com/day50-dev/Streamdown

Almost every model has a slight but meaningfully different opinion on what markdown is and how creative they can be with it.

Doing it well is a non-trivial problem.

Re: Preventing Flash of Incomplete Markdown when streaming AI responses

#12

Post by munch117 »

kherud wrote: post #3

Is there a general solution to this problem? I assume you can only start buffering tokens once you see a construct, for which there are continuations, that once completed, would lead to the previous text being rendered differently. Of course you don't want to keep buffering for too long, since this would defeat the purpose of streaming. And you never know if the potential construct will actually be generated. Also, t…

Generating simple HTML instead of markdown would have been a solution. But I guess that ship has sailed.

Re: Preventing Flash of Incomplete Markdown when streaming AI responses

#13

Post by graboy »

kherud wrote: post #3

Is there a general solution to this problem? I assume you can only start buffering tokens once you see a construct, for which there are continuations, that once completed, would lead to the previous text being rendered differently. Of course you don't want to keep buffering for too long, since this would defeat the purpose of streaming. And you never know if the potential construct will actually be generated. Also, t…

Yes. You can define a regex matching what you want, and every regex can be compiled into a state machine (https://en.wikipedia.org/wiki/Nondeterministic_finite_automa...). Then at each character you make a step in your state machine. You pause the output while the regex is not matching.

Re: Preventing Flash of Incomplete Markdown when streaming AI responses

#14

Post by monkeycantype »

porridgeraisin wrote: post #6
sim7c00 wrote: post #2

fun read, its weird interacting with chatgpt around markdown sometimes. it formats its own stuff with markdown, so if i ask it for markdown but dont explicitly specify a downloadable file, it will produce valid markdown up to where it conflicts with its own markdown, and then it gets choppy and chunked. its an issue of my prompting is what im sure some customer service rep would be told to tell me :p because theres m…

You can ask for it to put the markdown in a codeblock. It works well for me. It also works with latex.

I also ask for any nested codeblocks to be delimited with ~~~, not ``` so as not to break out of the code block
Post reply on HN