181–190 of 244 posts
Very cool point, and it applies to any tree data in general. I like to represent tree data with parent, type, and data vectors along with a string table, so everything else is just small integers. Sending the string table and type info as upfront headers, we can follow with a stream of parent and data vector chunks, batched N nodes at a time. Tye depth- or breadth-first streaming becomes a choice of ordering on the v…
... It might be a pursuit worth making a small library for.
Very simple API, takes a stream of string chunks and returns a stream of increasingly complete values. Helpful for parsing large JSON, and JSON being emitted by LLMs.
Extensively tested and performance optimized. Guaranteed that the final value emitted is identical to passing the entire string through JSON.parse.
Earlier quoted context omitted.
I already use streaming partial json responses (progressive json) with AI tool calls in production. It’s become a thing, even beyond RSCs, and has many practical uses if you stare at the client and server long enough.
how do you do that exactly?
Earlier quoted context omitted.
how do you do that exactly?
Not the original commenter but I’ve done this too with Pydantic AI (actually the library does it for you). See “Streaming Structured Output” here https://ai.pydantic.dev/output/#streaming-structured-output
I've been trying to create go/rust ones but its way harder than just json due to all the context/state they carry over
Earlier quoted context omitted.
hi dan! really interesting post. do you think a new data serialization format built around easier generation/parseability and that also happened to be streamable because its line based like jsonld could be useful for some?
I don’t know! I think it depends on whether you’re running into any of these problems and have levers to fix them. RSC was specifically designed for that so I was trying to explain its design choices. If you’re building a serializer then I think it’s worth thinking about the format’s characteristics.
As of right now, I could only replace the JSON tool calling on LLM's on something I fully control like vLLM, and the big labs probably are happy to over-charge a 20-30% tokens for each tool call, so they wouldn't really be interested on replacing json any time soon)
also it feels like battling against a giant which is already an standard, maybe there's a place for it on really specialized workflows where those savings make the difference (not only money, but you also gain a 20-30% extra token window, if you don't waste it on quotes and braces and what not
Thanks for replying!
Earlier quoted context omitted.
how do you do that exactly?
One way is to eagerly call JSON.parse as fragments are coming in. If you also split on json semantic boundaries like quotes/closing braces/closing brackets, you can detect valid objects and start processing them while the stream continues.
I think the problem with this is that it makes a very simple thing a lot harder. I don’t want to try and debug a JSON stream that can fail at any point. I just want to send a block of text (which I generate in 2ms anyway) and call it a day.
Compared with waiting on a blank page for ages, sometimes it's nice to see text content if it's useful, and to be able to click navigation links early. It's much better than pages which look like they have finished loading but important buttons and drop-downs are broken without any visible indication because there's JS still loading in the background. I'm also not fond of pages where you can select options and enter data, and then a few seconds after you've entered data, all the fields reset as background loading completes.
All the above are things I've experienced in the last week.
I have seen Dan's "2 computers" talk and read some of his recent posts trying to explore RSC and their benefits. Dan is one of the best explainers in React ecosystem but IMO if one has to work this hard to sell/explain a tech there's 2 possibilities 1/ there is no real need of tech 2/ it's a flawed abstraction #2 seems somewhat true because most frontend devs I know still don't "get" RSC. Vercel has been aggressively…
While RSC as technology is interesting, I don't think it makes much sense in practice. I don't want to have a fleet of Node/Bun backend servers that have to render complex components. I'd rather have static pages and/or React SPA with Go API server. You get similar result with much smaller resources.
Earlier quoted context omitted.
> Now reuse the same connection to request the nested data, which can all have more nested links in them, and so on. This still involves multiple round-trips though. The approach laid out in the article lets you request exactly the data you need up-front and the server streams it in as it becomes available, e.g. cached data first, then data from the DB, then data from other services, etc.
When you have an HTTP/2 connection already open a 'round-trip' is not really a gigantic concern performance-wise. And it gives the client application complete control and ver what nested parts it wants to get and in what order. Remember that the article said it's up to the server what order to stream the parts? That might not necessarily be a good idea on the client side though. It would probably be better for the cl…
However, sequentially-dependent requests are about as slow with HTTP/2 as HTTP/1.1. For example, if your client side, after loading the page, requests data to fill a form component, and then that data indicates a map location, so your client side requests a map image with pins, and then the pin data has a link to site-of-interest bubble content, and you will be automatically expanding the nearest one, so your client side requests requests the bubble content, and the bubble data has a link to an image, so the client requests the image...
Then over HTTP/2 you can either have 1 x round trip time (server knows the request hierarchy all the way up to the page it sends with SSR) or 5 x round trip time (client side only).
When round trip times are on the order of 1 second or more (as they often are for me on mobile), >1s versus >5s is a very noticable difference in user experience.
With lower latency links of 100ms per RTT, the UX difference between 100ms and 500ms is not a problem but it does feel different. If you're on <10ms RTT, then 5 sequential round trips are hardly noticable, thought it depends more on client-side processing time affecting back-to-back delays.