Reinventing pagination
?page=3&size=100
Progressive JSON
21–30 of 244 posts
Re: Progressive JSON
#22This would be good. I got really, really sick of XML, but one thing that XML parsers have always been good at, is realtime decoding of XML streams. It is infuriating, waiting for a big-ass JSON file to completely download, before proceeding. Also JSON parsers can be memory hogs (but not all of them).
Json is just a packing format that does have that limitation. If you control the source and the destination, could you possibly use a format that supports streaming better like Protobuf?
Re: Progressive JSON
#23This would be good. I got really, really sick of XML, but one thing that XML parsers have always been good at, is realtime decoding of XML streams. It is infuriating, waiting for a big-ass JSON file to completely download, before proceeding. Also JSON parsers can be memory hogs (but not all of them).
Json is just a packing format that does have that limitation. If you control the source and the destination, could you possibly use a format that supports streaming better like Protobuf?
Yes, but that also interferes with portability.
I’ve written a lot of APIs. I generally start with CSV, convert that to XML, then convert that to JSON.
CSV is extremely limited, and there’s a lot of stuff that can only be expressed in XML or JSON, but starting with CSV usually enforces a “stream-friendly” structure.
Re: Progressive JSON
#24I'll try to explain why this is a solution looking for a problem.
Yes, breadth-first is always an option, but JSON is a heterogenous structured data source, so assuming that breadth-first will help the app start rendering faster is often a poor assumption. The app will need a subset of the JSON, but it's not simply the depth-first or breadth-first first chunk of the data set.
So for this reason what we do is include URLs in JSON or other API continuation identifiers, to let the caller choose where in the data tree/graph they want to dig in further, and then the "progressiveness" comes from simply spreading your fetch operation over multiple requests.
Also often times JSON is deserialized to objects so depth-frst or breadth-first doesn't matter, as the object needs to be "whole" before you can use it. Hence again: multiple requests, smaller objects.
In general when you fetch JSON from a server, you don't want it to be so big that you need to EVEN CONSIDER progressive loading. HTML needs progressive loading because a web page can be, historically especially, rather monolithic and large.
But that's because a page is (...was) static. Thus you load it as a big lump and you can even cache it as such, and reuse it. It can't intelligently adapt to the user and their needs. But JSON, and by extension the JavaScript loading it, can adapt. So use THAT, and do not over-fetch data. Read only what you need. Also, JSON is often not cacheable as the data source state is always in flux. One more reason not to load a whole lot in big lumps.
Now, I have a similar encoding with references, which results in a breadth-first encoding. Almost by accident. I do it for another reason and that is structural sharing, as my data is shaped like a DAG not like a tree, so I need references to encode that.
But even though I have breadth-first encoding, I never needed to progressively decode the DAG as this problem should be solved in the API layer, where you can request exactly what you need (or close to it) when you need it.
Re: Progressive JSON
#25I'll try to explain why this is a solution looking for a problem. Yes, breadth-first is always an option, but JSON is a heterogenous structured data source, so assuming that breadth-first will help the app start rendering faster is often a poor assumption. The app will need a subset of the JSON, but it's not simply the depth-first or breadth-first first chunk of the data set. So for this reason what we do is include…
>The app will need a subset of the JSON, but it's not simply the depth-first or breadth-first first chunk of the data set.
Right. Closer to the end of the article I slightly pivot to talk about RSC. In RSC, the data is the UI, so the outermost data literally corresponds to the outermost UI. That's what makes it work.
It's encoded like progressive JSON but conceptually it's more like HTML. Except you can also have your own "tags" on the client that can receive object attributes.
Re: Progressive JSON
#26why send the footer above the comments? Maybe its not a footer then but a sidebar? Should be treated as a sidebar then? Besides this could all kinda be solved by still using plain streaming json and sending .comments last?
Re: Progressive JSON
#27Very cool point, and it applies to any tree data in general. I like to represent tree data with parent, type, and data vectors along with a string table, so everything else is just small integers. Sending the string table and type info as upfront headers, we can follow with a stream of parent and data vector chunks, batched N nodes at a time. Tye depth- or breadth-first streaming becomes a choice of ordering on the v…
If you send the tree in preorder traversal order with known depth, you can send the tree without node ids or parent ids! You can just send the level for each node and recover the tree structure with a stack.
Re: Progressive JSON
#28why send the footer above the comments? Maybe its not a footer then but a sidebar? Should be treated as a sidebar then? Besides this could all kinda be solved by still using plain streaming json and sending .comments last?
Part of the point I'm making is that an out-of-order format is more efficient because we can send stuff as it's ready (so footer can go as soon as it's ready). It'll still "slot in" the right place in the UI. What this lets us do, compared to traditional top-down streaming, is to progressively reveal inner parts of the UI as more stuff loads.
Re: Progressive JSON
#29Seems like some people here are taking this post literally, as in the author (Dan Abramov) is proposing a format called Progressive JSON — it is not.
This is more of a post on explaining the idea of React Server Components where they represent component trees as javascript objects, and then stream them on the wire with a format similar to the blog post (with similar features, though AFAIK it’s bundler/framework specific).
This allows React to have holes (that represent loading states) on the tree to display fallback states on first load, and then only display the loaded component tree afterwards when the server actually can provide the data (which means you can display the fallback spinner and the skeleton much faster, with more fine grained loading).
(This comment is probably wrong in various ways if you get pedantic, but I think I got the main idea right.)