Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Progressive JSON

overreacted.io

Post reply on HN
Print view View original

41–50 of 244 posts

Re: Progressive JSON

#41

Post by danabramov »

Velorivox wrote: post #12

99.9999%* of apps don't need anything nearly as 'fancy' as this, if resolving breadth-first is critical they can just make multiple calls (which can have very little overhead depending on how you do it). * I made it up - and by extension, the status quo is 'correct'.

To be clear, I wouldn't suggest someone to implement this manually in their app. I'm just describing at the high level how the RSC wire protocol works, but narratively I wrapped it in a "from the first principles" invention because it's more fun to read. I don't necessarily try to sell you on using RSC either but I think it's handy to understand how some tools are designed, and sometimes people take ideas from different tools and remix them.

Re: Progressive JSON

#42

Post by sopooneo »

hsbauauvhabzb wrote: post #22
ChrisMarshallNY wrote: post #19

This would be good. I got really, really sick of XML, but one thing that XML parsers have always been good at, is realtime decoding of XML streams. It is infuriating, waiting for a big-ass JSON file to completely download, before proceeding. Also JSON parsers can be memory hogs (but not all of them).

Json is just a packing format that does have that limitation. If you control the source and the destination, could you possibly use a format that supports streaming better like Protobuf?

I've heard two side the the Protobuf/streaming idea. On my first introduction, it seemed you could. But later reading leads me to believe it is only almost streamable: https://belkadan.com/blog/2023/12/Protobuf-Is-Almost-Streama....

* I do acknowledge you qualified the question with "better".

Re: Progressive JSON

#43

Post by danabramov »

owebmaster wrote: post #37
danabramov wrote: post #25

Earlier quoted context omitted.

>The app will need a subset of the JSON, but it's not simply the depth-first or breadth-first first chunk of the data set. Right. Closer to the end of the article I slightly pivot to talk about RSC. In RSC, the data is the UI, so the outermost data literally corresponds to the outermost UI. That's what makes it work. It's encoded like progressive JSON but conceptually it's more like HTML. Except you can also have you…

> Closer to the end of the article I slightly pivot to talk about RSC. Not again, please.

The best part about someone else's writing is you can just ignore it.

Re: Progressive JSON

#44

Post by danabramov »

turtlebits wrote: post #40

Progressive JPEG make sense, because it's a media file and by nature is large. Text/HTML on the other hand, not so much. Seems like a self-inflicted solution where JS bundles are giant and now we're creating more complexity by streaming it.

Things can be slow not because they're large but because they take latency to produce or to receive. The latency can be on the server side (some things genuinely take long to query, and might be not possible or easy to cache). Some latency may just be due to the user having poor network conditions. In both cases, there's benefits to progressively revealing content as it becomes available (with intentional loading stages) instead of always waiting for the entire thing.

Re: Progressive JSON

#45

Post by conartist6 »

danabramov wrote: post #41
Velorivox wrote: post #12

99.9999%* of apps don't need anything nearly as 'fancy' as this, if resolving breadth-first is critical they can just make multiple calls (which can have very little overhead depending on how you do it). * I made it up - and by extension, the status quo is 'correct'.

To be clear, I wouldn't suggest someone to implement this manually in their app. I'm just describing at the high level how the RSC wire protocol works, but narratively I wrapped it in a "from the first principles" invention because it's more fun to read. I don't necessarily try to sell you on using RSC either but I think it's handy to understand how some tools are designed, and sometimes people take ideas from differ…

I'm already thinking of whether there's any ideas here I might take for CSTML -- designed as a streaming format for arbitrary data but particularly for parse trees

Re: Progressive JSON

#46

Post by Velorivox »

danabramov wrote: post #41
Velorivox wrote: post #12

99.9999%* of apps don't need anything nearly as 'fancy' as this, if resolving breadth-first is critical they can just make multiple calls (which can have very little overhead depending on how you do it). * I made it up - and by extension, the status quo is 'correct'.

To be clear, I wouldn't suggest someone to implement this manually in their app. I'm just describing at the high level how the RSC wire protocol works, but narratively I wrapped it in a "from the first principles" invention because it's more fun to read. I don't necessarily try to sell you on using RSC either but I think it's handy to understand how some tools are designed, and sometimes people take ideas from differ…

I get that. Originally my comment was a response to another but I decided to delete and repost it at the top level — however I failed to realize that not having that context makes the tone rather snarky and/or dismissive of the article as a whole, which I didn't intend.

Re: Progressive JSON

#47

Post by xelxebar »

dmkolobov wrote: post #27
xelxebar wrote: post #8

Very cool point, and it applies to any tree data in general. I like to represent tree data with parent, type, and data vectors along with a string table, so everything else is just small integers. Sending the string table and type info as upfront headers, we can follow with a stream of parent and data vector chunks, batched N nodes at a time. Tye depth- or breadth-first streaming becomes a choice of ordering on the v…

If you send the tree in preorder traversal order with known depth, you can send the tree without node ids or parent ids! You can just send the level for each node and recover the tree structure with a stack.

Well the whole point is to use a breadth first order here. I don't think there's a depth vector analogue for breadth first traversals. Is there?

But, indeed, depth vectors are nice and compact. I find them harder to work with most of the time, though, especially since insertions and deletions become O(n), compared to parent vector O(1).

That said, I do often normalize my parent vectors into dfpo order at API boundaries, since a well-defined order makes certain operations, like finding leaf siblings, much nicer.

Re: Progressive JSON

#48

Post by xelxebar »

thethimble wrote: post #18
xelxebar wrote: post #8

Very cool point, and it applies to any tree data in general. I like to represent tree data with parent, type, and data vectors along with a string table, so everything else is just small integers. Sending the string table and type info as upfront headers, we can follow with a stream of parent and data vector chunks, batched N nodes at a time. Tye depth- or breadth-first streaming becomes a choice of ordering on the v…

You can even alternate between sending table and node chunks! This will effectively allow you to reveal the tree in any order including revealing children before parents as well as representing arbitrary graphs! Could lead to some interesting applications.

Good point! The parent vector rep is what allows arbitrary node order, but chunking the table data off chunks of node IDs is brilliant idea. Cheers!

Re: Progressive JSON

#49

Post by NoahZuniga »

yen223 wrote: post #35

I've never really thought about how all the common ways we serialise trees in text (JSON, s-expressions, even things like tables of content, etc), serialise them depth-first. I suppose it's because doing it breadth-first means you need to come up with a way to reference items that will arrive many lines later, whereas you don't have that need with depth-first serialisation.

Also it makes memory allocation easier
Post reply on HN