Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Progressive JSON

overreacted.io

Post reply on HN
Print view View original

61–70 of 244 posts

Re: Progressive JSON

#61

Post by tough »

danabramov wrote: post #36
goranmoomin wrote: post #29

Seems like some people here are taking this post literally, as in the author (Dan Abramov) is proposing a format called Progressive JSON — it is not. This is more of a post on explaining the idea of React Server Components where they represent component trees as javascript objects, and then stream them on the wire with a format similar to the blog post (with similar features, though AFAIK it’s bundler/framework speci…

Yup! To be fair, I also don't mind if people take the described ideas and do something else with them. I wanted to describe RSC's take on data serialization without it seeming too React-specific because the ideas are actually more general. I'd love if more ideas I saw in RSC made it to other technologies.

hi dan! really interesting post.

do you think a new data serialization format built around easier generation/parseability and that also happened to be streamable because its line based like jsonld could be useful for some?

Re: Progressive JSON

#62

Post by tough »

vinnymac wrote: post #52
goranmoomin wrote: post #29

Seems like some people here are taking this post literally, as in the author (Dan Abramov) is proposing a format called Progressive JSON — it is not. This is more of a post on explaining the idea of React Server Components where they represent component trees as javascript objects, and then stream them on the wire with a format similar to the blog post (with similar features, though AFAIK it’s bundler/framework speci…

I already use streaming partial json responses (progressive json) with AI tool calls in production. It’s become a thing, even beyond RSCs, and has many practical uses if you stare at the client and server long enough.

how do you do that exactly?

Re: Progressive JSON

#63

Post by hombre_fatal »

hsbauauvhabzb wrote: post #22
ChrisMarshallNY wrote: post #19

This would be good. I got really, really sick of XML, but one thing that XML parsers have always been good at, is realtime decoding of XML streams. It is infuriating, waiting for a big-ass JSON file to completely download, before proceeding. Also JSON parsers can be memory hogs (but not all of them).

Json is just a packing format that does have that limitation. If you control the source and the destination, could you possibly use a format that supports streaming better like Protobuf?

What stops you from parsing tokens from a stream like a SAX parser for JSON?

[ ["aaa", "bbb"], { "name", "foo" } ]

    Start array
    Start array
    String aaa
    String bbb
    End array
    Start object
    Key name
    String foo
    End object 
    End array

Re: Progressive JSON

#64

Post by harrall »

I don’t think progressive loading is innovative.

What is innovative trying to build a framework that does it for you.

Progressive loading is easy, but figuring out which items to progressively load and in which order without asking the developer/user to do much extra config is hard.

Re: Progressive JSON

#65

Post by aloha2436 »

slt2021 wrote: post #58

if you ever feel the need to send progressive JSON - just zip it and don't bother solving fake problem at the wrong abstraction layer

The article doesn't advocate sending it progressively to make it smaller on the wire. The motivating example is one where some of the data (e.g. posts) is available before the rest of the data in the response (e.g. comments). Rather than:

- Sending a request for posts, then a request for comments, resulting in multiple round trips (a.k.a. a "waterfall"), or,

- Sending a request for posts and comments, but having to wait until the commends have loaded to get the posts,

...you can instead get posts and comments available as soon as they're ready, by progressively loading information. The message, though, is that this is something a full-stack web framework should handle for you, hence the revelation at the end of the article about it being a lesson in the motivation behind React's Server Components.

Re: Progressive JSON

#66

Post by hobs »

harrall wrote: post #64

I don’t think progressive loading is innovative. What is innovative trying to build a framework that does it for you. Progressive loading is easy, but figuring out which items to progressively load and in which order without asking the developer/user to do much extra config is hard.

That's because its basically cache invalidation.

Re: Progressive JSON

#67

Post by yawaramin »

> I’d like to challenge more tools to adopt progressive streaming of data.

It's a solved problem. Use HTTP/2 and keep the connection open. You now have effectively a stream. Get the top-level response:

    {
      header: "/posts/1/header",
      post: "/posts/1/body",
      footer: "/posts/1/footer"
    }
Now reuse the same connection to request the nested data, which can all have more nested links in them, and so on.

Re: Progressive JSON

#68

Post by aperturecjs »

I previously wrote a prototype of streaming a JSON tree this way:

https://github.com/rgraphql/rgraphql

But it was too graphql-coupled and didn't really take off, even for my own projects.

But it might be worth revisiting this kind of protocol again someday, it can tag locations within a JSON response and send updates to specific fields (streaming changes).

Re: Progressive JSON

#69

Post by aloha2436 »

yawaramin wrote: post #67

> I’d like to challenge more tools to adopt progressive streaming of data. It's a solved problem. Use HTTP/2 and keep the connection open. You now have effectively a stream. Get the top-level response: { header: "/posts/1/header", post: "/posts/1/body", footer: "/posts/1/footer" } Now reuse the same connection to request the nested data, which can all have more nested links in them, and so on.

> Now reuse the same connection to request the nested data, which can all have more nested links in them, and so on.

This still involves multiple round-trips though. The approach laid out in the article lets you request exactly the data you need up-front and the server streams it in as it becomes available, e.g. cached data first, then data from the DB, then data from other services, etc.

Re: Progressive JSON

#70

Post by Aeolun »

I think the problem with this is that it makes a very simple thing a lot harder. I don’t want to try and debug a JSON stream that can fail at any point. I just want to send a block of text (which I generate in 2ms anyway) and call it a day.
Post reply on HN