Live data from Hacker News

Data fetching on the web still sucks

performancejs.com

31–40 of 98 posts

Re: Data fetching on the web still sucks

#31
post #5

I'm currently experimenting with React and WebSockets and they seem to be a perfect fit. No need to write wrappers for Fetch, network errors and reconnects can be handled on high-level, handlers for each message type can be mounted and unmounted on useEffect hooks, all back-end jobs can notify the user in realtime, all session-based client-side data can be updated in realtime (in single or multiple open tabs). I'm al…

We are using Phoenix Channels[0] for a websocket-based app and it's very easy to setup/work with. It should also scale easily.

[0] https://hexdocs.pm/phoenix/channels.html

Re: Data fetching on the web still sucks

#32
post #24

This post feels like a uninformed and undifferentiated rant against "things are too complex". Let's start with the first paragraph: What does the JavaScript fetch API have to do with data management? How can you compare the fetch() API with Swagger (an API documentation format) with Protobuf (a serialisation format)? That doesn't even make sense. Second paragraph: "The UI should automatically update everywhere the da…

Half of the stuff he mentioned are covered by apollo-graphql and libs in that eco-system: query batching, notifications (via subscriptions), automatic state updates via cache, typesafety with typescript and grapqhl-codegen. It doesn‘t seem like he even looked at the libraries he listed. :(

Re: Data fetching on the web still sucks

#34
post #24

This post feels like a uninformed and undifferentiated rant against "things are too complex". Let's start with the first paragraph: What does the JavaScript fetch API have to do with data management? How can you compare the fetch() API with Swagger (an API documentation format) with Protobuf (a serialisation format)? That doesn't even make sense. Second paragraph: "The UI should automatically update everywhere the da…

Author here.

Data fetching, caching, consistency, and UX are all closely related. If you treat them as separate problems, you're punting the problem onto product engineers, who won't solve it well. (See the last paragraph of the post, which suggests this same idea.)

> That is state management, yeah, and you can build proper state management with any HTTP library and any message serialisation format.

You're right that to do this manually it's just state management. But to automatically update the UI, it means your client data layer (eg. Apollo) needs to know & track the identity of fields; the data layer also needs to be able to subscribe to fetches anywhere in your app, not local fetches; it also means your protocol needs to support this identity (via agreed-upon ID fields); etc. These problems are all closely related.

> Request batching: How would that happen "automatically"? By waiting to fire requests and then batching them?

eg. Relay does this by statically combining requests throughout a React tree into a single request at the top of the tree. You could also do it dynamically, as you suggest. The tradeoff is often performance.

> UX when fetching data

Having engineers manually define loading states doesn't scale. React is approaching this problem with Suspense, and you could imagine standard loading states when fetches are in flight.

Re: Data fetching on the web still sucks

#35
post #24

This post feels like a uninformed and undifferentiated rant against "things are too complex". Let's start with the first paragraph: What does the JavaScript fetch API have to do with data management? How can you compare the fetch() API with Swagger (an API documentation format) with Protobuf (a serialisation format)? That doesn't even make sense. Second paragraph: "The UI should automatically update everywhere the da…

Half of the stuff he mentioned are covered by apollo-graphql and libs in that eco-system: query batching, notifications (via subscriptions), automatic state updates via cache, typesafety with typescript and grapqhl-codegen. It doesn‘t seem like he even looked at the libraries he listed. :(

Sorry if it was unclear, but my argument is that while many libraries solve pieces of the problem, no one library (or standard) solves all, or even most, of these.

Re: Data fetching on the web still sucks

#36
post #10

> Realtime updates. Sometimes, you want data to be pushed, not pulled (eg. for things like notifications). But the APIs for pushing and pulling data are often different, both on the client and on the server. Two and a half months since Google announced the removal of HTTP Push[1]. There'd been a number of threads open (some 5 years old now) asking the Fetch spec to please include some way to allow the page to be noti…

> this path was both realtime & still http/resource centric

There's always long-polling. :) Not quite as efficient as Server Push, but the logical behavior is similar in that the server pushes an HTTP resource at the client.

Re: Data fetching on the web still sucks

#37
post #24

This post feels like a uninformed and undifferentiated rant against "things are too complex". Let's start with the first paragraph: What does the JavaScript fetch API have to do with data management? How can you compare the fetch() API with Swagger (an API documentation format) with Protobuf (a serialisation format)? That doesn't even make sense. Second paragraph: "The UI should automatically update everywhere the da…

I think the post is a bit unfortunate in its wording and this seems to have sent you off on a wrong track.

From how I read it, this the post is not specifically about JS and a discussion of the specific technologies mentioned but rather concerned with the following very general situation:

1. There's a user sitting in front of a browser.

2. There's a backend server providing data and points of interaction with that data.

I think the central point of the post now is that we don't have a satisfying technical solution for this situation.

Let's take a look at one of the points you mentioned. Maybe you might find there's actually some valid points in the post and give it a more favourable reread.

> UX when fetching data: What does that have to do with any of the above?

Here, the author of the post writes: "It’s a big burden for engineers to have to manually add loading spinners and error states, and engineers often forget."

I think it's more or less clear how you could implement this using only plain vanilla js. Cumbersome but doable: A very manual, imperative process.

Now let's envision a technology from a possible future:

    const twitterFeed = createMagicDataSource("https://twitter.com/...")

    const feedComponent = magicRendererComponent(twitterFeed, state => /* HTML like declarative description of the visuals */)

Imagine this was everything you had write in your code to get the following:

* state gets automatically loaded when the first instance of the component is created

* updates are automatically visualized in the client based on the internals of the declaration in the renderer

* you don't have to specify if the updates are done by polling, websockets or whatever: the two magic methods figure this out by themselves.

* you don't have to specify how the data is fetched in the first place. giving the a URI to the `createMagicDataSource` function is enough.

* additional instances of the component don't fetch the data again

* updates are efficient: the sync method only exchanges exactly the data required, only the minimal visual updates are performed

* marking feeds as "seen" by the user is also done by magic and syncs across devices (same for other non-ephemeral ui state).

Now: And I'm sure you'd agree that we are not there yet technologically. But I hope you agree that this would be really nice.

Re: Data fetching on the web still sucks

#38
post #5

I'm currently experimenting with React and WebSockets and they seem to be a perfect fit. No need to write wrappers for Fetch, network errors and reconnects can be handled on high-level, handlers for each message type can be mounted and unmounted on useEffect hooks, all back-end jobs can notify the user in realtime, all session-based client-side data can be updated in realtime (in single or multiple open tabs). I'm al…

We are using Phoenix Channels[0] for a websocket-based app and it's very easy to setup/work with. It should also scale easily. [0] https://hexdocs.pm/phoenix/channels.html

When I was researching stuff I've read a lot of good things about Phoenix, I honestly think socket.io and ws NodeJS libraries are inferior compared to that.

It's just almost pure luck that the guy who wrote uWebSockets (written in C & C++) also wrote NodeJS bindings, otherwise we'll be stuck with the ones with sub-par api, docs, and performance.

Re: Data fetching on the web still sucks

#39
post #28
post #24

This post feels like a uninformed and undifferentiated rant against "things are too complex". Let's start with the first paragraph: What does the JavaScript fetch API have to do with data management? How can you compare the fetch() API with Swagger (an API documentation format) with Protobuf (a serialisation format)? That doesn't even make sense. Second paragraph: "The UI should automatically update everywhere the da…

I actually think the problems outlined are pretty valid. Yeah, there are solutions to them, but I think what the author is saying is that you have to subscribe to something batteries-included, like Meteor, or otherwise implement the solutions yourself from bits and pieces.

"You have to subscribe to something batteries-included, like Meteor, or otherwise implement the solutions yourself from bits and pieces."

...yes; that is, in fact, the conceptual dichotomy in solutions to essentially-complex problems. Either someone else solves them for you, or you have to solve them yourself.

I mean, yes, there is a part of the solution-space in-between these two extremes — a point where there's a batteries-included thing that someone else built but then gives away for free, perhaps as an open-source project you just have to run on your own infra. So it's mostly solved for you, and then you just do a little bit to "get the solution running" for your use-case. That point in solution-space is conspicuously absent for web data sync.

But, in domains with essential complexity, that middle-ground part of the solution-space is usually conspicuously absent. Because it takes continuous dedicated effort (i.e. labor; capital expenditure) to solve the complex problem in a cleanly-abstracted way. And it's very rare that anyone's going to go to the effort, unless they expect a return on their labor investment (by e.g. keeping the solution proprietary, and building a SaaS business around it.)

Re: Data fetching on the web still sucks

#40

Ember.js seems to check off a lot of those boxes, if you are willing to use a complete solution and not do the 'pick and choose' method of react and friends. While it has lost popularity in the last few years, it has been moving forward technically. It has become leaner and meaner. Really learning the "Ember Way" to do things can reduce some of the friction the author mentions. Ember Data can allow you to talk to mul…

I'm saddened to see that almost a decade after Ember's release, the front-end world still insists rolling their own (terrible) reimplementation of it. I would've expected that a decade later we would've settled on an Ember-like framework for the front-end and moved beyond constructing URLs and parsing JSON responses manually but apparently not.

The rest of the world moved on from Ember because Ember is a pain in the ass to use.
Post reply on HN