Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Workers API

developer.mozilla.org

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Web Workers API

#31

Post by MuffinFlavored »

reginaldo wrote: post #27
richardanaya wrote: post #26

A little known annoying detail of web workers is that passing large data to them via postMessage is incredibly slow. The browser has to convert your javascript object into some sort of internal binary format and it slows down the thread that's doing the sending.

Depends on the type of large data, but since 2011 there's "Transferable" [1], where objects like ArrayBuffer, MessagePort, and ImageBitmap can be transferred with a low overhead [2]. Now, if you're passing large arbitrary object graphs, you're out of luck. [1] https://developers.google.com/web/updates/2011/12/Transferab... [2] https://developer.mozilla.org/en-US/docs/Web/API/Worker/post...

Any libraries that make this nicer/easier to do instead of rewriting the same thing over and over?

Re: Web Workers API

#32

Post by The_rationalist »

yutijke wrote: post #25

The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use. This is clearly visible in the lack of any library ecosystem around them. We don't have any highly used threadpool or executor libraries using workers. Everyone seems to be manually setting up a worker and setting up the job scheduling logic from scratch. All the boilerplate and restrictions really restr…

There are solutions to those problems: https://github.com/GoogleChromeLabs/comlink https://github.com/Bnaya/objectbuffer

Re: Web Workers API

#33

Post by The_rationalist »

MuffinFlavored wrote: post #31
reginaldo wrote: post #27

Earlier quoted context omitted.

Depends on the type of large data, but since 2011 there's "Transferable" [1], where objects like ArrayBuffer, MessagePort, and ImageBitmap can be transferred with a low overhead [2]. Now, if you're passing large arbitrary object graphs, you're out of luck. [1] https://developers.google.com/web/updates/2011/12/Transferab... [2] https://developer.mozilla.org/en-US/docs/Web/API/Worker/post...

Any libraries that make this nicer/easier to do instead of rewriting the same thing over and over?

https://github.com/Bnaya/objectbuffer and comlink

Re: Web Workers API

#34

Post by csmpltn »

yutijke wrote: post #25

The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use. This is clearly visible in the lack of any library ecosystem around them. We don't have any highly used threadpool or executor libraries using workers. Everyone seems to be manually setting up a worker and setting up the job scheduling logic from scratch. All the boilerplate and restrictions really restr…

> "The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use."

Can anybody explain how did this ever make it into the official spec and the default implementation?

The browser will make a network call to fetch your Web Worker .js file on every instantiation of your Web Worker. Instantiations of the same Web Worker "module" aren't cached, so in a thread-pool scenario your browser would be fetching the same file over and over again.

What were the people designing this even thinking? This is so wasteful and frustrating. The API simply sucks.

Re: Web Workers API

#35

Post by dmitriid »

csmpltn wrote: post #34
yutijke wrote: post #25

The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use. This is clearly visible in the lack of any library ecosystem around them. We don't have any highly used threadpool or executor libraries using workers. Everyone seems to be manually setting up a worker and setting up the job scheduling logic from scratch. All the boilerplate and restrictions really restr…

> "The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use." Can anybody explain how did this ever make it into the official spec and the default implementation? The browser will make a network call to fetch your Web Worker .js file on every instantiation of your Web Worker. Instantiations of the same Web Worker "module" aren't cached, so in a thread-pool scen…

> Can anybody explain how did this ever make it into the official spec and the default implementation?

This and service workers. Both are... weird, to say the least

Re: Web Workers API

#36

Post by TAForObvReasons »

csmpltn wrote: post #34
yutijke wrote: post #25

The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use. This is clearly visible in the lack of any library ecosystem around them. We don't have any highly used threadpool or executor libraries using workers. Everyone seems to be manually setting up a worker and setting up the job scheduling logic from scratch. All the boilerplate and restrictions really restr…

> "The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use." Can anybody explain how did this ever make it into the official spec and the default implementation? The browser will make a network call to fetch your Web Worker .js file on every instantiation of your Web Worker. Instantiations of the same Web Worker "module" aren't cached, so in a thread-pool scen…

"best guess": API decisions are opinionated, reflecting the designers' views of how users should write code.

For example, FileReader API is async in the main thread. There's an equivalent FileReaderSync for sync operations but that is only available in Web Workers. Why isn't FileReaderSync available on the main thread? Because the designers didn't want people to do things that would block the main thread.

The Web Worker argument probably went along the lines of "If we allow users to pass arbitrary function objects, they may try to do something that accesses local variables from the site where the worker is created. That is obviously an error, so we should design the API so that it can't happen. Creating a separate script creates a clear mental separation and avoids that class of error"

Re: Web Workers API

#37

Post by fabiospampinato »

yutijke wrote: post #25

The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use. This is clearly visible in the lack of any library ecosystem around them. We don't have any highly used threadpool or executor libraries using workers. Everyone seems to be manually setting up a worker and setting up the job scheduling logic from scratch. All the boilerplate and restrictions really restr…

Workers are awesome but you are right, working with them can be painful without the right tooling.

Personally I've written my own libraries for abstracting all this away and I'm having a blast working with workers now, maybe check them out:

- WorkTank [1]: This abstracts away the difference between browser workers and Node worker threads, it makes it easy to make worker pools, and it can transfer simple functions to a worker at runtime too.

- WorkTank loader: This abstracts away loading asynchronous function from a worker basically, you just add ".worker" to your file name and that file and all its dependencies are transparently moved to a worker (or worker pool), all the rest of the app (TS types for example) won't even notice anything happened, it just works, transparently.

You might want to check out the more popular "comlink" library too, although it didn't work for me for whatever reason when I tried it, and it doesn't support worker pools I believe.

[1]: https://github.com/fabiospampinato/worktank

[2]: https://github.com/fabiospampinato/worktank-loader

Re: Web Workers API

#38

Post by csmpltn »

dmitriid wrote: post #35
csmpltn wrote: post #34

Earlier quoted context omitted.

> "The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use." Can anybody explain how did this ever make it into the official spec and the default implementation? The browser will make a network call to fetch your Web Worker .js file on every instantiation of your Web Worker. Instantiations of the same Web Worker "module" aren't cached, so in a thread-pool scen…

> Can anybody explain how did this ever make it into the official spec and the default implementation? This and service workers. Both are... weird, to say the least

When I'm seeing half-arsed APIs like this being introduced into so-called "modern" incarnations of the web, I start doubting the technical chops of the people involved and the process as a whole. Is it truly possible that there wasn't anyone with sufficient experience in building concurrency/multi-threading/parallelism APIs involved in building this?

Failing to account for the thread-pool scenario, as an example, is just mind boggling.

Re: Web Workers API

#40

Post by Jasper_ »

yutijke wrote: post #25

The fact that creating a worker requires you to pass a module (file) name makes them extremely unergonomic to use. This is clearly visible in the lack of any library ecosystem around them. We don't have any highly used threadpool or executor libraries using workers. Everyone seems to be manually setting up a worker and setting up the job scheduling logic from scratch. All the boilerplate and restrictions really restr…

A big complaint I have is that the built-in scheduling for workers assumes a very specific scenario. You can send messages to a single worker, but the message loop has to be operated by the browser. You also don't know the length of the message queue, even though the browser has it available, so you can't easily send a message to the worker with the least work queued. If a worker starts working on low-priority items, and you want to interject with a high-priority message, you also can't interrupt the worker, nor can the worker loop the messages on its own accord. You also can't re-sort the message queue, it's FIFO.

Basically, any sort of work scheduling that you would like to do to queue high-priority messages, or have workers share a pool of work, is impossible to build with the onmessage-style of WebWorkers. It feels like an API made by someone who had read about threads once, rather than someone who's built an actual many-workers processing system like this. The event loop being unpumpable from user code feels like a giant kick in the face.

My workaround for this was to kill workers working on low-priority items, relaunch them, and resort the queue, but that all feels like a mess. Also, at some point a Chrome update made this strategy crash. I ended up just removing the WebWorker code; more trouble than it was worth for marginal code improvements.

I know that SAB and atomics add new low-level primitives to support this, but SAB is still poorly supported on widely deployed platforms. And you still need to do the serialization yourself.

Post reply on HN