Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incremental – A library for incremental computations

github.com

Post reply on HN
Print view View original

21–30 of 83 posts

Re: Incremental – A library for incremental computations

#21

Post by zitterbewegung »

fadesibert wrote: post #6

Goldman took the same approach with instrument pricing ~30 years ago. I recall long discussions about "Node Purpling" in my ~13 year tenure there. Computer Science has evolved, and AFAICT this is not a graph approach, but things like differentiation are computationally expensive, and therefore you want to minimize the number of times you do it to as close to the theoretical minimum. Edit: Related HN discussion https:…

Yea and this created "bank python" informally a good article is here.

https://calpaterson.com/bank-python.html

The best description about how it became a problem is one of the paragraphs.

"New starters take an exceptionally long time to get up to speed - and that's if they don't resign in fit of pique as soon as they see the special, mandatory, in-house IDE (as I nearly did). Even months in, new starters are still learning quite fundamental new things: there is a lot that is different."

I think it took me til I was there around two and a half years to fully comprehend it when I was working on it. Not much modern training til they figured out they had to teach it again that was better. The worst part is to make an UI around it coding it and it wasn't approved for new projects.

Re: Incremental – A library for incremental computations

#22

Post by avallach »

jitl wrote: post #16

This style of reactive programming is quite popular in JavaScript UI frameworks these days under the moniker “signals”, with a proposal for standardization here: https://github.com/tc39/proposal-signals#-javascript-signals... It’s used by frameworks Vue, SolidJS, Svelte, Ember, Angular, and there’s a few different implementations for React like Mobx and Jotai. There’s a few different algorithms for how to propagate c…

Another notable example: JetBrains Noria ( https://blog.jetbrains.com/fleet/2023/02/fleet-below-deck-pa... ).

As the authors highlight: "Noria is not a UI framework at its core. Instead, it’s a platform for incremental computations.". But currently they happen to use it to optimize gui rendering in JetBrains Air IDE.

Re: Incremental – A library for incremental computations

#23

Post by geysersam »

Eliah_Lakhin wrote: post #12
RandomBK wrote: post #4

One thing I've never fully grokked is how this differs from an observable pattern where one can publish new values to inputs, propagate that through the computation, and push newly computed values to listeners. I guess there's probably optimizations around change detection and stopping the propagation if there's no change (though observables can do that as well). The stabilize command also makes things interesting as…

It depends on how you define the observable pattern. The fundamental components here are laziness and weak connections between graph nodes. Node values are getting materialized only when you observe them, and the system is flexible for live structural changes. Usually, you don't need to materialize the entire graph when you need to observe just some nodes. Additionally, you can halt computations at any point in time…

What does weak connections mean in this context?

Re: Incremental – A library for incremental computations

#24

Post by ronfriedhaber »

This is cool.

As far as I can tell, incremental the library aims to solve the problem of partially hydrating a computation graph when source data is altered. This approach is similar to the one pursued by (well designed) build systems and is common in the FP world. [2] This has many use cases and is very cool.

In addition, in the sphere of incremental computation, there exists Differential Dataflow, Timely Dataflow (adjacent), and DBSP. Systems like Feldera are built on DBSP. Materialize is lead by some DD guys.

Personally, I am pursuing an orthogonal approach specifically for the problem of financial data and financial workloads, There exists huge, very important problems to solve! [1]

[1] https://modolap.com

[2] Signals And Threads episode on the subject https://signalsandthreads.com/build-systems/

Re: Incremental – A library for incremental computations

#25

Post by edparcell »

jitl wrote: post #16

This style of reactive programming is quite popular in JavaScript UI frameworks these days under the moniker “signals”, with a proposal for standardization here: https://github.com/tc39/proposal-signals#-javascript-signals... It’s used by frameworks Vue, SolidJS, Svelte, Ember, Angular, and there’s a few different implementations for React like Mobx and Jotai. There’s a few different algorithms for how to propagate c…

[dead]

Re: Incremental – A library for incremental computations

#26

Post by arthurbrown »

Another related but semantically distinct package with great developer ergonomics is the FRP library React -- https://erratique.ch/software/react/doc/React/index.html

Very satisfying to use when you manage to find a problem that is suited to this type of approach.

Re: Incremental – A library for incremental computations

#27

Post by geokon »

Where is it actually explained how it works..?

Usually these kinds of systems either don't scale dynamically or have caching issues. The first example, a spreadsheet, is "easy" because there are a fixed amount of cells to track. A GUI can be a lot harder (imagine sub windows and sub-sub windows dynamically popping up and tracking some redundant and some unique "computations". Entities can appear and then be removed at random). Though the wording carefully says "constructing views" so maybe it doesn't handle dynamism

Re: Incremental – A library for incremental computations

#29

Post by vanderZwan »

RandomBK wrote: post #19
Eliah_Lakhin wrote: post #12

Earlier quoted context omitted.

It depends on how you define the observable pattern. The fundamental components here are laziness and weak connections between graph nodes. Node values are getting materialized only when you observe them, and the system is flexible for live structural changes. Usually, you don't need to materialize the entire graph when you need to observe just some nodes. Additionally, you can halt computations at any point in time…

Laziness and weak connections makes sense as differentiators. However I'm not sure Excel is such a great illustration in that case, as it's neither lazy nor weakly connected; at least at the surface.

So I remember having physics homework a quarter century ago at university where we had to use Excel to determine a static electric potential field, by defining all cells in the grid as "the average value of the four surrounding cells", except the edge cells and source/sink cells which would get actual concrete values.

I distinctly remember being amazed that it would just iterate until it reached equilibrium (maybe we had to change some setting somewhere first though, I never really used Excel before or after that). I think you could change a value mid-iteration, and the grid would just continue on with the previously calculated cell values, instead of start over from scratch. That's basically "weakly connected", isn't it?

(anyway, the real challenge is to find a practical use for Excel's hidden fractal powers https://www.youtube.com/watch?v=b-Fa6HtvGtQ)

Re: Incremental – A library for incremental computations

#30

Post by shikck200 »

Balinares wrote: post #28

Whenever I see something in OCaml I assume it's Jane Street and that ends up correct a surprising amount of the time.

Jane street was/is a top user of ocaml, they kind of flushed their reputation down the toilet with they massive fraud in the asian markets. IIRC the entire company got banned from trading inside India, hongkong etc.

After they got caught most of their online posts/videos either got deleted, or locked so you could not comment on them.

I had hoped their front guy for ocaml, Ron Minsky had atleast made some sort of statement. Currently no ones know how deep that fraud went, but what we know ocaml was used in the fraud, damagine ocaml ecosystem too.

Post reply on HN