Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are breaking 20 year old system design

zknill.io

Post reply on HN
Print view View original

21–30 of 31 posts

Re: LLMs are breaking 20 year old system design

#21

Post by pizza234 »

manueltgomes wrote: post #9

> Long running work: an agent doing a 10 minute task isn’t a ‘request’, it’s a long-running async process. Correct, but we solved this a long time ago when we started sending files to servers to be converted, for example. We either got a 'job_id' or a call to a webhook when the job was finished."

The article argues that although this is an asynchronous process:

> Long running work: an agent doing a 10 minute task isn’t a ‘request’, it’s a long-running async process.

it should not be stateful at the database/storage level:

> Stateful compute: an agent might run multiple turns of a conversation, might process multiple tool calls, and relies on accumulated context. That state is not really ‘database state’, it’s the agents memory.

According to the author, the problem is already solved, but implemented with the wrong design assumptions.

(Uploading a file for conversion could be framed as a slightly different problem from the author's, though, due to size constraints)

Re: LLMs are breaking 20 year old system design

#22

Post by zknill »

grugdev42 wrote: post #19

Article doesn't make sense. Some of the "horizontally scaled" servers have their own state. A local cache, a temporary filesystem etc. Also, has teh author never heard of long running queued jobs? Or long running scheduled jobs? They ultimately report back into the DB (updating their status etc). This article reeks of someone using AI to make huge leaping jumps of logic. The "single source of truth" rule has survived…

And once those long running jobs have reported their status back to the database, how will the client find out about that status?

Please, please, please don't say "polling". Because you've clearly missed the entire argument of the article if you say polling.

Re: LLMs are breaking 20 year old system design

#23

Post by zknill »

ahofmann wrote: post #12

To me this makes no sense. Nothing in web development changes because of long running requests, there are plenty of solutions for this. The most easy one is to just listen long enough on a http request for the answer. The routing problem can be mitigated with session pinning. Http2 and 3 have solutions for streaming data, websockets can be used, and pub/sub also. Heck, we could push the LLM response in a k2v system/r…

The industry decided a long time ago that sticky sessions was a terrible idea. They only half-solve the problem, while suffering from session loss on server loss and imbalanced load over time.

Re: LLMs are breaking 20 year old system design

#25

Post by dial9-1 »

zknill wrote: post #22
grugdev42 wrote: post #19

Article doesn't make sense. Some of the "horizontally scaled" servers have their own state. A local cache, a temporary filesystem etc. Also, has teh author never heard of long running queued jobs? Or long running scheduled jobs? They ultimately report back into the DB (updating their status etc). This article reeks of someone using AI to make huge leaping jumps of logic. The "single source of truth" rule has survived…

And once those long running jobs have reported their status back to the database, how will the client find out about that status? Please, please, please don't say "polling". Because you've clearly missed the entire argument of the article if you say polling.

postgresql has LISTEN and NOTIFY. redis and kafka have pubsub. this is a solved problem

Re: LLMs are breaking 20 year old system design

#26

Post by OutOfHere »

bilbo-b-baggins wrote: post #16

Claude code runs as a nearly stateless server using session JSONL files as a conversation database, sending stateless API requests to Anthropic, etc. This post doesn’t seem to understand how these systems work at the core of agent harnesses.

That is a limited argument because sometimes the harness can run itself in the cloud, not on the client side. Something then has to maintain the communication between the harness and the client.

Re: LLMs are breaking 20 year old system design

#28

Post by ahofmann »

zknill wrote: post #23
ahofmann wrote: post #12

To me this makes no sense. Nothing in web development changes because of long running requests, there are plenty of solutions for this. The most easy one is to just listen long enough on a http request for the answer. The routing problem can be mitigated with session pinning. Http2 and 3 have solutions for streaming data, websockets can be used, and pub/sub also. Heck, we could push the LLM response in a k2v system/r…

The industry decided a long time ago that sticky sessions was a terrible idea. They only half-solve the problem, while suffering from session loss on server loss and imbalanced load over time.

The services, I ran didn't care about decisions "the industry" made. They worked just fine.

Re: LLMs are breaking 20 year old system design

#29

Post by zknill »

dial9-1 wrote: post #25
zknill wrote: post #22

Earlier quoted context omitted.

And once those long running jobs have reported their status back to the database, how will the client find out about that status? Please, please, please don't say "polling". Because you've clearly missed the entire argument of the article if you say polling.

postgresql has LISTEN and NOTIFY. redis and kafka have pubsub. this is a solved problem

Isn't the point that you no longer have a connection to the client?

So you can be notified by the database, but you can't (with the stateless HTTP + loadbalancer design explained in the article) get that notification back to the client. Because the client isn't connected anymore; so how does the client know that there's new information?

Re: LLMs are breaking 20 year old system design

#30

Post by gatlin »

zknill wrote: post #29
dial9-1 wrote: post #25

Earlier quoted context omitted.

postgresql has LISTEN and NOTIFY. redis and kafka have pubsub. this is a solved problem

Isn't the point that you no longer have a connection to the client? So you can be notified by the database, but you can't (with the stateless HTTP + loadbalancer design explained in the article) get that notification back to the client. Because the client isn't connected anymore; so how does the client know that there's new information?

*I guess there would have to be some mechanism for the database to push notifications to the client. This is not a fundamentally unsolvable or particularly interesting problem.*
Post reply on HN