Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

421–430 of 615 posts

Re: Things we learned about LLMs in 2024

#421

Post by j_bum »

layer8 wrote: post #312
lumost wrote: post #214

Earlier quoted context omitted.

When I started my career in 2010, google was a semi-serious skill. All of the little things that we know how to do now such as ignoring certain sites, lingering on others, and iteratively refining our search queries were not universally known at the time. Experienced engineers often relied on encyclopedic knowledge of their environment or by "reading the manual". In my experience, LLM tools are the same, you ask for…

One difference is that skillful googling still only involved typing a few keywords or a short phrase and some syntax, and then knowing how to skim the results and iterate, and how to operate your browser efficiently. With LLMs, you have to type a lot more (and/or use voice input), and often also read more, it’s also not stateless/repeatable like following a web link, and most output looks the same (as opposed to the…

I also find LLMs to be more exhausting than Googling, but for me they’ve been ultimately more enriching and efficient.

Specifically, I’ve been using Kagi Assistant over the past 1.5 months for serious and lengthy searches, and I can’t imagine going back to traditional search.

I’m currently sold on this model of LLM assisted search (where explicit links are provided) over the old Google foo skills I developed during grad school.

Example search topics include deep dives and guidance for my first NAS build, finding new bioinformatics methods, and other random biomedical info.

Re: Things we learned about LLMs in 2024

#422

Post by jiggawatts »

fragmede wrote: post #413
jiggawatts wrote: post #270

Earlier quoted context omitted.

At the risk of sounding impolite or critical of your personal choices: this, right here, is the problem! You don’t understand how medicine works, at any level. Yet you turn to a machine for advice, and take it at face value . I say these things confidently, because I do understand medicine well enough to not to seek my own answers. Recently I went to a doctor for a serious condition and every notion I had was wrong.…

To be fair though, humanity doesn't know how some medicines work at a fundamental level either. The method of action for Tylenol, lithium, and metformin, among others isn't fully understood.

True, but modern "western"[1] medicine is not about the specific chemicals used, or even knowing how they exactly work at a chemical level, but the process for identifying what does and what does not work. It's an "evidence based" science with with experiments designed to counter known biases such as the placebo effect. Much of what we consider modern medicine was developed before we were entirely sure that atoms actually existed!

[1] It isn't actually western, because it's also used in the east, middle-east, south, both sides of every divide, etc... In the same sense, there is no "western chemistry" as an alternative to "eastern alchemy". There's "things that work" versus "things that make you feel slightly better because they're mild narcotics or stimulants... at best."

(I don't want to focus too much on Chinese herbal medicine, because I see the same cargo-culting non-scientific thinking in code development too. I've lost count of the number of times I've seen an n-tier SPA monstrosity developed for something that needed a tiny monolithic web app, but mumble-mumble-best-mumble-practices.)

Re: Things we learned about LLMs in 2024

#424

Post by worldsayshi »

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

I ponder if LLM:s are very useful but at a quite narrower set of tasks than we expect. Like fuzzy manipulation of logical specifications.

I.e. over time it constitute a fundamental shift in how we interact with abstractions in computers. The current fundamentals will still remain but they will become increasingly malleable. Details in code will become less important. Architecture will become increasingly important. But at the same time the cost of refactoring or changing architecture will quickly drop.

Any details that are easily lost when passing through an LLM will be details that have the highest maintenance cost. Any important details that can be retained by an LLM can move up and down the ladder of abstraction at will.

Can an LLM based solution maintain software architectures without introducing noise? The answer to that is the difference between somewhat useful and game changing.

Re: Things we learned about LLMs in 2024

#425

Post by zahlman »

BeetleB wrote: post #242
dartos wrote: post #198

> There’s a flipside to this too: a lot of better informed people have sworn off LLMs entirely because they can’t see how anyone could benefit from a tool with so many flaws. The key skill in getting the most out of LLMs is learning to work with tech that is both inherently unreliable and incredibly powerful at the same time. This is a decidedly non-obvious skill to acquire! I wish the author qualified this more. How…

I think most tech folks struggle with it because they treat LLMs as computer programs, and their experience is that SW should be extremely reliable - imagine using a calculator that was wrong 5% of the time - no one would accept that! Instead, think of an LLM as the equivalent of giving a human a menial task. You know that they're not 100% reliable, and so you give them only tasks that you can quickly verify and corr…

> Don't use LLMs where accuracy is paramount. Use it to automate away tedious stuff.

My programmer mind tells me that "tedious stuff" is where accuracy is the most important.

Re: Things we learned about LLMs in 2024

#427

Post by wrs »

dxbydt wrote: post #381
wrs wrote: post #293

Earlier quoted context omitted.

It's rather purple prose, but it's entirely meaningful. Maybe it doesn't seem to mean anything until after you know some linear algebra, though...

its been a long time, but when i was taught this material, i was told there are only 3 cases - x+y=1, x+y=2 clearly has no solution since two numbers can’t simultaneously add to both one and two. x+y=1,2x+2y=2 clearly has infinitely many solutions. There’s only one equation here after canceling the 2, so you can plug in x’s and y’s all day long, no end to it. x+y=1, 2x+y=1 clearly has exactly one solution (0,1) after…

Sure, but saying something in an ornate way is not the same as “saying nothing”.

Re: Things we learned about LLMs in 2024

#428

Post by mhh__ »

post-it wrote: post #90
sowbug wrote: post #84

Simon has mentioned in multiple articles how cool it is to use 64GB DRAM for GPU tasks on his MacBook. I agree it's cool, but I don't understand why it is remarkable. Is Apple doing something special with DRAM that other hardware manufacturers haven't figured out? Assuming data centers are hoovering up nearly all the world's RAM manufacturing capacity, how is Apple still managing to ship machines with DRAM that perfo…

Apple designs its own chips, so the RAM and CPU are on the same die and can talk at very high speeds. This is not the case for PCs, where RAM is connected externally.

It's on the same package but the same die?

Re: Things we learned about LLMs in 2024

#429

Post by zahlman »

torginus wrote: post #177
datadrivenangel wrote: post #33

Earlier quoted context omitted.

It's justified if AGI is possible. If AGI is possible, then the entire human economy stops making sense as far as money goes, and 'owning' part of OpenAI gives you power. That is of course, assuming AGI is possible and exponential, and that marketshare goes to a single entity instead of a set of entities. Lots of big assumptions. Seems like we're heading towards a slow-lackluster singularity though.

I was thinking about how the economy has been actively makes less sense and gets divorced more and more from reality year after year, AI or not. It's the simple fact that the ability of assets to generate wealth has far outstripped the abiliy of individuals to earn money by working. Somehow real estate has become so expensive everywhere that owning a shitty apartment is impossible for the vast majority. When the worl…

> When the world's population was exploding during the 20th century, housing prices were not a problem, yet somehow nowadays, it's impossible to build affordable housing to bring the prices down, though the population is stagnant or growing slowly.

In Canada, the population is still growing at a fairly impressive rate (https://www.macrotrends.net/global-metrics/countries/CAN/can...), and that growth tends to concentrate in major population centres. There are advocacy groups that seek to push Canadian population growth well above UN projections (e.g. the https://en.wikipedia.org/wiki/Century_Initiative "aims to increase Canada's population to 100 million by 2100") through immigration. In Japan, where the population is declining, housing prices are not anything like the problem we observe in North America.

There's also the supply side. "Impossible to build affordable housing" is in many cases a consequence of zoning restrictions. (Economists also hold very strongly that rent control doesn't work - see e.g. https://www.brookings.edu/articles/what-does-economic-eviden... and https://www.nmhc.org/research-insight/research-notes/2023/re... ; real "affordable housing" is just the effect of more housing.)

Post reply on HN