Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

171–180 of 615 posts

Re: Things we learned about LLMs in 2024

#171

Post by fullstackchris »

datadrivenangel wrote: post #137
philipkglass wrote: post #108

Earlier quoted context omitted.

I believe that AGI cannot be exponential for long because any intelligent agent can only approach nature's limits asymptotically. The first company with AGI will be about as much ahead as, say, the first company with electrical generators [1]. A lot of science fiction about a technological singularity assumes that AGI will discover and apply new physics to develop currently-believed-impossible inventions, but I don't…

I don't recall editing my message, but HN can be wonky sometimes. :) Nothing is truly exponential for long, but the logistic curve could be big enough to do almost anything if you get imaginative. Without new physics, there are still some places where we can do some amazing things with the equivalent of several trillion dollars of applied R&D, which AGI gets you.

Sure but think of the Higgs... how long that took for just _one_ particle. You think an AGI, or even an ASI is going to make an experimental effort like that go any bit faster? Dream on!

It astounds me that people dont realize how much of this cutting edge science stuff literally does NOT happen overnight, and not even close to that; typically it takes on the order of decades!

Re: Things we learned about LLMs in 2024

#172

Post by gre »

lolinder wrote: post #141
simonw wrote: post #112

Earlier quoted context omitted.

Yeah, the other skill you need to develop to make the most of AI-assisted programming is really good manual QA.

Have you found that to be a good trade-off for large-scale projects? Where I'm at right now with LLMs is that I find them to be very helpful for greenfield personal projects. Eliminating the blank canvas problem is huge for my productivity on side projects, and they excel at getting projects scaffolded and off the ground. But as one of the lead engineers working on a million+ line, 10+ year-old codebase, I've yet to…

I had some success converting a react app with classes to use hooks instead. Also asking it to handle edge cases, like spaces in a filename in a bash script--this fixes some easy problems that might have come up. The corollary here is that pointing out specific problems or mentioning the right jargon will produce better code than just asking for the basic task.

It's very bad at Factor but pretty good at naming things, sometimes requiring some extra prompting. [generate 25 possible names for this variable...]

Re: Things we learned about LLMs in 2024

#173

Post by jacobgkau »

ffsm8 wrote: post #158
cactusfrog wrote: post #145

Earlier quoted context omitted.

Every waste of money is not a Ponzi scheme.

I agree, the core aspect of a ponzi scheme is that it redistributes the newly invested funds to previous investors, making it highly profitable to anyone joining early and incentivising early joiners to get new investors. This just doesn't hold true for open ai

Doesn't it hold true for investment in AI (or potentially any other industry that experiences a boom) in general?

Anyone who bought in at the ground floor is now rich. Anyone who buys in now is incentivized to try and keep getting more people to buy in so their investment will give a return regardless of if actual value is being created.

Re: Things we learned about LLMs in 2024

#174

Post by fullstackchris »

EMIRELADERO wrote: post #149
GOD_Over_Djinn wrote: post #140

Earlier quoted context omitted.

This sounds like magic, not science.

What do you mean by this? Is there any fundamental property of intelligence, physicality, or the universe, that you think wouldn't let this work?

Not OP but yes. Electron size vs band gap, computing costs (in terms of electricity) any other raw materials needed for that energy, etc... sigh... its physics, always physics... what fundamental property of physics do you think would let a vertical take off in intelligence occur?

Re: Things we learned about LLMs in 2024

#177

Post by torginus »

datadrivenangel wrote: post #33
shihab wrote: post #20

Earlier quoted context omitted.

The last OpenAI valuation I read about was 157 billion. I am struggling to understand what justifies this. To me, it feels like OpenAI is at best few months ahead of competitors in some areas. But even if I am underestimating the advantage, it's few years instead of few months, why does it matter? It's not like AI companies are going to enjoy the first-mover advantage internet giants had over the competition.

It's justified if AGI is possible. If AGI is possible, then the entire human economy stops making sense as far as money goes, and 'owning' part of OpenAI gives you power. That is of course, assuming AGI is possible and exponential, and that marketshare goes to a single entity instead of a set of entities. Lots of big assumptions. Seems like we're heading towards a slow-lackluster singularity though.

I was thinking about how the economy has been actively makes less sense and gets divorced more and more from reality year after year, AI or not.

It's the simple fact that the ability of assets to generate wealth has far outstripped the abiliy of individuals to earn money by working.

Somehow real estate has become so expensive everywhere that owning a shitty apartment is impossible for the vast majority.

When the world's population was exploding during the 20th century, housing prices were not a problem, yet somehow nowadays, it's impossible to build affordable housing to bring the prices down, though the population is stagnant or growing slowly.

A company can be worth $1B if someone invests $10m in it for 1% stake - where did the remaining $990m come from? Likewise, the stock market is full of trillion-dollar companies whose valuations beggar all explanation, considering the sizes of the markets they are serving.

The rich elites are using the wealth to control access to basic human needs (namely housing and healthcare) to squeeze the working population for every drop of money. Every wealth metric shows the 1% and the 1% of the 1% control successively larger portions of the economic pie. At this point money is ceasing to be a proxy for value and is becoming a tool for population control.

And the weird thing is it didn't use to be nearly this bad even a decade ago, and we can only guess how bad it will get in a decade, AGI or not.

Anyway, I don't want to turn this into a fully-written manifesto, but I have trouble expressing these ideas in a concise manner.

Re: Things we learned about LLMs in 2024

#178

Post by cruffle_duffle »

datadrivenangel wrote: post #103
dxbydt wrote: post #92

Earlier quoted context omitted.

> best LLMs are able to accelerate you https://www2.math.upenn.edu/~ghrist/preprints/LAEF.pdf - this math textbook was written in just 55 days! Paraphrasing the acknowledgements - ...Begun November 4, 2024, published December 28, 2024. ...assisted by Claude 3.5 sonnet, trained on my previous books... ...puzzles co-created by the author and Claude ...GPT-4o and -o1 were useful in latex configurations...doing proof-rea…

Going faster isn't good if the quality drops enough that overall productivity decreases... Infinite slop is only a good thing for pigs.

Just use ChatGPT to summarize its own output. It’s like running your JPEG back through the JPEG compressor again!

Re: Things we learned about LLMs in 2024

#179

Post by hyperpape »

Animats wrote: post #10

> Some of those GPT-4 models run on my laptop That's an indication that most business-sized models won't need some giant data center. This is going to be a cheap technology most of the time. OpenAI is thus way overvalued.

This seems like a non-sequitur unless you’re assuming something about the amount that people use models.

Most web servers can run some number of QPS on a developer laptop, but AWS is a big business, because there are a heck of a lot of QPS across all the servers.

Re: Things we learned about LLMs in 2024

#180

Post by trenchgun »

nox101 wrote: post #133
logicchains wrote: post #68

Earlier quoted context omitted.

They can spit out LaTeX, and a PDF from that is going to look much nicer than a Google doc (and display the same everywhere). As an added bonus, the recruiter can't randomly rewrite parts of it (at least not so easily).

The recuiter isn't going to print out your resume. They're going to read in their computer or iPad or phone.

For sure they will read a pdf and not a google doc.
Post reply on HN