GPT-4 level models have gotten 240x cheaper in 2 years:
https://x.com/_philschmid/status/1827955011071959118
Inference is free and instant
11–20 of 25 posts
Re: Inference is free and instant
#12> No matter how fast Searle is, he won't be able to come up with a beautiful and original Chinese poem that has the creative spark special to humans Why not? > Of course, at some level of complexity, it will be stuck in a local maximum of work quality simply because the book has no guide on how to solve the problem at hand. I find this a pretty un-optimistic view, especially from someone building a coding autopilot.…
> Why not?
Unsure if this is a useful answer. But Searle/LLM could make something that looks like it has a creative spark, and that's it.
Why I think that's different is in the case of a human artist, they create something because they have something they want to say. Whatever they produce is a way of saying 'this is what the world feels like to me, is it the same for you?'. And if it is, it resonates.
But I cannot see how an LLM would 'want' to say anything. If we're talking psychoanlytically of where wanting comes from, and call it a desire to fill a void of how incoherent you actually are, then an LLM doesn't go through that process.
Maybe Searle does, and still wants the characters to make you feel a certain way, in which case the comparison doesn't fit.
Re: Inference is free and instant
#13we've like barely trained these things? the entirety of common crawl is 424 terabytes. that's merely 6 days of 8K raw video.
I think this is a good thing to keep in mind. If you compare that to how much information a young human gets as input for example, it really puts things into perspective.
Re: Inference is free and instant
#14> No matter how fast Searle is, he won't be able to come up with a beautiful and original Chinese poem that has the creative spark special to humans Why not? > Of course, at some level of complexity, it will be stuck in a local maximum of work quality simply because the book has no guide on how to solve the problem at hand. I find this a pretty un-optimistic view, especially from someone building a coding autopilot.…
> Why not? Unsure if this is a useful answer. But Searle/LLM could make something that looks like it has a creative spark, and that's it. Why I think that's different is in the case of a human artist, they create something because they have something they want to say. Whatever they produce is a way of saying 'this is what the world feels like to me, is it the same for you?'. And if it is, it resonates. But I cannot s…
> If we're talking psychoanlytically of where wanting comes from, and call it a desire to fill a void of how incoherent you actually are, then an LLM doesn't go through that process.
Ironically, many people complain LLMs are too incoherent, with all their confabulations and hallucinations.
But I agree. Desire is a good verb. I think that's what differentiates us from the 'machines'. In art, we try to create meaning. From our lives. From our discontents. Even a million LLMs cannot be in deficit of meaning; they are precisely tuned to their own capacity. Whereas something strange about humans is our endless desire for 'more'.
Re: Inference is free and instant
#15we've like barely trained these things? the entirety of common crawl is 424 terabytes. that's merely 6 days of 8K raw video.
This is for LLM's which deal mainly with text. An entire book can be stored as .42 MB according to https://www.quora.com/How-many-megabytes-are-in-a-book.
424 terrabytes text is over a billion books worth of data. On the common crawl website it even says "Over 250 billion pages spanning 17 years." That's an impressive amount of information.
Re: Inference is free and instant
#16> No matter how fast Searle is, he won't be able to come up with a beautiful and original Chinese poem that has the creative spark special to humans Why not? > Of course, at some level of complexity, it will be stuck in a local maximum of work quality simply because the book has no guide on how to solve the problem at hand. I find this a pretty un-optimistic view, especially from someone building a coding autopilot.…
> Why not?
The 'original' part is more important than the 'beautiful' part - which should have been more clear in my writing. This argument also triggers the question "is true originality even possible" but I think the difference for LLMs at the moment is their incapability of building non-obvious analogies. I've yet to be inspired something written by an AI and I don't think simply overfitting a model with all human generated data is enough for that. As I also mentioned in the blog, I would be happily proven in future.
> _if_ you split the process up appropriately I believe this pre-requisite is very important. LLMs so are terrible at planning and splitting a complex task into simpler steps. This might be natural limitation of `next token prediction`. For complex planning, each step should be the result of both the previous and speculative future steps. We try to tackle this by dividing a plan into two: a macro and a micro plan but still a lot to improve there.
p.s. thanks! aider is awesome too!
Re: Inference is free and instant
#17> No matter how fast Searle is, he won't be able to come up with a beautiful and original Chinese poem that has the creative spark special to humans.
This analogy seems flawed to me. Searle is in an empty room but LLMs are not. They are constantly learning from user inputs and data is continuously being made more available for LLM ingestion. I still don't think that an LLM will completely replace humans at pure creativity but I don't see why it can't come close. Especially since we're only 2 years in to this craze.
Re: Inference is free and instant
#18[deleted]
Re: Inference is free and instant
#19> No matter how fast Searle is, he won't be able to come up with a beautiful and original Chinese poem that has the creative spark special to humans Why not? > Of course, at some level of complexity, it will be stuck in a local maximum of work quality simply because the book has no guide on how to solve the problem at hand. I find this a pretty un-optimistic view, especially from someone building a coding autopilot.…
> Why not? Unsure if this is a useful answer. But Searle/LLM could make something that looks like it has a creative spark, and that's it. Why I think that's different is in the case of a human artist, they create something because they have something they want to say. Whatever they produce is a way of saying 'this is what the world feels like to me, is it the same for you?'. And if it is, it resonates. But I cannot s…
I'm not convinced we do "want" to say anything, though. The combinations of physical inputs (which mostly translate to hormones i imagine?) and data inputs seem to drive my behavior to such a degree that i question if i could really do anything else at any given moment.
The whole free will debate seems a bit out of scope (and out of my reach, hah), but nonetheless it feels interesting in the LLM context.
edit: Note that i don't necessarily think LLMs are there or even can be. We seem to technologically small to produce the complexity in ourselves. Nonetheless i'm always interested in how far reduced complexity can take us.
Re: Inference is free and instant
#20I think the discussion around "exponentials" with top-end LLM (think 3.5 sonnet, gpt-4 not the smaller models) scaling is really pointless. The heuristic we have for what to expect from performance is just scaling, which has worked pretty well. These benchmarks are imperfect in lots of ways, aren't necessarily sensitive to showing exponential progress and it is difficult to predict step changes in capability in advan…
> The best lens for future performance of large models is uncertainty.
100% agree. I think to better way to phrase my argument there would be to reject the notion that LLMs are destined to get exponentially smarter (twitter fallacy). This is not to say I believe they are not going to get any smarter in the future. We simply don't know and building a company/product on the expectation of another Moore's Law is dangerous.