31–40 of 70 posts
I’ve seen parts of the Congress testimony of Sam Altman amd it felt like the guy was on a power trip with his regulation ideas. What he accomplished with OpenAI is for the history books, but wanting monopoly over AI development in the US makes him look like an up and coming villain.
What OpenAI accomplished with GPT, happened before Altman joined as CEO.
Earlier quoted context omitted.
We started from thin clients connecting to room-sized mainframes made by giant corporations, and we ended at small cellphone-size devices mostly acting as thin clients connecting to datacenter-building-sized services run by giant corporations, even though the small devices could theoretically do much more than the original thin clients. Being able to run locally won't change much, if the local LLMs are owned by Micro…
It will honestly depend on just how much better the cloud versions are relative to the local versions. If the cloud versions are significantly better (like they are now) than local LLMs, then cloud will continue to be the way. If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. In…
'Better' has many meanings. Ultimately LLM performance depends not only on processor performance, but also on data. One could imagine real-time self-training, where the real-time availability of (allegedly anonymized) data from other users leads to a stable advantage over local LLM.
Another issue is deployment. Imagine this: You are a software developer using an LLM in your commercial product. You have two/three choices.
A) Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers.
B) Include a local LLM, and deal with all the lovely deployment issues, especially when the guys running a 8th gen Intel laptop with an iGPU and 4GB memory call and ask why they can't run your product.
Perhaps there will be a third option: C) Call a local API provided by the OS, which may or may not call a cloud API in the background.
One can see how even if the performance is totally equivalent, Cloud API may end up with an advantage.
IMHO, this is not a desirable future, but to avoid it we need to start by cutting down on the triumphalism.
[EDIT: >If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy.
Cost for whom? The local user will pay less if the call is remote. The cloud AIs can make up for their costs in data. Unfortunately, users often ignore their privacy. ]
Earlier quoted context omitted.
1) Giant serverfarms are expensive, but Microsoft has money and savings on scale. This can compare favourably to running locally. 2) People use what LLMs software makers give them and don't care at all about on premise. Software makers want something reliable they can deploy at scale, and not end up debugging problems on a client 4GB machine with an iGPU and multitude of OSs. The actually deciding people want to dele…
A long time ago, in order to get computers to do anything useful, the hardware required huge rooms. Today, a little cell phone is significantly more powerful than a room sized computer back in the days. LLMs shouldn't be any different. Today, they require giant server farms to run. Tomorrow, they will run on little robots/cell phones. A few things will allow this: 1. Chip makers like Apple, Qualcomm, AMD, Intel, Nvid…
Even if that weren't true and you look exclusively at on-prem and in-hand software, it took 25-30 years to shrink supercomputers to pocket size.
Moore's Law is certainly struggling. Modern developments are at least as much about architectural refinement as transistor density. That won't change unless there's a completely new game changer technology. (Optical? Quantum?) Whatever it is, it hasn't been invented yet. So it's unlikely to be productised within the next ten years.
There's no huge technical benefit to running models locally. Aside from subscription costs - which would be balanced by more expensive hardware - and maybe response time, if the cycles are there, there's not a lot you can do with a local model that you can't do with a hosted model and an API.
The real issue with models is training. If you want to train locally you need access to a huge dataset.
BigCo has that. You don't.
Earlier quoted context omitted.
A long time ago, in order to get computers to do anything useful, the hardware required huge rooms. Today, a little cell phone is significantly more powerful than a room sized computer back in the days. LLMs shouldn't be any different. Today, they require giant server farms to run. Tomorrow, they will run on little robots/cell phones. A few things will allow this: 1. Chip makers like Apple, Qualcomm, AMD, Intel, Nvid…
Today a little cell phone spends much of its time accessing hardware in huge rooms. Even if that weren't true and you look exclusively at on-prem and in-hand software, it took 25-30 years to shrink supercomputers to pocket size. Moore's Law is certainly struggling. Modern developments are at least as much about architectural refinement as transistor density. That won't change unless there's a completely new game chan…
The vast majority of this behavior are basic CRUD operations. The iPhone's single thread speed is likely faster than the server it's calling. For example, the Apple A16 contains 16 billion transistors - which is significantly more than a Zen4 CCD.
>Even if that weren't true and you look exclusively at on-prem and in-hand software, it took 25-30 years to shrink supercomputers to pocket size.
True but technology moves at a faster pace now. Chips get exponentially faster, not linearly. Do I expect a cell phone in 2027 to be as fast as a server room in 2023? No. But I expect the forces I outlined above to make it so that local LLMs will be useful enough for people to keep on their phone or computer? Yes. That's my bet.
>The real issue with models is training. If you want to train locally you need access to a huge dataset.
The assumption is that we will be downloading and updating models similar to how apps are currently deployed.
Earlier quoted context omitted.
1) Giant serverfarms are expensive, but Microsoft has money and savings on scale. This can compare favourably to running locally. 2) People use what LLMs software makers give them and don't care at all about on premise. Software makers want something reliable they can deploy at scale, and not end up debugging problems on a client 4GB machine with an iGPU and multitude of OSs. The actually deciding people want to dele…
1) Did you do the math? Makes no intuitive sense to me. 2) did you do a survey? Because I see no reason to just assume no one wants or needs on-prem. 3) you're extrapolating the current state of affairs into the future with no logical justification for doing so. 4) This just relies on 1-3 and is also only speculation. I can easily see scenarious where big tech fail to pivot into AI properly and go down.
A) Running a remote cloud LLM costs about nothing in dollars to the user.
B) Running a remote cloud LLM does cost for the provider. Note however Microsoft and Google are the cloud, so they don't have to pay the absurd profit margins. They already have specialized hardware and can bring their costs down.
C) Microsoft and Google can probably make money from your data, even if it's 'anonymized' LLM requests.
D) Data for convenience is a typical bargain, e.g. in the search space. There's good reason to assume it can work here too.
2-3) I'm not arguing no one needs on prem. I am extrapolating the current state of affairs into the future, because, well, we don't have much else to go on. The current state of affairs isn't good.
4) Yes, this is called a logical extrapolation. I don't think you have much to base scenarios where big tech goes down upon, but I'll be glad to hear an argument. Hopefully being based on more than predictions of local LLM performance.
Whether scaling laws hold or not, is up for debate. What isn't up for debate, is that: 1) Giant serverfarms are expensive 2) People want on premises/on machine solutions 3) LoRA Tuning of small models continues to excel 4) Thus specialized models continue to evolve at a fast pace 5) Open source foundation models on which tuning can be done are accelerating by the week 6) Performance doesn't matter once a model is "go…
Tbf, local models are still completely crap for that because the base model isn't as good. Software development is expensive so if a 1% better model cuts 10 hours of work it's already worth it.
The problem is more in that you can't send private company data to OpenAI and back because they'll use it for training and leak your IP.
Earlier quoted context omitted.
It will honestly depend on just how much better the cloud versions are relative to the local versions. If the cloud versions are significantly better (like they are now) than local LLMs, then cloud will continue to be the way. If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. In…
>It will honestly depend on just how much better the cloud versions are relative to the local versions. 'Better' has many meanings. Ultimately LLM performance depends not only on processor performance, but also on data. One could imagine real-time self-training, where the real-time availability of (allegedly anonymized) data from other users leads to a stable advantage over local LLM. Another issue is deployment. Ima…
>Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers
In the not too distant future, I expect most applications to be replaced by LLM assistants. So the above scenario won't really happen. You will interact with one LLM, maybe it's a local flavor or maybe it's a cloud flavor. The LLM will then call other services for you. So I don't expect services to call LLMs but I do expect LLMs to call other services.
Yes, we are getting a little ahead of ourselves.
Earlier quoted context omitted.
As I understand it, most people's objection to the calls for regulation are based on the fact that regulation raises the barriers to entry for competition. Regulatory compliance can be expensive and favors those with deep pockets. The more AI startups one can prevent from starting/growing now, the fewer competitors there will be in the future.
All of which is entirely reasonable and so is considering all of Samas possible motivations. But just declaring another parties intentions as you see fit is such an incredibly bad style in any discourse. When filling the gaps of your knowledge, there has to be a world in which Sama is in fact a good human being – in addition to the world where he is not – when that assumption does not logically conflict with anything…
I’ve seen parts of the Congress testimony of Sam Altman amd it felt like the guy was on a power trip with his regulation ideas. What he accomplished with OpenAI is for the history books, but wanting monopoly over AI development in the US makes him look like an up and coming villain.
Nothing that he said in Congress supports the claim that he wants monopoly over AI development. I find it fascinating how people are so happy to turn argumentation on it's head, and pretend it's the same thing. Surely you must know you are doing it? But what is to gain, I do not know.
Earlier quoted context omitted.
>It will honestly depend on just how much better the cloud versions are relative to the local versions. 'Better' has many meanings. Ultimately LLM performance depends not only on processor performance, but also on data. One could imagine real-time self-training, where the real-time availability of (allegedly anonymized) data from other users leads to a stable advantage over local LLM. Another issue is deployment. Ima…
> Another issue is deployment. Imagine this: You are a software developer using an LLM in your commercial product. You have two/three choices. > Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers In the not too distant future, I expect most applications to be replaced by LLM assistants. So the above scenario won't really happen. Y…
>In the not too distant future, I expect most applications to be replaced by LLM assistants. So the above scenario won't really happen.
Regardless we have the question of how to deploy the local LLM. If the frontend will be the OS, than we can be reliably sure that Microsoft and Google will have motivation to use their cloud hosted AI and you may have to fight the OS to get it to install your model. We may need regulation to ensure local LLMs are possible.