Hmm “Sam Altman won't tell you that GPT-4 has 220B parameters and is 16-way mixture model with 8 sets of weights” George Hotz said this in his recent interview with Lex Fridman. It looked like Lex knew this to be true by the way he reacted.
GPT-4 details leaked?
231–240 of 648 posts
Re: GPT-4 details leaked?
#232Earlier quoted context omitted.
It's not even remotely the same as anti-homeless spikes. We've just had the massive fine against 3M for knowing and hiding the risks of PFAS, and the top comments here were "increase the fines! Lock up the bosses!" Now we have a company going "we had to put a lot of effort into preventing this model from cheerfully outputting Al Quaida propaganda, explicit rape threats, and detailed instructions for an amateur to mak…
> "outputs al qaeda propaganda" So does Microsoft Word. Both require a human to tell the software what to output. > as if none of the other FAANGs could trivially cross any regulatory barriers that emerge. That's the point. This miracle technology does not belong to a few rich men, it belongs to us all. Tech oligarchs are provably not more responsible than the rest of us, and do not deserve to lock us out of the gard…
> So does Microsoft Word. Both require a human to tell the software what to output.
Clippy does what now?
Or do you mean "I can type", because if so you're minimising the very same capabilities that you're later describing as a miracle and saying belongs to us all.
> Tech oligarchs are provably not more responsible than the rest of us, and do not deserve to lock us out of the garden whose fruit they seek to pick.
"No more responsible than the rest of us" is a dangerously low standard.
The rest of us, collectively rather than each and every one of us, play lotteries, drive dangerously, addict ourselves to drugs, pickle our livers, and win Darwin awards.
For all our sophistication and sophistry, we're all just fancy balding primates with fairly similar tribal attitudes and motivations.
> YOU are not allowed to build the same robot to help boost your own income
Yes, obviously, with literal robots there are countless examples of public liability insurance and health & safety legislation. With computers, likewise, because they're connected to stuff.
Re: GPT-4 details leaked?
#233Earlier quoted context omitted.
The real moat is an abundance of high quality data.
Yeah they have the internet from before LLMs were used for anything, so the data is not poisoned. Not unlike carbon dating becoming useless for estimating age of anything made after nuclear atmospheric tests, or low-background steel.
You talk as if humans weren't perfectly capable of coming up with nonsense.
Blogs upon blogs full of worthless pap that is there for SEO reasons have existed for like a decade already.
Re: GPT-4 details leaked?
#234Earlier quoted context omitted.
Not sure if you're being sarcastic but I agree. Humans should not have AI research or advanced AI at all. It (a) removes purpose from people, (b) presents a situation that is too alien for human minds to handle, (c) increases the addictiveness of technology and thereby pushes us further into growing the technological system, (d) crosses the "adaptability threshhold", i.e. the point at which the PACE of technological…
It’s ironic you say: “we are playing with fire.” Playing with fire is, in large part, literally how humans have come to dominate this planet. Why stop now?
No, playing is how humans grow up to be adults that don't play, but think.
Re: GPT-4 details leaked?
#235Earlier quoted context omitted.
How do you know? do you have insider knowledge of this or is it just based on what they share publically?
as a fun ancedote, the Google Bard's implicit code execution update from *last month*, advertised by Sundar... no longer works https://twitter.com/swyx/status/1678495067663925248 i'd love to know whats going on in that team.
> i'd love to know whats going on in that team.
Seems like a good time to rewatch Silicon Valley and watch Hooli scramble to keep up.
Re: GPT-4 details leaked?
#236Earlier quoted context omitted.
> try to pressure the government into making it illegal for you to compete with us. I mean the guy who created GPT-4 literally demanded a ban of any system more powerful than GPT-4.
I don’t understand this. Won’t that hurt their progress on GPT-5?
The public position (as opposed to the rumour mills) is that they're not working on a 5, and don't intend to at least until they can figure out how to do it safely.
Re: GPT-4 details leaked?
#237> This, of course, is “only” a batch size of 7.5 million tokens per expert due to not every expert seeing all tokens. > Mixture of Expert Tradeoffs: There are multiple MoE tradeoffs taken: For example, MoE is incredibly difficult to deal with on inference because not every part of the model is utilized on every token generation. Are these experts able to communicate among them in one query? How do they get selected?…
You shouldn't take the "mixture of experts" too literally, it's yet another architecture to use internally for a gradient descent optimized graph of ops.
I obviously don't know how GPT-4 do it (or if it even does it) but think of partitioning your network into a couple of very isolated sub-graphs (the "experts"), and add another learnable network between the input tokens and the experts, that learns to route tokens to 1 or more expert sub-graphs. Then the gain is that you can potentially ignore running the unused sub-graphs completely for that token, and you can distribute them on other GPUs as except for the input and output they are independent of each other.
It all depends on the problem, data, and if the gradient descent optimizer can find a way to actually partition the problem usefully using the router and "experts".
Re: GPT-4 details leaked?
#238Earlier quoted context omitted.
It actually does make a difference. The genie is out of the bottle partially but it depends a lot on what we allow it it be used on. If we sit idly and allow for ingesting all what’s written for instance, including whats currently written and let bros make derivative works for a quick buck then we mostly killed the writer’s incentive to write or publish. If we slow down and not allow ripping one another off it could…
You can slow things down, but not by more than a few years, because of the gradual democratization of training foundation models. Right now training a model competitive with chatgpt can be done for $150K (microsoft orca 13b). In a few years the cost will be low enough that individuals can train models. At that point regulating it will require draconian dictatorships. I’m also very wary of the copyright angle on this,…
>> You can slow things down, but not by more than a few years, because of the gradual democratization of training foundation models.
Just to be clear, what's being "democratised" is the fine-tuning of second-tier, inferior-performance models; or pre-training of third-tier ones. In the game of training large neural nets, the players that can afford to train the largest models with the most amount of data and compute at any given time will continue to dominate for the foreseeable future.
To make it plain, maybe in a couple of years you'll be able to train GPT-4 on your student laptop (unlikely, but let's allow it for the sake of argument). You'll still not be able to get anywhere near the performance of GPT-6 or whatever OpenAI and Google will be able to train by then.
Academics, hobbyists and smaller companies will continue to play second fiddle to large corporations as long as the dominant paradigm is more data and more compute.
Re: GPT-4 details leaked?
#239If this is true, then: 1. Training took 21 yottaflops. When was the last time you saw the yotta- prefix for anything? 2. The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat.
>> The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat.
That really doesn't change anything at all. The more training large models gets cheaper, the more large corporations are able to train larger models than everyone else.
Suppose the gross price of rice was $0.001 a kg. That's dirt cheap! Yet, if I had a million dollars and you had a thousand dollars, I could still buy a thousand times more rice than you.
Re: GPT-4 details leaked?
#240Earlier quoted context omitted.
- HN on social media : The powers are too centralized, future is decentralization, question is how - HN on free software: is good - HN on copyright : WAY TOO MUCH PEASANTS CLAIMING INDIVIDUAL RIGHTS, RIGHTS THAT ARENT EVEN REAL, ART BE CENTRALIZED FOR MAXIMUM MONOPOLY
Copyright does very little for individuals. Most benefits from the copyright system are accrued to large corporations.
> Most benefits from the copyright system are accrued to large corporations
Citation fucking needed. Among those who study copyright and inequality, none suggest abandoning it [1][2].
Within the context of machine learning, one of the only pillars buttressing individuals against multi-trillion dollar corporations is copyright [3].
[1] https://journals.library.wustl.edu/lawreview/article/id/5108...
[2] https://www.jstor.org/stable/1339714
[3] https://sfstandard.com/2023/07/10/cruise-and-waymos-24-7-san...