191–200 of 267 posts
Earlier quoted context omitted.
Governments have killed more people than any other institution. Corporations have largely made our lives better.
Both governments and corporations have generally made our lives better. Both governments and corporations kill people (look up annual tobacco deaths, for just one fun example).
Earlier quoted context omitted.
USA hasn't killed more than dozens of millions. If that is chump change then it makes USA looks pretty good for an empire.
Well except that it definitely has. From more than 50 million people in pre-colonial times to less than 3 million in the 90s. USA has done genocide on a scale similar any other ethnic cleansing or to any communist regime.
USA didn't exist pre-colonial times, it started as a colony. Do you mean the British Empire? That one did way worse, yeah, but USA isn't the British Empire.
A lot of this was new to me, but it looks like Intel hopes to use this to demonstrate the linear scaling capacity of their Aurora nodes. Argonne installs final components of Aurora supercomputer (22 June 2023): https://www.anl.gov/article/argonne-installs-final-component... Aurora Supercomputer Blade Installation Complete (22 June 2023): https://www.intel.com/content/www/us/en/newsroom/news/aurora... Intel® Data Cent…
yes, I can't imagine the architecture of a supercomputer is the right one for LLM training. But maybe? If not, spending years to design and build a system for weather and nuke simulations and ending up doing something that's totally not made for these systems is kind of a mind bender. I can imagine the conversations that led to this: "we need a government owned LLM" "okay what do we need" "lots of compute power" "wel…
Earlier quoted context omitted.
Was it ever confirmed whether GPT-4 is a Mixture of Experts or not?
I just asked GPT-4, and it denies being a MoE model. User: Are you an MoE model? ChatGPT: No, I am not based on a Mixture of Experts (MoE) model. My underlying architecture is based on the GPT (Generative Pre-trained Transformer) framework, specifically the GPT-4 version. This architecture is a large-scale transformer-based neural network, but it does not use the MoE approach. In a GPT model like mine, the entire mod…
And OpenAI would not add such proprietary information there.
Earlier quoted context omitted.
> "No human requires the amount of training data we are already giving the models." Well, humans are also trained differently. We interact with other humans in real time and get immediate feedback on their responses. We don't just learn by reading through reams of static information. We talk to people. We get into arguments. And so on. Maybe the ideal way to train an AI is to have it interact with lots of humans, so…
Yeah I'm not totally convinced humans don't have a tremendous amount of training data - interacting with the world for years with constant input from all our senses and parental corrections. I bet if you add up that data it's a lot. But once we are partially trained, training more requires a lot less.
https://osf.io/preprints/psyarxiv/qzbgx
> Large language models show intriguing emergent behaviors, yet they receive around 4-5 orders of magnitude more language data than human children.
The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…
Learning in third person from the past data can only take AI so far. It needs to act and learn in the present, in first person, to be truly smart. No architectural change is needed, but the model needs to be placed in a real environment to get feedback signals.
Earlier quoted context omitted.
There's only one US Government and we know it did fairly bad things. There are many private entities, although often not benevolent, most still have space for the benefit of doubt. OpenAI, DeepMind or Anthropic haven't done anything that compares to, say, MKUltra or lying about Iraqi WMDs to start decade long wars.
This is kind-of making the parent’s point. There are lots of private companies that have done very bad things, too many to even count, as bad as your examples and worse. And there are thousands upon thousands of sub-organizations of the US government, some of which do very good things and are definitely benevolent, because there is no built-in profit motive. Private companies almost cannot be benevolent and survive,…
Absolutely not; there's no private company that's shot and bombed the life out of hundreds of thousands of innocent people in cold blood.
The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…
Earlier quoted context omitted.
I just realized that the NSA has probably been able to train GPT-4 equivalents on _all the data_ for a while now. We'll probably never learn about it but that's maybe scarier than just the Snowden collection story because LLMs are so good at retrieval.
Holy shit, you are right. They probably have 10-100x the data used to train gpt-4. Decades of every text message, phone call transcript, and so on. I can’t believe I haven’t seen anyone mention that yet. People keep saying we don’t have enough data. I think there is a lot more data than we realize, even ignoring things like NSA.
I couldn’t find the average message length, but I would guess it’s fairly short (with a fat tail of longer messages).
To make the math easy, let’s say the average length is ~10 tokens. I’d be surprised if that isn’t correct within a factor of 2 or so.
So we have 20 trillion tokens per year from text messages in the US alone. And this is high-quality conversational data.
The annual numbers were fairly constant in recent years (and then it drops off), so the past decade of US text messages is about 200 trillion tokens! That’s a metric fuck ton… Much larger than any dataset existing models have been trained on, I believe.
I would guess phone transcripts would be an order of magnitude larger at least. Talking is a lot easier than typing on a phone.
You could train an absolutely insane model with that amount of data… Damn.
[1] https://www.statista.com/statistics/185879/number-of-text-me...