Live data from Hacker News

Big LLMs weights are a piece of history

antirez.com

201–210 of 222 posts

Re: Big LLMs weights are a piece of history

#201
Naming antics aside, the article makes a good point I've heard previously about the importance of the Internet Archive.

Are there any search experiences that allow me to search like it's 1999? I'd love to be able to re-create the experience of finding random passion project blogs that give a small snapshot of things people and business were using the web for back then.

Re: Big LLMs weights are a piece of history

#202

“We should regard the Internet Archive as one of the most valuable pieces of modern history; instead, many companies and entities make the chances of the Archive to survive, and accumulate what otherwise will be lost, harder and harder. I understand that the Archive headquarters are located in what used to be a church: well, there is no better way to think of it than as a sacred place.” Amen. There is an active effor…

Anyone who takes even an hour to audit anything about the Internet Archive will soon come to a very sad conclusion. The physical assets are stored in the blast radius of an oil refinery. They don't have air conditioning. Take the tour and they tell you the site runs slower on hot days. Great mission, but atrociously managed. Under attack for a number of reasons, mostly absurd. But a few are painfully valid.

How significant is "in the blast radius of an oil refinery"? Once every how many years should I expect a typical oil refinery to explode? This really doesn't seem like it should be their first, second, fifth, or twelfth priority to"solve".

EDIT: asking Claude:

Based on historical data, major refinery explosions in developed countries might occur at a rate of approximately 1 in 1,000 to 1 in 2,000 refinery-years of operation. Using this very rough estimate, a single refinery might have approximately a 50% chance of experiencing a significant explosion somewhere between 700-1,400 years of continuous operation.

Re: Big LLMs weights are a piece of history

#203

Earlier quoted context omitted.

Their yearly budget is less than the budget of just the SF library system.

What I don't understand is how do they afford the storage costs? Surely it must be pricey to have 70+ petabytes of data that's only growing.

Their Form 990s are hard to decipher (many many entitites) and they don't talk much about how they're structured or how they run their backend.

A big chunk was outsourced to Sun at one point. And that name alone should tell you how current the information is. https://en.wikipedia.org/wiki/Sun_Modular_Datacenter

In 2020 at least one public filing shows expenses of $19.9MM with $9.2MM classified as wages. So no more than $900k/month in 2020 and maybe double that now. Recent data is messy due to Covid donations and lawsuits.

Re: Big LLMs weights are a piece of history

#204

Earlier quoted context omitted.

Anyone who takes even an hour to audit anything about the Internet Archive will soon come to a very sad conclusion. The physical assets are stored in the blast radius of an oil refinery. They don't have air conditioning. Take the tour and they tell you the site runs slower on hot days. Great mission, but atrociously managed. Under attack for a number of reasons, mostly absurd. But a few are painfully valid.

How significant is "in the blast radius of an oil refinery"? Once every how many years should I expect a typical oil refinery to explode? This really doesn't seem like it should be their first, second, fifth, or twelfth priority to"solve". EDIT: asking Claude: Based on historical data, major refinery explosions in developed countries might occur at a rate of approximately 1 in 1,000 to 1 in 2,000 refinery-years of op…

"11 years ago, the Chevron refinery exploded. It wasn’t a surprise."

https://apen4ej.medium.com/11-years-ago-the-chevron-refinery...

Keep in mind that Brewster bought the building because it looked like the icon, not vice versa. Not exactly the amount of thought that might be expected of an archival institution.

Re: Big LLMs weights are a piece of history

#205

“We should regard the Internet Archive as one of the most valuable pieces of modern history; instead, many companies and entities make the chances of the Archive to survive, and accumulate what otherwise will be lost, harder and harder. I understand that the Archive headquarters are located in what used to be a church: well, there is no better way to think of it than as a sacred place.” Amen. There is an active effor…

Anyone who takes even an hour to audit anything about the Internet Archive will soon come to a very sad conclusion. The physical assets are stored in the blast radius of an oil refinery. They don't have air conditioning. Take the tour and they tell you the site runs slower on hot days. Great mission, but atrociously managed. Under attack for a number of reasons, mostly absurd. But a few are painfully valid.

There should not be any physical centralization. Use a series of redundant IPFS pins and/or torrents or some decentralized database of some kind.

Re: Big LLMs weights are a piece of history

#206
post #205

Earlier quoted context omitted.

Anyone who takes even an hour to audit anything about the Internet Archive will soon come to a very sad conclusion. The physical assets are stored in the blast radius of an oil refinery. They don't have air conditioning. Take the tour and they tell you the site runs slower on hot days. Great mission, but atrociously managed. Under attack for a number of reasons, mostly absurd. But a few are painfully valid.

There should not be any physical centralization. Use a series of redundant IPFS pins and/or torrents or some decentralized database of some kind.

They've tried for years and nobody steps up. And as it turns out they couldn't even maintain torrent files at scale. Broken for years, and still no strategy for versioning them when metadata or files change.

Also until recently their whole model was storing physical material (on an active fault line next to an oil refinery) then allowing digital access to it. Courts ruled that illegal for modern works.

Re: Big LLMs weights are a piece of history

#207
Great idea. Slightly related idea: use the Internet Archive to build a dataset of 6502 machine code/binaries, corresponding manuals, and possibly videos of the software in action.. maybe emulator traces.

It might be possible to create an L LM that can write a custom vintage game or program on demand in machine code and simultaneously generate assets like sprites. Especially if you use the latest reinforcement learning techniques.

Re: Big LLMs weights are a piece of history

#208
post #128

Earlier quoted context omitted.

It's a crazy experience being just physically larger than most of the world. Especially when the size on the label carries some implicit shame/judgement. Like I'm skinny, I'm pretty much the lowest weight I can be and not look emaciated / worrying. But when shopping for a skirt in Asian sizes I was a 4XL, and usually an or L-2XL in European sizes. Having to shift my mental space that a US M is the "right" size for me…

The shame you feel is yours, it's not inherent to the sizing.

The shame is inherent to the crushing expectations put on women's appearances and the pressure to be small. It manifests in clothing sizing for the same reason it manifests standing on a scale, it's a measure of your smallness. And what makes it insidious is that the measures are juuust comparable enough across different people to make people feel bad for not having the same numbers as someone 5" shorter than you.

And my experience isn't unique in any way here and it's really hard to not see it pervasive through our culture.

Re: Big LLMs weights are a piece of history

#209
post #66

Earlier quoted context omitted.

Well, it did establish a new HQ in Canada… https://vancouversun.com/news/local-news/the-internet-archiv... (Edited: apparently just a new HQ and not THE HQ)

With this belligerent maniac in the White House who recently doubled-down on his wish to annex Canada [1], I wouldn't feel safe relocating there if the goal is to flee the US. [1] https://www.nbcnews.com/politics/donald-trump/trump-quest-co...

[dead]

Re: Big LLMs weights are a piece of history

#210
post #191
post #174

Earlier quoted context omitted.

To continue my earlier comment... I prefer not to call an LLM "intelligent" much less "outrageously intelligent". Why? The main reason is communication clarity -- and by communication I mean the notion of a sender communicating a meaning to a receiver. Not just symbolic information (a la Shannon), but a faithful representation in the recipient. The phrase "outrageously intelligent" can have many conflicting interpret…

Intelligence is closely related to the concept of attractiveness and gravitas. You say it depends on the agent. I say it's in the eye of the beholder. People aren't very good at explaining what attracts them either. The closest thing we have to a definition for intelligence is probably the LLMs themselves. They're very good at predicting words that attract people. So clearly we've figured it out. It's just such a sha…

> Intelligence is closely related to the concept of attractiveness and gravitas.

Interesting. I wonder why you make this connection. Do you know?

Your choice of definition seems to be what I would call "perception of intelligence". But why add that extra layer of indirection; why require an observer? I claim this extra level of indirection is not necessary. I eschew definitions with unnecessary complexity (a.k.a "accidental complexity" in the phrasing of Rich Hickey).

Here are some examples that might reveal problems with the definition above:

- DeepBlue (decisively beating Kasparov in 1997) showed a high level of intelligence in the game of chess. The notion of "being good at the game" is simpler (conceptually) than the notion of "being attractive to people who like the game of chess". See what I mean?

- A group of Somali pirates working together may show impressive tactical abilities, including the ability to raid larger ships, which I would be willing to call a form of tactical intelligence to achieve their goals. I grant the intelligent behavior even though I don't find it "attractive", nor do I think the pirates need any level of "gravitas" to do it. Sure, the pirates might use leadership, persuasion, and coordination to accomplish their goals but these concepts are a means to an end accomplishing the goal. But these traits are not necessary. Since intelligent behavior can be defined without using those concepts, why include them? Why pin them to the definition?

- The human brain is widely regarded as a intelligent organ in a wide variety of contexts relating to human survival. Whether or not I find it "attractive" is irrelevant w.r.t. intelligence, I say. If the neighboring tribe wants to kill me and my tribe (using their tribally-oriented brains), I would hardly call their brains attractive or their methods being nuanced enough to use "gravitas".

My claim is then: Intelligence should be defined by functional capability which leads to effectiveness at achieving goals, not by how we feel about the intelligence or those displaying it.

Post reply on HN