Live data from Hacker News

GPT-4 details leaked?

threadreaderapp.com

551–560 of 648 posts

Re: GPT-4 details leaked?

#551
post #197

Earlier quoted context omitted.

So far as anyone knows, this is not a derivative work, its transformative, and therefore not subject to any licensing requirement. You're right though, that's arguably still up for debate, but I think the precedent of transformative work is pretty well attested.

is it? Because condensation is literally part of the definition of derivative, and basically the weights are a condensed form of the input data. It's some sort of lossy compression, when looking at it from the right point of view. Summarization and translation are also clearly derivative. The definition of transformative I found: - add something new (context of other books I guess, this one might pass) - with a furth…

is it compression though? It's a model of the way our language works and contains a set of knowledge.

If I read 200 physics textbooks, I couldn't recall any of them exactly, but I could write another book... that would contain largely the same knowledge.

The same is true of AI, it's not "compression", it's "learning" (by recording statistics about relationships between pieces of information) -- it can't (usually) recreate whole works, it'll make stuff up and alter it and whathaveyou. How much of the sum total of the internet and all books can you compress into an n-gigabyte model? I've got an 8 gigabyte file here that seems remarkably smart, but it can't recite the contents of most books to me, though it knows their famous quotes and some excerpts.

The "compression" argument does seem to have some merit, but it also seems thin in the end and, in my opinion, unlikely to hold up in court, but we'll see.

I do think a very strong, probably winning argument could be made in court that it is transformative, but that's the nature of this whole debate, we don't know yet with certainty.

Re: GPT-4 details leaked?

#552
post #486

Earlier quoted context omitted.

To be clear, you just made up MoE details while MoE is actually well established and hails from decades old research?

This is common behavior for inference based learners who don’t hail from strong academic backgrounds. Many developers who are self taught utilize a similar method of learning, essentially using pattern recognition to make “educated guesses” that are then internalized as potential facts and tested at the earliest opportunity. In this instance the test was to project the incorrect information out onto a public forum co…

> Yes, this is done in lieu of actually looking up extended details on what something means.

If they were capable of understanding the extended details, they would already have an academic background in the subject. Laymen aren't going to have a clue what MoE means even if they went to the trouble of digging up the paper.

> Many developers who are self taught utilize a similar method of learning, essentially using pattern recognition to make “educated guesses” that are then internalized as potential facts and tested at the earliest opportunity.

Using pattern recognition skills to make an educated guess that is internalized as a potential fact sounds an awful lot like what LLMs do. At least when humans do it, we bother with the verification step instead of just acting like we know what we're talking about.

Re: GPT-4 details leaked?

#553

Earlier quoted context omitted.

Understanding is a moving target, though. Newtonian mechanics was incorrect for modeling the solar system as well - as it was eventually superseded - but that doesn't mean it wasn't a scientific understanding of planetary motion. Epicycles gave a better description of the movement of the planets, based on the observations available, which was entirely falsifiable. They were eventually superseded, and that's science a…

Interesting to say it's scientific because falsifiable. The objection is that it wasn't a theory, just fitting a function to data. It did "work" in that it captured some pattern: it was extremely good at generalizing/extrapolating/predicting. And was a "model" of something in the data. But there was no operational model behind it, of what was actually happening. Newtonian mechanics has a model, beyond curve fitting.…

This is incorrect: Epicycles were a model of planetary motion - the theory was that the planets moved around the earth, but also had additional circular motion as they moved along their path around the earth. This model explains the apparent geocentric motion of the planets, much as Newtonian gravity explains the apparent heliocentric motion of the planets (but is also wrong). Finding the exact parameters for the epicycles was the curve fitting part.

We now discount that model because it's based on an incorrect geocentric model of the solar system, but that doesn't mean that the model wasn't a model...

[edit] The CHomsky link is interesting - I just listened to an interview where he pooh-poohs LLMs at great length.

This point: "Statistical models have been proven incapable of learning language; therefore language must be innate, so why are these statistical modelers wasting their time on the wrong enterprise?" is interesting in that context - in the recent interview, Chomsky is now unhappy that LLMs can learn /any/ language, even unnatural ones, and therefore aren't good tools for understanding human language. Quite the reversal. I personally think it's a 'science progresses one funeral at a time' kind of situation...

Re: GPT-4 details leaked?

#554

Earlier quoted context omitted.

Science is not academia. You say you prefer to judge things as "what you can do". Well, how do you judge "what you can do"? Astrologists, homeopaths, podiatrists, Christian scientists (!!!) and other such "heretics and mad men" rejected by the scientific establishment, will all tell you that they "can do" stuff, and so will all their many paying customers. How do we know they can't do what they say? Because science g…

Placebos are known to work. I neither overestimate nor underestimate them. I understand them for what they are: placebos. And some people actually need them. Moreover, in social contexts, religion also "works" in many senses. A person asked me what me what can he do about depression and feeling of meaninglessness. Science would prescribe anti-depressants, medicate him and he would both have side-effects and the probl…

Sorry but I don't want to continue this discussion if you're accusing me of gaslighting others, and of being gaslit myself. Or of pretending to be a "spokesperson for science". I was hoping to have an honourable exchange.

Re: GPT-4 details leaked?

#555

Earlier quoted context omitted.

Interesting to say it's scientific because falsifiable. The objection is that it wasn't a theory, just fitting a function to data. It did "work" in that it captured some pattern: it was extremely good at generalizing/extrapolating/predicting. And was a "model" of something in the data. But there was no operational model behind it, of what was actually happening. Newtonian mechanics has a model, beyond curve fitting.…

This is incorrect: Epicycles were a model of planetary motion - the theory was that the planets moved around the earth, but also had additional circular motion as they moved along their path around the earth. This model explains the apparent geocentric motion of the planets, much as Newtonian gravity explains the apparent heliocentric motion of the planets (but is also wrong). Finding the exact parameters for the epi…

At some point I'd like to carefully study the history of that early era of science because I don't know it as well as I'd like. But I believe I understand that the epicyclical model (and it was a model, rather than a theory) did not in any way depend on geocentrism. For one thing, Coppernicus' model itself, while heliocentric, retained the epicycles of the earlier, geocentric model. Instead, the assumption on which the epicyclical model depended was the shape of the planets' orbits and of the planets themselves, which were considered to be necessarily circular, and spherical, respectively. I think this had to do with assumptions about the geometric perfection of the universe, as a creation of the gods. In any case, assuming that planetary orbits were circular an explanation was needed for the apparent "retrograde" motion of the planets (meaning it looks like they double back and turn against their original heading). Explaining this apparent motion was why epicycles were hypothesised in the first place.

The first time this necessarily circular model was abandoned was with Kepler's laws of planetary motion, which correctly identified the motion of the planets as elliptical, that for the first time explained their apparent retrograde motion without the need for epicycles. Then Newton's theory of universal gravitation explained how the planets could possibly be moving on elliptical orbits. In fact, I believe Newton's theory of universal gravitation explained how the planets could be turning around the sun without crashing down, despite not having anything to hold them up. I think this was the first big mystery that the ancients tried to answer- hence the name of "firmament" for the universe.

And Newton's theory was not the end of the story of course.

Re: GPT-4 details leaked?

#556

Earlier quoted context omitted.

>> The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat. That really doesn't change anything at all. The more training large models gets cheaper, the more large corporations are able to train larger models than everyone else. S…

At a certain point though, models become good enough for particular tasks. Once that happens for whatever my application is, I don't care if OpenAI has a model that's twice as good on some metric, because it's overkill for my use-case. I'm going to be happy using a smaller, cheaper model from a competitor.

One important milestone a model that is good enough to produce an acceptable quality of answer to x% of public users questions without any data being sent to the megacorps.

Re: GPT-4 details leaked?

#557

Earlier quoted context omitted.

>> We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. Just to be clear, there's no science being kept secret because there is no science being done. OpenAI's is a feat of engineering, borne aloft by a huge budget supporting a large team whose expertise lies in tuning neural net systems, and not in doing science. Machine learning, as it is practiced to…

> Machine learning, as it is practiced today, is not science. There is no scientific theory behind it and there is no scientific method applied. There are no scientific questions asked, or attempted to be answered. Total horseshit. There are tons of scientific papers on ML published. In fact it is MORE like traditional science than typical CS, because it is trying to reverse engineer how something we encountered in t…

[deleted]

Re: GPT-4 details leaked?

#558

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

Keep in mind, that was the idea originally. Then in 2018 Elon decided he wanted to be CEO, the board rejected that idea for now obvious reasons, and he reneged on 90% of a promised $1b donation. The only way forward was to become a for-profit company and do a normal funding round, which is what happened with Microsoft. Elon’s rug pull is why this happened.

OpenAI was so destitute, having only $100,000,000, that they had no choice but to fuck us all and lobby congress to make it illegal for us to use this new technology? That doesn't make sense.

If your choice is between $100M + doing earnest work, or $1B+ and debasing our free and competitive society, and you take the latter... what does that say about your collective character as an organization and as a set of people?

Re: GPT-4 details leaked?

#559

Earlier quoted context omitted.

Computer science is the study, that is looking at how computers and the use of computers can benefit mankind (in my short lay version). Software engineering is a subset of computer science where you build a real world application of the studies. The algorithms behind the Facebook feed and your favorite AI/ML product is the science. The code that powers the feed and chatgpt is the engineering. Wikipedia is not wrong I…

I agree with you. There's a lot of engineering, and science going into IT. But I'm a Software Engineer too, and while I'm clever and do clever things, I can assure you that there's no rigor or reason to call it engineering. This version of engineering is at most the re-use of the word, like how the word art is not just covering artistic expressions, but skillfulness too, even though when you apply a skill artfully, y…

Engineering is at all levels, I’ve designed systems that span global infrastructure and systems that run in embedded systems. I still think that design process is a key part of engineering. Figuring out how to put things together is part of design. I agree the more jr engineer you are, the less “engineering” you see.. but you can’t run until you learn to walk.

I talk to my friends who sent projects into space and friends who design deep sea drilling rigs. The engineering process is very similar.

Re: GPT-4 details leaked?

#560

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

>> We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. Just to be clear, there's no science being kept secret because there is no science being done. OpenAI's is a feat of engineering, borne aloft by a huge budget supporting a large team whose expertise lies in tuning neural net systems, and not in doing science. Machine learning, as it is practiced to…

Of course, 99.9% of science is engineering, not "science". We do science all the time without asking questions or attempting to come up with generalized answers.

Is a scientist doing a linear regression not doing science?

Anyway, alphafold, while not particularly "scientific", did answer one scientific question: it is possible to predict the structure of most proteins thru a combination of limited structural and extensive sequence information, combined with a sophisticated (and "non-scientific") algorithm. That was an open question for some time and their results convinced the community that their methods were right. What's amazing is that while it's entirely nonscientific, the results have been absolutely blockbuster in the scientific field. And even better, the only reason Alphafold was able to show this is because there was a well-defined protein structure leaderboard.

Post reply on HN