Live data from Hacker News

GPT-4 details leaked?

threadreaderapp.com

431–440 of 648 posts

Re: GPT-4 details leaked?

#431

Earlier quoted context omitted.

If anyone has a copyright claim to an LLM, the creators of the input data have more of a copyright claim than the company that trained it. There's a good chance they are not copyrightable at all. I'd bet there's a lot of people willing to take on that risk. However, they might still fall under trade secret law.

Why would an LLM be any less copyrightable than any other piece of software?

The software is the matrix multiplication and gradient descent. We are talking about the numbers in the matrices. They are the output of a training algorithm, so we can only talk about the copyright on the training algorithm, and on its input data.

Re: GPT-4 details leaked?

#432

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

>> We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. Just to be clear, there's no science being kept secret because there is no science being done. OpenAI's is a feat of engineering, borne aloft by a huge budget supporting a large team whose expertise lies in tuning neural net systems, and not in doing science. Machine learning, as it is practiced to…

I'm tired of science as a religion. People treat it as some gospel, like if you check out some criterions you're suddenly "scientific" and instantly get a sense of validity and authority that you shouldn't logically get.

I judge things as "what you can do", not "what can you predict". The only demonstration of knowledge and understanding is being able to do something. Not predict. Not "scientific method" and ridiculous "peer review" (actually peer pressure), not blind trials and not rigorous statistical analysis.

In the end of the day, you either manage to do something or you don't. So much of so called science had lost all contact with reality because our judgement of success isn't successfully doing something, it is successfully jumping through "scientific" hoops. Look at string theory and social sciences. The scientific process, instead of being a tool, became the purpose. It became a stamp of validity to seek. A stamp of validity with gatekeepers in the academia, in the peer review process, in the media coverage afterwards all the way to social media censorship and "fact checkers".

What used to be the frontier of creative people became a stagnant beurocratic machine worshipped like a new religion. The side of the heretics burned at the stake became the ones crying out heresy.

Enjoy your new brand of science. I'll stick to the older brand of heretics and mad men which did whatever it was the prevailing orthodoxy told them to avoid doing and thinking, and I'll remind you that the only real reason those are remembered is because they did something useful, not because of the social traditions they adhered to or the rigorous scientific standards they followed.

Re: GPT-4 details leaked?

#433

Earlier quoted context omitted.

If anyone has a copyright claim to an LLM, the creators of the input data have more of a copyright claim than the company that trained it. There's a good chance they are not copyrightable at all. I'd bet there's a lot of people willing to take on that risk. However, they might still fall under trade secret law.

Why would an LLM be any less copyrightable than any other piece of software?

For the same reason that phone books cannot have copyright.

Re: GPT-4 details leaked?

#434

Earlier quoted context omitted.

> great news for open source. Yes, and great news for shills, bad actors, agitators, trolls, foreign intel, and propagandists. I'm impressed by the tech but terrified because for once I cannot conceive of what this means for the future. My guess is that this kills the open web and laws get passed which bury it.

Everybody is self-soothing with the idea that OpenAI's (frankly, half hearted) push for regulation is just mundane regulatory capture and profit seeking, and not the fact that it will, at best, absolutely destroy everything about the internet and technology that we've come to love and know. Should a 4chan torrent show up like LLaMA, with weights and code for a base GPT4-level model, modern society is done. Golden age…

From your perspective, how would modern society be "done" if GPT-4 was generally available? How would it be substantially different from LLaMA?

Re: GPT-4 details leaked?

#435

Earlier quoted context omitted.

It's several reasons, first it's the lies and the abuse of a charity, that wouldn't be an issue if they had began as a private company instead of robbing a charity. But secondly, even if they were a private company, it's dishonest and reprehensible to claim to congress that you want to "protect the public" when you really only give a shit about protecting your moat, I'm not happy about that either. I'm also tired of…

> the abuse of a charity, A non-profit and charity are two different things. While a charity is a form of non-profit, it has to follow certain rules to qualify as one. Their profits must go towards the charity. A non-profit is a company that is set up not to make a profit. It is allowed to make a profit if it does. This is what OpenAI was. They switched to a "capped" for-profit model so that they could get more fundi…

> It is allowed to make a profit if it does.

A nonprofit is subject to the non-distribution constraint: any revenues that exceed expenses must be committed to the organization's purpose, not taken by private parties.

https://en.wikipedia.org/wiki/Nonprofit_organization

Re: GPT-4 details leaked?

#436

Earlier quoted context omitted.

>> We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. Just to be clear, there's no science being kept secret because there is no science being done. OpenAI's is a feat of engineering, borne aloft by a huge budget supporting a large team whose expertise lies in tuning neural net systems, and not in doing science. Machine learning, as it is practiced to…

Sure. But remember also that the standard “science” is largely a broken model. Academia sucks. It’s honestly bad. It’s overflowing with papers that are misleading, irrelevant, or fraudulent due to a mix of poor stats knowledge, bad incentives, and a failure to organize effectively. While people praise the scientific method, the majority of achievements we attribute to “science” are not derived from guess and check gr…

I think the search for tiny effects and the belief that they should drive treatment, is one of the biggest things wrong with medical research. Doctors will put millions of people on a pill that hits a vital metabolic pathway, for life, based on tiny (to my thinking) statistically significant (i.e. only a 5% chance they aren't based on a real effect) results.

These days I only believe large effects- like smoking causes heart disease

Re: GPT-4 details leaked?

#437

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

The only people pissed are a bunch of developers who want to use it for their own good.

GPT4 costs are ridiculously cheap for the value you get out of it. Any other company wouldn’t even release it to the public like they’ve done

Re: GPT-4 details leaked?

#438

Earlier quoted context omitted.

Great explanations! How about Multi-Query Attention?

This is the original paper: https://arxiv.org/abs/1911.02150 . The idea is that with a transformer you have many heads, say 64 for LLaMa, and for each head you have 1 "query" vector one "key" vector and one "value" vector per token. Most of the cost of inferencing models is loading the key and value vectors from GPU memory to the GPU itself. the idea behind MQA is that instead of having 64 queries, 64 keys, and 64 va…

Beautiful. Thanks!

Re: GPT-4 details leaked?

#439

Earlier quoted context omitted.

LLaMA 30B or 60B can be very impressive when correctly prompted. Deploying the 60B version is a challenge though and you might need to apply 4-bit quantization with something like https://github.com/PanQiWei/AutoGPTQ or https://github.com/qwopqwop200/GPTQ-for-LLaMa . Then you can improve the inference speed by using https://github.com/turboderp/exllama . If you prefer to use an "instruct" model à la ChatGPT (i.e. tha…

Just a reminder that LLaMA is not open—in order to use it legally you have to agree to Meta's terms, which currently means research use only. The versions circulating on torrents are essential pirated, and while I don't have an ethical problem with that at all you can't use it safely in a business. The open replacements for LLaMA have yet to reach 30B, let alone 65B.

People always bring this up like it’s a big deal, but most users aren’t interested in starting a business. We just wanna play with LLMs.

Frankly, I’m glad we don’t have a bunch of llamas in different skins being hawked like the current crop of “AI” startups that are just thin layers over OpenAI’s API.

Re: GPT-4 details leaked?

#440

Earlier quoted context omitted.

>> We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. Just to be clear, there's no science being kept secret because there is no science being done. OpenAI's is a feat of engineering, borne aloft by a huge budget supporting a large team whose expertise lies in tuning neural net systems, and not in doing science. Machine learning, as it is practiced to…

{Hypothesis, test, loop} is the scientific method, and I can guarantee it is being used when fine tuning an LLM.

The reason that this is Engineering as opposed to Science, is that the hypothesis is just, "hey, maybe this will work".

Nobody has really explained why it works.

Post reply on HN