211–220 of 525 posts
dude someone lied their way into being president of the united states all while people fact checked him basically immediately after each lie. lying doesnt make a difference.
The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
https://twitter.com/SmokeAwayyy/status/1646670920214536193 Sam explicitly said that there won't be GPT-5 in the near future, which is pretty clear evidence unless he's blatantly lying in public speaking.
The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
Does this mean we’ve reached the next AI winter? This is as good as it gets for quite a long time? Honest question :) perhaps this will postpone everyone’s fears about the singularity…
This is analogous to what’s happening with AI models. Sam Altman is saying we have reached the point where spending $100M+ trying to “beat” GPT-4 at everything isn’t the future. The next step is to chunk off a piece of it and turn it into something a particular industry would pay for. We already see small sprouts of those being launched. I think we will see some truly large companies form with this model in the next 5-10 years.
To answer your question, yes, this may be as good as it gets now for monolithic language models. But it is just the beginning of what these models can achieve.
Earlier quoted context omitted.
OpenAI has gone from open-sourcing its work, to publishing papers only, to publishing papers that omit important information, to GPT-4 being straight-up closed. And Sam Altman doesn't exactly have a track record of being overly concerned about the truth of his statements.
I had a fun conversation (more like argument) with ChatGPT about the hypocrisy of OpenAI. It would explicitly contradict itself and then began starting every reply with “I can see why someone might think…” and then just regurgitating fluff about democratizing AI. I finally was able to have it define democratization of technology and then recognize the absurdity of using that label to describe a pivot to gating models…
It's easy to recognize and laugh at the AI replying with the preprogrammed narrative, I'm still waiting for the majority of people realizing they are given the same training materials, non-stop, with the same toxic narratives, and becoming programmed in the same way, and that is what results in their current worldview.
And no, it's not enough to be "skeptic" of mainstream media. It's not even enough to "validate" them. Or to go to other sources. You need to be reflective enough to realize that they a pushing a flawed reasoning methods, and then abusing them again and again, to get you used to their brand of reasoning.
Their brand of reasoning is just basically reasoning with brands. You're given negative sounding words for things they want you to think are bad, and positive sounding words for things they want you to think are good, and continuously reinforce these connections. They brand true democracy (literally rule of the people) as populism and tell you it's a bad thing. They brand freedom of speech as "misinformation". They brand freedom as "choice" so that you will not think of what you want to do, but which of the things they allow you to do will you do. Disagree with the scientific narrative? You're "science denier". Even as a professional scientist. Conspiracy theory isn't a defined word - it is a brand.
You're trained to judge goodness or badness instinctively by their frequency and peer pressure, and produce the explanation after your instinctive decision, instead of the other way around.
Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so c…
The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
Does this mean we’ve reached the next AI winter? This is as good as it gets for quite a long time? Honest question :) perhaps this will postpone everyone’s fears about the singularity…
Another good example is in the coding landscape, which feels closer to existing. Ingest all of a company's code into a model like this, then start thinking about what you can do with it. A chatbot is one thing, the most obvious thing, but there's higher order product use-cases that could be interesting (e.g. you get an error in Sentry, stack trace points Sentry to where the error happened, language model automatically PRs a fix, stuff like that).
This shit excites me WAY WAY more than GPT-5. We've unlocked like 0.002% of the value that GPT-3/llama/etc could be capable of delivering. Given the context of broad concern about cost of training, accidentally inventing an AGI, intentionally inventing an AGI; If I were the BDFL of the world, I think we've got at least a decade of latent value just to capture out of GPT-3/4 (and other models). Let's hit pause. Let's actually build on these things. Let's find a level of efficiency that is still valuable without spending $5B in a dick measuring contest [1] to suss out another 50 points on the SAT. Let's work on making edge/local inference more possible. Most of all, let's work on safety, education, and privacy.
[1] https://techcrunch.com/2023/04/06/anthropics-5b-4-year-plan-...
Earlier quoted context omitted.
But we're still nowhere near that, or even near surpassing the skill of an average person at a moderately complex information task, and GPT-4 supposedly took hundreds of millions to train. It also costs a decent amount more to run inference on it vs. 3.5. It probably makes sense to prove the concept that generative AI can be used for lots of real work before scaling that up by another order of magnitude for potential…
There are a lot of employees at large tech consultancies that don't really do anything that can't be automated away by even current models. Sprinkle in some more specific training and I can totally see entire divisions at IBM and Accenture and TCS being made redundant. The incentive structures are perversely aligned for this future - the CEO who manages to reduce headcount while increasing revenue is going to be very…