41–50 of 768 posts
I feel like the whole ChatGPT bubble has really highlighted what feel like some fundamental shortcomings in the worldview that is represented in tech. That is, there seems to be a winner take all dynamic baked into the tech world. Maybe this arises from the simple fact that tech places a lot of power in the hands of individuals. But there's an emergent downside to this which is that it makes those who were already mu…
I work on a research team in FAANG. What it really feels like is one company made everyone else obsolete. And we are going to work working on NLP models that underperform ChatGPT by a huge margin. Twiddling my thumbs and keeping quiet while no one wants to recognize the elephant in the room. Also, there is no "working in AI", a few thousand people are doing real AI at most. The rest of us are calling an API.
Not sure why LLM would make Facebook (ads), Apple (hardware), Amazon (hosting, retail), Netflix (tv) obsolete. It's definitely something Google needs to think about, but there's no reason to think they won't again be the leader soon.
Once again, their ability to do computation on device and optimize silicon to do it, is unparalleled.
A huge Achilles heel of current models like GPT-4 is that they can’t be run locally. And there are tons of use cases where we don’t necessarily want to share what we’re doing with OpenAI.
That’s why if Apple wasn’t so behind on the actual models (Siri is still a joke a decade later), they’d be in great shape hardware-wise.
Earlier quoted context omitted.
I wonder how much of usefulness of ChatGPT that some report is caused by https://en.wikipedia.org/wiki/Novelty_effect Because it is new, and cool, and fascinating people are spending a bit more time with a bit more focus on the task with it compared to the old ways, hence the productivity increase. Once the novelty wears off…
It has helped me learn things more interactively because I can ask questions and get answers without having to read 10 blog posts or a book before understanding things. For my personality, it would be hard to recreate without having a living mentor available 24/7 that had knowledge of everything I wanted to ask of it. Does it get things wrong? Sure. But generally the subtle ways it might get stuff wrong would be the…
Earlier quoted context omitted.
Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?
Until we see otherwise, don't we have to assume there's some secret sauce? Bard doesn't match GPT4 and it isn't for a lack of trying. (though perhaps that will change, so far that's the case)
I feel like the whole ChatGPT bubble has really highlighted what feel like some fundamental shortcomings in the worldview that is represented in tech. That is, there seems to be a winner take all dynamic baked into the tech world. Maybe this arises from the simple fact that tech places a lot of power in the hands of individuals. But there's an emergent downside to this which is that it makes those who were already mu…
Human nature, not just tech
Earlier quoted context omitted.
Until we see otherwise, don't we have to assume there's some secret sauce? Bard doesn't match GPT4 and it isn't for a lack of trying. (though perhaps that will change, so far that's the case)
Google is going slow. They might be behind, but we haven't seen their best effort yet. Google has a 540 billion parameter model.
I work on a research team in FAANG. What it really feels like is one company made everyone else obsolete. And we are going to work working on NLP models that underperform ChatGPT by a huge margin. Twiddling my thumbs and keeping quiet while no one wants to recognize the elephant in the room. Also, there is no "working in AI", a few thousand people are doing real AI at most. The rest of us are calling an API.
Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?
GPT3.5 turbo is much more interesting probably, because they seem to have found out how to make it much more efficient (some kind of distillation?).
GPT4 if I had to make a very rough guess, probably flash attention, 100% of the (useful) internet/books for it's dataset, and highly optimized hyperparameters.
I'd say with GPT4 they probably reached the limit of how big the dataset can be, because they are already using all the data that exists. Thus for GPT5 they'll have to scale in other ways.
Earlier quoted context omitted.
Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?
I dont work at Google, I think other FAANGS underinvested in this area as they didnt think it was promising. But I will admit, I am suspicious that Google is incompetent. Probably they can come back given how much money they will be forced to throw at it. But Bard is clearly behind and I dont believe their "abundance of caution" arguments for why Bard cant even code.
> I am suspicious that Google is incompetent.
Google has put a lot of effort and investment into AI. With Bard I get the feeling they're not showing us what they really have - it's like for some reason they're holding back the good stuff, at least that's my suspicion.
I work on a research team in FAANG. What it really feels like is one company made everyone else obsolete. And we are going to work working on NLP models that underperform ChatGPT by a huge margin. Twiddling my thumbs and keeping quiet while no one wants to recognize the elephant in the room. Also, there is no "working in AI", a few thousand people are doing real AI at most. The rest of us are calling an API.
Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?
Earlier quoted context omitted.
Until we see otherwise, don't we have to assume there's some secret sauce? Bard doesn't match GPT4 and it isn't for a lack of trying. (though perhaps that will change, so far that's the case)
Bard would not trick anyone into thinking it was sentient, yet something they have supposedly did. I just think Google has far more to lose than Bing, so they are being more cautious.
Bard is overtly a reduced-resources model compared to the best version of the same technology (which, if true, is probably a boneheadedly bad choice for a public demo when everyone is already wowed by the people who got theirs out first, but easily explains that disparity. Though so does “guy who wanted public attention made stuff up well-calibrated to that goal.”)