And yet the Anthropic CEO is still claiming PhD level intelligence in next couple of years to Lex Friedman. It's starting to feel like the whole crypto pump and dump again
OpenAI, Google and Anthropic are struggling to build more advanced AI
471–480 of 622 posts
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#472Scientific benchmarks score are not necessary related to the rate of completion of tasks such as user persuasion. Software engineering is more important when the current state-of-the-art small language model is sufficient for soltion of our application.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#473Earlier quoted context omitted.
> Except our users experience the failed attempts (LLM replies that are wrong, even when backed by RAG) and it's incredibly hard to hide those from them. This has been my team's experience (and frustration) as well, and has led us to look at using LLMs for classifying / structuring, but not entrusting an LLM with making a decision based on things like a database schema or business logic. I think the technology and to…
> "let's only allow the LLM to do things we know it is rock-solid at." Even this is insanely hard in my opinion. The one thing that you would assume LLM to excel at is spelling and grammar checking for the English language, but even the top model (GPT-4o) can be insanely stupid/unpredictable at times. Take the following example from my tool: https://app.gitsense.com/?doc=6c9bada92&model=GPT-4o&samples... 5 models are…
I wouldn't expect an LLM to be good at spell checking, actually. The way they tokenize text before manipulating it makes them fairly bad at working with small sequences of letters.
I have had good luck using an LLM as a "sanity checking" layer for transcription output, though. A simple prompt like "is this paragraph coherent" has proven to be a pretty decent way to check the accuracy of whisper transcriptions.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#474Question for the group here: do we honestly feel like we've exhausted the options for delivering value on top of the current generation of LLMs? I lead a team exploring cutting edge LLM applications and end-user features. It's my intuition from experience that we have a LONG way to go. GPT-4o / Claude 3.5 are the go-to models for my team. Every combination of technical investment + LLMs yields a new list of potential…
We are just scratching the surface of what LLMs can do. Case in point, ESM3.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#475Question for the group here: do we honestly feel like we've exhausted the options for delivering value on top of the current generation of LLMs? I lead a team exploring cutting edge LLM applications and end-user features. It's my intuition from experience that we have a LONG way to go. GPT-4o / Claude 3.5 are the go-to models for my team. Every combination of technical investment + LLMs yields a new list of potential…
I have tried a few AI coding tools and always found them impressive but I don't really need something to autocomplete obvious code cases. Is there an AI tool that can ingest a codebase and locate code based on abstract questions? Like: "I need to invalidate customers who haven't logged in for a month" and it can locate things like relevant DB tables, controllers, services, etc.
I haven’t seen quite that, but it’s an interesting question; like a semantic search.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#476Question for the group here: do we honestly feel like we've exhausted the options for delivering value on top of the current generation of LLMs? I lead a team exploring cutting edge LLM applications and end-user features. It's my intuition from experience that we have a LONG way to go. GPT-4o / Claude 3.5 are the go-to models for my team. Every combination of technical investment + LLMs yields a new list of potential…
I have tried a few AI coding tools and always found them impressive but I don't really need something to autocomplete obvious code cases. Is there an AI tool that can ingest a codebase and locate code based on abstract questions? Like: "I need to invalidate customers who haven't logged in for a month" and it can locate things like relevant DB tables, controllers, services, etc.
Cursor (Claude behind the scenes) can do that, however as always, your mileage may vary.
I tried building a whole codebase inspector, essentially what you are referring to with Gemini's 2 million token context window but had troubles with their API when the payload got large. Just 500 error with no additional info so...
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#477Earlier quoted context omitted.
Whether self awareness is a requirement for AGI definitely gets more into the Philosophy department than the Computer Science department. I'm not sure everyone even agrees on what AGI is, but a common test is "can it do what humans can". For example, in this article it says it can't do coding exercises outside the training set. That would definitely be on the "AGI checklist". Basically doing anything that is outside…
I feel the test for AGI should be more like: "go find a job and earn money" or "start a profitable business" or "pick a bachelor degree and complete it", etc.
This is people's true desire. Make something like that while handling critisisms and fitting products to the market.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#478"We will have better and better models," wrote OpenAI CEO Sam Altman in a recent Reddit AMA. "But I think the thing that will feel like the next giant breakthrough will be agents." Is this certain? Are Agents the right direction to AGI?
It's marketing using buzz word rhetric. It's better to learn OOP if he trully think that. I also think OpenAI's PMF was to make the LLMs application towords better argument machine.
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#479Earlier quoted context omitted.
> "let's only allow the LLM to do things we know it is rock-solid at." Even this is insanely hard in my opinion. The one thing that you would assume LLM to excel at is spelling and grammar checking for the English language, but even the top model (GPT-4o) can be insanely stupid/unpredictable at times. Take the following example from my tool: https://app.gitsense.com/?doc=6c9bada92&model=GPT-4o&samples... 5 models are…
I wouldn't expect an LLM to be good at spell checking, actually. The way they tokenize text before manipulating it makes them fairly bad at working with small sequences of letters. I have had good luck using an LLM as a "sanity checking" layer for transcription output, though. A simple prompt like "is this paragraph coherent" has proven to be a pretty decent way to check the accuracy of whisper transcriptions.
Yes this is a tokenization error. If you rewrite the sentence as shown below:
https://app.gitsense.com/?doc=905f4a9af74c25f&model=Claude+3...
Claude 3.5 Sonnet will now misinterpret "GitHub as "Github"
Re: OpenAI, Google and Anthropic are struggling to build more advanced AI
#480Earlier quoted context omitted.
> why are so many experts so worried about AGI? FYI, I find this line of reasoning to be unconvincing both logically and by counter-example ("why are so many experts so worried about the Y2K bug?") Personally, I don't find AI foom or AI doom predictions to be probable but I do think there are more convincing arguments for your position than you're making here.
Fair enough, well put to both of these responses! I’m certainly biased, and can see how the events that truly scare me (after already assessing the technology on my own and finding it to be More Important Than Fire Or Electricity) don’t make very convincing arguments on their own. For us optimistic doomers, the AI conversation seems similar to the (early-2000s) climate change debate; we see a wave of dire warnings co…
I've written (and am writing) extensively why I think AGI cant be as bad as everyone thinks, from a first principles (i.e physics and math) standpoint:
https://chrisfrewin.medium.com/why-llms-will-never-be-agi-70...
Still have like 2-3 big posts to publish.
Long story short its easy to get enamored with an agent spitting out tokens out but reality and engineering are far far more complex than that (orders of magnitude)