141–150 of 156 posts
Earlier quoted context omitted.
same here. I first built the agentic workflow in python and later nextjs. uses dozens of llm apis. works well, and am also impressed with the results.
Curious what you mean by dozens of llm apis. Different models? Or different tool calls?
the whole contraption uses ~10 different models, but more can easily be plugged into the initial generation phase. happy to demo it sometime! [edit: email on profile].
Earlier quoted context omitted.
"Don’t fall into the trap of anthropomorphizing LLMs and assuming that failures which would discredit a human should discredit the machine in the same way." - https://simonwillison.net/2025/Mar/11/using-llms-for-code/#s...
What I got out of that essay is that you should discredit most responses of LLMs unless you want to do just as much or more work yourself confirming the accuracy of an unreliable and deeply flawed partner. Whereas if a human "hallucinated a non-existent library or method you would instantly lose trust in them." But, for reasons, we should either give the machine the benefit of the doubt or manually confirm everything…
> If your reaction to this is “surely typing out the code is faster than typing out an English instruction of it”, all I can tell you is that it really isn’t for me any more. Code needs to be correct. English has enormous room for shortcuts, and vagaries, and typos, and saying things like “use that popular HTTP library” if you can’t remember the name off the top of your head.
Using LLMs as part of my coding work speeds me up by a significant amount.
I feel like the bigger problem isn't whether these deep research products work, but rather the raw material, so to speak, that they're working with. For example, a lot of the "sources" cited in Google's AI Overview (notably not a deep research product) are not official, just sites that probably rank high in SEO. I want the original source, or a reliable source, not joeswebsite dot com (no offense to this website if i…
That'd what makes o3/o4-mini driven search notable to me: those models appear to have much better taste in which searches to run and which sources to consider.
Earlier quoted context omitted.
this remains true for pretty much all advice or information we receive. doctors, lawyers, accountants, teachers. there have been countless times that all of these professionals have given me bad advice or information sure, at least I have someone to blame in that case. but in my experience, the AI is at least as reliable as a person who I don't personally know
Plenty studies show that these models are better at catching and diagnosing than even a board of doctors are. Doctors are good at other things, and I hope the future will allow doctors to use these models together with their practice. The problem is when the ai makes a catastrophic prediction, and the layman can't see it.
I don't see how it is really different with AI
Earlier quoted context omitted.
Your logic is.... strange... Because it failed miserably at a very simple task of looking through some scattered charts, the human asking should blame themselves for this basic failure and trust it to do better with much harder and more specialized tasks?
I think you might as well be saying "robotics fail miserably at the very simple task of jogging around the block, so why should we trust the field to be able to accurately place millions of transistors within a 25cm square of silicon?" His point is that the two tasks are very different at their core, and deep research is better at teasing out an accurate "fuzzy" answer from a swamp of interrelated data, and a data sc…
I understand that there are fuzzy tasks that AIs/algorithms are terrible at, which seem really simple for a human mind, and this hasn't gone away with the latest generations of LLMs. That's fine and I wouldn't criticize an AI for failing at something like the instructions you describe, for example.
However in this case, the human was asking for very specific, cut and dry information from easily available NFL rosters. Again, if an AI fails at that, especially because you didn't phrase the question "just so", then sorry, but no, it's not much more trustworthy for deep research and data scientist inquiries.
What in any case makes you think the data scientists will use superior phrasing to tease better results under more complexity from an LLM?
Earlier quoted context omitted.
If you keep the prompt the same at some point the data will appear in training set and we might have answer. So even though today it might be a good check it might not remain as such a good benchmark. I think we need a way to keep updating prompts without increasing complexity in someway to properly verify model improvements. ARC Deep Research anyone?
Well, to test research capabilities, one could just adopt the year (2024->2025) in the prompt.
There are some NBA fan sites that do keep track of some of these tournament level final metrics.
Earlier quoted context omitted.
If you keep the prompt the same at some point the data will appear in training set and we might have answer. So even though today it might be a good check it might not remain as such a good benchmark. I think we need a way to keep updating prompts without increasing complexity in someway to properly verify model improvements. ARC Deep Research anyone?
Wouldn't somebody need to answer the question below? Or do you mean the discussion of its weakness might somehow make it stronger the next time it's trained?
I am not sure if the model will need the exact answer or just the backlinks to site where they can find them is enough. Maybe just documenting how to do it could do the job as well...