21–30 of 44 posts
[1] https://huggingface.co/spaces/vectara/leaderboard [2] https://github.com/vectara/hallucination-leaderboard/tree/ma...
From the paper abstract, > (4) we derive the optimal chain-of-thought length as [..math..] with explicit constants I know we probably have to dive into math and abandon metaphor and analogy, but the whole structure of a claim like this just strikes me as bizarre. Chain-of-thought always makes me think of that old joke. Alexander the great was a great general. Great generals are forewarned. Forewarned is forearmed. Fo…
Yesterday I used ChatGPT to transform a csv file. Move around a couple of columns, add a few new ones. Very large file. It got them all right. Except when I really looked through the data, for 3 of the excel cells, it clearly just made up new numbers. I found the first one by accident, the remaining two took longer than it would have taken to modify the file from scratch myself. Watching my coworkers blindly trust ou…
My take-away re: chain-of-thought specifically is this. If the answer to "LLMs can't reason" is "use more LLMs", and then the answer to problems with that is to run the same process in parallel N times and vote/retry/etc, it just feels like a scam aimed at burning through more tokens.
Hopefully chain-of-code[2] is better in that it's at least trying to force LLMs into emulating a more deterministic abstract machine instead of rolling dice. Trying to eliminate things like code, formal representations, and explicit world-models in favor of implicit representations and inscrutable oracles might be good business but it's bad engineering
[0] https://en.wikipedia.org/wiki/Datasaurus_dozen [1] https://towardsdatascience.com/how-metrics-and-llms-can-tric... [2] https://icml.cc/media/icml-2024/Slides/32784.pdf
From the paper abstract, > (4) we derive the optimal chain-of-thought length as [..math..] with explicit constants I know we probably have to dive into math and abandon metaphor and analogy, but the whole structure of a claim like this just strikes me as bizarre. Chain-of-thought always makes me think of that old joke. Alexander the great was a great general. Great generals are forewarned. Forewarned is forearmed. Fo…
Yesterday I used ChatGPT to transform a csv file. Move around a couple of columns, add a few new ones. Very large file. It got them all right. Except when I really looked through the data, for 3 of the excel cells, it clearly just made up new numbers. I found the first one by accident, the remaining two took longer than it would have taken to modify the file from scratch myself. Watching my coworkers blindly trust ou…
I expect future models will be able to identify when a computational tool will work, and use it directly
This topic is interesting, but the repo and paper have a lot of inconsistencies that make me think this work is hiding behind lots of dense notation and language. For one, the repo states: > This implementation follows the framework from the paper “Compression Failure in LLMs: Bayesian in Expectation, Not in Realization” (NeurIPS 2024 preprint) and related EDFL/ISR/B2T methodology. There doesn't seem to be a paper by…
This seems less accurate than `return 1.0` Using the unboundedly unreliable systems to evaluate reliability is just a bad premise.
From the paper abstract, > (4) we derive the optimal chain-of-thought length as [..math..] with explicit constants I know we probably have to dive into math and abandon metaphor and analogy, but the whole structure of a claim like this just strikes me as bizarre. Chain-of-thought always makes me think of that old joke. Alexander the great was a great general. Great generals are forewarned. Forewarned is forearmed. Fo…
Yesterday I used ChatGPT to transform a csv file. Move around a couple of columns, add a few new ones. Very large file. It got them all right. Except when I really looked through the data, for 3 of the excel cells, it clearly just made up new numbers. I found the first one by accident, the remaining two took longer than it would have taken to modify the file from scratch myself. Watching my coworkers blindly trust ou…
I'm struggling with trying to understand how using an LLM to do this seemed like a good idea in the first place.
From the paper abstract, > (4) we derive the optimal chain-of-thought length as [..math..] with explicit constants I know we probably have to dive into math and abandon metaphor and analogy, but the whole structure of a claim like this just strikes me as bizarre. Chain-of-thought always makes me think of that old joke. Alexander the great was a great general. Great generals are forewarned. Forewarned is forearmed. Fo…
Yesterday I used ChatGPT to transform a csv file. Move around a couple of columns, add a few new ones. Very large file. It got them all right. Except when I really looked through the data, for 3 of the excel cells, it clearly just made up new numbers. I found the first one by accident, the remaining two took longer than it would have taken to modify the file from scratch myself. Watching my coworkers blindly trust ou…
It had a small suggestion for the last sentence and repeated the whole corrected version for me to copy and paste.
Only last sentence slightly modified - or so I thought because it had moved the date of the event in the first sentence by one day.
Luckily I caught it before posting, but it was a close call.
Earlier quoted context omitted.
Yesterday I used ChatGPT to transform a csv file. Move around a couple of columns, add a few new ones. Very large file. It got them all right. Except when I really looked through the data, for 3 of the excel cells, it clearly just made up new numbers. I found the first one by accident, the remaining two took longer than it would have taken to modify the file from scratch myself. Watching my coworkers blindly trust ou…
> Yesterday I used ChatGPT to transform a csv file. Move around a couple of columns, add a few new ones. Very large file. I'm struggling with trying to understand how using an LLM to do this seemed like a good idea in the first place.
Of course this is the risk, not the proof. High risk answers can be correct, low ones can still be partly hallucinated. And then there is the factor of shit-in-shit-out training data. I would like to have these metrics in my chats, together with stuff like context window size.
I just want a badge that says "ai-generated" for content thats likely ai slop on LinkedIn, Reddit, X etc.