Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic Claude 3.5 can create icalendar files, so I did this

gregsramblings.com

Post reply on HN
Print view View original

51–60 of 174 posts

Re: Anthropic Claude 3.5 can create icalendar files, so I did this

#51

Post by dbaupp »

bonzini wrote: post #40
sorokod wrote: post #33

Earlier quoted context omitted.

Isn't asking an LLM "to write a Python validator" suffers from the 99.9% (or whatever the error rate for validators written by Claude) problem?

The difference is that you're asking it to perform one intellectual task (write a program) instead of 100 menial tasks (parse a file). To the LLM the two are the same level of complexity, so performing less work means less possibility of error. Also, the LLM is more likely to fail spectacularly by hallucinating APIs when writing a script, and more likely to fail subtly on parsing tasks.

In addition to what you say, it can also be easier for a (appropriately-skilled) human to verify a small program than to verify voluminous parsing output, plus, as you say, there's the semi-automated "verification" of a very-wrong program failing to execute.

Re: Anthropic Claude 3.5 can create icalendar files, so I did this

#54

Post by ks2048 »

disillusioned wrote: post #25
wenc wrote: post #3

You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. But LLMs can get t…

The "off by one" predilection of LLMs is going to lead to this massive erosion of trust in whatever "Truth" is supposed to be, and it's terrifying and going to make for a bumpy couple of years. (Or the complete collapse of objective knowledge, on a long enough time horizon.) It's one thing to ask an LLM when George Washington was born, and have it return "May 20, 2020." It's another thing to ask it, and have it matte…

Off topic, but a funny thing about asking about George Washington's birthday is there are two possible answers because of British calendar reform in 1750 (although we've settled on recognizing the new-style date as his birthday).

footnote [a] on wikipedia: https://en.wikipedia.org/wiki/George_Washington#cite_note-3

Re: Anthropic Claude 3.5 can create icalendar files, so I did this

#56

Post by stavros »

buro9 wrote: post #7

I'm finding Claude to hallucinate less than ChatGPT, and to be far more accurate at coding than CoPilot. Pleasantly surprised on both counts. Example hallucinations from ChatGPT include researching the dates of historical events for the company I work at, trivially verifiable by me but I was being lazy... ChatGPT told me about blog posts that never existed and I could prove never existed, Claude was spot on with date…

I made a mobile app using an LLM (I'd never written React Native before) and GPT-4 was constantly making mistakes after the code was more than a hundred lines or so. GPT-4o was even worse.

Claude, in comparison, dealt with everything much more competently, and also followed my instructions much better. I use Claude nowadays, especially for coding.

Re: Anthropic Claude 3.5 can create icalendar files, so I did this

#57

Post by jrflowers »

stavros wrote: post #43
disillusioned wrote: post #25

Earlier quoted context omitted.

The "off by one" predilection of LLMs is going to lead to this massive erosion of trust in whatever "Truth" is supposed to be, and it's terrifying and going to make for a bumpy couple of years. (Or the complete collapse of objective knowledge, on a long enough time horizon.) It's one thing to ask an LLM when George Washington was born, and have it return "May 20, 2020." It's another thing to ask it, and have it matte…

This is semi-offtopic, but "trust but verify" is an oxymoron. Trusting something means I don't have to verify whether it's correct (I trust that it is), so the saying, in the end, is "don't verify but verify".

Yep.

https://en.m.wikipedia.org/wiki/Trust,_but_verify

Re: Anthropic Claude 3.5 can create icalendar files, so I did this

#58

Post by jessekv »

sorokod wrote: post #33
wenc wrote: post #3

You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. But LLMs can get t…

Isn't asking an LLM "to write a Python validator" suffers from the 99.9% (or whatever the error rate for validators written by Claude) problem?

All tests have this problem. We still write them for the same reasons we do double-entry bookkeeping.

Re: Anthropic Claude 3.5 can create icalendar files, so I did this

#59

Post by dan-robertson »

wenc wrote: post #3

You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. But LLMs can get t…

If it’s correct 99.9% of the time, and the piano lessons are every two weeks, that’s one error in piano lesson scheduling over 40 years. That sounds good enough to me to not verify.

Re: Anthropic Claude 3.5 can create icalendar files, so I did this

#60

Post by MattGaiser »

wenc wrote: post #3

You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. But LLMs can get t…

Have two LLMs do the task and compare.
Post reply on HN