Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Check your fucking sources, people

brodzinski.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Check your fucking sources, people

#51

Post by larkost »

tinfoilhatter wrote: post #25
ubertaco wrote: post #17

Earlier quoted context omitted.

Snopes (like anywhere) is only as reliable as its track record of collecting firsthand sources and accurately reporting on their contents. Which is to say: pretty good so far, in their case. For the future? Who knows. But they've done well up to now, at least.

Actually no, their track record is not great: https://en.wikipedia.org/wiki/Snopes#2010s

Is this a meta-joke about links that don't prove what they are supposed to? If so, I don't really get it.

Most of the 2010s section is about some drama about managers/hosters. The only thing that is even remotely applicable is they fact checking a satirical website, and needing to add a "Labeled Satire" tag to clear up confusion around the intensions of the linked site (as opposed to combating people who use the article as an argument without labeling it as such).

Re: Check your fucking sources, people

#52

Post by Kim_Bruning »

kakugawa wrote: post #19
wing-_-nuts wrote: post #2

Ironically, 'source checking' is something AI is quite good at.

Have we forgotten how bad LLMs were at citing sources when they first came out? So, we had to build a lot of structure (harness engineering) and frontier labs had to do specific training to try to compensate for this. So, LLMs are inherently bad at citing sources. A lot of effort has been put in to improve this behavior, but it's compensating for an inherent flaw.

Huh? Oh! Were they still treating the LLM as an "oracle box"/online chatbot at the time? (as opposed to a more agentic workflow?)

If they weren't, ignore I said the following, and please tell me what else was going wrong (and with what models and harnesses!).

Models weights are like Wikipedia. A nice starting point, but should never be referenced directly. You need to have your agent actually go out onto the internet and do the research. Now the actual references will be in your agent's actual Context (memory), so then it'd at least be rather more surprising if they don't cite correctly.

I do realize there's still corner cases even in the best setups though; So a final crosscheck sweep is never not a good idea.

Re: Check your fucking sources, people

#53

Post by tinfoilhatter »

larkost wrote: post #51
tinfoilhatter wrote: post #25

Earlier quoted context omitted.

Actually no, their track record is not great: https://en.wikipedia.org/wiki/Snopes#2010s

Is this a meta-joke about links that don't prove what they are supposed to? If so, I don't really get it. Most of the 2010s section is about some drama about managers/hosters. The only thing that is even remotely applicable is they fact checking a satirical website, and needing to add a "Labeled Satire" tag to clear up confusion around the intensions of the linked site (as opposed to combating people who use the arti…

https://www.dailymail.com/news/article-4730092/Snopes-brink-...

It was far more than drama about managers / hosters.

Re: Check your fucking sources, people

#54

Post by AlienRobot »

The way it works, has always worked, and should always work, is that you read some information on article X, and you just quote it as verbatim as linguistically possible. Nobody, literally nobody, reading your articles wants you to paraphrase it. If you don't quote things verbatim, you are doing it wrong. If you don't have sources, you are making it up. Your job is essentially just putting N+1 quotes from other articles into your article, and nobody wants you to do more or less than this.

People just have a fundamentally misguided idea of what they should be doing. You just quote. That is all. Nobody needs your originality as a writer. They just want the quotes, the sources, and, optionally, a synthesis, conclusion, and summary. That is the "work" you need to do and if you do just this, that is enough value already even if it feels like plagiarism by people who don't know what the word plagiarism means.

Everyone knows information came from somewhere. Where did you read it? We all know you didn't just wake up one day and remembered it from a past life or something like that. Why are you trying to pretend you just know it? Where are the links? Where are the screenshots?

If you are giving people the sources that you should have been giving all along, in the correct way, then you don't need to "check your sources." Because "your sources" are literally where you got that information from in first place, so you have already checked them.

Hence, if ChatGPT gave you a source, then your source isn't the source that ChatGPT gave you, because you didn't read it. Your source is, literally, ChatGPT. You should be writing "Smith, 2015, as cited by ChatGPT." Because you didn't read Smith, 2015. You read ChatGPT!

Re: Check your fucking sources, people

#55

Post by teddyh »

justin66 wrote: post #11
6stringmerc wrote: post #3

Also relevant: the derision and mockery directed at JD Vance as a “couch fucker” even used by John Oliver. I read “Hillbilly Elegy” and wondered why it wasn’t in there. Snopes cleared it up in a matter of minutes. Why he hasn’t sued people into oblivion is his prerogative, but it’s a fascinating case study that we are, indeed, living in a Post-Truth environment.

Did anyone actually believe that was anything more than a joke? It was a disgusting and weird thing to suggest about a disgusting and weird guy, and highly immature, but it's only libel if it's presented as being true.

In only slightly more isolated regions of the internet than here, it used to be, when this story was new, that whenever this story was brought up, with dozens or more credulous comments with varying attempts at humor, there was at least one person who commented that it was fake, often with citations. As time went on, for each new instance, the number of comments dwindled, and the lone debunking comment appeared more and more rarely. Now I never see it, even though the story still occasionally appears, with a handful of credulous comments. So I would assume that yes, many people believe it.

Re: Check your fucking sources, people

#56

Post by Kim_Bruning »

louiereederson wrote: post #13

Late last year I tried asking ChatGPT to summarize a collection of 10 researchers' views/findings on a topic and provide representative quotes. It initially looked plausible but when I checked the links, the quotes were from clearly AI generated summaries of actual interviews. The paraphrasing was also plausible but subtly and profoundly incorrect. I haven't tested this again on the latest models though, so not sure…

ChatGPT is horrible overall, for sure, but how exactly did you ask it to summarize, and what model was it exactly?

(I'm not saying "you're holding it wrong", I'm asking "how were you holding it"?

Did you tell it to pull in the sources, did it do so automatically, or were you working from just the base weights? )

Re: Check your fucking sources, people

#57

Post by wat10000 »

tinfoilhatter wrote: post #46
wat10000 wrote: post #35

Earlier quoted context omitted.

What lies exactly?

https://www.dailymail.com/news/article-4730092/Snopes-brink-... https://foodbabe.com/do-you-trust-snopes-you-wont-after-read... There's a plethora of examples on the internet of Snopes engaging in this type of behavior, if you're actually interested in learning about their problematic approach to their work.

Your links are mostly about ownership, supposed sources of bias, and internal politics. I want to know what they've actually gotten wrong. The only example I'm seeing there is a claim about glyphosate, and the article is by the person Snopes said got it wrong, definitely no axe to grind there.

That example seems extremely weak. Is that all there is?

Re: Check your fucking sources, people

#58

Post by ubertaco »

tinfoilhatter wrote: post #53
larkost wrote: post #51

Earlier quoted context omitted.

Is this a meta-joke about links that don't prove what they are supposed to? If so, I don't really get it. Most of the 2010s section is about some drama about managers/hosters. The only thing that is even remotely applicable is they fact checking a satirical website, and needing to add a "Labeled Satire" tag to clear up confusion around the intensions of the linked site (as opposed to combating people who use the arti…

https://www.dailymail.com/news/article-4730092/Snopes-brink-... It was far more than drama about managers / hosters.

Okay, so you're linking a known tabloid whose sensationalized headline is still not about the actual content of the Snopes website being inaccurate, and instead is leaning hard on ad-hominems derived from legal drama between the then-divorcing husband-and-wife team who ran the website originally (with the husband continuing to run it after the wife had stopped working on it years ago).

The only thing in the Wikipedia section you linked that's actually about the content on the Snopes website is the thing where they had to create a label for "Satire" after people got mad that a right-wing satire site (literal, actual, intentional "fake news" but for comedy purposes) had its knowingly-false stories labeled as "false".

(don't come at me with "it was bias"; I lived in the right-wing evangelical bubble through my whole childhood and young adulthood all the way through to the early 2010s; I know the boy-who-cried-persecution complex that lives there, and I also know what the Babylon Bee both was and is quite well; they were never trying to be a real news source, so getting mad that their comedic fiction was labeled "false" is really a stretch).

You haven't exactly shaken my faith in their ability to do the thing they do: find primary sources, present them, and give a verdict based on those primary sources.

Re: Check your fucking sources, people

#59

Post by flail »

andai wrote: post #42

When Bard (now known as Gemini) first came out in Europe, I think mid 2023, I tested it out. AI search was still a new thing in those days, and I was excited to see what Google's solution would be like. I had high hopes. I asked it a question I knew the answer to. It searched the web, and told me the opposite of the truth. (Not nonsense, but a logical inversion of the actual fact. A common failure mode with earlier L…

Fundamentally, yes, it is a different "search engine."

BTW, as critical as I can be to AI, using an argument that something didn't work 3 years ago, so it must be crap, doesn't work in this context. 3 years ago, AI could barely generate several lines of consistent code. Now, it generates working apps with a prompt (it's another discussion how good the code is, but still).

I guess 3 years ago, Gemini couldn't tell how many r's are in the word refrigerator.

Same for research. At some point, I switched from ChatGPT and Gemini to Perplexity as it promised AI-powered search. It worked visibly better. Until it didn't, as GPT and Gemini models made a leap.

Back to the point, as long as we understand that, for now, it's all just a probabilistic machine generating the most likely output, no one should expect bulletproof answers. Search was/is way more deterministic than LLMs.

Post reply on HN