Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Bullshit Detector – agent skills that fact-check videos and articles

github.com

Post reply on HN
Print view View original

31–40 of 80 posts

Re: Show HN: Bullshit Detector – agent skills that fact-check videos and articles

#31

Post by bulbar »

westoque wrote: post #28
simonw wrote: post #21

Earlier quoted context omitted.

Something I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like ruling out random blogs, considering media publications with a good reputation for fact checking, and double-checking information that seems unlikely.

I don't think random blogs should be ruled out. Just because a blog is random doesn't mean its untrue.

It should not be seen a credible source though.

It can be, but that's hard to correctly judge on at scale.

Re: Show HN: Bullshit Detector – agent skills that fact-check videos and articles

#32

Post by autonomousErwin »

mysterydip wrote: post #7

What do you use as “ground truth”? The page says “independent sources”, and I’m sure there’s too many to list, but my question is how are they vetted as being truthful and how are two sources with opposite viewpoints reconciled?

I could imagine running multiple different AIs seeing if they agree if something is true.

I imagine this is similar to how Twitter's/X's community notes work. Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true.

So from that approach, you actually want to have two AIs that are on the other side of the spectrum of whatever you're trying to find out if it's true e.g. if a conservative AI and a liberal AI both agree that something is false, it's highly likely to be false.

Re: Show HN: Bullshit Detector – agent skills that fact-check videos and articles

#33

Post by skorniienko »

gutechh wrote: post #25
skorniienko wrote: post #14

Earlier quoted context omitted.

There is no whitelist source. It can't rate sources for truthfulness. When sources conflicts - skill drops verdict to misleading or unverifiable and both are linked. Can't pick a winner at the moment. That's probably the weakest part and still a judgement call for a model.

Maybe you read them already, but wikipedia has a bunch of interesting pages about sourcing and truth like: https://en.wikipedia.org/wiki/Wikipedia:Verifiability,_not_t... and https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources

I'll look at reliable-sources, and you are right "verifiability, not truth" is basically what I landed on too. Model doesn't get to decide what's true, it just needs to cite something.

Will read reliable-sources page properly.

Re: Show HN: Bullshit Detector – agent skills that fact-check videos and articles

#36

Post by mtweak »

autonomousErwin wrote: post #32
mysterydip wrote: post #7

What do you use as “ground truth”? The page says “independent sources”, and I’m sure there’s too many to list, but my question is how are they vetted as being truthful and how are two sources with opposite viewpoints reconciled?

I could imagine running multiple different AIs seeing if they agree if something is true. I imagine this is similar to how Twitter's/X's community notes work. Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true. So from that approach, you actually want to have two AIs that are on the other side of the spectrum of w…

This is extremely problematic. Much of news is highly syndicated, so what looks like 10 credible sources are actually just 2. I think the best you can aim for is empirical sources: receipts, videos, photo evidence, public disclosures directly at the source. Archiving is ok if disclosed.

I don't think LLMs can discern truth when the majority of news sources are incentivized primarily for views.

Re: Show HN: Bullshit Detector – agent skills that fact-check videos and articles

#37

Post by skorniienko »

lynx97 wrote: post #24

> Long output? Redirect to a file and read it from there: Hmm, smells AI generated. Why should an LLM (this is from a SKILL.md) care about $LINES?

Done, fixed in v0.4.1. Reason why agent reads from a file - it can do it in chunks or pass the path to subagent and keep it out from own context. I've tested it on 3hrs long interview YT video - worked smoothly.

Re: Show HN: Bullshit Detector – agent skills that fact-check videos and articles

#38

Post by skorniienko »

o10449366 wrote: post #35

god i wish these low effort projects were banned from Show HN the entire premise of a "bullshit detector" that's entirely vibe-coded is laughable

why it can't be vibe-coded? Would you be more pleased if it would be poorly hand-coded and claimed "No AI used during development"? What's the point?

I don't mind that I've used AI to build this, and I'm open about it ;)

Re: Show HN: Bullshit Detector – agent skills that fact-check videos and articles

#39

Post by BugsJustFindMe »

ck2 wrote: post #16

Washington Post gave up tracking Trump lies last term in 2021 because it became impossible by human hands with 21+ per day and over 30,000 in their database but with "AI" now it's possible not only to do non-stop but in REALTIME you could even just restrict the source of the check to the paper's own reporting the past fifty years * https://www.washingtonpost.com/graphics/politics/trump-claim... I'd like to see that b…

I'm not sure AI helps for that particular task. If you just assume that 100% of everything he says is a lie you'll be 99% correct. Anyone who believes anything he says at this point is living in a constructed reality where facts never matter. Fact checking seems more useful for less pathological cases where it might make a difference.

Re: Show HN: Bullshit Detector – agent skills that fact-check videos and articles

#40

Post by skorniienko »

egeozcan wrote: post #23

Can someone please use this skill against the claims in its own repo?

Done, fixed 2 misleading lines in README. You can see example report in repo examples.

Thanks for that, btw! At least someone questioned it :)

Post reply on HN