Show HN: I'm 16 and building an AI based startup called Factful with friends

Post by helloduck1234 » Wed, May 01, 2024, 8:43 PM UTC

itishappy wrote: post #59

I'm having a lot of fun with this! Prompt: > Factful.io uses a hand-coded natural language parser to determine factual accuracy to within a tolerance of 12.3 standard units of factfulness. No revisions, it must already be perfect! Prompt: > The only remaining bias in Factful.io is that which is legally required by the USDA. No revisions, it must already be perfect! Prompt: > Factful.io checks everything you enter aga…

Post by omneity » Wed, May 01, 2024, 8:59 PM UTC

codetrotter wrote: post #63

visarga wrote: post #48

Earlier quoted context omitted.

Sample a bunch of LLMs with the same question, if they disagree much then they are unsure. You can even sample the same LLM with high enough temperature, text augmentations, different prompts or different demonstrations. When they are correct they say the same thing, but when they make mistakes, they make different ones. This only works for factual or reasoning tasks, but that's where it matters.

But how do you know if the LLMs agree, when all of them word the response differently For example LLM 1: Yes, it is true that fireworks were invented in China LLM 2: Fireworks were indeed invented in China

Post by codetrotter » Wed, May 01, 2024, 9:08 PM UTC

omneity wrote: post #73

codetrotter wrote: post #63

Earlier quoted context omitted.

But how do you know if the LLMs agree, when all of them word the response differently For example LLM 1: Yes, it is true that fireworks were invented in China LLM 2: Fireworks were indeed invented in China

This is trivially achievable with function calling, assuming the model you use supports this (which most models do at this point). Define a function `reportFactual(isFactual: boolean)` and you will get standardized, machine-readable answers to do statistics with.

Post by echelon » Wed, May 01, 2024, 9:09 PM UTC

helloduck1234 wrote: post #31

dweinus wrote: post #17

Andrew, you and your friends should be proud. It's really encouraging to see people, especially in your generation, thinking seriously about the problem of misinformation. There are fundamental challenges you all will face in this idea: - most current LLMs are trained on large amounts of web data that itself contains facts, opinions, and misinformation. These things are treated equally, so I would expect the LLM to g…

Yea, thank you for your feedback!

Post by lazyasciiart » Wed, May 01, 2024, 9:15 PM UTC

helloduck1234 wrote: post #5

zero-sharp wrote: post #3

[flagged]

Damn, what the MVP does right now, is check for already established facts. So basically, what we define as fact is something that has proof. For example, we stray away from political views as those are opinions, and stay with “facts”. As for disputed facts, we would also stay away from them unless they have a significantly larger backing than the other. The source currently is LLMs that are trained on huge amounts of…