[flagged]
OP's project aside, even a flawed solution is progress towards a good solution.
Yes, I could have been more positive. Though I worry about overpromising with technology, especially on this topic.
Show HN: I'm 16 and building an AI based startup called Factful with friends
61–70 of 163 posts
Re: Show HN: I'm 16 and building an AI based startup called Factful with friends
#62Andrew, you and your friends should be proud. It's really encouraging to see people, especially in your generation, thinking seriously about the problem of misinformation. There are fundamental challenges you all will face in this idea: - most current LLMs are trained on large amounts of web data that itself contains facts, opinions, and misinformation. These things are treated equally, so I would expect the LLM to g…
Yea, thank you for your feedback!
> “…which has the potential to do as much harm as good.”
I find this is one of the more difficult things for people to learn to fully integrate into their psyche. Many people never learn to truly care about this and everything it means. They go on forever primarily caring about what’s good for them personally.
Re: Show HN: I'm 16 and building an AI based startup called Factful with friends
#63Earlier quoted context omitted.
There's a whole art to prompting a LLM to say it's unsure. I need to write a blog post about this, it's deep.
Sample a bunch of LLMs with the same question, if they disagree much then they are unsure. You can even sample the same LLM with high enough temperature, text augmentations, different prompts or different demonstrations. When they are correct they say the same thing, but when they make mistakes, they make different ones. This only works for factual or reasoning tasks, but that's where it matters.
But how do you know if the LLMs agree, when all of them word the response differently
For example
LLM 1: Yes, it is true that fireworks were invented in China
LLM 2: Fireworks were indeed invented in China
Re: Show HN: I'm 16 and building an AI based startup called Factful with friends
#64Impressive! Really well done, Andrew.
One piece of feedback. I used the sample text and it gave me a sentence about a gold fish's memory and when I checked it, it had three errors. I had to update each error individually and over a longer sentence/thought/paragraph that might get cumbersome. It would be nice if there was a way to see a fully corrected sentence and point out the changes so I can do a one click change for a single sentence.
Re: Show HN: I'm 16 and building an AI based startup called Factful with friends
#65Earlier quoted context omitted.
Sample a bunch of LLMs with the same question, if they disagree much then they are unsure. You can even sample the same LLM with high enough temperature, text augmentations, different prompts or different demonstrations. When they are correct they say the same thing, but when they make mistakes, they make different ones. This only works for factual or reasoning tasks, but that's where it matters.
But how do you know if the LLMs agree, when all of them word the response differently For example LLM 1: Yes, it is true that fireworks were invented in China LLM 2: Fireworks were indeed invented in China
Ask another model if the two statements are in agreement of course! ;)
Re: Show HN: I'm 16 and building an AI based startup called Factful with friends
#66How do you evaluate “factuality” without knowing all the facts, though? That’s the downfall of all such services - eventually (or even immediately) they begin to just push their preferred agenda because it’s easier and more profitable. That said, at 16 you’re just learning, and literally whatever you accomplish will be a great achievement, so go down these paths and learn your lessons
Hi there, thank you for your feedback! I think we could potentially go down the route of a web3 approach where we get the public consensus on the facts.
But that doesn’t preclude lying by omission, which is a strategy employed by mass media in nearly every news article in $CURRENT_YEAR
Re: Show HN: I'm 16 and building an AI based startup called Factful with friends
#67Impressive! Really well done, Andrew. One piece of feedback. I used the sample text and it gave me a sentence about a gold fish's memory and when I checked it, it had three errors. I had to update each error individually and over a longer sentence/thought/paragraph that might get cumbersome. It would be nice if there was a way to see a fully corrected sentence and point out the changes so I can do a one click change…
Yea, already done! It will be pushed tomorrow, when less people is on.
Re: Show HN: I'm 16 and building an AI based startup called Factful with friends
#68are you aware of http://disinformationindex.org?
Re: Show HN: I'm 16 and building an AI based startup called Factful with friends
#69Is there more where we can read about how it works? I tried testing it with the sentence “Toledo is the largest city in Ohio” and it’s suggestions were to replace this with paragraphs of text about what constitutes a city in Ohio and history about Ohio, which did happen to include that Columbus is currently the largest city in Ohio. That doesn’t seem strictly helpful as a fact checker as at first glance it isn’t even…
Yea we are working on it. It is a thing that we are trying to fix, it giving too much or too little suggestions.
Re: Show HN: I'm 16 and building an AI based startup called Factful with friends
#70> It evaluates context, factuality, coherence, and more to ensure the accuracy and credibility of content. Based on what it does the evaluations? Is there some set collection & hierarchy of sources and if so will I be aware of it after it is done?
yes, we are working on it. But currently you can check each source by pressing the little i on the popup