Anyone tried this? https://journaliststudio.google.com/pinpoint/about
Loki: An open-source tool for fact verification
11–20 of 74 posts
Re: Loki: An open-source tool for fact verification
#12My friend’s startup: https://factiverse.ai/
Re: Loki: An open-source tool for fact verification
#13So only thing they open sourced is the prompts [1] and code to call LLM APIs? There are plenty of such libraries out there. And the prompts seem to be copied from here [2]? [1] https://github.com/Libr-AI/OpenFactVerification/blob/main/fa... [2] https://github.com/yuxiaw/Factcheck-GPT/blob/main/src/utils/...
regarding your last concern, I found that yuxiaw is their COO[1], so it can't be considered a copy?
Re: Loki: An open-source tool for fact verification
#14So only thing they open sourced is the prompts [1] and code to call LLM APIs? There are plenty of such libraries out there. And the prompts seem to be copied from here [2]? [1] https://github.com/Libr-AI/OpenFactVerification/blob/main/fa... [2] https://github.com/yuxiaw/Factcheck-GPT/blob/main/src/utils/...
regarding your last concern, I found that yuxiaw is their COO[1], so it can't be considered a copy? [1] https://www.librai.tech/team
Ok but bigger issue is there is evidence that the LLMs are not better than specialized models for fact-checking. https://arxiv.org/abs/2402.12147
Re: Loki: An open-source tool for fact verification
#15It’s a bit misleading to call it open source tool when it relies on proprietary LLMs for everything.
Presumably the LLMs are swappable -- today the proprietary ones are very powerful and accessible, but the landscape may yet change.
Well but they don’t mention that it is clickbait to call it open source fact checking tool which needs LLMs to do everything. Also code is not designed to easily swap with a free locally running LLM.
Re: Loki: An open-source tool for fact verification
#16You might want to look into integrating DebateSum or OpenDebateEvidence (OpenCaseList) into this tool as sources of evidence. They are uniquely good for these sorts of tasks:
Re: Loki: An open-source tool for fact verification
#17Overall great idea though, I'll be definitely checking it back in the future. A few things that hit me out of the box:
* The idea behind using Serper is great, however it would be cool if other search engines/data sources can be used instead, ie. Kagi or some private search engine/data. Reason for the latter: there are tons of people who are sourcing all sorts of information which will not immediately show up on google and some might never do. For context: I have roughly 60GB (and growing) of cleaned news article with where I got them from and with a good amount of pre-processing done on the fly(I collect those all the time).
* Relying heavily on OpenAI. Yes, OpenAI is great but there's always the thing at the back of our minds that is "where are all those queries going and do we trust that shit won't hit the fan some day". It would be nice to have the ability to use a local LLM, given how many and how good there are around.
* The installation can be improved massively: setuptools + entry_points + console_scripts to avoid all the hassle behind having to manage dependencies, where your scripts are located and all that. The cp factcheck/config/secret_dict.template factcheck/config/secret_dict.py is a bit.... Uuuugh... pydantic[dotenv] + .env? That would also make the containerizing the application so much easier.
Re: Loki: An open-source tool for fact verification
#18Feedback on the example gif: at the moment it's almost comically useless. First you're bored watching the beginning 90% while commands are slowly being typed, and then the bit that's actually interesting and worth reading scrolls too fast and then resets to the beginning of the gif before there's a chance to read it.
Re: Loki: An open-source tool for fact verification
#19Interesting. In the Nordics, we have a couple of sites dedicated to fact checking news stories, done by real people. I think these kinds of automated tools can be helpful too, but needs to be tied to reliable sources. This became pretty apparent to me with the tech news coverage of xz, too. Lots of accidental (or sometimes intentional?) misinformation being spread in news articles. I wrote about it a bit[0], it was pretty sad to see big international publishers publishing an article based entirely on the journalist's misunderstandings of the situation. Facts and truth is important, especially as we see gen AI furthering the amount of legitimate looking content online that might not actually be true.
[0] - https://open.substack.com/pub/thetechenabler/p/trust-in-brea...
Re: Loki: An open-source tool for fact verification
#20> This tool is especially useful for journalists, researchers, and anyone interested in the factuality of information. Sorry, I think an individual who is not only aware of reliable sources to verify information, and who is not familiar enough with LLMs to come up with appropriate prompts and judge output should be the last person presenting themselves as the judger of factual information.
Thanks for your response. When discussing fact-checking capabilities, the key question is always: Can we guarantee that it will always offer the correct justification? While it's unfortunate, errors can occur. Nonetheless, we prioritize making the checking process both interpretable and transparent, allowing users to understand and trust the rationale behind each assessment.
We present the results at each step to help users understand the decision process, which can be seen from our screenshot at https://raw.githubusercontent.com/Libr-AI/OpenFactVerificati...
We will try our best to ensure this tool makes a positive difference