Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikimedia Enterprise – APIs for LLMs, AI Training, and More

enterprise.wikimedia.com

Post reply on HN
Print view View original

31–40 of 166 posts

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#31

Post by Tao3300 »

tux3 wrote: post #17
s1k3s wrote: post #5

> Real-time access to Knowledge Wikipedia knowledge is basically "democracy" knowledge, i.e. the more people decided to support an idea, the "truer" it gets. That's not knowledge at all!

That's exactly what almost all knowledge is. When was it that you last verified something by yourself, with an experiment? You didn't test the things you know. You know things because you could see they were the consensus, and so you had no reason to challenge them. If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unre…

We trust the consensus of published, peer-reviewed experts. That's different than the kind of Demos that does things like declare war on Persia, kill Jesus and Socrates, or edit Wikipedia.

> If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unreliable people. So with the Wiki.

Right. That makes Wiki kind of unreliable. Not completely. And not to the point of uselessness, but you should trust it about as far as you can throw it.

> Human knowledge isn't from the platonic realm.

Citation needed ;)

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#32

Post by tux3 »

s1k3s wrote: post #22
simonw wrote: post #18

Earlier quoted context omitted.

Read up on Wikipedia's "reliable source" policies. Information on Wikipedia is meant to be backed up by a verifiable source, partly to prevent a situation where knowledge only makes it onto Wikipedia if enough of the editors agree that it should be true. Molly White made a great video and write-up explaining this a few months ago: https://blog.mollywhite.net/become-a-wikipedian-transcript/#...

I know about that, but it's basically the same thing since reliable sources in the wikipedia terms are a set of sources that we collectively decide to trust. What's interesting about wikipedia sources is that it won't allow you to directly quote a person even though they are a well known trustworthy information source. Instead you must back up your statements through a 3rd party (usually media-related) entity. This i…

>it won't allow you to directly quote a person even though they are a well known trustworthy information source

Little known fact: That is actually allowed in some limited situations, but only reluctantly, and with a lot of care.

For instance if someone is recognized as an established researcher in their field with publications in top academic journals and then they make a statement on their website about something they have expertise in, you can actually cite that if you have no better source! Even though it's a direct self-published quote.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#33

Post by btown »

paxys wrote: post #9
mushufasa wrote: post #2

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

It also seems Wikimedia isn't trying to relicense the content in any way that strips its e.g. CC-SA status, but rather providing the licenses as context alongside each API call. https://helpcenter.enterprise.wikimedia.com/hc/en-us/article...

It's worth noting that https://creativecommons.org/faq/#artificial-intelligence-and... itself takes the general stance that "as a general matter text and data mining in the United States is considered a fair use and does not require permission under copyright."

But as a practical matter, I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training by the very platform that solicited their volunteer services, regardless of their works' legal status and Wikimedia's technical respect of that legal status.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#34

Post by TheGlav »

Tao3300 wrote: post #31
tux3 wrote: post #17

Earlier quoted context omitted.

That's exactly what almost all knowledge is. When was it that you last verified something by yourself, with an experiment? You didn't test the things you know. You know things because you could see they were the consensus, and so you had no reason to challenge them. If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unre…

We trust the consensus of published, peer-reviewed experts. That's different than the kind of Demos that does things like declare war on Persia, kill Jesus and Socrates, or edit Wikipedia. > If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unreliable people. So with the Wiki. Right. That makes Wiki kind of unreliable.…

We humans also trust the consensus of non-peer reviewed truth all the time. Tell that group of children that the opposite sex doesn't have cooties, and there's a good chance they'll laugh at you. Look at any online community, and it's the same. We humans are great at it and do it all the time.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#35

Post by toomuchtodo »

bawolff wrote: post #23

While i am sympathetic to wmf finding alternative funding streams, I do get nervous about these sort of things due to the inherent conflict of interest and incentives to canabalize the free offerings. I'm not saying that is happening now, but will it happen eventually? Additionally, originally it was promised this would all be open source, and officially they are sticking with that, but it seems like they are going w…

I’d rather Wikipedia offer it versus a for profit enterprise using their dataset. Cut out the middleman and vertically integrate.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#37

Post by rkagerer »

tux3 wrote: post #17
s1k3s wrote: post #5

> Real-time access to Knowledge Wikipedia knowledge is basically "democracy" knowledge, i.e. the more people decided to support an idea, the "truer" it gets. That's not knowledge at all!

That's exactly what almost all knowledge is. When was it that you last verified something by yourself, with an experiment? You didn't test the things you know. You know things because you could see they were the consensus, and so you had no reason to challenge them. If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unre…

When was it that you last verified something by yourself, with an experiment?

Just a few weeks ago. I did a simple experiment to check whether the Super-94 at my local Chevron is indeed ethanol-free. It wasn't.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#38

Post by violet13 »

itishappy wrote: post #8
mushufasa wrote: post #2

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

Wikimedia is Wikipedia's parent entity, but it's also the parent of: MediaWiki Wikibooks Wikidata Wikifunctions Wikimedia Commons Wikinews Wikiquote Wikisource Wikispecies Wikiversity Wikivoyage Wiktionary https://en.wikipedia.org/wiki/Wikimedia_Foundation

It sounds like an impressive list, but many of these are ghost towns and are of little value to machine learning. Wikiversity is a mess... Wikipedia is the crown jewel and probably the only thing of unique commercial value for ML.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#39

Post by yreg »

kristianpaul wrote: post #36

Interesting, the current DB dump is not for everyone, but if they also offer an LLM trained on Wikipedia data that answers and provide actual valida citation , please do. (not sure if duckduckgo stop offering that)

AFAIK a good way to provide better answers and avoid hallucinations would be to compute embeddings for all sections of text in Wikipedia and then when a user asks a question create an embedding from that question.

Use it to find the X closest embeddings to the question being posed, lookup their original articles, feed them all into context of an LLM and then ask it to answer the question based on that context (alone).

Contexts are becomming quite large so it's possible to put a lot of stuff in there. LLMs answering questions based on a giben text seem to be more reliable than those that are simply trained/fine tuned on some library of texts. p

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#40

Post by tux3 »

Tao3300 wrote: post #31
tux3 wrote: post #17

Earlier quoted context omitted.

That's exactly what almost all knowledge is. When was it that you last verified something by yourself, with an experiment? You didn't test the things you know. You know things because you could see they were the consensus, and so you had no reason to challenge them. If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unre…

We trust the consensus of published, peer-reviewed experts. That's different than the kind of Demos that does things like declare war on Persia, kill Jesus and Socrates, or edit Wikipedia. > If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unreliable people. So with the Wiki. Right. That makes Wiki kind of unreliable.…

>We trust the consensus of published, peer-reviewed experts. That's different than the kind of Demos that does things like declare war on Persia, kill Jesus and Socrates, or edit Wikipedia.

Ye of little faith! The Demos, after much bickering, have also decided to largely trust the consensus of published, peer-reviewed experts.

But what you really have to ask when you say you trust the consensus, is who forms the consensus of peer-reviewed experts?

It's rare to get an explicit consensus from an actual organization. When Cochrane does a large meta-analysis, and whisper "moderate evidence", I stop reading and immediately trust them with my life. Unfortunately, they very rarely have confidence in anything.

Most of the time, the consensus of published, peer-reviewed expert is also not something people form on their own. When has your neighbor last read and synthesized the literature to determine what the consensus is on hydroxyapatite in toothpaste, before going to the store?

Individual experts, I also trust only as far as I can throw them. The consensus of experts I'm happy to rely on. But that, very often, also comes from trusting the Demos, I'm afraid.

Post reply on HN