Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikimedia Enterprise – APIs for LLMs, AI Training, and More

enterprise.wikimedia.com

Post reply on HN
Print view View original

51–60 of 166 posts

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#51

Post by kemayo »

bawolff wrote: post #23

While i am sympathetic to wmf finding alternative funding streams, I do get nervous about these sort of things due to the inherent conflict of interest and incentives to canabalize the free offerings. I'm not saying that is happening now, but will it happen eventually? Additionally, originally it was promised this would all be open source, and officially they are sticking with that, but it seems like they are going w…

There's a whole statement-of-principles thing that at least implies that the intention isn't to cannibalize the existing offerings: https://meta.wikimedia.org/wiki/Wikimedia_Enterprise/Princip...

Though I imagine that only works so far as you feel you can trust the Foundation to stick to those principles, so that's complicated. :D

There's also a bunch of FAQs here that sort of get at how the funding streams are supposed to integrate into the existing structures and how it's supposed to avoid pushing out the free services: https://meta.wikimedia.org/wiki/Wikimedia_Enterprise/FAQ

(As I said elsewhere, I work for the WMF but I don't work on anything related to this so I'm just commenting as someone who has more experience searching through our public info than most HN commenters would...)

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#52

Post by emw »

btown wrote: post #33
paxys wrote: post #9

Earlier quoted context omitted.

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

It also seems Wikimedia isn't trying to relicense the content in any way that strips its e.g. CC-SA status, but rather providing the licenses as context alongside each API call. https://helpcenter.enterprise.wikimedia.com/hc/en-us/article... It's worth noting that https://creativecommons.org/faq/#artificial-intelligence-and... itself takes the general stance that "as a general matter text and data mining in the Unite…

> I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training

As someone who avidly edited Wikipedia for 6-8 years, I am happy to see my volunteer work used for LLM training. I also agree some other editors likely aren't.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#53

Post by emw »

violet13 wrote: post #38
itishappy wrote: post #8

Earlier quoted context omitted.

Wikimedia is Wikipedia's parent entity, but it's also the parent of: MediaWiki Wikibooks Wikidata Wikifunctions Wikimedia Commons Wikinews Wikiquote Wikisource Wikispecies Wikiversity Wikivoyage Wiktionary https://en.wikipedia.org/wiki/Wikimedia_Foundation

It sounds like an impressive list, but many of these are ghost towns and are of little value to machine learning. Wikiversity is a mess... Wikipedia is the crown jewel and probably the only thing of unique commercial value for ML.

Wikipedia indeed seems the most valuable for ML, by far. Wikidata, Wikimedia Commons, and Wiktionary also seem useful there.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#54

Post by falcor84 »

jahewson wrote: post #41
yreg wrote: post #39

Earlier quoted context omitted.

AFAIK a good way to provide better answers and avoid hallucinations would be to compute embeddings for all sections of text in Wikipedia and then when a user asks a question create an embedding from that question. Use it to find the X closest embeddings to the question being posed, lookup their original articles, feed them all into context of an LLM and then ask it to answer the question based on that context (alone)…

Unfortunately this still results in plenty of hallucinations.

What do you mean? Have you or someone else already followed this exact approach?

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#55

Post by philipwhiuk »

tux3 wrote: post #25
paxys wrote: post #10

Earlier quoted context omitted.

If you want monetary compensation you should not be contributing to Wikipedia. Otherwise apply to work there as an employee.

After seeing this comment, onlookers report that the Wikimedia Foundation appears to have just vanished in a puff of volunteerism

The Wikimedia Foundation are paid employees.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#56

Post by dools »

rkagerer wrote: post #37
tux3 wrote: post #17

Earlier quoted context omitted.

That's exactly what almost all knowledge is. When was it that you last verified something by yourself, with an experiment? You didn't test the things you know. You know things because you could see they were the consensus, and so you had no reason to challenge them. If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unre…

When was it that you last verified something by yourself, with an experiment? Just a few weeks ago. I did a simple experiment to check whether the Super-94 at my local Chevron is indeed ethanol-free. It wasn't.

Now imagine you were able to go and edit the sign that says it is ethanol free to add the details of your test and dispute the claim, that would improve the knowledge.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#57

Post by riku_iki »

cdme wrote: post #50
riku_iki wrote: post #7

Earlier quoted context omitted.

are they obligated?..

No, but nobody is obligated to contribute to or support them either.

that's not really true. Because of network capture, those who wants to contribute into some public free human knowledge, obligated to do it through wikimedia and contribute to wikimedia revenue streams.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#58

Post by zozbot234 »

ks2048 wrote: post #16
paxys wrote: post #9

Earlier quoted context omitted.

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. A…

DBPedia has been providing machine-readable info drawn from Wikipedia infoboxes for a long time, and that's absolutely an open project. But there are drawbacks to that approach, nowadays Wikipedia users are broadly expected to input that info into Wikidata (a language-independent project, providing its data under the CC0 license) and the Wikipedias are gradually moving towards fetching the information from there as well. The English Wikipedia is somewhat behind on that, so DBPedia might still be useful for parsing info from it.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#59

Post by pwarner »

thih9 wrote: post #30

Is it’s ok to train AIs on CC-licensed data? What about attribution, should the AI be distributed with a note about the dataset’s origin? I could imagine a poorly trained model that returns content nearly identical to original; who would enforce CC rules then?

https://creativecommons.org/2023/08/18/understanding-cc-lice...

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#60

Post by kemayo »

ks2048 wrote: post #16
paxys wrote: post #9

Earlier quoted context omitted.

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. A…

Machine-readable infoboxes are surprisingly painful because an "infobox" isn't a top-level thing in mediawiki. Rather, it's something that emerged from community practice -- a template that's slightly-standardized, and has a sprawling network of subtemplates for specific categories of article. Then the assorted other language wikipedias have their own variants of these templates with their own names, etc.

Pulling it into a mediawiki extension (or core) and making it part of the page-level metadata gets suggested pretty frequently, but it's a bit contentious amongst the hardcore editors who'd need to actually adopt such things. The template-based nature of the current infoboxes mean that they're very accessible to the community, and it's easy to spin off new variants or make changes without getting programmers to help you.

There's slow movement towards getting the sort of data that winds up in infoboxes into wikidata, but it's still somewhat spotty.

(If you've never done it, it can be quite edifying to install mediawiki for yourself and seeing how much of the surrounding infrastructure of wikipedia is absent because it's all templates.)

Post reply on HN