Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikimedia Enterprise – APIs for LLMs, AI Training, and More

enterprise.wikimedia.com

Post reply on HN
Print view View original

41–50 of 166 posts

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#41

Post by jahewson »

yreg wrote: post #39
kristianpaul wrote: post #36

Interesting, the current DB dump is not for everyone, but if they also offer an LLM trained on Wikipedia data that answers and provide actual valida citation , please do. (not sure if duckduckgo stop offering that)

AFAIK a good way to provide better answers and avoid hallucinations would be to compute embeddings for all sections of text in Wikipedia and then when a user asks a question create an embedding from that question. Use it to find the X closest embeddings to the question being posed, lookup their original articles, feed them all into context of an LLM and then ask it to answer the question based on that context (alone)…

Unfortunately this still results in plenty of hallucinations.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#42

Post by mtmail »

mushufasa wrote: post #2

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

I'm using those dumps for a monthly data process. It breaks regularly, mirrors disappear, there's only very few mirrors with the full data and little history. (I'm running a mirror for OpenStreetMap but don't have the disk capacity or bandwith for wikipedia's mirror requirements).

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#43

Post by schlauerfox »

elric wrote: post #29
paxys wrote: post #9

Earlier quoted context omitted.

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

CC-BY-SA to be specific. How does attribution of derivative works work in LLMs (etc)? When is the content they produce required to be CC-BY-SA as well (by virtue of the -SA part of the license)?

LLM output can't be copywritten as it's not the work of a human. The company would presumably have to include attribution to every piece of data the system is trained on for every query? Seems absurd, but then how would attribution work? Not really considered when the license was written.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#44

Post by yreg »

thih9 wrote: post #30

Is it’s ok to train AIs on CC-licensed data? What about attribution, should the AI be distributed with a note about the dataset’s origin? I could imagine a poorly trained model that returns content nearly identical to original; who would enforce CC rules then?

It shouldn't be much of a problem to ship the LLM along with attributions. (List of all sources used in the dataset - not a problem, unless they are secret, shady or illegal.) Wikipedia is one of the easy ones, since you need to attribute just 'Wikipedia' for the entire corpus instead of many individual users.

The bigger issue is when a user uses the model to generate some text. Should they attribute it when using it somewhere?

That doesn't seem very practical, since it seems that soon most of the text will be edited by LLMs and those seem to be trained on most of the web -> so pretty much everything would need to be attributed to everything. Unless someone puts a stop to this, which I find improbable.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#45

Post by riku_iki »

codetrotter wrote: post #24
ks2048 wrote: post #16

Earlier quoted context omitted.

I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. A…

There’s also Wikidata which has machine readable data for everyone https://m.wikidata.org/wiki/Wikidata:Main_Page > Wikidata is a free and open knowledge base that can be read and edited by both humans and machines. > Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wiktionary, Wikisource, and others.

Yes, but it is different dataset, many(majority?) infoboxes are not in wikidata

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#46

Post by kemayo »

bawolff wrote: post #28
kemayo wrote: post #15

Earlier quoted context omitted.

My understanding is that Enterprise is mostly about smoother delivery of content. E.g. Google has those informational summaries in search results, and it's easier for it to keep those current if there's a stream of updates it can subscribe to, rather than having to constantly download the full Wikipedia database dumps and parse them out. It also puts it all into conveniently formatted responses, and tries to do some…

> to keep those current if there's a stream of updates it can subscribe to, rather than having to constantly download the full Wikipedia database dumps and parse them out. I mean presumably prior to this they were using the [free] parsoid rest api along with the [free] event stream api. I highly doubt they were parsing the dumps. Its not even clear to me what the core value proposition of the new api is over the old…

No idea whether it's any quicker than the event stream APIs, but it sounds like the machine-readable aspect does a lot of processing that just using parsoid wouldn't get you. Parsoid is only a wikitext-to-HTML service, so you'd still need to do significant work on it to get meaning out of that HTML. The data dictionary page[1] suggests that it's exposing things like "how many citation-needed templates are in this?" (presumably localized, since those template names change across wikis), which I could see being handy for services like that.

That said, I suspect that the real value is just the "reliable" bit on that summary page.

> Written agreements, no additional content restrictions, SLAs, 24/7 support, and contractual guarantees against surprise breaking changes make working with Wikimedia Enterprise safe and reliable at scale.

i.e. If I was building a big enterprise thing using wikipedia data, getting a SLA and a promise that us rando open source devs won't completely change the APIs on a whim would be nice.

[1]: https://enterprise.wikimedia.com/docs/data-dictionary/

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#47

Post by low_tech_punk »

Nitpick, the purple blue gradient is overused, especially as the background of source code examples. It hinders readability and feels off-brand for a Wikimedia project. This might be foreshadowing a descent into chasing trends. I just hope Wikimedia does not follow Mozilla's track in spending budget on fluff work.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#48

Post by w10-1 »

mtmail wrote: post #42
mushufasa wrote: post #2

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

I'm using those dumps for a monthly data process. It breaks regularly, mirrors disappear, there's only very few mirrors with the full data and little history. (I'm running a mirror for OpenStreetMap but don't have the disk capacity or bandwith for wikipedia's mirror requirements).

Right, the incentives are adverse: if they lower quality of service for free offerings, they'll be rewarded with more paying customers for reliable service.

But quality does take money.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#49

Post by afh1 »

simonw wrote: post #18
s1k3s wrote: post #5

> Real-time access to Knowledge Wikipedia knowledge is basically "democracy" knowledge, i.e. the more people decided to support an idea, the "truer" it gets. That's not knowledge at all!

Read up on Wikipedia's "reliable source" policies. Information on Wikipedia is meant to be backed up by a verifiable source, partly to prevent a situation where knowledge only makes it onto Wikipedia if enough of the editors agree that it should be true. Molly White made a great video and write-up explaining this a few months ago: https://blog.mollywhite.net/become-a-wikipedian-transcript/#...

... and Wikipedia does not consider itself a reliable source. Rightfully so. Open any politics-related article and you'll see why.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#50

Post by cdme »

riku_iki wrote: post #7
deegles wrote: post #4

Looking forward to seeing the details on how they will handle revenue sharing with all of the people who contribute to them.

are they obligated?..

No, but nobody is obligated to contribute to or support them either.
Post reply on HN