Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikimedia Enterprise – APIs for LLMs, AI Training, and More

enterprise.wikimedia.com

Post reply on HN
Print view View original

21–30 of 166 posts

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#21

Post by bawolff »

nickpsecurity wrote: post #3

I was hoping more groups did stuff like this. The free sites doing it could handle some copyright issues if their EULA had a built-in license for distribution. In my previous analysis, (IIRC) I found that Wikipedia articles were under a copyleft license with attribution requirements. Does how Wikipedia Enterprise delivers this bypass that where neither use nor derivatives have those requirements? Or are they ignoring…

All licensing requirements still apply.

However keep in mind that different wikimedia content is under different licenses (e.g. wikidata is public domain). Additionally in the united states, if you take just the factual content (removed from any creative context) the facts are not copyrightable so the license does not apply.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#22

Post by s1k3s »

simonw wrote: post #18
s1k3s wrote: post #5

> Real-time access to Knowledge Wikipedia knowledge is basically "democracy" knowledge, i.e. the more people decided to support an idea, the "truer" it gets. That's not knowledge at all!

Read up on Wikipedia's "reliable source" policies. Information on Wikipedia is meant to be backed up by a verifiable source, partly to prevent a situation where knowledge only makes it onto Wikipedia if enough of the editors agree that it should be true. Molly White made a great video and write-up explaining this a few months ago: https://blog.mollywhite.net/become-a-wikipedian-transcript/#...

I know about that, but it's basically the same thing since reliable sources in the wikipedia terms are a set of sources that we collectively decide to trust. What's interesting about wikipedia sources is that it won't allow you to directly quote a person even though they are a well known trustworthy information source. Instead you must back up your statements through a 3rd party (usually media-related) entity. This is both good and bad, considering that journalists may not be the best at evaluating certain information, especially in the science or politics field.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#23

Post by bawolff »

While i am sympathetic to wmf finding alternative funding streams, I do get nervous about these sort of things due to the inherent conflict of interest and incentives to canabalize the free offerings. I'm not saying that is happening now, but will it happen eventually?

Additionally, originally it was promised this would all be open source, and officially they are sticking with that, but it seems like they are going with the model of throwing code over the fence like once a year, which does not really meet my expectations.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#24

Post by codetrotter »

ks2048 wrote: post #16
paxys wrote: post #9

Earlier quoted context omitted.

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. A…

There’s also Wikidata which has machine readable data for everyone

https://m.wikidata.org/wiki/Wikidata:Main_Page

> Wikidata is a free and open knowledge base that can be read and edited by both humans and machines.

> Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wiktionary, Wikisource, and others.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#25

Post by tux3 »

paxys wrote: post #10
deegles wrote: post #4

Looking forward to seeing the details on how they will handle revenue sharing with all of the people who contribute to them.

If you want monetary compensation you should not be contributing to Wikipedia. Otherwise apply to work there as an employee.

After seeing this comment, onlookers report that the Wikimedia Foundation appears to have just vanished in a puff of volunteerism

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#26

Post by bawolff »

ks2048 wrote: post #16
paxys wrote: post #9

Earlier quoted context omitted.

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. A…

Just fyi, i think the plan for machine readable infoboxes is stick the data in wikidata, which has its own dumps in machine readable format as well as a publicly accessible SPARQL query endpoint.

Its actually quite cool. If you have never played with it i encourage checking out some of the example queries (there is a button labelled examples) on https://query.wikidata.org/

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#27

Post by nickpsecurity »

tux3 wrote: post #17
s1k3s wrote: post #5

> Real-time access to Knowledge Wikipedia knowledge is basically "democracy" knowledge, i.e. the more people decided to support an idea, the "truer" it gets. That's not knowledge at all!

That's exactly what almost all knowledge is. When was it that you last verified something by yourself, with an experiment? You didn't test the things you know. You know things because you could see they were the consensus, and so you had no reason to challenge them. If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unre…

“That's exactly what almost all knowledge is.”

It’s worse than that. Much knowledge comes from authorities or peers. The sources could be unpopular or barely reviewed. Yet, people are likely to believe specific types of sources. It seems to be hardwired for some purpose.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#28

Post by bawolff »

kemayo wrote: post #15
nickpsecurity wrote: post #3

I was hoping more groups did stuff like this. The free sites doing it could handle some copyright issues if their EULA had a built-in license for distribution. In my previous analysis, (IIRC) I found that Wikipedia articles were under a copyleft license with attribution requirements. Does how Wikipedia Enterprise delivers this bypass that where neither use nor derivatives have those requirements? Or are they ignoring…

My understanding is that Enterprise is mostly about smoother delivery of content. E.g. Google has those informational summaries in search results, and it's easier for it to keep those current if there's a stream of updates it can subscribe to, rather than having to constantly download the full Wikipedia database dumps and parse them out. It also puts it all into conveniently formatted responses, and tries to do some…

> to keep those current if there's a stream of updates it can subscribe to, rather than having to constantly download the full Wikipedia database dumps and parse them out.

I mean presumably prior to this they were using the [free] parsoid rest api along with the [free] event stream api. I highly doubt they were parsing the dumps.

Its not even clear to me what the core value proposition of the new api is over the old api from the perspective of google. Maybe SLAs? A way to justify donating money without it looking like a donation?

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#29

Post by elric »

paxys wrote: post #9
mushufasa wrote: post #2

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

CC-BY-SA to be specific. How does attribution of derivative works work in LLMs (etc)? When is the content they produce required to be CC-BY-SA as well (by virtue of the -SA part of the license)?

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#30

Post by thih9 »

Is it’s ok to train AIs on CC-licensed data? What about attribution, should the AI be distributed with a note about the dataset’s origin?

I could imagine a poorly trained model that returns content nearly identical to original; who would enforce CC rules then?

Post reply on HN