Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikimedia Enterprise – APIs for LLMs, AI Training, and More

enterprise.wikimedia.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#61

Post by resolutebat »

emw wrote: post #53
violet13 wrote: post #38

Earlier quoted context omitted.

It sounds like an impressive list, but many of these are ghost towns and are of little value to machine learning. Wikiversity is a mess... Wikipedia is the crown jewel and probably the only thing of unique commercial value for ML.

Wikipedia indeed seems the most valuable for ML, by far. Wikidata, Wikimedia Commons, and Wiktionary also seem useful there.

Wikivoyage is underrated and that was not helped by the acrimonious split with Wikitravel (which was acquired a predatory marketing company), but it finally seems to be pulling ahead.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#62

Post by molticrystal »

tux3 wrote: post #17
s1k3s wrote: post #5

> Real-time access to Knowledge Wikipedia knowledge is basically "democracy" knowledge, i.e. the more people decided to support an idea, the "truer" it gets. That's not knowledge at all!

That's exactly what almost all knowledge is. When was it that you last verified something by yourself, with an experiment? You didn't test the things you know. You know things because you could see they were the consensus, and so you had no reason to challenge them. If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unre…

I thought knowledge, at least the best type comes from primary sources and from repeatable experiments with explicit premises as much as possible. This makes it sound like all knowledge is hearsay. If it is, what is the point of a place like Wikipedia or even an encyclopedia?

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#63

Post by nestorD »

falcor84 wrote: post #54
jahewson wrote: post #41

Earlier quoted context omitted.

Unfortunately this still results in plenty of hallucinations.

What do you mean? Have you or someone else already followed this exact approach?

The approach described above is what is commonly referred to as RAG[0]. I am not aware of someone having used it on Wikipedia but, from experience and while it helps, it does not eliminate all hallucinations.

[0]: https://en.wikipedia.org/wiki/Large_language_model#:~:text=t...

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#64

Post by ImaCake »

mushufasa wrote: post #2

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

Having the data available for download and a paywalled API is pretty standard practice for a lot of this kind of data. Copernicus weather/satellite data is often setup like this. There's a free download option, a free (limited) API, and then if you wanna get fancy you can use a more feature rich paid API. Many other weather services do similar. It seems to work pretty well with the usual caveats.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#65

Post by ImaCake »

kristianpaul wrote: post #36

Interesting, the current DB dump is not for everyone, but if they also offer an LLM trained on Wikipedia data that answers and provide actual valida citation , please do. (not sure if duckduckgo stop offering that)

You can do this with the copilot chat feature in MS Edge. I just tried to ask it to use only wikipedia and it gave me four references, two of which were wiki. So at least you can get it to spit out references with a bias

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#66

Post by sp332 »

yreg wrote: post #44
thih9 wrote: post #30

Is it’s ok to train AIs on CC-licensed data? What about attribution, should the AI be distributed with a note about the dataset’s origin? I could imagine a poorly trained model that returns content nearly identical to original; who would enforce CC rules then?

It shouldn't be much of a problem to ship the LLM along with attributions. (List of all sources used in the dataset - not a problem, unless they are secret, shady or illegal.) Wikipedia is one of the easy ones, since you need to attribute just 'Wikipedia' for the entire corpus instead of many individual users. The bigger issue is when a user uses the model to generate some text. Should they attribute it when using it…

Including a list of every Wikipedia author is possible but very inconvenient. https://en.m.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_...

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#67

Post by hehdhdjehehegwv »

emw wrote: post #53
violet13 wrote: post #38

Earlier quoted context omitted.

It sounds like an impressive list, but many of these are ghost towns and are of little value to machine learning. Wikiversity is a mess... Wikipedia is the crown jewel and probably the only thing of unique commercial value for ML.

Wikipedia indeed seems the most valuable for ML, by far. Wikidata, Wikimedia Commons, and Wiktionary also seem useful there.

One of my favorite LLM applications is getting them to write wikidata queries. The data is amazing, but the query language is nothing but pure hell.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#68

Post by croemer »

dools wrote: post #56
rkagerer wrote: post #37

Earlier quoted context omitted.

When was it that you last verified something by yourself, with an experiment? Just a few weeks ago. I did a simple experiment to check whether the Super-94 at my local Chevron is indeed ethanol-free. It wasn't.

Now imagine you were able to go and edit the sign that says it is ethanol free to add the details of your test and dispute the claim, that would improve the knowledge.

For that to make it into Wikipedia you'd have to first write an article in a reputable source.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#69

Post by tux3 »

philipwhiuk wrote: post #55
tux3 wrote: post #25

Earlier quoted context omitted.

After seeing this comment, onlookers report that the Wikimedia Foundation appears to have just vanished in a puff of volunteerism

The Wikimedia Foundation are paid employees.

They sure are!

I think the parent comment to mine was edited, and now my reply looks weird. This is what I remember replying to:

"If you want monetary compensation you should not be contributing to Wikipedia"

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#70

Post by PokestarFan »

kemayo wrote: post #60
ks2048 wrote: post #16

Earlier quoted context omitted.

I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. A…

Machine-readable infoboxes are surprisingly painful because an "infobox" isn't a top-level thing in mediawiki. Rather, it's something that emerged from community practice -- a template that's slightly-standardized, and has a sprawling network of subtemplates for specific categories of article. Then the assorted other language wikipedias have their own variants of these templates with their own names, etc. Pulling it…

I once tried to export a simple infobox template with dependencies and it exported 30-40 different template/module pages.
Post reply on HN