Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikimedia Enterprise – APIs for LLMs, AI Training, and More

enterprise.wikimedia.com

Post reply on HN
Print view View original

121–130 of 166 posts

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#121

Post by sciurus »

yareal wrote: post #91
tw04 wrote: post #76

Earlier quoted context omitted.

What are you talking about? The AVERAGE CD right now is 5%. My local CU is almost 6%. US bonds are currently ~4.5% - if you consider those unstable, I guess the US economy isn't stable - and if the US economy crashes, wikipedia will be the least of their or our worries. Wikimedia's expenses are almost ENTIRELY going to staff. Their balance sheet for 2023 included $101m in expenses for salaries and benefits out of a t…

An engineer costs $500k a year. Salary, benefits, office space, equipment, hr, legal, and other overhead. The engineer will only see a fraction of that, of course. If you told me it took a hundred engineers to run Wikipedia I'd say, that's not totally unreasonable. Features, design, api, scaling, moderation, there's a ton for engineers to be doing.

Your overall point still stands, but FWIW Wikimedia pays less than its peers. To compare two active listings

Senior Security Engineer at Mozilla (https://boards.greenhouse.io/mozilla/jobs/5803609): $124,000 to $199,000 plus bonus

Senior Security Engineer at Wikimedia (https://boards.greenhouse.io/wikimedia/jobs/5890112): $105,000 to $164,000

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#122

Post by michaelt »

miffy900 wrote: post #120

I'm reading the API docs https://enterprise.wikimedia.com/docs/ And they don't have an OpenAPI spec available to download? So they seriously expect developers to just manually write their own client code by reading and translating those example CURL commands by hand??! Seriously it's 2024! Not having a spec to download and insepct for any API is a sign of incompetence. When tools like Postman or https://github.com/Op…

If only there was some code generation technology that could take in human-readable specifications, not just machine-readable ones....

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#123

Post by madeofpalk »

w10-1 wrote: post #48
mtmail wrote: post #42

Earlier quoted context omitted.

I'm using those dumps for a monthly data process. It breaks regularly, mirrors disappear, there's only very few mirrors with the full data and little history. (I'm running a mirror for OpenStreetMap but don't have the disk capacity or bandwith for wikipedia's mirror requirements).

Right, the incentives are adverse: if they lower quality of service for free offerings, they'll be rewarded with more paying customers for reliable service. But quality does take money.

What are the incentives for more paying customers? Wikimedia Foundation, being a non-profit, has a lot less incentives to chase money at the expense of product quality.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#124

Post by Contax »

Brybry wrote: post #90
bawolff wrote: post #80

Earlier quoted context omitted.

This is a rediculous thing to say. You think the interest off 250 million would be enough to run the wikipedia website? Do you have any experience operating web properties at scale to come to this conclusion? Did you base this on anything at all?

I don't agree that it's pure greed but hosting costs for the Wikimedia Foundation in the FY2022-2023 were $3.1 million. [1][2] [1] https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2... [2] https://wikimediafoundation.org/wp-content/uploads/2023/11/W... (page 4, pdf page 6 for expenses)

I don't know what it takes at the scale of something like Wikipedia, but at my own scale, a single/self-employed developer, yes, hosting costs are a small part, but then I have to eat, a place to live, replace hardware, pay bills and taxes and much much more, things none of my sites could exist without.

Hosting is nothing without coders and maintainers, regardless of if the main content is contributed free, and decent coders and maintainers, software and hardware engineers, aren't cheap and don't count as hosting costs.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#125

Post by emw »

lambdaone wrote: post #106
emw wrote: post #52

Earlier quoted context omitted.

> I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training As someone who avidly edited Wikipedia for 6-8 years, I am happy to see my volunteer work used for LLM training. I also agree some other editors likely aren't.

Given that all Wikipedia editors have explicitly consented to their content being released under the Creative Commons Attribution-ShareAlike 4.0 License, they don't get a choice about their content being used for any purpose. Redistribution of content is an entirely different matter, and the legal status of copyrighted material in relation to LLM training is an open issue that is currently the subject of litigation.

Wikimedia Foundation’s perspective on this [1]:

> "it is important to note that Creative Commons licenses allow for free reproduction and reuse, so AI programs like ChatGPT might copy text from a Wikipedia article or an image from Wikimedia Commons. However, it is not clear yet whether massively copying content from these sources may result in a violation of the Creative Commons license if attribution is not granted. Overall, it is more likely than not if current precedent holds that training systems on copyrighted data will be covered by fair use in the United States, but there is significant uncertainty at time of writing."

The new Wikimedia Enterprise APIs facilitate attribution. For example, the "api.enterprise.wikimedia.com/v2/structured-contents/{name}" response [2] includes an "editor" object in a "version" object. So the Wikipedia editor who most recently edited the article seems quite feasible to attribute. ML apps could incorporate such attribution in their offering, and help satisfy the "BY" clause in the underlying CC-BY-SA 4.0 license for Wikipedia content.

---

1. https://meta.wikimedia.org/wiki/Wikilegal/Copyright_Analysis...

2. https://enterprise.wikimedia.com/docs/on-demand/#article-str...

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#126

Post by yareal »

sciurus wrote: post #121
yareal wrote: post #91

Earlier quoted context omitted.

An engineer costs $500k a year. Salary, benefits, office space, equipment, hr, legal, and other overhead. The engineer will only see a fraction of that, of course. If you told me it took a hundred engineers to run Wikipedia I'd say, that's not totally unreasonable. Features, design, api, scaling, moderation, there's a ton for engineers to be doing.

Your overall point still stands, but FWIW Wikimedia pays less than its peers. To compare two active listings Senior Security Engineer at Mozilla ( https://boards.greenhouse.io/mozilla/jobs/5803609 ): $124,000 to $199,000 plus bonus Senior Security Engineer at Wikimedia ( https://boards.greenhouse.io/wikimedia/jobs/5890112 ): $105,000 to $164,000

The rule of thumb is that employees cost the company double what they pay the employee. So, still hundreds of thousands per employee.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#127

Post by ZunarJ5 »

emw wrote: post #52
btown wrote: post #33

Earlier quoted context omitted.

It also seems Wikimedia isn't trying to relicense the content in any way that strips its e.g. CC-SA status, but rather providing the licenses as context alongside each API call. https://helpcenter.enterprise.wikimedia.com/hc/en-us/article... It's worth noting that https://creativecommons.org/faq/#artificial-intelligence-and... itself takes the general stance that "as a general matter text and data mining in the Unite…

> I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training As someone who avidly edited Wikipedia for 6-8 years, I am happy to see my volunteer work used for LLM training. I also agree some other editors likely aren't.

As another editor, I think they might be a vocal minority. :)

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#128

Post by croes »

miffy900 wrote: post #120

I'm reading the API docs https://enterprise.wikimedia.com/docs/ And they don't have an OpenAPI spec available to download? So they seriously expect developers to just manually write their own client code by reading and translating those example CURL commands by hand??! Seriously it's 2024! Not having a spec to download and insepct for any API is a sign of incompetence. When tools like Postman or https://github.com/Op…

Then create one, share it and save us some time.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#129

Post by StableAlkyne »

mushufasa wrote: post #2

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

It does exist, it is just incredibly painful to use -- it's a raw SQL dump of each individual page, which is missing some of the dynamically generated parts of the page you might care about (like categories, unless you separately download that table too). You also have to then run it through wiki software locally if you want the rendered page instead of markup.

Don't get me wrong, it is wonderful that the Wikipedia team offers this, and I am grateful they give anything at all for offline usage. It just feels like it's intended more as a side product of their backup process, rather than something you're really supposed to use.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#130

Post by trustno2 »

croes wrote: post #128
miffy900 wrote: post #120

I'm reading the API docs https://enterprise.wikimedia.com/docs/ And they don't have an OpenAPI spec available to download? So they seriously expect developers to just manually write their own client code by reading and translating those example CURL commands by hand??! Seriously it's 2024! Not having a spec to download and insepct for any API is a sign of incompetence. When tools like Postman or https://github.com/Op…

Then create one, share it and save us some time.

so enterprise very wow
Post reply on HN