Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikimedia Enterprise – APIs for LLMs, AI Training, and More

enterprise.wikimedia.com

Post reply on HN
Print view View original

81–90 of 166 posts

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#81

Post by bawolff »

tw04 wrote: post #76
ff317 wrote: post #75

Earlier quoted context omitted.

Over the long term of many years you're /lucky/ if a stable very-low-risk investment can net ~3% when accounting for inflation. Thus $250M could maybe net you roughly $7.5M/year. Exactly how many network links, servers, and engineering staff do you think that buys? It's way under what it operates on today, which is way under what it ideally should be for site like Wikipedia. And that's /just/ the operational engineer…

What are you talking about? The AVERAGE CD right now is 5%. My local CU is almost 6%. US bonds are currently ~4.5% - if you consider those unstable, I guess the US economy isn't stable - and if the US economy crashes, wikipedia will be the least of their or our worries. Wikimedia's expenses are almost ENTIRELY going to staff. Their balance sheet for 2023 included $101m in expenses for salaries and benefits out of a t…

> Their hosting was $3m

Its kind of unclear what this includes. Computer equipment is a separate line, and wikimedia owns its own servers, so presumably that is separate from server costs. You don't have to buy new servers every year so some servers might simply have been purchased in other years, although maybe that gets ammortized, i dont know.

Additionally when you host your own servers you need staff to operate them. When using something like AWS, this would be part of your AWS fees, but if you operate your own servers then you have to pay that part separately. Its probably cheaper overall in the end when you are wikipedia scale, but the costs break down differently.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#82

Post by boramalper »

mtmail wrote: post #42
mushufasa wrote: post #2

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

I'm using those dumps for a monthly data process. It breaks regularly, mirrors disappear, there's only very few mirrors with the full data and little history. (I'm running a mirror for OpenStreetMap but don't have the disk capacity or bandwith for wikipedia's mirror requirements).

> I'm running a mirror for OpenStreetMap

Curious why you're running a mirror for OSM, is it public or for your private use? If you don't mind, I'd love to learn more about it; sounds super interesting to me.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#83

Post by riedel »

bawolff wrote: post #23

While i am sympathetic to wmf finding alternative funding streams, I do get nervous about these sort of things due to the inherent conflict of interest and incentives to canabalize the free offerings. I'm not saying that is happening now, but will it happen eventually? Additionally, originally it was promised this would all be open source, and officially they are sticking with that, but it seems like they are going w…

I think commercial offerings are good to offer as an alternative to donations for institutions that want to support something and cannot easily make a donations.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#84

Post by nextaccountic »

paxys wrote: post #9
mushufasa wrote: post #2

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

> machine parsable formats,

Isn't the point of wikidata to offer machine-parseable formats?

Also: how can those formats be locked into a paywall?

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#85

Post by adultSwim »

I support this and think it should be a model for other organizations. They already publish the data for free. It's already being used to train models. Companies are already using those models to profit. I'm not sure I understand the concern.

I hope to see their APIs expand in future work, successfully tying together machine learning with their prior work on publishing a knowledge graph enriched with high quality structured metadata.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#86

Post by lenerdenator »

btown wrote: post #33
paxys wrote: post #9

Earlier quoted context omitted.

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

It also seems Wikimedia isn't trying to relicense the content in any way that strips its e.g. CC-SA status, but rather providing the licenses as context alongside each API call. https://helpcenter.enterprise.wikimedia.com/hc/en-us/article... It's worth noting that https://creativecommons.org/faq/#artificial-intelligence-and... itself takes the general stance that "as a general matter text and data mining in the Unite…

> But as a practical matter, I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training by the very platform that solicited their volunteer services,

I think that will heavily depend on just what the money goes to.

A better user experience, tightening up the code behind things, fewer nag screens for donations? Justifiable.

Jimmy Wales going from decently-compensated to a bad case of Founder Syndrome? Not so justifiable.

Founder Syndrome: a psychological condition wherein a person who starts a venture believes the venture should earn them a net worth on the order of billions of dollars, regardless of its actual economic value, and is willing to seriously enshittify the venture's product or service in order to make this delusion come true, especially in preparation for an IPO. See also: /u/spez, SPAC, Facebook, Unity Engine

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#87

Post by yareal »

tw04 wrote: post #74
bawolff wrote: post #23

While i am sympathetic to wmf finding alternative funding streams, I do get nervous about these sort of things due to the inherent conflict of interest and incentives to canabalize the free offerings. I'm not saying that is happening now, but will it happen eventually? Additionally, originally it was promised this would all be open source, and officially they are sticking with that, but it seems like they are going w…

> While i am sympathetic to wmf finding alternative funding streams Why are you sympathetic to it? Their fund, at this point, can make enough off interest on a basic CD to not just pay for every possible cost they could have until the end of time, but make the maintainer(s) obscenely wealthy without breaking a sweat. https://upload.wikimedia.org/wikipedia/foundation/3/3e/Wikim... $250m - they’re doing this out of gre…

Your own link shows investment income as about $3-4m. Nowhere near enough to pay for infrastructure and engineers, let alone all the other things you need to operate Wikipedia.

If you want to argue against finding Wikipedia you are going to have to do so in something approximating reality. Because the idea that they could somehow operate on one percent of their first budget is completely off base.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#88

Post by remus »

tw04 wrote: post #76
ff317 wrote: post #75

Earlier quoted context omitted.

Over the long term of many years you're /lucky/ if a stable very-low-risk investment can net ~3% when accounting for inflation. Thus $250M could maybe net you roughly $7.5M/year. Exactly how many network links, servers, and engineering staff do you think that buys? It's way under what it operates on today, which is way under what it ideally should be for site like Wikipedia. And that's /just/ the operational engineer…

What are you talking about? The AVERAGE CD right now is 5%. My local CU is almost 6%. US bonds are currently ~4.5% - if you consider those unstable, I guess the US economy isn't stable - and if the US economy crashes, wikipedia will be the least of their or our worries. Wikimedia's expenses are almost ENTIRELY going to staff. Their balance sheet for 2023 included $101m in expenses for salaries and benefits out of a t…

> What are you talking about? The AVERAGE CD right now is 5%. My local CU is almost 6%. US bonds are currently ~4.5% - if you consider those unstable, I guess the US economy isn't stable - and if the US economy crashes, wikipedia will be the least of their or our worries.

If you want to live off the interest you have to worry about inflation which essentially devalues your pot by x% per year, so if you really need y% for running costs you really need about x*y% to do it long term.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#89

Post by ryan_lane »

tw04 wrote: post #76
ff317 wrote: post #75

Earlier quoted context omitted.

Over the long term of many years you're /lucky/ if a stable very-low-risk investment can net ~3% when accounting for inflation. Thus $250M could maybe net you roughly $7.5M/year. Exactly how many network links, servers, and engineering staff do you think that buys? It's way under what it operates on today, which is way under what it ideally should be for site like Wikipedia. And that's /just/ the operational engineer…

What are you talking about? The AVERAGE CD right now is 5%. My local CU is almost 6%. US bonds are currently ~4.5% - if you consider those unstable, I guess the US economy isn't stable - and if the US economy crashes, wikipedia will be the least of their or our worries. Wikimedia's expenses are almost ENTIRELY going to staff. Their balance sheet for 2023 included $101m in expenses for salaries and benefits out of a t…

Hosting means nothing without the staff. Hardware, networking, datacenters, etc are the cheap part because the staff are good at their jobs.

You and the other set of trolls that think that Wikimedia can run itself need to appreciate that just because you work for a non-profit doesn't mean you should work for slave wages, or that you should be forced to work with the bare minimum amount of staff to keep things running without being able to make improvements to the infrastructure, reader experience, editor experience, or data consumer experience.

In comparison to similar services, Wikimedia has a relatively small overall budget that's well spent.

Re: Wikimedia Enterprise – APIs for LLMs, AI Training, and More

#90

Post by Brybry »

bawolff wrote: post #80
tw04 wrote: post #74

Earlier quoted context omitted.

> While i am sympathetic to wmf finding alternative funding streams Why are you sympathetic to it? Their fund, at this point, can make enough off interest on a basic CD to not just pay for every possible cost they could have until the end of time, but make the maintainer(s) obscenely wealthy without breaking a sweat. https://upload.wikimedia.org/wikipedia/foundation/3/3e/Wikim... $250m - they’re doing this out of gre…

This is a rediculous thing to say. You think the interest off 250 million would be enough to run the wikipedia website? Do you have any experience operating web properties at scale to come to this conclusion? Did you base this on anything at all?

I don't agree that it's pure greed but hosting costs for the Wikimedia Foundation in the FY2022-2023 were $3.1 million. [1][2]

[1] https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...

[2] https://wikimediafoundation.org/wp-content/uploads/2023/11/W... (page 4, pdf page 6 for expenses)

Post reply on HN