Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

June 2023 Data Dump is missing

meta.stackexchange.com

Post reply on HN
Print view View original

161–170 of 268 posts

Re: June 2023 Data Dump is missing

#161

Post by shagie »

danShumway wrote: post #151
shagie wrote: post #141

Earlier quoted context omitted.

> but it looks increasingly like services will have to lock down content or shut down due to AI models frontrunning them with their own content. This feels like its slightly off to me. An LLM that was trained on job postings to be able to categorize them isn't trying to do job postings ( https://wfhmap.com/algorithm/ ) but rather be able to do meaningful classification of bulk unstructured data. An LLM trained on red…

There are a couple of different motivations a company could have around blocking API access to prevent AI scraping: A) scraping itself is too expensive. I suspect that's probably not the case with SO because they blocked backup . Downloading the database from the Internet Archive doesn't cost SO any money. B) the AI is going to replace the original creators (or more likely, devalue their work and push wages lower) an…

On the B/C test: From Jody Bailey (SO staff) https://meta.stackexchange.com/a/390040 (in full) posted 7 minutes ago (as I write this)

> Stack Overflow senior leadership is working on a strategy to protect Stack Overflow data from being misused by companies building LLMs. While working on this strategy, we decided to stop the dump until we could put guardrails in place.

> We are working on setting up the infrastructure to do this correctly in the age of LLMs --- where we continue to be open and share the data with our developer community but work to set up a formal framework for large AI companies that want to leverage the data.

> We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community. We are working to design and implement appropriate safeguards and still sorting out the details and timelines. We will provide regular updates on our progress to this group.

---

On reddit, for the A/C test... yea, I'm going to be cynical there that they're looking to sell the information (and it isn't so much trying to protect users). But also that 3rd party clients are not showing ads and may be poorly behaved when provided a free API with (what at one time) was generous rate limiting.

Re: June 2023 Data Dump is missing

#162

Post by irrational »

NoZebra120vClip wrote: post #114
slg wrote: post #81

Earlier quoted context omitted.

>Email is a great example where most people wouldn't be interested in a version of email that only let's you email other @gmail.com users. Having a email address that can contact anyone, a phone number that can ring any other phone number etc instead of being locked into a single corporation network is a clear value add that people care about. That is only because those technologies predate those companies. Normal pe…

Surely, normal people get exhausted and burned out with how many accounts they need to create on all the platforms, just to stay in touch with everyone in their circle of friends. It was already crazy for me in the 90s days of Instant Messaging, and pidgin was a Godsend to consolidate all those accounts and friends-list into one interface. Surely normal people hate on the sheer volume of apps they need to install on…

> normal Internet social life

I guess I'm abnormal? They only way I contact any friends/family is text message or email. I've never used Facebook, Instagram, or whatever other platforms people use to "keep up with a normal Internet social life". Yet, I have a plethora of friends and don't feel like I have any problem keeping up with family. Maybe people don't really need these platforms.

Re: June 2023 Data Dump is missing

#163

Post by jzb »

the_pwner224 wrote: post #33
lumb63 wrote: post #16

This, along with recent Reddit goings-on has made me realize a major risk with the current structure of online communication. Take either Reddit or Stack Exchange as examples. They build a platform, and users contribute their time, thought, energy, and knowledge to build a community on that platform. Those companies can then gatekeep and restrict access to all that the community built, when all they did is provide th…

A decentralized system will never work because 99% of users do not care at all; the centralized systems are easier to sign up for and use. It's been demonstrated over and over and over again. Even if the underlying tech is decentralized, the community will settle around one or a few big instances (for example, Gmail and GitHub) which often end up having significant control over the trajectory of the entire ecosystem.…

"It's been demonstrated over and over and over again."

Well, yes. But it's also been demonstrated over and over again the risks of centralized sites. Maybe, just maybe, one of these days that lesson will stick and communities will take a longer term view. It seems like the cycle is happening a bit faster each time now, so maybe folks will get tired of the "damn, time to move to another site, again..." thing.

Or not. Convenience tends to trump lots of other considerations most of the time.

Re: June 2023 Data Dump is missing

#164

Post by richardw »

arp242 wrote: post #132
solardev wrote: post #60

Earlier quoted context omitted.

They're hard to capitalize. You need a skilled founder willing to forego profits, or a group of well intentioned devs with management experience, to organize the business side of things. You have to figure out non investment sources of income (charge for services, donations, etc. maybe no ads or at least no ad networks if you're worried about privacy or ethics). It's a lot of work, especially trying to scale that up…

The difficulty now is that you'll have to compete against Stack Overflow, whereas Stack Overflow had to compete with the paywalled Experts Exchange. Both for-profit and non-profit will have a very hard time unless Stack Overflow really goes off the rails in Quora-style it will probably stick around. Also I think people underestimate how much effort goes in to moderating Stack Overflow, and how delicate the entire sys…

SO also had pretty serious influencers in the tech community to bootstrap everything.

However I’d guess this is existential for them. I’d guess an AI is checking and generating a huge percentage of homework (and work) and that’ll only increase. The place to attack SO is through the chat / code gen UI and git repo’s. New languages and frameworks are probably on GitHub in their most hardened state approximately as quickly as snippets land on SO, and ChatGPT’s upvote is a good indicator of quality sans SO.

Re: June 2023 Data Dump is missing

#165

Post by usr1106 »

abetusk wrote: post #72

As a reminder, all the SE sites have content under a Creative Commons, By Attribution, Share Alike license, allowing for, among other things, commercial re-use [0] [1]. Yes, it sucks that the SE sites are getting more draconian about allowing access to their content but the SE sites are well insulated against it completely disappearing precisely because they're under a libre/free license. Note that Reddit [2], nor HN…

I haven't studied the legalize, but I assume if they put all answers behind a paywall tomorrow nothing can be done. I don't think the license says they must share.

Re: June 2023 Data Dump is missing

#166

Post by danShumway »

pc_edwin wrote: post #134
lumb63 wrote: post #16

This, along with recent Reddit goings-on has made me realize a major risk with the current structure of online communication. Take either Reddit or Stack Exchange as examples. They build a platform, and users contribute their time, thought, energy, and knowledge to build a community on that platform. Those companies can then gatekeep and restrict access to all that the community built, when all they did is provide th…

I believe we'll have more of these oh s** moments soon when people will finally realise why we need web3. Yes the whole space was full of scammers, charlatans but the technology and point was to create a substrate for networks on the internet. The idea that these networks and communities need to run on centralised servers is archaic. The technology exists where people should be able to own their own network (follower…

Let's be honest though, the primary thing that attracted most powerful people to web3 wasn't decentralization, it was reintroduction of artificial scarcity into digital spaces. Web3 billed itself as empowering users, but it always had an undercurrent of commodification and gatekeeping.

And that's exactly why (ignoring the scammers or pump-and-dump businesses) it saw such heavy investment from VC/tech types. The promise they were interested in wasn't democratization even if that's what they told their users -- what they were interested in was taking a plentiful resource (digital bits) and building a scarce asset that they could use to further entrench exclusivity, status, and monopolistic control over what that asset represented.

Read back over every sales pitch for web3 games. At some point they always devolve into talking about how ordinary users will be able to rent seek: to "license" characters/weapons/gear and passively earn income from other players, or to hoard exclusive tokens/releases in the game and speculate on their future value. Web3 looked at infinite digital spaces and its response was, "infinity is a problem that we need to solve." And it's revealing to look at most web3 branded metaverse attempts and see just how quickly they reintroduced real-world concepts like housing/space scarcity (why on earth would we want a housing market in a digital space with no physical constraints?), and how quickly they leaned into cosmetics and customization as a monetization strategy rather than a user right to free expression.

In general, if a technological "paradigm" is primarily associated with and primarily popular with VC firms, it's probably not being developed with the user in mind.

On the other hand: federation, interoperability, mobile identities, and legal efforts to build a right to data export existed independently of web3 and have shown a lot more promise when it comes to actually increasing user agency.

Re: June 2023 Data Dump is missing

#167

Post by karim79 »

What irks me about this is that 100% of their data is provided for free, by the community that they have fostered, the people like myself who have answered > 2500 questions[0], and now SO feels hard-done-by by LLMs using all their hard work to create tools like CodeGPT, GitHub copilot, etc.

Were it really a site for helping developers to improve their skills and increase their productivity through the give-and-take model that SO was, at least once upon a time, SO should perhaps take a deep breath and realise that this might not change a thing apart from causing their contributors to feel like they were never part of it in the first place.

I'm not sure if I've correctly articulated that, but I do find SO's stance to be quite revealing. It feels to me like they're crying foul that ChatGPT and the how many other systems out there are stealing their revenue. None of the contributors (apart from the employee ones, I suppose) ever got paid any currency other than high-fives in the form of rep, medals, the gamified stuff, moderation rights, and at certain rep levels some swag in the form of t-shirts and the usual.

I never wanted any money from SO, but the revelation of this attitude has left me feeling, well, a little sad to say the least.

[0]https://stackoverflow.com/users/70393/karim79

Re: June 2023 Data Dump is missing

#168

Post by jstarfish »

nologic01 wrote: post #116

twitter, reddit, stack overflow... the digital version of burning the library of alexandria it was always a broken system built on dodgy contracts, but it is still sad to see how unceremoniously everything implodes will any lessons be learned? unlikely.

All of our institutions are headed by the likes of Caligula, Nero and Elagabalus so it's only ever a matter of time before the charlatans in charge set it on fire themselves. Never count on anything lasting longer than a year. Motivations can change overnight.

With one exception, there are no instances of anything crowdsourced/community-supported that aren't later paywalled, gatekept or destroyed to prevent exfiltration. It's always an advance-fee scheme. The longer the duration of time, the more the terms are corrupted until the people expecting delivery on the original promise end up being told "what promise?" (The exception is piracy sites. Ironically the illegal nature of the activity seems to keep the owners honest.)

Never work for free, for any promise of long-term future payout, "exposure," or any other bullshit. When they fuck you over--and they will, because you made it so easy--you'll be too broke (and broken) to sue. Every inch, every day you give them is just more time for them to find ways to cheat you.

(You'll learn this lesson the hardest way in making concessions to a high-conflict ex-spouse armed with a 50/50 child custody agreement...they get you to agree to let the kid stay with them during your scheduled time, more and more, until they can prove the kid is basically with them 100% of the time-- then you get slapped with a vastly-increased child support order. You can't claw anything back because they have commitments now. Thus, you get cheated out of both your relationship and your money.)

Re: June 2023 Data Dump is missing

#170

Post by redbell »

b3morales wrote: post #78
lumb63 wrote: post #16

This, along with recent Reddit goings-on has made me realize a major risk with the current structure of online communication. Take either Reddit or Stack Exchange as examples. They build a platform, and users contribute their time, thought, energy, and knowledge to build a community on that platform. Those companies can then gatekeep and restrict access to all that the community built, when all they did is provide th…

This was one of the promises originally of Stack Overflow: all the content is Creative Commons licensed so that if they "turned evil" (I believe it was Joel that put it this way) the community could, in a way, create a fork. https://web.archive.org/web/20230203170609/https://stackover... Unfortunately the dumps themselves are not a legal requirement, just a gentleman's agreement, so realistically exercising this abil…

> This was one of the promises originally of Stack Overflow: all the content is Creative Commons licensed

This reminds me of the promise OpenAI was built on. Unfortunately, it turned out to be a bold claim to be respected and too good to be true [0]

0. https://news.ycombinator.com/item?id=34979981

Post reply on HN