Live data from Hacker News

I scraped all of OpenAI's Community Forum

julep-ai.github.io

21–30 of 68 posts

Re: I scraped all of OpenAI's Community Forum

#22

I did a bit of data scraping for fun in the past, but I was never quite sure of the legality of what I was doing. What if I was breaking some law in some jurisdiction of some country? Was someone going to track me down and punish me? OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

A precursor to this would have been that Linkedin lawsuit Microsoft lost, allowing that one company to scrape all of Linkedin (technically "public information").

Re: I scraped all of OpenAI's Community Forum

#23

I did a bit of data scraping for fun in the past, but I was never quite sure of the legality of what I was doing. What if I was breaking some law in some jurisdiction of some country? Was someone going to track me down and punish me? OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

We were really heading someplace with The Semantic Web aka The Real Web 3.0 [1]

Alas we have to fight against the machines in order to properly read the internet thru machines.

I believe Discourse knowingly keeps its data easy to scrape though, so kudos to them!

[1]: https://en.wikipedia.org/wiki/Semantic_Web

Re: I scraped all of OpenAI's Community Forum

#24

(disclaimer: I work for Discourse) Discourse has an AI plugin that admins can run on their community to generate their own sentiment analysis (among other things), though it's not quite as thorough as this write up! https://meta.discourse.org/t/discourse-ai-plugin/259214 We're always interested to see how public data can be used like this. It's something that can be a lot more difficult on closed platforms.

> helps you keep tabs on your community by analyzing posts and providing sentiment and emotional scores to give you an overall sense of your community for any period of time [...]

> Toxicity can scan both new posts and chat messages and classify them on a toxicity score across a variety of labels

Is that within the defined data processing purposes of all Discourse setups? Does the tool warn admins they might need to update their policies before being able to run this tool, perhaps needing to seek consent (depending on their jurisdiction and ethics)? It sounds somewhat objectionable, trying to guess my mental state from what I write without opt-in

Edit: and apparently it also tries to flag NSFW chat messages, does Discourse have PM chats where this would flag private messages for admins to read or is it only public chats that this bot runs on?

> tagging NSFW image content in posts and chat messages

Re: I scraped all of OpenAI's Community Forum

#25

I did a bit of data scraping for fun in the past, but I was never quite sure of the legality of what I was doing. What if I was breaking some law in some jurisdiction of some country? Was someone going to track me down and punish me? OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

> OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

Cloudflare gives a shit.

My household had to use our 5G internet for most things for a week or two until our IP reputation recovered.

Re: I scraped all of OpenAI's Community Forum

#26
post #25

I did a bit of data scraping for fun in the past, but I was never quite sure of the legality of what I was doing. What if I was breaking some law in some jurisdiction of some country? Was someone going to track me down and punish me? OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

> OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like. Cloudflare gives a shit. My household had to use our 5G internet for most things for a week or two until our IP reputation recovered.

Yeah it’s probably worth renting a server if there’s any doubt about whether it’s wholly appropriate to do something

Re: I scraped all of OpenAI's Community Forum

#28
post #4

Love it, just for the sole reason of turning something OpenAI made into a dataset for everyone else :D

I don’t think OpenAI are gonna lose any sleep over this.

Isn’t a “community forum” like this basically just: “we’re not gonna spend money on providing adequate customer support so instead here is a forum where y’all can talk amongst yourselves and we’ll give you some badges and imaginary points for doing the customer support yourselves”?

Re: I scraped all of OpenAI's Community Forum

#29
post #4

Love it, just for the sole reason of turning something OpenAI made into a dataset for everyone else :D

I don’t think OpenAI are gonna lose any sleep over this. Isn’t a “community forum” like this basically just: “we’re not gonna spend money on providing adequate customer support so instead here is a forum where y’all can talk amongst yourselves and we’ll give you some badges and imaginary points for doing the customer support yourselves”?

They probably just sic a customer service GPT on it and use it to train the other ones...
Post reply on HN