Live data from Hacker News

I scraped all of OpenAI's Community Forum

julep-ai.github.io

11–20 of 68 posts

Re: I scraped all of OpenAI's Community Forum

#11

I didn't even knew they have community forums. Looking at the main homepage (openai.com), the only external links I can find are to chatgpt and their docs hosted on platform.openai.com. The other links lead to their socials, github and soundcloud (of all places). Maybe I'm not looking thoroughly enough, so I may be wrong, tho!

I would also love to see these forums both to post and to lurk

Re: I scraped all of OpenAI's Community Forum

#12
(disclaimer: I work for Discourse)

Discourse has an AI plugin that admins can run on their community to generate their own sentiment analysis (among other things), though it's not quite as thorough as this write up! https://meta.discourse.org/t/discourse-ai-plugin/259214

We're always interested to see how public data can be used like this. It's something that can be a lot more difficult on closed platforms.

Re: I scraped all of OpenAI's Community Forum

#14
I did a bit of data scraping for fun in the past, but I was never quite sure of the legality of what I was doing. What if I was breaking some law in some jurisdiction of some country? Was someone going to track me down and punish me?

OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

Re: I scraped all of OpenAI's Community Forum

#15
That's super cool, thanks for sharing! I will share this as an easy to follow example of what we can with AI.

> Allowing a Q&A interface using these embeddings over the post contents could speed up research over the community posts (if you know the right questions to ask :P). Let's view some posts similar to this one complaining about function calling

That's indeed a great thing to surface, and that's exactly how the the OpenAI forum selects the "Related Topics" to show at the end of every topic. We use embeddings for this feature, and the entire thing is open-source: https://github.com/discourse/discourse-ai/blob/main/lib/embe...

We also embeddings for suggesting tags, categories, HyDE search and more. It's by far my favorite tech of this new AI/ML gen so far in terms of applicability.

> Using Twitter-roBERTa-base for sentiment analysis, we generated a post_sentiment label (negative, positive, neutral) and post_sentiment_score confidence score for each post.

We do the same, with even the same model, and conveniently show that information on the admin interface of the forum. Again all open source: https://github.com/discourse/discourse-ai/tree/main/lib/sent...

Disclaimer: I'm the tech lead on the AI parts of Discourse, the open source software that powers OpenAI's community forum.

Re: I scraped all of OpenAI's Community Forum

#16

I didn't even knew they have community forums. Looking at the main homepage (openai.com), the only external links I can find are to chatgpt and their docs hosted on platform.openai.com. The other links lead to their socials, github and soundcloud (of all places). Maybe I'm not looking thoroughly enough, so I may be wrong, tho!

I would also love to see these forums both to post and to lurk

https://community.openai.com/ (when you are logged in on platform.openai.com, there is a link from the menu)

Re: I scraped all of OpenAI's Community Forum

#17

I did a bit of data scraping for fun in the past, but I was never quite sure of the legality of what I was doing. What if I was breaking some law in some jurisdiction of some country? Was someone going to track me down and punish me? OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

Do you think it would be better if someone did track you down and punish you? Which world do you want to live in?

Re: I scraped all of OpenAI's Community Forum

#18

> Number of Posts with negative sentiment, grouped by Topic > # 1 Result: Python Packaging Checks out

A pro-tip for using the OpenAI API is to not use the official Python package for interfacing with it. The REST API documentation is good, and just using it in your HTTP client of choice like requests is roughly the same LOC without unexpected issues, along with more control.

Re: I scraped all of OpenAI's Community Forum

#19

I did a bit of data scraping for fun in the past, but I was never quite sure of the legality of what I was doing. What if I was breaking some law in some jurisdiction of some country? Was someone going to track me down and punish me? OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

Do you think it would be better if someone did track you down and punish you? Which world do you want to live in?

I think large companies should be punished for stealing from people to make themselves richer.

Re: I scraped all of OpenAI's Community Forum

#20

> Number of Posts with negative sentiment, grouped by Topic > # 1 Result: Python Packaging Checks out

A pro-tip for using the OpenAI API is to not use the official Python package for interfacing with it. The REST API documentation is good, and just using it in your HTTP client of choice like requests is roughly the same LOC without unexpected issues, along with more control.

I've found this happens with a lot of first party clients. At work, we use LaunchDarkly for feature flags and use their code references tool to keep track of where flags are being referenced. The tool uses their first party Go client to interact with the API but the client doesn't handle rate limiting at all even though they have rate limiting headers clearly documented for their API.
Post reply on HN