Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bluesky Social Dataset (235M posts from 4M users)

zenodo.org

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Bluesky Social Dataset (235M posts from 4M users)

#31

Post by nl »

ilrwbwrkhv wrote: post #25
infotainment wrote: post #10

I’m glad to see a new platform that isn’t completely locked down, allowing analysis like this. The trend toward everything being a walled garden is unfortunate.

[flagged]

This is wonderful! Openness means open to all.

And Open data decreases barriers to entry.

If all data was in a state where people had to pay for it then only large companies can use it. With open data it's very foreseeable in 10 years time it will be very likely a hobbyist can train a performant LLM at home from scratch.

Re: Bluesky Social Dataset (235M posts from 4M users)

#32

Post by nl »

mplewis wrote: post #29
jph00 wrote: post #26

Earlier quoted context omitted.

Actually, since this isn’t locked up by the big copyright holders, we can all use it and profit.

How will you use it to profit? You don’t have sweetheart cloud deals on ML training clusters. This benefits big players, not us.

"us" is relative.

There are plenty of people on HN who have their own ML training clusters and aren't really big tech. For example natfriedman has https://andromeda.ai/

And right now, today I can fine tune LLMs on this scale of data at home. In 5 or 10 years people will be able to training from scratch at home.

Computational resource barriers are temporary. Licensing is forever.

Re: Bluesky Social Dataset (235M posts from 4M users)

#33

Post by afavour »

infotainment wrote: post #10

I’m glad to see a new platform that isn’t completely locked down, allowing analysis like this. The trend toward everything being a walled garden is unfortunate.

I’m conflicted. I agree with everything you say but I’m concerned about Bluesky eventually being flooded with AI posts trained on its public dataset. Being open could very easily lead to downfall.

Re: Bluesky Social Dataset (235M posts from 4M users)

#34

Post by tptacek »

afavour wrote: post #33
infotainment wrote: post #10

I’m glad to see a new platform that isn’t completely locked down, allowing analysis like this. The trend toward everything being a walled garden is unfortunate.

I’m conflicted. I agree with everything you say but I’m concerned about Bluesky eventually being flooded with AI posts trained on its public dataset. Being open could very easily lead to downfall.

I feel like LLM models have had the opportunity to be trained on a sufficient amount of social media posts at this point that it's unlikely to matter.

Re: Bluesky Social Dataset (235M posts from 4M users)

#35

Post by afavour »

tptacek wrote: post #34
afavour wrote: post #33

Earlier quoted context omitted.

I’m conflicted. I agree with everything you say but I’m concerned about Bluesky eventually being flooded with AI posts trained on its public dataset. Being open could very easily lead to downfall.

I feel like LLM models have had the opportunity to be trained on a sufficient amount of social media posts at this point that it's unlikely to matter.

I’m curious to see how things look, say, ten years from now. The way people use social networks, the language they use, even the memes they trade in changes over time. I can absolutely imagine an out of date AI giving themselves away by repeating todays equivalent of “rawr xD” to a future audience.

Re: Bluesky Social Dataset (235M posts from 4M users)

#40

Post by verdverm »

ks2048 wrote: post #20
yawnxyz wrote: post #4

Earlier quoted context omitted.

I wonder how much time it takes to run this / what the script is / how resource intensive it is? Bsky is public right, so do you get rate limited? Do you scrape or use an official API? So many questions Also, I feel like only recently there's been an influx of people who have actually interesting things to say so I'd love to see nextyear's dataset

I was checking out the Python API today (the "firehouse" via "atproto" package) and got 5000 posts in 7.5 seconds.

I believe they are enabling(ed?) filters so you can control how much and what you actually get from the firehose
Post reply on HN