Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter shut off API access; users volunteering their own data for an open API

omarshehata.substack.com

Post reply on HN
Print view View original

81–90 of 106 posts

Re: Twitter shut off API access; users volunteering their own data for an open API

#81

Post by bangaladore »

nicbou wrote: post #80
bangaladore wrote: post #50

Here's a thought: someone "trustworthy" should maintain a Chrome extension or Tapermonkey script that automatically scrapes data from various social media sites in a fully anonymized fashion. As people browse Twitter, Reddit, or XYZ, the posts/comments are sent to some aggregation system. It might be against TOS, but certainly far less than scraping, and you couldn't tell, as it's the user driving what gets scraped.…

I was thinking of that exact strategy for two scraping problems I have. It would be a good way to gently scrape Berlin.de for appointments, and Immobilienscout24 for new flats. From what I can tell, it would work fine so long as the user is actively looking at those pages.

Yeah, something generic to work for any use case would be nice, but privacy becomes more difficult as you need to tailor the situation to each site to maintain privacy (i.e. only pull the information on the apartment/flat listing, or public tweet or reddit comment, etc...)

Re: Twitter shut off API access; users volunteering their own data for an open API

#82

Post by smileybarry »

wcarss wrote: post #45
smileybarry wrote: post #43

Earlier quoted context omitted.

It's gotten dramatically worse since premium started rewarding payouts for engagement/views, alongside zero moderation. Now there's an actual incentive to be a jerk, a spammer, and make repeated bad faith arguments / "just asking questions" -- people reply, your tweet is seen by more people, you get paid a share of ad revenue. Then you have the regurgitated messages in replies (AI or otherwise) endlessly copied and d…

Why not, um... stop using it? Mastodon has a lot of great people, and a lot less of all this stuff.

I have a Mastodon account, a Bluesky account, and apps for both. Mastodon doesn’t replace the comedy part of Twitter, and Bluesky is great but 99% of local social media is still on Twitter. So I’m “trapped” on Twitter until they move to Bluesky, but they won’t because a part of that crowd is addicted to the revenue share off their political tweets.

Re: Twitter shut off API access; users volunteering their own data for an open API

#84

Post by mewpmewp2 »

bangaladore wrote: post #81
nicbou wrote: post #80

Earlier quoted context omitted.

I was thinking of that exact strategy for two scraping problems I have. It would be a good way to gently scrape Berlin.de for appointments, and Immobilienscout24 for new flats. From what I can tell, it would work fine so long as the user is actively looking at those pages.

Yeah, something generic to work for any use case would be nice, but privacy becomes more difficult as you need to tailor the situation to each site to maintain privacy (i.e. only pull the information on the apartment/flat listing, or public tweet or reddit comment, etc...)

I wonder if those websites as a response might start adding fingerprint code in the source to indicate whose user session it was.

Re: Twitter shut off API access; users volunteering their own data for an open API

#85

Post by electrondood »

yodsanklai wrote: post #24
criticalfault wrote: post #9

Users should just get off this continued tragedy and API access wouldn't be an issue

To me, Twitter is by far the most toxic social network. It has the power to turn the most interesting and smartest people into bitter trolls. Maybe some people get value from it, but it requires serious self discipline to not get dragged into stupid arguments.

It's intentionally engineered to be as emotionally provocative as possible, to drive engagement, to drive and revenue.

My For You feed kept turning into videos of public fights and Karen's calling the cops, etc. no matter what feedback I gave or what I clicked on.

Deleted it and haven't looked back. Won't click on a link from x.com either.

Re: Twitter shut off API access; users volunteering their own data for an open API

#87

Post by miki123211 »

shadowgovt wrote: post #51
miki123211 wrote: post #42

Earlier quoted context omitted.

Twitter was "special" because they actually had an API. No other social media site (besides Reddit) had one. Facebook kind of tried, although theirs was always a lot more limited, and got shut down when it turned out Cambridge Analytica used it for widespread election fraud[1]. Both Twitter and Reddit did basically the same thing at roughly the same time, Twitter just had the misfortune to be under the control of Elo…

Indeed. CA's impact on election outcomes was likely negligible; Antonio Martínez's "Chaos Monkeys" makes the case that Trump using CA was more indicative of the overall notion that Trump's team was spending money to try a bit of everything (online and offline) and the Clinton campaign wasn't. Their tactical failure was believing they could redirect the money to down-ticket races because Clinton / Trump was such an ob…

What Mastodon is doing seems suitably ironic in this situation.

For those unaware, Mastodon's APIs are extremely open and it's very easy to scrape, to the point of providing you with a "firehose" of all public posts that an instance sees, both local and federated, with no authentication required. They also have an extreme anti-scraping culture, anybody who admits to running any kind of scraper which is not strictly opt-in, even for benign / scientific purposes, is very quickly shunned and blocked. Most instances also have a "disallow scraping via robots.txt" policy by default.

The results? I posted a canary token[1] link on a medium-sized, well-federated, well-protected instance which disallows scraping, and it got hit by some shady social media crawler in a fraction of a second. It started getting hit by many other strange crawlers later on, and it still keeps getting visits (mostly from Google now).

Re: Twitter shut off API access; users volunteering their own data for an open API

#88

Post by shadowgovt »

miki123211 wrote: post #87
shadowgovt wrote: post #51

Earlier quoted context omitted.

Indeed. CA's impact on election outcomes was likely negligible; Antonio Martínez's "Chaos Monkeys" makes the case that Trump using CA was more indicative of the overall notion that Trump's team was spending money to try a bit of everything (online and offline) and the Clinton campaign wasn't. Their tactical failure was believing they could redirect the money to down-ticket races because Clinton / Trump was such an ob…

What Mastodon is doing seems suitably ironic in this situation. For those unaware, Mastodon's APIs are extremely open and it's very easy to scrape, to the point of providing you with a "firehose" of all public posts that an instance sees, both local and federated, with no authentication required. They also have an extreme anti-scraping culture, anybody who admits to running any kind of scraper which is not strictly o…

It's an ecosystem of people that seem insistent on the idea that you can put stuff online behind no authentication wall and expect to stay exclusively on servers you believe it should be on.

And I don't know what to tell them, because that's not how the internet has ever worked.

Re: Twitter shut off API access; users volunteering their own data for an open API

#89

Post by shiroiushi »

CaptainFever wrote: post #72
wcarss wrote: post #45

Earlier quoted context omitted.

Why not, um... stop using it? Mastodon has a lot of great people, and a lot less of all this stuff.

Not parent, but I did use Mastodon for about a year. I ended up moving back to Twitter because: 1. I just couldn't vibe with the culture there. From my POV, Mastodon is made out of pearl-clutchers and politics. 2. So much drama. The FediSearch drama. The Raspberry Pi incident. It's just so tiring and you feel like you need to walk on eggshells all the time. 3. So much drama. You would just pray that your admin didn't…

>4. So much drama. Pray that your server doesn't shut down, because you can't import your posts elsewhere.

I would have thought a well-designed decentralized system would allow you to, for instance, download/export all your posts, and then import them all to a new server, in case you have to move servers for some reason.

Re: Twitter shut off API access; users volunteering their own data for an open API

#90

Post by shiroiushi »

asdff wrote: post #74
criticalfault wrote: post #9

Users should just get off this continued tragedy and API access wouldn't be an issue

People should do a lot of things. They shouldn't smoke and they should work out, but here we are in 2024 where the phillip morris stock has outperformed ibm over the last 5 years and the obesity crisis shows no signs of letting up. Knowing something is bad is the first step of course but clearly not enough to drive a real behavioral change, and we aren't even fully in agreement as a society that social media is harmf…

>we aren't even fully in agreement as a society that social media is harmful like how cigarettes or obesity are known to be harmful.

There's more than one society on Earth, so it's hard to say that "society" "knows" that smoking is bad without specifying the society. As an American who lives outside the US, one striking place where the US clearly outperforms peer nations, in my daily experience, is with smoking: Americans just don't smoke any more unless they're poor and uneducated. There's no smoking inside (except bars), there's no smoking within XX feet of business entrances, etc. In the US, smokers have really become pariahs since the 80s and early 90s. Some of that has been undone with the rise of "vaping", but still, if you hang out with college-educated people, you probably won't meet any smokers. It's just not like that in other countries: people of all social classes still smoke. Here in Japan, it's quite common to see some salaryman riding a bicycle while puffing on a cigarette, and many restaurants have smoking areas (usually sealed off with separate ventilation in modern buildings). There's even some old locals-only eateries that have (non-separate) smoking inside. Europe isn't any better.

As for obesity, only truly delusional Americans try to claim that "fat is healthy". All the rest know it isn't, even if they're obese, but the society makes it very hard to not be overweight, mainly because everyone drives cars and doesn't walk.

Post reply on HN