Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When imperfect systems are good: Bluesky's lossy timelines

jazco.dev

Post reply on HN
Print view View original

141–150 of 315 posts

Re: When imperfect systems are good: Bluesky's lossy timelines

#141

Post by crabbone »

Anecdotally, I ran into a similar solution "by chance".

Long ago, I worked for a dating site. Our CTO at the time was a "guest of honor" who was brought in by a family friend who was working in the marketing at the time. The CTO was a university professor who took on a job as a courtesy (he didn't need the money nor fame, he had enough of both, and actually liked teaching).

But he instituted a lot of experimental practices in the company. S.a. switching roles every now and then (anyone in the company could apply for a different role except administration and try themselves wearing a different hat), or having company-wide discussions of problems where employees would have to prepare a presentation on their current work (that was very unusual at the time, but the practice became more institutional in larger companies afterwards).

Once he announced a contest for the problem he was trying to solve. Since we were building a dating site, the obvious problem was matching. The problem was that the more properties there were to match on, the longer it would take (beside other problems that is). So, the program was punishing site users who took time to fill out the questionnaires as well as they could and favored the "slackers".

I didn't have any bright ideas on how to optimize the matching / search for matches. So, ironically, I asked "what if we just threw away properties beyond certain threshold randomly?" I was surprised that my idea received any traction at all. And the answer was along the lines of "that would definitely work, but I wouldn't know how to explain this behavior to the users". Which, at the time, I took to be yet another eccentricity of the old man... but hey, the idea stuck with me for a long time!

Re: When imperfect systems are good: Bluesky's lossy timelines

#142

Post by chrisweekly »

ChuckMcM wrote: post #135
an_ko wrote: post #83

Earlier quoted context omitted.

> It was acquired by IBM in March 2015, and the service was discontinued. — https://en.wikipedia.org/wiki/Blekko Perhaps GP has a more interesting answer though.

That's the correct answer, IBM wanted the crawler mostly to feed Watson. Building a full search engine (crawler, indexer, ranker, API, web application) for the English language was a hell of an accomplishment but by the time Blekko was acquired Google was paying out tens of billions of dollars to people to send them and only them their search queries. For a service that nominally has to live on advertising revenue ge…

Not my Q but thanks for the interesting history.

Also, (for other readers), I'm a huge fan of Kagi. Highly recommended.

Re: When imperfect systems are good: Bluesky's lossy timelines

#143

Post by cavisne »

AWS has a cool general approach to this problem (one badly behaving user effecting others on their shard)

https://aws.amazon.com/builders-library/workload-isolation-u...

The basic idea is to assign each user to multiple shards, decreasing the changes of another user sharing all their shards with the badly behaving user.

Fixing this issue as described in the article makes sense, but if they did shuffle sharding in the first place it would cover any new issues without effecting many other users.

Re: When imperfect systems are good: Bluesky's lossy timelines

#144

Post by Retr0id »

sphars wrote: post #69

When I go directly to a user's profile and see all their posts, sometimes one of their posts isn't in my timeline where it should be. I follow less than 100 users on Bluesky, but I guess this explains why I occasionally don't see a user's post in my timeline. Lossy indeed.

If another user you follow reposted or replied to a post, it can affect its order in your following feed. You shouldn't be seeing any loss as described in the article from following only 100 users.

Re: When imperfect systems are good: Bluesky's lossy timelines

#145

Post by fc417fc802 »

verdverm wrote: post #44
fc417fc802 wrote: post #27

Earlier quoted context omitted.

Are you suggesting the "big few" can't largely censor a given account? I don't see how ATProto is doing noticeably better than the scenario where a large ActivityPub instance blocks your external account.

Generally, yes. Currently, because Bluesky requires the use of their labeler if you use their app, this could happen. Two points of note 1. You can participate in Bluesky without the Bluesky app, so you can remove this requirement by using an alternative app 2. The most blocked account is blocked by around 0.25% of the full network ( https://clearsky.app/ ) This second point does not account for users banned from Blu…

> does not account for users banned from Bluesky by Bluesky for breaking the ToS or PDS abuse.

Then you are missing the point. I am asking how much censorship power the largest node in the network has.

If being blocked by the largest provider means 95% of users can't see me anymore then the situation is strictly worse than Mastodon vs ActivityPub-at-large.

Re: When imperfect systems are good: Bluesky's lossy timelines

#146

Post by pornel »

I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served).

When you have a celebrity account, instead of fanning out every message to millions of followers' timelines, it would be cheaper to do nothing when the celebrity posts, and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline. When millions of followers do that, it will be cheap read-only fetch from a hot cache.

Re: When imperfect systems are good: Bluesky's lossy timelines

#147

Post by ericvolp12 »

pornel wrote: post #146

I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served). When you have a celebrity account, instead of fanning out every message to millions of followers' timelines, it would be cheaper to do nothing when the celebrity posts, and later…

This is probably what we'll end up with in the long-run. Things have been fast enough without it (aside from this issue) but there's a lot of low-hanging fruit for Timelines architecture updates. We're spread pretty thin from a engineering-hours standpoint atm so there's a lot of intense prioritization going on.

Re: When imperfect systems are good: Bluesky's lossy timelines

#148

Post by pjc50 »

gamedever wrote: post #134
PaulHoule wrote: post #85

An airline reservation system has to be perfect (no slack in today's skies), a hotel reservation can be 98% perfect so long as there is some slack and you don't mind putting somebody up in a better room than they paid for from time to time. A social media system doesn't need to be perfect at all. It was clear to me from the beginning that Bluesky's feeds aren't very fast, not like they are crazy slow, but if it saves…

Miscommunication leads to bad outcomes. One missed message out of order could easily lead to a fight, a lawsuit, a flash mob, threats of violence - that then need to be taken seriously, swatting, DOXxing, etc... Msg 1: I hate ___insert_controversal_person_category_here___ Msg 2: Is the kind of statement that really sets me off Msg 1 has a very different meaning if you don't see Msg 2.

This can already happen without help from the platform.

Re: When imperfect systems are good: Bluesky's lossy timelines

#150

Post by jsnell »

pphysch wrote: post #118
jsnell wrote: post #96

Earlier quoted context omitted.

What alternative design did you have in mind, given that a Twitter-like data model of individual follows is likely a strict product requirement? There are obviously other ways of doing it (doing the timeline propagation in a batch job, fanning out the reads rather than the writes), but they've got their own problems. Probably worse ones.

Wouldn't a hybrid approach makes sense? Periodically classify users as hot/cold based on their activity, build hot-follower timelines on write, and build cold-follower timelines on read.

You'd still have exactly the same hot write path, it'd just have maybe 50% of the load. That could be a legit optimization, but not having it hardly warrants an incredulous "seriously?" like the OP's.

(And the same for the inverse hybrid strategy of quarantining the writes of highly followed users and handling their fan-out at read time. A neat optimization, and maybe even absolutely once you have accounts with 100M followers. But the vast majority of posts would still be handled via the original strategy.)

Post reply on HN