Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When imperfect systems are good: Bluesky's lossy timelines

jazco.dev

Post reply on HN
Print view View original

111–120 of 315 posts

Re: When imperfect systems are good: Bluesky's lossy timelines

#111

Post by artee_49 »

I am a bit perplexed though as to why they have implemented fan-out in a way that each "page" is blocking fetching further pages, they would not have been affected by the high tail latencies if they had not done this,

"In the case of timelines, each “page” of followers is 10,000 users large and each “page” must be fanned out before we fetch the next page. This means that our slowest writes will hold up the fetching and Fanout of the next page."

Basically means that they block on each page, process all the items on the page, and then move on to the next page. Why wouldn't you rather decouple page fetcher and the processing of the pages?

A page fetching activity should be able to continuously keep fetching further set of followers one after another and should not wait for each of the items in the page to be updated to continue.

Something that comes to mind would be to have a fetcher component that fetches pages, stores each page in S3 and publishes the metadata (content) and the S3 location to a queue (SQS) that can be consumed by timeline publishers which can scale independently based on load. You can control the concurrency in this system much better, and you could also partition based on the shards with another system like Kafka by utilizing the shards as keys in the queue to even "slow down" the work without having to effectively drop tweets from timelines (timelines are eventually consistent regardless).

I feel like I'm missing something and there's a valid reason to do it this way.

Re: When imperfect systems are good: Bluesky's lossy timelines

#112

Post by nightpool »

josefresco wrote: post #103
rconti wrote: post #97

Earlier quoted context omitted.

No, overbooking is a business decision justified by the fact that, statistically, not all passengers will actually show up for their flight, and lower load factors cost money.

What is the "no show" rate?

A 2019 study of 5 European airports in 2019 had no-show rates of 14.4%: https://www.ozion-airport.com/product/comparative-analysis-n...

However, my understanding is that airlines have much more sophisticated per-flight and per-passenger models that calculate the predicted no-show factor based on the historical rates for that particular route (e.g. you're more likely to get more no-shows in business class flying from NYC to SF compared to holiday travelers with a reservation on the Florida Keys)

Re: When imperfect systems are good: Bluesky's lossy timelines

#113

Post by artee_49 »

josefresco wrote: post #103
rconti wrote: post #97

Earlier quoted context omitted.

No, overbooking is a business decision justified by the fact that, statistically, not all passengers will actually show up for their flight, and lower load factors cost money.

What is the "no show" rate?

I think you'll have to pay a team millions to figure that out, it is unlikely to be a static rate but rather decided based on multiple traits like time of year, time of flight, distance of flight, cost of ticket, etc.

Re: When imperfect systems are good: Bluesky's lossy timelines

#114

Post by SteveNuts »

nightpool wrote: post #112
josefresco wrote: post #103

Earlier quoted context omitted.

What is the "no show" rate?

A 2019 study of 5 European airports in 2019 had no-show rates of 14.4%: https://www.ozion-airport.com/product/comparative-analysis-n... However, my understanding is that airlines have much more sophisticated per-flight and per-passenger models that calculate the predicted no-show factor based on the historical rates for that particular route (e.g. you're more likely to get more no-shows in business class flying from…

That blows my mind, I would expect maybe 1 or 2 passengers per plane at most. I'm trying to think of what factors would cause that many no-shows, it has to be mostly missed connections?

I can't imagine spending hundreds of dollars and just not showing up.

Re: When imperfect systems are good: Bluesky's lossy timelines

#115

Post by exabrial »

dang wrote: post #9

[stub for offtopicness]

I honestly am annoyed to use websites and services like this. Annoys the crap out of me and everyone else, but since it's petty much forced down their throats, the "eventually" is "eventually everyone stops complaining".

Re: When imperfect systems are good: Bluesky's lossy timelines

#116

Post by autobodie »

gkoberger wrote: post #31
autobodie wrote: post #24

Earlier quoted context omitted.

Absolutely. The profit motive is the root of most evil. It is a shame that so many are trained to believe it is the only motive available.

I completely agree with this... but without profit, people can't get paid, and they'll stop building. I do hate this incredibly need for growth, of course, but financial growth is necessary to pay people and give them raises and allow them to have upward mobility at the company. I hope Bluesky is able to find a model that works for them AND for consumers. (I do know it's an open protocol, so it'll live on without Blu…

> but without profit, people can't get paid, and they'll stop building

I wholeheartledly disagree. People build things all the time for things other than profit. In fact, most of the greatest things ever built were a loss for those who built them.

Dignity is the best motivator. Profit only supercedes dignity when dignity is not on offer.

Re: When imperfect systems are good: Bluesky's lossy timelines

#117

Post by anamexis »

immibis wrote: post #75
verdverm wrote: post #11

Earlier quoted context omitted.

It's not that it is "better" but that the choice is individual, not up to the mastodon server. In Mastodon, you trade Elon for some other group of individuals, so what happens if they make decision on moderation or content you do not agree with? ATProto is designed around accounts that are independent of data host, application, and moderation, all in the name of giving users individual control over these things. It's…

You have the opportunity to demonstrate this. I am banned from Bluesky. (They didn't tell me why - just a generic "you violated community guidelines") Tell me, concretely, how people can choose to continue following me, even though I am banned. Profile: immibis.bsky.social

Follow the instructions under "Self-hosting PDS" here: https://github.com/bluesky-social/pds

Re: When imperfect systems are good: Bluesky's lossy timelines

#118

Post by pphysch »

jsnell wrote: post #96
timewizard wrote: post #82

> This process involves looking up all of your followers, then inserting a new row into each of their Timeline tables in reverse chronological order with a reference to your post. Seriously? Isn't this the nut of your problem right here?

What alternative design did you have in mind, given that a Twitter-like data model of individual follows is likely a strict product requirement? There are obviously other ways of doing it (doing the timeline propagation in a batch job, fanning out the reads rather than the writes), but they've got their own problems. Probably worse ones.

Wouldn't a hybrid approach makes sense?

Periodically classify users as hot/cold based on their activity, build hot-follower timelines on write, and build cold-follower timelines on read.

Re: When imperfect systems are good: Bluesky's lossy timelines

#119

Post by abound »

artee_49 wrote: post #111

I am a bit perplexed though as to why they have implemented fan-out in a way that each "page" is blocking fetching further pages, they would not have been affected by the high tail latencies if they had not done this, "In the case of timelines, each “page” of followers is 10,000 users large and each “page” must be fanned out before we fetch the next page. This means that our slowest writes will hold up the fetching a…

I interpreted this as a batch write, e.g. "write these 10k entries and then come back". The benefit of that is way less overhead versus 10k concurrent background routines each writing individual rows to the DB. The downside is, as you've noted, that you can't "stream" new writes in as older ones finish.

There's a tradeoff here between batch size and concurrency, but perhaps they've already benchmarked it and "single-threaded" batches of 10k writes performed best.

Re: When imperfect systems are good: Bluesky's lossy timelines

#120

Post by [deleted] »

autobodie wrote: post #116
gkoberger wrote: post #31

Earlier quoted context omitted.

I completely agree with this... but without profit, people can't get paid, and they'll stop building. I do hate this incredibly need for growth, of course, but financial growth is necessary to pay people and give them raises and allow them to have upward mobility at the company. I hope Bluesky is able to find a model that works for them AND for consumers. (I do know it's an open protocol, so it'll live on without Blu…

> but without profit, people can't get paid, and they'll stop building I wholeheartledly disagree. People build things all the time for things other than profit. In fact, most of the greatest things ever built were a loss for those who built them. Dignity is the best motivator. Profit only supercedes dignity when dignity is not on offer.

[deleted]
Post reply on HN