Delivering Billions of Messages Exactly Once
segment.com
Delivering Billions of Messages Exactly Once
1–10 of 141 posts
Re: Delivering Billions of Messages Exactly Once
#2... or they could have used BigQuery with a primary key on message ID.
Re: Delivering Billions of Messages Exactly Once
#3Site seems to be down. Any ideas how big these HN hugs of death usually are? How big of a traffic spike brings these servers down?
Re: Delivering Billions of Messages Exactly Once
#4Site seems to be down. Any ideas how big these HN hugs of death usually are? How big of a traffic spike brings these servers down?
It's not down
Re: Delivering Billions of Messages Exactly Once
#5Isn't the new feature of Kafka about this?
Re: Delivering Billions of Messages Exactly Once
#6"The single requirement of all data pipelines is that they cannot lose data."
Unless the business value of data is derived after applying some summary statistics, than even sampling the data works, and you can lose events in an event stream, while not changing the insight gained. Originally Kafka was designed to be a high throughput data bus for analytical pipeline where losing messages was ok. More recently they are experimenting with exactly once delivery.
Re: Delivering Billions of Messages Exactly Once
#7Site seems to be down. Any ideas how big these HN hugs of death usually are? How big of a traffic spike brings these servers down?
hm seems to be up for me? you might've caught our autoscaler mid-scaling!
Re: Delivering Billions of Messages Exactly Once
#8Site seems to be down. Any ideas how big these HN hugs of death usually are? How big of a traffic spike brings these servers down?
It's not down
Hmm weird. I get: This site can’t be reached
segment.com refused to connect.
Try:
Checking the connection
ERR_CONNECTION_REFUSED
Maybe something local to me only?
Re: Delivering Billions of Messages Exactly Once
#9Would something like AWS SQS not scale for something like this? We currently push about 25k daily transactions over SQS, obviously no where near the scale of this, just wondering about what limitations we will bump into potentially.
Re: Delivering Billions of Messages Exactly Once
#1052 requests, 5.4 MB and 8.63 seconds to load a simple blog post. With a bonus XHR request every 5 seconds.