The Slotted Counter Pattern
planetscale.com
The Slotted Counter Pattern
1–10 of 34 posts
Re: The Slotted Counter Pattern
#2Makes sense, but I can't help but feel it's a solution at the wrong abstraction level. It's a shame the DB can't figure this out for you.
Re: The Slotted Counter Pattern
#3Interesting. This basically implements a parallel algorithm for summation/counting, by calculating a partial sum in each slot, and then merging the results with every read. This approach could be applied more generally to values in a commutative monoid, and there are probably other parallel algorithms that could be implemented in a database a similar way.
Re: The Slotted Counter Pattern
#4Wouldn't querying the count be slow using a WHERE clause?
Re: The Slotted Counter Pattern
#5Makes sense, but I can't help but feel it's a solution at the wrong abstraction level. It's a shame the DB can't figure this out for you.
Yeah, it seems like it would be possible for the DB engine to aggregate all these increments into one update. If you have two increments by one each in the queue, why not make it a single increment by two? I'm not sure though how much computing power it would need to figure that out...
Re: The Slotted Counter Pattern
#6Makes sense, but I can't help but feel it's a solution at the wrong abstraction level. It's a shame the DB can't figure this out for you.
Yeah, it seems like it would be possible for the DB engine to aggregate all these increments into one update. If you have two increments by one each in the queue, why not make it a single increment by two? I'm not sure though how much computing power it would need to figure that out...
wouldn't that break the atomic and isolated rule of ACID?
Re: The Slotted Counter Pattern
#7At the extreme end of this, you could also append a new row for every event and count them. If the number of rows would be too big over some period of time, you could similarly aggregate them occasionally and clear the "scratch" table.
Re: The Slotted Counter Pattern
#8Wouldn't querying the count be slow using a WHERE clause?
it was indexed which should help, but i assume that the reads are far less common than the inserts. reads could even be scheduled/automated and stored in a cache table if they need to be faster and ok being a little stale
Re: The Slotted Counter Pattern
#9Basically same as "Sharding counters" (2008)
https://download.huihoo.com/google/gdgdevkit/DVD1/developers...
Also in Brett Slatkin's "Building Scalable Web Apps with App Engine" (2008)
Re: The Slotted Counter Pattern
#10Basically same as "Sharding counters" (2008) https://download.huihoo.com/google/gdgdevkit/DVD1/developers... Also in Brett Slatkin's "Building Scalable Web Apps with App Engine" (2008) https://youtu.be/Oh9_t5W6MTE?t=1181
Was going to post this. "Sharding" seems like a better term for communicating the idea.