111–120 of 154 posts
Projects tend to gain more and more functionality to match the new workloads they're being used to accomplish, and it must certainly be a difficult decision for project visionaries. Do I listen to my users and implement features that will solve their new woes, but in return accept increased complexity and higher learning barriers? Complexity sucks, but it's even harder to say no to users in pain. I wonder if this ope…
I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…
Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.
Example:
c - h72gsb32 - 0000 - udoc - l363eofy
The groups, in order, are:
1. 'c' - identifies this as a cuid, and allows you to use it in html entity ids. The fixed value helps keep the ids sequential.
2. Timestamp
3. Counter - a single process might generate the same random string. The weaker the pseudo-random source, the higher the probability. That problem gets worse as processors get faster. The counter will roll over if the value gets too big.
4. Client fingerprint. For example, the first two chars are extracted from the process.pid. The next two chars are extracted from the hostname.
5. Pseudo random (Math.random())
One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…
No not really. It's called waterfall and we had it for decades.
And it was largely abandoned in favour of agile development because despite architects spending months designing something there was always something that was left out. Or the scope changed during those months. Or millions of other things that change whilst your hidden away from your customers/end users instead of delivering them new value every week.
For me personally the truth is somewhere between agile and waterfall. Some upfront design but not the slow code you refer to.
1. Consider adding an example for a stateful event stream processor client that saves the last read stream offset in redis, together with its current state and continues reading from that offset as an atomic operation. For example, a client that sums a stream of numbers, in order to have effectively once semantics would need to persist to redis the sum and offset together.
2. Consider adding a stream read deduplication example to mitigate clients that reinserted the same event twice. It is not clear how the client should behave if it didn't get an ack and it resents an event. What is the correct resending semantics so the reader would effectively dedup? What is the right data structure used to dedup message ids without consuming too much memory, etc...?
I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…
The dot doesn't make that a decimal, any more than it makes IP addresses or version numbers decimals. As for treating them as decimals inadvertently, well, hopefully client libraries will expose IDs as pairs of integers, not as strings. If users convert them into strings and then back into meaningless pseudo-decimals, well, great, we'll have an entertaining post about someone's outage to read.
I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…
The dot doesn't make that a decimal, any more than it makes IP addresses or version numbers decimals. As for treating them as decimals inadvertently, well, hopefully client libraries will expose IDs as pairs of integers, not as strings. If users convert them into strings and then back into meaningless pseudo-decimals, well, great, we'll have an entertaining post about someone's outage to read.
One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…
> This is how actual quality software that will stand the test of time gets designed and made .. No not really. It's called waterfall and we had it for decades. And it was largely abandoned in favour of agile development because despite architects spending months designing something there was always something that was left out. Or the scope changed during those months. Or millions of other things that change whilst y…
Have you considered adding a semantic layer inside streams that allows each client to consume a substream? In effect the stream becomes multiplexed substreams.
If substreams makes the design too complex... have you considered server side stream 403 semantics? When a stream is manually deprecated it enters an immutable state and provides a redirect response with a link to another stream. This would allow multiplexing and demultiplexing streams without changing the client implementations too much.
For completeness I would state the obvious when fifo grouping is needed: 1. Scaling stateful event processing by splitting streams and adding more clients (CPU limit) 2. Scaling cross region replication by splitting streams and adding more tcp connections (network limit) 3. Handling more throughput by splitting a stream into two redis nodes (disk I/O limit)
many clients will even not need that timestamp anyways.
Earlier quoted context omitted.
Because they do not mean a position, but a special ID.
It seems that "$" is a special ID for the last message, as opposed to the last possible message. I would humbly suggest that "^" would be a suitable symbol for the first message in a stream. ^ and $ are used in regex (and vim) in a similar way. That way you could write "XREAD BLOCK 5000 STREAMS newstream ^" and get all the messages in a stream from the beginning, and then block until a new message comes in all with a…