61–70 of 154 posts
Why not just use sequence ID? I'm confused about why a timestamp is important. The sequence ID gives us ordering, is always guaranteed to be increasing.
Because with the way stream IDs are conceived you also get time-based range queries for free. With time series this is very important in many use cases.
Projects tend to gain more and more functionality to match the new workloads they're being used to accomplish, and it must certainly be a difficult decision for project visionaries. Do I listen to my users and implement features that will solve their new woes, but in return accept increased complexity and higher learning barriers? Complexity sucks, but it's even harder to say no to users in pain. I wonder if this ope…
A new data structure + associated commands being supported in Redis is like, say... a new filesystem being supported in the Linux kernel. It's a few files that you could just avoid compiling in if you didn't want them, and which add code-paths that are never run unless you intentionally use that specific new thing.
Projects tend to gain more and more functionality to match the new workloads they're being used to accomplish, and it must certainly be a difficult decision for project visionaries. Do I listen to my users and implement features that will solve their new woes, but in return accept increased complexity and higher learning barriers? Complexity sucks, but it's even harder to say no to users in pain. I wonder if this ope…
It's also why software is a pop culture. Sophistication and completeness is seen as complexity and cruft by each successive generation, who start something new and simple. I don't think it's very avoidable. Tech is genuinely getting incrementally better, but it's usually in a sawtooth pattern.
My favorite aspect of this cycle is when some features in the complex software become seen as so useful as to be required and standard, so when the new simpler version is created they have to figure out a novel way of providing that useful functionality in a simple and elegant way. And they do it, sometimes knowing they have made a significant advance and sometimes without knowing.
https://engineering.linkedin.com/distributed-systems/log-wha...
Earlier quoted context omitted.
In any context its used where the demand (by whatever measure you care to use: bandwidth, throughput, message durability, etc.) doesn't justify it or isn't a good use case of Kafka, for starters. That happens all the time, because every data and infrastructure engineer in the Bay Area wants to put Kafka on his resume.
But that doesn't explain why Kafka has any minimum the output required. Does it have usability issues? A good tool should be able to be used at any scale.
Kafka also has the JVM, which requires a lot of love to scale in my experience. I do not want my programmers messing around with GC options when writing to what should (to them) be exposed just like a regular file handle (except distributed across many systems). I strongly prefer to avoid Java applications at all costs - in my experience it takes years and years and years for Java based infrastructure to become relatively stable & reliable (see ElasticSearch 5.0, or ask anyone who has been oncall for a Tomcat based application). This is almost certainly personal bias, but it's my bias regardless.
Redis also has a _massive_ number of tooling / monitoring / ecosystem advantages, including hosted options, and can run on a single instance without configuration changes from the developers perspective.
I also have personal reasons to prefer Salvatore's work over the work of Confluent.
Could the difference between `MAXLEN ~ 1000000` and `MAXLEN 100000` be handled internally, by marking the overflowing items as deleted until a whole block can be removed? Looks like this tombstone functionality is already planned, would make the API simpler.
I'm kind of thrown by the idea of putting the timestamp / stream-id in the XADD command, I would have thought the server would assign that, since one of the strengths of redis's single threaded nature is consistency: what's in redis is the truth. If you allow clients to specify timestamp, what happens when ntpd isn't running on some? I probably misread or misunderstood it.
Could you allow specifying `$` as the timestamp to tell the server you want it to use whatever it thinks the current time is as the timestamp / stream-id?
Projects tend to gain more and more functionality to match the new workloads they're being used to accomplish, and it must certainly be a difficult decision for project visionaries. Do I listen to my users and implement features that will solve their new woes, but in return accept increased complexity and higher learning barriers? Complexity sucks, but it's even harder to say no to users in pain. I wonder if this ope…
I'd agree in general, but Redis is architected in a way where these features aren't really new "layers", but rather just horizontal modular additions. A new data structure + associated commands being supported in Redis is like, say... a new filesystem being supported in the Linux kernel. It's a few files that you could just avoid compiling in if you didn't want them, and which add code-paths that are never run unless…
> A new data structure + associated commands being supported in Redis is like, say... a new filesystem being supported in the Linux kernel. It's a few files that you could just avoid compiling in if you didn't want them, and which add code-paths that are never run unless you intentionally use that specific new thing.
Isn't it exactly how features are implemented in Apache? Most people think that Apache is too bloated, but the same people also leave a lot of those modules enabled without trying to spend time to understand if they actually need them.
Is there a reason to choose millis as the granularity instead of micros or nanos? Is it because there's a stronger expectation of machines in a cluster agreeing on what milli it is "now" vs the other granularities? I'm kind of thrown by the idea of putting the timestamp / stream-id in the XADD command, I would have thought the server would assign that, since one of the strengths of redis's single threaded nature is c…
127.0.0.1:6379> xadd stream * a 1 b 2
1506977609865.0
"xadd","stream","1506977609865.0","a","1","b","2"
127.0.0.1:6379> xadd stream 10.0 a 1 b 2
(error) ERR The ID specified in XADD is smaller than the target stream top item
The reason why it was chosen to use milliseconds instead of nanoseconds is because, for most applications to query for sub-millisecond ranges is likely not useful, so to see even larger numbers in the ID maybe is just unpleasant if not useful, however we are still in time to change this if there are good motivations. But being the time the one produced by the local host, after a failover the IDs are generated by another host. Milliseconds can still more or less match with good time synchronization, but nanoseconds? So it's like if this additional precision will be just used to store non-valid info.