Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
1–10 of 10 posts
Re: Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
#2> In short, we believe that systems design is about contracts between reality and reality as we see it; reality as we see it is just a fiction. We believe systems exist in two forms: their actual reality and our fictional understanding of them. To us, the programmers, the system's reality is a fiction; we cannot see it, we can only observe it through additional SRE observability and monitoring mechanisms. To the system, this observability is a fiction. Whether due to tooling problems or just measurement error, the fiction presented via metrics, logs, and traces resembles the system itself, but is decidedly not the system itself.
this is very McLuhan/systemantics of you! all abstractions are leaky, but some abstractions let you look at the leaks.
TIL about setsums - one wonders if `fn setsum([String]) -> Digest` works then "nested setsums" must also work for very large scales.
one thing i missed from this post, which otherwise would score perfect marks for a technology introduciton, is benchmarks vs your comparisons on warpstream and friends.
Re: Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
#3> In short, we believe that systems design is about contracts between reality and reality as we see it; reality as we see it is just a fiction. We believe systems exist in two forms: their actual reality and our fictional understanding of them. To us, the programmers, the system's reality is a fiction; we cannot see it, we can only observe it through additional SRE observability and monitoring mechanisms. To the syst…
You will always find a copy of McLuhan on the Chroma teams bookshelf !
Re: Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
#4> In short, we believe that systems design is about contracts between reality and reality as we see it; reality as we see it is just a fiction. We believe systems exist in two forms: their actual reality and our fictional understanding of them. To us, the programmers, the system's reality is a fiction; we cannot see it, we can only observe it through additional SRE observability and monitoring mechanisms. To the syst…
Can you elaborate on what you mean by "nesting" setsums? It's an associative and commutative construction, so you can run e.g. map-reduce to map every parquet file to a setsum and then reduce to roll them up.
Re: Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
#5Hi,
I'm Robert, one of the authors of wal3. Here to answer any questions or explain more about our engineering philosophy.
Happy hacking, Robert
Re: Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
#6Hi, I'm Robert, one of the authors of wal3. Here to answer any questions or explain more about our engineering philosophy. Happy hacking, Robert
[deleted]
Re: Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
#7> In short, we believe that systems design is about contracts between reality and reality as we see it; reality as we see it is just a fiction. We believe systems exist in two forms: their actual reality and our fictional understanding of them. To us, the programmers, the system's reality is a fiction; we cannot see it, we can only observe it through additional SRE observability and monitoring mechanisms. To the syst…
You will always find a copy of McLuhan on the Chroma teams bookshelf !
Is there a particular book of McLuhan that ya'll recommend?
Re: Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
#8Is this deeply coupled with Chroma or something we can fork out for other applications? I would love to explore this for lightweight change data capture on S3.
Re: Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
#9Is this deeply coupled with Chroma or something we can fork out for other applications? I would love to explore this for lightweight change data capture on S3.
Absolutely! We took care during the build to address chroma’s needs, but I believe the only requirement is chroma storage. Long term it’ll appear on crates.io as a standalone package.
Re: Wal3: A Write-Ahead Log for Chroma, Built on Object Storage
#10Always exciting to see Robert Escriva's next database project. Building on top of the incredible engineering of S3 is clever, I'm definitely going to be looking at and learning from the implementation in wal3.
Looking forward to the benchmarks on using S3 for piece and the design of the scale out architecture.