Viewing profile — tdrhq

Yes, but our SaaS tool required our customers to not use sparse or shallow checkouts since we needed the git commit-graph. Using upload-pack allowed us to remove that constraint, s…

Azure DevOps has an additional requirement that Git clients support a protocol feature called "multi-ack". We don't support it yet, and we didn't think we need it. Rather than bloc…

Thanks! (author here)

> The paper that definitionally is Raft doesn't tell you how to interact with durable storage. That's being a bit pedantic. Yeah, I did mean that any respectable library implementi…

> Upgrades are especially challenging Yes indeed! But this doesn't apply to a startup in the Explore phase, where you don't need replication, and how we did it for a long time. Thi…

We used an existing library called bknr.datastore to handle this part, so we didn't have to reinvent the wheel :) I mentioned that at the end of the blog post, but I wanted to buil…

To clarify, as I think some people have misunderstood: we used an existing library called bknr.datastore to handle the "database" part of the in-memory store, so we didn't have to …

Good catch on the metrics! We do use Preset for metrics and dashboards, and obviously Preset isn't going to talk to our in-memory database. So we do have a separate MySQL database …

Ah, sure: I did not consider Redis at all. My goal in the Explore phase was to keep the data in the same process as my code, and replacing MySQL with any other database doesn't rea…

Raft does do persistence and crash recovery, at least of the transaction logs. What you need from your side (and there are libraries that already do this): a) A mechanism to snapsh…

This is fascinating, thanks for the data! I agree with the the other reply to this: I probably should've said that it's easy to get a machine with 100s of GB of RAM instead of sayi…

> It's quite odd that an argument grounded on performance claims I probably did a bad job then, because everything in the blog post was meant to be developer productivity claims, n…

Good question! And this comes to the difference between Explore phase and Expand phase. In the Explore phase, data migration was just running code on the production server via a RE…

Absolutely. By the way, if it wasn't clear from my blog post, in the Explore phase, I used an existing library to do this. It was only in the Expand phase that I put this existing …

> Wait, so you’re blocking on a Raft round-trip to make forward progress? That’s the correct decision wrt durability, but… Yeah. I hope it was clear in my post that the goal was de…

I haven't seen either of these! But I have to say, my inspiration here came from existing libraries. (My only innovation here is taking an existing library that did the whole trans…