The Kivaloo Data Store
tarsnap.com
The Kivaloo Data Store
1–10 of 78 posts
Re: The Kivaloo Data Store
#2Note: The performance values mentioned on that page (on an EC2 c1.medium instance using spinning-rust disks!) is wildly out of date. I'll get around to updating them some day.
For reference, on my laptop (Dell Latitude 7390 with an i7-8650U CPU):
* Bulk inserts run at ~600,000/second (up from 125,000).
* Bulk extracts run at ~660,000/second while in RAM (up from 30,000) and ~220,000/second from disk (up from 20,000).
* Bulk updates run at ~700,000/second in RAM dropping to ~300,000 from disk (up from 110,000 dropping to 60,000).
* Random reads run at ~800,000/second in RAM dropping to ~20,000 from disk (up from 220,000 dropping to 11,000).
* Random mixed run at ~400,000/second in RAM dropping to ~10,000 from disk (up from 30,000 dropping to 4,000).
* Hot-spot reads run at ~800,000/second in RAM dropping to ~500,000 from disk (up from 220,000 dropping to 60,000).
Note that "in RAM" means "the dataset fits into RAM" -- in all cases data is durably stored to disk.
There is almost certainly room for improvement; I haven't done extensive profiling yet.
Re: The Kivaloo Data Store
#3Note: The performance values mentioned on that page (on an EC2 c1.medium instance using spinning-rust disks!) is wildly out of date. I'll get around to updating them some day. For reference, on my laptop (Dell Latitude 7390 with an i7-8650U CPU): * Bulk inserts run at ~600,000/second (up from 125,000). * Bulk extracts run at ~660,000/second while in RAM (up from 30,000) and ~220,000/second from disk (up from 20,000).…
> It was designed to satisfy the needs of the Tarsnap online backup service for high-performance key-value storage, although it is not yet being used for that purpose
Does the fact that you're still maintaining Kivaloo this many years later imply it's now being used by Tarsnap?
Re: The Kivaloo Data Store
#4Note: The performance values mentioned on that page (on an EC2 c1.medium instance using spinning-rust disks!) is wildly out of date. I'll get around to updating them some day. For reference, on my laptop (Dell Latitude 7390 with an i7-8650U CPU): * Bulk inserts run at ~600,000/second (up from 125,000). * Bulk extracts run at ~660,000/second while in RAM (up from 30,000) and ~220,000/second from disk (up from 20,000).…
> It was designed to satisfy the needs of the Tarsnap online backup service for high-performance key-value storage, although it is not yet being used for that purpose Does the fact that you're still maintaining Kivaloo this many years later imply it's now being used by Tarsnap?
Tarsnap has recently started using Kivaloo. Over time I intend to use it far more -- but since Tarsnap is a backup service, I'm starting with the least critical parts first.
Re: The Kivaloo Data Store
#5Note: The performance values mentioned on that page (on an EC2 c1.medium instance using spinning-rust disks!) is wildly out of date. I'll get around to updating them some day. For reference, on my laptop (Dell Latitude 7390 with an i7-8650U CPU): * Bulk inserts run at ~600,000/second (up from 125,000). * Bulk extracts run at ~660,000/second while in RAM (up from 30,000) and ~220,000/second from disk (up from 20,000).…
These are actually some really good numbers for a certain application I'm looking at (https://news.ycombinator.com/item?id=24191307), especially with bulk inserts being so high.
A few questions:
- Why 255 bytes to 255 bytes? Does this have any performance consequences?
- Your laptop is SSD right?
- Is there any more documentation on how to use Kivaloo? (not just facts about it)
Re: The Kivaloo Data Store
#6Slighlty offtopic, but it's ironic how a service with the tagline "Online backups for the truly paranoid" serves insecure connections to its website and doesn't employ HSTS. Especially since the user is expected to download the client for the service from that website.
Kind of a reckless decision.
Re: The Kivaloo Data Store
#7Note: The performance values mentioned on that page (on an EC2 c1.medium instance using spinning-rust disks!) is wildly out of date. I'll get around to updating them some day. For reference, on my laptop (Dell Latitude 7390 with an i7-8650U CPU): * Bulk inserts run at ~600,000/second (up from 125,000). * Bulk extracts run at ~660,000/second while in RAM (up from 30,000) and ~220,000/second from disk (up from 20,000).…
These are actually some really good numbers for a certain application I'm looking at ( https://news.ycombinator.com/item?id=24191307 ), especially with bulk inserts being so high. A few questions: - Why 255 bytes to 255 bytes? Does this have any performance consequences? - Your laptop is SSD right? - Is there any more documentation on how to use Kivaloo? (not just facts about it)
The 255 byte limit is because I store keys and values as a one-byte length followed by the relevant data. For Tarsnap what I typically want is ~40 byte keys and data (hence using those sizes for benchmarking).
Yes, my laptop has a Intel 660p 512 GB NVMe disk.
No documentation per se, although I hope the library interfaces are reasonably understandable. I'd be happy to help though -- this code deserves to be used!
Re: The Kivaloo Data Store
#8Slighlty offtopic, but it's ironic how a service with the tagline "Online backups for the truly paranoid" serves insecure connections to its website and doesn't employ HSTS. Especially since the user is expected to download the client for the service from that website. Kind of a reckless decision.
Especially since the user is expected to download the client for the service from that website.
You mean the GPG-signed client code?
Re: The Kivaloo Data Store
#9Slighlty offtopic, but it's ironic how a service with the tagline "Online backups for the truly paranoid" serves insecure connections to its website and doesn't employ HSTS. Especially since the user is expected to download the client for the service from that website. Kind of a reckless decision.
I'm pretty sure the preferred download method is via FreeBSD ports ;)
Re: The Kivaloo Data Store
#10Slighlty offtopic, but it's ironic how a service with the tagline "Online backups for the truly paranoid" serves insecure connections to its website and doesn't employ HSTS. Especially since the user is expected to download the client for the service from that website. Kind of a reckless decision.
Especially since the user is expected to download the client for the service from that website. You mean the GPG-signed client code?
Well yeah but the GPG key is also served insecurely: