Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building blobd: single-machine object store with sub-ms reads and 15 GB/s upload

blog.wilsonl.in

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Building blobd: single-machine object store with sub-ms reads and 15 GB/s upload

#11

Post by supriyo-biswas »

Unroasted6154 wrote: post #6
moi2388 wrote: post #5

Earlier quoted context omitted.

And means it is acceptable for many others. There is a whole world outside of s3 you know.

It's a bit weird to present it as an alternative to S3 when it looks like a persistent cache or k/v store. A benchmark against Redis would have been nice for example. The benchmark for rocks DB is also questionable as the performance depends a lot on how you configure it, and the article's claim that it doesn't support range read doesn't give me confidence in the results. Also for the descried issue of small images f…

I'd have to assume it's a blob store for their search engine (or similar) project: https://blog.wilsonl.in/search-engine/

Re: Building blobd: single-machine object store with sub-ms reads and 15 GB/s upload

#12

Post by rrauch »

rockwotj wrote: post #8

> Direct I/O means no more fsync: no more complexity via background flushes and optimal scheduling of syncs. There's no kernel overhead from copying and coalescing. It essentially provides the performance, control, and simplicity of issuing raw 1:1 I/O requests. Not true, you still need fsync in direct I/O to ensure durability in power loss situations. Some drives have write caches that means acknowledged writes live…

Looks like the author is well aware:

  /// Even when using direct I/O, `fsync` is still necessary, as it ensures the device itself has flushed any internal caches.
  async fn sync(&self) {
    let (fut, fut_ctl) = SignalFuture::new();
    self.sender.send(Request::Sync { res: fut_ctl }).unwrap();
    fut.await
  }
Full code here:

https://github.com/wilsonzlin/blobd/blob/master/libblobd-dir...

Re: Building blobd: single-machine object store with sub-ms reads and 15 GB/s upload

#13

Post by rockwotj »

actionfromafar wrote: post #10
rockwotj wrote: post #8

> Direct I/O means no more fsync: no more complexity via background flushes and optimal scheduling of syncs. There's no kernel overhead from copying and coalescing. It essentially provides the performance, control, and simplicity of issuing raw 1:1 I/O requests. Not true, you still need fsync in direct I/O to ensure durability in power loss situations. Some drives have write caches that means acknowledged writes live…

You mean in volatile memory?

yes thanks

Re: Building blobd: single-machine object store with sub-ms reads and 15 GB/s upload

#15

Post by moi2388 »

Unroasted6154 wrote: post #6
moi2388 wrote: post #5

Earlier quoted context omitted.

And means it is acceptable for many others. There is a whole world outside of s3 you know.

It's a bit weird to present it as an alternative to S3 when it looks like a persistent cache or k/v store. A benchmark against Redis would have been nice for example. The benchmark for rocks DB is also questionable as the performance depends a lot on how you configure it, and the article's claim that it doesn't support range read doesn't give me confidence in the results. Also for the descried issue of small images f…

Yes, those are fair points.
Post reply on HN