Buffer: Composable Buffers for Go • Hacker News Board

Re: Buffer: Composable Buffers for Go

#11

Post by lobster_johnson » Sun, Apr 05, 2015, 11:54 PM UTC

TillE wrote: post #10

stinos wrote: post #2

// Buffer 32KB to Memory, after that buffer to 100MB chunked files buf := buffer.NewUnboundedBuffer(32*1024, 100*1024*1024) pool := NewFilePool(100*1024*1024, "") // "" -- use temp dir I seem to have a twofold feeling about this code: on one hand I really like the brevity of these statements, but on the other hand it does come at the cost of being rather cryptic: without the comments one can only guess what exactly i…

That's one problem that named arguments solve pretty nicely. Unfortunately, I don't think Go supports them.

You have anonymous strict literals, though, which are nearly as succinct and can serve the same role.

Re: Buffer: Composable Buffers for Go

#12

Post by heavenlyhash » Sun, Apr 05, 2015, 11:54 PM UTC

djherbis wrote: post #7

Hey author here, happy to answer your questions.

Have you / would you consider extending this concept to buffers that are still single writer but support many readers -- something like a `FreshReader() io.Reader` function that returns a reader that starts from the beginning again? (I have an application that needs this semantic and ended up just rolling it quick-and-dirty; this sounds like very similar concepts done up with better gift wrapping.)

Re: Buffer: Composable Buffers for Go

#13

Post by MetaCosm » Mon, Apr 06, 2015, 12:03 AM UTC

TillE wrote: post #10

stinos wrote: post #2

// Buffer 32KB to Memory, after that buffer to 100MB chunked files buf := buffer.NewUnboundedBuffer(32*1024, 100*1024*1024) pool := NewFilePool(100*1024*1024, "") // "" -- use temp dir I seem to have a twofold feeling about this code: on one hand I really like the brevity of these statements, but on the other hand it does come at the cost of being rather cryptic: without the comments one can only guess what exactly i…

That's one problem that named arguments solve pretty nicely. Unfortunately, I don't think Go supports them.

There are a lot of ways to make arguments nicer in Go. The simple on is to just use a configuration struct (or anon struct) at that has nice names. But, for complex stuff, functional options as laid out by Dave Cheney: http://dave.cheney.net/2014/10/17/functional-options-for-fri... (video: https://www.youtube.com/watch?v=24lFtGHWxAQ)

Re: Buffer: Composable Buffers for Go

#14

Post by djherbis » Mon, Apr 06, 2015, 12:09 AM UTC

heavenlyhash wrote: post #12

djherbis wrote: post #7

Hey author here, happy to answer your questions.

Have you / would you consider extending this concept to buffers that are still single writer but support many readers -- something like a `FreshReader() io.Reader` function that returns a reader that starts from the beginning again? (I have an application that needs this semantic and ended up just rolling it quick-and-dirty; this sounds like very similar concepts done up with better gift wrapping.)

I'm currently working on something that does that sort of thing. https://github.com/djherbis/fscache

It doesn't have the composability features yet, but it already handles 1 writer with many concurrent readers. Let me know if that helps!

Re: Buffer: Composable Buffers for Go

#15

Post by elithrar » Mon, Apr 06, 2015, 2:23 AM UTC

Nice work. buffer.NewPool is a great convenience over writing your own sync.Pool.

I was previously using https://github.com/oxtoacart/bpool as a 64K buffer pool for rendering (concurrently) template/html contents—so I can check for the errors from template.Render—before then using io.Copy to copy the "known good" contents to the http.ResponseWriter. I may have to look into using this.