Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transfering Files with gRPC

kreya.app

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Transfering Files with gRPC

#11

Post by CamouflagedKiwi »

I've done this before, using Google's semi-standard ByteStream messages. It works, but is a bit of work, and I really don't love how you're building on top of a protocol that completely solves streaming contents of arbitrary size, which gRPC drops, and you have to reinvent again in the application layer.

I know it's not easy to solve given how protobuf-centric it is, but this is the worst piece of gRPC for me. The 4MB limit is a terrible size, it's big enough to rarely hit in test cases but small enough it can hit you in production. If you control it all you can just lift that number to something arbitrarily big to avoid most things just failing (although you probably don't want to use that as an actual solution for streaming files of any size), but in practice a lot of "serious" setups end up contorting themselves remarkably to try to avoid that limit.

Re: Transfering Files with gRPC

#13

Post by augusteo »

Building on sluongng's point about schema evolution: we ended up in a weird middle ground on a project where we used gRPC for metadata and presigned S3 URLs for the actual bytes.

The metadata schema changed constantly (new compression formats, different checksum algorithms, retry policies). Having protobufs for that was genuinely useful. But trying to pipe multi-gigabyte files through gRPC streams was painful. Memory pressure, connection timeouts on slow clients, debugging visibility was terrible.

S3 presigned URLs are the boring answer, but they work. Your object storage handles the hard parts (resumability, CDN integration, the actual bytes), and your gRPC service handles the interesting parts (authentication, metadata, business logic).

Re: Transfering Files with gRPC

#14

Post by kruador »

I would add a further advantage of plain HTTP (REST) compared to gRPC. Splitting the response into blocks and having the client request the next block, as in the gRPC solution, causes round-trip delays. The server can't send the second block of data until the client requests it, so the server is essentially idle until the client has received all packets of the first block, parsed them and generated the next request.

In contrast, while HTTP/2 does impose framing of streams, that framing is done entirely server-side. If all one end has to send to the other is a single stream, it'll be DATA frame after DATA frame for the same stream. The client is not required to acknowledge anything. (At least, nothing above the TCP layer!)

It probably wasn't noticeable in this experiment as, if I'm reading it correctly, the server and client were on the same box, but if you were separated by any significant distance, plain HTTP should be noticeably faster.

Re: Transfering Files with gRPC

#15

Post by chasil »

a-dub wrote: post #6
avianlyric wrote: post #5

Earlier quoted context omitted.

Because you may already have robust and sensible gRPC infrastructure setup and working, and setting up the correct HTTP infrastructure to take advantage of all the benefits that plain old HTTP provides may not be worth it. If moving big files around is a major part of the system you’re building, then it’s worth the effort. But if you’re only occasionally moving big files around, then reusing your existing gRPC infras…

this. also, http/s compatibility falls off in the long tail of functionality. i've seen cache layers fail to properly implement restartable http. that said, making long transfers actually restartable, robust and reliable is a lot more work than is presented here.

Is see that QUIC file transfer protocols are available, including a Microsoft SMB implementation.

These would be the ultimate in resumability and mobility between networks, assuming that they exploit the protocol to the fullest.

Re: Transfering Files with gRPC

#17

Post by jeffbee »

augusteo wrote: post #13

Building on sluongng's point about schema evolution: we ended up in a weird middle ground on a project where we used gRPC for metadata and presigned S3 URLs for the actual bytes. The metadata schema changed constantly (new compression formats, different checksum algorithms, retry policies). Having protobufs for that was genuinely useful. But trying to pipe multi-gigabyte files through gRPC streams was painful. Memory…

Sending bulk data by reference is a common pattern. Even inside Google when I was there bulk data was sometimes placed on ephemeral storage and sent by reference, and 100MB was considered a "dangerously large" protobuf that would log a warning during decode.

Re: Transfering Files with gRPC

#18

Post by tuetuopay »

And then a C programmer comes in and slams sendfile. That’s the main advantage of HTTP/1.1. Of course TLS throws a wrench in it, but once kTLS is actually good (ahem), it’ll work.

In all seriousness, don’t do large file transfers over gRPC, except in a pinch for small files. As soon as e.g API gateways are introduced in the mix, stuff can go south very quickly: increased allocation, GC pressure, network usage, etc. Just use presigned S3 URLs.

Re: Transfering Files with gRPC

#19

Post by hpdigidrifter »

jayd16 wrote: post #10
avianlyric wrote: post #5

Earlier quoted context omitted.

Because you may already have robust and sensible gRPC infrastructure setup and working, and setting up the correct HTTP infrastructure to take advantage of all the benefits that plain old HTTP provides may not be worth it. If moving big files around is a major part of the system you’re building, then it’s worth the effort. But if you’re only occasionally moving big files around, then reusing your existing gRPC infras…

gRPC runs over http. What infra would be missing? If you happen to be on ASP.NET or Spring Boot its some boilerplate to stand up a plain http and gRPC endpoints side by side but I guess you could be running something more exotic than that.

http/2 is nothing like http/1

feel free to put them both behind load balancers and see how you go

Re: Transfering Files with gRPC

#20

Post by jasonjei »

I have PTSD from Google Protobufs. Sometimes the cost of a less-efficient protocol or traditional REST is worth it over an overengineered solution. Protobufs can be fine, but it's largely overkill. Debugging with protobuf was the price we paid for an "efficient" protocol
Post reply on HN