Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-Contained Pure-Go Web Server with Lua, MD, HTTP/2, QUIC, Redis Support

github.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Self-Contained Pure-Go Web Server with Lua, MD, HTTP/2, QUIC, Redis Support

#51

Post by tedunangst »

jand wrote: post #41
tyingq wrote: post #8

"Files that are sent to the client are compressed with gzip, unless they are under 4096 bytes." That's interesting. Is that a common optimization? I hadn't heard of any other web server doing that.

Not a direct answer, but also interesting to consider: As everything will end up in a packet when sent through the network stack, you might want to choose your minimum input size in such way, that you generate gzip-compressed output big enough. Why big enought? Nagle's algorithm [1] So yet another reason to think about 'what to gzip'. [1] https://en.wikipedia.org/wiki/Nagle%27s_algorithm

If you read the page you linked, it says the algorithm applies to data of any size.

Re: Self-Contained Pure-Go Web Server with Lua, MD, HTTP/2, QUIC, Redis Support

#52

Post by hombre_fatal »

codebeaker wrote: post #49
tyingq wrote: post #8

"Files that are sent to the client are compressed with gzip, unless they are under 4096 bytes." That's interesting. Is that a common optimization? I hadn't heard of any other web server doing that.

Also, according to the spec HTTP servers may not always honour the value in the `Accept-Encoding` header[0]. > Even if both the client and the server supports the same compression algorithms, the server may choose not to compress the body of a response, if the identity value is also acceptable. I've actually run into this twice in my career and it has been a surprise to those around me in both cases. Both times in th…

Makes more sense to use the verb "negotiating" with Accept-* headers rather than "honoring".

This makes obvious sense once you consider that the client tells the server which compression formats it supports in every request yet not every data format is compressible, nor does the server necessary support any candidate compression format.

For example, the server wouldn't gzip a jpeg since it's already compressed.

All Accept-* headers are like this. e.g. the server doesn't necessarily support any of the languages requested in the Accept-Language header, but it doesn't hurt to ask. You always have to inspect the response headers to see the result of negotiation.

Re: Self-Contained Pure-Go Web Server with Lua, MD, HTTP/2, QUIC, Redis Support

#53

Post by Malic »

tyingq wrote: post #8

"Files that are sent to the client are compressed with gzip, unless they are under 4096 bytes." That's interesting. Is that a common optimization? I hadn't heard of any other web server doing that.

I don't know about "under 4096 bytes" but I have heard of not compressing data that is under ~1500 bytes. Part of the thinking is this - if your result data (plus HTTP overhead) is already smaller than the data payload of an IP packet (MTU settings come into play here), then you are spending CPU time that will not save you any network I/O time.

Re: Self-Contained Pure-Go Web Server with Lua, MD, HTTP/2, QUIC, Redis Support

#54

Post by penagwin »

yetanotherme wrote: post #27
takeda wrote: post #17

Earlier quoted context omitted.

- increases amount of time to fetch and run container (it actually is quite noticeable when you have app that scales out and you updating it) - it increases amount of storage to store multiple versions of containers (when you have an internal app and do frequent releases it adds quickly) - it increases amount of data transferred on every deployment - increases amount of memory used (the whole point of containers, was…

True. Luckily, most Go binaries can be upx'd ( https://upx.github.io/ ) for a fraction of their original size. Just put it into your Dockerfile as a part of the build process.

This works and helps with storage/transit. Word of warning though that many AV like to flag UPX'd executables (only important if it's not an internal tool), and it'll take even longer for it to start, and will use more memory. My understanding is it essentially does standard compression on the executable and appends the decompression part on the front of it.

Re: Self-Contained Pure-Go Web Server with Lua, MD, HTTP/2, QUIC, Redis Support

#57

Post by xyproto »

Malic wrote: post #53
tyingq wrote: post #8

"Files that are sent to the client are compressed with gzip, unless they are under 4096 bytes." That's interesting. Is that a common optimization? I hadn't heard of any other web server doing that.

I don't know about "under 4096 bytes" but I have heard of not compressing data that is under ~1500 bytes. Part of the thinking is this - if your result data (plus HTTP overhead) is already smaller than the data payload of an IP packet (MTU settings come into play here), then you are spending CPU time that will not save you any network I/O time.

Yes, also compressing and decompressing a small amount of bytes may take longer than just sending it uncompressed.

Re: Self-Contained Pure-Go Web Server with Lua, MD, HTTP/2, QUIC, Redis Support

#58

Post by takeda »

codebeaker wrote: post #49
tyingq wrote: post #8

"Files that are sent to the client are compressed with gzip, unless they are under 4096 bytes." That's interesting. Is that a common optimization? I hadn't heard of any other web server doing that.

Also, according to the spec HTTP servers may not always honour the value in the `Accept-Encoding` header[0]. > Even if both the client and the server supports the same compression algorithms, the server may choose not to compress the body of a response, if the identity value is also acceptable. I've actually run into this twice in my career and it has been a surprise to those around me in both cases. Both times in th…

"Accept-Encoding" means only that the client also understands specific encoding (in this case compression) it is still up to the server to chose what to dot. There was a time when browsers didn't support any compression. This header was introduced to signal to server what is acceptable by the client, that's why the header allows specifying multiple compression algorithms.

Similar thing is with header that's quite useful, but for some reason very few sites honor it: "Accept-Language" browser can specify which languages are preferred, but it is up to server to honor it (for example given language version is not available).

Re: Self-Contained Pure-Go Web Server with Lua, MD, HTTP/2, QUIC, Redis Support

#59

Post by takeda »

DarkWiiPlayer wrote: post #34
tyingq wrote: post #8

"Files that are sent to the client are compressed with gzip, unless they are under 4096 bytes." That's interesting. Is that a common optimization? I hadn't heard of any other web server doing that.

I heared somewhere (Some blogs comment section, I believe) that gzip actually reduces the security of HTTPS; maybe someone can confirm / explain that?

In scenario when an attacker can see encrypted message (e.g. monitoring network traffic) and can affect part of the message that is being encrypted, he can use the compression to his advantage. He can for example try different inputs and observe the length of encrypted message, if with the certain input the length drops that means the given input contains string that's similar to another part of the decrypted message and the compressing algorithm did its job and used that to reduce the size.

This was mentioned in 2012 when CRIME[1] (also included BEAST[2] exploit) and later BREACH[2] vulnerability (when it was considered cool to come up with cool sounding name, creating a logo and a website for specific vulnerabilities)

[1] https://en.wikipedia.org/wiki/CRIME

[2] https://en.wikipedia.org/wiki/Transport_Layer_Security#BEAST...

[3] https://en.wikipedia.org/wiki/BREACH

Re: Self-Contained Pure-Go Web Server with Lua, MD, HTTP/2, QUIC, Redis Support

#60

Post by emilfihlman »

DarkWiiPlayer wrote: post #34
tyingq wrote: post #8

"Files that are sent to the client are compressed with gzip, unless they are under 4096 bytes." That's interesting. Is that a common optimization? I hadn't heard of any other web server doing that.

I heared somewhere (Some blogs comment section, I believe) that gzip actually reduces the security of HTTPS; maybe someone can confirm / explain that?

This shouldn't be true in any sane system.
Post reply on HN