Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-world HTTP/2: 400GB of images per day

99designs.com.au

Post reply on HN
Print view View original

31–40 of 72 posts

Re: Real-world HTTP/2: 400GB of images per day

#31

Post by rgbrenner »

xzyfer wrote: post #27
rgbrenner wrote: post #22

Founder of NuevoCloud here. If I read this right, you guys used Cloudflare for http 2. So let me ask you this, when you did your comparison, were all of the images cached (ie: x-cache: hit) at the edge? The reason I ask is because cloudflare, last I checked, still hasn't implemented http2's client portion. So when a file is not cached, it does this: client edge node origin server. Http2 is only used for the short hop…

To the best of my knowledge you are correct about how CloudFlare works. For context this data was collected over the period about a month on real production pages with significant traffic. The edges were well and truely primed.

although cloudflare doesn't manage caches on a per account basis. Each PoP has a single LRU cache that's used for all customers. In other words, even if you've primed it, your files may have been pushed out of the cache by a larger customer.

In order to know this hasn't occurred, you really have to check the hit rate cloudflare is reporting (for static files that rarely change, this should be near or at 100%)... and when you're doing side-by-side comparisons (like the speed index), you have to actually check the x-cache headers to verify that a cache miss hasn't occurred. Otherwise, you wouldn't actually know that a significant portion of traffic isn't being sent over http 1.1 (because of cache misses).

Re: Real-world HTTP/2: 400GB of images per day

#32

Post by xzyfer »

cagenut wrote: post #30

Did I read this right that http1 was with cdn A (unnamed?) and http2 was with cdn B (cloudflare)? If so, you really can't draw any conclusions about the protocol difference when the pop locations, network designs, hardware and software configurations could easily have made the kinds of differences you're seeing.

You read it correct.

By not moving our render blocking assets like CSS, JS and fonts over to the http/2 we rule out performance changes due to improvements to head of line block.

Our images were always on a separate hostname so the DNS lookup over is the same. We also did some initial benchmarking and found the new CDN to be more efficient than the old one.

Re: Real-world HTTP/2: 400GB of images per day

#33

Post by noahcollins »

xzyfer wrote: post #14
noahcollins wrote: post #13

Thanks for posting your findings - very useful data. It would be interesting to see the Webpagetest waterfalls in greater detail if you're able to share that. You planning to use your resource hints to enable server push at CDN edge?

Server push at the edge is problem atm. Current push semantics require the HTML document say which resources to push. That's an issue if you're serving assets off a CDN domain. Asset domains make less since with h2 from a performance perspective but there are still security concerns that need to addressed.

Good point if you're using push for page content that varies, like images in the the 99designs portfolio and gallery. That gets into dynamic caching territory.

As a first step, I'm focused on using push to cut latency between TTFB and processing of render-blocking static assets. Serving those from same domain as the base page, it should be easy for origin to supply edge with the list of resources push. Either in the HTML or with the `link` response header. It also means my critical assets are not longer behind a separate DNS lookup.

In the design gallery, this type of push approach could help you regain control of loading priority and get your fonts loading before that wall of images.

Re: Real-world HTTP/2: 400GB of images per day

#34

Post by foota »

wongarsu wrote: post #12

I don't think that the server is in charge of priorisation here. The server can do it, but there is no reason to push this responsibility onto the server when the browser can do it much better (for example the server can't know what's in the viewport). I expect this will be quickly sorted out by more mature HTTP/2 implementations in browsers. Downloading every image at once is obviously a bad idea, and I expect such…

I think the real solution hear is for the browser to be able to communicate some sort of priority to the server, without having to download a limited number of files at once.

Re: Real-world HTTP/2: 400GB of images per day

#35

Post by xzyfer »

foota wrote: post #34
wongarsu wrote: post #12

I don't think that the server is in charge of priorisation here. The server can do it, but there is no reason to push this responsibility onto the server when the browser can do it much better (for example the server can't know what's in the viewport). I expect this will be quickly sorted out by more mature HTTP/2 implementations in browsers. Downloading every image at once is obviously a bad idea, and I expect such…

I think the real solution hear is for the browser to be able to communicate some sort of priority to the server, without having to download a limited number of files at once.

Browser do currently do this. H2 has two types of prioritisations: weighted, and dependency.

All browsers implement weighted resource prioritisation and weigh resources by content type. This is a hold over from what they do for HTTP/1 connections.

Firefox has dependency resources prioritisation. https://bitsup.blogspot.com.au/2015/01/http2-dependency-prio...

The spec purposely leave how these heuristics should work to the implementor. Things will change and implementations will diverge over time.

The server ultimately being in control means we can tell the server what resources are important for specific pages with absolute knowledge of the page.

Re: Real-world HTTP/2: 400GB of images per day

#36

Post by dpc_pw »

I did not do any real tests and I might be completely wrong etc. but it seems to me that http2 is going to perform poorly over wireless links like 3g.

With http1 one had N tcp connections, and with the way tcp slowly increases the bandwidth used, and rapidly decreases it when packet is lost, even if any packet were dropped (which will happen quite a lot on 3g) other tcp streams were not delayed, or blocked, and can even utilize the leftover bandwidth, yielded by the stream that lost the packet.

With http2 however there's one tcp connection, so dropped packets will cause under-utilization of the bandwidth. On top of that dropped packets will cause all frames after them, to be delayed in kernel receiving buffer until the dropped packet is retransmitted, while in http1 case they would be available at the app level right away.

HTTP2 being implemented on top of TCP always seemed like a weird choice. It should have been UDP, IMO. That's why network accelerators like PacketZoom make so much sense. Note: I work in PacketZoom, I did not do any in-depth research on HTTP2, and this is my opinion, not necessarily of the company.

Re: Real-world HTTP/2: 400GB of images per day

#37

Post by xzyfer »

pedalpete wrote: post #24

We've recently moved to Google Cloud Storage from AWS because of http/2. We had a bottleneck of the browser waiting when serving multiple large (8+files * 10mb+each). I'm wondering if 99designs looked at any sort of domain sharding to get around the timing issues. If I understand correctly, wouldn't this get around the priority queue issue? Your js,fonts, etc. coming from a different address than your larger images,…

The priority queue isn't the issue. In fact the priority queue is what kept our first paint times tanking because browsers prioritised render blocking resources instead of images.

The issue was due to the variance of image size. An image that is significantly larger than the page average will be loaded slower since all images get an equal share of bandwidth (priority). Adding sharding wouldn't help since the client only has a fixed amount of bandwidth to share and all images would still get the same share of it. Sharding could help if the bandwidth bottle neck was at the CDN but that's rarely going to be the case.

Re: Real-world HTTP/2: 400GB of images per day

#38

Post by xzyfer »

noahcollins wrote: post #33
xzyfer wrote: post #14

Earlier quoted context omitted.

Server push at the edge is problem atm. Current push semantics require the HTML document say which resources to push. That's an issue if you're serving assets off a CDN domain. Asset domains make less since with h2 from a performance perspective but there are still security concerns that need to addressed.

Good point if you're using push for page content that varies, like images in the the 99designs portfolio and gallery. That gets into dynamic caching territory. As a first step, I'm focused on using push to cut latency between TTFB and processing of render-blocking static assets. Serving those from same domain as the base page, it should be easy for origin to supply edge with the list of resources push. Either in the…

The priority queue isn't the issue. In fact the priority queue is what kept our first paint times tanking because browsers prioritised render blocking resources instead of images.

The issue was due to the variance of image size. An image that is significantly larger than the page average will be loaded slower since all images get an equal share of bandwidth (priority).

We could further improve first paint times by pushing render blocking resources but we'd need to be serving those resources off the 99designs domain (with current push implementations). This opens us up to a class security issues we avoid by having an asset domain i.e. types of reflected XSS and serving cookies on assets.

For now we'll wait for the webperf working group to address the limitations with server push semantics.

Re: Real-world HTTP/2: 400GB of images per day

#39

Post by e1g »

runeks wrote: post #10

I'm really looking forward to see how much HTTP/2 will increase performance for my Bitcoin payment channel server: https://github.com/runeksvendsen/restful-payment-channel-ser... Just now I finished separating the front-end and back-end - by a RESTful protocol - and this roughly halved performance compared to using a native library (from ~2000 payments/second on my laptop to ~1000). I expect HTTP/2 to make a greater…

Useful related project: http://www.grpc.io/ is an excellent layer on top of HTTP2 for comms between backend services. It's from Google, and used by Docker and Square among others. It even comes with a rest-focused gateway https://github.com/grpc-ecosystem/grpc-gateway

Re: Real-world HTTP/2: 400GB of images per day

#40

Post by noahcollins »

xzyfer wrote: post #38
noahcollins wrote: post #33

Earlier quoted context omitted.

Good point if you're using push for page content that varies, like images in the the 99designs portfolio and gallery. That gets into dynamic caching territory. As a first step, I'm focused on using push to cut latency between TTFB and processing of render-blocking static assets. Serving those from same domain as the base page, it should be easy for origin to supply edge with the list of resources push. Either in the…

The priority queue isn't the issue. In fact the priority queue is what kept our first paint times tanking because browsers prioritised render blocking resources instead of images. The issue was due to the variance of image size. An image that is significantly larger than the page average will be loaded slower since all images get an equal share of bandwidth (priority). We could further improve first paint times by pu…

Interesting note on the impact of image size variation on queue, thanks for elaborating.

Serving those resources from the 99designs domain is worth a look. I considered the cookies and security trade offs as well. I found H2 compressed cookies enough to perform better than a separate cookieless domain for static assets, due to the DNS savings. DNS times can be bad at high percentiles. Reflected XSS addressed with a Content Security Policy. But I'm fortunate to have user base that supports CSP well.

Post reply on HN