Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenFreeMap survived 100k requests per second

blog.hyperknot.com

Post reply on HN
Print view View original

81–90 of 140 posts

Re: OpenFreeMap survived 100k requests per second

#81

Post by biker142541 »

nielsole wrote: post #77
biker142541 wrote: post #75

Curious how this would have compared to a static pmtiles file being read directly by maplibre. I’ve had good luck with virtually equal latency to served tiles when consuming pmtiles via range requests on Bunnycdn.

https://github.com/hyperknot/openfreemap?tab=readme-ov-file#...

Interesting, I should benchmark this. I have only used Bunnycdn so far and latency seemed similar to most tile providers like Maptiler and others (but a very limited test). This was using the full planet pmtiles file. Bunnycdn also makes it easy to prevent downloading the entire file, either in case you care about anyone using it or just want to prevent surprise downloads for those exploring network tab.

Re: OpenFreeMap survived 100k requests per second

#82

Post by biker142541 »

biker142541 wrote: post #81
nielsole wrote: post #77

Earlier quoted context omitted.

https://github.com/hyperknot/openfreemap?tab=readme-ov-file#...

Interesting, I should benchmark this. I have only used Bunnycdn so far and latency seemed similar to most tile providers like Maptiler and others (but a very limited test). This was using the full planet pmtiles file. Bunnycdn also makes it easy to prevent downloading the entire file, either in case you care about anyone using it or just want to prevent surprise downloads for those exploring network tab.

Quick benchmark of pmtiles directly in maplibre vs served tiles, both via Bunnycdn and 5 areas sampled using same style. Total impact on page end to end load time: 39ms longer with cached range requests from pmtiles than cached tiles. Individual requests are comparable in the 20-35ms range, so the slight extra time seems to be from the additional round trip for range headers (makes sense).

Re: OpenFreeMap survived 100k requests per second

#83

Post by RandomBacon »

RandomBacon wrote: post #41
jwilk wrote: post #27

Earlier quoted context omitted.

From the HN Guidelines https://news.ycombinator.com/newsguidelines.html >: > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".

That guideline is decent I guess. I am disappointed that they edited another guideline for the worse: > Please don't comment about the voting on comments. It never does any good, and it makes boring reading. It used to just say, don't complain about voting. If the number of votes are so taboo, why do they even show us the number or user karma (and have a top list)?

We can't even talk about the guidelines?

Re: OpenFreeMap survived 100k requests per second

#86

Post by justinclift »

hyperknot wrote: post #42
sour-taste wrote: post #36

Since the limit you ran into was number of open files could you just raise that limit? I get blocking the spammy traffic but theoretically could you have handled more if that limit was upped?

I've just written my question to the nginx community forum, after a lengthy debugging session with multiple LLMs. Right now, I believe it was the combination of multi_accept + open_file_cache > worker_rlimit_nofile. https://community.nginx.org/t/too-many-open-files-at-1000-re... Also, the servers were doing 200 Mbps, so I couldn't have kept up _much_ longer, no matter the limits.

> so I couldn't have kept up _much_ longer, no matter the limits.

Why would that kind of rate cause a problem over time?

Re: OpenFreeMap survived 100k requests per second

#87

Post by motorest »

TylerE wrote: post #68
andai wrote: post #53

From the screenshot I wanted to say, couldn't this be done on a single VPS? Seemed over engineered to me. Then I realized the silly pixels are on top of a map of the entire earth. Dang! I'm curious what the peak req/s is like. I think it might be just barely within the range supported by benchmark-friendly web servers. Unless there's some kind of order of magnitude slowdowns due to the nature of the application. Edit…

"€20/month from Hetzner" is great until you actually need it to be up and working when you need it.

> "€20/month from Hetzner" is great until you actually need it to be up and working when you need it.

I managed a few Hetzner cloud instances, and some report perfect uptime for over a year. The ones that don't, I was the root cause.

What exactly leads you to make this sort of claim? Do you actually have any data or are you just running your mouth off?

Re: OpenFreeMap survived 100k requests per second

#89

Post by wiradikusuma »

"Wplace.live happened. Out of the blue, a new collaborative drawing website appeared, built from scratch using OpenFreeMap." -- as a founder, you know you're working on the wrong thing when there's a "fun project" getting daily traffic more than what you'd get in a lifetime :)

Re: OpenFreeMap survived 100k requests per second

#90

Post by bigstrat2003 »

Sesse__ wrote: post #64
hyperknot wrote: post #31

Earlier quoted context omitted.

We are talking about an insane amount of data here. It was 56 Gbit/s (or 56 x 1 Gbit servers 100% saturated!). This is not something a "caching server" could handle. We are talking on the order of CDN networks, like Cloudflare, to be able to handle this.

> We are talking about an insane amount of data here. It was 56 Gbit/s. This is not something a "caching server" could handle. You are not talking about an insane amount of data if it's 56 Gbit/s. Of course a caching server could handle that. Source: Has written servers that saturated 40gig (with TLS) on an old quadcore.

I realize that what constitutes "insane" is a subjective judgement. But, uh... I most certainly would call 56 Gbps insane. Which is not to say that hardware which handles it doesn't exist. It might not even be especially insane hardware. But that is a pretty insane data rate in my book.
Post reply on HN