Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenFreeMap survived 100k requests per second

blog.hyperknot.com

Post reply on HN
Print view View original

131–140 of 140 posts

Re: OpenFreeMap survived 100k requests per second

#131

Post by nielsole »

biker142541 wrote: post #82
biker142541 wrote: post #81

Earlier quoted context omitted.

Interesting, I should benchmark this. I have only used Bunnycdn so far and latency seemed similar to most tile providers like Maptiler and others (but a very limited test). This was using the full planet pmtiles file. Bunnycdn also makes it easy to prevent downloading the entire file, either in case you care about anyone using it or just want to prevent surprise downloads for those exploring network tab.

Quick benchmark of pmtiles directly in maplibre vs served tiles, both via Bunnycdn and 5 areas sampled using same style. Total impact on page end to end load time: 39ms longer with cached range requests from pmtiles than cached tiles. Individual requests are comparable in the 20-35ms range, so the slight extra time seems to be from the additional round trip for range headers (makes sense).

What I haven't understood is why not write a program that serves the pmtiles individually on the server? Might not be as performant (but easily hit 1k RPS) but wouldn't require mounting another filesystem which practically rules out a containerised environment.

Re: OpenFreeMap survived 100k requests per second

#132

Post by piperswe »

jonathanlydall wrote: post #58
hyperknot wrote: post #52

Earlier quoted context omitted.

They are actually static files. There is just too many of them, about 300 million. You cannot put that in Pages.

Is CloudFlare’s R2 an option for you?

That, or enabling Cache Reserve to automatically have a global cache in R2 instead of only caching on the PoP-level

(disclaimer/source: I am a CF employee)

Re: OpenFreeMap survived 100k requests per second

#133

Post by CoolCold »

Aeolun wrote: post #101
toast0 wrote: post #54

Earlier quoted context omitted.

I'm pretty sure your open file cache is way too large. If you're doing 1k/sec, and you cache file descriptors for 60 minutes, assuming those are all unique, that's asking for 3 million FDs to be cached, when you've only got 1 million available. I've never used nginx or open_file_cache[1], but I would tune it way down and see if you even notice a difference in performance in normal operation. Maybe 10k files, 60s time…

If you do 200Mbps on a hetzner server after cloudflare caching, you are going to run out of traffic pretty rapidly. The limit is 20TB / month (which you’d reach in roughly 9 days).

You are probably talking about VMs - those do have traffic limits. Servers, on the other side, with default 1Gbit NICs doesn't (let's say until you consume 80%+ of bandwidth for months)

Quoting:

> Traffic

>All root servers have a dedicated 1 GBit uplink by default and with it unlimited traffic. Inclusive monthly traffic for servers with 10G uplink is 20TB. There is no bandwidth limitation. We will charge € 1 ($1.20)/TB for overusage.

Re: OpenFreeMap survived 100k requests per second

#134

Post by CoolCold »

toast0 wrote: post #54
hyperknot wrote: post #42

Earlier quoted context omitted.

I've just written my question to the nginx community forum, after a lengthy debugging session with multiple LLMs. Right now, I believe it was the combination of multi_accept + open_file_cache > worker_rlimit_nofile. https://community.nginx.org/t/too-many-open-files-at-1000-re... Also, the servers were doing 200 Mbps, so I couldn't have kept up _much_ longer, no matter the limits.

I'm pretty sure your open file cache is way too large. If you're doing 1k/sec, and you cache file descriptors for 60 minutes, assuming those are all unique, that's asking for 3 million FDs to be cached, when you've only got 1 million available. I've never used nginx or open_file_cache[1], but I would tune it way down and see if you even notice a difference in performance in normal operation. Maybe 10k files, 60s time…

> [1] to be honest, I'm not sure I understand the intent of open_file_cache... Opening files is usually not that expensive

I may have a hint here - remember, that Nginx was created in the times of dialup was a thing yet and having single Pentium 3 server was a norm (I believe I've seen myself that wwwXXX machines in the Rambler DCs over that time).

So my a bit educated guess here, that saving every syscal was sorta ultimate goal and it was more efficient in terms of at least latency by that times. You may take a look how Nginx parses http methods (GET/POST) to save operations.

Myself I don't remember seeing large benefits of using open_file_cache, but I likely never did a proper perf test here. Say ensure use of sendfile/buffers/TLS termination made much more influence for me on modern (10-15 years old) HW.

Re: OpenFreeMap survived 100k requests per second

#135

Post by Ericson2314 »

bspammer wrote: post #97
Ericson2314 wrote: post #80

Oh wow, TIL there is finally a simple way to actually view OpenStreetMap! Gosh, that's overdue. Glad it's done though!

What was wrong with the main site? Genuine question https://www.openstreetmap.org

Oh.... last I checked they didn't have that?

Re: OpenFreeMap survived 100k requests per second

#136

Post by Ericson2314 »

drewda wrote: post #122
Ericson2314 wrote: post #80

Oh wow, TIL there is finally a simple way to actually view OpenStreetMap! Gosh, that's overdue. Glad it's done though!

The OSM Foundation has been serving raster tiles for years and years (that's what's visible by default on the slippy map at www.openstreetmap.org): https://wiki.openstreetmap.org/wiki/OpenStreetMap_Carto After on and off experimentation by various contributors, OSMF just released vector tiles as well: https://operations.osmfoundation.org/policies/vector/

Thanks, I'm just out of date

Re: OpenFreeMap survived 100k requests per second

#137

Post by MaKey »

Aeolun wrote: post #129
MaKey wrote: post #125

Earlier quoted context omitted.

Small addition: That limit applies to Hetzner Cloud servers, their dedicated servers have unlimited traffic.

Depends on your connection I think. Mine do 1Gbit/sec but have a 20TB limit. The 100Mbit ones are unlimited (last I checked)

Yes, it does. 1 GBit/s dedicated servers have unlimited traffic, 10 GBit/s have 20 TB of traffic included. I'm not aware of a 100 MBit/s offer. Not sure which offer you are using but it sounds like a Hetzner cloud server not a dedicated server.

Re: OpenFreeMap survived 100k requests per second

#138

Post by immibis »

ranger_danger wrote: post #130
Aeolun wrote: post #98

Earlier quoted context omitted.

> They’re unreliable as soon as you have to deal with their support who have the technical knowledge of a brick. Since I never have to, that’s perfect isn’t it? If you need support from Hetzner you are using the wrong host.

Simply running "ipfs daemon" is enough to get hetzner to threaten canceling your service. They send nasty letters with pcap dumps of your "hacking" attempts and it doesn't matter how wrong you think they are.

clearly not true because I ran an ipfs daemon before and didn't get any nasty letters.

I got nasty letters during the letterbombing attack on Tor relays. They were nasty letters forwarded by Hetzner from the people who actually wrote them. Pointing Hetzner to the page about the attack made them go away.

Re: OpenFreeMap survived 100k requests per second

#139

Post by Aeolun »

CoolCold wrote: post #133
Aeolun wrote: post #101

Earlier quoted context omitted.

If you do 200Mbps on a hetzner server after cloudflare caching, you are going to run out of traffic pretty rapidly. The limit is 20TB / month (which you’d reach in roughly 9 days).

You are probably talking about VMs - those do have traffic limits. Servers, on the other side, with default 1Gbit NICs doesn't (let's say until you consume 80%+ of bandwidth for months) Quoting: > Traffic >All root servers have a dedicated 1 GBit uplink by default and with it unlimited traffic. Inclusive monthly traffic for servers with 10G uplink is 20TB. There is no bandwidth limitation. We will charge € 1 ($1.20)/…

Huh, this must have changed after I concluded my contract with them (several years ago).

Huh, archive.org tells me its been unlimited since at least 5 years ago, so I guess I must’ve just seen someone mention 20TB and felt it was a reasonsble limit :)

Re: OpenFreeMap survived 100k requests per second

#140

Post by ranger_danger »

immibis wrote: post #138
ranger_danger wrote: post #130

Earlier quoted context omitted.

Simply running "ipfs daemon" is enough to get hetzner to threaten canceling your service. They send nasty letters with pcap dumps of your "hacking" attempts and it doesn't matter how wrong you think they are.

clearly not true because I ran an ipfs daemon before and didn't get any nasty letters. I got nasty letters during the letterbombing attack on Tor relays. They were nasty letters forwarded by Hetzner from the people who actually wrote them. Pointing Hetzner to the page about the attack made them go away.

> clearly not true because it never happened to me personally

https://en.wikipedia.org/wiki/False_equivalence

Here's one of several emails they've sent me over the years, each one happening right after running "ipfs daemon": https://0x0.st/Kryd.png

There's lots of other easily-searchable accounts of people having the exact same experience:

https://github.com/ipfs/kubo/issues/4343

https://discuss.ipfs.tech/t/moved-ipfs-node-result-netscan-d...

https://github.com/ipfs/js-ipfs/issues/3982

https://github.com/filecoin-project/lassie/issues/121

https://github.com/covalenthq/refiner/issues/140

Post reply on HN