My £4 a month server can handle 4.2M requests a day • Hacker News Board

Re: My £4 a month server can handle 4.2M requests a day

#141

Post by blacklion » Tue, Sep 07, 2021, 10:50 PM UTC

joering2 wrote: post #113

blacklion wrote: post #101

Earlier quoted context omitted.

Crypto markets are very small :-) I'm working and company which process "real" exchanges, like NASDAQ, LSE, and, especially, OPRA feed. We've added 20+ crypto exchanges in our portfolio this year, and all of them are processed on one old server which is unable to process NASDAQ Total View in real-time anymore. On the other hand, whole OPRA feed (more than 5Gbit/s or 65B/day, yes, it is billions, messages of very opti…

Could you share some more about that very optimized binary protocol? I know there are ways to be more efficient than JSON but since you call it crappy, your solution must be much much better. Honestly interested to readup more.

It is not "our" protocol, it is protocol designed by exchange and we need to support it, as we can not change it :). Simple binary messages, with binary encoded numbers, etc. No string parsing, no syntax, nothing like this, only bytes and offsets. Think about TCP header, for example.

JSON is very inefficient both in bytes (32 bit price is 4 bytes in binary and could be 7+ bytes as string, think "1299.99" for example) and CPU: to parse "1299.99" you need burn a lot of cycles, and if it is number of cents stored as native 4-byte number you need 3 shifts and 4 binary ors at most, if you need to change endianness, and in most cases it is simple memory copy of 4 bytes, 1-2 CPU cycle.

When you have binary protocol, you could skip fields which you are not interested in as simple as "offset = offset + " (where is compile-time constant!) and in JSON you need to parse whole thing anyway.

Difference between converting binary packet to internal data structure and parsing JSON with same data to same structure could be ten-fold easily, and you need to be very creative to parse JSON without additional memory allocations (it is possible, but code becomes very dirty and fragile), and memory allocation and/or deallocation costs a lot, both in GC languages and languages with manual memory management.

Re: My £4 a month server can handle 4.2M requests a day

#142

Post by nine_k » Tue, Sep 07, 2021, 10:52 PM UTC

fhood wrote: post #67

nostrademons wrote: post #20

People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…

People may underestimate how fast modern machines are, but that is probably in part because, at least in my fairly relevant experience, I have literally never seen a CPU bottleneck under normal circumstances. Memory pressure is nearly always the driving issue.

The CPU is rarely used up to 100% because most code fails to utilize several cores efficiently.

OTOH a service loading the single core with the main thread is a frequent sight :( Interpreted languages like Python can easily spend 30% of time just on the deserialization overhead, converting the data from a DB into a result set, and then into ORM instances.

Re: My £4 a month server can handle 4.2M requests a day

#143

Post by ochoseis » Tue, Sep 07, 2021, 10:53 PM UTC

nostrademons wrote: post #83

fizwhiz wrote: post #46

Earlier quoted context omitted.

Would you mind describing your stack in more detail? Did you use gRPC with Go?

Sure, startup is defunct now and I think arbitrage & data on centralized exchanges is a dead market now. Wall Street HFTs got into the arbitrage game, and the data sites laypeople actually visit are the ones started in 2014. Codebase was pure server-side Kotlin running on the JVM. Jackson for JSON parsing, when the exchange didn't provide their own client library (I used the native client libraries when they did). Th…

Would you use Kotlin again for the back end? Having not yet used it for that purpose, it seems like you’d get the benefit of the JVM ecosystem along with a nice language (but perhaps too many power-features).

Re: My £4 a month server can handle 4.2M requests a day

#144

Post by zxcvbn4038 » Tue, Sep 07, 2021, 10:56 PM UTC

Combined with a CDN and you can do a lot more, I like to think in terms of origin requests rather than raw req/s. The problem I have is getting people to really understand how caching works at the cdn and browser layers and design their frontends and backend responses around that. There is also lot you can do with edge compute to clean the incoming requests before the CDN evaluates them to increase cache hits. Even if you are trying to give a "real time" view of data, caching it for even a second and allowing stale data to be served while it is updating can reduce origin requests significantly. I've seen people hammer sites hundreds of times a second looking for changes that only happen once every fifteen seconds or once per minute - the best thing you can do for yourself is handle all those requests at the CDN level (eventually you'll do log analysis and see the activity and can take other measures, but in the mean time don't let all of those requests go to the origin). Your CDN is probably giving you better rates for network egress then Amazon or Google anyway - the later are more focused on incentivizing you to use their ecosystem exclusively by penalizing you for sending data "outside". Cheap VPS hosts discourage you exceeding your bandwidth allocation because they are overselling their capacity and heavy usage upsets that - so again you want to shift as much as you can to your CDN.

Re: My £4 a month server can handle 4.2M requests a day

#145

Post by wpietri » Tue, Sep 07, 2021, 10:57 PM UTC

ppeetteerr wrote: post #97

wpietri wrote: post #68

Earlier quoted context omitted.

An awful lot of professional programmers work in such heavyweight contexts that they don't have a good idea of how fast modern hardware can be. I was talking with an architect at a bank whose team was having trouble getting under a 2-second maximum for page views. They blamed it on having to make TCP requests to other services, and said something like "at a couple hundred milliseconds per request, it adds up quickly!…

This is a person running a blog and pulling HTML data from a database. Is it that impressive that they are getting 50r/s?

It's not.

My point isn't that it's impressive in some ultra-tuned performance sense. It's that doing pretty mundane things on pretty basic servers is still very fast compare with a) the past, or b) what a lot of developers are used to professionally. That's why it is interesting to the crowd here.

Re: My £4 a month server can handle 4.2M requests a day

#146

Post by throwaway20371 » Tue, Sep 07, 2021, 11:02 PM UTC

Zababa wrote: post #107

dreyfan wrote: post #23

Earlier quoted context omitted.

Broadly speaking people on HN have no clue how to setup a performant httpd/app server and are impressed by abysmal performance/cost metrics like this or the MangaDex post. Everything these days is obscured through multiple layers of SaaS offerings and unnecessary bloat like kubernetes. ~10k rps (it was concurrent connections but close enough) was state of the art in 1999. Now 22 years later ~50 rps is somehow impress…

> ~10k rps (it was concurrent connections but close enough) was state of the art in 1999. Now 22 years later ~50 rps is somehow impressive. I honestly don't understand how that can be true. I'm not suggesting you're lying of course, but when you put it this way it's almost like people are actively trying to slow their programs down. I have a few ideas on why that might be the case (switch to slow interpreted language…

Software has become slower and less efficient because we have faster hardware to run it on and nobody's asking for the software to get more efficient. Back in the day you had inefficient, expensive hardware with limited software. There was demand to make software as performant as possible so it didn't cost you ten grand to serve a popular website. But now you can load up on hardware for pennies on what it used to cost. 16 cores? Sure! 32 gigs of ram? No sweat. Software can now be bloated and slow and it won't break the bank. On top of that, we now have free CDNs, free TLS certs, even free cloud hosting, and domains are a couple bucks. You can serve traffic for free today that would have cost $50K 15 years ago.

Sometimes there's also a misunderstanding of metrics that leads devs to not think about performance tuning. Like, "4.2M requests a day" is clearly incorrect at this 50rps benchmark. Traffic is not linear, it's bursty. You will never serve 50rps of human traffic steadily for 24 hours. If you're serving 4.2M requests per day, 90% of it will be in a 12 hour window, peaking at whenever people have lunch or get off work, with a short steep climb leading to a longer tail. So to not crash your site at peak visitorship, you realistically need to handle 300+ rps in order to achieve 4.2M requests per day. (But also that's requests per second... if under load it takes 5 seconds to load your site, you can still serve a larger amount of traffic, it's just slower... so a different benchmark is also "how many requests before the server literally falls over")

Re: My £4 a month server can handle 4.2M requests a day

#147

Post by modeless » Tue, Sep 07, 2021, 11:02 PM UTC

You don't need £4. As long as you can structure your site as a set of static files with interactivity done client side, even if some of the files change every minute or two, you can serve everything for $0 with Cloudflare in front of any free host. I've served 1M pageviews a day for $0 with Cloudflare + App Engine free tier and there's no reason it wouldn't scale to 100M or beyond.

Re: My £4 a month server can handle 4.2M requests a day

#148

Post by SilverRed » Tue, Sep 07, 2021, 11:03 PM UTC

fhood wrote: post #67

nostrademons wrote: post #20

People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…

People may underestimate how fast modern machines are, but that is probably in part because, at least in my fairly relevant experience, I have literally never seen a CPU bottleneck under normal circumstances. Memory pressure is nearly always the driving issue.

In my experience it is almost always the database holding things up. If your app does not use a database or it makes very simple use of it, then I'm not surprised it is blazing fast. As soon as you need to start joining tables and applying permissions to queries, it all gets slow.

Re: My £4 a month server can handle 4.2M requests a day

#149

Post by blacklion » Tue, Sep 07, 2021, 11:04 PM UTC

paraph1n wrote: post #117

joering2 wrote: post #113

Earlier quoted context omitted.

Could you share some more about that very optimized binary protocol? I know there are ways to be more efficient than JSON but since you call it crappy, your solution must be much much better. Honestly interested to readup more.

There are many binary encoding protocols. A popular one is protobufs[1], which is used by gRPC. [1]: https://developers.google.com/protocol-buffers

"old skool" exchanges uses either FIX (old and really vernose), FAST (binary encoding for FIX) or custom fixed-layout protocols.

Most big USA exchanges uses custom fixed-layout protocols, where each message is described in documentation, but not in machine-readable way. European ones still use FAST.

I didn't seen FIX in the wild for data feeds, but it is used for brokers, to submit orders to exchange (our company didn't do this part, we only consume feeds).

I don't know why, but all Crypto Exchanges use JSON, not protobufs or something like this, and didn't publish any formal schemes.

Fun fact: one crypto exchange put GZIP'ed and base64'ed JSON data into JSON which pushed to websocket, to save bandwidth. IMHO, it is peak of bad design.

Re: My £4 a month server can handle 4.2M requests a day

#150

Post by tristor » Tue, Sep 07, 2021, 11:06 PM UTC

The comments in this thread surprise me quite a lot. I suppose it shouldn't, but it does. This post + the response calls to the surface how badly basic system operations knowledge is needed in the industry and how much of it is missing from the toolkit of most developers.