Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My £4 a month server can handle 4.2M requests a day

mark.mcnally.je

Post reply on HN
Print view View original

121–130 of 479 posts

Re: My £4 a month server can handle 4.2M requests a day

#121

Post by adtac »

habibur wrote: post #89
markandrewj wrote: post #38

Normally benchmarks for things like this are measured in how many concurrent requests can be handled, i.e the C10K problem, not by how many requests you are able to serve in a day. It's also well known that you can serve a large amount of requests on limited hardware. https://en.wikipedia.org/wiki/C10k_problem "By the early 2010s millions of connections on a single commodity 1U rackmount server became possible: over…

Right. I calculated what 5m/day converts into. And it's like 60 req/sec. Considering non even distribution and spikes, I would assume its like 200req/sec.

unrelated but 4x increase isn't really a spike

Re: My £4 a month server can handle 4.2M requests a day

#122

Post by nostrademons »

joering2 wrote: post #113
blacklion wrote: post #101

Earlier quoted context omitted.

Crypto markets are very small :-) I'm working and company which process "real" exchanges, like NASDAQ, LSE, and, especially, OPRA feed. We've added 20+ crypto exchanges in our portfolio this year, and all of them are processed on one old server which is unable to process NASDAQ Total View in real-time anymore. On the other hand, whole OPRA feed (more than 5Gbit/s or 65B/day, yes, it is billions, messages of very opti…

Could you share some more about that very optimized binary protocol? I know there are ways to be more efficient than JSON but since you call it crappy, your solution must be much much better. Honestly interested to readup more.

Some Googling turned up this protocol descriptor:

https://uploads-ssl.webflow.com/5ba40927ac854d8c97bc92d7/5bf...

If you're optimizing for latency JSON is pretty terrible, but most people who use it are optimizing for interoperability and ease of development. It works just fine for that, and you can recover decent bandwidth just by compressing it.

Re: My £4 a month server can handle 4.2M requests a day

#123

Post by great-potential »

sleepyhead wrote: post #106
great-potential wrote: post #65

Earlier quoted context omitted.

I mean this is just basic stuff, for me it just sounds like a developer putting a site in production, no auto-scaling, patching ... might as well outsource this to a CDN since there is no database/redis/varnish ...

So...someone else's web server instead?

No offense but I think you've missed the point.

Re: My £4 a month server can handle 4.2M requests a day

#124

Post by meltedcapacitor »

dirkg wrote: post #92
nostrademons wrote: post #20

People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…

may be OT, but how do you subscribe to these trade feeds, is there a unified service or do you need to do it individually for each source, and how much does it cost approximately ? I'm guessing if you put all this data into Kinesis or message queues it would end up costing quite a bit more.

Never heard of a crypto exchange that charges for data feeds, the norm is free and fast. One of the positive of the industry compared to old school finance.

They're rent seeking in other ways though, no worries.

Re: My £4 a month server can handle 4.2M requests a day

#125

Post by nostrademons »

dirkg wrote: post #92
nostrademons wrote: post #20

People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…

may be OT, but how do you subscribe to these trade feeds, is there a unified service or do you need to do it individually for each source, and how much does it cost approximately ? I'm guessing if you put all this data into Kinesis or message queues it would end up costing quite a bit more.

There are probably unified services that let you do it - I was kinda competing in this area but didn't want to deal with enterprise sales, and it's a bit of a hard sell anyway.

If you do it individually, there are public developer docs for each exchange that explain how their API works. It's generally free as long as you're not making a large number of active trades.

Re: My £4 a month server can handle 4.2M requests a day

#126

Post by celsoazevedo »

sigg3 wrote: post #63

Put Wordpress on it, and do a new battery of TTFB tests ;)

Even WordPress would work fine if we use a plugin like WP Super Cache (no idea why they don't cache things by default). It wouldn't beat a simple static page, but WordPress + Cache plugin + cheap VPS can easily handle #1 on HN.

Re: My £4 a month server can handle 4.2M requests a day

#127

Post by bochoh »

giancarlostoro wrote: post #80
Gepsens wrote: post #49

Earlier quoted context omitted.

I'm running a crypto trading platform I'm developing on 30$ on DigitalOcean. I coded exclusively in Rust and recently added a dynamic interface to python. Today during the BTC crash it spiked at 20k events/s, and that's only incoming data.

is that $30 for a single droplet or is it spread out between a few different services? I'm kind of curious since I use DO for small projects myself.

I’m also curious on this. My stack currently is a SQLite file -> ORM -> .net core on Linux on a single box

Re: My £4 a month server can handle 4.2M requests a day

#128

Post by idworks1 »

One of my proudest moment in my career is when I lowered our app processing time from ~8hrs to 17 minutes. When I deployed my first update, it reduced it to 2 hours. The sysadmin immediately contacted me that there was something unusual. I confirmed the results but he was skeptical.

Then with my second update, he told me that the app must be broken or that the script must be dying. There is no way it could complete this fast.

What was the issue? We processed terabytes of data. Each and every single line processed created a new connection to the database and left it hanging. A try catch was added when the connections failed and restarted the process. Removing the connection from the for loop and properly handling it reduced the time drastically.

And... why would you loop through millions of records when you can use batches? Also this was a phperlbashton* script. I turned it into a single PHP script and called it a day.

As a consequence, backup time was reduced to 2 hours as opposed to 12 hours (no one was allowed on the website until the back up was done).

Modern machines are incredibly fast.

* PHP/Perl/Bash/Python

Re: My £4 a month server can handle 4.2M requests a day

#129

Post by JJMcJ »

danudey wrote: post #90
nostrademons wrote: post #20

People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to we…

A lot of that is due to absolutely lousy code. We had a system management backend at my last company. Loading the users list was unbearably slow; 10+ seconds on a warm cache. Not too terrible, except that most user management tasks required a page reload, so it was just wildly infuriating. Eventually I took a look at the code for the page, which queried LDAP for user data and the database for permissions data. It did…

Your pattern is quite powerful: get data from several sources and do the rearranging on the client (which might be a web server), instead of multiple interactions for each data item.

For SQL you can also do a stored procedure. Sometimes that works well if you are good at your DBMS's procedure language and the schema is good.

Re: My £4 a month server can handle 4.2M requests a day

#130

Post by great-potential »

idworks1 wrote: post #128

One of my proudest moment in my career is when I lowered our app processing time from ~8hrs to 17 minutes. When I deployed my first update, it reduced it to 2 hours. The sysadmin immediately contacted me that there was something unusual. I confirmed the results but he was skeptical. Then with my second update, he told me that the app must be broken or that the script must be dying. There is no way it could complete t…

omg what is this, we need some Brendan Gregg posts to elevate the level here, this is HN not reddit :D
Post reply on HN