Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use one big server

specbranch.com

Post reply on HN
Print view View original

321–330 of 601 posts

Re: Use one big server

#321

Post by nicoburns »

Gigachad wrote: post #306
manigandham wrote: post #200

Earlier quoted context omitted.

Why would you break apart a microservice? Any why do you need to use/split into microservices anyway? 99% of apps are best fit as monolithic apps and databases and should focus on business value rather than scale they'll never see.

Where I work we are looking at it because we are starting to exceed the capabilities of one big database. Several tables are reaching the billions of rows mark and just plain inserts are starting to become too much.

Yeah, the at the billions of rows mark it definitely makes sense to start looking at splitting things up. On the other hand, the company I worked for split things up from the start, and when I joined - 4 years down the line - their biggest table had something like 50k rows, but their query performance was awful (tens of seconds in cases) because the data was so spread out.

Re: Use one big server

#322

Post by thayne »

mike_hearn wrote: post #267
thayne wrote: post #265

Earlier quoted context omitted.

One issue I've seen with this is that if you have a single, very large database, it can take a very, very long time to restore from backups. Or for that matter just taking backups. I'd be interested to know if anyone has a good solution for that.

Presumably it doesn't matter if you break your DB up into smaller DBs, you still have the same amount of data to back up no matter what. However, now you also have the problem of snapshot consistency to worry about. If you need to backup/restore just one set of tables, you can do that with a single DB server without taking the rest offline.

> you still have the same amount of data to back up no matter what

But you can restore/back up the databases in parallel.

> If you need to backup/restore just one set of tables, you can do that with a single DB server without taking the rest offline.

I'm not aware of a good way to restore just a few tables from a full db backup. At least that doesn't require copying over all the data (because the backup is stored over the network, not on a local disk). And that may be desirable to recover from say a bug corrupting or deleting a customer's data.

Re: Use one big server

#323

Post by arwhatever »

Recent team I was on used one big server.

Wound up spawning off a separate thread from our would-be stateless web api to run recurring bulk processing jobs.

Then coupled our web api to the global singleton-esque bulk processing jobs thread in a stateful manner.

The wrapped actors up on actors on top of everything to try to wring as much performance as possible out of the big server.

Then decided they wanted to have a failover/backup server but it was too difficult due to the coupling to the global singleton-esque bulk processing job.

[I resigned at this point.]

So yeah color me skeptical. I know every project's needs are different, but I'm a huge fan of dumping my code into some cloud host that auto-scaled horizontally, and then getting back to writing more code that provides some freeeking busines value.

Re: Use one big server

#324

Post by thayne »

hamandcheese wrote: post #314
mgiampapa wrote: post #291

Earlier quoted context omitted.

This isn't really a backup, it's redundancy which is good thing but not the same as a backup solution. You can't get out of a drop table production type event this way.

If you stop at the first bullet point then you have a backup solution.

It doesn't solve the problem that sending that snapshot to a backup location takes a long time.

Re: Use one big server

#325

Post by drewg123 »

efortis wrote: post #156

Some comments wrongly equate bare-metal with on-premise. Bare-metal servers can be rented out, collocated, or installed on-premise. Also, when renting, the company takes care of hardware failures. Furthermore, as hard disk failures are the most common issue, you can have hot spares and opt to let damaged disks rot, instead of replacing them. For example, in ZFS, you can mirror disks 1 and 2, while having 3 and 4 as h…

720Gb/s actually. Those last 20-30Gb/s were pretty hard fought :)

Re: Use one big server

#326

Post by dsr_ »

hamandcheese wrote: post #310
dsr_ wrote: post #282

Earlier quoted context omitted.

Here's the way it works for, say, Postgresql: - you rsync or zfs send the database files from machine A to machine B. You would like the database to be off during this process, which will make it consistent. The big advantage of ZFS is that you can stop PG, snapshot the filesystem, and turn PG on again immediately, then send the snapshot. Machine B is now a cold backup replica of A. Your loss potential is limited to…

Do you even have to stop Postgres if using ZFS snapshots? ZFS snapshots are atomic, so I’d expect that to be fine. If it wasn’t fine, that would also mean Postgres couldn’t handle power failure or other sudden failures.

You have choices.

* shut down PG. Gain perfect consistency.

* use pg_dump. Perfect consistency at the cost of a longer transaction. Gain portability for major version upgrades.

* Don't shut down PG: here's what the manual says:

However, a backup created in this way saves the database files in a state as if the database server was not properly shut down; therefore, when you start the database server on the backed-up data, it will think the previous server instance crashed and will replay the WAL log. This is not a problem; just be aware of it (and be sure to include the WAL files in your backup). You can perform a CHECKPOINT before taking the snapshot to reduce recovery time.

* Midway: use SELECT pg_start_backup('label', false, false); and SELECT * FROM pg_stop_backup(false, true); to generate WAL files while you are running the backup, and add those to your backup.

Re: Use one big server

#327

Post by rszorness »

thayne wrote: post #265
belak wrote: post #64

Earlier quoted context omitted.

This is absolutely true - when I was at Bitbucket (ages ago at this point) and we were having issues with our DB server (mostly due to scaling), almost everyone we talked to said "buy a bigger box until you can't any more" because of how complex (and indirectly expensive) the alternatives are - sharding and microservices both have a ton more failure points than a single large box. I'm sure they eventually moved off t…

One issue I've seen with this is that if you have a single, very large database, it can take a very, very long time to restore from backups. Or for that matter just taking backups. I'd be interested to know if anyone has a good solution for that.

Try out pg_probackup. It works on database files directly. Restore is as fast as you can write on your ssd.

I've setup a pgsql server with timescaledb recently. Continuing backup based on WAL takes seconds each hour and a complete restore takes 15 minutes for almost 300 GB of data because the 1 GBit connection to the backup server is the bottleneck.

Re: Use one big server

#328

Post by nicoburns »

oceanplexian wrote: post #172
darkr wrote: post #162

Earlier quoted context omitted.

> Customers who have invested millions of dollars > … > an hour of outage would lose us $1M+ in business Given (excluding us-east-1) you’re looking at maybe an hour a year on average of regional outage, sounds like best case break even on that investment?

I'm going to say that an hour a year is wildly optimistic. But even then, that puts you at 4 nines (99.99%) which is comparatively awful, consider that an old fashioned telephone using technology from the 1970s will achieve on average, 5 9's of reliability, or 5.26 minutes of downtime per year, and that most IT shops operating their own infrastructure contractually expect 5 9's from even fairly average datacenters an…

I was amused when I joined my current company to find that our contracts only stipulate one 9 of reliability (98%). So ~30 mins a day or ~14 hours a month is permissible.

Re: Use one big server

#329

Post by riku_iki »

lmm wrote: post #315
cfors wrote: post #56

Yep, there's a premium on making your architecture more cloudy. However, the best point for Use One Big Server is not necessarily running your big monolithic API server, but your database. Use One Big Database. Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want t…

If you do this then you'll have the hardest possible migration when the time comes to split it up. It will take you literally years, perhaps even a decade. Shard your datastore from day 1, get your dataflow right so that you don't need atomicity, and it'll be painless and scale effortlessly. More importantly, you won't be able to paper over crappy dataflow. It's like using proper types in your code: yes, it takes a b…

> Shard your datastore from day 1

what about using something like cocroach from day 1?

Re: Use one big server

#330

Post by efortis »

drewg123 wrote: post #325
efortis wrote: post #156

Some comments wrongly equate bare-metal with on-premise. Bare-metal servers can be rented out, collocated, or installed on-premise. Also, when renting, the company takes care of hardware failures. Furthermore, as hard disk failures are the most common issue, you can have hot spares and opt to let damaged disks rot, instead of replacing them. For example, in ZFS, you can mirror disks 1 and 2, while having 3 and 4 as h…

720Gb/s actually. Those last 20-30Gb/s were pretty hard fought :)

Yeah. Thank you!
Post reply on HN