Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use one big server

specbranch.com

Post reply on HN
Print view View original

551–560 of 601 posts

Re: Use one big server

#551

Post by pulse7 »

fleddr wrote: post #331

Our industry summarized: Hardware engineers are pushing the absolute physical limits of getting state (memory/storage) as close as possible to compute. A monumental accomplishment as impactful as the invention of agriculture and the industrial revolution. Software engineers: let's completely undo all that engineering by moving everything apart as far as possible. Hmmm, still too fast. Let's next add virtualization an…

Every new generation wants to invent a wheel until they learn it is already invented.

Re: Use one big server

#552

Post by Jolter »

tzs wrote: post #286
stevenjgarner wrote: post #6

If you are not maxing out or even getting above 50% utilization of 128 physical cores (256 threads), 512 GB of memory, and 50 Gbps of bandwidth for $1,318/month , I really like the approach of multiple low-end consumable computers as servers. I have been using arrays of Intel NUCs at some customer sites for years with considerable cost savings over cloud offerings. Keep an extra redundant one in the array ready to sw…

I don't understand the pedestal approach. Do you put your server in the pedestal, so the pedestal is in effect your data center?

I suppose a NUC or two will easily fit in there.

Re: Use one big server

#553

Post by cogman10 »

banku_brougham wrote: post #475
cogman10 wrote: post #434

Earlier quoted context omitted.

I did in the original comment. We have over 200 monolith applications each accessing overlapping schemas of data with their own sets of stored procedures, views, and direct queries. To migrate a portion of that data out into it's own database requires, generally, refactoring a large subset of the 200 monolith apps to no longer get all the data in one query, but rather a portion of the data with the query and the rest…

>Including things like a bunch of random scripts checked into who knows where that are apparently mission critical and reading/writing across the entire database. This hits pretty hard right now, after reading this whole discussion. When there is a galaxy with countless star systems of data its good to have locality owners of data who publish for their usage as domain leaders, and build a system that makes subscripti…

100% agreed and that's what I've been trying to promote within the company. It's simply hard to get the momentum up to really affect this change. Nobody likes the idea that things have to get a little slower (because you add a new layer between the data) before they can get faster.

Re: Use one big server

#554

Post by adam_arthur »

julianbuse wrote: post #506
adam_arthur wrote: post #260

I'm building an app with Cloudflare serverless and you can emulate everything locally with a single command and debug directly... It's pretty amazing. But the way their offerings are structured means it will be quite expensive to run at scale without a multi cloud setup. You can't globally cache the results of a worker function in CDN, so any call to a semi dynamic endpoint incurs one paid invocation, and there's no…

I'm not sure if you know this, and it might not be useful to you even if you do, but workers can interact with the cache directly: https://developers.cloudflare.com/workers/runtime-apis/cache...

Yes, but the worker is in front of the cache (have to pay for an invocation even if cached), and the worker only interacts with the closest cache edge node, not the entire CDN.

But yeah, there are a few hacky ways to work around things. You could have two different URLs and have the client check if the item is stale, if so, call the worker which updates it.

I'm doing something similar with durable objects. I can get it to be persistent by having a cron that calls it every minute and then setting an alarm loop within the object.

It's just super awkward. It feels like a design decision to drive monetization. Cloudflare would be perfect if they let you have a persistent durable object instance that could update global CDN content

It's still the best serverless dev experience for me. Can do everything via JS while having transactional guarantees and globally distributed data right at the edge

Re: Use one big server

#555

Post by adam_arthur »

anonymousDan wrote: post #467
adam_arthur wrote: post #260

I'm building an app with Cloudflare serverless and you can emulate everything locally with a single command and debug directly... It's pretty amazing. But the way their offerings are structured means it will be quite expensive to run at scale without a multi cloud setup. You can't globally cache the results of a worker function in CDN, so any call to a semi dynamic endpoint incurs one paid invocation, and there's no…

As someone who has only dabbled with serverless (Azure functions), the difficulty in setting up a local dev environment was something I found really off-putting. There is no way I am hooking up my credit card to test something that is still in development . It just seems crazy to me. Glad to hear Cloudflare workers provides a better experience. Does it provide any support for mocking commonly used services?

Yes, you can run your entire serverless infrastructure locally with a single command and close to 0 config.

It's far superior to other cloud offerings in that respect.

You can even run it live in dev mode and remote debug the code. Check out miniflare/Wrangler v2

Just wish they would have ability for persistent objects. Everything is still request driven, yet I want to schedule things on subminute schedules. You can do it today, but it requires hacks

Re: Use one big server

#556

Post by thayne »

nightshift1 wrote: post #377
thayne wrote: post #324

Earlier quoted context omitted.

It doesn't solve the problem that sending that snapshot to a backup location takes a long time.

Unless your storage is already mirrored off-site. Ex: EMC srdf

so say the disk fails on your main DB. or for some reason a customer needs data from 6 months ago, which is no longer in your local snapshots. In order to restore the data, you have to transfer the data for the full database back over.

With multiple databases, you only have to transfer a single database, not all of your data.

Re: Use one big server

#557

Post by anticristi »

fleddr wrote: post #331

Our industry summarized: Hardware engineers are pushing the absolute physical limits of getting state (memory/storage) as close as possible to compute. A monumental accomplishment as impactful as the invention of agriculture and the industrial revolution. Software engineers: let's completely undo all that engineering by moving everything apart as far as possible. Hmmm, still too fast. Let's next add virtualization an…

Let me add fuel to the fire. When I started my career, users were happy to select among a handful of 8x8 bitmap font. Nowadays, users expect to see a scalable male-doctor-skin-ton-1 emoji. The former can be implemented by bliting 8 bytes from ROM. The latter requires an SVG engine -- just to render one character.

While bloatware cannot be excluded, let's not forget that user expectations have temendously increased.

Re: Use one big server

#558

Post by hot_gril »

journey_16162 wrote: post #492
hot_gril wrote: post #361

The way I code now after 10 years: Use one big file. No executable I'm capable of writing on my own is complex enough to need 50 files spread across a 3-layers-deep directory tree. Doesn't matter if it's a backend, a UI, or what. There's no way your React or whatever tutorial example code needs that either. And you don't gain any meaningful organization splitting into files when there are already namespaces, classes,…

Reminds me of a company I used to work at which took a similar approach. We used one file per person policy, each developer had their own file that contained functionality developed by them, named like firstName_lastName.ext - everyone owned their file so we didn't have to worry about merge conflicts.

On the team at my day job, it'd be very bad for each person to strictly "own" their code like that because things get handed off all the time, but in some other situations I can see it making sense.

Re: Use one big server

#559

Post by hot_gril »

hot_gril wrote: post #354
kedean wrote: post #348

Earlier quoted context omitted.

Many databases can be distributed horizontally if you put in the extra work, would that not solve the problems you're describing? MariaDB supports at least two forms of replication (one master/replica and one multi-master), for example, and if you're willing to shell out for a MaxScale license it's a breeze to load balance it and have automatic failover.

Not without big compromises and a lot of extra work. If you want a truly horizontally scaling database, and not just multi-master for the purpose of availability, a good example solution is Spanner. You have to lay your data out differently, you're very restricted in what kinds of queries you can make, etc.

Clarification, you can make unoptimized queries on Spanner with a great degree of freedom when you're doing offline analysis, but even then it's easy to hit something that's too slow to work at all, whereas in Postgres I know it'd not be a problem.

Re: Use one big server

#560

Post by arinlen »

jd_mongodb wrote: post #542
Aeolun wrote: post #509

Earlier quoted context omitted.

DynamoDB (and Mongo) is nice, right up until you need those relations. I haven’t found a document oriented database that gives me the consistency guarantees of a RDBMS yet.

You must not have looked at MongoDB. We have been delivering fully consistent ACID transactions since 4.0 which shipped several years. Yes, Jepsen did find some issues with the initial release of ACID transactions and yes, we fixed those problems pretty rapidly.

> Yes, Jepsen did find some issues with the initial release of ACID transactions (...)

By "some issues" you mean lost data and violated causal by default.

https://www.infoq.com/news/2020/05/Jepsen-MongoDB-4-2-6/

And no, the issues were not fixed. Jepsen pointed out that "the newer MongoDB 4.2.6 has more problems" including “retrocausal transactions”.

https://jepsen.io/analyses/mongodb-4.2.6

From Jepsen's report:

> Jepsen evaluated MongoDB version 4.2.6, and found that even at the strongest levels of read and write concern, it failed to preserve snapshot isolation. Instead, Jepsen observed read skew, cyclic information flow, duplicate writes, and internal consistency violations.

Post reply on HN