Use one big server • Hacker News Board

Re: Use one big server

#421

Post by sanderjd » Wed, Aug 03, 2022, 2:24 AM UTC

throwfh80h82 wrote: post #406

endisneigh wrote: post #402

Earlier quoted context omitted.

Without details it's hard to really get anything from this. Could you share some?

I can share some. Had a similar experience as the parent comment. I do support "one big database" but it requires a dedicated db admin team to solve the tragedy of the commons problem. Say you have one big database. You have 300 engineers and 30-50 product managers shipping new features every day accountable to the C-Suite. They are all writing queries to retrieve the data they want. One more join, one more N+1 query…

I think this hits the nail right on the head, and it's the same criticism I have of and article itself: the framing is that you split up a database or use small vms or containers for performance reasons, but that's not the primary reason these things are useful; they are useful for people scaling first and foremost, and for technical scaling only secondarily.

The tragedy of the commons with one big shared database is real and paralyzing. Teams not having the flexibility to evolve their own schemas because they have no idea who depends on them in the giant shared schema is paralyzing. Defining service boundaries and APIs with clarity around backwards compatibility is a good solution. Sometimes this is taken too far, into services that are too small, but the service boundaries and explicit APIs are nonetheless good, mostly for people scaling.

Re: Use one big server

#422

Post by makeitdouble » Wed, Aug 03, 2022, 2:44 AM UTC

throwfh80h82 wrote: post #406

endisneigh wrote: post #402

Earlier quoted context omitted.

Without details it's hard to really get anything from this. Could you share some?

I can share some. Had a similar experience as the parent comment. I do support "one big database" but it requires a dedicated db admin team to solve the tragedy of the commons problem. Say you have one big database. You have 300 engineers and 30-50 product managers shipping new features every day accountable to the C-Suite. They are all writing queries to retrieve the data they want. One more join, one more N+1 query…

For whatever reason I've been thrown into a lot of companies at that exact moment when "hardware is cheap" and "not my problem" approaches couldn't cut it anymore...

So yes, it's super painful, and requires a lot of change in processes, mindsets, and it's hard to get everyone to understand things will get slower from there.

On the other end, micro-services and/or multi-DB is also super hard to get right. One of the surprise I had was all the "cache" that each services started silently adding on their little island when they realized the performance penalty they had from fetching data from half a dozen services on the more complicated operations. Or the same way DB abuse from one group could slow down everyone, and service abuse on the core parts (e.g. the "user" service) would impact most of the other services. More that a step forward, it felt a lot like a step sideways and continuing to do the same stuff, just in a different way.

My take from it was that teams that are good at split architectures are also usually good at monolith, and vice-versa. I feel from the parent who got stuck in the transition.

Re: Use one big server

#423

Post by disqard » Wed, Aug 03, 2022, 2:50 AM UTC

I didn't see the COST paper linked anywhere in this thread [0].

Excerpt from abstract:

We offer a new metric for big data platforms, COST, or the Configuration that Outperforms a Single Thread. The COST of a given platform for a given problem is the hardware configuration required before the platform outperforms a competent single-threaded implementation.

[0] https://www.usenix.org/conference/hotos15/workshop-program/p...

Re: Use one big server

#424

Post by octoberfranklin » Wed, Aug 03, 2022, 2:52 AM UTC

stevenjgarner wrote: post #65

saulrh wrote: post #28

Earlier quoted context omitted.

> competitively provide residential > Internet service to offset costs. I uh. Providing residential Internet for an apartment complex feels like an entire business in and of itself and wildly out of scope for a small business? That's a whole extra competency and a major customer support commitment. Is there something I'm missing here?

It depends on the scale - it does not have to be a major undertaking. You are right, it is a whole extra competency and a major customer support commitment , but for a lot of the entrepreneurial folk on HN quite a rewarding and accessible learning experience. The first time I did anything like this was in late 1984 in a small town in Iowa where GTE was the local telecommunication utility. Absolutely abysmal Internet…

> You can get a small rack-mount sheet metal pedestal with an AC power meter and an HVAC unit for under $2k.

Things have changed a lot and the dominant carriers are no longer willing to interconnect with small guys.

The anti-small bias now extends to the Department of Transportation in most states (which "owns" the right of way). In Washington, WSDOT has an entire set of rules for "financially small" (their term) telecoms, basically designed to prevent them from existing. They claim this is to prevent "financially small" providers from defaulting on damage they cause to the roadway ("not able to abate or correct their environmental damage").

https://www.google.com/search?hl=en&q=wsdot%20%22financially...

Re: Use one big server

#425

Post by sanderjd » Wed, Aug 03, 2022, 2:53 AM UTC

I didn't want to write a top-level comment and I'm sure few people will see this, but I scrolled down very far in this thread and didn't see this point made anywhere:

The article focuses almost entirely on technical questions, but the technical considerations are secondary; the reason so many organizations prefer cloud services, VMs, and containers is to manage the challenges of scaling organizationally, not technically.

Giving every team the tools necessary to spin up small or experimental services greases the skids of a large or quickly growing organization. It's possible to set this up on rented servers, but it's an up front cost in time.

The article makes perfect sense for a mature public facing service with a lot of predictable usage, but the sweet spot for cloud services is sprawling organizations with lots of different teams doing lots of different mostly-internally facing things.

Re: Use one big server

#426

Post by gtd » Wed, Aug 03, 2022, 2:56 AM UTC

threeseed wrote: post #308

Guid_NewGuid wrote: post #101

Earlier quoted context omitted.

I think a strong test a lot of "let's use Google scale architecture for our MVP" advocates fail is: can your architecture support a performant paginated list with dynamic sort, filter and search where eventual consistency isn't acceptable? Pretty much every CRUD app needs this at some point and if every join needs a network call your app is going to suck to use and suck to develop.

> if every join needs a network call your app is going to suck to use and suck to develop. And yet developers do this every single day without any issue. It is bad practice to have your authentication database be the same as your app database. Or you have data coming from SaaS products, third party APIs or a cloud service. Or even simply another service in your stack. And with complex schemas often it's far easier to…

> It is bad practice to have your authentication database be the same as your app database.

No, this is resume-driven-development, Google-scale-wannabe FUD. Understand your requirements. Multiple databases is non-trivial overhead. The only reason to add multiple databases is if you need scale that can't be handled via simple caching.

Of course it's hard to anticipate what level of scale you'll have later, but I can tell you this: for every tiny startup that successfully anticipated their scaling requirements and built a brilliant microservices architecture that proactively paved the way to their success, there's a 100 burnt out husks of companies that never found product market fit because the engineering team was too busy fantasizing about "web-scale" and padding their resume by overengineering every tiny and unused feature they built.

If you want to get a job at FAANG and suckle at the teat of megacorporations who's trajectory was all based on work done in the early 2000s, by all means study up on "best practices" to recite at your system design interview. On the other hand, if you want to build the next great startup, you need to lose the big co mentality and start thinking critically from first principles about power to weight ratio and YAGNI.

Re: Use one big server

#427

Post by octoberfranklin » Wed, Aug 03, 2022, 2:57 AM UTC

stevenjgarner wrote: post #128

xen2xen1 wrote: post #110

Earlier quoted context omitted.

What other benefits are there to being a "public telecommunication utility"?

The benefit that is obvious to the regulators is that you can charge money for services. So for example, offering telephone services requires being a LEC (local exchange carrier) or CLEC (competitive local exchange carrier). But even telephone services have become considerably unregulated through VoIP. It's just that at some point, the VoIP has to terminate/interface with a (C)LEC offering real dial tone and telephon…

> the competitive right to place your own facilities on telephone/power poles or underground in public right-of-way under the Telecommunications Act of 1996

This is not true. The FCC doesn't regulate the first pole attachment by a given attacher to poles owned by a given owner. The pole owners are basically free to use all sorts of lame reasons for refusing your first pole attachment request.

The FCC only gets involved when a company already has some (even just one) attachments and is getting rejected or stonewalled on making more attachments.

If you think about it, this is typical captured regulator behavior. The phone companies already have attachments to the electric utility's poles wherever they operate. So this lets the phone companies call in the FCC on any pole dispute. But it provides zero assistance to any new market entrants who want to compete with the existing phone company.

It also makes the decision to allow the first attachment a much harder decision for the pole owner, with the result being that the electrical utilities are incentivized to exclude new telecoms from competing with the phone company. But of course these new telecoms aren't trying to provide electrical services, so it doesn't look anticompetitive to a superficial analysis.

Re: Use one big server

#428

Post by octoberfranklin » Wed, Aug 03, 2022, 2:58 AM UTC

eek2121 wrote: post #339

stevenjgarner wrote: post #128

Earlier quoted context omitted.

The benefit that is obvious to the regulators is that you can charge money for services. So for example, offering telephone services requires being a LEC (local exchange carrier) or CLEC (competitive local exchange carrier). But even telephone services have become considerably unregulated through VoIP. It's just that at some point, the VoIP has to terminate/interface with a (C)LEC offering real dial tone and telephon…

> Other benefits to being a "public telecommunication utility" include the competitive right to place your own facilities on telephone/power poles or underground in public right-of-way under the Telecommunications Act of 1996. You will need to enter into and pay for a pole attachment agreement. Of course local governments can reserve the right to tariff your facilities, which has its own ugliness. Note that in many p…

Because AT&T never allowed Google their first attachment. Ma Bell is not stupid.

Re: Use one big server

#429

Post by andreimackenzie » Wed, Aug 03, 2022, 2:59 AM UTC

sanderjd wrote: post #421

throwfh80h82 wrote: post #406

Earlier quoted context omitted.

I can share some. Had a similar experience as the parent comment. I do support "one big database" but it requires a dedicated db admin team to solve the tragedy of the commons problem. Say you have one big database. You have 300 engineers and 30-50 product managers shipping new features every day accountable to the C-Suite. They are all writing queries to retrieve the data they want. One more join, one more N+1 query…

I think this hits the nail right on the head, and it's the same criticism I have of and article itself: the framing is that you split up a database or use small vms or containers for performance reasons, but that's not the primary reason these things are useful; they are useful for people scaling first and foremost, and for technical scaling only secondarily. The tragedy of the commons with one big shared database is…

These are real problems, but there can also be mitigations, particularly when it comes to people scaling. In many orgs, engineering teams are divided by feature mandate, and management calls it good-enough. In the beginning, the teams are empowered and feel productive by their focused mandates - it feels good to focus on your own work and largely ignore other teams. Before long, the Tragedy of the Commons effect develops.

I've had better success when feature-focused teams have tech-domain-focused "guilds" overlaid. Guilds aren't teams per-se, but they provide a level of coordination, and more importantly, permanency to communication among technical stakeholders. Teams don't make important decisions within their own bubble, and everything notable is written down. It's important for management to be bought in and value participation in these non-team activities when it comes to career advancement (not just pushing features).

In the end, you pick your poison, but I have certainly felt more empowered and productive in an org where there was effective collaboration on a smaller set of shared applications than the typical application soup that develops with full team ownership.

Re: Use one big server

#430

Post by throwawaylinux » Wed, Aug 03, 2022, 2:59 AM UTC

fleddr wrote: post #331

Our industry summarized: Hardware engineers are pushing the absolute physical limits of getting state (memory/storage) as close as possible to compute. A monumental accomplishment as impactful as the invention of agriculture and the industrial revolution. Software engineers: let's completely undo all that engineering by moving everything apart as far as possible. Hmmm, still too fast. Let's next add virtualization an…

> Software engineers: let's completely undo all that engineering by moving everything apart as far as possible. Hmmm, still too fast. Let's next add virtualization and software stacks with shitty abstractions.

That's because the concept which is even more impactful than agriculture and the computer, and makes them and everything else in our lives, is abstraction. It makes it possible to reason about large and difficult problems, to specialize, to have multiple people working on them.

Computer hardware is as full of abstraction and separation and specialization as software is. The person designing the logic for a multiplier unit has no more need to know how transistors are etched into silicon than a javascript programmer does.