Live data from Hacker News

Self-Hosting Dozens of Web Applications and Services on a Single Server

cprimozic.net

141–150 of 285 posts

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#141
post #12

Is your 64MB memory ECC? Why do you RAID-1 NVMe? They are likely to fail simultaneously so maybe make them independent and schedule regular backups from one volume to the other, or to a remote disk drive somewhere.

I don't think OVH offers any dedicated servers without ECC. Their cheaper offspring, Kimsufi (now a separate company since quite a few years) may, although I'm not sure. Even the cheapest OVH dedicated server, starting at about 50 EUR / month, has 16 GB of RAM and it's ECC.

Many Kimsufi offerings are not fitted with ECC, I don't think some of the lower-end CPUs (the Atoms) even support it at all.

Their middle-budget brand SoYouStart seens to only offer machines with ECC, I started using one recently, so I assume you are right and the main brand operates likewise.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#142
post #62

Earlier quoted context omitted.

Where did you read that? It doesn't sound implausible, but if we're talking 70W idle for a 24/7 server, that does add up so I'd be interested in where the cut-off point is. Edit: just realized the 70W figure is in a sibling thread, not a (grand)parent, but I'm still interested regardless!

Some sources, not very recent but with great links to more detailed studies: https://css.umich.edu/factsheets/green-it-factsheet https://www.fcgov.com/climatewise/pdf/computers.pdf Arguably, none of these account for the pollution due to extraction/refinement of materials, which is another concern raised with building more stuff. Recycling is also a polluting process (usually less so than new extraction) but unfortun…

have you come across Harun Šiljak's fantastic piece in the Science for the People magazine titled 'Opening This Article Voids Warranty'?

"Repair is not an auxiliary, optional part of the economy. Repair is care, at every stage of the supply chain, and at every scale. Those in dire need of repair for their devices are, just like Terry Pratchett’s Theory of Economic Injustice predicts, in the crosshairs of widening economic inequality.4 Already-impoverished consumers are encouraged to buy ever-newer products (and other planned obsolescence mechanisms have co-evolved with the boom of overproduction for the Global North). These products are notoriously hard to repair by design and require repeated purchases, exposing the long-term scheme of the manufacturers. Mineral extraction necessary for the manufacture of new hi-tech devices inflicts death and immiseration on surrounding populations.5 A community that sees no value to repair is a community that cannot respond to the crisis of capitalism, and is merely its hostage. Repair, as an act of reclaiming technology, is ongoing in the Global North and South with complementary driving forces and problems."

[...]

"A classic strategy of anti-repair design is avoiding modularity. If the parts of the device are strategically bundled together so that failure of one part requires replacement of a whole unit, it is not “repair” anymore. While it happens with mechanical components as well, the electronic version of this strategy is worthy of closer examination. Unlike a gearbox whose miniaturization still results in an assemblage of separate gears that might be smaller and harder to replace, miniaturization in electronics was driven by manufacture of monolithic semiconductor structures. Control systems that have previously been implemented with, for example, a collection of discrete transistors and diodes (basic electronic components that are easily replaceable) have been revamped as embedded systems: for the same functionality they now use a microchip with software running on it. Access to the software is not provided. Access to the chip itself is not provided as it is commonly “globbed” (covered with a black blob of epoxy). On top of this, the manufacturer takes precautions to prevent you from replacing the entire controller with a different, customized controller on your own. Here I return to the robotic arm: what kind of a controller do you want such a mechanism to have? The odds are that the same arm might be deployed in a myriad of different settings and scenarios, and needs tweaking of the controller. The “body without organs” controller under the blob of epoxy offers no modularity, no chance to expand, reduce, or in any other way customize inputs, outputs, or processing capabilities. The individual components that might be at a higher risk of damage (e.g. transistors) don’t exist anymore, so every failure takes the entire block of silicon down with it. And finally, if product support is discontinued, the manufacturer goes out of business, or chooses to change the business model into selling a service rather than a product, the controller is expected to become an unusable brick. To make things worse, by making non-standard interfaces with mechanics and/or placing restrictive licenses on its software, the brick is supposed to be irreplaceable, hence rendering the entire robotic arm unusable (even if the mechanics are perfectly sound).

The loss of repairability is not a consequence of technological progress or increasing complexity—it should arguably be the opposite. Complex systems science pioneer W. Brian Arthur explains the two primary mechanisms of improving a technology: “internal replacement,” or changing the existing parts; and “structural deepening,” which means adding new components.10 Neither of these require that new parts and components cannot be modular, replaceable, and repairable. Complexity, in fact, is all about modularity and heterogeneity and can be an argument in favor of repair. The concepts of internal replacement and structural deepening, if anything, are the philosophy of repair as a creative process. New parts or mechanisms that come from repair contribute to an invention: potential new applications of the device, a new approach to manufacturing, and personalization of the item. A creatively repaired device is where the social network merges with the technological one. However, that is not in the interests of the manufacturing lobby: this network is one of capital accumulation.

The other aforementioned strategy of disabling repair is the legal one. To keep this grip of capital on the technology and knowledge of manufacturing and repair, the opponents of repair create the illusion of illegality: Looking under the hood should be taboo, understanding how things work should be perceived as illegal, and the concept of patents and protection of intellectual property should be regurgitated as respect for science and protecting the world from anarchy. Big manufacturers such as Apple also grasp at other legal straws such as privacy concerns.11

Bogus legal barriers run against the basic principles of science and engineering. Take, for example, the concept of reverse engineering. Finding out how a piece of hardware or software works by observing its inputs and outputs is an essential part of repair in modern technological systems. Often portrayed as illegal, this activity does not go against trade secrets laws. Instead, it becomes an issue concerning terms and conditions agreed to by user and manufacturer.12 Among legal contracts, “terms and conditions” represent a world of their own, with clauses that are often void, unenforceable, or plain illegal.13 The “opening box voids warranty” stickers mentioned earlier are a blatant example, but not the only one. Through lobbying, manufacturers erect new legal barriers where previously there had been none: when the Unitron Mac 512, a Brazilian clone of the Apple Macintosh was developed in the mid-eighties, it infringed no laws in Brazil. Only after heavy lobbying from Apple and threats of sanctions from the US government did the Brazilian government introduce a completely new “software law” to stop the sale of the reverse-engineered Macintosh.14"

source: https://magazine.scienceforthepeople.org/vol24-2-dont-be-evi...

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#143

Thanks for this article, it's great to see people caring for their server (does it have a name?) and not defaulting to the serverless craze. Here's a few thoughts :) > there is some small downtime when I deploy new versions of things since I don't have any load balancing or rolling deployments It's entirely possible to achieve, depending on your stack. `nginx -s reload` will reload the entire config without killing e…

> there is some small downtime when I deploy new versions of things

Some time ago I was looking for an easier way to fix this. It seemed to me that a good way would be to have the reverse proxy (e.g. nginx or similar) hold the requests into the app restarts. Fit the user, this would mean a ~10s hiccup rather than 504 errors.

I didn't find an easy way to do it with nginx though and was sort of disappointed. Maybe other reverse proxies make this easier? Or maybe there is a stand-alone tool to do this?

[edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so]

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#144
post #66

Three independent, but somewhat related thoughts on this topic: 1). On HOWTO articles about infra (1/2): I'd like to see more articles that lead with requirements, rather than setups that then justify the setup with requirements. Like, congrats, you managed to host a bunch of web applications via containers on a dedicated server. It's really nice for a super personal project and I'm sure it helped OP gain a lot of op…

AWS is complexity-as-a-service which abstracts away so much that you pay a HUGE price when it's time to scale -- and it's a confusing matrix of tradeoffs. I find it easier to login to a server, "sudo apt-get install" and tail some logs, than to try to manage the monstrosity that is AWS. Products that go horizontal-cloud-first are easy to scale, but burn an order of magnitude more money, with far worse realtime performance. At the end of the day, AWS is just an abstraction on top of CPU, RAM and disk -- if your app makes bad use of underlying CPU, RAM and disk, then no amount of cloud magic is going to fix that, it'll just mask the problem and you'll end up paying an arm and a leg. AWS/GCloud are actually incentivized to make it hard to debug and optimize performance, they would want you to waste resources on their platform.

See also: https://news.ycombinator.com/item?id=29660117

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#145

I’m using the same server type from OVH in Hillsboro and it’s great. They frequently do sales. Highly recommend anyone interested to go for the nvme upgrade. I rent three now total, one for production sites, one as a development machine strictly for vscode and another for development / staging / misc. waaaaaay overkill but it’s been a huge quality of life improvement. For containers I just use a couple docker compose…

I’m running collocated servers with simple docker compose file too. Actually https://mailsnag.com is hosted on one of the servers and is using single docker compose file for deployment. Not sure why the author had to record every single docker run command when docker compose file manages all of that for you.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#146
Traefik Proxy[0] was a game changer for my self-hosted setup of Docker containers.

Traefik can read labels applied to Docker containers (easily done with docker-compose) and setup the proxy for you as containers come and go. Even maintains the Lets Encrypt certificates seamlessly.

[0] https://traefik.io/traefik/

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#147
post #144
post #66

Three independent, but somewhat related thoughts on this topic: 1). On HOWTO articles about infra (1/2): I'd like to see more articles that lead with requirements, rather than setups that then justify the setup with requirements. Like, congrats, you managed to host a bunch of web applications via containers on a dedicated server. It's really nice for a super personal project and I'm sure it helped OP gain a lot of op…

AWS is complexity-as-a-service which abstracts away so much that you pay a HUGE price when it's time to scale -- and it's a confusing matrix of tradeoffs. I find it easier to login to a server, "sudo apt-get install" and tail some logs, than to try to manage the monstrosity that is AWS. Products that go horizontal-cloud-first are easy to scale, but burn an order of magnitude more money, with far worse realtime perfor…

Exactly, at this decade it's still a ripoff = huge margin for them, but market is there, one can't deny that. It'll even out eventually so they're on par with smaller shops and dedicated/colocation, eventually they'll have no choice but to cut margins and leverage scale to kick out smaller competition, they'll delay it, of course, as much as possible with lock-in dark techniques etc – but at the end there is no reason, in theory, why they should not be more attractive price wise with anything else you can come up with. At this decade-tick timeline they're happy taking huge margins and people just throw money without much of first principle etc thinking.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#148
post #144
post #66

Three independent, but somewhat related thoughts on this topic: 1). On HOWTO articles about infra (1/2): I'd like to see more articles that lead with requirements, rather than setups that then justify the setup with requirements. Like, congrats, you managed to host a bunch of web applications via containers on a dedicated server. It's really nice for a super personal project and I'm sure it helped OP gain a lot of op…

AWS is complexity-as-a-service which abstracts away so much that you pay a HUGE price when it's time to scale -- and it's a confusing matrix of tradeoffs. I find it easier to login to a server, "sudo apt-get install" and tail some logs, than to try to manage the monstrosity that is AWS. Products that go horizontal-cloud-first are easy to scale, but burn an order of magnitude more money, with far worse realtime perfor…

In the long run and for most higher global nationwide scale AWS/GCP will be cheaper from 1) general reduced price every few quarters/years 2)private contract. It’s important to realize those cloud providers does a whole bunch of work to reduce cost internally so customers can pay less (eventually)

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#149
post #28

Earlier quoted context omitted.

Raid-1 is doing exactly what you recommend without any effort. A perfect replica of the disk. And if the other one dies, who cares, the beauty of raid-1 is you don‘t need the other one to have a full copy.

I think the idea here is that RAID1 forces both SSDs to write every block at the same time. With identical SSDs and very similar write endurance profiles you're likely to have them both give up at the same time. Even just a nightly rsync would decorrelate what is right now nearly perfect correlation.

> identical SSDs ... you're likely to have them both give up at the same time

I wouldn't say much more likely than with traditional drives, unless you are getting towards EOL in terms of how much rewriting has been done but after that much time I'd expect randomness to separate things out at least a bit.

The main concern I have with either drive type is finding out blocks that haven't been touched in ages have quietly gone bad, and you don't notice until trying to read them to rebuild the array once a failed drive has been replaced - that applies equally unless you run a regular verify. Other failure modes like the controller dying are less likely to happen concurrently, unless there is a power problem or some such in the machine of course, but again these might affect all drive types and this is one of the reasons you need proper backups as well as RAID (the age-old mantra: RAID is not a backup solution, RAID increases availability and reduces the chance you'll need to restore from backup).

Having said that, my home server deliberately has different drives (different controllers, highly unlikely that even if the memory on each is from the same manufacturer it is from the same batch) in its R1 mirror of SSDs, just in case. The spinning metal drives it also has in another array were bought in a way to decrease the chance of getting multiple from one batch in case it is a bad batch.

> nightly rsync

The problem with that and other filesystem level options is that depending on the filesystem and what you are running, some things might be missed due to file-locking. As RAID is block-device level this is never going to be the case, though of course in either case you can't catch what is in RAM and not yet physically written.

Of course this problem will be present for most off-device backup solutions too, so you could use the same mitigation you have there for backing up between the drives too.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#150
Yet another modern "cloud native" era developer discovers how overpriced and underpowered most cloud offerings are.

Here's what they did folks: they set cloud prices a while back, did not drop them as Moore's Law delivered more power but instead pocketed the profit, and meanwhile ran an entire "cloud native" development push to encourage development practices that maximize cloud lock-in (both through dependence on cloud services and encouraging complexity).

Oh, and bandwidth is ludicrously overpriced in the cloud. You can transfer hundreds of terabytes a month outbound for <$500 if you know where to look. No I don't mean from static data either. Look at bare metal hosting.

Post reply on HN