Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-Host All the Things?

tedium.co

Post reply on HN
Print view View original

281–290 of 295 posts

Re: Self-Host All the Things?

#281

Post by XCSme »

e2e4 wrote: post #115
margorczynski wrote: post #29

As for hosting your own apps I found Hetzner VPS or something similar to be very good. Just pack them up into a docker-compose with your CI/CR pushing an image into a repository and you can host a lot of low-medium traffic solutions on a single box with the cost being a fraction of "the Cloud" (especially PaaS). On the box there is a single Nginx acting as a reverse proxy to the exposed compose ports offloading SSL.…

Similar, but am using CapRover for docker images management and setup.

I also use CapRover. The initial setup was harder than I expected and there definitely is some learning curve and learning how to package your own apps or how to even make some of the pre-made apps to work (it doesn't really work out of the box). Once you get used to it though, it makes it really easy to add new applications and put them live. If I have an idea for a website, I can just click new WordPress instance, point the domain and done. If I want to also add a forum for that website, I just click new Disqus instance.

Re: Self-Host All the Things?

#282

Post by 8fingerlouie »

Terretta wrote: post #263
8fingerlouie wrote: post #215

Earlier quoted context omitted.

> I had fun doing this until I had kids. As i keep telling people, self hosting is fun as long as your user count is 1. When it grows beyond that, you suddenly have a SLA. I self hosted almost everything (e-mail is pointless from privay concerns), and when we had kids i moved to a dual Synology setup with a single proxmox server for running services. Fast forward some years and electricity suddenly costs about an arm…

> As i keep telling people, self hosting is fun as long as your user count is 1. When it grows beyond that, you suddenly have a SLA. This is the principle I.T. departments fail to grasp. > Kids also brings all sorts of devices with new and exiting exploits/vira... Curiosity: while vira is arguably less wrong, hackers of a certain age would have expected viri or virii , which are more wrong: https://en.wikipedia.org/w…

> Meanwhile, in "Kids also brings" – I fully support what you did there!

It of course also helps that in 2023, literally all school work, for better or for worse, is done through the cloud. I wrote printers above, and yes, they do have access to the printers, but apart from our 3D Printers, the laser/inkjet printers have seen very little use.

Here the schools use Microsoft, which means assignments are done in Word/Excel, and handed in online either through a school portal, or shared from OneDrive.

I won't get into the privacy details, but we do have some fairly strict laws concerning kids and identity protection (a thing that recently got Google kicked out from the educational sector), so while not ideal it is probably not as bad as it sounds.

Apart from school work, their needs are mostly only local peer to peer networking for games, and/or internet access, and all can be accomplished by simply sticking them on a "less restricted" guest network, while at the same time making reasonably sure they're not wiping out the rest of the households computers :)

The firewall also runs a very small subset of IDS/IPS rules, mostly concerning malware/bot rules, and we use a NextDNS profile per subnet to filter out the worst.

> Curiosity: while vira is arguably less wrong, hackers of a certain age would have expected viri or virii

My bad, i used the latin plural form of virus, which is vira. In any case, my network setup should keep most vira, viruses or virii out :)

Re: Self-Host All the Things?

#283

Post by 8fingerlouie »

grepfru_it wrote: post #259
8fingerlouie wrote: post #215

Earlier quoted context omitted.

> I had fun doing this until I had kids. As i keep telling people, self hosting is fun as long as your user count is 1. When it grows beyond that, you suddenly have a SLA. I self hosted almost everything (e-mail is pointless from privay concerns), and when we had kids i moved to a dual Synology setup with a single proxmox server for running services. Fast forward some years and electricity suddenly costs about an arm…

I hosted email until my email to a college student was rejected with no way of contacting either him or the admins of his school. That was the straw on the camels back. I still self host apps today but my hardware is old enough that it costs more in power and cooling than what I get out of it, and the roi on new hardware doesn’t justify the means

> and the roi on new hardware doesn’t justify the means

That was my takeaway as well, considering that a 4 bay synology uses more in electricity than purchasing the same storage in the cloud (up to a certain point, datahoarders need not apply).

On top of that i then need to purchase new hardware every 3-6 years if i want reasonable assurance that my data is still there, and doing the math on a 5 year TCO, i would end up paying around double what i pay now, and still have worse data integrity.

I haven't done the math on where the breakaway point is, but i have around 10TB of cloud storage (including backups), as well as DNS services, static web hosting, mail, and a few other curiosities, and i average €25/month on cloud services.

Comparing that to a 4 bay synology with 4x6TB WD Red drives, you end up with €1276 in hardware costs (current prices here). Over a 5 year period that's €21.2/month for the hardware alone. Assuming the Synology draws 10W, and each WD Red draws an average of 5W, that's 30W of power, totalling around 22 kWh/month, which at €0.6/kWh adds up to an additional €13/month.

So in total around €35/month to self host what i can host in the cloud (including backups!) for €25/month.

Re: Self-Host All the Things?

#284

Post by vineyardmike »

stonewall wrote: post #85
novok wrote: post #80

Earlier quoted context omitted.

How much power does it take? I've realized with some services it's cheaper to use it than the electricity and hardware cost.

I almost certainly don't save any money considering electricity cost. I have a dell r630 for compute and an r730xd that I use as a NAS. Then I have one switch for the rack and a POE switch for the house. Probably 3-5amps total? If I started over, I would probably choose more efficient gear. That said, I don't mind paying for the electricity too much. I enjoy the warm fuzzies of knowing my data lives under my roof.

> Probably 3-5amps total?

A raspberry pi draws 2+ amps. Your dual Xeon server is drawing a lot more power. That said, typically you’d want to measure in watts because amps is relative. Eg a RPI is 2A at 5V while a computer is probably 5A at 120V - an order of magnitude more total energy consumed.

Re: Self-Host All the Things?

#285

Post by cb321 »

triyambakam wrote: post #266
__MatrixMan__ wrote: post #262

Earlier quoted context omitted.

I'm tickled to that you're interested. "I have this cool idea involving algorithms on strings" isn't exactly everybody's favorite conversation. Here's the old version, hardly even proves the concept: https://github.com/MatrixManAtYrService/gnize Python was giving me trouble though, so I'm switching to Nim for performance reasons, and also because I can compile it to C, objective-C, and JavaScript for use by a wider v…

> "I have this cool idea involving algorithms on strings" isn't exactly everybody's favorite conversation. Haha yeah I get you. I wouldn't have expected it would be for me either but the approach and use case you shared is really cool. Thanks for the links. It will be neat to see the development in Nim, as well.

You both might be interested in this little Nim program "framed" to frame & digest text for near-duplicate detection:

    https://github.com/c-blake/ndup/blob/main/framed.nim
which does a rolling hash with the late Bog Uzgalis' Buz Hash which is (IMO) about 10x simpler than Rabin fingerprinting. It's really just xor out old, xor in new.

In my context of near duplicate detection I worry about false negatives as well as false positives and displaying near-bys to the user & such. So, things are set up for random seeds for very independent framings & digests (to avoid unlucky ones). { This is very different from the more common "backup/rsync/transfer" context of "content-based slicing" aka "content-defined chunking". }

Re: Self-Host All the Things?

#286

Post by __MatrixMan__ »

cb321 wrote: post #285
triyambakam wrote: post #266

Earlier quoted context omitted.

> "I have this cool idea involving algorithms on strings" isn't exactly everybody's favorite conversation. Haha yeah I get you. I wouldn't have expected it would be for me either but the approach and use case you shared is really cool. Thanks for the links. It will be neat to see the development in Nim, as well.

You both might be interested in this little Nim program "framed" to frame & digest text for near-duplicate detection: https://github.com/c-blake/ndup/blob/main/framed.nim which does a rolling hash with the late Bog Uzgalis' Buz Hash which is (IMO) about 10x simpler than Rabin fingerprinting. It's really just xor out old, xor in new. In my context of near duplicate detection I worry about false negatives as well as fa…

Oh wow, this is great. We're up against similar problems here, but what I've got as unnamed musings you've got as links to relevant Wikipedia articles. I've got some reading and thinking to do.

I'll probably just include both algorithms, make it a parameter, and do a bunch of testing (which I'll be sure to publish). I can't put my finger exactly on why I feel better about Rabin fingerprints, maybe buzhash just needs time to sink in.

It's only recently that I've been thinking of this as a framing problem. The right words eluded me because I only care about the frames near the annotation. So instead of "multiple statistically independent framings" I've been thinking of separate "feature channels" (one for each prime polynomial in GF2) which cause different substrings to be identified as features, but I think it's more or less the same thought deep down.

I want a more or less uniform distribution of features so that wherever a user wants to put an annotation, there's always something nearby to anchor it to. In the event of a large gap I've imagined schemes to switch to different framings until its filled . Or maybe I show the features to the user and let them click something to "change the channel" until they get a set of features that works for their particular annotation.

Putting this as a nearly-duplicate-files problem makes me wonder if it could be used to arrange files into a graph based on how they differ. Like when browsers unhelpfully just increment a counter and redownloads the the file every time you click a link to it. It's hard to keep track of the fact that copy 6 is the one I want, but if they were rendered as a sort of constellation of files, where the edges are longer for more dissimilarity... That would be fun to play with.

Anyway, I'll be looking through this more in the future. Thanks for sharing it.

Re: Self-Host All the Things?

#287

Post by cb321 »

__MatrixMan__ wrote: post #286
cb321 wrote: post #285

Earlier quoted context omitted.

You both might be interested in this little Nim program "framed" to frame & digest text for near-duplicate detection: https://github.com/c-blake/ndup/blob/main/framed.nim which does a rolling hash with the late Bog Uzgalis' Buz Hash which is (IMO) about 10x simpler than Rabin fingerprinting. It's really just xor out old, xor in new. In my context of near duplicate detection I worry about false negatives as well as fa…

Oh wow, this is great. We're up against similar problems here, but what I've got as unnamed musings you've got as links to relevant Wikipedia articles. I've got some reading and thinking to do. I'll probably just include both algorithms, make it a parameter, and do a bunch of testing (which I'll be sure to publish). I can't put my finger exactly on why I feel better about Rabin fingerprints, maybe buzhash just needs…

You are welcome. Thanks are too rarely offered. :-)

You may also be interested in word stemming ( such as used by snowball stemmer in https://github.com/c-blake/nimsearch ) or other NLP techniques, but I don't know how internationalized/multi-lingual that stuff is, but conceptually you might want "series of stemmed words" to be the content fragments of interest.

Similarity scores have many applications. Weights on graph of cancelled downloads ranked by size might be one. :)

Of course, for your specific "truncation" problem, you might also be able to just do an edit distance against the much smaller filenames and compare data prefixes in files or use a SHA256 of a content-based first slice. ( There are edit distance algos in Nim in https://github.com/c-blake/cligen/blob/master/cligen/textUt.... as well as in https://github.com/c-blake/suggest ). Or, you could do a little program like ndup/sh/ndup to create a "mirrored file tree" of such content-based slices then you could use any true duplicate-file finder (like https://github.com/c-blake/bu/blob/main/dups.nim) on the little signature system to identify duplicates and go from path suffixes in those clusters back to the main filesystem. Of course, a single KV store within one or two files would be more efficient than thousands of tiny files. There are many possibilities.

Re: Self-Host All the Things?

#288

Post by cobertos »

fareesh wrote: post #256
cobertos wrote: post #228

Earlier quoted context omitted.

Seafile works well as a Dropbox clone. Been using it for a couple years now without much issue. My files are replicated to my main computer and I push files from my phone to the Seafile instance over WebDAV (automatically with Autosync in Android) and download specific files through the app.

Yeah I was more interested in cloud backups than syncing

For that I had to make a custom service that periodically runs rclone. Duplicati does cloud backups, but it has a record of being a bit sus

Re: Self-Host All the Things?

#289

Post by fareesh »

cobertos wrote: post #288
fareesh wrote: post #256

Earlier quoted context omitted.

Yeah I was more interested in cloud backups than syncing

For that I had to make a custom service that periodically runs rclone. Duplicati does cloud backups, but it has a record of being a bit sus

Is there a server / host that provides storage at a price cheaper than Dropbox though?

Re: Self-Host All the Things?

#290

Post by einhverfr »

sgt wrote: post #231
einhverfr wrote: post #198

At PGConf India, one of the keynotes addressed exactly this topic. The largest stock broker firm in India had made the decision to self-host everything. The CTO made a number of points that I think are missed in this discussion and article, namely: 1. You may think you are a software company, but HR, accounting etc are just as critical to your operations as the customer product. Therefore there isn't really a distinc…

Agreed fully. You might risk slightly more downtime, but the overall benefit of owning it yourself is well worth it long term.

Over time, if you come to understand the technology, you can fix things a managed service cannot, so you might actually risk less downtime if you prioritize that.

At least that's my experience based on fighting weird bugs on managed database services.

Post reply on HN