Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I am building a cloud

crawshaw.io

Post reply on HN
Print view View original

191–200 of 589 posts

Re: I am building a cloud

#191

Post by codethief »

LoganDark wrote: post #63
farfatched wrote: post #19

Nice post. exe.dev is a cool service that I enjoyed. I agree there is opportunity in making LLM development flows smooth, paired with the flexibility of root-on-a-Linux-machine. > Time and again I have said “this is the one” only to be betrayed by some half-assed, half-implemented, or half-thought-through abstraction. No thank you. The irony is that this is my experience of Tailscale. Finally, networking made easy. O…

I find it difficult to configure Tailscale for my use case because they seem to completely not support making ACL rules based on the identity of the device rather than a part of the address space. I'm not configuring a router here, I'm configuring a peer-to-peer networking layer... or at least I'm supposed to be...

> because they seem to completely not support making ACL rules based on the identity of the device rather than a part of the address space

Could you rephrase that / elaborate on that? Isn't Tailscale's selling point precisely that they do identity-based networking?

EDIT: Never mind, now I see the sibling comment to which you also responded – I should have reloaded the page. Let's continue there!

Re: I am building a cloud

#192

Post by mattbee »

Tepix wrote: post #16
Manfred wrote: post #11

Earlier quoted context omitted.

Companies buy cloud services because they want to reduce in-house server management and operations, for them it's a trade-off with hiring the right people. But you are right, when you can find the right people doing it yourself can be a lot cheaper.

I get the feeling that with LLMs in the mix, in-house server management can do a lot more than it used to.

The internet of 20 years ago was awash with info for running dedicated servers, fragmented and badly-written in places but it was all there. I can absolutely believe LLMs would enable more people to find that knowledge more easily.

Re: I am building a cloud

#193

Post by abdjdoeke »

dajonker wrote: post #74

> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…

I dunno the more people dig into this approach they will probably end up just reinventing Kubernetes.

I use k3s/Rancher with Ansible and use dedicated VMs on various providers. Using Flannel with wireguard connects them all together.

This I think is reasonable solution as the main problem with cloud providers is they are just price gouging.

Re: I am building a cloud

#194

Post by Nnnes »

Medowar wrote: post #151
zuhsetaqi wrote: post #135

Earlier quoted context omitted.

Just for comparison I use the cheapest netcup root server: RS 1000 G12 AMD EPYC™ 9645 8 GB DDR5 RAM (ECC) 4 dedicated cores 256 GB NVMe Costs 12,79 € Results with the follwing command: fio --name=randreadwrite \ --filename=testfile \ --size=5G \ --bs=4k \ --rw=randrw \ --rwmixread=70 \ --iodepth=32 \ --ioengine=libaio \ --direct=1 \ --numjobs=4 \ --runtime=60 \ --time_based \ --group_reporting IOPS Read: 70.1k IOPS W…

That is a bit of a unfair comparison. The Hetzner and DO instances are shared hosting, you are using dedicated ressources. Using a Netcup VPS 1000 G12 is more comparable. read: IOPS=18.7k, BW=73.1MiB/s write: IOPS=8053, BW=31.5MiB/s Latency Read avg: 5.39 ms, P99.9: 85.4 ms, max 482.6 ms Write avg: 3.36 ms, P99.9: 86.5 ms, max 488.7 ms

Hetzner has dedicated resources too, but they also have 2 levels of shared resources, "Cost-Optimized" and "Regular Performance". The 3900 IOPS CX23 above is "Cost-Optimized".

Here are some "Regular Performance" shared resource stats

Hetzner CPX11 (Ashburn, 2 CPUs, 2GB, 5.49€ or $6.99/month before VAT)

read: IOPS=36.7k, BW=144MiB/s, avg/p99.9/max 2.4/6.1/19.5ms

write: IOPS=15.8k, BW=61.7MiB/s, avg/p99.9/max 2.4/6.1/18.7ms

Hetzner CPX22 (Helsinki, 2 CPUs, 4GB, 7.99€ or $9.49/month before VAT)

read: IOPS=48.2k, BW=188MiB/s, avg/p99.9/max 1.9/5.7/10.8ms

write: IOPS=20.7k, BW=80.8MiB/s, avg/p99.9/max 1.8/5.8/10.9ms

Hetzner CPX32 (Helsinki, 4 CPUs, 8GB, 13.99€ or $16.49/month before VAT)

read: IOPS=48.3k, BW=189MiB/s, avg/p99.9/max 1.9/6.2/36.1ms

write: IOPS=20.7k, BW=81.0MiB/s, avg/p99.9/max 1.8/6.3/36.1ms

Re: I am building a cloud

#195

Post by codethief »

sahil-shubham wrote: post #131

The point about VMs being the wrong shape because they’re tied to CPU/memory resonates hard. The abstraction forces you to pay for time, not work. I ended up buying a cheap auctioned Hetzner server and using my self-hostable Firecracker orchestrator on top of it ( https://github.com/sahil-shubham/bhatti , https://bhatti.sh ) specifically because I wanted the thing he’s describing — buy some hardware, carve it into as…

> My agents work inside sandboxes

Out of interest, what sandboxing solution do you use?

Re: I am building a cloud

#197

Post by regularfry »

hkt wrote: post #157
goombaskoop wrote: post #133

Earlier quoted context omitted.

> the average devops knows k8s and will slap it onto anything they see This is certainly the case from all the third person accounts I hear. Online. I never actually met a single one that is like that, if anything, those same people are the ones that are first to tell me about their Hetzner setups.

DevOps here. The trouble is that we are literally expected to do this everywhere we go. I've personally advocated for approaches which use say, a pair of dedicated servers, or VMs as in GPs example. If you want it outside of AWS/GCP/Azure, you're regarded as a crazy person. If you don't adopt "best practices" (as defined by vendors) then management are scared. Management very often trust the sales and marketing depar…

There is a kernel of validity lurking in the heart of all this, which is that immutable images you have the ability to throw away and refresh regularly are genuinely better than long-running VMs with an OS you've got to maintain, with the scope for vulnerabilities unrelated to the app you actually want to run. Management has absorbed this one good thing and slapped layer after layer of pointless rubbish on it, like a sort of inverse pearl. Being able to say "we've minimised our attack surface with a scratch image" (or alpine, or something from one of the secure image vendors) is a genuinely valuable thing. It's just the all of the everything that goes along with it...

Re: I am building a cloud

#198

Post by sgt »

psviderski wrote: post #175
sgt wrote: post #167

Earlier quoted context omitted.

That looks pretty interesting. Is it being used in production yet (I mean serious installs) ?

Yes but at small scale. Myself and a handful of others from our Discord run it in production. The core build/push/deploy workflows are stable and most of the heavy lifting at runtime is done by battle-tested projects: Docker, Caddy, WireGuard, Corrosion from Fly.io. Radboud University recently announced they're rolling it out for managing containers across the faculty which is the most "serious install" I know about,…

Did you improve the security concerns? E.g the way it executes in a `curl | bash` level. I was a bit concerned about that.

Re: I am building a cloud

#199

Post by tete »

dajonker wrote: post #93
adamtulinius wrote: post #89

Earlier quoted context omitted.

If you spin up Kubernetes for "a couple of containers to run your web app", I think you're doing something wrong in the first place, also coupled with your comment about adding SDN to Kubernetes. People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes.

I totally agree, but that's not what happens in reality: the average devops knows k8s and will slap it onto anything they see (if only so they can put in on their resume). The average manager hears about k8s, gets convinced they need and hires beforementioned devops to build it.

> the average devops knows k8s

If you'd know Kubernetes, you know not to use it. I say that as someone who used to do consulting for it.

The reality is that yet again "making money" completely collides with efficient, quality, sane productive work.

For me one of the main reasons to leave that space is that I couldn't really deal with the fact that my work collides with a client's success. That said I have helped to get off that stuff and other things that they thought they needed, that just wasted time and money. It just feels odd going into a company that hired you to consult on a topic only to end up telling them "The best approach for you is not doing that at all". Often never. Like some people thought "Well, if we have hundreds of thousands or even millions of users" and the reality was that even in these scenarios if you went away from that abstract thought and discussed a hypothetical based on their product they realized that they'd still be better off without it. Besides the fact that this hypothetical often was in a future that made it likely that they said they'd likely have completely different setup so preparing for that didn't even make sense.

I think a big thing related to that was/is the microservice craze where people end up moving to a complex architecture for not many good reasons and then they increase complexity way faster than what they actually deliver in terms of the product, because it somehow feels good. I know it does, I've been there. When in reality the outcome often is just a complex mess with what could have been a relatively simple monolith. And these monoliths do work. And in the vast majority of cases they are easy to scale, because your problem switches from "how do we best allocate that huge amount of very different services across our infrastructure" to (for the most part) "how do we spin up our monolith on one more server" which tends to be a way easier to tackle service.

And nothing stops you from still using everything else if you want. Just because it's a monolith doesn't mean you need to skip on any of the cloud offerings, etc. For some reason there seems to be that idea that if you write a monolith you are somehow barred from using modern tooling, infrastructure, services, etc. Not sure where that comes from.

Re: I am building a cloud

#200

Post by sahil-shubham »

codethief wrote: post #195
sahil-shubham wrote: post #131

The point about VMs being the wrong shape because they’re tied to CPU/memory resonates hard. The abstraction forces you to pay for time, not work. I ended up buying a cheap auctioned Hetzner server and using my self-hostable Firecracker orchestrator on top of it ( https://github.com/sahil-shubham/bhatti , https://bhatti.sh ) specifically because I wanted the thing he’s describing — buy some hardware, carve it into as…

> My agents work inside sandboxes Out of interest, what sandboxing solution do you use?

Not sure what you mean. I use the above linked personal project, bhatti, which internally uses Firecracker microVMs.
Post reply on HN