261–270 of 376 posts
Earlier quoted context omitted.
According to geekbench, an i9 4790 processor released a decade ago is ~5 times slower than i7 14700. 4790's go for $30 at ebay, vs $300 for 14700, so price/performance seems to be in favor of older hardware:)
What about power consumption? When running a server 24/7, power is likely to be a bigger cost concern than the one-off cost of purchasing the processor.
For a desktop, yeah, you’re generally better off buying newer from a performance/$ standpoint. For servers, the calculus can shift a bit depending on your company’s size and workloads. Most smaller companies (small is relative, but let’s go with “monthly cloud bill is I have three Dell R620s, which are over a decade old. They have distributed storage via Ceph on NVMe over Mellanox ConnectX3-PRO. I’ve run DB benchmarks (with realistic schema and queries, not synthetic), and they nearly always outclass similarly-sized RDS and Aurora instances, despite the latter having multiple generations of hardware advancements. Local NVMe over Infiniband means near-zero latency.
Similarly, between the three of them, I have 384 GiB of RAM, and 36C/72T. Both of those could go significantly higher.
Those three, plus various networking gear, plus two Supermicro servers stuffed with spinning disks pulls down around 700W on average under mild load. Even if I loaded the compute up, I sincerely doubt I’d hit 1 kW. Even then, it doesn’t really matter for a business, because you’re going to colo them, and you’re generally granted a flat power budget per U.
The downside of course is that you need someone[s] on staff that knows how to provision and maintain servers, but it’s honestly not that hard to learn.
[0]: https://www.guru3d.com/review/core-i7-4790k-processor-review...
[1]: https://www.tomshardware.com/news/intel-core-i9-14900k-cpu-r...
Earlier quoted context omitted.
To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to h…
This is exactly what we do, but with Spark instead. We develop the functions locally in a package and call necessary functions for the job notebooks, and the job notebooks are very minimalistic because of this
Earlier quoted context omitted.
What happens when the single machine fails?
Machine failures are few and far between these days. Over the last four years I've had a cluster of perhaps 10 machines. Not a single hardware failure. Loads of software issues, of course. I know this is just an anecdote, but I'm pretty certain reliability has increased by one or two orders of magnitude since the 90s.
Earlier quoted context omitted.
That‘s nice, but the question is if you (i.e. your company) needs this ecosystem.
Need no. But it’s nice especially since I have the know how. Creating a new deployment is just super fast. Developers can deploy their own apps etc. And then there is Helm. If we only decide by “need” then most of the time we also wouldn’t need object oriented programming.
Don’t get me wrong, I like K8s and run it at home, and I’d take it any day over ECS or the like at work, but it’s not like you can’t achieve a very similar outcome without it.
Earlier quoted context omitted.
Genuine question: say you have 3-4 services and a bunch of databases that make up your product, what's the alternative to plemping them all into K8s according to you?
For personal stuff I simply run systemd services and that does scale quite a lot (as in, you can rely on it for more production services) I believe.
Earlier quoted context omitted.
Genuine question: say you have 3-4 services and a bunch of databases that make up your product, what's the alternative to plemping them all into K8s according to you?
If you use AWS, it's probably easier to use ECS that takes away some of the complexity from you.
Earlier quoted context omitted.
> because people realized "You're not Google. Your company will never be Google." Is that also why almost no one is using microservices and Kubernetes?
Maybe add /s ;). It may decline number of hot headed responses
Earlier quoted context omitted.
That‘s nice, but the question is if you (i.e. your company) needs this ecosystem.
Genuine question: say you have 3-4 services and a bunch of databases that make up your product, what's the alternative to plemping them all into K8s according to you?
Don't get me wrong, maybe they're good choices, but absent any other facts I'd start asking questions about what makes each service necessary.
Earlier quoted context omitted.
To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to h…
Ok, so question (because I really like the DAG approach in principle but don't have enough experience to have had my fingers burned yet): The way you use Airflow, what advantage does it have over crontab? Or to put it another way, once you remove the pipeline logic, what's left?
With cron you have to be more thoughtful about failover especially when convincing others to write failure safe cron in invoked code. With airflow you shouldn’t be running code locally so you can have a mini framework for failure handling.
Cron doesn’t natively provide singleton locking so if the system bogs down you can end up running N of the same jobs at the same time which slows things down further. Airflow isn’t immune to this by default but it’s easier to setup centralized libraries that everything uses so more junior people avoid this when writing quick one off jobs.