Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New setup for 2020

changelog.com

Post reply on HN
Print view View original

31–40 of 58 posts

Re: New setup for 2020

#31

Post by tgsovlerkhgsel »

lifeisstillgood wrote: post #16

>>> We no longer provision load balancers, or configure DNS; we simply describe the resources that we need, and Kubernetes makes it happen. This is (part) of what keeps me in the stone age. You are provisioning load balancers and DNS - but just one step removed through k8s And my prior is that we need to understand that, be aaare of it, have a model of what is going on to help develop and debug. And so it feels a bit…

By using K8s and similar technologies, you're buying standardization with underlying complexity and reduced efficiency.

In many cases, it's a good tradeoff, because you can now use standard tooling on everything.

Just like it's cheaper to ship an entire (physical) shipping container that's half-full than to ship the same stuff loosely. Or why companies will send you two separate letters on the same day with a small note that this is more efficient for them than collating them.

I assume that k8s also makes it much easier to move to a different cloud provider if you're unhappy with one (or the new one offers better pricing). Instead of rewriting your bespoke scripts that only you understand, anyone familiar with the technology will know which modules to swap to make it work with the new provider.

Re: New setup for 2020

#32

Post by tgsovlerkhgsel »

PietKachelhout wrote: post #30
brobinson wrote: post #28

Earlier quoted context omitted.

With all these negatives, there must be a really compelling reason to stay. What is it?

The owner of the company thinks they are great because you can call them when there is a problem. I'm having a hard time convincing him that these are issues I've never had at other providers. Certainly not so many. We are moving everything away. Most of our servers are with another provider already. And we haven't had any similar issues there. I've never called them! And I forgot to mention the connectivity issues a…

This is something more companies need to realize: Yes, you should be easily and quickly reachable by phone so that when (not if) things go wrong, I can get good service and resolve it quickly.

But if I'm calling you, you have most likely already failed. If I'm calling for information, your documentation has failed to make that information accessible (it either wasn't documented, or not easy enough to find). If I'm calling to resolve an issue (technical or billing), it would have been a lot better if it didn't happen in the first place.

Re: New setup for 2020

#33

Post by enos_feedler »

lifeisstillgood wrote: post #16

>>> We no longer provision load balancers, or configure DNS; we simply describe the resources that we need, and Kubernetes makes it happen. This is (part) of what keeps me in the stone age. You are provisioning load balancers and DNS - but just one step removed through k8s And my prior is that we need to understand that, be aaare of it, have a model of what is going on to help develop and debug. And so it feels a bit…

Isn’t this true of all operating systems? Debugging is going to require some knowledge about the OS and the syscalls it makes but you don’t want to write directly to the machine as an OS would. Same goes for internet connected clustered machines

Re: New setup for 2020

#34

Post by brainzap »

lifeisstillgood wrote: post #16

>>> We no longer provision load balancers, or configure DNS; we simply describe the resources that we need, and Kubernetes makes it happen. This is (part) of what keeps me in the stone age. You are provisioning load balancers and DNS - but just one step removed through k8s And my prior is that we need to understand that, be aaare of it, have a model of what is going on to help develop and debug. And so it feels a bit…

I have switched from dev to cloud engineer, and from what I have experienced all abstractions leak and you need to master all layers.

Re: New setup for 2020

#35

Post by gerhardlazu »

PietKachelhout wrote: post #30
brobinson wrote: post #28

Earlier quoted context omitted.

With all these negatives, there must be a really compelling reason to stay. What is it?

The owner of the company thinks they are great because you can call them when there is a problem. I'm having a hard time convincing him that these are issues I've never had at other providers. Certainly not so many. We are moving everything away. Most of our servers are with another provider already. And we haven't had any similar issues there. I've never called them! And I forgot to mention the connectivity issues a…

I use Digital Ocean, Scaleway, Linode, Google, Cloudflare & Amazon on a daily basis, and I have experienced networking issues on all providers this year. It's all public, some even wrote lengthy post-mortems, most have been posted on HN.

When failures happen, it's always a series of unfortunate incidents. When we've hit issues with Linode, we reached out and worked on what we can improve in our changelog.com setup, and discussed about the improvements that we can expect on the Linode end. Our common interest is a more resilient system, which requires a healthy collaboration, and Linode has been a great technology partner for us. Expect to see these write-ups on changelog.com as soon as these improvements have shipped, and we have hard data to support the claims ; )

I'm sorry to hear that things have not been as smooth for you on Linode. I hope that you will find an infra provider that you will be able to rely on and work with as we do. Not all collaborations will work out, and that's OK. It's also OK to be annoyed, fed up with the way things are and look for something different, something more suitable for you. My only ask is that you share your migration story with the changelog.com community. That is something that I would want to hear about.

Re: New setup for 2020

#36

Post by gerhardlazu »

manigandham wrote: post #19

The static files should definitely be on some kind of object storage like S3, that's what it's built for. Much faster, more reliable, more scalable, and likely much cheaper too. As for persistent volumes, might be better to just offload Postgres to a managed DB service and downsize the K8S instances, or use something like CockroachDB which is natively distributed and can make use of local volumes instead.

Yes, it does make sense to move static files to object storage, especially the mp3s. There is some ffmpeg-related refactoring that we need to do before we can do this though, and it's not a quick & easy task, so we have been deferring it since it's not that high priority, and there are simpler solutions to this particular problem (i.e. improved CDN caching).

Other static files such as css, js, txt make sense to remain bundled with the app image, which is stateless and a prime candidate for horizontal scaling. Also, CDN caching makes small static files that change infrequently a non issue, regardless of their origin.

The managed Postgres service from Linode's 2021 roadmap is definitely something that we are looking forward to, but the simplest thing might be to provision Postgres with local volumes instead. We are already using a replicated Postgres via the Crunchy PostgreSQL Operator, so I'm looking forward to trying this approach out first.

CockroachDB is on my list of cool tech to experiment with, but that will use an innovation token, and we only have a few left for 2021, so I don't want to spend them all at once.

Re: New setup for 2020

#37

Post by gerhardlazu »

jacques_chester wrote: post #12

I'd be interested in learning more about the move from Concourse to Circle (I'm a notorious Concourse fanboy). What went well, what didn't, what you miss, what prompted it -- that sort of thing.

The primary reason behind the move was not wanting to manage CI. Since there were no options for a managed Concourse in 2018, we migrated to Circle, one of the Changelog sponsors at the time.

Concourse worked well for us, we didn't have any issues that were being enough to remember. You may be interested in this screenshot that captured the changelog.com pipeline from 2017: https://pipeline.gerhard.io/images/small-oss.png

I missed the simple Concourse pipeline view at first, but CircleCI improved by leaps and bounds in 2020, and the new Circle pipeline view equivalent is even better (compared to Concourse, clicking on jobs always works): https://app.circleci.com/pipelines/github/thechangelog/chang...

The Circle feature which I didn't expect to like as much as I do today, is the dashboard view (list of all pipeline/workflow runs). This is something that Concourse is still missing: https://app.circleci.com/pipelines/github/thechangelog

My favourite Circle 2020 feature is the Insights: https://app.circleci.com/insights/github/thechangelog/change.... Yup, we were one of the first ones to ask for it in 2019.

In 2021, I expect us to spend one migration credit on GitHub Actions, as a Circle replacement. Argo comes as a second close, but that requires an innovation credit which is more precious to us. Because we are already using GitHub Actions for some automation, it would make sense to consolidate, and also leverage the GitHub Container Registry, as a migration from Docker Hub. Watch https://github.com/thechangelog/changelog.com to see what happens : )

Re: New setup for 2020

#38

Post by gerhardlazu »

lifeisstillgood wrote: post #16

>>> We no longer provision load balancers, or configure DNS; we simply describe the resources that we need, and Kubernetes makes it happen. This is (part) of what keeps me in the stone age. You are provisioning load balancers and DNS - but just one step removed through k8s And my prior is that we need to understand that, be aaare of it, have a model of what is going on to help develop and debug. And so it feels a bit…

K8S is an API that the majority is agreeing on, which is rare. There is a lot of amazing tooling, a staggering amount of ongoing innovation, all built on solid concepts: declarative models, emitted metrics (the /proc equivalent, but with larger scope) and versioned infrastructure as data (a.k.a. GitOps).

For someone that is known as the King of Bash (self-proclaimed) - https://speakerdeck.com/gerhardlazu/how-to-write-good-bash-c... - and after a decade of Puppet, Chef, Ansible and oh wow that sweet bash https://github.com/gerhard/deliver - even if all my workstations and work servers (yup, all running k3s) are provisioned with Make (bash++), I still think that K8S is the better approach to running production infrastructure. The advantage to using simple and well-defined components (e.g. external-dns, ingress-nginx, prometheus-operator etc.) that adhere to a universal API, and are maintained by many smart people all around the world, is a better proposition than scripting in my opinion.

At the end of the day, I'm in it for the shared mindset, great conversations and a genuine desire to do better, which I have not seen before K8S & the wider CNCF. I will go on a limb here and assume that I love scripting just as much as you do, but go beyond this aspect and you will discover that it's more to it than "thin install scripts that deploy containers" (which are not just glorified jails or unikernels).

Re: New setup for 2020

#39

Post by mjgs »

lifeisstillgood wrote: post #16

>>> We no longer provision load balancers, or configure DNS; we simply describe the resources that we need, and Kubernetes makes it happen. This is (part) of what keeps me in the stone age. You are provisioning load balancers and DNS - but just one step removed through k8s And my prior is that we need to understand that, be aaare of it, have a model of what is going on to help develop and debug. And so it feels a bit…

I’m seeing things as you are seeing them currently.

I love the idea of using Kubernetes, it sounds amazing initially, but then every single article I read about it turns into some epic blog post that leaves me worried that the whole house of cards could easily come crashing down.

Maybe in the future the abstraction will become rock solid, easy to install and manage and ‘just work’, but it doesn’t feel like that to me now. There’s too many ‘we had to come up with / use hack X to integrate it with software Y’.

Until then, if you are on a small team with a small budget, I reckon keeping it simple is the better approach. Standard OSs with some bash scripts for provisioning, build and deploy. Even if it’s more manual work, and takes a bit longer, having an understanding of the platform you are building on is crucial.

BTW if you are looking for a description of a way todo this sort of thing ‘the boring way’:

Robust NodeJS Deployment Architecture

https://blog.markjgsmith.com/2020/11/13/robust-nodejs-deploy...

Re: New setup for 2020

#40

Post by gerhardlazu »

jgalt212 wrote: post #27
jlgaddis wrote: post #24

Earlier quoted context omitted.

Not knowing what changelog.com is, I was wondering the same thing... from the header of the page, I could tell that there's a blog, a podcast, and a newsletter. Without some additional insight that I don't have, it does seem that this is an enormously over-engineered "solution" for a website -- I've NFI why it requires 99.99% uptime! Perhaps they look it as a goal or challenge, an opportunity to showcase their knowle…

It's kind a brand marketing paradox. 1. We're great engineers because we can set and maintain such an impressive set-up. 2. We're terrible engineers because all of this could probably done one server for dynamic content + S3 for dynamic content. Or not even S3, maybe just some Cloudflare or Akamai caching. Of course, like the posters above, I could be missing something due to my outsider/consumer view of changelog.

We are both! I would also add lazy to that paradox. My surname is a letter off, and that's at close as it gets : )

The devil is in the details, there is more to it than dynamic & static content, we are using Fastly, otherwise we couldn't serve all the traffic that we do.

The best part is that it's all public - https://github.com/thechangelog/changelog.com - and we welcome contributions, especially those that simplify our setup without compromising on resiliency and availability. I'm looking forward to yours ; )

Post reply on HN