Great read! Especially the part on ingress class which I didn't know about. Would you mind sharing some of your grafana dashboards?
Painless NGINX Ingress
11–20 of 51 posts
Re: Painless NGINX Ingress
#12Why does everyone use reverse proxies? It seems complex and inefficient. Why not serve xhr's and other dynamic content from the app server(s) and static content from a static webserver?
What would you use to provide a single endpoint to multiple instances of an app server?
What about just exposing the multiple instances of app server, and have the frontend code select one for load balancing or failover purpouses? There could be a load balancing config read by the client, or you can have static rules in the frontend js, like choosing shard number based on a hash from the client ip address.
Round-robin DNS might also work or complement this.
Re: Painless NGINX Ingress
#13I think this blog post is one turn of the crank away from a truth we're all about to learn: don't hand roll your own Kubernetes ingress.
Dealing with the traffic handling between your users and your code is not a trivial problem. Like all good ops problems, you can fix it with good tools, deep knowledge of those tools, fine-grained observability, and smart people running all that.
This has been the recipe for a couple of really successful SaaS offerings. Individual servers? Datadog. CDN? Akamai / Fastly.
Disclaimer: I work at one of those companies, Turbine Labs, and we're trying to make ingress better. Here's a presentation from our CEO on Kubernetes ingress, and why the specification creates the problems that this blog post is trying to fix. https://www.slideshare.net/mobile/MarkMcBride11/beyond-ingre...
Re: Painless NGINX Ingress
#14This is a great read. I know the single cluster for all env is something that is sort of popular but it's always made me uncomfortable for the reasons stated in the article but also for handling kube upgrades. I'd like to give upgrades a swing on a staging server ahead of time rather than go straight to prod or building out a cluster to test an upgrade on. I tend to keep my staging and prod clusters identical , even…
+100 to this. Why would any sane Op/Inf/SRE choose not to have at least account-level isolation - is it only a matter of cost due to under-utilization?
I prefer to have everything 100% isolated for dev / qa / stage / prod, and have process and tooling in place to explicitly cross the streams. This comes from a history of pain with random dev-to-prod (or worse, prod-to-dev) access and dealing with "real companies" with things like audit requirements.
Having them separate lets you do things like @odammit suggests, upgrade your cluster in staging without affecting your developers or customers.
If you don't want to go that far, you can set up separate AWS accounts that are all tied together via an organization, and you can set up IAM roles and whatnot to share your API keys between accounts. That gives you at least some isolation, but still lets you GSD the same way as if you have a single account.
Re: Painless NGINX Ingress
#15ingress+overlay network confusion was the reason why we moved from k8s to Docker Swarmkit.
I still keep hoping for kubernetes kompose (https://github.com/kubernetes/kompose) to bring the simplicity of Docker Swarmkit to k8s.
Or will Docker Infrakit bring creeping sophistication first and eat kuberentes lunch ? (https://github.com/docker/infrakit/pull/601)
Re: Painless NGINX Ingress
#16This is a great read. I know the single cluster for all env is something that is sort of popular but it's always made me uncomfortable for the reasons stated in the article but also for handling kube upgrades. I'd like to give upgrades a swing on a staging server ahead of time rather than go straight to prod or building out a cluster to test an upgrade on. I tend to keep my staging and prod clusters identical , even…
> This is a great read. I know the single cluster for all env is something that is sort of popular but it's always made me uncomfortable for the reasons stated in the article but also for handling kube upgrades. I'd like to give upgrades a swing on a staging server ahead of time rather than go straight to prod or building out a cluster to test an upgrade on.
I've been doing patch-level upgrades in-place since the beginning, and never had a problem. For more sensitive upgrades, this is what I do: create a new cluster using based on the current state in order to test the upgrade in a safe environment before applying it to production.
And for even more risky upgrades, I go blue/green-like by creating a new cluster with the same stuff running in it, and gradually shifting traffic to the new cluster.
Re: Painless NGINX Ingress
#17This is a great read. I know the single cluster for all env is something that is sort of popular but it's always made me uncomfortable for the reasons stated in the article but also for handling kube upgrades. I'd like to give upgrades a swing on a staging server ahead of time rather than go straight to prod or building out a cluster to test an upgrade on. I tend to keep my staging and prod clusters identical , even…
+100 to this. Why would any sane Op/Inf/SRE choose not to have at least account-level isolation - is it only a matter of cost due to under-utilization? I prefer to have everything 100% isolated for dev / qa / stage / prod, and have process and tooling in place to explicitly cross the streams. This comes from a history of pain with random dev-to-prod (or worse, prod-to-dev) access and dealing with "real companies" wit…
> Why would any sane Op/Inf/SRE choose not to have at least account-level isolation - is it only a matter of cost due to under-utilization?
In our particular case, yes, pretty much. We are a small company with a small development team, so even if I would want to split accounts to different teams, we would end up having one account for 2-3 users, which doesn't make a lot of sense now.
Re: Painless NGINX Ingress
#18Why does everyone use reverse proxies? It seems complex and inefficient. Why not serve xhr's and other dynamic content from the app server(s) and static content from a static webserver?
Off the top of my head: load balancing, hiding details of app servers, compressing responses and multivariate testing. All of which could be done at the app server level sure, but then that would shift that complexity to your app and your developers. Oh and job security, obviously.
You could do all of those, except hiding app servers, with the client based technique I outlined in the nearby other comment. It would just be a tweak to the rule that the frontend uses to choose the the app server.
Re: Painless NGINX Ingress
#19Great read!
>> "Let me start by saying that if you are not alerting on accept queue overflows, well, you should."
Does anyone know how to effectively keep a tab on this on a docker container running nginx open source? I have an external log/metrics monitoring server that could alert on this, but I'm asking more on the lines of how to get this information to the monitoring server.
Re: Painless NGINX Ingress
#20[deleted]