Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You don’t need to be “enterprise-ready” or “scalable”

gorelay.co

Post reply on HN
Print view View original

131–140 of 185 posts

Re: You don’t need to be “enterprise-ready” or “scalable”

#131

Post by xputer »

fasteddie31003 wrote: post #34

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

I'm sad to say this even happens at FAANG. So many engineers seem to either not want to or not know how to optimize their own workflow as they're building it. It's really frustrating when you want to contribute something to their project and the development workflow for verifying the application still starts after making any changes is "upload the 5GB deployment bundle to AWS". Like come on, you can't seriously expect people to be productive that way.

Re: You don’t need to be “enterprise-ready” or “scalable”

#132

Post by tluyben2 »

bob1029 wrote: post #35
fasteddie31003 wrote: post #34

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

> Is this common at other companies that you cannot run the whole app locally to develop and test against? Yes. I work in B2B banking software, so having an "authentic" environment to test against is extraordinarily complex. Even the big vendors in the space seem unable to maintain accurate facsimiles of production with their own decades-old core software, so smaller vendors stand no chance unless they adjust their e…

Yep, work in b2b banking as well and there is a huge issue with banking ‘test’ environments. Our partners (100s of companies, mostly banks/brokers/insurers/etc) mostly don’t even have test envs and if they do, they don’t represent production at all (sometimes the api is a completely different version etc). Our software allows to spin up a test/dev env locally or where ever but nothing much works without the partners and to dev/test those, those, you will have to dev/test against their production systems with real money.

In the beginning (more than 20 years ago), I insisted that we implement test systems ourselves so we didn’t have to test in production; that was naive; it was expensive and there is so much complexity that this does not work well.

Re: You don’t need to be “enterprise-ready” or “scalable”

#133

Post by raffraffraff »

cubes wrote: post #11

Hard agree. History is littered with dead startups that designed for scale before they had enough usage to justify it. Within reason, having users knock your site over resulting in failures like the Twitter Fail Whale is a good problem to have. With that said, you need to be prepared to scale up quickly once you have this problem. There's a reason Facebook counts its users in the billions, and Friendster is a footnot…

So how do you square they circle "Don't design to scale but scale up quickly when you need to"? You can't rewrite your stack just because you took on a new customer that is suddenly killing you.

I don't think that it's stupidly hard to get basic things set up for scaling at the start. These don't apply to everyone but:

- use a cloud provider, they let you grow

- write infrastructure as code (terraform, pulumi, cdk), it's not that hard and it makes building your 2nd, 3rd data center easier

- for the love of God think for 10 minutes about a scalable naming convention for data centers, infrastructure components, subdomains, IP ranges etc. This type of stuff causes amazing headaches if you get it wrong, and all you have to do is consider it at the start. Many AWS resources can't be renamed non-destructively, and are globally namespaced. Be aware of this. Don't paint yourself into a really stupid corner.

- even if you're not doing microservices, consider containers and an orchestrator that has scalability baked in (like Kubernetes). If you use a managed Kubernetes with managed node groups or fargate, you can basically forget about compute infrastructure from this point on

- deploy Prometheus, Grafana, Loki etc day one and build basic dashboards. With Kubernetes, you can get this installed quickly and within a day you'll at least be able to graph request/error count and parse logs.

- deploy an environment for developers to deploy to and use the product (even load test it)

The development cycle is now set up for building what you need to build. You can easily find developers at any level who have experience with containers, Kubernetes, Grafana so they can onboard quickly, be productive and troubleshoot their own stuff. They can refactor the monolith without having to invent service discovery, load balancing, ingress etc.

My personal experience with Kubernetes began at a start-up with a tiny team, none of whom had prior experience with it. We had started building a single monolithic app. But within a short time we figured out how to make a helm chart, add ingress and boom, we were "online". Every time someone was about to spend time and effort inventing something, we would quickly scan through the Kubernetes docs and CKAD material to see if we could avoid having to build it at all. We leverage cloud stuff too, where it was cheap and easy and scalable (like AWS SQS/SES/SNS/CodeArtifact/ECR/Parameter Store)

Re: You don’t need to be “enterprise-ready” or “scalable”

#134

Post by Existenceblinks »

Tldr; use frequent iteration / feedback loop as enterprise-advantage alternative to build trust. Ahh, people really do like fashion though, peer testimonial, network effect etc involving kinda semi-real value construct just like the enterprise does (just spend more money because they can afford to)

Re: You don’t need to be “enterprise-ready” or “scalable”

#135

Post by WJW »

bonestormii_ wrote: post #115
temp8964 wrote: post #82

Earlier quoted context omitted.

> they were being developed by FTPing files to production server or even editing them live with vim or something. Many editors / IDEs (e.g. Notepad++) can directly connect to the FTP server, so working on a FTP servers looks almost same as working on a local folder. You just make changes to the files, and then refresh the web browser to see the changes.

lol, yes, sure. But why don't you sound horrified? Version control at least is essential for something of any size with any number of people working on it. You must be able to "revert" a set of changes quickly and reliably if the application has any importance at all.

> You must

Don't want to be rude here, but version control is clearly not "required" in the strict sense. Plenty of software has been developed without it. Now I wouldn't want to work at a place without version control either, but it is as required for developing software as seat belts are required for driving a car. Less even, since at least for seat belts they are required by law.

Re: You don’t need to be “enterprise-ready” or “scalable”

#136

Post by kijin »

fasteddie31003 wrote: post #80
ma2rten wrote: post #77

Earlier quoted context omitted.

At Google, it is very team-specific but it is generally not possible to run servers locally for testing and there generally is no staging environment, but it might be possible to bring up some part of the stack on Google's clusters. Those severs may read production data or some test data. Other than that there are static checks, unit tests, code reviews and the release process to ensure reliability.

How do you develop features if you can't run the server locally? Do you just write code, get it approved in a code review and push it to production?

Production is not the only alternative to local. Companies can set up servers, either in the cloud or on premise, specifically for development and/or staging purposes.

This is very useful if the production environment is hard to replicate on each developer's local machine. For example, a team where some people prefer the MacBook Air and others prefer beefy Windows desktops might set up a cluster of Linux servers for shared development and testing. A common alternative is to tell everyone to run a bunch of VMs/containers locally, but this isn't always feasible depending on the size and complexity of the production cluster.

Re: You don’t need to be “enterprise-ready” or “scalable”

#137

Post by vbezhenar »

fasteddie31003 wrote: post #34

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

It is the case in my company. Production and staging is bunch of docker-composes scattered around several servers. We had to create another staging environment, I think it took several days to configure it. People run services locally, but not entire stack, that takes too much effort.

Right now I’m trying to understand and deploy Kubernetes. Scalability and availability is good, but one of the reasons that I want to make it possible to roll out entire stack locally.

Basically it’s lack of devops. We don’t have separate role and developers do the minimal job of keeping production running well enough, they got other tasks to do. There should be someone working on improving those things full time.

Re: You don’t need to be “enterprise-ready” or “scalable”

#138

Post by jamesfinlayson »

scarface74 wrote: post #119
jamesfinlayson wrote: post #114

Earlier quoted context omitted.

Unsurprising reasons for everything - security department would have been reluctant to allow development from inside AWS, and running in Docker was to ensure everyone (i.e. all developers, testers and TeamCity agents) were running on the same versions of runtimes etc (Docker was chosen as the easiest solution for this problem at the time).

Damn, I first got into AWS not just to avoid administering systems. But to also avoid system administrators. That being said, how do you get on different versions if everyone uses the same package manager config file (requirements.txt/package.json/the wonderful system that Go uses/whatever Nuget/C# uses). I’m also spoiled because I’ve never been in a position where I wasn’t the developer and leading the “DevOps” init…

It's more to make sure everyone is using Java 11.x, Java 17.x, Python 3.x, Node x etc - we're on Macs but TeamCity agents are Linux and Docker is easy way to pin the runtime on both operating systems.

Docker also fixes the "I forgot to brew install a specific version" in this case.

Re: You don’t need to be “enterprise-ready” or “scalable”

#139

Post by vbezhenar »

WJW wrote: post #135
bonestormii_ wrote: post #115

Earlier quoted context omitted.

lol, yes, sure. But why don't you sound horrified? Version control at least is essential for something of any size with any number of people working on it. You must be able to "revert" a set of changes quickly and reliably if the application has any importance at all.

> You must Don't want to be rude here, but version control is clearly not "required" in the strict sense. Plenty of software has been developed without it. Now I wouldn't want to work at a place without version control either, but it is as required for developing software as seat belts are required for driving a car. Less even, since at least for seat belts they are required by law.

People manually control versions all the time. Ctrl + C, Ctrl + V on a project folder to make a version. I have doubts that someone developed any nontrivial software without that approach.

Re: You don’t need to be “enterprise-ready” or “scalable”

#140

Post by raffraffraff »

fasteddie31003 wrote: post #34

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

I'm at a company that works like this, and it's awful. There are also anti-patterns around git branching and container deployment. The staging environment is broken every other day and production deployment requires the full time effort of a very senior engineer who cherry-picks through git commits from one branch to the production branch (there are thousands of differences between these branches). I'm honestly done with them. They don't even know what sensible dev/test/deploy looks like.
Post reply on HN