Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You don’t need to be “enterprise-ready” or “scalable”

gorelay.co

Post reply on HN
Print view View original

141–150 of 185 posts

Re: You don’t need to be “enterprise-ready” or “scalable”

#141

Post by iovrthoughtthis »

raffraffraff wrote: post #133
cubes wrote: post #11

Hard agree. History is littered with dead startups that designed for scale before they had enough usage to justify it. Within reason, having users knock your site over resulting in failures like the Twitter Fail Whale is a good problem to have. With that said, you need to be prepared to scale up quickly once you have this problem. There's a reason Facebook counts its users in the billions, and Friendster is a footnot…

So how do you square they circle "Don't design to scale but scale up quickly when you need to"? You can't rewrite your stack just because you took on a new customer that is suddenly killing you. I don't think that it's stupidly hard to get basic things set up for scaling at the start. These don't apply to everyone but: - use a cloud provider, they let you grow - write infrastructure as code (terraform, pulumi, cdk),…

i didn't see the words "it depends" anywhere in this

thats the whole point of engineering, picking/building the right solutions for the problems and context

some stuff just needs a vps, a php file and an sqlite db and it will scale

other stuff might need container orchestration or anywhere in-between (or beyond). it should depend on the context though and we should be prepared to change strategy as needed

Re: You don’t need to be “enterprise-ready” or “scalable”

#142

Post by chupasaurus »

nerdponx wrote: post #83
halfmatthalfcat wrote: post #53

Earlier quoted context omitted.

I haven't found a wholistic integration testing framework where you can effortlessly mock out upstreams as running services , so that you don't have to ham-fist integration testing code into your app or rely on actual, live upstreams which are probably prone to breakage anyway.

I've done things like this in ad-hoc setups with Docker Compose and environment variables, managed with shell scripts. I would be very interested in a more thoughtful alternative that was integrated with testing.

I've done a thing automating whole environment setups with VMware cloning (undocumented feature because there is it's own product on top of it) and Docker via CI, in a bank. Basically we did golden images of VMs with databases having all accounts for test cases and prepared images for services to be deployed, while getting all necessary variables from CI and VSphere to write down in configs and Consul/Vault via scripts. Initial version control for the env could be done via same prefixes in the Git branches of services, after the deployment each env had a way to deploy whatever via CI's Web UI or API.

Running automated integration tests on the whole env is just an added step in the pipeline.

Re: You don’t need to be “enterprise-ready” or “scalable”

#143

Post by chupasaurus »

jamesfinlayson wrote: post #108
scarface74 wrote: post #38

Earlier quoted context omitted.

Why is it a big deal to run your app locally? You just give each developer access to a cloud account (either one account or one per developer) with wide enough guard rails. Heck, when I am in a place with bad internet, I spin up a Cloud 9 (Linux) or Amazon Workspace account (Windows) and do everything remotely. I’m not wasting my time trying to use LocalStack or SAM local.

It would be quicker for me to run things locally - in my team we all have access to an AWS account each but to test my code remotely I need to compile code, build my container, push my container to ECR then deploy the container to ECS - that last step takes minutes.

> deploy the container to ECS - that last step takes minutes

Either you wind up an instance for each container, it takes too long to start the app with less resources or the image size is in dozens of GB, everything could be avoided.

Re: You don’t need to be “enterprise-ready” or “scalable”

#144

Post by exdsq »

avg_dev wrote: post #122
exdsq wrote: post #97

Earlier quoted context omitted.

Yea, it’s common that one doesn’t exist. I work as a test engineer and am usually the one to build and support those environments, I’ve yet to find a company that has it all working well before I join. I actually made a killing building integration testing environments of blockchains and sticking them in CI too.

Can you elaborate on the blockchain testing environments/CIs thing please?

Sure! So when testing you want a deterministic environment - starting from genesis, couple accounts, the ability to build or wipe state, etc… sometimes projects use multiple chains, so maybe Ethereum and Polkadot along with a bridge between them. Building this out in Docker, managing it in CI, and building a test framework to allow devs to easily complete actions like sending a tx or waiting for an event to occur in a block has been my bread and butter for three years (and taken me from a median salary to a top 1% rate). I’ve recently moved to another industry but it was good fun.

Re: You don’t need to be “enterprise-ready” or “scalable”

#145

Post by grinich »

dt3ft wrote: post #13
bob1029 wrote: post #9

Earlier quoted context omitted.

I am on calls all week with our banking clients about getting SAML/SSO squared away once and for all. We are having a lot more nervous takes around legacy LDAP/AD auth mechanisms these days.

I developed SAML/SSO integration for an application used by the Swiss UBS which initially relied on individual accounts, not even AD. Tricky stuff.

Small plug for http://WorkOS.com (where I work).

It’s an API for easily adding SAML, SCIM, and more. Currently being used in production by Vercel, Planetscale, Webflow, and +200 other apps.

Re: You don’t need to be “enterprise-ready” or “scalable”

#146

Post by kcartlidge »

matt_heimer wrote: post #111
fasteddie31003 wrote: post #34

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

A lot of it depends on the complexity and the technology stack. Microservices based architecture with massive amounts of microservices and other service dependencies, its a pain to do more than local unit testing and mocks. Are you using FaaS or a lot of other cloud features? Also difficult to fully test locally. Extract as much logic out of the cloud dependant code. Often there will be sub-sections of large apps tha…

> Microservices based architecture with massive amounts of microservices and other service dependencies, its a pain to do more than local unit testing and mocks.

For microservices if you use something like Consul for service discovery within those microservices then you can hit them with, eg, Postman and change your local Consul registrations so everything goes to a main environment except the handful of services you're working on - which you redirect to your local machine for debugging.

Re: You don’t need to be “enterprise-ready” or “scalable”

#147

Post by FpUser »

winrid wrote: post #127
FpUser wrote: post #102

Earlier quoted context omitted.

Postgresql. No framework (well there is rather low level one of my own but it is fluid). Using 4 libs for JSON, postgres, HTTP, logging. I expose basically JSON based RPC. Peers / Clients post JSON commands and receive JSON replies. Upon startup all data (except couple of giant tables) is sucked from DB into RAM into highly efficient data structures hence most reading requests are handled in microseconds. Those 2 gia…

Nice! A lot of game servers are similar to this too, which you of course know. In my case I usually just "snapshot" the in memory structures to disk... BTW for others considering this approach, the DB will do similar things that FpUser is doing in the app layer - keeping hot/recent stuff in memory. But when tail latency and cost really matters this is a good technique.

>"DB will do similar things"

Not really as the storage data layout and application data layout are different. If I were to use database only relying on it's caching abilities it would lead to complex queries that are definitely way less performant than getting data from in memory structures optimized for this particular application / usage patterns.

On top of that some requests require some relatively complex calculations that again are executed against optimized in memory layout and would suffer greatly if done against the database.

I did some experiments and the difference was very significant.

Re: You don’t need to be “enterprise-ready” or “scalable”

#148

Post by raffraffraff »

nijave wrote: post #39
bob1029 wrote: post #35

Earlier quoted context omitted.

> Is this common at other companies that you cannot run the whole app locally to develop and test against? Yes. I work in B2B banking software, so having an "authentic" environment to test against is extraordinarily complex. Even the big vendors in the space seem unable to maintain accurate facsimiles of production with their own decades-old core software, so smaller vendors stand no chance unless they adjust their e…

Worked with banking software before; it can be incredibly complicated. I worked on proof of concept infrastructure for a Websphere monolith that ran on AIX servers. It had at least 10-15 services it connected to and each one of those were huge Java monoliths, too. Some of those apps had 8+ copies of /each/ environment (8x dev, 8x test, 4-8x stage/uat, 1-2x prod) so you could do integrated tests without stepping on ot…

It's funny, I worked in banking about 15 years ago and the technology stack was the same. Java monoliths, Websphere on AIX (plus mainframes naturally)

Re: You don’t need to be “enterprise-ready” or “scalable”

#149

Post by raffraffraff »

vbezhenar wrote: post #137
fasteddie31003 wrote: post #34

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

It is the case in my company. Production and staging is bunch of docker-composes scattered around several servers. We had to create another staging environment, I think it took several days to configure it. People run services locally, but not entire stack, that takes too much effort. Right now I’m trying to understand and deploy Kubernetes. Scalability and availability is good, but one of the reasons that I want to…

This: "Basically it's lack of devops". Companies think they can avoid hirig any infra/devops people and make all of that stuff the job of junior developers. They should contract someone it to set things up, at the very least.

Re: You don’t need to be “enterprise-ready” or “scalable”

#150

Post by andyjohnson0 »

fasteddie31003 wrote: post #34

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

In my experience, yes.

Often it's due to complexity. Systems accrete stuff (features, subsystems, dependencies, whatever) over time due to changing business requirements. The more this happens, the harder it becomes to integrate them properly without breaking other parts of the system. And the system is running the business, often 24 hours a day, which makes migrations hard to do. So you end up of something thats basically too complex to run locally.

And particularly in regulated industries like banking you have the security teams locking everything down hard. Problems accessing systems and getting realistic test data, etc.

Post reply on HN