Live data from Hacker News

You don’t need to be “enterprise-ready” or “scalable”

gorelay.co

81–90 of 185 posts

Re: You don’t need to be “enterprise-ready” or “scalable”

#81
post #37

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

I work at a company that has similar practices to what you describe (we don’t run our services locally). While it’s hard to get used to at first, I think the silver lining is that it forces everyone to write good unit tests, since it’s the only efficient way to run their code before publishing a PR. Having worked on the opposite end (at a team where we mostly tested locally and had almost no automated tests), I much…

My last job was working at a company where you could not bring up the app locally. It didn't start that way. When the company was smaller I could develop a whole feature from the backend to the frontend and see the whole flow. As we grew it became increasingly difficult to run the app locally. Then it became impossible. Bugs grew and development time increased honstly 4x to do the same feature I could have done 3 years ago. I now think a good clean developer experience is key to a stable app.

Re: You don’t need to be “enterprise-ready” or “scalable”

#82
post #51

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

I work at a VOIP services provider and we operate multiple brands which are companies we’ve acquired over several years. One thing that has rang true of every company we’ve acquired is that there is typically barely even a way to run one piece locally, let alone the whole thing. Some of these when I took over there wasn’t even version control, and they were being developed by FTPing files to production server or even…

> they were being developed by FTPing files to production server or even editing them live with vim or something.

Many editors / IDEs (e.g. Notepad++) can directly connect to the FTP server, so working on a FTP servers looks almost same as working on a local folder. You just make changes to the files, and then refresh the web browser to see the changes.

Re: You don’t need to be “enterprise-ready” or “scalable”

#83
post #39

Earlier quoted context omitted.

Worked with banking software before; it can be incredibly complicated. I worked on proof of concept infrastructure for a Websphere monolith that ran on AIX servers. It had at least 10-15 services it connected to and each one of those were huge Java monoliths, too. Some of those apps had 8+ copies of /each/ environment (8x dev, 8x test, 4-8x stage/uat, 1-2x prod) so you could do integrated tests without stepping on ot…

I haven't found a wholistic integration testing framework where you can effortlessly mock out upstreams as running services , so that you don't have to ham-fist integration testing code into your app or rely on actual, live upstreams which are probably prone to breakage anyway.

I've done things like this in ad-hoc setups with Docker Compose and environment variables, managed with shell scripts. I would be very interested in a more thoughtful alternative that was integrated with testing.

Re: You don’t need to be “enterprise-ready” or “scalable”

#84

I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they in…

lol. It sounds like they hire smart people. But smart people =/= good engineers. Good engineers keep things as simple as they can.

Re: You don’t need to be “enterprise-ready” or “scalable”

#85
post #77

Earlier quoted context omitted.

At Google, it is very team-specific but it is generally not possible to run servers locally for testing and there generally is no staging environment, but it might be possible to bring up some part of the stack on Google's clusters. Those severs may read production data or some test data. Other than that there are static checks, unit tests, code reviews and the release process to ensure reliability.

How do you develop features if you can't run the server locally? Do you just write code, get it approved in a code review and push it to production?

Automated tests where you're just running a tiny subset of the code you're working on

Re: You don’t need to be “enterprise-ready” or “scalable”

#86

Earlier quoted context omitted.

Local dev is great until production gets complex enough that you lose parity. That'll happen even earlier now that we run x86 on the server but ARM locally. At my company we don't support local dev nor have any sort of staging environment. We have development linux servers connected to prod services and dbs. There are no pragmas or build flags to perform things differently in "dev". Things are gated with feature flag…

Local dev isn't the end, it is the beginning. With it it allows you to get further than you would otherwise - and most importantly, when (not if) you do discover a problem on a lower environment (or prod) it gives you a place to try and replicate it. You run everything locally so hopefully you can make a change to mirror the problem... Not a guarantee but at least you have a good starting point. Also your local dev s…

Yep. This is one reason I've always ran linux on my dev machine anywhere that would let me - sure, there shouldn't be any difference between OSes, but otherwise it's one more place for differences between dev and prod to creep in.

Re: You don’t need to be “enterprise-ready” or “scalable”

#87

I'm working on a project where the backend devs decided to use Cassandra to manage a few hundred entities. Due to different access modes they had to denormalize the data into multiple tables and we're constantly running into consistency issues. A fucking sqlite db with a few tables and indexes would do just fine. Also, the tool has maybe a dozen DAUs. But it's web scale, so, yeah…

Why would a team decide to use nosql to manage "a few hundred" entities? Nosql databases excel at simple lookups and gets by ID, rather than trying to sort and join pretty much ad hoc.

Why would you ever use SQL? The SQL transaction model doesn't make a lot of sense for most use cases (including essentially anything that uses the internet), and trivial usability things like being able to have collection columns without having to define a separate table all add up.

Re: You don’t need to be “enterprise-ready” or “scalable”

#88
post #39

Earlier quoted context omitted.

Worked with banking software before; it can be incredibly complicated. I worked on proof of concept infrastructure for a Websphere monolith that ran on AIX servers. It had at least 10-15 services it connected to and each one of those were huge Java monoliths, too. Some of those apps had 8+ copies of /each/ environment (8x dev, 8x test, 4-8x stage/uat, 1-2x prod) so you could do integrated tests without stepping on ot…

I haven't found a wholistic integration testing framework where you can effortlessly mock out upstreams as running services , so that you don't have to ham-fist integration testing code into your app or rely on actual, live upstreams which are probably prone to breakage anyway.

One way to balance is contract testing where you run a test suite against your mock/stub and the same test suite against the real thing. Your mock/stub would usually just have hardcoded data that aligns with the test suite to pass.

Another way is creating separate code libraries to abstract away external services. For instance, you might write ObjectStorage library that has real integration tests that use S3 but in your application you just stub out the ObjectStorage library. A lot of frameworks have verifying doubles or "magic mocks" or similar that will stub the method signatures so that can help make sure you're calling methods correctly (but wouldn't help catch data/logic issues).

Live upstreams are also prone to inconsistent state and race conditions--especially when you try to run more than 1 instance of the test suite at once.

For web services, I also like using "fixtures". You just manually run against a real service and copy the requests to a file (removing any sensitive data) than have your stub load the data from the file and return it. If you hit any bugs/regressions, you can just copy the results that caused them into a second file and add another test case.

Re: You don’t need to be “enterprise-ready” or “scalable”

#89
post #86

Earlier quoted context omitted.

Local dev isn't the end, it is the beginning. With it it allows you to get further than you would otherwise - and most importantly, when (not if) you do discover a problem on a lower environment (or prod) it gives you a place to try and replicate it. You run everything locally so hopefully you can make a change to mirror the problem... Not a guarantee but at least you have a good starting point. Also your local dev s…

Yep. This is one reason I've always ran linux on my dev machine anywhere that would let me - sure, there shouldn't be any difference between OSes, but otherwise it's one more place for differences between dev and prod to creep in.

Heck, in my previous $WORK, I saw differences from local (running Debian) to production running CentOS. Luckily one cluster of Jenkins build nodes were also running CentOS and those caught a subtle incompatibility between system libraries that would have blown up if we had merged and deployed to production. My local Debian testing env did not catch it.

Re: You don’t need to be “enterprise-ready” or “scalable”

#90
post #66
post #35

Earlier quoted context omitted.

> Is this common at other companies that you cannot run the whole app locally to develop and test against? Yes. I work in B2B banking software, so having an "authentic" environment to test against is extraordinarily complex. Even the big vendors in the space seem unable to maintain accurate facsimiles of production with their own decades-old core software, so smaller vendors stand no chance unless they adjust their e…

My gf works in UX for a major bank and they simply issued her a bank account in her name (with her SSN etc -- it's a real account, with ATM and debit cards and the like). If she and some co workers want to test something out relating to accounts they actually use these accounts to minimize this kind of problem. They must have some special code that says that these accounts can have their fees zeroed and probably are…

Previously led data teams in banks and you’re right. There are dozens of flags for test accounts. When auditing, we screen out these accounts. Of course, with a proper data cleaning process, analytics gets a lot easier
Post reply on HN