Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software Infrastructure 2.0: A Wishlist

erikbern.com

Post reply on HN
Print view View original

121–130 of 202 posts

Re: Software Infrastructure 2.0: A Wishlist

#121

Post by Archelaos »

rurounijones wrote: post #113
pgwhalen wrote: post #54

Earlier quoted context omitted.

Your comment makes me realize how lucky I am at my job: we do this production environment replication via “SAN magic” and it is so insanely useful.

How do you deal with replicating customer data / data the developer / tester should not have access to? Do you copy that into the ephemeral test environment as well? How are permissions managd for accessing that data? Does it have the same restrictions as in prod? (i.e. even in this env the developer / tester has no access to systems / data they would not have in prod).

Using real person-related data for testing was made illegal in Europe by GDPR. Even with explicit consent of each affected person, the tester needs to make sure that the test really requires real person-related data, otherwise it would violate the principle of "privacy by design" from GDPR. Without consent it is entirely forbidden, as of my understanding of the GDPR.

Due to GDPR I had to implement an anonymization feature for the production database anyway. So I am able to clone the production database for testing and run the anonymization function on every entry in the test database before using it.

Re: Software Infrastructure 2.0: A Wishlist

#122

Post by cryptica »

>> Software infrastructure (by which I include everything ending with *aaS, or anything remotely similar to it) is an exciting field, in particular because (despite what the neo-luddites may say) it keeps getting better every year

The fact that many people think that software infrastructure is getting worse is a very bad sign in itself (it cannot be ignored). There has always been religious wars over software development practices, but never to the extent we have today.

Re: Software Infrastructure 2.0: A Wishlist

#123

Post by olavgg »

jiggawatts wrote: post #47

Years and years ago I saw an advertisement by a SAN array vendor where their gimmick was that they supported very cheap read/write snapshots of very large datasets, in combination with a set of transforms such as data masking or anonymisation. Their target market was same as the OP: developers that need rapid like-for-like clones of production. The SAN could create a full VM server pool with a logical clone of hundre…

I want to a copy of our Azure SQL database dumped to my local dev MS SQL instance very often. At the moment I do this by using SSMS, where I generate a script of the Azure SQL database and use sqlcmd to import that script. It is a really painful process, and sqlcmd sometimes just hang and becomes unresponsive.

Why is there no simple export/import functionality?

Re: Software Infrastructure 2.0: A Wishlist

#124

Post by cowsandmilk »

klintcho wrote: post #33
stadium wrote: post #28

Earlier quoted context omitted.

There is AWS aurora serverless https://aws.amazon.com/rds/aurora/serverless/

I know this perspective is not relevant for most companies: But for side-projects and a-like the cold-start time for Aurora is like 15 - 30 seconds, and the first query always times out. Having it always on (if just 1 "compute") will cost you 30 USD a month. I'm hoping for Aurora to eventually be closer to DynamoDB pricing and startup (I'm fine with 5 - 10 second cold start as long as it doesn't time out the first re…

Have you evaluated aurora serverless v2? It is supposed to have much faster cold starts.

Re: Software Infrastructure 2.0: A Wishlist

#125

Post by [deleted] »

Archelaos wrote: post #121
rurounijones wrote: post #113

Earlier quoted context omitted.

How do you deal with replicating customer data / data the developer / tester should not have access to? Do you copy that into the ephemeral test environment as well? How are permissions managd for accessing that data? Does it have the same restrictions as in prod? (i.e. even in this env the developer / tester has no access to systems / data they would not have in prod).

Using real person-related data for testing was made illegal in Europe by GDPR. Even with explicit consent of each affected person, the tester needs to make sure that the test really requires real person-related data, otherwise it would violate the principle of "privacy by design" from GDPR. Without consent it is entirely forbidden, as of my understanding of the GDPR. Due to GDPR I had to implement an anonymization fe…

[deleted]

Re: Software Infrastructure 2.0: A Wishlist

#126

Post by nelsonenzo »

XorNot wrote: post #24

Some of this I agree with, even if the overall is to me asking for "magic" (I wouldn't mind trying to get there though). For me the one I really want is "serverless" SQL databases. On every cloud platform at the moment, whatever cloud SQL thing they're offering is really obviously MySQL or Postgres on some VMs under the hood. The provisioning time is the same, the way it works is the same, the outage windows are the…

They did, it's called Big Table by Google. Closed behind thier cloud offering few use.

Re: Software Infrastructure 2.0: A Wishlist

#127

Post by t_sawyer »

nine_k wrote: post #14
skywhopper wrote: post #3

While it’s a good idea to lay out a vision for the future of infrastructure as a service, it’s annoying to read this sort of article that seems to believe it ought to be simple to provide true serverless magical on-demand resources for random code. Fact is it’s a lot more complicated than that unless you’re just a small fry without any really hard problems to solve. It’s silly to complain that AWS doesn’t have a serv…

> It’s silly to complain that AWS doesn’t have a service that allows any random programmer to deploy a globally scalable app with one click. For one thing, you aren’t AWS’s primary target audience. I won't be surprised if AWS makes disproportionately more money off smaller operations which don't want to care about honing their infra, but want something be done quickly and with low effort. They quench the problem with…

They don't need to eat a slice of Heroku's pie. Heroku pays AWS for each of their customers already. Their infrastructure runs on top of AWS.

Re: Software Infrastructure 2.0: A Wishlist

#128

Post by tammerk »

Literally author is asking for something in the cloud which is abstracted away, fast, easy to use and easy to debug. This is not possible imho. There are always trade-offs. You can't get all of them for "free". e.g serverless services will be slower than your own setup, like always. Debugging, I'm not sure but I don't think it will be as easier as your own setup ever.

Re: Software Infrastructure 2.0: A Wishlist

#129

Post by jiggawatts »

mcny wrote: post #120
jiggawatts wrote: post #47

Years and years ago I saw an advertisement by a SAN array vendor where their gimmick was that they supported very cheap read/write snapshots of very large datasets, in combination with a set of transforms such as data masking or anonymisation. Their target market was same as the OP: developers that need rapid like-for-like clones of production. The SAN could create a full VM server pool with a logical clone of hundre…

> Meanwhile, in the Azure cloud: Practically nothing has a "copy" option. Deploying basic resources such as SQL Virtual Machines or App Service environments can take hours. Snapshotting a VM takes a full copy. Etc... To add to the list of etc, I can't query across two databases in azure SQL.

You can't even copy an Azure SQL Database across subscriptions! This is something that comes up regularly at my day job: there's an Enterprise Agreement production subscription, and an Enterprise Dev/Test subscription for non-production that's much cheaper. This would be great, except for stupid limitations like this.

Re: Software Infrastructure 2.0: A Wishlist

#130

Post by jiggawatts »

olavgg wrote: post #123
jiggawatts wrote: post #47

Years and years ago I saw an advertisement by a SAN array vendor where their gimmick was that they supported very cheap read/write snapshots of very large datasets, in combination with a set of transforms such as data masking or anonymisation. Their target market was same as the OP: developers that need rapid like-for-like clones of production. The SAN could create a full VM server pool with a logical clone of hundre…

I want to a copy of our Azure SQL database dumped to my local dev MS SQL instance very often. At the moment I do this by using SSMS, where I generate a script of the Azure SQL database and use sqlcmd to import that script. It is a really painful process, and sqlcmd sometimes just hang and becomes unresponsive. Why is there no simple export/import functionality?

You can export Azure SQL Databases to a "BACPAC" file, which is much faster. This can then be restored onto on-prem SQL equally fast.

It's about 20-30 lines of PowerShell to do this, and then you can just double-click the script and drink a coffee.

Alternatively, for data-only changes you can use Azure SQL Sync, but I've found that it's primitive and a bit useless for anything complicated...

Post reply on HN