Can cloud infrastructure become truly automatic?

#1

Post by rash805115 » Mon, Feb 04, 2019, 8:16 PM UTC

AWS, GoogleCloud, and Azure are the 3 main players that provide IaaS, but I have often found myself learning a whole lot before I can get a working setup with CICD and routing and scaling and database and logging setup correctly.

In my mind, the future is where you don't need to define infrastructure but based on your code infrastructure is derived. E.g. you can detect from configuration or npm/maven package list that you are using mongodb so it is automatically created and maintained.

I am questioning can we really automate cloud infrastructure to the degree where all you need to do is upload your code, and proper infrastructure is guessed, build, and is managed and maintained without any developer input.

Finally, if I start working on such a project, would any of you want to use it?

Re: Can cloud infrastructure become truly automatic?

#2

Post by wmf » Mon, Feb 04, 2019, 8:42 PM UTC

Magic provisioning + utility pricing = your bill is enormous and you have no idea why?

Re: Can cloud infrastructure become truly automatic?

#3

Post by LinuxBender » Mon, Feb 04, 2019, 8:58 PM UTC

Several companies have written their own implementation of datacenter automation, but it would be great if you published an open source framework that people could contribute to. What hardware do you plan to support? Which devices have you already automated?

We stand up servers, switches and routers fully automated, just like VM's, but I doubt we could transplant our code easily into another company.

Re: Can cloud infrastructure become truly automatic?

#4

Post by rash805115 » Mon, Feb 04, 2019, 10:32 PM UTC

wmf wrote: post #2

Magic provisioning + utility pricing = your bill is enormous and you have no idea why?

@wmf I should have explained better in my post that I am not talking about automating an existing AWS/Google infrastructure. What other companies are doing is they are generating automated scripts which will create an infrastructure for you and then you maintain it. What I am proposing is a new PaaS service which will parse your configuration and create an infrastructure on demand, publish your application in that infrastructure and send you a working link back. The developer does not have to interact with the infrastructure at all.

Re: Can cloud infrastructure become truly automatic?

#5

Post by wmf » Mon, Feb 04, 2019, 10:40 PM UTC

rash805115 wrote: post #4

wmf wrote: post #2

Magic provisioning + utility pricing = your bill is enormous and you have no idea why?

@wmf I should have explained better in my post that I am not talking about automating an existing AWS/Google infrastructure. What other companies are doing is they are generating automated scripts which will create an infrastructure for you and then you maintain it. What I am proposing is a new PaaS service which will parse your configuration and create an infrastructure on demand, publish your application in that in…

Right, that sounds cool but also dangerous in the sense that I could write code that requires excessive infrastructure under the hood without realizing it (just like people pull in 10,000 npm packages).

Re: Can cloud infrastructure become truly automatic?

#6

Post by rash805115 » Mon, Feb 04, 2019, 10:42 PM UTC

LinuxBender wrote: post #3

Several companies have written their own implementation of datacenter automation, but it would be great if you published an open source framework that people could contribute to. What hardware do you plan to support? Which devices have you already automated? We stand up servers, switches and routers fully automated, just like VM's, but I doubt we could transplant our code easily into another company.

Hi @LinuxBender. Sorry for my rusty explanation. I do not mean to automate IaaS by writing scripts and generating a cloud template because that is "developer-aware" infrastructure. Once the infra is built, you still have to maintain it and that is a big issue. What I am proposing is a new PaaS service that can provision and manage infra for that service transparently without being given any input from developer.

Re: Can cloud infrastructure become truly automatic?

#7

Post by rash805115 » Mon, Feb 04, 2019, 10:46 PM UTC

wmf wrote: post #5

rash805115 wrote: post #4

Earlier quoted context omitted.

@wmf I should have explained better in my post that I am not talking about automating an existing AWS/Google infrastructure. What other companies are doing is they are generating automated scripts which will create an infrastructure for you and then you maintain it. What I am proposing is a new PaaS service which will parse your configuration and create an infrastructure on demand, publish your application in that in…

Right, that sounds cool but also dangerous in the sense that I could write code that requires excessive infrastructure under the hood without realizing it (just like people pull in 10,000 npm packages).

Yes, but for this specific scenario I can monitor usage of each service and alert them. But I understand your concern in a broader sense that there are a lot of unknowns and people can do some real damage. But we can start small and see how it goes. Question is do you feel humanity should take a step in this direction to automate infra this way or it just too crazy?

Re: Can cloud infrastructure become truly automatic?

#8

Post by garettmd » Tue, Feb 05, 2019, 2:18 PM UTC

I think this is an interesting idea. The closest thing to this that I can think of is Terraform, but that's declarative, rather than automatic. I think what you're proposing would have a lot of challenges. Maybe you could make it work in the JS world with npm/maven; but I think it would be very difficult to make it automatic enough for a broad enough collection of languages and frameworks to be sufficiently useful. That said, I'd like to see where it goes, so good luck if you do attempt something.

Re: Can cloud infrastructure become truly automatic?

#9

Post by rash805115 » Tue, Feb 05, 2019, 7:37 PM UTC

garettmd wrote: post #8

I think this is an interesting idea. The closest thing to this that I can think of is Terraform, but that's declarative, rather than automatic. I think what you're proposing would have a lot of challenges. Maybe you could make it work in the JS world with npm/maven; but I think it would be very difficult to make it automatic enough for a broad enough collection of languages and frameworks to be sufficiently useful. T…

Hi @garettmd. Thanks for your encouraging words. I have a working solution for Javascript and node applications here @ https://plugins.quadnix.com/ and here https://www.youtube.com/embed/-8jaCIbZ2Is For every application you launch in this platform, the infra is being generated on the fly. So far things have not gone that complicated because I am heavily using docker and simple apps just need an apache/node server on front to work. But I have plans to add support for database and logs, and those could become little complicated. I also accept your point that not all frameworks can be supported, but lets just start with few and see how far we can take it. If you are a developer, would you be willing to try it out?