Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Visualize AWS Cloud Diagrams

aws.amazon.com

Post reply on HN
Print view View original

41–50 of 58 posts

Re: Visualize AWS Cloud Diagrams

#41

Post by toksvaeth »

viraptor wrote: post #31
toksvaeth wrote: post #25

Hi there, I'm from AWS and my team are responsible for development of this project (AWS Perspective). It's important to note that Perspective is not a traditional diagramming tool like draw.io or Visio. Perspective is intended to display resources that already exist rather than draw new ones. To do this, Perspective inventories resources in your accounts (Lambdas, DBs, EC2 instances, ENIs, EBS volumes etc.) and can a…

I may be missing some complexity that's not clear from the description, but since all the resources should be available in AWS Config already, why is it using so much extra infrastructure? Why isn't it a "point me at AWS Config dump, I'll give you an explorable static page/SVG" script instead? (The cost part would have to reach out to billing and/or cloudwatch, but still...) Or expressed a bit differently: what are t…

Hi,

That's a good question.

Whilst Perspective sources a lot of data from AWS Config, it also supplements and enriches the data from other sources where resources are not supported or where we wanted to show more detail. We'll be adding in other data sources in future, too.

Additionally, the AWS Config query language is quite helpful but doesn't expose a graph of the resources. Our approach is to pull the individual relationship data out and represent those in a graph DB (Amazon Neptune). Backing on to a graph database gives us the ability to let users navigate through the relationships of interest on the fly, rather than dumping what would inevitably be a complicated mess on the screen if we just showed everything.

Using this approach is also one of the ways we've tried to solve the UI/UX challenges presented in this space. Rather than say "show me all the things" that would lead to a complex mess of interconnections (and probably explode browsers) we instead encourage a workflow that starts with a resource or workload and explores outwards from there. It also allows us to implement some interesting features in future -- we intend to keep working on the project.

The ElasticSearch cluster is used by the discovery process to locate resources based on the metadata we collect; it's part of the way we enrich the data and relationships beyond what we get from Config.

Re: Visualize AWS Cloud Diagrams

#42

Post by staticassertion »

This feels like a strange and expensive way to solve this problem.

1. Why do you need a graph database? You could almost certainly keep the entire thing in memory, who has >16GB of infrastructure?

2. 500+ dollars a month for a service that I'll use very infrequently?

3. I feel like all I actually want is something that takes CloudFormation and gives me a picture, for free, in a python script on my laptop. But ok, fine, what about things not in CF, and other metadata sure, but then see (1) and (2).

If this were, say, one dollar a month (or serverless) and the infrastructure was totally abstracted away I'd be a lot more excited.

edit: OK so (2) isn't really their fault, since this isn't an AWS service, but still. And 600 bucks isn't that bad for many companies, I just think there's a bit of a pricetag shock on this one.

Re: Visualize AWS Cloud Diagrams

#43

Post by tidepod12 »

viraptor wrote: post #31
toksvaeth wrote: post #25

Hi there, I'm from AWS and my team are responsible for development of this project (AWS Perspective). It's important to note that Perspective is not a traditional diagramming tool like draw.io or Visio. Perspective is intended to display resources that already exist rather than draw new ones. To do this, Perspective inventories resources in your accounts (Lambdas, DBs, EC2 instances, ENIs, EBS volumes etc.) and can a…

I may be missing some complexity that's not clear from the description, but since all the resources should be available in AWS Config already, why is it using so much extra infrastructure? Why isn't it a "point me at AWS Config dump, I'll give you an explorable static page/SVG" script instead? (The cost part would have to reach out to billing and/or cloudwatch, but still...) Or expressed a bit differently: what are t…

Speculation on my part, but this seems to be much more geared towards data analysis of your environment rather than just having an SVG for reference.

I'm assuming like most things AWS, this is not targeted at your small-sized startup running 2 VPCs, a handful of subnets, and 10-20 EC2 instances. This is likely built specifically for major enterprises running a hundred thousand instances, with hundreds of VPCs, thousands of subnets, hundreds of thousands of security groups, etc. In such environments, a static SVG isn't feasible and you might want something that allows more analytics instead, which I think is the entire point of using Neptune.

Re: Visualize AWS Cloud Diagrams

#44

Post by jcims »

staticassertion wrote: post #42

This feels like a strange and expensive way to solve this problem. 1. Why do you need a graph database? You could almost certainly keep the entire thing in memory, who has >16GB of infrastructure? 2. 500+ dollars a month for a service that I'll use very infrequently? 3. I feel like all I actually want is something that takes CloudFormation and gives me a picture, for free, in a python script on my laptop. But ok, fin…

The graph database is useful for external analysis of the relationships between your resources. I've built this in the past using just AWS Config and, while incredibly useful, it is extremely limited due to Config's very limited scope.

If this covers a reasonable percentage of AWS services it would be more than cost justified at $1-2k/mo for our environment.

Re: Visualize AWS Cloud Diagrams

#45

Post by jcims »

toksvaeth wrote: post #25

Hi there, I'm from AWS and my team are responsible for development of this project (AWS Perspective). It's important to note that Perspective is not a traditional diagramming tool like draw.io or Visio. Perspective is intended to display resources that already exist rather than draw new ones. To do this, Perspective inventories resources in your accounts (Lambdas, DBs, EC2 instances, ENIs, EBS volumes etc.) and can a…

Do you have a list of supported services? I've built a very similar solution using AWS Config to feed Neo4j and it has been incredibly useful, but Config's coverage is so anemic that we are left with just a tiny portion of our infrastructure visible.

Re: Visualize AWS Cloud Diagrams

#46

Post by staticassertion »

jcims wrote: post #44
staticassertion wrote: post #42

This feels like a strange and expensive way to solve this problem. 1. Why do you need a graph database? You could almost certainly keep the entire thing in memory, who has >16GB of infrastructure? 2. 500+ dollars a month for a service that I'll use very infrequently? 3. I feel like all I actually want is something that takes CloudFormation and gives me a picture, for free, in a python script on my laptop. But ok, fin…

The graph database is useful for external analysis of the relationships between your resources. I've built this in the past using just AWS Config and, while incredibly useful, it is extremely limited due to Config's very limited scope. If this covers a reasonable percentage of AWS services it would be more than cost justified at $1-2k/mo for our environment.

Yeah, thinking about it more I do feel that this is a justifiable cost.

Re: Visualize AWS Cloud Diagrams

#47

Post by shmoogy »

This seems crazy for someone in a smaller account. I just want something that will chart out all my current resources allocated in a VPC/account and send to me once a week or so - so I can spot check if anything got accidentally left on.

I kind of feel like you would need $20k+/month spend for it to be worth $500 a month to do this.

Re: Visualize AWS Cloud Diagrams

#48

Post by toksvaeth »

jcims wrote: post #45
toksvaeth wrote: post #25

Hi there, I'm from AWS and my team are responsible for development of this project (AWS Perspective). It's important to note that Perspective is not a traditional diagramming tool like draw.io or Visio. Perspective is intended to display resources that already exist rather than draw new ones. To do this, Perspective inventories resources in your accounts (Lambdas, DBs, EC2 instances, ENIs, EBS volumes etc.) and can a…

Do you have a list of supported services? I've built a very similar solution using AWS Config to feed Neo4j and it has been incredibly useful, but Config's coverage is so anemic that we are left with just a tiny portion of our infrastructure visible.

We sure do: https://docs.aws.amazon.com/solutions/latest/aws-perspective...

If there are more you'd like to see added, please raise an issue: https://github.com/awslabs/aws-perspective

Re: Visualize AWS Cloud Diagrams

#49

Post by degenerate »

Hnrobert42 wrote: post #15
hlandau wrote: post #3

Is this a joke? I feel like this is a parody of AWS, and the notion of using as many AWS services as possible to make something otherwise simple.

Not to mention this is something for which AWS already has all the data. They should just be providing this diagram as a free part of the AWS experience.

I've always found it annoying that they have all the data readily available to bill you for, but not to SHOW you, your overall utilization.

Re: Visualize AWS Cloud Diagrams

#50

Post by ctvo »

tidepod12 wrote: post #40
ctvo wrote: post #32

The decentralized nature of Amazon / AWS rears its head again. It's much harder, I assume, to get security / teams to allow another team to crawl through its customer data (the Lambda team owns which customers have how many Lambdas running where, for example). Now imagine doing that for every AWS resource to form a graph of relationships. To get around that they deployed something that customers are wholly responsibl…

>To get around that they deployed something that customers are wholly responsible for. I doubt they were "getting around" anything. The customer being responsible for it is likely the entire point, because the customer who asked for this to be created probably specifically requested that (based on my experience working with AWS SAs). This post is an "AWS Solutions Implementation". This isn't a service like S3/EC2/Lam…

> I doubt they were "getting around" anything. The customer being responsible for it is likely the entire point, because the customer who asked for this to be created probably specifically requested that (based on my experience working with AWS SAs).

I doubt a customer asked AWS: hey, can I get a graph of how the resources in my accounts all interconnect while maintaining the clusters for the data stores, handling upgrade, security, etc. myself and paying hundreds of dollars a month for the pleasure?

The alternative is a customer asked for a way to see how resources were connected and how much it was costing them, and AWS, unable to deliver because of N hurdles (I mentioned security, but no doubt others), delivered this instead. Again: this should be a managed service and free.

Post reply on HN