Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI for AWS Documentation

awsdocsgpt.com

Post reply on HN
Print view View original

81–90 of 134 posts

Re: AI for AWS Documentation

#81

Post by scarface_74 »

avereveard wrote: post #78
8note wrote: post #54

Earlier quoted context omitted.

This all looks about right, but how would somebody without aws experience be able to validate it without making a mess?

It still isn't tho, this will work for testing, maybe, but each lambda will be it's own connection and that will exhaust db resources real fast, you're supposed to have a pooling proxy between lambdas and RDS.

I asked ChatGPT the same question again and then followed up with:

“Is there a more efficient way to connect to the database”

It then went off the rails. It gave me generic answers about using connection pooling that your language framework provides.

Of course that doesn’t help with each Lambda being invoked separately.

I then asked “But each Lambda runs based on its own vm”

It then correctly said that while each Lambda invocation would have its own pool and couldn’t be shared, it could be shared with multiple connections during one invocation.

Which is technically correct. But not the answer I was looking for.

In all fairness, it’s the same type of answers I would expect from a junior to mid developer with limited experience with AWS. I would hire a person who could answer that well.

Re: AI for AWS Documentation

#82

Post by cypress66 »

underlines wrote: post #58

RAG is very difficult to do right. I am experimenting with various RAG projects from [1]. The main problems are: - Chunking can interfer with context boundaries - Content vectors can differ vastly from question vectors, for this you have to use hypothetical embeddings (they generate artificial questions and store them) - Instead of saving just one embedding per text-chuck you should store various (text chunk, hypothe…

What context size is used? Can you "brute force" your way by using let's say 16k or 32k of context?

Re: AI for AWS Documentation

#83

Post by tomhamer »

underlines wrote: post #58

RAG is very difficult to do right. I am experimenting with various RAG projects from [1]. The main problems are: - Chunking can interfer with context boundaries - Content vectors can differ vastly from question vectors, for this you have to use hypothetical embeddings (they generate artificial questions and store them) - Instead of saving just one embedding per text-chuck you should store various (text chunk, hypothe…

Marqo provides automatic, configurable chunking (for example with overlap) and can allow you to bring your own model or choose from a wide range of opensource models. I think e5-large would be a good one to try. https://github.com/marqo-ai/marqo

Re: AI for AWS Documentation

#84

Post by jacurtis »

cle wrote: post #49
scarface_74 wrote: post #39

Earlier quoted context omitted.

Why? You can ask the same questions to ChatGPT and get the same or better answers. I also know from personal experience with ChatGPT, that you can use it to: - convert Python/boto3 to any language that has an AWS SDK - convert CloudFormation to Terraform or the CDK - write scripts that use the SDK You will get the occasional hallucination.

Here's one example question that ChatGPT utterly fails at, but that this answers fine: "What is Amazon CodeCatalyst?" ChatGPT: "I'm sorry, but as of my knowledge cut-off in September 2021, there was no service, tool, or product known as Amazon CodeCatalyst offered by Amazon Web Services (AWS). [...]"

A lot of stuff in AWS has changed since Q3 2021.

Re: AI for AWS Documentation

#86

Post by jacurtis »

scarface_74 wrote: post #41
WrtCdEvrydy wrote: post #38

Earlier quoted context omitted.

> Inexperienced folks won’t know better. I've seen answers like this in Stack Overflow before. Technically correct, but basically broken in every other way: cybersecurity's future is gonna be interesting.

One of my pet peeves about almost every answer on Stack Overflow is that the answer always starts with client=boto3.client( , access-key,secret-key) There is never a reason to include those last two parameters. It’s always a security risk no matter how you do it.

For that matter, using an access-key or secret-key at all is basically an anti-pattern now.

In the past few months AWS has added to their UI a popup box that asks you why you are creating access keys, it makes you answer it before generating the keys. Once you give it the answer it tells you better ways to do almost anything you're trying to do, and you have to agree to ignore those suggestions one more time before actually generating the keys.

Basically the standard access keys are really of no use anymore. You should be using roles, OIDC, IAM Identity Center, cognito federated identities, or something else. There are a million ways to identify yourself that are more secure than that and I wish tutorials stopped assuming that you should generate access keys because in no practical 2023 use case should you be doing that anymore. Especially not in any sort of sensitive or corporate environment.

Re: AI for AWS Documentation

#87

Post by ec109685 »

Asked it whether you should use a shared vpc:

Made this up as reason for dedicated vpc: “Better performance: By using dedicated hardware, you have better control over the performance of your instances in the VPC. This can be beneficial for applications that have stringent performance requirements.”

Re: AI for AWS Documentation

#88

Post by [deleted] »

scarface_74 wrote: post #81
avereveard wrote: post #78

Earlier quoted context omitted.

It still isn't tho, this will work for testing, maybe, but each lambda will be it's own connection and that will exhaust db resources real fast, you're supposed to have a pooling proxy between lambdas and RDS.

I asked ChatGPT the same question again and then followed up with: “Is there a more efficient way to connect to the database” It then went off the rails. It gave me generic answers about using connection pooling that your language framework provides. Of course that doesn’t help with each Lambda being invoked separately. I then asked “But each Lambda runs based on its own vm” It then correctly said that while each Lam…

[deleted]

Re: AI for AWS Documentation

#89

Post by BoorishBears »

scarface_74 wrote: post #81
avereveard wrote: post #78

Earlier quoted context omitted.

It still isn't tho, this will work for testing, maybe, but each lambda will be it's own connection and that will exhaust db resources real fast, you're supposed to have a pooling proxy between lambdas and RDS.

I asked ChatGPT the same question again and then followed up with: “Is there a more efficient way to connect to the database” It then went off the rails. It gave me generic answers about using connection pooling that your language framework provides. Of course that doesn’t help with each Lambda being invoked separately. I then asked “But each Lambda runs based on its own vm” It then correctly said that while each Lam…

Any time you ask ChatGPT a technical question on something you're not familiar with it should be three parts:

- What are the considerations?

- Implement it with the considerations

- Did we forget anything?

It understands the need for a proxy from step 1:

https://chat.openai.com/share/7ca37130-a771-457c-8742-a5f941...

Re: AI for AWS Documentation

#90

Post by jacurtis »

8note wrote: post #54
scarface_74 wrote: post #40

Earlier quoted context omitted.

Asking the same question to Chatgpt To connect a Lambda function to an RDS (Relational Database Service) instance and query it safely, you can follow these general steps: 1. Configure your Lambda function's execution role: Ensure that your Lambda function has the necessary permissions to access the RDS instance. The role should include the `AWSLambdaVPCAccessExecutionRole` and the appropriate RDS permissions. 2. Plac…

This all looks about right, but how would somebody without aws experience be able to validate it without making a mess?

This to me is why I don't use ChatGPT. Every time I use it I get answers that are questionable in validity. I'd say that 3/4 of my questions have returned incorrect answers at least partially. Maybe 10% are wildly incorrect with the remainder being partially incorrect.

So if it is wrong with MOST of the questions that I am able to validate myself, then how can I trust it on the questions that I am unable to validate myself.

The whole reason I started even doing this passive research is because I had an employee late last year who used to be a decent employee (he was never an all-star, but he got the job done to an average/satisfactory level), and started to all of a sudden perform incredibly poorly.

He was submitting code and solving problems that were just really bad. He was always just an average producer, and was always teetering on the edge during performance reviews, doing just well enough that we kept him around. But he quickly started to make mistake after mistake. Several code reviews I found really strange artifacts and comments in his code that were blatant mistakes. I confronted him about them over the course of several performance reviews and said he blamed stackoverflow "copy pasta". Eventually I actually fired him, since it was getting to a point that almost everything he submitted or produced was problematic in some way and he was burning more of my time than he was saving. So I ended up firing him.

While firing him, in front of HR he finally broke down and admitted that he has been using ChatGPT for everything and he begged us to let him stay and he would stop using it altogether. I of course didn't care at that point and we let him go. But I started to realize the increase of mistakes were all due to ChatGPT leading him astray.

That whole experiment really taught me that ChatGPT is not ready for primetime. If you blindly trusted ChatGPT you will find yourself in the wrong place most of the time. The problem is that unless you already know the answer to the question you are asking, it is very difficult to tell where chatgpt's answers might be correct and were they might be incorrect (because it is usually a mixture of both). This makes it entirely useless for asking questions that you are not comfortable validating.

Post reply on HN