Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Athena: Query S3 Using SQL

aws.amazon.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Amazon Athena: Query S3 Using SQL

#21

Post by asteadman »

brilliantcode wrote: post #15
nulagrithom wrote: post #13

Wondering if I could use this like SQLite for Lambdas. I'd like to build some serverless apps, but the commitment to a monthly fee from DynamoDB puts me off. Could I use Athena to drive down my cost to zero as long as the app is unused?

DynamoDB is like $5 or $10 bucks a month? but I understand the need to keep it to a minimum. Athena is really interesting and if it can be as it is advertised "Serverless SQL" then they've got a killer product in the pipes: A future where developers no longer need to spend time on scaling, configuring, maintaining, strategizing deployments but upload code and instantly begin reaping the benefits of serverless tech. T…

Re: users auth. Isn't that what Cognito is supposed to be? I mean, I don't fully understand it, but I think so.

As for the cold boot issue, I thought the standing solution was to have a "fast-exit" ping-like code-path within the lambda. Query it on a regular basis (you can even do it with a lambda scheduled-event). That way your lambda should be kept warm.

Re: Amazon Athena: Query S3 Using SQL

#23

Post by nulagrithom »

asteadman wrote: post #19
nulagrithom wrote: post #13

Wondering if I could use this like SQLite for Lambdas. I'd like to build some serverless apps, but the commitment to a monthly fee from DynamoDB puts me off. Could I use Athena to drive down my cost to zero as long as the app is unused?

Note the 10MB minimum "charge" per Query. For small datasets under 10MB, you'd only get up to 200 Queries for the minimum billable $0.01. That would be a fairly small number of queries, so probably not that useful. Plus you'd have all kinds of issues regarding consistency if your data was dynamic (s3 is a blob store, not a database, normal s3 consistency guarantees still apply). I'm confused though. The monthly fees…

Only thing you're missing is that I didn't realize DynamoDB's free tier was a non-expiring free tier.

Re: Amazon Athena: Query S3 Using SQL

#24

Post by brilliantcode »

asteadman wrote: post #21
brilliantcode wrote: post #15

Earlier quoted context omitted.

DynamoDB is like $5 or $10 bucks a month? but I understand the need to keep it to a minimum. Athena is really interesting and if it can be as it is advertised "Serverless SQL" then they've got a killer product in the pipes: A future where developers no longer need to spend time on scaling, configuring, maintaining, strategizing deployments but upload code and instantly begin reaping the benefits of serverless tech. T…

Re: users auth. Isn't that what Cognito is supposed to be? I mean, I don't fully understand it, but I think so. As for the cold boot issue, I thought the standing solution was to have a "fast-exit" ping-like code-path within the lambda. Query it on a regular basis (you can even do it with a lambda scheduled-event). That way your lambda should be kept warm.

TIL Cognito!

That completely flew under my radar, not sure why I didn't see it before (oh that's right I was heads down in Azure).

With Athena the circle is complete for me.

That fast exit ping thing is pretty cool, any more information regarding that?

Your comment is probably the most valuable one I came across to date since signing up, I wish there was a way to award a gold star like on reddit :D

There's very little objection at this point in moving to a Serverless architecture = Athena (SQL) + Lambda (CPU) + Cognito (User).

Re: Amazon Athena: Query S3 Using SQL

#25

Post by neximo64 »

raghavsethi wrote: post #20
neximo64 wrote: post #6

Any examples of queries and what this can do? S3 was file storage as far as i thought?

Athena (Presto) supports standard ANSI SQL - you can query data that's stored in S3.

How does that work though, so say my bucket has 10,000 json files in it and I want all of them with the name attributes being like '%john'. Is that possible?

Re: Amazon Athena: Query S3 Using SQL

#26

Post by ktamura »

spullara wrote: post #10
estefan wrote: post #5

Earlier quoted context omitted.

When I tried it it was slower than bigquery. Plus you've got to mess about creating hive schemas.

I don't know why you are getting downvoted. For all those data formats you have to painstakingly make table schemas for them before you can query them. Not like Snowflake or BigQuery. One of the biggest strikes against Presto IMHO.

It's not Presto per se, but running any data processing workload against unoptimized data formats is the issue.

Then again, both BigQuey and Snowflake require that you move data into their storage engine (Redshift too), and that's an additional step that's proportional to the size and complexity of your data. At the same time, it's stupid to store your logs as OLAP optimized formats and completely lose legibility. In sum, Athena trades off performance for convenience.

No matter what database vendors say, you can't defy the principles of computer science.

Re: Amazon Athena: Query S3 Using SQL

#27

Post by nimrody »

Would be useful if AVRO files were supported. This was the data can also be imported into Redshift if needed (Redshift does support Avro).

Other formats are schema-less (JSON,CSV, etc.) or not supported by Redshift (ORC, Parquet). Perhaps less efficient for some queries (AVRO is not a columnar format) but still useful.

Re: Amazon Athena: Query S3 Using SQL

#28

Post by ktamura »

danso wrote: post #11

This looks very neat. I'm someone who deals with a lot of plaintext data from a variety of sources, and so I find using ack/grep and csvkit to be efficient enough for my purposes of exploration. I love using SQL and SQLite but rarely do it for "fun" -- that is, I'll use it when I've committed to building a project, but not for exploration. This seems like it could lighten the friction quite a bit. If anyone from AWS…

The real question to ask is, will Amazon contribute back to open source? Presto itself is plenty proven and scalable: after all, it was created at Facebook.

Re: Amazon Athena: Query S3 Using SQL

#29

Post by asteadman »

brilliantcode wrote: post #24
asteadman wrote: post #21

Earlier quoted context omitted.

Re: users auth. Isn't that what Cognito is supposed to be? I mean, I don't fully understand it, but I think so. As for the cold boot issue, I thought the standing solution was to have a "fast-exit" ping-like code-path within the lambda. Query it on a regular basis (you can even do it with a lambda scheduled-event). That way your lambda should be kept warm.

TIL Cognito! That completely flew under my radar, not sure why I didn't see it before (oh that's right I was heads down in Azure). With Athena the circle is complete for me. That fast exit ping thing is pretty cool, any more information regarding that? Your comment is probably the most valuable one I came across to date since signing up, I wish there was a way to award a gold star like on reddit :D There's very littl…

What's your usecase for Athena (Really curious how this changes anything)?

You can hook your lambda up to a cloudwatch scheduled event (http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Sc...). If you don't do much when this happens (like, you exit immediately), then you won't be charged much compute time. Can't find the reference to how often you should ping it to keep it warm, probably varies, IIRC hourly would be plenty.

Re: Amazon Athena: Query S3 Using SQL

#30

Post by brilliantcode »

asteadman wrote: post #29
brilliantcode wrote: post #24

Earlier quoted context omitted.

TIL Cognito! That completely flew under my radar, not sure why I didn't see it before (oh that's right I was heads down in Azure). With Athena the circle is complete for me. That fast exit ping thing is pretty cool, any more information regarding that? Your comment is probably the most valuable one I came across to date since signing up, I wish there was a way to award a gold star like on reddit :D There's very littl…

What's your usecase for Athena (Really curious how this changes anything)? You can hook your lambda up to a cloudwatch scheduled event ( http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Sc... ). If you don't do much when this happens (like, you exit immediately), then you won't be charged much compute time. Can't find the reference to how often you should ping it to keep it warm, probably varies, IIRC hourly…

building user management, security has always been a "build your own wheel". While I think a properly secured SQL table with encrypted keywords is certainly possible the risk remains. We see even for prolific and large websites their entire user tables are dumped online on a regular basis.

Off-loading this burden on a cloud vendor whose sales are directly tied to the security, gives me peace of mind. Ashley Madison had no financial incentive to keep their user base as secure as possible or actively defend against it using R&D. Yahoo Mail as well. But AWS and Active Directory are products that sell this security and for them to fuck up on an epic scale would mean their end.

so the tldr is piggybacking on the tremendous resources from a cloud giant which frees up resources to focus on the core product.

This is not to say that it's impossible to roll out your own wheel, I'm just saying it makes more sense to align the financial incentives of vendors maintaining the user base security. This may or may not mean I'm open to focusing on companies solely focused on user auth/management products, it's tough to beat the branding and trust built into AWS & Azure that developers are voting everyday with their money & data.

Post reply on HN