Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Athena: Query S3 Using SQL

aws.amazon.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Amazon Athena: Query S3 Using SQL

#31

Post by xmly »

brilliantcode wrote: post #24
asteadman wrote: post #21

Earlier quoted context omitted.

Re: users auth. Isn't that what Cognito is supposed to be? I mean, I don't fully understand it, but I think so. As for the cold boot issue, I thought the standing solution was to have a "fast-exit" ping-like code-path within the lambda. Query it on a regular basis (you can even do it with a lambda scheduled-event). That way your lambda should be kept warm.

TIL Cognito! That completely flew under my radar, not sure why I didn't see it before (oh that's right I was heads down in Azure). With Athena the circle is complete for me. That fast exit ping thing is pretty cool, any more information regarding that? Your comment is probably the most valuable one I came across to date since signing up, I wish there was a way to award a gold star like on reddit :D There's very littl…

My solution currently is : Aws API gateway + Cognito + lambda + DynamoDB for webservice. S3 for html/css/js. CodeDeploy + Cloudformation for deployment.

Athena does not gurantee for timing, use it for async call and offline processing

Re: Amazon Athena: Query S3 Using SQL

#32

Post by brilliantcode »

xmly wrote: post #31
brilliantcode wrote: post #24

Earlier quoted context omitted.

TIL Cognito! That completely flew under my radar, not sure why I didn't see it before (oh that's right I was heads down in Azure). With Athena the circle is complete for me. That fast exit ping thing is pretty cool, any more information regarding that? Your comment is probably the most valuable one I came across to date since signing up, I wish there was a way to award a gold star like on reddit :D There's very littl…

My solution currently is : Aws API gateway + Cognito + lambda + DynamoDB for webservice. S3 for html/css/js. CodeDeploy + Cloudformation for deployment. Athena does not gurantee for timing, use it for async call and offline processing

super cool! would love to find out more about your workflow using those AWS components.

what did you mean by your last sentence?

Re: Amazon Athena: Query S3 Using SQL

#33

Post by bsg75 »

ktamura wrote: post #26
spullara wrote: post #10

Earlier quoted context omitted.

I don't know why you are getting downvoted. For all those data formats you have to painstakingly make table schemas for them before you can query them. Not like Snowflake or BigQuery. One of the biggest strikes against Presto IMHO.

It's not Presto per se, but running any data processing workload against unoptimized data formats is the issue. Then again, both BigQuey and Snowflake require that you move data into their storage engine (Redshift too), and that's an additional step that's proportional to the size and complexity of your data. At the same time, it's stupid to store your logs as OLAP optimized formats and completely lose legibility. In…

You don't replace them with an OLAP format, you can pair them with an OLAP engine to aggregate, filter, or analyze. Elastic Search and Splunk are one approach, SQL query engines are another.

Apache Drill is a schema discovery on read approach that can handle some of this. Its not perfect, but it does simplify some of the process where its capabilities fit the task at hand.

Re: Amazon Athena: Query S3 Using SQL

#34

Post by sciurus »

brilliantcode wrote: post #15
nulagrithom wrote: post #13

Wondering if I could use this like SQLite for Lambdas. I'd like to build some serverless apps, but the commitment to a monthly fee from DynamoDB puts me off. Could I use Athena to drive down my cost to zero as long as the app is unused?

DynamoDB is like $5 or $10 bucks a month? but I understand the need to keep it to a minimum. Athena is really interesting and if it can be as it is advertised "Serverless SQL" then they've got a killer product in the pipes: A future where developers no longer need to spend time on scaling, configuring, maintaining, strategizing deployments but upload code and instantly begin reaping the benefits of serverless tech. T…

Just because you can query it with SQL doesn't make it a relational database suitable for use for OLTP. Athena is built on Presto, so see https://prestodb.io/docs/current/overview/use-cases.html for an explanation.

Re: Amazon Athena: Query S3 Using SQL

#35

Post by gbrits »

brilliantcode wrote: post #32
xmly wrote: post #31

Earlier quoted context omitted.

My solution currently is : Aws API gateway + Cognito + lambda + DynamoDB for webservice. S3 for html/css/js. CodeDeploy + Cloudformation for deployment. Athena does not gurantee for timing, use it for async call and offline processing

super cool! would love to find out more about your workflow using those AWS components. what did you mean by your last sentence?

Not OP, but athena returns results for most queries in a couple of seconds (quote is somewhere in the blogpost) this would likely not be enough for your typical request/response flows.

Re: Amazon Athena: Query S3 Using SQL

#38

Post by fhoffa »

ktamura wrote: post #26
spullara wrote: post #10

Earlier quoted context omitted.

I don't know why you are getting downvoted. For all those data formats you have to painstakingly make table schemas for them before you can query them. Not like Snowflake or BigQuery. One of the biggest strikes against Presto IMHO.

It's not Presto per se, but running any data processing workload against unoptimized data formats is the issue. Then again, both BigQuey and Snowflake require that you move data into their storage engine (Redshift too), and that's an additional step that's proportional to the size and complexity of your data. At the same time, it's stupid to store your logs as OLAP optimized formats and completely lose legibility. In…

Note that BigQuery has been able to read files straight from GCS, Drive, and even Google Spreadsheets for a while:

https://cloud.google.com/bigquery/federated-data-sources

(I'm Felipe Hoffa and I work for Google https://twitter.com/felipehoffa)

Post reply on HN