Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

trino.io

Post reply on HN
Print view View original

51–60 of 70 posts

Re: The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

#51

Post by jerryjerryjerry »

bitsondatadev wrote: post #36
jerryjerryjerry wrote: post #27

One of the features I'm interested in (or would like to have) from Trino or Presto is the workload management which can better manage different types of queries and allocate resources accordingly. This becomes important when more applications adopt Trino or Presto as a distributed SQL database/platform, where the impact from different queries or workloads can be mitigated, besides the dedicated resources (CPU, MEM, e…

For managing difrerent workloads, check out this blogs and this videos from Shopify, Salesforce, Goldman Sachs, and Electronic Arts, respectively: - https://engineering.salesforce.com/how-to-etl-at-petabyte-sc... - https://shopify.engineering/faster-trino-query-execution-inf... - https://trino.io/episodes/33.html - https://www.youtube.com/watch?v=-5mlZGjt6H4 All use the Lyft "Presto but really Trino"-Gateway project…

Yes, definitely it may help if going with multiple clusters, however, there are also many scenarios that we don't want to maintain multiple clusters. For example, when we come to a SaaS platform, multi-tenant is pretty typical where different tenants may have different workloads, and workload management would be needed for different users, or even within the same tenant. So the "built-in" workload management (besides other features for multi-tenant) would be a big plus.

Re: The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

#52

Post by dmitrykoval »

jjoonathan wrote: post #17
gavinray wrote: post #8

I recently had to write SQL query generation for AWS Athena, which is based off Presto 0.217 It turns out that the dialect doesn't support LATERAL joins with a LIMIT in them. The below query only works if you remove the LIMIT clause. https://i.stack.imgur.com/rdB1s.png This makes saying things like "Fetch all artists where ..., for each artist fetch their first 3 albums where ..., and for each album fetch the top 10…

AWS Athena: selling a buggy, old, stale copy of someone else's work (Presto / Trino) for high prices and getting away with it because you control the platform. If that's not peak Amazon, I don't know what is.

other arguments aside .. Athena costs $5 per 1TB scanned and also supports predicates pushdown to S3 Select. I wouldn't call this expensive, at least in comparison to self hosted Presto.

Re: The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

#53

Post by hashhar »

dmitrykoval wrote: post #52
jjoonathan wrote: post #17

Earlier quoted context omitted.

AWS Athena: selling a buggy, old, stale copy of someone else's work (Presto / Trino) for high prices and getting away with it because you control the platform. If that's not peak Amazon, I don't know what is.

other arguments aside .. Athena costs $5 per 1TB scanned and also supports predicates pushdown to S3 Select. I wouldn't call this expensive, at least in comparison to self hosted Presto.

At a certain scale it does become very expensive. It's easy math.

When your monthly Athena bill crosses whatever it would cost to have 5 or 10 EC2 machines it'll be cheaper to use Trino. At my previous workplace we moved from ~$40,000/month to ~$18,000/month by replacing Athena.

Athena is a very good tool to start with - unless you have super large scale you'll probably not outgrow it. But when you do there's Trino.

I do contribute to Trino - although I was merely a user when that cost reduction happened.

Re: The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

#55

Post by bitsondatadev »

georgewfraser wrote: post #48
bitsondatadev wrote: post #7

Earlier quoted context omitted.

I mean, the hive migration path is one thing. Now that Iceberg is taking over the old Hive model, data lakes are all the rage again. The other thing I would say is that Trino and Presto are not one-trick ponies or just hive replacements. There's also the ability to query across multiple systems that is, to me, the feature that future proofs a lot of architectures. It inherently frees you up to fiddle with your data i…

Yeah I think that is the key question: will data lakes become the dominant paradigm? There is certainly a lot of talk around them, though I see a ton of companies are still just going all in on a conventional data warehouse, but they tend not to talk about it because it’s not a new or interesting thing to do.

Yeah, though a lot of Fivetran customers are likely the type that would go all in on paying for a conventional data warehouse where people using open source stacks may be the ones that are using open ingestion alternatives.

We see a pretty even mix from the Trino/Starburst lens. Bigger companies like to mix and match.

Re: The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

#56

Post by slt2021 »

jjoonathan wrote: post #17
gavinray wrote: post #8

I recently had to write SQL query generation for AWS Athena, which is based off Presto 0.217 It turns out that the dialect doesn't support LATERAL joins with a LIMIT in them. The below query only works if you remove the LIMIT clause. https://i.stack.imgur.com/rdB1s.png This makes saying things like "Fetch all artists where ..., for each artist fetch their first 3 albums where ..., and for each album fetch the top 10…

AWS Athena: selling a buggy, old, stale copy of someone else's work (Presto / Trino) for high prices and getting away with it because you control the platform. If that's not peak Amazon, I don't know what is.

AWS is just managed open source as a service

Re: The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

#57

Post by dmitrykoval »

hashhar wrote: post #53
dmitrykoval wrote: post #52

Earlier quoted context omitted.

other arguments aside .. Athena costs $5 per 1TB scanned and also supports predicates pushdown to S3 Select. I wouldn't call this expensive, at least in comparison to self hosted Presto.

At a certain scale it does become very expensive. It's easy math. When your monthly Athena bill crosses whatever it would cost to have 5 or 10 EC2 machines it'll be cheaper to use Trino. At my previous workplace we moved from ~$40,000/month to ~$18,000/month by replacing Athena. Athena is a very good tool to start with - unless you have super large scale you'll probably not outgrow it. But when you do there's Trino.…

Sure, I agree, above certain usage threshold hosted Trino becomes totally justified. But then, some engineering time to maintain the cluster has to be factored in as well.. for anything ad-hoc in nature, I would start with Athena by default.

Re: The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

#58

Post by tfehring »

hashhar wrote: post #53
dmitrykoval wrote: post #52

Earlier quoted context omitted.

other arguments aside .. Athena costs $5 per 1TB scanned and also supports predicates pushdown to S3 Select. I wouldn't call this expensive, at least in comparison to self hosted Presto.

At a certain scale it does become very expensive. It's easy math. When your monthly Athena bill crosses whatever it would cost to have 5 or 10 EC2 machines it'll be cheaper to use Trino. At my previous workplace we moved from ~$40,000/month to ~$18,000/month by replacing Athena. Athena is a very good tool to start with - unless you have super large scale you'll probably not outgrow it. But when you do there's Trino.…

I'm not sure the math is so easy. Even knowing the direct cost savings in hindsight, engineers' time is expensive, and it's not obvious that the ongoing engineering cost of maintaining Trino on an EC2 cluster would be that far below $22k/month. Even if you get a net cost savings on an ongoing basis (which, granted, you probably do), you may have a long payback period for the initial engineering time spent evaluating solutions and getting the deployment spun up.

And that's all with the benefit of hindsight - it's hard to know a priori how much cheaper your own deployment will be compared to a managed service or how long it will take to implement. Of course, anecdotes like yours help with that, so thanks for sharing your experience!

Re: The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

#59

Post by pbronez »

RobinL wrote: post #33
jjoonathan wrote: post #17

Earlier quoted context omitted.

AWS Athena: selling a buggy, old, stale copy of someone else's work (Presto / Trino) for high prices and getting away with it because you control the platform. If that's not peak Amazon, I don't know what is.

I think many users just see they can execute a query on huge data cheaply and incredibly quickly and are delighted. That's certainly my experience. It's one of the backends available in Splink, our FOSS record linkage software and it's revolutionary how it allows users to execute large scale probabilistic record linkage ridiculously cheaply. It wasn't long ago you needed very expensive proprietary software plus a big…

Splink looks cool. I'm familiar with Tamr and Senzing, but this is the first FOSS option I've come across.

Re: The SQL query engine Trino (formerly PrestoSQL) recaps a decade of innovation

#60

Post by mrwnmonm »

We are building a SaaS BI tool.

To enable the users to connect to their databases... we have a form that collects the database credentials from the user, saves it in a secure way, and when the user writes or uses an SQL query, we establish a database connection right away (from our server), execute it, and return the results, and we keep the connection alive for like 15mins.

But with serverless architecture, first query could go to instance 1, so instance 1 will establish a db connection, then the second query could go to instance 2, so instance 2 will establish another one. You could end up with a lot of unnecessary connections.

If you use AWS RDS (for yourself), beside lambda for example, AWS have RDS Proxy to solve this problem.

So I was thinking about using Trino like the RDS Proxy, but for more databases, and for our customers database, not ours. Is that doable with Trino?

Post reply on HN