Froid: Optimization of Imperative Programs in a Relational Database [pdf] • Hacker News Board

Re: Froid: Optimization of Imperative Programs in a Relational Database [pdf]

#11

Post by karthiksr » Mon, Dec 24, 2018, 6:53 PM UTC

maslam wrote: post #9

karthiksr wrote: post #6

Earlier quoted context omitted.

Thank you. The shipped preview has only a bit more than 1500LOC. The VLDB paper was presented at Rio in Aug this year already, but I'll try to come over to LA anyways :)

Karthik, I'm no Spark expert but almost all advice I read is to avoid UDFs if at all possible. Examples below: - https://medium.com/teads-engineering/spark-performance-tunin... - https://www.inovex.de/blog/efficient-udafs-with-pyspark/

Thank you for those pointers.

There are definitely some differences between the kind of UDFs that Spark supports and the kind that Froid handles. For one, Spark UDFs cannot invoke a Spark SQL query in their definition AFAIK, whereas TSQL functions can. But still, some techniques might be applicable. Definitely worth digging further!

Re: Froid: Optimization of Imperative Programs in a Relational Database [pdf]

#12

Post by prince617 » Tue, Dec 25, 2018, 4:36 AM UTC

maslam wrote: post #7

karthiksr wrote: post #2

I am a co-author of the Froid paper, and am around if people have any questions/comments/feedback. Froid is now available as a feature of SQL Server 2019 preview. The feature is called "Scalar UDF Inlining" https://blogs.msdn.microsoft.com/sqlserverstorageengine/2018... Available to try out for free here: https://www.microsoft.com/en-us/sql-server/sql-server-2019

Thank you for the paper - it is well-written and succinct. Karhik, do you think this approach can be applied to Apache Spark as well (given its well-known slowness with UDFs)?

You might want to check out this related work: http://casper.uwplse.org

Re: Froid: Optimization of Imperative Programs in a Relational Database [pdf]

#13

Post by karthiksr » Tue, Dec 25, 2018, 8:52 AM UTC

prince617 wrote: post #12

maslam wrote: post #7

Earlier quoted context omitted.

Thank you for the paper - it is well-written and succinct. Karhik, do you think this approach can be applied to Apache Spark as well (given its well-known slowness with UDFs)?

You might want to check out this related work: http://casper.uwplse.org

Thank you. Casper is very interesting work, and I am aware of it. Program synthesis offers an alternative approach to such problems, with different trade offs and characteristics.

The paper includes a brief discussion on synthesis-based techniques, and the reasoning behind Froid's design choices.