Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stochastic gradient descent written in SQL

maxhalford.github.io

Post reply on HN
Print view View original

141–150 of 187 posts

Re: Stochastic gradient descent written in SQL

#141

Post by rukuu001 »

spprashant wrote: post #135
jackschultz wrote: post #13

>A machine learning algorithm which can be trained using SQL opens a world of possibilities. The model and the data live in the same space. This is as simple as it gets in terms of architecture. Basically, you only need a database which runs SQL. First paragraph of the conclusion, and this very much fits with the mindset that's been growing in me in the data world over the past few years. Databases are much more powe…

The problem with throwing everything in a database is you end up with brittle stored procedures all over the place, which are painful to debug. There is no good support for version control or testing, which means you end up creating a dozen copies of each function named (sp_v1, sp_v2,.., etc.). It much more harder to practice iterative development which the rest of software development seems implements effectively. A…

No comment re rDBs supporting parallelized ML but re stored procedures - if your workflow evolves to treat them as ‘1st class’ code assets they’ll be just the same as the rest of your code.

We always had them in version control, unit tested etc. The tools are there if you want to use them.

Re: Stochastic gradient descent written in SQL

#142

Post by swyx »

Lemaxoxo wrote: post #28
swyx wrote: post #17

Earlier quoted context omitted.

had a very good chat with https://postgresml.org/ last week which is focusing on bringing ML to postgres: https://youtu.be/j8hE8-jZJGU

I'm watching it, it's really good. Montana makes a great point: you can move data to the models, or move the models to the data. Data is typically larger than models, so it makes sense to go with the latter.

thanks for watching! i should really up the production quality haha but also this is what i can kinda manage with my existing workload. idk how the pro youtubers make these calls interesting

Re: Stochastic gradient descent written in SQL

#143

Post by hdespiritu »

zackmorris wrote: post #71

This is great! Moving away from the proprietary nature of GPUs and complex math gatekeeping should help democratize AI. Has anyone converted stuff like gradient descent to set theory? https://support.unicomsi.com/manuals/soliddb/7/SQL_Guide/2_G... https://www.sqlshack.com/mathematics-sql-server-fast-introdu... https://www.sqlshack.com/learn-sql-set-theory/ Right now AI algorithms kind of look imperative and stateful…

Your question "Has anyone converted stuff like gradient descent to set theory?" doesn't make sense from the perspective that gradient descent uses differential calculus to find min/max points of an objective function and differential calculus requires a lot of additional assumptions on top of set theory.

That being said, current deep learning libraries such as JAX and Pytorch use automatic differentiation to efficiently compute partial derivatives used for optimization algorithms such as gradient descent and it's not clear to me what the level of effort would be to convert that to something that could run efficiently in SQL?

Re: Stochastic gradient descent written in SQL

#144

Post by teruakohatu »

zackmorris wrote: post #71

This is great! Moving away from the proprietary nature of GPUs and complex math gatekeeping should help democratize AI. Has anyone converted stuff like gradient descent to set theory? https://support.unicomsi.com/manuals/soliddb/7/SQL_Guide/2_G... https://www.sqlshack.com/mathematics-sql-server-fast-introdu... https://www.sqlshack.com/learn-sql-set-theory/ Right now AI algorithms kind of look imperative and stateful…

> This is great! Moving away from the proprietary nature of GPUs and complex math gatekeeping should help democratize AI.

I understand the sentiment but you can run ML on plain CPUs, in fact it is easier to do so, you just don't get the massive speed benefits of GPUs. Hugging face's public models are all run on CPUs to save costs.

> Right now AI algorithms kind of look imperative and stateful to me, like state machines.

The maths is not stateful or imperative, only our implementation. Much like how functional languages are executed on a stateful CPU.

Research in parallelization is ongoing.

Re: Stochastic gradient descent written in SQL

#145

Post by hdespiritu »

glomgril wrote: post #87

Incredible post. I laughed when I saw the title, snickered at the first paragraph, and then proceeded to be blown away by the rest of it. Thought I was in for a joke and instead I'm thinking about the nature of ML Ops and what it's become.

My sentiment exactly. The premise of the article comes across a little naive because there are so many fine-tuned libraries for specialized hardware architectures that already do this computation very efficiently.

However, it did make me wonder what this might look like on a gpu-accelerated database engine that is designed to leverage the SIMD parallelism of GPGPU architectures.

Beyond using SQL/NoSQL databases for CRUD apps I am not a "database guy", so I'm not sure about the feasibility, but it would be interesting to see it implemented.

Re: Stochastic gradient descent written in SQL

#146

Post by westurner »

Lemaxoxo wrote: post #91
Terretta wrote: post #86

Title here is wrong. Title in article and headings in article are right: ONLINE gradient descent It's specifically not stochastic. From the article: Online gradient descent Finally, we have enough experience to implement online gradient descent. To keep things simple, we will use a very vanilla version: - Constant learning rate, as opposed to a schedule. - Single epoch, we only do one pass on the data. - Not stochast…

Hehe I was wondering if someone would catch that. Rest assured, I know the difference between online and stochastic gradient descent. I admit I used stochastic on Hacker News because I thought it would generate more engagement.

What are some adversarial cases for gradient descent, and/or what sort of e.g. DVC.org or W3C PROV provenance information should be tracked for a production ML workflow?

Gradient descent: https://en.wikipedia.org/wiki/Gradient_descent

Stochastic gradient descent: https://en.wikipedia.org/wiki/Stochastic_gradient_descent

Online machine learning: https://en.wikipedia.org/wiki/Online_machine_learning

adversarial gradient descent site:github.com inurl:awesome : https://www.google.com/search?q=awesome+adversarial+gradient...

https://github.com/EthicalML/awesome-production-machine-lear...

Robust machine learning: https://en.wikipedia.org/wiki/Robustness_(computer_science)#...

Robust gradient descent

Re: Stochastic gradient descent written in SQL

#147

Post by gbrits »

Lemaxoxo wrote: post #25
noloblo wrote: post #21

Earlier quoted context omitted.

Does this work with Postgres?

Postgres has excellent support for WITH RECURSIVE, so I see no reason why it wouldn't. However, as I answered elsewhere, you would need to set some stateful stuff up if you don't want the query to start from scratch when you re-run it.

I for one would welcome a second blogpost with the stateful example. I’ve been doing running totals (basically balances based on adding financial tx per account) but had to do it in some client code because I couldn’t figure out how to do a stateful resume in Sql

Re: Stochastic gradient descent written in SQL

#148

Post by simonw »

Lemaxoxo wrote: post #102
simonw wrote: post #65

I tried to replicate this in SQLite. The first few steps worked OK, e.g. https://lite.datasette.io/?json=https://gist.github.com/simo... (I replaced "figures" with "raw" due to the way Datasette Lite assigns a default table name to the imported JSON) But the more complex recursive queries gave me this error and I'm not sure how to work around it: recursive reference in a subquery: state E.g. https://lite.datasette.io…

That's too bad, I would have expected it to work out of the box. Other than rewriting the query in a different way, I'm not sure I see an easy workaround. Are you still working on this?

No I managed to drag myself away from it and get back to what I'd been intending to work on!

Re: Stochastic gradient descent written in SQL

#149

Post by mr_toad »

naasking wrote: post #124
chrstr wrote: post #85

Earlier quoted context omitted.

create table x as (select * from person); select name from x where ...; there you go, just configure your editor to display "create table x" as "declare x = " ;) or even a version with lazy evaluation: create view x as (select * from person); select name from x where ...;

You're still not getting it. First-class status means that anywhere a value or variable can be used, a query or table should also be able to appear, and vice versa. This means a table or query can appear as a return type, a parameter to a stored procedure or query, a variable, and so on. SQL just does not have this, it instead has 15 different second class ways to handle tables and queries that try to make up for the…

> You're still not getting it. First-class status means that anywhere a value or variable can be used, a query or table should also be able to appear, and vice versa. This means a table or query can appear as a return type, a parameter to a stored procedure or query, a variable, and so on.

I doubt you could implement a query planner that would cope with that degree of flexibility. Which means you’d be forced to deal with the mechanics of the query, pushing you away from declarative SQL and into procedural and functional programming. At which point you might as well ditch SQL anyway.

Re: Stochastic gradient descent written in SQL

#150

Post by mr_toad »

zackmorris wrote: post #71

This is great! Moving away from the proprietary nature of GPUs and complex math gatekeeping should help democratize AI. Has anyone converted stuff like gradient descent to set theory? https://support.unicomsi.com/manuals/soliddb/7/SQL_Guide/2_G... https://www.sqlshack.com/mathematics-sql-server-fast-introdu... https://www.sqlshack.com/learn-sql-set-theory/ Right now AI algorithms kind of look imperative and stateful…

Most ML algorithms are numerical approximations, so yes, they look procedural. You can try and find analytical solutions, but except for the simplest models they’re intractable.
Post reply on HN