Live data from Hacker News

Stochastic gradient descent written in SQL

maxhalford.github.io

121–130 of 187 posts

Re: Stochastic gradient descent written in SQL

#121

In the comments here so far, we see a pattern we've seen before. When someone suggests doing something in SQL, there's a lot of concern about SQL being a very limited programming language where it's hard to do proper engineering. Here's I would really love to know: why is it that SQL is, to first order, the only language used to interact with databases, and SQL has about the same features as it did in the 70s? It see…

I believe the PC revolution gave a few generations of programmers the mistaken belief that imperative languages with familiar features (Basic, Pascal, C, Perl, Python, Java, etc.) are the only or the best way to interact with computers. Forms of interaction developed earlier (Forth, Prolog, SQL, etc.) were learned later, grudgingly, if at all. SQL was viewed with disdain, a necessary evil to be held at arm's length. That stalled evolution of SQL, the success of other query languages, or really anything that didn't fit the Algol-style "general purpose programming language" blinders.

Mercifully, finally, the cold war against SQL in particular and against relational databases in general seems to be thawing. Articles like this one exemplify that trend

Re: Stochastic gradient descent written in SQL

#122

Earlier quoted context omitted.

Basically, but queries are not first class in SQL. You can't assign a query to a variable, or pass it as a parameter to a stored procedure, for example. This would make SQL composable: declare @x = (select * from Person) select Name from @x where Birthdate

You can do composition through CTE or Table-Valued Function.

CTE and TVF still treat tables as second class citizens, so while they enable some forms of composition, they're still very restricted. This has been the consistent story with SQL, 15 ways to use queries and tables, all to work around the limitation that they are not first class values.

Re: Stochastic gradient descent written in SQL

#123

This is great. The only thing I dislike from this is using these variables to try to predict Adj Close when they are not at all correlated. There are countless meaningful correlations in financial data that would have been just as easy to play around with. One truly valuable example would be to look at trading multiples of comparable companies. Sticking to P/E would be easier as P is easily observable and forward-loo…

Do you have some data/resources on this? I'm a total snowflake at this, but I'm willing to learn.

In hindsight this is harder than it seems if you don't already have access to data, so I'll try to be as informative as possible in my response

It's hard to find this information out there, so here's ~all you need to know.

Data is usually behind paywalls, unfortunately. Industry standards are Bloomberg terminal (ridiculously expensive, 5 digits $), FactSet (very expensive, 4 digits), Capital IQ (expensive, not sure)... but there are a number of up-and-coming startups trying to disrupt the space so you may be able to grab data from them. I think https://atom.finance has a 7-day free trial you could use to play around with.

P/E simply means the company's _P_rice per share divided by _E_arnings per share. Cancel out the "per share" terms and you get total market capitalization (which is the value of the total equity) divided by net income (since "earnings per share" really means "net income per share")

So the "P" is easy to get. It's your Adj Close.

The "E" is trickier as it can mean a lot of things. Diluted EPS from financial statements? Last year's EPS? Management's guidance for EPS? None of those are actually correct even if they are all "EPS"

Importantly--and contrary to 99% of the info you will find online--the most relevant EPS number are forward estimates of EPS, usually for the next twelve months ("NTM"). That is based on an average or median of analyst estimates which is called "consensus". These are analysts from financial institutions who build their own little models based on their own views of where the business is going to go, informed by recent earnings, management's color in earnings calls and filings, etc.

Believe it or not, as hairy as that sounds, EPS is fairly easy to get as it's a metric that has less room for interpretation than, say, EBITDA.

So you're not going to go out there, read all these (paid) analyst reports, find their EPS, calculate the median, etc. Bloomberg, Capital IQ, FactSet do this for you and it's easily observable for the end user (that's their business).

The thing is, as you may have guessed, "next twelve months" are a moving target across time. Analysts usually provide estimates for the current fiscal year (i.e. FY 2023, ending 12/31/2023 for most companies) and the following year, ending 12/31/2024. Let's call these FY0_EPS and FY1_EPS, for simplicity

You might be tempted to just take a moving average of these two estimates, so that on 1/1/2023 it is 100% of FY0_EPS + 0% of FY1_EPS, on 1/2/2023 it is 99.9% + 0.1% and gradually "move forward in time" as the days pass. That sort of works (and definitely checks the box for a proof-of-concept like in your post) but for the sake of completeness, I'll just say that the right-er approach is to only "move forward in time" when new earnings are released. So it doesn't matter if we're in 1/1/2023 or 2/1/2023--what matter is what is the latest reported quarter. Take Coca-Cola for instance (https://www.bamsec.com/companies/21344/coca-cola-co). Let's roll the tape backward one year. They reported FY 2021 earnings on 2/22/2022, at which point analysts published new estimates in revised models, so on from that day forward until the next quarterly earnings we take 100% FY0_EPS + 0% FY1_EPS, in which these correspond to estimates for FY 2022 and FY 2023, respectively.

On 4/1/2022, Coca-Cola reported Q1 2022 results, analysts published new estimates, and we now take 75% FY0_EPS + 25% FY1_EPS. On 7/1/2022, we move forward another quarter so 50% + 50%, then 25% + 75% starting on 10/26 and then back to square one with 100% + 0% except FY0_EPS now means FY 2023 vs FY 2022 previously, and FY1_EPS means FY 2024

So your table is something like (I'm making up numbers)

    +------------+--------+-----------+---------+---------+-------------+-------------+-------------+
    | Date       | Ticker | Adj_Close | FY0_EPS | FY1_EPS | Period      | NTM0_Weight | NTM1_Weight |
    | 01/01/2022 |     KO |     90.10 |   20.00 |   24.00 | Q1 2022     |        1.00 |        0.00 |  
    | 01/02/2022 |     KO |     91.14 |   20.00 |   24.00 | Q1 2022     |        1.00 |        0.00 |  
    | 01/03/2022 |     KO |     89.30 |   20.00 |   24.00 | Q1 2022     |        1.00 |        0.00 |  
    | 01/04/2022 |     KO |     91.09 |   20.00 |   24.00 | Q1 2022     |        1.00 |        0.00 |  
    | 01/05/2022 |     KO |     92.01 |   20.00 |   24.00 | Q1 2022     |        1.00 |        0.00 |  
    | 01/06/2022 |     KO |     89.05 |   20.00 |   24.00 | Q1 2022     |        1.00 |        0.00 |  
    ...
    | 07/02/2022 |     KO |     89.05 |   19.50 |   23.20 | Q2 2022     |        0.75 |        0.25 |
With that you can take NTM0_Weight and NTM1_Weight to calculate NTM_EPS by multiplying those weights by FY0_EPS and FY1_EPS. And then can take AdjClose / NTM_EPS to calculate P/E

Why is this useful? Because in theory you can take the average P/E of companies X, Y and Z in one industry and compare it to a fourth company W. Is W's P/E multiple above or below the industry average? You now know if they are over or undervalued, respectively, which means you know if you should buy or sell that stock (if you believe you picked the right "comparable" companies in that industry)

This is just one example... there are all sorts of similar analyses done daily in the financial services industry. I'm not saying it's easy to extract alpha from trading on these, but that's the framework

Re: Stochastic gradient descent written in SQL

#124
post #85

Earlier quoted context omitted.

Where is the assignment to a variable? Where can you construct a query using a variable in table/query position? That's the whole point of being first class and composable, a query becomes like any other value so you should be able to parameterize any query by another query assigned to a variable that may have been set inside an if-statement, or accepted as a parameter to a stored procedure. You know, the same kinds…

create table x as (select * from person); select name from x where ...; there you go, just configure your editor to display "create table x" as "declare x = " ;) or even a version with lazy evaluation: create view x as (select * from person); select name from x where ...;

You're still not getting it. First-class status means that anywhere a value or variable can be used, a query or table should also be able to appear, and vice versa. This means a table or query can appear as a return type, a parameter to a stored procedure or query, a variable, and so on.

SQL just does not have this, it instead has 15 different second class ways to handle tables and queries that try to make up for the fact that they are not first-class values. These include CTEs, table valued functions, views, etc.

Re: Stochastic gradient descent written in SQL

#125

Earlier quoted context omitted.

Hehe I was wondering if someone would catch that. Rest assured, I know the difference between online and stochastic gradient descent. I admit I used stochastic on Hacker News because I thought it would generate more engagement.

Then just call it Non-stochastic Gradient Descent? You can't editorialize titles per HN guidelines https://news.ycombinator.com/newsguidelines.html

Thanks, I wasn't aware.

Re: Stochastic gradient descent written in SQL

#126

Earlier quoted context omitted.

Then just call it Non-stochastic Gradient Descent? You can't editorialize titles per HN guidelines https://news.ycombinator.com/newsguidelines.html

Thanks, I wasn't aware.

My pleasure. You can still edit the title, by the way ;-)

Re: Stochastic gradient descent written in SQL

#127
post #100

Earlier quoted context omitted.

What if I wrote a very long, complicated query that I'd like to test against different tables (like test tables), and let's say I can't use stored functions or procedures. How could I pass different tables to my query?

CREATE TABLE data_a AS (SELECT 'a' AS test_case, 1 AS value); CREATE TABLE data_b AS (SELECT 'b' AS test_case, 2 AS value); CREATE VIEW data AS (SELECT * FROM data_a UNION ALL SELECT * FROM data_b); CREATE VIEW complicated_query AS (SELECT test_case, value+1 FROM data); SELECT * FROM complicated_query WHERE test_case = 'a'; SELECT * FROM complicated_query WHERE test_case = 'b';

You just reinvented defunctionalization, which is a transformation from a domain that has first-class values to a domain where support is only second-class. Defunctionalization is typically used in programming languages to simulate first-class functions in languages where functions are only second-class citizens, like C and Pascal.

This perfectly illustrates my point. You had to manually defunctionalize your data model and queries to support what I'm saying should be inherently part of SQL.

Re: Stochastic gradient descent written in SQL

#128

Earlier quoted context omitted.

You cannot abstract over stored procedures either, so that's still not composable.

No idea what this means

Think of first-class functions. Can't do that with stored procedures, just like you can't do that with queries or tables in SQL.

Re: Stochastic gradient descent written in SQL

#129
post #55

Earlier quoted context omitted.

The issue is that many relational databases are not horizontally scalable, so you want to be frugal with their resources.

The resources I'm familiar with are I/O, memory, and CPU. The only one I believe can be spared in the database by using that resource outside of the database, is CPU. When the database is far from saturated on CPU and latency and throughput are determined by I/O and memory, using CPU on some other machine that isn't the database can't possibly have any impact on latency and throughput.

> When the database is far from saturated on CPU

The issue here is if you scale enough saturate the database, you'll have to rewrite essentially all your code if you're a typical CRUD webapp. Basically all of your business logic is about data retrieval. There's probably some companies that can get away with this, but it would be way too expensive for most.

Post reply on HN