Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code

posthog.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code

#31

Post by -warren »

lovasoa wrote: post #10

The thing I would have liked to know is why they don't use an existing fast SQL parser. Was being slightly incompatible with all existing SQL dialects a product requirement?

I think thats exactly what indirectly happened. This guy didnt optimize the parser. Someone else did -- years ago. That work was pulled into the LLM and made it look like magic.

Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code

#35

Post by robbie-c »

sscaryterry wrote: post #26

Good read, but "70x" is always misleading.

In what way? This was a geometric mean of the improvements from a small test corpus. In production, where it only parses longer SQL that didn't hit the parser cache, the mean parse time went down by 454x, across millions of parses.

Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code

#36

Post by veidelis »

choilive wrote: post #20
bitlad wrote: post #14

Earlier quoted context omitted.

I think AI is powertool. Period. If you give it to people who are skill, it will create a mess. I think democratization of intelligence is going to be interesting. You could say the same with same about internet. I think it is part of evolution. May be intelligence or expertise is what does not make us special. May be it is that we are ingenious amd creative with tools and thats how we evolve.

There are some studies that suggest human brain sizes have been shrinking over the last 20,000 years. The theory is that as civilization developed the demand for individual humans to be independently intelligent has weakened because we developed a "collective brain" and also self-domesticated to be more cooperative.

Honestly there might be truth to it, I don't get the downvotes - why?

Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code

#38

Post by jakewins »

I’ve had very good success in similar setups where you have some sort of “oracle” and can generate enormous corpuses of test data, such that you really, really trust the LLM code must work for the inputs you expect it’ll ever need to handle.

Makes me think of all the algorithms we specify in proof languages and then hand-implement in production languages - this setup could maybe let you just specify the proof of an algorithm and then let LLMs derive efficient implementations with the (slow) proof as an oracle

Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code

#40

Post by duendefm »

Daishiman wrote: post #7
duendefm wrote: post #5

Well despite my current anti AI sentiment, I have to admit that after reading the article, It was a good use of AI, done by someone with good technical skills. Still I have the feeling that this only works because of the vast accumulated knowledge pre-AI, and if everybody keeps going in this path, it will end up making everyone not advancing their knowledge at the pace they did before. I feel that this AI immersion i…

> till I have the feeling that this only works because of the vast accumulated knowledge pre-AI I'm not about to say that there's nothing new under the sun, but parsers are a really well-understood problem where 99.9% of people don't need frontier knowledge and wouldn't be in a position to use it anyway. And I don't think that people doing research on parsers would ever rely on LLMs for precisely that reason. But we'…

My point is, we have programming languages like C and C++, we have operating systems like Linux and FreeBSD, we have an empire of software and knowledge accumulated because of the intellectual battles fought by people before AI. With AI, we all are getting our coding easier (and are kind of being forced to), in a way that we will skip these kind of battles. That is, if we all use AI to make our job easier it will have some short term gain but we will end up as a whole ceasing to advance human knowledge with new stuff that has to come from real intellectual work. Like, I don't see people coming up with new outstanding technology if we all sucumb to be AI dependent.
Post reply on HN