[dead]
I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
21–30 of 79 posts
Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
#22[flagged]
Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
#23Well despite my current anti AI sentiment, I have to admit that after reading the article, It was a good use of AI, done by someone with good technical skills. Still I have the feeling that this only works because of the vast accumulated knowledge pre-AI, and if everybody keeps going in this path, it will end up making everyone not advancing their knowledge at the pace they did before. I feel that this AI immersion i…
I think AI is powertool. Period. If you give it to people who are skill, it will create a mess. I think democratization of intelligence is going to be interesting. You could say the same with same about internet. I think it is part of evolution. May be intelligence or expertise is what does not make us special. May be it is that we are ingenious amd creative with tools and thats how we evolve.
> democratization of intelligence
I'm not trying to be pedantic; I think this is an interesting topic and there's a worthwhile distinction to make here. It isn't really being democratized for a couple reasons (at least).
One, access to information isn't truly knowledge in and of itself. People allowing information from LLMs to pass through their brains are not necessarily retaining any of it, and their ability to synthesize and utilize disparate information from LLMs isn't inherently improved by this technology. So the premise of knowledge isn't very sturdy in my mind.
Two, LLMs function across very broad fields of capability, accuracy, content, and so on, and the best models are not accessible to many people. I find people tend to mean the technology is widely available and accessible when they say 'democratization', but that's not necessarily true nor what that word means to begin with.
True democratization would mean something more like "everyone participates in, shapes, regulates, and grows this technology with their own inputs". I don't think that's what happens at all, and in fact, it has been quite the inversion of that so far.
I mention all of this because I agree that it will be interesting to watch what happens, but I don't agree that it will be for the same reasons. I worry about it specifically because there is not an egalitarian distribution of knowledge, and it is not democratically built or shared.
Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
#24Earlier quoted context omitted.
I think AI is powertool. Period. If you give it to people who are skill, it will create a mess. I think democratization of intelligence is going to be interesting. You could say the same with same about internet. I think it is part of evolution. May be intelligence or expertise is what does not make us special. May be it is that we are ingenious amd creative with tools and thats how we evolve.
> May be it is that we are ingenious amd creative with tools and thats how we evolve. And every time you use the AI to be ingenious or creative, that will be added to the training data. Then someday the AI can be ingenious and creative without you! (It might take a few more breakthroughs. But investors will literally spend trillions chasing those breakthroughs.) The endgame here is to replace all human intelligence a…
> And every time you use the AI to be ingenious or creative, that will be added to the training data
That's part of evolution.
We as humans have always outsmarted the tools.
Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
#25How long did this take?
It took about 2 days to get a proof of concept, and about a week to get something I could ship to production. I skipped a few features for the PoC (like XML tag support, token positions), so most of the delta was adding those back in!
If you didn’t need to look at the code at all, why not write it in asm instead of Rust, and make it even faster?
Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
#26Good read, but "70x" is always misleading.
Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
#27I cannot believe they're sticking to their guns on this website design. It's awful.
Yeah. It locked up my browser. What a pile.
Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
#28[flagged]
This is true and unkind at the same time.
Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
#29Earlier quoted context omitted.
It took about 2 days to get a proof of concept, and about a week to get something I could ship to production. I skipped a few features for the PoC (like XML tag support, token positions), so most of the delta was adding those back in!
If you didn’t need to look at the code at all, why not write it in asm instead of Rust, and make it even faster?
Ha I did consider that! But 70x is plenty fast enough (we still have to query an actual database!) and the parser runs in a shared process on untrusted input, so it wasn't worth the security risk
Re: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
#30This is the type of problem for which LLM generation is great for.
If you have an oracle, and your problem is largely just a pure function, it's pretty good at generating something that both works and is fast.