Very interesting, 2012 marks an inflection point, a change of the regime. I noticed that at that time the discourse shifted from the founder's concerns to that of the employees and became less interesting for me.
Hacker News Activity Analysis with GPT-4 Agent
41–50 of 54 posts
Re: Hacker News Activity Analysis with GPT-4 Agent
#42Earlier quoted context omitted.
We will probably enable the Hackernews data as demo data in the future. Today we only have some e commerce demo data when you sign up. Although you could connect the public Big query dataset and do it yourself. Your 2nd question would require some preprocessing per message that should probably be done as part of data preparation and not at query time.
OK, I'm a bit confused then - if this needs prep work to ask questions, what is AI bringing to the table?
After looking at the HN demo, it seems like AI is for streamlining the sql query -> chart building process.
A service to preprocess the data with custom prompts would be neat.
Re: Hacker News Activity Analysis with GPT-4 Agent
#43Earlier quoted context omitted.
OK, I'm a bit confused then - if this needs prep work to ask questions, what is AI bringing to the table?
After looking at the HN demo, it seems like AI is for streamlining the sql query -> chart building process. A service to preprocess the data with custom prompts would be neat.
> A service to preprocess the data with custom prompts would be neat.
It is pretty cool to mess about with it. I posted it last week, but didn't get any nibbles: https://github.com/MittaAI/mitta-community/tree/main/cookboo...
Re: Hacker News Activity Analysis with GPT-4 Agent
#44Love how the demo falls pray to what I dont have a term for, "the SQLers assumption"? It asks ChatGPT to write SQL to get sales data, ChatGPT (or most SQLers) trust that every year-month combo has atleast one entry - which means the graphs its presenting could be wrong. Because if there was no entries for a year-month it it will skip that year-month and make it look like you never had a 0 month. I've made this mistak…
My personal hope is that time_bucket_gapfill(), interpolate(), and others will prove so popular in Timescale that DBMSes adopt them and they become part of the SQL standard. When I have to use a "naked" system without them to do analysis and reporting (read: making dashboards), I wind up creating similar UDFs.
Re: Hacker News Activity Analysis with GPT-4 Agent
#45Has anyone seen a project such as this which is open source? I am not saying this project should be, it’s just that my pet project is something very similar and I am sure some people must be building this in the open?
The biggest project I am aware of is the SQL agent from langchain. It definitely gets you started and is great imo for single developers or small technical teams.
We played with Langchain before deciding on your solution. I think the biggest value add from Dot here, aside from the prompt engineering, is the infra around spinning up the meta tables that describe columns. I think you could get good results with Langchain if you precomputed them like Dot does. Having metadata describing the content (which Langchain tries to do on the fly by default by querying the top 5 or so rows, which is slower and leads to… mixed results) turns the tool from basically useless to pretty good instantly.
Re: Hacker News Activity Analysis with GPT-4 Agent
#46Earlier quoted context omitted.
OK, I'm a bit confused then - if this needs prep work to ask questions, what is AI bringing to the table?
After looking at the HN demo, it seems like AI is for streamlining the sql query -> chart building process. A service to preprocess the data with custom prompts would be neat.
That's exactly what Dot is about.
Preprocessing goes into data transformation territory and given the millions of comments on HN it's also getting expensive quite fast.
From BigQuery and Snowflake I know they have remote/external function that you can use to just plug in the OpenAI API.
Re: Hacker News Activity Analysis with GPT-4 Agent
#47I’m guessing the bot has access to the schema of the data and then builds sql queries to fetch subsets into python for plotting. Is that right? You could potentially stage the query in two parts — one in which it builds the query that you execute , and the 2nd in which you provide data for it to analyse/visualise.
That's along the lines of what Dot is doing. For the full details you can expand "Explanation > Full Logs".
Re: Hacker News Activity Analysis with GPT-4 Agent
#48Very interesting, 2012 marks an inflection point, a change of the regime. I noticed that at that time the discourse shifted from the founder's concerns to that of the employees and became less interesting for me.
It is interesting to see that exponential growth stopped and linear growth set in. I wasn't around at that time, so I'd be curious if it's the reason you give.
For me HN is the highest signal place on the internet for founder and tech content.
Re: Hacker News Activity Analysis with GPT-4 Agent
#49Earlier quoted context omitted.
The biggest project I am aware of is the SQL agent from langchain. It definitely gets you started and is great imo for single developers or small technical teams.
We played with Langchain before deciding on your solution. I think the biggest value add from Dot here, aside from the prompt engineering, is the infra around spinning up the meta tables that describe columns. I think you could get good results with Langchain if you precomputed them like Dot does. Having metadata describing the content (which Langchain tries to do on the fly by default by querying the top 5 or so row…
Thank you for sharing that.
Building the training space for Dot is definitely an important part of the value prop.
Re: Hacker News Activity Analysis with GPT-4 Agent
#50Interesting Stuff. OpenBB has also implemented an LLM/AI-based solution using GPT to query stock/trading data in QnA format. I want to do something similar with an e-commerce website using RDBMS(MySQL/pgSQL). Does anyone know any such solution? Like, if I am running a t-shirt store, my users can query like: "Do you have a round neck t-shirt in red color in XL size" and it returns all relevant results
In general that sounds more like a classical search engine to me.
Here I would check out Algolia (https://www.algolia.com)