Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

151–160 of 165 posts

Re: DuckDB Internals Part 1

#151

Post by fartfeatures »

otterley wrote: post #133
fg137 wrote: post #106

Earlier quoted context omitted.

Sorry, I don't think your non-software analogy that existed before 2025 helps at all in this specific topic about software consultancy.

Meanwhile, people are in fact making actual money solving problems for others by building use-case-specific solutions. Perhaps you’re unaware of them, but they do exist. They prefer to operate off the radar.

This is something that can't escape people making such statements but never seems to be addressed. Perhaps it is unaddressable as the original statement is grounded in emotion not reason?

Re: DuckDB Internals Part 1

#152

Post by pknerd »

otterley wrote: post #133
fg137 wrote: post #106

Earlier quoted context omitted.

Sorry, I don't think your non-software analogy that existed before 2025 helps at all in this specific topic about software consultancy.

Meanwhile, people are in fact making actual money solving problems for others by building use-case-specific solutions. Perhaps you’re unaware of them, but they do exist. They prefer to operate off the radar.

> by solving problems for others by building use-case-specific solutions.

You mean come up with a solution that fits for a usecase and write a blog or make a video of it?

Re: DuckDB Internals Part 1

#153

Post by My_Name »

One of my projects was using a RRD database to store 10-second samples from 20 data streams with a RRD time of 1 year, which works out to around 82 million entries. I display this interactively on a raspberry pi and I used to use RRDTool in python, but switching to my own code leveraging mySQL (and using some judicious rollup tables) means that instead of a larger time span taking ~5 seconds to update, it is now fast enough to give me 20fps no matter the viewed range.

Not sure DuckDB will provide any benefit to my specific case, but I'll check it out.

Re: DuckDB Internals Part 1

#154

Post by pknerd »

otterley wrote: post #133
fg137 wrote: post #106

Earlier quoted context omitted.

Sorry, I don't think your non-software analogy that existed before 2025 helps at all in this specific topic about software consultancy.

Meanwhile, people are in fact making actual money solving problems for others by building use-case-specific solutions. Perhaps you’re unaware of them, but they do exist. They prefer to operate off the radar.

> They prefer to operate off the radar.

If they are off the radar then how do people approach them?

Re: DuckDB Internals Part 1

#155

Post by pknerd »

lazarus01 wrote: post #92
gortok wrote: post #79

Earlier quoted context omitted.

> Like businesses don't care about the tool/tech itself, how do I find and approach them, and for which niche. You probably don't realize this, but you're asking one of the hardest questions when starting a business, and one of the questions others are least likely to be able to answer for you. "finding" a niche, and connecting to the business folks inside that 'niche' is hard, and is inherently a personal journey. T…

I use to be in sales before I became an MLE. There is a theory called diffusion of innovation. The simple explanation is that there are 5 different cohorts of buyers. Early adopters, visionaries, pragmatists, conservatives and laggards. Early adopters and visionaries are risk takers, who will make bold moves to achieve order of magnitude results. This is called the early market, which represents 13% of the market. Th…

Thanks for the response, being a someone not live in the West, the only way I can reach out is...Interne

Re: DuckDB Internals Part 1

#156

Post by xnx »

tomjakubowski wrote: post #107
skeeter2020 wrote: post #103

Earlier quoted context omitted.

>> DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop I agree, and the dirty (not so) secret big data providers like Snowflake try to hide: the majority of your work is not big data and WILL fit on your local machine. My last company was spending $2M/yr on contract with Snowflake, and another million between Fivetran and Matillion. Of the 1200 clients usi…

Folks have been beating this drum for as long as I've worked in software, dating to the Hadoop era, and it remains true today. So much of "big data" only appears big because it's poorly stored, or is represented wastefully (in persistent storage or in memory).

SSD was a big turning point. Before that, there was more case for a database server.

Re: DuckDB Internals Part 1

#157

Post by lazarus01 »

aabdi wrote: post #111
lazarus01 wrote: post #92

Earlier quoted context omitted.

I use to be in sales before I became an MLE. There is a theory called diffusion of innovation. The simple explanation is that there are 5 different cohorts of buyers. Early adopters, visionaries, pragmatists, conservatives and laggards. Early adopters and visionaries are risk takers, who will make bold moves to achieve order of magnitude results. This is called the early market, which represents 13% of the market. Th…

Probably should tell guy to read the crossing the chasm book? Seems useful in this context. Define the smallest market possible or something like that. I’m not sales though.

Agreed. That book is the bible in applying innovation diffusion theory to your marketing and product strategy.

Re: DuckDB Internals Part 1

#158

Post by lanstin »

nxm wrote: post #150
skeeter2020 wrote: post #103

Earlier quoted context omitted.

>> DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop I agree, and the dirty (not so) secret big data providers like Snowflake try to hide: the majority of your work is not big data and WILL fit on your local machine. My last company was spending $2M/yr on contract with Snowflake, and another million between Fivetran and Matillion. Of the 1200 clients usi…

A good portion of users querying Snowflake at large companies are not technical so you can’t expect them to run DuckDB, not to mention data access controls

Greybeam (the company writing the blog) offers a service to proxy Snowflake and route queries in the fly to DuckDB or Snowflake based on predicted size. Saves a lot of money.

Re: DuckDB Internals Part 1

#159

Post by fartfeatures »

pknerd wrote: post #154
otterley wrote: post #133

Earlier quoted context omitted.

Meanwhile, people are in fact making actual money solving problems for others by building use-case-specific solutions. Perhaps you’re unaware of them, but they do exist. They prefer to operate off the radar.

> They prefer to operate off the radar. If they are off the radar then how do people approach them?

EO/IR

Re: DuckDB Internals Part 1

#160

Post by fastasucan »

0xferruccio wrote: post #17

DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…

>DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop

It also works great for data that doesn't fit on my laptop.

Post reply on HN