Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

71–80 of 165 posts

Re: DuckDB Internals Part 1

#71

Post by ashu1461 »

0xferruccio wrote: post #17

DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…

Can you please expand more on the claude analysis part. What exactly you analysed and what outcome it helped with ?

Re: DuckDB Internals Part 1

#72

Post by fg137 »

faangguyindia wrote: post #40
pknerd wrote: post #29

Earlier quoted context omitted.

Just curious whether one can earn money making these exts?

You can definately offer consultation or custom integration.

That's going to be a difficult business in this age unless you have some uniquely strong ideas and products.

Re: DuckDB Internals Part 1

#73

Post by hilariously »

jkubicek wrote: post #4
steve_adams_86 wrote: post #2

> DuckDB has received widespread adoption because it's just so damn easy to use. This was a major factor in my initial adoption. Since then it has stuck because it’s also absurdly capable, versatile, and fast. If it wasn’t so easy to use I suspect I wouldn’t have adopted it when I did. The ergonomics are crazy. It still impresses me regularly.

What do you use it for? I’m perpetually interested in using DuckDB, but it doesn’t seem to do anything I need.

Honestly as someone whose super SQL focused and spends less time focusing on python I just can write generic SQL to transform things in memory to do whatever I want, its very helpful for that.

Re: DuckDB Internals Part 1

#74

Post by hilariously »

levanten wrote: post #63
mcv wrote: post #59

Is everything becoming columnar? Parquet stores data per column instead of per row because it improves compression. I get that. Arrow apparently is columnar, and now DuckDB also gets its efficiency by treating data as columns instead of rows? I still need to wrap my head around how that works, but it's a fascinating development.

It depends on your task. In analytics where you need to scan lots of data points within few columns, then columnar storage is very much the best. But for transactional workloads where you have to deal with specific entities, row based would be more advantageous. There are hybrid systems that try to be both at the same time but in my experience they end not doing either very well.

Often used to be referred to as HTAP, and yeah in most data engineering its moving things from OLTP to OLAP forms, and OLAP pretty much always benefit from columnar compression for aggregations and rollups.

Re: DuckDB Internals Part 1

#75

Post by hilariously »

conradkay wrote: post #52
snissn wrote: post #32

I'm just curious - is duckdb too slow for people? This benchmark from clickhouse shows it being fairly slow compared to some options: https://jsonbench.com/

That's for their `JSON` data types. In DuckDB it's just a string meaning lots of queries will have to do JSON parsing on every row, but the inserts are very fast. Definitely a bit of a footgun and when you actually just need STRUCT or MAP. There's a talk about ClickHouse's approach from its creator: https://www.youtube.com/watch?v=xHj9mysh0GI , but the gist is that it maintains (sub)columns to store different paths i…

Not just that, you can SELECT * FROM read_json('folder/*') and read an entire schematically consistent folder of json files, or parquet files, or basically anything it supports with its various functions. Duckdb is insanely useful.

Re: DuckDB Internals Part 1

#76

Post by wiredfool »

jkubicek wrote: post #4
steve_adams_86 wrote: post #2

> DuckDB has received widespread adoption because it's just so damn easy to use. This was a major factor in my initial adoption. Since then it has stuck because it’s also absurdly capable, versatile, and fast. If it wasn’t so easy to use I suspect I wouldn’t have adopted it when I did. The ergonomics are crazy. It still impresses me regularly.

What do you use it for? I’m perpetually interested in using DuckDB, but it doesn’t seem to do anything I need.

Few different use cases, other than just a general swiss army knife for vaguely tabular data.

* fastapi + duckdb + parquet for the backend for a relatively high profile website

* wasm duckdb + react for a few visualization websites

* yaml driven ETL from lots of sources, principally ugly spreadsheets, into usable data. More T than E or L really

Re: DuckDB Internals Part 1

#77

Post by fartfeatures »

fg137 wrote: post #72
faangguyindia wrote: post #40

Earlier quoted context omitted.

You can definately offer consultation or custom integration.

That's going to be a difficult business in this age unless you have some uniquely strong ideas and products.

I'm not convinced that is true. The JCB digger didn't put groundworkers out of business. A consultant that can get more done in a day is worth a lot more than one who can't. There is still skill required in wielding the tool of the day and that skill is marketable.

Re: DuckDB Internals Part 1

#79

Post by gortok »

pknerd wrote: post #68
faangguyindia wrote: post #40

Earlier quoted context omitted.

You can definately offer consultation or custom integration.

Thanks for your kind response. Could you guide further? Like businesses don't care about the tool/tech itself, how do I find and approach them, and for which niche. Thanks in Advance

> Like businesses don't care about the tool/tech itself, how do I find and approach them, and for which niche.

You probably don't realize this, but you're asking one of the hardest questions when starting a business, and one of the questions others are least likely to be able to answer for you.

"finding" a niche, and connecting to the business folks inside that 'niche' is hard, and is inherently a personal journey.

There's an old writing adage, "Write about what you know", and the same adage works in business: Do business with what you know.

Your question goes into another issue that you have to resolve when building a business: going into a platform specialization necessarily means folks know about that platform or they know they need you to solve a problem they have with that platform.

In general, there are two ways out of each problem:

1. Build an ecosystem with DuckDB at its center that solves a business problem that a particular niche cares about. 2. Build a reputation solving problems with DuckDB that would attract those that know they have a problem with DuckDB.

Honestly, best of luck here, becoming successful at business is hard if you're not already in tune with why folks buy and ensuring you're selling something they want to buy from you.

Post reply on HN