21–30 of 60 posts
Earlier quoted context omitted.
Yes, SQLite is row oriented. It's not a very space efficient format because it also doesn't support compression or compact representations of numbers in binary. But it doesn't rely on the JVM and a typical JVM ecosystem. It is a big benefit for some use-cases, like dealing with numerical data on the edge.
If you are searching for the analytics equivalent you should check out duckdb.
Earlier quoted context omitted.
If you are searching for the analytics equivalent you should check out duckdb.
DuckDB is about querying existing files or am I missing something?
Earlier quoted context omitted.
I agree that sqlite has a number of similar benefits - openness, table abstractions and concurrent transactions. It’s also a library so close to how delta, iceberg and hudi are implemented. I’m glad it’s had an uptick in interest recently but I haven’t yet seen it mentioned for analytics yet. I assume it’s row rather than column oriented?
Yes, SQLite is row oriented. It's not a very space efficient format because it also doesn't support compression or compact representations of numbers in binary. But it doesn't rely on the JVM and a typical JVM ecosystem. It is a big benefit for some use-cases, like dealing with numerical data on the edge.
I don't see any mention of sqlite. Is a sqlite file not the same thing they're talking about here? Pretty sure it has a spec and hasn't changed formats in many years so if you wanted to read it out in something that isn't sqlite, I imagine it wouldn't be too hard.
Earlier quoted context omitted.
That comparison blog seems biased toward Hudi.
Biased in what way? The authors provide solid arguments for why they think Hudi is a good tool.
Earlier quoted context omitted.
Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions. Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vend…
> Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. This is all very interesting, and thank you for taking the time to explain. Any good starting points for someone who would like to know more?
I've been dabbling with implementing a database, and adopting a format would be much easier.
I don't see any mention of sqlite. Is a sqlite file not the same thing they're talking about here? Pretty sure it has a spec and hasn't changed formats in many years so if you wanted to read it out in something that isn't sqlite, I imagine it wouldn't be too hard.
The main disadvantage of sqlite compared to these other formats is that sqlite is designed for single machine processing. This can be problematic for huge datasets or complex queries. These other formats easily support using clusters to process your data.
Is there any such open source table format for row based data? I've been dabbling with implementing a database, and adopting a format would be much easier.