Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Parquet, Iceberg and Data Lakehouses

davidgomes.com

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Understanding Parquet, Iceberg and Data Lakehouses

#91

Post by justsomehnguy »

hermitcrab wrote: post #89
pitah1 wrote: post #37

Good in-depth insights into each format. This complements nicely with a site I created called tech-diff ( https://tech-diff.com/file/ ) where it provides a summary of the file formats.

"Comma-Separated Values (CSV) is a text file format that uses commas to separate values in plain text." Except when it uses semi-colons. Or pipes. Or something else.

The most amusing thing is what there are symbols designated as separators

https://en.m.wikipedia.org/wiki/C0_and_C1_control_codes#Fiel...

Re: Understanding Parquet, Iceberg and Data Lakehouses

#92

Post by hermitcrab »

justsomehnguy wrote: post #91
hermitcrab wrote: post #89

Earlier quoted context omitted.

"Comma-Separated Values (CSV) is a text file format that uses commas to separate values in plain text." Except when it uses semi-colons. Or pipes. Or something else.

The most amusing thing is what there are symbols designated as separators https://en.m.wikipedia.org/wiki/C0_and_C1_control_codes#Fiel...

Oh yes. I wrote about that here:

https://successfulsoftware.net/2022/04/30/why-isnt-there-a-d...

And discussed at HN here:

https://news.ycombinator.com/item?id=31220841

Re: Understanding Parquet, Iceberg and Data Lakehouses

#93

Post by broner »

lysecret wrote: post #52

I am currently working with about 100TB data on GCP with BigQuery as a query engine and simple hive partitioning like /key3=000/key2=002/. We are happy because we can run all the queries you want and it is insanely cheap. But latency is reaching quite high levels (it doesn't matter so much for us) but I was wondering, if implementing Iceberg would improve this? Has anyone experience with this? Overall this kind of ar…

This looks pretty neat if you're ok moving to AWS https://www.boilingdata.com/

Re: Understanding Parquet, Iceberg and Data Lakehouses

#95

Post by justsomehnguy »

hermitcrab wrote: post #92
justsomehnguy wrote: post #91

Earlier quoted context omitted.

The most amusing thing is what there are symbols designated as separators https://en.m.wikipedia.org/wiki/C0_and_C1_control_codes#Fiel...

Oh yes. I wrote about that here: https://successfulsoftware.net/2022/04/30/why-isnt-there-a-d... And discussed at HN here: https://news.ycombinator.com/item?id=31220841

Oh! Great article and, if the evil twist of fate would force me to write an export, I hope I would remember and use .usv

But I can't agree on 'default tools' - anything bigger than 3 columns is PITA to look at and I just Ctrl+H to replace the commas with tabs or \r\n (or whatever), depending on the format. At some point 'standart tools' doesn't matter anymore, because it doesn't worth to even use them on such files; think of 'jq'.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#96

Post by hermitcrab »

justsomehnguy wrote: post #95
hermitcrab wrote: post #92

Earlier quoted context omitted.

Oh yes. I wrote about that here: https://successfulsoftware.net/2022/04/30/why-isnt-there-a-d... And discussed at HN here: https://news.ycombinator.com/item?id=31220841

Oh! Great article and, if the evil twist of fate would force me to write an export, I hope I would remember and use .usv But I can't agree on 'default tools' - anything bigger than 3 columns is PITA to look at and I just Ctrl+H to replace the commas with tabs or \r\n (or whatever), depending on the format. At some point 'standart tools' doesn't matter anymore, because it doesn't worth to even use them on such files;…

>I just Ctrl+H to replace the commas with tabs or \r\n (or whatever)

That may not work if the CSV is escaped/quoted (e.g. because the data contains commas).

Or you could use our Easy Data Transform software. It pretty much handles all that horrible CSV stuff for you. ;0)

Re: Understanding Parquet, Iceberg and Data Lakehouses

#97

Post by justsomehnguy »

hermitcrab wrote: post #96
justsomehnguy wrote: post #95

Earlier quoted context omitted.

Oh! Great article and, if the evil twist of fate would force me to write an export, I hope I would remember and use .usv But I can't agree on 'default tools' - anything bigger than 3 columns is PITA to look at and I just Ctrl+H to replace the commas with tabs or \r\n (or whatever), depending on the format. At some point 'standart tools' doesn't matter anymore, because it doesn't worth to even use them on such files;…

>I just Ctrl+H to replace the commas with tabs or \r\n (or whatever) That may not work if the CSV is escaped/quoted (e.g. because the data contains commas). Or you could use our Easy Data Transform software. It pretty much handles all that horrible CSV stuff for you. ;0)

Of course, but most of the time I just need to have a quick glance at the data, or find a single value which I know where it should be.

Slightly on-topic: I found what LibreOffice Calc has a decent input wizard which most of the time works and good enough.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#98

Post by fbdab103 »

jamesblonde wrote: post #12

I disagree with this strongly - "The best way to store Apache Arrow dataframes in files on disk is with Feather. However, it’s also possible to convert to Apache Parquet format and others." The best way to build your own non-JVM lakehouse is to use Iceberg for metadata, Parquet for the Data, Query with DuckDB using Arrow tables (read Parquet directly into Arrow is very low cost), and then use Arrow->Pandas or Polars…

At one point, I thought Feather did not carry any long-term format guarantees. Presumably that has now changed, but I still feel like Parquet is the best future proof option on the table.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#99

Post by pitah1 »

hermitcrab wrote: post #89
pitah1 wrote: post #37

Good in-depth insights into each format. This complements nicely with a site I created called tech-diff ( https://tech-diff.com/file/ ) where it provides a summary of the file formats.

"Comma-Separated Values (CSV) is a text file format that uses commas to separate values in plain text." Except when it uses semi-colons. Or pipes. Or something else.

I was using the following RFC for the formal definition of CSV https://www.rfc-editor.org/rfc/rfc4180.html.

When the delimiter is different, you have a different file format. TSV (tab-separated values) is an example of this.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#100

Post by iamcreasy »

wokwokwok wrote: post #25

It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talkin…

I am familiar traditional JSON to relational table ETL pipelines, but how does ETL pipelines differ in data lake/house when processing data dumps, such as PDF/images?
Post reply on HN