What is really missing for parquet's wide adoption is support in Excel.
Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
21–30 of 37 posts
Re: Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
#22Very nice that it can show the metadata. If you rather focus on the data itself, a Swiss army knife in the terminal is VisiData [1] . It works with many formats from CSV to Parquet. You'd need to install Pyarrow I think to read Parquet files. VisiData is great to not only peek into the file but filter it, sort, compute simple metrics and even can plot a histogram or scatterplot for ex. I avoided a lot of Jupyter notebooks by using VisiData :)
Re: Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
#23I found a similar tool called nail-parquet[1] which has some nice query functions. I packaged[2] it up for nixpkgs but it’s stuck in merge limbo…
[1] https://github.com/Vitruves/nail-parquet [2] https://github.com/NixOS/nixpkgs/pull/449066
Re: Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
#24Similar tool for JSONL files: I built JSONL Viewer Pro after repeatedly crashing VS Code trying to inspect multi-GB training datasets and IoT device logs with nested objects.
Native Mac/Windows app with multi-threaded parsing (simdjson), automatic nested object flattening, and handles 10M+ rows instantly.
For HN: Use code HN100 for free access
https://iotdatasystems.gumroad.com/
Built with C++ for native performance (~6MB app, not Electron).
Would love feedback from folks working with large JSONL files.
Re: Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
#25Beautiful, I'm currently deep into getting our data into iceberg from firehose and I'm really curious what metadata is written, are bloomfilters being written for the columns i want? Has my compaction and sort jobs helped min-max statistics on those columns?
Will take a look when i get to my laptop!
Re: Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
#26Similar tool for JSONL files: I built JSONL Viewer Pro after repeatedly crashing VS Code trying to inspect multi-GB training datasets and IoT device logs with nested objects. Native Mac/Windows app with multi-threaded parsing (simdjson), automatic nested object flattening, and handles 10M+ rows instantly. For HN: Use code HN100 for free access https://iotdatasystems.gumroad.com/ Built with C++ for native performance…
Super quick feedback - opening that link on my phone shows me two options next to each other, seemingly with the same name / description (followed by …) and same pricetag. I had to turn my phone sideways to see that there is a windows and a Mac version.
I think you can afford the extra characters to show the whole page in portrait mode. (iPhone 16 pro Safari)
Re: Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
#27This looks very handy, thank you for working on this and making it open source.
I did submit a feature request for vi keybindings; though I could look into contributing this myself if I find a bit of spare time.
The other thing that surprised me was the size of the binaries: 90MB for a TUI tool (x64 Linux)? I wonder what the bulk of that is? Is there an issue with LTO? An other commenter noticed as well.
It also looks like you are building against a relatively recent glibc (2.34), which limits compatibility with older systems. Building against an older glibc can be hard to do, so I am not faulting you here, and you do provide a musl fallback, which is appreciated (mandatory notice that the musl allocator can dramatically degrade the performance of rust programs, just in case you were not aware of this).
A few more ideas for improvements (you probably already have your own laundry list):
- Mouse support?
- Seeing that you do have graphs, it would be fun to see a scatter plot as well as a distribution plot under statistics in the "Row Groups" tab (though you probably pull these from the metadata, so that would require further processing, which may be out of scope).
Re: Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
#28Similar tool for JSONL files: I built JSONL Viewer Pro after repeatedly crashing VS Code trying to inspect multi-GB training datasets and IoT device logs with nested objects. Native Mac/Windows app with multi-threaded parsing (simdjson), automatic nested object flattening, and handles 10M+ rows instantly. For HN: Use code HN100 for free access https://iotdatasystems.gumroad.com/ Built with C++ for native performance…
Quick update: Mac ZIP had a corruption issue that's now fixed.
Anyone who downloaded in the last few hours - please re-download!
Also just added a Data Plot feature for visualizing numeric columns.
Thanks to everyone who reported the issue!
Re: Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
#29Similar tool for JSONL files: I built JSONL Viewer Pro after repeatedly crashing VS Code trying to inspect multi-GB training datasets and IoT device logs with nested objects. Native Mac/Windows app with multi-threaded parsing (simdjson), automatic nested object flattening, and handles 10M+ rows instantly. For HN: Use code HN100 for free access https://iotdatasystems.gumroad.com/ Built with C++ for native performance…
Super quick feedback - opening that link on my phone shows me two options next to each other, seemingly with the same name / description (followed by …) and same pricetag. I had to turn my phone sideways to see that there is a windows and a Mac version. I think you can afford the extra characters to show the whole page in portrait mode. (iPhone 16 pro Safari) https://imgur.com/a/aTxO3sp
I will change the description. Thank you!
Re: Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files
#30what was wrong with using a python repl with pyarrow/polars/duckdb for this?