Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

apps.microsoft.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#11

Post by paddy_m »

Do you have a demo video?

What are you using for processing (polars)?

Marketing note: I'm sure you're proud of P Core/V Core, but that doesn't matter to your users, it's an implementation detail. At a maximum I'd write "intelligent execution that scales from small files to large files".

As an implementation note, I would make it simple to operate on just the first 1000 (10k or 100k) rows so responses are super quick, then once the users are happy about the transform, make it a single click to operate on the entire file with a time estimate.

Another feature I'd like in this vein is execute on a small subset, then if you find an error with a larger subset, try to reduce the larger subset to a small quick to reproduce version. Especially for deduping.

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#12

Post by TheTaytay »

Thank you for this. I find myself increasingly using CSVs (TSVs actually) as the data format of choice. I confess I wish this was written for Mac too, but I like the trend of (once again) moving data processing down to our super computers on our desk...

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#13

Post by morganherlocker »

geodel wrote: post #4
esafak wrote: post #2

[flagged]

I don't know if for Desktop App (most likely electron based) people expect any better.

As a point of comparison, I just downloaded the Windows binary for duckdb (which provides a nice TUI for similar tasks) and it was 9.84MB. People can and should expect better.

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#14

Post by hilti »

TheTaytay wrote: post #12

Thank you for this. I find myself increasingly using CSVs (TSVs actually) as the data format of choice. I confess I wish this was written for Mac too, but I like the trend of (once again) moving data processing down to our super computers on our desk...

… I‘m trying to use our super computers in our pockets, like an iPhone ;-) But still struggling with the way how to present CSV data effectively on a small screen, although it‘s huge in terms of pixels compared to computer screens from the 90s

It‘s interesting to research how capable applications like Lotus123 have been even on low resolutions like 800x600 pixel compared to today’s standard

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#15

Post by RyanHamilton »

TheTaytay wrote: post #12

Thank you for this. I find myself increasingly using CSVs (TSVs actually) as the data format of choice. I confess I wish this was written for Mac too, but I like the trend of (once again) moving data processing down to our super computers on our desk...

QStudio allows querying CSV on mac via DuckDB: https://www.timestored.com/qstudio/csv-file-viewer I've been improving the Mac version a lot lately, key bindings, icon, an App package to download. So if you find any problems please raise a github issue.

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#16

Post by [deleted] »

paddy_m wrote: post #11

Do you have a demo video? What are you using for processing (polars)? Marketing note: I'm sure you're proud of P Core/V Core, but that doesn't matter to your users, it's an implementation detail. At a maximum I'd write "intelligent execution that scales from small files to large files". As an implementation note, I would make it simple to operate on just the first 1000 (10k or 100k) rows so responses are super quick,…

[deleted]

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#17

Post by hermitcrab »

TheTaytay wrote: post #12

Thank you for this. I find myself increasingly using CSVs (TSVs actually) as the data format of choice. I confess I wish this was written for Mac too, but I like the trend of (once again) moving data processing down to our super computers on our desk...

If you are wrangling CSV/TSV files on Mac, it might be worth taking a look at Easy Data Transform.

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#18

Post by paddy_m »

TheTaytay wrote: post #12

Thank you for this. I find myself increasingly using CSVs (TSVs actually) as the data format of choice. I confess I wish this was written for Mac too, but I like the trend of (once again) moving data processing down to our super computers on our desk...

Ok, if we are all tagging and promoting our own projects, check out mine.

I created Buckaroo to provide a better table viewing experience inside of notebooks. I also built a low code UI and auto cleaning to expedite the wrote data cleaning tasks that take up a large portion of data analysis. Autocleaning is heuristically powered - no LLMs, so it's fast and your data stays local. You can apply different autocleaning strategies and visually inspect the results. When you are happy with the cleaning, you can copy and paste the python code as a reusable function.

All of this is open source, and its extendable/customizable.

Here's a video walking through autocleaning and how to extend it https://youtu.be/A-GKVsqTLMI

Here's the repo: https://github.com/paddymul/buckaroo

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#19

Post by marcellus23 »

paddy_m wrote: post #11

Do you have a demo video? What are you using for processing (polars)? Marketing note: I'm sure you're proud of P Core/V Core, but that doesn't matter to your users, it's an implementation detail. At a maximum I'd write "intelligent execution that scales from small files to large files". As an implementation note, I would make it simple to operate on just the first 1000 (10k or 100k) rows so responses are super quick,…

> Marketing note: I'm sure you're proud of P Core/V Core, but that doesn't matter to your users, it's an implementation detail. At a maximum I'd write "intelligent execution that scales from small files to large files".

Speaking personally, "intelligent execution that scales from small files to large files" sounds like marketing buzz that could mean absolutely nothing. I like that it mentions specifically switching between RAM and disk-powered engines, because that suggests it's not just marketing speak, but was actually engineered. Maybe P vs V Core is not the best way to market it, but I think it's worth mentioning that design.

Re: Show HN: CSV GB+ by Data.olllo – Open and Process CSVs Locally

#20

Post by olllo »

TheTaytay wrote: post #12

Thank you for this. I find myself increasingly using CSVs (TSVs actually) as the data format of choice. I confess I wish this was written for Mac too, but I like the trend of (once again) moving data processing down to our super computers on our desk...

Thank you! I completely agree—TSVs/CSVs are such a simple yet powerful format, and it's great to hear you're making good use of them. I'm also a big fan of doing as much as possible locally—our machines are incredibly capable these days. Good news: I'm currently working on the macOS version of Data.olllo and plan to submit it to the Mac App Store soon. Stay tuned!
Post reply on HN