Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

github.com

Post reply on HN
Print view View original

91–100 of 141 posts

Re: Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

#91

Post by thamer »

Some (most?) tools that output data in columns and fit each one to the largest value in that column need to scan the whole file as a first pass just to start displaying data.

Not only is it the case with this tool, but from what I'm reading in main.rs it looks like it's also loading the whole file in memory. I was going to say that scanning the file was a deal-breaker, but if true this is much more resource-intensive.

This looks like a nice tool, but these design choices seem to limit its use to relatively small files. It could be updated to have a read-ahead buffer instead and adjust its output as new lines are discovered with values of different width, although doing this without a jarring resize could be challenging.

Could someone with better knowledge of Rust than mine confirm this?

I see the full dataset being loaded here[1] and the column widths being computed here.[2]

[1] https://github.com/alexhallam/tv/blob/main/src/main.rs#L183-...

[2] https://github.com/alexhallam/tv/blob/main/src/main.rs#L218-...

Re: Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

#92

Post by eevilspock »

flusteredBias wrote: post #49
contravariant wrote: post #37

Earlier quoted context omitted.

The NA detection and higlighting is nice but I'm not sure how I feel about showing anything other than the exact textual value. I don't mind abridging quotes when they're not necessary, but showing "N/A", NA,, etc. as the same value is a bit iffy.

It is rough. There are many ways that different tools put NAs, na, N/A, "", etc. in a file. To chose only "NA" would mean I would be excluding the output of other tools. I chose accessibility over specificity. #trade-offs.

Simple: provide CLI switches to let the user decide what they want for NA detection (current behavior as default, user can provide alternate NA values, per the source file or the natural language it is expressed in), and how they want them displayed, whether as-is, blank or a consistent custom value (as-is should be the default).

Re: Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

#94

Post by eevilspock »

thamer wrote: post #91

Some (most?) tools that output data in columns and fit each one to the largest value in that column need to scan the whole file as a first pass just to start displaying data. Not only is it the case with this tool, but from what I'm reading in main.rs it looks like it's also loading the whole file in memory. I was going to say that scanning the file was a deal-breaker, but if true this is much more resource-intensive…

I like this idea. I don't think it would be jarring if the read-ahead buffer was a minimal number of lines, i.e. looking like distinct pages. The default could be at least the line height of the terminal, or some multiple.

There could be an option to redisplay the header row for resized "pages".

There could be a CLI switch giving the user control, i.e. make everyone happy.

Re: Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

#95

Post by killjoywashere »

eevilspock wrote: post #94
thamer wrote: post #91

Some (most?) tools that output data in columns and fit each one to the largest value in that column need to scan the whole file as a first pass just to start displaying data. Not only is it the case with this tool, but from what I'm reading in main.rs it looks like it's also loading the whole file in memory. I was going to say that scanning the file was a deal-breaker, but if true this is much more resource-intensive…

I like this idea. I don't think it would be jarring if the read-ahead buffer was a minimal number of lines, i.e. looking like distinct pages. The default could be at least the line height of the terminal, or some multiple. There could be an option to redisplay the header row for resized "pages". There could be a CLI switch giving the user control, i.e. make everyone happy.

I think data scientists will recognize this problem, and there's a well-used solution: .head()

Just show me the top 5 rows. That's all most people are looking for.

cat data/a.csv | tv --head

Re: Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

#96

Post by fiddlerwoaroof »

killjoywashere wrote: post #95
eevilspock wrote: post #94

Earlier quoted context omitted.

I like this idea. I don't think it would be jarring if the read-ahead buffer was a minimal number of lines, i.e. looking like distinct pages. The default could be at least the line height of the terminal, or some multiple. There could be an option to redisplay the header row for resized "pages". There could be a CLI switch giving the user control, i.e. make everyone happy.

I think data scientists will recognize this problem, and there's a well-used solution: .head() Just show me the top 5 rows. That's all most people are looking for. cat data/a.csv | tv --head

Or:

    head -n5 data/a.csv | tv

Re: Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

#97

Post by a1369209993 »

psadauskas wrote: post #59
contravariant wrote: post #37

Earlier quoted context omitted.

The NA detection and higlighting is nice but I'm not sure how I feel about showing anything other than the exact textual value. I don't mind abridging quotes when they're not necessary, but showing "N/A", NA,, etc. as the same value is a bit iffy.

When presented with a similar problem, I tend to use non-ascii characters. For example, in my `~/.psqlrc` I have: \pset null ␀ Looks like this in output: 40 | ␀ | 2021-09-23 20:42:32.536571 | ␀ 41 | 15 | 2021-09-23 20:42:33.177474 | ␀ 42 | 19 | 2021-09-23 20:42:33.212133 | ␀ 43 | ␀ | 2021-09-23 20:42:33.247346 | ␀

I was going compain about null bytes in text (never, period), but then realized you actually did mean the U+2400 SYMBOL FOR NULL[0] character itself. That's surprisingly viable (though you do now have to worry about the string "\xE2\x90\x80" ending up in your data).

0: Which is actually incorrectly named - it should be "SYMBOL FOR NUL".

Re: Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

#98

Post by eli »

fiddlerwoaroof wrote: post #96
killjoywashere wrote: post #95

Earlier quoted context omitted.

I think data scientists will recognize this problem, and there's a well-used solution: .head() Just show me the top 5 rows. That's all most people are looking for. cat data/a.csv | tv --head

Or: head -n5 data/a.csv | tv

Unless your csv has embedded line breaks

Re: Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

#99

Post by franga2000 »

killjoywashere wrote: post #95
eevilspock wrote: post #94

Earlier quoted context omitted.

I like this idea. I don't think it would be jarring if the read-ahead buffer was a minimal number of lines, i.e. looking like distinct pages. The default could be at least the line height of the terminal, or some multiple. There could be an option to redisplay the header row for resized "pages". There could be a CLI switch giving the user control, i.e. make everyone happy.

I think data scientists will recognize this problem, and there's a well-used solution: .head() Just show me the top 5 rows. That's all most people are looking for. cat data/a.csv | tv --head

> Just show me the top 5 rows. That's all most people are looking for.

Is it? I'd wager that can't be more than half its use at most. Accessing a specific section that could be at any section of the file is very common in my experience, as is truly random access. Both of these, as well as the first few rows use case, are far better served by a page system.

Re: Show HN: Tidy Viewer – a cross-platform CSV pretty printer for viewer enjoyment

#100

Post by baggiponte »

flusteredBias wrote: post #45
ComputerGuru wrote: post #40

XSV [0] can also pretty-print (minus the colors), but that's just the tip of the iceberg as far as what it can do. It's very handle for quick statistical analysis of CSV input. [0]: https://github.com/BurntSushi/xsv

I love xsv! I mention in the readme that command line data manipulation tools are great compliments to tv. https://github.com/alexhallam/tv#tools-to-pair-with-tv

that's exactly the comment I was looking for! xsv is super powerful and I think you might both draw inspiration from one another. I read above that tv reads everything into memory: maybe you can exploit some xsv tricks to avoid that. I feel tv looks great to visualise the outcome at the end of a pipeline, perhaps with xsv. I am no Ruby expert either, but this can become a cool Homebrew binary: people on macOS will use it too!
Post reply on HN