Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qsv: Efficient CSV CLI Toolkit

github.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Qsv: Efficient CSV CLI Toolkit

#11

Post by fbdab103 »

quasarj wrote: post #10
snidane wrote: post #9

This looks great! Please consider removing any implicit network calls like the initial "Checking GitHub for updates...". This itself will prevent people from adoption or even trying it any further. This is similar to gnu parallel's --citation, which, albeit a small thing - will scare many people off. Consider adding pivot and unpivot operations. Mlr gets it quite right with syntax, but is unusable since it doesn't wo…

Wait, who is scared off by parallel's --citation?

I refuse to use parallel due to that obnoxiousness.

At minimum, it is not installed by default, so it is already a negative to just using xargs. That it then puts that barrier in my way makes it an easy tool to skip.

Re: Qsv: Efficient CSV CLI Toolkit

#12

Post by fbdab103 »

foehrenwald wrote: post #2

related: https://github.com/johnkerl/miller I am wondering who really uses these tools and for what since there are R and python data science tools available?

I write Python every day, but still use miller here and there. If I am doing a "simple" operation (eye of the beholder), being able to pipe it on the command line is great.

To do a comparable amount of manipulation in Python takes a lot more boilerplate (imports, command line arguments, diety-can-we-default-to-Int64 already?, etc), plus you have to ensure you have a virtual environment with correct dependencies. Which is more or less standard numpy+pandas, but a single executable tool to do some data workup is always appreciated.

I am never performance constrained, but I have been told that miller is one of the slower tools in this space, but I still reach for it do to its wide format support.

Re: Qsv: Efficient CSV CLI Toolkit

#13

Post by quasarj »

fbdab103 wrote: post #11
quasarj wrote: post #10

Earlier quoted context omitted.

Wait, who is scared off by parallel's --citation?

I refuse to use parallel due to that obnoxiousness. At minimum, it is not installed by default, so it is already a negative to just using xargs. That it then puts that barrier in my way makes it an easy tool to skip.

I just don't understand what barrier you are talking about. I just checked, it doesn't even whine at you when you use it, the help just notes that you should cite it if you publish a paper where you used it. And... anyone publishing papers knows about citation requirements lol. Anyone else can ignore it. What is this barrier?

Re: Qsv: Efficient CSV CLI Toolkit

#14

Post by dima55 »

quasarj wrote: post #13
fbdab103 wrote: post #11

Earlier quoted context omitted.

I refuse to use parallel due to that obnoxiousness. At minimum, it is not installed by default, so it is already a negative to just using xargs. That it then puts that barrier in my way makes it an easy tool to skip.

I just don't understand what barrier you are talking about. I just checked, it doesn't even whine at you when you use it, the help just notes that you should cite it if you publish a paper where you used it. And... anyone publishing papers knows about citation requirements lol. Anyone else can ignore it. What is this barrier?

In addition to being annoying, it raises questions about whether it is free software or not. Some people care a whole lot about that. And some people have higher standards about being nagged. And lots and lots of time was spent discussing solutions, for instance: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=915541

Re: Qsv: Efficient CSV CLI Toolkit

#15

Post by dima55 »

snidane wrote: post #9

This looks great! Please consider removing any implicit network calls like the initial "Checking GitHub for updates...". This itself will prevent people from adoption or even trying it any further. This is similar to gnu parallel's --citation, which, albeit a small thing - will scare many people off. Consider adding pivot and unpivot operations. Mlr gets it quite right with syntax, but is unusable since it doesn't wo…

You can get quite far by piping to other tools and/or using DSLs. pivoting can almost certainly be done by the luau support in qsv (or `vnl-filter`, for instance). Summing and grouping is something that `datamash` does well (or qsv luau probably, or `vnl-filter --eval`). Adding a column once again can be done with luau or `vnl-filter`.

Would you be more likely to use this tool if it had even more stuff in it requiring reading even more documentation? That's a genuine question.

Re: Qsv: Efficient CSV CLI Toolkit

#16

Post by mbreese »

snidane wrote: post #3
foehrenwald wrote: post #2

related: https://github.com/johnkerl/miller I am wondering who really uses these tools and for what since there are R and python data science tools available?

Out of core computations. While your python and R script will choke after reading few hundred megs, my compiled binary cli will keep streaming through many such files with memory usage sitting somewhere near zero.

That’s just the effect of streaming IO vs reading in the file into memory all at once. That has nothing to do with the language you use, but how you process the data.

I keep multiple little Python scripts around to do things like sum lists of numbers (think extracting a column with awk, then calculating a sum). Compiled vs an interpreted script really doesn’t matter. What matters is using the right algorithm for the job. R and Python data science libraries like to read in all of the data at once into one single data structure. That’s the anti-pattern to avoid if at all possible.

(But they are very handy for small datasets of complex calculations that require the entire dataset in memory. )

Re: Qsv: Efficient CSV CLI Toolkit

#17

Post by quasarj »

dima55 wrote: post #14
quasarj wrote: post #13

Earlier quoted context omitted.

I just don't understand what barrier you are talking about. I just checked, it doesn't even whine at you when you use it, the help just notes that you should cite it if you publish a paper where you used it. And... anyone publishing papers knows about citation requirements lol. Anyone else can ignore it. What is this barrier?

In addition to being annoying, it raises questions about whether it is free software or not. Some people care a whole lot about that. And some people have higher standards about being nagged. And lots and lots of time was spent discussing solutions, for instance: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=915541

Ah, I see, they have changed it (or possibly the version on my system has had the --will-cite patched out, as discussed in this bug).

Okay, I accept your argument about Free Software. However, I find it interesting that it's a GNU project... they are generally the most hardline Free Software people.

Re: Qsv: Efficient CSV CLI Toolkit

#18

Post by fbdab103 »

quasarj wrote: post #13
fbdab103 wrote: post #11

Earlier quoted context omitted.

I refuse to use parallel due to that obnoxiousness. At minimum, it is not installed by default, so it is already a negative to just using xargs. That it then puts that barrier in my way makes it an easy tool to skip.

I just don't understand what barrier you are talking about. I just checked, it doesn't even whine at you when you use it, the help just notes that you should cite it if you publish a paper where you used it. And... anyone publishing papers knows about citation requirements lol. Anyone else can ignore it. What is this barrier?

To slippery slope this, what happens if more tools start adopting this behavior? Curl now asks you to buy Daniel Stenberg a coffee on each use. Wget asks you to support Ukraine. Caddy wants you to invest in their startup. Each of which may come with their own `--ignore-annoyance-flag` I need to learn. The best I can do is vote with my feet.

I also do not care for the citation requirement. I utilize tons of tools in my work which go unstated. I do not feel the need to cite Linux, DNS, htop, Make, Diet Coke, my Kinesis keyboard, etc. Sadly, reliable plumbing gets no respect. Especially for a tool which is more or less interchangeable with some shell scripting. Unless I am trying to shore up the references list, I am going to cite directly relevant work.

At some point, you no longer need to note that your work was powered by electricity.

Re: Qsv: Efficient CSV CLI Toolkit

#19

Post by jasonjayr »

fbdab103 wrote: post #18
quasarj wrote: post #13

Earlier quoted context omitted.

I just don't understand what barrier you are talking about. I just checked, it doesn't even whine at you when you use it, the help just notes that you should cite it if you publish a paper where you used it. And... anyone publishing papers knows about citation requirements lol. Anyone else can ignore it. What is this barrier?

To slippery slope this, what happens if more tools start adopting this behavior? Curl now asks you to buy Daniel Stenberg a coffee on each use. Wget asks you to support Ukraine. Caddy wants you to invest in their startup. Each of which may come with their own `--ignore-annoyance-flag` I need to learn. The best I can do is vote with my feet. I also do not care for the citation requirement. I utilize tons of tools in m…

Vim has solicited donations for Uganda since forever.

Re: Qsv: Efficient CSV CLI Toolkit

#20

Post by sidpatil »

hermitcrab wrote: post #4
foehrenwald wrote: post #2

related: https://github.com/johnkerl/miller I am wondering who really uses these tools and for what since there are R and python data science tools available?

Also: https://github.com/BurntSushi/xsv https://csvkit.readthedocs.io/en/latest/

qsv is a fork of xsv — the latter hasn't been maintained in a while.
Post reply on HN