XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal • Hacker News Board

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#21

Post by gpvos » Sun, Mar 30, 2025, 4:18 PM UTC

I tend to use csvkit for more complicated transformations, and OCaml's csvtool[0] for the simpler ones. For intermediate transformations I wrote my own csved[1] script, which reads for every line of a CSV reads it into @F, applies a Perl expression to that array, then writes it out. With the -h option you can also use the %F hash to access fields by name. It's very fast.

It looks like xsv and xan are in the "csvkit but faster" niche, which is nice, but now I must learn another set of commands.

And there are now many more recent utilities called csvtool, including a Perl and a Python one.

[0] https://github.com/Chris00/ocaml-csv

[1] https://github.com/gpvos/csved

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#22

Post by rixed » Sun, Mar 30, 2025, 4:43 PM UTC

pradeepchhetri wrote: post #8

I prefer to use clickhouse-local for all my CSV needs as I don't need to learn a new language (or cli flags) and can just leverage SQL. clickhouse local --file medias.csv --query "SELECT edito, count() AS count from table group by all order by count FORMAT PrettyCompact" ┌─edito──────┬─count─┐ │ agence │ 1 │ │ agrégateur │ 10 │ │ plateforme │ 14 │ │ individu │ 30 │ │ media │ 423 │ └────────────┴───────┘ With clickhou…

How does it compare with duckdb, which I usualy resort to? What I like with duckdb is that it's a single binary, no server needed, and it's been happy so far with all the CSV file I've thrown at it.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#23

Post by SoftTalker » Sun, Mar 30, 2025, 5:22 PM UTC

pradeepchhetri wrote: post #8

I prefer to use clickhouse-local for all my CSV needs as I don't need to learn a new language (or cli flags) and can just leverage SQL. clickhouse local --file medias.csv --query "SELECT edito, count() AS count from table group by all order by count FORMAT PrettyCompact" ┌─edito──────┬─count─┐ │ agence │ 1 │ │ agrégateur │ 10 │ │ plateforme │ 14 │ │ individu │ 30 │ │ media │ 423 │ └────────────┴───────┘ With clickhou…

I used to use q for this sort of thing. Not sure if there are better choices now as it have been a few years.

https://harelba.github.io/q/

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#24

Post by SwamyM » Sun, Mar 30, 2025, 6:08 PM UTC

dlkmp wrote: post #16

Can't help but thinking how handy PowerShell is out of the box for tasks like this. Translating the examples from the ReadMe, having read the file with: $medias = Get-Content .\medias.csv | ConvertFrom-Csv Previewing the file in the terminal xan view medias.csv $medias | Format-Table Reading a flattened representation of the first row xan flatten -c medias.csv $medias | Format-List Searching for rows xan search -s ou…

Yes, I find PowerShell is criminally underrated for these type of tasks. Even though it's open source and cross-platform, the stigma from it's Windows-centric days is hard to overcome.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#25

Post by pradeepchhetri » Mon, Mar 31, 2025, 2:27 AM UTC

rixed wrote: post #22

pradeepchhetri wrote: post #8

I prefer to use clickhouse-local for all my CSV needs as I don't need to learn a new language (or cli flags) and can just leverage SQL. clickhouse local --file medias.csv --query "SELECT edito, count() AS count from table group by all order by count FORMAT PrettyCompact" ┌─edito──────┬─count─┐ │ agence │ 1 │ │ agrégateur │ 10 │ │ plateforme │ 14 │ │ individu │ 30 │ │ media │ 423 │ └────────────┴───────┘ With clickhou…

How does it compare with duckdb, which I usualy resort to? What I like with duckdb is that it's a single binary, no server needed, and it's been happy so far with all the CSV file I've thrown at it.

clickhouse-local is similar to duckdb, you don't need a clickhouse-server running in order to use clickhouse-local. You just need to download the clickhouse binary and start using it.

  clickhouse local
  ClickHouse local version 25.4.1.1143 (official build).

  :)

There are few benefits of using clickhouse-local since ClickHouse can just do lot more than DuckDB. One such example is handling compressed files. ClickHouse can handle compressed files with formats ranging from zstd, lz4, snappy, gz, xz, bz2, zip, tar, 7zip.

  clickhouse local --query "SELECT count() FROM file('top-1m-2018-01-10.csv.zip :: *.csv')"
  1000000

Also clickhouse-local is much more efficient in handling big csv files[0]

[0]: https://www.vantage.sh/blog/clickhouse-local-vs-duckdb

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#26

Post by rixed » Wed, Apr 02, 2025, 6:29 AM UTC

pradeepchhetri wrote: post #25

rixed wrote: post #22

Earlier quoted context omitted.

How does it compare with duckdb, which I usualy resort to? What I like with duckdb is that it's a single binary, no server needed, and it's been happy so far with all the CSV file I've thrown at it.

clickhouse-local is similar to duckdb, you don't need a clickhouse-server running in order to use clickhouse-local. You just need to download the clickhouse binary and start using it. clickhouse local ClickHouse local version 25.4.1.1143 (official build). :) There are few benefits of using clickhouse-local since ClickHouse can just do lot more than DuckDB. One such example is handling compressed files. ClickHouse can…

Wanted to try it.

Debian package is of poor quality: not even sure if clickhouse local is included in there, I believe so but there is no manpage, no doc at all, and no `clickhouse-server -h`.

Went to the official page looking for a tarball to download, found only the `curl|sh` joke.

Went to github looking for tagged tarballs, couldn't find any. Looked for INSTALL.md, couldn't find any.

Will try harder later, have to weep my tears for now.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#27

Post by pradeepchhetri » Wed, Apr 09, 2025, 3:06 AM UTC

rixed wrote: post #26

pradeepchhetri wrote: post #25

Earlier quoted context omitted.

clickhouse-local is similar to duckdb, you don't need a clickhouse-server running in order to use clickhouse-local. You just need to download the clickhouse binary and start using it. clickhouse local ClickHouse local version 25.4.1.1143 (official build). :) There are few benefits of using clickhouse-local since ClickHouse can just do lot more than DuckDB. One such example is handling compressed files. ClickHouse can…

Wanted to try it. Debian package is of poor quality: not even sure if clickhouse local is included in there, I believe so but there is no manpage, no doc at all, and no `clickhouse-server -h`. Went to the official page looking for a tarball to download, found only the `curl|sh` joke. Went to github looking for tagged tarballs, couldn't find any. Looked for INSTALL.md, couldn't find any. Will try harder later, have to…

> Debian package is of poor quality

Can you elaborate more please? I would love if you can say what all can be improved to make debian package up to standards.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#28

Post by zX41ZdbW » Wed, Apr 09, 2025, 4:14 PM UTC

rixed wrote: post #26

pradeepchhetri wrote: post #25

Earlier quoted context omitted.

clickhouse-local is similar to duckdb, you don't need a clickhouse-server running in order to use clickhouse-local. You just need to download the clickhouse binary and start using it. clickhouse local ClickHouse local version 25.4.1.1143 (official build). :) There are few benefits of using clickhouse-local since ClickHouse can just do lot more than DuckDB. One such example is handling compressed files. ClickHouse can…

Wanted to try it. Debian package is of poor quality: not even sure if clickhouse local is included in there, I believe so but there is no manpage, no doc at all, and no `clickhouse-server -h`. Went to the official page looking for a tarball to download, found only the `curl|sh` joke. Went to github looking for tagged tarballs, couldn't find any. Looked for INSTALL.md, couldn't find any. Will try harder later, have to…

ClickHouse is a single binary. It can be invoked as clickhouse-server, clickhouse-client, and clickhouse-local. The help is available as `clickhouse-local --help`. clickhouse-local also has a shorthand alias, `ch`.

This binary is packaged inside .deb, .rpm, and .tgz, and it is also available for direct download. The curl|sh script selects the platform (x86_64, aarch64 x Linux, Mac, FreeBSD) and downloads the appropriate binary.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#29

Post by rixed » Sat, Apr 19, 2025, 2:41 PM UTC

pradeepchhetri wrote: post #27

rixed wrote: post #26

Earlier quoted context omitted.

Wanted to try it. Debian package is of poor quality: not even sure if clickhouse local is included in there, I believe so but there is no manpage, no doc at all, and no `clickhouse-server -h`. Went to the official page looking for a tarball to download, found only the `curl|sh` joke. Went to github looking for tagged tarballs, couldn't find any. Looked for INSTALL.md, couldn't find any. Will try harder later, have to…

> Debian package is of poor quality Can you elaborate more please? I would love if you can say what all can be improved to make debian package up to standards.

Thank you for your interest.

My comment was really about the state of documentation ("there is no manpage, no doc at all, and no `clickhouse-server -h`"). More specifically:

  % dpkg -S clickhouse-server | grep bin
  clickhouse-server: /usr/sbin/clickhouse-server
  % man clickhouse-server
  No manual entry for clickhouse-server
  % man clickhouse       
  No manual entry for clickhouse
  % /usr/sbin/clickhouse-server --help
  Unknown option specified: help
  % /usr/sbin/clickhouse-server -h    
  Unknown option specified: h
  % ls -l /usr/share/doc/clickhouse-server
  total 60
  -rw-r--r-- 1 root root   235 Dec  5  2022 changelog.Debian.amd64.gz
  -rw-r--r-- 1 root root  1437 Dec  5  2022 changelog.Debian.gz
  -rw-r--r-- 1 root root 33174 Dec 20  2018 changelog.gz
  -rw-r--r-- 1 root root 15057 Oct 29  2022 copyright