XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#1

Post by Yomguithereal » Thu, Mar 27, 2025, 3:50 PM UTC

XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal
github.com

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#2

Post by evolve2k » Sat, Mar 29, 2025, 10:23 PM UTC

Something that would be insanely useful is if your tool could be users to do validations.

For example being able to define data types for each column and say required columns. And then run your tool as a validator and take the errors as an array that’d be amazing!

Coming from a dev who’s just over processing CSV files back into my apps.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#3

Post by ranger_danger » Sat, Mar 29, 2025, 10:30 PM UTC

evolve2k wrote: post #2

Something that would be insanely useful is if your tool could be users to do validations. For example being able to define data types for each column and say required columns. And then run your tool as a validator and take the errors as an array that’d be amazing! Coming from a dev who’s just over processing CSV files back into my apps.

https://digital-preservation.github.io/csv-validator/

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#4

Post by sevg » Sat, Mar 29, 2025, 10:34 PM UTC

This looks really nice to use!

There are a lot of tools that one can use on Linux cli to work with csv. But many of them have become unmaintained. Or have terrible docs. Or have really awkward usage (looking at you, “yq”).

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#5

Post by doug_durham » Sat, Mar 29, 2025, 11:16 PM UTC

I use Pandas for most of my CSV work. It's super fast and very powerful. There's a bit of a learning curve. I can then use Python scripts to manipulate massive CSV files.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#6

Post by jgord » Sun, Mar 30, 2025, 12:10 AM UTC

I was going to mention BurntSushis excellent xsv, also written in rust ... but I see xan mantions its a fork + rewrite of xsv.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#7

Post by crashabr » Sun, Mar 30, 2025, 1:32 AM UTC

evolve2k wrote: post #2

Something that would be insanely useful is if your tool could be users to do validations. For example being able to define data types for each column and say required columns. And then run your tool as a validator and take the errors as an array that’d be amazing! Coming from a dev who’s just over processing CSV files back into my apps.

You should take a look at the Frictionless data library https://framework.frictionlessdata.io/docs/guides/validating...

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#8

Post by pradeepchhetri » Sun, Mar 30, 2025, 1:45 AM UTC

I prefer to use clickhouse-local for all my CSV needs as I don't need to learn a new language (or cli flags) and can just leverage SQL.

    clickhouse local --file medias.csv --query "SELECT edito, count() AS count from table group by all order by count FORMAT PrettyCompact"

   ┌─edito──────┬─count─┐
   │ agence     │     1 │
   │ agrégateur │    10 │
   │ plateforme │    14 │
   │ individu   │    30 │
   │ media      │   423 │
   └────────────┴───────┘

With clickhouse-local, I can do lot more as I can leverage full power of clickhouse.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#9

Post by reagle » Sun, Mar 30, 2025, 2:04 AM UTC

This could be a nice pre-filter for VisiData…?

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#10

Post by sitkack » Sun, Mar 30, 2025, 2:18 AM UTC

pradeepchhetri wrote: post #8

I prefer to use clickhouse-local for all my CSV needs as I don't need to learn a new language (or cli flags) and can just leverage SQL. clickhouse local --file medias.csv --query "SELECT edito, count() AS count from table group by all order by count FORMAT PrettyCompact" ┌─edito──────┬─count─┐ │ agence │ 1 │ │ agrégateur │ 10 │ │ plateforme │ 14 │ │ individu │ 30 │ │ media │ 423 │ └────────────┴───────┘ With clickhou…

I use SQLite in a similar manner, but I'll have to check this out.