Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

github.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#13

Post by aorth »

strunz wrote: post #12
ctippett wrote: post #11

See also: csvkit ( https://csvkit.readthedocs.io )

Yep, been using csvkit for years and it's so great. There's nothing I haven't been able to do, is there some reason to use Xan instead?

I use both csvkit and xsv. The syntax for csvkit is a bit easier for most of my uses, but xsv is way faster when I have larger files.

I know someone who uses csvtk (Golang), but haven't tried it yet. https://github.com/shenwei356/csvtk

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#14

Post by hantusk »

evolve2k wrote: post #2

Something that would be insanely useful is if your tool could be users to do validations. For example being able to define data types for each column and say required columns. And then run your tool as a validator and take the errors as an array that’d be amazing! Coming from a dev who’s just over processing CSV files back into my apps.

Reading CSV into a duckdb table will give you that, along with a table for the errors and reason for error: https://duckdb.org/docs/stable/data/csv/reading_faulty_csv_f...

Could definitely be done as a small little bash script

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#16

Post by dlkmp »

Can't help but thinking how handy PowerShell is out of the box for tasks like this.

Translating the examples from the ReadMe, having read the file with:

  $medias = Get-Content .\medias.csv | ConvertFrom-Csv
Previewing the file in the terminal

  xan view medias.csv
  $medias | Format-Table
Reading a flattened representation of the first row

  xan flatten -c medias.csv
  $medias | Format-List
Searching for rows

  xan search -s outreach internationale medias.csv | xan view
  $medias | Where-Object { $_.outreach -eq "internationale" } | Format-Table
Selecting some columns

  xan select foundation_year,name medias.csv | xan view
  $medias | Select-Object -Property foundation_year, name | Format-Table
Sorting the file

  xan sort -s foundation_year medias.csv | xan view -s name,foundation_year
  $medias | Sort-Object -Property foundation_year | Select-Object -Property name, foundation_year | Format-Table
Deduplicating the file on some column

  # Some medias of our corpus have the same ids on mediacloud.org
  xan dedup -s mediacloud_ids medias.csv | xan count && xan count medias.csv
  $medias | Select-Object -ExpandProperty mediacloud_ids -Unique | Measure-Object; $medias | Measure-Object -Property mediacloud_ids
Computing frequency tables

  xan frequency -s edito medias.csv | xan view
  $medias | Group-Object -Property edito | Sort-Object -Property Count -Descending
It's probably orders of magnitude slower, and of course, plotting graphs and so on gets tricky. But for the simple type of analysis I typically do, it's fast enough, I don't need to learn an extra tool, and the auto-completion of column/property names is very convenient.

Re: XAN: A Modern CSV-Centric Data Manipulation Toolkit for the Terminal

#19

Post by account-5 »

dlkmp wrote: post #16

Can't help but thinking how handy PowerShell is out of the box for tasks like this. Translating the examples from the ReadMe, having read the file with: $medias = Get-Content .\medias.csv | ConvertFrom-Csv Previewing the file in the terminal xan view medias.csv $medias | Format-Table Reading a flattened representation of the first row xan flatten -c medias.csv $medias | Format-List Searching for rows xan search -s ou…

I find Nushell even better for these usecases:

    $medias = open .\medias.csv
The above is the initial read and format into table.

I'm currently on my phone so can't go through all the examples, but knowing both PS and nu, nu has the better syntax.

EDIT:

Get data and view in table:

    let $medias = http get https://github.com/medialab/corpora/raw/master/polarisation/medias.csv
    $medias
Get headers:

    $medias | columns
Get count of rows:

   $medias | length
Get flattened, slight more convoluted (caveat there might be a better way):

    $medias | each {print $in}
Search rows:

    $medias | where $it.outreach == 'internationale'
Select columns:

    $medias | select foundation_year name
Sort file:

    $medias | select foundation_year name | sort-by foundation_year
Dedup based on column:

    $medias | uniq-by mediacloud_ids
Computing frequency and histogram

    $medias | histogram edito
Post reply on HN