21–30 of 34 posts
Do people really have CSV files with more than 1M rows? How do you search and update such files? Why not use some database?
Why is this a yearly cost? I don’t understand. There’s already open source utilities [0] for users who aren’t proficient in Unix commands. If anything this should be the definition of a one time fee. You’re free to charge whatever you like, but it seems odd that anyone would pay you year after year to use your app. It’s not the price, as $40 isn’t that much, but the value and principle of the thing. [0] https://githu…
The target users are non-technical users. Think accountants. They are technical and not be able to use command line tools. Other than that, I appreciate the feedback.
Second, why do accountants need to pay a yearly fee for such a basic utility?
This site probably isn't your target audience....
Unless you want competition and people to realize people would pay for such a thing
It would be fascinating if there ended up being a market for this as it’s so counterintuitive for me.
Do people really have CSV files with more than 1M rows? How do you search and update such files? Why not use some database?
It would be nice if excel worked with 50M rows to do quick filters and pivots and stuff.
But of course there are other tools for that.
Comically, I’m not sure how splitting a 50M line csv into 50 files helps as you can’t filter or pivot across 50 files unless you want to do lots of manual stuff.
It's interesting how people use and abuse Excel. How many people try to use Excel to process millions of rows? Time to use tools that can directly query csv files to aggregate the data.
I happen to build that kind of tool as well: https://superintendent.app Anyway, if you don't know SQL, you are kinda out of luck and stuck with Excel.
Earlier quoted context omitted.
I happen to build that kind of tool as well: https://superintendent.app Anyway, if you don't know SQL, you are kinda out of luck and stuck with Excel.
Or one of the billion tools designed to work with big csvs.
awk can do this super easily. Here's an example snippet that not only shards, but compresses your shards. ``` (NR - 1) % shard_size == 0 { # ready to start a new shard current_n = current_n + 1 output_file = sprintf("%s%04d.%s.bz2", target_prefix, current_n, file_type) print "writing to " output_file > "/dev/stderr"
# close any prior-opened output_command (else will err on too many open files)
if(output_command != "")
close(output_command)
output_command = "bzip2 > " output_file
# print header
print headrow | output_commandNR != 1 { print $0 | output_command }
```
This of course assumes that each line is a single record, so you'll need some preprocessing if your CSV might contain embedded line-ends. For the preprocessing, you can use something like the `2tsv` or `select -e ...` command of https://github.com/liquidaty/zsv (disclaimer: I'm its author) to ensure each record is a single line.
You can also use something like `xsv split` (see https://lib.rs/crates/xsv) which frankly is probably your best option as of today (though zsv will be getting its own shard command soon)
Earlier quoted context omitted.
The target users are non-technical users. Think accountants. They are technical and not be able to use command line tools. Other than that, I appreciate the feedback.
The target users don't have that amount of data believe me I worked as an accountant