Not knowing what the data is like; I'd save it in a comma- or tab-separated text format. From there, it's just a few lines of bash.
Ask HN: How would you chunk a large Excel file?
31–40 of 49 posts
Re: Ask HN: How would you chunk a large Excel file?
#32This turns into a massive graph theory nightmare problem if there are lots of references going on
Re: Ask HN: How would you chunk a large Excel file?
#33Re: Ask HN: How would you chunk a large Excel file?
#34It can't be. Excel's maximum row limit is only ~1 million.
Re: Ask HN: How would you chunk a large Excel file?
#35 #!/usr/bin/perl -CSD -w -Mstrict -Mwarnings -MText::CSV
# chunk.pl -- split csv files into chunks
# Usage message and exit if needed
if (!@ARGV || $ARGV[0] eq '-h') {
print "Usage: $0 input_csv [chunk_size] [output_filename_format] [separator]\n";
print "Example: $0 input.csv 500 'input-%08d.csv' ','\n";
exit;
}
# Set command-line arguments
my ($INFILE, $CHUNKSIZE, $FMT, $SEP) = @ARGV;
$CHUNKSIZE //= 500;
$FMT //= "data-%08d.csv";
$SEP //= ",";
# Initialize CSV, file handles, and counters
my $csv = Text::CSV->new({ binary => 1, auto_diag => 1, sep_char => $SEP, eol => "\n" });
my ($i, $f, $out) = (0, 1, undef);
open my $in, "getline($in)) {
if ($i % $CHUNKSIZE == 0) {
close $out if defined $out;
open $out, ">:encoding(UTF-8)", sprintf($FMT, $f++) or die "Cannot open output file: $!";
}
$csv->print($out, $row) or die "Failed to write row: $!";
$i++;
}
# Clean up: close file handles
close $out if defined $out;
close $in;Re: Ask HN: How would you chunk a large Excel file?
#36You can use my SQL spreadsheet app: https://superintendent.app I had a similar problem at work where I needed to do some formula on a 5 GB CSV file. Excel can't handle more than 1M rows. Database through command line is too clunky. I did try to split the CSV into multiple files before but using formula on top of multiple files isn't easy. Eventually I built a Desktop GUI wrapper on SQLite, and it grew into Superinten…
Re: Ask HN: How would you chunk a large Excel file?
#37Not knowing what the data is like; I'd save it in a comma- or tab-separated text format. From there, it's just a few lines of bash.
Or https://github.com/BurntSushi/xsv
Re: Ask HN: How would you chunk a large Excel file?
#38Re: Ask HN: How would you chunk a large Excel file?
#39Re: Ask HN: How would you chunk a large Excel file?
#40If the input is not a spreadsheet, use a tool appropriate to the scale of the data store. Provide an api for access from Excel if end users need Excel.
If input is an Excel but the integrity of the data as a spreadsheet doesn’t matter,
If it is an Excel already, Excel provides end users with access from Excel. If the Excel is likely to outgrow Excel, use a tool appropriate to the scale of the data. Provide an api for access from Excel if end users need Excel.
If it-is-Execl is your problem, breaking it up into multiple Excels is slicing the magic broom. Good luck.
Good luck.