Live data from Hacker News

Ask HN: How would you chunk a large Excel file?

news.ycombinator.com

31–40 of 49 posts

Re: Ask HN: How would you chunk a large Excel file?

#32
Does each chunk have to execute properly? Do you have to minimize chunk size by maximum working inter-cell references? Can you split huge chunks with too many references by an intermediate chunk with stage based value propagation?

This turns into a massive graph theory nightmare problem if there are lots of references going on

Re: Ask HN: How would you chunk a large Excel file?

#35
I would save my data in CSV format, then use this. Save the code below as chunk.pl (remove leading spaces) and call it as "perl chunk.pl" :

    #!/usr/bin/perl -CSD -w -Mstrict -Mwarnings -MText::CSV
    
    # chunk.pl -- split csv files into chunks
    
    # Usage message and exit if needed
    if (!@ARGV || $ARGV[0] eq '-h') {
        print "Usage: $0 input_csv [chunk_size] [output_filename_format] [separator]\n";
        print "Example: $0 input.csv 500 'input-%08d.csv' ','\n";
        exit;
    }
    
    # Set command-line arguments
    my ($INFILE, $CHUNKSIZE, $FMT, $SEP) = @ARGV;
    $CHUNKSIZE //= 500;
    $FMT //= "data-%08d.csv";
    $SEP //= ",";
    
    # Initialize CSV, file handles, and counters
    my $csv = Text::CSV->new({ binary => 1, auto_diag => 1, sep_char => $SEP, eol => "\n" });
    my ($i, $f, $out) = (0, 1, undef);
    open my $in, "getline($in)) {
        if ($i % $CHUNKSIZE == 0) {
            close $out if defined $out;
            open $out, ">:encoding(UTF-8)", sprintf($FMT, $f++) or die "Cannot open output file: $!";
        }
        $csv->print($out, $row) or die "Failed to write row: $!";
        $i++;
    }
    
    # Clean up: close file handles
    close $out if defined $out;
    close $in;

Re: Ask HN: How would you chunk a large Excel file?

#36
post #19

You can use my SQL spreadsheet app: https://superintendent.app I had a similar problem at work where I needed to do some formula on a 5 GB CSV file. Excel can't handle more than 1M rows. Database through command line is too clunky. I did try to split the CSV into multiple files before but using formula on top of multiple files isn't easy. Eventually I built a Desktop GUI wrapper on SQLite, and it grew into Superinten…

I am quite enjoying the different approaches people are recommending. Good job.

Re: Ask HN: How would you chunk a large Excel file?

#37
post #31
post #6

Not knowing what the data is like; I'd save it in a comma- or tab-separated text format. From there, it's just a few lines of bash.

Or https://github.com/BurntSushi/xsv

xsv is great. I quite like the fork called qsv as well; it has some features that were helpful to avoid some piping I didn't want to do. There are a lot of other additions. https://github.com/jqnatividad/qsv

Re: Ask HN: How would you chunk a large Excel file?

#40
Why would you want to break it up?

If the input is not a spreadsheet, use a tool appropriate to the scale of the data store. Provide an api for access from Excel if end users need Excel.

If input is an Excel but the integrity of the data as a spreadsheet doesn’t matter,

If it is an Excel already, Excel provides end users with access from Excel. If the Excel is likely to outgrow Excel, use a tool appropriate to the scale of the data. Provide an api for access from Excel if end users need Excel.

If it-is-Execl is your problem, breaking it up into multiple Excels is slicing the magic broom. Good luck.

Good luck.

Post reply on HN