Live data from Hacker News

Ask HN: How would you chunk a large Excel file?

news.ycombinator.com

1–10 of 49 posts

Ask HN: How would you chunk a large Excel file?

#1
Let's say you had an Excel file with 10,000 rows and you wanted to break it up into many Excel files each with 500 records. Each new file should have the header fields from the original. How would you do it? I did it by writing a node script but I'm wondering if there's an easier way.

Edit: Guys this is just an example. I'm looking for a general solution. It could be 10 million rows.

Re: Ask HN: How would you chunk a large Excel file?

#3
Bet it took you more time.to write that code than if you'd done it manually.

If you actually had to do it 'properly' there are actually a ton of options:

  - do it old school with a VBA macro
  - use the newer js macro stuff
  - xslx files are just a zip file of XML so could just do it in pretty much any language

Re: Ask HN: How would you chunk a large Excel file?

#8
Write a script. I know there are decent api for excel and files.

I dont know the api or recommended scripting language. This would be a good case for chatgpt or equivalent type task. Enough to get started.

edit: I asked chatgpt, it recommended python and 'pandas' for interacting with excel

    python
    import pandas as pd
    # Load the data from an Excel file, assuming headers are in the first row by default
    data = pd.read_excel('path/to/your/file.xlsx')
    # Define the number of records per chunk
    chunk_size = 500
    # Split the data into chunks and write each chunk to a new Excel file
    for i in range(0, len(data), chunk_size):
        # Extract the chunk of data based on the current index range
        chunk = data.iloc[i:i + chunk_size]
        # Write the chunk to a new Excel file, including headers as column names
        chunk.to_excel(f'output_{i // chunk_size + 1}.xlsx', index=False)
I asked about the first 'row', and it claims panda includes that in each chunk, but I don't know about that. It's at least a place to start to iterate from. Would need to iterate further with real code/tests.

Re: Ask HN: How would you chunk a large Excel file?

#9
I would use R. Phoneposting now but something like

  library(tidyverse)
  library(readxl)
  library(writexl)
  read_excel("file.xlsx") %>%
    group_by(group_id =     row_number() %/% 20) %>%
    group_walk(~ write_xlsx(.x, paste0("file_", .y, ".xlsx")))

edit: updated to write xlsx instead of csv

Re: Ask HN: How would you chunk a large Excel file?

#10
It's been well over a decade since I last dealt with Excel, but I remember you could actually query the data with SQL without opening the file, like you would with any flat-file db. If the size is the problem. It was poorly documented but I'd done it a few times and it worked really well. The best part being it was simple, fast and worked even with locked files. Otherwise I don't understand the question.
Post reply on HN