At least you were given a programming problem. I would have played along a little bit and at least discussed how the layers work. Instead of pandas, use the csv module so he could see a loop iterating over header and data rows as tuples. If he pushed further, outline how you might code the csv reader yourself if it didn't exist. Move the discussion into the difficulties of correctly handling a file format like csv and real-world issues like malformed inputs, validation, and error-handling.
In my case about a decade ago, I became obstinate when given one of those silly math-trivia-puzzle problems like how many ping pong balls will fill an elevator or how long is a string. This was for a relatively senior track FAANG interview, where my background at the time was in HPC, distributed systems, and provisioning systems that were precursors to today's IaaS bread and butter. In my case, the interviewer was adamant that I derive some figures that depended on basic geometry and knowing the diameter of the earth and ratios of landmass to ocean surface. But, he wanted me to first SWAG these figures, after I told him I wasn't a geospatial geek and didn't memorize such facts.
I decided to be difficult. If I were somehow faced with such a task, I would first consult reference materials and even see if I could find the actual answer he wanted, since it sounded only one step removed from what you could find in the CIA World Fact Book or similar. Only if that failed, would I dig up source facts and try to derive an answer. I would focus on getting the answer, not on entertaining myself with a Martin Gardner puzzle at my employer's expense and risk. I would never have to make seat of the pants estimates and act on them rather than doing due diligence. I'd either have done homework, or if there was really no time (like some system availability crisis, if we pretend I was an ops person), I'd choose a pre-planned contingency action to make time.
They didn't go forward with an offer, and I felt a bit of relief at that.