Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

161–170 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#161

Excel is NOT a database. I don't understand the people who use it as such (and I've come across many who do). Then they're surprised when it all goes tits-up.

It's hard to say from here what would be the best thing for them to use, but simply using a modern spreadsheet format instead of the ancient xls seems fine for interchange.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#162
post #157
post #139

Friends don't let friends use Excel. Unfortunately Excel is the great swiss army knife of software and it's hard to avoid.

And what do they use instead of Excel? LibreOffice? Some expensive custom database software?

Yep, that's the problem. There's a gap in the space between the generality of Excel and the performance of a SQL DB with no UI that nobody wants to fill.

Microsoft Access sits there, but nobody uses it for some reason.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#163

I love how Excel has become a de facto database - even though it absolutely shouldn’t be.

The first rule of software is "If it works, it'll get used."

This is why people's email inboxes double as their TODO lists.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#164

Earlier quoted context omitted.

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

Who? Almost anyone who wanted to get something done without going through the red tape of working with IT. Even in a private corporation that red tape is enough to make many a things end up in excel (or for the more tech savvy, access). Government is surely even worse (better?) at having red tape so the pressure was likely stronger.

Yup. What also helps is that the data literally sits on your drive, or your network drive. No SaaS vendor is holding it hostage.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#165

So the limit for Excell is 65,000 rows for the old format, and one million-plus rows for the new format (from the article). Then I wonder, is there any tool that mimic Excell but with Sqlite as the backend? The limit of rows in Sqlite is 2 raised to the power of 64 (18446744073709551616 or about 1.8e+19). https://sqlite.org/limits.html

apparently part of the problem here was simply the fact that they used the columns [1] (of which there are far fewer, even on newer versions of excel) instead of the rows

[1] https://twitter.com/standupmaths/status/1313055411285774336?...

edit: retracting this as the person who posted that tweet made a correction in one of the replies

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#166
post #134

I've been bitten by this exact issue (albeit on non critical data), save a file as XLS it will silently drop rows beyond 65k - save it as XLSX and it's all good..

The silent part there is the real surprising part to me. You'd think the exporter would die noisily if it still had data in the buffer and it ran out of file.

I'm betting the exporter isn't running out of file, but out of loop counter.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#167
I wrote an energy reporting system based on Excel. Based on that, I think Excel use falls into 2 categories

a, extremely simple reporting with few workbooks

b, serious misuse of technology

Billion dollar companies were born based on replacing Excel in workflows. The problem is not that they have an old version, the real problem is that they use such a system in the critical path.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#168

Earlier quoted context omitted.

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

Who? Almost anyone who wanted to get something done without going through the red tape of working with IT. Even in a private corporation that red tape is enough to make many a things end up in excel (or for the more tech savvy, access). Government is surely even worse (better?) at having red tape so the pressure was likely stronger.

Yes exactly. An endless battle: IT want to stamp out spreadsheets and centralise everything but it all becomes an ossified bottleneck of development projects. Meanwhile people need to get stuff done so they use Excel.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#169

So the limit for Excell is 65,000 rows for the old format, and one million-plus rows for the new format (from the article). Then I wonder, is there any tool that mimic Excell but with Sqlite as the backend? The limit of rows in Sqlite is 2 raised to the power of 64 (18446744073709551616 or about 1.8e+19). https://sqlite.org/limits.html

XLSX's limit is 2^20, or just over 1.04 million (as the article says).

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#170
post #165

So the limit for Excell is 65,000 rows for the old format, and one million-plus rows for the new format (from the article). Then I wonder, is there any tool that mimic Excell but with Sqlite as the backend? The limit of rows in Sqlite is 2 raised to the power of 64 (18446744073709551616 or about 1.8e+19). https://sqlite.org/limits.html

apparently part of the problem here was simply the fact that they used the columns [1] (of which there are far fewer, even on newer versions of excel) instead of the rows [1] https://twitter.com/standupmaths/status/1313055411285774336?... edit: retracting this as the person who posted that tweet made a correction in one of the replies

about 26^3 columns, or 17576
Post reply on HN