Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK Covid-19 data underreported due to file size limit exceeded

theargus.co.uk

Post reply on HN
Print view View original

11–20 of 24 posts

Re: UK Covid-19 data underreported due to file size limit exceeded

#11

Post by zelos »

mellosouls wrote: post #9

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct,…

I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?

Re: UK Covid-19 data underreported due to file size limit exceeded

#12

Post by JonoBB »

mellosouls wrote: post #9

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct,…

Apparently each case was recorded in a column, and the number of columns had reached the maximum (16,384).

https://twitter.com/MaxCRoser/status/1313046638915706880

Re: UK Covid-19 data underreported due to file size limit exceeded

#13

Post by arethuza »

JonoBB wrote: post #12
mellosouls wrote: post #9

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct,…

Apparently each case was recorded in a column, and the number of columns had reached the maximum (16,384). https://twitter.com/MaxCRoser/status/1313046638915706880

The document referenced in that tweet doesn't seem to mention columns?

Re: UK Covid-19 data underreported due to file size limit exceeded

#14

Post by s_dev »

zelos wrote: post #11
mellosouls wrote: post #9

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct,…

I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?

A normalised correctly structured SQL DB?

Re: UK Covid-19 data underreported due to file size limit exceeded

#15

Post by josephpmay »

zelos wrote: post #11
mellosouls wrote: post #9

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct,…

I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?

We’ve actually had a similar issue in the US with some Electronic Initial Case Report documents (to be clear, we mostly rely on Electronic Lab Results-ELR rather than eICR in the US to count COVID numbers, so this hasn’t resulted in an under-count)

Rather than be a problem with the database having arbitrary file size limits, it actually was a problem with file size limits within network intermediaries. In most cases, the issue is less the limit, and more that the files themselves were too large because of how they were created. I don’t think this is the same issue as the UK (I don’t think they are using the eICR standard there yet), but it’s an example of how you could have a problem with file size limitations without storing data in Excel.

Re: UK Covid-19 data underreported due to file size limit exceeded

#16

Post by zelos »

s_dev wrote: post #14
zelos wrote: post #11

Earlier quoted context omitted.

I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?

A normalised correctly structured SQL DB?

I'm not a backend developer, but isn't that likely to be in the terabytes? This seems to be issues around tens of thousands of rows.

Re: UK Covid-19 data underreported due to file size limit exceeded

#17

Post by s_dev »

zelos wrote: post #16
s_dev wrote: post #14

Earlier quoted context omitted.

A normalised correctly structured SQL DB?

I'm not a backend developer, but isn't that likely to be in the terabytes? This seems to be issues around tens of thousands of rows.

A normalised SQL DB will have zero redundancy. It would also easily handle these data set sizes. The only reason Excel was chosen was because clearly some clerk just felt more familiar with the tool and didn't have the experience to know better.

The reality is though for storing this kind of relational data -- it's a solved problem -- SQL would have been the correct tool to employ.

Re: UK Covid-19 data underreported due to file size limit exceeded

#18

Post by bArray »

zelos wrote: post #11
mellosouls wrote: post #9

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct,…

I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?

A database running from a FAT disk partition for example.

Re: UK Covid-19 data underreported due to file size limit exceeded

#19

Post by netsharc »

zelos wrote: post #11
mellosouls wrote: post #9

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct,…

I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?

It's probably more the web server configured to only allow x MB maximum for file uploads. Or since the government has been shown to be incompetent, maybe it's the maximum size for email attachments..

Re: UK Covid-19 data underreported due to file size limit exceeded

#20

Post by bar8uyr »

zelos wrote: post #11
mellosouls wrote: post #9

Note: for all the people reckoning this is an Excel issue as reported on Twitter: a) The Twitter link is referencing the Daily Mail. https://www.dailymail.co.uk/news/article-8805697/Furious-bla... b) The Mail does not source it's claim. c) All those Excel references in the news seem to postdate (at time of writing) speculation on Excel in IT-heavy forums like this. While I can believe the Excel conjecture is correct,…

I hope you're right, but most reports do at least mention "exceeding maximum file size". That seems like a fairly basic error: what reasonable (national-scale) data storage formats would have an arbitrary file size limit?

Originally I presumed this was an error importing Excel files and an upload file size limit..
Post reply on HN