Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The implementation of the UK Covid-19 dashboard

techcommunity.microsoft.com

Post reply on HN
Print view View original

41–50 of 118 posts

Re: The implementation of the UK Covid-19 dashboard

#41

Post by greatgib »

First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on...

On the other side, I have the feeling that this thing that clearly over-engineered. Just look at data their diagram... If I'm not wrong there is one writer and multiple reader for the data, or at least multiple writers on one side and multiple readers on another side, without a need for "real time" consistency.

So, this thing could probably have been better splitted to not have the use for "scaled" databases

Re: The implementation of the UK Covid-19 dashboard

#42

Post by mslot »

trebligdivad wrote: post #5

I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…

The dashboard has to deal with a complex data integration problem, with different sources with differences in completeness, accuracy, age, and granularity (at many levels), daily corrections in past data, changes in data structure and semantics over time, large data volume, 4pm traffic spikes. Moreover, an API that allows you to select different metrics for different areas. Being able to simply write a SQL query or update and have it be fast regardless of volume is quite a life-saver if you have a tiny (mostly 1 person) team and development speed/adaptability is essential.

Some example queries issued by the dashboard: https://github.com/publichealthengland/coronavirus-dashboard... https://github.com/publichealthengland/coronavirus-dashboard... https://github.com/publichealthengland/coronavirus-dashboard...

Re: The implementation of the UK Covid-19 dashboard

#43

Post by mmcnl »

trebligdivad wrote: post #5

I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…

I agree. For example, the Dutch corona dashboard (coronadashboard.rijksoverheid.nl) is a statically rendered dashboard using Next that gets updated daily. No backend and it's super fast.

Maybe I'm not objective because I'm Dutch myself, but from both a user-facing and technical perspective I think the Dutch dashboard is by far the best corona dashboard in the world. It's very fast, has a lot of detailed visualizations, provides a lot of context and has fair amount of accessibility features.

Re: The implementation of the UK Covid-19 dashboard

#44

Post by amelius »

shroompasta wrote: post #38
rm445 wrote: post #26

Earlier quoted context omitted.

> Microsoft > Ban FAANG I know exactly what you mean, but is that what we're doing now? Including Microsoft in FAANG but not changing the acronym?

to be fair, microsoft should've been included in the first place. they've been teetering top 3 market cap for a while now.

They should replace Netflix in FAANG or include Disney, HBO and a bunch of others.

Re: The implementation of the UK Covid-19 dashboard

#45

Post by sdoering »

greatgib wrote: post #41

First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on... On the other side, I have the feeling that this thing that clearly over-engineered. Just look at data their diagram... If I'm not wrong there is one writer and multiple reader for the data, or at least multiple writers on one side and multiple readers on anothe…

> First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on...

The article states it was written by Claire Giordano from San Francisco. Not sure where you got the UK Government official from.

To me it read like a b2b marketing piece and showcase. Kind of: We can power this, so we can power your BI dashboard as well.

Taking this into account it was a nice write up and from a data analyst's and consultant's pov interesting to read.

Re: The implementation of the UK Covid-19 dashboard

#46

Post by GraemeMeyer »

glogla wrote: post #31

I skimmed the article and it seems interesting. On the data side, they have ~7.5 billion total records and they add in 55 million new a day. On the web side, they have ~1 million daily unique users and 100k concurrent users at peak ("concurrent" means "in one minute" is seems). I'm no expert on the web part, but I'm kind of curious why they went with the design they did for the data part. The design, and the chosen t…

A few other commenters have pointed out the same thing - I’m wondering if it’s simply the skill set they had on hand when the need arose

Re: The implementation of the UK Covid-19 dashboard

#47

Post by mslot »

glogla wrote: post #31

I skimmed the article and it seems interesting. On the data side, they have ~7.5 billion total records and they add in 55 million new a day. On the web side, they have ~1 million daily unique users and 100k concurrent users at peak ("concurrent" means "in one minute" is seems). I'm no expert on the web part, but I'm kind of curious why they went with the design they did for the data part. The design, and the chosen t…

OLAP stores are relatively fast at answering a single query on a large data set, but basically none of them can handle high throughput with subsecond response times (e.g. when the whole country checks statistics for their own postcode at 4pm).

OLTP stores are relatively bad at aggregating across a lot of data.

Analytics dashboards with many users, a lot of ever-changing data, and many different views exist in a gray area between OLAP and OLTP often referred to as real-time analytics or operational analytics. The queries are usually somewhat lighter / less ad-hoc / more indexed than in OLAP, but there can be hundreds or thousands of them per second with different filters and aggregations.

There are some specialized real-time analytics databases like Druid. Citus (used in the article) allows you to run such workloads at scale on PostgreSQL.

Re: The implementation of the UK Covid-19 dashboard

#48

Post by StringyBob »

greatgib wrote: post #41

First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on... On the other side, I have the feeling that this thing that clearly over-engineered. Just look at data their diagram... If I'm not wrong there is one writer and multiple reader for the data, or at least multiple writers on one side and multiple readers on anothe…

Not as much tech detail, but for an alternative source, here’s an intro by the team dashboard lead - https://ukhsa.blog.gov.uk/2022/01/20/reporting-the-vital-sta...

Also - I’ve been really impressed by the openness of the team actually doing the work - eg threads like https://twitter.com/pouriaaa/status/1476892793729654787

and in particular this analysis of debugging a problem that the dashboard encountered - which also gives a lot more background context: https://dev.to/xenatisch/cascade-of-doom-jit-and-how-a-postg...

Re: The implementation of the UK Covid-19 dashboard

#49

Post by DrBazza »

greatgib wrote: post #41

First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on... On the other side, I have the feeling that this thing that clearly over-engineered. Just look at data their diagram... If I'm not wrong there is one writer and multiple reader for the data, or at least multiple writers on one side and multiple readers on anothe…

> First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on...

Maybe, I'm naive, or not cynical enough, but I just read this as a case study of customer using Azure to provide the general public with information in a robust fashion.

In fact, if anything, the whole article is remarkably light on pushing Azure, and quite heavy on architecture details.

The open source code (on Github) uses Postgres (not MSSQL), and Python (not C# or Powershell), and in fact has a screen shot of Jetbrain's Pycharm, and not VSCode.

In fact it's probably quite an MS agnostic article.

Even though gov.uk is actually a really good IT company, I'm quite pleased that they're using "the cloud" rather than trying to create their own.

Post reply on HN