The implementation of the UK Covid-19 dashboard • Hacker News Board

Re: The implementation of the UK Covid-19 dashboard

#51

Post by smarx007 » Sun, Jan 23, 2022, 2:41 PM UTC

lloydatkinson wrote: post #24

trebligdivad wrote: post #5

I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…

I had the same thoughts and then it was confirmed how insane this setup is part way through: “At the time of writing, the Citus distributed database cluster adopted by the team on Azure is HA-enabled for high availability and has 12 worker nodes with a combined total of 192 vCores, ~1.5 TB of memory, and 24 TB of storage. (The Citus coordinator node has 64 vCores, 256 GB of memory, and 1 TB of storage.)” That’s beyon…

Did you look at the 3 different (non-trivial) APIs they are offering on top of the dashboard? Though I have a hard time understanding why use PostgreSQL instead of ClickHouse, for example.

Re: The implementation of the UK Covid-19 dashboard

#52

Post by mdm12 » Sun, Jan 23, 2022, 2:41 PM UTC

nojito wrote: post #30

zlib wrote: post #8

https://github.com/publichealthengland powered by a lot of Python and Ruby, nice.

And a bunch of F# https://github.com/publichealthengland/coronavirus-dashboard...

Nice catch!

It surprises me how much more popular F# is in Europe compared to the US. I finally got a professional F# gig in the states (\o/), but there were very few options. It makes me wonder, are universities in Europe providing a more functional-first approach to CS education, or is something else going on?

Re: The implementation of the UK Covid-19 dashboard

#53

Post by amelius » Sun, Jan 23, 2022, 2:47 PM UTC

StringyBob wrote: post #50

amelius wrote: post #3

I would have preferred to see it implemented as a spreadsheet in the cloud.

Well, you can just grab the data very easily: https://coronavirus.data.gov.uk/details/download

Yeah, not what I prefer.

As a programmer, I want to make demands about UX too, for a change ...

Re: The implementation of the UK Covid-19 dashboard

#54

Post by Rastonbury » Sun, Jan 23, 2022, 2:49 PM UTC

sdoering wrote: post #45

greatgib wrote: post #41

First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on... On the other side, I have the feeling that this thing that clearly over-engineered. Just look at data their diagram... If I'm not wrong there is one writer and multiple reader for the data, or at least multiple writers on one side and multiple readers on anothe…

> First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on... The article states it was written by Claire Giordano from San Francisco. Not sure where you got the UK Government official from. To me it read like a b2b marketing piece and showcase. Kind of: We can power this, so we can power your BI dashboard as well. Taki…

Coming from consulting, this is exactly what it is - from a pure engineering standpoint it may be lacking but if you've read any other 'case study' targeted as business people this goes really deep. Normally it's SEO fodder crammed with jargon and buzzwords

Re: The implementation of the UK Covid-19 dashboard

#55

Post by discordance » Sun, Jan 23, 2022, 2:55 PM UTC

azalemeth wrote: post #22

axiosgunnar wrote: post #11

> Government project > awarded to Microsoft Hey Europe, want to stop being several decades behind in IT compared to US/China? One simple trick: Ban FAANG from public procurement in Europe! It‘s a no-brainer really. Buy locally, ideally giving small companies and startups a chance. You will have to do it anyway very soon if you want your privacy laws to be taken seriously. There might be a couple of months of friction…

I really can't agree with this more. My university gives >$2e6/year to Microsoft alone. I'd much rather it gave >$2e6/year to providing jobs for locally employed people, rather than buying someone another yacht.

Curious how many students and faculty at your university are served by that $2m/year?

I don’t disagree with you, but if I were the CTO of a 10000+ seat organization, and a Microsoft/Google/etc told me they could provide email, storage, sharing/collaboration, office apps, security etc for a few bucks per user / month… that’s a pretty compelling deal.

Re: The implementation of the UK Covid-19 dashboard

#56

Post by xbar » Sun, Jan 23, 2022, 3:01 PM UTC

mmcnl wrote: post #43

trebligdivad wrote: post #5

I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…

I agree. For example, the Dutch corona dashboard (coronadashboard.rijksoverheid.nl) is a statically rendered dashboard using Next that gets updated daily. No backend and it's super fast. Maybe I'm not objective because I'm Dutch myself, but from both a user-facing and technical perspective I think the Dutch dashboard is by far the best corona dashboard in the world. It's very fast, has a lot of detailed visualization…

Is there a dashboard of dashboards somewhere?

Re: The implementation of the UK Covid-19 dashboard

#57

Post by benbristow » Sun, Jan 23, 2022, 3:02 PM UTC

axiosgunnar wrote: post #11

> Government project > awarded to Microsoft Hey Europe, want to stop being several decades behind in IT compared to US/China? One simple trick: Ban FAANG from public procurement in Europe! It‘s a no-brainer really. Buy locally, ideally giving small companies and startups a chance. You will have to do it anyway very soon if you want your privacy laws to be taken seriously. There might be a couple of months of friction…

Get your point.

Those small companies and startups are going to end up using Microsoft/Amazon/Google for their hosting/cloud-services anyway so FAANG still win in the end.

Re: The implementation of the UK Covid-19 dashboard

#58

Post by odiroot » Sun, Jan 23, 2022, 3:03 PM UTC

axiosgunnar wrote: post #11

> Government project > awarded to Microsoft Hey Europe, want to stop being several decades behind in IT compared to US/China? One simple trick: Ban FAANG from public procurement in Europe! It‘s a no-brainer really. Buy locally, ideally giving small companies and startups a chance. You will have to do it anyway very soon if you want your privacy laws to be taken seriously. There might be a couple of months of friction…

> Ban FAANG from public procurement in Europe!

Is it any better if SAP / Telekom get similar contracts (what usually happens in Germany)?

Re: The implementation of the UK Covid-19 dashboard

#59

Post by jll29 » Sun, Jan 23, 2022, 3:07 PM UTC

mdm12 wrote: post #52

nojito wrote: post #30

Earlier quoted context omitted.

And a bunch of F# https://github.com/publichealthengland/coronavirus-dashboard...

Nice catch! It surprises me how much more popular F# is in Europe compared to the US. I finally got a professional F# gig in the states (\o/), but there were very few options. It makes me wonder, are universities in Europe providing a more functional-first approach to CS education, or is something else going on?

Never seen a job ad on anything sharp other than C# in Europe.

There are occasionally LISP and Clojure jobs from what I can tell.

(It's also hard to find people on the talent side. I needed a Haskell developer with NLP skills in 2005, and could not find one so we had to port our codebase to Java.)

Re: The implementation of the UK Covid-19 dashboard

#60

Post by vlovich123 » Sun, Jan 23, 2022, 3:07 PM UTC

mslot wrote: post #42

trebligdivad wrote: post #5

I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…

The dashboard has to deal with a complex data integration problem, with different sources with differences in completeness, accuracy, age, and granularity (at many levels), daily corrections in past data, changes in data structure and semantics over time, large data volume, 4pm traffic spikes. Moreover, an API that allows you to select different metrics for different areas. Being able to simply write a SQL query or u…

See the other comment about the Dutch dashboard. Covid data isn’t changing that quickly. Having the frontend render something more static simplifies the design. No sql queries are even needed and you don’t need to scale out your database.