Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The implementation of the UK Covid-19 dashboard

techcommunity.microsoft.com

Post reply on HN
Print view View original

71–80 of 118 posts

Re: The implementation of the UK Covid-19 dashboard

#71

Post by notahacker »

discordance wrote: post #55
azalemeth wrote: post #22

Earlier quoted context omitted.

I really can't agree with this more. My university gives >$2e6/year to Microsoft alone. I'd much rather it gave >$2e6/year to providing jobs for locally employed people, rather than buying someone another yacht.

Curious how many students and faculty at your university are served by that $2m/year? I don’t disagree with you, but if I were the CTO of a 10000+ seat organization, and a Microsoft/Google/etc told me they could provide email, storage, sharing/collaboration, office apps, security etc for a few bucks per user / month… that’s a pretty compelling deal.

And the idea of a university rolling its own office apps suite to provide local employment sounds like the sort of decision likely to end up in tears, or at least in students relying on Excel and Word on their own computers anyway

Re: The implementation of the UK Covid-19 dashboard

#73

Post by aembleton »

amelius wrote: post #53
StringyBob wrote: post #50

Earlier quoted context omitted.

Well, you can just grab the data very easily: https://coronavirus.data.gov.uk/details/download

Yeah, not what I prefer. As a programmer, I want to make demands about UX too, for a change ...

Why not build your own UX on top of the data?

Re: The implementation of the UK Covid-19 dashboard

#74

Post by alexchamberlain »

mslot wrote: post #42
trebligdivad wrote: post #5

I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…

The dashboard has to deal with a complex data integration problem, with different sources with differences in completeness, accuracy, age, and granularity (at many levels), daily corrections in past data, changes in data structure and semantics over time, large data volume, 4pm traffic spikes. Moreover, an API that allows you to select different metrics for different areas. Being able to simply write a SQL query or u…

I think the point was that the analyst needs to do that, but the frontend doesn't really need to - it could render a bunch of statically aggregated data and the spikes become another CDN problem.

Re: The implementation of the UK Covid-19 dashboard

#75

Post by mdm12 »

jll29 wrote: post #59
mdm12 wrote: post #52

Earlier quoted context omitted.

Nice catch! It surprises me how much more popular F# is in Europe compared to the US. I finally got a professional F# gig in the states (\o/), but there were very few options. It makes me wonder, are universities in Europe providing a more functional-first approach to CS education, or is something else going on?

Never seen a job ad on anything sharp other than C# in Europe. There are occasionally LISP and Clojure jobs from what I can tell. (It's also hard to find people on the talent side. I needed a Haskell developer with NLP skills in 2005, and could not find one so we had to port our codebase to Java.)

It's anecdotal, but on the F# Software Foundation's slack workspace[1], 4 out of the last 5 postings in the #jobs channel were in Europe.

No doubt, any company that picks a niche programming language as their business's lingua franca is taking a risk. For me, though, that is an indicator that they care about quality and do not have a culture of treating engineers as replaceable assembly line parts.

[1] https://fsharp.org/guides/slack/

Re: The implementation of the UK Covid-19 dashboard

#76

Post by lloydatkinson »

smarx007 wrote: post #51
lloydatkinson wrote: post #24

Earlier quoted context omitted.

I had the same thoughts and then it was confirmed how insane this setup is part way through: “At the time of writing, the Citus distributed database cluster adopted by the team on Azure is HA-enabled for high availability and has 12 worker nodes with a combined total of 192 vCores, ~1.5 TB of memory, and 24 TB of storage. (The Citus coordinator node has 64 vCores, 256 GB of memory, and 1 TB of storage.)” That’s beyon…

Did you look at the 3 different (non-trivial) APIs they are offering on top of the dashboard? Though I have a hard time understanding why use PostgreSQL instead of ClickHouse, for example.

No I didn’t tbh, I didn’t read much further. Notice how one sentence says Postgres was chosen because it was somebody’s preference

Re: The implementation of the UK Covid-19 dashboard

#77

Post by samhw »

lloydatkinson wrote: post #24
trebligdivad wrote: post #5

I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…

I had the same thoughts and then it was confirmed how insane this setup is part way through: “At the time of writing, the Citus distributed database cluster adopted by the team on Azure is HA-enabled for high availability and has 12 worker nodes with a combined total of 192 vCores, ~1.5 TB of memory, and 24 TB of storage. (The Citus coordinator node has 64 vCores, 256 GB of memory, and 1 TB of storage.)” That’s beyon…

> could be generated statically a couple of times a day

Hell, let's do some partial evaluation: just bake the computed HTML into the source code and recompile that a few times a day. No need to even read from a file when you can fetch it from rodata.

As for the reason why they did it this way, I assume it's a combination of CV-driven development along with the hackernoon-reading-junior-engineer-meets-cunning-salesperson effect which others have noted.

Re: The implementation of the UK Covid-19 dashboard

#78

Post by samhw »

DrBazza wrote: post #49
greatgib wrote: post #41

First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on... On the other side, I have the feeling that this thing that clearly over-engineered. Just look at data their diagram... If I'm not wrong there is one writer and multiple reader for the data, or at least multiple writers on one side and multiple readers on anothe…

> First, when you see public officials doing a blog post on "Microsoft.com" website instead of on a public website, you know that something fishy is going on... Maybe, I'm naive, or not cynical enough, but I just read this as a case study of customer using Azure to provide the general public with information in a robust fashion. In fact, if anything, the whole article is remarkably light on pushing Azure, and quite h…

> Even though gov.uk is actually a really good IT company

For anyone who's wondering, the relevant team here is GDS[0]. We hired a bunch of engineers from there at one of my previous companies - which was doing some quite gnarly technical work - and they were superb. I believe the US equivalent is 18F.

[0] The Government Digital Service in full, but no definite article for the initialism.

Re: The implementation of the UK Covid-19 dashboard

#79

Post by stickfigure »

axiosgunnar wrote: post #69
odiroot wrote: post #58

Earlier quoted context omitted.

> Ban FAANG from public procurement in Europe! Is it any better if SAP / Telekom get similar contracts (what usually happens in Germany)?

At least the tax euros stay in Europe, and you get local expertise, tech hubs etc

Don't Google Microsoft et al have offices in Europe?

Re: The implementation of the UK Covid-19 dashboard

#80

Post by samhw »

nojito wrote: post #67
snthd wrote: post #61

Elsewhere[0] Microsoft have redefined "Open Source" to not include the right to redistribute, or to host on a cloud service. So while there's nothing wrong here with calling an MIT project open souce, it's not inconsistent with their own definition, and useable as propaganda. [0] https://azure.microsoft.com/en-gb/services/developer-tools/d... >Is Azure Data Studio open source? >Yes, the source code for Azure Data Stu…

How is the most common open source license not open source?

I mean, whatever the truth may be, you're begging the question somewhat by calling it an 'open source license'.
Post reply on HN