91–100 of 118 posts
I would have preferred to see it implemented as a spreadsheet in the cloud.
But then this would let you perform statistical analyses on _their_ data, I'm not sure they're such a big fan of that...
Elsewhere[0] Microsoft have redefined "Open Source" to not include the right to redistribute, or to host on a cloud service. So while there's nothing wrong here with calling an MIT project open souce, it's not inconsistent with their own definition, and useable as propaganda. [0] https://azure.microsoft.com/en-gb/services/developer-tools/d... >Is Azure Data Studio open source? >Yes, the source code for Azure Data Stu…
This has been an argument for at least 25 years that I've been around this stuff.
I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…
I had the same thoughts and then it was confirmed how insane this setup is part way through: “At the time of writing, the Citus distributed database cluster adopted by the team on Azure is HA-enabled for high availability and has 12 worker nodes with a combined total of 192 vCores, ~1.5 TB of memory, and 24 TB of storage. (The Citus coordinator node has 64 vCores, 256 GB of memory, and 1 TB of storage.)” That’s beyon…
E.g. 12 worker nodes and 192 vCores means they've picked 16 core nodes. 1.5TB of memory across 12 nodes means 128GB per node. 24TB of storage is just 2TB per node.
So it's 12 relatively mid sized servers/VMs.
They could certainly do it with much less, and I have no interest in looking up what 12 nodes of that spec would cost on Azure, but at Hetzner it'd cost less than 1500 GBP/month including substantial egress. At most cloud providers the bandwidth bill for this likely swamps the instance cost, and the developer cost to develop this is likely many times the lifetime projected hosting cost even with that much overkill.
If they happen to have someone familiar with query caching and CDNs, I'm sure they could cut it significantly very quickly, and even an entirely average developer could figure out how to trim that significantly over time. But even at (low) UK government contract rates it's not worth much time to try to trim a bill like that much vs. just picking whatever the developers who worked on it preferred.
Earlier quoted context omitted.
I had the same thoughts and then it was confirmed how insane this setup is part way through: “At the time of writing, the Citus distributed database cluster adopted by the team on Azure is HA-enabled for high availability and has 12 worker nodes with a combined total of 192 vCores, ~1.5 TB of memory, and 24 TB of storage. (The Citus coordinator node has 64 vCores, 256 GB of memory, and 1 TB of storage.)” That’s beyon…
My suspicion is that since this has to do with COVID, there is no real limit on what the cost should really be. As for using the setup for other things, that seems less likely given this expensive setup.
Elsewhere[0] Microsoft have redefined "Open Source" to not include the right to redistribute, or to host on a cloud service. So while there's nothing wrong here with calling an MIT project open souce, it's not inconsistent with their own definition, and useable as propaganda. [0] https://azure.microsoft.com/en-gb/services/developer-tools/d... >Is Azure Data Studio open source? >Yes, the source code for Azure Data Stu…
BSDL, GPL, MIT License. This has been an argument for at least 25 years that I've been around this stuff.
Microsoft is defining their product, which you can't redistribute[0], as "Open Source".
[0] https://github.com/microsoft/azuredatastudio/blob/4012f26976...
I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…
I had the same thoughts and then it was confirmed how insane this setup is part way through: “At the time of writing, the Citus distributed database cluster adopted by the team on Azure is HA-enabled for high availability and has 12 worker nodes with a combined total of 192 vCores, ~1.5 TB of memory, and 24 TB of storage. (The Citus coordinator node has 64 vCores, 256 GB of memory, and 1 TB of storage.)” That’s beyon…
Alternatively, we're building https://www.polyscale.ai/ that is a good fit for this type of use case. It's a global database cache and integrates with Postgres/MySQL etc. We host PoP's globally so the database reads are offset and local to users.
Agree with the other comments in that this feels like a shiny use case to quote to other prospects, but all good :)
I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…
To add to this the scale of the data is presumably quite small as well. The geographical resolution is probably not super fine, there's only a handful of different kinds of data (deaths, vaccination whatnot) and the time resolution doesn't have to be too fine either (a day?). Even if you wanted to query it in very sophisticated ways you wouldn't need a database.
Earlier quoted context omitted.
Yes. This was never pure ONS data. It was always processed/massaged/tampered with. Although horrifyingly it did allow you to gauge the Westminster mindset and understand what Draconian measures they were planning to introduce sightly ahead of time because the data would have to reflect this when lord Boris got up on the podium...
Despite the downvotes the only incorrect statement here is that Boris has a lordship. Go export the ONS data yourself and analyse it unless your too lazy to see what I mean
If you're going to make such claims then the onus is you to provide the evidence.
Earlier quoted context omitted.
The dashboard has to deal with a complex data integration problem, with different sources with differences in completeness, accuracy, age, and granularity (at many levels), daily corrections in past data, changes in data structure and semantics over time, large data volume, 4pm traffic spikes. Moreover, an API that allows you to select different metrics for different areas. Being able to simply write a SQL query or u…
See the other comment about the Dutch dashboard. Covid data isn’t changing that quickly. Having the frontend render something more static simplifies the design. No sql queries are even needed and you don’t need to scale out your database.
I guess that implies that using Next for a "static site" is not a great idea.
I do wonder if the power of a distributed database is really needed here; it gets ~1 update a day, so there's no need to have clever consistency stuff. Most of the queries relate to either today's data (you open the map and zoom in to see how doomed your area is today), or the graphs showing a standard set of history (e.g. cases over the last year). You'd think you could extract that data to be static and not require…
I agree. For example, the Dutch corona dashboard (coronadashboard.rijksoverheid.nl) is a statically rendered dashboard using Next that gets updated daily. No backend and it's super fast. Maybe I'm not objective because I'm Dutch myself, but from both a user-facing and technical perspective I think the Dutch dashboard is by far the best corona dashboard in the world. It's very fast, has a lot of detailed visualization…