61–70 of 172 posts
The asteroid detection capability is amazing: https://rubinobservatory.org/news/rubin-first-look/swarm-ast...
Among many other uses: https://m.youtube.com/watch?v=h6QYjNjivDE
Jesus H Christ, the Universe is big.
Earlier quoted context omitted.
I've followed this project for over a decade and the amount of data they are moving around is fairly routine, given their budget size and access to computing and networking resources. The total storage (~40-50PB) is pretty large, but moving 10TB around the world isn't special engineering at this point.
It's not about the size of the data in bytes, it's also the amount of changes that need to be detected and alerts that need to be sent out (estimated at millions a night). Keep in mind the downstream consumers of this data are mostly small scientific outfits with extremely limited software engineering budgets.
Earlier quoted context omitted.
It's not about the size of the data in bytes, it's also the amount of changes that need to be detected and alerts that need to be sent out (estimated at millions a night). Keep in mind the downstream consumers of this data are mostly small scientific outfits with extremely limited software engineering budgets.
Again, nothing special. The small outfits aren't going to be doing the critical processing.
Here's the SDSS view[0] of this featured[1] section from the Virgo Cluster, in comparison, to put the staggering depth of these exposures in their proper context, [0] https://aladin.cds.unistra.fr/AladinLite/?target=12%2026%205... [1] https://rubinobservatory.org/gallery/collections/first-look-...
With an opacity slider, for easy comparison: https://aladin.cds.unistra.fr/AladinLite/?baseImageLayer=CDS...
The amount of data this thing will be putting out every night is insane. For years now the community has been building the infrastructure to be able to efficiently consume it for useful science, but we still have work to do. Anyone interested in the problem of pipelining and distributing 10s of TB of data a night should check out the LSST and related GitHubs.
Is this not the same problem high resolution spy satellites have? Seems like a fair bit of crossover at least?
Earlier quoted context omitted.
Again, nothing special. The small outfits aren't going to be doing the critical processing.
…they do the science
Here's the canonical example: https://home.cern/science/computing/grid and a lab that didn't have enough horsepower using a different grid: https://osg-htc.org/spotlights/new-frontiers-at-thyme-lab.ht...
Personally, I have pointed the grid folks (I used to work on grid) towards cloud, and many projects like this have a tier 1 in the cloud. The data lives in S3, metadata in some database, and use cloud provider's notification system. The scientists work in adjacent AWS accounts that have access to those systems and can move data pretty quickly.
My God, it's full of stars
brings up that old paradox - should any line of sight ultimately end up at a star?
The wikipedia article is quite good - https://en.wikipedia.org/wiki/Vera_C._Rubin_Observatory (Edit: Treasure trove of details in the references if any of your interests are adjacent to this) The image of the woman holding the model of the sensor is nice because it includes a moon for scale. Question I was curious about is whether or not the focal plane was flat (it is). This is an interesting tidbit: > Once images a…
>The prompt products are alerts, issued within 60 seconds of observation, about objects that have changed brightness or position relative to archived images of that sky position. Transferring, processing, and differencing such large images within 60 seconds (previous methods took hours, on smaller images) is a significant software engineering problem by itself.[64] >This stage of processing will be performed at a cla…