Live data from Hacker News

The IceCube Neutrino Detector at the South Pole Hits Paydirt

spectrum.ieee.org

21–30 of 109 posts

Re: The IceCube Neutrino Detector at the South Pole Hits Paydirt

#21
post #3

Earlier quoted context omitted.

Nah, the IceCube neutrinos are probably photonuclear with local photons rather than CMB photons.

Is this how other people hear software developers when we talk about code?

Except that we physicists can usually understand you :)

(since many of us spend most of our time writing code too...)

Re: The IceCube Neutrino Detector at the South Pole Hits Paydirt

#22
post #17

Earlier quoted context omitted.

> just to insert a CD and press reset That may have been all he did that time, but imagine if stuff had gone sideways, he'd have to troubleshoot it from Antarctica! I thought working from home was bad...

I work on an experiment at the south pole (not IceCube). sshing into fix stuff is painful, even when the "good" satellite is up. Protip: pressing up several times and then enter is very dangerous since the command that you will actually run may not have appeared on your screen yet.

Similar experience to being a hacker-cracker 20 yrs ago, bouncing off servers around the world over a modem connection to telnet into some poor SunOS 4 boxes in South Korea or at GSFC (NASA). Sometimes you had to wait minutes before anything appeared on the screen while TCP was doing its thing. God forbid if you accidentally (yp)cat’d some large file..:)

Re: The IceCube Neutrino Detector at the South Pole Hits Paydirt

#23
post #17

Earlier quoted context omitted.

> just to insert a CD and press reset That may have been all he did that time, but imagine if stuff had gone sideways, he'd have to troubleshoot it from Antarctica! I thought working from home was bad...

I work on an experiment at the south pole (not IceCube). sshing into fix stuff is painful, even when the "good" satellite is up. Protip: pressing up several times and then enter is very dangerous since the command that you will actually run may not have appeared on your screen yet.

Jusr amoment ago learned about mosh.org that sounds like would be helpful for that kind of situations.

Re: The IceCube Neutrino Detector at the South Pole Hits Paydirt

#24
post #16
post #5

My coworker went down to the South Pole to install Debian on the IceCube cluster (funny story: it's hard to cool servers there because the air is really dry) ~14 years ago. Basically, a 48 hour trip both ways just to insert a CD and press reset. We were all jealous.

I would have thought that for the time and cost of doing that, they could just ship the CD with a booklet of photographic instructions. An instruction manual SO SIMPLE that even a scientist at the South Pole could understand it. Or they could make a video chat call and it could be explained. (I don't mean scientists are so dumb, but that they might have Antarctic Stare...) https://en.wikipedia.org/wiki/Polar_T3_syndr…

Former IceCube winterover here. Things at Pole often don't go smoothly, and detector uptime is very valuable considering the cost of building and operating it.

Video chat isn't a realistic option - not the greatest connectivity down there...

Re: The IceCube Neutrino Detector at the South Pole Hits Paydirt

#25
post #7

Earlier quoted context omitted.

Yes. And the same goes with finance people, and people in lots of other fields. Jargon rich conversations that outsiders think is mumbo-jumbo.

I'm the author of the Spectrum article: this is the exact reason why I get paid for my job :)

You did a great job of inspiring me to understand something I have, literally, not even the foggiest iota of a clue about. I finished your article in awe at the human species.

Re: The IceCube Neutrino Detector at the South Pole Hits Paydirt

#26
post #7

Earlier quoted context omitted.

Yes. And the same goes with finance people, and people in lots of other fields. Jargon rich conversations that outsiders think is mumbo-jumbo.

I'm the author of the Spectrum article: this is the exact reason why I get paid for my job :)

>"After 3.9 billion years of hurtling unhindered through the vast reaches of the universe, a ghostly neutrino particle died on 22 September 2017. It was annihilated when it collided with an atom in the frozen darkness two kilometers beneath the surface of the south polar ice cap."

Your article says the neutrino came from far away, which disagrees with that post that says it was probably "photonuclear with local photons".

Re: The IceCube Neutrino Detector at the South Pole Hits Paydirt

#27
post #20

Earlier quoted context omitted.

> just to insert a CD and press reset That may have been all he did that time, but imagine if stuff had gone sideways, he'd have to troubleshoot it from Antarctica! I thought working from home was bad...

Yes, that's why he was sent. In case something went wrong, he was qualified to fix it in production. I've come to an appreciation for people who have jobs where they don't do much most of the time, but when the shit hits the fan, they do exactly what was required.

You can think about it much like load managing servers:

A job has a task load that varies over time. Those tasks also have some maximum latency before Bad Things happen. Thus, you need enough worker capacity to handle the peak load. You can allocate new workers on demand, but that also has a certain warm-up time which can be problematic if it's lower than the maximum latency.

For most white-collar service-oriented jobs, the variance of that task load is quite smooth. If you're a software dev not working directly on a production service, there aren't that many urgent surprises. Also, the max latency is really high — many tasks can be put off till tomorrow, next sprint, etc.

That makes it easy to allocate just enough workers to handle your average task load, keep them busy all day, and rarely need to spin up or spin down new ones.

But some jobs just have really high variance or really low latency. No one knows when a patient is going to show up at a trauma hospital but when one does, you need the surgeon on site right now and not somewhere stuck in traffic.

For that kind of work, the logical thing to do then is to have spare idle worker capacity. It seems inefficient, but it's cheaper than the cost of missing your latency targets when a spike happens.

You can reduce the idle capacity needed by reducing variance. For example if you flew all your trauma patients to one hospital, that actually lets you allocate surgeons more efficiently. Because then at that point you have spikes frequently enough that the aggregate number of patients in a given day becomes more consistently predictable. Instead of one surgeon each sitting on their thumbs most of the day at ten city hospitals, you get three surgeons that reliably have a couple of patients a day at the regional hospital. (But of course there is the overhead of getting all the patients there.)

Also, you can reduce idle capacity by lowering warm-up time. That's why doctors have pagers — it reduces the time between a patient coming in and the doctor being ready to help.

Or you can increase the max latency, though this can be hard based on the nature of the work. For example, if you're better able to stabilize patients, you may be able to wait longer until the doctor is on-site.

In this case, the "install Debian" job has really high variance. 99% of the time it's a click, 1% of the time, you've gotta know some Linux internals. The max latency is low because you don't want the IceCube array non-operational for long. And the warm-up time is really high — fly someone out to the South Pole.

So the reasonable response is to throw money at idle capacity and send someone out pre-emptively.

Re: The IceCube Neutrino Detector at the South Pole Hits Paydirt

#28
post #26

Earlier quoted context omitted.

I'm the author of the Spectrum article: this is the exact reason why I get paid for my job :)

>"After 3.9 billion years of hurtling unhindered through the vast reaches of the universe, a ghostly neutrino particle died on 22 September 2017. It was annihilated when it collided with an atom in the frozen darkness two kilometers beneath the surface of the south polar ice cap." Your article says the neutrino came from far away, which disagrees with that post that says it was probably "photonuclear with local photo…

That's exactly why the author gets paid. By local I meant near the source, not near us.

Re: The IceCube Neutrino Detector at the South Pole Hits Paydirt

#29
post #7

Earlier quoted context omitted.

Yes. And the same goes with finance people, and people in lots of other fields. Jargon rich conversations that outsiders think is mumbo-jumbo.

I'm the author of the Spectrum article: this is the exact reason why I get paid for my job :)

You did great. Precise, informative, and evocative. Really fun to read.
Post reply on HN