This is neat, though there are far better methods for doing this type of thing. Now that the Nexrads are dual-pol, you can look at the correlation between the horizontal and the vertical polarizations to filter out the vast majority of the noise. The noise you are filtering out is primarily insect returns. If you watch it, it will get worse at some times of the day rather than others. If inversions happen, you'll als…
Dual pol wasn't deployed when we first started Dark Sky. Now that it's becoming available, it's definitely going to make our job a lot easier!
Not knowing much about it, though, it'll take some time for us to become comfortable with the data.
This is neat, though there are far better methods for doing this type of thing. Now that the Nexrads are dual-pol, you can look at the correlation between the horizontal and the vertical polarizations to filter out the vast majority of the noise. The noise you are filtering out is primarily insect returns. If you watch it, it will get worse at some times of the day rather than others. If inversions happen, you'll als…
Dual pol wasn't deployed when we first started Dark Sky. Now that it's becoming available, it's definitely going to make our job a lot easier! Not knowing much about it, though, it'll take some time for us to become comfortable with the data.
Well it's very impressive that you got that performance without using any dual-pol parameters. Abstract submission for the AMS radar conference is next week, and I'm sure they'd be interested in this.
http://www.ametsoc.org/MEET/fainst/201336radar.html
Meteorologist turned HN junkie here: this is really cool and I love the commenter suggesting using dual pol data to do the heavy lifting for you.
My question about your radar data is this; it's beautiful but I absolutely hate the color palette you are using, from a weather perspective - why not just stick with the 'standard' green-to-red colors we all know and love? The purples are different, for sure - but really hard to make sense of!
I thought it was great. Could you comment on why all the radar noise in the sample image seems to be almost perfectly confined to the eastern half of the US? Is that just an artifact of the time of day (perhaps the sun was setting the in middle of the US at the time)? I'd love to know more about the neural network too, but I have no experience with them so I'm not even sure what to ask.
Exactly. Those enormous Bagel Blobs tend to occur after the sun goes down (and only in the warmer months. Two months ago, they weren't there). The image was generated around 10:30 PM EST, so they hadn't had time to propagate further west. Regarding neural nets, I have to admit that I didn't have much experience with them either before working on Dark Sky. They do, however, seem to be a little more forgiving than othe…
Meteorologist turned HN junkie here: this is really cool and I love the commenter suggesting using dual pol data to do the heavy lifting for you. My question about your radar data is this; it's beautiful but I absolutely hate the color palette you are using, from a weather perspective - why not just stick with the 'standard' green-to-red colors we all know and love? The purples are different, for sure - but really ha…
The color palette of the images in the blog post was adjusted to highlight areas of precipitation at the expense of intensity discrimination. Dark Sky itself uses a wider range, but it's still a similar blue to purple to salmon to yellow color table.
The reason we went with that over the standard green-to-red color table is because I personally find the standard colors to be jarring and more confusing to the casual user because of the increased contrast between intensity bands (at least, it certainly was to me back when I was a casual weather app user!)
The tradeoff is that there isn't as fine an intensity discrimination at lower intensities. In theory I can see how that can be a problem, but I haven't noticed it in practice.
Dual pol wasn't deployed when we first started Dark Sky. Now that it's becoming available, it's definitely going to make our job a lot easier! Not knowing much about it, though, it'll take some time for us to become comfortable with the data.
Well it's very impressive that you got that performance without using any dual-pol parameters. Abstract submission for the AMS radar conference is next week, and I'm sure they'd be interested in this. http://www.ametsoc.org/MEET/fainst/201336radar.html
Also, correct me if I'm wrong (and I very well could be), but the most egregious noise -- the giant Bagel Blobs -- aren't insect reflections, but rather night-time temperature inversions. Can dual-pol correlation help identify those?
Seeing this on HN, I realize I didn't go into a lot of technical detail in the blog post (we still aren't quite sure who our audience is with the Forecast blog). So if anyone has any specific questions, ask away.
Did you try any other ML algorithms on your data before using neural nets? I know neural nets are getting trendy again but a simple SVM or a linear classifier can provide very good performance.
Interesting... as part of a little site I threw together, I have a ton of radar data for the region of Italy where I live, and have always thought it'd be fun to do something with it:
Seeing this on HN, I realize I didn't go into a lot of technical detail in the blog post (we still aren't quite sure who our audience is with the Forecast blog). So if anyone has any specific questions, ask away.
Did you try any other ML algorithms on your data before using neural nets? I know neural nets are getting trendy again but a simple SVM or a linear classifier can provide very good performance.
We did, but they didn't work quite as well. But that could absolutely be my lack of experience with how best to normalize and format the input vectors.
In the end, NNs seemed less "fussy", if that makes any sense.
Well it's very impressive that you got that performance without using any dual-pol parameters. Abstract submission for the AMS radar conference is next week, and I'm sure they'd be interested in this. http://www.ametsoc.org/MEET/fainst/201336radar.html
Also, correct me if I'm wrong (and I very well could be), but the most egregious noise -- the giant Bagel Blobs -- aren't insect reflections, but rather night-time temperature inversions. Can dual-pol correlation help identify those?
It's likely a combination of both. A quick way to discern would be to look at velocity, zdr, and rhohv. For ground clutter and returns caused by the inversion, you will expect to see very low velocities(close to 0) relative to the background precipitation. Additionally Zdr will end up looking like a roughly random field. If it is insects, then the velocity will roughly match the background precipitation, but you will have a high zdr(as insects look like very very oblate bags of water). In both cases you should get a drop in rho_hv, the correlation coefficient between the channels which will help to differentiate it from actual precipitation.
Also it looks like you're primarily concerned with pulling out the rainfall in several cases. For this, I'd look at the specific differential phase(K_DP) as it is a much better estimator of rainfall than reflectivity. In general, it is linearly proportional(exponential, with an exponent close to 1) to the rainfall rate. I'm not sure how good the nexrad estimators are for kdp though.