Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cleaning Radar Images using Neural Nets and Computer Vision

blog.forecast.io

Post reply on HN
Print view View original

1–10 of 21 posts

Re: Cleaning Radar Images using Neural Nets and Computer Vision

#4

Post by m_mueller »

Hey there. I've done a GPGPU implementation of a physical weather prediction model[1]. I'm interested in how you get your weather data. Do you use WSM for modeling? Also, have you considered licensing out Dark Sky for data assimilation purposes to weather agencies? What I hear from the weather researchers this is one of the hardest aspects of this field. I think that once you have a product that's able to integrate an automated feedback loop into existing weather models for all kinds of weather data, you've basically won the weather game.

[1]https://github.com/muellermichel/Hybrid-Fortran

Re: Cleaning Radar Images using Neural Nets and Computer Vision

#5

Post by MBCook »

thegrossman wrote: post #3

Seeing this on HN, I realize I didn't go into a lot of technical detail in the blog post (we still aren't quite sure who our audience is with the Forecast blog). So if anyone has any specific questions, ask away.

I thought it was great.

Could you comment on why all the radar noise in the sample image seems to be almost perfectly confined to the eastern half of the US? Is that just an artifact of the time of day (perhaps the sun was setting the in middle of the US at the time)?

I'd love to know more about the neural network too, but I have no experience with them so I'm not even sure what to ask.

Re: Cleaning Radar Images using Neural Nets and Computer Vision

#6

Post by saidajigumi »

m_mueller wrote: post #4

Hey there. I've done a GPGPU implementation of a physical weather prediction model[1]. I'm interested in how you get your weather data. Do you use WSM for modeling? Also, have you considered licensing out Dark Sky for data assimilation purposes to weather agencies? What I hear from the weather researchers this is one of the hardest aspects of this field. I think that once you have a product that's able to integrate a…

Regarding data, have you looked at the API[1] they provide, also used to power Forecast.io and Dark Sky?

[1] https://developer.forecast.io/

Re: Cleaning Radar Images using Neural Nets and Computer Vision

#7

Post by thegrossman »

MBCook wrote: post #5
thegrossman wrote: post #3

Seeing this on HN, I realize I didn't go into a lot of technical detail in the blog post (we still aren't quite sure who our audience is with the Forecast blog). So if anyone has any specific questions, ask away.

I thought it was great. Could you comment on why all the radar noise in the sample image seems to be almost perfectly confined to the eastern half of the US? Is that just an artifact of the time of day (perhaps the sun was setting the in middle of the US at the time)? I'd love to know more about the neural network too, but I have no experience with them so I'm not even sure what to ask.

Exactly. Those enormous Bagel Blobs tend to occur after the sun goes down (and only in the warmer months. Two months ago, they weren't there).

The image was generated around 10:30 PM EST, so they hadn't had time to propagate further west.

Regarding neural nets, I have to admit that I didn't have much experience with them either before working on Dark Sky. They do, however, seem to be a little more forgiving than other kinds of classifiers (Support Vector Machines, and the like).

Check out the examples for the FANN C library: http://leenissen.dk/fann/wp/

Or in ruby: https://github.com/tangledpath/ruby-fann

Re: Cleaning Radar Images using Neural Nets and Computer Vision

#8

Post by josephhardin »

This is neat, though there are far better methods for doing this type of thing. Now that the Nexrads are dual-pol, you can look at the correlation between the horizontal and the vertical polarizations to filter out the vast majority of the noise.

The noise you are filtering out is primarily insect returns. If you watch it, it will get worse at some times of the day rather than others. If inversions happen, you'll also see some beam bending that can cause you to pick up powerlines/roads, etc. You can verify it is mostly bugs by looking at the differential reflectivity. The differential reflectivity(difference of horizontal and vertical returned powers) will be somewhat random for ground clutter, and higher values for bugs.

With this approach, without knowing exactly how you do it, I'd be worried that it would have the tendency to filter out initial formations of stratiform clouds, and just leave convective formations. I can point you to some theory on how a lot of the filtering is done for research in atmos if you'd like, feel free to message me(I'm working on my Ph.D. in weather radar).

Re: Cleaning Radar Images using Neural Nets and Computer Vision

#9

Post by geofffox »

MBCook wrote: post #5
thegrossman wrote: post #3

Seeing this on HN, I realize I didn't go into a lot of technical detail in the blog post (we still aren't quite sure who our audience is with the Forecast blog). So if anyone has any specific questions, ask away.

I thought it was great. Could you comment on why all the radar noise in the sample image seems to be almost perfectly confined to the eastern half of the US? Is that just an artifact of the time of day (perhaps the sun was setting the in middle of the US at the time)? I'd love to know more about the neural network too, but I have no experience with them so I'm not even sure what to ask.

The color scale is reduced, but that seems to be ground clutter caused by a temperature inversion. It forms after dark and usually breaks up after a few hours of daylight.

I am a meteorologist and have used radar for nearly thirty years. I have yet to see a foolproof system -- and I've looked for them.

Re: Cleaning Radar Images using Neural Nets and Computer Vision

#10

Post by josephhardin »

josephhardin wrote: post #8

This is neat, though there are far better methods for doing this type of thing. Now that the Nexrads are dual-pol, you can look at the correlation between the horizontal and the vertical polarizations to filter out the vast majority of the noise. The noise you are filtering out is primarily insect returns. If you watch it, it will get worse at some times of the day rather than others. If inversions happen, you'll als…

Also I didn't see it mentioned, but are you pulling in level 2 or level 3 data?
Post reply on HN