The different "colors" of noise correspond to the altered frequency amplitudes of the random frequencies that make up the noise. White noise should be totally flat with an equal contribution of all frequencies. Brown noise has some of the highs rolled off. There's other colors like pink which have different characters.
The nature of the sound is still noise. To an untrained ear it might be hard to detect the different frequency reasponses and it is highly dependent on the quality of speaker you're listening to it on as each speaker imposes a frequency response curve on top of the noise itself.
If you want to detect the difference i'd recommend wearing headphones and switching back and forth between the two. There, you should be able to hear a difference assuming it's actual brown noise.
I use a website called mynoise.net which allows you to select from a variety of noises, one of which is white noise. The key to mynoise.net is that it includes an equalizer that allows you to "tune" your noise to the environment. When working in a loud library i will sit there with my headphones and mynoise.net on white rain and pull each equalizer knob up and down until it perfectly covers the noise around me. I do that for the entire frequency table and I've got a perfectly tuned white noise/sound blocker for the people's voices around me. It's honestly amazing