Ten years ago, in grad school, I co-wrote a video conferencing module in VHDL [0]. I haven't touched HDLs since but here's what I remember very clearly from that project:
* It took > 3 minutes to compile our code.
* DMA made a huge performance difference once we figured it out.
* Realizing that we had to be one with the clock tick took a lot of time. Understanding synchronous based programming (if that's the term) was a paradigm shift for my partner and I.
* The utter delight when we got frames streaming across the wire. The performance (though over a LAN) was silky smooth and you could tell immediately this was different than the run-of-the-mill x86 desktop program.
[0] - http://www1.cs.columbia.edu/~sedwards/classes/2009/4840/repo...