@liuliu - in the last graph, there's quite a jump from epoch 26 to 27. I'm curious to find out what might be causing this.
That's when learning rate changed to a smaller number. The graph mainly shows that with different initialization scheme, the network starts descending initially faster.