Oddly, the first language I ever spoke to my wife was actually Japanese, the usual Japanese greeting for a first-time meeting, "はじめまして. どうぞよろしく." Over the years, we have grown strongly to prefer speaking English with each other (from initially mostly speaking Mandarin with each other) because she finds it more congenial to speak what is really on her mind when speaking English. That's a cultural difference between American culture and Taiwanese culture--greater frankness in family conversations in America.
We were quite resolute in speaking Mandarin whenever we were together, whether living in the United States or in Taiwan, as our two older sons were growing up. I would speak to them in English if I was alone with one or both of them. They switched effortlessly from English to Chinese or back as my wife was present or not.
Literacy is HARD to maintain in languages in which the relationship between speech and writing is more remote than in English, as is surely the case in Chinese. I know many, many, many native speakers of Chinese who received their primary, secondary, and even higher educations in Chinese-speaking countries who forget how to write many Chinese characters if they spend a lot of time abroad. Computer input used to be nasty for Chinese, but it is coming along now even in American versions of Windoze. Literature is also more interesting to read if it is uncensored by the government, which gives English-language literature an enormous worldwide draw. But it is definitely life-enriching and thought-provoking to know two or more languages to reasonably high proficiency, and I have enjoyed spending the majority of my life able to communicate in Chinese.
One considerable advantage for the child who grows up bilingual is learning yet more languages as second languages when an adult more readily than do adults who grew up speaking only one language. By diligent study of linguistics, after having some foreign language study (German) that began in elementary school, I acquired enough Chinese to work professionally as an interpreter and a translator, and have enough reading German to be able to do research in that language, and smatterings of other languages. But all the native bilingual members of my family do much better than I do per unit of time in learning languages, so they have many choices before them as occasion arises to learn other languages for various purposes. That helps with second-language acquisition of an understandable pattern of pronunciation, too.