There's an absurd number of crops that would effectively disappear from American (and probably even global) diets if California started to downsize its agriculture. The Central Valley is very unique in having both fertile soil and year-round reasonably-temperate weather. That's a resource which - if anything - ought to be prioritized and expanded to as high of a degree as possible.
Water is quite literally the only issue, and it's a very recent one. It's also an easy one to solve: follow Israel's lead and build desalinization plants up the wazoo. That, or pray that we mountain folks keep getting decent winters to sustain the summers. Pushing ranching (which is disproportionately-water-intensive) out-of-state is another option, as is figuring out ways to reduce ranching's water impact.
As for the low proportion of California's economy to which agriculture contributes, it's worth remembering that agriculture doesn't exactly have high margins. California's farms in particular are also heavily staffed with undocumented labor (which I reckon will skew a lot of data in all sorts of crazy directions).
Put simply: crops ain't the problem, and scapegoating the "dumb and dirty" farmers was already a tired trope five years ago, let alone now.