If the site can render responsively, is there a reason it can't be zoomed?
Both the reasons looking for zoom, and your reasons for not providing it seem reasonable. Is there some technical barrier for not enabling zoom?
Zoom is a first-class gesture on mobile, it is the second one after scrolling that you always expect to be there. On a desktop, you generally have more desktop than you need for a window, so it is possible to resize the window to adjust the size of the viewport. Responsive sites work nicely in the case. On mobile, there is no "window" and the device's dimensions themselves are the viewport. You can reorient the device to enable changing the width, but without zoom, you are taking away any ability to change the viewport.
Maybe zoom isn't a critical feature, but it is part of how the devices work. As a result, it is an expected interaction for those on mobile. Have you ever used a piece of desktop software where you couldn't resize the window - it drives you crazy. Ignoring expected behavior is contrary to good usability.