32bit support isn't just bad bloat due to larger pointers.
Only doing 32bit and not doing amd64/x86_64 is also leaving a lot of performance on the table:
x86 traditionally has very few registers to work with, amd64 gives you quite a few more general purpose registers, so your hand-fiddled assembler will be faster, and if you use compiled code, your compiler will have an easier time with optimizations, faster function calls due to more register arguments, less register-to-memory spills and more execution station parallelism.
Memory use is also an important point, your 32bit application will be limited to at best 4GB per process, usually more like 2GB (due to the kernel/user boundary at 2/2GB, absent weird tricks). With games currently being multiple times that size, you have to swap out assets and load new ones very frequently, which is why loadpoints are getting more common and level design has to add more corridors and doors to support those load points and segment level data into 2GB chunks.
And the newer processor features, especially SIMD stuff like AVX and SSE work better with 64bit registers, if they work at all. Many support libraries that can utilize SIMD only do so in 64bit versions. Some features aren't even available if the hardware is in 32bit mode. And compilers can do automatic SIMD conversions, but those work far better (if they work at all) in x86_64 mode because of the insufficient number of registers in 32bit mode for copying around and splitting up stuff from a SIMD register.
I think the real reason that especially game developers are opposed to amd64 is that they used to be stuck on 32bit on Windows for a long time and now have to be dragged into the future by force ;)