Yup the PCI Pinhole trick will get you key to more RAM on the VM.
ESXi it's self dosent require a card for output when running. At least it wont complain. You'll end up "with the VM powered off" at a point where you can no longer recover the host aka you'll see cnc-register or something like that at boot and it appear that the host isnt booting. At that point the devices has been allocated to a VM which is why passthrough works on a host with a single video card. With your setup having the 2nd video card in, you need to assign it to another VM to remove the host access. Or just dont plug a monitor in.
But I'm going to guess your talking about the console video or the VM video driver... Nope that driver has to stay but it dosent have to be enabled for output. Don't disable it in device manager. All you need to do is set your VM's primary monitor to the video card and you'll be good.
5.1 is still broken with USB passthrough. I tried that the other day but 5.0 u2 while it was unstable at first worked out on a fresh load of the host and I havent turned back since.