Have you checked if ESXi has actually booted? It looks like "Multiboot could not setup the video subsystem" is just a warning message, and it should continue to try to boot the OS (without any further updates to the display) regardless. There's no guarantee that it will succeed in booting the OS, but it looks like it will try to do so... You might need to hook up another monitor to get it configured (e.g. if you want to enable SSH), but after that it might not need a monitor at all. (Disclaimer: I have never tried this! 
)
Cheers,
--
Darius