derickso: Mind if I send you a PM? I don't want to derail the thread but am curious about a few things. I though about including my HTPC (Win 7 - MCE for live tv with HDHomeRun tuners and XBMC for stored content) as a VM but was cautious about stability and such.
jhusen: Maybe I did not give the sytem enough time to do the switch over and was being to impatient 
Then again, I was doing this until 2 in the morning as I don't get too much time otherwise with the wife and baby. As for passing the wrong USB controller, I did that already. I installed ESXi on a USB stick and the first one I tried was the lucky one with the install on it.