1. Check your external firewall configuration from the rules here - http://pubs.vmware.com/view-51/topic/com.vmware.ICbase/PDF/view-51-security.pdf (page 19). Sometimes people forget UDP 4500 and UDP 500.
-- Checked.
2. Check that the firewall is enabled for the specific profile on the Connection Server and Security Server. We've seen cases where the customer has a profile which disables the firewall/IPsec.
-- Double Checked.
3. Go through the steps in "Prepare to Upgrade or Reinstall a Security Server" on your Connection Server to allow installation of the Security Server. See page 57 here http://pubs.vmware.com/view-51/topic/com.vmware.ICbase/PDF/view-51-installation.pdf Also, set a pairing password.
-- Triple Checked.
I have given up. I finally opened a case with VMware Support. I'll keep you updated.
This View 5.1 installation kicked my butt with all the new "security" stuff like SSL, cert, etc.
When I installed 4.6 and then 5.0, it wasn't that difficult.
I just followed your Video at vimeo. 