I would guess that the misconfiguration is either with your external firewall (between Security Server and Connection Server) or that there is a problem with your configuration of the firewalls on the servers themselves. The firewall must be running on both servers.
1. Check your external firewall configuration from the rules here - http://pubs.vmware.com/view-51/topic/com.vmware.ICbase/PDF/view-51-security.pdf (page 19). Sometimes people forget UDP 4500 and UDP 500.
2. Check that the firewall is enabled for the specific profile on the Connection Server and Security Server. We've seen cases where the customer has a profile which disables the firewall/IPsec.
3. Go through the steps in "Prepare to Upgrade or Reinstall a Security Server" on your Connection Server to allow installation of the Security Server. See page 57 here http://pubs.vmware.com/view-51/topic/com.vmware.ICbase/PDF/view-51-installation.pdf Also, set a pairing password.
After carefully doing these 3 steps, repeat your Security Server installation and this should get you up and running.
If it still doesn't work in your environment, raise a support request and someone will walk you through it.
Let us know what it was as it will help others. Thanks.
Mark