Configured memory : 4 GB
Limit: 512 MB
This is bad. You are, in fact, lying to the operating system, telling it you have 4GB of memory when you only have 512MB. You will NEVER get 4GB of physical memory with this configuration - the other 3.5GB will always come from the ESXi swap file so your performance is going to be horrible unless the swap is on SSD or something like a FusionIO card.
If you really only have 512MB of memory to give to the guest, then configure it for 512MB but don't apply a limit.