Hi,
VPN comes up if you replace your WI-FI modem/router with the same ISP, right? Please check your non-working modem/router to see if they allow port 4500 traffic also. As there will be NAT-T and return traffic is NATed correctly and port 500 and 4500 both are open or not.
Thanks,
Vikas