Are you saying that the 2, 3 & 4 networks are NOT on any SSG interface but you simply have a route forwarding that traffic to another router or firewall?
If so, what is the return route situation for traffic coming from the ISP to these subnets?
Is the SSG the default gateway or route for those devices?
If not, you might need to source nat the traffic you forward to them from the ISP.