Hi,
A simple test to narrow down on a routing isue would be to NAT the traffic before it leaves the SSG.
On the policy that permits traffic from Site-B to the server, enable Src-NAT and use the SSGs interface IP to NAT the traffic. This will mask the Site-B subnet behind the SSGs internal interface IP.
If this works, you can either leave the NAT in place or sort out the routing issue - maybe route added on the server, pointing to SSG as the gateway to reach Site-B.