If traffic is initiated from SpokeB (PA) side then it works, if it's initiated from SpokeA to SpokeB then it fails. Can you please ensure that the bidirectional policy is configured properly with the correct subnets,service,application etc
I think, if you see the sessions details for specific session ID in the PA then at bottom it gives some reason why traffic is failing.
Thanks,
Vikas