Just want to make sure I understand the descripton correctly.
You are saying you contact multiple remote sites via the polycom calls and only ONE remote site shows the issue.
What are the differences between that remote site and the working sites?
Are these all sites internal to you company or outside?
Internal company sites may have access to more ports for communications to work than external entities.
I think the first step here is to try to be certain which side we want to concentrate on for troubleshooting.