This will happen if the remote party does not send a 'mix-minus' from their sound board (audio mixer), or if their headphones are turned up too loud.

Video: What is Mix Minus / Cleanfeed?

All parties should always use headphones rather than speakers. Consumer type earbuds and similar will commonly bleed the incoming audio back onto their built-in microphone.