It's rude playing music when you're talking with someone. It will keep distracting you no matter how much you reduce the volume. They go off when I want to communicate with someone.
I used to play music while talking to someone. My own way is that I will reduce it to the bare minimum so that the person won't see me as a rude person, I will only nod my head as the music is playing but if the conversation is more interesting, I will pause the music totally.
I listen to music every day but there are some songs which I can't play outloud because of my daughter. There are some curse/F words in raps I don't want her hearing at a very young age.