Interacting selectively and non-covalently with a phosphorylated amino acid residue within a protein.