Interacting selectively and non-covalently with a pyrimidine nucleobase, an organic nitrogenous base with a pyrimidine skeleton.