Interacting selectively and non-covalently with xanthophylls, any of several neutral yellow to orange carotenoid pigments containing oxygen.