The chemical reactions and pathways resulting in the breakdown of xanthophylls, oxygen-containing carotenoids.