Any process that modulates the frequency, rate or extent of vascular smooth muscle cell differentiation.