Any process that modulates the frequency, rate or extent of cellular response to amino acid starvation.