Any process that modulates the frequency, rate or extent of delayed rectifier potassium channel activity.