Any process that modulates the frequency, rate or extent of the passage or uptake of molecules by the mitochondrial membrane.