The process in which the potential difference across the mitochondrial membrane is reduced from its steady state level.