The aggregation, arrangement and bonding together of a set of components to form a mitochondrial respiratory chain complex.