Any process that stops, prevents, or reduces the frequency or rate of initiation of conjugation with cellular fusion.