Any process that stops, prevents, or reduces the frequency, rate, or extent of cell adhesion mediated by integrin.