net: qca_spi: Avoid re-sync for single signature error