gpio: 74x164: Fix crash during .remove()