vxlan: fix out of order operation on module removal