iavf: Add waiting so the port is initialized in remove