btrfs: sync ioctl, handle errors after transaction start