idr: reduce the unneeded check in free_layer()