lib: add error checking to hex2bin