clk: bcm: rpi: Run some clocks at the minimum rate allowed