cxgb4: fix large delays in PTP synchronization