NR UE loses synchronization

I've recently noticed that NR UE would lose synchronization with gNB a few minutes after connecting. Further debugging revealed that MR !2893 (merged) which modified the time synch is the problem because when I revert that merge, there is no problem.

Steps to reproduce

1. Run the gNB with USRP
2. Run UE with USRP
3. Wait for few minutes and the UE would go out sync and finally stop