Based on observation, this udelay(20) was apparently too high and caused
subsequent failure to calibrate DDR when U-Boot was compiled with certain
toolchains. Lowering this delay fixed the problem.

Instead of permanently lowering the delay, calculate the correct delay
based on the original comment, that is, obtain EOSC1 frequency and use
it to calculate the precise delay.

Signed-off-by: Marek Vasut <marex@denx.de>