Fix overflow when negating minimum int16_t value in precoding
This closes #955 (closed)
It prevents undefined behavior caused by the minimum int16_t value (-32768) in two's complement representation. Because possible negation by multiplying with -1 during precoding cause an overflows since +32768 cannot be represented in 16-bit signed integers and end up again in -32768 because of overflow, here reveald by using the SIMD functions.
Workaround we use a slightly smaller number than 1 in Q15 Format to prevent this error. Q15 representation of (integer value -32767) -1 + 2^-15 = -0.999969482421875
See Note
c16_t convert_precoder_weight(double complex c_in)
{
return (c16_t) {.r = round(SHRT_MAX*creal(c_in)), .i = round(SHRT_MAX*cimag(c_in))};
}
Thanks to @lthomas and @mjoang_oai for the support here.
Edited by Robert Schmidt