Numerical accuracy

The filter computes in complex64 and reports one peak per bin. A float64 numpy correlation of the same inputs is the answer it has to agree with. This sweeps injected SNR from pure noise to 1000 and measures the disagreement -- run while this page was built, not cached.

2.8e-07
worst relative error, any point
4.8
float32 ULPs at worst
100%
lowest index agreement
statistic
Relative error vs a float64 correlation (median)5e-81e-71.04.05.06.08.012.020.050.0200.01000.0injected SNR (0 plotted at 1)relative errorn = 1024n = 4096n = 16384n = 65536float32 eps
Distribution of relative error (7680 trials, all lengths and SNRs)044087913191758float32 eps1e-92.2e-94.6e-91e-82.2e-84.6e-81e-72.2e-7relative error against a float64 correlationtrials
The distribution is a single pile a few float32 epsilons wide with no tail. That is the claim the percentiles above cannot make on their own: a median of 5e-8 is consistent with a tight pile or with a narrow core plus occasional large errors, and those are different statements about the arithmetic.

What is being compared

ReferenceThe whole correlation in float64: |IFFT(data * conj(template))| * n, computed by numpy on the identical inputs.
ErrorThe reported magnitude against that reference evaluated at the lag the filter reported. That isolates arithmetic from tie-breaking: comparing peak magnitudes instead would blame the filter whenever two nearby lags swapped places.
Index agreementSeparately, how often the reported lag is the float64 argmax. Where two lags sit within rounding of each other either is a correct answer, so this is reported rather than asserted -- but over random noise exact ties are vanishingly unlikely, and it is 100%% here.
InputComplex Gaussian noise with one copy of the template injected at a random lag, as a phase ramp on the spectrum -- exactly a circular shift, so the signal lands where intended with no resampling error of its own. 192 trials per point.
The curves are flat in SNR, and that is the result. A relative error that tracked the signal would mean something was computed absolutely and then divided -- invisible on noise, and growing on exactly the loud events a search most needs to get right. Injected SNR spans 0 to 1000 here, so a proportional term would show as three orders of magnitude of growth. tests/test_precision.py asserts both this and the absolute bound.
All numbers (40 rows)
ninjected snrtrialsmedianp90worstULPs at worstindex agreement
102401925.67e-081.20e-071.84e-073.1100%
102441925.59e-081.33e-071.92e-073.2100%
102451925.22e-081.22e-072.33e-073.9100%
102461925.02e-081.20e-071.84e-073.1100%
102481926.18e-081.18e-072.14e-073.6100%
1024121925.87e-081.37e-071.90e-073.2100%
1024201925.58e-081.41e-072.48e-074.2100%
1024501925.50e-081.23e-071.90e-073.2100%
10242001926.96e-081.44e-072.23e-073.7100%
102410001927.17e-081.33e-072.11e-073.5100%
409601925.35e-081.32e-072.16e-073.6100%
409641924.81e-081.25e-072.36e-074.0100%
409651926.03e-081.19e-071.59e-072.7100%
409661925.32e-081.23e-072.33e-073.9100%
409681925.97e-081.38e-072.08e-073.5100%
4096121925.15e-081.22e-071.62e-072.7100%
4096201926.03e-081.36e-072.83e-074.8100%
4096501926.98e-081.44e-072.39e-074.0100%
40962001926.40e-081.47e-072.11e-073.5100%
409610001926.67e-081.22e-071.80e-073.0100%
1638401925.29e-081.27e-072.08e-073.5100%
1638441926.05e-081.47e-072.76e-074.6100%
1638451925.94e-081.27e-072.52e-074.2100%
1638461925.68e-081.39e-072.29e-073.8100%
1638481925.20e-081.33e-072.06e-073.5100%
16384121925.69e-081.25e-071.95e-073.3100%
16384201926.02e-081.50e-072.17e-073.6100%
16384501926.46e-081.35e-072.21e-073.7100%
163842001928.57e-081.59e-072.10e-073.5100%
1638410001927.26e-081.39e-071.91e-073.2100%
6553601926.10e-081.47e-072.18e-073.7100%
6553641927.42e-081.56e-072.25e-073.8100%
6553651927.47e-081.55e-072.58e-074.3100%
6553661927.05e-081.67e-072.48e-074.2100%
6553681926.61e-081.53e-072.24e-073.8100%
65536121927.38e-081.51e-072.26e-073.8100%
65536201928.63e-081.79e-072.35e-073.9100%
65536501929.13e-081.65e-072.44e-074.1100%
655362001929.55e-081.76e-072.82e-074.7100%
6553610001929.87e-081.61e-072.32e-073.9100%
← See it work
Hardware comparison →