Module

Fonctionnalités

v0.1 — Base

• Smoothing Savitzky-Golay sur lat/lon

v0.2 — Nettoyage avancé

• Détection de sauts GPS (outlier jumps via vitesse implicite)
• Marquage des blackouts GPS (séquences NaN > seuil)
• Correction de la dérive cold start (biais exponentiel décroissant)
• Smoothing adaptatif HDOP (fenêtre variable selon la qualité GPS)

v0.3 — Multi-rate

• Reconstruction de la vitesse à partir du GPS quand speed_mps est absent
• Calcul haversine entre points GPS valides non consécutifs (multi-rate)
• Forward-fill + lissage de la vitesse reconstruite

v0.4 — Adaptatif

• Détection automatique de la fréquence IMU et GPS
• Validation D0 en entrée (colonnes, ranges, monotonie)
• Forward-fill intelligent lat/lon (interpolation linéaire, max 30s gap)
• Distance hybride (speed * dt prioritaire, haversine en fallback)
• Paramètres de smoothing adaptatifs à la fréquence détectée
• Rapport de qualité enrichi dans les artifacts

Algorithmes clés

Détection de fréquence

• Fréquence IMU = 1 / médiane(dt) sur toutes les rows
• Fréquence GPS = 1 / médiane(dt) entre rows où lat/lon changent
• Multirate = IMU_Hz > GPS_Hz × 1.5

Reconstruction de vitesse (v0.3+)

• Identifier les indices des GPS valides (lat/lon non NaN)
• Pour chaque paire consécutive de GPS valides : distance = haversine(lat1, lon1, lat2, lon2) vitesse = distance / dt
• Écrêtage à 200 km/h (55.6 m/s)
• Forward-fill + lissage moyenne glissante (5 points)

Distance hybride (v0.4)

• Si speed_mps disponible (> 50% non NaN) : dist = |speed| × dt
• Sinon : haversine entre points GPS consécutifs

Tests unitaires (12 tests)

1. Cohérence distance (dist_m vs speed*dt < 20%)
2. Reconstruction vitesse multi-rate
3. Détection rotation IMU (pitch 15°)
4. Détection freinage brusque
5. Hz detection 10/24/100 Hz (3 tests)
6. Validation D0 (colonnes manquantes, OK)
7. Forward-fill GPS
8. GPS Cleaner v0.4 complet sur multi-rate

Datasets validés

Field Validation — experimental telematics prototype (urban environment, France, 2025)

Hardware disclosure: the field dataset described in this section comes from an experimental telematics prototype device deployed by an industrial partner in a delivery vehicle, not from a commercial Teltonika FMC880. The Teltonika FMC880 was also installed in the vehicle but did not produce usable GPS data during this campaign. A validation on the Teltonika FMC880 is planned for the next deployment campaign.

Dataset: 14 trips, 351,356 points, experimental telematics prototype device, light commercial delivery vehicle.

Multi-rate characteristics:

• IMU: 50 Hz nominal, burst pattern of 50 frames per cycle (1 s capture at 50 Hz, then ~1 s gap, 25 Hz effective)
• GPS: 1 Hz (polyline-encoded)
• Effective multi-rate ratio: 25:1 (50 Hz nominal but 25 Hz effective due to duty cycle)

Key findings for D1 GPS Cleaner:

1. Burst pattern detection: The new detect_burst_pattern() (v0.5) correctly identifies the bimodal dt distribution: 50 frames at 50 Hz native followed by ~1020 ms gap, yielding 25 Hz effective rate. The cleaner now reports: Burst sampling: 50 frames @ 50 Hz, effective 25 Hz (gap 1020 ms). Field-validated on 612 bursts in a single 19-min trip from the prototype device.

2. Outlier detection: 2,205 points flagged as outliers (>55.6 m/s implied speed). These come from GPS polyline interpolation artifacts, not actual sensor errors — a new failure mode not seen in research datasets.

3. Blackout detection: 56 ticks marked as GPS blackout. Consistent with the 1 Hz GPS rate and inter-trip gaps.

4. Gravity-compensated accelerometer: The field device firmware subtracts gravity before transmission. The cleaner’s QSD (Quasi-Static Detection) still works because it uses variance thresholds, not absolute gravity magnitude. However, the speed_reconstruction from IMU double-integration is affected (initial condition shifted by ~0.45 m/s² bias).

5. Contribution — Burst sampling: No published GPS/IMU cleaning pipeline documents this burst pattern. The 50-frame burst creates a distinctive bimodal dt signature: 1 s of 50 Hz data followed by ~1 s of silence. Naive median-based frequency detection reports 50 Hz (correct nominal but misleading on data density). The new detect_burst_pattern() function (v0.5) returns a richer descriptor: {nominal_hz: 50, effective_hz: 25, burst_size: 50, gap_ms: 1020}. Downstream stages (smoothing windows, SQS scoring) should use effective_hz, not nominal_hz.

New row for datasets table:

v0.6 — Physical GPS quality scoring (2026-05-30)

Motivation (field, FMC880 fleet): the firmware-reported satellite count is not a reliable quality measure. On recent FMC firmware it reports satellites in view across all constellations, inflated up to 66 (physically impossible for a fix), while an older firmware unit reports a sane ~14. A binary satellites >= 8 gate therefore lets every fix through and silently fails as a quality filter (this also polluted the no-gyro yaw-baseline pool selection). Diagnosis on the shared-IMEI unit: median sat 34-47, max 66; HDOP, by contrast, was sane (median ~0.5).

Method — compute_gps_quality()

Following the uncertainty-aware GNSS literature (Stølsmark 2009, who represents position uncertainty as a scalar accuracy radius supplied by the receiver and read as a standard deviation; the HDOP × UERE relation below is the usual textbook DOP error model and is not attributed to a specific source here), we replace the satellite count by a physical, per-fix quality score with two factors. The kinematic factor is our own design: Vasconcelos et al. (2017) also apply outlier detection to speed and acceleration streams, but they treat the outliers as driving manoeuvres to classify, not as sensor errors to filter.

1. GNSS accuracy from DOP (standard model): accuracy_radius_m ≈ HDOP × UERE with UERE ≈ 5 m for consumer GPS. Position confidence is a linear ramp 1 (r ≤ 5 m) → 0 (r ≥ 50 m). This yields a meaningful metres estimate, not a count: HDOP 0.8 → 4 m (conf ≈ 1), HDOP 8 → 40 m (conf ≈ 0.22).
2. Kinematic consistency (outlier proxy): the discrepancy between Doppler speed and position-derived speed |v_doppler − v_haversine| / (v + 5). A fix whose position jumps incoherently with the reported speed is down-weighted.

gps_quality = position_confidence × kinematic_factor ∈ [0,1], exposed per fix as gps_quality and gps_accuracy_radius_m (our own fields; the scalar accuracy radius follows Stølsmark 2009). The satellite count is kept only as a weak presence floor, never as a measure. When HDOP is absent (older firmware), the score falls back to the floor + kinematics.

Impact

• The no-gyro yaw-baseline pool selection now filters on HDOP ≤ 2.0 (good fix) instead of the inflated satellite count, fixing a silent selection bug.
• gps_quality becomes a first-class signal for downstream weighting (adaptive smoothing, SQS GPS dimension) — a continuous confidence rather than a threshold.

Companion fix — density-invariant distance (BUG-030)

The incremental distance was dist = speed × dt, which over-estimated by +70 % to +330 % once the upsampler densified a 1 Hz trace (it integrates the Doppler speed over the bridged gaps, and the upsampler additionally re-sampled the incremental dist_m column, multiplying the total ×5). Fix: compute_dist_m now uses haversine over positions (linear interpolation preserves polyline length: 50.8 km vs 51.2 km raw, vs 84 km in speed×dt) and the upsampler recomputes dist_m after densification. Result: ±2 % on both FMC 10 Hz and FMM 1 Hz upsampled.

Tests

test_gps_quality.py (5: HDOP→accuracy, good/bad HDOP, kinematic penalty, no-HDOP fallback), test_distance_consistency.py (density invariance + multi-rate ffill).