GeneXpert molecular testing machines across seventeen Cameroon laboratories export raw CSV result files that vary by language and character encoding, feeding several TB research studies and a diagnostic-efficiency manuscript.
Raw exports varied by machine language (English/French) and encoding (UTF-16-LE, UTF-8-SIG, windows-1252, latin-1), study IDs were malformed or hand-typed, pooled testing produced duplicate sample records, and tests spanning midnight produced invalid negative durations.
I built a high-performance parser with a BOM-plus-chardet encoding cascade and a process-pool that auto-switches to streaming above ten thousand files, then a cleaning layer that applies 200+ French→English mappings, extracts study IDs by project-specific regex, corrects datetime rollovers, and resolves pooled cartridges back to individual specimens.
A single automated run consolidates 1,721 raw machine exports into a 133,556-row analysis-ready dataset, and a pooled-testing analysis across 14,404 pools quantified where pooling saves cartridges without missing cases.