Clinical research data engineering — record linkage for treatment outcomes

Probabilistic record linkage: reconnecting TB treatment records to their original enrolment

make sure every patient on treatment is linked back to their screening record

TB treatment medication and a treatment-monitoring consultation.
Reconnecting every TB treatment record to its original enrolment across six research cohorts.

Tuberculosis treatment records arrive in a separate clinical database and often lack the unique participant IDs used across six research cohorts, breaking the link between a patient's treatment and their original enrolment.

The problem

Without a shared identifier, manual reconciliation of treatment records against six studies was slow, error-prone and unscalable, leaving treatment outcomes disconnected from enrolment and screening data.

What I did

I built a two-stage probabilistic matcher — exact phone-hash lookup, then a demographic fuzzy match (rapidfuzz on names, filtered by age and gender) with weighted scoring — that reconciles treatment records to the right participant and syncs the result back to the central outcome database, with national-programme target tracking on top.

6 cohorts
Consolidated and matched
2-stage
Phone + fuzzy demographic match
0.80
Match-confidence threshold
Hourly
Automated Airflow refresh
record-linkagerapidfuzzREDCapAirflowPower BIPython
Result

Treatment records are now linked automatically to their enrolment records and written back to the central database, and three Power BI dashboards track treatment progress against national-programme targets by site and region.

LET'S TALK

Ready to get started?

Tell me about your project, question or goal. I reply within 24 hours, in English or French.

Get in touch