Data quality in an HIV vaccine efficacy clinical trial in South Africa: through natural disasters and with discipline | BMC Medical Research Methodology

Data entry metrics

During the trial, 447,130 case report forms were entered into the database, and 94% were entered within 7 days of the visit date. There were 8716 adverse events entered, and 85% were entered within 3 days of the date that the adverse event was reported to the trial site. The data query ratio was 5 per 100 case report form pages, and 75% of query responses were within 7 days of the visit date (Table 1).

Protocol deviations

There were a total of 5407 participants enrolled, amongst which 4074 protocol deviations were reported over 12,456.1 person years of follow-up, corresponding to a protocol deviation rate of 32.7 [95% CI: 31.7–33.7] per 100 person years. The ratio of deviations to enrolments (4074/5407) was 75 (95% CI: 73.0–77.6) per 100 enrolments. At least one protocol deviation was reported for 2687 participants. There was a median of 1 protocol deviation per participant (IQR 1–2).

The median time from deviation to site awareness was 31 days (IQR 0–146). The fourteen sites had a median of 275 deviations per site (IQR 116–425) (Table 2).

Deviation type categories were omitted data and/or procedures (69%), error in data or procedures (13%), use of materials not approved by the research ethics committee (10%), consent errors (4%), and needless data collection or procedures (3%) (Table 3).

At each site, a median number of 382.5 (IQR 342–486) participants was enrolled (Table 2). Enrolments at high-enrolling sites ranged from 407 to 511 participants and from 64 to 358 participants at low-enrolling sites, with equal numbers of sites in each category. The protocol deviation to enrolment ratio was 0.48 at high-enrolling sites (1598/3313) versus 1.18 at low-enrolling sites (2476/2094). Low-enrolling sites had a higher proportion of omitted data and procedures and use of materials not approved by the Ethics Committee compared to high-enrolling sites. However, there was a higher proportion of errors, consent documentation errors and needless data and procedures at high-enrolling sites compared to low-enrolling sites (Table 4, Supplementary Table 1).

From qualitative analyses, using quantitative methods as supportive evidence, four main deviations themes are discussed.

Deviation theme 1: disasters can give rise to protocol deviation trends

The national response to COVID-19 began soon after trial unblinding, affecting trial conduct. Of omitted data in the trial, 54% (1525/2815) was stated to be because the South African government imposed lockdown at the beginning of the COVID-19 pandemic. This was linked with omissions of physical examinations and laboratory samples, while procedures like questionnaires, counselling, and event reporting were conducted telephonically.

“The participant missed the following visit 08.0 assessment due to the conduct of a telephonic visit: vital signs assessment, collection of blood, pregnancy and STI samples. The telephonic visit was conducted as a result of the COVID-19 lockdown period in the country.”

There was also record of a climate disaster, flooding, linked with protocol deviations. One site reported that their city “experienced extreme weather conditions, resulting in a cancellation of the full meeting” of their ethics committee. The ethics committee therefore could not provide continuing review, and the annual recertification lapsed over six days when the trial site conducted study procedures with nine participants.

Deviation theme 2: subroutines introduce an opportunity for deviation

The concept of a “subroutine” is borrowed from the field of computer programming, where it means a sequence of programming instructions for a specific scenario. The concept is similar to an algorithm in clinical care – “if this happened, then do that”. In clinical trials, subroutine errors occur when staff do not follow the dissimilar sequence of procedures required for different participants or scenarios.

In HVTN 702, many omissions were linked with not following subroutines. A prominent example of deviations related to a multi-subroutine was stool swab collection. The main informed consent form provided participants with the option to collect swabs of their stool, which initiated the first subroutine for staff to either collect or not collect the samples. The second subroutine was that staff should ask only participants who collected the stool swab to complete a dietary and gastrointestinal symptom questionnaire. The third subroutine was the timing of the swab sample: staff should collect the stool swab before study product administration at the month 0 visit, and any time during the month 6.5 visit. There were protocol deviations reported for each subroutine, e.g. collecting stool samples from participants who had not agreed to it on their consent form, not completing the required questionnaires when stool samples were completed, and collecting samples after study product administration at the month 0 visit.

“The research nurse erroneously collected stool samples and participant did not consent for stool sample collection on the signed informed consent form…”

Protocol deviations reported for some other subroutines included not following subroutines specified in the study specific procedures guide for anaemic participants, omitting to collect dried blood spot samples for participants whose visits were on specific calendar days as specified in the study specific procedures guide or collecting it at the wrong time during the visit, and not tailoring sample types for sexually transmitted infection tests by sex at birth.

“Dried blood spot specimen taken after vaccination.”

Deviation theme 3: document fragmentation presents obstacles for consent documentation

More common than implementing incorrect consent form versions was the prevalence of giving participants study-related brochures before its approval by the ethics committee. The template informed consent form in the protocol made provision for four issues to be covered in more depth in brochures: trial-permitted birth control methods, trial procedures, vaccine-induced seroposititvity (VISP), and participant bill of rights and responsibilities. Site investigators could submit these as separate documents to their ethics committees for approval to distribute to participants with the consent form, or copy the information from 3 of these brochures (with the exception of vaccine-induced serposititvity information) into their site consent form in lieu of brochures.

Distributing consent-related brochures to participants before ethics committee approval was common to 403 deviations.

“Participant was handed with VISP and Bill of rights brochures at screening before they were reviewed and approved by ethics committee.”

Furthermore, deviations were reported when the structure of a consent document fragmented the completion of fields across multiple pages. Deviations reported that staff and participants omitted to complete any fields situated on pages other than the signature page of the consent form.

“Pharmacogenetic ICF [VERSION, DATE] page 3 of 4 no option was chosen by participant regarding limited pharmacogenetic testing in error. Participant continued the study without confirming if they still want [to] allow use of samples for limited pharmacogenetic testing or not.”

Deviation theme 4: visit scheduling

Not checking the visit window when scheduling participants and/or before conducting procedures was linked with visit procedures being performed unnecessarily.

“Pregnancy test performed out of window as a result of incorrect visit scheduling.”

In many of deviations reported for missed visits, calls had not been made to remind participants of their upcoming visits. In some cases, staff had not recognised that the visit window was open for a participant who was at the site presenting for ancillary care, and did not take the opportunity to do visit procedures. In other cases, a visit was split over more than one day, and participants did not return for the second part of the split visit.

Preventive actions

Preventive action categories were: (i) no preventive measures (n = 1903); (ii) discipline, training and/or awareness (n = 963); (iii) quality review, checking and verifying (n = 839), and (iv) changing the process and/or tools (n = 633). In 254 cases, more than one category of preventive action had been applied.

Preventive action theme 1: systems-based actions are unusual

Some narratives did not provide anything further than corrective action. There was also evidence from the narratives that some staff were unclear about the difference between corrective and preventive actions. In an example of a laboratory sample not collected from the participant, the narrative for the corrective action described the efforts undertaken to obtain the sample after deviation awareness, and the preventive action narrative describes the corrective action further:

“Participant visit window for visit 13.0 is still open. Closes on the 11 may 2019. CT/NG [Chlamydia trachomatis, Neisseria gonorrhoeae] urine will be collected at this visit when the participant is found. CT/NG urine kits are available.”

Some narratives proposed that the preventive action would be to stop the incorrect action and simply follow the correct action in future. Specific plans on how the site would change course were not always evident in the narratives. In an example of missed counselling and questionnaires, the narrative stated the site’s steps to prevent future occurrences of this type of deviation:

“Staff to always follow visit schedule and do procedures according to visit requirements.”

Instead, many narratives offered preventive actions centred around people, and there was a dominant perception that an error would prompt people to change behaviors, become more “vigilant” and exercise more care. The onus was placed on people to remember to improve their performance after training and awareness campaigns through email reminders or wallcharts.

“Staff member will avoid being distracted by the participant’s conversation and pay more attention to visit procedures…”

“Staff member responsible was made aware of the error and the importance of following the schedule of procedures as stipulated in Appendix H of the Protocol was emphasised. ‘Appendix H’ Page was printed, laminated and placed in the counselling rooms for easy reference.”

For a haemoglobin test that could not be done because of a clotted sample:

“Nurses reminded of the gentle inversion technique to prevent clotting.”

Systems-based preventive actions were less usual. They focused on correcting or adding more detail to forms (e.g. laboratory requisition forms, medical history forms), adding more detail to checklists to distil visit requirements (e.g. visit flow checklists), introducing trackers (e.g. for informed consent versions), adding staff members to perform critical requirements which were not being done (e.g. assigning staff members to call participants to remind them of their scheduled visits, allocating staff members to call participants for safety follow up), and procuring tools with advanced technology (e.g. imaging technology that locates veins for successful blood sampling to prevent insufficient blood sampling, and syncronising digital clocks to a device with atomic time instead of digital clocks to prevent time documentation discrepancies).

In 2% (n = 79) of the 4074 protocol deviation narratives, site staff mentioned that they had conducted root cause analysis of that particular deviation. Staff often narrated that they found a human root cause – i.e. they thought that the error arose from a person.