Acute respiratory infections (ARIs), primarily pneumonia, are the leading infectious causes of death among children under 5 years of age globally, accounting for an estimated 800,000 pneumonia‐related deaths in 2017. 1 Deaths from pneumonia in children result mostly from delayed presentation to appropriate healthcare providers, wider issues of quality of care such as referral challenges and access to oxygen and inappropriate treatment. 2 Classification of fast breathing, as a sign of pneumonia, by CHWs and first‐level health facility workers (FLHFWs; collectively known as frontline health workers) is based on manually counting the number of breaths in 60 s in children under five (U5) years of age with cough and/or difficulty breathing. This allows the health worker to assess whether the RR is high enough for a particular age to prescribe antibiotics, and treat suspected pneumonia, as defined by the World Health Organization (WHO) integrated management of childhood illnesses (IMCI) guidelines 3 for FLHFWs and the iCCM guidelines 4 for CHWs. In practice, frontline health workers indicate that counting respiratory rate (RR) can be difficult because children breathe irregularly and faster than adults; the child may not be calm and still for a full minute; and it is difficult to define what is and is not a breath. 5 Misclassification of the observed rate remains high 6 , 7 and often leads to inappropriate treatment. 8
Low blood oxygen saturation, or hypoxaemia, is a sign that can be observed among a variety of different diseases, including pneumonia, that has been identified as a marker of severity and a predictor for morbidity and mortality in children with respiratory illness. 9 However, hypoxaemia, if characterised at all in low‐ and middle‐income country (LMIC) settings, is typically identified based on clinical findings alone, 10 and the inability of healthcare workers to promptly detect and refer these children, whose lives are in danger, leads to an adverse prognosis in many of these children. While pulse oximetry is a reliable and non‐invasive method for identifying children with hypoxaemia, pulse oximeters are rarely available outside higher‐level facilities in resource‐constrained countries. 11 A modified iCCM algorithm would therefore be required to include oxygen saturation (SpO2) less than 90% as a sign of hypoxaemia and as a referral and treatment sign. This can then be used by the HEWS, in combination with the current RR cut‐offs, in the sick child consultations.
United Nations International Children's Emergency Fund (UNICEF)’s Acute Respiratory Infection Diagnostic Aid (ARIDA) project 12 was initiated as a response to the calls for better devices that diagnose symptoms of pneumonia. 13 , 14 Two ARIDA devices have been developed and commercialised in response to UNICEF’s Request for Proposals, 15 and field trials were conducted in Ethiopia and Nepal to assess usability and acceptability of devices whose attributes meet the specifications outlined in the UNICEF ARIDA target product profile (TPP) 16 for use by frontline health workers at the lowest levels of health care. The study reported in this paper is one of the field trials in Ethiopia funded through UNICEF’s partnership with ‘La Caixa’ Foundation and conducted by Malaria Consortium.
Ethiopia was selected to host the ARIDA acceptability field trial due to the high burden of ARIs in children U5 (16% in 2016), 17 availability of a CHW cadre (locally called health extension workers [HEWs]) who deliver community‐based management of pneumonia 18 and availability of first‐line amoxicillin dispersible tablets (Amox DT). Since 2010, Ethiopia has scaled up iCCM in most regions following a national policy change supporting community‐based management of childhood illnesses through HEWs. 19 All HEWs are women, educated to at least tenth grade, trained for 1 year in iCCM and other health preventive and curative interventions. They are government paid and equipped to assess, classify and manage uncomplicated pneumonia, malaria, diarrhoea and severe acute malnutrition and provide preventive and curative health services. 18 Trained HEWs are deployed to a health post to work at the sub‐district (kebele) level and serve a population of approximately 5000 people. Through iCCM, any child 2–59 months with fast breathing pneumonia and no danger signs can be treated at the health post; sick children under 2 months are referred to a health centre. 18 The next level of the Ethiopian health system is the health centre, which is staffed by around 20 health professionals (FLHFWs), provides preventative and curative services to approximately 25,000 people and serves as a referral centre and practical training institution for HEWs. In 2016, the Ethiopian oxygen roadmap was published, which has a vision for pulse oximeters to be available and used at all levels of the Ethiopian health system. 20
This paper reports findings of a cross‐sectional study to determine usability, through exploring whether HEWs in Ethiopia using Rad‐G could adhere to the required steps to assess and classify pneumonia in children under five with cough and/or difficult breathing.
Full details of the methods have been described elsewhere. 21
Masimo developed the Rad‐G (Figure 1) which uses differential light absorbance technology to measure SpO2 and derives respiration rate (RR), pulse rate (PR) and perfusion index (Pi) from the photo plethysmography waveform or Pleth (RRp™) in children 0 to 59 months and classifies the RR and SpO2 of the child according to WHO IMCI/iCCM guideline cut‐offs. 3 , 4 The device, designed for LMIC use, is comprised of a handheld unit with a liquid crystal display touchscreen user interface and a detachable universal sensor probe suitable for children above 3 kg and which is placed on the child's finger or toe. The device also has an animated display to help keep the child calm during use (Figure 1).
The study was conducted in community settings and first‐level health facilities in Sodo Zuria, Damote Sore and Damote Gale districts of Wolaita zone in Southern Nations, Nationalities, and Peoples' Region (SNNPR), Ethiopia. SNNPR and the Wolaita zone were selected because of the high burden of ARIs, sufficient number of HEWs with experience, relative political stability (no insecurity that might have affected the implementation of the study) and availability of oxygen at the district hospital in case a child was referred. Wolaita zone has logistic and operational feasibility for data collection and quality assurance, as the health posts and health centres are accessible to road transport. In addition, Malaria Consortium has strong relationships with SNNPR health bureau and the Wolaita zonal health department through implementing different projects in Wolaita zone in the past, such as the TRAction study. 22 All available HEWs (134) in the three districts were selected for the study. All selected HEWs were trained in and using iCCM at their health posts (65). For the acceptability element of the study, a convenience sample of twenty FLHFWs (one per health centre) who were trained in IMNCI and working in the under five clinic of their health centre was selected to participate in consultation with the heads of the three district health offices.
This was a cross‐sectional study using direct observation of HEW consultations and semi‐structured interviews with frontline health workers and caregivers. The study group decided to directly observe HEWs only, with the assumption that FLHFWs would be able to use Rad‐G as well as, or better than FLHFWs due to their higher education level and longer training.
One hundred and thirty‐four HEWs and 20 FLHFWs were provided with a 2‐day training on the WHO requirements to assess fast breathing plus the measurement of SpO2 as a sign of hypoxaemia, detailed in a job aid (see Appendix S1), trained on how to use Rad‐G and completed pre‐ and post‐training assessments to test their knowledge of iCCM/IMNCI and Rad‐G. A pass mark of 75% was required of all HEWs participating in the study. Immediately after the training 133 HEWs were directly observed using Rad‐G (observation 1) to assess and classify pneumonia during two consecutive child consultations in their usual workplace (the health post). HEWs were observed completing nine assessment, classification, treatment and referral steps from the WHO requirements to assess fast breathing and the device manufacturer instructions for use (IFU; Table 1). HEWs were then left to use the device routinely for 2 months before being directly observed a second time (observation 2), again for two consecutive child consultations. The sample size was calculated assuming 71% of HEWs (calculated as 75% complete correct RR assessment, of which 95% complete correct RR classification) would adhere to the required assessment and classification steps for the primary outcome, with 95% confidence and 7.5% precision, including an adjustment of 10% for non‐participation or exclusion for any other reason and a design effect to account for clustering at HEW level of 1.7. 23
| Consultation step | Definition | Source of step | |
|---|---|---|---|
| 1 | Child calm before Rad‐G attempt | Calm: not actively crying or moving | WHO requirements to assess fast breathing WHO requirements to assess fast breathing |
| 2 | Correct mode selected | Screening mode | Device manufacturer instructions for use |
| 3 | Correct age group | Age group recorded by HEW on Rad‐G device matches screening checklist | WHO requirements to assess fast breathing WHO requirements to assess fast breathing |
| 4 | Correct probe position | Fully inserted | Device manufacturer instructions for use |
| 5 | Correct probe direction | Picture on top of finger or toe | Device manufacturer instructions for use |
| 6 | Child not eating/feeding during Rad‐G attempt | No eating/breastfeeding | WHO requirements to assess fast breathing WHO requirements to assess fast breathing |
| 7 | Child calm during Rad‐G attempt | Calm: not actively crying or moving | WHO requirements to assess fast breathing |
| 8 | Correct classification using Rad‐G | According to iCCM guidelines, based on screening age group and breathing status of the child | WHO requirements to assess fast breathing |
| 1‐8 | Correct assessment and classification (steps 1–8) – primary outcome | HEW correctly completed all steps 1–8 | Device manufacturer instructions for use and WHO requirements to assess fast breathing |
| 9 | Correct treatment and referral guidance using RAD‐G classification and HEW's assessment of other symptoms (yes/no?) | According to iCCM guidelines, based on age group recorded during child screening, and breathing status of the child. N.B HEW will not be marked as 'incorrect' if there was stock‐out of antibiotics, caregiver refused treatment or other valid reason for no treatment recorded | WHO requirements to assess fast breathing |
To ensure there were sufficient children visiting the health post on the day of the HEWs’ observation, HEWs were encouraged to mobilise caregivers of sick children aged U5. Observations were conducted by twelve research assistants (either medical officers or degree‐qualified nurses). Research assistants were trained to screen U5 children for inclusion criteria and then silently observe the HEW assessing a child under five presenting for illness (aged 0 to <2 months) or for cough and/or difficult breathing (aged 2–59 months) according to the WHO requirements to assess fast breathing and device manufacturer IFU. HEWs had up to three attempts to obtain a RR classification with Rad‐G; if these attempts were unsuccessful, the HEW reverted to using standard practice (ARI timer, smartphone timer or watch). Each step of the HEWs’ assessment, classification and treatment and/or referral (Table 1) of the child was independently recorded by two research assistants in a digital tablet‐based data collection form using CommCare (version 2.38.1, Dimagi). Research assistants photographed Rad‐G with the RR and SpO2 result, classification and age displayed, to provide source documents for verification purposes. Data were synced daily to a protected cloud server and validated and cleaned by the data manager. Between the first and second observations, HEWs were encouraged to use Rad‐G during routine practice, but were allowed to revert to standard practice if required, and instructed to record which device they used in their patient register using coloured stickers (one patient register per health post). FLHFWs were not directly observed conducting a sick child assessment using Rad‐G but were encouraged to use Rad‐G at the health centre for the 2 months prior to their semi‐structured interview. A half‐day focused training was provided to all HEWs participating in the study before observation 2, as it was felt by the research team that participants needed further clarification on the importance of probe fit and signal strength in achieving a successful reading. This was based on their experience during observation 1, where health workers needed further support to get a stronger signal by fitting the probe correctly.
We interviewed a purposive sub‐sample of 14 HEWs immediately after observation 2, making sure to include HEWs with a range of years’ experience practising as a HEW. We also interviewed 14 caregivers of children who were assessed by these HEWs, to explore their reactions and experience of the device use on their child. A convenience sample of five FLHFWs who were available on the day we visited health facilities was also interviewed. Topic guides were developed using a comprehensive conceptual framework of acceptability of healthcare interventions. 24 Questions relating to seven facets of acceptability (attitude, burden, perceived effectiveness, ethicality, intervention coherence, opportunity costs and self‐efficacy) were included to prompt caregivers to evaluate their experience of the device and health workers to evaluate their experience of using the device. Six research assistants (all Ethiopian nationals with some prior experience of qualitative methods) were trained over 3 days to conduct interviews with health workers and caregivers. The topic guides were translated into the local languages and pilot tested with HEWs and caregivers at three health posts. Following pilot testing, some questions were revised and minor amendments made to the translation of specific words and phrases. All semi‐structured interviews were conducted in the local language and audio‐recorded with participants’ consent. Interviews were conducted with HEWs, FLHFWs and caregivers until data saturation was reached.
Descriptive information about the HEWs is presented in frequencies and percentages, including numbers trained and numbers completing first and second observation, sex, district, number of years qualified as a HEW, last integrated refresher training and last supervision. Descriptive information about the number of children enrolled, number of evaluations that started, number of evaluations completed by Rad‐G and standard practice and child age and sex was also presented. The primary outcome was calculated as the proportion of U5 children consultations where HEWs using Rad‐G adhered to WHO requirements to assess fast breathing and device manufacturer IFU after 2 months of routine use. This analysis was disaggregated by age group, respiration rate and SpO2 classification. Secondary outcomes including the proportion of HEWs correctly performing steps reflecting the device manufacturer IFU and the steps that reflect the WHO requirements to assess fast breathing are also presented and the difference in the proportion of consultations that were completed correctly between observation one and observation two. For the main outcomes, the most conservative estimates were used, that is if the two research assistants disagreed on how the HEW performed a step in the assessment, the one that recorded an inconsistency/error for that step was used over the one who recorded that the step was performed correctly. The mean time taken to complete the full assessment was calculated for Rad‐G from the time when the HEW turned on the Rad‐G (prior to probe placement) to when the Rad‐G displayed both SpO2 and RR readings. The number of children who were assessed for signs of respiratory illness by HEWs with Rad‐G or standard practice during routine care was also presented. Univariable logistic regression analyses were performed to assess the relationship between the time since (i) a HEW’s last routine iCCM integrated refresher training; (ii) a HEW’s last routine supervision; or (iii) qualification as a HEW and the proportion of consultations where HEWs adhered to WHO case management and device manufacturer IFU.
For the qualitative analysis, data collected via semi‐structured interviews with caregivers and frontline health workers were analysed separately. Transcripts were created from the recordings of the interviews, translated and checked for accuracy. We carried out a thematic analysis of the qualitative data using MAXQDA (VERBI Software, 2016) to manage data coding, searching and retrieval. One author reviewed all transcripts to identify possible codes; the list of codes was refined in discussion with the research team, and final coding frames were developed for the HEW/FLHFW interviews and the caregiver interviews. After coding all data, matrices were used to display data for each emerging theme and further explore similarities and differences within the data. Each theme was critically analysed by the research team until the final themes were agreed upon.
The study was approved by the Liverpool School of Tropical Medicine Research Ethics Committee (ref. 18‐026) and the SNNPR Regional Health Bureau Ethics Committee (ref. 6‐19/10426). Written consent for observations and semi‐structured interviews was obtained from each HEW prior to the observation and from each caregiver whose under five child was assessed by a HEW during an observation.
One hundred and thirty‐four HEWs completed the training, of whom 133 were directly observed using Rad‐G between September and December 2018. Those observed had an average of 7.7 (S.D 4.3)‐year experience as a HEW, 78.2% had received iCCM integrated refresher training within 3 years, and 52.6% had received their last supervision within 3 months prior to the study (Table 2). Fifteen HEWs from Sodo Zuria, Damote Sore and Damote Gale participated in the semi‐structured interviews (SSI). Twenty female FLHFWs completed the training, of whom eight participated in a semi‐structured interview (SSI) from Sodo Zuria (38%), six from Damote Sore (46%) and six from Damote Gale (15%). Fifteen caregivers (33%) in each of the three districts participated in SSIs.
| District | ||||
|---|---|---|---|---|
| Sodo Zuria | Damote Sore | Damote Gale | Overall | |
| Number of HEWs completed 2‐day training on iCCM and Rad‐G | 49 | 38 | 47 | 134 |
| Average pre‐training score (%) | 63.6 | 56.9 | 64.5 | 61.7 |
| Average post‐training score (%) | 87.9 | 88.7 | 88.5 | 88.7 |
| Number of HEWs participating in observation 1 using Rad‐G | 49 | 38 | 46 | 133 |
| Mean years' experience as a HEW (SD) | 7.0 (4.1) | 8.3 (4.2) | 8.1 (4.5) | 7.7 (4.3) |
| Received iCCM Integrated Refresher Training ≤3 years ago (%) | 98.0 | 97.4 | 41.3 | 78.2 |
| Received last supervision ≤3 months ago (%) | 40.8 | 65.8 | 54.3 | 52.6 |
| Number of FLHFWs completed training (N) | 8 | 6 | 6 | 20 |
| Average pre‐training score (%) | 54.8 | 64.1 | 56.4 | 58.3 |
| Average post‐training score (%) | 80.8 | 79.5 | 71.8 | 77.3 |
Two hundred and sixty‐six children were enrolled for the first observation and 260 children for the second observation (Figure 2). Of these, 264 (observation 1) and 259 (observation 2) Rad‐G consultations were started; the reasons for not starting were as follows: (1) the child not being calm enough to start, and (2) the Research Assistants (RAs) recorded different age data, and therefore, these data were excluded during analysis. For observations at the first time point, just over half the children were male, and for the second time point, 54% were female (Figure 2). For observation 1 49.2% of consultations were completed on the first attempt. This increased to 81.1% for the second observation. The main reasons for unsuccessful attempts during observation 1 were that the child was not calm or was moving (44.6%) and the Rad‐G displayed an error or ‘motion detected’ message (52.3%). While the overall number of unsuccessful attempts was less for observation 2, the main reasons remained the same (Table 3). During observation 1, 19.7% and 1.5% of children were determined to have fast breathing and hypoxaemia by Rad‐G, respectively; these percentages were similar during observation 2 (20.1% and 1.9%).
| Reason for Rad‐G unsuccessful attempt a | Observation 1 | Observation 2 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1st attempt | 2nd attempt | 3rd attempt | Total | % | 1st attempt | 2nd attempt | 3rd attempt | Total | % | |
| Child not calm or moving | 99 | 70 | 49 | 218 | 44.6 | 44 | 28 | 16 | 88 | 51.8 |
| Child pulls off Rad‐G | 0 | 1 | 0 | 1 | 0.2 | 0 | 0 | 0 | 0 | 0.0 |
| Rad‐G displays error message | 66 | 45 | 26 | 137 | 28.0 | 20 | 11 | 8 | 39 | 22.9 |
| Child starts feeding stops assessment | 0 | 2 | 1 | 3 | 0.6 | 0 | 0 | 0 | 0 | 0.0 |
| HEW could not start Rad‐G | 0 | 1 | 0 | 1 | 0.2 | 0 | 0 | 0 | 0 | 0.0 |
| Other | 0 | 1 | 0 | 1 | 0.2 | 1 | 0 | 0 | 1 | 0.6 |
| Rad‐G signal strength low | 0 | 0 | 2 | 2 | 0.4 | 1 | 1 | 1 | 3 | 1.8 |
| Probe attached with picture facing down | 4 | 3 | 0 | 7 | 1.4 | 0 | 0 | 0 | 0 | 0.0 |
| Rad‐G displays 'motion detected' symbol | 61 | 37 | 21 | 119 | 24.3 | 19 | 12 | 8 | 39 | 22.9 |
After 2 months using Rad‐G, HEWs adhered to the WHO requirements to assess fast breathing and device manufacturer IFU in 85.3% (95% CI 80.2–89.3) of consultations with children under five. This was an absolute increase of 8.9% from the first observation immediately after training (Table 4). When broken down by assessment steps, initially the HEWs found it more difficult to calm the child (84.8%; 95% CI 79.9, 88.7) and select the right age group (89.8%; 95% CI 85.4, 92.9) but both proportions improved in observation 2 (95.8%; 95% CI 92.5, 97.6 and 95%; 95% CI 91.5, 97.1, respectively). After 2 months, HEWs gave the correct treatment and referral in 95.8% (95% CI 92.3, 97.7) of consultations based on the classification results shown by Rad‐G (Table 4).
| Consultation steps | Observation 1 (after training) | Observation 2 (after 2 months) | |||||
|---|---|---|---|---|---|---|---|
| n | % | 95% CI | n | % | 95% CI | ||
| 1 | Child calm before Rad‐G attempt | 224 | 84.8 | 79.9, 88.7 | 248 | 95.8 | 92.5, 97.6 |
| 2 | Correct mode selected | 264 | 100.0 | 98.6, 1.0 | 258 | 99.6 | 97.3, 99.9 |
| 3 | Correct age group | 237 | 89.8 | 85.4, 92.9 | 246 | 95.0 | 91.5, 97.1 |
| 4 | Correct probe position | 253 | 95.8 | 92.6, 97.7 | 258 | 99.6 | 97.3, 99.9 |
| 5 | Correct probe direction | 257 | 97.3 | 94.5, 98.7 | 254 | 98.1 | 95.4, 99.2 |
| 6 | Child not eating/feeding during Rad‐G attempt | 244 | 92.4 | 88.5, 95.1 | 257 | 99.2 | 96.9, 99.8 |
| 7 | Child calm during Rad‐G attempt | 249 | 94.3 | 90.8, 96.6 | 251 | 96.9 | 93.9, 98.5 |
| 1–7 | Cumulative assessment (steps 1–7) | 180 | 68.2 | 62.3, 73.6 | 227 | 87.6 | 83.0, 91.1 |
| 1–7: Denominator = Children who were assessed (assessment started) | 264 | 259 | |||||
| 8 | Correct classification using Rad‐G | 183 | 93.8 | 89.4, 96.5 | 229 | 96.2 | 92.9, 98.0 |
| 1–8 | Correct assessment and classification (steps 1–8) – primary outcome | 149 | 76.4 | 69.9, 81.9 | 203 | 85.3 | 80.2, 89.3 |
| 9 | Correct treatment and referral guidance using Rad‐G classification and HEW's assessment of other symptoms (yes/no?) | 185 | 94.9 | 90.7, 97.2 | 228 | 95.8 | 92.3, 97.7 |
| Denominator = Children with RR and SpO2SpO2 classification (Rad‐G) | 195 | 238 | |||||
| Intention to treat | |||||||
| 8 | Correct classification using Rad‐G | 183 | 69.3 | 63.5, 74.6 | 229 | 88.4 | 83.9, 91.8 |
| 1–8 | Correct assessment and classification (steps 1–8) – primary outcome | 149 | 56.4 | 50.4, 62.3 | 203 | 78.4 | 72.9, 83.0 |
| Further analysis | |||||||
| 2, 4, 5 | Manufacturer instructions for use correctly performed (steps 2, 4, 5) | 248 | 93.9 | 90.3, 96.3 | 252 | 97.3 | 94.4, 98.7 |
| 1–3, 6–8 | Adherence to WHO requirements to assess fast breathing (steps 1, 3, 6, 7, 8) | 151 | 57.2 | 51.1, 63.1 | 208 | 80.3 | 75.0, 84.7 |
| Denominator = Children who were assessed (assessment started) | 264 | 259 | |||||
There was no significant difference (p = 0.58) in the proportion of HEWs adhering to each of the steps after 2 months for the 2‐ to <12‐month‐olds (88.3%; 95% CI 78.8, 93.9) compared to the 12‐ to 59‐month‐olds (87.4%; 95% CI 81.6, 91.6; Table 5). The largest variation in assessment steps between age groups was seen for both ‘child calm in advance’ (94.8% vs. 96.6%) and ‘child calm during assessment’ (94.8% vs. 98.3%) for the 2‐<12 months and 12‐ to 59‐month groups, respectively; the difference was not significant (p = 0.93). HEWs completed all eight steps correctly more frequently for normal breathing children (89.9%; 95% CI 84.9, 93.3) compared to fast breathing children (78.8%; 95% CI 65.2, 88.1), but again was not found to be significant (p = 0.97). The logistic regression analysis found no significant association between correct assessment and classification of children after 2 months of routine use and the number of months since last iCCM integrated refresher training (p = 0.28), nor the number of months since last supporting supervision (p = 0.17). There was a statistically significant positive association between number of years since qualification as HEW and ability to adhere to relevant guidelines using the Rad‐G (OR 1.13; 95% CI 1.04–1.22, p < 0.001). The mean performance time, that is from the moment the Rad‐G was turned on to when the reading was displayed (inclusive of up to three attempts) was 06:58 min (range 1 min 32 s to 36 min 32 s) for observation 1, which decreased to 5 min 42 s (range 1 min 30 s to 22 min 31 s) for observation 2. The proportion of correct assessments relating to the manufacturer's IFU after 2 months of routine utilisation was 97.3% (95% CI 94.4–98.7), representing an increase of 4.3% from when HEWs were assessed for observation 1 immediately after training (Table 4). This compares to adherence to the steps included in the revised iCCM guidelines, which was lower at 80.3% (95% CI 75.0–84.7), but which had also increased by 23.1% from observation 1 (Table 4).
| Consultation steps | Observation 2 (0–<2 months) N = 7 |
Observation 2 (2–<12 months) N = 77 |
Observation 2 (12–59 months) N = 175 |
Observation 2 (fast breathers) N = 52 |
Observation 2 (normal breathers) N = 207 |
|||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| n | % | 95% CI | n | % | 95% CI | n | % | 95% CI | n | % | 95% CI | n | % | 95% CI | ||
| 1 | Child calm before Rad‐G attempt | 6 | 85.7 | 25.7, 99.0 | 73 | 94.8 | 86.7, 98.1 | 169 | 96.6 | 92.5, 98.5 | 49 | 94.2 | 83.0, 98.2 | 199 | 96.1 | 92.4, 98.1 |
| 2 | Correct mode selected | 7 | 100.0 | 59.0, 1.0 | 76 | 98.7 | 91.0, 99.8 | 163 | 93.1 | 88.2, 96.1 | 52 | 100.0 | 96.4, 1.0 | 206 | 99.5 | 96.9, 99.9 |
| 3 | Correct age group | 7 | 100.0 | 59.0, 1.0 | 76 | 98.7 | 91.0, 99.8 | 175 | 100.0 | 97.9, 1.0 | 47 | 90.4 | 78.4, 96.1 | 199 | 96.1 | 92.4, 98.1 |
| 4 | Correct probe position | 7 | 100.0 | 59.0, 1.0 | 76 | 98.7 | 91.0, 99.8 | 175 | 100.0 | 97.9, 1.0 | 52 | 100.0 | 96.4, 1.0 | 206 | 99.5 | 96.6, 99.9 |
| 5 | Correct probe direction | 7 | 100.0 | 59.0, 1.0 | 75 | 97.4 | 89.9, 99.4 | 172 | 98.3 | 94.8, 99.5 | 50 | 96.2 | 85.3, 99.1 | 204 | 98.6 | 95.6, 99.5 |
| 6 | Child not eating/feeding during Rad‐G attempt | 7 | 100.0 | 59.0, 1.0 | 77 | 100.0 | 95.3, 1.0 | 173 | 98.9 | 95.5, 99.7 | 50 | 96.2 | 85.3, 99.1 | 207 | 100.0 | 98.2, 1.0 |
| 7 | Child calm during Rad‐G attempt | 6 | 85.7 | 25.7, 99.0 | 73 | 94.8 | 86.7, 98.1 | 172 | 98.3 | 94.8, 99.5 | 49 | 94.2 | 83.0, 98.2 | 202 | 97.6 | 94.3, 99.0 |
| 1–8 | Cumulative assessment (steps 1–7) | 6 | 85.7 | 25.7, 99.0 | 68 | 88.3 | 78.8, 93.9 | 153 | 87.4 | 81.6, 91.6 | 41 | 78.8 | 65.2, 88.1 | 186 | 89.9 | 84.9, 93.3 |
| 1–8 | Correct classification using Rad‐G (yes/no?) | 6 | 100.0 | 54.1, 1.0 | 61 | 96.8 | 87.8, 99.2 | 162 | 95.9 | 91.5, 98.0 | 43 | 84.3 | 71.1, 92.1 | 186 | 99.5 | 96.2, 99.9 |
A total of 74 health posts reported data on device use using stickers in their patient registers, of which 71 reported data correctly. A total of 579 pneumonia consultations for children U5 were completed between October–December 2018 (median = 63 days) at these 71 health posts. Of these, 486 (83.9%) were completed with Rad‐G (a mean of 6.6 Rad‐G consultations per health post), 57 (9.8%) were completed with their standard practice device (ARI timer, phone timer or watch), and 36 (6.2%) were completed with an unrecorded device. Of the assessments completed with a known device, 89.5% were completed with Rad‐G.
We identified three overarching themes relating to the usability and acceptability of Rad‐G among HEWs and FLHFWs and acceptability to caregivers: (1) adherence to modified iCCM guidelines and manufacturer IFU, (2) usability of Rad‐G and (3) acceptability of Rad‐G.
When discussing the results displayed on Rad‐G after assessments, many HEWs and FLHFWs expressed no difficulty in reading the oxygen and RR levels and that the display was ‘readable’, ‘clear’ and the colours helped to interpret positive and negative readings. Some FLHFWs and just two HEWs reported being able to refer cases based on oxygen level, which they were unable to do before. This indicates that the SpO2 function on the Rad‐G, when taken in isolation, was still seen as having value over no device. Many HEWs and FLHFWs mentioned using the job aid to help with classification and referral, specifically mentioning how easy or straightforward it was to identify classification categories using the charts in the job aid. Some HEWs mentioned using the job aid to compare the findings on the screen with the charts to decide on the treatment. Notably, some health workers talked about how Rad‐G supported them to improve their antibiotic prescribing behaviour. For example, some FLHFWs reported that they had previously prescribed ‘drugs’ or ‘treated’ children ‘without counting the RR’. Some HEWs said that previously they would treat cases ‘without proper diagnosis’, based on observation only or based on breath counting which ‘isn't accurate’. (See Appendix S1 for quotes).
Many HEWs and FLHFWs expressed initial hesitation or fear of the device, explaining how they expected it be ‘confusing’ or ‘difficult to operate’, that they ‘hesitated’ due to lack of knowledge about the device and how they ‘feared’ to use it at first. These initial concerns were alleviated with practice, and most health workers said their ‘skills’, ‘knowledge’ and ‘ability’ to use the device had improved with time and experience. There was considerable discussion about the user interface and errors that impacted health workers’ ability to use the device. For example, many HEWs and FLHFWs mentioned that the device displays an ‘error’ message if the child moves or ‘there is motion’ and this ‘distorts’ the assessment, ‘affects the result’ and makes it difficult to ‘get the correct result’. Some mentioned having to start the assessment again, while others resorted to the ‘previous’ or ‘manual’ method with a few saying they ‘give up’ using it until the next day. Another problem with the interface related to selecting age categories; some FLHFWs explained that the categories are too close to each other resulting in other categories ‘being tapped’ by mistake, while HEWs and FLHFWs mentioned difficulty selecting the age of the child because mothers sometimes did not know the actual age of the child, which then led to inaccurate results.
Some FLHFWs and many HEWs said having the results displayed in the device made it easier to communicate, share or explain the results to caregivers. Across all districts, caregivers described feeling happy about the result of the assessment with the Rad‐G device, because the child ‘has no disease’ or because their child was ‘healthy’. Caregivers from all three districts indicated they would recommend the device to others, and the main reason was because the device ‘identifies disease’; others were keen to express that they would recommend the device based on the fact the it shows the ‘correct result’ or helps ‘rule out whether the child is positive or negative’.
There was widespread agreement among HEWs and FLHFWs and strong views against working on other tasks while Rad‐G is attached to the child and measuring the RR. Many health workers expressed the need to maintain their attention on the child, that ‘concentration’ is needed and that your ‘business should be only with the device and the child being assessed’. They felt a lot of care was needed to control the motion of the child and to ‘focus on the measurement’ and follow the assessment process. HEWs and FLHFWs agreed that older children were generally more easily distracted with the animal images or the ‘game’ on the device screen, although the specific age group varied from those over 1 year, over 2 years or older. Several HEWs talked more generally about children being distressed when they saw the device or the red light, but this was transient, and many calmed down when shown the pictures or the ‘game’ on the device screen (see Appendix S2 for quotes).
There was consensus among FLHFWs that they preferred Rad‐G compared with previous manual counting methods. The main reason stated was that Rad‐G provided an ‘accurate’ result, despite this study not measuring device accuracy. Others suggested it was easier than counting, that it was better than ‘simply observing’. A few mentioned that it enabled them to provide ‘better treatment’ and the fact it measured oxygen saturation was helpful. A few mentioned the durability of the device and that it does not rely on batteries that can easily run out.
HEWs also preferred Rad‐G compared with the previous method of counting RR, with most reporting Rad‐G was ‘easier’ than counting sometimes ‘repeatedly’ and the potential for ‘missing the correct rate’. Some emphasised the device was ‘faster’ than manual counting and gave the result more ‘quickly’ therefore reducing the assessment time. In contrast to FLHFWs, only very few HEWs mentioned the accuracy of the result as a reason for their preference for Rad‐G, rather they reported that their ability to classify pneumonia cases had improved, and this was important to avoid ‘guessing based on respiratory count’ and the ‘difficulties’ associated with classifying based on counting breaths.
HEWs and FLHFWs talked a lot about children's and mother's fear of the device and particularly the red light within the finger probe. According to the health workers, children would cry and become ‘afraid’ or ‘scared’ of the red light, thinking it is ‘fire’, while mothers also feared that the red light was fire or could burn they were also concerned that the red light might ‘suck blood’ from their child. FLHFWs described how caregiver's initial fear was usually allayed when they took time to ‘explain’ or ‘convince’ them.
When discussing their trust in the device and the results, many FLHFWs talked about checking the result several times before being confident in the result. Others mentioned checking the result with their own clinical observations and comparing the device result with their own ‘estimation’, and a few had ‘total trust’ in the device ‘regardless of drawbacks’ because it is ‘modern’ or the ‘latest technology’. Both HEWs and FLHFWs perceived that caregivers generally accepted Rad‐G, and many said this was because it is a ‘new device’ which had generated a lot of interest in the community. Across all three districts, caregivers reported broad acceptance of the Rad‐G. A common view among caregivers was that the ‘government brings only good things’ that do not cause harm and ‘what government brings does not worry’. FLHFWs and HEWs talked a lot about how having the device available at the health post or health centre enhanced either their own credibility or that of the service provided. Health workers explained that credibility was linked to caregivers’ preference for assessment with the device, and their belief that their children recover quickly if assessed with a ‘medical device like Rad‐G rather than giving treatment without a device’. HEWs were more likely to reflect that their own ‘motivation to work more’ had increased because people ‘appreciated my work’ or the device enabled them ‘to deliver an effective service’. Another common view among HEWs and FLHFWs was that the availability of the device for diagnosis combined with free treatment encouraged caregivers to the health centre and the health post, and those caregivers whose children recovered after ‘treatment using Rad‐G’ also encouraged others to attend. (See Appendix S3 for quotes).
This is the first study to examine usability and acceptability of a multimodal device with RR counting capabilities and pulse oximetry at the community level. The results are encouraging, demonstrated by high levels of usability, with HEWs adhering to the required steps in assessing and classifying U5 children using Rad‐G in more than three‐quarters of assessments. Furthermore, the data show that through adequate training and continued practice, using Rad‐G over a 2‐month period the HEWs’ ability to adhere to the required guidelines improved, as did their ability to calm the child before the attempt and get successful readings with the device, both for RR and SpO2 . Furthermore, we saw a significant association between the number of years since being trained and the adherence of the user. Therefore, it is key to provide adequate training, including clear instruction on using both clinical signs and device readings to correctly diagnose patients and adequate focus on device specific training, particularly on probe fit and signal strength. This was highlighted by HEWs when interviewed, who also stated that they found the support of the job aid valuable when classifying children using Rad‐G. Given the number of failures and difficulties expressed by HEWs using the device during observation 1 the research team conducted a half‐day focused training to support HEWs further on probe fit and signal strength issues with using Rad‐G. This reflects the findings in other recent studies which highlight the need for a task‐based active learning approach to training when introducing pulse oximetry in new settings. 25
High levels of acceptability were reported in this study, with both HEWs and FLHFWs expressing desire to keep using the device and feeling it improved their self‐efficacy and encouraged care seeking behaviours when combined with free treatment. Again, this reflects the findings of previous studies, which requested the development of these types of multimodal devices, offering both RR and SpO2 measurements. 25 , 26 Health workers also explained that they needed to provide a lot of explanation to alleviate children's fear of the red light emitted by the probe. This could have contributed to the high proportion of children who were not calm before and during assessments. Health workers also reported that trust in the device was low at first and that HEWs said they knew not to trust the result if the child was not calm.
A number of issues with the usability of Rad‐G were highlighted. Firstly, further work needs to focus on a potential redesign of the probe to ensure better fit, especially in the younger children. Such a probe has recently been developed and tested by Lifebox 26 and Masimo have also planned further development of their probe for the Rad‐G. Alternatively, multiple probes may need to be developed to allow these types of devices to be used effectively on all ages of children, as argued by King et al. 27 Secondly, the age cut‐offs need to be mutually exclusive to avoid any confusion when HEWs select the age group of the child. Finally, the classification options need to be simplified for the device to be effectively used by HEWs at the community level.
In this study, the average time to obtain a reading with the Rad‐G device for HEWs immediately after training was close to 7 min, ranging from 1 min 32 s to 36 min to 32 s. This reduced by just over a minute after 2 months of practice and still with a large range. While Masimo has developed the ‘Signal Extraction Technology’, or Masimo SET, which has been shown to better account for movement artefacts, 28 the manufacturer has confirmed that this technology has not been applied to the RR measurements from Rad‐G. This could explain the longer times this device was taking to get a successful reading. Further developments by the manufacturer to increase ease of use and reduce reading errors due to motion disturbance are therefore recommended to shorten this time period, as this may otherwise become a major hurdle for its wider use if rolled out in a busy health post or health centre.
A strength of our study is that data quality was maintained as source documents were used to verify the age selected by the HEW and the RR/SpO2 and classification of Rad‐G. A limitation of this study is that it did not compare the usability and acceptability of Rad‐G versus standard practice, nor did we use a ‘gold standard’ to review the HEWs’ management of the child, as done in similar studies of other RR diagnostic aids. 7 , 29 A further limitation is that a second refresher training was deemed necessary by the research team in the period between the two observations, in order to improve skills on probe fit and signal strength. This may have influenced results positively during observation 2. It also proved challenging to recruit children less than 2 months old from the health post, given that cultural barriers discourage women from taking their newborns out of the home when the babies are young. Instead, newborns were seen by HEWs during home‐based post‐natal check‐ups. A recruitment drive towards the end of data collection increased the proportion of young infants in the study, but the numbers were still small, and the study was not sufficiently powered to detect a difference in adherence to assessment and classification steps between age groups. While the research team took steps to minimise the possibility that HEW’s changed their behaviour when being observed, by silently observing child assessments and not interfering with the assessments, it is possible that the results are influenced by the ‘Hawthorne effect’. There is potential for courtesy bias in the qualitative data; it is possible that some interviewees may have responded in ways they felt were appropriate rather than reflecting their own experience or views. To minimise this, the interviewers ensured the HEW was not present for the caregiver interview and were trained to build rapport with the interviewee to make them feel comfortable.
The findings suggest that with adequate training and a comprehensive job aid, Rad‐G supports HEWs in SNNPR, Ethiopia to adhere to the required modified iCCM guidelines and manufacturer's IFU in children under five. HEWs and caregivers both found the device acceptable and usable. Specific modifications to the user interface and the probe could improve the usability and also the time taken to get a reading. The findings from this study support the rationale for further studies on performance, cost‐effectiveness and implementation, of this and other multimodal devices to inform policy decisions in countries with a high burden of childhood pneumonia.
We declare there is no conflict of interest as these are discrete pieces of work.
This study is conducted in partnership with UNICEF and was funded through a grant from the ‘La Caixa’ Foundation. The manufacturer of the devices (Philips and Masimo) did not contribute any funding for this work and did not participate in any data analysis or interpretation. The Malaria Consortium research team has subsequently received funding from the Philips Foundation to conduct future work on developing an improved RR reference standard.
This study was funded by ‘la Caixa' Foundation. The authors would like to acknowledge the data manager (Yenealem Reta), site coordinator (Eskinder Goshu) and all research assistants. We also express thanks to the HEWs, FLHFWs and caregivers of children under five in the SNNPR region, Ethiopia, who kindly participated in the study. The authors would like to thank the advisory committee, the SNNPR regional health bureau and Federal Ministry of Health for their and support of the study.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Tweets
Usability and acceptability of a multimodal respiratory rate and pulse oximeter device in case management of children with symptoms of pneumonia: A cross‐sectional study in Ethiopia
Twitter
Facebook
WhatsApp
LinkedIn
