Assessing the quality of evidence for verbal autopsy diagnosis of stroke in Vietnam

Introduction

Mortality is one of the most important health indicators for measuring a country's health and development. Knowledge about the distribution of causes of death in populations is essential for public health planning, resource allocation and measuring the impact of interventions. Although complete and reliable mortality statistics are the backbone of population health assessment, program evaluation and epidemiological research; mortality data are often poor in developing countries because of inadequate death registration.[1] Medically certified cause-of-death data are available only for less than one-third of over 57 million deaths occurring worldwide annually.[2] With weak vital registration systems, and a low proportion of people who die under medical care, the verbal autopsy (VA) method is a potentially important alternative to ascertain cause of death in developing countries.[34]

The accuracy of VA methods in correctly ascertaining the cause for each individual death is crucial in order to have confidence in the data collected through such methods. The VA method is based on the assumption that the distinct symptom complexes pertaining to specific causes of deaths can be recognized, remembered and reported by lay respondents. In addition to this information content in the completed questionnaire, the VA method also relies on the physician reviewer's judgment in applying diagnostic guidelines to assign specific causes of death. This judgment would inherently depend on the content and quality of information in the completed VA questionnaire.

Accuracy of the VA procedure is generally established by carrying out validation studies. By definition, validation involves a comparison of the underlying cause of death derived from the VA with a ‘true’ or reference underlying cause for the same death, derived from either a pathological autopsy (‘gold standard’) or clinical records.[5] As hospital records have their own limitations in terms of availability and quality; choice of the reference diagnosis for the underlying cause has been a basic challenge for studies assessing the validity of VA procedures. Further, validation is only possible in cases for which reference diagnoses are available, and this is clearly not the case for most deaths in developing countries, which occur outside health facilities, and often with no medical attention during the illness preceding death. Therefore, in such cases, the next best alternative for verification would be to assess the content and quality of evidence recorded in the VA questionnaire, to examine the utility of recorded responses to construct a diagnosis of the cause of death.

In the absence of mortality data from routine vital registration systems, Vietnam has recently pilot tested a sample mortality surveillance system, using verbal autopsies to ascertain causes of death.[67] Briefly, the sample consisted of 192 communes distributed through 16 provinces across Vietnam. Details of the sampling plan, data collection and analysis are described elsewhere.[78] In regard to cause of death ascertainment, all field data collection and processing in the main project including VA questionnaires and physician review was based on standard international guidelines[7] adapted to the Vietnamese context.[9–11] The study has identified stroke as the leading cause of death in Vietnam with the estimated age-standardized mortality rates from stroke being 139/100000 for males and 70/100000 for females.[12]

Given this magnitude of stroke mortality in Vietnam, there is a need to evaluate the quality of stroke diagnoses from VA, to support these findings. We attempted to validate the VA diagnoses in a sample of deaths that had occurred in hospitals, but could not access medical records to derive reference diagnoses, and therefore could not perform a validation study. As an alternative, we attempted to verify the accuracy of VA evidence for deaths diagnosed as stroke by physicians in Vietnam. Stroke has typical and specific symptoms that could be recalled and reported during the VA interview, and hence is suitable for assessment of evidence recorded in VA questionnaires. The overall objective of this manuscript is to describe a study on the quality of supporting evidence for diagnosis of stroke in a sample of VA questionnaires. We first reviewed the content in each VA questionnaire, then applied defined criteria to the VA responses to categorize them into cases with high, medium or low-quality of evidence for stroke diagnosis; and finally assessed the influence of contact with health facilities prior to death on the quality of evidence for stroke deaths.

This knowledge would enable an understanding of the quality of information that aided diagnosis of stroke as well as provide a basis for strengthening verbal autopsy data collection. Moreover, stroke is a leading cause of death in developing countries.[13] As the majority of deaths in developing countries occur in rural areas and since VA is used for ascertaining cause of death,[14–17] it is important to know the reliability of VA for stroke deaths.

Materials and Methods

Study population

The overall mortality surveillance activity covered 16 provinces across Vietnam, and field work was supported by a network of five medical universities located in different regions of the country. Each medical university functioned as a node for fieldwork coordination, data management and processing, and archival of all physical study records pertaining to their respective geographic areas of responsibility. This study on quality of evidence for stroke diagnoses from VA was conducted in four provinces in the north and middle of Vietnam namely Hanoi, Hai Duong, Quang Ninh and Thanh Hoa [Figure 1], where fieldwork was supported by the Hanoi Medical University. All VAquestionnaires for deaths that occurred in 2008 in the four selected provinces were collected and arranged. At first, the final causes of death were studied for each of these questionnaires, in terms of the ICD codes entered by the diagnosing physicians, and all questionnaires with an underlying cause of death from cardiovascular disease were selected for the study. A total of 396 deaths were identified with underlying causes related to the broad cause group of cardiovascular diseases. On closer scrutiny, we found that the vast majority (326) of deaths were from cerebrovascular diseases i.e., stroke (ICD codes (I60-I69)). The remaining deaths comprised of 50 from ischemic heart disease (I20-I25); 11 from hypertensive heart disease (I13), and 9 from other cardiovascular conditions. For the purpose of assessing quality of evidence for stroke diagnosis, we selected the 326 VA questionnaires with deaths assigned to cerebrovascular diseases as the study sample. We also evaluated the remaining 70 cardiovascular cases for evidence of recording a positive history of paralysis or prior history of stroke.

Close

Figure 1

Study area selected for the VA content analysis study

Export to PPT

Of the 326 stroke cases, there were 187 cases with hospital admission and the rest 139 were without hospital admission. All cases that visited a health facility and had a hospital admission at least once anytime during the course of their illness, irrespective of the duration of hospital stay or whether the death occurred at hospital or at home, were considered as cases with hospital admission. In contrast, those who had never been admitted to a hospital during their illness were considered as cases without hospital admission.

Statistical analysis

All data was double entry verified. STATA 11 and Microsoft Excel were used to analyze the data. The frequency distribution of paralysis was calculated on the basis of positive response to the structured question of the symptom and/or mention of paralysis in the free text. Chi-squared test was used to examine the statistical significance of differences in symptoms reported between cases with and without hospital admission. Descriptive analysis was carried out for the duration and site of paralysis for cases that had a positive response to the symptom of paralysis. Duration and site of paralysis were only mentioned in the structured question. So, all the frequency and percentage were calculated based on the responses to the structured question on duration and site of paralysis. A comparison was made between cases with and without hospital admission for the duration and site of paralysis also.

Paralysis accompanied with unconsciousness strengthens the diagnosis of stroke. Hence, the frequency of the symptom of unconsciousness was calculated by using the number of cases with positive response to unconsciousness out of the total cases of stroke. In the VA questionnaires, like paralysis, unconsciousness was mentioned either in the structured question or in the free text recorded through the narration given by the respondent. Thus, the response to unconsciousness was analyzed in structured question or free text alone; in both the structured question and free text and in either of the two. The percentage of paralyzed stroke deaths with and without unconsciousness was calculated and a comparison was made between cases admitted to hospital and those who were not.

Based on the findings from preliminary descriptive analysis, an attempt was made to provide a summary score of quality of information in VA questionnaires that was available to support the physician diagnosis for each case; and aggregated for all cases from each cause. For example, good-quality evidence for stroke would have positive symptoms (e.g., paralysis) recorded in both structure and free text and a history on stroke, or diagnosis from hospital. Low-quality evidence for stroke would have fewer positive symptoms or no direct clue in the VA questionnaire for the diagnosis [Table 1].

Table 1 Quality of evidence categories for stroke

Results

Table 2 provides some baseline characteristics of our study sample. Fifty-seven percent of the 326 cases diagnosed as stroke were reported to have an admission to hospital during the illness. In our sample, males accounted for a slightly higher proportion of stroke deaths (57%). In terms of age, as expected mortality due to stroke for those over 60 years was higher than those under 60 years of age. However, the differences between gender and age groups were not statistically significant (P > 0.05).

Table 2 Distribution of stroke by province, gender and age group

Evidence of stroke in VA

Overall, 72% of stroke deaths had a positive response to the structured question of paralysis in the verbal autopsy questionnaire. As high as 80% of the stroke cases with hospital admission had a positive response to paralysis in the structured question as against 60% cases of stroke with no hospital admission [Table 3]. The difference in the response to paralysis in cases with and without hospital admission was statistically significant (P < 0.001).

Table 3 Response to paralysis in stroke cases with and without hospital admission

Besides the positive response to paralysis in the structured question, there was a mention of paralysis in the free text in 52% of cases with a diagnosis of stroke. Of the 92 (28%) cases for which there was either no paralysis or no information about paralysis in the structured question, 12 were found to have a mention of paralysis in the free text. In this way, in 246 (76%) cases out of 326 cases of stroke, there was an evidence of the presence of paralysis either in the structured question or in the free text [Table 3].

The remaining 80 cases with no evidence of paralysis either in the structured question or in free text were examined further for the presence of a history of stroke. Out of these 80 cases, a history of stroke was present in 47 cases. Hence, in 293 cases (90%), out of the 326 cases for which a diagnosis of stroke was made; the diagnosis was based on either the presence of the evidence of paralysis in the verbal autopsy questionnaire or on the basis of a positive response to history of stroke. The difference in the hospital and nonhospital groups was statistically significant (P < 0.001) with 94% of the cases diagnosed as stroke having an evidence of either paralysis or past history of stroke amongst those having hospital admission as against 85% among those without hospital admission. The distribution and inter-relationship of cases with presence of the symptoms of stroke is shown as Venn diagrams in Figure 2.

Close

Figure 2

Breakup of the stroke cases according to the response to the questions pertaining to evidence of paralysis and history of stroke in verbal autopsy questionnaire. (a) Positive response to the structured question of paralysis (b) Mention of paralysis in free text (c) Positive response to history of stroke

Export to PPT

It can be seen from the Venn diagrams in Figure 2 that all 175 (94%) cases that had a hospital admission and 118 (85%) cases that had no hospital admission had evidence of paralysis or history of stroke in the verbal autopsy questionnaires.

Of the remaining 12 cases with hospital admission, in four cases a hospital diagnosis of stroke was mentioned in the VA form, and four cases had a history of long standing hypertension. Similarly, of the remaining 21 cases without hospital admission, nine cases had a history of hypertension either in the structured question or in free text.

In the remaining 16 cases (four with hospital admission and 12 without hospital admission), there was no objective evidence in the verbal autopsy questionnaires to support a diagnosis of stroke. However, when analyzed further, almost all cases were found to have some evidence in the form of nonspecific subjective symptoms that might have been considered by the diagnosing physicians in conjunction with their own clinical judgment (as depicted in the flow chart in Figure 3).

Close

Figure 3

Flow chart depicting the potential hierarchy of symptom based evidence to support VA diagnosis in 326 cases of stroke

Export to PPT

As with paralysis and past history of stroke, there was either a positive response to the structured question of unconsciousness or a mention of unconsciousness in the free text in the verbal autopsy questionnaires. The positive response to structured question was present in 63% of the cases whereas an evidence of unconsciousness was found in overall 72% of stroke cases either in the structured question or in free text.

More than half (55%) of the stroke cases with a positive response to the structured question of paralysis had asymmetrical paralysis; involving either one half of the body (hemiplegia) or a single limb (monoplegia), 34% had paralysis of the whole body and 10% cases had other indeterminate body parts involvement. The duration of paralysis ranged from half a day to 11 years. Of the 234 cases with a positive response to the structured question of paralysis, approximately one third (31%) had paralysis for more than one year. Twenty-seven percent of cases experienced paralysis ranging from 1 month to 1 year and the remaining 42% were reported to live with paralysis for less than 1 month.

Quality of medical evidence

Table 4 summarizes the proportion of cases with high, medium and low-quality evidence based on the criteria defined in Table 1. It can be seen that 43% of the cases (141 out of 326) had high-quality evidence and 48% (157 out of 326) had medium-quality evidence. The remaining 9% cases had low-quality evidence. Whereas, as many as 48% of the cases with hospital admission had a high-quality evidence, only 37% cases without a hospital admission belonged to the high-quality evidence category. This difference between cases with and without hospital admission was found to be statistically significant (P = 0.001).

Table 4 Distribution of quality of evidence to support the diagnosis of stroke

Influence of contact with health facility

There was a significant difference in the positive responses to almost all the symptoms in cases with and without hospital admission. As high as 80% of the cases in the hospital group had a positive response to the structured question of paralysis while 60% in the nonhospital category had this response (P < 0.001). Similarly 73% of the cases in the hospital category had a history of stroke in the structured question as against 63% in the nonhospital group (P = 0.03). The difference was evident in the quality of evidence categories also as 48% cases with hospital admission had a high-quality evidence but only 37% cases with no hospital admission belonged to the high evidence category (P = 0.001).

When considering paralysis as a cardinal sign of stroke, it is importrant to assess the potential for VA questionnaires to record paralysis but the reviewing physician assigns a cause of death other than stroke. To study this phenomenon, we reviewed the 70 cases that were assigned to cardiovascular conditions other than stroke, for the presence of paralysis. As seen in Table 5, only 23% of nonstroke cases had a mention of paralysis in either the structured response or the free text. Our review identified that the questionnaires in all these cases had substantial evidence to support the diagnosis of the other major cardiovascular condition recorded as the underlying cause of death, thereby resulting in the physician overlooking the presence of paralysis in these 23% of cases. This assessment underscores the element of subjectivity in VA interviews, as well as in clinical judgment applied in ascertaining causes of death from VA

Table 5 Response to paralysis in nonstroke CVD cases with and without hospital admission

Discussion

Paralysis was considered as a typical cardinal manifestation of stroke from the VA point of view since respondents can be expected to have a good recall of paralysis, especially when it is unilateral. Although paralysis alone may be an indication of conditions other than stroke, its presence in the VA questionnaire was taken as a certain indicator for stroke. The VA questionnaire used in Vietnam is designed to capture specific responses to structured questions as well as the narration of the respondents in the form of open ended free text. Hence, the response to paralysis was looked for in both these sections. As one can expect, there were more cases with positive response to the structured question (72%) than the mention of paralysis in the free text (52%) since a specific description of the symptom might have helped some respondents recall the condition of the patient who otherwise forgot to mention the symptom in their narration. Consistent with this expectation, there was good overlap of the mention of paralysis in the cases with a positive response to the structured question with two-thirds of respondents having given a positive response to structured question also mentioning paralysis in their narration. Hence, the structured question appears to be more sensitive in recording the symptom that is considered to be a typical clinical presentation of stroke.

On the other hand there were nine cases in which the respondents gave a negative response to the structured question but mentioned paralysis in their narration (out of a total of 73 cases with a negative response). There was also one case with mention of paralysis in the free text with a ‘don’t know’ response to the structured question (out of a total of 8 cases with a ‘don’t know’ response) and two instances with mention of paralysis in free text in cases where structured question responses were left blank by the interviewers. Although we have considered all these instances as a positive evidence of stroke in the VA in terms of presence of paralysis, it may be likely that the interpretation of ‘paralysis’ is different amongst respondents and interviewers depending on the prevalence of local dialects and fine shades of meaning. The interviewers, being paramedical personnel might have a more technical perspective of the term than respondents whose interpretation might be more general. For example, the term “liet” was used in the verbal autopsy questionnaire for “paralysis” but “liet” may also mean “transient limping” in some areas. Some respondents who actually denied the presence of paralysis mentioned “weakness” of the limbs or “inability to move” the limbs in their narration. From the clinical point of view, these cases cannot be considered as having paralysis as a symptom of stroke.

This difference in understanding meanings of the same word is also evident in the description of the term "stroke". With reference to stroke, there are two words in Vietnamese; viz. “dot quy” and “tai bien mach mau nao” that when translated into English mean stroke (or cerebrovascular condition). However, Vietnamese community perspectives on “dot quy” and “tai bien mach mau nao” seem to be different in many ways. The term “tai bien mach mau nao” has a more technical element and is generally used by the medical fraternity. This term resembles a clinical perception more than a community understanding of the condition. As a result, the positive responses to the structured question on history of stroke were more frequent in cases with hospital admission as this term was used for the particular question. In some cases, “tai bien mach mau nao” was mentioned in the free text by the respondents. However, all these cases had been in contact with health facilities in some way or the other.

The cases without an evidence of paralysis were examined for the presence of a history of stroke in the VA. Like paralysis, both structured question and free text were examined for the presence of a history of stroke. Respondents of 25 cases who responded ‘no’ to the history of stroke when specifically asked about it (structured question), mentioned about a history/diagnosis of stroke in their narration. This may again be due to differences in the understanding of interviewers and respondents. It is possible that the respondents might not have mentioned the exact word ‘stroke’ but the interviewers interpreted it by their explanation of the signs and symptoms of the disease (section 6.3). It may also seem apparent that a positive response to the past history of stroke is only likely from respondents of cases where the patient had a hospital admission. However, in contrast to this assumption, 71% of the cases in the nonhospital admission category also reported a past history of stroke (either in structured question or free text). This could be due to the fact that the cases without a hospital admission might have consulted a physician on an out-patient basis or have been in contact with a private health facility from where they became aware of the diagnosis of stroke.

With 90% of stroke cases diagnosed based on strong and medium-quality evidence, we can be confident that VA is a viable alternative to identify deaths caused by stroke in the current study. When a complete set of symptoms associated with stroke including paralysis, history of stroke, history of hypertension and unconsciousness was considered, one-third plus (37%) of the VA diagnosed stroke cases had this set of symptoms. However, it should be noted that history of hypertension is highly subject to recall by the respondents as is unconsciousness, although to a lesser extent. Also, there might have been some cases having both paralysis and unconsciousness but only unconsciousness was reported by the respondents in the VA as they might have been unable to recognize paralysis in presence of deep unconsciousness.

The differences in the responses to various symptoms in the cases by their status of hospital admission shows that the relatives of the deceased with hospital admission were better able to recall the symptoms. This finding is in accordance with the results of many previous studies that show the direct relation between patient access to health care facilities and the recall of symptoms by the relatives.[20–22] This is an inherent source of bias and one of the major limitations in all validation studies that consider facility assigned diagnosis as the gold standard for comparison.

In addition to the limitations of VA described above, our study findings have potential limitations in terms of generalizability. As per the overall protocols, VA questionnaires were archived at each of the five medical universities that were charged with coordinating the mortality surveillance activity. Owing to time and resource constraints, we were able to confine our activities to only one medical university, and we chose to access and analyse the VA questionnaires from the four northern provinces for which activities were coordinated by Hanoi Medical University. Hence, our sample of 326 stroke deaths was only 22% of all stroke deaths in the national sample, which could limit the generalisabiity of our findings on quality of evidence in VA questionnaires. Further, we only explored the remaining 70 deaths assigned cardiovascular causes for potential misdiagnoses, in terms of presence of paralysis but classification to causes other than stroke. There is a likelihood of deaths assigned noncardiovascular causes also to have recorded paralysis but misdiagnosed to other causes. The overall surveillance protocol was based on physician review of VA questionnaires based on specific training programs in line with international standard guidelines for cause of death assignment from VA, adapted to the Vietnamese context. In case of diagnostic uncertainty, physician reviewers could assign a nonspecific cause, which accounted for 13.7% of all deaths in the overall data.[7] We, therefore, anticipate that physician reviewers would have based their diagnoses according to the standard guidelines, and used the option of assigning a nonspecific cause where necessary. Despite this, there are likely to misclassification errors in the overall data, but we did not make an assessment of such errors, due to resource constraints. Given the magnitude of stroke mortality in the overall data, we chose to focus on the quality of evidence supporting stroke diagnosis from VA in a sample of cases for which stroke had been assigned as the underlying cause of death. To some extent, this possibly affects generalization of our findings on the quality of evidence to all stroke deaths in the overall study sample.

Conclusions

Verbal autopsies are a useful approach to ascertain stroke deaths in the community. Our finding that about 90% of stroke deaths in our sample had some evidence of paralysis in the VA provides a measure of reliability of the estsimated stroke mortality rate from the national mortality surveillance system, although there could be limitations in the generalisability from our sample. The review of the specific symptom questions and an analysis of the respondents’ narration recorded in the form of free text in the verbal autopsy questionnaires shows that VA is effective for stroke and VA content analysis can be utilized as an adjunct to conventional validation studies in developing countries where availability of medical records is a problem. To obtain a broader perspective and to validate (or invalidate) the current findings, it would be necessary to conduct further research on the content analysis of the verbal autopsy procedure.

The understanding of the VA content would also form the basis for improvement in the quality of interviews and collection of data to achieve better quality information in future.