Development and content validity analysis of artificial intelligence-generated Indonesian language insomnia questionnaire based on the International Classification of Sleep Disorders, Third Edition

INTRODUCTION

Insomnia is one of the sleep disorders with the highest prevalence and burden. Global prevalence of insomnia varies but is still considered high, ranging from 2.3% to 25.5%.^[1] A study showed that the prevalence rate in Indonesia is higher, ranging from 33.3% with sub-threshold insomnia and 11.0% of clinically significant insomnia.^[2] Despite the global magnitude of the problem, insomnia remains less reported into medical centers, with various local studies reporting that only 39.8% patients with insomnia symptoms consulted a general practitioner^[3] and only 25.6% patients with chronic insomnia symptoms have active health-seeking behavior for their problem.^[4] As a result, there are unmet needs between insomnia diagnosis and its proper treatment. Hence, a set of easy self-assessment questions to detect insomnia symptoms and to alert its user to seek medical help is required to fill the gap. To solve this problem, several questionnaires have been invented to bridge the diagnostic gap between health professionals and patients. The ever-changing insomnia criteria over the years further challenge the validity of the questionnaire. To date, there are at least two most established insomnia diagnostic criteria: Diagnostic and Statistical Manual, Fifth Edition (DSM-5) and International Classification of Sleep Disorders, Third Edition (ICSD-3).^[5] The original ICSD-3 is currently adopted by the Indonesian (ID) Neurology Association (Perhimpunan Dokter Neurologi Seluruh Indonesia [PERDOSNI])^[6] and has the potential to be developed into a local screening questionnaire. However, currently, in Indonesia, an ICSD-3-based insomnia questionnaire available in the local language does not exist. This prompts the additional need to develop a questionnaire in the local language beside of satisfying the currently accepted diagnostic criteria.

Artificial intelligence (AI) is a computational technology aimed to simulate human intelligence in doing tasks requiring the complicated nature of human intelligence such as problem solving, decision making, and humane communication.^[7] There is an AI function specifically designed to understand and analyze human language, which is called natural language processing (NLP).^[8] A large language model (LLM) is a form of AI, trained on a vast amount of linguistic data and capable of applying NLP.^[9] The LLM is known to be able to bypass medical language barrier and has been proposed to address the translational issue with non-native English (EN)-speaking patients.^[10] However, the ability of LLM being able to create a valid questionnaire under strict, specified, and standardized criteria is still being explored. Hence, in this study, we try to explore the AI ability, in the form of LLM, into creating an ICSD-3-based insomnia screening questionnaire in ID.

MATERIALS AND METHODS

Translation, review, adjudication, pretest, and documentation (TRAPD) and TRAPD homolog

The workflow of this study is homologous based on the conventional questionnaire translation method of TRAPD, which is followed by psychometric analysis of criterion validity, construct validity, and reliability.^[11,12] Traditionally, the initial Translation and Review process was done by a clinical Expert Panel. The results were then consulted with the linguistic Expert Panel who provided back translation. The back translation result was then resubmitted to the clinical Expert Panel for feedback response. These whole feedbacks between clinical and linguistic experts are summarized as the Adjudication process. In the pre-test stage, the final questionnaire then pre-tested to achieve data. Documentation is an important stage in which data and process transparency can be audited. This whole conventional TRAPD process is summarized in Figure 1 (blue shade).

Close

Figure 1:

Comparison between large language model (LLM)-based questionnaire development and conventional translation, review, adjudication, pretest, and documentation (TRAPD)-based questionnaire translation process. The questionnaire generation process used in this study is homolog to the conventional TRAPD approach. The translation and review process being bypassed entirely by LLM is the main difference between TRAPD and this study’s methods. Early grammatical revisions in the LLM workflow can be seen as a prompter task with expert panel supervision. In TRAPD, the content validity of a novel questionnaire item, including a translated version, can be assessed as early as the review process by clinical experts has been done. The criterion validity, construct validity, and reliability instead require at least pretest results from target respondents, which are not covered in this study. Color code: Orange shade: The scope of the current study; Gray shade: Part of TRAPD homolog which is not the scope of the current study; Blue shade: Conventional TRAPD process; Purple: Position of human profession/task relative to each process. The color code is based on Prompter tasks on the TRAPD analogue for LLM-based questionnaire development process: Dark orange: Idea grounding, Light orange: Construct engineering, Light yellow: Questionnaire realization, Pale yellow: Grammatical revision

Export to PPT

The TRAPD homolog is an attempt in this study to implement AI in the form of LLM into the conventional TRAPD method. In this study, the GPT-4.0 is tasked to specifically perform only the translation and review stage, bypassing the clinical expert role in the conventional TRAPD method. In this study, we try to minimize the human input on the LLM to achieve a questionnaire product that is ideally independent from direct human influence. The minimization is achieved by segregating the human and LLM roles and by segregating expert influence during the questionnaire creation process. This minimization resulted in a study workflow homologous to the TRAPD [Figure 1, orange shade].

The role of LLM is limited to initial Translation and Review based on this study aim. Due to the nature of the original TRAPD where language expert role is started in the Adjudication stage and the requirement of satisfying content validity from the previous stages, the TRAPD homolog in this study did not involve linguistic evaluation, e.g., back translation and linguistic validity analysis [Figure 1, gray shade]. With consideration where the LLM-generated item content validity is unknown, then the adjudication, pre-test, and following psychometric analysis of criterion validity, construct validity, and reliability were not performed until this study produced enough evidence to initiate the mentioned stages.

RESULTS

DISCUSSION

This study demonstrates the ability of an LLM in creating an insomnia questionnaire with a prompt engineered for a certain topic. The performance of an LLM in creating a specified questionnaire has been explored in several studies and has produced satisfying results. A study has compared the medical-examination questionnaire performance between GPT-based LLM and humans, in which no statistically significant difference was found in item difficulty between both makers.^[17] There are even comparisons between LLMs in the making of a medical-examination questionnaire with significant differences between validity and difficulty on each model,^[18] suggesting that difference in performance is caused by different training datasets and LLM architecture. The completeness of training data has been a major issue for LLM performance, especially in the field of medicine. This prompts the need for manual, human-involved benchmarking for LLM regarding potential performance issues, such as uncertainty and data presentation accuracy.^[19]

The LLM in this study is able to cite ISI and generate ICSD-3-based questionnaire items, although it cannot initially identify ICSD-3 diagnostic criteria for insomnia. It is important to notice that although the LLM in this study is able to cite ISI, the comparison between the coverage of the ICSD-3 construct in the LLM-generated and ISI items is noticeably different [Table 2]. The ISI is mainly based on DSM-Fourth Edition (DSM-IV), one of several classification systems that describe insomnia.^[20] The ever-changing insomnia diagnostic criteria provide a basis for various classification systems and screening instrument development. For example, the Athens Insomnia Scale (AIS) is based on the International Classification of Diseases, Tenth Edition,^[21] ISI is based on DSM-IV,^[20] Sleep Condition Indicator (SCI) is based on DSM-5,^[22] and Insomnia Screening Scale (ISS) is based on ICSD-2.^[23] The DSM is the earliest to describe insomnia diagnostic criteria by comparing it to other mental disorders. The DSM was revised in 2022 as DSR-5-Text Revision (DSM-5-TR), and it contains eight diagnostic criteria for insomnia. The ICSD-3 was released in 2014, 8 years earlier than DSM-5-TR.^[24] Compared to DSM-5, the original ICSD-3 does not segregate insomnia from sleep difficulty caused by mental illness, drug and substance abuse, or organic comorbidities. The segregation was only applied in the text revision (ICSD-3-TR) in 2023 under one criterion only.^[25] In a coincidental manner, this recent segregation may help explain our finding which is the expert panel’s relatively lower Aiken V score for the sixth item, which is in turn generated based on pre-revised ICSD-3. Based on these evolving criteria, the LLM in this study has been shown to be qualitatively able to produce an instrument containing questions needed to fulfill the data required for the most recent diagnostic criterion – the ICSD-3-TR. Compared to all of the previously mentioned instruments, the LLM-generated questionnaire also demonstrated fewer questions (7 items) compared to AIS (8 items),^[21] SCI (8 items),^[22] and ISS (26),^[23] while on par with ISI (7 items).^[20] Limitations also observed relative to all other mentioned questionnaires, such as uneven generation of Likert scale between each item as well as the decision of the LLM to explain how the total score should be calculated but not explaining how the score should be categorized, leaving the questionnaire open to further validity analysis.

This study demonstrates the ability of LLM in creating a questionnaire directly in ID language. Since the LLM is mainly trained in EN, a combination between the conventional questionnaire development method and the conventional instrument translation method should be used. Because this study does not involve a human expert in the questionnaire generation process, a process dedicated to content validation is mandatory. Hence, we deploy similar methods used in the development of recovering quality of life measure instrument. The methods revolve around four processes which are (1) generation of candidate questionnaire items, (2) content validity assessment, (3) psychometric evaluation, and (4) selection of final questionnaire items.^[26] Item generation traditionally requires experts across multidisciplinary fields to ensure the content validity, as demonstrated in the creation of the Adolescent Insomnia Questionnaire.^[27] However, in this study, the questionnaire item generation phase is handled solely by AI with an unknown dataset completeness without expert intervention. Studies showed that item generation and data extraction under non-standardized prompts and unknown LLM dataset transparency will lead to accuracy, consistency, and reliability issues.^[28-30] Combined with our findings, we suggest the need for prior LLM dataset confirmation during item generation, which in this study was realized as the deployment of specific questionnaire generation phases in this study.

The use of auto-translation-capable LLM in this study provides insight into how we should address the foreign instrument translation issue in the era of AI. Questionnaire translation is usually done conventionally by performing six different stages encompassing five essential processes of TRAPD,^[11,12] in which only the first two stages are covered in this study. The first stage, “Translation”, traditionally requires language translation by native and bilingual speakers, which is bypassed in this study by AI. The second stage, “Review”, requires an expert round to assess the content validity and to modify questionnaire wording, which is proven to be unskippable even in this study as evidenced by multiple revisions during questionnaire generation.

Aside from content validity, linguistic validity is also important to measure comprehensiveness when the questionnaire is presented into respondents. Linguistic validity affects the degree of understanding of respondents when filling out the screening questionnaire due to sociocultural variations in society. Studies showed that every questionnaire in development requires its own validation by a linguist.^[31,32] Linguistic validity can be measured using a simple Likert scale, involving bilingual experts^[33] as well as real target respondents.^[34] Considering the unknown or parameter of linguistic validity of AI and AI products, the role of linguists is still needed in parallel to the content validation process.

Content validity of an instrument can be examined from several methods, such as Aiken V and CVI, which all require involvement of a human expert for critical assessment.^[15,16] Content validity of non-ICSD-3-based insomnia questionnaire has been reported previously. The most prominent insomnia screening instrument for professionals, the Pittsburgh Sleep Quality Index, has S-CVI/Ave of 0.905, which indicates a good overall content validity.^[35] The Holland Sleep Disorders Questionnaire has an item CVI between 0.83-1, suggesting a high validity value.^[36]

Content validity is often not deeply explored because the conventional instrument creation process already directly involves experts in their field. This lack of exploration has been recorded in several studies. For example, a systematic review on the structural validity of ISI showed that validity component is mostly explored as internal consistency, test– retest reliability, construct validity, and diagnostic validity, but not content validity.^[37] However, in the age of AI capable of creating similar instruments with unknown source transparency, analysis of content validity has become more relevant than ever.

The use of LLM for creating a novel clinical questionnaire is not without ethical concern. As demonstrated in this study, LLM claims to create novel questionnaire items based on ISI. The actual ISI, however, only covers Criteria A and Criteria B of chronic and short-term insomnia based on ICSD-3. This is because ISI itself is based on totally different insomnia diagnostic criteria, the DSM-IV.^[37,38] There are clearly questionnaire choices with more updated criteria or even a straight ICSD-based choice such as SCI or ISS; however, the GPT-4.0 reason behind choosing ISI as the first answer remains in question. This finding indicates a major bias in LLM output. It is worth noting that the LLM itself cannot be free from bias, since bias is the natural state of any machine incorporating a neural network. This “natural bias” can be explained by the iteration of a random number sequence in a specific length known as the “seed”, which is the staple method used by machine learning workflows.^[39-41] Each number value and position in the seed provides “weight” relative to the input given, and this further dictates the following answer spectrum.^[42] This answer spectrum was then correlated in the neural network hidden layer to generate the final output, resulting in dynamic but natural-looking responses. This initial randomization, however, becomes a major problem when using machine learning as a scientific tool because it may reduce,^[43] inflate,^[39,40] or even eliminate scientific reproducibility, unless the exact initial seed is known.^[44] The neural network hidden layer itself is another ethically controversial subject. A hidden layer is a specified and correlated linear network of a trained dataset to recognize given input.^[45] Concerns had risen from multiple societies and disciplines given the nature of these trained datasets, especially in LLM without clear data training disclosure.^[46-49] These include but are not limited to how the data are obtained, what happened to licensed data, how the data can be recognized in the hidden layer, how the data are trained, and how the data are validated before the training.^[47,50-52] Due to the nature of machine learning, these trained data are also able to update themselves based on further response to the output, creating substantially different and more desirable output – hence the learning part of the machine learning.^[42] This learning capability, combined with a hidden seed, may further complicate the former bias problem, since under the learning influence, the output is updated too, technically eliminating exact output reproducibility.^[40] Further complicating this problem is the nature of the dataset used in LLM training. A review in 2024 showed that an incomplete dataset may affect LLM output due to bias caused by imbalanced sample size, non-random missing data, uncaptured/not-easily-available data, misclassification during dataset labeling and input, and imbalanced origin of data publication (racial and ethnicity bias).^[53] While these ethical issues can be avoided by creating custom-made, task-specific LLM using open-source options, the rarity of expertise and resources currently creates difficulties to avoid the use of more accessible but unspecialized LLM with questionable data disclosure. In the dataset completeness context, several technical strategies have been proposed to mitigate this bias problem. The strategies include but are not limited to minority data oversampling, data augmentation, usage of a standardized reporting checklist, avoiding deliberate acts of filling missing data (non-imputation), compliance with unified expert consensus, and usage of standardized data criteria for predictive purposes.^[53] Based on our findings and the existing evidence, human intervention in the form of an expert validator is still required to mitigate biases that may happen in LLM-generated products such as a diagnostic-screening questionnaire.

There are several weaknesses limiting this study. The first is the limitation on the content validity analysis only. A good questionnaire should satisfy not only content validity but also construct validity, criterion validity, and reliability, which is not evaluated in this study. Further investigation involving a population of respondents addressing initial psychometric data covering construct validity, criterion validity, and reliability is required to fully implement this questionnaire. Second, although this study covers local expert opinions on the auto-translated revision, this study also did not consider linguistic evaluation for the addressed linguistic validity. Based on the Adjudication phase in the TRAPD homolog, we deemed that linguistic evaluation of the LLM product is an important aspect, yet an unknown subject that still requires investigation from future research. We also suggest future studies aimed at comparing the results between blinded Adjudication and blinded pre-test stage parallel to their non-blinded counterparts, to investigate the effects of minimum human intervention during LLM workflow. Third, this study relies on a commonly available, unspecialized LLM with an unretrievable random seed value and unknown dataset completeness, which hinders exact reproducibility of the generated item under the same prompt. Our study results suggest that proprietary LLMs may provide very powerful computational speed and access, but their limited dataset has been proven to severely limit the response results, no matter how detailed the given prompt is. This demonstrates that a specific LLM with a more transparent dataset and open-sourced model architecture is much more compatible, valuable, and important in the field of medicine.

CONCLUSION

This study shows that AI is able to generate an insomnia questionnaire and satisfy content validity requirements. This study also shows the general ability of AI in bypassing language barriers, as well as demonstrating that human intervention is still required to address various specific issues involving topic stability and language diction. The final product of this study is meant to be a set of easy, self-assessment questions to detect insomnia symptoms, and to alert is user when to seek medical help. While some of the starting requirements for a good questionnaire have been evaluated in the study, further studies, such as in the scope of linguistic validity and questionnaire psychometry, are still required to examine whether LLM is ready for truly assisting the medical diagnostic process.