This paper is in the following e-collection/theme issue:

Research Instruments, Questionnaires, and Tools (663) Information Quality in Digital Media (374) Information Seeking, Information Needs (525) eHealth Literacy / Digital Literacy (458) Assessing and Building eHealth / Digital Literacy in Populations (46) Infodemiology and Infoveillance (1454) Information Retrieval (323)

Published on in Vol 27 (2025)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/62884, first published .
Development and Validation of the Media Health Literacy Scale: Assessment Tool Development Study

Development and Validation of the Media Health Literacy Scale: Assessment Tool Development Study

Development and Validation of the Media Health Literacy Scale: Assessment Tool Development Study

Authors of this article:

Sangyoon Shin1 Author Orcid Image ;   Seungyeon Kim2 Author Orcid Image ;   Youngshin Song3 Author Orcid Image ;   Hyesun Jeong4 Author Orcid Image ;   Yun Mi Yu5 Author Orcid Image ;   Euni Lee1 Author Orcid Image
Article Authors Cited by Tweetations Metrics

    Original Paper

    1Research Institute of Pharmaceutical Sciences, Natural Products Research Institute, College of Pharmacy, Seoul National University, Seoul, Republic of Korea

    2College of Pharmacy, Dankook University, Cheonan si, Chungnam, Republic of Korea

    3College of Nursing, Chungnam National Univerisity, Daejeon si, Republic of Korea

    4Department of Nursing, College of Nursing & Health, Kongju National University, Gongju-si, Republic of Korea

    5Department of Pharmacy and Yonsei Institute of Pharmaceutical Sciences, College of Pharmacy, Yonsei University, Incheon, Republic of Korea

    Corresponding Author:

    Euni Lee, PharmD, PhD

    Research Institute of Pharmaceutical Sciences, Natural Products Research Institute

    College of Pharmacy

    Seoul National University

    1 Gwanak-ro, Gwanak-gu

    Seoul, 08826

    Republic of Korea

    Phone: 82 2 740 8588

    Email: eunilee@snu.ac.kr


    Background: Advancements in IT have transformed methods for accessing and conveying health-related information. While technical advancements offer more options for people to choose their preferred information sources, injudicious dissemination of incorrect or unverified health-related information by internet-based media poses a threat to society. The concepts of media health literacy (MeHlit) and eHealth literacy have emerged for assessing one’s ability to understand and use health-related information from media sources. However, tools to evaluate the level of MeHlit within the domain of communication or follow a solid validation process are scarce.

    Objective: This study aimed to develop a validated tool to evaluate the level of MeHlit in adults in South Korea.

    Methods: A 2-step tool development process, including item development and validation processes, was carried out. At first, tool development studies were identified by a systematic review of the literature. A conceptual framework was established from the review by constructing an affinity diagram, and an item pool was generated. Face validation was conducted to assess whether the items measured MeHlit properly. Content validation was conducted to assess the overall relationship between domains by calculating the content validity index. Construct validation processes, including exploratory and confirmatory factor analyses, were completed with 1000 adults. Internal consistency of the Media Health Literacy Scale (MHLS) was assessed with Cronbach α. Concurrent validation was conducted to validate the MHLS’s performance by comparing it with an established tool, the Korean version of the eHealth Literacy scale (K-eHEALS).

    Results: A total of 13 published studies from the systematic review was used to develop the conceptual framework and an item pool of 65 items was created, including 3 domains (access, critical evaluation, and communication) and 9 subdomains. Through face and content validation processes, the MHLS was refined to comprise 3 domains, 6 subdomains, and 29 items. A total of 1000 participants were recruited for exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). Five subdomains were identified through EFA, and CFA demonstrated a good model fit (chi-square [Cmin χ2/df] under 2.659, root mean square error of approximation=0.058 [90% CI 0.053-0.062], comparative fit index=0.927, and standard root mean residual under 0.067). Following the EFA and CFA, Cronbach α scores of 0.915 and 0.932, respectively, were obtained, indicating that the tool had good reliability. A positive correlation was found between the MHLS and K-eHEALS from the concurrent validity evaluation, indicating that the MHLS can assess the target concept similarly as the K-eHEALS (Pearson correlation coefficient=0.736, P<.001).

    Conclusions: The MHLS was developed and validated in a step-by-step process to assess individuals’ ability to access, critically evaluate, and communicate health-related information through media platforms. This validated tool can serve in identifying deficiencies in specific MHLS areas and subsequently providing targeted education.

    J Med Internet Res 2025;27:e62884

    doi:10.2196/62884

    Keywords

    media; internet; media health literacy; ehealth literacy; survey development; validation; health-related information; communication 


    Advancements in IT have significantly affected how people convey information through various types of media. One of the key changes in transforming communication through the use of new media, including social network services and video-sharing platforms, is a shift from the one-way transmission of information by mass media institutions before the internet to a multidirectional exchange of information [1]. This change in the modes of communication has affected people’s lives as they use health care resources. The multidirectional exchange of health information has empowered patients enabling them to choose services that align with their needs [2]. Although the ease of information delivery has had positive effects, incorrect or unverified information also started to circulate and has led to numerous cases of harm caused by inaccurate health information [3].

    The increase of interest in health information on the internet led to the emergence of the concepts of media health literacy (MeHlit) and eHealth literacy. MeHlit was defined as “the ability to identify health-related content across various types of media, recognize its influence on health behavior, critically analyze the content, and express an intention to respond through action” [4]

    and eHealth literacy as “the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem” [5].

    Several tools have been developed to evaluate media health and eHealth literacy, including a tool to assess MeHlit, the eHealth Literacy Scale (eHEALS), and the eHealth Literacy Questionnaire [5-13]. The translated versions of the eHEALS (K-eHEALS) and the Digital Health Literacy Instrument (Covid-DHL-K) are used in South Korea [14,15]. Previously developed tools include assessment domains that access, understand, and critically appraise information from the internet, or have an intention to change their behaviors; however, they do not provide a communication domain that evaluates the dissemination of health information through the internet. Studies that evaluated internet users’ experiences and behavior through the internet are important in this era of frequent exchange of health information and user opinions; however, not many tools for evaluating the behavior are available, and validated tools are even more scarce.

    In the report of the “Survey on the Internet usage” in South Korea, the country’s internet penetration rate was over 99% [16] compared with the global internet penetration rate of 66% [17]. The rate of owning smart devices such as smartphones or tablet PCs is 97.3% [16], and the proportion of people who search for health-related information such as diseases, hospitals, and nutrition through the internet is 60.5% [16]. It can be inferred that the scale of internet accessibility and use in South Korea is significantly greater. Although South Korea is a country with a wide availability of internet access [16] and high literacy rates [18], the rates of health literacy are known to be low [19]. Similar to the global context, problems caused by inaccurate information have also consistently occurred in South Korea [20,21].

    Therefore, the objectives of our study are to develop a MeHlit tool, the Media Health Literacy Scale (MHLS), to assess the experiences and behaviors of consumers in accessing, critically evaluating, and communicating health-related information on the internet and to validate the developed tool with responsible processes that can be used in South Korea.


    Overview

    The study process included the development and validation of a tool to evaluate the MeHlit of Korean adults. The concept of MeHlit was operationally defined as “one’s ability to access, critically evaluate and communicate health-related information by new media,” based on studies related to MeHlit and eHealth literacy [4,22,23]. New media was defined as any channels that people could access and communicate through the internet (eg, social network services, video sharing platforms, etc) using PCs, tablet PCs, smartphones, or any device that can access the internet. Health-related information was defined as any information related to health, such as diseases, medicine, treatment, exercises, nutrients, and other types of information related to health [24]. The tool development was carried out through the processes of item development and validation.

    Conceptual Framework and Item Generation

    A systematic literature review was conducted to generate foundational data for the conceptual framework and item pool. Searches using the Medline, Embase, Cochrane Library, KoreaMed, and Research Information Sharing Service databases were conducted to find published studies on tools to evaluate MeHlit and eHealth literacy. Additional search was conducted to identify tools to evaluate media literacy using the same databases, adding Elton B. Stephens Company (EBSCO) and the Web of Science. The keywords used for the literature searches are presented in Multimedia Appendix 1. Studies focusing on the MeHlit, eHealth literacy, and media literacy were eligible for inclusion. Studies that developed an assessment tool to evaluate the literacies and provided a model describing the literacies were also eligible for inclusion. Studies were excluded if they (1) focused on the translation of already developed tools; (2) were not written in either English or Korean; (3) or were not available in full text. The literature search and selection process were performed independently by 2 researchers (SS and SK). Any discordance among the reviewers during the process of literature selection was resolved through mutual agreement or by involving a third researcher (EL) in a discussion.

    The tools identified from the systematic review were used to create an appropriate model consistent with the operational definition of MeHlit. An affinity diagram, usually used when various relevant concepts or issues are complex to grasp [25], specified and categorized domains and subdomains within the model; the final conceptual model was constructed by organizing these domains and subdomains [25].

    An item pool was created by collecting items identified from the systematic review by listing the tools and items from those tools. Each item was then assigned to the appropriate domain and subdomain within the final conceptual model, with similar items being merged and newly developed items being added. Items presented in English were translated into Korean by 3 internal researchers. The translated items were then modified by internal researchers and a Korean linguist who corrected grammatical errors and ensured cultural adaptation. Each item evaluating MeHlit used a 5-point Likert scale ranging from 1=“strongly disagree” to 5=“strongly agree.”

    Face Validation

    Using the initial draft of the survey, face validation was completed with the involvement of 6 internal researchers who have direct experience in interacting with or counseling patients about health-related information. During this process, the appropriateness of each item assignment to a domain or subdomain was reviewed by the researchers based on the objectives of the MHLS. The descriptions of the items and the subdomains were revised to enhance clarity, and redundant items were merged or rearranged to other subdomains after modification, if necessary.

    Content Validation

    Content validation was conducted to assess whether items were not only appropriately assigned to the domains and subdomains of the conceptual framework but also relevant to the overall theme of the MHLS, with scores graded by experts [26]. A total of 15 experts (4 medical doctors, 5 nurses, and 6 pharmacists), who met the inclusion criteria of having direct experience in interacting or counseling patients in their practice settings, were recruited to conduct content validation.

    The expert panel provided feedback on the tool in general, domains, subdomains, and items by rating their level of agreement on a scale of 1 to 4, where 1 indicated that items were not related to domains and subdomains, 2 that they were slightly related, 3 that they were related, and 4 that they were highly related. The scoring was then converted, with scores of 1 and 2 coded as 0, implying disagreement, and scores of 3 and 4 coded as 1, implying agreement that items were allocated properly. The scores for each item were summed and divided by the number of participating experts (n=15) to calculate the content validity index (CVI), based on which items with a CVI score of 0.8 or higher were retained; those with a score between 0.7 and 0.8 required further modification, enhancement, or deletion; and those with a score below 0.7 were removed from the item pool [26].

    Pilot Test

    A pilot test was conducted with general consumers by snowball sampling. A total of 30 participants representing general consumers who were not health care professionals completed the revised MHLS draft generated from the content validation and provided comments for further revision.

    Construct Validation

    A survey was conducted for construct validation using the MHLS semifinalized draft prepared based on the content validation and the pilot testing. The process included descriptive analysis, item analysis, factor analysis, and reliability tests. Sample size calculation was based on the literature indicating the appropriate number of participants for conducting factor analysis. The number of participants over 500 was considered very good [27,28]; the ratio of the number of participants and the number of survey items would be sufficient as 10:1 [29]; the number of participants should be equal or exceed 20 times more than the number of factors [27]. Based on the calculation, the study determined that 500 participants in each exploratory and confirmatory factor analysis (CFA) were necessary [30]. A total of 1000 participants aged 19 and above were recruited for each exploratory factor analysis (EFA) and confirmatory factor analysis. The survey participants were eligible if they met the inclusion criteria of having searched for health-related information on the internet or media and having delivered health-related information through media platforms. Health care professionals were excluded because they were considered familiar with health-related information due to their training in looking for and evaluating such information. A stratified sampling scheme was used to reflect the sex and age corresponding to the population distribution in South Korea [31]. The survey was conducted between December 14 and November 22, 2021 [31].

    Item analysis was conducted to evaluate the quality of each item. The mean, skewness, and kurtoses of each item and the item-total correlation were calculated. Items with skewness and kurtosis values outside the range of ± 1 were excluded [32]. Regarding item-total correlation, an item with a correlation value of less than 0.3 indicated a low correlation and was excluded [33].

    The Kaiser-Meyer-Olkin (KMO) measure and Bartlett sphericity test were used to assess the suitability of the respondent data for factor analysis [28,34]. Following the EFA results, eigenvalues, variance percentages, and factor loadings of items were identified. Items with factor loadings below 0.40 were removed. All items were then reviewed to determine whether they were grouped with similar items in a factor and those that were considered to have different characteristics from other items in the factor were eliminated after discussion with the researchers [28]. The process of EFA was conducted with SPSS (version 26; IBM Corp) [35].

    The model provided by the EFA was validated through CFA. Model fit was assessed based on chi-square tests, root mean square error of approximation (RMSEA), comparative fit index (CFI), and the standardized root mean residual (SRMR). The model was considered to have a good fit when each index satisfied the following scores: RMSEA under 0.5, CFI over 0.95, and SRMR under 0.08 [34]. The internal consistency reliability was tested by calculating the Cronbach α value. A Cronbach α value of 0.7 or higher indicated acceptable internal consistency and this tool was determined to be reliable [36]. The process of CFA was conducted with SPSS Amos (version 26; IBM Corp) [37].

    Concurrent Validation With the K-eHEALS

    The MHLS tool was validated using the Korean version of the eHealth Literacy scale (K-eHEALS) to conduct concurrent validation, which aims to confirm the construct validity of the MHLS by comparing it with previously developed tools. This process involved examining the correlation between the K-eHEALS, which was a translated version of eHEALS) and had similar evaluation criteria, and the MHLS tool developed in this study, using Pearson correlation coefficients [38]. Since the K-eHEALS did not include items corresponding to the Communication domain of the MHLS, the comparison was conducted using the K-eHEALS and MHLS scores, excluding the communication domain. The process of calculating a correlation score was conducted with SPSS (version 26; IBM Corp) [35].

    Ethical Considerations

    The study protocol adhered to an appropriate ethical review and was approved by the Institutional Review Board of Seoul National University (2108/002-01). The study conformed to the principles embodied in the Declaration of Helsinki. The participants were recruited through the Tillion Internet research panel managed by a survey company, Panel Marketing Interactive (PMI Co, Ltd). A web-based survey was accessed by the participants through the website of the company. Before participating in the study, the participants were informed of the purpose, procedures, data use, and their rights. Then the participants signed a consent form stating that their participation was voluntary and they could withdraw at any point. As data protection measures, the collected data from the web-based survey were anonymized and no personal identifiers were collected. Participants who completed the survey received a reward in the form of company points redeemable on the web-based platform.


    Draft of the MHLS

    A systematic review searching media literacy, MeHlit, and eHealth literacy found literature that suggested models of the literacies. The results showed 13 models of related literacies [9-12,23,39-45]. Through the affinity diagram, domains of models were grouped with domains that present similar meanings. From the results of the affinity diagram, the European Association for Viewers Interests (EAVI) model, which included the domains of use, critical understanding, and communicative abilities, was selected as the main framework because of its comprehensive characteristics and similarity with operational definition. All domains from collected models were regrouped under these 3 domains of the EAVI model (Multimedia Appendix 2). The conceptual framework to develop survey items was modified and constructed based on the result of the affinity diagram, resulting in 3 domains (access, critical evaluation, and communication) and 9 subdomains (Table 1). The item pool was constructed using the collected questionnaires (65 items).

    Table 1. Change in domains, subdomains, and number of items.

    		MHLSa draftAfter face validationAfter content validationAfter construct validation
    DomainsSubdomainsItemsSubdomainsItemsSubdomainsItemsSubdomainsItems
    Access317312311210
    Critical evaluation32911111019
    Communication3192102828
    Total965633629527

    aMHLS: Media Health Literacy Scale.

    Face and content validation were conducted qualitatively. During face validation, the domains, subdomains, and all items in the conceptual framework were reviewed. As a result, the number of subdomains decreased from 9 to 6 and the number of items decreased from 65 to 33 (Table 1). Content validation resulted in a reduction of 4 items, considering the relationship between items and domains. One item with a CVI score higher than 0.8 was deleted because of duplication, and another with a CVI score between 0.8 and 0.7 was deleted because of a poor relationship with MeHlit. A total of 6 items with a CVI score under 0.7 should have been deleted, but 2 items, each within the “evaluation of quality on health-related information” subdomain of the critical evaluation domain and the “experience of communicating health-related information” subdomain of the communication domain were modified and retained because of their importance as judged by internal researchers. The scale-level CVI score based on the average method was 0.83 and the scale-level CVI score based on universal agreement method was 0.88.

    In the pilot test, 30 general consumers were recruited through snowball sampling. Their comments from the process were used to modify the MHLS draft. These comments were related to the operational definition of health-related information and the mode of accessing the internet, such as through PC or mobile. Therefore, a descriptive statement on the meaning of health-related information in our study was added to the MHLS in a box format. We also elaborated on the description that various modes of accessing the Internet were acceptable (Multimedia Appendix 3).

    Construct Validation of the MHLS

    Following content validation of the MHLS draft, item analysis, factor analysis, and reliability test were conducted to achieve construct validity (Multimedia Appendix 4). A total of 1000 participants answered the MHLS items after content validation and were divided into 2 groups: 498 in the EFA and 502 in the CFA. The mean (SD) age of the participants was 42.49 (SD 12.37) years in the EFA and 42.58 (SD 12.31) years in the CFA. Male participants comprised 271 of 498 participants in the EFA and 273 of 502 in the CFA (54.4%) and college graduates or higher comprised 416 of 498 participants in the EFA and 421 of 502 in the CFA (83.7%). A total of 381 of 498 participants in the EFA and 402 of 502 in the CFA were employed (78.2%). The characteristics of participants are summarized in Table 2.

    Item analysis revealed a mean of 3.20 to 4.32. The skewness and kurtosis of whole items were in the range of ± 1, and the item-total correlation of whole items was higher than 0.3 (Multimedia Appendix 5).

    The KMO index was 0.938, and Bartlett sphericity showed a statistically significant result (P<.001), with both indices indicating the EFA was appropriate. The EFA identified 5 factors that explained 52.726% of cumulative variance. Direct oblimin rotation was used and 2 items were deleted considering the factor loading on each factor. One was excluded because the factor loading was under 0.4; the other one about the experience with accessing health-related information via media was excluded because the factor loading was over 0.4 with the utilization of digital devices. Factors from EFA were named considering items that described each factor: “digital device operational skills,” “experience and ability of accessing health-related information,” “evaluation of quality of health-related information,” “experience of communicating health-related information,” and “responsible communication of health-related information.” The EFA identified 5 subdomains and 27 items (Table 3).

    Table 2. Characteristics of participants in construct validation.
    CharacteristicsExploratory factor analysis (n=498), n (%)Confirmatory factor analysis (n=502), n (%)
    Age (years)

    		19-29103 (20.7)104 (20.7)

    		30-39117 (23.5)117 (23.3)

    		40-49125 (25.1)126 (25.1)

    		50-59104 (20.9)104 (20.7)

    		6049 (9.8)51 (10.2)
    Sex

    		Male271 (54.4)273 (54.4)

    		Female227 (45.6)229 (45.6)
    Employment status

    		Yes381 (76.5)402 (80.1)

    		No117 (23.5)100 (19.9)
    Education level

    		Middle school graduate1 (0.2)1 (0.2)

    		High school graduate81 (16.3)80 (15.9)

    		College graduate or higher416 (83.5)421 (83.9)
    Average monthly income

    		5th quintile53 (10.6)45 (9.0)

    		4th quintile75 (15.1)77 (15.3)

    		3rd quintile128 (25.7)134 (26.7)

    		2nd quintile141 (28.3)144 (28.7)

    		1st quintile101 (20.3)102 (20.3)
    Table 3. Exploratory factor analysis of the Media Health Literacy Scale.
    Subdomains and itemsFactor loadings

    		12345
    Digital devicea operational skills

    		I can access the internet via a digital device.

    		–.819

    I can install programs or applications on digital devices.

    		–.732

    I can operate a digital device to navigate a search engine.

    		–.754

    I can operate a digital device to communicate or post information.

    		–.697

    Experience and ability to access health-related informationb

    		When I come across health-related information, I browse the internet for additional related information.


    		.519
    When I want to get health-related information, I generally browse the websites of health-related public institutions or hospitals.


    		.498
    I have experience finding desired health-related information on the internet.


    		.446
    I know where to search for health-related information on the Internet.


    		.627
    I know which search terms to use to find health-related information on the internet.


    		.592
    I can choose the desired information from the abundant pool of health-related information available on the internet.


    		.500
    Evaluation of quality on health-related information

    		I consider whether the health-related information I come across on the internet is accurate..454



    I check whether the health-related information I come across on the internet is up to date..586



    I check the sources of the health-related information I come across on the internet..683



    I check whether the health-related information I come across on the internet is provided by health care professionalsc..613



    I check whether the health-related information I come across on the internet is corroborated by scientific evidenced..603



    I check for intentions or purposes (eg, political leaning or advertising) behind the health-related information I come across on the internet..561



    I check whether the health-related information I come across on the internet highlights only the positive effects or conceals the risks..606



    I check the accuracy of the health-related information I come across on the internet by cross-referencing multiple websites..520



    I check the accuracy of the health-related information I come across on the internet by consulting a healthcare professional..609



    Experience in communicating health-related information

    		I press the “recommend”, “do not recommend”, “like”, or “dislike” buttons on health-related information posts I come across on the internet to express my opinione.
    		.723


    I write comments on health-related information posts I come across on the internet to express my opinion.
    		.817


    I forward health-related information posts I come across on the internet to others via messaging apps.
    		.676


    I post texts, images, or videos with health-related information on the internet (eg, blog, online forum, YouTube, social media).
    		.710


    Responsible communication of health-related information

    		I check whether the health-related information on the internet is accurate when I post, forward, or express opinions through the internet.



    		.585
    I believe that the health-related information that I post, forward, or comment on over the internet (including likes, recommendations, and comments) may affect the health or well-being of others or society.



    		.561
    I check whether the health-related information I post, forward, or comment on over the internet (including likes, recommendations, and comments) contains any material that violates the law.



    		.520
    I disclose the source of the online health-related information when I post or forward it.



    		.499
    Eigenvalue8.764.261.471.200.86
    Proportion of variance (total 52.726%)30.4714.383.672.731.48

    aDigital device: smartphone, computer, and tablet PCs.

    bHealth-related information: news, advertisements, articles, blogs, content on YouTube, and other social media.

    cHealth care professionals: physicians, doctors of Korean medicine, dentists, pharmacists, nurses, nursing assistants, and so on.

    dScientific evidence: evidence provided by health care professionals, professional organizations, journal articles, government documents, and so on.

    eOpinions: likes, recommendations, and comments.

    CFA was conducted with a 5 factors model derived from the EFA. In total, 502 sex- and age-stratified samples from 1000 participants were included in the CFA. The CFA results were presented as: Cmin χ2 (df) under=2.659, RMSEA=0.058 (90% CI 0.053-0.062), CFI=0.927, and SRMR under 0.067. All indicators represented a good fit for the model. The structural equation model is presented in Figure 1.

    Internal consistency was confirmed with Cronbach α. The Cronbach α score of the dataset used for EFA was 0.915, and the domain scores were as follows: “digital device operational skills”=0.888, “experience and ability of accessing health-related information”=0.826, “evaluation of quality on health-related information”=0.870, “experience of communicating health-related information”=0.864, and “responsible communication of health-related information”=0.748. The Cronbach α score of the dataset used for CFA was 0.927, and the domain scores were as follows: “digital device operational skills”=0.911, “experience and ability of accessing health-related information”=0.824, “evaluation of quality on health-related information”=0.888, “experience of communicating health-related information”=0.849, and “responsible communication of health-related information”=0.804. Scores indicated acceptable internal consistency.

    ‎
    Figure 1. Structural equation model for confirmatory factor analysis. HRI: health-related information.

    Concurrent Validation of the MHLS

    The correlation between the K-eHEALS and MHLS scores, excluding the communication domain, had a positive relationship. This indicates that the MHLS can measure the target concept similar to the K-eHEALS. The Pearson correlation coefficient was 0.736, showing statistical significance (P<.001).


    Principal Findings

    Increased health concerns and interests in health-related information [46] have resulted in a proliferation of related information in the media, which has affected public behavior [47,48]. Although various tools have been developed to evaluate how people interact with health-related information in the media [5-13,45], there has been no tool to comprehensively assess the concepts following a rigorous validation process. The MHLS was developed to address this gap by offering quantitative scores measuring MeHlit across the domains of access, critical evaluation, and communication. We believe the MHLS tool can identify consumers with relatively low scores in specific domains and the potential to cause harm by disseminating misinformation, disinformation, or malinformation.

    The development process was based on a framework that proposed these 3 abovementioned domains. The MHLS, structured by access, critical evaluation, and communication, can measure the proficiency of individuals who interacted with health-related information via media. In the study by Nazarnia et al [8], the MeHLit tool was heavily focused on the capability of users to appraise health-related information with the domains of goal appraisal, content appraisal, implicit meaning appraisal, visual comprehension, and audience appraisal skill. In addition, the eHEALS, consisting of 6 domains covering traditional, health, information, scientific, media, and computer literacy [5], has been used in several studies to assess eHealth literacy [49-51]. Surveys like the Digital Health Literacy Instrument, eHealth Literacy Questionnaire, and eHEALS surveys address users’ capability without collecting their experience of using the internet. Levin-Zamir et al [6] and Fleary [7] also developed tools to evaluate MeHlit with the domains of identification or recognition, influence or critical analysis, and action or reaction. While these tools evaluated the user’s intention to act or react, the intention-behavior gap has been documented in the literature [52], tools evaluating not only consumers’ intentions but also their behaviors are needed.

    The main difference between the preexisting tools and the MHLS tool in our study is the inclusion of the communication domain, which elucidates how individuals interact with health-related information. We believe the MHLS tool distinguishes itself from already developed tools by revealing consumers’ behavior on communicating health-related information they have searched for and their opinions, allowing assessment of both the transmission of health-related information and individuals’ responsibility in health-related information dissemination. In addition, where harm by the dissemination of misinformation is prominently evident [3,53], the tool can grasp the characteristics of people who have shared misinformation without realizing it and identify weaknesses to improve prevention of harm caused by misinformation.

    The MHLS demonstrated structural strengths through well-executed development and validation processes. During the development stage, a systematic review of relevant literature was conducted to establish the conceptual framework and formulate items. Since there are various ways to develop items and a conceptual framework, [54,55] appropriate methods were chosen based on the conditions of the research environment. The drafted items underwent a series of validation processes to ensure qualitative relationships with the items recommended in the literature [56]. Construct validation, including factor analysis (both exploratory and confirmatory), and reliability and concurrent validity tests were conducted to quantitatively validate the tool [34,36,38]. Compared with the already developed tools, the EFA scores were demonstrated with KMO, which identifies whether an appropriate number of participants were recruited. EAVI scores were 0.859, 0.896, and 0.93 and the MHLS scored 0.938, meaning the number of participants included was very good [8-10]. CFA scores were presented as CFI and RMSEA, which explains the fitness of the final model. Previous studies reported CFI values of 0.93 and 0.94 and RMSEA values of 0.049, 0.051, and 0.07, which represent a good fit of the model [8,9,13], while the MHLS had CFI and RMSEA scores of 0.927 and 0.058. The reliability of the already developed tools according the Cronbach α scores ranged from 0.74 to 0.93 [5,6,8-10] and the Cronbach α of the MHLS was 0.915 in the EFA and 0.927 in the CFA, indicating good reliability. In comparison with already developed tools, all validation steps of the MHLS scored similarly to the past tools, satisfying the criteria for being good. This quantitative comparison between already developed tools shows that meticulous development and validation procedures support the validity of the MHLS [57].

    The EFA results recommended that the model include factors with eigenvalues greater than 1.0, but our model included a factor with an eigenvalue less than 1.0 because the model demonstrated a stronger relationship between items and factors based on factor loadings and item allocation [58]. Subsequently, the CFA results indicated that the model had a good fit, surpassing the acceptable criteria. Structural equation modeling provided additional support for the model’s strong fit. The model effectively explained MeHlit, beginning with the construction of a framework and ensuring several types of validity.

    To further confirm the validity of the MHLS, our study conducted concurrent validation with the K-eHEALS [38]. The eHEALS was originally developed to assess eHealth literacy among individuals using IT, while the K-eHEALS is the translated version specifically designed for Korean adults [5,14]. As eHealth literacy shares several similarities with MeHlit, analyzing the correlation between the K-eHEALS and the MHLS can validate the effectiveness of the MHLS based on the correlation coefficient scores. The results indicated a positive correlation between the tools, suggesting that the MHLS effectively measures the level of MeHlit among survey participants.

    Limitations

    There are several limitations in our study. First, generalization of the MHLS can be challenging because the characteristics of participants did not represent those of the general public. The proportion of college graduates or higher in our study was about 84%, which is significantly higher than the national statistics of 53% [59]. Moreover, because the panel used in the study recruited participants by email or over the internet, those who were not familiar with the recruiting method were not included. Therefore, only 10% of participants were aged 60 years or more, compared with approximately 14.7% in the Korean population [60]. Participants older than 70 years and more were not represented in this study. Since the reported rate of searching the internet among people 70 years and older was more than 50% in South Korea and there was a high representation of people with college-level or greater education, the exclusion of these populations indicates that the study population may not ideally represent the general public in South Korea. Therefore, further studies to ensure the validity of the MHLS are needed, especially targeting older adults and consumers with low educational levels.

    Second, the MHLS tool was a home-grown tool for consumers in South Korea. Therefore, the application of the MHLS to other cultural environments should consider cultural and linguistic differences. We believe cultural and societal characteristics, including the rate of internet penetration, health information–seeking behaviors, education levels, literacy or health literacy rates, health policy, health care access, and service products could vary across cultures; they should be considered when translating the MHLS for other cultures and should go through additional validation processes.

    Conclusions

    The MHLS was developed and validated in a step-by-step process to assess individuals’ ability to access, critically evaluate, and communicate health-related information through media platforms. The findings from our study suggest that the validated MHLS tool performed well at measuring MeHlit in South Korea.

    Acknowledgments

    This work was supported by the National Research Foundation of Korea (NRF) grant, funded by the Korean government (Ministry of Science and ICT; grant numbers 2020R1A2C1101560 and 2022R1F1A1064147), and the 4 phases of the Brain Korea 21 program in 2024.

    Data Availability

    The datasets generated or analyzed during this study are available from the corresponding author on reasonable request.

    Authors' Contributions

    All authors contributed to the conceptualization and design of the study. SS and SK contributed to the data curation, the formal analysis, and the methodology. EL contributed to the project administration and the supervision of the study. SS, SK and EL contributed to the validation of the study. SS and EL contributed to the visualization of the study. SS wrote the original draft.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Search keywords.

    DOCX File , 13 KB
    Multimedia Appendix 2

    Affinity diagram.

    PNG File , 250 KB
    Multimedia Appendix 3

    Media Health Literacy Scale.

    DOCX File , 63 KB
    Multimedia Appendix 4

    Checklist for Reporting Results of Internet E-Surveys (CHERRIES) checklist.

    PDF File (Adobe PDF File), 552 KB
    Multimedia Appendix 5

    Item analysis results.

    DOCX File , 15 KB
    1. Leonardi PM, Vaast E. Social media and their affordances for organizing: a review and agenda for research. Acad Manag Ann. 2017;11(1):150-188. [FREE Full text]
    2. Finkelstein J, Knight A, Marinopoulos S, Gibbons MC, Berger Z, Aboumatar H, et al. Enabling patient-centered care through health information technology. Evid Rep Technol Assess (Full Rep). 2012;(206):1-1531. [Medline]
    3. Bin Naeem S, Kamel Boulos MN. COVID-19 Misinformation online and health literacy: A brief overview. Int J Environ Res Public Health. 2021;18(15):8091. [FREE Full text] [CrossRef] [Medline]
    4. Levin-Zamir D, Bertschi I. Media health literacy, eHealth literacy, and the role of the social environment in context. Int J Environ Res Public Health. 2018;15(8):1643. [FREE Full text] [CrossRef] [Medline]
    5. Norman CD, Skinner HA. eHEALS: The eHealth literacy scale. J Med Internet Res. 2006;8(4):e27. [FREE Full text] [CrossRef] [Medline]
    6. Levin-Zamir D, Lemish D, Gofin R. Media Health Literacy (MHL): Development and measurement of the concept among adolescents. Health Educ Res. 2011;26(2):323-335. [CrossRef] [Medline]
    7. Fleary SA. Development and validation of the adolescent media health literacy scales: Rasch measurement model approach. JMIR Pediatr Parent. 2022;5(2):e35067. [FREE Full text] [CrossRef] [Medline]
    8. Nazarnia M, Zarei F, Rozbahani N. Development and psychometric properties of a tool to assess Media Health Literacy (MeHLit). BMC Public Health. 2022;22(1):1839. [FREE Full text] [CrossRef] [Medline]
    9. Seçkin G, Yeatts D, Hughes S, Hudson C, Bell V. Being an informed consumer of health information and assessment of electronic health literacy in a national sample of internet users: Validity and reliability of the e-HLS instrument. J Med Internet Res. 2016;18(7):e161. [FREE Full text] [CrossRef] [Medline]
    10. van der Vaart R, Drossaert C. Development of the digital health literacy instrument: Measuring a broad spectrum of health 1.0 and health 2.0 skills. J Med Internet Res. 2017;19(1):e27. [FREE Full text] [CrossRef] [Medline]
    11. Kayser L, Karnoe A, Furstrand D, Batterham R, Christensen KB, Elsworth G, et al. A multidimensional tool based on the eHealth literacy framework: Development and initial validity testing of the eHealth literacy questionnaire (eHLQ). J Med Internet Res. 2018;20(2):e36. [FREE Full text] [CrossRef] [Medline]
    12. Paige SR, Stellefson M, Krieger JL, Miller MD, Cheong J, Anderson-Lewis C. Transactional eHealth literacy: Developing and testing a multi-dimensional instrument. J Health Commun. 2019;24(10):737-748. [FREE Full text] [CrossRef] [Medline]
    13. Kayser L, Rossen S, Karnoe A, Elsworth G, Vibe-Petersen J, Christensen JF, et al. Development of the multidimensional readiness and enablement index for health technology (READHY) tool to measure individuals' health technology readiness: Initial testing in a cancer rehabilitation setting. J Med Internet Res. 2019;21(2):e10377. [FREE Full text] [CrossRef] [Medline]
    14. Chang SJ, Yang E, Ryu H, Kim HJ, Yoon JY. Cross-cultural adaptation and validation of the eHealth literacy scale in Korea. Korean J Adult Nurs. 2018;30(5):504-515. [CrossRef]
    15. Chun H, Park E-J, Choi SK, Yoon H, Okan O, Dadaczynski K. Validating the digital health literacy instrument in relation to COVID-19 information (COVID-DHL-K) among South Korean undergraduates. Int J Environ Res Public Health. 2022;19(6):3437. [FREE Full text] [CrossRef] [Medline]
    16. Choi M, Kim T, Myung S. 2023 Survey on the Internet Usage. National Information Society Agency. 2023. URL: https://www.nia.or.kr/site/nia_kor/ex/bbs/View.do?cbIdx=99870&bcIdx=26715&parentSeq=26715 [accessed 2025-04-14]
    17. Kemp S. Internet use in 2024. Datareportal. URL: https://datareportal.com/reports/digital-2024-deep-dive-the-state-of-internet-adoption [accessed 2025-01-28]
    18. Statistics Korea. IMD competitiveness in education. URL: https://www.index.go.kr/unity/potal/main/EachDtlPageDetail.do?idx_cd=1526 [accessed 2024-11-09]
    19. Bae J, Kim H. Health Literacy in Korea: Findings from a Nationally Representative Survey. Research@KIHASA. 2023. URL: https://repository.kihasa.re.kr/en/handle/201002/42084 [accessed 2025-04-14]
    20. Lee JJ, Kang K, Wang MP, Zhao SZ, Wong JYH, O'Connor S, et al. Associations between COVID-19 misinformation exposure and belief with COVID-19 knowledge and preventive behaviors: Cross-sectional online study. J Med Internet Res. 2020;22(11):e22205. [FREE Full text] [CrossRef] [Medline]
    21. Suarez-Lledo V, Alvarez-Galvez J. Prevalence of health misinformation on social media: Systematic review. J Med Internet Res. 2021;23(1):e17187. [FREE Full text] [CrossRef] [Medline]
    22. Jabareen Y. Building a conceptual framework: Philosophy, definitions, and procedure. Int J Qual Methods. 2009;8(4):49-62. [CrossRef]
    23. Renee H. Digital and Media Literacy: A Plan of Action. New York, NY. Aspen Institute; 2010.
    24. Health information. Korea Disease Control and Prevention Agency. URL: https:/​/health.​kdca.go.kr/​healthinfo/​biz/​health/​gnrlzHealthInfo/​gnrlzHealthInfo/​gnrlzHealthInfoMain.​do;jsessionid=a8_Gx7dKbzXVVScGBtcjOTnG4lVdUlmtv2jUbeKE.​healthinfo20?lclasSn=0 [accessed 2020-11-09]
    25. Tague NR. The Quality Toolbox, Second Edition. Milwaukee, WI. ASQ Quality Press; 2005.
    26. Yusoff M. ABC of content validation and content validity index calculation. J Med Educ. 2019;11(2):49-54. [CrossRef]
    27. Tak J. Psychological Testing: An Understanding of Development and Evaluation Method. Seoul, Korea. Hakjisa; 2007.
    28. Williams B, Onsman A, Brown T. Exploratory factor analysis: a five-step guide for novices. Australas J Paramed. 2010;8(3):1-13.
    29. Nunnally J. Psychometric Theory. New York, NY. McGraw-Hill; 1978.
    30. Comrey AL, Lee HB. A First Course in Factor Analysis. London, UK. Psychology Press; 2013.
    31. Kim S, Yu YM, You M, Jeong KH, Lee E. A cross-sectional survey of knowledge, attitude, and willingness to engage in spontaneous reporting of adverse drug reactions by Korean consumers. BMC Public Health. 2020;20(1):1527. [FREE Full text] [CrossRef] [Medline]
    32. Huck SW, Cormier WH, Bounds WG. Reading Statistics and Research. London, UK. Pearson; 2004.
    33. Gable RK, Wolf MB. The reliability of affective instruments. In: Instrument Development in the Affective Domain: Measuring Attitudes and Values in Corporate and School Settings. Dordrecht, Netherlands. Springer; 1993:201-235.
    34. Thompson B. Exploratory and Confirmatory Factor Analysis: Understanding Concepts and Applications. First Edition. Washington, DC. American Psychological Association; 2004.
    35. IBM SPSS Statistics. URL: https://www.ibm.com/products/spss-statistics [accessed 2025-05-02]
    36. Taber KS. The use of Cronbach’s Alpha when developing and reporting research instruments in science education. Res Sci Educ. 2017;48(6):1273-1296. [CrossRef]
    37. Arbuckle JL. IBM® SPSS ® Amos™ 26 User’s Guide. 2019. URL: https://www.ibm.com/docs/en/SSLVMB_26.0.0/pdf/amos/IBM_SPSS_Amos_User_Guide.pdf [accessed 2025-05-02]
    38. DeVon HA, Block ME, Moyle-Wright P, Ernst DM, Hayden SJ, Lazzara DJ, et al. A psychometric toolbox for testing validity and reliability. J Nurs Scholarsh. 2007;39(2):155-164. [CrossRef] [Medline]
    39. Ofcom’s Strategy and Priorities for the Promotion of Media Literacy. Ofcom. 2004. URL: https://www.ofcom.org.uk/media-use-and-attitudes/media-literacy/strategymedialit [accessed 2025-05-02]
    40. Celot P. Study on Assessment Criteria for Media Literacy Levels. European Commission. 2009. URL: https://ec.europa.eu/assets/eac/culture/library/studies/literacy-criteria-report_en.pdf [accessed 2025-05-02]
    41. Kayser L, Kushniruk A, Osborne RH, Norgaard O, Turner P. Enhancing the effectiveness of consumer-focused health information technology systems through eHealth literacy: A framework for understanding users' needs. JMIR Hum Factors. 2015;2(1):e9. [FREE Full text] [CrossRef] [Medline]
    42. Chun H, Yeon S, Lee C, Lee S. An Evaluation of a Media Literacy Program. Korea Communications Commission. 2016. URL: https://www.yesform.com/wdata/doc-420914.php# [accessed 2025-04-14]
    43. Kim K, Lee S, Kim G. Developing a media literacy index and measuring the difference in media literacy levels between regions in South Korea. Korea Communications Commission. 2017. URL: https://www.yesform.com/wdata/doc-461162.php [accessed 2025-04-14]
    44. Zakaria N, AlFakhry O, Matbuli A, Alzahrani A, Arab NSS, Madani A, et al. Development of Saudi e-health literacy scale for chronic diseases in Saudi Arabia: Using integrated health literacy dimensions. Int J Qual Health Care. 2018;30(4):321-328. [FREE Full text] [CrossRef] [Medline]
    45. Karnoe A, Furstrand D, Christensen KB, Norgaard O, Kayser L. Assessing competencies needed to engage with digital health services: Development of the eHealth literacy assessment toolkit. J Med Internet Res. 2018;20(5):e178. [FREE Full text] [CrossRef] [Medline]
    46. Bujnowska-Fedak MM, Węgierek P. The impact of online health information on patient health behaviours and making decisions concerning health. Int J Environ Res Public Health. 2020;17(3):880. [FREE Full text] [CrossRef] [Medline]
    47. Centola D. Social media and the science of health behavior. Circulation. 2013;127(21):2135-2144. [CrossRef] [Medline]
    48. Wu Y, Shen F. Exploring the impacts of media use and media trust on health behaviors during the COVID-19 pandemic in China. J Health Psychol. 2022;27(6):1445-1461. [CrossRef] [Medline]
    49. Paige SR, Krieger JL, Stellefson M, Alber JM. eHealth literacy in chronic disease patients: An item response theory analysis of the eHealth literacy scale (eHEALS). Patient Educ Couns. 2017;100(2):320-326. [FREE Full text] [CrossRef] [Medline]
    50. Sudbury-Riley L, FitzPatrick M, Schulz PJ. Exploring the measurement properties of the eHealth Literacy Scale (eHEALS) among baby boomers: A multinational test of measurement invariance. J Med Internet Res. 2017;19(2):e53. [FREE Full text] [CrossRef] [Medline]
    51. Kim K, Shin S, Kim S, Lee E. The relation between eHealth literacy and health-related behaviors: Systematic review and meta-analysis. J Med Internet Res. 2023;25:e40778. [FREE Full text] [CrossRef] [Medline]
    52. Sheeran P, Webb T. The intention–behavior gap. Soc Personal Psychol Compass. 2016;10(9):503-518. [FREE Full text] [CrossRef]
    53. Lazard AJ, Nicolla S, Vereen RN, Pendleton S, Charlot M, Tan H, et al. Exposure and reactions to Cancer treatment misinformation and advice: Survey study. JMIR Cancer. 2023;9:e43749. [FREE Full text] [CrossRef] [Medline]
    54. Krause N. A comprehensive strategy for developing closed-ended survey items for use in studies of older adults. J Gerontol B Psychol Sci Soc Sci. 2002;57(5):S263-S274. [FREE Full text] [CrossRef] [Medline]
    55. Kenaphoom S. Establishment of a survey research conceptual framework on management. ASEAN J Manag Innov. 2015;4(1):63-77. [FREE Full text]
    56. Burton LJ, Mazerolle S. Survey instrument validity part I: principles of survey instrument development and validation in athletic training education research. Athl Train Educ J. 2011;6(1):27-35. [CrossRef]
    57. Wood TM, Zhu W. Measurement Theory and Practice in Kinesiology. Champaign, IL. Human Kinetics; 2006.
    58. Cliff N. The eigenvalues-greater-than-one rule and the reliability of components. Psychol Bull. 1988;103(2):276-279. [CrossRef]
    59. Education at a Glance 2023: OECD Indicators. Organisation for Economic Co-operation and Development. URL: https://www.oecd.org/en/publications/2023/09/education-at-a-glance-2023_581c9602.html [accessed 2025-05-02]
    60. Korean Statistical information Service: Projected Population by Age (Korea). Statistics Korea. URL: https://kosis.kr/visual/populationKorea/PopulationDashBoardDetail.do [accessed 2024-11-09]

    ‎
    CFA: confirmatory factor analysis
    CFI: comparative fit index
    CVI: content validity index
    EAVI: European Association for Viewers Interests
    EBSCO: Elton B. Stephens Company
    EFA: exploratory factor analysis
    eHEALS: eHealth Literacy Scale
    K-eHEALS: the Korean version of the eHealth Literacy scale
    KMO: Kaiser-Meyer-Olkin
    MeHlit: media health literacy
    MHLS: Media Health Literacy Scale
    RMSEA: root mean square error of approximation
    SRMR: standardized root mean residual

    Edited by X Ma; submitted 05.06.24; peer-reviewed by C Baur, F Zarei; comments to author 27.09.24; revised version received 18.11.24; accepted 29.03.25; published 05.05.25.

    Copyright

    ©Sangyoon Shin, Seungyeon Kim, Youngshin Song, Hyesun Jeong, Yun Mi Yu, Euni Lee. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 05.05.2025.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.