Are diabetes consumer education materials easy to understand and act on? An evaluation of multilingual Australian resources
Diabetes was ranked as the seventh leading cause of death in Australia in 2017-2018, affecting 1.2 million Australians (4.9%).1 The prevalence of diabetes is higher in migrant groups in Australia.2, 3 This disparity is likely to increase with increasing immigration from China where the prevalence of diabetes is currently 10.9%, with another 35.7% of the population at high risk of developing the condition.4
Diabetes complications cost the Australian healthcare system AUD$14.6 billion per year, as well as negatively impacting quality of life.5, 6 However, the burden of diabetes can be mitigated through appropriate self-management. Diabetes education is an important part of self-management interventions and often includes written educational materials.7, 8, 9 Hence, it is crucial that these materials are accessible and effective.
Key words: Health literacy, diabetes, multilingual, patient education materials, PEMAT
A growing body of research has explored strategies to improve the effectiveness of written health materials. A systematic review found that strategies to improve written communication (such as using simple language and relevant illustrations) were effective in improving knowledge and cognitive outcomes.10 However, there is some evidence that these strategies may not have been incorporated into key national diabetes resources. Our recent assessment of the readability of eight national diabetes consumer education materials, for example, found that on average these materials were written at a grade 10 reading level, well above the grade 8 reading level recommended by health literacy guidelines.11
Although this study provided a useful indication of the understandability of national diabetes materials, readability is a relatively narrow assessment of the health literacy demand of materials, and could only be calculated for English-language materials.11 The Health Literacy Universal Precautions Toolkit recommends using a range of methods to examine the quality of written materials, as many features cannot be captured by a single test.12 The Patient Education Materials Assessment Tool (PEMAT) provides a broader assessment of the health literacy demands of educational materials.13 This validated tool assesses two constructs: understandability (which includes assessments of readability as well as structure, layout and design); and actionability (assessment of strategies that help users identify what to do based on the information provided). A recent American study used the PEMAT to assess 13 diabetes education materials that were available online from national not-for-profit organisations and pharmaceutical companies. 14 This study found that the readability of materials was not associated with PEMAT scores, very few materials had adequate understandability, and only one material had adequate actionability. Together, these results highlight the importance of a multimodal approach to assessing the health literacy demands of diabetes education materials.
The PEMAT has the added advantage that it can be used to assess education materials that are written in languages that do not have readily-available readability formulas. The ability to concurrently assess English and translated materials is particularly important given Australian research has found that people from culturally and linguistically diverse communities prefer health information to be available in both English and their first language.15
This study aimed to examine the understandability and actionability of printed Australian diabetes materials available in English and Chinese, and to explore the factors that could improve the understandability and actionability of such materials.
This was a non-experimental study that involved assessing the understandability and actionability of diabetes consumer education materials using a validated health literacy instrument.13
Selection of consumer education materials
A search was conducted to obtain the publicly and freely accessible diabetes materials in a printable format (in PDF) and available in English and Simplified Chinese. To ensure content accuracy and professional credibility, only the latest versions of materials jointly published by the Australian government (National Diabetes Services Scheme) and the national peak diabetes organisation (Diabetes Australia) were selected. Information only in English, or in video, pictorial or graphic form was excluded, as were any materials not directly related to diabetes self-management. The search was conducted in February to March 2019.
Categorisation of the consumer education materials
A previous study examining the readability of similar materials from the same publishers categorised materials into four main content categories: general, medical information, nutritional, and physical activity.11 In this study, the nutritional information and physical activity information sections were combined into one category (‘lifestyle information’), so that three categories – general, medical and lifestyle information – were applied to the study
The Patient Education Materials Assessment Tool for printed materials (PEMAT-P) was developed and validated by Shoemaker and colleagues to assess understandability (17 items) and actionability (7 items).12, 13 Each item is scored on a binary scale (agree (1) or disagree (0)) excluding seven items which include a ‘not applicable’ option. Results are expressed as the percentage of items coded ‘agree,’ such that a higher percentage suggests the consumer education material is more likely to be accessible and easier for a person to act on the information presented. A score of at least 70% is considered indicative of reasonable understandability and actionability.13 The PEMAT-P was validated to be used by both healthcare professionals and laypersons with consistent and reliable results among different assessors, and has been previously used to assess Australian consumer education materials.13, 14, 16
After receiving training from the Patient Education Material Assessment Tool User’s Guide,17 an assessment panel comprising three health professional members and two consumer panel members (people with diabetes) scored materials using the PEMAT. All panel members spoke Chinese as their first language and had high English proficiency. See Appendix 1 and Appendix 2 for the detailed characteristics.
Two health professional panel members (SL, JZ) assessed the English texts, and two health professional panel members (SL, YL) assessed the Chinese texts. Disagreements between members were resolved through face-to-face meetings and/or teleconferences. When there was a discrepancy that could not be resolved between two members, the third diabetes educator from the panel (JZ or YL) was consulted.
Consumer members independently read and assessed the English texts. A consensus agreement was not required for the consumer panel members for logistical reasons. Mean and range of PEMAT scores for each material were calculated.
The overall score for each material was the average of the single (consensus) health professional score and the mean of the two consumer scores. These overall scores were averaged to calculate mean scores for each content category.
Our search identified eight materials that met inclusion criteria (Table 1).
Table 1: Mean understandability and actionability scores from the PEMAT-P assessment for individual consumer education materials by content categories and languages.
|Consumer education material title||English texts||Chinese texts|
|Understandability (%)||Actionability (%)||Understandability (%)||Actionability (%)|
|Understanding type 1 diabetes||71||47||69||47|
|Understanding type 2 diabetes||73||40||69||40|
| Understanding gestational diabetes |
|Making healthy food choices||78||100||78||100|
|Pregnancy and diabetes||77||89||75||89|
The mean understandability scores in all content categories were 70% or above in both English and Chinese (range 69-76%), except for the general information category for which the Chinese text was just under 70% (Table 1). The ‘Making healthy food choices’ had the highest mean understandability in both English (78%) and Chinese (78%), while the two materials under the general information category (‘Diabetes-related complications’ and ‘Understanding gestational diabetes’) had the lowest mean understandability scores (67% and 69% respectively) in both languages (Table 1). For each content category, mean understandability scores were similar for English and Chinese texts (with a difference of up to one percentage point).
Average scores for actionability were between 77% and 84% in the medical and lifestyle information category in both languages, but were lower for general information with the lowest mean score of 47% in both languages (Table 1). The materials related to food choices from the lifestyle information category achieved the maximum score for actionability (100%) from all the panel members in both languages (Table 1). Three of the four materials in the general information category scored the lowest mean actionability (40%, 47% and 47% respectively) of all eight materials (Table 1). Average actionability scores for the general and lifestyle information categories were exactly the same across English and Chinese texts (47% and 84%, respectively), but there was a 3%difference between English (80%) and Chinese (77%) scores in the medical information category (Table 1).
Across all materials and both languages, two assessment items for the understandability ‘provides a summary’ and ‘uses a visual aid whenever possible’ were adhered to less than 13% and 38% of the time, respectively (Table 2). The actionability items ‘provides tangible tools whenever it could help’ and ‘use visual aid whenever possible to help take action’ were adhered to less than 50% and 38% of time (Table 2).
Table 2: Proportion (%) of Australian national multilingual diabetes consumer education materials meeting agreement for Patient Education Material Assessment Tool (PEMAT) items, by language, and assessment panels.
|Health professional panel||Consumer panel||Health professional panel||Consumer panel|
|Consumer DL||Consumer BH||Consumer DL||Consumer BH|
|Makes its purpose completely evident||100||100||100||100||100||100|
|No distracting information||100||100||100||63||100||100|
|Word Choice and Style|
|Common, everyday language||13||88||75||13||100||100|
|Medical terms are defined and used only to familiarise readers||100||50||38||88||75||88|
|Use of Numbers|
|Numbers are clear and easy to understand||88||88||100||88||100||100|
|Does not expect readers to do calculation||100||100||100||100||100||100|
|Chunk information (long paragraph) into short sections||0||63||100||0||88||100|
|Sections have informative headers||100||100||100||100||88||100|
|Presents information in a logical sequence||100||100||100||100||100||100|
|Provides a summary||0||13||13||13||13||13|
|Layout & Design|
|Uses visual cues on key points||100||88||88||100||100||100|
|Use of Visual Aids (VA)|
|Uses VA whenever possible||13||25||38||13||25||38|
|VA reinforce rather than distract||13||75||100||13||25||38|
|VA have clear titles and captions||0||0||0||0||0||0|
|VA are clear and uncluttered||100||100||100||100||100||100|
|Tables are simple with short, clear role and column headings*||n/a||n/a||n/a||n/a||n/a||n/a|
|Identifies at least one action for the user||75||88||100||75||100||100|
|Addresses the user directly||100||100||100||100||100||100|
|Breaks down actions into explicit steps||63||63||88||63||75||88|
|Provides tangible tools whenever it could help||50||38||50||25||25||38|
|Instructions and examples for calculations*||n/a||n/a||n/a||n/a||n/a||n/a|
|Explains how to use the charts, diagrams etc.*||n/a||n/a||n/a||n/a||n/a||n/a|
|Use VA whenever possible to help take |
*: n/a indicates the mean could not be calculated as some or all materials for this PEMAT item was rated ‘not applicable’.
This study is the first to demonstrate that the whole series of consumer education materials published in 2018 by Diabetes Australia and the National Diabetes Services Scheme on eight different topics had adequate understandability with modest to adequate actionability, as assessed by a validated instrument (the Patient Education Materials Assessment Tool). One of the most important contributors to lower understandability scores was the absence of visual aids such as graphs, tables, charts and pictures. While minimum recommendations for understandability were met for all consumer education materials in both English and Chinese, only two content categories – medical and lifestyle information – achieved the actionability recommendation in both languages. In addition, there was little difference found in understandability and actionability scores between the English and the translated Chinese texts of the same consumer education materials.
Key contributors to lower understandability and actionability scores in this study related to not providing summaries, tangible tools or visual aids to support the consumer in understanding the information and taking action. Nevertheless, there are some exceptions to the above poorer scores. The ‘Making Healthy Food Choices’ fact sheet included different visual aids – the healthy eating plate model chart, the standard alcohol serving size graph, and the recommended food choice images (low glycaemic index carbohydrates and healthy fats) next to the relevant text. These visual aids could be useful in conveying key messages in making healthy food choices for people with diabetes and contribute to a person’s confidence to apply this information in their daily life. An important contributor to actionability scores was the use of a ‘tangible tool.’ For example, the ‘Managing Hypoglycaemia’ fact sheet included a step-by-step checklist for treatment of hypoglycaemia to clearly guide people to promptly and correctly treat this medical emergency. As a result, this factsheet scored high in actionability. Another study exploring hypoglycaemia consumer education materials also found visual aids and cues could contribute to a greater understandability and actionability.14
Variable actionability scores across different content categories in our study suggest that the content of education materials may influence actionability. As outlined above, materials from the lifestyle (‘Making Healthy Food Choices’) and medical categories (‘Managing Hypoglycaemia’) used titles that oriented the reader’s expectations towards taking action. As such, from the start, these materials naturally lent themselves to ‘tangible tools’ such as a meal plan and checklist. By comparison, the materials in the general information category had the lowest actionability. A previous US study also reported very low actionability for materials that provide general information about diabetes and insulin.14 This study concluded that actionability could be improved by including, for example, instructions on how to administer and store insulin.14 We suggest that this can be extended even more broadly, for example, by including ‘next steps’ sections in general information materials. This might include steps to obtain more information, connect with relevant services, or set short-and long-term goals. This could, in turn, broaden the intended impacts of those materials through new connections to services or programs, and opportunities to practise skills relevant to the general information provided. In addition, it is worth considering how the titles of educational materials shape expectations of content. For example, three of the four titles for general information materials used the word ‘understanding’. Adapting these titles to incorporate a sense of action could also improve actionability.
Another key finding from our study is the translated texts had similar mean PEMAT scores as the English texts, suggesting that translation had a minimal impact on a text’s understandability and actionability. While the similar PEMAT scores in different languages of the same source text (English) is encouraging and reinforces the importance of having high understandability and actionability in the source texts to support the development of effective translated versions, it is possible the PEMAT is not sensitive enough to detect linguistic nuances or issues such as mistranslation. The translation process is often perceived to be a challenging aspect of creating multilingual materials as evidenced by a linguistic study which looked at an earlier version of the same patient education materials and found a number of mistranslations or inappropriate translations for the target culture and audience.18 For instance, the hypoglycaemia treatment ‘glucose gel’ was mistranslated as glucose jelly and ‘private health insurance’ was translated as private health investment fund. 18 This provides further support for a multimodal approach to assess consumer education materials, incorporates a broad range of assessments like readability, actionability, understandability, cultural accessibility and linguistic comprehensibility.
Limitations and strengths
This study builds on our previous work which examined the readability of the previous version (published in 2016) of materials published by Diabetes Australia and NDSS, but moved beyond rigid algorithms that only consider a few surface linguistic features like sentence length and word length;11 the PEMAT offers more critical evaluation of the quality of materials than the readability testing. Our finding that mean readability scores were above the recommended reading level for most materials,11 but that they had adequate understandability scores as assessed by PEMAT, provides a more nuanced understanding of the strengths and limitations of national diabetes materials. This finding is also in line with previous studies which have found that the relationship between readability and PEMAT scores is often inconsistent or negatively associated.14, 19 However the PEMAT is still limited in its scope in that it does not assess, among other things, cultural suitability, and the accuracy, and comprehensiveness of the content as well as the linguistic analysis in the translated texts like syntactic and discourse construction.13
The decision to include consumers as panel members in this study is both a strength and a limitation. On the one hand, both consumer panel members were young skilled migrants who spoke good English. Their age, education and prior knowledge about diabetes would contribute to their assessments of understandability and/or actionability. As such, their PEMAT assessments might be more reflective of people with higher health literacy. It is important to acknowledge consumers have a diversity of experiences that vary with socioeconomic status, treatment therapies, durations and types of diabetes and other comorbidities. To address this limitation, we also included assessments from bilingual diabetes health professionals who have extensive experience working with a range of consumers. On the other hand, the validity of the findings is strengthened by including consumers and health professionals in the PEMAT assessment panel. Previous studies that assessed the quality of the printable health information have generally been carried out by researchers alone. 14, 16, 19, 20 As far as all authors are aware, this is the first time the diabetes materials in languages other than English have been tested with health professionals and consumers head-to-head by using the valid health literacy tool, PEMAT-P.
It would be interesting to investigate PEMAT assessments of diabetes materials with other language and cultural groups and with a broader panel of assessors. Further work should also continue to employ and compare multiple assessment instruments such as readability scores and PEMAT, but also extend this analysis to linguistic discourse, translation accuracy and appropriateness, and cultural suitability. It may be worth considering the Patient-Oriented and Culturally-Appropriate (POCA) model, which attempts to address the above factors.21 In particular, the POCA model’s ‘informational practicality’ domain assesses information practicality in a culturally appropriate manner. 21 This could prompt resource developers to include referrals to culturally specific services to warrant a higher actionability. As such the assessment using the POCA model would provide additional information about the understandability of materials for consumers from culturally and linguistically diverse backgrounds.
The multilingual consumer education materials published by Diabetes Australia and the National Diabetes Services Scheme are appropriate to be included as a part of the diabetes education for self-management, and could help improve health inequity, and reduce the disparity in diabetes health outcomes in culturally and linguistically diverse communities in Australia. The findings from this study can be used to increase the confidence health care professionals have in the national diabetes consumer education materials, and encourage them to incorporate the consumer education materials into their practice for people with diabetes. For those developing and revising consumer education materials, it is recommended to use multiple health literacy assessment tools and pay particular attention to the PEMAT items that were overlooked in this study.
The national multilingual diabetes consumer education materials demonstrate an overall favourable understandability and actionability in both English and the Chinese language. Ensuring adequate understandability and actionability of English templates for translated materials is a useful first step in creating translated documents that adhere to the universal precautions approach to health literacy. Overall, this study provides evidence that Australian national diabetes materials largely adhere to recommendations for effective written health education materials, which may help the culturally and linguistically diverse communities to further increase their diabetes self-management understanding and facilitate desirable behaviour change.
- Health professionals and consumers used the Patient Education Materials Assessment Tool (PEMAT) to assess Australian national diabetes consumer education materials in English and Chinese languages.
- The study found the materials met recommendations for effective written communication in both languages, and are appropriate to be used in diabetes education.
- Resources with the highest scores incorporated visual aids, everyday language, active voice and minimal complex medical terms.
Julie Ayre, Danielle Marie Muscat
The authors acknowledge the contribution of the consumer panel members Dingxing Lin (Sydney) and Bin Han (Sydney), and the health professional panel members Jennifer Zhen (Credentialled Diabetes Educator, Redland Diabetes, Metro South Hospital and Health Service, Queensland Health) and Professor Yutao Lan (Diabetes Nurse Educator, Diabetes Research in Nursing Education, Guangdong Medical and Technology University), for their valuable input in assessing the diabetes consumer education materials using PEMAT, and the guidance Associate Professor Meng Ji has provided to the author throughout the PhD research.
Australian Bureau of Statistics. National health survey: first results, Australia, 2017â18, 4364.0.55.001.30(48). Canberra: ABS; 2018. [cited 2019 Oct 5]. Available from:Â www.ausstats.abs.gov.au/ausstats/subscriber.nsf/0/4B3976684C09F43FCA258399001CE630/$File/4364.0.55.001%20-%20national%20health%20survey,%20first%20results,%202017-18.pdf2.
Australian Bureau of Statistics. National health survey: first results, 2017â18, Table 4: long-term health conditions by population characteristics â Australia (dataset). Canberra: ABS; 2018. [cited 2019 Oct 5]. Available from: www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/4364.0.55.0012017-18?OpenDocument3.
Colagiuri R, Thomas A, Buckley A. Preventing type 2 diabetes in culturally and linguistically diverse communities in NSW. Sydney: NSW Department of Health; 2007. [cited 2019 Nov 5]. Available from: www.static.diabetesaustralia.com.au/s/fileassets/diabetes-australia/68e22e06-4281-4fb1-8d06-59af53719d85.pdf4.
Wang L, Gao P, Zhang M, Huang Z, Zhang D, Deng Q, Wang L, et al. Prevalence and ethnic pattern of diabetes and prediabetes in China in 2013. JAMA. 2017;317(24):2515â23.5.
Lee CMY, Colagiuri R, Magliano DJ, Cameron AJ, Shaw J, Zimmet P, Colagiuri S. The cost of diabetes in adults in Australia. Diabetes Research and Clinical Practice. 2013;99(3): 385â90.6.
Ueno H, Hirono I, Ryo S, Yoshihiko I, Yumiko O, Toshimasa Y, Kiuchi T, et al. The association between health literacy levels and patient-reported outcomes in Japanese type 2 diabetic patients. SAGE Open Medicine. 2019;7:1â10.7.
Colagiuri S, Dickinson S, Girgis S, Colagiuri R. National evidence based guideline for blood glucose control in type 2 diabetes. NHMRC Clinical Practice Diabetes Guidelines. Canberra: Diabetes Australia and NHMRC; 2009:108â68, 190â204. [cited 2019 Oct 15]. Available from: www.static.diabetesaustralia.com.au/s/fileassets/diabetes-australia/659c89a3-dcc2-4a2e-86e5-cc1d09956c60.pdf8.
The Royal Australian College of General Practitioners. General practice management of type 2 diabetes: 2016â18. Victoria: RACGP; 2016:120â4. [cited 2019 Aug 5]. Available from: www.racgp.org.au/FSDEDEV/media/documents/Clinical%20Resources/Guidelines/Diabetes/General-practice-management-of-type-2-diabetes_1.pdf9.
National Institute for Health and Care Excellence. Type 2 diabetes in adults: management. NICE guideline. London: NICE; 2015:9â13. [cited 2019 Nov 29]. Available from: www.nice.org.uk/guidance/ng28/resources/type-2-diabetes-in-adults-management-pdf-183733861549310.
Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Annals of Internal Medicine. 2011;155(2):97â107.11.
Lin S, Ayre J, Muscat MD. How readable are Australian multilingual diabetes patient education materials? An evaluation of national English-language source texts. Public Health Research & Practice. Forthcoming 2020.12.
Brega AG, Barnard J, Mabachi NM, Weiss BD, DeWalt DA, Brach C, West DR, et al. Assess, select, and create easy-to-understand materials: tool #11. Agency for Healthcare Research and Quality health literacy universal precautions toolkit (2nd ed.). Rockville, US: AHRQ. 2015:45â6. [cited 2019 Nov 30]. Available from: www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/literacy-toolkit/healthlittoolkit2-tool11.html13.
Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability an actionability for print and audiovisual patient information. Patient Educ. Couns. 2014;96(3):395â40314.
Lipari M, Berlie H, Saleh Y, Hang P, Moser L. Understandability, actionability, and readability of online patient education materials about diabetes mellitus. American Journal of Health-System Pharmacy. 2019;76(3):182â6.15.
Cultural and Indigenous Research Centre Australia. Consumer health information needs and preferences: perspectives of culturally and linguistically diverse and Aboriginal and Torres Strait Islander people. Sydney, Australia: Australian Commission on Safety and Quality in Health Care. 2017:13â5. [cited 2019 Nov 15]. Available from: www.safetyandquality.gov.au/sites/default/files/migrated/Consumer-needs-and-preferences-Perspectives-of-culturally-and-linguistically-diverse-and-Aboriginal-and-Torres-Strait-Islanders.pdf16.
Mastroianni F, Chen Y, Vellar L, Cvejic E, Smith JK, McCaffery KJ, Muscat DM. Implementation of an organisation-wide health literacy approach to improve the understandability and actionability of patient information and education materials: a pre-post effectiveness study. Patient Educ. Couns. 2019;102(9):1656â61.17.
Shoemaker SJ, Wolf MS, Brach C. The Patient Education Materials Assessment Tool (PEMAT) and userâs guide. Rockville, US: Agency for Healthcare Research and Quality. 2013:1â67. [cited 2019 Nov 29]. Available from: www.ahrq.gov/professionals/prevention-chronic-care/improve/self-mgmt/pemat/index.html18.
Lin S, Ji M. Assessing linguistic comprehensibility of healthcare translation using the Patient-Oriented and Cultural-Adapted (POCA) model. In: Ji M, Taibi M, Crezee IHM (Ed.), Multicultural Health Translation and Interpreting. London: Routledge. 2019:69â84. [cited 2019 Aug 14]. Available from: www.taylorfrancis.com/books/e/978135100039019.
Wong K, Keefe K, Gilad A, Chong-Sun Li CJ, Levi JR. Parental actionability of education materials regarding laryngotracheal reconstruction. JAMA Otalaryngology-head & neck surgery. 2017;143(9):953â4.20.
Guan Y, Maloney KA, Roter DL, Pollin TI. Evaluation of the informational content, readability and comprehensibility of online health information on monogenic diabetes. Journal of Genetic Counseling. 2018;27(3):608â15.21.
Lin S, Ji M. Developing Patient-Oriented and Culturally-Adapted (POCA) health translation model. In: Ji M (Ed.), Cross-Cultural Health Translation. London: Routledge. 2019:30â46. [cited 2019 Sept 24]. Available from: www.taylorfrancis.com/books/e/9780429054402