Are diabetes consumer education materials easy to understand and act on? An evaluation of multilingual Australian resources

Key words: Health literacy, diabetes, multilingual, patient education materials, PEMAT

A growing body of research has explored strategies to improve the effectiveness of written health materials. A systematic review found that strategies to improve written communication (such as using simple language and relevant illustrations) were effective in improving knowledge and cognitive outcomes.¹⁰ However, there is some evidence that these strategies may not have been incorporated into key national diabetes resources. Our recent assessment of the readability of eight national diabetes consumer education materials, for example, found that on average these materials were written at a grade 10 reading level, well above the grade 8 reading level recommended by health literacy guidelines.¹¹

Although this study provided a useful indication of the understandability of national diabetes materials, readability is a relatively narrow assessment of the health literacy demand of materials, and could only be calculated for English-language materials.¹¹ The Health Literacy Universal Precautions Toolkit recommends using a range of methods to examine the quality of written materials, as many features cannot be captured by a single test.¹² The Patient Education Materials Assessment Tool (PEMAT) provides a broader assessment of the health literacy demands of educational materials.¹³ This validated tool assesses two constructs: understandability (which includes assessments of readability as well as structure, layout and design); and actionability (assessment of strategies that help users identify what to do based on the information provided). A recent American study used the PEMAT to assess 13 diabetes education materials that were available online from national not-for-profit organisations and pharmaceutical companies. ¹⁴ This study found that the readability of materials was not associated with PEMAT scores, very few materials had adequate understandability, and only one material had adequate actionability. Together, these results highlight the importance of a multimodal approach to assessing the health literacy demands of diabetes education materials.

The PEMAT has the added advantage that it can be used to assess education materials that are written in languages that do not have readily-available readability formulas. The ability to concurrently assess English and translated materials is particularly important given Australian research has found that people from culturally and linguistically diverse communities prefer health information to be available in both English and their first language.¹⁵

This study aimed to examine the understandability and actionability of printed Australian diabetes materials available in English and Chinese, and to explore the factors that could improve the understandability and actionability of such materials.

Methods

Study design

This was a non-experimental study that involved assessing the understandability and actionability of diabetes consumer education materials using a validated health literacy instrument.¹³

Selection of consumer education materials

A search was conducted to obtain the publicly and freely accessible diabetes materials in a printable format (in PDF) and available in English and Simplified Chinese. To ensure content accuracy and professional credibility, only the latest versions of materials jointly published by the Australian government (National Diabetes Services Scheme) and the national peak diabetes organisation (Diabetes Australia) were selected. Information only in English, or in video, pictorial or graphic form was excluded, as were any materials not directly related to diabetes self-management. The search was conducted in February to March 2019.

Categorisation of the consumer education materials

A previous study examining the readability of similar materials from the same publishers categorised materials into four main content categories: general, medical information, nutritional, and physical activity.¹¹ In this study, the nutritional information and physical activity information sections were combined into one category (‘lifestyle information’), so that three categories – general, medical and lifestyle information – were applied to the study

Assessment tool

The Patient Education Materials Assessment Tool for printed materials (PEMAT-P) was developed and validated by Shoemaker and colleagues to assess understandability (17 items) and actionability (7 items).^{12, 13} Each item is scored on a binary scale (agree (1) or disagree (0))  excluding seven items which include a ‘not applicable’ option. Results are expressed as the percentage of items coded ‘agree,’ such that a higher percentage suggests the consumer education material is more likely to be accessible and easier for a person to act on the information presented. A score of at least 70% is considered indicative of reasonable understandability and actionability.¹³ The PEMAT-P was validated to be used by both healthcare professionals and laypersons with consistent and reliable results among different assessors, and has been previously used to assess Australian consumer education materials.^{13, 14, 16}

 Scoring and analysis

After receiving training from the Patient Education Material Assessment Tool User’s Guide,¹⁷ an assessment panel comprising three health professional members and two consumer panel members (people with diabetes) scored materials using the PEMAT. All panel members spoke Chinese as their first language and had high English proficiency. See Appendix 1 and Appendix 2 for the detailed characteristics.

Two health professional panel members (SL, JZ) assessed the English texts, and two health professional panel members (SL, YL) assessed the Chinese texts. Disagreements between members were resolved through face-to-face meetings and/or teleconferences. When there was a discrepancy that could not be resolved between two members, the third diabetes educator from the panel (JZ or YL) was consulted.

Consumer members independently read and assessed the English texts. A consensus agreement was not required for the consumer panel members for logistical reasons. Mean and range of PEMAT scores for each material were calculated.

The overall score for each material was the average of the single (consensus) health professional score and the mean of the two consumer scores. These overall scores were averaged to calculate mean scores for each content category.

Results

Our search identified eight materials that met inclusion criteria (Table 1).

Table 1: Mean understandability and actionability scores from the PEMAT-P assessment for individual consumer education materials by content categories and languages.

The mean understandability scores in all content categories were 70% or above in both English and Chinese (range 69-76%), except for the general information category for which the Chinese text was just under 70% (Table 1). The ‘Making healthy food choices’ had the highest mean understandability in both English (78%) and Chinese (78%), while the two materials under the general information category (‘Diabetes-related complications’ and ‘Understanding gestational diabetes’) had the lowest mean understandability scores (67% and 69% respectively) in both languages (Table 1). For each content category, mean understandability scores were similar for English and Chinese texts (with a difference of up to one percentage point).

Average scores for actionability were between 77% and 84% in the medical and lifestyle information category in both languages, but were lower for general information with the lowest mean score of 47% in both languages (Table 1). The materials related to food choices from the lifestyle information category achieved the maximum score for actionability (100%) from all the panel members in both languages (Table 1). Three of the four materials in the general information category scored the lowest mean actionability (40%, 47% and 47% respectively) of all eight materials (Table 1). Average actionability scores for the general and lifestyle information categories were exactly the same across English and Chinese texts (47% and 84%, respectively), but there was a 3%difference between English (80%) and Chinese (77%) scores in the medical information category (Table 1).

Across all materials and both languages, two assessment items for the understandability ‘provides a summary’ and ‘uses a visual aid whenever possible’ were adhered to less than 13% and 38% of the time, respectively (Table 2). The actionability items ‘provides tangible tools whenever it could help’ and ‘use visual aid whenever possible to help take action’ were adhered to less than 50% and 38% of time (Table 2).

Table 2: Proportion (%) of Australian national multilingual diabetes  consumer education materials meeting agreement for Patient Education Material Assessment Tool (PEMAT) items, by language, and assessment panels.

*: n/a indicates the mean could not be calculated as some or all materials for this PEMAT item was rated ‘not applicable’.

Discussion

This study is the first to demonstrate that the whole series of consumer education materials published in 2018 by Diabetes Australia and the National Diabetes Services Scheme on eight different topics had adequate understandability with modest to adequate actionability, as assessed by a validated instrument (the Patient Education Materials Assessment Tool). One of the most important contributors to lower understandability scores was the absence of visual aids such as graphs, tables, charts and pictures. While minimum recommendations for understandability were met for all consumer education materials in both English and Chinese, only two content categories – medical and lifestyle information – achieved the actionability recommendation in both languages. In addition, there was little difference found in understandability and actionability scores between the English and the translated Chinese texts of the same consumer education materials.

Key contributors to lower understandability and actionability scores in this study related to not providing summaries, tangible tools or visual aids to support the consumer in understanding the information and taking action. Nevertheless, there are some exceptions to the above poorer scores. The ‘Making Healthy Food Choices’ fact sheet included different visual aids – the healthy eating plate model chart, the standard alcohol serving size graph, and the recommended food choice images (low glycaemic index carbohydrates and healthy fats) next to the relevant text. These visual aids could be useful in conveying key messages in making healthy food choices for people with diabetes and contribute to a person’s confidence to apply this information in their daily life. An important contributor to actionability scores was the use of a ‘tangible tool.’ For example, the ‘Managing Hypoglycaemia’ fact sheet included a step-by-step checklist for treatment of hypoglycaemia to clearly guide people to promptly and correctly treat this medical emergency. As a result, this factsheet scored high in actionability. Another study exploring hypoglycaemia consumer education materials also found visual aids and cues could contribute to a greater understandability and actionability.¹⁴

Variable actionability scores across different content categories in our study suggest that the content of education materials may influence actionability. As outlined above, materials from the lifestyle (‘Making Healthy Food Choices’) and medical categories (‘Managing Hypoglycaemia’) used titles that oriented the reader’s expectations towards taking action. As such, from the start, these materials naturally lent themselves to ‘tangible tools’ such as a meal plan and checklist. By comparison, the materials in the general information category had the lowest actionability. A previous US study also reported very low actionability for materials that provide general information about diabetes and insulin.¹⁴ This study concluded that actionability could be improved by including, for example, instructions on how to administer and store insulin.¹⁴ We suggest that this can be extended even more broadly, for example, by including ‘next steps’ sections in general information materials. This might include steps to obtain more information, connect with relevant services, or set short-and long-term goals. This could, in turn, broaden the intended impacts of those materials through new connections to services or programs, and opportunities to practise skills relevant to the general information provided. In addition, it is worth considering how the titles of educational materials shape expectations of content. For example, three of the four titles for general information materials used the word ‘understanding’. Adapting these titles to incorporate a sense of action could also improve actionability.

Another key finding from our study is the translated texts had similar mean PEMAT scores as the English texts, suggesting that translation had a minimal impact on a text’s understandability and actionability. While the similar PEMAT scores in different languages of the same source text (English) is encouraging and reinforces the importance of having high understandability and actionability in the source texts to support the development of effective translated versions, it is possible the PEMAT is not sensitive enough to detect linguistic nuances or issues such as mistranslation. The translation process is often perceived to be a challenging aspect of creating multilingual materials as evidenced by a linguistic study which looked at an earlier version of the same patient education materials and found a number of mistranslations or inappropriate translations for the target culture and audience.¹⁸ For instance, the hypoglycaemia treatment ‘glucose gel’ was mistranslated as glucose jelly and ‘private health insurance’ was translated as private health investment fund. ¹⁸ This provides further support for a multimodal approach to assess consumer education materials, incorporates a broad range of assessments like readability, actionability, understandability, cultural accessibility and linguistic comprehensibility.

Limitations and strengths

This study builds on our previous work which examined the readability of the previous version (published in 2016) of materials published by Diabetes Australia and NDSS, but moved beyond rigid algorithms that only consider a few surface linguistic features like sentence length and word length;¹¹ the PEMAT offers more critical evaluation of the quality of materials than the readability testing. Our finding that mean readability scores were above the recommended reading level for most materials,¹¹ but that they had adequate understandability scores as assessed by PEMAT, provides a more nuanced understanding of the strengths and limitations of national diabetes materials. This finding is also in line with previous studies which have found that the relationship between readability and PEMAT scores is often inconsistent or negatively associated.^{14, 19} However the PEMAT is still limited in its scope in that it does not assess, among other things, cultural suitability, and the accuracy, and comprehensiveness of the content as well as the linguistic analysis in the translated texts like syntactic and discourse construction.¹³

The decision to include consumers as panel members in this study is both a strength and a limitation. On the one hand, both consumer panel members were young skilled migrants who spoke good English. Their age, education and prior knowledge about diabetes would contribute to their assessments of understandability and/or actionability. As such, their PEMAT assessments might be more reflective of people with higher health literacy. It is important to acknowledge consumers have a diversity of experiences that vary with socioeconomic status, treatment therapies, durations and types of diabetes and other comorbidities. To address this limitation, we also included assessments from bilingual diabetes health professionals who have extensive experience working with a range of consumers. On the other hand, the validity of the findings is strengthened by including consumers and health professionals in the PEMAT assessment panel. Previous studies that assessed the quality of the printable health information have generally been carried out by researchers alone. ^{14, 16, 19, 20} As far as all authors are aware, this is the first time the diabetes materials in languages other than English have been tested with health professionals and consumers head-to-head by using the valid health literacy tool, PEMAT-P.

Future directions

It would be interesting to investigate PEMAT assessments of diabetes materials with other language and cultural groups and with a broader panel of assessors. Further work should also continue to employ and compare multiple assessment instruments such as readability scores and PEMAT, but also extend this analysis to linguistic discourse, translation accuracy and appropriateness, and cultural suitability. It may be worth considering the Patient-Oriented and Culturally-Appropriate (POCA) model, which attempts to address the above factors.²¹ In particular, the POCA model’s ‘informational practicality’ domain assesses information practicality in a culturally appropriate manner. ²¹ This could prompt resource developers to include referrals to culturally specific services to warrant a higher actionability. As such the assessment using the POCA model would provide additional information about the understandability of materials for consumers from culturally and linguistically diverse backgrounds.

Implications

The multilingual consumer education materials published by Diabetes Australia and the National Diabetes Services Scheme are appropriate to be included as a part of the diabetes education for self-management, and could help improve health inequity, and reduce the disparity in diabetes health outcomes in culturally and linguistically diverse communities in Australia. The findings from this study can be used to increase the confidence health care professionals have in the national diabetes consumer education materials, and encourage them to incorporate the consumer education materials into their practice for people with diabetes. For those developing and revising consumer education materials, it is recommended to use multiple health literacy assessment tools and pay particular attention to the PEMAT items that were overlooked in this study.