Health Measurement Scales: Methodological Issues

The use of the Appropriate Scoring System

Based on an extensive literature search, the majority of scales that have been constructed using 1, 2 or 3 thresholds for each component and assigning scores of 1, 2, 3 when the attitude is towards or not a healthy behavior (or vice versa). In a recent publication [9] we investigated whether the number of classes of scale components influences the diagnostic accuracy of the tool. The accuracy was measured through the area under the Receiver Operating Characteristics (ROC) curve (i.e. the Area Under Curve, AUC). Particularly, based on simulated data, a scale with infinity number of components (I∞) from the Normal distribution was initially developed. Afterwards, 10 other new scales were developed from the previous one using 100-quantiles, 50-quantiles, 15-quantiles, 10-quantiles, 8-quantiles, 6-quantiles, quintiles, quartiles, tertiles and the median of I∞ scale. A positive association between the number of classes and the diagnostic accuracy of the scales (measured using the sensitivity and AUC) was observed (Fig. 1). This finding was also confirmed by the choice of natural logarithm as the function that best describes the relationship between the number of classes of each scale component and the measures of diagnostic accuracy of the tool. The previous findings lead to the conclusion that the maximum diagnostic accuracy of a scale is achieved when the maximum number of classes in each component is used.

The latter association remained stable when some components were un-correlated with the outcome, while it was not detected when all index components were un-correlated with the outcome.

The finding of the aforementioned work is important for health practice and research, since it strongly suggests using as many classes as it can be used for a scale’s components. For example, in order to measure the level of anxiety (a potential cardiovascular risk factor) one may use a scale with many possible responses (like not at all, 3-5 per month, 1-2 per week, daily etc) instead of using a simple “yes/no” answer. Taking also into account that most clinicians wish to have a diagnostic tool that accurately predicts the truly diseased individuals, this conclusion may substantially reduce the level of misclassification in screening procedures for high risk people. However, one may claim that scales constructed using small number of classes may be more comprehensive and easier applied in daily clinical practice. As discussed above they result in a tool with low diagnostic accuracy. The latter could be an explanation for the lack of significant findings from some studies that used small-range scales [10, 11]. Moreover, one may also argue that large-range components may lead to lack of reproducibility, which is of importance for assessing etiology in medical research. The latter consideration is still under investigation.

Nevertheless, it should be mentioned that the use of continuous instead of discrete components in order to develop a composite scale is always preferable. However, in many cases the use of continuous components is not feasible for various practical reasons, like the inability of a person to quantitatively measure a feeling or behaviour.

The findings discussed suggest the use of large-range components. These considerations are of clinical as well as methodological importance since their application in practice will provide researchers with a methodological framework to develop more accurate predicting tools of a health related outcome.

The use of Weights

The contribution of each component to the total score of a composite scale is a major, unresolved issue. The majority of the presented scales have been developed assigning the same weight to all components. However, it is widely accepted that in the majority of the scales the components do not have the same relationship to a health outcome. For example, consider a composite scale that aims to evaluate the risk for developing cardiovascular disease, which is based on arterial blood pressure, lipids and glucose levels. It is rational to believe, based on the evidence, that all the aforementioned components do not have the same effect on the development of cardiovascular disease. Moreover, several studies examining the relationship between dietary scales and disease outcomes have reported no or moderate associations [12, 13]. The latter could also be a result of the use of un-weighted scales to evaluate the food-health relationships. One may claim that using unequally distributed scores of a scale’s components is an indirect weighting method. Specifically, some investigators have suggested using different scaling distributions between the components (e.g. in the example given before one may assign score 1 in those who have hypertension, or abnormal lipids levels, but score 2 for those reporting diabetes, since the later has stronger associations with the disease; in those without these conditions score 0 would be assigned). However, the previous “weighting” system is arbitrary, lacks of scientific basis and therefore, cannot be provided as a methodological guideline.

In a very recent work, 3 weighting methods for scale components were proposed and tested using both simulated and empirical data [14]. In particular, the odds ratios obtained from unadjusted logistic regression that evaluated the effect of individual scale components on a hypothesized health outcome were suggested as specific weights. Although the use of these specific weights was associated with improvements in the diagnostic ability of the scales, this method shares some important limitations. Specifically, the suggested weights are based on the relative relationship of each scale’s item with a particular, simulated outcome. Therefore, even for the same component of a scale, different weights could be proposed when applying the scale to different health outcomes. In addition, the influence of the inter-correlations and synergistic effect of the scale items, which is common in real practice, are ignored using the odds ratios obtained from unadjusted models as potential weights. Thus, the use of odds ratios obtained from multiple logistic regression models may improve the accuracy of the scale.

Then, and under the concept of the same work, the weights of the scales’ components were derived based on odds ratios obtained from multiple logistic regression models when each index component was entered as independent variable and a total score of the rest of components (i.e. summation of the initial scores of these components) was entered in the model. The latter was made to control for the potential confounding effect of the other components. It should be noted that a score of the components was preferred instead of all components separately, because it is very common that scale items are highly correlated, since they aimed to describe the same characteristic. Afterwards, another scoring procedure was also suggested. Particularly, the weights of scale components were obtained by multiplying the aforementioned weights with the Deviance, which is a measure of the importance of the component in predicting outcome.

Finally, weights were suggested by multiplying the weights obtained from the odds ratios with the inverse of the variance of the specific odds ratio, which represents the effect size of the association. The current findings highlight that the predictive capacity of the weighted indices constructed using the weights mentioned above is higher compared to that of the un-weighted index (Fig. 2). However, no meaningful differences were observed between the different scoring procedures suggested in the referenced work. The previous findings were confirmed when empirical data were used from an epidemiological study [14].

Although the use of the suggested weights seems to improve the diagnostic accuracy of health measurement scales, some important issues of using them should be further discussed. At first, the proposed weights are based on the relative relationship of each scale item with the specific outcome. Thus, different weights should be proposed for different health outcomes for the same components. For example, let assume a scale that is based on food group consumption. Based on the literature all foods included influence both the development of hyperlipidemia, hypertension and diabetes; however, they do not have the same impact on several health outcomes, such as coronary heart disease or stroke. Therefore, different weights should be used for specific foods according to their relationship with coronary disease or stroke. In other words, several food-based scales should be developed based on what the scale intends to predict. Another limitation of the aforementioned methods of scoring is that specific weights for scale components should be given based on specific datasets. Thus, it is not certain that weights derived from 1 dataset will improve the predictive ability of corresponding indices developed from another dataset. In other words the issue of lack of generalization arises. A solution to the aforementioned consideration is using the effect size estimates (i.e., odds ratios) from available metanalyses of studies investigating the same topic.

Thus, despite the clarity and consistency of the weights presented in previous works, further research is needed in order to examine whether other effect size measures may be used as weights of a scale components.

The Number of Components and the Level of Inter-correlation between the Components

In the above paragraphs the range of values and the use of weights of the scale components were discussed. However, the number of components used or the correlation structure of the components remains an important, undiscovered topic for the development of an accurate health measurement scale. In one of our recent studies in this field we showed that the diagnostic accuracy of a scale increases as the number of components increases, if and only if all components are related to the outcome that the scale intends to evaluate [15]. Moreover, the diagnostic accuracy is improved when non- or low inter-correlated components are used.

Specifically, based on simulated data we showed that the diagnostic accuracy of a scale developed using non- or low inter-correlated components is higher compared with that of a scale that was developed using moderate to high inter-correlated components. The latter reveal the issue of the explained variability of the components that are high correlated. In addition to the previous findings, the diagnostic accuracy of a scale increases as the number of components used increases, too. Moreover, all scale components should be related to the outcome that the scale aims to predict, since the simulated and empirical data of this work showed that the diagnostic accuracy of a scale developed using both related and un-related to a specific outcome components is lower compared with a scale developed using only the components correlated with the outcome. This could be an explanation why several health scale measurements, especially from the nutrition epidemiology field, have failed to observe significant associations with health outcomes [13]. The majority of these dietary scales have developed based on general and not disease-specific dietary recommendations. Furthermore, in the aforementioned work we suggested that the diagnostic accuracy of a single component is higher than of the accuracy of a scale that has been developed using inter-correlated components and with only some of them associated with the outcome.

Finally, the correlation structure of the components does not affect the already reported association between the number of classes and the diagnostic accuracy of the scale. (Fig. 3). In particular, it was observed that the diagnostic accuracy increases as the number of classes increases, irrespective of the correlation structure of the components. All the previous findings were also confirmed when empirical data were used [15].

The aforementioned findings, suggest the use of low- or non- inter-correlated components with a high level of association with the investigated outcome, in order to obtain an accurate scale. The number of components used for the development of a scale is important only when low- or non- inter-correlated components are used; in this case the diagnostic accuracy increases as the number of components increases.