The Nursing Home Rating System Revisited
In my June 30, 2008 blog post "Can Apples and Oranges be Compared?" I wrote about the anticipated implementation of the nursing home rating system that the federal government was going to introduce. Since that time the Centers for Medicare and Medicaid (CMS) have introduced this system, leading to a system of ranking that has produced controversy and outrage among many long-term care professionals. Nursing homes are rated on three major measures: health inspections, nursing home staffing and quality measures. Out of these data CMS produces a ranking system based on stars with a one star ranking being the lowest and a five star ranking being the highest. Each of the three categories is ranked on one to five stars and each nursing home obtains an overall ranking of one to five stars based on these three criteria.
In my previous article I mentioned that you cannot compare apples and oranges and yet, this is apparently what the federal government has attempted to do in their ranking system. When you look at this system I am very puzzled on how many of the overall ranks are finally attained. I see many facilities that appear to average three or four stars yet receive an overall ranking of two or three stars. Conversely, I also see many that appear to average three or four stars in the areas that are examined, yet they come away with an overall ranking of two or three stars.
This is interesting. I then wonder how they weigh the respective areas. Is one of the three areas weighted differently, or are they all weighted the same, which in turn in some way should lead to the overall rankings that are found. However, I am not sure how they come away with these rankings. Looking at this system reminds me of going into a college course with the professor not providing any information on how grading takes place. Intuitively the system looks quite appealing. We like simplistic systems that do not require much thought. When we hear that a hotel or restaurant is five stars, we automatically assume that it is a quality hotel or restaurant. We often fail to question how the rankings were put together. With the nursing home ranking system it appears to be another consumer friendly ranking system, but what does it really mean, and how are the overall star rankings attained.
A major question that I have to raise is how do individuals at the federal level put together an overall ranking of every nursing facility in the United States by looking at 1) health inspection surveys; 2) nursing home staffing; and 3) quality indicator measures. Of the three variables that are examined, nursing home staffing is the only highly objective and highly empirical variable that can be found. You can count and average out this variable with high levels of objectivity. However, the other two variables, health inspection surveys and quality indicator measures have tremendous variability. Health survey inspection teams have great variability. Some teams provide more citations than others.
In addition, states vary considerably in the survey team's provision of health citations, with some states differing quite dramatically in the average number of citations given. Furthermore, the quality indicator reports provide information on how nursing homes rate in certain areas as compared to other nursing homes, but it fails to take acuity into consideration. For instance, a nursing home can rank very high in pressures sores but it may also have a terribly high acuity of clientele as compared to another nursing home that does not admit residents that are as severely ill or infirm. These are just a few of the major problems that can be found in comparing data of this nature.
So how do those that conduct the analysis to provide these rankings create a common benchmark to compare all nursing care facilities equally? With all these confounding factors that can work to influence the results what type of statistical control exists, if any. It would be naïve to assume that the data that is being used is not compromised in any way. However, it appears that those at the government level that are conducting this rating are assuming that no variability or biases in the data exist and take the data at full face validity. This assumption is an important methodological flaw. Furthermore, how do they measure the data? When things are examined in the scientific community the methodology is always made public to assure that others can examine it as well. However, in this case, what types of measuring tools where used and how the measurements lead to this ranking system that in turn achieved the results are for the most part a mystery.
So where does that leave us. It leaves us with an apparent system that is suppose to rank nursing homes objectively. However, what type of objectivity really exists in this measure? Do nursing homes that take a higher level of severe residents get penalized unfairly in this ranking system? Do nursing homes within lower socioeconomic regions that fail to obtain the funding that other nursing care facilities obtain get penalized by this system unfairly? Can those who do the rating, not knowing if the data that they are using is compromised in some way, say that they are doing a totally objective analysis? These are important questions that this new rating system brings up and needs to answer.
Unfortunately, when people here that a rating system has been done, they often view it as a totally reliable system that one must take unquestionable heed of as if it is the final word. But as I have mentioned, this supposed ranking system poses too many questions. It is interesting to note that as I have mentioned in my previous article, it is very difficult, if not impossible, to say that hotels, restaurants, and nursing care facilities can be ranked on the basis of stars. In an industry that deals with people's lives, with a byzantine complexity, it is interesting that we now feel that we can rank this complexity on a system of stars.