Health Risk Appraisal (HRA)

The reviewers do not summarize the conclusions of prior reviews on this topic including those by David Anderson et al., Robin Soler et al., Ron Goetzel and Catherine Heaney, and the Rand review for CMS [Centers for Medicare and Medicaid Services].  These should be cited at the beginning as a foundation for this review.

We added a brief summary of each review to Chapter 1: Introduction.  The summaries are contained in a new section called 'Earlier Literature Reviews'

It is unreasonable to expect that merely administering a survey instrument, with feedback, will change anyone's behavior, let alone prevent the occurrence of a disease.  Previous literature reviews, including those by Rand, Anderson, and Community Guide, came to the same conclusion.

• The focus of the review is primarily on the use of an HRA in isolation when the focus should be on the entire process of administering an HRA, providing feedback, and following up with behavior change and risk reduction support programs.  The Community Guide to Preventive Services, in its review of the worksite literature (which is referenced in this review) differentiated between two types of HRA applications: 1) an assessment of health risks with feedback, when used alone, ("HRA Alone"), and 2) an assessment of health risks with feedback as a gateway to more intensive and prolonged health promotion and risk reductions interventions ("HRA Plus").  That review concluded that an HRA Alone intervention is largely ineffectual while an HRA Plus intervention is effective in achieving long-term behavior change (such as quitting smoking) and significant reductions in biometric risk factors like cholesterol and blood pressure.  At a minimum, the HRA Plus process would involve the administration of the HRA and production of a feedback report that would form the foundation of a personal prevention plan.  However, for the HRA Plus to be most effective, it needs to also include the following components that complement the provision of an HRA with a feedback report:
• Multiple or serial administrations of HRAs, with longitudinal feedback provided to participants on their health risk status,
• Ongoing health education programs, provided through pamphlets, books, videos, or interactive computer programs,
• Motivational interviewing, counseling, and coaching provided face-to-face or telephonically to support behavior change and risk reduction,
• Referral to community resources such as fitness facilities, self-help support groups, or neighborhood volunteer programs, and
• Referral to local or national health promotion vendors and services such as smoking quit lines and wellness coaches.

Without the above follow-through" activities, the HRA alone program will not succeed.  It is unreasonable to expect that merely administering a survey instrument, with feedback, will change anyone's behavior, let alone prevent the occurrence of a disease.  Previous literature reviews, including those by Rand, Anderson, and Community Guide, came to the same conclusion.

• I agree with the statement: "We believe the process following HRA questionnaire administration, namely feedback and recommendations, provides participants with a sense of engagement that encourages behavioral change.

• The criteria applied for the evidence review (e.g., the Jadad scale) are applicable to clinical trial studies but not to "real world" evaluation studies conducted at workplace or community-based settings.  Of the 115 studies reviewed, 53 were done at worksites and 23 in communities.  The requirements for inclusion of studies in the "good" category are overly stringent given the realities of conducting this type of research in applied settings.  For example, studies lost points if there was no double blinding, tracking of withdrawals, and reporting of adverse events.  Again, these criteria are very relevant for drug or clinical trials but not to workplace based programs when employees at an entire worksite are encouraged to become more physically active or eat a healthier diet.  The worksite interventions often involve changes in the physical and social environment, mass communication efforts, policy changes, and creation of support groups for workers wishing to improve their health.  In fact, introducing environmental and policy interventions to improve lifestyle may be more potent than individual counseling, as has been shown to be the case with tobacco use.  Quality scores were biased downward because the Jadad scale's adverse effects question was largely inapplicable to evaluating Health Risk Appraisals (HRAs). As the reviewers admit: "Low quality ratings could reflect poor reporting (perhaps prompted by journal word restrictions) instead of poor research. "

Our discussion of short- versus long-term outcomes occupies a very small portion of the TA.  To avoid drawing disproportionate attention to this issue, we deleted the word 'tangible' from the TA to prevent the formation of an implied hierarchy where intermediate outcomes are given less importance than disease incidence.

• Fielding JE, Mason T, Knight K, et al.  A randomized trial of the IMPACT worksite cholesterol reduction program.  Am J Prev Med 1995;11(2):120-3. PMID:7632447 OVID-Medline. Exclude: Does not report health outcomes—yes it does.
• Goetzel RZ, Ozminkowski RJ, Bruno JA, et al. The long-term impact of Johnson & Johnson's Health & Wellness Program on employee health risks. Journal of Occupational & Environmental Medicine 2002;44(5):417-24. PMID:12024687 OVID-Medline. Exclude: No comparison group—yes there is.
• Heaney C A; Goetzel R Z., A review of health-related outcomes of multi-component worksite health promotion programs. American journal of health promotion: AJHP 1997;11(4):290-307.
• Can health promotion programs save Medicare money? Ron Z Goetzel, David Shechter, Ronald J Ozminkowski, David C Stapleton, Pauline J Lapin, J Michael McGinnis, Catherine R Gordon, Lester Breslow. Am J Health Promot 22(1):suppl 1-7, iii.
• Short Meghan E; Goetzel Ron Z; Young Jared S; Kowlessar Niranjana M; Liss-Levinson Rivka C; Tabrizi Maryam J; Roemer Enid Chung; Sabatelli Adriano A; Winick Keith; Montes Myrtho; Crighton K Andrew, Measuring changes in lipid and blood glucose values in the health and wellness program of Prudential Financial, Inc. Journal of occupational and environmental medicine / American College of Occupational and Environmental Medicine 2010;52(8):797-806.
• Goetzel Ron Z; Baker Kristin M; Short Meghan E; Pei Xiaofei; Ozminkowski Ronald J; Wang Shaohung; Bowen Jennie D; Roemer Enid C; Craun Beth A; Tully Karen J; Baase Catherine M; DeJoy David M; Wilson Mark G. First-year results of an obesity prevention program at The Dow Chemical Company. Journal of occupational and environmental medicine / American College of Occupational and Environmental Medicine 2009;51(2):125-38.
• Goetzel Ron Z; Roemer Enid C; Pei Xiaofei; Short Meghan E; Tabrizi Maryam J; Wilson Mark G; Dejoy David M; Craun Beth A; Tully Karen J; White John M; Baase Catherine M.  Second-year results of an obesity prevention program at the Dow Chemical Company. Journal of occupational and environmental medicine / American College of Occupational and Environmental Medicine 2010;52(3):291-302.
• Goetzel, R.Z., Sepulveda, M., Knight, K., Eisen, M., Wade, S., Wong, J., Fielding, J. "Association of IBM's 'A Plan For Life' Health Promotion Program with Changes in Employees' Health Risk Status."  Journal of Occupational Medicine 36:9, September 1994, 1005.

We added this article to the TA.

We added this article to the TA.

We added a summary of this article to the introduction, as per a previous comment from the reviewer.

This article is a review, not primary research, and would therefore not be included in a systematic review.

This article was excluded because it was published after our literature search cut-off date of June 2010.

This article was excluded because treatment and control subjects all received HRAs; the authors are really comparing environmental interventions independent of HRAs.

This article was excluded because it is a companion paper to the previous publication, which was also excluded.

We added this article to the TA.

Our definition of an HRA, namely that it included questionnaires or clinical tests, feedback, and recommendations, was based on the RAND report (RAND Santa Monica CAL. Evidence Report and Evidence-based Recommendations: Health Risk Appraisals and Medicare. Contract no.: 500-98-0281. Baltimore, MD: Centers for Medicare and Medicaid Services, 2003).  We agree the notion of what constitutes an HRA is subject to debate and we have mentioned this issue in the introduction. However, to guide the review, we had to select one specific definition and felt a broad-based definition was valid given that the consensus in health promotion appears to be that HRAs are more than questionnaires or clinical tests undertaken to collect information on risk factors for disease.

We made this change in the discussion as well.

Other reviews of HRAs are in the literature.  However, the general form of a TA is to provide a brief introduction and background before listing the key questions and turning to the methods.

The TA was commissioned to address specific key questions, so our focus was directed to these questions.

We replaced the word 'study' with 'article' (and vice versa) in many places throughout the TA to better differentiate between articles and studies.

We added a brief summary of 10 studies with non-HRA controls to the results for key question #2.  These studies confirm our assertions regarding the effects of feedback and recommendations.

We are referring to the number of follow-up contacts undertaken as part of the HRA itself.  To clarify this point, we added the word 'contacts' or 'contact' to create the phrases "followup contacts" or "followup contact". We also deleted the last sentence of p. 76, paragraph 6 (i.e., "However, the true impact of HRAs cannot be adequately assessed over such a small number of contacts and future research should involve more frequent followups.").

We added the following definition of general health to the text: "HRAs targeting general health collect data on an assortment of risk factors without a specific interest in any one disease (e.g., CVD) or behavioral area (e.g., smoking cessation, physical activity)".

The notes to Table 5 have been moved to the end of the Table.

Specific articles may be counted in more than one column in Figure 8.  We revised the footnote to Figure 8 to make this fact clearer.

We made the requested change to the text.

We made the requested change to the text.

We made the requested change to the text.

We made the requested change to the text.

We made the requested change to the text.

We made the requested change to the text.

We made the requested change to the text.

We made the requested change to the text.

We made the requested change to the text.

We made the requested change to the text.

To address such issues: only studies reporting on seniors should be studied, only studies reporting quantitative outcomes (morbidity, mortality. health risk changes, medical utilization) with comparison groups can be analyzed.  Here, only 16 studies were eligible on the age criterion; workplace programs cannot be generalized to seniors.  Only a few of these reported outcomes, and those outcomes are not reported in meta-analytic fashion, they seem to be only checkmarks that the program collected the data!

Thus, by studying seniors for whom outcomes were measured in quantitative detail you could approach the charge with far fewer studies to evaluate in far greater depth.  Here, clearly abstracts were used instead of detailed study review.  No quantitative data are presented. It's all checklists. Many good studies are not included orl were excluded. Some, easily accessed, were said to be unfindable.

Hypotheses for HRA characteristics likely to improve health include at the top: (1) theory-based, as with self-efficacy or readiness to change; (2)"tailored", with computer triage giving every subject a specific feedback on initial hra and a progress report, in detail, on subsequent hra's, (3) brief, to increase completion and to focus on major health risks, not marginal ones, to decrease questionnaire burden. (4) Positive, self-efficacy prose is more effective that scary threats. None of these appear to have been examined.

The authors admit failure in their discussions of key items 2 and 3 on page es-5.

An irony here is that this charge was previously given by CMS to RAND, "Evidence Report and Evidence-based Recommendations: Health Risk Appraisals and Medicare, contract # 500-98-0281; (1998-2003).  This study, while far from perfect, was infinitely better than this one, and found generally positive results.  As a result, CMS is funding the Senior Risk Reduction Study, the most rigorous attempt yet to answer these questions with a randomized design and adequate numbers of randomly chosen persons aged 67-74.  Did no one associated with this study know of this work?

Additionally there are thoughtful reviews of this subject by Chapman, Pelletier, Aldana, Goetzel, and others, critical but generally positive.  They describe theory and practice as well as citing the interventions judged most likely to succeed.  Did nobody think to synthesize prior work?

I looked up about 20 articles on seniors in the report that I knew well and felt to be among the best. Generally, they were not found, were excluded on some technicality of a check list, weren't read carefully, or showed reviewer flaws such as not considering claims data endpoints blinded and objective.

I'm out of time and energy so will stop here.  This is a sad case in an important area.  Sloppy, sloppy, sloppy.

To address key question #3, we included studies reporting 'health outcomes'.  Most health outcomes (in HRA studies of seniors and non-seniors alike) were intermediate markers such as blood pressure.  All of these health outcomes were "quantitative".  We discussed the limitations of generalizing workplace studies to seniors.  We did not conduct a meta-analysis because the study data were far too heterogeneous to combine statistically, and we pointed this out in the methods chapter.

The Evidence Table in Appendix D reports specific quantitative outcomes.

The scope of the TA (key questions 1 and 2) encompassed all groups, seniors and non-seniors.  We included studies reporting numerical (i.e., quantitative) outcomes, which are shown in the Evidence Table in Appendix D.

We employed standard systematic review methods to screen articles for inclusion in the TA: the first two screening levels involved reading titles and abstracts; the third level involved reading the full article text (see Chapter 2: Methods for further explanation).

In the absence of a list of specific articles from the reviewer, we are unable to assess whether certain excluded articles should have been included in the TA.

We excluded 'readiness-to-change' outcomes because they are not health outcomes.  HRAs had to involve individualized feedback to be included in the TA.  We were not asked to look at length or 'positivity' of HRAs.

We state for key question #2 that the evidence was essentially too heterogeneous for us to uncover any patterns that would indicate whether specific characteristics of HRAs were associated with better health outcomes.  For key question #3, we conclude that the results of studies in the 'under age 65 years' group cannot be generalized to the '65 years and over' group.

We are aware of the RAND study and patterned our definition of HRAs after the definition in the RAND study.

The TA is a synthesis of prior work, provided the prior work addressed any of the three key questions and met our inclusion/exclusion criteria.

In the absence of references for these specific articles, we cannot address the reviewer's comment.

In the absence of specific details, we cannot address the reviewer's concern.

In our continuing efforts to be a market leader and monitor emerging trends, we appreciate the opportunity to provide comments on the 'Health Risk Appraisal Technology Appraisal Report' draft dated January 19, 2011. StayWell understands this is a draft document and the goals of this report were to describe key HRA features, the features associated with a successful HRA and the applicability HRA to the Medicare population as a whole. StayWell appreciates this opportunity to provide comments and looks forward to providing further input if opportunities arise.

We believe this report will have the greatest value to policy makers if the relevant literature is viewed within the context of the typical role of HRA and the fact that past research has been conducted with limited financial and technical support and often in environments where optimal experimental controls to assure internal validity are not feasible. While the evidence based on this body of research is inherently imperfect, we believe ample evidence is available to guide policy makers and generally supports the use of HRA as a core component of a comprehensive wellness program.

Our comments focus on distinguishing the impact of HRA as a stand-alone tool versus the impact of health risk assessment within the context of a comprehensive wellness program. We believe the evidence, though far from ideal, is substantial enough to recommend against implementing a stand-alone HRA tool and to recommend for implementing a Medicare HRA tool within the context of a broader wellness strategy focused on assisting Medicare recipients in making changes in their daily lifestyle and health practices identified by the HRA.

The following comments draw heavily on a review of the literature on the impact of worksite-based HRA on health-related outcomes, which was completed by StayWell expert staff at the invitation of the Centers for Disease Control and Prevention and published in the American Journal of Health Promotion(1).  Although a considerable number of studies on the impact of health risk assessment within the context of comprehensive wellness programs have been published since that time (2,3) the findings and conclusions we drew in 1996 remain largely unchanged.

Taken as a whole, the studies reported in the literature reviews by Aldana (2) and Baicker et al.(3) provide imperfect but relatively consistent evidence supporting the effectiveness of HRA in the worksite setting when used as a component of a more comprehensive wellness program.  This research also generally provides evidence that 'more is better' when it comes to providing support for individuals attempting to change unhealthy behaviors, but it is not clear whether this refers to the number or types of intervention strategies or to the number of personal contacts with program staff or outreach tools. 

Based on published literature, there is little evidence that the use of stand-alone HRA is sufficient to produce long-term changes in health-related behaviors or health-risk status.  While several of the studies report positive changes in seat belt use and self-reported physical activity associated with HRA participation, as well as positive changes in other health-related outcomes (e.g., systolic blood pressure, body mass index), evidence supporting causal inference is lacking due to the many threats to internal validity in these studies. Probably the best conclusion at this time is that evidence for the impact of HRA per se on health-related behaviors is weak.  Few studies have been completed, and almost none have been well controlled. It is important to recognize, however, that behavior change is not the typical objective of HRA within the context of a comprehensive wellness process.  Rather, HRA objectives focus on health-related outcomes in the early 'prebehavioral' aspects of the change process, such as increased awareness of risks and commitment to change.  This suggests that research should focus on assessing the impact of HRA on movement through the initial steps in the change process.(4,5)   It is unfortunate that most published studies of HRA effectiveness focus almost exclusively on behavioral outcomes and risk reduction, which only reflect movement through the later, more visible behavioral stages of change.(4,5)  As was also noted in previous reviews of HRA effectiveness,(6,7) if the objectives of HRA focus on 'prebehavioral' health outcomes, then evaluating the effects of HRA on health behavior outcomes is misguided.

Another limitation of the research to date is the general lack of relevance between published HRA research and the evolving nature and role of HRA in practice.  Rather than the science of HRA informing its definition and application, practitioners have drawn upon relevant research in related fields (8,4,5) and their own experience to push the definition and use of HRA well beyond its origins and far ahead of its scientific literature.  This limitation is a direct consequence of inadequate funding for basic HRA research. As Schoenbach and colleagues recognized in their review of HRA research,(6) ?most of the studies have been carried out with minimal funding, by persons outside the mainstream of scientific research, and without benefit of external review.? This observation from more than 20 years in the past could as well be made today. For HRA research to be credible or helpful to policy makers and to wellness practitioners, more relevant, better controlled studies focused on appropriate health-related measures are needed. Accordingly, if a Medicare HRA process including follow-up wellness interventions is implemented within the mandated annual wellness visit, we strongly recommend including a strong research component including pilot studies that systematically vary key elements of the program design to assess their relative effectiveness.

Future research should move beyond the general question of the impact of HRA on health-related outcomes, such as changes in participants' knowledge, motivation and behavior, to specific issues of HRA design and delivery approaches.  For example, it is important to know not only whether HRA is effective, but also how each characteristic of HRA influences its effectiveness.  What are the best risk-projection methods with respect to user comprehension and motivation?  Should there be more or less emphasis on mortality and morbidity?  Does presentation of historical data enhance HRA effectiveness?  What types of design format for the HRA participant report—text, charts, statistical information, etc.—are most effective?  How should HRA feedback be tailored to different target populations—varying in age, sex, education, ethnicity, culture, and socioeconomic status—to maximize effectiveness within each?  Many generally agreed-upon assumptions by practitioners regarding answers to these questions have evolved over time, but these assumptions have been informed almost solely by practice because rigorous research on these questions is virtually non-existent. Again, we recommend research be conducted to address these questions, particularly as they apply to the Medicare population for the purposes of this program.

Biometric screening has played an important historical role in the use of HRA in assessing health risks.  There is a trend toward simplified screening protocols, however, as HRA instruments have evolved from mortality-based to habit-based in nature and as screening recommendations have become more targeted.(9)  Screening is sometimes even eliminated from the assessment process if it is either not feasible or cost-prohibitive.  For these reasons, the role of screening in the effectiveness of the health risk assessment process should be systematically explored.  The reliability, validity, and effectiveness of HRA in the absence of screening must be better understood. When screening is included in the health risk assessment process, a better understanding is needed of how to integrate screening measurements into HRA feedback to maximize comprehension and effectiveness.(8) Again, relevant research is sorely needed and there may be some opportunities to test certain aspects of how best to integrate screening data in this program.

HRA has played an important role in worksite wellness for 30 years and will play an increasingly important role in the future as HRA technology increasingly integrates the emerging knowledge of behavior-change processes, HRA-generated data become an increasingly valuable vehicle for targeting and tailoring follow-up behavioral interventions, and organizations demand greater accountability for measuring the impact of wellness programs on health risks.  If HRA research is to support the future development of this core wellness technology, a more relevant and sophisticated HRA science base is essential. As the subject review makes clear, research with stronger internal validity that can still be generalized to the Medicare population and other populations is a priority need as HRA and wellness play a greater role in mainstream health care delivery.

References:

(1)Anderson DR, Staufacker MJ. The impact of worksite-based health risk appraisal on health-related outcomes: A review of the literature.  American Journal of Health Promotion 1996; 10:499-508.

(2)Aldana SG. Financial impact of health promotion programs: A comprehensive review of the literature. American Journal of Health Promotion 2001;15:296-320.

(3)Baicker K, Cutler D, Song Z. Workplace wellness programs can generate savings. Health Affairs 2010;29(2):1-10.

(4)Prochaska J, DiClemente C.  Stages of change in the modification of problem behaviors.  In:  Hersen M, Eisler R, Miller P, eds.  Progress in Behavior Modification (Volume 28, pp. 183-213).  New York:  Sycamore, 1992.

(5)Prochaska J, Norcross J, DeClemente C.  Changing for Good.  New York:  William Morrow, 1994.

(6)Schroenbach V, Wagner E, Beery W.  Health risk appraisal:  Review of evidence of effectiveness.  Health Services Research 1987; 22(4):553-580.

(7)Edington D, Yen L.  Reliability, validity, and effectiveness of health risk appraisals. In:  Peterson K, Hilles S, eds.  The Society of Prospective Medicine Handbook of Health Risk Appraisals, Second Edition.  Indianapolis:  Society of Prospective Medicine, 1994.

(8)Terry P.  The role of health risk appraisal in the workplace:  Assessment versus behavior change.  American Journal of Health Promotion 1987; 2(2):18-22,36.

(9)U.S. Preventive Services Task Force.  Guidelines for Clinical Preventive Services. http://www.ahrq.gov/clinic/prevenix.htm

Thank you for reviewing and commenting on the draft report.

Thank you for this comment.

Policymaking is beyond the mandate of the TA.

Comment noted.

We are unable to address this comment because it does not relate specifically to the TA.

We are unable to address this comment because it does not relate specifically to the TA.

We are unable to address this comment because it does not relate specifically to the TA.

We are unable to address this comment because it does not relate specifically to the TA.

We are unable to address this comment because it does not relate specifically to the TA.

We are unable to address this comment because it does not relate specifically to the TA.

As the industry leader in well-being improvement, we currently provide our health promotion, chronic care management, wellness and prevention services, both domestically and internationally, to approximately 40 million people on behalf of more than 1,000 employers and 100 health plans.  Our mission is to create a healthier world by delivering solutions that:

1. Keep healthy people healthy.
2. Reduce health-related risks.
3. Assure the provision of evidence-based care to those who are ill.

Healthways has provided health risk assessments (HRA) for more than 25 years. Our current instrument is the Healthways Well-Being Assessment? (WBA).  The WBA is based on the latest evidence-based behavioral science available, including the work of Dee Edington, PhD, University of Michigan, James Prochaska, PhD, University of Rhode Island, and Janice Prochaska PhD, Pro-Change Behavior Systems, Inc.; Ron Kessler, PhD, Health Management Research Center; and Gallup, the worldwide leader in research of human nature and behavior. The WBA is configurable for different populations, is certified by the National Committee for Quality Assurance (NCQA), and is currently being implemented by commercial health plans, private sector employers and the Federal Government's Office of Personnel Management, Department of Interior and the General Services Administration.

Despite the HTA authors' findings of relatively consistent improvement among short term health measures for HRA participants, we are not surprised to learn that the literature review did not lead to conclusive evidence supporting a specific feedback or recommendation protocol.  In our experience, developing and implementing interventions designed to modify behavior to achieve improved health is a complex, multi-component process.  In a 2008 JAMA article titled, 'The Science of Improvement,' Berwick discusses the limitations of traditional study designs in evaluating effectiveness of these types of interventions. We concur with Berwick's assessment that the effectiveness of interventions geared toward behavior change is sensitive to many influences, environmental and otherwise.

Similarly, we have found that taking into account a more comprehensive view of the environmental and other influences that affect individuals substantially improves the effectiveness of interventions intended to drive behavior change. We have invested heavily in the study and measure of the factors that drive individual behavior and the effectiveness of interventions intended to change it.

In partnership with Gallup, Healthways created the Gallup-Healthways Well-Being Index (WBI).  With over 1 million surveys of individual well-being completed, the WBI is a rich database of benchmark information on the well-being of the people across the United States.  Using the WBI, we created the aforementioned Healthways Well-Being Assessment (WBA), a NCQA-certified, comprehensive tool that captures both 'traditional' HRA information and relevant information about an individual's home and/or other environments and their access to basic essentials needed for achieving and maintaining health and well-being.

Through our work with the WBA we have found that understanding the context of an individual's daily life, health and lifestyle behaviors that otherwise may seem irrational, may actually be rooted in specific circumstances which may not be related to physical or emotional health per se. By arraying personal information over a broad range of domains that incorporate many environmental factors, the WBA enables a comprehensive view of an individual's well-being and provides important context for developing interventions to improve both health and overall well-being.

Based on our success in developing interventions that drive behavior change to improve well- being, we strongly recommend that the CDC and CMS require the use of HRA tools that are sufficiently robust to improve the likelihood of their effective use.  For example, we recommend that HRA tools meet the standards of a recognized accreditation organization such as NCQA, URAC or JCAHO, and have established benchmarks for measuring well-being.  Notwithstanding the HTA findings of inconclusive evidence, our own experience demonstrates that by setting an appropriately high bar in the forthcoming HRA guidance, CMS can help ensure that providers have access to high-quality tools that will help them, and CMS, succeed in achieving the triple aim of improving the patient experience, improving health and reducing cost. 

In summary, in our experience, HRAs can and do succeed in their primary function of collecting data against which actionable intervention plans can be developed.  In addition, well-designed HRAs that are integrated with a comprehensive and targeted intervention program are often successful in achieving their broader goal of improving health outcomes and overall well-being. We would welcome the opportunity to discuss this issue further with CMS and to share our research on the evidence of effectiveness with the CDC as it moves forward on the development of guidance on HRAs.

Thank you for this information.

Thank you for the comment.

Thank you for this information.

Thank you for this information.

Thank you for this information.

This TA does not discuss coverage policy-related issues. Please direct coverage-related comments or questions to the Center for Medicare and Medicaid Services (CMS).

Please contact CMS and CDC directly.

2. The definition of HRA as they posit in the article I might agree with; however, many would argue that "the HRA" is the sampling instrument and "the report" and "information given to inform and change behavior" generated by "the HRA" is a fundamentally different question for consideration.

3. There are many HRAs tools in existence and most do not have strong published information about the multi-variate correlation between the quality and reliability of the "sampling instrument" and "the report" and "the health outcome".

a. I think this absence of scientific information creates a vacuum that could lead the research team in the wrong direction if this paper is the only source of awareness about HRAs;
b. There are many innovative sampling tools and methods with valuable health information reports attached that are not "in the published science" at the "level of evidence quality" that is preferred by these reviewers.  HOWEVER, if they make fundamental policy decisions based on what is in this selected body of literature they may be missing some great opportunities to guide the market and researchers in what needs to be done to further research the value of HRA instruments and how HRAs could be best used. In my editorial opinion;  I would clearly argue there is absence of evidence to make sound decisions (especially based on what I know about the market and seeing how few studies actually passed "muster" for inclusion in this review). I hope the review results in a decision that more research is needed; I think that is about all that can be concluded based on the research that was evaluated.

4. I didn't see information about internal validity assessment (other than one small reference in one study) related to the sampling instruments they assessed.  I think this is a concern; if the studies they selected did not use valid instruments and they were only looking for the health outcome of the "intervention" post sampling; there is a fundamental weakness in the assessment. There was some modest assessment of Inter-rater reliability.

The reviewer did not specify what he thought was "amiss", so we could not address any concerns he might have with the TA.

Thank you for this observation.

Thank you for this observation.

Thank you for this observation.

Thank you for this observation.

We present reliability and validity data in our response to key question 1a (Chapter 3: Results).

The HRA studies reviewed looked at outcomes across all different risk factors, including smoking cessation, weight loss, physical activity, cardiovascular health, and so on.  To compare outcomes across these categories is comparing apples to oranges. On the one hand, we may not expect to be equally successful in all these areas, for reasons such as variation in the effectiveness of the therapy options to address different conditions.  On the other hand, the potential overall health effect of the risk factors observed may be different across these risk factors as well.

We included Table 5 in the TA to delineate outcome types (e.g., obesity, cardiovascular health) by article and give readers a sense of the variety of outcomes used in the literature.  Statistically significant outcomes are identified in the table, on a per article basis, to provide the type of individual-level detail suggested by the comment.

Readers must consider article quality in their appraisal of the evidence.

Moreover, they point out that the vast majority of the participants in these studies were not senior citizens. Since some of the critical risk factors don't emerge until later in the life span (e.g., cognitive impairment), it's necessary to ask different questions of the elderly.  So how can we generalize from the results of younger cohorts to a Medicare population? 

The HRA studies reviewed were published over the last 30 years.  Technology has gone through rapid changes during that time.  While the authors indicated that many of the instruments studied provide 'personalized' feedback, that doesn't mean they were tailored, as we understand that term.  This would especially apply to instruments that were not computer based.  Research has shown again and again that the degree/depth of tailoring determines the effectiveness of health-related messaging, as demonstrated by a variety of measures (e.g., eye movements, fMRI responses, behavior change, and so on).  We would argue that HRAs without tailored feedback are going to be inherently less effective than their more tailored counterparts.

We agree with comment about generalizability (see key question #3 in Results and Discussion).

Thank you for this observation.

This study served more as a review of various research studies which used health questionnaire (not necessary a HRA) as a measurement tool—describing key features of those selected research studies and examining which features were associated with successful research design or implementation.  In this way, the HRA itself is no longer the subject of major study; thus, the authors failed to answer the research questions that they originally proposed.

There are three major weaknesses in this report as to the research design and implementation:

1) a lack of understanding, learning, description, introduction and review of the origin, development and current status of the HRA technology—HRA features were not thoroughly and rigorously studied throughout this proposal.

2) no efforts given for the direct assessment of the HRA technology—No surveys and no research were conducted to directly evaluate the HRA technology via HRA providers or HRA users; instead, the study used literature searches to evaluate how various researchers have used the health assessment questionnaire as a tool.

Theoretically, a technology assessment should focus on the components of the HRAs: the quality of its questionnaire and the health risk appraisal algorithm, the totality and the usefulness of the profile report, the value of the follow-up strategy and the possible implementation and outcomes.   Instead the current report expanded the assessment much beyond the HRA technology.

3) use of "better health outcome" as the only outcome measure for the success of the HRA technology. The HRA has been used as a tool for health education and awareness.  A person who takes an HRA and receives a profile or even several follow-ups would not necessarily receive a "better health outcome."  This reviewer does not believe there is a causal relationship between simply taking an HRA and a "better health outcome."  In worksite health promotion and disease prevention, the HRA is positioned as a gateway program to help individuals improve health awareness, attitude, and belief.  "Better health outcome" is often refers to health behaviors and health status.  Programs for behavioral change—such as smoking cessation, weight control, and increase in physical activity—are not part of the HRA technology, but parts of the comprehensive health promotion and disease prevention program that may complement the HRA.  In this proposal, the authors were asked and attempted to evaluate the HRA technology; however, they actually selected and evaluated comprehensive health programs that used some HRA technology as a gateway program by using "better health outcome" as a criterion in their literature search.  The authors expanded the definition of "HRA technology" too broadly. This particular limitation is related to the first weakness that the authors showed in the current report—that is, a lack of basic knowledge about the HRA technology, including both the HRA science and its applications.

Overall, the study design is poor and does not fulfill the purposes of this RFP.  The authors' definition of the HRA was incomplete, and their report assessed the research conducted or published on using health assessments (not all of which could rightfully be classified as an HRA), which deviated from the stated purpose of the study (to assess the HRA technology).  Because the study's inclusion/exclusion criteria lack a basic understanding of the applications of HRA technology in the field of health promotion and disease prevention, it provides very limited information to fulfill one of the major purposes of the grant—providing useful references for the implementation of the Affordable Care Act to offer an annual HRA to the Medicare population. 

Our definition is consistent with the Centres for Disease Control and Prevention (CDC) definition of an HRA. CDC defines HRA as "...an assessment tool used to evaluate an individual's health. An HRA could include a health survey or questionnaire (see Employee Health Survey); physical examination, or laboratory tests resulting in a profile of individual health risks often with accompanying advice or strategies to reduce the risks." (Centers for Disease Control and Prevention. Workplace Health Promotion—Glossary Terms. Atlanta: Centres for Disease Control and Prevention. Available at: http://www.cdc.gov/workplacehealthpromotion/glossary/#H. Accessed on: March 9, 2011.) We added the CDC definition to the text of the TA.

The TA was designed to answer the key questions using the aforementioned definition of HRA.

Please refer to our previous comment above.

Our mandate was not to provide an in-depth review of the origins of HRAs.

The reviewer provides insufficient detail regarding the HRA features that he claims were not "thoroughly and vigorously studied" in the TA.  Without further detail, we cannot address the comment

AHRQ-commissioned technology assessments are systematic reviews and do not involve primary data collection.

The TA responded to the key questions in the task order from AHRQ.  We used an expansive definition of HRA that was consistent with previous reviews in the area (RAND Santa Monica CAL. Evidence Report and Evidence-based Recommendations: Health Risk Appraisals and Medicare. Contract no.: 500-98-0281. Baltimore, MD: Centers for Medicare and Medicaid Services, 2003).

Key question # 2 directed us to examine "better health outcomes".  AHRQ, CMS, and CDC approved our use of the HRA definition quoted in a previous comment above.

See previous comment.

Therefore, this reviewer believes the HRA technology was well-defined in the field of health promotion and disease prevention until the early 2000s, when the SPM disbanded.  Between the late 1970s and the late1980s, the CDC and university research groups became the major forces behind HRA promotion and research.  The SPM was the main academic agent contributing significantly to the development of HRA technology. Since the 1990s, the rapid development of computer technologies and the commercialization of the HRA by the private sectors expanded HRA technology and widened its functions, academic areas, and health focuses.  As reviewed by the SPM in 1999, there were at that time 46 HRA instruments, 28 Quality of Life/Health Status Assessments and 32 Lifestyle-Specific Health Assessments, where the later two categories of assessment were expanded from the HRA but used similar logic as the HRA.  Thus, this reviewer agrees with authors' definition of the HRA that the "HRA are techniques or processes of gathering information to develop health profiles, using the profiles to estimate further risks of adverse health outcomes, and providing persons with feedback on means of reducing their health risks."  However, this reviewer did not see the authors fully integrate the complete definition above into the study inclusion criteria, especially those functions "using the profiles to estimate further risks of adverse health outcomes".

The section on reviews of the history of HRA was not adequate.  This shortcome consequently guided this report into the direction of how the health assessment was used as a tool in research studies and away from the stated direction of the features of the HRA technology.

The authors also reviewed the importance of implementing an HRA program among the elderly.  This is the important background information.

Finally, they formulated three important key research questions as follows:

1. What are the characteristics of the provision of HRAs?
2. What characteristics of HRAs are associated with better health outcomes?
3. What is the generalizability of the data in Questions1 and 2 to the Medicare population or subpopulation?  

The first key question was on target.

Another question, concerning the utilization (and repeated utilization) rates and the numbers of individuals who have completed the HRA, should be the direct outcome measures of any technology assessment but were omitted by the authors.  Instead the authors put "better health outcomes" as the successful HRA technology outcome measure, which caused the error in the research design.

HRA technology is the only one of the many programs to associate with "better health outcomes" either in a research or program setting.  As a result, the authors changed the HRA technology assessment to an assessment of a research study or a program implementation where an HRA might be included as a measurement tool or as a program component to improve personal health.

We added a reference to the Hyners et al. (eds.) handbook in the introduction to the TA.

Thank you for the HRA overview.

Key question #2 required that we focus on health outcomes.  Estimating future risk is not a health outcome per se and was therefore excluded from the TA.

Please refer to our previous comments above.

Noted.

CMS formulated the questions, not the authors.

Noted.

The TA addressed CMS's questions, which did not pertain to utilization rates, but concerned health outcomes.

Please refer to our previous comments above.

Unfortunately, the literature review (using an inclusion criterion of RCT or cohort and case studies) would not be sufficient to adequately answer the first Key Question, especially with all the sub-questions listed beneath it.  Thus, the authors' search criteria on research articles focused primarily on the 2^nd Key Research Question for "better health outcome," This focus resulted in the following shortcomings: 1) some valuable HRA-related studies were excluded due to "no health outcome measured"; 2) some studies that did not include a true HRA program component (just a health assessment questionnaire) were included; and  3) some studies had "better health outcomes" that did not directly result from the HRA program.

Using literature review to respond the 3^rd research question was a questionable strategy.  If the authors spent enough time researching the HRA history, developments, and its current status, this question would have been proposed differently.  Originally, the HRA technology was developed from and used among healthy adult populations for health education.  Both the CDC and Carter Center's HRA stated clearly that HRA instruments were suitable for people aged 19 to 64 and those free of diseases.  Because of this qualification, the authors would have difficulty finding existing publications that used an HRA only within the Medicare population.  In fact, there have been studies on retirees using an HRA.

In addition to literature reviews using publication databases, this reviewer believes three research approaches are needed: 1) extension of the literature search to those HRA users who were selected by the authors with a telephone interview, in order to collect detailed information on the features of HRAs they used;  2) a survey of HRA providers for the features of their HRA instruments and their HRA usage status; and, 3) a search through the Internet to focus on HRA providers and their HRA features.

The authors used selected keywords to search titles and abstracts through publication databases and also "hand searched" the American Journal of Health Promotion.  However, this reviewer believes that running a literature search based on author—rather than on keywords—would be a more focused strategy.   In general, most HRA technologies have been developed by certain research groups, i.e. Dee Edington and The University of Michigan Health Management Research Center; Nicolas Pronk and his colleagues; and so on.  All of these groups have extensive research publications on the use of the HRA as a tool for worksite health promotion programs.  However, few studies from these researchers were selected in this current proposal.

The reason that these publications that used an HRA as a measurement or program tool were not selected for the review might be due to the authors' inclusion/exclusion criteria.

The authors only included the studies of randomized controlled trials (RTC) or observational studies with comparison groups (e.g. cohort, case control).  However, as the authors acknowledged, most HRAs have been used at worksites among voluntary, convenient samples.  Since the authors used this research design/sample selection criterion, they missed a majority of the research publications related to the HRA, especially those publications with a focus on the HRA features listed by the authors as sub-questions of the Research Question 1.  For example, only three of the 100+ journal articles using HRA data published by Edington and his associates were selected by the authors in the current study.

Using these exclusion criteria made the  main purpose of this study more difficult to achieve — that is, the discussion of the applicability of HRAs to the Medicare population, because the Affordable Care Act authorizes Medicare to cover an annual HRA.  This Act would result in HRA distributions to voluntary, convenient samples among the Medicare population, a similar distribution to most of the HRA-related research studies at worksites.

The authors were not only biased in their exclusion criteria but also in the inclusion criteria.  As the authors acknowledged in their introduction, one of the key characteristics of the Health Risk Appraisal was its appraisal function (or, to use the authors' words, "using the profiles to estimate further risks of adverse health outcomes").  However, this inclusion criterion was not used by the authors.

The authors stated that "our definition of an HRA contained three components: participants provided self-reported information to identify individual risk factors for diseases; participants received individualized health-related feedback based on information they provided; and the information was used to give participants at least one recommendation or intervention to promote health, sustain function, or prevent disease."  Thus, by changing the definition, the authors switched the research target from an HRA technology with a program component to any health assessment  without an appraisal process during the questionnaire to profile procedure.  Of the 115 studies the authors selected to study, some of those studies inadvertently used a non-HRA assessment as a tool due to this excessively-wide inclusion criterion.

The reviewer's comment is welded in his definition of an HRA, which is different from the definition in the TA.  Please refer to our previous comments on this issue above.

We included studies with comparison groups to evaluate health outcomes in HRAs with the three components of our definition (data collection + feedback + recommendation) versus HRAs with less than these three components (or versus some other non-HRA program).

We addressed the key question as proposed by CMS.

We found few publications in the Medicare population; all of these publications related to persons aged 65 years or over.

Given the interest in using HRAs in the Medicare population, a search of the literature to find information in this group, as well as an assessment of whether studies done in persons aged 19-64 years can be generalized to the Medicare population, are valid lines of inquiry.

Collection of primary data (points 1 and 2) is not a component of a systematic review and is therefore beyond the scope of the TA.  Point 3 would be relevant if we were asked to develop a compendium of HRA risk factor assessment questionnaires, which was not part of our mandate.

Search strategies based on well-known authors or research groups can introduce bias into systematic reviews, e.g., only a handful of relevant studies, focused on limited interventions, with a narrow range of conclusions, would likely be included in such reviews.  Objective search strategies use keywords linked to the specific research questions guiding the review.  A professional medical librarian, with expertise in systematic reviews, developed our search strategy.

We extracted data on articles that passed three levels of screening based on our inclusion/exclusion criteria.  If studies from the researchers cited in the reviewer's comment (e.g., Edington, Pronk) were excluded from the TA, then these studies did not meet our specified criteria, which were geared to answering our three key questions.

We agree.

We were tasked with assessing whether certain characteristics of HRAs produced better health outcomes.  A basic tenet of evidence-based medicine/practice is to evaluate health outcomes in comparative studies.  Consequently, we included studies with comparison groups and excluded case series (no comparison groups).

Key question # 2 directed us to examine "better health outcomes".  Therefore, we followed standard systematic review methodology and designed our inclusion/exclusion criteria to meet the demand of this question.  While HRAs can be used to estimate future adverse health risks, this specific issue was not within the scope of the TA.

Please refer to our previous comments above.

Of the 538 citations promoted to full text screening, 423 were excluded and 115 proceeded to full data extraction and quality assessment.  One of this reviewer's publications was selected as the extracted study.  However, the review does not appear to be correct and the subsequent conclusion was completely wrong.  As shown in Table 3 of the Results, in the "HRA Instruments Used in the Extracted Studies" column, the "Instrument Reference for this study (reference 26)" was listed as "No details on HRA or GM program—This HRA is a product of the StayWell Company, which has won the C. Everett Koop Award in the past.  See reference for Gold, 2000 in this chart" and at the "Tools/Instruments" column was listed "HRA in tandem with General Motors LifeSteps program."   Actually, the HRA used in this study was not a product of the StayWell Company, and the GM LifeSteps program was described in detail in this article.  Furthermore, at the time of study, General Motors had not even applied for the Koop Award and the LifeSteps program have never used StayWell Company's HRA until today.  Based on these misleading statements from the authors, this reviewer questions the quality of the authors' work, although the authors claimed they had "reviewed the extracted data to confirm the accuracy of the work." 

However, the poor quality of this particular review and its erroneous conclusion were not the primary problems with this proposal.

Rather, the main problem was the authors' focus on studies that might not be involved in HRA technology.

The authors should focus their study on HRA technology, and this reviewer sees no value in Table 1 and Table 2 of the results section, where the authors tried to assign a quality score for the studies published as traditional academic research.  The assessment of the quality of the extracted publications had very little relationship to the assessment of HRA technology and the HRA features, such as its questionnaire, algorithm, profile, and follow-up capacity.

Again, relying on literature searches as a research tool may not achieve the main research objectives indicated in the study.  Five key sub-questions of Research Question 1 on the characteristics and the provision of HRAs could not be adequately answered due to insufficient data, mainly because the researchers under review did not present those HRA characteristics in their publication (given that these publications used the HRA chiefly  as a personal health measurement tool or part of overall health promotion programs).

The Table 3 in the study was necessary and informative.  However, the structure of the table was dictated by the different studies, not by the HRA provider or the HRA itself.  For example: If this table were categorized by the HRA (i.e. HRA for research purpose; HRA from commercial use, such as HealthTrac HRA; StayWell HRA; WebMD HRA, etc), it would be more focused and more useful.

In Table 4, a drop-out rate was presented as a major outcome measure on success of the HRA without any indication of a participation rate.  This drop-out rate was confusing and not clearly defined.  If an HRA participant completed an HRA questionnaire and then received a profile, at what point could a drop-out rate be measured? If dropping out occurs before referral to the follow-up programs, another measure—perhaps follow-up rate of the HRA participants—had to be presented before this dropout rate to make the measurement meaningful.  However, the dropout rate of the HRA follow-up program or a comprehensive health promotion program with a health assessment component was not really the outcome of HRA and could not lead to a "better health outcome." 

We agree with the reviewer that some detail of the LifeSteps Program is contained in the article and we deleted the phrase "No details on HRA or GM program" from the table.

The LifeSteps Program won the Koop Award in 2004 and we mentioned this in the table as having won the award "in the past".  To clarify, we changed the table cell entry to read "The LifeSteps Program won a C. Everett Koop National Health Award in 2004".

When we looked into the LifeSteps Program, we visited the http://www.lifesteps.com webpage and were directed to a Web site copyrighted by The StayWell Company, LLC.  This prompted us to write "This HRA is a product of the StayWell Company."  Of course, our definition of HRA includes the full LifeSteps Program, not just a risk-factor questionnaire.  Consequently, our comment in the table appears consistent with the current state of affairs; i.e., LifeSteps is a product of StayWell.  To clarify this issue, we added the following underlined word to the text in the table cell: "This HRA is currently a product of the StayWell Company."

We deleted the reference to Gold in the table cell.

The reviewer is unclear about what he considers the "erroneous conclusion".

Please refer to our previous comments about differences in defining HRAs.

Quality assessment is an integral part of summarizing a body of evidence in a systematic review and forms a basic tenet of systematic review methodology.

Please refer to our previous comments.

The standard method of presenting extracted data in systematic reviews is by study.  Since we did not limit our definition of HRAs to data collection instruments alone, we would not have been able to communicate any practical information using a table structure patterned after the reviewer's suggestion.

The drop-out rate is the number of participants who completed a study (data available at the last follow-up time point) divided by the number of participants who began the study at baseline.  We added this definition to the text.

Table 5 lists six outcome domains used in the extracted HRA studies.  The row for each study indicates whether the specific outcome was included in the study in question.  The asterisk indicates whether the outcome was statistically significant at the 5% level in the study in question.  Table 5 is introduced in the text as follows: "Many articles reported benefits for intervention groups in domains such as general health, lowered cholesterol, reductions in blood pressure, reduced fat intake, or improved physical activity".  Table 5 does not pertain to age and gender.

Comments on the authors' discussions on the five sub-questions regarding Key Research Question 1 are as following:

a. Specific HRAs.
The authors stated, "most articles were concerned with general or cardiovascular health assessment... Articles using questionnaires only often asked participants to self-report previous diagnoses or risk factor for disease (e.g. smoking).  HRAs designed for specific objectives such as improving diet rather than improving general health often utilized questionnaires to elicit information on items like participants' food intake."  This reviewer does not agree with the authors on their inclusion criteria. Those "specific" HRAs classified by the authors were not true HRs; they were a health assessment questionnaire and should be excluded from this study.

b.      HRA administration methods.
The authors were able to classify of 115 articles based place of HRA distribution—at the workplace (53 articles) vs. the non-workplace.  This observation was the only major finding they could report for this sub-question.  The authors stated that "no specific type of HRA was associated with one place of administration versus another."  This inadequate finding arose primarily from a poor research strategy (described earlier) and a lack of knowledge and understanding of HRA technology. 

c.  The training of personnel who administered HRAs.
The authors found "21 different descriptions in 78 articles" from the 115 articles they selected. They stated, "we relied on authors' descriptions of training… although most authors did not detail the specific training regiments required of staff."  If the authors focused on HRA users or providers as the major data resources for this report, this problem would be avoided. 

d.  The methods and frequencies of follow-up for HRA.
The authors were able to provide some non-systemic answers regarding this question.  They stated: "In workplace HRAs, meeting were a favorite means of contact...  Some community-based HRAs also involved in-person meetings...  The increasing popularity of the Internet…however, HRAs administered entirely online might not reach groups at highest risk for chronic disease. Persons in these groups may be less likely to have Internet access... This issue was largely unstudied in the extracted articles, so more work is needed to assess the efficacy of online HRA delivery...  Future research should involve more follow-up. ...This prevented us from assessing the durability of HRA over time.  ...Further research is required to elicit the specific factors that influence participation in HRAs."  In this sub-question, the authors confused two different concepts: methods of HRA delivery and follow-up of the persons who completed an HRA.  The authors were unable to reach clear conclusions in either area and simply suggested further research on the topics.

e. The characteristics of the patient population who received HRAs.
This research sub-question, in principle, was not correct.  The HRA is a technology that was developed and has been used mainly for healthy populations, not patient populations.  The question asks who received HRAs and the authors' answers were limited to "typical participants," which is not an informative or useful answer. Conceivably, this question could be important for Research Question 3 and the Affordable Care Act, which authorizes Medicare to cover an annual HRA for the Medicare population and not just Medicare patients. Since a blanket distribution of an annual HRA to the Medicare population will be forthcoming, it will be vital to learn the demographic characteristics of the people aged 65 or over who received an HRA or who were more likely to respond an HRA. Such information about demographics and response characteristics among the elderly could be very instructive but was not addressed in this proposal.   

The authors concluded Key Research Question 2: "the evidence did not suggest a clear set of characteristics (of HRAs) that were associated with better health outcomes."  Their conclusion was correct and resulted from their inadequate determination of the research question.  In actuality, Research Question 2 should focus on the fundamental measurement of a technology or product such as the HRA on user numbers and rates.  The discussion determined that these "better health outcomes" were beyond the scope of the study and the data that the authors collected for this report.

Based on their inclusion/exclusion criteria, the authors "found 16 (of 115) articles included the members of the Medicare population" for Key Research Question 3—the generalizability of the data (HRA related) to the Medicare population or subpopulation.   However, the authors "cannot readily generalize results from HRA studies in persons aged less than 65 years to persons aged 65 or over."  However, there were at least two published studies from Edington's group that did not make the first two rounds of eliminations but focused on Medicare population; there is also another article that addresses a  retiree population that may be relevant to the current study. The three articles are listsed below:

• Musich, Shirley A., Aartee Phatak, Timothy McDonald, David Hirschland, Dee W. Edington.  Self-Reported Utilization of Preventive health Services Among Retired Employees 65 Years and Older. Journal of American Geriatrics Society 49(12):1665-1672, 2001.
• Ronald J. Ozminkowski, Ron Z. Goetzel, Feifei Wang, Teresa B. Gibson, David Shechter, Shirley Musich, Joel Bender, Dee W. Edington. The Savings Gained From Participation in Health Promotion Programs For Medicare Beneficiaries. Journal of Occupational and Environmental Medicine 48(11):1125-1132, 2006.
• Yen, Louis, Alyssa B. Schultz, Timothy McDonald, Laura Champagne, Dee W. Edington.  Participation in Employer-Sponsored Wellness Programs Before and After Retirement. American Journal of Health Behavior 30(1):27-38, 2006.

One purpose of this TA is to provide an evidence-based perspective on the impact of HRAs on health outcomes.  Assessment of evidence requires consideration of quality (Agency for Healthcare Research and Quality. Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews, Version 1.0 [Draft posted Oct. 2007]. Rockville, MD. Available at: http://effectivehealthcare.ahrq.gov/repFiles/2007_10DraftMethodsGuide.pdf Plugin Software Help)

The conclusions are based on the articles included in the TA.

HRAs as per our definition were not associated with one specific place of administration versus another place of administration

The key question asked us to delineate the training of personnel who delivered HRAs.  The many descriptions are not a "problem", merely a statement of the contents of the extracted studies.

We explain what we mean by "follow-up" in Chapter 4: Discussion, under the section for Question 1d.  'Follow-up' is the number of contacts between the HRA program and participants over time (relevant for us since we defined HRAs as being more than the collection of risk factors).  The reviewer's comment arises since he defines HRAs differently than us.

CMS determined the wording of the question.  The important word in the question is "populations", which included any individual enrolled in an extracted study.  This particular question was not restricted to the Medicare population.

Key question #2 specifically mandated us to obtain evidence on health outcomes, not to collect data on user numbers or counts.

This article does not have a comparison group (all participants received the same intervention).  The outcome measure is the percentage of persons who availed themselves of preventive services or health screenings within clinically recommended timeframes (not a health outcome).  This article does not meet the criteria for inclusion in the TA.

The outcome in this article is cost savings.  Costs are not a health outcome as defined in the TA, so this article does not meet the criteria for inclusion in the TA.

The outcome in this article is participation rate in an HRA, which is not a health outcome, so this article does not meet the criteria for inclusion in the TA.

The organization of Table 4 (Methods and Frequencies of Follow-up) was inadequate but its contents were necessary.  The table combined HRA questionnaire distribution, HRA delivery method, HRA profile contents, recruiting for HRA participation, and HRA follow-ups.  This table should be organized according to HRA features and broken into several tables in a conceptually similar manner as that of the HRA process procedure: i.e. HRA questionnaire distribution and delivery; HRA profile contents and HRA feedback delivery; and, HRA follow-up programs. 

The contents of Table 5 attempted to document "better health outcome" evidences based on the extracted articles selected by the authors.  The authors called Table 5 an "Evidence Table."  However, these "evidences" documented the intervention programs that were associated with the HRA programs and were essentially independent of the HRA technology used in the HRA programs.

In addition, the categories used in this table were not exclusive of one another.  Table 5 provided the outcomes resulted from health promotion programs or activities presented in those selected articles, but these were the evidences of the comprehensive programs with a health assessment component.  These evidences, if any, were not the direct results of HRA technology but were, rather, the result of an broad health management program that included an HRA as part of its armamentarium.  In other words, the authors incorrectly attributed those "evidence" to HRA technology.  

In conclusion, three of the five tables did not provide valid information for HRA technology assessment and the other two tables were useful but poorly organized.  In addition, this reviewer questions the content validity of these tables based on the authors' inaccurate descriptions on an article written by the reviewer (Reference 26).

The standard method of presenting extracted data in systematic reviews is by study.

We defined an HRA as being inclusive of these programs.

Please see our previous comment, as well as other comments related to our definition of an HRA.

The reviewer's comments on the tables stem from his divergent view of the definition of an HRA.  As we mentioned earlier, a consensus definition of an HRA is lacking and the tables are consistent with the definition of an HRA that was used in this report.

Regarding reference 26, the information in the table was fundamentally correct: LifeSteps did win the 'Koop Award' and it is currently attached to StayWell.

• Hyners GC, Peterson KW, Travis JW, Dewey JE, Foerster JJ, Framer EM, Eds. The Society of Prospective Medicine Handbook of Health Assessment Tools.  Indianapolis, IN:  The Society of Prospective Medicine.  1999:135-144.

Figures 2 & 3 did not provided valid information for HRA technology assessment, the same as for Tables 1 and 2.

Figure 4 supposedly summarized the objectives of the HRA.  The intention was good but the execution faltered.   Theoretically, if the objective of an HRA is based on general health issues, it should also include objectives for reducing cardiovascular risks, improvement of smoking, weight, and physical activity behaviors, and the other objectives listed in the same axis in the figure.  This reviewer believe that only those articles with an objectives on general health (60) or cardiovascular diseases (49) could be defined as using an HRA, while the others used assessments based on single behaviors—smoking (10); weight (14); physical activity (31); or other (34). Strictly speaking, these single-focus assessments  should not be considered HRAs. 

Thus, out of the 115 articles that made the authors' final cut, there would be maximally 109 articles that could be considered to contain an HRA (with no overlap on 60 general health and 49 cardiovascular disease articles).  If Figure 4 presents the same data as Table 5, there may be another 10-20 or more articles that would be excluded—since the authors classified some HRAs with the objectives on both general health and cardiovascular disease. 

Figure 5 was a good for the type of HRAs. 

Figure 6 was a good for the method of HRA administration.

Figure 7 had a good concept but it was not germane to the study, since it showed the number of articles selected that reported training on HRA administration.  The figure could not generate an outcome to designate the quality of an HRA and also had little to do with HRA technology. 

Figure 8 suffered from the same problems as Table 4, in that it mixed two different concepts together: HRA profile delivery method and follow-up on HRA participants.

Figure 9 contained useful information but again, it counted HRA profile delivery as a 1st time follow-up. 

Figures 10-11 showed the scatter plot between "Numbers of Methods of Follow-up/Feedback" and "Frequency of Follow-up/Feedback." Both figures should be deleted from the report since the data "failed to suggest possible linkages between tenacity of follow up and dropout rates."  With the mixed concept in HRA follow-up and HRA profile delivery and the loosely defined "drop out" rate from the extracted articles, there were no way to find any correlations between these confusing measurements created by the authors.

Figures 12-13 attempted to answered Research Question 1-e on the mean age and gender distributions of the people who received an HRA.  However, the authors presented both average values based the HRA participants' information from the extracted articles, which is probably the only way they could derive the numbers from the publication. 

In sum, six of 13 figures (figures 2, 3, 10-13) provided invalid information to answer the research questions on HRA technology assessment.  In the reviewer's opinion, further clarifications are needed for the other 5 of the 7 figures.

Please see previous comments regarding different definitions of HRAs.

The stated objectives of several studies involved single behaviors.  Based on our definition of an HRA, single-behavior studies could be included in the TA, provided they met our other inclusion/exclusion criteria.  Please see previous comments regarding our definition of an HRA.

We reviewed the numbers in Figure 4 and Table 5; we updated the numbers in Figure 4 to match the numbers in Table 5.  Please see our previous comment regarding the number of articles defined as including an HRA.

Thank you.

Thank you.

We included this figure to summarize the data in Question 1c.  This figure is unrelated to quality and was not connected to quality in the text.  The figure itself was not meant to "generate an outcome".

Please see previous comments regarding different definitions of HRAs.

Please see previous comments regarding different definitions of HRAs.

Please see previous comments regarding different definitions of HRAs, as well as our clarification of the drop-out rate.

Information should not be deleted from systematic reviews because it fails to show correlations (linkages, associations etc.).  No correlation (linkage, association, etc.) is a relevant result.

Thank you for the comment.

Please see above comments on the figures

Appendix B contains the authors' screening forms for the research articles.  The authors used the JADAD scale and Newcastle-Ottawa Scale as screening criteria to select articles; as mentioned previously, those exclusion criteria or score methods set by the authors might exclude the majority of publications focused on the HRA. 

In addition, the Screening Questions for the HRA developed by the authors included significant numbers of the health promotion/wellness program studies without a true HRA program component.  The Question 2 reflected this problem:  Does this paper refer to health risk appraisal (sometimes called health risk assessment) OR focus on a health promotion/wellness program targeted to a specific, individually identifiable, population (such as employees at particular worksites)?  It was clear that the phrase with the "OR" drew those program studies without a true HRA program component into the current report.

Appendix C listed excluded studies.  In fact, the first two studies suggested by the reviewer for the authors to review for Medicare population were in this list of exclusions.  The one study authored by Musich et al was rejected due to "no comparison group"—actually, the comparison group data on active employees aged 65 or younger than 65 was published in the same year.

 Another paper written by Ozminkowski et al was rejected because it "does not report health outcomes"—however, this article reported savings gained from participation in health promotion programs for Medicare Beneficiaries.  The savings in the study were the medical expenditures which should be considered as "health outcomes."  The authors' of this proposal appear to define health outcomes very narrowly—perhaps limited only to health behaviors.

Please see previous comments.

Please see previous comments regarding different definitions of HRAs.

The articles report data on the same program in two age groups, which means the articles taken together form a stratified analysis, but not a comparison of two programs.  We defined comparison groups to be based on different HRA programs.

We did not consider costs or expenditures to be health outcomes (go to p.4—'Literature Search Strategy'). Indeed, health economics separates health outcomes from costs in the calculation of the incremental cost-effectiveness ratio, which is the primary means of comparing programs in cost-effectiveness and cost-utility analysis.