[This Transcript is Unedited]

DEPARTMENT OF HEALTH AND HUMAN SERVICES

NATIONAL COMMITTEE ON VITAL AND HEALTH STATISTICS

SUBCOMMITTEE ON QUALITY HEARING

OCTOBER 18, 2010

National Center for Health Statistics
3311 Toledo Road, Auditorium A
Hyattsville, MD 20782

Proceedings by:
CASET Associates, Ltd.
Fairfax, Virginia 22030
(703) 266 8402

CONTENTS


P R O C E E D I N G S

Agenda Item: Introductions

DR. TANG: We have a rich set of testimonies before us. Before we get started, we wanted to have everyone just go around the room perhaps and give introductions including the speakers who are here.

DR. MIDDLETON: Blackford Middleton from Partners Healthcare, Brigham and Women's Hospital, co-chair of quality subcommittee and member of the NCVHS Full Committee, and no conflicts of interest.

DR. GREEN: Larry Green, University of Colorado, member of the Full Committee, member of the Quality Subcommittee, no conflicts.

DR. TANG: Paul Tang, member of the Full Committee and co-chair of the Subcommittee on Quality, no conflicts. Any other members on the phone before we go around the room please?

DR. CARR: Hi Paul. It is Justine Carr, Caritas Christi Healthcare, chair of the Full Committee, member of the Subcommittee, no conflicts.

DR. TANG: Thanks Justine. Anybody else on the phone from the committee?

(No response)

(Introductions around the room.)

DR. TANG: Thank you very much. Okay. This is a very interesting hearing and it is sort of a continuation of work that the Quality Subcommittee has been doing over the years -- talked about maybe a few years back -- we talked about quality measures in anticipation of the arrival of the electronic health record system. Unfortunately over the past year and half or so, it looks like that day is coming sooner than we -- we were worried that sooner than we thought, and that is a good thing.

We last talked about meaningful measures, piggy backing on the meaningful use initiative as CMS that is providing additional incentive for adopting and effectively using electronic health record systems. We are continuing on the journey of improving measures that really matter. That is meaningful measures.

And today we are starting to look forward towards a roadmap of if really we could have without being tethered to our path. In the past most of these measures were developed and defined based on claims data or things that you would get on from paper records.

With the incoming electronic medical record systems and the personal health record systems that other electronic resources of data it would appear that you could gleen other information that might be even more useful than the information we have had in the past. That is sort of where we have our sights set untethered by what technology might be available and what data sources might become available in the future.

We would like to ask the question first, what information would be useful to various stakeholders? Whether it is to consumers, to providers, the health care professionals, the accreditors, the payers, the employers. What would be useful in helping them make decisions related to health and not only health care? That is sort of a setting and the context for this hearing. Anything further, Blackford, you want to add?

DR. MIDDLETON: No. I will just say welcome again to everybody and echo Paul's comments. I think this is a very exciting time for measurement and measurement from health care information technologies and a variety of sources. But again considering this from a quality roadmap point of view, what are the sources of data that might be relevant to future measures of healthcare both quality and measurement of value, which is increasingly under intense scrutiny and focus. And with a 3 to 5-year time frame, is our perspective getting these multiple views from different stakeholders would be extremely helpful.

DR. TANG: Let me just review the panels we have for today, and thanks to all the panelists for really participating on short notice. The first panel is going to concern information - what information and measures would consumers benefit from as they make decisions about their health and health care?

The second panel looks at the same kinds of things. What information and measures will providers desire and make use of in order to measure and continuously improve their quality and their accountability?

The third has to do with professional organizations and the accrediting organizations and regulators. What information can they use to perform their tasks in terms of overseeing professional accreditation?

And finally, the payers and purchasers, how do they assess the value of the care and the services provided in the health industry?

That is what we have on tap for today, and we are going to begin with our first panel which has to do with looking at from the consumer point of view. And our first testifier is going to be Eva Powell, who is the Director of Health Information Technology at the National Partnership for Women and Families. Thank you, Eva.

Agenda Item: Panel I – What Information and Measures Will Consumers Need to Execute Their Functions As Educated Purchasers and Stewards of Their Own Health?

MS. POWELL: Thanks Paul and thanks for the opportunity to speak to you today. I will start out a little bit by talking about the current environment, which I won't go into great detail about because we all know what that is, but if you will advance to the first slide, the consumers and quality.

We have done a lot of work with consumers and consumer advocates to talk about quality and to get from them what their concepts of quality is as well as understand what things could go into leading them into more active involvement and evaluation of quality and making decisions based on data.

And what we have learned is that there are a number of barriers in the way of this and one is the fact that there are a couple of other things that are much more prominent on consumer radar screens with regard to health care and quality. And those two things are cost and access. There are more immediate concerns for most consumers than quality as those of us in the quality world think of it. And some of the reason for that is there are so many hurdles for consumers to overcome before they ever encounter health care of any sort whether it is quality or not. They have to find a provider. They have to navigate the insurance with some. They have to make an appointment which often times is not easily accomplished.

Then creating some sort of relationship with the providers is something once they find that provider that most people want to achieve. Then also determining how they are going to pay for care aside from navigating the insurance system whatever out of pocket cost they have consumers have to figure out how they are going to pay for this and what exactly am I going to end up paying?

Consumers really need data and information that is meaningful and useful to them. Any data and information that can first of all help them clear these first barriers would be useful, but I think my point in talking about this is more to keep these things in mind as we move toward quality measurement from the consumer perspective and what is important to them. I think if we come with data and measures that are more meaningful, I know we can -- overall experience with health care system then they will come closer to using that.

In thinking about consumer needs I think it is useful to think obviously in the broader context of health reform. On this slide I just kind of started with the goals and objectives have already been laid out by the Secretary in her draft of the national quality strategy and these are the three primary goals. What I have done is pulled out some of the objectives where I think there is the most opportunity and immediate need for consumer focused data.

First of all, person-centered care, which is one of the objectives under better care will really require better consumer use of data not just at the individual level which is I think what most of us think about in terms of consumer data and how they might use that. It is on an individual level that makes decisions about providers or insurance plans, and I think that is certainly an important area to focus on.

However, there are two other areas that I think we need to bear in mind that I think some measures and some data will serve all three purposes, but I think there may be some differences that we need to look at for the other two purposes and those are a governance purpose. If we are going to have a person-centered health care system, that is not going to be achieved if all the other stakeholders are making all the decisions and then we hand it down to consumers. Consumers must be involved in governance and they will need data to help them make decisions in their roles in that capacity.

And then also on the local implementation of that, and I think that has a governance piece to it, but I am thinking mainly there about consumer involvement and the actual quality improvement process which really isn't a prominent thing right now. But I think if we are talking about systems improvement and really overall learning health care system, which we are ultimately after, perhaps not in the 3 to 5-year timeframe, but ultimately we need to be thinking about how can we learn from consumers' experiences. And I think at any point that a process touches a consumer that is a point where that consumer can help shape and improve that process. There is a data need there as well.

One of the things we have learned from consumers about their concerns and their needs with regard to quality is that very consistently their problems with the quality or their perception of quality issues revolved around communication and coordination primarily. Whatever data and measures that are developed ultimately really must address those problems as well as others, but those are two of the key issues that consumers will be looking to evaluate.

And actually this brings up an issue of a data source because for much of this, for the communication and coordination dimensions of quality there will be times when consumers themselves are really the most valid force of the information because they are the one consistent actor, if you will, in the process.

And then that I think leads to another issue with regards to reform as we start implementing some of the newer payment models as well as delivery models. If we are going to engender consumer trust in those models then I think one way to achieve that is to include consumer generated data in our accountability structure for those models.

And that helps us get at the concept of value, value to whom. And obviously every stakeholder has a stake in the value and has perhaps a slightly different definition of what value is and all of them are relevant. But right now there is really not a way of measuring and holding people accountable for the things that are of value to consumers. Data as we move forward needs to be able to enable that.

So how can these data need to be translated into actual measures? As I thought about this, I think there are a number of criteria that can help direct our efforts. First of all measures have to resonate with consumers in their overall experiences with the health care system. Data also has to be available on multiple different levels, those for individual providers and for overall facilities or group practices. And then data also has to be available across provider settings.

Another criterion is that the data must really show real differences between providers. In other words, if we have data that has been developed using a methodology that shows that 90 percent of the providers are in the average or acceptable category and then there may be two or three outliers on the good end and two or three on the bad end, that is really not very useful for consumers or really for anyone in terms of making decisions based on quality. There are the methodological issues probably that need to be worked on there, but also in the presentation of that information needs to be done in such a way that they are very clearly some differences in specific dimensions of quality.

Then flowing from that we also need to have data that are comparable so that regardless of whether that person is making a decision between Dr. A in this practice and Dr. B in another practice or with another health plan the data needs to be comparable so that it is valid to be used for decision making both in terms of helping consumers use that data, but also in helping those who may not understand that there could be differences that makes the data noncomparable to keep them from making bad decisions on data that they thought was comparable, but really wasn't.

And then finally comprehensive information is really important. I think there are new pieces of information that we don't yet have and that is an area of development in terms of consumer-generated data. Those are perhaps some of the newer concepts that we will need to work on. But some of the older dimensions of quality, if you will, such as the clinical quality measures and some outcomes measures and developing them more fully. I think it is not an either or for consumers and for any other stakeholder really to be able to judge the quality of a provider. There needs to be a comprehensive dashboard of measures, if you will.

And the reason for that and I think this might go for other providers or other stakeholders as well, but consumers vary and individual consumers have different needs. What is really important to one consumer may not be so important to another consumer in terms of decision making. We need to find what those specific dimensions of quality are that are broadly important to all consumers, but then present that information in such a way that people continue to use those things that are most important to them.

And included in that full picture you have on the slide here. I won't go through each and every one of these, but outcomes data is critically important and specifically patient-focused outcomes measures; things such as functional status, health-related quality of life, safety. All of those things are dimensions of quality that will really be very useful and very meaningful to consumers.

Often times consumers when they are making a decision about a treatment or providers for other people who have what I have and who have similar circumstances and presentations, what can I expect if I choose A over B? Those dimensions, those kinds of outcomes are the things that I think will really resonate with consumers.

And then also in the area of specialty care there should be some priority given to those specialties who have -- where there is more opportunity for consumers to make the choice such as OB/GYN, orthopedics where there is a long process, a decision-making process and there is not an urgent need for care that sometimes by necessity circumvents the use of quality data.

One more note, I put cost at the bottom of this slide. I wasn't certain how to handle this. This is perhaps something for the committee to discuss. In terms of quality and decision making there needs to be transparency of cost information. This is perhaps a separate issue, but at the same time if we are talking about quality, efficiency, that is where this kind of comes in. If we are talking about the data needs of consumers then cost data is a data need. When we are talking about different potential treatments that may not have a huge difference in potential outcomes, but if there is a huge difference in costs then that is a really important thing for a consumer to know. I hesitate to talk about that strictly in a measurement sense, but it is certainly a data need.

Finally, the last slide here, I just wanted to lay out a few ideas for where we might go in terms of next steps for meeting consumer data needs. First of all, it is prioritized development, endorsement and use of patient-centered outcome measures, and then secondly to identify some robust methodologies for collecting and using patient-contributed data, and then finally coordinating an integrated and consistent approach to measurement as well as reporting among all public and private stakeholders.

And this is I think, a place where we can think about what Paul and Blackford brought up early on is the context that we are discussing this includes rapid advancement in the HIT world and there are certainly plenty of opportunities to leverage what is going on there. And some ideas for that given that that is where I have been 24/7 most of my time is to collect and use the right information from appropriate team members.

And what I mean by this is and I guess this would relate to a data source is that most of the work being done in HIT is very physician centric in terms of we are talking about that from the sense of the physician will be entering the data in the EHR, the physician is being held accountable and will receive incentive money and there are some understandable reasons for that.

But what I worry about with regard to that is that there are many very relevant and extraordinarily necessary pieces of information from a consumer's standpoint that will never come from a physician and one example -- in my former life I was a social worker for 10 years at a teaching hospital and there were many decisions made about the care a person was going to receive based not solely but in large part on that person's social situation. If there was a very complex treatment that was clearly the indicated treatment for what that person needed but their special situation did not support successful implementation of that treatment then I am not sure that we can say moving ahead with what is clinically appropriate is appropriate and is good quality care.

Until we begin to incorporate not just the clinical information but also social information as well as information from other disciplines that get at functional status such as physical therapy and occupational therapy those are all elements of data that are extremely important if we are really going to get to quality outcomes that are patient centered.

Standardizing information particularly with shared care plans maybe also care management tools. These are all things that are HIT pieces, but also could contain information that ultimately could be fed into an HER, into clinical decision support systems to support the collection of data and all of this is being discussed in some of the efforts that NQF has going on with the quality data set, but those are all direct lines to data sources and the data needs of the consumers.

And then finally some of the basic pieces of the puzzle here I think are just basic electronic communication methods and how can we start employing those to communicate with patients about their individual care, but also to receive feedback from them on their experience of care and get at quality that way. And that is certainly an area for development, but one with the opportunities in front of us in the HIT world we can begin now working towards.

Thank you for the opportunity.

DR. TANG: Excellent. Thank you very much Eva. Lots of good information here, precisely answering the questions that we asked. Thank you.

Before I go on I wanted to, one, acknowledge Mike Fitzmaurice who joined us and maybe you want to comment, introduce yourself and your conflicts if any.

DR. FITZMAURICE: Michael Fitzmaurice, Agency for Health Care Research and Policy. I apologize for being late.

DR. TANG: Thank you. Any other folks on the phone, if you would introduce yourself, including either public or speakers.

DR. NELSON: Gene Nelson from Dartmouth. I will be a speaker.

DR. TANG: Great. Thanks Gene.

MS. HIBBARD: Judy Hibbard from the University of Oregon. I am a speaker.

MS. BOWMAN: This is Danielle Bowman from Health and Human Services, Office of Healthcare Quality.

DR. TANG: Anyone else? Wonderful.

DR. CARR: It is Justine. The presentations are great. Are there slides? I know you have the slides there. Are they available electronically or on the web?

DR. TANG: There will be shortly I am told.

DR. CARR: They will be posted on the website?

DR. TANG: Yes.

DR. CARR: Okay. Great.

DR. TANG: Why don't we open for some questions for Eva and we will take questions for each panelists and then at the end we will have some open discussion with all the panelists.

DR. GREEN: Eva, can you say more about that bottom word that you said you didn't know quite how to present called cost? What do consumers want to see to make comparisons about cost? What is the information they are looking for?

MS. POWELL: I believe the consumers can use just absolute cost information which really isn't available to many people if any right now. My line of thought here is that again using the example I used before. If there is a situation where there is a shared decision making process with the person's physician over potential treatment, there is not a clear, preferable, clinical decision in terms of which one is better and there is a discussion about this is what this treatment involves, this is what that treatments involves. What are you particular circumstances? Which of these or your concerns about those? The decision making process somehow seems to me to be incomplete if there is not information available to the consumer such that they can use that information in their own decision making about which one is better for them.

In other words, if this is a person who has limited means and there is clearly significantly less expensive route and when I say less expensive I mean less expensive from their own pockets.

The difficulty here is that this very rapidly goes down into the death panel discussion and I think that is the reality in the environment we are working in we have to be weary of. At the same time the complete lack of that information is not helpful to anyone. I think this is a bigger issue than consumers. I think it is an issue system wide.

But thinking from the consumer perspective in an environment where it is so difficult when you are making a decision about a treatment to know exactly what it is that I personally am going to have to pay. I just somehow think that that is a huge gap in the availability of information. And I do view it differently than quality data. At the same time it seems to me like the two pieces go together. That is why I mentioned it and mentioned it last. Does that answer your question?

DR. GREEN: Yes, it is helpful. Can we tease it down just a little bit more? If I heard you right, you are saying consumers want to know what it is going to cost them out of pocket for the options that appear to be equally satisfying, another comparison. Is that correct?

MS. POWELL: I think in the instances when there is a comparison to be made, in other words, if they are thinking treatment then yes they would use the comparison between the two treatments. But if there is a treatment they decided on and they are also in the midst of deciding about a provider or they are overwhelmed by the thought of this amount of cost coming out of their pocket, there may also be an opportunity to use that as a criterion for choosing a provider. Someone from the payer group may speak more to this point about how that might engender changes in behavior from the provider's perspective, but it seems to me like from a consumer perspective most consumers by the time they get to the point of talking about different treatment options they probably have already chosen a provider.

But there may be instances when people and again I am thinking out and how patients might ultimately use data. If they are going to comparison shop then they are going to comparison shop on what their clinical needs are and what is going to meet their health care needs. But if they are paying a lot out of pocket then that is something they are going to want to compare on as well.

DR. GREEN: I get that. I want to go to the word stewardship and your quality dashboard. I would like to try to unite them and get your opinion about this. If I heard you right, you are saying you have consumers need a dashboard that gives them information that allows them to make comparisons. We are really interested in quality. It is a quality hearing here. The dashboard has measures of quality of some sort. I am hearing you say you are not quite sure how to bring this up, but it needs to say something about cost.

And then in our questions that we are asking in this panel we are interested in helping people be stewards of their health. And you are suggesting that people might actually be -- the consumer might be sensitive to the need to make decisions about their care based on quality and cost and access. That is what I heard.

MS. POWELL: Well and those definitely. The cost and the access are the things when you talk to consumers initially and talk to them about the health care system, those are the things top of mind usually. If you peel a little further, you get to the --

DR. GREEN: Let me nail down just to almost a yes or no question. Do you think consumers give a damn about the total cost of their care or do they just care about their out of pocket cost?

MS. POWELL: I think probably just out of pocket cost right now. These are questions that I think we will see when we have information available.

DR. MIDDLETON: I wanted to follow up Larry's questions. Eva, again, thank you for an excellent presentation and a great overview. I am stuck on this patient centeredness issue a little bit. I certainly think I know what it means in many ways and all of us have an image of what it is. As a clinician if you are evaluating treatment A versus B for the patient before you it matters whether or not there is social context or environmental or clinical or even financial context allows you to think about that recommendation of a specific treatment for a specific patient.

But when we turn to trying to measure this idea, it becomes very complicated very quickly. Because of the confounding of patient preferences and costs, if you will, externalities even at a level. When you think about patient centeredness and I am just thinking at the top of my mind of miles per gallon. If I am going out to shop for a car, I can look at miles per gallon. I have a very clear indicator of some level of one assessment of value for two different cars. What would be that thing if you were coming to see me or Paul for your next clinical encounter, God help you? If you wanted to choose between two doctors, what is the right thing? What is that essential measure or what are the flavors of it that you think have to be incorporated in a single measure or in a composite measure that a user can use, an end user?

MS. POWELL: I think bringing up the concept of a composite is a helpful place to start because of the nature of the individual differences. When I think about this, I think about the questions that I would ask if I were in the situation of making these decisions. And, one, getting at what you mentioned about the individual differences and patient preferences does this provider act according to patient preferences. If you ask that question then that is what relevant to me. I don't really care what all his patient preferences are and how many times he gave advice using an Arabic interpreter versus a Spanish interpreter. That doesn't matter to me.

What matters to me is this a provider that really acknowledges this as a dimension of quality that he or she must work toward achieving. In terms of how to form the measure or collect data on the measure I think that is where we get to some of the ideas I talked about before incorporating social assessment information. I think of it also as a clinical decision support role which can then feed into measurement in the same way as clinical information feeds into clinical decision support than ultimately why can't social information feed into that. In other words, if there is a -- I will use my own experience.

I worked for six years on a transplant program and there were a number of times when after all the assessment was done the psychologist and I recommended not listing a person for transplant because of social factors. And sometimes that fed into the overall decision not to list that person. Sometimes we got trumped by other things and almost without fail that person did not do well with transplant and it was a rough and rocky and just a difficult road from start to finish.

It always made me to think if perhaps we can incorporate those social factors particularly for those treatments or procedures that really require a lot of a patient and family or a specific thing from a patient and family once they are no longer in contact with the health care system or not in daily contact or under the care of that provider institution then that would then become a flag. It wouldn't necessarily say okay always if this is the case then this person doesn't get this treatment obviously. It would be a flag just as CDS is a flag for doing particular things, adding additional supports, doing further assessment, those kinds of ideas. Does that get at your question?

DR. GREEN: Yes. Thank you. And a quick follow up. You also mentioned a couple of times you alluded to the governing bodies or the need for governing bodies or governance to help consumers or help processes enable consumers in the ways you described. I am not sure what those governing bodies are. Can you say a little bit more about that?

MS. POWELL: I think the federal governing bodies which I think we have done a good job of having consumer presence on there. But there also and again thinking in the HIT world, the state HIE effort is one example. How often are consumers involved in the decision making on the local level? We at the National Partnership are starting to get more and more calls from consumers who actually have been very active in their state efforts to this point and because of local politics whatever decisions they have already made that have been very consumer friendly have been kind of disregarded in favor for the more politically relevant of viable situation. There is just a lot to be considered on the local level because there are differences there. There may be a different decision made in one area than another because of the individual local circumstances and that is neither good nor bad, but consumers must be involved in that decision-making process if they are going to accept this.

And I think there are different questions. Some of the same questions but I guess in my thinking that is where the population quality measures come into play.

DR. CARR: Blackford.

DR. MIDDLETON: Yes.

DR. CARR: Blackford, it is Justine. I would like to make an observation about the connection between what we are hearing this morning and what we heard in July on the health plan ID testimony. At that testimony we heard the importance of having electronic transactions permit both the physician and the patient to know what is covered in their program either on that day or going forward. I think I am hearing a lot of endorsement from a different audience here from the patient perspective that how important that is in the moment.

MR. QUINN: Something that Eva said that really resonated with me is the idea that consumers need to themselves reflected in measures of quality and other aspects of care. I think we are going to hear this a little bit from the other speakers in this panel, but there are some methodological issues that need to be addressed especially with regard to clinicians accepting these measures if there is consumer input involved.

This leads me to think about and this is the same for other quality measures in other domains the difference between internal quality improvement efforts and the measures that are collected for internal feedback, growth, improvement, and then measures that are reported publicly or externally, for external agencies. I think that if you were to talk to most clinicians they would say the measures that are reported externally are ones that could be standardized and other things, but aren't necessarily the ones that they really internally run their practice based on. It would be good to see where that intersection of meaningful consumer measures that are methodological sound enough that clinicians can accept them for internal use. And then I think that there is going to be a higher level of methodological soundness that will be required for them to be used externally.

MS. POWELL: I would agree. I think the place to start is the internal place because that is I think where you are going to prove the value to the individual clinician is to their practice environment. And then it is also where you work out the methodological kinks that obviously need to be addressed before we go to something like public reporting and accountability.

DR. TANG: That is something to build on this then Mike. Going back to -- one of the things that struck me about you made -- first of all I really liked the attributes you brought up in your presentation in terms of what is meaningful for a consumer. One of the statements you made was in saying everybody is 90 percent of something isn't that helpful. Let me try a couple of things and get your reaction because I think this is something that we talk about a lot in the various quality committees that are operating in this area.

One is let me build a case from Blackford's miles per gallon. Miles per gallon is done in a very standardized rigid way and you have a number and there is no such thing as a standardized driver. One question would be would it make more sense to you to stratify that. This is what a city driver would get. This is what a person who does long commutes on a highway would get so that it can be closer to what is meaningful for me because I know I do X.

The other approach I want to take is there are -- I don't think you used the patient satisfaction which I think is good because there are companies out there that do these patient satisfaction surveys. And it turns out for whatever reason at least it seems to be that patients who return their surveys always rate doctors highly or rate them high. Therefore, most everybody is in the 90s percent in terms of satisfaction on these surveys. But then these companies will turn them into percentile which means you could be 1 percent different in the patients who rate you high and yet it makes a 10, 20, 30-point difference in the percentile. And is that meaningful?

Two pieces. One, is it better to standardize, i.e., risk adjust on the measures so that you get a number that is in theory comparable but doesn't actually answer your question and the other is what is the implications of having these things by percentile when everything is bunched around let's say the 90 percent.

MS. POWELL: I hadn't thought about it that way before in terms of stratifying by consumer characteristics but I think ultimately if we could figure out what those characteristics might be, I think, yes, that would be helpful. I am not sure. Sorry, so your second part of your question again.

DR. TANG: The second one was turning these high percentages of favorable opinion about patient satisfaction, so-called patient satisfaction, turning them into percentile which ends up making small differences in percentages of favorable opinion into large differences in percentile ranking because they are all clustered around high numbers.

MS. POWELL: I think with that there are a number of ways to get at that. First of all, yes, I didn't say patient satisfaction because we prefer to talk about patient experience because they are two very different things. Patient satisfaction generally has more to do with was my food hot, was the nurse nice and those are all important things, but they don't have real bearing on the actual quality of care. Patient experience, the things being evaluated saying the HCAP surveys and the other CAP surveys get more at things that do come closer or in some cases -- to have a bearing on quality. I think that is really -- when we are talking about measurement and data, that is where we need to focus.

And I think this is also where HIT comes into play and I am not a statistician so the methodological issues I will leave to someone else, but I think part of -- what you say about those who return the surveys are the ones that are enamored with their physicians. If we can make the completion of the survey -- if we can gather survey results from a higher number of people then my guess is that we will reduce that effect somewhat and HIT is a perfect opportunity to address that issue. I understand that there are methodological issues there, but it is still an opportunity. And I suspect that those methodological issues can be worked out. That would be one thing I would say about that.

And that also addresses, I think, a concern on the provider end of the expense of doing those kinds of survey. It also enables again back to the CDS idea I had before; the electronic gathering of that kind of information. And if there is a way of gathering that information at the point of care, you may be more likely to get overall, yes, I am really satisfied with this. However, there was this one time and this was not a very satisfying experience and this is why. Those are the kinds of things that I think are lost in the current way those surveys are done because it is after the fact, but that also -- that moves us towards this learning health care system.

It also is an opportunity for HHT and to use the electronic environment to feed that information back to the provider in a way that informs their ongoing practice, but it also then gets it -- it kind of addresses Matt's earlier comment about the internal use versus the reporting use. I think perhaps that might be a way to start addressing some of the things Matt said. I don't know if that really addresses your question. But I do think that the more we actually gather that information and report that back certainly to the clinician, but also make that information available to consumers so that they begin to understand the various dimensions of quality and that is good.

DR. FITZMAURICE: I, too, enjoyed your presentation and I think in good part because of your background having spent so many years in social service in a hospital. You put your feet in the shoes of the patient and said what kinds of decisions do they need to make and then how can I help them make the decisions. As part of a research agency, we ask how do we get consumers to use federal, state, health plan or association supplied quality measures since that is a good part of our business at AHRQ. We ask are the data good, is the method good so it gets at your robustness. Are the quality measures meaningful to patient outcomes and were the providers patients adequately described? Were they in fact sick or were they not?

I want to ask, one, is it a good thing for quality measures to -- here is a quality measure that we found out about the provider, but also to present a provider feedback or a provider statement saying my patients were sicker or I won't say I had a bad day, but look at all that I did do during a day to put into a balance or perspective.

Secondly, you talked about what I interpret as personalizing the quality measures, giving them functional status, tell them about how this treatment affects their quality of life. Is it safe? Would you go as far as to say some of the quality measures given to patients should be disease specific and tailored to the individual's characteristics? Maybe they have -- conditions. Maybe there is something else about their family situation or their ability to afford care that the quality measures ought to be tailored to that. I am asking more the first question. How do we get consumers to use them? And I liked a lot of your answers and I am probing to get more should we have physician feedback also presented to the patients. Should we take into account more what is meaningful for a particular patient who is going to be reading the quality measures or looking at it?

MS. POWELL: I think on your specific question about should we include the provider feedback in terms of kind of an explanation behind whatever score they got. I hesitate to look favorably on that only because everyone says their patients are different and yes there is a need for risk assessment and incorporating some of those methodological issues. The patient seen at a small community hospital are not the same as those seen at the tertiary medical center.

But I think we can -- again, not being a statistician it seems to me there are methods to accommodate that via methodology rather than add that complexity and confounding factor for consumers to have to figure out. And that also sometimes ends up being an excuse. And I have spent the majority of my career in the provider setting so I certainly identify with that. And there are. Every patient is different and you are under the gun in the provider setting. I understand those arguments, but I think ultimately there are ways to deal with that and adding that dimension or that statement would only confuse consumers.

In terms of how to get consumers to use the information, again, the biggest piece is making it relevant and we have talked about some of those ways. Still consumers trust their providers over pretty much any other stakeholder. I think a lot of that boils down to do the providers see value in this.

In another life I worked for the QIO in North Carolina. I know from that experience and that experience was primarily with hospitals, but the more advanced hospitals, the hospitals that really took quality seriously in the early days beat CMS to the punch in putting their quality measures on their own website. I look at that as being a really innovative thing to do and a lot of those hospitals frankly were very small community hospitals.

I think a hospital that does that, a provider that does that that takes quality seriously and understands the value of quality measurement to the point that they are putting it in places where patients and consumers see it and then that brings up the opportunity for it to be part of a conversation which then becomes part of that provider/consumer relationship. I think that is how you get consumers to use it once you have made the data relevant to them.

DR. TANG: So Larry because we are short on time, would you mind if we did it at the end? Could we do that? Okay. Thanks very much, Eva.

Next on the panel is Judy Hibbard and Judy is a professor at the Institute for Policy Research and Innovation at the School of Architecture and Allied Arts at the University of Oregon. And Judy is going to be talking about patient activation. Judy.

MS. HIBBARD: Good morning and I am sorry I can't be there with you this morning. Assuming you have my slides. I actually would like to start by answering the question that someone asked about patient centered. What does it mean to be patient centered? The way that I look at it is through the lens of my own research. I look at it as the degree to which you as a provider understand my job as a patient in managing my health and my condition and actually help me do my job. To me that is what patient centered is.

Let's go back to the quality measurements and what is going to be meaningful to consumers and patients. One of the issues that has been a major barrier for consumers to use quality information is they don't understand it. We can go to the first slide. What is meaningful in what we have seen and studied is that people do understand the patient experience data and many find it meaningful and useful. Based on that I think if we had more patient-generated data it would also be useful and actionable to patients.

I want to talk about two major categories: the patient-generated data that focuses on improvements and health status and functioning and the improvements in people's ability to self manage their condition. Let's go to slide number three and just think about the criteria for these types of measures. Of course we want valid and reliable measures. We want measures that can be used to improve performance that are actionable on the provider side. But we also can find measures and use measures that can be used to actually inform the medical encounter and to improve the care of the patient providing the information.

We want measures that are sensitive enough to reflect changes resulting from medical intervention. And also I believe we want measures that are going to shift the focus of performance to an area that consumers value and understand.

Let's go to the next slide. I understand that Eugene Nelson will be talking about the patient-generated data that focuses on health status and functioning. I am going to talk about the ability to self manage. Now if we start with the assumption that an important outcome of high-quality care is that the patient should be gaining in their ability to self manage then we should be measuring and tracking improvements in this construct which I will refer to as activation as an intermediate outcome of care. In activation it means that the patient has the knowledge, the scale, and the confidence to manage their health and their health care and to also understand that they have a role to play that believes this knowledge and skill.

This is actually something that can be measured. We can go to slide five. We measured it. We developed a measure of patient activation that assesses these characteristics. We used RASCH modeling analysis to do this and we have come up with a very robust measure.

Now what I want to do is just tell you a little bit about the measure itself and what we have learned and how this relates to these questions of the criteria that I laid out in slide three. The patient activation measure is a 13-item measure. In slide five what you are seeing is the built-in difficulty structure of the measure that is how hard it is for a person to say, yes, that is true about me. And we have them measure, translated into 16 different languages. We have evaluated the data on about six of those languages and that difficulty structure is maintained across language and culture as far as we have observed.

Slide six just shows the actual questions in the measure. We can go to slide seven. This is where some of the key learnings have been. We have had an opportunity just to learn so much once we have a valid measurement. One of the things we have observed is that people appear to be -- that this idea of activation is developmental and people appear to go through these different levels on their way to becoming effective managers of their health.

In in-depth interviews we have seen that people who are low on this dimension tend to be overwhelmed with the job of managing their health. They may not understand that they have to take a role. They have had a lot of experience with failure. They have tried to do it. They couldn't and they become passive as a result. I think this is -- and at the high end you can see the mirror opposite.

This is important when we think about what happens to people in the medical encounter where especially those with chronic illness are given a big long list of all the things that they need to do to change about their life. For people who are low on this dimension, they may try to do it, but when they can't their natural human reaction is to do nothing. They end up having another failure. In essence by not really understanding who we are dealing with in a medical encounter we are setting a lot of people for failure and kind of keeping them in that passive position.

I am going to just go to the next slide. We have seen in multiple studies actually all over the world with different investigators the activation measure is predictive of preventive behaviors, healthy behaviors, disease specific self-management behaviors, and information seeking behaviors.

If we go to slide nine, I will share with you another major aha we had in this and this is looking at how someone manages their hypertension. These are divided into the percentage of people who actually engage in the behavior by level of activation. This is what we call a behavior map. We developed many of these. And the big lesson from this was that almost every behavior is related to level of activation, but as behaviors become more difficult and complex and requires sustained action fewer people in all the levels do that behavior.

Go to the next slide. However, behaviors that focus on information seeking many of them are only done by the highest activated group. I am sorry to say that is also true about quality information using quality information that it is the most activated group that will seek out, know about, or use quality information and we have seen this in multiple studies. And I think this has implications for how we promote and use quality information and also thinking about the importance of people becoming more active and activated.

Go to slide number 11. One of the things that we sort of gleaned from all of this is that we can actually use activation levels to think about what is the realistic action step for a patient. Many of the behaviors we are asking people to do right now are kind of way beyond them. It is kind of like throwing non-swimmers into the deep end of the pool. We are asking people to do things that they are not ready to do.

But if we start with behaviors that are more feasible for people to take on, we actually increase their opportunity to experience success and that is how people gain confidence. At the low end of activation it is really a matter of not having confidence that you can affect your health. The point is that this is actionable information in a medical encounter.

Slide number 12 we needed to see if this was a changeable construct and we did observe over a 6-month period we observed chronic disease patients and we saw that some people did increase significantly in activation and those people had statistically significant improvements in 11 of 18 behaviors. When activation improved behaviors, multiple behaviors changed. Those who did not increase or stayed stable or declined in their behaviors had the same trajectory.

Slide 13 shows and 14 we look at how activation is related to outcomes. This is slide 13 is a study by AARP that looked at people who recently had hospitalization who had multiple chronic illnesses. And what they saw was that people who were low activated had more than twice likely chance of being readmitted to the hospital after they were discharged. They were also more likely to experience a medical error. They had more chance of having poor care coordination and having harm as a result of that poor care coordination. And not surprisingly they were more likely to lose confidence in the system. Activation is related to many of the outcomes we are very concerned about.

Slide 14 shows a study that was done by the care management institute at Kaiser Permanente. They measured activation in chronic disease patients in 10 states and then went back and looked at -- 2 years later they looked at the records of the diabetic patients in that sample and they saw that activation score was predictive of whether or not patients were adherent to their diabetic testing, whether or not they had good glycemic control 2 years later, and whether or not they had had a hospitalization in this period. The measure is robust enough to predict future outcomes.

Slide 15 is a summary of a study that was done to actually intervene and try and increase activation in an intervention. And a study in disease management there was a comparison group. What we observed there was that activation increased more in the intervention group than the group that was getting coached in the usual way. The intervention group was coached tailored to their level of activation and they had better clinical outcomes and reduced utilization.

Slide number 16 just says what we know cross sectionally that those patients who say that their provider helped them to learn to monitor, set up an exercise program that those patients are more activated although we don't know the direction of causality there.

We can go to slide 17 to summarize here. We do have valid measures, measures that are changeable and are sensitive to intervention. The activation measure actually focuses attention on important outcome of care that currently goes unmeasured.

The other thing that this does is it changes assumptions about what good care entails for both the provider and the patient.

I went through that rather quickly but I want to see if people have comments or questions.

DR. TANG: Great. Thank you very much Judy. Larry.

DR. GREEN: Judy, thanks. Are you in Oregon?

MS. HIBBARD: I am.

DR. GREEN: Thank you for joining us so early. Judy, I have overlapping questions here that are all connected. I think you are telling us to one of your answers to our question about what information can consumers need is that they need to know their activation level.

Then going back to Paul's point about stratification are you suggesting that one way to stratify Eva's dashboard is by activation level of patients?

MS. HIBBARD: I am actually saying that I think that what patients need to know is that providers do a good job or don't do a good job in supporting their patients' ability to self manage. It is like a measure, yes, on a dashboard, but what does the provider's panel look like. Are they gaining in activation as compared to another provider? So they can see how really supportive are people getting the help they need to do their job.

DR. GREEN: Got it.

DR. MIDDLETON: Good morning Judy and thanks for joining us. A terrific presentation - extremely interesting. I guess the first question I have is about the activation measure and what we heard from Eva Powell in the first panel was that consumers are very concerned perhaps most concerned about costs of care and access even though they have differential out of payment and the outer payment payment only pays for part of health care costs, et cetera.

MS. HIBBARD: Can you repeat what you said?

DR. MIDDLETON: Can you hear me now?

MS. HIBBARD: Yes, I can hear you.

DR. MIDDLETON: Okay. We are doing a Verizon commercial. The question, Judy, is about measure. What we heard from Eva Powell was that patients or consumers were most concerned about cost and access, yet, reviewing the patient activation measure 13 questions not one of them deals with access or cost. Do you think those are independent measures of patient activation since Eva also said patients who don't have access or can't pay for their care sometimes become passive, et cetera? Is that an important thing to consider beyond the patient activation measure or how does the patient activation measure consider cost and access?

MS. HIBBARD: Good because I wanted to get into the cost question too. What we have seen in studies is that while every aspect of health behaviors related to activation cost is not. It is true that people are concerned about cost, but I think that there is a danger in the way we talk to consumers about cost. And that is that what is being -- what is seen in public reports is often there is just a measure of cost or efficiency that is just added on to quality information.

When consumers do not understand quality which is the case with much of the measures that are out there right now, they will use cost as a signal for quality. Higher cost for many people will signal quality. How we communicate that our cost is really that we have to be sensitive to that and understand that.

One way to get around that is to provide cost information within quality strata so people can see that they can still get high quality and not have to pay top dollar for it. But they are activation and costs are really separate issues. The activation measure focuses on really mean managing by health and how confident that I can do this.

DR. MIDDLETON: And how about the access dimension or issue or is it the same thing in your mind?

MS. HIBBARD: We haven't explored that as much, but I do want to say one thing about this whole area of what consumers care about. I think it is really important. And that is that we have seen this over and over again. When you ask people about an area that they know is quite important but it is complex and it is unfamiliar territory, they will give you an answer about what is important to them, but it is a very changeable idea because people's ideas are fluid right now. Their preference is about or their ideas about what is important with quality can be changed just by the way you ask a question, what you presented just before because people don't know. When we say oh people care about cost, of course they care about cost, but they care about other things too when they understand them. My point is we can't just go with what people say is important right now because that is just too fluid and too changeable. When people really understand the quality arena, their ideas will become more fixed, but right now they are not.

DR. MIDDLETON: Just one more quickie if I may. The data you presented, for example, on slide 13 showing that the less activated patients might experience -- do experience more medical error. I wonder if you could just talk through that a little bit in terms of what is the biological plausibility or mechanism of that.

MS. HIBBARD: Yes. Okay. Good. So what we have observed with the less activated is they really don't understand that they have to play a role. If you can imagine being in the hospital, the activated patient will ask about their medications, will be more involved in what is happening if they are able. But the less activated patient assumes this is not their job and that everyone is going to take care of them. That is why I think that they lose confidence because their experiences don't match their expectations.

DR. MIDDLETON: Okay. That is helpful. And last one for me is just on this critical question of the predictive nature of the patient activation measure. Are the data you are showing on slide 14, for example, from a prospective control trial in some way that actually assesses the independent prediction capability of a PAM on utilization or I am concerned that there may be extraordinary, confounding that is that the higher activated providers or health systems, if you will, are influencing patient activation of course the other way. How do you sort that out?

MS. HIBBARD: This is all within Kaiser. They are all Kaiser patients in this particular study. It was the survey that measured activation with them in 2004 and then in 2000 they went back and looked at the data. It is prospective but they went back and looked at the data and pulled all of the records for the people, diabetics who were in the survey and looked prospectively at how well it predicted what happened over the next 2 years.

DR. MIDDLETON: I get that sort of a historical prospective methodology, but how well do you --

MS. HIBBARD: But there was no intervention except the usual care for everyone.

DR. MIDDLETON: And do you have a sense of what factors they controlled for or what interactions they might have controlled for?

MS. HIBBARD: They did control for multivariate -- I am trying to read the small print here. Age, gender, race, comorbidities, and number of diabetic-related prescriptions.

DR. MIDDLETON: Okay. So if it is actually the case that the PAM is predictive in a prospective way, should we make it -- I can't remember what number of vital sign we are up to, but if we have --

MS. HIBBARD: I do think it is a vital sign. It tells you about the patient's ability to do the job you are asking them to do. That is to go home and change the way that they eat or exercise or take their medications.

DR. MIDDLETON: Thank you. That is very helpful.

DR. TANG: This is Paul Tang. I wonder if on the 13-item PAM, how is that scored? Is there just a point for yes? Is it a yes/no and one point assigned for each of these?

MS. HIBBARD: No. It is actually based on the RASCH analysis. It is a simple summative score divided by the number of questions. And then if you don't have the ability to run it through RASCH then we do have like a score table. The 0 to 100 scale is what it produces. And we have seen it just changes in like a few points are meaningful in terms of behaviors and outcomes.

DR. TANG: Let me check an understanding I think you said. If you have a PAM score at one point in your life, you know that people with these scores behave in certain ways or don't. If the same individual changes their PAM score at a later point in life, do they then adopt the behaviors of the other folks who were in that Pam score at baseline?

MS. HIBBARD: Yes. People do but you know like so in the study where we measured 18 behaviors and people changed 11 of them, but we don't which 11. The idea is that and people start to feel more in control. They are going to do multiple things different not just like one thing different.

DR. TANG: And then your final response to one of Blackford's questions. You talked about how the PAM score sort of assesses whether an individual would "do the things they are asked to do by their health care professional." And I am wondering whether you could look at it the other way around. Should we be actually taking into account both their PAM score and what Eva was talking about the social determinants in health when you "ask them to do something"? Do you see what I am saying? It is sort of the flip side of that.

MS. HIBBARD: Yes. That is right. That is the whole idea of thinking about sort of setting someone up for success instead of failure is ask them to do something at least an initial step. What we have observed is that people -- their motivation kicks in when they start to experience a little bit of success. They feel more motivated when they think they can actually do this. Yes, asking them to do something that is realistic and then sort of thinking about it more as a journey or teaching someone to swim. You don't ask them to swim laps right away if they don't know how to float. You start out smaller and kind of build to the skill base.

DR. TANG: As our lesson to you it sounds like one's activation is more malleable and there must be some kind of constraints on that than let's say contrast that to your cultural background. You have a certain culture. You have a certain belief system. It is not as if we should be changing their belief system. We, the health care professionals, should be working to develop a care plan or health plan that takes into account their belief system, their cultural background. You are saying the opposite in PAM, not quite the opposite, but in addition to taking that into account you believe you can change one's activation over time.

MS. HIBBARD: Yes. We have seen it over and over again. Yes, you can. And sometimes it is as simple as telling someone who has a big long list of things that they need to do don't do all right now. Just work on this one thing. And the one thing should be what they want to work on because that is where there is the most motivation. Now when they start to see they can actually do this then their motivation increases. Their abilities increase to step up to the next challenge.

We have seen people increase 8, 10 points in a month just because doing something like giving them permission to do one thing. It is very freeing for the individual. If you feel like you can't do everything, often you just do nothing.

DR. TANG: Great. Thank you.

MS. POWELL: Thanks. I apologize. Judy, thank you so much for your presentation. I agree with everything you said. I just wanted to add something quickly that I should have said particularly in reference to Larry's question about cost. The question was specific to out-of-pocket costs and I answered that. Yes, consumers are concerned about their out-of-pocket cost. But what I should have said at the time is that cost as we talk about that in reference to consumers and to consumers perhaps is better termed burden and I think, Judy, you were getting at this because cost to consumers -- out-of-pocket cost is just one of those costs. There are also costs in time. There is cost in work hours and income and potential loss of income if there is an absence from work too often. A cost of attention. There are multiple costs which really add up to burden which perhaps is the better way to talk about this with patients.

And I wondered, Judy, if those things are linked to activation particularly as it gets into some of the kind of other life demands kind of costs. Have you all looked at that?

MS. HIBBARD: No, we haven't. Only like actual money costs. But that is an interesting question. I don't know. One of the things that we have observed that is relevant is that the more activated are actually -- have better experiences in care and are more satisfied. And I think that is because they know how to get what they need and we have seen also in the study for health systems change that use the PAM that even after you control for income and insurance status, the low activated are much more likely to have unmet medical needs. And I think that is because they come up against a problem, a barrier and they give up. It is all about people's sense of being able to get what they need and their skills in doing so.

DR. TANG: A final question, Mike, if you could keep it short. We are running out of time.

DR. FITZMAURICE: Sure. I have a good short one. Is PAM related to patient outcome? You presented some evidence that it seemed so at least. Blackford mentioned about should this be another vital sign which triggers in me a thought. And by the way, thank you for a very thought-provoking presentation. I wasn't provoked but my mind certainly was provoked.

My question is if this is related patient outcomes then should the provider be held accountable for his or her patients achieving a higher score. Is it something like patient exercise where if I exercise more, I am going to be healthier and be better, but yet it is my own motivation and maybe not my doctors? Is the lack of exercise the provider's fault? Is the lack of progression in a patient's activation somehow is the provider accountable for that or is it something --

MS. HIBBARD: That is a very good question. And we are actually in a study right now where we are linking patient PAM scores with a score with a continual measure for clinicians about their beliefs in the -- basically their support for the patient role. What we are seeing is that they are linked. At least statistically that those clinicians who are more supportive their patients have higher scores. Now we need to look at that longitudinally and we need to see if physicians do make a difference, but I believe that we are going to see that physicians can make a difference and what we are going to see is that people -- that their panels look different in terms of their activation profile and in which case I would say yes.

DR. TANG: Thank you, Judy. This has been very rich and provocative and interesting. I think it feeds in very well with this discussion we are having.

Gene, I think you are up next and we will give you the full time.

DR. NELSON: Thank you. Am I coming through okay by phone?

DR. TANG: You sure are. Thank you very much.

DR. NELSON: Great. Are my slides ready to --

DR. TANG: Yes, they are too and I will just give a quick introduction saying that Gene is the Director of Population Health Measurement Program at Dartmouth Institute and Director of the Population Health and Measurement.

DR. NELSON: Thank you. The first question that I was asked to comment upon is what data and measures will consumers need to be successful patients and purchasers of care. The short answer is they will need a triple aim compass with feed forward and feedback data. And I hope by the end of my brief remarks that you will have a sense of what that means.

Amory Lovins, the chief scientist for the Rocky Mountain Institute, has a question. And it is how the kilowatt-hour of electricity is like a day in the hospital. And the answer is nobody wants either. What do consumers want? Hot showers and cold beer. What do patients want? Better health, better care and lower costs.

What I will do briefly is to focus in on a few questions that consumers need data to answer, a few of the critical questions like, first, can I get the care I want and need exactly when I want and need it. Second, am I getting the vest value care for me given who I am and what health care can do? Third, how can I get the data I need as care is provided, to make good decisions and to review the success of the care that I have gotten?

What I will do is provide you with a couple of case studies to put these questions into context and to show how this might work. First, can I get the care I want and need exactly when I want and need it?

Go to slide five please. And this is Amy. We have Amy's permission to tell her story. This is a true story. Amy, at the time that I met her, was a 38-year-old single woman, new to the area teaching school for disabled children. And she found a lump in her breast.

Amy at the invitation of Dr. Dale Collins who runs our comprehensive breast cancer program was accompanied by Leslie Lamb, one of my quality fellows, to just find out what Amy's journey would be like for the first 6 months of her care.

This slide six portrays the 21 different visits made in the first 6 months to 14 different frontline clinical Microsystems for her first 6 months of care. And Amy would tell you and she told the comprehensive breast cancer team as well that at times her care was delightful. The wig lady. The first time she met with her oncologist, Dr. Pam Ely. Sometimes her care was dreadful. For example, having to have stat lab that weren't stat and actually her first meeting with the surgeon that did not go well.

If we step back and we look at Amy's care from a distance, we have Amy who is a person, a patient who is part of a larger population of patients: women with newly diagnosed breast cancer. And they entered into our health care system and they start a journey in health care in hopes of getting the best results possible. There are tests. There are diagnoses. There are treatments. There are decisions to be made. There are often shared decisions that are made based on preferences and best available evidence.

And then we can pop out to the right side to say what were the outcomes at 1 month, at 6 months, at 1 year, at 2 years, at 5 years. What were the health outcomes? What were the experiences that the person had and what were the costs that evolved over time?

So the next slide eight, Amy and people like her, want data to answer key questions as the care progresses over time. What are my chances of surviving? What are the risks of different treatments? How will I feel physically and mentally? Will I be able to get back to work to teach my children at school? What can I do to help myself to get the best results that I can get? Will I be treated the way I would like to be? Can I afford the best treatments that I want and need?

Slide nine then puts these questions into the form of a compass that Amy was in a sense, using in the early part of her care journeys. She was concerned about mortality, the risk of death, and the opportunities for survival. She was concerned about the risk of recurrence, the north point. She was concerned about the morbidity and the complications associated with different treatment options. Going north again, she was concerned about how she would feel physically and mentally. Could she go back to being a teacher? Would she be tired out or would she have vitality?

What were her care experiences? What was her level of activation as her care evolved? What were her experiences with the delivery system? Amy had good insurance. She wasn't worried about her direct medical care costs. She was concerned about being out of school, substitute teachers doing perhaps not as well what she needed to do and understanding her community was paying for that.

Here are two more cases that I will use. I will use Brian and Betty and the context being the Dartmouth Spine Center. Brian and Betty are fictitious cases but absolutely typical. Brian, a 45-year-old man with a herniated disk and good insurance. Betty, a 64-year-old woman with spinal stenosis, no insurance, single woman, lost her job.

The context being the Dartmouth Spine Center where the motto is back to work, back to play one person at a time and to achieve this in a way that is patient centered. The idea is to bring together all the disciplines that are needed to care for people with spine problems, to engage in shared decision making, to use outcomes tracking at the individual patient and the clinical population level and to use the measures that accumulate over time on outcomes and costs and experience for collaborative learning and research.

Slide 12 shows that when the spine center was open, an idea that was novel was to create a very rich information environment using feed forward patient reported data to help inform what the initial plan of care would be and to evaluate how that plan of care is working for the patient; and then feeding back that data to the patient and clinical team over time to monitor progress, to refine treatment regiments, as well as to aggregate the data for collaborative learning in the spine network and for an NIH trial on back surgery.

Slide 13 goes back to Brian and Betty. Brian with the herniated disk, and Betty with the spinal stenosis, and people like them have key questions. What are my chances of becoming pain free? What are the risks and benefits of different treatments? How will I feel physically and mentally? How fast will I be able to get back to work and back to my life? What can I do to help myself to get the best results? Will I be treated the way I would like to be? Can I afford the best treatments that I want and need?

In slide 14 you will see an image of how the patient-based feed forward data works that the individual when they make a visit to the spine center can get a touch pad or they can come in over the Internet and in a secure portal and they complete a health questionnaire about how they are doing.

On the next slide, slide 15, creates a summary report that is produced each time the individual makes a visit to the spine center. If you could see the details of this summary report, you would see that you are putting critical information about the patient all on one page for the patient and the clinician or the clinical team to work on together: the health history, the health habits, any red flag considerations.

And then to the right you will see that this particular person had six visits over a 19-month period. On the right hand side you can actually see that the physical function was improving from rather low levels to getting near the national average. And perhaps surprisingly the mental health was drifting downward causing a red flag to be concerned about loss in mental health.

The bottom panel shows the disease status as measured by the Oswestry Disability Index, and lower numbers are better. This person's disease status was decreasing over time.

The bottom left has the patient's expectations met or not, for example, symptom relief. Definitely yes at 19 months. Sleep better. Probably yes at 19 months. Dr. Weinstein, the founder of the spine center, would say I can't be a good doctor without this data. I won't have information I need to formulate the treatment plan with the patient or to judge the goodness of the treatment plan for that patient.

The next slide then, slide 16, gives you an image of a compass for Brian, upper left, herniated disk and for Betty to learn about, upper right. And this was data that is now given to patients in the form of a calculator customized to them at the plan of care and also reinterpreted at the Center for Shared Decision Making. What Brian would learn about would be that if we look at the compass on the left herniated disk, people that got surgery the improvement in physical function, north, was 44. People that did not get surgery, this was a randomized controlled trial of surgery, not surgery, 13 centers, the gain was very substantial, but less, 32 points on the SS36 physical component score.

If we go west, we see the disease burden dropped 38 points for surgery, 24 points for nonsurgery, both big drops, but again the surgery is better. If we asked the person at 2 years, these are data at 2 years of follow up for asking these patients at 2 years how much they were helped by their care 76 percent of the surgical patients felt they helped a great deal, 58 percent of the nonsurgical patients.

And we go south and we can see there was a cost involved that was greater for surgery. These are direct and indirect costs, medical care as well as indirect social cost on average, $25,000 versus $10,195. That is what Brian would learn and would be able to make a decision based other people's prior results and his own preferences.

Betty, if you go to the far right would learn about this. This is our people with stenosis. Betty is about 64 years old. The average person with stenosis is about mid-60s. But you can see that the blue and yellow bars on clinical status changes, functional status changes, and how much the person perceived they were helped by treatment were pretty close and the cost were different, more for surgery, $26,000 versus $13,000. Betty does not have insurance so she would be able to take this information into account in making a decision.

If you will go to the next slide, then wrapping up with recommendations. Go ahead to slide 18. The belief here is that people will need a compass to navigate the health care system. People or consumers or patients are on a journey and they are trying to regain or maintain their health. Their health care needs evolve as their health status changes and as treatments work or do not work and therefore patients need data to answer the key questions about health and about their care experiences and about costs as their conditions change. They need data on health status, their own, and likely outcomes given the database on prior patients like them to match the best care plan to health state and to monitor the impact of the care plan on outcomes, and to make informed decisions about the best treatment options for me given my preferences. They will need feed forward and feedback data on care experiences of others like themselves and on costs.

Slide 19 goes back to this initial thought that my triple aim compass as an individual in a point in time will in fact evolve over time as my health and health needs change. The cardinal points on the compass needed to navigate the care system, health outcomes, disease status, functional status, risk status, care experiences, activation, the characteristics of care, safe, timely, effective, efficient, equitable patient centered the care experience, perceived health benefit, how much was I helped. And then the last major point being costs of care, direct medical care costs especially out-of-pocket costs as was noted earlier as well as indirect social costs related to not being able to work or somebody else having to serve as the care giver, et cetera.

The next slide, slide 20, suggests that we will need a longitudinal measurement framework that connects the patient's experiences over time with data and measures. And here you have an image of the NQF patient-focused longitudinal measurement framework with some of the triple aim compass points underneath. The logic is that we have people at risk and some of those people will move to the right with an initial onset of a disease or a problem such as an acute myocardial infarction or the need for total joint replacement or newly diagnosed cancer, and then follow-up care over time, months, years often times rehabilitation, self care, et cetera. And that we would like to be able to connect the measures of risk function of disease of experiencing cost to people as they move through the progression longitudinally.

The last slide, the conclusion then, is it goes with Wayne Gretzky who was asked how he could score so many goals and he answered that he always tried to skate to where the puck is going to be. If we think out 3 to 5 years and there is a informal/formal Gretzky that has been working this with NQF and many others involved that it would be helpful for us to adopt the longitudinal measures framework, and to populate with triple aim measures, and to create the mechanisms to feed forward the data especially patient-reported data or consumer-reported data as care is given to patients, and to feedback the data on the outcomes of populations of similar patients to support decision making and care program improvement and patient-centered research.

DR. TANG: Excellent Gene. Thank you very much. This has been truly an outstanding panel. Let me open it up for some questions.

DR. GREEN: I want to go back to the question I want to ask Eva and also Judy and Gene ask you to chime in on this. After such a splendid presentation, this is going to sound like such a mundane question. What can the three of you tell us about the consumer's quantitative skill set? Can consumers do percents?

MS. HIBBARD: About half of Americans are what we would say are innumerate; that is, they have difficulty deriving meaning from numbers. Putting meaning on numbers for people is kind of what we need to do.

DR. MIDDLETON: Gene, this is Blackford Middleton. A question for you. Looking at the compasses on slide 16, I thought this was a terrific and very readily accessible, graphical representation of a whole bunch of data, so sort of you know visual representation tough T award for you. I guess what I don't understand from a clinical perspective is when you show the map like this to Brian, this compass, what is the conversation and ultimately how are his utilities or preferences incorporated into this decision-making process?

DR. NELSON: That is a great question and I will also loop back to what Larry just asked. To help an individual make sense out of this kind of data it helps to have several different approaches to customize them to the individual. There is a Center for Shared Decision Making where a person can go to discuss these kinds of information with a live person to see video tapes of people like them who have had surgery or not surgery for herniated disks and graphic representations that are more first patient lay-person friendly.

For example, a way of interpreting Brian's compass might be that for people that got surgery 24 out of 100 achieved substantial benefit in terms of their pain or in terms of their physical function. Twenty-one out of 100 that didn't get surgery 2 years later had similar results.

We work with people who are experts at helping non-numerative people interpret this kind of information. And one of those good approaches is people like you X number out of 100 would benefit versus Y number out of 100.

DR. MIDDLETON: The tailoring for Brian has already occurred and sort of taken the evidence base and customizing it to his specific condition or to what degree has that occurred?

DR. NELSON: It occurs to the conversation with the clinician or clinical team or at the Center for Shared Decision Making or with the use of the risk calculator that I mentioned.

DR. MIDDLETON: Thank you.

DR. TANG: Any other questions? Mike.

DR. FITZMAURICE: Gene, this is Mike Fitzmaurice. Again, an excellent presentation and thank you for your time. In looking at this I see sometimes there is a sequence of a sequence of people who might want to do none surgery for a while and then they turn to surgery particularly with back functions. Is there another chart that says these patients put it off for 2 years and then they went to surgery and they did -- 35 of them had a good result as opposed to 44 who went to surgery right away and 21 who never went to surgery? Is there a sequence like that that should be investigated?

DR. NELSON: There is not, but it would be very good if it were to be very helpful.

DR. TANG: Thanks again, Gene. If there are any overall questions for all three panelists, we could entertain a couple of those now.

DR. NELSON: If I might make a comment on the issue of value for the individual. One way of thinking about it is the health outcomes that they achieve and the care experiences that they have experienced in relationship to their costs, the cost that they pay out of pocket as well as the cost paid on their behalf. I think it is possible to develop a value equation that can work well for an individual as well as for a defined population of people by taking into account outcomes and experiences in relationship to cost.

DR. GREEN: Gene, this is Larry again. Let me ask you another question. When I heard you talk about in your location and with your examples that the person then goes to this center. They go to a center and get help and then interpreting the information and measures that they needed to execute their functions as purchasers and stewards of health care. Are you implying, suggesting, recommending any of those verbs that we need to start planning on having such centers scattered all across the United States for everyone? And part two, are you assuming that consumers are on their own here that they are not going to have a doctor? They just got to figure this out on their own.

DR. NELSON: Great questions. I think the first recommendation about shared decision making is to enable shared decision making in multiple ways and multiple venues so that it can take place as part of the clinical team, clinician, patient interaction. You may have a real center like a Center for Shared Decision Making at the Dartmouth-Hitchcock Center or the VA. You might have a virtual center for shared decision making.

If you think, of course, about what the work that the Foundation for Informed Medical Decision Making is doing and what others are doing to bring together health information enabled answers to critical questions and data and stories and videos and peer to peer as well as professional to peer support. You can imagine having in 5 years some very excellent virtual centers for shared decision making using social networking software using health information, intelligence technology, using knowledge management as part of that.

You can imagine if you look at Clay Christensen's work and Innovator's Prescription and the model that I am speaking about is the distributed model where professional knowledge and people that have a shared concern come together virtually through a rich, Internet-enabled information environment.

MS. POWELL: This is Eva. To follow on that point about the centers I think another element of this -- I kind of see this as a two-pronged issue. One is developing the knowledge base around shared decision making and the tools which some of the entities already mentioned have done some great on and will continue. But the other element of this is knowing how to use those tools and that is a skill set that really needs to be incorporated in all of our clinical training whether that be medical schools or other professional schools because really this is an issue that each member of the care team needs to have some knowledge of how to do. Of course the shared decision-making process is driven by the physician, but then there are others who have input into those decisions and into the decision-making process. It shouldn't be physician exclusive I guess I should say. I don't think we can underestimate the value of that particularly as we are talking about making these changes in the current environment that really does not support this very well. That is something that we just need to begin working on as well.

DR. TANG: Great. Thank you very much again to the panelists for an outstanding panel this morning. And we will break for lunch and resume at 1 o'clock. Thank you.


A F T E R N O O N S E S S I O N

Agenda Item: Panel II – What Information and Measures to Providers need to Improve Quality and Increase Accountability?

DR. TANG: This is going to be the second part of our today's hearings and then we will have a final one tomorrow morning before we have some subcommittee deliberations. We had a wonderful panel this morning on the consumer perspective on information that they would find useful that would address the -- let me use the quality word it is really beyond that, really the whole quality and meaningful measures that affect their health care decisions and health decisions. This afternoon we will start off with a panel that has the same question about what information and what measures would help the health care professional address their needs to, one, understand the quality of the care they provide as well as to increase their accountability.

We have Dr. Fred Rachman, Yael Harris, Theresa Cullen and Karen Kmetik. We will start out with Dr. Fred Rachman from the Alliance of Chicago Community Health Services. And he is on the phone I believe.

DR. RACHMAN: Hello?

DR. TANG: Why don't you give us a few more words just so we can adjust the volume?

DR. RACHMAN: Okay. Is this any better?

DR. TANG: Is there any way you could either get closer or situated differently because it is a little faint?

DR. RACHMAN: Let me just try one thing -- disconnected I will call right back in. Hello?

DR. TANG: That is certainly better. It is better. It is still a bit faint.

DR. RACHMAN: I am not sure what I can do on this end. I am speaking directly into a hand set on a land line.

DR. TANG: Is there anybody else on the phone? I am wondering whether anybody on the phone can hear me better.

MS. KMETIK: Hi. This is Karen Kmetik. I can hear Fred very well.

DR. TANG: We can hear Karen better than Fred.

DR. RACHMAN: How about if I shout? Is that better?

DR. TANG: Let's do it for now. We will give it a try.

DR. RACHMAN: I don't mean to shout at you all but we will try that. Did you receive the slides I sent fairly late?

DR. TANG: Yes.

DR. RACHMAN: Great. I will just try to walk through really briefly. I wasn't sure if folks needed any background on the community health centers because it is largely from that frame that I am speaking. I am assuming that most of you have some basic background, but just to highlight that there are currently more than 1100 federally-funded community health centers and the health reform package calls for tremendous growth in this sector. And it is certainly a major linchpin in the provision of primary health care and not only that has a model of health care that really encompasses much of what we are contemplating in terms of concepts of medical home and a focus on quality, primary and preventive health care. I just wanted to make sure that folks knew that. It is really the claim from which I am speaking.

I am going to skip ahead to the third slide that really talks about the role of health centers and health information technology. Not only is there this background that the health centers have in primary care and medical homes, but there is a tremendous focus and body of experience working with chronic disease management. Informal programs that incorporate pushing out evidence-based practice recommendations and then a set of reporting on quality and outcomes measures related to those evidence-based practice guidelines.

In addition to that there has been now for almost a decade an investment by HRSA in funding clinical information systems evolving to electronic health record systems in this setting as well. I think these are interesting facts about health centers from which to contemplate the questions you have put in front of us today.

Should I press on or do you want me to try to call back in and perhaps have another speaker go?

I would like you to move if you would to slide five. Just to put as background that these health centers again to reinforce that the health centers have been now reporting on multiple quality measures that are imposed upon them. First of all, HRSA has a set of measures attached to community health centers. There are measures attached to these. There are chronic disease programs that I mentioned. Many of the centers receive Ryan White HIV funding which have both core measures to that funding program, may also have state measures, and also have quality measures for centers choosing to participate in quality improvement programs attached to Ryan White.

Different third party payer measures that all of the health care world is subject to now both private insurance and state Medicaid measures as well as individual funding that centers may receive around specific grant programs or other projects.

Just a little word about our organization, Alliance. Next slide. We are what are called a HRSA health center controlled network. We are funded to provide a common infrastructure bridging multiple health centers and our focus has been on health information technology. Early on we were happy to partner with Karen's group at the AMA to look at layering incorporating clinical practice guidelines and quality measures into a commercial EMR, and having had some success at that we are now providing a centrally hosted solution with common data elements, common clinical decision support and common performance reporting that is spanning 28 health centers operating as 100 delivery sites nationally. Again, that is the background from which I would like to share some perspectives to you.

It is going to be hard perhaps for you to see this on paper, but if you look at slide 10 this is an example of a clinical decision support screen that is in our system that marries a practice guideline recommendation to actual structured data entry and reporting that is then used if you skip ahead two slides to be able to publish clinical outcome measures comparing practice sites, health centers, providers could be sliced by health disparity groups against those measures to aid in health disparities' work and quality improvement work.

What I would say from this work is it has given some insight to us about some of the challenges, first of all, against multiple practices if you look at the next slide. Underneath there is a huge change management process to get providers to accept common practice guidelines and to adhere to standardized ways of recording data. And then there must be strategies to make sure that in fact the data of capture methods as designed are actually able to be carried out.

Against that the next two slides really just talk and this is some background you, just some of the challenges that we have as a nation in implementing health information technology.

I would like to really focus on slide 15 which are really some considerations as a jumping off point that I would like to share with you. First of all, following from the multiple measures that our centers are required to report on and I would say that is a proxy for what is happening to all kinds of practice sites out there. There is definitely a need for alignment of measures so that the complexity of data capture all the way through to acting on date is simplified by not having so many multiple variations on the same content.

The second is and this should be obvious is that we are only working with specified measures that are set up to be collected in electronic format, that are validated and have been tested so that we are sure we are putting this effort to good use.

The third is acknowledging that there are complexities of data capture and I will come to this in a minute, but some data can get in very simply but others require very, very complicated work flow considerations in order to capture accurately. And if we are going to try to use and interpret this data, it is important to acknowledge that.

Certainly we all talk about health disparities, but if we want to use any of these measures for health disparities, we are going to have to be very thoughtful about the accuracy and consistency with which we are capturing information about the disparities groups, who we are including, how we are defining them.

From a health center model we are very aware that more than medical care is important in global health of an individual and we really believe that this is part of the reason we have had such disappointing outcomes in house. The pitch that we broaden are consideration of what is included in health and quality measures to include other parts of the health care system; things like behavioral health, case management, other social, sociocultural data on patients that may be necessary in order to make these measures meaningful.

Thinking about those levels of accountability. Each time we think of quality measure thinking at what level that is actionable because the goal of this should really be around driving improvement and reduction of disparities not just measuring performance.

Just a couple of last slides and then I will finish up. I know I am probably close on time. I would like you to look at the pyramid slide which I am guessing if you are not looking at it as a presentation is much too small to really look at. But it really is just meant to reflect the hierarchy of how data gets in. At the bottom we have some relatively simple data. We are having a direct electronic data element, something such as a lab result where if the -- this kind of a big if but if the interface is correct and the order is being captured, that data element is seamlessly captured and available for quality reporting just in the care process.

And all the way at the other end of the spectrum is some kind of quality result where we need information coming in from outside of the particular institution in which the practitioner is based and is now a very, very complex health information exchange or continuity process. And measures like that which may be among the most valuable for us from a system perspective pose tremendous complexity for us to measure.

The next consideration if you go back a slide is looking at how we are defining the measures. Really from a practice standpoint we really want to be pushing to practices, evidence-based practice guidelines. And we want to be using electronic systems to prompt decision making at point of care around those guidelines. But in fact in many cases where the practice is being called upon to report a quality measure, which may be less intense than what the actual evidence-based guideline requires, we may actually be having in some cases decision support that is around the grade or around the performance measures rather than going a little bit deeper to the guideline itself.

And then finally just punching up that point if you look at the next to last slide with the overlapping circles, where do the quality measures come from? Are they system level measures that are coming through a health information exchange? Are they coming from the EHRS which is provider or institution based or should we be looking at some source of data that is coming actually around the patient marrying multiple sources?

The last slide I want to leave us with is just this old thought on the performance improvement world which says that the real aim in measuring quality is not to tell us how we are doing, but to tell us what is important. I would pitch that is we are structuring quality measures for ourselves that we want to have measures that communicate to everyone where we should be focusing in involving our health care system rather than judging performance. Thanks.

DR. TANG: (Inaudible)

DR. RACHMAN: Nothing changed on my end.

DR. TANG: (Inaudible)

DR. MIDDLETON: Dr. Rachman, Blackford Middleton. Really loved your presentation even despite the challenges we had on our end listening to it. I think it is terrific work you are doing and I was particularly impressed with the dashboard ideas and the sort of shared decision support ideas I understand you are doing. I think you skipped over the demonstration slide, but that looks like a GE Office Centricity product, is that right?

DR. RACHMAN: That is correct.

DR. MIDDLETON: And what are they actually receiving from the Alliance. Is it a decision support exchange or can you explain that in more detail?

DR. RACHMAN: Sure. I wish I could be there in person and show you dynamically, but what I endeavored to show was a sample of a screen within the EMR that a practitioner can pull up that summarizes for them all of the practice recommendations, evidence-based recommendations around a particular chronic disease. In this example it is diabetes. And can snapshot very easily for the provider where the patient sits with regards to that recommendation. It is pulling from other parts of that same visit and encounters any activities that have currently been done or have done in the past and without going into other order screens makes it easy for the practitioner to address those care recommendations. It is an attempt to marry the work flow at point of care to an opportunity to review in an organized fashion a particular performance recommendations or practice recommendations and quality measures.

DR. MIDDLETON: That screen design and its construction based upon on the guideline information, is that being --

DR. RACHMAN: I am hoping I was audible.

DR. MIDDLETON: We heard you loud and clear. Thank you. Can you hear me okay?

DR. RACHMAN: Hello?

DR. MIDDLETON: Can you hear us now?

DR. RACHMAN: I can't hear you. Do you think it would be worth my trying to call back in?

PARTICIPANT: Can you hear us now?

DR. RACHMAN: I can hear you perfectly.

PARTICIPANT: It may have been the mic.

DR. MIDDLETON: It is Blackford Middleton again. Can you hear me now?

DR. RACHMAN: I can hear you perfectly.

DR. MIDDLETON: Verizon is getting a nickel every time I say that. In terms of this screen and the decision support you are providing, do I understand correctly? Is that remotely provided by your community or is that something that the user of this application in their clinic is doing?

DR. RACHMAN: I am not sure I understand the question. We have centrally designed this screen. All of the health centers that are using the application access the application remotely. It is housed in a level three data center. But all are using the same clinical content and data capture methods. Does that answer your question?

DR. MIDDLETON: Yes, it does. Thank you. In terms of governance around arriving at those common data definitions and arriving at a common implemented specification of that diabetes guideline, for example, what is the -- can you describe the process at which you arrive at that consensus? What is the governance mechanism there?

DR. RACHMAN: We have a clinical committee that meets and reviews content and guidelines. However, we made the decision using the Harvard Getting to Yes principles that we would base our clinical decision support on nationally established guidelines. Again, for the one challenge that we have is that there is not necessarily consensus on all of the guidelines, but we very much appreciate the work that the AMA has done to try to drive consistent set of measures and drive consensus. I am sure Karen will be talking about that. And in large part we have adopted those measures where they exist.

DR. MIDDLETON: Thank you. And my last question then is to make that dashboard, the Alliance total reports with all the different individual measures lined up and the stoplight summary. It shows in the prior slide that data is returned to a data warehouse in a de-identified form for you to report on. Is that right?

DR. RACHMAN: That is correct and this is because of our -- this is a technology that our commercial vendor happens to have. It is one of the reasons that when we began this work 8 years ago, we chose that product. Our hope certainly is through the certification standards that we are living into an age where all EMR products will have that capability baked on, but currently most EMR products are designed as transactional databases around single patients and very tremendously in their ability to report at this population level native to the EMR.

DR. MIDDLETON: Thank you.

DR. TANG: I have a question. Paul Tang. Follow up on Blackford's question about the use of standardized measures. You said that you followed the national guidelines. The national clinical guidelines are often different or let's put it the other way around. The standard quality measures are often different from the guidelines, the clinical guidelines. Let's take diabetes, for example, where the guidelines still say to manage a diabetic to an A1c of less than 7 percent. The measure is the percent of folks who are over 9 percent. When you say you adopt the national guidelines in your measure, how would you reconcile those two, the difference between the quality measure and the clinical guideline?

Are you still there or can you hear us?

MS. KMETIK: Fred, if you can hear me, it is Karen. We cannot hear you at all.

DR. RACHMAN: Was that a question for Paul Tang or for me?

DR. TANG: It was from Paul Tang to you.

DR. RACHMAN: I am sorry. Yes, that is what I tried to attempt on that slide and I get got confusing with my comment about the AMA. Yes, you are absolutely correct. What we prompt in the EMR is against the measure although that may be different from the underlying guideline. It is really a question that I am raising is what should we really be prompting again. Should we be prompt against the guideline or a measure? And if we are trying to prompt against the guideline, it becomes even more -- that process of driving consensus becomes a lot more complex I think as you were probably getting at in your question.

DR. TANG: Correct. I think Matt wants to follow up and Mike Fitzmaurice wants to.

MR. QUINN: Hi Dr. Rachman. This is Matt Quinn from AHRQ. Thank you for your presentation. I really enjoy looking at this diamond that you put together and calling out what is necessary for these different types of measurement activities and where the data is coming from and what needs to occur to support that. Looking forward 3 to 5 years do you think that the approach for us to measure what we need to measure what is most critical is sequentially starting from the bottom going to the top or where would you prioritize in focusing our efforts?

DR. RACHMAN: Again, my attempts of saying that we should be measuring quality to tell people what is important and not how they are doing I guess what I meant to convey by that is I think right now at this point in time we should decide how do we want to focus this system in terms of getting us to where we want to be 10 years from now. And I think it is totally fine to have measures which are more complex to collect as long as we are certain that we are interpreting that data with caution and we are realizing that what we might be measuring is our inability to capture the information as much as we are measuring performance. I think some of the nervousness about promulgating quality measures comes from providers who are fearful that they are going to be penalized for poor performance and that leads us to dumb down our measures.

I guess my hope is that that pyramid is something we keep in mind and we caution our payers or our funders to be mindful about how we are putting these measures forward.

DR. TANG: Thank you. Mike.

DR. FITZMAURICE: Yes, I also enjoyed your presentation very much. I have a set of questions. Let me just give you the questions and you can choose which ones you want to answer and which ones you don't. Is it hard to get community health centers to capture and report valid and accurate quality measures? Is there much extra man power involved? What is the response of community health centers seeing their own quality measure results compared with other community health centers? And do patients get to see the dashboard? Is it posted periodically where they can see it?

DR. RACHMAN: Great questions. I will try to do all of them. Is it hard? I would say it is no harder in community health centers than in any practice setting. One of my roles now is as co-director of the Chicago Health Information Technology Regional Extension Center and as you know through meaningful use we are trying to push this out all the way through to small, private practices and I would say it is probably even more challenging in those setting than in community health centers. But there certainly are a number of challenges. I tried to capture some of those on the slides which maybe I didn't cover or my voice cut out. Yes, it is very intensive in terms of resources and labor. I do not think individual centers or individual practices are equipped to do this.

And I also really hoped I made the point that we can make it a little bit easier by being more thoughtful about how we select the quality measures and trying to drive some alignment so that we are at least simplifying the field there.

What is the results -- I hope I am not skipping questions here, but what are the -- how do health centers react about comparing data? Well, one of the reasons I am thrilled to work in community health center world is that we have a culture of collaboration and sharing and typically what happens is when health centers see an area where another center is performing better, they seek that center out and start asking questions relentlessly to try to figure out if there are some lessons learned there that can be applied. That by in large has been the response we have seen.

And then your question on whether we are making them available to patients. Not yet. But we really want to. We do publish the dashboards to the health centers. I actually shouldn't say not yet. There may be some health centers without my knowledge that are publishing them on their websites. I know that several of them at least that tack them up in the hallways in their health centers, but so far I have not seen actual proactive attempts to push this data out to the patients.

However, our centers are embarking over the next year or so in developing patient portal technology and it is definitely in our mindset to figure out ways to have these measures sit out there to be shared between the practices and the patients. I hope I hit all your questions.

DR. FITZMAURICE: Thank you very much.

DR. TANG: Any other questions? Okay. Thank you very much, Dr. Rachman, and we will move on to Yael. Yael has taken up a new responsibility at HRSA as Director of the Office of Health Information Technology and Quality. Welcome. I got your title right.

MS. HARRIS: Thanks so much and thanks for having me here. I apologize that I didn't have much time to put this together so I will try to be creative, but obviously not as insightful as Dr. Rachman has been. Thank you.

Basically I aptly titled 2010 a quality measurement odyssey because I think we are really at a crossroads here between where quality measure has been in the past which albeit was pretty advanced considering just about a decade ago we didn't have things that reported on the Internet. We didn't have information driving decision making. I think we have made quite of tremendous strides in the last decade, but I really think we are now with the age of EHR really at a crossroads in terms of what is available to us and also realizing what the limitations are in the quality measures that we have to date. And I think that some of those gaps have been really recently identified because of the move towards meaning use.

For example, most of our measures are disease specific. There is really no consideration or underlying factors or multiple chronic conditions. It assumes that every diabetic is treated exactly the same not taking into account any other conditions that they may have. If those are taken into account conditions like mental health that may affect the effectiveness of some of these interventions and I think we are very unidimensional in our approach.

I also think we have not been very person centric. We have been very disease centric. Looking at it as a disease as opposed to as an individual with a collection of diseases that we are trying to help improve.

Also measures have really been developed without specifying how they are going to be used. And my favorite example is I was actually at what was HCFA at the time when I was in a meeting and we were deciding on what was called the Nursing Home Quality Indicators Project and I was in one room with a group of stakeholders where we said these are quality indicators. They will not be used to determine what is good quality. They will be used to help just identify areas that need further investigation. Down the hall the administrator of HCFA at the time was making an announcement that the quality indicators project would become publicly reported and updated on nursing home compare. I then had to walk out of one meeting into the next meeting and say I apologize. We made a mistake. And these indicators that we just said are indicators are actually measures and will be publicly reported by this time next year. I think we need to be cognizant as we create measures what their purpose is for, what their intent for, and think through how we structure them.

Another concern and this stems from -- I was just at a meeting last week for MedPAC and they were developing quality measures for long-term care hospitals. And someone in the room said well, you know CMS always relies on those averages. That shows they don't know statistics. I was like I designed that -- compare websites. Thank you very much. But I fully acknowledge that.

The limitations of statistics using -- relying on averages and seeking out averages are not really telling us what is good quality. We really need to have a better understanding of statistics. The issue of small denominators using rolling averages to make sure we have a large enough denominator report means we don't really understand these measures and what should be reported and how they can be used to improve quality. I think it is understanding outliers which highlight unique circumstances when you need to use one or two details to identify issues. When an average is appropriate and when an average is not appropriate and when the denominator should be suppressed. Is it because we don't really see it as a quality problem or just that we feel that statistics don't point that direction.

I tried to find a nice picture of a landscape and I just thought this one had more white space for me to type on so I picked this one. No appeal to any specific nation or country. But I just really think as we think through quality measurement, I think the critical thing is thinking about the landscape. Everything has changed since the passage of ARRA stimulus bill and the passage of health reform. Whether or not Congress repeals health reform I think the landscape is completely different and will never go back to the way it was and I think we need to be cognizant of how it has changed the landscape and make sure that as we think forward through quality measurement, we think forward with all these aspects in mind and think through how these are going to impact what we are doing.

The first thing I had was patient-centered medical home and I don't think you can go anywhere now without hearing that word. I don't know if we have a standard definition. Obviously many people use it many different ways. CMS is coming out with their demonstration projects around it but I hear a whole lot of different entities. There is a patient-centered commission that I get emails from literally every other minute. There is a whole bunch of people at the table. I am sure everyone is in it for the best but I don't know if we know what we mean when we say patient-centered medical home or where we intend to be except that we all have the best of intentions.

But I really think it has a lot to do with about measurement because when you talk about care coordination and we don't need to talk about it in the context of one single disease or in context of a single provider. I think we need to think of care coordination across all providers again at the person centered interest. A person doesn't care where they are getting their treatment and a lot of times we don't know where the best treatment can be received. After acute care stay, do they go to a long-term hospital, a patient rehabilitation facility, a skilled nursing facility, a home health agency? Some of the same patients go to all these different settings and we don't know where they are getting the best quality. That is why we need measures that we can standardize across all those settings so we can determine what are we getting. What is the bang for out buck because these centers don't -- these places don't all provide -- have the same costs. We don't know if they all provide the care and the same quality and so we really need measures that can help us determine the best treatment for the patient dependent on the whole patient as a whole not just one disease diagnosis.

Also we can't just focus on disease as a corporeal phenomenon. I think we fail to look at a holistic view which includes behavioral health and oral health. Oral health is something that HRSA has been very involved with because of the fact that we see how it affects health care in the long term. And mental health I have heard championed for a long time but for some reason it just keeps getting missed which is if you cut off the head sure you can treat the body, but we all know how the mind affects everything. If you are treating an illness and you have a mental illness underlying that, it is going to exacerbate the conditions. You are not going to be able to treat the conditions acceptably. So really when we look at measures, we have to look at person-centered approach and consider disease prevention, health promotion, looking at the whole person.

The next big thing is the ACOs. And I just saw a draft of the CMS rule and it hasn't come out for prime time yet. It is still in the department for clearance. The ACOs will be a reality very soon. I think a lot of us are optimistic and a lot of us are skeptical at the same time about what will happen with ACOs. But I think we need to consider them as we go forward in quality measurement because one of the critical features of an ACO will be measuring quality and how they impact quality.

What is an ACO and what is its role in the patient-centered medical home? Is it really just about controlling medical costs? Is it about being a partner to achieve the patient medical home? Is it just about reducing redundant tests? Who is responsible for the cost and who does the measure affect? Do they impact how the ACO gets reimbursed, et cetera? I think these are things that we need consider as we move forward. I don't have the answers. You guys are the experts, but I wanted to make sure you guys are thinking through these things as you move forward because I think these will all come into play in the next decade of quality measurement.

And then meaningful use which is the one I am most familiar with having worked with Paul on the first draft of meaningful use. And I think there are a lot to be said here and a lot that hasn't been said yet. First of all, I think this goes without saying is that we need to have the measures dictated by the need not by what is available. For the longest time we have been going for the low-hanging fruit and that makes sense in a lot of our quality efforts because there is just so much to tackle that why don't go forward with what we can feasibly touch and feel and do right now. But as we move forward stage two and stage three of meaning use only having measures that are easy to measure means we are missing a whole spectrum of the quality and I often refer to quality like a diamond. It has so many facets. And just because there are parts we can't measure doesn't mean they are not important features. And if we don't have a measurement for them, they won't be built into meaningful use and if they are not built into meaningful use no one will ever work on them.

You guys are really in a drive seat about transforming quality in America because if you just look at the facets of the diamond we can see, meaningful use is going to miss the mark tremendously and I think that is a real tragedy.

Going beyond that I think there is unlimited potential in electronic data. I think it opens the door to so many measures we have previously haven't thought about. Just thinking about patient errors everyone says you can only measure near misses. No one reports on the real misses because the patient is dead. That doesn't get figured into the denominator.

I think with electronic health records we have a real opportunity to measure things that were either too burdensome to collect information on or were too difficult to identify such as safety measures, duplication of resources, et cetera. There is really an opportunity to use electronic health records to broaden what kind of information we can collect and measure.

And while it may initially be burdensome to collect it actually allows us to measure things that previously we would say could not be measured because we couldn't collect the data. Now if it is somehow built into the EHR while initially it is burdensome to collect, it is part of providing a quality focus of care so it is something that physicians should be collecting and then something that would automatically feature into the calculation of a measure.

I think it is also the value of real time use of quality data and this has been worked on very closely by a number of colleagues being led by NIH and ONC, but I know CDC and AHRQ and others are at the table about creating a learning health care system so really using data in real time. So not just having a quality measure that will be reported for last quarter or something that the doctor could use say you know last week I had 20 diabetics and I only did a retinopathy on 80 percent of them. It is more than -- it is real-time data to generate decisions.

Forget having someone read every single journal in the United States of America which I have always joked would require every physician in America to spend 2 months of their year just hold up reading through every issue of JAMA and New England Journal of Medicine, et cetera which no one has time to do. But beyond that having real-time education of information so you can generate quality measures in real time looking at data because at that time you would have access to the entire repository of patients to say all right, we don't know what the best treatment is for fibromyalgia. But we know that across America these are the different treatments have been prescribed and this is having the most common effect. It is a chance for actually real-time measurements to be created as we move forward not just prospectively thinking and that is something to take into mind.

Also, the flow of information. I think care transfers have been highly emphasized. It is something that the QAI program focused on in their last scope of work. They are now thinking through their next scope of work. But I think the real critical thing is that having information is a necessary but not sufficient condition to good quality care. I think we can all agree to that. And I think information flow is the one thing that an EHR can give you or information technology can give you even if it is just a CCD. Having information in real time means that you have 24/7 access to information so clinicians can make decisions from their bedside instead of saying just send the patient to the ED because I don't feel like coming in to look at my chart today. It means having people make informed decisions, having clinical decision support at their ready.

I was just in New York and heard a colleague talk about my favorite story was he said it was a dark and stormy night. But basically he had been seeing patients all day at his clinic in New York City which is a community health center that treats a lot indigent people in New York City. And he saw a child come in and he was close to the day, but the mother said I have been here twice. I have seen two different MDs. Please see my daughter. I can't get her fever down. I don't know what is wrong. It is 6 o'clock. It is raining. He just wants to get home to his family. He looks at the patient and he says I see nothing. Low-grade fever. Child is -- there is nothing I can do -- electronic health record and out pops something that said have you checked for West Nile Virus. And he said, you know the sad thing was I entered those 2 weeks ago when the CDC notice came out. I took that CDC notice and I plugged it into our epic system so that those alerts would pop up if someone saw a low-grade fever and these symptoms. And he goes but here I was the one who entered it into the system and it didn't occur to me because I was so tired and it had been a long day until the system popped up.

This is what I mean about the use of technology to really change how care is provided and I think those are things that we don't even realize until we will start taking more advantage of them, but that is the real future and what measurement can offer us.

And then finally innovations. I work on a mobile health work group across all of HHS and someone gave me the example. They said when the cell phone first came out and people carried a purse with a brick in it. They would bring out the brick and bring it to their ear and people were like wow you are talking on the phone. Could that person who was talking on this wonderful new novel device that takes up this much space and is 10 pounds in their purse even imagine this type of device and what this device could do? And it has only been a couple of decades.

We can't even begin to imagine what kind of innovation there is for us in the next decade. I think we need to be cognizant of that as we move forward in measurement because we need to realize what innovations can offer us in terms of health management and health care coordination and treatment.

Then the big one is pay for performance. I don't want this is anything new, but I think it is growing in terms of its role as it expands to multiple care settings and multiple providers and its role in terms of care coordination.

I think we need to think about measures in terms of how they are going to be used. As I mentioned before the quality indicators for nursing homes were never considered to be public reported quality measures and now some of them are going into a pay for performance demonstration for nursing homes that CMS is working on.

I think we need to consider the measures in the context of how they are going to be used. Are they actionable? A measure that we can't really do anything about is nice to know, but it is nothing we can do. We want a measure that we can actually do something with that we know how to improve and what the right steps are. Evidence-based medicine.

Are they measures appropriate for pay for performance? It doesn't mean that they may not be good measures for us to indicate where further review is needed where further risk management is needed, but we need to be cognizant of which measures are good for pay for performance and which measures have a lot of subjectivity and may not be the most qualified measures for pay for performance.

What are realistic goals? In some cases some measures we say they are never events. We know they should never happen and they really should be avoidable. I would say 99 percent of the time they should be avoidable, maybe even 100 percent. It depends on who you talk to.

Other times we know they are going to happen. We know you are never going to get to 100 percent of people having this or that or that. What is reasonable? What should our goal be because that is what we should strive for and that is what we should aim for?

And then we need to talk about what is average? Is average acceptable because if you have outliers they are moving that average up or down? Do we want to say that the average is what we want to achieve or do we want bimodal distribution? Sometimes it depends on different populations. These are the things I think as we think through measures, we need to take into account.

And then I also want people to consider what adverse behaviors might this measure create? Putting this measure out there and having people report on it. Is it going to cause people to act differently? And I will give you an example. Falls. In nursing homes, now a fall is considered anything that is -- where anyone drops from one position to a lower position whether or not they are safely lowered into the position it is a fall. Now you have thousands of people reporting falls all over the universe. Do we think falls are a bad thing or a good thing? If it is a controlled fall, if someone knows the person is at risk for falls is following them and catches them, but doesn't restrain them I would say that is a good thing. If we don't know if a person is at risk for falls and we just tie them down to avoid the problem, we have 0 percent falls in that facility but every single person is tied down to their chair. I wouldn't say that is very good quality.

Taking into account how different measures bounce one another and what adverse effects may happen by reporting on a measure unless we think it through in its context.

The next thing that I want to talk about was population health because this is growing by leaps and bounds and it is really big focus. First of all, the cultural context of population health in terms of how people think about things. One example is reporting of pain. Different cultures announce their pain differently. We have the standard face of scale that people use, but there are a lot of other measures that may be affected by people's culture and by people's ability to react to them. We just think about them in terms of the different populations we treat both by finances, sociocultural phenomenon, location, rural verses urban, et cetera.

Also the availability of resources. One of the things that always get cited are facilities that are high on Medicaid always have worse quality. Is that a factor of their resources or of the types of care they provide or of the types of patients that seek care there? I don't think anyone has really teased that out. There have been papers blaming everything. But it is really a question of how do we create a fair playing field to see what is achievable and what we could achieve in that setting and try to figure out what are the factors that contribute to poor quality so we can work to improve them. And then you need to take into account the type of population that is coming there.

And finally, the implications for risk adjustment. I think a lot of times people over risk adjust to the point where there really isn't much difference, but then at the same time if you don't risk adjust everyone says my place is unique. I have this or I have that or I treat more of this. Where do you find the fine line? Risk adjustment is a necessary condition, but you don't want to use it to the detriment of actually identifying real problems going on.

And then impact on behaviors and attitudes. What is acceptable in one population may be very different depending on the type of clinician you are, depending on the type of care practice you are providing care in and depending on the type of patient population you are serving. Taking those into context as we develop measures is very important.

Just two more slides. I think patient preferences are a critical thing. I know that there has been a lot of work that the NQF has been doing on, the CAHPS, the Consumer Assessment of Health Plans. We really don't have a good measure of patient satisfaction. But I think there are measures of patient experience are critical and it is no longer just black and white in terms of I was satisfied or unsatisfied. There are varying gradations of that. I think we need to take that into account. I think we need to take into account the patient's experiences, the provider's experiences.

Levels of subjectivity. Obviously if mom died, they are going to have a bad estimate of the care provided regardless of whether the doctor did everything for them. How do you take that into account, I don't know but I do think it is something and I think when you are talking about a patient looking at quality measures to choose where does he seek care, this is the information they find most valuable and most relevant, not how many foot exams you have done for your diabetes patients or how many of your patients had an obstetrical exam in the last 12 months. Patients want something that is meaningful to them and so this is something that we really need to develop.

Finally, I think we need to do an increased focus on people with multiple comorbid conditions, not just looking at one situation, but looking at people with multiple comorbid conditions and their experiences and what their expectations are because this is a growing segment of our population especially as we get more baby boomers and we really need to be cognizant of the fact that they are not one disease. They are multiple diseases and their experience of care is very different because they are managing and coordinating through multiple provider types.

These didn't really fit into any rubric, but I just wanted to put these down there. One of the things that we have been thinking about in HRSA although we are not really in a position to say anything, but we are on the board of directors now for the NQF and on the NPP, National Priorities Partnership that we have been putting our word in edgewise is the need to develop scales or indices. So, rather than just one disease kind of a scale or a set of conditions that we are measuring, looking at a person as a whole, not as an individual diagnosis.

Also taking into consideration new technologies and how these affect the care that we provide whether for better or for worse. I think the funniest story I ever hear is those of you who haven't heard it. When the new invention came, I think it was the 1920s, the American Medical Association said this was the worst technology in the world. It was going to separate the patients from their providers, create this disconnect between physicians and patients. Does anyone know what it is? Exactly, the stethoscope. It was in the 1800s. I apologize. Thank you. I knew it was pretty early on. I didn't realize it was that early on.

When you think about it, people are resistant to technology but technology can transform health care. Can anyone imagine practicing medicine now without a stethoscope? I even have one at home just for my asthmatic daughter and I am not a clinician just so I can hear how her lungs sound. Imagine clinicians not using it. Just to think about what technology means in the future because it definitely has a strong fit into where we go with quality measurement.

I also want to consider future changes to health reform whether health reform is repealed. I certainly hope not, but it is on the table. Whether another bill is passed to change some things. We need to take all these things into consideration because it is going to change the landscape and what is deemed important to clinicians and to patients.

And finally, I think the need for flexibility. As we move forward we need to be flexible about what are the different thresholds we want for quality improvement. Are there different thresholds? Are there two deviations above the mean and that is what we are striving for? What are we going for? Is there room for revisions? A lot of times measures have been developed and then they are set in stone. And I know CMS has a quality measurement review process, but it can only do so much and have so much time there. They are stretched very thin. Once a measure is determined to be appropriate for a setting, it goes into that quality measurement standpoint and there are several years before it is relooked at to assess whether it is valuable.

And as we develop new assessment tools we can't revise the measures per se and I know I keep referring to nursing homes, but for example the quality measures that were developed for nursing homes were developed on what is called the MDS, the minimum data set 2.0. That system effectively stopped September 31. October 1 they started the MDS 3.0. It will quite a while before they can figure out how to do measures using MDS 3.0 so we have no measures for nursing homes going forward right now. I think it is being reported on very old data that is irrelevant to the new system.

I think this is something that stands to reason for all of our health care settings because as we move forward with new requirements for certification of EHRs, et cetera, we need to consider are the measures flexible to accommodate those changes.

And finally, the need for evidence-based science. Health reform has really pushed for that. Directly AHRQ is leading the way on that but I think we need to really be cognizant of clinical effectiveness research and relying on what we know is effective before we put out measures that we want to improve on but we don't know how to improve on.

This is my contact information if you have questions. And I am happy to answer questions at this time. Thank you.

DR. TANG: Thanks Yael. I think because of the time, we won't have time to ask questions right now. Perhaps we might have a little time at the end but we have two more speakers. Thank you very much.

The next up is Dr. Cullen, and Dr. Cullen is the CIO for the Indian Health Service.

DR. CULLEN: (mic off) I asked our sites about it. It turns out we are evaluating that at one of our sites. It is interesting to note that is a proprietary product so the site had to pay to use it which I didn't know about. So at the same time a two-question, non-proprietary evaluation. The subject about consumers understanding percentages, we have done lots of focus groups and we have real problems with that. Probably less than 15 percent of our patients understand percentages. That even extends to the concept of BMI, which seems like that should be pretty easy because it is a number like hemoglobin A1c, but that ability to translate a BMI into whether I am overweight or obese has been difficult for our patient population.

The dashboard approach. I just wanted to -- Blackford, I am sure you know this. We actually run our dashboard on a live database at the time of care. We don't export to -- we do export to a data warehouse, but we run the ability to do on the fly queries on a cache database. People have indicated they thought we couldn't do that. We have been running it on databases that have up to 300,000 patients. We actually asked 20 people to not do it in the middle of the day because you can get some efficiency productivity impact, but overall it is fine.

I ook somewhat of a different approach to these questions what you are going to see here today. I do give you some specifics about measurement, but I think that there are some major issues in architecture and design, and I am looking probably not at 2013, perhaps not even at 2015, but really trying to do a roadmap on what I think we should be doing as opposed to continuing on this perpetual quest we have right now to come up with the measures. It is about accountable measurable about health care organizations.

The other thing I should say I do work for Indian Health Service. I have to use every opportunity to plug my little agency. There are slides at the end I am not going to talk about them, but they tell you about us so if you don't know about it.

I went back to the quality of health care for 2001. And also at the end of this is what we need to change because I think that we need to have some heuristic approach to get us out of the chasm we seemed to be diving into. And I just want to step back a little bit about what we are doing in health IT because I don't see this changing right now with what is going out there. It is still driven by visit. It is still episodic. It is still based on reimbursement. I, in fact, moonlight in a private ER. They love our private system because my average EM code went from a three to four and they are getting a lot more money. They are really intrigued by the fact that I am not clear my patients are getting better health care even though this is what I do for a living.

I do think that this traditional perspective insures that the provider sees the data in the way they have always seen it in a SOAP-based format which there are days and God bless, Larry, that perhaps we need to get beyond SOAP. I don't know where to go, but I do think that we are constraining ourselves by saying that the way we have always done medical decision making and outcomes evaluation is predicated on our recording data within that format.

And as we know right now we only measure the data that is contained in obtained from this format. We have measures that reflect that appropriately. IHS has embraced this for 12 years. We have been doing clinical quality measures. We have used what has been available when there haven't been standards. We have extended standards. It has been highly successful. We have great data to show that we have been to have significant change at the point of care, but I think it is limited and I think we need to go beyond this. And I think it is limited because it doesn't incorporate a changing model of health care delivery. And I am going to talk about a measure that I think can help us get at that.

Also, the way we define the measures is really based on the evidence base and available standards and it is not, for instance, reflective of an American Indian, Alaskan native communities. It may not be reflective of other ethnic groups. It may not be reflective of patients with disabilities when we try to get into smaller denominators. What I think we need is malleable systems and learning health care systems. Everybody is talking about that we need quality measures that do that. I am going to go quick because of time here.

I think we need to step back a second and I realize it is not the purview of this committee perhaps, but I think that if we don't step back we are just going to continue on the path where we are going. I think that we need to ensure that the health IT -- I feel like a broken record sometimes. That the health IT systems are designed to facilitate a measure the business process reengineering as well as the outcome. And right now that is difficult to do because the health IT systems are designed to get what we have said we wanted to do which is billing reimbursement and outcome measures.

And that in order to do that we need to make sure that the essential domains that are captured there include that first question that I think Charlie made at NCI has really taught me and I think he is right. Does this data need to be interoperable? We, in fact, ran our pharmacy application. It has lots of data fields, about 250. It turns in only about 15 of them need to be exposed. We can do an API for the whole app but we really only need those 15. The local site needs that other data for things they do, but I think that there are some questions we need to ask and we need to relook at.

The data sources that we limit ourselves to and we all talked about this are really based on what we have traditionally have done and I think we need to get beyond that. The interesting data on narrative medicine, that whole concept if we go back to history and physical really helping us decide 95 percent of the time what is wrong with patients. I think we haven't figured that out well. We haven't figured out how to record that well in the health care system obviously. There is a subjective line in a note. But we aren't getting it. We are not teasing out these domains that I think could in the long run save -- ensure that the health care system became more efficient from resource utilization.

The impact on communication. That was talked about this morning and it is interesting. Once again these slides are really a compilation of my staff and they are on to this communication. They think it is the critical thing. They think it is what we are not doing well. And as you can see they believe that that communication far extends beyond just the patient and the provider. It really goes into the clinic into the community. It goes among the care teams. And because we are predominantly a primary care setting, it extends to how we share data with the specialists that ensures that we can get the appropriate outcomes.

And this morning or maybe it was this afternoon. Somebody talked about Amy, the patient Amy, and what she does when she goes through that health care system. Our docs are that blunt. They are out here. They are sending that patient through that and they can't figure out how to navigate that.

Because of our cultural issues and that we are rural we have many patients that choose to not do things. They are well informed. They are making decisions. I don't want to transplant. I am really okay with not having a transplant. It is an okay decision. But we let them go adrift out into a health care system that kind of pummels them and says no that must be the wrong decision for you and actually it may be the right decision for that patient. We need to somehow have the system do it.

Let me go back to quality measures which are what you guys really want. I think they need to be dynamically designed. It goes to the question earlier about what do you do when there is a difference between guidelines and there is a difference between a measure. If you program that statically which we do for our overall reports to Congress and they want one number, hemoglobin A1c is less than seven, we give them that number. But we also facilitate and allow there to be dynamic measurement at the site where a provider can say I am looking at all this new data on diabetes and I am not going to get anybody below seven and a half or eight. I know that there is a guideline. I will give you that data, but at the same time this is what I am doing with my population. It really begs the question of what happens with reimbursement and the scary thing she does while it is a measure but now it is tied to reimbursement. But I think that it is important that we do that.

We also need to include that there is different data sets. I have been on some groups lately and we have been really driven by what are the major mortality issues in the country. Well, perhaps it is really YPLL. I don't know what it is but the systems need to be able to facilitate that and this framework of specific models of care and Yael talked about this. I actually think that this is critical. This is old stuff to a large extent that traditional patient-centered care and then the medical home which I know is seemingly a novel concept, but I think all of us in primary care have really focused on that for years. It goes back to Paul Nutting with work in community-oriented primary care that is from the ‘70s that we just haven't figured out how to embed in the health IT because once again it is a visit centric reimbursement centric system.

We need community-based quality measures and you can read here. I think it is really important for us at least that while there be national measures that the facility and/or the provider has the option of saying my patients are special and I am running my special population and they are my denominator and I am going to see what happens to them. Because I think if we start saying to providers no, nobody gets to be special especially in primary care, we are going to lose them. They all are special and we all relate to our patients differently. To have us say at a national level that there needs to be national measures I think people are fine with that, but I think that until we give the providers the ability to be able to have some flexibility in how they define their denominator, people will push back there.

I am going to go through those things really quickly. Incorporate safety. These are some really specific ideas that are things that might be important. The longitudinal data view. I am actually going to give you a slide of a new application we just released to prove to you that this can be done. Tag patients with appropriate risk assessment. Some of these are process. They don't all need to be outcome. I am going to talk about that because I think the ability in fact to recognize process measures is really critical. Early sentinel awareness. The ability to quickly identify patients at risk due to blah and the blah is what the system needs to enable and facilitate. That is really the measurement. It is did you recognize H1N1. But can you do that again for something else that is coming down the pike really quickly and we are not going to have time to do it.

This is where I want to spend just a little bit of time. The provider -- I have actually talked a lot about that. How flexible can the system be? Patients confidence and self care. But really this improved care planning and care coordination. We have an improving patient care initiative. It is now at 120 sites. Everybody in that initiative would say you know what you are -- they hate my old measures that I program 10 years ago. I was so proud of myself. But they are like they are important but you know what is more important is I do this intervention and 4 weeks from now I can assess whether it made a difference. I don't really care whether you want that hemoglobin A1c at seven. Well, they do care.

But they want to say I changed patient registration. I changed the patient experience. I want to be able to measure it in 4 weeks. In some ways the measurement is does the system itself facilitate process improvement as opposed to us trying to define what process improvement is. And that reflects that this burden of measurement is high and that we need to make sure that it is passive and that the investment in it is minimal.

Patient centered. These are things people have already talked about today. The one thing that we have really been focusing on is this extending the views to the family, to however the person defines their family, to the community and the population there. mHealth has already been talked about and appropriate patient knowledge.

I think timing gets really difficult and my whole team struggled with what to do here. This is pretty traditional stuff.

Efficiency. I think there are ways to really do this. We should focus on groups. We should be using more of them. A measure is how many patients got an order through group order entry. How many patients got care through a shared decision making process? How many patients got care through standing orders as opposed to saying computerized provider or order entry requires me to hit a button every time somebody gets a Pneumovax. Thank God CMS backed off on that a little.

But I think that if we really want to be efficient we need to look beyond that traditional doctor/patient relationship. And you can see some other options here.

We are also really convinced that there are these regional differences in utilization, that they are really difficult to tease out, that it is not clear. It is just access to care. Obviously the patient and the provider themselves, have a lot to do with that and how to measure that in a way.

Equitable. I think that this one is not as difficult as we have made it. There are five performance measures and if you slice and dice those denominators, do you come up with different access to care? Do you come up with different outcomes? I think that the problem is that the health IT systems have not really been designed to ensure that that can happen.

If we did all that, would it improve quality and accountability? I don't think so. I think there needs to be some other things. I think we need to -- this is probably why I am 5 to 8 years out, I really think we need to start looking beyond and see what is true disruptive health IT technology. I think just continuing to incorporate this formatting giving me access to everything I have had may not make a difference.

I work in a system. I still provide clinical care where the providers are expected to see 30 patients a day. No ifs, ands or buts. That is it. Sometimes it is more than that. In that kind of setting which is primary care what you want to do -- and you are rural so you can't send everybody to a specialist. You have to figure out what is wrong to a large extent. This system is helping and our docs love health IT. They have embraced it. But there is more that we need to offer them.

And finally I just want to show you this. This is just a new application we just released 2 weeks ago. And the reason why I want to show this to you is that -- you know that slide of Amy who went to 26 different places and it was crazy and there probably wasn't any longitudinal thing. I actually think it is not that difficult. I think it is related to design. This is a big funnel. You can throw in whatever event you want. It can be a lab. It can be a procedure. It can be a result. And it allows you to have a longitudinal track. We have not worked out all the glitches. We know it works. But what happens is on one screen instead of having a problem list, it says abnormal PAP smear with 25 things after it because we see patients for their lives. I can go now to this page that tracks everything and allows me to trigger patient-specific reminders that override because they are more conservative than the reminders that are in this system or in fact extend them.

With that I am going to stop. Thanks.

DR. TANG: Thank you very much. I think we will press on. We only have 10 minutes. We can go maybe 15. And we will see if we can catch any questions at the end.

The next up is Karen Kmetik from AMA.

MS. KMETIK: Hi everyone. I am Karen. I am with the AMA and the PCPI and I am sorry I am not with you in person. And I think given the time and I am sure you want to have some questions. I am going to be brief and try to maybe comment on something that I did not hear emphasized by the other speakers who were so eloquent in their remarks - and I know I learned a lot.

One way to look at the question of what information measures do providers need to improve quality and increase accountability, perhaps another item to put on the table is they need confidence in the data and that is true really for anyone who is going to use the data of course whether it is a physician or another health care provider or an organization that is trying to determine accountability or whether it is a patient who wants to be certain that their preferences are recorded. I would just put forward that we need to be mindful that for the data to really be useful in improving care and to be used for accountability. We all need to have confidence in the data.

I will just give one very brief example of how we might want to look at this. There are a set of measures for HIV/AIDS and to the point I think, Fred, you made earlier about these measures need to be aligned. Here is a great example where we did all sit around the table. Our group with NCQA, with HRSA, with the Infectious Disease Society of America we said let's all get on the same page. What is important here for patients with HIV/AIDS? We have a set of aligned measures and I have no doubt that over time those very thoughtful groups will come back around to the table and add measures such as what is the shared gold treatment. What is the patient-reported outcome? I have no doubt that that will be added.

We take those measures and then groups are trying to implement them. We have talked with Lisa Backus at the VA and she is actually integrated those measures into the VA EHR system. And we talked with Michael Horberg from Kaiser and he has actually implemented those national measures into the Kaiser EHR system. And they are both sharing very valuable information about variation across the measures and issues that they are now going to focus their QI efforts. That is all terrific.

But what we started missing though is how confident are we in those data by which we are going to make some very important decisions whether it is right there when you are talking with the patient about what to do next or whether you are sharing information externally for some decision making.

And to something to maybe think about is as we move forward in this enterprise with these great opportunities, how can we also be relaying to providers others confidence in the data that they are seeing, that they are making decisions on? One step is in this case for the HIV/AIDS measure we actually worked with Fred's group who spoke earlier and said can you test these for us to make sure that if somebody who wasn't at Kaiser or wasn't at the VA implemented these measures in their EHR. What should they look out for? Are the specifications written in a way that an automatic query to that system is going to present similar results as to if an EHR in a different location was doing an automatic query.

And it is from that kind of testing that we learn a tremendous amount about what we need to do differently in terms of the data collected in the EHR, what we need to tell providers as Fred said. We learn about what are some of those glitches that we need to resolve as a nation in terms of sharing data or in terms of using standard vocabularies. But I just offer for the conversation that providers and others need confidence in the data and we need a way to continually evaluate measures, test out new ideas for measures. We are putting together a network to do this and I would be happy to share this with the committee after the day here to try to make this happen in a more real-time basis so we don't have measures out there and even the next generation of measures and we don't have a lot beside that a way to help improve the reliability of that data. I think I will stop there and happy to be part of the conversation.

DR. TANG: Thank you, Karen. Open for questions. Maybe one crosscutting question and I am not sure -- we have heard some opinions. Let me refer back to Dr. Rachman saying they use national consensus and I don't know whether these are guidelines or measures. But one of the things that I think that multiple speakers talked about was the difference between what you use internally and public reporting. That is a quandary that goes on in many circles including National Quality Forum, but the whole notion of some of these publically -- these NQF endorsements publicly reported measures are designed to be just that, publicly reported. And they seem to carry a different and someone comments on the purpose, a different purpose and people designed the measures differently. Is the question to improve the system or to ace the test I guess is one way to ask the question? I wonder if the panelists have a perspective on that.

DR. RACHMAN: This is Fred again. I think that we should think nationally about putting out a sort of standards that are going to be based on data elements and content that again will send a message out to what we think is important for people to focus on. And I think it is inevitable that the consumers of these measures are going to include those that want to give grades. But as much as possible I think we should focus on having a set of measures that are in fact aiming at a vision for where we want to be as a nation in 10 years and work hard advocating that those that are using the measures for grades are resting somewhere short of the goal. They are measuring someplace that is going to encourage people to participate and move forward and encourage people not to pull away from the notion of establishing national measures.

I guess also just commenting on what one does with an individual institution, I think the opportunity is there to use the same data elements and the same reporting and set higher bars. But again I would pitch that we make sure that the underlying content the data elements that are populated in the measures are ones that will be useful for those kinds of purposes at the individual institution level. I hope that made sense.

MS. KMETIK: This is Karen. I would just add -- if we work hard at getting the data entered in a uniform way, the next question shouldn't be so problematic. It would then give people the freedom to do local variation as they need to with their patient populations, but also be able to contribute to the national measures which are very important as well. Maybe it is just changing the focus a little bit to say let's make sure we have the data in the EHR that we need for all these different purposes knowing that some are going to be interested in some measures and others and other measures and that is probably okay.

DR. TANG: Let me ask you a different question. Let's pick one disease let's say even though Yael talked about not being disease focused. If we pick one domain, diabetes, would you propose assuming you use -- the same data is available to you. This data is in the EHR, let's say. Would you define different measures that you would use for improving your quality as a provider from ones that you would report publicly or do they think they should be the same measures and perhaps different thresholds as someone mentioned or not?

MS. CULLEN: I can give you our experience with that, but it is far beyond just a specific diabetes measure. It is when we moved from specific diabetes measures to a comprehensive diabetic measure. And what we saw was we went from rates of 80 percent to 22 percent as soon as we combined six or seven things. And what happened then, Paul, was the sites decided what is the most important thing for me here based on the morbidity that I am seeing in my population. Do I see amputations? Do I see dialysis? Do I see blindness? And then they would set a higher goal for themselves in whatever that one part of the comprehensive measure is.

We actually have seen in diabetes at least the whole issue of hemoglobin A1c is concerning. People are backing off from it. We have given the sites the option to not use seven at this point because of the data. Some of it is that we report seven. Some of it is that -- we are taking a hit on it but some of it is that the providers want to be able to say you are not responding quickly enough. The national measure groups are not responding quickly enough to the new information that is coming out. I need the latitude.

DR. TANG: Let me just ask the concrete question though. Would you propose -- this is a roadmap for quality measures, are there different quality measures for inside versus -- you said that there could be different thresholds. You said that you could have some local priority and where you want to focus your efforts, but are you also saying that we can use the same measures, the same quality of measures for quality improvement inside and public reporting outside?

MS. CULLEN: I think you can use the same measure. The issue becomes what are they doing with the measure result. It is not that you can't measure. It is whether it will be putative or negative or you will lose CMS funding if you don't have 50 percent of population of hemoglobin A1c of seven.

DR. TANG: Do the other panelists want to comment?

DR. RACHMAN: I am not sure I understand the question. It seems to me if we are promulgating a set of national measures, those would become internal measures just by default and I don't think we could -- at the same time I don't think we could limit an individual institution from building onto those measures. But I do think that we should be thoughtful that the measures were promulgating at the national level are ones that we are sure will be useful to drive improvement efforts at the individual institution and practice level.

DR. TANG: That is I think another vote for saying use the same definition of the quality measures with respect to some domain process.

DR. GREEN: Paul, I want to shift to another area and notice like Karen did something that has not been talked about before we leave this provider segment of the hearing. It is interesting to me -- I missed it if someone brought this up, but part of the answer to the question is what information of measures providers need to improve quality and increase accountability. I want to make it very, very explicit and call that -- providers need to know for whom they are responsible.

Say it another way. They need an answer to the question who is my patient. And this connects back to what Yael was talking about avoiding measures that are so disease specific. It also goes back to Judy's comments earlier this morning about all of that interesting stuff about the PAM and about how it characterizes the patient, the type of patient that the disease metric is being applied to.

I just want to beat this to death for a moment. Providers who do not have an unduplicated and accurate account of who their patient is are going to have a very, very difficult time dealing with quality measures that are patient centered, person specific, or summed up to population specific. To go to this conversation about the seven and for the A1c. Well, it really helps if you don't get the same patient counted six times in the dataset.

When I said I was going to beat it to death, I was really serious, Paul. This subcommittee is part of the whole committee. Revisited at our last meeting the fact that in the HIPAA law there were some identifiers that were identified as necessary and as we all know we have had all we want about the provider identifier. Thank goodness we don't have to do those hearings again. But we still don't have the individual identifier for reasons we understand. If you don't have the identifier, the use of all these quality measures from a provider's perspective can lack traction. It can lack confidence. It can lack all sorts of characteristics.

Made worse by Judy's comments about how so much of what we need to actually measure quality depends upon who the person is that the measurement is about because for somebody a nine is the greatest success story of their life and for other people a seven is an absolute disaster. We have to know more about the individuals not just the diseases like Yael was saying. You can't do that if you don't know who they are.

MS. CULLEN: I would offer one comment though because we have been in a dialogue with CMS over meaningful use. The other caveat here is depending upon your practice. If you are in a group practice, group primary care practice and you share patients because of your lay out or something like that, the issue becomes how do you report then on those shared patients. Right now meaningful use in 2011 will probably include many patients that are duplicative in the denominator depending upon who is reporting on them.

DR. GREEN: I would accept that comment readily, but in our question we have the word accountability and that begs the question about is it the group that is accountable or is it someone in the group that is accountable. Back to NCVHS work, it helps if you know who the provider is. It helps if you know who the patient is and helps if you know what the problem is. This is so foundational and I am disturbed just a little bit, not a lot, just disturbed a bit that as we ask this question about what providers need that we are sort of hurdling into more details and a level of granularity about this that I am concerned about because it begs this prior question for accountability.

DR. RACHMAN: I guess I attempted to deal with this with that slide saying, where does accountability lie. Is it with the individual provider or practice? Is it with the system? Is it shared with the patient? I would pitch that the real fear here is about how the measure is going to be applied or how it is going to be used, but that if we back away from providing the data and measuring it we are losing an opportunity to identify where the problems exist. I guess that is what I would say.

Health centers operate from Wagner's chronic disease model that says that the patient/provider interaction rests within a whole larger health system. To that extent I think providing this information at practitioner level as long as we are not then beating the provider over the head with what we are learning provides the opportunity to uncover where we need to do better even if it is just doing better with having access to total information around the patient becomes a measure of how we are doing with health information exchange.

DR. MIDDLETON: I have a big question for all members of the panel. I am concerned about -- Karen, your comment I thought was very apropos about the reliability of the data and I am concerned about the reliability of the data because from the get go it is extraordinarily variable in how it is acquired, what we record when we see similar patients, how we say what we see and how it is documented in the record. On the in-bound side it is extraordinarily variable. And then even when we try to support decision making in health care with HIT, the decision support is variable both in its representation and then in its localization with whatever threshold is relevant to particular hemoglobin A1c or what have you.

And the same logic is variably expressed in how we measure the performance delivered. That is a hemoglobin A1c rule has a corollary hemoglobin A1c report that tries to assess based upon who the patients are for the appropriate denominator, et cetera whether or not that population is under good control. We also use different definitions of disease. The way you define disease in the Indian Health Service might be different than how we define it in Boston. The value sets are different, et cetera.

This variability is just rampant. The question is how we begin to chip away at that. What are the relevant standards or what are the high value initial steps to take to try to reduce the unwarranted variability either in recording documentation and observation or decision support or reporting?

MS. KMETIK: This is Karen. I will just take the first crack at that. I would say to shine a really bright light on it just one way to begin to go about it and you are right in that is expediential as we add all these opportunities for error. But shine a big, big light on it. For example, again, if we take a particular measurement set notwithstanding the question about patients have more than one issue, but if you start there and when we talk about it at the national measurement set let's side by side say and today here are a dozen places that are entering data that would be required for these measures and here are the different ways they are doing it. And here are the different vocabularies they are using even though we know the nation is to be moving toward the standard vocabulary. And here are half a dozen other issues. I mean, Blackford, I just don't know how we will chip at it until we put a really bright light on it and step by step try to resolve those issues that we can.

MS. CULLEN: I would say our experience and it is probably why I don't talk about reliability of the data because we have almost gotten beyond that because in 2004 when we said the only way you will be evaluated is based on these definitions of the data. It has to be entered in these data fields. It needs to be consistent. We did 18,000 chart reviews that year and we continue to do chart reviews to document the data. And even though the incentive for us was never financial, but it was improving health status.

It kind of goes along with what Karen did. We made it all public and said do you think your PAP smear rate is greater than 30 percent then put your PAP smears in this way in a standard fashion and lock down the datasets. It is easy for us to do because we are an integrated health system. But it is easy for the software vendors to do also and say this is the way the queries are done. This is the education for the doctor when the data comes in from an immunization registry the first time you do a QA check on it. The problem is I don't think you can -- I think you can accelerate that a little, but that is a couple year processes where the patient and the providers will finally trust the data.

DR. TANG: Thank you both. Any other questions? I am going to turn over it to Blackford with a 10-minute deficit. We are going to have a little break here. What 10 or 15 -- 10-minute break. We will resume back at 50 after 2. Thank you.

(Break)

Agenda Item: Panel III – What Do Professional Organizations, Accreditation Organizations, and Regulators Need to Assess Clinical Performance Across the Continuum?

DR. MIDDLETON: Why don't we reconvene? I apologize for the short break but I think if we can get started up again, we will be able to finish on time, which will be an equally important goal.

Our third panel is all about the professional organizations, the accreditation organizations, and regulators addressing the question what do we need to assess clinical performance across the continuum. And we have Kevin Weiss from the American Board of Medical Specialties, Margaret VanAmringe, I hope I am doing that right, from the Joint Commission, and Rebecca Lipner from the ABIM. And Kevin is coming late. So Margaret, why don't you take it away please?

MS. VANAMRINGE: Thank you very much for the opportunity to meet with you. My presentation is going to be a little bit different but hopefully get to the points also of what we need for measurement purposes.

Just a quick background. The Joint Commission is very well known for its accreditation programs which you can see in the dark blue colored, but I first want to mention we have a lot of certification program because this really bears I think on the topic today. We have about 19,000 health care programs that we are accrediting or certifying today and they do really cut across a continuum of health care.

When you look at certification programs, they are usually unique to a specific medical condition or a particular topic in health care that is very timely like for example, it could be on health information technology, use safe adoption, something of that nature. But the one that we are most well known for is the primary strokes certification that we do and we do all of the hospitals that currently have primary stroke designation.

We also programs for the federal government in lung volume reduction surgery and in ventricular assist device. Those are the kinds of nature of the certification programs, a lot of disease specific or procedural specific type of care.

On the accreditation programs while clearly we have the majority of hospitals who you see down in the bottom left. We have about 95 percent of the hospital beds are accredited under the Joint Commission. We have laboratories. Under home care we have a large number of about 4800 home care organizations that we are accrediting. Under long-term care that also includes hospice and other types of post-acute, office-based surgery. And ambulatory is an interesting program because ambulatory runs the gamut from something like ambulatory surgery programs to school health clinics, prison health, retail clinics like minute clinics that you see in CVS and almost everything else in between. These are all programs that we are seeking to have some kind of measurement for which makes it very interesting you see.

You can see the programs going down the middle of this particular slide. And across the top are the kinds of things that we are looking to have measures in, everything from care coordination transitions of care, patient safety, moving across to comorbidities and other types of areas. These are really the typical areas including efficiency that you are going to probably hear from most of the organizations that are presenting to you. I don't want to go into a lot of depth on any of them.

I will say under patient safety having measures for harm is very, very important because when you have measures of harm and you can look at why harm occurred, you actually can delve much deeper into care processes and get an understanding of many of the causal effects which can inform the requirement standards and other types of measures. It is a very interesting group that I would say needs some additional attention.

We also on the right you can see it is very important to start harmonizing measures for facilities and health care organizations such as what the Joint Commission is accrediting and certifying with physician measures. And there is probably a lot more work that needs to be done in that area.

Some other important areas that I would like to mention is we are now beginning to look at what does it take to measure the new delivery models which are coming out of health care reform. In some cases not out of health care reform, but they are just more ways to try do clinical integration or other types of integration. We intend to have programs in the accountable care, primary care home settings. We have already launched the beginning of the primary care home.

But it is a real challenge given the broad array of participants that we think ought to be in these new delivery models. As I said before, if you want to bring in retail clinics you really need to figure out in accountable care organizations or primary care home, how those are going to fit in.

And we would like to see more attention to patient experiences across the delivery sites in these new delivery models and the management of chronic conditions because so often the physicians aren't talking to each other. If someone has multiple comorbid conditions, it is difficult to get that kind of measure that looks across how well everything is being coordinated and the cost experience.

And I would also say that a very important challenge that we are going to have especially I think in the ACO area is that we are often looking at merging information from different payers across different patient populations and we need to have data consistency so we can really benchmark and be able to look at what we are seeing and I think we are going to have to start facing that challenge right now.

And the last thing on this slide is one of our most important areas we would like to see is measures that are more sensitive to the failures and care transitions. And I say this because as we look at our sentinel event database which has collected a lot of information over the years on serious adverse events what you see is the number one safety failure underneath these events is communication, hands-off transitions. That is the number one. Now, it is not the only one because often if you have failure in communication, you also have five other things. But if you really plotted out what is the one single thing that you could pull out of the majority of these and that is probably about 85 percent of them, it is failure in transitions of care.

The Joint Commission is moving in a new measurement direction that I really wanted to take the opportunity to talk to you about and this is under Dr. Chassin, our new CEO and president. For the Joint Commission we are trying to see if we can't move really as much as the world as possible to a higher bar for programs that do look at accountability in the programs and I will define that in a minute. And we are really hoping that the Federal Government states other stakeholders that are involved in accountability programs will join us. And we are not looking to have less measurement. We are actually looking to improve the measurement so we have better cost benefit for measurement, better results, higher quality of care, and we really have to start now. And that was one of the things I was asked to speak about is what do we need to start now if we are going to have some fruition in 3 to 5 years from now.

We believe that there should be first of all a national goal that can focus on measures that maximize health benefits to patients and examine the current measures. And we have thousands of measures today that have gone through various processes including the NQF processes that are included in national quality programs and that we should use some established criteria which I will offer what we think they should be and we should make sure that we are replacing any poor measures that don't result in good quality improvement with better measures. And we suggest that all quality measures used in national transparency and payment programs be vetted against some specific criteria for accountability programs.

Accountability programs for us are ones that are used for accreditation, certification, public reporting, and value-based purchasing. That kind where there is a clear significant consequence in the public to the metric. Clearly accreditation if you are not achieving the metrics, your reputation may be at stake, ability to participate in Medicare and Medicaid. Sometimes for the Joint Commission hospital programs it is whether your residency program will be able to continue for value-based purchasing updates, obviously financial incentives, medical use incentives, penalties. And public reporting can certainly drive market share and other investments.

It is really important to have a high bar. And the reason we think we can do this now is to just go back a little bit in history. In 1998 the Joint Commission launched for hospitals it ORYX program which was the first national reporting program for hospitals. And in 2002 we said all right. We are going to take ORYX and make sure that all the measures that are used in a core program are standardized so we can benchmark and we started that. In 2004, we began putting that standardized information out on the web and CMS was following close behind in its reporting program. And in 2005 using these measures went live with the reporting program which we were very pleased about.

You can see if you look at this graph in 2005 to 2006 when CMS decided to make public reporting part of its requirements, you can see the large increase we had in measures being complied with. The reason for this I think is that between 2002 and 2005 while we were collecting data at the Joint Commission and we were publishing it to some degree, the consequences we not so great for people. And it wasn't until they went on a CMS website the people said I better look at those data and see if they were really accurate. You saw a big charge in the reporting and the compliance.

And in fact if you look at 2002 the overall performance that we had with measures that we believed meet these criteria which I will get to in a second was about 82 percent. By 2009 it was over 95 percent. This has allowed us because of the amount of experience we have had to look at which measures are working and which ones don't. And to decide that we really need a more formal process than just the Joint Commission's activities to assess that experience, learn from it, and act on it. And we need to really find an approach to do that.

We put out recently this year -- Dr. Chassin in the New England Journal wrote up his thinking on this and published it. You can see here there are four real attributes to the measures that we think we really need in accreditation and other oversight programs.

Here I am talking about process measures. I am going to get outcome measures in a minute, but we are talking here just about process because that is primarily what we have been working with. We believe that these are the four attributes that oversight programs, value-based purchasing programs and other types of public reporting programs should be adhering to. We need to really evaluate the evidence to make sure that the research behind the measures is one that shows that the specific processes that are being measures actually relate to improved outcomes. That there is some accuracy here that it captures the evidence-based care process and I am going to describe that later a little bit more when I show you some examples of ones that don't. That there is proximity. That is that the process is more approximately close to the outcome and not a very distally related to the actual patient outcome. And that that there aren't any adverse effects or little or few adverse effects to actually implementing the measure.

The measures that we are looking for need to have clinical integrity. Well, we have seen is that when clinicians believe that the particular process measures they are using actually will lead to better performances. You get adherence to the measures. The collection of data is better. It is more accurate. Physicians believe in it. And if they don't they really turn away from the improvement efforts that need to follow the measurement because obviously we are not measuring just to get data. We are measuring to try to drive improvement from the measurements. And it often leads to workarounds and a lot of wasted effort as organizations try to pull resources on a measure that turns out not to be that clearly important.

Some examples of accountability measures, again, process measures that we have seen in our programs that have clinical integrity that meet the four criteria that I mentioned earlier are the aspirin at arrival, beta blockers, ACE inhibitors, the antibiotic prophylaxis, and the new perinatal measures that we are putting in place. And all of these meet these criteria. These are all Joint Commission measures. They are also used by except for the new perinatal measures by CMS. And we think that these are the ones to focus on.

The non-accountability measures that have been in our system that we are removing from our system are things such as smoking cessation counseling and heart failure discharge instructions because they do not measure the actual process. They just measure whether some box was checked often on a form. There were workarounds for those. And you are not really looking at the quality of the discharge instructions or the quality of smoking cessation counseling. Oxygenation and left ventricular function assessment are very distal to the actual outcome of the patient in surgery for care for left ventricular. And the last two -- they actually can have some untoward consequences and we really saw that with first dose of antibiotics with people getting antibiotics who really did not need them.

What we are doing now with the Joint Commission is and I think there are four of these at least -- is we are going to be including the accountability measures. We are trying to work them into our accreditation standards so that now that we have seen that people can actually achieve a very high compliance with these and as you saw from the slide earlier there are a large number of people who are achieving over 90 percent now. We are trying to look at ways to bake those into our accreditation decisions. We want to work with other stakeholders like CMS to eliminate any measures that don't meet the accountability criteria, again, process measures. And include only accountability measures in our program going forward and we will also try to provide evidence for organizations and help organizations understand the evidence behind the accountability measures.

And then lastly because we don't believe that measures are just ending at the measurement stage. Our Center for Transforming Health Care that we created a year and a half ago will help organizations improve on accountability measures and will offer solutions exchange which is a web-based tool with solutions for achieving higher reliability on the measures.

We are going to also try to create a recognition program to further reward those who have top performances on the accountability measures.

Our next step is to go to outcome measures which we have not done yet, but that will be what we are working -- that is what we are working on now. Clearly clinical outcome measures are going to require risk adjustment and I think we sometimes have a long way to go in risk adjustment on many measures. That is a challenge that we have moving forward. The measure obviously must capture the outcome and have minimal or no adverse effects. Some excellent sources of outcome measures with clinical integrity that we have seen, or the patient experience measures that we have been using so far in this country for hospitals and others, functional outcomes for congestive heart failure and joint replacement.

The cardiac surgery programs listed that some states have, STS' registry and American College or Surgery NSQIP. These are all good outcome measures that we think would meet good clinical integrity standards, but we really need to also move into this outcome area.

There are some problematic mortality measures that are out there that are outcome measures that we need to take care of. And what happens is we often see that the risk factors are not using the factors that are really influencing a patient's risk of dying. You can see, for example, that sometimes there is misclassification of hospitals using mortality measures when you come back and study it. And very often the crucial clinical factors that are known by clinicians to have a major impact are missing and I think this is a very important point because if we can't get the clinical factors right in the measures then we are really not going to have the buy in of the people who need to do the improvement and this is a very critical issue.

We have seen, for example, in Massachusetts recently, I am sure you saw this summer, when they tried to look at a mortality measure for hospitals and they used different types and evaluated different types of vendor systems. They finally somewhat threw up their hands and said there really isn't one that we can use at this time because all four were giving different results.

This is very important. Even Medicare's Pneumonia Model and we certainly applaud Medicare for going down this route. But if you look at a Fine article that was printed in the New England Journal a number of years ago and this is not -- this has pretty much stayed constant. These are very much the important risk factors for treating pneumonia patients. And yet if you look in the Medicare model only two of those risk factors are actually included. We need to figure out ways to include more of the risk factors that we need to do that.

That is on clinical issues. I just wanted to end on this particular note because I can't leave without saying this. We are a little bit concerned at the Joint Commission and that sometimes the move to putting measures into the electronic health record are threatening the integrity of some of our measures and we have seen that with the VTE and the stroke measures. When they have been simplified to go into the electronic health record, some important clinical information gets left out. And we really think that we need to do a better job nationally in conducting some type of evaluation when we go to eSpecifications of whether or not we have changed significantly what the derived measure is. This is something that I know maybe too much in the weeds although I am sure I am getting a lot of people's -- you have been dealing with this probably form other people.

We are looking at ways that we can also create mapping to some of the language that is currently in the eHealth records which uses SNOMED, LOINC and RxNorm, but some of the information that we may need to get in for the measure may not be captured in those languages. Can we collect data in another way and map it into those languages and get some type of consistency and reliability there? We would like to make sure that at the national level that as we move to meaningful use phases two and three, we look at the measures that we want to use and make sure that any EHR specifications have not damaged the measure themselves and what kind of work can we do to make sure that that hasn't happened if we want to use these measures going forward because I am sure there are fixes. It is a question of getting the right fix.

We also need to resolve a couple of questions. Who is the measure steward when the eSpecifications are not created by the original steward and that was the case with the Joint Commission measures? We were the measure steward but somebody else did the eSpecifications. And then now who owns those because we didn't do the e-specs but we are responsible for the actual measure itself that is derived from the medical record abstraction process. And where is the locus of accountability nationally to fix these problems and to make sure that they are getting the attention?

We will be raising these things to ONC and others, but I wanted to put them on the plate because it doesn't make sense to have really good clinical measures and then find out when we get to the measure collection process that we are not getting the information. Some of the clinical nuances that we really need to make sure that the measure is accurate, maybe lost.

I am sorry I really raced through that.

DR. MIDDLETON: That was terrific. Thank you very much, Margaret. And we will take a few minutes for questions now.

DR. TANG: Thanks, Margaret. Can we drill down on this question because that is actually the question we want to ask is how do we use new sources? We are technology agnostic, but how -- assuming that these data exist what would the desirable measures look like? Your question about how you lost some of the integrity when it got converted certainly concerns me. I think there is a little caveat though, for the short term because of the meaningful use criteria that have to be put into place yesterday and then now tomorrow. They did retooling, so-called retooling of existing measures and I can see how there is some fidelity that could have been lost. Can you go ahead and go through the VTE example so we know what was lost and can imagine or see how we would deal with in the future how we would not lose that. Presumably some of the issue is not wanting to continue with the manual review because that is high cost and you can't scale that and the other piece is the number of exclusions that are in these definitions that contribute to the high cost.

MS. VANAMRINGE: There are so many exclusions but I think also there is what we see often being lost is contraindication information of that kind of clinical nuance. I am not the expert at the Joint Commission I will tell right now on the weeds and the measures. When people talk to me at the Joint Commission, I think they are talking a different language than I speak. What I would love to do is get back to you. We do have a list and I think some slides on the particular things that got lost in VTE and stroke and if I could submit that to the committee, I would love to do that. But I may not be able to go down too much in the weeds on the specifics.

DR. TANG: That would be very helpful. Thanks.

DR. MIDDLETON: If you were supplying that information in another deck or if there is another way to get this is a related question. I guess if you think about the measurement processes and the evolution towards standards-based measures and then eSpecifications which has a very rigorous specification, if you will, with current terminology standards and the like. What would be your preferred standards for those measures? And then when you think about that, how do we trade off what is available in the EMR again in a technology agnostic way, but some things will be there and some things won't be. How do we get the information that is not from an IT source into an IT source?

MS. VANAMRINGE: I think there are probably ways to map some of the information. If some of it is in administrative data but we don't have a way to actually get it so that it can be linked into the value sets that are in LOINC and SNOMED. We could probably do some kind of a translation to get that done. I think it is an eminently solvable problem at least for some. I don't believe there will be perfect congruence, but we may not ever be able to live with perfect congruence in the near term. That is something that -- there are tradeoffs, but our think our concern is that we don't know right now unless we start evaluating this how much did we lose and where are those bias direction. That is our concern and we believe there needs to be some investment in doing this testing and then seeing if different mapping strategies will get us to a higher level of fidelity.

DR. FITZMAURICE: Mike Fitzmaurice. When you get down into say the weeds, does the Joint Commission specify the vocabulary for the quality measures, that is, for the variables reported the name, definition, how it is represented? You might say here is a set of ICD-9 codes that we want included, but these ICD-9 codes should be excluded and maybe it is the same thing with some procedure codes. These count. But if somebody has had this procedure could, we don't want to count them in the denominator there in exclusion.

MS. VANAMRINGE: Yes. That is absolutely true and that is one of the things we could map actually is the ICD-9 codes and make sure that what we need in the measure gets mapped and see how close it could be to one of the values that is in there already and most of the EHR system. Absolutely. We get down to the specifications. We have data dictionary. We spend a lot of time on that and I think that is one of the areas that we can be very helpful in doing --

DR. FITZMAURICE: A second question. Are you preparing for ICD-10, that is, here is what we have for ICD-9. Now if we were to go to ICD-10, here are the codes that we would include and exclude.

MS. VANAMRINGE: Absolutely. We have been doing that for a couple of years now.

DR. FITZMAURICE: Very good.

DR. GREEN: Margaret, thank you for really clear presentation. A quantitative question. What would you estimate to be the total number of measures that the Joint Commission uses to assess clinical performance across the continuum?

MS. VANAMRINGE: Across the continuum that is -- we don't have measures right now that go across the continuum. That is part of our concern. Our measure work has largely been in hospitals and then the other areas where we have measures are, for example, primary care stroke certification where there is a measurement requirement specific measure. But many of the other programs were using measures that were already other people's measures such as the MDS-derived measures for nursing homes, oasis, for home care. We have not really created and this is what we are looking at right now for accountable care organizations. What measures do we have to create that really go across this continuum and we are also looking in the post-acute area not necessarily for ACOs, but just what can we look at in that area. We don't really have good measures now.

DR. GREEN: I think the answer to that was zero.

MS. VANAMRINGE: I would have to say zero if it were me. Yes.

DR. GREEN: So let's back up a little bit. Do you have any opinion or insight into of all the measures that the Joint Commission uses that are site specific, location centric? Do you have a guess about which ones are applicable in other settings or cross settings?

MS. VANAMRINGE: Cross settings?

DR. GREEN: Or is this going to have to be a slug fest place by place, site by site?

MS. VANAMRINGE: Yes, I think it would have. We are just beginning to pull that kind of thing together. It is very difficult because now that we have a certain inventory of measures we are beginning to look at where do we apply them and to what type of organization. But I would be hard pressed to give an answer to say how many in that category.

DR. MIDDLETON: Two quick questions. Matt.

MR. QUINN: Something that as we look ahead to things that -- I have used you guys as an example several times, as someone who has done a really good job in specifying measures so that they are applied consistently. Back in 2001 and 2000 I was a product manager of GE's core measures in ORYX stuff and I went on to Quantros who does a whole passel of that stuff today. And one of the things that was really refreshing as a product manager was that we were given specific ways of implementing those measures in the systems down to the very detailed level and there was a testing system that accredited and certified that we were measuring things consistently on behalf of our clients.

Two thoughts. One is my product development folks said we currently measure some of the core measures slightly differently in our system that is used for quality improvement internally. Should we change it to reflect the core measures? And I said yes, absolutely, even though that meant it was different than what it had been in the past and we had a lot of benchmark data on it.

The second was the question is all of this going to be electronic data that is supporting these core measures. And at the time even though they were administrative data that we were sourcing, we had to create a system that allowed people to go collect data from paper charts and actually enter it into the system. There it was electronic in some form, but it looked both at that and I would just propose that as a model that could get us over the hump on electronic health record data in addition to this.

MS. VANAMRINGE: I agree. And I think you raised the point of data collection standardization. That is really critical. When we originally started out with our perinatal measures, we saw that you could specify numerator and denominator. You could have a good data dictionary, but you actually could collect the same data about 20 different ways and that does not make for I think reliable information which is one of the reasons we are concerned about when you start getting into accountable care organizations and other delivery reforms. How are you going to make sure that those data are collected in a standardized way?

We would also like to be able to take many of the measures that are coming out of other sources whether they are state-based reporting resources and HSN, other types of data collection sort of tracks registries and so forth. But the issue still comes up what about validation. What about data accuracy and how do you really assure that if you are going to have a program accountability where somebody's finances and reputation are held to a high standard on those particular things?

MR. QUINN: Just as a super quick follow up. One of the things in the model that I described with core measures is that there was vendor intermediary in that and as somebody who saw what was coming from hospitals towards this and it wasn't consistent and it wasn't -- many times there were many iterations in going back. Do you think that that is necessary if we are going to have quality reporting?

MS. VANAMRINGE: We have talked about that a lot within the Joint Commission. The vendor system has evolved over time. There are fewer vendors than there used to be. But right now we feel for data accuracy that is the best thing we have. Eventually obviously everyone wants to move to a different system, but to me I see very little attention paid often to the data infrastructure in this country and how are we going to get information especially if we are going to start aggregating from different sources in a way that is consistent and reliable. I don't think we do nearly enough data validation. At least the vendors are held to data collection methodologies and they have to be audited. I feel that right now information that is coming into perhaps value-based purchasing and meaningful use needs to potentially have a greater audited capacity.

DR. MIDDLETON: Just one last question. I don't feel we can leave your presentation without the following. In a way we think about accreditation and all the usual process outcomes manners that you have been discussing. And I guess when we think about where meaningful use is going with increased attention on clinical decision support, do you feel the Joint Commission will ever be looking at the quality of clinical decision making itself or perhaps secondarily the quality of the knowledge bases underlying clinical decision support systems perhaps warding off the FDA investigation around regulatable devices and what not? What would be your --

MS. VANAMRINGE: I would say more the latter than the former. I don't really see us evaluating individual clinical decisions, but more sort of the infrastructure below that and what goes into that and I think clinical decision support is just critical and ways to really improve that. I see us perhaps getting more and more into that area as we look not only solutions for health care organizations, but we are looking at where the failures are and often there is an information failure.

DR. MIDDLETON: I am sure we could go on. But thank you very much for a terrific presentation. Welcome Dr. Kevin Weiss. Blackford Middleton. Nice to meet you. And why don't you go ahead and take it away if you are ready.

DR. WEISS: Also I see that you have Dr. Lipner joining you today. Has she presented -- I don't want to preempt what she might want to talk about with -- certification and board certification. And I have my presentation ready to load up. However, it may be more helpful to maybe just give you the presentation for the record and actually just enter in a conversation with you, but I can do either way if you would like because you may have some very specific questions about this whole credentialing environment as seen by the physician community that you may get a little bit of when Dr. Lipner presents to you particularly -- I am assuming you are going to be presenting from the ABIM sort of experience of that maintenance certification. Great. Would you be willing to -- you want me go with the fall presentation and leave that here or do you want me to actually maybe just give you a more --

DR. MIDDLETON: Why don't you go ahead and step through the presentation and be use as much of it as you would like, but I also think the conversations that we have had all day long have been extraordinarily valuable. We have time for that as well.

DR. WEISS: I will step quickly through the presentation and then I will give you some time just to maybe cue up some thoughts. I just want to start by saying that it has been -- for me this is kind of nice. It has been about 20 years since I last spoke in front of this committee that you are now members on. I did last time when I was an NIH fellow here at the National Center for Health Statistics. It has probably been about 20 years since I have been back in this part of time. It is kind of a treat to be back here -- big round circle.

I come to you in my role as President and CEO of the American Board of Medical Specialties, ABMS. I assume that what we can do is -- if I say next slide or something like that. Just to give you a sense of that framework which I think most of you know but at least you would see it from our perspective is we have the regulatory side, the legislative regulatory sense of physician licensure which are the state medical boards and then we have this national -- and those are state based. And we have this national program of credentialing that was started over 80 years and ABMS is now 75 years old, 76 years old where the profession said it needed to create a cross the board standard that it could use to define for the public what defines specialty medicine, specialty-based medicine. And over the years accreted to the American Board of what used to be called the Advisory Board of Medical Specialties eventually became the ABMS, American Board of Medical Specialty. From a looser affiliation to a more formal affiliation with standard setting obligations the ABMS was created and these are the boards that exist in the ABMS.

ABMS now through its 24 boards oversees the certification and the continual recertification of 750,000 physicians in the United States which is about somewhere about 80 percent of the doctors who are practicing right now carry one or more certificates from ABMS. We are now the only certification branch that has a national program. The osteopathic association, the AOA has their version as well as there are scattered -- I think we have lost count at 50 to 100 other individual places where you can get a certificate that would give you a sense of being board certified, but it is not an ABMS process for sure.

I like to use this framework when I describe the world that we live in versus what else is going on in terms of the accountability and transparency in health care quality. It really shows what I call the three tectonic plates of accountability that are going on. One your left side is this physician accountability both driven by both the regulatory which is the licensing boards, but also the self regulatory, the physicians. Now mind you the licensing boards are predominant physicians often have public members. The certifying boards are predominantly in some cases near exclusively all physicians.

And then we have these two other accountability frameworks which we are going to talk about both the purchaser and the consumer and I would like to talk a minute about both of those because that is what is colliding here and the need for data underneath all this is colliding as well.

For the new world with these tectonic plates is kind of evolving. It is being driven by purchaser and plan needs driven by the big purchasers including the Federal Government. If you tape once, you have this whole consumer reporting health plan reporting and part of what you have been talking throughout the day on these. And then what you have on the next is maintenance certification. Maintenance certification started behind that firewall that I showed you in the last slide, but really is now beginning to reach out through interest and transparency to serve these needs of the market place in better ways.

And right behind us is as of spring of this year you will see maintenance of licensure which now is official. Have you had a discussion from anyone from the Federation of State Medical Boards? MOL, I will talk with you a little bit briefly. Kind of view it as a program that is very reminiscent of what I will show you for MOC, but a little bit lighter in terms of its obligations and there are obvious reasons for that.

On the purchaser accountability you probably know this better than I do so I am just going to blow through this next slide. I think there are like four or five bullets and if you can stop before you tap again. It is all about this report card. It is the idea that you can actually aggregate data and create a performance measurement, numerator, and denominator. It can meet all that performance requirement for the science of performance measurement and you can report that back. And the purchaser has been doing this now driving the insurers and other people to do it. It has been administrative. It is what Arnie Epstein likes to call munching on administrative data, but really nothing more. We need to clinically enrich it in some fashion. They have put on hemoglobin A1c and blood pressure is attached to it. But it is a very fragile, rudimentary sort of system to actually look at medicine.

The other problem is the lack of uniformity which you are all familiar with. I imagine is part of the interest of the committee. And the fact that the robustness of the data to look in granular fashion you just can't get there because when you start drilling down individual physicians, you really start to get into numbers problems which drive any statistician crazy.

On the Federal Government you probably know this better than I do, but we have all the versions. But for the physician it is really resting around PQRI which started out as a truly voluntary program, the physicians voluntary reporting issue of PVRI. It became PQRI. Incentives are positive. They are going to negative. Eventually you are not going to make as much money if you don't follow them. But the really limited data and CMS is struggling in a very positive way to try to make this more worthwhile. The physician community has not bought into it because they don't believe in the data. The entire surgical community doesn't hold ownership to it because it has these skip data which are really evidence based but don't really lead to individual surgical accountability. There are lots of problems with it. They are trying to fix it by putting in registries. But ultimately it is this carrot that is becoming a stick that is going to drive people to it even though if it is an albeit imperfect system. We are beginning to align with it.

I think there is one more bullet here. And this is what happened in the past 4 or 5 weeks which you may or may not closely aligned with and that is CMS led a series of contracts, five of them, which are to develop efficiency measures and that was a big release into the world of performance measurement. ABMS is working on one of those contracts to develop those measures working with Brandeis University and the AMA's PCPI.

Consumer accountability is what it is all about because I think so much of what we are doing in performance measurement rests on that. The boards are beginning to try and figure out what does it mean to be more transparent because historically we have said board certified, yes or no. Now we are talking about board certified and participating in MOC, yes or no. Is that enough for the public or does the public need more information to make good decisions? Do other people need more information? That is a key point I will come back to at the very end of this presentation.

What we find from consumer surveys done by Gallup poll and such is that the public actually think that board certification is really important. They tend to want to have a doctor who is certified. They say that they would leave their doctor if they weren't certified, but they don't always know if the doctor is certified. It is kind of a confused moment. And if you ask them and we haven't done this yet, but we will on new surveys ask them. Do you think that your doctor is recertifying? They are going to assume yes which is a big assumption because only about half the doctors in the country are actively going through recertification activity.

We are left with the consumer movement that I like to think of as the wild, wild west which is any information is good when you haven't got good information. You just suck up anything and you end up with things like ZAGATS which is being supported by one of the health plans which will go unnamed as a way to support consumer information. It is rating your doctor kind of thing. A, B, C, and D. Now the boards have been struggling with this issue and should we as part of a transparency talk about rating doctors. I think we keep on coming back to a clear no. And the reason is there aren't enough doctors in the United States and if you start taking doctors who are just below average and telling people they are just below averaging, you are going to be driving more and more patients to a smaller and smaller workforce that is going to drive everyone crazy. What the answer is the way to solve the problem of quality is the old adage to raise boats, raise the tide. I didn't do that right, but I think you kind of know it. Rising tide raises all boats I think is how it goes better. This doesn't work for us.

We began to form -- we, being the boards, began to formally explore what it means to take that physician accountability plate and start talking with these other two plates because we had a very large meeting with NQF co-sponsorship about a year ago and the results from that will -- the proceedings from that are going to be released in the next month or so. I would be happy to forward those to you. But basically they said and this is the quotes from -- not formal quotes from Janet, but I will paraphrase it. That there was real clear understanding that the credentialing environment created by the boards is essential to health care quality in this country, but it is insufficient particularly if we don't align to the other activities that are going on. If we have a misaligned plate in those three plates, that it not helpful to the public. On the other hand, the other two plates really can't work without us. It is really kind of a new dialogue, new day.

And then of course health care reform really does drive it home. You probably know these again better than I do. These are the five sections of the ACA that talk about quality in one way or another. This is the first federal health legislation that really does tie quality into performance in the health care system and they do it in a number of different places in here, one of which is related to us. I will get back to that shortly.

What do we do on the credentialing side and what do we do on the licensing side? Well, the Federation for State Medical Boards is a similar organization to the ABMS except it doesn't set standards. The FSMB brings their state licensing boards together and suggest best practices and they have suggested one now call Maintenance of Licensure. I will get back to that.

ABMS, on the other hand, brings our boards together and has the authority from those boards to actually set standards. For example, the biggest thing that ABMS does is it states whether another board can come to be or not, whether specialty can come to be. It also set standard for the recertification program which is very important as we go forward.

Maintenance of Licensure after what Hank Chaudhry who heads the FSBM, my counterpart of the FSBM -- so to speak states that after a 7-year labor and delivery they have delivered MOL, Maintenance of Licensure. And that happens spring of this year at the FSBM's annual meeting. They voted it in. I believe it was unanimous or unanimous minus one territory. I can't remember which one it was. Overwhelmingly support.

It has three parts and you will hear me talk later about MOC having four parts. The three parts have to do with education, some sort of objective assessment, and then some sort of practice improvement activity. Education, which they are translating into CME, some sort of objective assessment, which they haven't figured out yet and then the third has to do with performance activity, improvement activity, which probably is more like a PI CME, practice improvement CME activity.

The next one is what is MOC in that light? Well, MOC is four parts. The first is making sure you have a valid license and now emerging, developing standards of a patient in peer survey that will be appearing as part of a professionalism piece. And then it has education. An objective assessment, which is a secure proctored exam. And then it has a practice improvement activity that is more formally defined. The three parts of MOL and the parts two, three, and four of MOC are the near equivalence except the standards for MOC are set much higher than they currently are for MOL. It is a higher bar that we have currently.

Here is what MOC is. It is maintenance certification program. It was set up to promote lifelong learning for the specific point of improving patient care. And then we have these six competencies. It is not just knowledge. It is not just practice. It is also these other element system-based practice and professionalism. Elements of care that really comprehensively define what you want to see in a good evaluation of an individual physician.

The four components I went through. And if you look at components two, three and four which are listed there, that is the same if you would renumber them as two becomes one, three becomes two and four becomes three of MOL near equivalency. Again, the bar higher right now in MOC than it is being set currently in MOL for lots of obvious reasons.

The question is why four parts, and why not just do performance measurement. Why not make it easy and do what NQF is suggesting we just take the performance measures and bring them into practice? And the answer is you can't evaluate a doctor and performance measurements alone. I can do as a primary care doc you can measure my hypertension and you can measure my diabetes care, but to get to thyroid nodules on me in terms of do I know what to do with thyroid nodules, you don't have the numbers. You don't have the denominator to get your numerator on that. If you want to know how I manage stiff neck and fever, it isn't going to happen through a performance measurement. The frequency is too low. You have to find other modalities for assessment of a doctor and to get performance in it. In addition, you are never going to measure professionalism through a performance measure that looks like an NQF measure. You have to look at different ways.

The other piece is you need to know that the doctor has a certain amount of core knowledge and knowledge we know decays with time at least my knowledge decays with time, unless you keep on refreshing it and refreshing it and refreshing it. This comprehensive evaluation of physician is where we are after. One part of it has to do with the performance measurement work that you spend a lot of time I would think considering.

I wanted to show this slide because it says we are only 10 years young. It was 10 years ago that we decided -- we, the board, decided to do this complex thing. It has only been about 4 years since all 24 boards have engaged in their first cohort of doing it in terms of actually implementing it. It is a very young science on our part and we are learning and one of the criticisms people have of us is it all looks very different from board to board. The answer is yes. We are in a very rapid experimental cycle which you will see from Rebecca very soon for Dr. Lipner is going to be a really nice view of what ABIM has done. I think they have really shown the way, but there are other great examples as well.

And then I want to just for the last couple of slides to get us right into the question of data which is -- where the committee might be focused is this part four. I wanted to wrap this in the knowledge that just doing performance measure data and getting clinical data out is not going to be enough to certify a physician. It takes more. But you still need it to certify a physician because you need to know how a physician is doing and the common things they do and are they doing them well as best we can measure them with flow of data.

We now have in this thing called part four which is practice improvement four basic pathways that are emerging. The first is the practice audit which means in simple terms that physicians go into their practice, they pull charts, they abstract data, clinical data most commonly, usually 20, 30, 40 records and then they enter that in. They do a practice improvement activity where they try to improve the care based upon feedback and then they do that a second time. And the PIMs, the practice performance improvement modules or is it practice or performance. Practice improvement modules that you will hear from maybe ABMS. It is just a quick essentially wonderfully good example, a high good science-based way of doing it.

Next is the outcomes assessment through data bases and one of our boards, the thoracic surgery board, uses the participation in the STS registry which you I am sure know. If not it is very easy to sort of -- this is the cardiovascular database that is maintained by the Society of Thoracic Surgeons. There are a number of other boards who are building their own little registries or working their societies to build a registry-based data to get clinically -- direct clinical data in so that they can evaluate on an ongoing basis in a series of patients.

The third which is a brand new area that we are working on and it probably has a lot of meaning to even the Joint Commission is that we are working to say can a group of -- that there is a lot of improvement activity going on where physicians participating in a group activity. You can identify the physician's participation, but it is being driven by a group activity. I think the nice one nationally is the Cystic Fibrosis Foundation has for years had a cystic fibrosis national registry and the centers all participate in it and they send the data in and that data gets reflected publicly. And their centers are all judged by lifespan and quality of life of kids growing older with cystic fibrosis. And the physicians participating in that even if their practice per se is not but they are part of a center and they are doing this actively. They should get credit for that.

Now we are doing this more specifically at the institution level now as opposed to just national. And the example that is being ruled out and Dr. Green knows about it because it is American Board of Family Medicine is doing this with the American Board of Medicine and Pediatrics and we are now scaling it up is at Mayo Clinic where we are actually saying if Mayo is doing quality improvement activities can physicians who are doing those who have recognizable activity in those use the data from those improvement activities to be part of part four activity. And that institutional recognition of what I am calling institutional accreditation for part four activity is when we are rapidly trying to explore and evolve.

And the final one which shouldn't go unnoticed but it is one that only represents a small number of boards is where you don't have patient care but you have direct patient responsibilities so pathology, diagnostic radiology. Those are where you actually want to know that someone is making sure that those past slides are being read right and there is a peer-to-peer activity that goes on. Actually with pathology it prescribed by I think it is CLIA. And that is recognized by our world.

With all that in context what are the emerging data needs? It is all around part four. We really have a good sense of what the other data needs are. And it all rests on this very simple construct of patient-level data with accountability to individual physicians. The patient-level data you spend a lot of time talking about in terms of how do you get patient data, a good data out of EHR. It is the second piece to this statement that makes it so important to the boards in order for us to be able to tell the public what we need to tell them and that is linking that patient data to a doctor and that isn't easy. And I think if there is one thing I can come out of you in terms of your thinking here is no matter what you are doing in terms of trying to understand the data and how to make a hemoglobin A1c look the same in all electronic health records, how do you attach the ordering of that to a doctor or to another a licensed physician or to another licensed health professional because that is going to be key as you go forward as you make recommendations?

And it is not always easy to manage that accountability because of obvious reasons. The downside is that is what the boards have to do because we certify individual physicians. That is going to be key.

The other question in data is whether the member boards are finding their data. Most of our data is not coming out of records through data entry as I have mentioned. A little bit of it is coming through these registries and we have almost none coming through electronic health records. The question to us is as we start to get more open in transferring and moving data how do the credentialing and the accrediting bodies get access to that data to do the need that the public needs us to do and that is another key issue in terms of I would encourage you to think about.

That is particularly true of the electronic health record data, but we are going directly to patients now for the first time. Some of our boards have already started this where they are actually asking for outcomes data. We are starting with experience with care surveys, but it is not that far of a leap to do what they are doing in the UK where these four conditions -- I think I is hip and -- well, I know it is hip fracture where they are actually saying 6 months after a hip operation how are you doing on your hip repair. And we should be doing that in the US and the boards will start to consider that and I am hoping we will start to adopt that. It is not just experience with care we will be getting. How do we get that on a more national basis is a question.

I think this is my pet ultimate which is what can we do to help. Well, I think it directly relates to the last two slides so I won't read this. And just close by saying that it is easy to think that we can build a system, a performance measurement and satisfy the need for public improvement and care that somehow we can measure everything in a performance measure and get there and that if we move the electronic data out of the electronic health record, that is the answer. What I come to say is no. That gives you part of the answer and it gives you part of the answer on the big, broad brush stroke thing. It is those smaller brush strokes, those more subtle brush strokes, those low frequency, high risk, intermediate frequency activities that have to be part of the everyday diagnostics when you start talking about appropriateness and misadventures in care that are just as important as those high frequency, high priority items that we are all working to define for the national priorities.

And if we don't make sure that the data stream is not just about those high priority items, the hemoglobin A1c sort of speak, then we are going to miss the boat. I will stop there. I hope this has been helpful.

DR. MIDDLETON: That was really terrific, Kevin. Thank you very much. Rich slides and excellent presentation and lots of issues highlighted. Given the time I just want to ask if there are two burning questions and then we will move on to our next speaker. Paul.

DR. TANG: One question. On part four you mentioned the Mayo activities and I think let me just if I got this interpreted correctly you were saying in a sense participation in the Mayo activity could deem them in fulfillment of the part four of your MOC, correct?

DR. WEISS: That is the direction we are trying to move. Yes.

DR. TANG: Likewise you might say it could relate to meaningful use in one way or another in its later forms where we are measuring outcomes, et cetera than if you do this then it shows that you have one and you can use it to improve your practice, that kind of thing.

DR. WEISS: Absolutely. Now take one more step further and say that under -- as of about 4 weeks ago, and you may know this, we are under a contract from ONC to develop MOC components both for part two, part three and part four. And the intent of that is to build components that will relate to meaningful use assessment. We will be over the next year or two developing formal projects in part four that will look at the question of meaningful use. We will be looking at part two in terms of what do people understand in their own self assessment of meaningful use and then also developing items that will end up on the high stakes' exam that will question whether physicians actually know what meaningful use is. We are directly going to be approaching that.

DR. TANG: I get that for part two. How does part four -- I thought part four would go the other way around like the Mayo serving as qualification, satisfaction for part four of your MOC.

DR. WEISS: What we can do very simply in that is part of what I am calling the accreditation standards for what an institution can participate. The institution would have to meet certain obligations for all of its physicians if we are going to participate in improvement activity. That is where that can be inserted.

DR. FITZMAURICE: You had mentioned that low-frequency events are just as important as the high-frequency events. Are you saying that the development of quality measure that would look at rare bad things that happen and so you might keep account of them and put that over some denominator that if it could be reported could be accurate data could be an important quality measure.

DR. WEISS: Absolutely. And again I think the example that comes quickly to mind which is the thyroid nodule. In an everyday general medicine practice you may see one or two a month maybe, but yet when you do it you can really take a misadventure in care if you don't know what you are doing well. You can miss a thyroid cancer. It would be a horrible thing. That is a feature of something if you have that low frequency. But that persistent over practice you probably start to have accumulative risk for seeing a thyroid cancer in your practice probably every 5 to 10 years as opposed to diabetes which you are doing it 20 times a week.

DR. MIDDLETON: Thanks very much. Why don't we go ahead and switch gears. Dr. Lipner from the American Board will be presenting. I am going to ask the committee members at least and consultants here indulgence to go over a little bit. We are just running about 10 minutes behind schedule so if we could have 10 more minutes that would be perfect. Dr. Lipner, please take it away.

DR. LIPNER: Thank you. I am very glad that Kevin went first because I was hoping that he would give you the big overview of the ABMS. What I am going to be talking about today is really from the perspective of one of those 24 boards that is working under ABMS. We happened to be the biggest board so we have 300,000 of those diplomats. But we have sort of entered this into what do we need to assess clinical performance across the continuum in a different way than all the other boards. We are sort of leading some of the research that is going into part four particularly but also part three.

I just sort of put this out there. We all know that clinical performance assessment is complex and as Kevin mentioned we look at it from the ABMS as clinical performance is really a function of multiple competencies. It is not just about those measures that you are measuring in practice, but it is about your diagnostic reasoning skills. Can you come up with the right diagnosis? If you look at a chart, you don't know that they come up with the right diagnosis. You just see the measures there. The clinical care is important: the outcomes measures, the process measures, the mortality, morbidity.

Communication with patients and peers. How do you get along with your patients? How do you get along with your peers? And that includes things like team work, coordination of care. It is the big picture.

Are you able to work within a system and if you change systems, can you adapt to the different systems? It is not just about the system that you are currently working in but can you move to another system.

And finally, about professionalism. Do you come to the job with the right characteristics and the right approach? Are you -- simple things like do you have disciplinary actions against you. Some basic things like that all the way to sort of your ethical values and how do you work with others.

To do that, we obviously need different types of data and measures to assess these different competencies. Now, they are not going to come all from EHR. I would love them to come from the EHR. As a researcher it would be lovely to get the data electronically and without a whole lot of effort.

But when we assess these we need to do it at multiple levels. We need to look at things at the patient level, at the physician level, and at the system level because they are all interrelated. For us to think they are not is really making it much too simplistic.

We in statistics in my department is sort of what we call the psychometrics department, but as statisticians, we note that patients are nested within the physician. In other words, patients pick who they go to. That is an important feature culturally as well as just where you live and in the kinds of patients that you see, but they choose them. We have to take that into account when we look our reliability, when we look at our validity, when we do our risk adjustment, et cetera.

Physicians are also nested within the system they work and that is also another function. There are some people and in some of our research we have people in really great systems that aren't doing very well. You wonder why. They have all the bells and whistles, but what is going on there. It is more than just the system. We know that there is a real big interaction between these three things.

We need to make the most accurate decisions about physician's clinical performance that we can at this point in time. And I think things have gotten better over the last 10 years and they will get better, but we need to do the best job we can now and what we think of as moving the curve along. Let's show people where they are now and believe it or not our physicians on our board basically say you show me where I am compared to other people and I am going to want to move up because it is just that type A personality. We all want to do better.

And we need to evaluate that performance improvement over time. As a nation and as physicians as a whole, we need to look at that. Are we doing better over time? Not just this individual physician but as a whole.

In terms of looking at the data obviously we need high quality data and a lot of people have talked about this, but I just wanted to sort of call out this little here is the standards that we use in our testing industry for measurement and we really abide by that book. It is like our bible. How do we know that we are doing a good job at the measures that we are giving you?

But we do look at it from three different levels. We talked a lot about the data itself, the data elements, and the accuracy of those elements. Are they complete? Do we have a lot of missing data in there? Is it filled up with the right data? Are they comparable across systems even EHRs? Am I getting the same things when I look at that data? And are they timely like are they old ones or can we dynamically change them and what is in that? That is very important that that is all accurate.

Now at the board we have assumed that what we are getting is that accuracy although we know for sure that it is probably not. And right now at the board we have collected data and asked physicians to collect it themselves. They have pulled their own charts. They self report. And that actually is a very positive thing. We thought they would really be annoyed with us to have to keep pulling those charts, but the first 25 charts that they have pulled they said wow. I didn't realize this wasn't in the chart. It is an eye opener. It really says aha. Something is wrong with my bookkeeping. From that perspective we haven't spent a lot of time at that point that we recognize that it is really important.

Then when you roll up those data elements into measures based on clinically-based guideline hopefully, we need to assure that they are reliable, valid, and feasible. Can we actually calculate them, get them? Do we have enough patients per physicians as Kevin was mentioning to collect them because if we don't, that measure is not going to be reliable? And I will talk a little about what we hope to do as more on a composite level where we aggregate measures up to a composite because we don't feel the measures in and of themselves by themselves are reliable enough for me to make a decision about you as a physician. Are you certified or not certified? We have really high stakes decision that we are making and we need to make sure that what we make those decisions and what we base those decisions on are high reliably and valid measures.

The other thing -- so I talked a little bit about this, the classifications and decisions. We didn't talk much about that, but for us that is really important. For us it is whether you got certified or not. For others it is whether you got paid more or not. There could be many levels of decisions that you could make about a physician, but when you do that you need to have it be very accurate because you will get people not believing in those as decisions. The people that we think should be certified should be others that think should be certified. There should be other measures to say yes. This is the right person and yes, this person probably shouldn't have been certified. We need to make sure that those decisions are accurate. There are high consequences to not being certified in this country. You can't get jobs. You can't get paid at a higher level.

And finally we need to make sure what we base is on appropriate and planned sample design so that -- even though when we EHRs you will have lots of data and you think you don't need to worry about samples, but there will always be measures where you don't have enough patients during that annual period that you are looking at. What do you do and how much is enough? How many patients do we need?

I just gave some examples of the specific things that we need to do our work. We have talked a little bit about these here in various different settings, but we need the clinical data. That could come from EHRs. That does come from the EHRs. We need patient data. We need the demographics of the patient. Does the physician know the demographics of the patient or do we have to get that from the patient?

The other week we asked physicians -- we did a focus group and asked physicians can you tell us the insurance of your patients like go get the insurance? And they like had either no clue to where to look or just didn't want to. They didn't want to go from the chart to the administrative data, but somewhere you know they are getting paid.

Those kinds of things are really important in terms of potential patient risk adjustors. Again, we have the debate in our board about should we risk adjust or should we stratify. Should we show people how they are doing based on certain patient populations or should we actually put the risk adjustment into the measures? And that is an ongoing debate and it is sort of a little bit of both.

We also think it is important to look at the patient self-care or what we are calling here experience of care measures. We have found that those are actually informative and helpful in predicting outcomes of care. We look at it a little bit differently. We ask the patient and we do a lot of what Judy said this morning, but we do a very specifically for that particular disease. We will ask the patient did the doctor or somebody in the office tell you how to check your feet, something really very focused and very actionable. And we are actually putting those patient care measures into this composite, which is sort of a little bit out there. I will show you that in a minute.

The physician's data. Obviously we need basic demographics on the physician. For us specialization is important. Are you an endocrinologist or a general internist? What kinds of patients are you seeing and what kind of education did you have when you got there?

Diagnostic reasoning is important to us. Are you making errors and you are taking care of this patient, but for the wrong thing. We need to know that. I mentioned the unrestricted medical license and disciplinary actions.

And then we need to know what kind of system you are in. You should see the span of physicians meeting certification programs. It goes from the solo practitioner to the people in Mayo clinics to group practices of sizes of 10 or so. All of that is important in terms of us understanding the relationships between all these competencies and between the patient, physician and the system.

ABIM defines the field of internal medicine and these are the subspecialty areas that we have now. We have general internal medicine. Everybody has to go through and get certified in that, but once they do that they can actually go on to fellowship training and get specialized in any of these areas. Right now we have over 50 percent of our physicians is specializing in one of these areas. We have a broad span of areas to deal with in our program.

These define the field of internal medicine because they are broad. We are not talking about just thyroid disease. We are talking about a little bigger field.

What we are dealing with is very different types of physicians, but I just pulled out this slide to demonstrate to you that in our MOC program as of the end of 2009, we actually have 16 percent of our physicians are still in a practice of one physician and it is clearly defined, not working part time with others, one, by themselves. And then if you look 10 or less we have 50 percent of the practices are in 10 or less. It really still feels like we have a long way to go to get to the accountable care organizations or a bigger picture of pulling these physicians together.

We also have a variety of things that motivate physicians to participate in our program and surprisingly this was a study that was done in 2006, but we have repeated that study. We haven't published it yet. But professional image is really what drives them to the program at least this is what they say. As you can see, required for employment and monetary benefits, again this is self reported, is not as important to them as keeping up to date and doing well for their patients. When we ask the question a different way of are you required to do this for yours, we got like a 65 percent saying yes we are required for some either a health plan, insurance companies. Somebody else is requiring us to do that. But the idea here is that we have a wide variety of physicians and motivations and backgrounds.

What we would like to do in thinking about assessment is we look at it from and this is a very theoretical approach to clinical assessment, but sort of to get you out of the mode of the outcomes and processes of care. There are many different ways to assess and you can do some things. The bottom level is very elementary. Do you have the knowledge? And the top level, can you actually show it? Are you actually doing the right things in your practice? I will just talk about these from a medical standpoint, but at the bottom level when you are doing assessment, you can actually assess things in a broader way. I think this is a little bit what Kevin was talking about. The things that you don't see normally in practice how the heck are you going to know whether if that person comes to your office, do you know how to take care of them? One way of doing that is by one of these lower levels of types of assessment.

When you move up the pyramid to the top, it is much narrower what you can assess and it is less standardized. And that is what we are facing with all these processes and outcomes of care. Everybody is in different types of practices. They see different patients. We really can't get very good standardized measures. We are trying and we are getting there, but it is hard. We can't cover everything. We use sort of this spectrum of things to test different types of competencies in different ways.

I will talk about two areas that we do at the American Board of Internal Medicine and the second level is the diagnostic reasoning. We are assessing diagnostic reasoning using clinical vignettes and that is what our exam does. It is basically little case scenarios, snippets. Can you handle this? Can you diagnose the person appropriately? And then we talk about the performance and practice and --

For clinical diagnostic reasoning I am not going to spend a lot of time on this, but why do we do this? Who cares? If you can take care of your patient, why do you care about this? And one of the reasons is that you could test broadly. The other is that there is a whole theory about problem representation. If you have the right schema in your head to represent problems and you have a really foundation of the way you think about diagnosis, you will be able to come up hypotheses that are meaningful and not just gathered from incidental information. You don't just sort of pull the pieces together and go online and look up. Well, it is this, this and this. What can it be? You actually think about it in a very logical and representative way.

We think this is an important thing to test because if you don't have that and we have seen it where people can scramble and use resources, but they are still not getting the right answer what is it that they are missing. This is a really important thing in education. And Judy Bowen has a really nice article in New England Journal that kind of describes that. But that is what we are trying to do through the secure exams.

Going over to the practice improvement modules this is what we use to assess performance and practice. And again right now it is all physicians pulling their own charts. Eventually with the meaningful use contract that we have we are actually going to put one of our practice improvement modules which we call our comprehensive care practice improvement module into electronic forms so that we can actually get data from electronic health records directly. We are going to have a whole learning experience through that. It has been very helpful today that we hear about that experience.

But we have approached this with is a three-pronged data collection. It is not just about the chart data, the clinical data, but it is also about the patient experience of data as well as the practice data. We actually use NCQA's practice survey in a lot of our practice information in our practice survey to collect information about the practice. It had practically 100 questions in it and we wanted to see whether it was related to anything else.

We take those three pieces of information and feed back in performance report to the physician based on how they are doing and I will show you how feedback relative to how everybody else is doing as well. And then they have to do a quality improvement cycle based on something that they are not doing well on. And then they remeasure it afterwards. We want to see if they have improved. We don't hold them to improving although all we hold them to at this point is --

What we have and this is written up in two different papers. What we have is a diabetes composite score which I think you actually have seen. I don't know how many of you were on this committee last year, but David Ruben presented some of this last year. And basically the bottom line is that the composite measure that we came up with is much more reliable than any of the individual measures. And we make the classification decision based on that composite. It is much more reproducible than a decision that you can make in any individual measure. And we basically get fewer false-positive decisions when we do that. We want to protect ourselves against certifying somebody who shouldn't be certified and not certifying somebody who should be.

The composites allow for a more comprehensive assessment. But performance and feedback on those individual measures is still important. Even though you say okay here is how you did relative to everybody else, you still need to know where you need to improve. Saying it at this level but giving all the information is very helpful.

And then we have done a classifications decision based on a very scientific approach to what we call standards setting. We didn't just have experts sitting around the room and say 70 percent. We actually did a methodology that holds up fairly well scientifically.

I won't go through the developing composite measure steps, but we really did base it on the data that we had in our program. In order for us to get our expert panel to really believe what they were doing is they needed to see the data. Tell me how people are actually doing on the measures.

We looked at 20,000 patient charts and 18,000 patient surveys so a lot of data. We actually had more data than that now, but we based our data on that. And we looked at individual measures as well as the importance of those individual measures.

This is kind of what you saw last year. It has changed a little bit I think. This is again for diabetes' care. These are the intermediate outcome measures, the clinical process measures, and we have patient experience measures in there. It is part of our composite at this point.

Any one of these measures by themselves we didn't feel as reliable enough to say you pass or fail on it. We ended up coming up with thresholds for those individual measures and important weights for those individual measures as well.

What is different about our composite than others is that we give people partial credit for doing well so that not everybody is like in or out. It is not an all or none. We actually give them something. If most of their patients are doing well in something, they should be recognized for that. We give them a partial credit.

It is though a compensatory model. You can make up for something. If you are not doing well in one thing and doing well in another although these things are so interrelated as you know in medicine, you can make up for doing bad on something.

What we did notice is that you can't make up -- you can't kind of pass this thing if it is all about patient experience measures. And the patient experience measures by themselves did not hold up really well by themselves but together with the rest of it it did hold up.

DR. TANG: (Inaudible)

DR. LIPNER: It is the percent of patients that need to meet this goal.

DR. TANG: And literally you said a threshold for LDL less than 100 up to 23.8 percent.

DR. LIPNER: Yes. Let me just sort of take a step back. We started with what we were calling tell us about the competent physician and we set the -- purposely have the group think about it with a very low bar. Who would you just like not have your mother go to? It was purposely set that way. Now it doesn't have to be. You could change the mentality of the group and say an average physician, an excellent physician. But we purposely did this because we thought it would be less threatening and it would get physicians engaged in the process of this is easy to do and they will see that they can actually achieve this.

We actually haven't set this in our program. All it is is performance feedback at this point. But it was based on the data and surprisingly people were doing pretty badly in that measure. They sort of came up with a value without the data and then looked at the data and adjust it.

DR. TANG: So if you don't reach 23.8, you get 0 for that row.

DR. LIPNER: No. You actually get something, but very little. You will get something.

This just shows you a particular doctor and their performance and how they would do. What we did do for clinical process measures is what you just said. We said you have to achieve at least this much. We couldn't justify giving you anything if you didn't meet that minimum threshold.

How did we do in terms of accuracy of our classifications for this competency measure? We said these people are competent and these aren't. And basically this is -- I don't want to get into all the statistics here, but at that dashed vertical line is where we set the cuts for. And there out of one we were at .98, very high classification accuracy which is really, really good. We felt really good about our composite. And partially we are there because it is such an easy measure. It is such an easy thing to achieve and most people are achieving it. As you can see if you set the bar at different points, that reproducibility or that accuracy would change a bit, but still be pretty good.

And this is sort of what we came out with for those 957 physicians and I think David showed that to you last year. But the idea here is to give feedback to physicians. Where are you on this curve? Dr. Smith was at 66. He is not even at the mean. Dr. Smith would say wow. I need to improve. I am not even at the average.

But we also wanted to give them an idea of how bad is bad and we said anybody below here really doesn't meet this very minimal standard. And we only had 4 percent of our physicians in that category and these were volunteers who did this. When we actually did this again with non-volunteers, it was a little bit higher.

And then we are giving feedback based on the individual measures as well and we are dividing it into these quartiles to just give them a sense of where they feel on these measures. We are also thinking about stratifying them to comparing them to physicians like themselves, which I think will be very helpful for them.

All in all this worked out really well. The other thing that I just wanted to point out is the composite score interpretation was very valid. What we looked at was endocrinologists did perform better than internists, which is what we expect. Those classified as incompetent scored lower on their diagnostic reasoning exam. There was a significant difference. They also have low overall ratings in residency so from their program directors. There was something about them that was different. And they were also more likely to be in solo practice. It was sort of like these were things that really stood out that made us feel like okay. It is working in the right direction.

Moving forward and this is sort of what the meaningful use grant is going to be working on is the comprehensive care plan where we are now going to take the challenge of not just diabetes but looking across the spectrum of general internist practice. And we have already done a study on this where we had chart abstractors abstracting the data. And I won't get into the details of it other than -- let me just show the summary in the next slide. The complexity increases across multiple conditions. But measurement of chronic disease care and preventive services was feasible. Not every single bitty thing, but on a general level we got enough there that we can get enough charts and enough data. Acute care conditions were not well documented and we really couldn't measure them at all. And again performance of practice here was correlated with diagnostic reasoning.

And finally basically high quality data is important and for us access to the actual broad data elements is really critical mostly to continue doing the research that we need to do to understand the interrelationships. If everything is rolled up in measures and measures keep changing over time, we are never going to understand how these things interact with one another. And obviously the connectivities to the electronic databases are really critical as well.

DR. MIDDLETON: This is really terrific and I am sure we can go on for a long time, but we are at the end of the day. Let me ask if there are three burning questions.

DR. TANG: One, you answered the question that I asked David a year ago. Thank you. I got my answer now.

Let's go back to the purpose of this because I find it very helpful and I want to do some offline stuff with you in meaningful use. But what do we take away on the national quality measurement roadmap? I understand the parts of how you assess everything from the competency to the practice of an individual physician. How do we relate that to national quality measurement?

DR. LIPNER: It is a question that I don't think there is a great answer for it. I think the thing that I think we have to start with is reducing redundancy between all the different players that are working on the same thing. I think that is -- as I sit here listening and have been in many other places, everybody is approaching things in different ways and that is great. But bringing that research together and really trying to understand. What do we know now and what can we implement? We are looking for a patient experience of care of measure. Can we come up with one that we could all be happy with or can we come up with a variety or at least constructs that are always going to go into these things?

Similarly with I think with the electronic health records we are a little further along because I think the NQF has really been and we looked to others to develop to give us the measures. We really don't want to create the measures ourselves although when we started this a number of years ago there just weren't measures in some areas and you saw all the disciplines that we had. I think that is one that we have to also focus on, the too hard box. What can't we measure currently that maybe at some point we can measure and how do we collect the data to get there?

DR. TANG: So it is a little bit of begging the question it sounds like. We all agree that we want access to raw data in real record systems. And I suppose the only thing and this may be similar to the conclusion we had last time under meaningful measures. Would ABIM and ABMS be willing to be at the table since we all want the same thing and we don't want to duplicate work? Should we all just work together and find a forum to do that now that we have a much richer sandbox? We didn't have a sandbox for any of us. We all did our own thing. Now that the sandbox is being built should we work together on figuring out a way not only for common measures, but sort of cross demeaning and that kind of thing?

DR. MIDDLETON: One thought you mentioned NQF a couple of times and I think the current work underway there in terms of utilization measures, measures of CDS and structural measures of HIT adoption that framework of course might be amenable or lend itself to some measurement framework for cognitive services. That would be very interesting to pursue building upon that infrastructure.

DR. GREEN: I want to thank you for coming and also you, Kevin, for taking the time to do this. And I want to take us back to the topic of the hearing about developing a quality measurement roadmap that considers future information needs and data sources. Just notice that this is an addition to the data sources that is still immature but extremely promising, and it ties back to NCVHS' work in a couple of powerful ways in my view. A big one is those years that NCVHS spent on looking at reuse of data. Collect once, use often for multiple purposes, and that is coming back, I think, to what you are calling out, Paul.

Another aspect of why this is an important thing I think for this subcommittee and the full committee to consider, is what you showed us at least three times in your slides in different ways. But it is the reach and to what is actually the big platform with health care delivery. The small physician office dwarfs all other. It is not a competition. We can have the integrated delivery systems that we want, but like we heard from HRSA at the end of the day is 16 million people. We heard from HIS at the end of the day. It is 1.8 million. What she just talked to us about starts touching almost everybody. And this really matters to our objective here is driving quality improvement and quality measurement all the way right to the front lines where communities meet medicine. This is a lovely set of presentations.

With that said and I don't want to get myself into trouble here with anybody, but it still is an immature enterprise and we have miles to go and we have much to learn about how an MOC performs. With that said there is a lot of progress. A couple of examples to try to provide some evidence to support that, The American Board of Family Medicine is a CMS registry, already made that step. Already report once, use it three times. Get your CME credit as a physician, do your board work and do PQRI. One quality improvement process. The clinical community simply loves that. Just loves it.

And these questions we had about the Mayo model and that Kevin answered. There are a lot of those experiments now that are launched and there is going to be a lot of more. Some of them are university based. Some of them are health professions association based. But these are destined in my view, to become another data source and we ought to put it on the roadmap.

DR. MIDDLETON: That was the right way to end the day. Thank you, Larry. To set the challenge for us. Thank you, Dr. Lipner and Dr. Weiss again. What a terrific day. Three panels - one more to go tomorrow. And thank all staff and support of everybody for staying an extra 15 minutes. We will see you in the morning. We are adjourned.

(Whereupon, at 4:26 pm, the meeting was adjourned.)