AMERICAN STATISTICAL ASSOCIATION (ASA) MEETING OF THE ASA COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION (EIA) OF THE UNITED STATES DEPARTMENT OF ENERGY Washington, D.C. Thursday, October 20, 2005 2 1 COMMITTEE MEMBERS: 2 NICOLAS HENGARTNER, Chair Los Alamos National Laboratory 3 MARK BERNSTEIN 4 RAND Corporation 5 JOHNNY BLAIR Abt Associates 6 MARK BURTON 7 University of Tennessee 8 JAE EDMONDS University of Maryland 9 MOSHE FEDER 10 Research Triangle Institute 11 BARBARA FORSYTH Westat 12 WALTER HILL 13 St. Mary's College of Maryland 14 NAGRAJ NEERCHAL University of Maryland 15 DARIUS SINGPURWALLA 16 LECG 17 RANDY SITTER Simon Fraser University 18 EIA PERSONNEL: 19 BOB ADLER 20 MARGOT ANDERSON 21 COLLEEN BLESSING 22 TOM BROENE BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 3 1 EIA PERSONNEL (CONT'D): 2 GUY CARUSO 3 JOHN PAUL DELEY 4 ALOULOU FAWZI 5 HOWARD BRADSHER-FREDRICK 6 STAN FREEDMAN 7 CAROL FRENCH 8 DWIGHT FRENCH 9 BILL GIFFORD 10 BEHJAT HOJJATI 11 SUSAN HOLTE 12 ALETHEA JENNINGS 13 RAY KASS 14 ROBERT KING 15 NANCY KIRKENDALL 16 TOM LORENZ 17 RUEY-PYNG LU 18 PRESTON McDOWNEY 19 RENEE MILLER 20 MICHAEL MORRIS 21 ERIK RASMUSSEN 22 MARK RODEKOHR BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 4 1 EIA PERSONNEL (CONT'D): 2 TOM RUTHERFORD 3 BHIMA SASTRI 4 MARK SCHIPPER 5 JOHN STAUB 6 LAWRENCE STROUD 7 AMY SWEENEY 8 PHILLIP TSENG 9 KEN VAGTS 10 ANGELA VEITCH 11 SHAWNA WAUGH 12 PAULA WEIR 13 ALSO PRESENT: 14 SUSAN LISS Federal Highway Administration 15 NANCY McGUCKIN 16 United States Department of Transportation 17 WILLIAM WEINIG ASA 18 KATHLEEN WERT 19 ASA 20 21 22 * * * * * BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 5 1 C O N T E N T S 2 AGENDA SESSION: PAGE 3 Administrator's Remarks 13 4 Committee Updates: 5 Since Spring 2005 Meeting 31 6 Results of Simulation Study 39 for the EIA-914 7 EIA's Regional Short-Term 46 8 Energy Outlook 9 Short-Term Forecasting Performance 71 Measures and Accuracy Evaluation 10 Preserving EIA Trustworthy 117 11 Datasets, Model Documentation and Contextual History 12 Learning from the Past: 187 13 Updating Data Quality Efforts 14 Can Discrepant Estimates 232 Be a Good Thing? 15 Survey Self-Assessments 277 16 17 18 * * * * * 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 6 1 P R O C E E D I N G S 2 (8:32 a.m.) 3 DR. HENGARTNER: Ladies and 4 gentlemen, oyez, oyez, oyez. Good morning, 5 everybody, welcome to this meeting. This is 6 an ASA, not an EIA, meeting. This committee 7 periodically provides advice to the EIA. 8 This meeting is open to the public and public 9 comments are welcome. Time will be set aside 10 for comments at the end of each morning and 11 afternoon sessions. Written comments are 12 welcome as well and may be sent to either the 13 ASA or the EIA. 14 All attendees including guests and 15 EIA employees should sign in at the register 16 in the hall and include their e-mail 17 addresses. So there's a little bit more of 18 the formalities. The rest rooms are located 19 either on one end of the hall at this end and 20 the other end of the hall. 21 The telephone in this room shares a 22 line number. Let me give you that. It's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 7 1 202-586-3071 and you can also get messages 2 through the following number if you need, 3 586-6273 and 586-6202 and probably it might 4 be a nice gesture to turn off cell phones. 5 Kathleen Wert with the ASA Meeting 6 Department is here and available for the 7 committee members' questions on expense 8 reimbursements, and I'm sure she will be 9 happy to help us, she has in the past been 10 very helpful. In commenting each committee 11 member is asked to speak toward a microphone. 12 The microphones are here if you don't speak 13 clearly and loudly we will be reminded. The 14 transcriber will appreciate that. 15 The speakers from the audience are 16 asked to use the microphone that's over 17 there. We did have a portable one but 18 because of the echo I think that might be 19 better. Also, if you need help Bill Weinig 20 will help you with this. 21 Okay, the announcements. First I'd 22 like everybody to introduce themselves in the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 8 1 room. I will start with myself. My name is 2 Nick Hengartner and I'm the chair of the 3 committee. 4 MS. KIRKENDALL: Nancy Kirkendall, 5 Director of the Statistics and Methods Group, 6 EIA. 7 MR. CARUSO: Guy Caruso, 8 Administrator at EIA. 9 DR. FEDER: Moshe Feder, RTI, 10 committee member. 11 MR. BERNSTEIN: Mark Bernstein, 12 member of the committee. 13 DR. BURTON: Mark Burton, 14 University of Tennessee. 15 DR. EDMONDS: I'm Jae Edmonds at 16 the Pacific Northwest National Laboratory. 17 MS. FORSYTH: I'm Barb Forsyth. 18 I'm a member of the committee. 19 DR. SITTER: Randy Sitter, Simon 20 Fraser University. 21 MR. McDOWNEY: Preston McDowney, 22 Statistics and Methods Group, EIA. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 9 1 MR. TSENG: Phillip Tseng, SMG, 2 EIA. 3 MS. BLESSING: Colleen Blessing, 4 National Energy Information Center, EIA. 5 MS. MILLER: Renee Miller, 6 Statistics and Methods Group, EIA. 7 MS. FRENCH: Carol French, Office 8 of Oil and Gas, EIA. 9 MS. VEITCH: Angela Veitch, 10 Petroleum Division, EIA. 11 MR. SASTRI: I'm Bhima Sastri. I'm 12 with the Office of Integrated Analysis and 13 Forecasting, EIA. 14 MR. STAUB: John Staub, EIA. 15 MS. HOJJATI: Behjat Hojjati, EIA. 16 MS. JENNINGS: Alethea Jennings, 17 Statistics and Methods Group, EIA. 18 MR. FAWZI: Aloulou, Office of 19 Integrated Analysis and Forecasting, EIA. 20 MS. WEIR: Paula Weir, EIA. 21 MR. LORENZ: Tom Lorenz, EIA. 22 MR. BRADSHER-FREDRICK: Howard BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 10 1 Bradsher-Fredrick, EIA. 2 MR. VAGTS: Ken Vagts, EIA. 3 MS. HOLTE: Susan Holte, EIA. 4 MS. SWEENEY: Amy Sweeney, EIA. 5 DR. HENGARTNER: Thank you very 6 much. For your information, under the terms 7 of the DoE grant to the ASA Nancy Kirkendall 8 here may chair but must attend each meeting. 9 And she's authorized to adjourn the meeting 10 if she determines this to be in the public 11 interest. She must approve all meetings of 12 the advisory committee and every agenda. 13 Also she may designate a substitute in her 14 absence. 15 I think that we have a good agenda. 16 We've all seen a little bit of what's going 17 to be on the program. I'm looking forward to 18 it. There are going to be four breakout 19 sessions and a few presentations here. 20 We have also one new committee 21 member I hope to be able to introduce to you 22 shortly and with both sadness and pleasure BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 11 1 I'd like to recognize the participation of 2 two of our former members. It's going to be 3 their last session. One is Mark Bernstein 4 here and the other Randy Sitter. Both of 5 them have been with us for six years and 6 their contributions have been valuable to the 7 committee and we'd like to recognize this. 8 MR. CARUSO: Thank you, Nick. Let 9 me start with Mark since he's the closest 10 one. There's couple of small tokens of 11 appreciation for you, Mark. There's one from 12 Nancy and me and this one's from Secretary 13 Bodman. So, Mark, thank you. 14 MR. BERNSTEIN: Thanks so much. 15 MR. CARUSO: We'll miss you here. 16 MR. WEINIG: Could you and Mark 17 come up towards the front so we can get a 18 couple of additional pictures? Thank you. 19 DR. HENGARTNER: We're thinking of 20 putting Mark's picture on the wall here. 21 The second member who's going to 22 depart us and whose last meeting it is as BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 12 1 well is Randy Sitter and I'd like also to ask 2 us to acknowledge him. 3 DR. SITTER: Thank you very much. 4 DR. HENGARTNER: So just a note on 5 the program. The first address this morning 6 will be done by Guy Caruso on the EIA general 7 state of affairs. Then Nancy Kirkendall as 8 the past director of the EIA Statistics and 9 Methods Group will tell us about what's going 10 on. And other summary presentation will be 11 from Preston McDowney, Margot Anderson, and 12 that's it for the locals. Good. Now, the 13 important thing, lunch. 14 DR. SITTER: Oh, already? 15 DR. HENGARTNER: Lunch will be as 16 usual served on the first floor at 12:25. 17 It's in the same corridor. Just follow me. 18 I'll find it. And also there's going to be 19 dinner tonight in honor of Mark. We are 20 going to go to a pub and I know that Bill 21 Weinig has sampled them and has chosen us the 22 best pub in DC. To make life easy for our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 13 1 organization can I have a show of hands of 2 who will be attending? 3 Thank you very much. Tomorrow 4 morning there's going to be breakfast at 7:45 5 and we will resume the meetings here at 8:30. 6 Finally, just a reminder, I am the Chair and 7 my responsibility is to keep us on time and 8 therefore we're 20 minutes late. Don't 9 worry, we'll catch up, so thank you very much 10 and I'd like to ask Guy to give us the state 11 of affairs. 12 MR. CARUSO: Thank you, Nick, and 13 good morning to the members of the committee 14 and everyone else joining us this morning. 15 It's a pleasure always to welcome the ASA 16 Energy Statistics Committee and, as I have 17 said many times, we find the work and advice 18 of your committee extremely useful and that 19 will show up in my remarks this morning as 20 well as presentations throughout the next two 21 days. It's been put to some very practical 22 use just in recent days in our analysis of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 14 1 the hurricanes and in our short- term outlook 2 and winter fuels outlook particularly. It's 3 always sad to see members of the committee 4 leave but Mark and Randy have been with us 5 for six years and they had extremely useful 6 comments and then some of the other 7 activities with Randy, some of Randy's 8 students and other activities, so we wish 9 them well and we'll be seeing you in other 10 venues. 11 Let's very quickly go over what's 12 been a very busy six months since we've last 13 met. Most recently, of course, our services, 14 our analysis, our data have been under close 15 scrutiny as the analysis in the aftermath of 16 both hurricanes Katrina and Rita have had 17 such a devastating effect on our energy 18 infrastructure in addition to the more 19 important personal tragedies of the 20 individuals involved but we've had a large 21 number of request for our services, for our 22 analysis, for our testimony, and just last BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 15 1 week released our winter fuels outlook, which 2 we'll talk a little bit more about later, and 3 one of the things we've put to very good use 4 is your advice on regional short-term energy 5 outlook. I'll present a couple of examples 6 to that but Margot Anderson, whose office is 7 responsible for producing that analysis, will 8 go into more detail later. 9 And I'll also say a little bit 10 about the redo of our website which we're 11 very excited about and again your comments on 12 that last meeting were taken into account and 13 you'll see that hopefully by the end of 14 November when the new website is up and 15 running. 16 And couple of comments about the 17 budget, the last time we mentioned that we 18 now have an external review team and they are 19 about to meet for the second time and we'll 20 talk a little bit about that and I think 21 that's the highlights. 22 Increased activity due to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 16 1 hurricanes, of course, has been the request 2 we've had on a regular basis for our data and 3 for our analysis and clearly, as always, it 4 moves financial markets, it does shape 5 policy. It was very instrumental in the 6 decisions made by Secretary Bodman and 7 recommendations by Secretary Bodman to the 8 President about utilization of the SPR and 9 the activities of the International Energy 10 Agency. 11 We've had nine testimonies since 12 Labor Day before Congress and numerous 13 briefings and there's been enormous activity 14 on the website, particularly on the situation 15 report that Margot Anderson's office 16 publishes every day, so just when we thought 17 things couldn't get any busier I think again 18 we're being asked to provide the latest 19 information and we're tracking as shown 20 something like this which we released last 21 week what's been happening to US crude oil 22 production in the Gulf of Mexico since the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 17 1 onset of Katrina and Rita. And we, of 2 course, project that out through the winter 3 and it's instrumental in our analysis of 4 where we think crude oil prices are going but 5 perhaps even more critical this winter will 6 be natural gas, which has suffered, I think, 7 even a larger blow to the infrastructure than 8 oil. 9 It's still about 10 percent of our 10 gas productions off line as we stand here 11 today and the restoration is coming perhaps 12 slower than I think many of us thought it 13 would be and it's partly because the on- 14 shore infrastructure for processing the gas 15 and distributing that gas has had such 16 extensive damage. 17 And then, of course, the refinery 18 situation probably is improving a bit quicker 19 than the production of oil or natural gas. 20 Nevertheless, as we've all witnessed as we 21 pull up to the pump, the high price of 22 gasoline but probably more important for this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 18 1 time of the year, heating oil prices are even 2 higher than gasoline, which is relatively 3 unusual for heating oil to be selling at a 4 premium to gasoline but as you can see by 5 this chart that the upper, the darker, line 6 there is heating oil and diesel fuel. Prices 7 will stay above gasoline for the foreseeable 8 future certainly through the winter and 9 that's largely because of the refineries. 10 Ten percent of our refineries still remain 11 offline. 12 So we're expecting very tight 13 markets for crude products of particularly 14 heating oil and jet fuel but gasoline as 15 well. And natural gas the average consumer 16 is going to be paying about $16/1,000 cubic 17 feet in the US and that's at least a 50 18 percent increase over what you as a consumer 19 would have been paying a year ago. 20 So it's a huge political issue and 21 I think there's a lot of frustration in 22 Congress and elsewhere about what to do about BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 19 1 it because there are very few levers in the 2 short term especially for natural gas because 3 there's no world market for natural gas 4 compared with crude oil products. So prices 5 have attracted an enormous increase in the 6 imports of gasoline but in the case of 7 natural gas there are no available spare 8 supplies on the world market. 9 I mentioned that the work that we 10 did with you and the advice you gave us on 11 the regional short-term energy outlook was 12 invaluable and our August STEO included the 13 regional projections and had been very useful 14 in our ability to make specific briefings to 15 individual congressmen or policy makers about 16 the different regions. And so once again the 17 work of this committee has translated itself 18 into a better product for EIA and I think a 19 more useful product in policy analysis and 20 we've had independent expert reviews looking 21 at these models and Margot will go into that 22 in much more detail in a few moments. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 20 1 This is an example of how we've 2 been able to utilize that and we now break 3 down the heating bills, in this case natural 4 gas, by four regions. The model itself, we 5 used nine census regions, as you may recall, 6 but for purposes of presentation we are using 7 four regions but the STEO itself goes into 8 much more detail. So the richness of the 9 data and the analysis has really been useful 10 and, of course, as you all know, every time 11 you give the customer a little more detail 12 they want more and it raises more questions. 13 Yesterday when I presented this to the House 14 Energy Committee those who lived in different 15 regions wanted to know why are we paying 55 16 percent more this year when other regions are 17 only 29 percent so it generates more 18 questions and more work for Margot and her 19 team and she'll talk about that. 20 In a continuing effort to improve 21 the longer term outlook we've been working to 22 improve the National Energy Modeling System, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 21 1 NEMS. We're extending in this year's outlook 2 our projection period to 2030 for the first 3 time. That will be out, if all goes well, 4 about the end of November with a reference 5 case and I think it's going to be pretty 6 interesting because our price assumptions 7 obviously are going to be much higher this 8 year than they were last year and there are 9 some pretty interesting results with respect 10 to the long-term impact of higher prices of 11 oil, of all energy but oil and natural gas in 12 particular, what it means for the mix of 13 generation for electricity, what it means for 14 imports and the demand for products and the 15 economic effect of those prices on what we'd 16 call nonconventional sources of both oil and 17 natural gas such as the oil sands of Canada. 18 Even coal to liquids is breaking some new 19 modeling ground and we'll be adding also the 20 latest Manufacturing and Energy Consumption 21 Survey, MECS. 22 I didn't want to put anything on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 22 1 this chart because it's still very tentative 2 but we've asked the Secretary and now OMB but 3 it's still in the very preliminary stages. 4 We've asked for an substantial increment of 5 money over the next three years to completely 6 redo NEMS. As many of you know, and 7 certainly Jay was part of that, this is a 8 modeling system now about 15 years old and 9 we've been trying to patch it up and you've 10 seen some of those patches in some of the 11 sessions we've had with you. Clearly it 12 needs a real top to bottom look and if we get 13 that funding I think we can make a lot of 14 improvements and incorporate some of the 15 structural changes that have occurred in the 16 industry and a number of other things that 17 we're gleaning from the data and so hopefully 18 I'll be able to report to you in the spring 19 meeting that we now have that approval from 20 OMB and that's the budget for '07 that would 21 be presented by the President in January. 22 I just wanted to give you a little BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 23 1 heads up about that because we'll certainly 2 be asking for your advice and assistance as 3 we move forward on that. It's not just a 4 question of if but when and if we don't get 5 as much money as we've asked for we're still 6 going to try to move, do something, because 7 it's clear that a number of the components of 8 NEMS need updating. 9 The other topic which you've been 10 directly involved in is the National Natural 11 Gas Production Survey. I think it was one of 12 the first items that we discussed when I got 13 here three and a half years ago and it's 14 finally come to fruition. We now have the 15 914 data on the website and we'll be running 16 that in parallel with the existing system for 17 a while to do our data testing and quality 18 control and Ken Botts and Nancy's teams have 19 both worked very hard on this and Preston's 20 going to talk a bit more about natural gas in 21 his presentation. And we're also changing a 22 little bit on the data for the Texas Natural BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 24 1 Gas and we'll talk about that in more detail. 2 I mentioned about the '07 budget. 3 Well, '06 budget here we are what, three 4 weeks into the fiscal year and this budget 5 hasn't been approved yet. As many of you 6 know, it's being debated and we're on a 7 continuing resolution. What was reported out 8 of our appropriations committee was about $86 9 million for '06. It's up $2 million more 10 than '05 and keeping our number of FTEs at 11 369. 12 The '06 budget obviously is going 13 to be subject to pretty intense debate 14 because of the need to pay for the federal 15 costs in restoration and recovery in the Gulf 16 Coast so most people are expecting there will 17 be some impact on everyone's budget so we'll 18 be working closely with our congressional 19 appropriation staffers to encourage them to 20 think of EIA as part of the solution and not 21 part of the problem. So maybe we won't 22 suffer too badly but stay tuned on that one. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 25 1 I mentioned the website being 2 redone. The website hits continue to exceed 3 our expectations and clearly during September 4 and thus far in October has been as much 5 activity as I think we've ever seen and 6 Margot's office's products of the daily 7 situation report and this winter fuels 8 Outlook, STEO, getting a lot of attention. 9 And I mentioned the external review 10 Team, I think, last meeting. We may have 11 mentioned it but I'm not sure. Did we, 12 Nancy? 13 MS. KIRKENDALL: We mentioned it. 14 I think we gave them Denny Ellerman's name as 15 Chair but I'm not sure we told them about the 16 other members yet because I don't think that 17 was fixed yet. 18 MR. CARUSO: This external review 19 Team we're getting, we're anticipating an 20 extremely useful report coming from Denny 21 Ellerman's team and we have outstanding 22 members, Kathy Cooper, who is a former BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 26 1 Undersecretary of Commerce now back in Dallas 2 teaching, Jay Hakes, my predecessor as 3 administrator, Paul Joskow, whom you all know 4 is the head of the MIT Energy and Environment 5 Policy Center, and Phil Sharp, now the 6 Director of Resources for the Future. 7 We met in July and had a very good 8 discussion about their views and where we 9 think that they could be useful in mapping 10 out where EIA might be going over the next 11 five years, what our mandate to them was, and 12 Denny, as he always does, has been very 13 thorough and even while he's spending a year 14 abroad on sabbatical getting lots of e-mails 15 and requests and they're going to meet in 16 about a month and that will be the critical 17 bringing together of the team and then after 18 that will prepare a report for us and it will 19 be very useful as we look out in our 20 strategic plan and particularly as we respond 21 to OMB's request that we look at the whole 22 program. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 27 1 They criticized our part submission 2 because we had submitted individual reviews 3 of individual components of what EIA does but 4 we had never done a complete review of what 5 EIA's mission is, what it should be, are we 6 doing the right things, so that's what this 7 is all about and I think it's going to be 8 extremely useful and with that team that you 9 see there I can't help but think that it will 10 be very important to EIA's future planning. 11 As I mentioned, we're going to have 12 a new home page in our website and we're 13 hoping it will be available by the end of 14 November. We had invaluable comments from 15 this group at the last meeting. It's now 16 gone through internal testing and review and 17 I've been looking at it myself over the last 18 few days and I think it's going to be 19 extremely popular, even make EIA's website 20 even more popular. 21 With that, Nick and members of the 22 committee, I once again want to thank you for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 28 1 everything you've done for EIA and I know 2 you'll see from the further sessions how 3 important your advice has been and how we put 4 it to use and why we know that we've still 5 got plenty of room to improve and we look 6 forward to working with you, Nick, and the 7 rest of the committee. Thanks. 8 DR. HENGARTNER: Thank you very 9 much. 10 MR. CARUSO: I'll be happy to 11 answer any questions. 12 DR. HENGARTNER: The strategy is if 13 someone in the committee has a question we 14 have a tent and we lift it. Questions, 15 anyone? I had actually a few questions 16 concerned with the response to natural 17 disasters and, more generally, disruption to 18 the energy infrastructure. And the first 19 question is are we actually well equipped to 20 report in a timely manner such disruptions? 21 Secondly, are we gathering the right data to 22 be able to do that? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 29 1 MR. CARUSO: I think EIA itself is 2 pretty well equipped because we do our weekly 3 surveys particularly for petroleum and 4 natural gas. We have a ready-made network 5 that we can reach out to in an emergency. As 6 you could imagine, in an emergency like this 7 a number of our respondents were either 8 evacuated or offline so it was more difficult 9 but I think we've played a pretty important 10 role. The actual emergency response and 11 operational side of that are the 12 responsibility of another office within DoE 13 called Office of Energy Reliability but we've 14 been feeding our information through them in 15 some instances and then we have our own 16 product in the case of the daily situation 17 report on our website. 18 But clearly you find out in 19 situations like this there are gaps and 20 particularly on the natural gas side much 21 more difficult because there are 30,000 22 operators as opposed to the oil side where BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 30 1 you have a more limited universe. 2 DR. HENGARTNER: Yes, Jae. 3 DR. EDMONDS: As we go through 4 further sessions will we be hearing more 5 about plans for NEMS and the approach that 6 you're going to be taking as you rethink that 7 architecture? 8 MR. CARUSO: Yes, I would hope, 9 Jae, by April that the next meeting of this 10 group that that would be a fairly significant 11 part of the agenda although you may recall 12 the last time we did it it took three years. 13 And that's what we've requested, a three-year 14 budgetary plan. The first year we would 15 actually start using some of our own money in 16 '06 and try to develop the architecture and 17 the last time we did it by taking I think it 18 was between a handful and 10 people out of 19 their day to day jobs, creating a task force 20 and designing the NEMS with the assistance of 21 contractors as you remember. 22 And we even had a national academy BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 31 1 review of that and I think it was '91 or '92 2 that Jim Sweeney, I think, chaired. So the 3 answer is yes, but not this meeting, partly 4 because other than the in-house agreement 5 that this needs to be done we now need to get 6 OMB on board. I'm hopeful. 7 DR. EDMONDS: Right, well, I think 8 it's a very exciting prospect. 9 DR. HENGARTNER: Thank you very 10 much. 11 MR. CARUSO: Thank you, Nick. 12 DR. HENGARTNER: Next we will hear 13 from Nancy Kirkendall, who is going to talk 14 to us about what's going on in the Statistics 15 and Methods Group at EIA. 16 MS. KIRKENDALL: Well, actually 17 what I'll talk about is some of the 18 suggestions that you gave us before and what 19 we've done as a result of them. A lot of 20 that is what we're doing in SMG2, though, 21 since that seems to be how I entertain my 22 troops. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 32 1 We've talked to you a number of 2 times about our frames. We had an overall 3 team evaluation of all of our frames last 4 year and we talked to you about it in the 5 spring. As you know we've also had the 6 Census Bureau that's been evaluating five of 7 our frames and on Friday you'll be hearing 8 the last of their reports. They've completed 9 their study or it's almost complete at least. 10 So one of the recommendations you 11 made after the last time was that we might 12 not want to evaluate every frame on a 13 periodic basis every three to five years. 14 Even you thought that perhaps we should use a 15 more targeted approach, pick the frames that 16 are in most need of help and look into them 17 in more detail and come up with actual 18 recommendations for improvement. 19 So this is an ongoing effort. We 20 are going to think about your recommendations 21 and I've asked the frames team, which Howard 22 Bradsher-Fredrick is chairing, a group of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 33 1 people from throughout EIA, to think about 2 what we should be doing in the long term in 3 terms of frames evaluation. So we will 4 contemplate that and come back and talk to 5 you about it in the spring. 6 For one thing the Census Bureau's 7 report is new and we haven't had a chance to 8 look at it and try to figure out what we 9 should do about that too so you will have a 10 chance to give us some input on that 11 evaluation and what we should do going 12 forward. 13 As Guy mentioned, the web usability 14 testing was very useful. We've taken most of 15 your suggestions into account. These are 16 some of the specifics. They've done the 17 prototype and, as Guy mentioned, we are 18 actually testing it within EIA. This is the 19 redesigned system and what we're supposed to 20 be doing, we being EIA staff, is finding bugs 21 in it. So we're trying to debug it before it 22 goes live and hopefully the new system will BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 34 1 go live in November or December so we're 2 aiming to have something soon. 3 The external review team, in fact 4 we came to you I guess was it in the fall 5 last year when we first asked you what we 6 should do in terms of getting an external 7 review? So this whole effort came from some 8 of your recommendations. In the spring you 9 particularly suggested that we should do a 10 self-study before we have the external review 11 team come in. 12 And we did think about it, we came 13 up with a plan, we thought perhaps we'd have 14 senior managers in the EIA answer a subset of 15 the questions we were asking the committee to 16 provide that as answers to begin with. We 17 debated about how we could possibly do it in 18 the time frame we had because they were 19 having their first meeting in July. And then 20 when we asked Denny Ellerman if he thought we 21 should go ahead with this, he's the chair of 22 the external review team, he said he didn't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 35 1 think that we needed to do it. So we 2 breathed a great sigh of relief and didn't do 3 it so we didn't take your suggestion but we 4 did think about it a lot. We can do 5 something more like the preliminary review in 6 future iterations of the external review 7 team. 8 Some of your suggestions, however, 9 did get rolled into a new document we 10 prepared particularly for the external review 11 team. That's the overview document that 12 describes EIA and its products and we will 13 send it to you for a review. I think it's a 14 very nice document. We haven't really had 15 something that looked at EIA as a whole to 16 describe what we do and why we do things. 17 Future thoughts are that we should take that 18 document and edit it and make it available on 19 our website so any comments you have on the 20 document would be most welcome. 21 This is just, again, the list of 22 people from the external review team and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 36 1 their affiliations. It's really a great 2 group. Denny is fantastic. I'm chairing the 3 support team and just this week he sent out 4 assignments to all of his teammates about 5 what they're supposed to talk about at their 6 meeting in November so I think we're going to 7 have some really interesting results and 8 we'll talk to you about what their 9 recommendations are and what we plan to do 10 about it in the spring. 11 This is everything else. You've 12 given us some recommendations on the short- 13 term energy outlook and we've accepted and 14 implemented some of your suggestions and 15 you're going to hearing a lot more about STEO 16 from Margot. We asked you about incentives 17 in the residential energy consumption survey. 18 I believe that the incentive experiment has 19 been approved by OMB and we're going forward 20 with that. 21 The way these things go it will 22 probably be a long time before you hear about BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 37 1 the outcome of that experiment. This is a 2 very large survey, it takes a long time to 3 run, but we'll tell you the outcome of the 4 experiment once we get there. 5 On survey evaluations this was 6 Brenda Cox talking about coming up with a 7 template for evaluation of survey programs. 8 We're going to have her continue to finish 9 off the petroleum marketing ones and then 10 we've to digest what we can do with the 11 ideas. One of the things that we've already 12 done with template is to use it as an outline 13 for documentation for new surveys. So even 14 if we could have our documentation in a more 15 consistent form that's not a bad outcome of 16 that exercise. 17 Last time on the EIA 826, the 18 electric power sales and revenue report, we 19 talked about an experiment predicting annual 20 data from annual data so we could actually 21 find out how various estimation methods 22 worked. And your recommendation was that we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 38 1 should do a similar experiment to the best we 2 could with monthly data, which is a grand 3 idea, we do plan to do that, we haven't done 4 it yet. 5 You're going to be hearing a talk 6 tomorrow morning that is looking at the 826 7 and the 861 but they're trying to look at how 8 to tie together time series methods with more 9 stationary regression methods in that survey. 10 And that work resulted from an analyst who 11 looked at our historical data and noticed 12 some funny bumps in it. 13 The EIA 914, Guy has talked about 14 that. One of the interesting things, I think 15 it was on a slide but you probably didn't 16 catch it is in the State of Texas, as you 17 know, Randy's student did a lot of work. 18 That state had wonderful data with patterns 19 that could be used in predicting for the 20 future and she helped us come up with a nice 21 way of doing that. 22 Well, in January 2005 Texas changed BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 39 1 the way they're reporting data so there is 2 now a huge change in Texas data and it makes 3 us even more thankful that we now have the 4 914 to help us come up with our new 5 estimation methods. And with that I'll 6 introduce Preston, who's going to give you an 7 update on the 914. As you recall, he gave 8 you some simulation results a couple of 9 meetings ago and you made some 10 recommendations so this is really just 11 finishing up the report on what worked and 12 what didn't work in trying to estimate 13 natural gas production from a sample survey. 14 So Preston. 15 MR. McDOWNEY: Good morning. I'm 16 going to update for the simulation results 17 for the EIA 914. In the spring 2004 meeting 18 Inderjit Kundra introduced the development of 19 the 914 survey to collect monthly data 20 considering a probability proportion to size 21 sample at the time and it goes with the 22 estimated US production with a coefficient of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 40 1 variation of 1 percent at the US level and 5 2 percent at the regional level. 3 We have seven regions, Texas to 4 Federal Gulf, Louisiana, New Mexico, 5 Oklahoma, Wyoming, and the rest of United 6 States excluding Alaska. We're going to use 7 the presumed optimal allocation formula to 8 determine the sample size based on the 9 desired CV. We calculated that the formula 10 of a sample of 176 companies will give us the 11 1 percent CV at the US level but we needed 12 358 companies to get the desired CVs in each 13 region. 14 With that sample in 2004 at the 15 fall meeting I presented the simulation 16 studies. We used simulated four estimation 17 procedures. The first was the probability 18 proportion to size estimator, the weighted 19 least squares estimator, which is the ratio 20 estimator, ordinary least squares estimator, 21 and a difference estimator, which uses the 22 prior, in this case years, response as a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 41 1 predictor for the future. 2 For the weighted least squares and 3 the ordinary least squares procedures we had 4 two methods, the first using the noncertainty 5 group only in estimation procedure and the 6 second using both the certainty and the 7 noncertainty groups in estimation procedure. 8 For both methods we had three variations, one 9 using all the operators in the method, the 10 second removing the outliers, and the third 11 removing outliers and influential variables 12 observations. 13 The results of that simulation, 14 these are the percent errors for the total 15 natural gas production at the US level. All 16 of the procedures, methods, and variations 17 had a negative bias. The probability 18 proportion to size estimator had the smallest 19 bias but a very large variation and range of 20 errors. The weighted least squares, ordinary 21 least squares and the difference estimator 22 all had smaller ranges but a higher negative BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 42 1 bias. So we concluded in the fall that the 2 PPS sampler wouldn't give us the desired 3 accuracy. The ASA committee recommended 4 using the Hajek estimator as a nonbiased 5 estimator. We decided to use a 90 percent 6 cut-off sample at the national level and John 7 Wood made a presentation on the need for a 8 bias adjustment. 9 In the spring 2005 meeting John 10 Wood presented the preliminary January and 11 February 2005 data. He demonstrated the 12 estimation procedure, which is a variation of 13 the ratio estimator, and he presented the 14 adjustment for the negative bias. He 15 reported a 99.4 percent response rate that's 16 volume rated. 17 So after the fall meeting we 18 started Stage 2. We had a new frame and we 19 wanted to see if the frame made a difference 20 in our simulation procedures. We also 21 implemented the Hajek estimator and we 22 implemented a stratified sample. And this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 43 1 time we only did two regions, Texas and the 2 Federal Gulf, and they're the two largest 3 regions. 4 In the first stage we had annual 5 data in the frame and the sources were the 6 EIA-23 survey data. We had state data and 7 multiple third party data. And Stage 2 was 8 monthly data and this time we had a single 9 third party as the data source. The first 10 time we had the seven regions in a way 11 integrated so if we picked a company in Texas 12 if they were put in any other region we took 13 their data also. In the second stage we 14 simulated each region individually so there 15 was no integration here. 16 With the new frame we used the same 17 formula as in the first one but we got 18 different sample sizes. In Texas we had 96 19 companies and 20 in the Federal Gulf. In 20 Stage 2 there were 61 companies and 19 in the 21 Federal Gulf, and that's due just to the 22 frame. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 44 1 We had the same four estimation 2 procedures and we added the Hajek and the 3 stratified. This is the Hajek estimator, the 4 stratified sample versus random stratified 5 sample with replacement. We had six non- 6 certainty strata in Texas, two non-certainty 7 strata in the Federal Gulf. 8 We're going to summarize the whole 9 project by showing the percent errors in the 10 total estimated regional production and I did 11 that by just summing all the monthly 12 estimates within a region. The stratified 13 sample had the strongest negative bias and a 14 fairly large range of errors. The PPS sample 15 had a much reduced negative bias but a very 16 large range of errors. 17 The Hajek was a little less biased 18 and smaller range of errors. The difference 19 in the model-based methods, the difference 20 estimator had a very small bias and very 21 small range of errors. 22 I'm going to skip to the last two. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 45 1 These are the model-based methods. When you 2 add the certainty and the non-certainty 3 together you really reduce your range of 4 errors and moderately improve on the bias as 5 opposed to when you don't add the certainty 6 companies. 7 We get a similar story in the 8 Federal Gulf. The stratified sample, that's 9 the best prospect but there are only two 10 strata and seven companies in the non- 11 certainty category. The Hajek does give you 12 a much smaller ———————— than the PPS sample 13 in the Federal Gulf. The difference 14 estimator performance is pretty much the same 15 and the model-based does reduce the range of 16 errors but, unlike in Texas, actually 17 increases your bias. 18 So in general the probability 19 methods had the largest variation in many 20 cases especially you need to go back and look 21 at the months themselves with the most 22 negative bias. The difference estimator and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 46 1 the model-based estimators with the certainty 2 and the non-certainty companies had the 3 smallest variation and least bias. And we 4 didn't show them but outlier and influential 5 variables made little difference in the 6 results. So that's my update. 7 DR. HENGARTNER: Thank you very 8 much. We will now also hear the EIA regional 9 short-term energy outlook by Margot Anderson. 10 MS. ANDERSON: Good morning, 11 everyone. I want to thank Nancy and Bill for 12 inviting me to come and speak with you today 13 updating you on what we've done to the 14 regional Short-Term Energy Model. I was here 15 in April to talk to you about work that was 16 currently underway. As Guy mentioned in his 17 presentation, it's become a workhorse for us 18 over the last couple of months. 19 Back in the halcyon days of August 20 before the hurricanes when we were just 21 starting to launch the regional model little 22 did we know how useful it might be during the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 47 1 hurricanes and after to look at the aftermath 2 and to be able to disaggregate some of the 3 results at a finer level of detail than we 4 were able to before. 5 So I want to give you an update. I 6 want to recap from 2005. I want to look at 7 the schedule because there is still work for 8 us to do. I want to give you an overview of 9 the key components and some of the challenges 10 that we're facing as we go forward. I remind 11 you that the purpose was to expand the 12 national model into more regional detail than 13 we had had before. Clearly we are 14 constrained by the amount of subregional data 15 that we have available to put in to the 16 model. 17 In many cases we can go to the nine 18 census regions, as Guy mentioned. In some 19 cases we can't break it down into that much 20 detail but we do try to regionalize to the 21 extent that we can regionalize for most 22 energy sectors. And we did this in the hopes BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 48 1 of providing additional relevance to the 2 model, to enhancing our ability to look at 3 regional supply, demand, and prices, clearly 4 to answer the kinds of request we get at the 5 regional level. 6 Guy is absolutely right. I think 7 people would like a household projection. 8 We're unable to do that. We get a lot of 9 requests still, of course, at the state level 10 and the regions are not fine enough for a lot 11 of our customers who would like us to go into 12 more detail. We just don't have the state 13 level data to do that in most cases. 14 We want to add depth, greater 15 understanding of what's going on. It's not 16 just to feed the customers but it's for us to 17 understand how these markets are interacting 18 particularly at a regional level. And there 19 are lots of disparities in supply, demand, 20 and price across the regions and we thought 21 it would be helpful and enrich our analysis 22 of the short-term energy situation and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 49 1 outlook to have a more regionalized model. 2 So that was our motivation. We 3 talked about wanting to launch in the summer 4 of '05. I'm happy to report we were able to 5 launch in August, as I said, the halcyon 6 days. Before the hurricanes we thought it 7 might be smooth sailing, we would launch 70 8 percent of it in August, we would continue to 9 do the updating in September and October, in 10 November we'd have the whole thing ready. 11 Well, we got hit by Katrina and by 12 Rita and it meant that we weren't able to do 13 some of the development that we wanted to and 14 continue working on the electricity dispatch 15 model, which is still not operational but 16 will be shortly, and we wanted to focus more 17 of our efforts on using the regional model to 18 put out the September and the October STEO. 19 In September we did something 20 unusual because it had been just a few days 21 since the hurricane. For one of the first 22 times ever we did a non-single estimate. We BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 50 1 did a forecast that looked at a slow, medium, 2 and fast recovery, which isn't something we 3 normally do in the short-term energy outlook. 4 We tend to have a baseline outlook. But we 5 were able to look at three different recovery 6 patterns because in those first days we 7 didn't really know where it was going, we 8 didn't have a good assessment of the damages, 9 and we didn't want to commit ourselves to a 10 single baseline forecast so we had three 11 forecasts. 12 By the time Rita hit we had about a 13 good three weeks to figure out what was going 14 on in the Gulf and the October forecast went 15 back to a single baseline, a medium recovery 16 forecast, but again was also coupled with the 17 winter fuels outlook. You saw many of the 18 slides that were used in the winter fuels 19 outlook in Guy's presentation and we were 20 able to look at heating expenditures for the 21 winter in most cases for the nine census 22 regions, something we had not been able to do BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 51 1 before. 2 So, as I said, we're going to get 3 the electricity component working in the next 4 couple of months. The delay was really due 5 to the fact that the hurricanes just 6 sidestepped us from the development angle to 7 the use angle. We are ongoing in our expert 8 review. We have all of that documentation 9 ready to be reviewed and we are working with 10 SMG to put together a panel to review the 11 documentation. We made the decision to go 12 live without having the documentation 13 reviewed. 14 We just didn't feel that we could 15 do them in the sequence that we would 16 normally do, review and then go live. We 17 wanted to go live and we will look at this as 18 a living model. If we need to make the 19 adjustments based on comments we get back 20 from the expert reviewers we're more than 21 happy to do so but we really wanted to 22 implement the regional model so the expert BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 52 1 review will occur in '06, Bill, '06. 2 The recap, as I said earlier, we've 3 broken it into these sectors and we have the 4 regional components associated with each 5 sector, far more information than we had 6 before. We also have an expanded query 7 system associated with the STEO models so it 8 allows you to go in and get the data that we 9 use for the subregions that we're looking at. 10 We did get rid of the interactive 11 model that people could use to generate their 12 own estimates in STEO. It's too unwieldy to 13 make the regional model available in that 14 kind of a PC format. If we ever get there we 15 might be able to make it available sometime 16 in the future but for now we have not made 17 that feature available so that is one service 18 we're not providing to customers but we're 19 giving you all the detail in terms of the 20 data that we can and will be as transparent 21 as possible in the website with the 22 documentation. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 53 1 Regional detail varies. I 2 mentioned this earlier. It just depends on 3 what region and what field you're looking at 4 to determine how much data we can get to a 5 more disaggregated level. We talked last 6 time about there were strong regional 7 differences, availability of data, more 8 limited data for propane and for heating 9 fuels. 10 For heating oil it doesn't matter 11 as much because a lot of those fuels are used 12 in a very concentrated area. So sometimes it 13 doesn't matter as much that you don't have 14 the regional detail that you'd like to have. 15 These are the regions. I'm sorry it doesn't 16 show up better. It didn't color on the STEO 17 page. But this gives you an idea. We have 18 the census regions and census divisions and 19 this is how we've broken out in the winter 20 fuels outlook in the October STEO so you can 21 look at prices and expenditures for those 22 regions which has been very helpful this last BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 54 1 time out because we were able, as I said, to 2 go into more detail which matters more when 3 prices were higher and supplies were tighter. 4 In other years where you might have had 5 expected a more normal year there probably 6 would have been less emphasis on the regions 7 because there wasn't as dramatic a change 8 between this year and last year in terms of 9 expenditures. 10 And I think that showed up nicely 11 on Guy's presentation where he showed the 12 damage associated with the hurricanes and the 13 expectations on prices that are attributed in 14 part to the hurricanes, in part to the 15 general situation we find ourselves in tight 16 global markets and tight domestic markets. 17 But the winter fuels outlook 18 mattered more this year because of these 19 dramatic changes in the energy sector so it 20 was nice to be able to have additional 21 information for people to make decisions on 22 and for policy makers to consider where there BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 55 1 might be regional disparities associated with 2 heating expenditures. 3 Observations, I've been at EIA a 4 year, about 14 months, and so I'm looking at 5 it as somebody that jumped into the 6 development of the regional model while it 7 was ongoing. I was not there at the 8 beginning when a lot of the work was really 9 developed so I certainly can't take any 10 credit for that nor could I take credit for a 11 lot of the day to day work that goes on in 12 it. That is attributable to Mark Rodekohr 13 and Dave Costello, who are part of the STEO 14 team and lead the STEO team. 15 It is resource-intensive to do this 16 kind of work. That sounds like an obvious 17 observation, but I think probably more 18 resource-intensive than we might have 19 anticipated when we walked down this path; 20 however, the value that you get out of a 21 regional model is quite large. 22 You asked earlier about are we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 56 1 capable of looking at some of these energy 2 emergencies. The regional models can allow 3 us to do that better. As we develop it even 4 more and more it's even going to add to that 5 capability. 6 So we use something called the 7 rules of thumb when we look at international 8 supply disruptions and it's a fairly simple 9 way to look at supply shortages and how they 10 might affect GDP in the United States and how 11 they might affect oil prices. This model 12 would be able to help us look at disruptions 13 whether they are happening in the United 14 States due to a weather event or somewhere 15 else; it's almost irrelevant where it 16 happens. 17 The regional model will allow us to 18 look at some of these contingencies. So I 19 think this is a powerful tool particularly, 20 because we didn't have this capability on the 21 natural gas side or on the electricity side 22 which we will have in the not too distant BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 57 1 future. So, again, I said earlier, looking at 2 regional disparities and conducting 3 contingency analysis, it's going to be a very 4 powerful tool. 5 And it can also be a fulcrum for 6 data collection and data sharing within EIA. 7 My office is very dependent upon all the 8 other offices in EIA to get the data to put 9 into the model and so it means that there 10 often has to be more cooperation and 11 coordination across the offices to make sure 12 that we're getting the data that we need in a 13 timely fashion to put into the STEO. 14 So I think it enhances the 15 collectiveness of the data and how the data 16 are used in a forecasting and modeling 17 framework within EIA and has raised a lot of 18 questions about data that are collected and 19 the turn-around of data to get them prepared 20 and model ready for the STEO model. So it's 21 raised some issues and some challenges that 22 we have here in EIA on dealing with some of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 58 1 the data issues of making sure that STEO is 2 ready to go every month without fail. We 3 can't delay STEO. It's on the same kind of 4 schedule that our weekly data series are on. 5 It's not optional. It has to go out every 6 month. 7 I urge you to look at the regional 8 STEO. You can look at it in the new web page 9 in the not too distant feature. The STEO is 10 now completely redesigned and we think it's 11 much easier to navigate through the tables, 12 through the text, through the links. The 13 query system is there. We've added some 14 interesting little features that I think make 15 it a much more user-friendly product and a 16 much more attractive product. It's always 17 been a high quality product and remains so 18 today, I think, and the regional has only 19 enhanced it. 20 So I can answer any questions. Do 21 you want me to answer questions on the 22 regional model at this point before I go into BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 59 1 forecasting? 2 DR. HENGARTNER: Yes, Moshe. 3 DR. FEDER: Margot, do the regional 4 STEOs add up to the national STEO and how is 5 it accomplished if so? 6 MS. ANDERSON: Yes and no. We 7 still have a national forecast and sometimes 8 it's an adding up and sometimes it's a 9 national forecast that is decomposed down. 10 So it isn't such a seamless system that every 11 little regional area adds up to a national 12 estimate. Mark, stop me when I go into an 13 area I don't need to go into. 14 DR. FEDER: Whether you aggregate 15 up or disaggregate down it always adds up, 16 right? There's no -- 17 MS. ANDERSON: Well, it has to. I 18 mean, there is no separate national forecast 19 and then we match it to a separate system 20 that's a regional forecast. They're an 21 integrated system and so the real question is 22 are we getting a different national forecast BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 60 1 than we would've gotten without the regional 2 model. 3 DR. FEDER: That's a good question. 4 MS. ANDERSON: That's an 5 interesting question. We haven't answered it 6 yet. Mark -- 7 MR. RODEKOHR: Yes, you are. 8 MS. ANDERSON: Thank you, well, we 9 are. 10 MR. RODEKOHR: Yes, we do add 11 everything up but the regional area is 12 differed by the fuel. For example, petroleum 13 is based on a pad structure and natural gas 14 is based upon a census division structure. 15 But we do add it up and it's not a top-down 16 notion where you get a national forecast and 17 you share it out. It's the other way around. 18 We get the regional forecast and we build 19 them up. 20 MS. ANDERSON: But the question 21 about whether it's the same as we got with 22 just the national model. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 61 1 MR. RODEKOHR: When you add up 2 things over certain time spectrum they're not 3 a lot different in many respects. A good 4 example is what Guy did here with the fuel 5 outlook. When you look at the regional 6 differences on weather patterns and you add 7 them all up it really doesn't make a lot of 8 difference. But it's important to the 9 regions themselves as to those difference. I 10 don't know if that's helpful. 11 MS. ANDERSON: If we could really 12 disaggregate, looking at forecast errors 13 before the regional model and forecast errors 14 after the regional model, that might help you 15 determine levels of accuracy when we were 16 using just the national model or whether we 17 were using just the regional model. What 18 matters now is the model that we're using is 19 as accurate as we can possibly get it. 20 MR. RODEKOHR: Can I say one more 21 statement? It may be easier to get a much 22 better accurate forecast at a national level BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 62 1 because all sorts of things are balancing out 2 but from a regional perspective it really is 3 important to get those things better and 4 that's what we could not do before. 5 DR. HENGARTNER: Mark? 6 MR. BERNSTEIN: I might say I 7 really liked when you did the different 8 scenarios after Katrina stuff and I would 9 encourage you to think about continuing 10 something like that rather than giving a 11 single estimate. Even in the short term as 12 you go along there is always something that 13 you look out the next couple of months and 14 say well, there's something we're not 15 particularly sure of but there is some 16 uncertainty critical out there in the next 17 few months and it would be nice to recognize 18 that and perhaps let that one float. And it 19 may be different ones in different months 20 where is this winter going to be particularly 21 cold or something else. 22 And so I would encourage you to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 63 1 think about having a range even in the short- 2 term outlook. So that's just one comment. 3 The question is in the short- term 4 energy outlook are the regions the same for 5 fuels and electricity or are they like NEMS, 6 the regions are different? 7 MS. ANDERSON: No, the regions are 8 different but, getting back to the first 9 point, it's a source of some debate within 10 EIA or at least friendly conversation about 11 how much our forecasting models ought to try 12 and hit a bull's-eye. Now, the parenthetical 13 here is that, of course, in the winter fuels 14 outlook we do a colder and a warmer scenario 15 so that we give estimates depending on 16 whether we expect it to be 10 percent warmer 17 or colder from the NOAA forecast. 18 But many other forecasters will put 19 out reports that might look at cleverly 20 titled worst case scenario or intensive 21 hurricane season forecast and so they will 22 give this range of estimates depending on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 64 1 what they think might be something that would 2 disrupt the market over the coming months. 3 Certainly we know that these markets are not 4 smooth sailing and if you look back one year, 5 you look back five years, you look back 10 6 years there is always some unanticipated 7 event. And to be able, maybe, to anticipate 8 those better or to give a range on estimates 9 is certainly something that the NEMS model 10 does on a regular basis. 11 They run 30 cases or so every year. 12 They may highlight the reference case and 13 talk mostly about the reference case but 14 they're always doing these other scenarios. 15 We have tended not to do that in the STEO 16 framework mostly because of time constraints 17 because it has to get out every month and 18 then the text constraint of being able to 19 talk about those other scenarios but I don't 20 think it's an issue that we've put to rest 21 yet. 22 I'm always concerned about making BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 65 1 it appear as if we're trying to hit a 2 bull's-eye every month when in fact what 3 we're really trying to do is generate a 4 reasonable story and a discussion about what 5 we think is impacting the markets and to give 6 a fairly accurate assessment of what we think 7 the interactions are and what the prices are 8 going to be and demands, et cetera. 9 So I think there might be some 10 value in that because often people will take 11 the estimate and really misuse the point 12 estimate and they won't be nuanced by what we 13 all know as forecasters and researchers that 14 there are some wide bands here. And it's our 15 obligation to educate people about the 16 uncertainty that might be associated with 17 these estimates. So I think it's a point 18 well taken. 19 DR. HENGARTNER: Yes. 20 MR. RODEKOHR: One last point, if 21 you do look in the last 100 STEOs you will 22 notice on world oil prices there is a red BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 66 1 band and it has always been there. And so 2 you can't say we've ignored that uncertainty. 3 People just don't focus on it. 4 MS. ANDERSON: The confidence 5 interval that we put on some of the prices. 6 MR. RODEKOHR: Yes, on some of the 7 prices, right. 8 MS. ANDERSON: But we don't focus 9 on it either. I mean, we have the red band 10 but we don't talk about what that really 11 means and what's motivating us to do that and 12 how people should interpret that information 13 so point well taken. 14 DR. SITTER: How are you generating 15 that red band? Is that scenario variability? 16 MS. ANDERSON: Mark. 17 MR. RODEKOHR: The red band is 18 generated upon the standard deviations from 19 the model we use to do prices. Now, again, 20 that's the model we use but the real point 21 estimate on the prices is done by more of a 22 Delphi approach where we sit around and we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 67 1 talk to each other and we get that but we do 2 have a model and it does give you two 3 standard deviations away from it and that's 4 where the red band is generated from. 5 And I would like to say if you went 6 back historically and looked at how well we 7 did versus that red band the model's probably 8 pretty close. In other words 95 percent of 9 the time you're going to be within that band 10 understanding the limitations, there are 11 monthly prices, there are averages, et 12 cetera. So that's how we do it. 13 MS. ANDERSON: But we could test 14 that. 15 MR. RODEKOHR: Well, it's very 16 time-consuming, yes. 17 DR. HENGARTNER: What I liked about 18 Mark's initial comment that started this 19 discussion is that in fact the model is 20 changing. You're changing the assumption in 21 the model and so this red band although tied 22 to standard deviation is useful. The BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 68 1 question is how sensitive is it to model 2 departures that can come from external shocks 3 like the hurricanes. 4 MS. ANDERSON: Well, the red band 5 is only on a couple of price variables that 6 are exogenous to the model. 7 MR. RODEKOHR: That's right, 8 exactly. It's not all-encompassing. That's 9 correct. It's on the most price variable 10 that a lot of people come to the STEO for a 11 WTI. 12 DR. SITTER: Right, that was the 13 point I was trying to make is that's not 14 really addressing this issue. This is a 15 completely different measure of variability 16 than the issue of different scenarios. 17 Different scenarios could actually be handled 18 in a more Bayesian framework if you wanted 19 to. 20 MS. ANDERSON: If we use the STEO 21 mostly to generate the short-term energy 22 forecast I'm hopeful and my vision is that we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 69 1 would be able to use the STEO to do analysis 2 outside of just generating the forecast so 3 you would see that you might put in a couple 4 of different scenarios of what-ifs associated 5 with either some kind of an event or a change 6 in policy and be able to look at a short-term 7 impact by using the short-term model. 8 We've tended to use it for the STEO 9 product itself as opposed to a workhorse of 10 energy analysis and then in that case you 11 would certainly want to explore alternatives 12 but it's a point well taken. 13 DR. HENGARTNER: Well, thank you 14 very much. 15 MS. ANDERSON: You're welcome. 16 DR. HENGARTNER: Before continuing 17 I'd like to introduce to you Tom Rutherford, 18 who just joined us. He is a new committee 19 member from Colorado, correct? 20 MR. RUTHERFORD: Michigan. 21 DR. HENGARTNER: So welcome very 22 much to our committee. Those members of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 70 1 audience who have joined us since we started, 2 if you could please introduce you into the 3 microphone for the records, please. 4 MR. SINGPURWALLA: Darius 5 Singpurwalla, LECG. 6 MR. BLAIR: Johnny Blair, Abt 7 Associates. 8 MR. SCHIPPER: Mark Schipper, EIA. 9 MS. McGUCKIN: Nancy McGuckin, 10 Department of Transportation. 11 MS. WAUGH: Shawna Waugh, 12 Statistics and Methods Group. 13 MS. TAYLOR: Yvonne Taylor, EIA. 14 MR. RODEKOHR: Mark Rodekohr, EIA. 15 DR. HENGARTNER: Thank you very 16 much. Before continuing I again want to ask 17 if any member of the committee has some 18 questions for Nancy's or Preston's 19 presentations. I know it's a little bit back 20 but I'd like to wrap up the update for the 21 committee section. 22 DR. SITTER: I have one question BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 71 1 for Preston. Is it possible to get a 2 write-up together with the previous write-up 3 because this was too terse for me to 4 understand what has been done since the 5 previous write-up. 6 MR. McDOWNEY: Sure. 7 DR. SITTER: Thanks. 8 DR. HENGARTNER: Anything else? 9 Okay, thank you very much. So let's 10 continue. The next presentation will be on 11 short-term forecasting performance measure 12 and accuracy evaluation by Margot, yes. 13 MS. ANDERSON: I'm going to talk 14 about the STEO performance indicators 15 project. I have my partners in crime here as 16 usual, Mark Rodekohr and Mike Morris, who 17 have done all the yeoman's work on this. I'm 18 just here to try and provide context and put 19 into an approach that not only talks about 20 some of the technical matters but why we're 21 doing it and what we hope to do with it in 22 the future. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 72 1 This is a growing program. It came 2 out of a conversation that we had last time 3 we here, in April, about what to do, about 4 really understanding how well we're doing 5 with our forecast. And it isn't so much to 6 be a report card and to say you got an A or a 7 B or you were absolutely right or absolutely 8 wrong. But we really needed a way to do some 9 performance indicators to help us improve 10 what is now a much more complicated model as 11 well as to take a look at how well we're 12 doing. So it's really to serve a couple of 13 purposes to develop a systematic way to look 14 at performance indicators and forecast 15 errors. 16 So when I was here last in April we 17 discussed the necessity for developing some 18 diagnostic tools for gauging forecast 19 accuracy and we've got 1,500 variables in 20 this thing now that we could look at and that 21 we're forecasting and do we look at all of 22 them, do we look at every conceivable way to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 73 1 measure forecast accuracy or do diagnostics, 2 and what might be the minimum set to give us 3 the maximum information? So we had some 4 conversations about that. 5 We discussed how to use the 6 information not only to troubleshoot data 7 problems but to improve the model, maintain 8 confidence in EIA's forecasting capability. 9 We tend not to report forecast errors on a 10 regular basis for STEO to the general public. 11 We do keep some track internally to meet some 12 OMB requirements but we have not tended of 13 late to have a program that is transparent to 14 the users of STEO about how are you doing. 15 We also discussed how we might 16 share these results. It can be burdensome to 17 do this every month and to have a long report 18 about what we've got and what it means but 19 there may be a low cost way or a relatively 20 straightforward way to tell our readership 21 what we are showing in terms of forecast 22 errors and diagnostics. We think that's a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 74 1 good goal. 2 So what did we do? Well, since 3 April while we were working on the regional 4 model we also talked about doing some 5 preliminary work on looking at some forecast 6 errors and choosing some metrics and taking a 7 look at what they were telling us. These are 8 not final results. These are not looking at 9 every variable. I think we looked at about 10 40 variables. I'm going to show you about 15 11 to show you the kinds of metrics we're 12 looking at and what some initial results are. 13 So it's an ex-post analysis. We 14 examine how are the projections compare to 15 the data but we don't go back and revise the 16 data in this particular presentation. 17 Whether we need to do that in the future is 18 determined by how much did the data change 19 from revision to revision to revision. 20 That's time-intensive. 21 This analysis starts with monthly 22 forecast in '03. We can go back longer on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 75 1 many series. We just did '03 for this 2 exercise. And what we're going to be looking 3 at is our one-month, our six-month, and our 4 12-month ahead forecast errors so I'm not 5 going to be talking about any diagnostic 6 tools of looking at outliers and 7 distributional concerns about the data. This 8 is primarily a discussion about forecast 9 errors. 10 We looked at a couple of standard 11 measures, we looked at the mean absolute 12 error, we looked at the mean absolute percent 13 error, we looked at the bias to tell us the 14 percent of times that we were under or over- 15 predicting. In some cases we're under- 16 predicting 100 percent of the time but only 17 by a little bit so you want to look at a 18 couple of things to determine how well you're 19 doing. We looked at an F-statistic to look 20 at the statistical significance between the 21 actual data and the predicted data to tell us 22 something about how off we were, given the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 76 1 incredible variation in some of the data 2 series alone. 3 So some initial observations, no 4 surprise to anybody in this room. The price 5 data, particularly the international price 6 data like WTI, are really hard to predict and 7 we're usually wrong. Sometimes we're wrong 8 by a little bit, sometimes we're wrong by a 9 quite a bit more, but we're pretty good and 10 pretty consistent when we're looking at 11 variables that are within our control in the 12 sense that we have modeled the markets 13 carefully. 14 The demand, the consumption 15 variables, for most sectors are relatively 16 stable. The price variables are the ones 17 that give us fits. So those are some initial 18 observations. Obviously those that are in a 19 self-contained US system we tend to be able 20 to model that and where it tends to be less 21 variability there tends to be more knowledge. 22 The data also tend to be better. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 77 1 I hope you can read this. Mark, 2 you're the one that's most far but can you 3 read this at all? 4 MR. RODEKOHR: Barely, yes. 5 MS. ANDERSON: Well, squint. I'll 6 try and walk you through here. What we're 7 looking at, the month ahead is the 1, the 6, 8 the 12. The first three columns are the 9 absolute percent errors that tell you on 10 average how much you're off. Then there's 11 the bias that looks at the percent of time 12 that you're under or over and then the F- 13 statistic, which is testing the difference 14 between the data series and the forecasted 15 series to determine whether you are 16 statistically different. If so you better go 17 look and find out what's going on. 18 These are the prices, the WTI and 19 motor gasoline and heating oil and the 20 natural gas prices, and you can see that the 21 bias in some cases is 100 percent. So for 22 WTI since 2003 100 percent of the time we've BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 78 1 been under-predicting. When we're one month 2 ahead we're under-predicting 23 percent. The 3 error widens the further out we go. So not 4 so bad one month ahead, not so great six 5 months ahead, even worse 12 months ahead. 6 And that's the case on the bias as well. Our 7 predictions are progressively off as we go 8 out for the one or the six or the 12 month 9 ahead. 10 Again, that's not really surprising 11 and I would venture to say that if you were 12 looking at other forecasters that are playing 13 this game they've got a similar track record. 14 That doesn't mean we can't do better and in 15 fact we should be able to use this 16 information to circle back into the modeling 17 framework and say if we're under-predicting 18 all the time maybe we ought to examine what 19 our priors are about why we're under- 20 predicting. 21 If we're under-predicting just a 22 little bit, 1 or percent, I tend to be less BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 79 1 worried about it. If we're under-predicting 2 by these larger percentages that's something 3 that we need to examine in how we're coming 4 up with some of these price variables. 5 Again, the WTI is determined exogenously, 6 outside of the model. The others are related 7 to WTI so they feed off wherever we start 8 with WTI. While natural gas has its own 9 markets and prices are much more determined 10 within US borders it is still often pulled up 11 or triggered by what's going on in WTI. 12 So this is giving us some insight 13 into how we're doing. And one of the 14 questions that I'm hoping you guys can help 15 us with is to say well, why aren't you 16 looking at these four other statistics. Why 17 didn't you do this kind of analysis? We 18 would like some feedback on what might be 19 other approaches that we could take to 20 looking at these variables that would lend 21 themselves to fairly easy interpretation so 22 that if we put them out on a website BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 80 1 customers would get an easy understanding of 2 what this meant. We don't really want to 3 make this purely an academic type of 4 exercise. Jae. 5 DR. EDMONDS: Margot, would you 6 just remind me what's WTI again? 7 MS. ANDERSON: It's the West Texas 8 intermediate crude price. It's a benchmark 9 for crude oil prices. 10 DR. EDMONDS: Thank you. 11 MS. ANDERSON: Motor gasoline is 12 what you get at the pump. The blue line is 13 the one-month, the next dotted line is the 14 six-month, and, of course, the 12-month is 15 the shorter line, less at the series. You all 16 have these in your handouts. I'm giving you 17 some more statistics about what we're looking 18 at. The solid line is zero so it's showing 19 you how much above and beyond we are 20 predicting depending on how far out we go. 21 Again, it's not clear to me whether 22 the best estimate is the one-month ahead or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 81 1 the six-month ahead. Is it the 12 month 2 ahead? What is the best measure of a 3 forecast error if you don't want to flood 4 your audience with every conceivable 5 permutation of every kind of statistic or 6 variable to look at? So we have some 7 deciding to do about that minimal set of 8 variables and statistics to give maximum 9 information. 10 There are gasoline prices again. 11 This is just really a picture of what that 12 summary table showed you. Mike, correct me 13 if I'm wrong. The scales on these are all 14 different so your eyeball can't compare one 15 to another but the price variables, again, 16 have the higher F-statistics than some of the 17 demand variables, which I will get to. 18 Wellhead natural gas prices, this 19 is indicative of what a volatile market this 20 has been over the last couple of years and 21 how difficult it has been to anticipate these 22 changes in the natural gas market and I think BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 82 1 that this picture really demonstrates that 2 story. 3 A couple of demand variables, we 4 looked at natural gas and looked at 5 electricity sales data. Again, it looks a 6 whole lot better in terms of the F-statistic. 7 The bias is telling us that as we move out we 8 were over-predicting on the natural gas 9 demand. I suppose this is intuitive. The 10 further out you go the more difficult it is. 11 I guess that makes some sense. If the one- 12 month aheads are really based on last month 13 you may tend to always do better one month 14 ahead than you would six months ahead or one 15 year ahead. I'm thankful I'm not in John 16 Conti's shoes because he has to predict 20 17 years out and that's a difficult thing to do. 18 We've got pictures of these as well 19 of the petroleum demand. Again, we're much 20 closer to where we need to be, which is zero, 21 which is, as I said earlier, on these demand 22 and consumption variables we do tend to do BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 83 1 better, on natural gas demand a bit more off 2 the mark related to what I said earlier about 3 the vast changes in this market. 4 Total electricity demand, our 5 regional model will help us with this. None 6 of these are regional, by the way. These are 7 all national level estimates. It goes back 8 to the question that we were talking about 9 earlier and what Mark was saying about the 10 ability of the regional models to hit their 11 mark. We haven't yet incorporated into this 12 regional forecast errors because we don't 13 have the time series to do so but eventually 14 we would want to look at how the regional 15 models are doing as well. Right now it's 16 just on the national side. 17 So we are in the early stages of 18 this and we are looking for help and looking 19 for feedback on what we can do. We want to 20 write this up in a paper. As we move on we 21 want to write it up in a more academic paper 22 and justify why we're picking what we're BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 84 1 picking and being able to show a bit more on 2 what these data look like in terms of their 3 distributions, in terms of outliers, and in 4 terms of just some basic diagnostic tools of 5 looking at the data series. 6 I think being able to automate a 7 lot of this will really help us now that we 8 have this much larger model and to pick out 9 some anomalies that are inevitably going to 10 creep in when we have over 12,000 variables 11 that are in the model, maybe predicting 12 1,400, so it's a very complicated model. 13 So what statistic should we look 14 at? What variables should we look at? The 15 obvious ones are on the front page every day 16 but there are a lot of other interesting 17 variables that can provide tremendous insight 18 as to how well the model is doing that ought 19 to be added to our menu of variables that we 20 want to check out. It's, again, not just for 21 public consumption. It's to feed back into 22 making the model a better model. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 85 1 We certainly want to focus on the 2 areas where there are significant error and 3 where we can do something about it. In some 4 cases there is not much we can do about it. 5 There may not be that many variables or 6 structural equations in the model that we can 7 change depending on where the variable comes 8 from if it's determined exogenously. But 9 there are certainly some ways that we can 10 consider options to reduce the uncertainty or 11 the forecast error by adding variables. 12 There is seasonality in the models 13 and we haven't adjusted for that in these 14 forecast errors. And it might be wise if we 15 talked a little bit about the value of doing 16 some turning point analysis. There are quite 17 a few turning points in these kinds of data 18 and some of these metrics are not so great 19 when you get close to a turning point. And 20 so maybe doing turning point analysis might 21 be something that would be useful on some of 22 these series so that's something to consider BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 86 1 as well. 2 And we've got to figure out a way 3 to publish this without creating a huge 4 burden on the staff to do this but creating 5 as much transparency as we can. I think 6 we're obligated to our customer to let them 7 know how well we do and we hope that that 8 would engender others in this game to play 9 along. There are lots of people forecasting 10 and there are lots of people that don't have 11 the credibility that we have and don't have 12 the obligation to be transparent and 13 responsive. 14 And so I think this is a part of 15 EIA's overall agenda, to reveal what we can 16 reveal and to keep the discussion open and to 17 generate feedback, solicit feedback, and make 18 the adjustments where we can. These are 19 official government statistics and we need to 20 feel confident of them. I've said all along 21 when we started this process that it wasn't 22 so we could have an uh-huh and point a finger BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 87 1 at somebody and say see, I knew you were 2 wrong. 3 I have great confidence in this 4 model and I think we're doing probably pretty 5 well compared to what others do and I think 6 we're pretty accurate to the extent that we 7 can be. There is always room for improvement 8 but I would like to have a tool to be able to 9 say not only am I relatively confident but 10 that Guy Caruso gets up somewhere and talks 11 about the results from this model that he has 12 the confidence that he needs because he has 13 the statistics behind him to show that what 14 we're doing is as good as we can do given the 15 volatility of these markets and the sometimes 16 unpredictability that goes on in these 17 markets. So we're looking to you guys to 18 help us out on making this a better program. 19 Mike or Mark, did you want to add 20 anything to this presentation? 21 MR. RODEKOHR: Just one little 22 thing, I think exactly what you said is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 88 1 right. We used to publish the forecast 2 errors and I agree we ought to started it up 3 again. Budget reasons stopped it but 4 nevertheless I do think that that adds 5 something to what other people don't do. You 6 will find that the other private consultants 7 will always tell you when they were right. 8 They will never tell you when they were 9 wrong. And to the extent we can add an 10 historical record and be transparent about it 11 and say here is what we did right and here is 12 what we did wrong I think that's a value 13 added to what we're doing. That's all. 14 Michael. 15 MR. MORRIS: Again, my name is 16 Michael Morris. I work with the STEO team 17 and I look at petroleum demand issues 18 primarily. I don't want to duplicate or 19 replicate what Margot or Mark has said. I 20 just say that I have output for the 37 series 21 that we currently cover which are all 22 national in nature. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 89 1 We'll do the regional, perhaps, 2 starting in a year or two from now when we 3 have more of a track record behind us. And 4 if anybody wants to look at my wonderful F- 5 statistics and biases and all the rest of it 6 I'd be happy to show you so you get a flavor 7 of what this is about and I'd be happy to 8 stick around and show it to any interested 9 parties. Thank you, that's all I have to 10 say. 11 MS. ANDERSON: Thank you, Michael. 12 MR. MORRIS: Any other questions 13 for me or anybody else on this? 14 DR. HENGARTNER: So actually, we 15 have two discussants right now and you're one 16 of them, Tom. 17 MR. RUTHERFORD: Sure. 18 DR. HENGARTNER: So you may as well 19 start with your comments and you can weave in 20 your questions and then Moshe also will have 21 some comments and I probably will open up the 22 discussion to the floor after that. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 90 1 MR. RUTHERFORD: I'm a newcomer 2 here so I wasn't completely certain of what 3 my role was going to be but I'm very happy to 4 be here and I really found the presentation, 5 the discussion, and the broader thing quite 6 stimulating. I should also say at the outset 7 as a caveat that I'm really not a time series 8 forecaster in terms of the details of this. 9 I'm more of a user of this type of work for 10 the purpose of constructing policy-oriented 11 models. 12 Let me first begin by being a bit 13 of the devil's advocate and just reflecting a 14 bit on what the mission is because there is a 15 lot of discussion about customers as though 16 were a profit-making forecasting enterprise. 17 And I would submit that my view of it, and 18 this is just my personal view, is that the 19 purpose, the main reason, for having a model 20 and paying attention to what the model does 21 and how it forecasts is to have a better 22 handle on what data are important to be BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 91 1 collecting. 2 If we think about private goods and 3 public goods that are being produced within 4 the Department of Energy the public component 5 is the information, what data do we need to 6 generate in order that someone could do a 7 good forecast. And by doing a forecasting 8 model you get a good handle on what data is 9 important and could be useful. There is this 10 public component to that in that once you 11 produce the data then it's useful to all 12 sorts of people. 13 And clearly the types of things 14 you're forecasting there's a strong incentive 15 for the private sector to want to do this 16 right as well. So if you can produce the 17 data that permit the private sector to do 18 these forecasts well you've accomplished a 19 large fraction of your objective. I don't 20 think you have to worry about you're the one 21 who is forecasting for mom and pop's delivery 22 service and that if they don't know what the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 92 1 prices are they're going to go out of 2 business. 3 The fact of the matter is that mom 4 and pop's store has their own incentive to 5 look at what information they need and what 6 forecasts they need to buy and there should 7 be a market that they have for going out to 8 buy the forecast; however, the private 9 sector, no individual firm that's doing the 10 forecasting has the incentive to go out and 11 do the surveys to collect the data they need 12 to do a good forecast so the purpose of the 13 government agency being involved is to 14 identify what data is important and to help 15 do it. 16 Having said that, I think we also 17 have to be a little bit humble. I don't know 18 how many of you enjoy watching hurricane 19 forecasts. This has been a good year for 20 doing that. It's one of the things that 21 whenever I watch hurricane forecasting done 22 by modelers I'm intrigued by the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 93 1 interrogation they do with the modelers about 2 these things. 3 I'm completely humbled by the 4 precision with which they can forecast these 5 things. I mean, there are four different 6 models and all of them predict that four 7 hours from now this storm is going to turn 90 8 degrees to the right. How did they do that? 9 Well, it's because the physical sciences, 10 storms and so on, they don't care what other 11 agents anticipate the storm is going to do, 12 right. There is much less simultaneity. 13 Thunder clouds don't care about 14 what the forecaster's telling them is going 15 to happen whereas in our game, the social 16 sciences, you forecast something, you affect 17 the market, it goes on. It's a very 18 difficult game. So I think we have to 19 humbled about what out objective is and we 20 can't hold the accomplishments of NOAA and 21 the various weather forecasting units up as a 22 metric for evaluating the precision of our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 94 1 work because we're in a much harder 2 environment here. 3 So those are my two basic points, 4 that what you're doing is hard but you also 5 think about the mission and that the idea of 6 using the model as a means of identifying 7 what data are crucial to the enterprise is 8 really the most useful thing you can do with 9 this. And so that'd be my basic reaction 10 about what to put in the model. 11 I think that this analysis if you 12 can address it from the standpoint of what 13 additional data, had we had it, would have 14 improved the forecast and using it as a means 15 of doing a cost-benefit calculation for what 16 additional surveys you need to do or what 17 additional data items you need would be a 18 very useful way to approach the problem in 19 thinking about what the ultimate mission is. 20 I didn't have enough time to read 21 any of the background materials about the 22 model and the details. It looks like more of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 95 1 a statistical exercise. As a structural 2 modeler I'm a little bit nervous therefore 3 about the uses of the model for doing policy- 4 oriented work because I think that you have 5 to think about what instruments, what levers, 6 are in there and who knows what and how do 7 decisions get made so I have some questions 8 about that. I didn't completely understand 9 but hopefully in the next year or something 10 I'll learn more about it and be able to give 11 you more feedback. 12 So, just to conclude, I apologize 13 for not being able to tell you more about 14 specifically what variables to put in and so 15 on but I can just share with you my general 16 perspective on what the objective is. 17 DR. HENGARTNER: Thank you very 18 much. Moshe? 19 DR. FEDER: Actually I've been 20 preempted by what you said about Wilma 21 because when I was watching last night I was 22 wondering how come two models say it will go BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 96 1 up to Maine and another model said that it'll 2 go to the Yucatan Peninsula in Mexico and 3 actually I had Wilma written here so I'm 4 going to cross this out. 5 And then I wanted to talk about 6 regional forecasting errors and Margot said 7 we are doing it yet but I'm going to talk 8 about it anyway because it relates to one of 9 my suggestions. But I just want to focus on 10 one question, which is the forecasting errors 11 and how to measure them, what to do about 12 them, and I'm going to explain it just a 13 little bit. 14 And one technical comment, I don't 15 know if other statisticians in the audience 16 were confused by the term "bias" that you 17 heard there. It's an indicator of bias but 18 it's not what we call bias so I suggest 19 another name. It's the frequency you are 20 under- or overestimating, and when I looked 21 at the slides because I didn't have the 22 benefit of the actual presentation I said how BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 97 1 come it's minus 100 percent, I don't 2 understand and so I can cross out this too 3 and a lot of it. But I wanted to mention a 4 few things here. Actually what Mark said 5 about the way they do the estimation Mark was 6 talking about the way you construct a model 7 and I share what you said, Tom, about how you 8 construct your models. 9 Demographers actually do the same 10 thing when they provide us, statisticians, 11 with population projections. They use a lot 12 of discussions, brainstorming. They have 13 inputs from statisticians which we do have 14 standards ready which they put into the model 15 and then they use their judgment to come with 16 actually, according to John Row, very good 17 estimates so I'm sure your method works well 18 and what we can contribute is a little bit on 19 how to estimate the accuracy. And what I 20 think you should do, and I looked at some of 21 the plots, looks to me like there could be 22 some serial correlation which you could BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 98 1 exploit and the other part, which you said 2 you are not doing yet, original patterns. 3 So I would look at the prediction 4 errors both temporally and specially and see 5 is there anything that we are missing, and 6 that could contribute to finding out which 7 data you need to augment your model. Say, 8 why is it that always in the northeast we 9 underestimate and why do we overestimate in 10 California or why is it that we have a long 11 period of underestimation and than we have 12 overestimation. Could we use auto-regressive 13 models or something else? And you mentioned 14 seasonality, which was another thing that I 15 crossed out because I was going to talk to 16 you about it. 17 So in the interest of catching up 18 with time just one more comment, mean 19 absolute error is a good measure but I think 20 mean squared error is another tool. Actually 21 the F-statistic may be answering that, Mark. 22 Is it? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 99 1 MR. RODEKOHR: It is. 2 DR. HENGARTNER: Thank you very 3 much, Moshe. Yes, Jae. 4 DR. EDMONDS: I thought that was a 5 fascinating presentation. I think it's 6 actually interesting that everybody brings up 7 this hurricane analog and I think it is 8 actually an interesting analog. I think the 9 hurricane modeling is very similar to your 10 problem and that is it's a specialty, it's 11 really a short-term forecast. It's a much 12 more of a statistically based model. And 13 then after they do that they get in a room 14 and they say do we believe any of this 15 whereas the weather models are structural 16 models. 17 Weather models are based on physics 18 and hurricane models are based on statistics 19 and judgment. And in fact I remember sitting 20 at the beach in South Carolina and there was 21 a hurricane coming and we're looking at the 22 weather forecasts and the weather forecasts BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 100 1 were clear skies, low wind, and then the next 2 part of the weather forecast was we're 3 evacuating. And that's because the hurricane 4 is a subgrid phenomenon and so it does not 5 show up in the weather model. It is just not 6 there so the weather model is totally 7 incapable of dealing with it. 8 But I have actually a real question 9 which I think may be helpful. When you were 10 looking at the forecasts and comparing to the 11 observed ex-post I presume that you were 12 looking at the forecast as given and the ex- 13 post? 14 MS. ANDERSON: Yes. 15 DR. EDMONDS: My hypothesis is that 16 the WTI, which I now know what it means, and 17 the other variables are strongly correlated 18 to an exogenous input, which would be the 19 world oil price. Is the world oil price an 20 input? 21 MS. ANDERSON: No, the world oil 22 price is determined outside of the model. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 101 1 DR. EDMONDS: That's what I mean. 2 It's an input to the system. 3 MS. ANDERSON: It's an input to the 4 system, yes. 5 DR. EDMONDS: It's an input to the 6 system and so you are taking that as given. 7 My question is what would this have -- 8 MR. RODEKOHR: I don't want to 9 interrupt you but the world oil price is a 10 basket of about roughly 300 crude oil types. 11 What we're forecasting in our short-term 12 outlook is a representative price for crude 13 oil that Americans pay. The world oil price 14 is not an input into the WTI. What we are 15 forecasting is the WTI price. 16 MS. ANDERSON: Right, but a lot of 17 it is coming at the output from the model. 18 DR. EDMONDS: Let me just say the 19 question I was going to ask. If you went 20 back and put in all the real input variables 21 and ask the question how would my forecast 22 have changed if I had known exactly what the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 102 1 world market was doing and you'd gotten all 2 that right my hypothesis is that you would 3 look really good. 4 MS. ANDERSON: Absolutely. 5 DR. EDMONDS: Anyway is that 6 tendency to have under-forecast the WTI and 7 the other variables simply because you were 8 in a period at which the input values were 9 actually under and in fact if you went back 10 and you put in the real values whether you'd 11 look better. That's the question. 12 MS. ANDERSON: And I think, Mark, 13 if we'd gone back before '03 I bet the 14 forecast would have been more stable. 15 MR. RODEKOHR: I think there is a 16 little bit of confusion here, though, and I'd 17 like to try to clear it up. When we do the 18 world oil price or WTI or IRAP projections we 19 don't use a model. We use inputs from 20 different models and we use judgment. That's 21 what determines the forecast. So there is no 22 precise model that we use. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 103 1 And I know what you're getting at. 2 You're saying if you had a model and you went 3 and put it back, actual events, would you get 4 it closer? And the answer is generally yes 5 but because it's a lot of judgment there is 6 no model per se in this instance. 7 Let's think petroleum demand. 8 There is a model and if you went back and put 9 into that model actual weather, actual GDP 10 growth, as opposed to what we assumed will we 11 get it more accurate? Answer is yes. 12 MS. ANDERSON: But even the world 13 oil price that we're using for the Delphic 14 method and we're still under-predicting it, 15 other if the model is just our brains somehow 16 or other we're not taking something into 17 consideration and it goes back to this data 18 issue of what's missing from our Delphic 19 method that we're not taking into 20 consideration that might dampen the error on 21 that much less adding variables to all of 22 these other models. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 104 1 DR. EDMONDS: Right, so it has to 2 do with is your problem in the STEO. 3 MS. ANDERSON: Some of it is, some 4 of it isn't. 5 DR. EDMONDS: Or is it in fact that 6 you needed a better a better way of feeding 7 the STEO? 8 MS. ANDERSON: Well, back to your 9 question about correlation on these petroleum 10 product prices are often all driven by this 11 Delphic method of WTI. Even though there are 12 models for the motor gasoline price there are 13 more components within the model that help us 14 determine what that is but we know a large 15 component of that is driven off of whatever 16 we're deciding in the Delphic process so they 17 are correlated. We can explain more what's 18 going on in the motor gasoline situation than 19 we really can with the equations for a world 20 oil price. 21 MR. RODEKOHR: For example, if I 22 know when the hurricanes were coming, what BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 105 1 damage it was doing, if I had known who was 2 going to blow up a pipeline in Iraq or 3 somewhere else, or when the Argentineans went 4 out on strike or when the Nigerians went out 5 on strike, then yes, I could do a better job. 6 But I don't know that and so you're stuck. 7 DR. EDMONDS: Right, but the 8 question I had had to do with partitioning. 9 If you think about drawing the boundary 10 around your short-term forecasting tool is 11 the problem inside that box or is the problem 12 that you've really got a world out there 13 where you're taking inputs and you've got to 14 live with some fundamental uncertainties 15 around it. And so this apparent bias may in 16 fact just be an artifact of the fact that 17 during that period you were having this ramp- 18 up for a lot of reasons that were going on 19 and a part of it -- 20 MR. RODEKOHR: You can't just do 21 it. 22 DR. EDMONDS: Exactly, and so BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 106 1 actually inside the box is working great. 2 MS. ANDERSON: It's working much 3 better. That's for sure, yeah. 4 DR. EDMONDS: And so I think that 5 little diagnostic will help you to know 6 whether you've got a serial correlation 7 problem in the model or whether it's outside 8 the model. 9 MS. ANDERSON: And that's why you 10 break them up because you knew there is a 11 whole set of variables that we're probably 12 doing the best we possibly can. You'd want 13 to investigate that but all these 14 international price variables are not really 15 within the system so you don't want to 16 penalize yourself for something that may not 17 get any better than what you're doing right 18 now. And we focus mostly on the variables 19 that may be more under your modeler's 20 control. 21 MR. RODEKOHR: I agree but, as I 22 was trying to say, there are so many random BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 107 1 events in this world that it -- 2 MR. CARUSO: Could I? 3 DR. HENGARTNER: Yes. 4 MR. CARUSO: The random events, 5 obviously we can't predict them. But 6 probably a better example for what Jae's 7 point is in responding to Tom is two years 8 ago most of the world thought that in 2004 9 the world oil market would grow by 1-1 and a 10 half million barrels a day in '04 which would 11 have been a 1 and a half percent growth. 12 What happened was China grew on its own by 1 13 million barrels a day in 2004. No one 14 predicted that because the data was bad on 15 China. 16 If we had known China was going to 17 grow a million barrels a day in 2004 we 18 probably would have still under-forecasted 19 the price of WTI but we probably would have 20 gotten closer. So that's a perfect example 21 of why if we had better data in that 22 particular one variable, which is critical. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 108 1 It's, of course, external to the model. 2 DR. HENGARTNER: Thank you very 3 much. Mark. 4 MR. BERNSTEIN: I think you're 5 going to find that there is somewhat more 6 than that external data that's going to go on 7 for the reason for under-forecasting prices. 8 I actually do have a weather forecasting 9 story which I'm not going to tell anyone 10 because we're running behind but maybe later. 11 But, Margot, you mentioned that you think 12 you're doing as good or better than private 13 forecasters and one of the things that you 14 might ask yourself is are they also 15 underestimating prices on a regular basis. 16 And I know you guys get some of 17 those forecasts and it wouldn't be 18 necessarily too difficult to ask yourself 19 that quick question and this is my guess 20 because we're actually doing a report on 21 gasoline prices and why gasoline prices don't 22 seem to match the equilibrium estimates that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 109 1 economists try to make on them is that there 2 is fundamentally something missing in the 3 models. I'm not sure what they are yet. 4 One of the things is futures and 5 options and trader behavior and things like 6 that which affect the markets in very 7 unpredictable ways but may be getting more 8 predictable over time at least in the bias in 9 the direction. Our observations have been 10 that in most cases gasoline prices have been 11 under-forecasted across a variety of 12 different types of forecasts and so there is 13 something missing from that and I would 14 encourage you to look to see if everybody is 15 doing it but then fundamentally thinking 16 about what goes beyond the model that you 17 can't really model and how much of that is 18 driving prices. 19 What we've seen over the last 20 couple of years is impact of futures and 21 options and the trader behavior having more 22 and more of an impact on gasoline prices than BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 110 1 they had, say, three or four years ago and as 2 that becomes more prevalent it's going to be 3 harder to predict that type of behavior. 4 And so it's something just to keep 5 in mind as you're thinking about evaluating 6 these things. What we're actually doing is 7 we're building a production/supply curve for 8 California for gasoline and looking at 9 demands and then comparing the actual prices 10 to what an equilibrium would tell you should 11 have happened under those conditions. After 12 we get that done we'll show it to you. It 13 may give you some insights as to what's 14 beyond the model. 15 DR. HENGARTNER: Randy has been 16 very patient. 17 DR. SITTER: It's just a very short 18 comment, I hope. Tom has talked about your 19 focus should be more on what data you should 20 collect. And Jae commented on the fact that 21 there are certain things completely out of 22 your control and it's hopeless to try and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 111 1 model. Conditional on those you're not going 2 to have much chance. 3 But some of the comments from your 4 colleagues are defending that in a sense so I 5 think Thomas is correct. Your focus may not 6 be correct. We're doing the best we can to 7 predict and trying to somehow defend that and 8 really that's not necessarily the point. 9 I think that what you're doing is 10 using one summary statistic and what can be 11 done here and what Jae was getting is that if 12 you break down the sources of where your 13 errors are coming from you'll better 14 understand what data you're missing or where 15 your model structure may be wrong or needs 16 some more components. And how well you are 17 or aren't doing right now is more about what 18 you should be doing and not how you have 19 done. I just get a sense that that's not 20 really the focus. 21 MS. ANDERSON: No, I agree and I 22 hope that I said that I don't think that it's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 112 1 about hitting us hard. I think that's the 2 wrong way to market, looking at accuracy or 3 diagnostics, because I think it's about using 4 feedback to improve the model. I hadn't 5 really thought about it in terms of the data 6 that you're missing. I'm thinking of it as a 7 specification of the model. 8 We ultimately can't get away from 9 trying to make the model better at predicting 10 what it's doing but the idea, and this gets 11 back to doing scenarios, I don't think that 12 we ought to have our primary goal as saying 13 we hit a target and then to flagellate 14 ourselves when we're off the target. It's 15 really to have really good robust structural 16 models. 17 But how do you get at that unless 18 you're looking at a forecast accuracy 19 measure? It can be a diagnostic tool to open 20 up the model to tell you what you might do 21 better. And I don't know how else to do that 22 unless I look at these kinds of metrics. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 113 1 DR. SITTER: No, I think the 2 metrics are fine. I just think that the 3 process of the exercise of investigating why 4 they're giving you a particular pattern is to 5 break it down. Like you talked about, for 6 example, the WTI price percolates down into 7 the others so if you had the WTI price 8 correct would you have gotten the others 9 correct? Would they have been satisfactorily 10 correct? 11 If that's true then you can focus 12 on why your WTI forecast is not correct. And 13 I think that Jae's comment was that basically 14 he can even go one step further. I mean, if 15 you'd had the correct inputs from things you 16 don't really control or events like that 17 would you in fact have got it correct? If 18 that's true then stop beating yourself about 19 the model and start focusing on the things 20 that you are in control of. 21 And Mark said that that's probably 22 not going to be entirely the case but still BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 114 1 these kinds of exercises will give you a much 2 better idea of what proportion of your 3 problem is coming from what source and then 4 you use a Pareto principle if you want to get 5 better; that is, the things you can fix that 6 are causing the most problems you fix first. 7 That's all. 8 MS. ANDERSON: Helpful, thank you. 9 DR. HENGARTNER: Thank you. Yes. 10 MR. FAWZI: My name is Aloulou. 11 I'm from OIAF. Whether you're tracking the 12 natural gas production report, the price, and 13 the deviation between different indexes by 14 region and why they deviate compared to NYMEX 15 or Henry Hub so -- 16 MS. ANDERSON: I can't answer that 17 but I think there is somebody in our 18 organization that's been working on natural 19 gas prices and how they relate to NYMEX. 20 There are some folks that are actually 21 looking at that issue. Bill —————— can 22 probably help you much more on that question BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 115 1 than I think we could but people are trying 2 to answer this question, is NYMEX a good 3 predictor of natural gas prices. 4 MS. KIRKENDALL: Don't we have a 5 session coming up tomorrow? I think we have 6 a session on that tomorrow, a breakout 7 session. 8 MS. ANDERSON: It may be less of a 9 STEO question than it is a broader question 10 about natural gas prices. 11 DR. HENGARTNER: Thank you very 12 much. I like to go to the break, one of 13 those important parts of the meeting. I want 14 to remind you that after the break we are 15 going to go to breakout sessions. Mark, 16 Johnny, Mark, Moshe, Tom, Susan, and Randy 17 are going to go to Room 8E and those who I 18 haven't named are going to come downstairs to 19 Room 5E, that's the fifth floor, 069, and 20 I'll try to gather you before we leave. So 21 let's take a 15-minute break. Thank you. 22 (Recess) BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 116 1 MR. WEINIG: Ladies and gentlemen, 2 I'd like to introduce Mark Schipper with the 3 Energy Information Administration, Energy 4 Markets and End Use, and Mark will be talking 5 about vehicle energy use what he did and what 6 it tells. Mark? 7 MR. SCHIPPER: Good morning. 8 MS. KIRKENDALL: Mark, before we 9 start this is supposed to be a discussion 10 session so I would like to get some of you 11 folks from the audience to come here and sit 12 around the table. One would be Dwight French 13 and anybody else from your staff that you 14 think ought to come down. 15 MR. SCHIPPER: How about some DoT 16 folks? 17 MS. KIRKENDALL: DoT folks would be 18 good too. 19 MR. FRENCH: Dwight French, Energy 20 Information Administration. 21 MS. LISS: Susan Liss, Federal 22 Highway Administration. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 117 1 MR. GIFFORD: Will Gifford, EIA. 2 MR. BROENE: Tom Broene, EIA. 3 MR. ADLER: Bob Adler, EIA. 4 MR. FREEDMAN: Stan Freedman, EIA. 5 MR. SCHIPPER: Is that everyone? 6 Good morning. Thank you for the opportunity 7 to talk to you today about a new product that 8 EIA has produced which is out of the 9 mainstream of what the consumption surveys 10 have produced in the past because we 11 hopefully did some work on vehicle energy 12 use; however, it was a study that did not 13 originate here at EIA. For this reason I 14 have some acknowledgment for the Federal 15 Highway Administration, which conducted the 16 initial survey that EIA then used to augment 17 with energy-related data. 18 As part of the Federal Highway 19 Administration I think they coupled with the 20 Bureau of Transportation Statistics and the 21 National Highway Traffic Safety 22 Administration to conduct the National BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 118 1 Household Travel Survey and DoE's Oak Ridge 2 National Laboratory was very helpful in 3 estimating vehicle miles traveled which EIA 4 subsequently used and modified as needed so I 5 just want to acknowledge their help in all 6 this. 7 The reason we undertook this work, 8 I gave a quick slide in terms of what 9 transportation energy use has done in this 10 country since 1949 to the latest number was 11 2003; however, I think the 2004 Annual Energy 12 Review is out. Transportation has increased 13 its share of the energy in this country with 14 a couple of pauses in it looks like in the 15 '74, and the '79 to '82 period and '90-'91. 16 And the latest was 2001, which was the year 17 the National Household Travel Survey was 18 conducted, but I just want to give an 19 overview as to one reason why we're looking 20 at transportation especially as it relates to 21 oil because petroleum in terms of motor 22 gasoline is about two-thirds of the petroleum BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 119 1 use of the transportation sector and what it 2 boils down to is light duty vehicles and how 3 they're used and the energy consumed by them 4 is important for the nation. 5 And basically why we're interested, 6 my office as part of EMEU conducts some 7 demand side energy surveys. Basically we 8 work on the demand side of the energy 9 equation. We survey residential buildings, 10 manufacturing plants, and commercial 11 buildings and up until 1994 we conducted a 12 residential transportation study. That 13 started in '83 and looked like it was 14 conducted every three years until 1994 when 15 it was discontinued. After that there was a 16 dark period where there were no 17 transportation statistics in terms of light 18 duty vehicles at EIA. 19 However, my office decided to 20 investigate using the 2001 National Household 21 Travel Survey as a replacement for that data 22 series and through the next release is the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 120 1 National Personal Travel Survey which DoT 2 plans to implement in 2008. I put a question 3 mark by that because funding is always an 4 unknown when it comes to these large surveys. 5 We are working with the folks at 6 Federal Highway. I think that's the reason 7 for the name change is that it's now a 8 Federal Highway Administration product only 9 and so it's been renamed the National 10 Personal Travel Survey. 11 What we did, basically we augmented 12 the National Household Travel Survey with 13 energy-related data. What that means is 14 we've put in fuel economy estimates for the 15 vehicles, we imputed a fuel type, we also 16 imputed the energy consumption based on fuel 17 economy and vehicle miles traveled and then 18 through data series available here at EIA we 19 embedded energy prices to calculate energy 20 costs for the year. 21 A little example here, our energy 22 consumption estimate was essentially an BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 121 1 on-road, in-use fuel economy measure as the 2 inverse of that multiplied by VMT at an 3 annual level. There was a little bit more 4 involved and that's in the paper in how that 5 number is calculated but in a general 6 nutshell that's how it was calculated. 7 Our data sources, as I mentioned, 8 the fuel economy estimates came from the 9 National Transportation Safety 10 Administration, their corporate average fuel 11 economy program, and because they were a 12 participant in the NHTSA survey they were 13 more than willing to provide us with that 14 data series unlike the Environmental 15 Protection Agency, which had the similar 16 data, however because of restrictions we're 17 unable to release that to EIA for our use. 18 What that meant is that EIA had to 19 do additional work because there are 20 different calculations used by the corporate 21 average fuel economy program to give credit 22 to flex fuel vehicles, to alternative fuel BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 122 1 vehicles, which goes through more of a 2 political process in obtaining credits and 3 how those credits are added over time. So 4 EIA had to go through a process of removing 5 all that information and unfortunately it 6 took a very long time to do; however, we were 7 able to do so with the help of the corporate 8 average fuel economy folks. 9 And EIA's data experts came into 10 play because the corporate average fuel 11 economy program didn't come into existence 12 until 1978 so anything that was pre-1978 13 model years had to rely on EIA's previous 14 residential transportation studies and some 15 data experts who just had a good feel for 16 fuel economy measures and that would be 17 Dwight French, who sits over here on my left. 18 The fuel price data by month by 19 state came from EIA supplier surveys. I've 20 listed the two here. I believe they're still 21 going in terms of their coverage. I think 22 they're even expanding their coverage to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 123 1 select cities within states and so whether we 2 can subsequent to that 2008 study use that 3 information we can evaluate whether that is a 4 possibility. And the federal, state, and 5 local excise taxes came from FHWA. 6 I provide here a little schematic 7 for how it was all put together. Basically 8 we have the three databases from the fuel 9 economy estimates from the EPA, NHTSA, the 10 safety administration, and the NHTS which 11 provides the vehicle miles traveled which was 12 the reason I threw EIA on here is because we 13 modified those VMT estimates to represent a 14 possession of the year. In other words DoT 15 provides an annual estimate for VMT; however, 16 EIA was only interested in how long that 17 vehicle was possessed by the household and 18 the VMT associated with that possession. 19 The EIA provided the retail price 20 information and essentially, to put these 21 boxes together, the work that EIA did is 22 embedded in this red box where we essentially BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 124 1 select a vehicle from the corporate average 2 fuel economy database, adjust its fuel 3 economy to on-road, in-use values, and then 4 merge that with VMT estimates for the actual 5 possession to calculate energy consumption, 6 which is then subsequently used to calculate 7 vehicle fuel expenditures from the retail 8 fuel price database. 9 Essentially that whole process 10 comes under linking process of one to many. 11 Take an example of our Volkswagen Jetta 2001 12 automobile. That's all the information that 13 we're provided from the NHTS sample 14 respondent so when we link that to the CAFE 15 program, which has a host of information, a 16 rich data source in terms of its number of 17 cylinders, engine size, fuel type, we come up 18 with I think there are around eight different 19 configurations of a Volkswagen Jetta 2001 20 model year ranging from a fuel economy 21 standard of 52.6 miles per gallon, which is a 22 turbo diesel, not many of them sold in this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 125 1 country, to a Volkswagen Jetta around 25.1 2 miles per gallon. 3 So it's linking one to many and our 4 job in that relationship is to select one. 5 This one to many relationship, I want to 6 quantify it so that you could come up with a 7 reasonable approximation of how good this 8 linking process was for EIA. This is a chart 9 of the number of eligible donor vehicles and 10 a number of sample vehicles that were 11 matched. 12 Unfortunately you all don't have 13 the slides but the table provides the number 14 of eligible donor vehicles. In other words 15 that Volkswagen Jetta would be a vehicle on a 16 one to one match. We had around 5200 17 vehicles where there was a one to one match. 18 Where there were two to five 19 vehicles we had almost 20,000 vehicles that 20 were matched with only donor vehicles of two 21 to five. And the quality of it goes from top 22 to bottom where it was presumed that the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 126 1 imputation quality is very high at the top 2 and very low at the bottom of this table, and 3 we have a little over 47,600 cases of sample 4 vehicles from the NHTS. 5 How we adjusted fuel economy, we 6 start with EPA's unadjusted fuel economy 7 values, which come directly from the CAFE 8 program. We then adjust that for on-road 9 shortfall. In other words the vehicle as it 10 hits the road, what does it get on-road. 11 Now, that's an estimate or adjustment that 12 has been in agreement since 1985. In fact 13 EPA uses those adjustments when you view the 14 fuel economic sticker on a new vehicle. By 15 law there is a sticker placed on every new 16 vehicle that identifies the highway and city 17 and combined fuel economy estimate so that 18 users or purchasers have clear information 19 when they make their purchase choice and fuel 20 economy is one of their justifications for 21 the purchase. 22 The second adjustment is something BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 127 1 unique to the transportation surveys that EIA 2 used to do where we adjust for in-use. In 3 other words adjust for the location of the 4 vehicle, the driving patterns, the usage of 5 that vehicle, which is something that the EIA 6 has applied from a paper produced by Crawford 7 in 1983. 8 The reason I put the site down here 9 is that that model hasn't been adjusted or 10 recalibrated since 1983, I think, using fuel 11 economy information obtained through travel 12 and fuel purchase diaries so they actually 13 went out and talked to the users of these 14 vehicles and tracked their travel as well as 15 their consumption of energy. 16 The details are very rich because 17 this is a new product that EIA has produced 18 and to maintain a high level of transparency 19 of what we did and all the downfalls, all the 20 pitfalls, and all the imputations that we did 21 because we had essentially added additional 22 information to a survey based on reported BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 128 1 information. So it's 100 percent imputed 2 information. 3 So we made all this information 4 available on this website and we verified the 5 procedure for adjustment in 1991 and 1994 6 RTECS; however, because we don't have fuel 7 purchase diaries or travel diaries linked 8 there is no way that we could verify the 9 procedures that we used for 2001 NHTS study. 10 The adjustments, I basically just 11 threw the equations out here. The first two 12 equations deal with the shortfall adjustment 13 produced by Hellman that people are in 14 agreement since '85 on. The last two 15 equations deal with an additive adjustment 16 where we have the gallons per mile ratio. 17 That's a GPMR equation which we add delta to 18 which is Crawford in his '83 publication 19 developed to give a total fuel economy 20 in-use, on-road estimate of the GPMR 21 statistic and essentially here it is in terms 22 of the clarity of it all. It's well BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 129 1 documented in the appendices. 2 So I wanted to give an example 3 going back to that Volkswagen Jetta where the 4 composite MPG is 25.1 miles per gallon. 5 Let's just presume that we selected that 6 vehicle. By month I circled the annual to 7 remind me that I should tell you that we make 8 our adjustment by month; however, we publish 9 annual estimates, not monthly estimates; 10 however, we do build out from the months. 11 Using a straight fuel economy 12 estimate directly from the CAFE program, we 13 would have given the vehicle a 25.1 mile per 14 gallon estimate. Using the shortfall, which 15 would drop it by the Hellman-designated 16 shortfall adjustment, we would have dropped 17 that estimate to 21.3 miles per gallon; 18 however, based off of that delta regression 19 estimate however it was approximated, I don't 20 know if it was regression, we can track what 21 the fuel economy annualized estimate would 22 have been had the vehicle traveled the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 130 1 typical amount of 15,000 miles. You can see 2 the fuel economy varies by month and that in 3 my opinion is dealing more with the use of 4 the vehicle whether it was used for highway 5 or city driving where as you drive more or 6 VMT increases you're driving more on a 7 highway level than a city. 8 So you see that the summer months 9 you peak and you go more towards the estimate 10 provided directly as a composite estimate. 11 Now, if we went to a household with only 12 4,000 miles of travel that is the 13 distribution that we would see by month and 14 the annual estimate would be around 15 miles 15 per gallon so you can see there is a wide 16 range. We really adjust the fuel economy 17 estimate based on the annualized VMT estimate 18 and if we go between 4,000 miles all the way 19 up to 15,000 miles you can see the trend 20 basically converges to that 21.6 miles per 21 gallon estimate, which would be a 15,000 22 miles annual VMT. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 131 1 Knowing that we're using 100 2 percent imputed estimates, EIA conducted some 3 sensitivity analysis of our national energy 4 consumption estimates from this product and 5 also provides a lot of guidance to users of 6 the day. There's a very large report making 7 its way through a Category 1 review. I 8 believe the committee obtained a copy of that 9 but not for citation or distribution. Let me 10 know. Did you all receive a report? Okay. 11 In that sensitivity analysis 12 instead of selecting one vehicle we made room 13 for three selections of fuel economy estimate 14 for a given vehicle. One would be the sales- 15 weighted value of fuel economy, in other 16 words of those eight Volkswagen Jettas we 17 would pick the sales-weighted fuel economy 18 estimate for that Volkswagen Jetta in the 19 sample database. At the national level that 20 type of a procedure or protocol lowered the 21 consumption estimate by 2 percent. 22 When we selected the maximum fuel BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 132 1 efficiency designated by the 95th percentile 2 of those sales we lowered the consumption by 3 9 percent, which you would expect. We're 4 picking the most fuel-efficient vehicles 5 automatically for each of the sampled 6 vehicles and to cover our lower balance we 7 picked the minimum fuel economy estimate, 8 which was designated the fifth percentile by 9 sales, which raised our consumption by 7 10 percent. 11 So with those numbers along with 12 our sampling or nonsampling errors we in our 13 report and in our documentation provide notes 14 to the readers through an author's note that 15 the use and the usefulness of these data 16 should be evaluated against the researcher's 17 project requirements. If you are requiring a 18 very exact estimate for your research 19 requirements there is no need to use these 20 numbers; however, if this type of a range of 21 uncertainty is okay for your research then 22 please feel free to use it or misuse it BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 133 1 because the tendency is to misuse the numbers 2 rather than to use them correctly. 3 In addition to that, and I believe 4 this is the first time that EIA put out 5 tables in this way, we added some 6 interactivity within the tables in an Excel 7 spreadsheet where the user can specify based 8 on the sample count cut-offs and the measures 9 of precision. In other words each estimate 10 is supported by an underlying sample. If a 11 researcher requires 20 vehicles or 20 12 households to support their estimate or their 13 quality of the estimate that they're looking 14 for we allow them to change those parameters 15 or the measure of the precision in terms of 16 the CV or relative standard error, allow them 17 to change the viewing of the tables. In 18 other words they show up as red italicized 19 numbers rather than the typical black bolded 20 numbers. 21 We haven't had any feedback as to 22 whether this is helpful but we put it out BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 134 1 there as one more level of guidance for our 2 users. And we are working with NPTS for 3 their 2008 and this is probably going to take 4 place in four areas of interest. One is the 5 verification of our EIA adjustments, in other 6 words verification of that Crawford 1983 7 model. 8 What we would need to verify that 9 is fuel purchase and travel diaries and I put 10 a little qualifier on here that this is very 11 unlikely to happen. It's very expensive to 12 do and from current discussions unless there 13 is a lot of funding for it it is very 14 unlikely to take place. Now, to minimizing 15 the imputation errors there are some aspects 16 that are likely to occur and those would be 17 collecting additional vehicle 18 characteristics. 19 As I mentioned before, we only have 20 the make model, model year, and the type of 21 vehicle. If we had additional information 22 related to, for example, the type of fuel or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 135 1 the price paid for that fuel or the number of 2 the cylinders of the fuel or the gold 3 standard would be the vehicle identification 4 number, which has embedded in I think a 10 to 5 16-digit code on each vehicle, a way to 6 decode all the engine characteristics of that 7 vehicle that again would be the gold standard 8 but again putting a probability on collecting 9 vehicle identification numbers I put fairly 10 unlikely. 11 Capturing the vehicle disposition 12 data, as I mentioned before, EIA was 13 interested in the actual time of possession, 14 not an annualized VMT estimate, and it's very 15 likely that DOT will go ahead and capture 16 that information in its survey instrument 17 which would be very helpful. Rather than EIA 18 discerning that information from odometer 19 readings and auxiliary information provided 20 by the respondent we would have that as a 21 reported data item. 22 Also the last item is the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 136 1 Statistical Efficiency Act. I don't know if 2 DOT is going to use that for the NPTS but 3 there was a signed into law by Bush II about 4 a year ago? 5 MR. FRENCH: December '02, yes. 6 MR. SCHIPPER: December '02 which 7 allows statistical agencies to share data for 8 statistical purposes. We went through a 9 legal process of sharing data last time. The 10 ability to easily share information makes 11 things a lot easier especially when we're 12 working alongside of DOT but trying not to 13 hamper their survey processing because they 14 have their own daily users to satisfy and 15 their own scheduling to work with. 16 The question that we had for the 17 committee is this is a new product by EIA. 18 It's 100 percent imputed product. It comes 19 from a solid source, the National Household 20 Travel Survey, but the question that we had 21 is do the extensive assumptions that we make 22 to create this data cause any concerns when BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 137 1 we label this as an official EIA data 2 release. Even though we put all caveats in 3 the documentation, in the report, on the 4 spreadsheets that are sent out as a data 5 file, is that enough or should we have 6 additional concerns or should we continue to 7 expend the resources to do this again in 8 2008? 9 And if we do move ahead are there 10 any ground rules that we should establish in 11 methods, data requirements, quality of 12 information with DoT before we undertake such 13 work because the more information that we can 14 provide DoT at the front end of a project and 15 and how clear we are the better the product 16 will be at the end in where to focus their 17 time with their contractor vetting the 18 information because, again, EIA is working 19 alongside DoT. We're not the primary focus 20 of their study. 21 Also just general improvements, 22 data, methods, and so on, and thank you for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 138 1 your time and attention and you don't have 2 the slides but there was some mention how EPA 3 calculates their fuel economy which is then 4 handed off to the safety administration, 5 NHTSA. 6 I threw in the testing protocols 7 that EPA goes through for the city and the 8 highway. You can tell which is city and 9 which highway by the miles per hour. And 10 that the testing protocols are done in some 11 general summaries and that they are done with 12 the vehicles air-conditioning or other 13 accessories turned off. So I just threw that 14 in to give you an idea of how EPA comes up 15 with their numbers to begin with and I'll 16 look forward to answering your questions and 17 getting your input. 18 DR. FEDER: What if I delegate you 19 to lead the discussion? 20 MR. BERNSTEIN: Well, but why don't 21 you start off if you have some comments? 22 DR. FEDER: No, actually before you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 139 1 raised some questions that we would like 2 probably to answer after people have made 3 their comments. So why don't we start? 4 Mark, go ahead. 5 DR. BURTON: Well, since this is a 6 relatively new data product who's using it 7 and how and what motivated the development of 8 this initially? 9 MR. SCHIPPER: I'd say the first 10 motivation was that EIA had a data gap in its 11 transportation statistics. Since 1994 we did 12 not have any information related to light 13 duty vehicles and household use of energy. 14 EIA through its national energy modeling 15 system at one time had demographic 16 information embedded in it in terms of light 17 duty vehicle use. I believe in the last go 18 around that information or that module of the 19 NEMS model was ripped out because it didn't 20 have that information. Now the question is 21 is this information good enough to go back 22 and look at re-evaluating whether that type BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 140 1 of a module could be replaced or improved in 2 the modeling system. Those would be the two 3 items. Dwight, do you have any additional 4 items as to who was looking for this 5 information? 6 MR. FRENCH: Well, everybody looks 7 for it. We have the same set of customers as 8 we would have for any of our consumption 9 information from Congress and congressional 10 staff, the education, the energy research 11 community, and so forth. 12 DR. BURTON: I just wondered if 13 there were specific research questions that 14 had motivated the development of the data. 15 So you're saying that from an internal 16 standpoint potentially you send the revised 17 NEMS and then anybody else that wants it -- 18 MS. KIRKENDALL: I think one reason 19 they did the analysis was that they had lost 20 the residential transportation energy 21 consumption survey in '94 so there were data 22 available that had been used within EIA. It BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 141 1 was lost because of the resources. 2 MR. RUTHERFORD: I will tell you 3 I've gotten comments from the academic and 4 energy research community between 1994 and 5 now that say how can you call yourself an 6 energy statistics community when you don't 7 have information about energy use for two- 8 thirds of the oil consumption. I mean, in 9 essence we don't know what the fuel economy 10 of the US household vehicle fleet is. This 11 is our best approach to try to get at it but 12 to ask somebody what the fuel economy of the 13 vehicle fleet is nobody can give you an 14 absolute certain answer to that. 15 MS. LISS: If I may add, the 16 household vehicle fleet has expanded 17 considerably since 1994. I mean, what we 18 looked at in 2001-2002, it's a huge growth in 19 the number of vehicles. 20 DR. BURTON: And, I would assume, 21 significant change in the composition in 22 terms of fuel economy. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 142 1 MS. McGUCKIN: SUVs were introduced 2 after 1994 as a major market segment. 3 MR. BERNSTEIN: I agree that 4 absolutely this needed to be there. You 5 asked a question if she could do it in 2008. 6 I mean, for a variety of reasons yes, but the 7 most critical reason internally for EIA which 8 gets back to Mark's comment is can this be 9 used by NEMS and STEO to do a better 10 forecast. If it can that should guide what 11 you choose to do because that ought to be the 12 primary use. Internally you try to get a 13 hand on vehicles and vehicle use. 14 The other question is so that's one 15 in terms of whether you should do it in 2008. 16 You really have to answer that question 17 because if they can't use it you might think 18 about whether this is the best approach to 19 take. 20 DR. FEDER: Since there were three 21 questions raised I would like to maybe go one 22 question at a time and actually that's just BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 143 1 as good question as any so any more comments 2 on should this be redone in '08 and what 3 should the emphasis be? 4 MR. SCHIPPER: Within the EIA you 5 have the informal relationships with the NEMS 6 modelers and I'm aware that the NEMS in terms 7 of the transportation side will look at this 8 information of VMT estimates of energy use as 9 a benchmark in their own mind in looking at 10 the information whether their modeling makes 11 sense or agrees or disagrees and how it 12 disagrees. The use of this information in 13 NEMS on a formal or an informal basis because 14 you are going to have modelers interested in 15 going hey, I'll take any information out 16 there. Knowing the caveats, knowing how it 17 was put together, this is beneficial to me as 18 a modeler to provide additional information 19 related to my model; however, formally am I 20 taking this number directly and embedding it 21 in a process that is codified in the NEMS 22 systems. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 144 1 Which level are we talking about, 2 the formal or the informal, or is the 3 informal less of a justification for doing 4 this work than the formal process? I would 5 assume so but in terms of resources this was 6 a lot of time by energy analysts, not survey 7 logistics, so it was a whole different skill 8 set that went into doing this work in knowing 9 the transportation sector and household and 10 vehicle use rather than knowing survey 11 logistics of how to get information from a 12 household so there are differences. 13 DR. FEDER: Can we go back? I'm 14 sorry. 15 MR. RUTHERFORD: I just had a 16 question of information. I think about the 17 use of this type of information or this type 18 of survey. A policy question is if we're 19 faced with the need to try to abate energy 20 use at the household level how effective is 21 the CAFE program versus other types of 22 instruments for achieving the same objective. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 145 1 I was just wondering in the survey 2 does it give you very much insight into this 3 question about rebound effect, because, I 4 mean, CAFE is a very blunt instrument because 5 there's this relationship between miles per 6 gallon and number of miles driven that's 7 quite central to the analysis and I just 8 wonder. It's that elasticity I'm wondering 9 about rather than actually what the shares 10 are in the current fleet. I just wonder if 11 that's something that -- 12 MR. SCHIPPER: Probably the fuel 13 purchase diaries, that type of information. 14 EIA conducts weekly gasoline surveys of how 15 the price changes with how much energy has 16 been supplied so you get that elasticity 17 tracked over time to get that effect of the 18 price. But are you talking about changing 19 fuel economy in terms of I'll get a vehicle 20 that is 10 percent more fuel efficient, I'm 21 going to then drive 20 percent more, that 22 type of elasticity? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 146 1 MR. RUTHERFORD: Yes, I mean, 2 basically there is this effect that if you 3 try to target fuel economy then that doesn't 4 necessarily control how much fuel use you 5 manage. 6 MR. SCHIPPER: Right, and I think 7 CAFE does use the rebound effect in their 8 calculation or setting of the CAFE targets 9 and I just don't have enough information as 10 to whether this study could provide 11 elasticities in the way that you are looking 12 for. By the time you get the elasticities 13 it's too far back in the past and things have 14 changed. You want to have the elasticities 15 at least on a quarterly basis rather than 16 three years, four years, back. 17 MR. FRENCH: I also wonder if at 18 this point if cost elasticities don't swamp 19 any MPG elasticities because the price is 20 changing so rapidly. Certainly that's true 21 on a short term. I think we are on the cusp 22 of being on some elasticity changes in the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 147 1 short to mid-term and probably more likely in 2 the longer term. Before we didn't do 3 anything on CAFE over 20 years and VMT just 4 keeps going up inexorably so it isn't as if 5 by not doing anything we kept VMT under 6 control. That's not true. Maybe VMT would 7 have gone up faster or something if we had 8 instituted it but those are hypothetical-s. 9 MS. LISS: One thing we also look 10 at is time elasticities, travel time, travel 11 time budgets, because from our viewpoint 12 we're seeing some saturation in travel. 13 Because of the congestion and the time it 14 takes people are only willing to spend X 15 amount of time behind the wheel each day. 16 MS. McGUCKIN: I was just going to 17 say that you should possibly look up the 18 Victoria Transport Policy Institute. They 19 have the travel elasticities report on their 20 website right now. It's very interesting how 21 inelastic miles is. People just travel. 22 They don't care how much it costs, they don't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 148 1 care what their MPG is, they just do what 2 they have to do every day. 3 MR. BERNSTEIN: I must say that 4 over the time period that those estimates 5 were made you basically had declining real 6 prices of gasoline with increasing miles per 7 gallon for the most part. Anecdotal evidence 8 in the last six months might suggest 9 otherwise as public transit systems have seen 10 increasing ridership and we have had when we 11 have been trying to do some of these things 12 with other bits of data and the problem is 13 the time series of data sees very little 14 variation in price over a long enough period 15 of time to get anything useful out of it and 16 it's mostly declining real prices. And so 17 it's hard to say what that really tells you 18 about consumer behavior over the long run if 19 you actually begin to see. Now, these last 20 couple of years of volatile prices, hopefully 21 there'll be enough information and data 22 coming out for us to be able to get some BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 149 1 sense as to how consumers actually do react. 2 DR. BURTON: Is that the silver 3 lining that we'll have enough variation in 4 the data to make some good estimates? 5 MR. BERNSTEIN: I hope so. 6 DR. BURTON: I was going to add to 7 what Mark said in that you're right. Miles 8 tend to be very insensitive; however, the 9 fuel prices will affect modal choice and all 10 you've to do is look at transit numbers of 11 the last six months to see that that's very, 12 very true. 13 DR. FEDER: Can we just answer the 14 first question that Mark had concerning the 15 assumptions that went into the study that are 16 extensive, as you pointed out, and I was 17 wondering. I don't know most of the terms 18 VMT and what NEMS exactly is and so on. My 19 concern was actually about that. Can you 20 just explain first why you asked this 21 question? Maybe then they can answer it. 22 MR. SCHIPPER: Since it's a new BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 150 1 product that EIA has produced, in other words 2 we did not go out and draw a sample and 3 collect this information directly from the 4 respondents. We imputed it based on survey 5 responses. 6 And in terms of should we be doing 7 this because EIA when it puts out information 8 it's taken literally. In other words it's 9 used and misused because it's the information 10 that EIA has produced. Even though we have 11 placed caveats on the quality of the 12 information we are still held to a different 13 standard of government statistics because it 14 was obtained from EIA's website or the report 15 was obtained from our National Energy 16 Information Center and has the EIA label on 17 it. So there was concern to say whether this 18 type of a product is something that EIA 19 should release. 20 DR. BURTON: Can I address that? I 21 do more transportation work than I do energy 22 work and I probably use BTS products more BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 151 1 than I use EIA products so I'm familiar with 2 data products in both worlds and the 3 limitations on the reliability of the BTS 4 products are just as real and they're just as 5 good about warning users about those 6 limitations and people still misuse them. 7 The trade analysis framework is a wonderful 8 example of how people have taken something 9 and used it in ways it was never supposed to 10 be used. 11 But at the end of the day what you 12 have done, I think, is at least as good as 13 any of the other products that I see on the 14 transportation side. I have no problem with 15 it at all nor would I have any problem using 16 it and defending it and I'd probably misuse 17 it too. When you need an answer you do just 18 about anything. 19 DR. SITTER: I disagree with your 20 characterization of what you did. I mean, to 21 me you have data and you basically modeled 22 with it. You can view it that way. Your BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 152 1 data was categorical, maybe, and you used 2 auxiliary information and so forth but 3 ultimately you had data and you built models 4 around that data. It wasn't data you 5 personally collected, other agencies 6 collected it, but I wouldn't really view it 7 as imputation. I don't think that's exactly 8 what you did. 9 Imputation suggests that you 10 release data with artificial data points and 11 that's not what really you did. I mean, I 12 really wouldn't characterize it that way at 13 all. 14 MR. BERNSTEIN: I'm not sure what 15 you called it but if you call it an estimate 16 of vehicle energy use like you call forecast 17 estimates then you get away from this issue 18 of is it actually data because I think 19 Randy's right. It's not primary. You're 20 using primary data but -- 21 DR. FEDER: Do the interactive 22 tables you mentioned offer any way of getting BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 153 1 some handle on the uncertainties? 2 MR. SCHIPPER: The only 3 uncertainties would be related to the measure 4 of precision associated with the estimate and 5 that the sample supporting that estimate in 6 terms of the actual responding household or 7 the number of vehicles. We didn't carry 8 along two or three different measures for 9 each statistic. We calculate it using the 95 10 percent sales weighted value. If we carried 11 all those along, basically did the study 12 three different ways, that would be a measure 13 of uncertainty associated with each estimate. 14 It's a possibility to do something like that; 15 however, I don't know if we would confuse the 16 readers to say well, which one should I use. 17 Why not the 60th percentile? Or 18 once we start giving flavors of the data 19 you're left with different paths to take. If 20 there is an agreement out there in 21 statistical community, say, you should be 22 looking at these types of values, 75 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 154 1 percentile and the 50 percentile or somewhere 2 along those lines, and that's something we 3 could evaluate. Otherwise the user who'd say 4 well, which one should I use. You gave me 5 three estimates. I like this one. I'm going 6 to use this one because it fits my research. 7 MR. BERNSTEIN: Well, but that's 8 okay too. 9 MR. SCHIPPER: And I really like 10 the idea that it's a model-based estimate 11 instead of an imputed estimate. 12 DR. BURTON: I think if we're going 13 to look for a way to describe that I would 14 say that it's a bit more deterministic, it's 15 not a purely stochastic model, and I think 16 there are probably some engineering aspects 17 that go into determining some of the 18 parameters so it may be a bit more 19 deterministic but it's still a model. I 20 agree with Randy. 21 MR. BERNSTEIN: In terms of the 22 assumptions I would say a couple of things. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 155 1 One is I believe EPA is changing how they are 2 doing ———————— 3 MR. SCHIPPER: They are looking 4 into by the end of the year trying to better 5 approximate the fuel economy that a person 6 would actually obtain driving a vehicle on 7 American roadways. 8 MR. BERNSTEIN: So what would be 9 interesting, of course, is well, in 2008 you 10 should be expecting to do something else but 11 also have a close idea as to your previous 12 ones and also be thinking about how to update 13 that 1983 estimate somehow, I mean, without 14 doing the detailed stuff that they did and 15 see what can be done to generate some of 16 those. 17 MR. FRENCH: Just a short comment 18 about that, when we first used the RTECS we 19 did a diary data collection and we did it 20 through the 1985 cycle of RTECS and so when 21 the '83 study came out and we decided after 22 '85 for budgetary reasons to go from a diary BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 156 1 collection to one where we were just 2 collecting VMT and modeling the miles per 3 gallon we looked at the '85 data versus the 4 '83 modeling procedure and, lo and behold, 5 for major population subgroups it was within 6 a couple of percent uniformly and so we went 7 ahead and we used it '88, '91, and '94. 8 At the time in '94 I said this 9 thing has been around for a decade now. 10 Should we test it? So in '95 we did a 11 research study. We did a small research 12 study, it cost us a couple of $100,000, but 13 we went out and did diaries on 500 vehicles 14 nationwide, subsample of the '91 RTECS, I 15 guess it was, and once again we found the 16 model for large population subgroups to be 17 within a couple of percent. I was expecting 18 that model approach to have deteriorated for 19 determining on-road but it hadn't, amazing to 20 me. 21 And so I was planning on continuing 22 to use that; however, the budget realities of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 157 1 the mid-'90s intervened and by '96-'97 time 2 frame we had decided to change the 3 consumption surveys and one of the things we 4 did was drop RTECS. 5 So we haven't really looked at that 6 since then but you're right. To the extent 7 that EPA changes their approach toward 8 determining what they're actually going to be 9 putting on the stickers and so forth that 10 whether in itself is good or bad as an 11 adjustment may not relate in the same way to 12 what EPA is going to produce in its new 13 adjusted miles per gallon estimates and so we 14 might have to think of changing that 15 supplemental adjustment or doing something 16 different for a couple of different reasons. 17 MR. SCHIPPER: We may have to have 18 two paths because they're not going to go 19 back and retest. They're only going to test 20 forward. So you're still going to have the 21 vehicle stock out there that's pre-2005, 22 2006, whenever EPA would institute a new BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 158 1 dynamometer test. 2 MS. LISS: And the average vehicle 3 age of the household vehicle fleet is 8.9 4 years so you're talking about a large number 5 of vehicles. 6 MR. SCHIPPER: Especially light 7 trucks, which stay around forever. 8 DR. SITTER: I was interested in 9 your answer to the question concerning 10 components of variance. From what you're 11 saying you do have estimates of error for 12 things like weighted sales estimate so if you 13 start just stepping back one thing your final 14 formula is a ratio-type estimate based on 15 three estimates, VMT, miles per gallon, and 16 cost per gallon. 17 MR. SCHIPPER: Right. 18 DR. SITTER: Do you have standard 19 errors on those three estimates? 20 MR. SCHIPPER: Standard errors in 21 the possible selections for each vehicle or 22 just the tallied estimate for each? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 159 1 DR. SITTER: Well, I mean, a 2 standard error based on the data that you 3 originally built these on. I mean, some of 4 them you've got to have a standard error in 5 the weighted estimated sales and then you use 6 that to weight the average miles per gallon. 7 You can treat those as fixed. I mean, these 8 are all sampling errors. 9 So ultimately you must be able to 10 percolate your sampling errors up in some 11 way. I think that was more of the question. 12 Your answer of giving three estimates, I 13 don't think that's really what was being 14 asked. 15 MR. SCHIPPER: So we do calculate 16 the precision for the fuel economy estimate 17 and the VMT estimate in a population sense. 18 Would that be basically the variance of the 19 three components, VMT multiplied by the 20 inverse of fuel economy times the price to 21 get the expenditure? Each one of those has 22 its own variability. I think in the sampling BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 160 1 sense we treat everything as a fixed value 2 and it's a composite of those three 3 components put together. Now, if you don't 4 add up the variability of each one of them 5 and the correlations among them is that where 6 you're trying to get? In other words it's 7 not a linear because you get a ratio. You're 8 right. 9 DR. SITTER: You could linearize it 10 so I think that was the question. Was it you 11 that asked, Moshe, about that? 12 MR. SCHIPPER: Well, it also could 13 be more interesting that the fuel economy 14 itself would be a less variable item because 15 in terms of what's out there, in terms of 16 population, the variability of fuel economy 17 is going to be much less than VMT. VMT is 18 going to drive the variability of the 19 estimate. Price, fixed value, we have a 20 month and a state item. I'd have to dig into 21 the results. I think that's a census of 22 the -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 161 1 MS. KIRKENDALL: No, but these are 2 samples and they do provide sampling errors 3 and that's probably going to be small, 4 though. 5 DR. SITTER: So I think, that was 6 the confidence band issue about your 7 interact. 8 MR. SCHIPPER: Another way to look 9 at it, instead of just one, take all three. 10 DR. SITTER: And then you linearize 11 the ratio and you can at least have some 12 confidence interval-type information when you 13 estimate. 14 DR. FEDER: I have a question that 15 maybe shows lack of understanding about what 16 this is about but the name of the study is 17 about energy use and there was some issue 18 about the cost. What are people mostly 19 interested in? Is it the expenditure or the 20 energy consumption or total volume? 21 MR. SCHIPPER: Again it depends on 22 the researcher. We would be interested in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 162 1 the energy use, the amount of energy consumed 2 in terms of what's the impact or the profile 3 of the light duty vehicle fleet in energy 4 required by the country; however, there will 5 be interest in knowing the expenditures for 6 certain types of households, people on the 7 lower income levels, elderly householders, 8 the amount of money that's spent for 9 transportation use or their personal 10 transportation use. That will be of interest 11 to other researchers but I think the main 12 statistic that EIA is looking for and, 13 Dwight, you correct me if I'm wrong, is the 14 consumption, the amount of energy consumed. 15 MR. BERNSTEIN: Let me come back to 16 Thomas' point related to this which is if I 17 were Guy Caruso sitting here today saying 18 what do I need to know to answer 19 congressional questions it is can we do 20 anything to reduce demand for gasoline to 21 reduce pressure on prices in the short term 22 and perhaps in the long term. And one of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 163 1 questions you guys need to ask yourselves is 2 is there anything in here that you can give 3 to him or to researchers who are studying 4 that question. Is there anything in here 5 that can get at that question? I mean, 6 that's the number one issue on the minds of 7 most politicians these days when it comes to 8 energy is gasoline prices. 9 What can this tell us about what we 10 can do in the near term or the long term, if 11 anything, to reduce pressure on prices which 12 comes about as demand issues as well as then 13 linking it up to the production and supply 14 stuff that you get from other parts of the 15 EIA? That's one of the real critical 16 elements you got to be thinking about. 17 MR. SCHIPPER: This has been a 18 lengthy process. The report that you all 19 were forwarded is up to over 200 pages long. 20 The initial report that came from the study 21 was appendices and methodology tables and a 22 very short write-up in the front, just a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 164 1 basic summary of here are some numbers, they 2 went up, they went down. 3 The current report embodies more 4 analysis, more how you can use this 5 information. One of the key components is a 6 composition of energy use based on household 7 types, fuel economy, vehicle types, vehicle 8 vintages, which really didn't pan out to be 9 anything, but there are items that we provide 10 analysis for over an above what we had 11 intended to do because we intended to come 12 out with a data product that here it is, here 13 are the caveats, here's something useful, but 14 in addition to that we put in a lot of 15 analysis work into trying to get at those 16 questions. 17 Now, we get at those questions 18 based on the year of the information that we 19 have so we're always based on the 20 characteristics and the advance that occurred 21 in the sample years. In 2001 we had 9/11, 22 which has a potential impact on those BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 165 1 numbers. So those things that we can't 2 control for may have been of interest to 3 other researchers to compare but that's one 4 of the caveats. We answer those questions 5 but it's only within the realm of the survey 6 —————————————————— collected so you go from 7 '94 to 2001. A lot of stuff has happened 8 between those two time frames and we lump it 9 all into one year or the other. 10 MS. McGUCKIN: Just to have fun, I 11 was going to say we look at those questions 12 at DoT. For instance, one of the things 13 people ask us is what kinds of trips would 14 people stop doing because of the rise of the 15 gas prices. Obviously it's very easy to do 16 for work trips but not so easy for shopping 17 and they may drop shopping trips or social 18 rec trips because of that. 19 So don't take the cost off. We 20 really do love this partnership because we 21 gain as much or maybe more. I don't mean to 22 say that while we're negotiating the next BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 166 1 ground but we get a lot from this 2 partnership. 3 MR. BERNSTEIN: That just 4 emphasizes the need to work together to 5 answer some of these questions that are going 6 to be critical for when your bosses have to 7 go up to give testimony in Congress on what 8 could we do in the next four months to reduce 9 pressure on gasoline prices. I mean, it 10 comes back around to that and there's clearly 11 a lot of information in DoT of things you do 12 with the data that these guys can actually 13 make use of and vice versa and I know it's 14 hard. 15 The other question I wanted to ask 16 just on the EPA data, you said they wouldn't 17 give it to you. So (a) do you know why, 18 which you don't have to answer now, but (b), 19 I mean, if you do a FOIA request or if 20 somebody outside EIA does a FOIA request is 21 there a reason why they wouldn't give it? 22 MR. SCHIPPER: Well, there was no BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 167 1 reason provided. EPA through DoE's fuel 2 economy guide does put out a rich data source 3 of fuel economy estimates. The second item 4 is the sales figures for each configuration 5 of the vehicle. That information EPA wasn't 6 willing to send to us. NHTSA was because 7 they had both the sales information and the 8 fuel economy estimate just so we had to go 9 through the second step of removing all the 10 CAFE program calculations because EPA will 11 conduct calculations for CAFE program and 12 then we just had to back everything out. 13 MR. BERNSTEIN: Well, I'll be happy 14 to make a FOIA request if you want —————————— 15 MR. SCHIPPER: And one more thing, 16 we find it beneficial to have all the travel 17 and trip information from the NHTS study or 18 now the NPTS. And there are users, I think, 19 Professor Golub at USC. Is it USC or -- 20 MS. McGUCKIN: I think it's UC 21 Berkeley, isn't it? 22 MR. SCHIPPER: Basically tying BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 168 1 these two data sets together and we put out a 2 public use file for that purpose so 3 researchers can do a lot of unique and 4 interesting things with energy data as well 5 as all the travel and trip information. 6 DR. FEDER: Just one question on 7 the NPTS, is there going to be any 8 demographic component and breakdown of travel 9 by work, shopping, or recreational travel so 10 we can answer some of the questions that Mark 11 was raising about how people respond to 12 rising costs and things like that? 13 MS. McGUCKIN: Yes, we get detailed 14 demographic and trip information from travel 15 diaries every household member keeps in a 16 single day so they report together so we also 17 get their household interaction and vehicle 18 use. 19 DR. BURTON: How available is that 20 data to the public? 21 MS. McGUCKIN: It's on the website. 22 You can download it. We also have a table in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 169 1 it to -- 2 DR. BURTON: You mean in that level 3 of desegregation? 4 MS. LISS: Oh, yes, the absolute 5 desegregate data set with identifying 6 information removed is on our website and in 7 addition on our website is an online analysis 8 table which allows you to go in and run the 9 data without having to go through the six 10 months of learning curve on this data set. 11 DR. FEDER: Actually it's a very 12 tangential thing but I should mention that 13 Nick and I are both interested in academics 14 and how they juggle and this data can find 15 its use for us. Apparently some people 16 travel long distance and carry the viruses 17 and diseases with them and it's good to know. 18 MS. McGUCKIN: How about tiger 19 mosquitoes, tiger mosquitoes got carried on 20 the freeway system from just the rush of air. 21 DR. HENGARTNER: It's actually more 22 important that similar kind of data is used BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 170 1 to build transportation models at the urban 2 level like TRANSIMS and they are used to do 3 academic modeling but that's really far out. 4 Actually I would rather wish to stop short 5 this discussion instead of starting one. If 6 we can talk offline I'd like to know more 7 what happened here. 8 If I can invite the rest of the 9 people out there. 10 Welcome back from the breakout 11 session. I would like to have them 12 summarized, the two breakout sessions, for 13 the rest of the committee. Walter Hill will 14 provide us the first summary about the 15 session on preserving EIA trustworthy data 16 sets, model documentation, and contextual 17 history. 18 MR. HILL: And I got much more than 19 I could easily summarize —————————————————— 20 couple of times that we need more money, more 21 staff. That was a key point ———————————— 22 there are many documents available —————————— BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 171 1 ———————————— three main points was the 2 mission —————————————— one of them might be 3 what you ideally would have and then what we 4 actually have —————————————— have open 5 documentation, have an honest system 6 —————————— very transparent. Unfortunately 7 there's no overall inventory. There's no way 8 of scheduling. There is no standard 9 available which makes it obviously difficult 10 to track records if you're looking for 11 something quite far back. 12 Much material now is in boxes. 13 They gave a number on the order of 10,000 so 14 there are tens of thousands of boxes of 15 records available so obviously searching for 16 something specific, say, pick a date, 1983, 17 1984, is a challenge. 18 Some of that you may find with an 19 online search but there's no guarantee that 20 you'll be able to search for records or 21 retrieve the records or ensure the 22 authenticity of a record from what's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 172 1 available now. 2 So that ———————— issues dealing 3 with the mission, problems with the mission. 4 Ideally the management ought to be in a way 5 that theres quality control, there's 6 credibility. Records are used to forecast so 7 you want information available for 8 forecasting. Accessibility should ensure 9 common language. You should be able to find 10 the responsible person or entity giving 11 records so if something is inconsistent you 12 could maybe contract on the source to correct 13 the inconsistency. There's actually a little 14 story given at the end, a problem in the way 15 records were kept where there was a record 16 that was lost. It was hard to track down 17 what had happened to that record. 18 From the academic side a number of 19 these things that are ideal are types of 20 things we would typically want, reliability, 21 validity, authenticity. A government agency, 22 they mentioned things that wouldn't be seen BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 173 1 —————— by academics ———————— want the records 2 secured. You want records certifiable and 3 you want the records stored and compliant 4 with government regulations. And those are 5 the types of things that it sounds like most 6 of those are available in one way or another 7 although there might be difficulty for 8 specific cases. Schedules might vary from 9 case to case. Reliability might vary from 10 case to case, for example. 11 The depository ought to have 12 procedural accountability, financial 13 sustainability. There should be standard 14 storage procedures in the ideal system. And 15 finally what actually exists and in what 16 actually exists there are records kept in 17 various formats, various time periods, from 18 various different agencies. 19 Some I noted may be not in the 20 PowerPoint presentation but notes ———————— 21 —————————— for example, in microfilm. I 22 remember that. Those are ———————————— in one BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 174 1 section and then online materials in another 2 section and then boxes are in another 3 section. So I can easily imagine that if 4 you're looking for material in 1981 oil 5 prices it may be difficult to find exactly 6 what you're looking for. 7 In those different schedules 8 available we've had some discussion about 9 when material should be ———————————————— 10 destroyed —————————————— deleted. That's an 11 issue. In some sense there's a trade-up 12 between having everything available with 13 minute by minute documentation would be too 14 much but insufficient information is also a 15 problem if you don't have enough so that a 16 researcher could track down annual 17 information then it's not enough so there is 18 some happy medium between those two extremes. 19 We were not actually sure where that should 20 be. Apparently there's not a standard about 21 what should be done. 22 It was recommended that procedures BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 175 1 ought to be standardized, there ought to be a 2 way of tracking documents that are stored, 3 and they're looking for ways to maintain the 4 information about models. Some models change 5 quite regularly. In my research I haven't 6 noticed that but looks like that's a severe 7 problem, something we would be concerned 8 about, models being changed from year to 9 year. 10 A couple of recommendations —————— 11 ———————————————— maybe one or two —————————— 12 it would be good if there was a central 13 content management place or procedure of some 14 sort so that if you're looking for record you 15 can go to one central place and there will be 16 documentation on where to go from that 17 central space. 18 And it would be good to have 19 information on what is needed over time. It 20 was noted that they're coming up on the 30th 21 year anniversary. Information going back to 22 early 1980s is there but how much do you want BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 176 1 to keep and how much is really needed to be 2 kept? 3 Also there's anecdotal talk about 4 somebody doing a dissertation in the early 5 1980s. The material isn't on record but it 6 is still possible to get information because 7 there's an oral history available but that 8 will disappear at some point ———————————— 9 —————————————————— might disappear ———————— 10 DR. HENGARTNER: Thank you very 11 much. Does anybody else who attended that 12 breakout session want to add to Walter's 13 summary. I just want to say again I think 14 that archiving is extremely important and 15 unfortunately under-appreciated by most. It 16 always seems like a drag on everybody but 17 having good records and history so that one 18 could go back and especially for researchers 19 it's a good thing. 20 And I really want to applaud the 21 EIA actually already thinking of trying to do 22 that in a systematic way. I think doing that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 177 1 in itself, the whole process, puts the EIA 2 ahead of the curve. So I just want to finish 3 this on a positive note that it's important 4 and it's a good thing that we are thinking 5 about this although I don't have a quick 6 answer to all the questions that were raised 7 during the breakout session. 8 The next breakout session was on 9 vehicle energy use, what we did and what it 10 can tell us, and I understand that Moshe is 11 going to summarize this session. 12 DR. FEDER: It's very hard to 13 summarize everything that's been said. It's 14 quite an impressive study actually that 15 combines a lot of data and many assumptions 16 that are made in its composition but let me 17 just briefly try to explain what this is all 18 about, the questions that the committee was 19 asked, and some of the answers that we gave. 20 The vehicle energy use study 21 actually is about a few things such as what 22 is the energy that's been used, what is the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 178 1 cost of that energy, and then I thought those 2 were the two questions but it turns out that 3 there's a lot of interest in how people would 4 respond to changes in cost and how policy 5 makers could reduce the consumption of oil 6 which is one of the questions that are 7 repeatedly asked. 8 So the process of this study was to 9 combine data from different sources. Just in 10 a nutshell how do you measure energy 11 consumption and its cost? You take a vehicle 12 and you know its theoretical energy 13 consumption in miles per gallon which you 14 multiply by the cost and there you are but in 15 fact the data come from different sources. 16 There's an adjustment. The EPA has a 17 theoretical number which is obtained by lab 18 tests. It's more efficient to look in the 19 lab than on the highway so there's an in-use, 20 on-road adjustment for the fact that highway 21 and city driving are different. They consume 22 more energy than in the lab. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 179 1 In addition we know that seasonal 2 variation affects our driving differently. 3 People who drive more in the summer because 4 of recreational needs consume less energy 5 because it's highway travel and so on and all 6 these adjustments are based on assumptions 7 that will be made. 8 In addition it turns out 9 information from the surveys and we had the 10 mention of the RTECS, which was discontinued 11 in 1994. There's the NHTS and so what I want 12 to explain is in 2008 there's going to be 13 NPTS, another survey. All those provide 14 typically a car manufacturer, a model name, 15 and a year. And it turns out that to almost 16 every such model there are many actual types 17 of vehicles that range quite variably in the 18 MPGs, the consumption. 19 So one example, a 2001 Jetta that 20 can range between 25.1 to way over so you 21 need to make an assumption which you can't 22 and they used a one to many linkage algorithm BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 180 1 and, of course, there's a lot of uncertainty 2 coming from that. Again there are the 3 adjustments for the on-road, in-use that 4 people make and other things as well. 5 The data, again there was a mention 6 here in the discussion that there was some 7 collection of data through diaries and then 8 there was some modeling done and comparisons 9 actually show that the modeling performed 10 quite well but we don't know how future use 11 is going to be in response to the rise in 12 energy cost and shortages which are described 13 as elasticities. I'm not sure I'm saying 14 this right but I'm not an econometrician. 15 And all these things combined lead to 16 estimates of energy use and energy cost. 17 The questions to the committee were 18 concerning the extensive assumptions that 19 went into the model, are those appropriate, 20 what can you do about it, any concerns about 21 it, and so on. Then we referred to 22 imputation and the committee said actually we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 181 1 should refer to this as modeling and it's 2 okay to make assumptions in modeling and be 3 understood about the uncertainties. Also one 4 of the products of this study is interactive 5 tables which allow for accounting of some of 6 the uncertainties especially about the 7 assumptions that are being made. 8 And another comment was made about 9 other studies that this is at least as good 10 as any other study that has been seen by the 11 committee members. So I think the people 12 here, and I'm not an expert in this, but the 13 others thought it was a good product. 14 The other question was should we do 15 this again in 2008 after the collection from 16 the NPTS and again what ground rules should 17 be set with regard to the DoT collaboration 18 because a lot of data is coming from DoT and 19 again definitely it should be redone in 2008. 20 And then the third question was 21 very encompassing. It was about what 22 improvements can be done to the data and to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 182 1 the methods and some of you may have heard 2 the responses because you already came into 3 the room where we were talking about it. And 4 I think one comment was made that the EPA is 5 changing the way it's doing some of the 6 statistics that are being used here 7 especially with regard to the fuel 8 consumption so definitely use that. 9 And another comment was made about 10 the methodology or the methods. One of the 11 committee members wondered about the 12 components of variance and how that affects 13 the standards, how that affects the output, 14 and it would be interesting to be able to use 15 all these sources of uncertainties and the 16 components of variance to know what range the 17 predictions from the model can be, not just a 18 single number. 19 Mark, maybe you want to add because 20 you probably understood much more than I? 21 MR. BERNSTEIN: No, I think you 22 covered things mostly. One of the things BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 183 1 that we did emphasize thinking about 2008 is 2 where else in EIA this information is going 3 to get used, particularly NEMS and STEO, and 4 while it is important for those of us outside 5 EIA to have this data because it is a big gap 6 but it's got to be used and if you're really 7 going to do it well in 2008 it's got to have 8 a market internal to EIA as well and so it 9 should be designed so that NEMS could take 10 the benefit from the effort. 11 DR. HENGARTNER: Anyone else from 12 that committee who wants to add to this? 13 MR. RUTHERFORD: I'll just add an 14 observation. I thought it looked like it was 15 quite a productive interaction with the 16 Department of Transportation. The EIA 17 presenter was apologizing about the fact that 18 they haven't done their own survey but my 19 impression was that this looked like a very 20 good way to work because they can double up 21 on the survey and share information back and 22 forth and so I was quite impressed. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 184 1 MR. BERNSTEIN: Yes, I agree you 2 guys ought to publicize the fact that you're 3 working well with DoT. 4 DR. HENGARTNER: I actually had a 5 question. Can you backtrack and find out how 6 consumers are going to react by buying 7 smaller cars because of energy consumption or 8 is that completely unrelated because there 9 seems to be an interaction here, at least 10 anecdotally? 11 MR. RUTHERFORD: We had a 12 discussion about rebounds and DoT and this 13 was one of the points we discussed. 14 DR. HENGARTNER: Well, thank you 15 very much. I don't want to be between the 16 committee and its lunch. Before we leave I 17 want to invite the public for any comments if 18 they have any. That's fine. 19 So the committee will adjourn for 20 lunch and reconvene at 1:25 p.m. up here to 21 continue with Learning from the Past, a 22 presentation by Renee Miller. Thank you very BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 185 1 much. 2 (Whereupon, at 12:23 p.m., a 3 luncheon recess was taken.) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 186 1 A F T E R N O O N S E S S I O N 2 DR. HENGARTNER: Good afternoon, 3 everybody. Welcome back to the committee 4 meeting. Before I start I would like to 5 remind you about dinner plans if Bill wants 6 to fill us in? 7 MR. WEINIG: Dinner tonight, 8 reservations are at 6:00 o'clock. It's at 9 Gordon Biersch. It's a beer pub and 10 restaurant. Nancy and I were there two weeks 11 ago. 12 MS. KIRKENDALL: For breakfast. 13 MR. WEINIG: The iced tea is 14 terrific. It's at the edge of Chinatown. So 15 it will be 9th and F Street NW. You can use 16 the red line, for example, to Chinatown, 17 Gallery Place and then exit to the south, 18 towards the river, and go to 9th and F. 19 DR. SITTER: If you want to walk a 20 little bit, I mean, if you're coming from 21 here you can take the blue or orange and then 22 get off at Metro Center and it's only three BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 187 1 blocks. Get off the 12th and F at Metro 2 Center and just walk three blocks rather than 3 changing trains. 4 DR. HENGARTNER: Excellent, good. 5 So let us start with the program hoping to 6 stay on time as much as possible. The next 7 talk will be by Renee Miller about learning 8 from the past, updating data quality efforts. 9 MS. MILLER: Good afternoon. I'm 10 pleased to be here to talk about our past 11 data quality efforts. These are projects 12 that I worked on for many years in the '80s 13 and early '90s and we're now thinking about 14 bringing back some aspects and we would like 15 your help to decide what we should do. 16 We'll start by talking about the 17 past because what's past is prologue, those 18 who cannot remember the past are condemned to 19 repeat it, and my favorite, this is like déjà 20 vu all over again. The past is prologue 21 doesn't sound so bad but we don't want to be 22 condemned and we don't want déjà vu all over BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 188 1 again and that's why we want your help. I'm 2 going to tell you about our past efforts, 3 what worked well and what didn't, and then 4 Alethea Jennings is going to tell you about 5 what we're thinking about for the future in 6 terms of a web product and then she will have 7 some specific questions for you. 8 Our past data quality efforts took 9 several different forms. Most of these were 10 done in the predecessor office to the 11 Statistics and Methods Group, which was 12 called the Office of Statistical Standards, 13 although some were done by the program 14 offices or working very closely with the 15 program offices. We had state of the data 16 reports which were detailed assessment 17 reports that were publicly available and we 18 presented information about what we knew 19 about the strains and limitations of the 20 data. We also prepared in-house reports that 21 had recommendations on how to improve data 22 quality and how to improve the process and we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 189 1 also at one point experimented with a quality 2 profile. This was also a publicly available 3 assessment report. 4 Some of you may be familiar with 5 this product. The Census Bureau or The 6 National Center for Education Statistics put 7 these out as well. During our business 8 reengineering efforts we also tried to 9 develop a summary measure of data quality and 10 we also present feature articles on data 11 comparisons. And I'll tell you a little bit 12 more about each of these efforts. 13 For the reports that became known 14 as the state of the data reports we looked at 15 results of validation studies. These were 16 very extensive studies done in the early days 17 of EIA. We were a new agency then trying to 18 establish our credibility and there was 19 concern that our respondents, particularly 20 the oil companies, weren't accurately 21 reporting to us all the time. 22 And so we conducted audits of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 190 1 company records and we carefully looked at 2 our frames and other aspects of the data 3 collection. We gained a lot of information 4 from these validation studies but they 5 weren't very popular with the respondents, as 6 you might imagine, or with the survey 7 managers and they were very expensive and so 8 when our budget was reduced in the early '80s 9 they were discontinued. 10 One of the reasons that they 11 weren't popular with the survey managers was 12 that the survey managers felt that the 13 results were misleading. When you conduct an 14 audit you often times find things that don't 15 quite look right and they thought that the 16 validation studies were making things look 17 worse than they look actually were because, 18 after all, the EIA data tracked fairly well 19 with data from other sources. So in order to 20 have a balanced approach in the state of the 21 data reports we also included comparisons 22 with other sources so we conducted the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 191 1 comparisons at the company level and at the 2 aggregate level. 3 Now, comparisons can raise more 4 issues than they resolve because you don't 5 know how good the comparative series is, you 6 don't know whether it's validated, sometimes 7 you don't have documentation, but never- 8 theless we found them useful in raising 9 questions about specific features of each of 10 the data collections. And I found them 11 particularly useful when we could do them at 12 the company level and we can match company 13 level data. 14 And then when we found 15 discrepancies we could call the respondents 16 and say how come you reported this on one 17 form and then on the other and back then, we 18 had very knowledgeable respondents and they 19 were able to answer these questions. I'm 20 told that this is more of a challenge now, 21 the respondents aren't as knowledgeable, and 22 so if we were to conduct an effort such as BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 192 1 this we might not get to the bottom of the 2 matter like we did in the earlier days. 3 Over time there were fewer 4 comparative series because there was a 5 consolidation of data collection efforts and 6 we no longer had the validation studies and 7 so we started relying on outlier detection 8 techniques to look for symptoms of problems 9 in the data. These were also conducted at 10 both the company and aggregate level. We 11 used graphical techniques. Sometimes we used 12 regression analysis to detect any anomalies 13 and, as with the comparisons, we followed up 14 because anomalies don't necessarily mean data 15 errors. There could have been a change in 16 the market or in the company structure. 17 This is an example of a data 18 comparison we did many years back. The data 19 are from 1977 to 1981. This is a comparison 20 of EIA motor gasoline supply data and federal 21 highway data and the dark line is the EIA 22 data and it was consistently below the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 193 1 federal highway data. So we initially 2 thought well, the federal highway data may 3 have included some double counting but then 4 we looked a little more and we realized that 5 we were missing the blenders. And so this 6 was beginning of an effort to get the 7 blenders into our survey And you will hear 8 more about this type of comparison a little 9 later on. 10 After we completed a state of the 11 data report for each of the major energy 12 areas we consulted with independent expert 13 reviewers to get ideas for what we should do 14 next and they recommended that for the next 15 round we prepare less detailed assessment 16 reports but that in addition we prepare 17 in-house reports that contain recommendations 18 on how we might fix some of the problems. 19 So we decided to look at the issues 20 arising from the data evaluations and try to 21 determine do these stem from the way the form 22 was designed, does the form need to be BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 194 1 changed. We also looked to see whether there 2 were going to be changes in the industry 3 structure that should be reflected in the 4 form design. These in-house reports evolved 5 over time, I don't think any two were exactly 6 the same, but they all contained those 7 features of looking at the form's design. 8 We also focused on response, 9 nonresponse, and processing error and we 10 looked to see whether the quality control 11 activities that we had in place were adequate 12 to control for those types of errors and we 13 also looked at standards pertaining to those 14 topics. And we developed recommendations, 15 which we then gave to the program office. 16 Another activity that we performed 17 a little later on was the development of a 18 quality profile. This report also presented 19 comparisons with other data series but it 20 wasn't its focus as it was for the state of 21 the data reports. The focus here was mainly 22 on sources of nonsampling error and I listed BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 195 1 them, coverage, nonresponse, measurement 2 error, data processing, imputation, and 3 estimation. 4 Actually Tom Jobine, who some of 5 you may know, did the quality profile for us 6 for the residential energy consumption 7 survey. And he looked at all the residential 8 energy consumption surveys that had been 9 conducted up until that point to see how they 10 changed so that if somebody was doing 11 longitudinal analysis they would be able to 12 see the changes over time. This report was 13 very well received in the statistical 14 community but unfortunately we didn't really 15 publicize it and so we didn't really receive 16 requests for it. 17 I understand the Census Bureau has 18 gone forward and is continuing to prepare 19 quality profiles. They call them accuracy 20 reports and they have them for the American 21 community survey and they cover these topics. 22 They are not quite as detailed as the one BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 196 1 that we did on the residential energy 2 consumption survey though. 3 Another effort at around the same 4 time was during our business reengineering 5 days when we decided to look for a summary 6 measure of data quality. And this came up 7 several times and we decided it couldn't be 8 done but nevertheless it kept coming again 9 and so finally we decided to try to do 10 something but ultimately decided that this 11 wasn't really possible and I'm bringing it up 12 not so much because I think we should revisit 13 trying to come up with the summary measure of 14 data quality but because there may be 15 something here that might be useful for a new 16 product. 17 What we did was we decided to look 18 at the dimensions of data quality and we 19 identified five dimensions, sampling error, 20 measurement error, coverage, nonresponse, and 21 what we called methodological consistency, 22 which was whether a series was consistent BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 197 1 over time. Several of these items are 2 similar to what was in the quality profile. 3 Ideally if we could quantify each 4 of these dimensions then the idea would be we 5 would somehow combine it and we would have 6 our summary measure but we didn't really have 7 quantitative information for each of these 8 measures. Well, we had sampling error and 9 you're thinking well, you have response rates 10 but we didn't have nonresponse bias. We 11 didn't have an estimate of measurement error. 12 That was the type of information that we got 13 from the validation studies that were 14 discontinued. 15 We really didn't have information 16 on coverage and we didn't know how to 17 quantify methodological consistency. So 18 since we couldn't come up with a quantitative 19 measure what we decided to do was to try a 20 qualitative measure so we decided to gather 21 all the information we had together for each 22 of the series and we figured well, we would BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 198 1 rate each series on each of these dimensions 2 on a 1 to 5 scale. 3 And we tried to develop guidelines 4 as to what was a 5 versus a 4 if we were 5 going to use a 1 to 5 scale like that. This 6 got very complicated very quickly. There 7 were concerns about the time it would take to 8 do this, whether we could ensure consistency, 9 and there was a comment made that well, even 10 if we could ensure consistency would we 11 really be providing the user with any more 12 information than they could get from reading 13 our explanatory notes. 14 So then a suggestion was made. 15 Perhaps we should focus on standardizing the 16 material in our explanatory notes. And this 17 was never really pursued but there may be a 18 possibility here if we could concisely 19 summarize the information we have on each of 20 these error types and present it. But since 21 we don't have information necessarily, we 22 don't have definite information for each of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 199 1 these error types, it's a real challenge 2 coming up with a concise statement and once 3 you start putting in all the caveats you get 4 back to the level of detail that we have in 5 our explanatory notes but I think this is 6 still something to think about. 7 Moving on to an activity that has 8 continued into the present, feature articles, 9 we have feature articles on data comparison 10 for petroleum supply and petroleum marketing 11 data that the Petroleum Division performs 12 annually and they're available on the web. 13 And we have a one-time product on data 14 comparisons for alternative transportation 15 vehicle fuels that was done by the Statistics 16 and Methods Group that's also a web article. 17 To summarize our data quality 18 efforts we could put them into three 19 categories. We had efforts to inform the 20 public, that was the state of the data 21 reports, the quality profile, the feature 22 articles. We had efforts to improve the data BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 200 1 or the process. That would be our in-house 2 reports to improve the data. And we also had 3 an attempt to measure data quality. We were 4 more successful with the efforts to inform 5 the public and to improve the data than we 6 were in coming up with the summary measure. 7 Overall what worked well was that 8 we had information on how different data 9 series tracked over time so that if questions 10 came up we had some answers. In working on 11 this we also gained a lot of knowledge. 12 There's stuff that I learned from looking at 13 the data series 20 years back that still 14 comes in handy. 15 Now, you may wonder well, do the 16 survey managers get much out of this. I'm 17 not sure we identified problems that they 18 didn't already know about but we were able to 19 help bound the problems so that when we had 20 discussions about them we all had the same 21 points of references. 22 In terms of format what worked well BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 201 1 was the question and answer format that we 2 tried for one of our reports. What didn't 3 work well was that we had a lengthy review 4 process. In general the process was lengthy. 5 It required a lot of resources, both EIA and 6 contractor support. We had a lot of input, 7 we produced a lot of output, but we weren't 8 necessarily sure of the outcome because at 9 least for the reports that were publicly 10 available we didn't have a lot of users. 11 So we're now looking to the future. 12 We're looking for something that's, number 1, 13 affordable and that would be useful and we're 14 just at the beginning stages of thinking 15 about this and I'll turn it over to Alethea 16 now, and she'll show you what we're thinking 17 about and get your comments. Thanks. 18 MS. JENNINGS: I'm going to take 19 this down for just a moment so you don't 20 confused. She'll be back. Good afternoon. 21 Renée has shared with you some of the quality 22 efforts that EIA has undertaken in the past. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 202 1 Some things worked well for us and some 2 things didn't work well for us. Now we'll 3 talk about the future. In order to take 4 advantage of what we think are the positive 5 aspects of our previous work we're 6 considering presenting data comparisons in a 7 question and answer format on the web. 8 Now, given our experience in 9 preparing various reports, feature articles, 10 and other types of data comparisons we think 11 a product of this type would actually provide 12 a useful and friendly product for our users. 13 We also discovered that there are several 14 advantages to providing this as a web 15 product. 16 The first advantage is that we 17 could present information as we obtain it 18 rather than waiting for a detailed product to 19 be completed. Initially we could always 20 provide general information and then provide 21 links for more details such as to methodology 22 papers or to the actual survey forms and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 203 1 instructions themselves. 2 Another advantage, as always, is 3 that it is cost effective. We always have to 4 be mindful of our budget and it's always a 5 good idea to keep that in mind. But the 6 third advantage is that since EIA staff would 7 develop the product for the web themselves 8 the staff members would benefit from the 9 learning process since there is an element of 10 training involved. So let's take a look at 11 our first example. 12 This is a demand comparisons graph. 13 If we look at the graph, which, by the way, 14 is in thousand barrels per day, this is based 15 on information currently available on motor 16 gasoline comparisons so we're using this as 17 our example. EIA presents information on 18 motor gasoline product supplied and motor 19 gasoline sales. Motor gasoline sales data 20 are also available from the Federal Highway 21 Administration. 22 This graph illustrates these three BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 204 1 data series, which are considered to be proxy 2 measures of motor gasoline demand, and you 3 can see quite clearly from looking at the 4 graph that the data points are relatively 5 close although there is some variation. With 6 regard to this graph we have some examples of 7 questions that we might pose as a means of 8 discussing data comparisons. 9 This is our potential format. This 10 is the way that we actually envision our web 11 product and it's just one of the formats that 12 we're thinking about where the graph will be 13 placed above the questions. This is another. 14 Now, question number 1, both EIA and the 15 Federal Highway Administration present data 16 on motor gasoline sales. The actual question 17 is why do this figures differ. 18 We could start in answering this 19 question by specifying the points of 20 collection, we could talk about who the 21 respondents are, and then we would provide 22 links to the surveys where those links are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 205 1 available. We could also explain that EIA 2 respondents are suppliers who sell to local 3 distributors, to local retailers, and/or 4 their end-users. These are respondents who 5 are asked to report sales volume by grade and 6 category. 7 Looking at the Federal Highway 8 Administration data, they collect data from 9 the states. Their respondents are wholesale 10 distributors who are asked to report gross 11 volume of gasoline and their data are taken 12 from the state taxation reports. 13 Let's take a look at our next 14 sample question. EIA also presents data on 15 motor gasoline product supplied. Is this the 16 same as demand and sales? What is the 17 relationship to consumption? Now, the 18 previous slide dealt with the differences 19 between the data series that come from 20 different agencies. This slide is actually 21 more about the concept that's represented. 22 We compute product supplied as net BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 206 1 production plus imports minus exports and 2 then we take into consideration the change in 3 stock. Now, product supplied is a proxy for 4 demand and consumption but it is not exactly 5 the same. This is a challenge for EIA 6 because of the fact that we do use the terms 7 interchangeably. Right now we're currently 8 in the process of revisiting the way that we 9 describe product supplied in our glossary. 10 Nevertheless we do think that it's worthwhile 11 to have a question included in the web 12 product that's based on concept. 13 Question number 3, do the sales and 14 product supply series show the same pattern 15 over time? While it seems that the series do 16 generally show the same pattern EIA faces a 17 couple of challenges. The first challenge is 18 in deciding how much detail to go into and 19 the second is in explaining why the series 20 don't always move in quite the same way over 21 time. And this is a challenge for us at EIA 22 because we may never be able to actually BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 207 1 determine why that is. 2 Our experience with comparison data 3 tells us that it's likely that we won't have 4 answers to all of the questions that the 5 comparisons pose so we have several questions 6 for the committee. First we'd like to ask 7 you what your general or your specific 8 thoughts are about what we are proposing. Do 9 you see any merit in going forward with a web 10 product to show the users what we currently 11 know? And finally we'd like to know if you 12 have any other ideas based on your experience 13 or from our past work that we should consider 14 for a web product on data quality. We'd 15 appreciate having your input. 16 DR. HENGARTNER: Thank you very 17 much. We'll start out first with two ASA 18 discussants, who I hope will then stimulate 19 the rest of us to have a broader discussion, 20 and I would like to start with Mark 21 Bernstein. 22 MR. BERNSTEIN: Thanks for the good BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 208 1 presentations. I notice you put Randy and me 2 on the same discussants and so both our last 3 meeting and couldn't get to hear who are 4 probably polar opposite in what we are 5 probably going to talk about but either way 6 and Nancy is going to be glad this is my last 7 one because she has heard me say this every 8 single time. And maybe one of you guys is 9 going to pick up this mantra. 10 This seems to be the same problem 11 I've had and maybe it's the presentation as 12 opposed to what you actually did because I 13 can never tell what you actually did was the 14 presentation but fundamentally what's your 15 objective, and believe the objective is how 16 to make sure the data quality is the best you 17 can make it and how to acknowledge when there 18 are data issues, it seems to me, the two 19 basic objectives. 20 I appreciate the concept of not 21 committing the same mistakes as in the past; 22 however, you've learned a lot over the years BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 209 1 and statistics have changed and data have 2 changed and sometimes spending a lot of time 3 going back in the past is irrelevant for the 4 future but I really think you've got to be 5 very clear what your objective is and you 6 start with the objective and you move from 7 there and I just didn't see the objective 8 come out in these presentations. 9 A little side note, the first thing 10 that occurred to me when I was reading this 11 and I saw those three data series, the first 12 question I would ask is why do we need all 13 three data series. Are two of the three 14 enough and could you save some money by 15 getting rid of one? I can understand where 16 the differences are and I see where they are 17 used but the fact of the matter is they don't 18 differ that much and if you could save money 19 by getting rid of one of them and so when you 20 are going to do these data comparisons if 21 you're doing these data comparisons for a 22 bunch of different stuff and you're finding BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 210 1 two or three data series answering similar 2 questions it's the first step in trying to 3 weed out things you don't need. 4 The other thing is what you need to 5 do to acknowledge when you know there are 6 problems in the data and how that has to get 7 reflected on the website. For example, I 8 think it's still there, and I haven't checked 9 recently, this issue I brought up maybe it 10 was a year ago or the last meeting. There 11 was a problem in the state level data for 12 2003 for four or five different states which 13 we uncovered while we were using the data, 14 and my researcher said there is this problem. 15 These things are all going up and down like 16 this. 17 So I asked Nancy. I was here at a 18 meeting. So, Nancy, can you find out what it 19 was? And the answer came back as yeah, we 20 knew there was a problem, but the fact of the 21 matter is on the web there was nothing that 22 said we know there is a problem and we're BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 211 1 working on fixing it. And so those of us 2 doing the research end up finding this 3 problem and not knowing what to do about it 4 or what we end up doing is extrapolating 5 between the two years because we couldn't 6 have —————————————— and it just wasn't 7 feasible. 8 But when you know or suspect there 9 is a data problem what you absolutely need to 10 have is either a pop-up, though a lot of 11 people have pop-up stoppers so that may not 12 work, but something very specific that says 13 either we know there is a problem in this 14 data and we were fixing it or you don't put 15 the data out there or we suspect there is a 16 problem, we're not sure what it is yet, but 17 be careful. You really need to have that out 18 there. 19 And given the fact that now 20 everything has basically been taken off the 21 web, all the data stuff, it's not that hard 22 to put something up there that says what your BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 212 1 latest feeling about the data is for those 2 people who use it. So I think you've got to 3 acknowledge when you know there are issues 4 and that that acknowledgement has to be up 5 there if the data is out there. That really 6 needs to happen, and that's my short comment 7 for ———————— 8 DR. HENGARTNER: Thank you very 9 much, Mark. Randy, do you want to respond to 10 Mark? 11 DR. SITTER: Well, I disagree with 12 everything that Mark said. 13 MS. FORSYTH: For six years? 14 DR. SITTER: Except what he said 15 today. It was a very nice presentation and a 16 nice write-up. I think you've done a lot of 17 work and so I'll start with I know absolutely 18 nothing about this basically. I don't use 19 data. I tried. Data just makes it harder. 20 So in any case I do have a few 21 comments but now that you know I know nothing 22 about this then you can take them as you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 213 1 like. Actually I think that this notion of 2 what is an objective is really important 3 because when I was reading it I was also 4 saying here I'm not sure what this was about 5 because in one sense you want to have as good 6 a quality as possible. 7 That, I think, was the objective 8 but a lot of what was presented was how can 9 we present the quality problems we have and I 10 think those are two different things and I 11 would hope the second one is not as important 12 except where it really does impact and that 13 to me is part of being good quality. 14 So there are different kinds of 15 presenting when we have a quality problem. 16 One is the measure of it and how can we 17 measure it and the other is we know there is 18 a problem, we should let you know there is a 19 problem, and letting you know there is a 20 problem is probably more important than 21 necessarily knowing exactly why. 22 And your example is a good one. I BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 214 1 mean, I have an outlier, okay? Any good 2 statistician if he's got one that looks 3 different from everybody else first thing he 4 should do is go and try and find out why 5 because it may not be a real outlier. I 6 mean, it may be a real outlier and not 7 something -- 8 DR. FEDER: Legitimate. 9 DR. SITTER: So if I know that you 10 think this is suspect I'm going to treat it 11 quite differently than if I don't know if you 12 really thinks it's real. So that you need to 13 know and I think that comes across in some of 14 the things that came to me. 15 So that really comes down to this 16 web-friendly thing. It was difficult for me 17 to tell who your users were. The questions 18 are very general and who they link to may be 19 more specific but I think that thought really 20 has to go into it if you're going to do this 21 kind of thing about who your audience is and 22 at what level you expect them to go into your BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 215 1 links. 2 The questions and answers you put 3 at the top looked to me to be to perhaps 4 statistically unsophisticated or not too 5 sophisticated a user, not the people who are 6 actually going to get into the data. And 7 what Mark was talking about are people that 8 are going to actually get into the data. 9 And for the people on the front end 10 of that if those are really your target then 11 you should at least let them know, for 12 example, in your example that even if they 13 collected exactly the same data they would 14 get different series and that's the part of 15 the issue because they would. 16 The way that everything was worded 17 on this thing was like these are real 18 numbers, they're just measuring three 19 slightly different things, no recognition of 20 sampling variability. They could in fact be 21 measuring exactly the same thing, which is 22 with noise, I mean it's hard to tell the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 216 1 difference. So that was one thing. 2 The other thing was you've got your 3 cost things if you're going to do something 4 like this and there are pros and cons to a 5 web-based structure. I don't know if it's a 6 pro or a con but there are inconsistencies 7 across products so different surveys you're 8 going to have different information up there. 9 That can be a strength if it's done 10 correctly. It can be confusing if it's done 11 poorly. 12 And then also within products 13 linking to reports that were created for a 14 completely different purpose can be more 15 confusing than it's worth sometimes with 16 these things. It's the easiest and cheapest 17 way to do it. You spend all your effort on 18 that very front-end part and then you just 19 slap links into it but the links are semi- 20 related or they were done for a different 21 purpose so they don't really tie in together 22 and if you want a good web product that means BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 217 1 that the things you're linking to you have to 2 know what they're being linked from and so 3 that certainly requires at the very least 4 some thought into how you write those things. 5 So I would suspect that where 6 you're going to start with is your 7 measurement of quality, then your internal 8 report, and then what you're going to put on 9 the web and there should be a pattern to what 10 that means and a strategy for how that's done 11 that doesn't increase workload too much but 12 recognizes that these things are going to be 13 linked from above in a web-based system. 14 So a simple example is to 15 completely change notation. I mean, it's the 16 nightmare if you link from three different 17 papers and they're all using different words 18 for different things like you use words 19 interchangeably, maybe they use them 20 differently, really a bad idea, and that can 21 happen so easily when you're doing this kind 22 of stuff. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 218 1 The other thing is if you're going 2 to do it when you're designing and 3 implementing a survey you're measuring and 4 judging these quality issues constantly so a 5 process that captures it so that it can be 6 put on what you're envisioning easily. 7 So, summarizing your explanatory 8 notes, you need to standardize what that 9 means not just on the web but as you're 10 developing them so that you don't have to do 11 it three times. I mean, you don't want you 12 going back and looking at what other people 13 have written, figure out how you're going to 14 put it on the web. That's just building work 15 for yourself and there is no real need to do 16 that. 17 And again that relates back to 18 these feature articles. There needs to be 19 some structure to them to standardize them 20 across and within products so that you know 21 these three levels and how you are going to 22 put them up there. Then it shouldn't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 219 1 increase your workload too much because 2 you're doing all those stuff now. It's just 3 that different segments of your organization 4 are doing different things and no doubt they 5 have different terminologies for it and so 6 forth and a different way of doing it. So 7 you don't want to make it too homogenized 8 because you need variety as well. The 9 different surveys are different but some key 10 things would be important. 11 So those are the thoughts that came 12 to mind and I don't know how they answer. I 13 guess those are the thoughts. Do you see 14 merit in a web-based product? Yes. 15 Advantage and disadvantage and other ideas 16 for a web product? No, I have no other ideas 17 for a web product. I like the question and 18 answer as a front end. I think that's okay. 19 DR. HENGARTNER: Johnny? 20 MR. BLAIR: Yes, just a couple of 21 comments. I think that there did seem to be 22 this mix of talking about data quality BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 220 1 improvement and reporting about data quality 2 and I think it's hard to mix those in the 3 discussion or in terms of really developing 4 what you're going to do with them and how do 5 you communicate about data quality. Well, 6 those are quite different things in a program 7 for improving data quality and then this 8 reporting aspect. 9 I certainly agree with the notion 10 that it's important to acknowledge when you 11 think that there is a problem in the data but 12 it's also, I think, a little bit of a tricky 13 issue in that what's a problem for one person 14 isn't a problem for someone else; that is, 15 how do you decide what's a problem? There 16 are going to be some maybe that almost 17 everyone would agree on the extreme outlier 18 or whatever but there are always going to be 19 some gray areas too where depending on your 20 use of the data whether it constitutes a 21 problem. 22 If you see a spike in a standard BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 221 1 error, well, for some people that's going to 2 be a problem, for others not, so I think one 3 of the things that leads to is the difficulty 4 of the question-answer approach is selecting 5 the questions and then stating the question 6 in a way that seems, for lack of a better 7 term, a fair statement because a problem can 8 be described in in different ways, the same 9 problem. Two different statisticians may put 10 a slightly different spin on it. 11 So I think that while the question- 12 answer approach might be a useful one that 13 it's probably only a starting point in that 14 you have to have some kind of mechanism for 15 responsiveness to your users who then ask 16 follow-up questions or who raise questions 17 that you didn't initially think of and you 18 probably can't respond to everything but 19 there has to be some sort of selection. 20 But I think that if the beginning 21 and end point is the questions and answers 22 that you generate that you're probably going BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 222 1 to have a fair amount of dissatisfaction from 2 some segment of your users and it's pretty 3 hard to satisfy everyone. 4 So I think aside from pointing out 5 the obvious things, and hopefully there 6 aren't a lot of those that you have each time 7 that you've got to have a laundry list of 8 here's our data set now here are the 50 9 problems with it so hopefully this isn't 10 something that happens a lot and that there 11 are more not so much things described as 12 problems but limitations in terms of the use 13 of the data, what can these data support and 14 what kinds of things can they support, so let 15 me just stop with that. 16 DR. HENGARTNER: Thank you very 17 much. I'd like also to respond to the web 18 questionnaire form. When I was listening to 19 the talk, I mean, I was looking at it. 20 That's a great teaching tool. Now, a lot of 21 the users like Mark here probably already 22 know all that stuff but for someone who is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 223 1 new starting to see the subtleties on what 2 you are measuring and reporting becomes 3 important because it's not just that it 4 sells, no, because sales are at what level, 5 what this supports, and it goes with Johnny 6 is saying, and in a way you do address data 7 quality in a sense and in the following way. 8 Then you say exactly what it is that you're 9 measuring. 10 Instead of saying here are sales 11 you say these are sales and they're sales at 12 this level and this is how we collected it. 13 And if you don't like the results, well, then 14 go to the Department of Transportation. But 15 in a way it's really just defining the 16 context. And again I think that's more of a 17 teaching tool. There's a need for that but 18 I'm not sure every one of your customers will 19 find that important but I found it very 20 interesting to listen to so I think there is 21 some value to it. Darius? 22 MR. SINGPURWALLA: Yes, I mean, to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 224 1 put it on the web, I think, is a great idea, 2 actually, to put it out there and let 3 everyone see it and be able to use it in the 4 way that they wanted to. And I think Johnny 5 made a lot of the points that I wanted to 6 make. 7 One of the questions I had was who 8 is the public. Like, is it going to be 9 someone like me who's just going to be 10 sitting around and my PC looking for data for 11 an analysis to use because sometimes what 12 works in industry, like Johnny said, what's 13 someone's problem might be the way that I'm 14 about to make a company a million dollars so 15 looking at those differences. 16 And in response to the question and 17 answer session thing one of the things that I 18 was wondering if you have the capability of 19 doing was actually putting, like, data or 20 some sort of rolled up version of data out on 21 the web for users to manipulate their own way 22 because it's impossible for you to probably BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 225 1 think of every question that I'm going to be 2 looking to answer but, I mean, in previous 3 projects that I've worked on I know that 4 we've been able to make web interfaces for 5 users to actually go like an Excel pivot 6 table like drag and drop and look at 7 different cross tabs and do their own plots 8 over things. If you have the budget to do 9 that and the technology know-how that might 10 be a good way to answer some of the 11 limitations of putting your own questions out 12 there will have as well. 13 DR. HENGARTNER: Nagaraj? 14 DR. NEERCHAL: I think I agree with 15 a couple of comments and I want to reiterate 16 that it seems to be that the two series in a 17 very general sense that the observation that 18 supports the data quality is the fact that 19 they track each other. The fact that they're 20 differing is something everybody expects them 21 to differ at a small scale. 22 It seems to me that those two BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 226 1 things come up together in that discussion 2 and I feel unsatisfied listening to that 3 answer. I think there's a grave question, 4 why do these two series differ, but when I 5 listen to the answer that's like, of course 6 everything is different in a way. And so I 7 think that you really want to phrase the 8 question directly in terms of quality, that 9 directly addresses quality rather that just 10 the why numbers differ. 11 Of course, two numbers are going to 12 differ so I feel that maybe you want to focus 13 your questions more towards quality. That's 14 what I felt. It's a great question but when 15 I listened to the answer I felt unsatisfied 16 that then it's something leaving unsaid. So 17 you're listing a number of things that 18 contribute to why they're different. That's 19 not really going to tell me a whole lot about 20 on the other hand the fact that they're 21 tracking each other. Should they track each 22 other? Are they measuring the same thing in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 227 1 a very general sense? 2 So once again I agree. I think 3 this is really not meant for experts. The 4 experts don't need this kind of feedback. 5 They probably have very focused questions in 6 their mind and they probably would want to 7 have a chance to e-mail to you those specific 8 questions. I think it's a great teaching 9 tool, definitely, very nicely done, and I 10 think there's a place for those things but I 11 think you want to focus your customer. 12 DR. HENGARTNER: Thank you. Moshe? 13 DR. FEDER: I just wanted to 14 respond to one minor point that Darius made 15 about having a web interface to the data. I 16 learned it in another very different context 17 that users with the intent of retrieving some 18 confidential information that they are not 19 entitled to because it provides information 20 with disclosure potential about the 21 respondent can do that by querying the 22 systems in different ways until they have BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 228 1 enough cross-sections to identify a person. 2 So I just want to caution if there's anything 3 in the data at the micro level that you put 4 behind such a system it's very interesting 5 mathematically and otherwise to build some 6 protection into the systems so that won't be 7 possible. That's my only comment. 8 DR. HENGARTNER: Tom? 9 MR. RUTHERFORD: I just was going 10 to put in my two cents again as someone who 11 tends to be oriented toward questions and 12 objectives. You don't develop the data just 13 for the sake of maintaining it but we 14 actually try to answer questions with the 15 data. This often happens with graduate 16 students they want to build a trade model so 17 they have to build a global data set and they 18 spend all their time working on peripheral 19 aspects of the data when really they won't 20 identify what they need to know about the 21 data set till they have their model working 22 and they know what they're focused on. Then BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 229 1 it becomes clear that, well, this elasticity 2 is important or this share or these are the 3 tax rates that really matter. 4 So my concern is just that I think 5 that having a group within DoE that 6 identifies data problems and tries to help 7 users identify what the cause of this or that 8 inconsistency is is really useful because if 9 problems are identified as part of someone's 10 actually trying to use the data to do an 11 analysis that focuses you on what matters. 12 But in the process of just going through the 13 whole data set and looking for potential 14 problems you could spend a lot of time and 15 perhaps not be as contributing as useful. 16 In other words the role of having 17 that sort of facility if someone who helps 18 identify things reduces the need for 19 extensive documentation because —————— can 20 get feedback without necessarily having to 21 look up every single detail. But my general 22 concern is just this broad brush approach to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 230 1 all the whole data set is not perhaps as 2 productive ———————— respond to specific 3 questions. 4 DR. HENGARTNER: Thank you very 5 much. Johnny, toss a coin. 6 MR. BLAIR: I was just going to 7 make one quick point and that is that the 8 language that you use is important when 9 you're talking about these issues. If you 10 label something as here's a data quality 11 problem versus here's something that has 12 somewhat a higher variance than we expected 13 it can come across quite differently. And 14 how people look at what it is that you're 15 putting out can leave a quite different 16 impression depending on what labels you put 17 on it. 18 So I just caution that when you 19 start putting these out, particularly keeping 20 in mind the wide audience that you have from 21 the students and novice users to the very 22 high level experts, that the language that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 231 1 you use is important and you want to give a 2 lot of thought to that, I think, as you're 3 developing this approach. 4 DR. HENGARTNER: Randy? 5 DR. SITTER: Just one comment on 6 improving quality generally, you can probably 7 just ask people what's your biggest quality 8 problem. They'll probably tell you. Focus 9 on that. The biggest problems should always 10 be focused on first and sometimes it's so 11 easy to lose track of that. 12 So ask yourself what's your biggest 13 problem. If you know what it is, which you 14 probably know your top three right now, just 15 focus on those things first. And I think 16 that if you use that strategy in quality 17 improvement generally you'll always have good 18 quality. And the other one in quality that's 19 always true is a watched pot never boils. 20 Keep that in mind. 21 DR. HENGARTNER: Thank you very 22 much. Well, that was a good discussion. I'd BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 232 1 like to now to move on to our next 2 presentation. Again I'd like to welcome 3 Renée Miller, who's going to discuss can 4 discrepant estimates be good. 5 MS. MILLER: This is actually a 6 follow-on to what we were just talking about 7 and I'd like to thank Lawrence Stroud for 8 helping me get the material together for 9 this. I originally called the talk Measuring 10 the Economy May not Be As Simple as 1, 2, 3 11 after a Washington Post headline that Paula 12 Weir pointed out to us. 13 In this article they noted that the 14 Census Bureau puts out two different sets of 15 poverty estimates. The lower line is based 16 on the official estimate and the other line 17 was based on a poverty measure that the 18 National Academy of Sciences recommended. 19 MS. KIRKENDALL: Also put out by 20 the Census Bureau. 21 MS. MILLER: Also put out by the 22 Census Bureau. So I thought well, this is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 233 1 adding knowledge and I think in general this 2 was well received by the statistical 3 community that the Census Bureau was 4 providing some additional information but yet 5 they got a headline, "Conflicting Data," and 6 this made us start to wonder about some of 7 our data series. Do we have conflicting 8 data? Are we causing confusion or are we 9 helping users? So I just want to show you 10 some examples and get your thoughts on this. 11 If you look at our petroleum data 12 and go through our petroleum navigator you 13 have the option of getting data on 14 consumption and sales and we show data on 15 both products supplied and sales volumes. 16 Now, these are the data that Alethea showed 17 you and so you saw that they weren't exactly 18 the same but we do provide some description 19 as to what these concepts are before you 20 actually get the data and in our minds these 21 are two separate series. 22 This goes back to what Mark was BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 234 1 saying. Well, do we really need both of 2 them? But through the years we've had both 3 of them and we consider these two distinct 4 series and I guess something to consider is, 5 first of all, is this confusing for users and 6 then there are other implications about what 7 we show. 8 To make things more interesting for 9 our consumption data I don't know how clearly 10 this comes out but we also have data from 11 end-users. So, for example, we have data 12 from manufacturing establishments, this is 13 our industrial page, and then we also have 14 sales and delivery data collected from energy 15 suppliers and we label it as such but yet 16 they're both used as surrogates for 17 consumption. 18 Now, the data from the 19 manufacturing energy consumption survey 20 aren't supposed to be exactly the same as the 21 data from the industrial sector because MECS 22 is a subset of the industrial sector. So BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 235 1 basically here we just hope that it's lower, 2 which it is in most cases, but over the years 3 there have been challenges in reconciling our 4 data from end-users with the data that we get 5 from suppliers. 6 Another example of series that may 7 be considered conflicting data, we have sales 8 of fuel oil and kerosene and then we also 9 have what we call adjusted sales of fuel oil 10 and kerosene. The adjusted series is bench- 11 marked to product supplied and we did this 12 because in the early days of our sales survey 13 I don't think we were entirely comfortable 14 with the frame. 15 We weren't sure it was complete and 16 so we benchmarked to product supplied and 17 then over time we have developed users who 18 were interested in this series so we show 19 both sets of numbers. If you click on 20 distillated fuel oil and kerosene, for 21 example, you'll get links to definitions and 22 to the explanatory notes and the explanatory BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 236 1 notes explain the difference between the two 2 series. 3 Another case where we may be 4 showing conflicting data, this is a new 5 situation, natural gas production. We've 6 been showing data obtained from the state 7 agencies and estimates and this is what we 8 show on our natural gas web page. And then 9 recently we have data from the EIA-914. You 10 take a completely different path to get there 11 and we say this is the first release of the 12 data and these data are not yet official. 13 And we thought we were doing a service by 14 providing these data. We could have held 15 back until we finished our evaluation but we 16 thought it was worthwhile getting comments. 17 But this is another situation where we have 18 two sets of numbers that represents the same 19 concept. 20 The situation isn't just limited to 21 data as far. As our forecasts, the short- 22 term energy outlook is produced monthly and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 237 1 the annual energy outlook, as its name 2 implies, comes out annually and if you look 3 at the summary tables at first it looks like 4 we're not showing data for the same years 5 because the AEO shows data for 2010, for 6 instance, and the short term energy outlook 7 shows 2005 and 2006. But we do have an 8 annual table for the annual energy outlook 9 and so there are some overlapping years and 10 since the forecasts were not made at the same 11 time they're not the same but we make it hard 12 for users to make the comparison. 13 The data are in different units. 14 The STEO, I believe, is in physical units and 15 the AEO is in Btu. And the STEO is in 16 current dollars and the AEO is in 2003 17 dollars so users can make the comparison, but 18 they can't do it that easily. 19 I started out by showing you census 20 data and in May there was a seminar that the 21 Committee on National Statistics put on where 22 not only people from Census but National BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 238 1 Center for Health Statistics and Bureau of 2 Labor Statistics talked about their 3 challenges and it was called "Discrepant 4 Estimates, Love Them or Leave Them." 5 One of the comments at that seminar 6 was well, maybe discrepant estimates aren't 7 such a bad thing. After all, you're showing 8 the variability in the data. So the question 9 for you is what do you think about discrepant 10 estimates. How can we help the public better 11 understand them? Do you think we should just 12 restrict what we show? Do we concentrate on 13 developing better caveats? 14 We welcome your thoughts on these 15 and other issues because there are 16 implications in terms of what we collect and 17 what we present. It's your turn now. Thank 18 you. 19 DR. HENGARTNER: Thank you very 20 much. 21 DR. SITTER: One last chance, Mark. 22 DR. HENGARTNER: I'd like to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 239 1 welcome Jae Edmonds to present his comments. 2 DR. EDMONDS: This would seem to 3 be, as Renée said, a follow-on to the 4 preceding presentations as a more general 5 phrasing of the problem but I think it really 6 is essentially the same problem which we 7 looked at in particular in the last session 8 and therefore much of what I would say has 9 already been said, not necessarily by 10 everyone but it has been said. 11 The place I would start is that if 12 you're worried about a journalist writing a 13 piece in which they observe that at some 14 point EIA said something and then they said 15 something else and those two could be 16 confused and you want to protect yourself 17 against naiveté or mischief then I think 18 there's really only one strategy and that is 19 disbanding EIA because there's just no other 20 way to do that. I mean, it's impossible. 21 The problem that you have raised 22 occurs all the time and in fact one of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 240 1 things that I did was one of my colleagues, 2 Jerry Stokes, started something called the 3 Atmospheric Radiation Measurement Program, 4 another DoE enterprise. Radiation in this 5 sense is solar radiation so you don't need to 6 be too worried about that. 7 But they will frequently put 8 together four instrument packages and get an 9 observation of the same event in the same 10 place at the same time, one from the ground, 11 one from an airplane, one from a balloon, and 12 one from a satellite, and the odds of all 13 four of these giving the same value is 14 vanishingly small. What he said is what do 15 you do about this. Well, he said, what we do 16 about it is we flag it. When there are 17 differences we just put a flag there. 18 When you need to have an official 19 number he said that what they do is the armed 20 scientists get together and they decide 21 whether one of them is the best quality for 22 that particular observation and choose it. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 241 1 Sometimes they choose none of the above. 2 They'll take an average sum of them. Some of 3 the times it will just be their best expert 4 judgment but they note it and they in fact 5 note that these other values exist. As they 6 say, it happens all the time, a good example 7 of where you just choose one and you let the 8 others go. 9 If you're looking for fossil fuel 10 CO2 emissions you can get them in two 11 flavors, one demand side and one based on 12 supply estimates, and the supply side is 13 invariably chosen as the estimate of 14 preference and that's what you get reported 15 out as your global fossil fuel CO2 emissions 16 for the world. But other instances you'll 17 actually have averaging going on. 18 I think you need to distinguish 19 between the situation in which you actually 20 have an observation for the same thing at the 21 same time using two different instruments and 22 you have different observations. And much of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 242 1 what you were discussing was in fact you were 2 measuring things which were really different 3 but you were worried about the potential for 4 confusion and so this notion of flagging, I 5 think, is a useful one. 6 The more difficult problem occurs 7 when there are multiple data from different 8 observational environments where they in 9 principle should have some kind of 10 relationship to one another. For example, if 11 you were measuring something for industry it 12 ought to invariably be a bigger number than 13 manufacturing alone because industry is a 14 superset of manufacturing. 15 If the source of the estimate, of 16 course, is different then in fact you can 17 have these situations in which you have this 18 flip-flopping of the relationship which you 19 would expect a priori and when that arises 20 that's an occasion for asking why and 21 probably a place where you again want to put 22 a flag with some notion that you're looking BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 243 1 into this or at least whatever the state of 2 play really is. 3 I think that those instances where 4 you're looking at data there you in principle 5 have an event which occurred and therefore in 6 principle you should be able to have a true 7 measurement. I think that's different than 8 the case in which you're looking at forecasts 9 where you're looking forward out of a 10 foundation of history and the fact that you 11 have two or three different models that were 12 designed for different purposes which give 13 forecasts for the same future date that may 14 not be the same I think the basic result that 15 you're trying to get out of models is some 16 kind of insight and in fact I don't worry 17 about trying to make sure that those can 18 always be reconciled. 19 Those in my mind can be very useful 20 in that they can give you a reflection that 21 the future in fact is not etched in stone and 22 there is not one future that that must BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 244 1 evolve. So those are the things that came to 2 my mind in response to your presentation. 3 DR. HENGARTNER: Thank you very 4 much. We have some time so I'd like to open 5 discussion some more about data quality and 6 the topic we've heard so far this afternoon. 7 Yes? 8 DR. BURTON: Just to follow up on 9 what you said, in a world without any 10 resource constraints more data are always 11 better. The idea that something would be 12 suppressed or laid aside or not be made 13 available because of the potential to 14 introduce confusion troubles me because there 15 are two possibilities. 16 One, you have the same things being 17 measured in two different ways or 18 alternatively you have two series that have 19 similar but not quite identical definitions. 20 In either case, looking at the differences 21 between the series, whether it's from two 22 different measurement directions or whether BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 245 1 it's from just similarly defined series, 2 looking at the differences between those can 3 sometimes be really instructive. We can find 4 things in there that are sometimes very 5 useful to us. 6 The only thing I'd ask is that you 7 be very careful in how things are labeled so 8 that I as a user when I look at them realize 9 that I have two different series and why. If 10 you've got petroleum supply as reported by 11 state agencies then it needs to say that if 12 it's petroleum supply as reported by 13 producers then as long as I can see those 14 differences I'm better off having both 15 series. 16 DR. HENGARTNER: Nagaraj? 17 DR. NEERCHAL: How did the Census 18 Bureau respond to that story? 19 MS. KIRKENDALL: Actually the 20 answer is that in response to the CNSTAT, the 21 Committee on National Statistics, study on 22 poverty measures the Bureau of the Census BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 246 1 came up with not just one extra poverty 2 measure but, I don't know, 14. And they have 3 been publishing all of these experimental 4 measures for a long time now. I think they 5 started when I was at OMB so I think it's, 6 like, five years. 7 And I think the thought was that 8 OMB at some point would pick one to make it 9 the new poverty measure but they haven't done 10 that and so we're still going along with all 11 these 14 different poverty measures so there 12 are some reasons. I think the article picked 13 on OMB for not picking one. They weren't 14 ready to make a decision. 15 MS. MILLER: Yes, I mean, the 16 Census Bureau points it out that it wasn't up 17 to them. 18 MS. KIRKENDALL: The Census Bureau 19 is not going to pick how to define poverty 20 for the United States. That's a 21 responsibility with the Office of Management 22 and Budget, and they wanted to do that, yes. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 247 1 DR. NEERCHAL: So that analogy 2 applied here also then? If there are two -- 3 MS. KIRKENDALL: We don't have any 4 series that I know of that are quite at that 5 same level of importance as the official 6 poverty measure for the United States. And 7 so we don't have things that are dictated 8 quite so much as to how we should do them. 9 DR. HENGARTNER: Mike? 10 MR. HILL: Actually I thought the 11 department example is a nice one. I actually 12 get that question from students, that they 13 have several different measures, so it's well 14 known enough so a lot of undergraduate 15 students know they —————————— two books they 16 have different numbers it's supposed to be 17 the same data for the same year for the same 18 phenomenon. You ———————————— different 19 numbers so I thought it was a good choice of 20 an example. 21 I just ———————————— the initial 22 part of that maybe as a response to one of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 248 1 the questions from the last what, last half 2 hour, last session. I mean, who is the 3 end-user? So in some sense one of the end- 4 users must be the readers of the Washington 5 Post or somebody on that level. If that's so 6 ———————————— the public so you're not dealing 7 with experts. You want to be fairly general 8 at that level. 9 You could imagine something at 10 another level you could have questions that 11 come up —————————————— in the Middle East in 12 January 1991. One should go check to see 13 whether there is a change in oil consumption. 14 At that level you want one set of data. And 15 then at another level you're actually at the 16 level ———————————— the last hour where you 17 want several different time series and you 18 want to check the differences between those 19 different time series. Those are several 20 different levels and different types of 21 users. Maybe that's your graduate student. 22 They're completely different types BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 249 1 of users and I just would think that you'd 2 want different presentations for each one 3 ———————————— but the problem is challenging 4 from that end. There are many different 5 users out there who'll want different data, 6 different time series. 7 DR. HENGARTNER: Thank you. 8 Johnny? 9 MR. BLAIR: Just another comment on 10 the same issue of when you've got these 11 comparisons that differ, that jumping right 12 to the comparison seems to me that you want 13 to be careful not to overlook that first you 14 need a clear description of each of the 15 individual pieces of it before you get to the 16 comparison both in terms of the description 17 of it and how it's labeled. You might just 18 bypass the issue if that part is done 19 correctly and completely first. 20 And that's in a sense a problem 21 with the web, though I suppose it can happen 22 with text too, is that someone goes right to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 250 1 the graphic and starts to look at things 2 before having read the background and part of 3 the problem is getting them to go back and 4 read the background and see what they're 5 looking at for each of the individual 6 measures. So I think that needs to be on 7 your list as well. 8 DR. HENGARTNER: Yes, Mark? 9 MR. BERNSTEIN: I think this came 10 up also in the previous session and I just 11 want to make it clear again in this session 12 which is the first thing you need to ask 13 yourself is should there be a discrepancy. 14 And the answer to that question leads you to 15 two different —————————————— to do about it. 16 If there shouldn't be a discrepancy because 17 they're answering different things you need 18 to explain yes, they should be different and 19 here is why they should —————————————— bring 20 up and then if there shouldn't be a 21 discrepancy then you've got to start dealing 22 with, well, why is there and what do we need BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 251 1 to do to fix it. 2 And so when you do see those things 3 the should question should come first and 4 then that sets you into two different paths. 5 DR. HENGARTNER: Johnny? 6 MR. BLAIR: Part of the 7 interpretation is what your expectation is, 8 that is, what it is that you expected to see 9 quite often in looking at data. On another 10 committee that I'm involved with they look at 11 test scores of 4th and 8th grade kids and 12 they look at differences in math scores and 13 reading scores and so forth and every now and 14 then I'm looking at trends over time. You 15 see a mean score go up radically or you see a 16 score go down radically and I discovered 17 early on as a member of that committee that 18 it's only when the score goes down radically 19 that it's considered an anomaly. So one has 20 to be careful and keep in mind that there are 21 expectations as to how you interpret these 22 things. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 252 1 MR. BERNSTEIN: That's interesting 2 to say because I've just been dealing with 3 that in my son's school and last year all of 4 a sudden the 4th graders are just like way up 5 here and everybody was touting it as a great 6 success and I said wait a minute. Next year 7 can be very different and you ought to be 8 worried about whether next year it's going to 9 go down like this and you're going to start 10 wondering oh, my God, we're not doing as well 11 though I don't think energy quite works that 12 way. 13 DR. HENGARTNER: Nagaraj? 14 DR. NEERCHAL: I really liked the 15 way Mark put it. I think that those two are 16 the really key issues that will make those 17 questions focus a little bit better. First 18 thing it's going to give you a definition of 19 what each series is and the clear definition 20 will go a long way into what the user is 21 going to expect to see. And another thing is 22 that I felt that way when I saw the graphs BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 253 1 and I know pictures are great but once you 2 pull those pictures, once I see that graph, I 3 want them to be the same regardless. It's 4 like a psychological thing. I say I want 5 these two series to be the same because they 6 are going so closely together. 7 So you can give me a long 8 explanation as to what they are doing. I 9 think are you sure, and that is particularly 10 the case when you're talking about this 11 annual model-based forecast. I don't think 12 it's possible for you. One or two web 13 screens may not be enough to explain the 14 differences between the forecasting methods. 15 My experience with this committee shows that 16 forecasting methods are quite complex and 17 they are quite different so you may be 18 setting yourselves up for a impossible task 19 of explaining two complicated models and they 20 should be different anyway mathematically 21 even if they are starting with the same or 22 identical data set. So you may want to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 254 1 temper the expectation of the user to some 2 extent in your write-up. 3 DR. HENGARTNER: Anybody else? 4 Yes, Renee? 5 MS. MILLER: I just wanted to 6 follow up on the comment that Jae made about 7 flagging. This also has to do with what Mark 8 was saying about identify whether there 9 should be a discrepancy. The situation we 10 have, like, with the manufacturing energy 11 consumption data and our data on industrial 12 sector consumption, so we don't expect them 13 to be exactly the same but we do expect the 14 data from the manufacturing energy 15 consumption survey to be lower than the 16 industrial sector figure. 17 So for residual fuel oil we've 18 noticed that it's not lower. We've been 19 looking into this for quite a while and we 20 haven't quite found out what's causing this. 21 So would that be something that would be 22 useful to put a flag on either or both? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 255 1 MR. BERNSTEIN: Absolutely. 2 MS. MILLER: But what would we say? 3 Wouldn't we be raising more questions? 4 MR. BERNSTEIN: Well, but you've 5 got to raise those questions. Absolutely, if 6 you know that the manufacturing fuel oil 7 should not be higher than the industrial and 8 that's for sure and it is you need to make 9 sure and that if somebody is going to use it 10 they're going to use it at their own peril. 11 But they need to know that they're using one 12 of those numbers at their own peril. 13 DR. SITTER: Well, this one is a 14 striking example of it doesn't matter who the 15 user is. If this is an unsophisticated user 16 that wouldn't know that that's supposed to be 17 true they're going to screw it up. If it's a 18 sophisticated user they're going to identify 19 it for you and they're just going to be very 20 puzzled by your data. They're going to be 21 asking you what's going on. So it's got to 22 be flagged for every level of user. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 256 1 MS. MILLER: Actually we haven't 2 gotten questions from the outside users. 3 We've gotten questions from internal users. 4 DR. SITTER: That's because the 5 unsophisticated users won't notice. They're 6 just using it. 7 MR. BERNSTEIN: But you might be 8 getting people using one or the other and the 9 fact is you know that one or the other isn't 10 exactly right. Maybe they're both not right 11 but they can't both be right so it's one or 12 the other. And you may get a lot of users 13 using one and a lot of users using the other 14 and you really should let them know that 15 there may be a problem here. I know that's 16 hard because then you get blasted for having 17 poor data. But on the other hand you've got 18 to be honest about it and up front. 19 MS. MILLER: I mean, the challenge 20 is we don't really know what to say. We've 21 looked into a number of things and so far -- 22 MR. BERNSTEIN: You have a little BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 257 1 thing that says the manufacturing fuel oil 2 data is higher than our total estimate for 3 industrial. We don't know why or we're 4 working on the problem but the fact is you 5 don't have to go into a lot of detail other 6 than the fact that you let people using each 7 one know that there is this issue hanging out 8 there and it's possible that a user out there 9 would actually have a solution that you 10 haven't thought of and somebody is going to 11 come up and say, well, maybe it's because of 12 this, this, and this. 13 DR. HENGARTNER: So you're 14 suggesting to have like a prize to explain? 15 MS. MILLER: Right, a prize. 16 DR. BURTON: This is the sort of 17 thing that as a data user absolutely scares 18 me to death because when I look at that I 19 don't know if this is one case where things 20 are messed up or whether they're all messed 21 up but this just happened to be the one where 22 the manufacturing exceeded the industrial. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 258 1 So, I mean, if I see even one record like 2 that it makes you disinclined to use the data 3 at all. 4 MS. MILLER: But if we flagged it 5 wouldn't that -- 6 DR. HENGARTNER: No, because then 7 that shows you care about the quality of your 8 data. The others that are not flagged people 9 will have confidence in them and they'll say 10 well, at least they made due diligence. If 11 the end-user finds that error without you 12 having flagged that then he's going to say 13 the same problem occurs in all the series. 14 DR. BURTON: That helps, flagging 15 helps, but that's the sort of problem that 16 you've just got to fix. 17 MS. MILLER: But that's the 18 problem. 19 DR. BURTON: Mark said fix it. 20 MR. HILL: May be three or four 21 different types of users. The sophisticated 22 user wants to see the flag there and wants to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 259 1 know that I'm not crazy, right. There is 2 something wrong and they know that it's wrong 3 and either they're fixing it or they're not 4 fixing it. Then with the nonsophisticated 5 user there's another problem —————————————— 6 —————————————— a hidden item here. Years ago 7 there would have been a hard copy some place. 8 I don't know if that's what you're worried 9 about, is a hard copy that says there's an 10 error here, and you don't want that hard copy 11 around for 100 years ———————————— now it's 12 all web-based so ———————————————— an error 13 you could put it out in whatever you've 14 designed for that. 15 DR. SITTER: And one of the things, 16 answering Mark's comment there, is just that 17 if you have a history, if you do this all the 18 time and they see that you fix it and you are 19 able to identify them as small problems, he's 20 not going to be as shaky in the future. You 21 know what I mean? He's seen this in the past 22 and, generally speaking, your data is of good BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 260 1 quality and you're keeping track of it, so of 2 a general reputation of dealing with these 3 things well and not these big global problems 4 that sometimes occur. That will give them 5 more confidence and they'll still be able to 6 use your data. 7 MS. MILLER: I think the tendency 8 has been once we have resolved the problem 9 then we describe and document and footnote 10 it. But when we haven't we just -- 11 MS. KIRKENDALL: We can't figure it 12 out. 13 MS. MILLER: We just keep looking. 14 MR. HILL: —————————————— actually 15 I've personally used a data set in one of the 16 class examples. It's an energy consumption 17 and there are two points that are off-scale 18 and I have no idea —————————— it would be 19 nice to have a note that we?ve got these two 20 points are —————————— something happened ———— 21 —————————— whatever the story is for those 22 two points it would be nice to know. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 261 1 MS. MILLER: You're saying even if 2 we don't have the story -- 3 MR. HILL: You don't —————————————— 4 MR. RUTHERFORD: It's really a 5 matter of psychology, right? It's a matter 6 of educating your users to the fact that this 7 is a process. This isn't something that's 8 given on high. It reminds me a bit of in 9 Russia under the Soviet system Gaz Komstat 10 (?) collected all the data and they did all 11 these surveys and everything was knock, 12 knock, bang, bang, everything was very tight. 13 And then under fiscal federalism there is 14 this formula they've adopted for how they 15 share funds from the center and suddenly 16 there is a lot more debate about what these 17 actual statistics are. 18 And in Russia it's as though 19 suddenly the world has changed. We knew 20 exactly how things were before and now 21 there's debate that goes back and forth and 22 people feel very unsettled by it whereas as BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 262 1 an American company you say Okay, well, this 2 is the fact of the matter. This is a survey. 3 How do you expect this to be so precise? 4 So I really agree completely with 5 Jae that you flag it and you say we're 6 working on this as what we know. And the 7 fact that the Washington Post has some writer 8 who has this misconception about how the 9 world works who takes off and writes some 10 crazy article, I wouldn't worry about that. 11 DR. SITTER: I think it's just a 12 matter of setting up a best practice and 13 following it. 14 MR. BERNSTEIN: How are most of the 15 issues uncovered or maybe, like, what are 16 they? Or is that too hard a question? 17 MS. MILLER: Well, I mean, this 18 particular issue with residual fuel oil came 19 up actually when the people who were doing 20 the long term forecasting were trying to 21 decide what to use as their base. Do they 22 use the MECS number or do they use the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 263 1 industrial number? So they were the ones who 2 noticed it so it was internal users. 3 MS. KIRKENDALL: We're fortunate 4 that we have a lot of internal users. We 5 have several groups of modelers who are very 6 knowledgeable heavy-duty users. 7 DR. BURTON: If I'm using data from 8 a new source that I haven't used in the past 9 when I start to use the data just in the code 10 I'll make sure that I've checked the things 11 I've got just to see how good the data are 12 before I take off. 13 DR. HENGARTNER: Just as an 14 example, Mark, if you'd find an error would 15 you go back and flag it to the EIA or what 16 would you do? 17 DR. BURTON: It depends on what 18 impact it could potentially have on the 19 analysis. There are times when I would and 20 there are times where if I've got 500,000 21 records I'd just throw it out. If it happens 22 100,000 times then I got to go do something BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 264 1 about it. 2 MR. BERNSTEIN: Well, in my case if 3 it was one state in that one year that had a 4 problem I probably would have ignored it but 5 there was a group of problems. Before I was 6 part of this group I probably have contacted 7 the EIA because we needed to use the data and 8 I would have said what's going on here. And 9 the problem I had in this case was that the 10 analyst knew there was a problem. So oh, you 11 didn't find this other state too? 12 MR. RUTHERFORD: That cost us a 13 week's worth of work, actually, and to me 14 it's a business setting type of thing. And 15 that was a week's worth of wasted money that 16 we could have avoided if there had been just 17 some flag there that said here are the states 18 that we know there is a problem. 19 DR. NEERCHAL: Could I ask a 20 related question? Do you have an e-mail or 21 something that if you find problems with this 22 data please let us know? If an unknown user BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 265 1 somewhere in the corner of the country is 2 using the data and finds out a data issue 3 can we motivate that person to come back to 4 you with a question? 5 MS. KIRKENDALL: Our e-mail, you 6 can send that to the webmaster on our website 7 and those e-mails do get back to people to 8 solve problems. 9 MR. BERNSTEIN: And all the reports 10 and things have a contact. 11 MS. KIRKENDALL: And they all have 12 contacts, yes. 13 DR. NEERCHAL: Do you get e-mails 14 like that in the past? 15 MS. KIRKENDALL: Yes, we get 16 e-mails with questions about quality and is 17 this wrong. We do get some comments like 18 that. 19 MS. MILLER: They don't all go to 20 one place. They go to a particular subject 21 area. 22 MS. KIRKENDALL: Some of them come BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 266 1 into the NEIC or the webmaster and he splits 2 them out and sends them out to the 3 appropriate person with the questions about a 4 certain product. He'll send it back to the 5 owner of the product to resolve. 6 DR. NEERCHAL: Keeping track of 7 that also could be a quality indicator. If 8 you have fewer and fewer questions about a 9 particular database it's a good indication 10 that they're weeding out the errors. 11 DR. SITTER: Sometimes if you have 12 one person or somebody that actually sees 13 them all in a more global sense you can 14 discover bigger problems that aren't specific 15 because they're happening in different places 16 for different surveys. 17 DR. HENGARTNER: Randy, do you 18 think one could detect a frame drifting by 19 just looking at the data or errors like that? 20 That's one of the issues always. I mean, 21 where does the source come from and the 22 sampling frame is -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 267 1 DR. SITTER: That's a good example 2 of something that you might see if you've got 3 somebody that's looking a little more 4 globally and not so hot spot specific is that 5 the complaints can be in relation to 6 something, some other source or something, 7 and you start to realize that maybe it's 8 something bigger than just your methodology. 9 MS. KIRKENDALL: Actually I think 10 that first graph that Renee showed that was a 11 little hard to see from that very old 12 document did illustrate that because EIA 13 compared its results to Federal Highway for 14 some years and there was, like, a consistent 15 difference and we couldn't explain the bias 16 completely but there are some anomalies and 17 then they started to get apart. 18 Then people realized, well, there's 19 something changing and we weren't capturing 20 it and they discovered the blenders. And 21 that's one of the reasons why the data 22 comparisons can be useful. I'm just looking BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 268 1 at them periodically and you can explain some 2 things and you can try to improve the 3 relationship between the series. But looking 4 for changes between the series can help you 5 when your frame's deteriorating, something's 6 changed in the industry, a number of things. 7 MR. SINGPURWALLA: Well, and for 8 repeatable surveys too back in my old job we 9 used to look at distributions of the variable 10 itself against the previous month or year or 11 whatever it was repeated over time. And 12 that's always shown that the boat had sprung 13 a few leaks, lots of, like, really simple 14 internal quality control measures that it 15 would sound like a SAS program setup or 16 something and you feed in your data set and 17 it just spits out one report. It's, like, 18 really quick to do and you can program it in 19 Excel to, like, automatically highlight 20 variations that are big. 21 DR. HENGARTNER: Any more questions 22 and comments about data quality? Yes. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 269 1 MS. WAUGH: Hi, I'm Shawna Waugh 2 and I work with Renee. I actually worked on 3 the residual fuel problem. Renee asked a 4 question earlier about how to label this. 5 And what occurred to me was that FDA, I think 6 it's FDA anyway, provides labeling, like, if 7 you buy things from the grocery store, if you 8 buy cigarettes. It's got warnings on it. 9 What kind of warning label can we put on a 10 problem like this? 11 MR. BERNSTEIN: Maybe it's more 12 like the side effects, little in the 13 commercials the little tiny thing at the 14 bottom, the side effects of this. 15 DR. EDMONDS: Don't take the 16 industrial data together with the 17 manufacturing. 18 MS. KIRKENDALL: In answer to your 19 question this was a debate a long time ago 20 and then the petroleum data, the imports 21 data, are of much less accuracy, especially 22 preliminary ones, than the other data. It's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 270 1 just really variable and it's hard to get 2 them right. 3 So preliminary estimates came from 4 the weekly and everybody knew that wasn't 5 very good. So the way they talked about 6 showing the difference was to label it with a 7 little skull and crossbones as a footnote 8 symbol. 9 MR. BERNSTEIN: But in all 10 seriousness, I mean, you do have to think 11 about how you say it and I think somebody 12 here, I think Johnny, said that how you say 13 it, the interpretation is either that there's 14 something bad or wrong or that on a positive 15 note there are issues in data all the time 16 and you're working on fixing it. So the 17 warning flag or the flag could be simply we 18 recognize there may be a discrepancy in this 19 data or there may be a problem in this data 20 that is being worked on. And if you have 21 specific questions here is how to get some 22 answers on it. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 271 1 DR. EDMONDS: I think it's just 2 being straightforward in the instance where 3 you have an occasion where manufacturing is 4 greater than industrial you just say we would 5 expect that the relationship between this 6 data set and the values in the other data set 7 would be as follows and they aren't and 8 therefore we are looking into this and watch 9 this site because when we uncover this we 10 will post the results of our investigation. 11 MR. RUTHERFORD: Reflecting, I was 12 thinking I had seen something like before. I 13 was trying to rack my brain as to what it is. 14 It's in compilers. 15 If you use the best Fortran 16 compiler around I think it's Leahy. It's 17 been around for a long time. Tom Leahy goes 18 back to UNIVAC, a long time back, but on his 19 website they post not just what are the 20 outstanding bugs they're trying to fix but 21 they give you the history of all the bug 22 fixes they've done back for 15 years. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 272 1 So really high quality software 2 manufacturers actually not only do they tell 3 you what the problem of the week is. They 4 tell you historically what problem was 5 generated when. And I think it's simply just 6 this idea that they want to reinforce this 7 notion that hey, we're in there with you, 8 we're doing this, we're working on this. By 9 not hiding it they're being up front about 10 the fact that they're actually actively 11 working on it. 12 DR. HENGARTNER: The entire open 13 software is like that. The Linux kernels, 14 they have the same thing. LaTeX is also the 15 same thing. Kanouth has the whole list of 16 errors, when he has found them, what he has 17 fixed. 18 DR. SITTER: And then, of course, 19 when they update the upgrade they put across 20 and these were all fixed in the previous -- 21 MR. RUTHERFORD: And now we're 22 going to introduce the following bugs. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 273 1 DR. SITTER: Actually I require 2 flags of my students. Whenever they give me 3 an estimated probability that's outside the 4 range 0 to 1 they must flag it or they will 5 get 0 no matter what they've done. And I 6 state that up front and I still give out a 7 few zeroes. 8 DR. HENGARTNER: Also the other 9 thing is just preliminary estimates, mark 10 them as such. 11 MS. KIRKENDALL: That we do. 12 DR. HENGARTNER: And people will 13 know, right? And they won't expect them to 14 be final results. 15 DR. SITTER: I think the point 16 being made here and I think underlying all of 17 this is that there's a notion of 18 embarrassment that we want our data to be 19 perfect. I think you have a bigger mission 20 and that is to educate the public that —————— 21 data is not perfect. It's not the way it 22 works. So implicitly that you're trying to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 274 1 give the best possible data and that you're 2 working on it is not something you should be 3 embarrassed about at all. You should be 4 proud of it. 5 DR. NEERCHAL: To add to that, I 6 think simply because you're putting a warning 7 on one data point and no warning on the other 8 the no-warning one is not perfect, obviously. 9 As far as you know you have not discovered 10 any errors on it. In front of every sentence 11 the understood phrase is, "I think." It 12 would soften your language quite a bit by 13 starting every sentence saying, "I think." 14 MR. BERNSTEIN: I do realize, 15 though, it differs and you do have this 16 political dimension, I mean, in which 17 somebody in the House and somebody on their 18 staff says hey, these guys got bad data over 19 here, let's cut their budget. I realize that 20 issue permeates some of it but if you 21 carefully craft it and if it's not showing up 22 on every database then you should be all BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 275 1 right. 2 DR. HENGARTNER: And that's also 3 something that the review committee can bring 4 up and say well, this is a desirable 5 practice. I mean, if the review committee 6 comes and tells us well, yes, this is a way 7 of practicing if the Congress starts to push 8 back and say we don't like this, well, yes, 9 but we were told by the experts, not ASA but 10 the experts, that this was a desirable 11 practice and that the users would understand 12 what it means. 13 DR. EDMONDS: And in fact you'd 14 much rather have your flag up there than have 15 some committee staffer discover it. 16 MR. BLAIR: And I think you also 17 want to have some way to convey the process 18 to the user, not just that you have the flag 19 but that there is something that says this is 20 in general how we deal with things that we 21 don't expect to see in the data, that we flag 22 it and then someone's working on it in the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 276 1 future. You might see it removed or you 2 might see an additional explanation or 3 whatever but some description of what the 4 process is. I think that goes along with 5 this idea of that you're on the user's side, 6 that you're not trying to conceal anything, 7 that you're flagging things, but you're also 8 working on them. 9 But I think that needs to be said 10 more generally whether there are flags or not 11 hanging on a particular thing so that's 12 constantly reinforced, what the process is. 13 DR. BURTON: And truthfully it 14 could be worse than having something like 15 this reported in the press. I actually had a 16 reporter call me one time. I had transposed 17 two numbers and instead of reporting it he 18 gave me a chance to correct it. Subsequent 19 to that I did thousands of dollars of free 20 work for him. 21 DR. HENGARTNER: Let's take a 22 15-minute break. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 277 1 (Recess) 2 DR. HENGARTNER: Welcome back. 3 This is the last session for this afternoon. 4 Tom is going to talk about survey 5 self-assessment and he told me in confidence 6 that it's going to be short. 7 MR. BROENE: That's right. Thank 8 you, Nicolas. 9 MS. FORSYTH: Well, my commentary 10 is going to be long. 11 MR. BROENE: We will see. The 12 other thing I didn't tell him is I really 13 have two talks here. When I did the practice 14 run everyone said you've got two talks and 15 you didn't integrate them. So I'll tell you 16 that up front. 17 I'll tell you first about what's 18 going on in Europe. What we know now, some 19 of it's a little old, some of it's third 20 hand, some of it's new. And then I'll tell 21 you about our efforts at EIA. That's really 22 the best way I could think of to do it. So BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 278 1 we'll find out first what's going on in 2 Europe. 3 So the basic idea is we heard about 4 self-assessments being done and we heard 5 about it in that Stat Sweden first. And 6 people said that it was well received and 7 that they were making good progress so we 8 decided to try something similar. And we did 9 discuss this with the committee about 18 10 months ago when we were just starting the 11 interviews. 12 So let's go to what I know about 13 what's going on in Europe. Most of this from 14 Sweden and from Eurostat is from a conference 15 in May of 2004 in Germany. We didn't attend 16 but David Marker of Westat attended. He gave 17 his notes to Nancy. Nancy gave one talk and 18 I'm summarizing what Nancy said so I hope 19 it's right. 20 So Stat Sweden started this in 2002 21 and Westat worked with them on the 22 development of their quality program. And BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 279 1 I'm not sure what the product is, if that's a 2 survey or a group of surveys, but they're 3 doing 200 products. They're about a third of 4 the way through them. There was someone 5 giving a talk at the joint statistical 6 meeting this summer and so they've done 69 of 7 them so far. 8 I'm not sure if everything is done 9 only once every five years or if some are 10 done more frequently. It could very well be 11 some are done more frequently than every five 12 years but they have a 20-page self- 13 assessment. The survey manager, either 14 individually or in a group, I'm not sure, 15 fills it out. And then that's reviewed by a 16 team of others from Stat Sweden, people in a 17 different office but within the agency. 18 Those auditors have had one day of 19 training on how to audit and they do two or 20 four audits during the course of the year. 21 The nice thing about this is that it's very 22 intensive. It's one week. The auditors come BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 280 1 in, review the assessment, they talk with the 2 survey staff, my guess is probably more than 3 once, and then they write up their report and 4 they are through in one week. 5 And then they have follow-up. They 6 have about the equivalent of one FTE. They 7 have people working part-time. I think they 8 said three people working one-third time. 9 And I'm not sure if that rotates but somebody 10 is following up keeping track of all the 11 recommendations. They're averaging 10 12 recommendations and two good suggestions, 13 examples to share, and then they keep track, 14 what's been resolved, what's being worked on, 15 what are they thinking about, and no progress 16 on yet. 17 And we're seeing this too. 18 Computer systems were the biggest category of 19 things needing improvement but that's not a 20 quick thing to fix usually. For Stat Sweden 21 the documentation was second and the easiest 22 to correct. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 281 1 DR. EDMONDS: Tom, will you remind 2 me what are we assessing? What is the self- 3 assessment focused on assessing? 4 MR. BROENE: For us it's on our 5 surveys and I think it is for Stat Sweden 6 also. It's how is your survey going, what's 7 working well, what isn't, what is your 8 response rate, what's your estimation 9 procedures. The only thing I don't know for 10 sure is if they are grouping their surveys at 11 Stat Sweden or possibly it could be like 12 output also. I'm not sure about that. But 13 for us we're looking at on a per survey 14 basis. 15 That's a quick summary of Stat 16 Sweden. I hope this is from Eurostat. It's 17 possible it's from the UN. But this is the 18 DSAP. They have a 55-page questionnaire that 19 the survey major fills out. Some of the 20 questions are real short. I mean, a big long 21 list of select your imputation procedure. 22 Others are rank order your unmet data needs BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 282 1 and priority and describe your efforts to 2 address those deficiencies and so that's a 3 55-page questionnaire. And then at the end 4 of each section you get a question rate 5 yourself. So, like, for coverage what's your 6 under coverage? We have a little, we have 7 some, virtually none. The further away you 8 are from the center the better the score. 9 And so this gives you a way of 10 displaying a dozen measures in one two- 11 dimensional surface. And then behind this is 12 a 55-page questionnaire that the survey 13 majors fill out. 14 I gave an earlier version of this 15 talk this summer and I was in the same 16 session with Maria, the head of the quality 17 management at Portugal's INE. She agreed to 18 let me use her slides provided I put in an ad 19 for the ISI meetings. I've heard good 20 reports of the past ISI meetings and there 21 are some posters in the hallway there. No 22 tickets but I've heard good reports and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 283 1 Portugal agreed to host this. Switzerland 2 had to back out at the last minute. 3 So they have 800 employees. 4 They've been in business about 70 years. 5 There are four parts to their quality 6 management system. The framework has some 7 ISO norms. They are following a European 8 foundations quality model. Documentation, 9 they have handbooks, they have checklists, 10 they have minimum documentation requirements. 11 I'm just going to cover the 12 involvement and the audits. Maria spent 13 about a third of her talk on this. So there 14 are 800 employees. How to get the 800 15 employees involved? She had eight slides on 16 this. This is my summary of those eight. 17 They have internal meetings once a month. 18 Everyone is invited. She thought it was 19 working well. 20 They have a suggestion box. You 21 can do it by paper or electronically. 22 Supposedly top management answers every BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 284 1 suggestion and then there are awards both for 2 the best suggestion during the year and for 3 the best improvement in quality. So those 4 are some of the things they are trying to get 5 the 800 employees involved. 6 In terms of the audits it's mainly 7 a matter of are you following the checklists 8 in the handbooks. I mean, if you've got the 9 good procedures laid out, the minimum 10 documentation requirements, then it's a 11 question of are you following them and do you 12 need any changes in those procedures. 13 Again, they have two or three 14 auditors on each team. I'm not sure if they 15 rotate. It looks like they were more on a 16 permanent basis in Portugal but I'm not sure 17 about that. But they were trying to really 18 involve the auditors. It wasn't just that 19 they come in and critique things. It's 20 really trying to work as a consultant and try 21 to address solutions. And so that was one of 22 the things they were trying to do there. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 285 1 That's what I know about what's 2 going on in Europe and that's a summary. But 3 we didn't know any of that when we started on 4 this. We started this in 2003 and we formed 5 a team. Bob King and Paula Weir are here. 6 They were on it. We had at least one 7 representative from each of our offices that 8 run a survey and we worked with David Marker 9 and David Morganstein at Westat. They came 10 in here and ran a couple of sessions for us 11 and helped us draft a list of questions. 12 And we tried to address all ongoing 13 aspects. We didn't look at the startup 14 issues. I don't know if there was a separate 15 handout of the questions. I'm not sure that 16 got distributed but if you're interested I 17 have that set of questions. It's half-dozen 18 pages including a summary of quality 19 measures. 20 The basic goal, again, this is just 21 one approach that we have to look at the 22 quality of our surveys and the basic goal was BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 286 1 to get their opinion. We didn't have what 2 Randy just suggested, the one question, 3 what's your biggest problem. Maybe we should 4 have asked that also. We didn't have that 5 but to get their assessment on the topics and 6 then get a list of what they know about their 7 data quality measures and their goals we 8 asked for a one year ahead goal. 9 So the format we are following is 10 to explain the purpose, have a group meeting 11 with myself and Nancy and Shawna Waugh of 12 SMG, and then Shawna and I met individually 13 with each survey major or at most two 14 surveys. We tried one meeting where we had 15 eight surveys and Nancy and I were pretty 16 confused. At least I was confused. That was 17 too much to absorb especially for someone 18 like me who's in a different office. 19 So we're not demanding proof, we do 20 ask follow-up questions, but, I mean, if 21 someone says they have 100 percent response 22 rate we ask them how they do it but we don't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 287 1 ask them for proof. So far we are posting 2 these internally. I heard some of the past 3 discussions just few minutes ago but the 4 decision so far was that we would hopefully 5 get a more honest appraisal if we did not 6 post these on the web. 7 And so the procedure so far is that 8 Shawna and I write the reports to spare the 9 survey managers the extra work of writing it. 10 And then they get to review it and they have 11 final say on what's included but then we're 12 just posting it internally. 13 We have not discussed most of this 14 with EIA management yet. The interviews are 15 completed but I'm still working on the 16 summary reports. We've had good cooperation. 17 One recommendation that was given 18 months 18 ago here that we didn't follow was to 19 interview everyone. We didn't get to the 20 contractors. We ran out of time. We did do 21 some of the IT staff. 22 There was a learning curve for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 288 1 myself and others on this. It takes a while 2 to do these especially since we are in a 3 separate office. And, of course, some 4 managers are easier to interview than others. 5 Most were pretty good. 6 We got a lot of variety and who 7 responds to our surveys. There's a lot of 8 staff turnover in the organizations that we 9 contact. They get a lot of new contacts in a 10 given year. Some of the contacts are 11 contract employees so they're not even 12 employees of the firm. The changes of the 13 data collected, that's a real understatement 14 there to say they're time consuming for us. 15 It can be very chaotic for some cases. 16 One survey manager suggested that 17 we should do a cost-benefit study before we 18 make changes to our forms, try to figure out 19 really how many of our respondents are going 20 to be able to provide the data, how easily, 21 how good is it, and then what's the magnitude 22 going to be when we get it. Is it going to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 289 1 be 1 percent of the total? Is it going to be 2 something bigger than that? 3 We got a wide variety of procedures 4 in terms of automated data collection. We're 5 all over the map. The most advanced is for 6 the EIA-23. It's an annual survey of oil and 7 gas reserves and several software firms are 8 marketing software to populate that 9 automatically. And they've also filed all 10 the forms required by the SEC. I ran over 11 this a while back, several months back, with 12 Howard Gruenspecht, our deputy. He already 13 knew about everything we told him through one 14 of our surveys and then Nancy asked some 15 details about a specific estimate for another 16 survey and he said well, that wasn't covered. 17 I don't know. So we got a lot. We're 18 running about five to six pages per survey 19 but still if you get into great detail unless 20 it was brought up it's not going to be 21 covered here. 22 I gave this talk in December of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 290 1 last year at FCSM and somebody said if you're 2 doing a self-assessment you've got to have 3 tough questions. There is some evidence that 4 two of our questions at least were too soft 5 because people contradicted each other when 6 they shouldn't have. Two survey majors in 7 the same group are contradicting each other. 8 So we asked about training. Do you 9 have good training for new staff? Or cost, 10 can you estimate what it costs to run this 11 survey? Well, one person said yes, the other 12 said no. So those questions, I think they're 13 still useful but they may not have been tough 14 enough. On the goals being selected they're 15 pretty reasonable. Most are projects already 16 underway. We did try to avoid making new 17 work. 18 That's a summary of our current 19 status and any comment is appreciated, 20 especially how often. Can we group these by 21 program? Do we need a longer questionnaire 22 and then a more rigorous review? Do we need BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 291 1 an expanded list of questions? And who 2 should be on the team? Should it include IT 3 people, for example? And that's it. 4 DR. HENGARTNER: Thank you very 5 much, Tom. I'd like to invite Johnny Blair 6 to provide his comments or responses to this 7 presentation. 8 MR. BLAIR: First my general 9 comment is that it looks like you're 10 progressing pretty well, that you've got a 11 reasonable structure that you are working 12 within. The kinds of things that you are 13 doing seem sensible and you're obviously now 14 focusing on obtaining this in a sense 15 baseline information about your surveys. 16 But what I want to do is to talk 17 about and try to respond to some of your 18 questions, at least, probably not all of them 19 but in the context of looking forward as well 20 as what you are doing right now to get this 21 off the ground because I think it's important 22 particularly with a self-assessment that you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 292 1 start now thinking about the direction that 2 it's going to go in and how you keep it 3 effective and keep people engaged and 4 effective is not necessarily uniform. 5 I think it has to be a dynamic 6 system that's going to respond to changes in 7 the survey changes and the economic climate 8 that you operate in and so forth. So I think 9 that keeping the program healthy is something 10 that you want to begin thinking about now 11 even though your primary focus is on getting 12 it off the ground. As far as the frequency, 13 for example, in the future I think that this 14 is a prime example of a flexibility that 15 depends to some extent on the results that 16 you get, that if something is looking really 17 great you may not need to follow up on that 18 one as soon as another survey appears to have 19 a serious problems. 20 Another issue in looking at the 21 frequency is part of the next your follow-up 22 is going to be to look at were the changes or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 293 1 the goals implemented that had been set and 2 so you have some expectation as to what's a 3 reasonable period of time to expect to have 4 goals implemented. If the goals are 5 primarily IT kinds of things there may be a 6 different expectation in terms of time to 7 implement than if the focus is, say, on 8 response rate or on doing something to 9 upgrade a sampling frame or whatever. 10 Also as far the frequency to keep 11 in mind, and I think this was something in 12 maybe the Swedish plan, that it's not 13 necessarily an all or nothing follow-up, that 14 it can be focused and that maybe a way to 15 make it more efficient. 16 Second issue, coming back to this 17 notion of keeping in mind that when I look at 18 this list that you have under the EIA 19 approach that begins with frames and sampling 20 error and response and imputation and this 21 long list is to keep in mind that you're 22 addressing different categories of survey BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 294 1 performance and survey quality; that is, 2 you've got a set here that relates directly 3 to total survey error as far as question 4 content or response rates and so forth. 5 And you've got other things that 6 relate more to the user as far as 7 documentation and communication, 8 confidentiality, and so forth and then 9 submissions related to cost. I think over 10 time as you're setting goals you want to do 11 two things. One, in each area I think you 12 want to, going back to Randy's advice, what's 13 the big issue there that needs to be 14 addressed and focusing on that because you 15 can't do everything but that also you want to 16 balance that with over time being sure that 17 you cover some of these other areas, that is, 18 the things that are related to the user as 19 well as the things that directly impact total 20 survey error. 21 You want to keep the staff engaged 22 in this. Any kind of self-assessment, that's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 295 1 important. You want them buying into it and 2 one of the best ways of getting people to buy 3 into something is letting them have input 4 into it. And so when you're thinking about 5 some of the questions that you're posing to 6 us also whatever mechanism, informal or 7 whatever, but to give your staff the 8 opportunity to comment, to have input on the 9 overall process as well as specific parts of 10 it that affect them. 11 As far as the team composition 12 there's the question about whether IT should 13 be on it. I think that some rotation is 14 possible. Who should be on it? It would 15 depend to some extent on your expectations. 16 You don't go into this completely blind. You 17 know something about the particular survey 18 and where there might be issues presumably 19 before you begin the assessment. 20 So certainly if there were 65 21 percent of the issues or whatever related to 22 IT you probably had some inkling of that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 296 1 before you actually collected the data and 2 get the 65 percent number and so that would 3 suggest in that particular example that you 4 want to have some representation from IT to 5 do up a proper assessment and to properly set 6 goals. 7 And also consider, though, that it 8 may deviate somewhat from the notion of 9 self-assessment but to look at, at least 10 periodically, certainly after you've gotten 11 beyond the baseline of bringing in someone 12 from the outside to participate and I'm 13 thinking mainly someone from their user 14 community who will have likely a different 15 perspective than a totally internal view. 16 I mean, this should be primarily 17 internal. That's the point of it. But it 18 doesn't have to be exclusively internal. You 19 can bring in someone user, consultant, 20 whatever, from outside even if they're only 21 involved in part of the process. You may get 22 some fresh or different perspectives. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 297 1 As far as this question about 2 review all the surveys in one program area in 3 the same year, assuming that's feasible just 4 operationally, there may be some advantages 5 to that particularly if similar goals come 6 out of the individual assessments and there 7 can be some cross-survey collaboration to 8 help address those goals. 9 So if you've got a program of 10 similar surveys and one of the goals relates 11 to some data collection issue that's to some 12 extent common across them then they can work 13 together. You might get a more efficient 14 process than if those assessments come out in 15 different years and then people are working 16 on essentially the same problem at different 17 times. So there are, I think, some 18 advantages to doing a program area. 19 Then just a couple of other points, 20 as far as whether you need to ask tough 21 questions I don't know that I would put it 22 that way. I'm not sure exactly what a tough BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 298 1 question is. I think you want to leave a 2 little room for interpretation, that you want 3 to ask specific questions and not leave a lot 4 of wiggle room because there obviously are 5 some potential even if not fully conscious 6 issues that I'm being assessed and if I say 7 that we're doing good training that that 8 reflects on our group and me individually. 9 So you don't want to leave a lot of room for 10 interpretation in a self-assessment set of 11 questions. 12 Anything else here? On the 13 question of whether you should add this 14 greatly expanded list of questions that's 15 hard to answer without looking at the 16 questions but I certainly would tend towards 17 fewer questions, identifying areas, and then 18 drilling down in those specific areas rather 19 than trying to hammer everything to death, 20 again with the idea that you can't address 21 everything. You have some idea where issues 22 are so why ask a huge number of questions. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 299 1 Ask enough to see if there is an unknown 2 problem that surfaces but you don't have to 3 hammer it into the ground. 4 I'll just wrap up by saying I think 5 that it's important as much as you want to 6 have a identifiable program with an identity 7 to it at the same time it's got to have some 8 degree of flexibility and it's got to have 9 elements in it that keep the staff engaged so 10 that after the first time when the novelty 11 wears off that it's not oh, no, here comes 12 this awful drudgery that I've got to do every 13 couple of years or whatever. I'll stop with 14 that point. 15 DR. HENGARTNER: Thank you very 16 much, Johnny, and the second discussant is 17 Barbara. 18 MS. FORSYTH: Right, so I want to 19 say first of all that I agree with Johnny 20 that it's a nice structure that you have set 21 up. You've got some processes that you 22 follow and the different surveys are exposed BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 300 1 to the same process so it's a fair process 2 and it's a transparent process so those are 3 all strengths, I think, and I think Johnny's 4 points about planning for the future and 5 planning for ways of keeping individuals 6 engaged are really good points since you will 7 be implementing this across time. 8 I looked at the statistics you had 9 up for Statistics Sweden about their self- 10 assessment program and I'm thinking that out 11 of 200 that they planed to implement they've 12 conducted 36 in two years and it's like this 13 is a very intensive process and so no, I 14 wouldn't increase the questionnaire but it's 15 clear that it's a process that involves not 16 only a lot of your investment but also a lot 17 of investment from the survey staff and so 18 anything to facilitate it and get feedback 19 quickly is obviously wise. 20 I wasn't really clear on the 21 purpose for the evaluation. So I understand 22 that it's at the survey level rather than at BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 301 1 a program level and I understand that what 2 you're looking at are some measures of survey 3 quality, and this is something that came up 4 earlier as well, you could be documenting 5 survey quality, you could be evaluating 6 survey quality, or you could be looking for 7 ways to improve survey quality or, of course, 8 you could be doing all three but what you do 9 the most of depends on which of those goals 10 is most important to you. 11 It sounds like the latter two 12 goals, evaluating survey quality and 13 identifying ways to improve survey quality, 14 are probably the more important goals but in 15 the results you present a lot of them are 16 documenting differences across surveys and 17 that's more documenting than evaluating or 18 improving. So the questions you ask people 19 to think about and the questions you take 20 into the assessments probably will vary 21 depending which of those goals are most 22 important. It's not clear to me yet which of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 302 1 those are or should be most important to you. 2 I liked a lot of things about your 3 approach. Things I really liked are first of 4 all that you were drawing on European 5 experiences. They've done a lot of quality 6 work and so it's nice that you're drawing on 7 it and making some analogies and identifying 8 some things that you're doing that are 9 different, perhaps. 10 Just as a sidebar, I don't know if 11 anyone else would be interested in this but I 12 think it would be really interesting to hear 13 a methods paper on what they're doing in 14 Europe compared with what you are doing here, 15 perhaps how the different approaches are 16 tailored to the different goals. At least my 17 understanding is in Europe they have pretty 18 structured quality improvement activities and 19 it sounds like this is less structured and so 20 it would be interesting to see how your 21 approach has been adapted for perhaps the 22 less structured, more informal, quality BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 303 1 improvement activities that you maybe 2 interested in. So, just as a sidebar, it 3 might be interesting to get a methodological 4 comparison and identify some dimensions for 5 tailoring. 6 I think it was smart to hire a 7 consultant to help you with the assessment 8 content. And I also think it was smart to 9 select a relatively narrow area to focus on 10 survey operations as opposed to survey 11 planning or survey design. Survey operations 12 is still pretty broad but it could be worse. 13 So I think it's pretty smart to select a 14 focused area like that. 15 I think Johnny said this. I really 16 like that your surveys pick annual goals. I 17 think they are annual. They pick goals. I 18 really like that that's an output of the 19 process. And I'm not sure that you mentioned 20 this in your talk but it was in the paper 21 that you post the successes. You let them 22 select successes that they'd like to be BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 304 1 posted so that the entire organization can 2 learn from hard lessons that have been 3 learned by specific surveys. 4 And I have in my notes I imagine 5 it's important to give them some control over 6 what you publicize. But I can see from what 7 Johnny said getting staff involved, giving 8 them some input into the process, that it's 9 probably very important motivationally to 10 give them some control over what gets 11 publicized. It seemed funny to me that you'd 12 let them edit what you wrote up in the 13 summary but I can see now why that would be 14 important, to make them part of the process. 15 I wondered a couple of things and 16 these are like methodological questions. So 17 I'm wondering how the survey has reacted to 18 you guys as interviewers. It seemed like you 19 had interviewers from a separate group come 20 in and talk to them about their survey and 21 that could work really well or it could pose 22 weird interview dynamics. Maybe that's what BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 305 1 you meant when you said some people are 2 easier to interview than others. But I'd be 3 interested in how that worked and whether you 4 think it would contribute to the process if 5 you bring other team members into the 6 interviewing activities. 7 I was interested in your result 8 that some results are out of date before the 9 summaries are available. In other words I'm 10 wondering why whether it's the nature of the 11 data that you're collecting that maybe the 12 data are so volatile that it's almost 13 impossible to get a summary out quick enough 14 or maybe the summary process can be adapted 15 to get a summary out more quickly, maybe 16 fewer interviewers and more time summarizing 17 in between them so that you can get data out 18 while they are still useful. So that was 19 something I wondered about, what makes them 20 on a date. Is it the nature of the data or 21 is it the nature of the reporting process? 22 And I'll finish up by saying Johnny BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 306 1 mentioned on asking tough questions and FCSM 2 suggested that you should be asking some 3 tough questions and I wondered whether in a 4 self-assessment there is really such a thing 5 as a tough question. I mean, I understand 6 the difference between objective and 7 subjective but I think maybe the toughness of 8 the question really is in the eyes of the 9 beholder and in the attitude of the 10 respondent; that is, a relatively subjective 11 question can still get really good data if 12 the person is seriously invested in the 13 process and can still uncover potential 14 problems in quality if the person is really 15 interested in identifying problems in 16 quality. 17 And so I'm not sure that toughness 18 is in the question alone. It might be part 19 of the process. And maybe "toughness" isn't 20 the right word, maybe it's "honesty" or 21 "self-examination," but I'm not sure the 22 questions alone are going to influence what BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 307 1 they were describing as toughness. 2 MR. BROENE: Well, why don't I 3 respond to a couple of those? 4 MS. FORSYTH: I think that's it 5 except, I mean, it really is good to have, I 6 think, as Johnny said, a systematic approach 7 for assessing and improving quality. 8 MR. BROENE: A couple of responses 9 and explanations, in terms of a separate 10 group that's what we saw being done in Europe 11 and so that's what we did. Also our group is 12 separate, the Statistics and Methods Group. 13 The biggest drawback is we don't know enough 14 about the individual survey so like some of 15 the surveys I had worked on and knew a great 16 deal about them and Shawna and some others 17 and so we knew about some of them. Others we 18 had never looked at them and so that's a 19 drawback. 20 The out of date was mainly for 21 doing it right at the turn of the calendar 22 year and they were making changes in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 308 1 procedures and so the documentation I had was 2 out of date. And even though I got that one 3 out pretty quick it was out of date by the 4 time I got it out. 5 MS. FORSYTH: So it was just the 6 timing? 7 MR. BROENE: The Stats Sweden 8 approach would have been nice. They did 9 everything in one week flat. And so I guess 10 that's all the auditors work on and that's a 11 much more intensive time commitment for the 12 survey managers than what we were doing. We 13 were having a much shorter time commitment, 14 maybe a day total. 15 DR. NEERCHAL: This morning we had 16 a breakout session on the documentation and 17 how often to update and things like that. It 18 seems to me that that question has to be part 19 of this self-assessment process. I think 20 this is a great time to really look at that 21 issue and for all the products coming out of 22 that particular this may be the right time to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 309 1 develop how frequent updating and all the 2 questions coming up and very complicated 3 ones, really, but this seems like a good time 4 to do that because you are anyway focused on 5 that survey for a few hours or few days. 6 And the other one in terms of 7 addressing the burden issue, would it be 8 possible to have two levels of details? You 9 have a two-page questionnaire asking very 10 important questions at a more frequent basis. 11 And every seven years you do a your thorough 12 thing. Is it possible so that you have a 13 seven-year cycle so every year you're only 14 doing one-seventh of them, perhaps? 15 But on a yearly basis you still 16 keep track of some important issues and that 17 could be self-reported. That could be 18 something self-reported. And I want to 19 really appreciate what you said about 20 toughness. Someone asked me how are you 21 doing. It depends on who is asking me what I 22 am going to say. On the way if someone says BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 310 1 hi I'm just going to say hi and if a friend 2 asks me hi I'm going to stop and say do you 3 have time, a couple of hours. 4 MS. FORSYTH: What a tough 5 question. 6 DR. HENGARTNER: Jae? 7 DR. EDMONDS: In trying to think 8 about this I go back one level and ask the 9 question well, what's the purpose of all 10 this, and actually Barbara took us back there 11 in her comments and I would say that the 12 purpose that I would put forward is that 13 we're trying to improve the quality of the 14 data, the product that this organization 15 delivers, and so I think that it's really 16 important to begin there because of many of 17 your questions about how do you go forward 18 out of this experience, I think, really need 19 to go back and set up a test, ask yourself 20 what was I trying to do and did this activity 21 get me what I wanted as efficiently as 22 possible and in a sense what were the costs BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 311 1 and what were the benefits. 2 Obviously you've pointed out it 3 takes a lot of time and a lot of energy and 4 in a sense you've diverted resources from 5 potentially other tracks. And so I think 6 it's really important to ask what did you 7 learn out of all of this. 8 Because you're going very fast and 9 over this it looks like essentially a blanket 10 survey of all the individuals that were 11 involved in producing these data products. 12 And you're going and asking them how are 13 things going here, what's going well, what's 14 not going well. And out of that you're 15 hoping to generate some new information that 16 you didn't already have. 17 And so I think one of the things 18 you want to go back and ask yourself is what 19 did you learn. And one of the things that 20 you said I think is useful. You said we took 21 this up to Howard and Howard said I knew 22 that, I knew that, I knew that. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 312 1 And so the question is what is it 2 that you are learning and how much time and 3 energy did you divert to learning that. So 4 the first thing is where is it that we are 5 doing well and where is it that we are doing 6 badly, and then I think the next question is 7 did this survey process generate some 8 solutions that we hadn't already identified. 9 And it may be that those solutions were down 10 there in the people who managed the product 11 but for some reason had never bubbled up to 12 the surface and you were getting that. 13 But I think that what I would 14 recommend and maybe you've already done this 15 and just in the time that you've got to 16 present I know you just get this little 17 sliver of time, maybe that's already 18 happened, but that needs to happen. You need 19 to go back and you ask yourself that 20 question. 21 And you probably want to put a 22 couple of hard questions to your European BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 313 1 friends. What do you guys get out of this? 2 You spent a lot of money. What did you learn 3 from this? What are you doing differently 4 today than you were doing before you did this 5 assessment because you're putting a lot of 6 resources into it. 7 The other thing that others have 8 brought up that I'll just put out as a bullet 9 is what are the incentives to the people 10 you're interviewing. In some ways it 11 reminded me of these Maoist circles where you 12 do self-criticism and that's probably not 13 going to work in this environment. And so is 14 there an incentive? Am I saying my data is 15 not that good, does that imply that I'm not 16 doing a very good job, and so what are the 17 incentives to get a true reading, assuming 18 that in fact you want to go forward with it, 19 that is, you feel that this is the best way 20 to generate information you weren't getting. 21 And then at the same time you think 22 about what are the steps if I go ahead on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 314 1 this trajectory ask yourself well, is there a 2 parallel path that gets me to the same goal 3 and can I do let's lay up two or three 4 options and then say which look like the most 5 efficient given that we had now this 6 experience that we had. So those are some 7 thoughts that I had just listening to the 8 presentation this afternoon. 9 DR. HENGARTNER: Johnny? 10 MR. BLAIR: Before I make my 11 comment I want to underline Jae's point about 12 asking follow-up questions of the European 13 agencies. I think keeping those contexts 14 open and following up with them is something 15 that will be very very valuable. 16 A couple of things I wanted to 17 mention, I think he mentioned that sometimes 18 you're going in and you know a fair amount 19 about the survey and sometimes you don't have 20 very much knowledge about it. That's an 21 important thing to go back to and address, 22 how much you know about the survey before you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 315 1 go into it, I think, both in terms of the 2 reception of staff to you. You're asking 3 questions and particularly when you have to 4 ask follow-up questions or probe or whatever 5 it's going to be much better. 6 I also think that having a better 7 understanding would also affect the reporting 8 process in writing up the results afterwards 9 and also even before writing up the results 10 structuring what the report is going to be 11 like. So I think the whole process is 12 affected by how much knowledge you bring to 13 it about the particular survey. 14 The second thing I wanted to go 15 back to, this issue of how you ask the 16 questions has a number of effects but I just 17 want to touch on two of them. One is that a 18 respondent in addition to being cooperative 19 and so forth and wanting to be forthcoming 20 has to understand what you're looking for, 21 has to understand what level of detail that 22 do these folks want in my response. A more BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 316 1 specific question conveys that better than a 2 less specific question. 3 So if you're asking about training 4 and just saying do you have a training 5 program as opposed to do you have written 6 procedures for training will help them 7 respond more specifically to that question 8 and hopefully give you more uniform responses 9 across your survey but will also convey to 10 the respondent what you're looking for and 11 what level of detail you would like for them 12 to respond at. So respondent attitude is key 13 and is the fundamental thing that they want 14 to be cooperative and forthcoming but I think 15 that the question, and this is why you want 16 it to be specific rather than "tough," that 17 how you ask the question can affect how they 18 respond not just to that question but to the 19 later questions in the survey. 20 DR. HENGARTNER: Thank you. Randy? 21 DR. SITTER: I'm done. 22 DR. HENGARTNER: I'd actually like BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 317 1 to ask one of those tough questions. It's 2 all good to do those self-assessments and I 3 find it's wonderful we're doing that. But 4 then in the back of my mind it's, like, what 5 are actionable results. What are we going to 6 do with it? And I think one way to motivate 7 people to participate in that is if indeed 8 you can benefit them doing something in the 9 future. So are there any results you can 10 already say yes, we've identified this and, 11 lo and behold, we've changed that and look at 12 that, our response rate is better? Do you 13 have any success story so far? 14 MR. BROENE: The only thing that's 15 close is something that was already underway 16 and that was training on how to deal with 17 respondents. We have instituted some 18 training. That's all I know of it. I 19 believe it's a one-day course of how to deal 20 with respondents and that was something that 21 some people have already had training in 22 that. Others don't really train the people BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 318 1 that deal with our respondents. A lot of 2 them are contractors, some of them are very 3 knowledgeable, some of them are not 4 knowledgeable. And so that was one of the 5 things we have found, that there are big 6 differences among our surveys and the amount 7 of training for either federal staff or 8 contractors. But then that training 9 procedure was already underway. 10 DR. HENGARTNER: Thank you. I also 11 have something I'd like to throw out for the 12 committee. I know that Johnny is not going 13 to be here tomorrow so that's why I'm 14 bringing it up. I was wondering if there 15 would be a sense of actually having a 16 USA-Euro Cup on the data quality essentially 17 at the next JSM or maybe JSM 2007, i.e., 18 invite some of our colleagues from Sweden, 19 Canada and have a discussion at JSM on this, 20 if that would be fruitful for the EIA we can 21 sponsor a session and if you think that the 22 general statistical community might be BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 319 1 interested in something like that? 2 Yes, Johnny? 3 MR. BLAIR: To respond to that, my 4 answer is yes. I think that would a good 5 idea and I think that's something that people 6 would definitely be interested in. And I 7 also just want to make another comment to 8 follow up on your previous point and that is 9 when you do this self-assessment or whatever 10 assessment that you do it's important to keep 11 in mind that it sets up expectations of the 12 people that are working on these surveys and 13 that it's important that they see something 14 come out of this on the other end 15 particularly for the next round. When you 16 come around the second time to do this if 17 they haven't seen some results come out the 18 first time you'll notice it in the response 19 you get. 20 DR. HENGARTNER: We don't need to 21 make a decision but I'd like to throw it out 22 in the air and depending on the response we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 320 1 get I'll try to see if it's worthwhile doing 2 it. 3 MR. SINGPURWALLA: What do you 4 mean, World Cup? 5 MS. FORSYTH: Competition. 6 DR. HENGARTNER: A competition. 7 What I'm thinking is essentially the 8 Europeans have one way of doing their quality 9 control, we seem to have a different take 10 although we're somewhat inspired, and what we 11 want to know is what works and what doesn't. 12 MR. SINGPURWALLA: Well, what works 13 better. 14 DR. HENGARTNER: Well, another way 15 of putting it. Good. At this stage I'd like 16 to invite the members of the general public 17 if anyone feels compelled to go to the 18 microphone and wants to raise questions. 19 If not I'd like to adjourn this 20 meeting until tomorrow. I remind you dinner 21 is at 6:00 o'clock. Those who are going back 22 to the hotel, depart at 5:30 from the lobby. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 321 1 That should give us enough time. Or you can 2 fend for yourself. 3 (Whereupon, at 4:24 p.m., the 4 PROCEEDINGS were continued.) 5 * * * * * 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382