UNITED STATES DEPARTMENT OF ENERGY MEETING OF THE AMERICAN STATISTICAL ASSOCIATION (ASA) COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION (EIA) Washington, D.C. Thursday, October 28, 2004 2 1 PARTICIPANTS: 2 F. JAY BREIDT 3 NICOLAS HENGARTNER 4 JOHNNY BLAIR 5 MARK BURTON 6 MOSHE FEDER 7 BARBARA FORSYTH 8 NEHA KHANA 9 NAGARAJ K. NEERCHAL 10 SUSAN M. SEREIKA 11 RANDY R. SITTER 12 HOWARD BRADSHER-FREDRICK 13 ROBERT RUTCHIK 14 NANCY KIRKENDALL 15 PRESTON McDOWNEY 16 GUY CARUSO 17 TOM BROENE 18 HENRY S. BROOKS 19 BRENDA COX 20 GRACE SUTHERLAND 21 SHAWNA WAUGH 22 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 3 1 PARTICIPANTS (CONT'D): 2 TOM LORENZ 3 PHILLIP TSENG 4 JOHN WOOD 5 HOWARD GRUENSPECHT 6 WILLIAM WEINIG 7 INDUJIT KUNDRA 8 JOE SEDRANSK 9 KAREN NORMAN 10 11 12 * * * * * 13 14 15 16 17 18 19 20 21 22 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 4 1 C O N T E N T S 2 AGENDA: PAGE 3 Greetings, Presentations and 5 Remarks 4 Updates for the Committee 34 5 Proposal for Analysis of Collected 62 6 Data 7 A Customer Evaluation of the Short 83 Term Energy Outlook (STEO) 8 Frames: Progress on and Lessons 126 9 Learned from Frames Evaluations 10 The EIA Short Term Regional Electricity 134 Model: Capabilities and Data 11 Requirements 12 Natural Gas Production, Frames, 219 Samples and Estimation 13 Methods for Assessing NEMS Solution 227 14 Data for Interpretive and Diagnostic Purposes 15 Introduction to Program Assessment 238 16 Rating Tool (PART) Program Evaluation 17 External Evaluations of Survey Programs 245 18 External Evaluations of Forecasting 306 and Models 19 How the ASA Energy Committee Might Help 317 20 in Program Evaluation under PART? 21 22 * * * * * BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 5 1 P R O C E E D I N G S 2 (8:30 a.m.) 3 CHAIRMAN BREIDT: Okay, I think 4 we're ready to begin. Welcome to the fall 5 meeting of the American Statistical 6 Association Committee on energy statistics 7 with the Energy Information Administration. 8 This meeting is being held under the 9 provision of the Federal Advisory Committee 10 Act. This is an ASA not an EIA committee, 11 which periodically provides advice to EIA. 12 The meeting is open to the public and public 13 comments are welcome. Time will be set aside 14 for comments at the end of each morning and 15 the afternoon sessions. Written comments are 16 welcome and may be sent to either ASA or EIA. 17 All attendees including guests and 18 EIA employees should sign the register in the 19 hall and should include their e-mail 20 addresses. Restrooms are at the end of the 21 hall towards the back of this room, back that 22 way, a fountain is in the same corridor on BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 6 1 the way, telephones in this room share a 2 single number. It is, if you're interested 3 202-586-3071. 4 Kathleen Wert with ASA meetings 5 department is here. She was out at the front 6 desk and is available for committee member 7 questions on expense reimbursements. Tara 8 Stull was the former committee meetings 9 liaison, and you may know that she has moved 10 to a new job in Boston where she'll be 11 working for the Boston Museum of Natural 12 History. And if you were up late last night 13 you may have seen pictures of her revelling. 14 So in commenting each participant is asked to 15 speak toward a microphone, we have a 16 transcriber to appreciate that and committee 17 members and speakers at the head table need 18 to speak clearly and into a microphone, Bill 19 always writes that for me, "clearly and into 20 a microphone". 21 These microphones are reasonably 22 sensitive so you may not need to really lean BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 7 1 in, they should be able to pick it up. And 2 speakers are asked to use a microphone, 3 which is available here, and Bill can help 4 out with that. So now I'd like each of us to 5 introduce ourselves. So, we'll begin with 6 the committee members and then we'll go out 7 into the audience. Please speak into a 8 microphone and give your name and your 9 affiliation. So my name is Jay Breidt, I am 10 with Colorado State University Department of 11 Physics and can we start here. 12 MR. McDOWNEY: Preston Mc Downey, 13 EIR, SMG. 14 DR. NEERCHAL: Nagaraj Neerchal, 15 UMBC, Math department. 16 MS. KHANNA: Neha Khanna, State 17 University of New York, Birghamton, Economics 18 and Environmental Studies. 19 MS. Forsyth: Burhara Forsyth from 20 Westate. 21 DR. FEDER: Meshe Feder, Research 22 Triangle Institute, Research and statistics BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 8 1 division. 2 DR. BURTON: Mark Burton, 3 University of Tennessee, Knoxville center for 4 Transportation Research. 5 DR. HENGARTNER: Nick Hengartner, 6 Los Alamo, National Laboratory Statistical 7 Science Group. 8 MR. CARUSO: Guy Caruso, 9 Administrator, Energy Information 10 Administration. 11 MS. KIRKENDALL: Nancy Kirkendall, 12 Statistics and Methods Group. 13 CHAIRMAN BREIDT: Okay we'll go 14 back to the audience and can we use the 15 microphone please? Yeah I'm pointing at you 16 -- yeah or maybe Bill, you want to do this. 17 MR. WEINIG: This'll be fine. 18 CHAIRMAN BREIDT: Ok I'm sorry. 19 MR. WEINIG: One last member of the 20 EIA, I'm Bill Weinig with the Energy 21 Information Administration. 22 MR. TSENG: I'm Phillip Tseng, EIA. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 9 1 MS. HOJJATI: Behjat Hojjati, 2 Energy Information Administration. 3 MS. WAUGH: Shawna Waugh, 4 Statistics and Methods Group. 5 MR. HOUGH: Richard Hough, US 6 Census Bureau. 7 MS. HAITOT: Vicki Haitot, US 8 Census Bureau. 9 MS. SNAPP: Eva Snapp, US Census 10 Bureau. 11 MS. LOFTIN: Lacey Loftin, US 12 Census Bureau. 13 MR. BOURNAZIAN: Jake Bournazian, 14 EIA. 15 MR. HSEN: Paul Hsen, Census 16 Bureau. 17 MR. LORENZ: Tom Lorenz, EIA. 18 MS. BLESSING: Colleen Blessing, 19 EIA. 20 MS. MILLER: Renee Miller, EIA. 21 MR. SEDRANSK: Joe Sedransk, CWRU 22 and EIA. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 10 1 MR. RUTCHIK: Bob Rutchik, EIA. 2 MR. WOOD: John Wood, EIA. 3 MR. WEINIG: Just pass the 4 microphone over. 5 MS. JENNINGS: Alethia Jennings, 6 EIA. 7 MR. FREEDMAN: Stan Freedman, EIA. 8 MS. TAYLOR: Yvonne Taylor, EIA. 9 MR. JOOSTEN: Jim Joosten, EIA. 10 MR. BRADSHER-FREDRICK: Howard 11 Bradsher-Fredrick, EIA. 12 MR. WOOD: John Wood, EIA, Reserve 13 and Production tables. 14 MS. SPENCER: Lynda Spencer, Kema 15 Incorporated. 16 MS. FRENCH: Carol French, EIA. 17 MR. ZHANG: Bin Zhang, EIA. 18 MR. MILLER: Herb Miller, EIA. 19 CHAIRMAN BREIDT: Okay. Bill is 20 checking the halls for anyone else. Okay, 21 it's now time for my favorite announcement, 22 for your information Nancy Kirkendall is the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 11 1 designated Federal officer for the advisory 2 committee. In this capacity Dr. Kirkendall 3 may chair, but must attend each meeting and 4 she is authorized to adjourn the meeting if 5 she determines this to be in the public 6 interest. She must approve all meeting of 7 the advisory committee and every agenda; also 8 she may designate or substitute in her 9 absence. 10 Okay, so we have a very interesting 11 agenda today, we have a number of breakout 12 sessions. This is one of the most technical 13 agendas I think we've seen in a long time. 14 So there's a lot of statistical and 15 econometric meet here, so this should be a 16 lot of fun. There are a total of 10 breakout 17 sessions and the first session this morning, 18 which is not a break out session, plenary 19 session. 20 We'll begin the briefing with EIA 21 Administrator Guy Caruso, on EIA's general 22 state of affairs and then Nancy Kirkendall BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 12 1 the director of EIAs Statistics and Methods 2 Group will tell us about EIA work since the 3 spring 2004 meeting and the advice we 4 provided. 5 Lunch for the committee, presenters 6 and invited guests will be on the first floor 7 at 12:15. It will be in the usual room. And 8 this evening we have reservation for dinner 9 at the Aroma Indian Restaurant 1919 "I" 10 Street Northwest. You can take the blue line 11 or the orange line to get there. It's pretty 12 close to the hotel GWU Inn that we're staying 13 at. Our reservations are at 6, so can I see 14 a show of hands for who will be attending the 15 dinner. 16 Okay, and tomorrow morning 17 breakfast for the committee will be here 18 again beginning at about 8 and we'll resume 19 at 8:30 tomorrow in this room. So I guess 20 that's it. Now it's my pleasure to recognize 21 Guy Caruso, Administrator of the Energy 22 Information Administration. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 13 1 MR. CARUSO: Thank you Jay and good 2 morning everyone and special welcome to ASA 3 committee members and I want to first start 4 by telling you how much we appreciate the 5 work that you have done and now my fifth 6 meeting since I've been here and I think 7 we've found this advice extremely valuable, 8 and as Jay said today's agenda I think shows 9 the kind of specific linkage to the work that 10 we've been doing that I'll talk a little bit 11 about and how important it can be and in the 12 work of EIA and in the I think the accuracy 13 and the timeliness of our information and the 14 analysis. 15 So thank you once again and I will 16 say a little bit more about Jay specifically, 17 at the end, because as you known this is his 18 last session as the chair and a so. Let me 19 start by talking a little but about what's 20 been going on at EIA since our last meeting 21 in spring. As you know every year EIA is 22 asked to do winter fuels outlook and this BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 14 1 year the attention on the outlook for this 2 winter was particularly acute with rising oil 3 and gas prices and it fairly is -- as we'll 4 talk about the interest not only in market 5 but on the hill and the public has been 6 higher than ever. 7 We're going to talk a bit about 8 confidentiality. I know this committee has 9 had a number of briefings on by Jay and Nancy 10 in the past and that's continuing to play out 11 with respect to the statistical agencies and 12 guidelines from OMB etcetera as well as more 13 general data access that we talked about in 14 this committee. 15 Budgeting is always an issue, of 16 course to be able to achieve the results we 17 hope to and I'll bring you up to date on 18 where we are and the fiscal year 2005 budget 19 and we've reached the committee last meeting 20 on our strategic plan which we came to 21 agreement on as organization and individual 22 offices since we last met and bring you up to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 15 1 date a little bit on the progress towards 2 that. 3 Winter fuels as I mentioned you 4 know, enormous interest not only because of 5 the type of markets both in oil and gas but 6 going into a political -- a presidential 7 election there's even more interest in what 8 we do than ever. As we do our monthly 9 forecasting and put out our weekly data these 10 are all areas, which this committee has 11 directly or indirectly had some influence 12 over and will continue to. 13 In today's meeting we're going to 14 talk about some of the surveys that feed into 15 this Short term energy outlook and 16 specifically some of the results and the 17 regional modeling of this short term energy 18 outlook and the customer survey of the short 19 term energy outlook which we'll be talking 20 about in fact in the next session of the 21 breakout session. 22 That just shows in this slide the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 16 1 specific sessions we'll be having today and 2 tomorrow that feed directly into the 3 methodology and results of our short term 4 energy outlook and so once again a very 5 topical very timely, very much appreciated 6 the advice we've gotten from you on our 7 survey's and on our frames and in particular 8 on the -- for the modeling of the short term 9 energy outlook which continues to be enhanced 10 particularly on the regionalization aspects 11 of that. Dave Costello will be here to talk 12 about that. I think that's later on today or 13 tomorrow. 14 Another one which really started 15 two years ago was our desire to more directly 16 collect monthly natural gas production data 17 and John Wood and his people in Dallas have 18 been working with this committee for, in fact 19 from the first meeting that I've been here -- 20 Five meetings now though and finally we have 21 -- we hope some good news and that is that 22 OMB has approved the natural gas production BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 17 1 survey forum 914 in Iowa. 2 There'll be a session later on 3 today that John will be talking to you about 4 it -- specifically about what we're doing on 5 that survey and what we hope to achieve and I 6 can assure you that the work that this 7 committee has done has directly led to the 8 development of that survey and we were hoping 9 for more timely and more accurate natural gas 10 production data, that's our goal and I know 11 that the winters of '02-'03 and '03-'04 are 12 when natural gas prices spike so much there 13 was an increase in political concern over 14 natural gas and this will be, we hope, an 15 important response from EIA. 16 On the budget side the congress 17 went on recess without having passed the 18 budget in our case, an interior budget, where 19 our budget is an interior committee. 20 So we're on a continuing resolution 21 for fiscal year '05 at the same spending 22 level of '04 which is always a challenge BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 18 1 because we were expecting a 5% increase going 2 from 81 million to 85 million this year. So 3 we're having to operate at a lower pace of 4 spending than we would've liked and that 5 directly affects some of the things including 6 the natural gas survey that I just mentioned 7 but, we will actually have a session led by 8 Howard Gruenspecht tomorrow morning to talk 9 about that and get your advice on operating 10 in this kind of tight budget scenario. 11 CIPSEA as you have been -- as this 12 has been on the agenda a number of times 13 here, continues to be in effect and we are 14 now working with OMB on the specific 15 guidelines for the agencies like our own, 16 like EIA, who are not directly, mentioned in 17 CIPSEA, but really come under it and we're 18 working very hard to tighten up security of 19 the data and we are doing things like 20 training and other areas of -- to basically 21 to improve the security of our data. One of 22 the areas we've talked about in this BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 19 1 committee is granting access to researchers 2 and we've started a inter- office team to 3 look into this issue, Nancy Kirkendall is 4 leading that and we've also have had meetings 5 with a number of other statistical agencies 6 and work with them on getting advice and 7 ideas on how they do it and some of the 8 agencies are represented here, that we work 9 with -- on that. We will continue to work -- 10 to come up with a way that we think can 11 satisfy the needs of researchers but also 12 meet the requirements under CIPSEA and other 13 confidentiality pledges. 14 The last meeting we talked a little 15 bit about how sensitive some of the EIA data 16 has become with respect to the financial 17 institutions and markets in general and it's 18 -- I think it's interest is heightened more 19 since then because several financial 20 instruments have been launched on NYMEX and 21 other for which EIA data is directly being 22 used for either derivatives or auctions and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 20 1 therefore when our weekly data for either oil 2 or natural gas comes out, Wednesday or 3 Thursday morning, there's considerable amount 4 of writing on it and therefore a lot more 5 interest there than there's ever been, by the 6 media, by the financial houses and the 7 public. 8 So this sort of feeds back to the 9 issue of confidentiality and need to even 10 have greater security of our data, which you 11 know I think EIA has not been subject to the 12 kind of scrutiny that other statistical 13 agencies that produce the principal economic 14 indicators such as Census BLA and BLS have 15 dealt with this for decades whereas it's kind 16 of new for EIA and we're, again working with 17 all of those agencies to work on data 18 safeguards and I think that this slide really 19 speaks to that. 20 Certainly we need to prevent the 21 unauthorized release of data and we're 22 finding that the scrutiny is when there are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 21 1 billions, or certainly hundreds of millions 2 of dollars riding on what that number is each 3 week. We're finding consequences that is 4 very, very close scrutiny by as I mentioned, 5 financial institutions but also the media, 6 which uses our data to inform their 7 customers. 8 One of the technical issues which 9 isn't directly an issue for this committee is 10 that our website is being utilized 11 intensively during those few moments just 12 when the data is to be released at 10:30 on 13 Wednesdays and Thursday mornings and many of 14 the consumers are using robots to -- the term 15 of it is pinging our system, hundreds and 16 thousands of times per second and that has 17 created a bandwidth problem for us in terms 18 of speed at which our customers can acquire 19 the data and so we're working on that as well 20 but that's really more of an IT issue but 21 clearly one of the consequences of this 22 heightened interest in EIAs data. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 22 1 The other thing, Jay Edmonds isn't 2 here, he's had a close interest in the 3 greenhouse gas collection -- data collection 4 issue, that is the guidelines for collecting 5 emissions data, voluntary reporting of 6 Greenhouse Gases has moved forward and is 7 close to being finalized at OMB for final 8 approval right now, but it will have a direct 9 impact on EIA because we will be required to 10 collect a great deal more of this greenhouse 11 gas emission data than under the current 12 system. Talked about the strategic plan 13 goals last time. 14 These are the 3 main goals and 15 there are a number of sub-elements under this 16 and perhaps the quality of data is one that 17 this committee has dealt mostly with and 18 that's -- the next slide talks about some of 19 the things that the -- this committee has 20 done on the survey quality assessments and 21 then again the issue of the quality of our 22 analysis and the sufficiency of the frames BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 23 1 which we'll talk about directly this morning. 2 And the 3rd goal, which is, more of 3 having to do with the motivation and the 4 leadership quality of EIA, motivation of the 5 staff and productivity. We have -- there 6 have been a couple of developments one, we 7 did a survey, an organizational assessments 8 survey here in EIA in May and June of this 9 year and Howard and I are very pleased with 10 the results which are shown in this slide, 11 comparatively we've improved in terms of the 12 staff's agreeing or strongly agreeing with 13 the statement that the management provides 14 quality and continues improvement by creating 15 and modeling in the organization of vision 16 values and quality. 17 I think you know -- this is fairly 18 very positive development and we know we can 19 still do better and we're looking for some of 20 the issues, that we're going to get on our 21 agenda -- today -- again to lead to further 22 improvements in those areas, in both of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 24 1 goals, one and goal three. The other thing 2 on goal one is we did a -- we participated in 3 American Customer Satisfaction Index surveys. 4 It's called ACSI on our website and for the 5 first time I believe. Is that accurate 6 Colleen? is that the first time we've ever 7 participated in ACSI? 8 MS. BLESSING: No we did it -- we 9 participated in 2000. 10 MR. CARUSO: 2000, okay. But we 11 hadn't done it for several years and we did 12 it again this year. And the ACSI score of 13 72, which we thought was very very pleased. 14 It was a bit higher than the Government 15 average and -- but I think as important were 16 some of the areas where we thought that we 17 could use to improve our website, which is 18 becoming increasingly used and in just the 2 19 years or so that I've been here. 20 And what we've done about improving 21 the website is to appoint a web champion, 22 John Weiner and John is taking that task on BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 25 1 and we're certainly looking at ways on 2 improving the navigation and other issues 3 that the ACSI survey pointed out and 4 certainly it will be coming to this committee 5 in future to ask for further assistance and 6 advice from you. 7 So that's a very quick run through 8 of some of the things that we've been doing 9 here at EIA and the events that have been 10 driving our agenda here in the last six 11 months and looking forward to again getting 12 the advice and guidance from this committee 13 during the next day and half. So thanks once 14 again Jay, I'd be happy to answer any 15 questions now or at any time during the next 16 two days. 17 CHAIRMAN BREIDT: Okay, thank you. 18 MR. CARUSO: Are there any 19 questions? 20 CHAIRMAN BREIDT: What is a web 21 champion, is that like a drug czar? 22 MR. CARUSO: Yeah, we have a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 26 1 committee within EIA, from each of the 2 offices that looks on their specific products 3 that goes on the web and in effect John is 4 going to be the coordinator of that group. 5 But it sort of, yeah it's kind of more of, 6 we're trying to give it a little more, let's 7 say imprimature, more like a czar you're 8 right about that. You know, if there aren't 9 any questions let me take this moment this 10 moment to more formally thank Jay Breidt who 11 has been the chairman since last year -- how 12 long has it been? 13 CHAIRMAN BREIDT: Two years. 14 MR. CARUSO: Two years. That went 15 by fast, and on the committee itself for six 16 years. I have a letter from Secretary 17 Abraham, which I'll read and then Nancy has a 18 certificate, which she'll present to you. 19 "Dear Dr. Breidt, Thank you for 20 your outstanding service as committee chair 21 and as a member of the American Statistical 22 Association Committee on energy statistics. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 27 1 You have contributed substantially to the 2 effectiveness of the committee during the 3 past six years. Your leadership in guiding 4 the committee and you're continuing examples 5 of vision, encouragement and sound judgment 6 have kept the committee productive and on 7 course. Your insightful comments and 8 technical discussions have helped the Energy 9 Information Administration in many technical 10 areas during the past 6 years. 11 In particular we appreciate your 12 advice on selecting estimation procedures 13 using relative standard errors, editing and 14 evaluating EIA survey testing and methods. 15 We thank you for your vision and help, and 16 hope that we may continue to call upon you 17 from time to time for advice and assistance. 18 Sincerely, Spencer Abraham, Secretary of 19 Energy." Thanks Jay. 20 MS. KIRKENDALL: And before you 21 disappear we also have a certificate and this 22 is from Guy and myself thanking you for your BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 28 1 work with the committee. Oh I guess I'm 2 supposed to present next. 3 CHAIRMAN BREIDT: So it's my 4 pleasure to introduce Nancy Kirkendall. 5 MS. KIRKENDALL: Okay well Guy went 6 through updates on a lot of the activities 7 that we had in the last meeting and I'm going 8 to sweep up with just a few additional ones. 9 There are 4 topics that I'll talk about. 10 Electric power, the revision policy and 11 survey testing methods and then estimation of 12 stock of other oils. So on electric power we 13 had a big session in the last meeting where 14 Doug talked about transmission papers, he'll 15 be here later, that paper is -- I think it's 16 been approved by Howard, it's about ready to 17 come out of the front office and they're 18 trying to collect names of people who would 19 like to have a hard copy. 20 It is going to be posted on the web 21 so you can always look at it there. But if 22 you're interested in a copy of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 29 1 publication let us know and we can make sure 2 that you get one. 3 We also talked about the focus 4 groups that Howard Bradsher-Fredrick and Phil 5 participated in, that also illuminated our 6 data needs on transmission. And the electric 7 power package, they actually did use the 8 information that we learned from the focus 9 groups and transmission paper. I think your 10 advice was that we needed to do these things 11 a little bit more quickly and coordinate them 12 a little bit better and of course, that's 13 good advice. 14 But it did make an impact on the 15 questionnaire and right now the electric 16 power package, it received a number of 17 questions from the industry, they're not sure 18 they would like to report transmission 19 outages to EIA. And so I guess EIA is in the 20 process of coming up with it's response back 21 to OMB. Of course that was one of the 22 recommendations from Doug's paper. I guess BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 30 1 that was all on that one. 2 All right revision policy -- we 3 talked about revision policy that was Alethia 4 Jennings and Renee Miller. Just to remind 5 you the goal of -- we have a standard that 6 says that we shouldn't revise our data too 7 much and I think that the committee wasn't 8 sure that -- they thought that was 9 particularly important. But we presented our 10 new policy on revising data that we had 11 previously called final, trying to make it 12 more accurate and the collaboration that goes 13 on within EIA to agree on changes like that. 14 And you saw the proposal was fine, 15 it is being implemented now and there've been 16 a number of changes that have been made that 17 way. Stan Freedman and Bob Rutchik talked 18 about their survey testing methods. We just 19 went through what we think are good 20 approaches on testing and we ask for your 21 assessment. Committee recommendations are on 22 the next slide and we're basically trying to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 31 1 follow your advice since then -- next slide 2 -- Stan has presented this paper in Amsterdam 3 and has gotten some good comments on it and 4 approval of the methods and he's also been 5 invited by Lynda Carlson who of course 6 started this whole cognitive testing approach 7 within EIA to conduct a workshop on our 8 methods at her office and she also wants us 9 to work with her going doing cognitive 10 testing and some of their surveys. 11 So there will be some cross- 12 fertilization. She also has some people who 13 have done cognitive testing, so we can learn 14 a little bit, they can learn a little bit, 15 that and might be a good collaboration. 16 We talk about -- the project was to 17 come up with a better way of estimating 18 weekly stocks of other oil. Basically this 19 is -- we have a weekly survey that gives us 20 the inventories of the major products and 21 other oils is sort of the rest, however, we 22 also like to publish it total. So we need an BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 32 1 estimate for the other, so that we can get a 2 good estimate in total. There was a problem 3 last year when stocks -- our estimate rather, 4 oil was pretty bad and so this has been a 5 project to try to do better. 6 The major improvement has been 7 caused by collecting propane data, so now 8 what we're estimating is much smaller than it 9 used to be and that was the thing that had 10 the biggest change during that problem time 11 period. We're still looking at ways of doing 12 a better job, estimating the residual -- next 13 slide -- I think you're recommendation was 14 that we needed to collaborate better because 15 we have many efforts in EIA that look at 16 estimating stocks of other oils including our 17 short term energy outlook folks as well as 18 the petroleum people. 19 So we're still working on this, we 20 have come up with a couple of our best 21 regression based methods and now we're doing 22 some comparisons and now we're working with BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 33 1 the other program officers to see what we can 2 come up with and that is basically what I 3 have as an update. Anybody have questions 4 about anything? Anything either Guy or I 5 forgot to talk about that you were 6 particularly remembering as an issue from 7 last time. Okay, thank you. 8 CHAIRMAN BREIDT: Thanks Nancy. 9 Okay we're now ready to move to break out 10 sessions and if the committee members take a 11 look at the break-out sessions. You might 12 notice that if you are not Barb then you 13 always go to the same place. So if you're 14 upstairs you always stay upstairs, if you go 15 downstairs you always stay downstairs, except 16 Barb who is the only exception -- 17 MS. FORSYTH: I need to work out. 18 CHAIRMAN BREIDT: Okay. Last time 19 we had some complicated fractional factorial 20 and people were -- complaining about that, so 21 this is a little simpler. So let's go ahead 22 and reconvene in breakout. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 34 1 (Recess) 2 MR. RUTCHIK: Good morning, I'm Bob 3 Rutchik. I'm of Statistics and Methods Group 4 of EIA. I want to welcome you to the third 5 breakout session. In the past year and a 6 half that we've had on EIA 920 combined heat 7 and power plant reports. The previous two 8 sessions were in April of last year and that 9 was on the EIA 920 pre-survey design visits. 10 In last October a year ago we had 11 another session on proposed respondent level 12 and aggregate level edits for the EIA-920. 13 The purpose of this session is to get your 14 feedback, -- the committee's feedback on 15 EIA's plan to evaluate the effectiveness of 16 the form EIA-920 which has been collecting 17 data since January of this year. We picked 18 the EIA-920 to apply this program to, that 19 I'll be discussing, because we at EIA have 20 extensive experience in the survey's design 21 and testing. We are very familiar with it 22 and very comfortable with it. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 35 1 This is the plan on which I'm going 2 to be asking your advice -- for which I'm 3 going to be asking the committees advice. 4 As you can see there are four 5 criteria, which we have proposed to use to 6 evaluate the EIA- 920. These criteria are 7 timeliness of response, ability to provide 8 data, accuracy of response and reduction in 9 processing time for the EIA. And shortly I 10 will go over measures to operationalize each 11 of these four criteria and I'll also be 12 asking your advice on the measures to 13 operationalize each. We also hope to use 14 this proposal as a template to evaluate other 15 surveys, which EIA has applied it's survey 16 design and testing program to do something 17 that Nancy talked about earlier. 18 So EIA will not just have a testing 19 and design program, we will have a testing, 20 design and evaluation program. Now before I 21 get to the measures for each of the criteria, 22 I just want to review the EIA-920 with you. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 36 1 The EIA-920 combines a heat and power plant 2 report, collects data from combined heat and 3 power plant facilities such as, papers mills, 4 refineries, steel mills and food processing 5 plants and so on. We collect data on four 6 variables. One is total fuel consumed by the 7 CHP facility, the second is fuel consumed to 8 generate electricity in that CHP facility, 9 third is electrical generation and fourth is 10 fossil fuel stock, coal and distillate oil. 11 The major focus of the survey is 12 fuel consumed to generate electricity. If 13 the respondent has understood the survey, the 14 EIA-920, the reported quantity for fuel 15 consumed to generate electricity should be 16 less than total fuel consumed because, 17 basically the CHP plant should be using it's 18 fuel for two purposes. One, to generate 19 electricity and the other for it's 20 manufacturing process, such as drying paper. 21 Two other things about the EIA-920 22 that I just forgot to mention is that it is a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 37 1 monthly form of a sample selected from all 2 CHP facilities and I think there are probably 3 about 600 -- and that could be wrong -- CHP 4 facilities in this country. There's also an 5 annual form that's completed by CHP 6 facilities that are not selected to respond 7 monthly. Okay, now into the variables that 8 we plan to use, or hope to use - - to 9 operationalize the four criteria that I 10 presented earlier. The first criteria, is 11 timeliness and under timeliness we want to 12 measure that or evaluate that by more 13 respondents reporting within the reporting 14 period. 15 What I mean by more respondents is 16 that before the EIA-920, EIA used to collect 17 data from CHP plants on the form EIA-906, 18 which is power plant reports, now that 19 EIA-906 collects from regulated utilities, so 20 add more respondents to the EIA-906, and two 21 fewer respondents getting reporting 22 extensions, EIA would get phone calls from BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 38 1 respondents saying, no I can't fill this up 2 by the 15th of the month can you give me an 3 extra 5 days or 10 days or so on. And also 4 I've been using the word "we" a lot here, I 5 think that's a good sign to mention that on 6 this project I'm working with my colleagues 7 Tom Broene, Stan Freedman, Guy Caruso and 8 Karen Norman, all of Statistics and Methods 9 Group. 10 The 2nd criteria that -- in our 11 plan is ability to provide data and we have a 12 number of measures that we plan to use to 13 evaluate that. One is response rate, and 14 again, the response rates -- you know, higher 15 than let's say then was on the EIA-906, 16 that's just one example. 17 Now there is reporting errors, 18 pardon me. Let's say that there's a problem 19 with a particular variable -- all right, in 20 the EIA-920, let's say that instead of 21 recording just fossil fuel stocks such as 22 coal -- you know, plants are recording BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 39 1 something in storage that is not a fossil 2 fuel. So there may be a problem with the 3 instructions or the survey itself not being 4 clear. The 3rd thing we have is analysis of 5 comments, you'll notice at the bottom of the 6 first page of the EIA-920 the section of 7 comments from respondents, so we're going to 8 analyze those respondents and also their 9 comments that from EIA staff, that is put in 10 the file for the EIA-920 so we're going to 11 look at that too, and also down the line 12 after we've -- you know, looked at response 13 rate, reporting errors and all sorts of 14 comments and other things we're going to talk 15 to the EIA-920 processing staff. 16 We're going to ask them a lot of 17 questions but main -- basically things like, 18 what you think is going right with this 19 survey, what do you think the problem area 20 is, what's the bulk of your pullbacks on, 21 what's the bulk of questions that you get 22 from respondents, what can we do to improve BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 40 1 the survey and so forth, a lot of things. 2 And then, after that probably one of the last 3 things we're going to do is, we're going to 4 follow-up on respondents and we're going to 5 ask them question as basic as you think this 6 survey is an improvement over the survey 7 EIA-906, do you understand what EIA wants in 8 question 2, that's a key question there, fuel 9 used to generate electricity and several 10 other questions. 11 Okay we have, two other criteria 12 and the 3rd one is accuracy of response. One 13 thing we're going to do statistically is 14 compare data on the EIA-920 to data collected 15 on the EIA-906, the power plant report. This 16 is a little bit of a pitfall if it's here, 17 because on the EIA-906, EIA collected data 18 from CHP plants for something called useful 19 thermal output, which is a fuel used for 20 manufacturing and other processes and combine 21 heat and power plants. 22 From useful thermal EIA then BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 41 1 calculated fuel used to generate electricity, 2 as the difference between useful thermal 3 output and total fuel consumed. It was EIAs 4 wariness about the accuracy of use of thermal 5 output in the calculated fuel used to 6 generate electricity now but that led to the 7 design and implementation of EIA-920, so the 8 comparison there is a little bit fraught as 9 one of my colleagues suggested yesterday. 10 Two, we're going to try to do this 11 also this act by comparison with sub groups 12 in the EIA-920 by prime mover steam turbine 13 configuration. There's different type of 14 turbines, there's steam turbines, gas 15 turbines and other types of setups. And also 16 by industry is there a difference between 17 paper mills versus food processors, or 18 plastic manufacturers versus refineries and 19 so forth. And finally, our final criteria in 20 the measures we've have for that, is 21 reduction in processing time for EIA. 22 We're going to look at -- you know, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 42 1 has there been a decrease in respondent call 2 backs and in other words that would be kind 3 of an indication if there's been a decrease 4 in call backs that respondents are having -- 5 are understanding the survey and in other 6 words they're filling in more items than, 7 let's say the EIA-906 and this will also 8 indicate that maybe the EIA is saving 9 resources on the EIA-920. 10 MR. RUTCHIK: So those are 4 11 criteria and on these criteria we just listed 12 out, we have measures we want to use 13 operationalize that criterion. We've only 14 done research so far on ability to provide 15 data and as I said there is preliminary 16 analysis and that's a very good phrase. We 17 looked at January to May data. We have 18 looked at about 140 respondents of the 19 approximately 300 responses that EIA got for 20 that period. We looked at respondent in the 21 EIA comments, then we compared the question. 22 One, total fuel consumption data to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 43 1 question two data, that's data -- fuel use to 2 generate electricity data. The comments 3 provided very little systematic information. 4 There is nothing in the comments that just 5 leaked out at you to say, this is going right 6 with the survey, this is going wrong, and 7 kind of all over the board and illuminated 8 very little. 9 The question -- two to question one 10 comparison though did yield to this breakdown 11 of the 140 or so responses that we looked 12 for. Seventy four respondents recorded a 13 fuel use to generate electricity at the CHP 14 plant was less then total consumption, in 15 other words, these people seem to have got 16 the basic concept of the survey. There were 17 six to eight respondents that did not seem to 18 get the concept of the survey. For most of 19 these, they report their fuel use to generate 20 electricity was equally to their total fuel 21 consumption. 22 There were some that did not report BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 44 1 consumption at all just before -- no data at 2 all, just had some other problems with their 3 responses. There is also some other 4 findings. I don't want to put findings in 5 quote. From our analysis a look at these 140 6 responses. 7 One is on waste heat fuel 8 consumption not reported. Let me explain 9 waste heat is. In combined heat in power 10 plants, some of them have what's called 11 combined cycle technology. And you are 12 getting the full brunt of my engineering 13 knowledge with this. Combined cycle 14 technology, in that fuel is, let's take coal, 15 is put into in a boiler churns(?) out turbine 16 electricity is generated. Then a process of 17 generating electricity, steam is produced, at 18 that time the plant CHP facility can do one 19 or two things with that steam, they can admit 20 it into the atmosphere. 21 Hence the phrase "waste heat," they 22 don't need it, they are not going to use it, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 45 1 it's waste, or they can recycle the steam and 2 use it, let's say as part of their 3 manufacturing processes, you know, to 4 manufacture the candy or again use my 5 favorite one, drive the paper in the paper 6 mill. Some are designed as this cognitive of 7 any views, we knew that the plants would have 8 problems reporting fuel consumption for waste 9 heat. 10 These are waste heat, we really do 11 not consume fuel for waste heating because 12 the production has already been done at the 13 first part of the cycle, so therefore they 14 would report zero fuel consumption for waste 15 heat, and that's what happened when we look 16 at the responses for the combined cycle 17 points in the 140 or so that we looked at, I 18 think we did about maybe about a dozen 19 combined cycle facilities of the 140 and none 20 of them I think reported any fuel consumed 21 with a waste heat. 22 The second kind of finding was a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 46 1 larger respondents appeared to report 2 correctly, what I mean by this is unless (?) 3 we have, let's say a large food processing 4 company or a large paper company and they 5 have eight facilities reporting eight or nine 6 -- 8 separate responses and as I looked at 7 them and Stan looked at them we see that 8 let's say six reported correctly, they 9 reported their fuel use to generate 10 electricity with less then total fuel. 11 Two, total fuel was equal to fuel 12 use to generate electricity. So three 13 quarters of the people in that company saying 14 that they got the idea of the survey, and it 15 also shows that this decentralized reporting 16 you know, the reports are coming to EIA or 17 EIA-920, it is not coming from the EIA from 18 the central headquarters, but from the 19 individual facilities. 20 And the final two things is that 21 our pre-survey design respondents reported 22 correctly in the 140 or so, there were about BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 47 1 four pre-survey design people that we have 2 visited and talked to and our education 3 efforts seemed to work, they seemed to get 4 the hang of the survey. There were about 6 5 or 7 cognitive respondents in this group, I 6 think was half and half on whether they got 7 the idea of survey or did not. 8 Now I can come up with a facile of 9 rationale for why it was this way with the 10 cognitive respondents versus the pre-survey 11 design respondents and I will. In the 12 cognitive respondents, we let's say visited 13 you know, just once facility out of let's say 14 3 or 4 of that company and that one facility 15 filled it out correctly but the others that 16 we didn't visit would not fill it out 17 correctly, so again direct education may 18 work. 19 Okay, so that's the only research 20 we've done on this so far. And these are the 21 questions I have for the committee. Are the 22 criteria, the full criteria that EAI has BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 48 1 proposed for the evaluation at the ENI-20, 2 are these fully valid? Are the other 3 criteria that EIA can use? And finally, are 4 the measures to operationalize each of the 5 four criteria -- are there criteria -- are 6 there measures other than the ones we 7 proposed that we can use still? 8 In other words, is there something 9 under ability to provide data that we have 10 not listed that we can use. Again to sum up 11 you know, the thrust of the question, is EIA 12 on the right track to assessing survey 13 effectiveness? Thank you very much and open 14 it up for the committee. Nicolas? 15 DR. HENGARTNER: I have a question 16 that is not addressing any of these issues 17 but I'm really thankful for you to have 18 provided that test form because the first 19 time I see what you were talking about, and 20 that I'm reading at the top and the sentence 21 says, report actual value. 22 MR. RUTCHIK: Very good. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 49 1 DR. HENGARTNER: These are not 2 available, report estimates the value, now if 3 I go down on the phone I see nowhere an 4 indication that the value you are collecting 5 are either real or estimate. That is an 6 issue. So that goes to the accuracy and the 7 ability to answer because if we don't know if 8 the numbers that we are getting are real or 9 estimate, then we have a problem. And there 10 is also no ability to correct as the fact 11 that you have an estimate as to make the 12 value in January and you get the real numbers 13 in February do you resubmit the form? What's 14 happening? 15 MR. RUTCHIK: Okay, this may not 16 fully answer your question, and that's 17 something I left out. When we did a 18 pre-survey design as it is in, they were not 19 cognitive interviews, the respondents told us 20 that it could measure directly fuel used to 21 generate electricity. It goes into the right 22 nature of the configuration of the CHP BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 50 1 plants. 2 But he said, through I'm backing 3 out of steam pressures and other things, that 4 they could provide us with an estimation. I 5 know this direct -- addressed it directly, 6 only question two but that's what they told 7 that stuff. What we were getting most of the 8 time in question two, if you used to generate 9 electricity are, you know, estimates, 10 evaluations done by the facility themselves 11 and as a matter of fact this was the just as 12 the talk of the first session we had on this 13 in April 2003. So I don't know that really 14 answered your question but these are some 15 questions -- calculations are right. 16 MR. BLAIR: May I follow up -- 17 MR. RUTCHIK: Yes, sure. 18 MR. BLAIR: To the next question 19 and that is, do you know from talking with 20 the respondents sort of what procedures they 21 use for estimation and do you agree with how 22 they are coming up with their estimates? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 51 1 MR. RUTCHIK: No, not at this 2 point, I don't know. But that would be a 3 good idea to use if, you know, we do 4 follow-ups. 5 MR. BLAIR: I think I mean just 6 about the second letter. If you are going to 7 continue these visits in what you described 8 as education that if you know sort of how 9 people are coming up with these estimates 10 that that could be some part of what you 11 continue to collect the information on and 12 also maybe build into part of your education 13 networks. 14 MR. RUTCHIK: But it is fun, I 15 think it's more basic one that you want to 16 know sort of which way it is estimated -- 17 DR. HENGARTNER: And the danger now 18 is that using question two, those estimated 19 value or how sophisticated, they already 20 estimated that as - - evaluated through a 21 survey by saying, well, hold the consumption 22 and the electrical estimate and consumption BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 52 1 producing -- well, not asking the price, they 2 could be the same. Well, you have to be a 3 little careful using those values to evaluate 4 how well the respondents are understanding 5 the form. As you are measuring out the 6 specification of the respondents, we need to 7 have as you say, calculating the pressure and 8 we are getting a specific way beyond 9 cognitive. 10 MR. RUTCHIK: Yeah. Anybody else 11 has anything they want to ask? John, I'm 12 sorry, John. 13 MR. BLAIR: I was just going to 14 come back to the criteria number, response 15 rate which is obviously important but 16 describe sort of separately and I wonder if 17 you are putting together sort of the unit 18 response rate and the item response rate, so 19 that for each of the variable each of the 20 questions that you can compute an individual 21 rate which is the unit response rate times 22 the item response rate. And that is what BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 53 1 really tells you I think what you have for 2 each of your data for instance. 3 MR. RUTCHIK: Good idea. I think 4 you should -- 5 MS. WAUGH: I'm looking at -- is 6 this on? 7 MR. RUTCHIK: Yeah. 8 SPEAKER: Can you tell me who you 9 are? 10 MS. WAUGH: Shawna Waugh from EAI, 11 I'm looking at some of the theory if you will 12 about survey talks about three factors, 13 timing, accuracy and cost. And it appears 14 that the EIA processing time might be related 15 to cost but I didn't actually see any 16 variable that would evaluate cost 17 effectiveness and that might be something to 18 also attempt operation on. 19 MR. RUTCHIK: Thank you. Okay is 20 there anybody else have any comments, 21 suggestions because all we got so far have 22 been very, very helpful. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 54 1 MR. BLAIR: Yes, maybe if you go 2 back for a bit to -- you talked about the 3 comments that you receive -- 4 MR. RUTCHIK: Yeah. 5 MR. BLAIR: And they were kind of 6 all over the place and there was no pattern. 7 I guess sort of two things and one is if I 8 get comments all over the place and the 9 comments seem sensible that worries me a bit 10 because, it sound like then that are 11 problems. 12 MR. RUTCHIK: Well, what I meant by 13 that I probably wasn't very clear, there 14 didn't seem to be any great mass or number of 15 comments on any one thing. I think I did 16 something where I did like a frequency chart, 17 a bar chart of frequency. You know, when I 18 did various categories and have two or three 19 on this, two or three on that, three of four 20 on this, three or four on that, there wasn't 21 something like let say 140 respondents and 22 not all those 140 -- well actually I looked BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 55 1 at the whole 300 actually, not everybody 2 makes a comment, so those who made comments, 3 there wasn't somehow but say 15 or 20 or 25 4 saying that we don't understand question two. 5 MR. BLAIR: Right. 6 MR. RUTCHIK: That's what I meant. 7 MR. BLAIR: Sure, but you also 8 suggest that you did some sort at least 9 informal coding of the comments in order to 10 -- 11 MR. RUTCHIK: Yeah, informal would 12 be a good word here. 13 MR. BLAIR: Okay, if now that you 14 are getting that was it 68 out 140 or 15 something, 142 that provided comments? 16 MR. RUTCHIK: No, I don't know how 17 many provided comments, we didn't count that 18 up. 19 MR. BLAIR: Oh that was different 20 -- 21 MR. RUTCHIK: That was a -- 22 MR. BLAIR: People who didn't get BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 56 1 it. 2 MR. RUTCHIK: Yeah, there was a 3 question one versus two comparison. We 4 didn't really count up how many people, of 5 responses -- respondents provided comments. 6 MR. BLAIR: Oh, okay, well and this 7 is kind of stupid(?) and you certainly want 8 to count that and depending on how many that 9 we have may be able to do some more formal 10 coding and track over time you know if there 11 are some differences but, how useful that is 12 depends on sort of how many comments you got 13 and if you got five comments you know what 14 can you do, but if you had 50 you know that's 15 something, something else. So if you have 16 some substantial numbers then I would you 17 know, look at developing some sort of a 18 coding for it. 19 MR. HOUGH: My name is Rick Hough 20 from Census Bureau, you guys investigated the 21 use of electronic reporting for your response 22 related issues? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 57 1 MR. RUTCHIK: I wish somebody here 2 from the program office was here, but I think 3 the survey is reported electronically. It's 4 not a mail in survey, respondents go to a web 5 site, they access the survey, they fill it 6 out and send it back electronically, I mean 7 that's a very welcome and a clear way of 8 explaining it but is it on an electronic 9 reporting system. 10 MR. HOUGH: Okay. 11 MR. BLAIR: I don't want to drag 12 this out, but, a question on the expert 13 review. You mentioned that the survey staff 14 with the people and I wonder if there might 15 be a larger, of course I'm actually not sure 16 what the survey says but includes and that 17 includes the people that are responsible for 18 EEO, the processing or people analysis with 19 the data you know for the users but you might 20 consider sort of to know exactly who is that 21 that are appropriate for this expert 22 revealing, maybe something part of you should BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 58 1 make it different information from the 2 processing staff, that you get from the data 3 collecting staff. 4 MR. RUTCHIK: Anybody else, 5 anything you wish to add. I want to thank 6 you first for participating and listening and 7 if you don't have anymore comments and if you 8 do please come forward I just want to thank 9 you because of the comments we got were very, 10 very clear in substance. Thank you. 11 SPEAKER: Good job. 12 CHAIRMAN BREIDT: Okay, I guess we 13 will get started with the summary of the 14 break out session. So first Johnny Blair 15 will summarize the proposal for announces of 16 collected data break out issues. 17 MR. BLAIR: EIA has begun 18 evaluation of redesign form 920 and in our 19 session we are basically commenting on the 20 evaluation plan and essentially comments were 21 very positive. There were three questions 22 that were posed to the committee, the first BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 59 1 one was, are the evaluation criteria the 2 right ones? There were four general criteria 3 which are timeliness of response to the 4 survey, ability to provide the data that is 5 requested, the accuracy of response, the 6 accuracy of those data and reduction in 7 processing time for the EIA. 8 And in general we thought yes, 9 those were appropriate criteria but we had 10 some comments, which I will come to later on 11 how to apply them. The second question to 12 the committee was, are there other criteria 13 that EIA can use and there were two points 14 that were raised here, the first which was 15 raised actually by someone in the audience 16 was that cost is not explicitly measured in 17 the criteria. There are some indirect 18 measures such as processing time, but we felt 19 that more could be done to track cost 20 directly in the track changing cost due to 21 the new form. 22 The second is that the form has on BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 60 1 page 2 interesting instruction, which is 2 report actual values if these are not 3 available, we put estimated values. And a 4 comment on this was that it would be useful 5 to know item by item for each respondent 6 which items are actual and which items are 7 estimated and so the suggestion was to make a 8 minor modification in the forms so that 9 respondents can indicate when they are 10 putting in estimated values and more over in 11 the follow-up with respondents which is also 12 a part of the evaluation plan to look at how 13 those estimates are arrived at, how are the 14 estimates being constructed and whether EIA 15 agrees in general with the estimation 16 procedures and if not what suggestion that 17 they can make to improve them and as part of 18 their education, ongoing education of 19 respondents, what can they do to improve 20 those estimates. 21 The third question to the committee 22 was what, what measures can be use to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 61 1 operationalize the criteria and just a couple 2 of comments here, one connected to -- related 3 to the expert review which is one procedure 4 that used right now. There is an expert 5 review by the survey staff and we suggested 6 that that might be expanded to include other 7 people such as the processing staff, data 8 analysts and more broadly other users to 9 participate in that expert review since 10 expert reviews are generally the low cost and 11 quick way to get response. 12 There should be really marginal 13 additional effort to expand the pool of 14 experts, they do their review. And finally 15 on that response rate which is obviously an 16 important criterion, really unit response 17 rate was focused on and some mention of item 18 response rate but the suggestion was that 19 item by item is question by question of that 20 the product that goes is really what one 21 should be looking at, the product of unit and 22 item response rate for each question to see BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 62 1 sort of what you're really getting at the 2 question level. 3 And those are the notes that I have 4 which actually probably took as long as our 5 session did, which very quickly but if any of 6 the other committee members want to correct 7 or add anything to what I've said, now is the 8 time. 9 CHAIRMAN BREIDT: Okay any comments 10 or question for Johnny? Okay thanks, Johnny. 11 And now I will summarize the secondary break 12 out session which is a customary evaluation 13 of the Short Term Energy Outlook STEO, this 14 is based on a random sample of 500 email 15 addresses from list of about 4,000 email 16 addresses. These are self-selected, it's a 17 list -- so these are people who have chosen 18 to repeat new updates monthly and these are 19 thought to be fairly representative of the 20 core users of the STEO so. 21 From that email list, you have a 22 logistically straightforward means of BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 63 1 sampling and there is a lot of automation 2 that can be built into it which is a nice 3 feature. The run for about 25% unit response 4 rate and one of the questions had to do with 5 response rate. Howard Bradsher-Fredrick 6 described the results of the survey, 7 generally showing pretty much satisfaction 8 with the STEO, it's fairly consistent for 9 results for other products particularly the 10 Annual Energy Outlook and the International 11 Energy Outlook and you might recall from the 12 April meeting those were reported on the 13 basis of this sample, the satisfaction 14 survey, in that case was a survey of NEMS 15 conference register, is that right. 16 So let's see, one of the questions 17 on the survey that was emailed to people was 18 whether or not that respondent would agree to 19 follow up for more detailed questions, so 20 again there is stage of self-selection. You 21 have the self-selected list of -- you have 22 those who agree to participate in the survey BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 64 1 and then further you have further self- 2 selection on those who might be willing to 3 follow it up. And still with all this 4 self-selection going on, you get something 5 that is not necessarily representative of all 6 STEO users uses or even necessarily of all 7 STEO core users but probably something useful 8 in terms of customer satisfaction. You get a 9 useful first look. 10 The questions for the committee 11 were, how often should customer surveys of 12 this type be conducted? Would once a year be 13 too frequent or not and is the response rate 14 adequate and then questions on other 15 suggestions. So on the first question, how 16 often should customer surveys of this type be 17 administered? The committee I think was in 18 general agreement, you didn't want to do it 19 every year sending out to everyone that would 20 be unduly annoying, so it was thought that 21 one thing you could do is to implement some 22 kind of rotating panel where you might BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 65 1 revisit some people so that you can track 2 change over time if there were changes that 3 were of interest, you do want to revisit some 4 of the same folks and a possible selection 5 for part of that core panel could be some of 6 these people who have self selected to hear 7 the survey again. 8 Beyond that you can have a rotating 9 component and since there is 4,000 names on 10 the list and that gets somewhat refreshed 11 over time as people add and drop, you can 12 rotate through the email addresses for the 13 sample as a 500 only over eight years so 14 that's not really unduly burdensome. The 15 second question has to do with the response 16 rate, it was 25.9% and would that be adequate 17 and again spent some what you mean by 18 adequate, this really isn't going to be a 19 representative sample of all STEO users again 20 but it should be a useful sample in terms of 21 trying to get some information on customer 22 satisfaction. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 66 1 So the list service is self 2 selected, the respondents are self selected 3 after four ways of email follow- up and still 4 the thinking is that you should be able to 5 get some useful information from those who 6 want to be heard, those who are actually 7 taking the time to respond to the survey are 8 those who really have an opinion and this 9 should give you useful first look at the kind 10 of those core users. 11 But you get something that's sort 12 of actionable where you might be able to work 13 on the STEO. The performance of STEO you 14 want to go back to those core uses who added 15 strong opinion and actually find out exactly 16 what is it that they have questions or 17 concerns about. And it seems that they do 18 have a number of mechanisms for giving 19 feedback to EIA on this product but through 20 the survey mechanism they do have this thing 21 where they can self select to be followed up 22 and nothing has been done with that yet but BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 67 1 it could be, so that can be very useful to 2 jot down with that follow-up to get some 3 discussion and specifics like model 4 components and things like that. 5 As far as increasing response rate, 6 there are things that you can do but with 7 multiple ways of email follow-up it's not 8 going to do the trick. At some point you 9 again have to change the mode of data 10 collection if you really want to increase the 11 response rate and the other possibility is 12 that, eventually there is going to be these 13 non respondents that are going to respond no 14 matter how many times you contact them to 15 this kind of conventional mode follow-up, you 16 might sub sample them and then make some 17 efforts to get out responses from them, may 18 with a simplified questionnaire. 19 So and maybe giving them some 20 incentive to respond, I don't know. Then the 21 other question was other suggestions and the 22 discussion kind of centered around other BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 68 1 modes of entry, sort of other entry points 2 for data collection and these included some 3 discussion of the American Consumer 4 Satisfaction Index ACSI which is a pop under, 5 must be careful not to say pup up. But pop 6 under methodology that is sitting there on 7 the web page after you close you browsers so 8 it has -- this has caused technical problems 9 where people have a dozen of these open or 10 something and they are irritating and the 11 other concern is that there is a sense of 12 frame bias because the more technological 13 savvy users have blocked all their pop ups 14 and pop unders and everything else. 15 So it's maybe not as useful as 16 other means of getting that data but it's 17 good to have for customer satisfaction not 18 necessarily the case that you need 19 representation, you need to give people the 20 opportunity of have their voices heard in 21 different mechanism, two different 22 mechanisms. So let's see and one of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 69 1 other possible mechanisms discussed was sort 2 of an invitation on the web page where you 3 were invited to participate in survey. Again 4 self-selection but again potentially giving 5 you useful information that could be followed 6 up on. So that was kind of the being focus 7 of our discussion I think. Is there any 8 comments from people who are there to follow 9 up on that question. 10 DR. FEDER: Jay when you said 11 rotating panel, are those panels going to be 12 interviewed more than once? 13 CHAIRMAN BREIDT: Yes -- 14 DR. FEDER: Okay. 15 CHAIRMAN BREIDT: I think of a core 16 panel that would be interviewed year after 17 year and the others would sort of rotate out. 18 DR. FEDER: I have one small 19 question, is that, what about new users is, 20 there any attention going to be paid to new 21 users? 22 CHAIRMAN BREIDT: Well, the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 70 1 interest through the new users who are core 2 users who self select and get in on the list 3 there you know, can get in there but I guess 4 it would be a good idea to try to get those 5 into the core as well, refresh that panel, 6 that didn't really come off -- there is a 7 concern about new users to kind of get a 8 lower level being more frustrated with 9 products and things like that, so it's a 10 definitely interesting sub group. Any other? 11 MS. KHANNA: Yeah, I think on the 12 clarification point. One of the things that 13 we talked about is the (?)N. We are talking 14 about listing this up under -- approach was 15 that -- even that would not get rid of self 16 selection or even the bias in the selection 17 because people who bookmark the pages would 18 not necessarily receive pop under or the 19 invitation, the letter that we sort of talked 20 about -- 21 CHAIRMAN BREIDT: Right. 22 MS. KHANNA: Having invitation but BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 71 1 even there there was going to be some issues 2 of bias even the potential sample that you 3 got -- 4 CHAIRMAN BREIDT: Right. 5 MS. KHANNA: Apart from 6 self-selection. 7 CHAIRMAN BREIDT: And that just 8 reminds me something that -- I think that 9 statistics is something like 10,000 hits on 10 the STEO page and 4,000 users on the lists 11 there and those are different population, 12 they overlap to some extent but this 13 particular survey is really aimed at those 14 core uses who have self selected to 15 participate in this email collection. 16 MR. BLAIR: Yeah, just a comment on 17 the official response rate and customer 18 satisfaction survey that there is some -- 19 there is some evidence in the literature in 20 fact that indicates that that non 21 respondents, excuse me, the respondents to 22 customer satisfaction surveys tend to have a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 72 1 more positive rating than non respondents 2 which seems somewhat kind of imperative but, 3 in fact the explanation or at least that's 4 been positive, is that respondents, our 5 sample members look at doing the survey as 6 kind of a favor for the agency doing the 7 survey, so there is a kind of issue of 8 reciprocity, so if they have a positive 9 feeling towards the agency or whomever is 10 doing the survey, there is someone more 11 likely to respond and those who are not so -- 12 this is something I think that you know, 13 maybe useful to keep in mind when you're 14 dealing with this low response surveys. 15 CHAIRMAN BREIDT: One other thing 16 we came is that -- the earlier respondents 17 were grumpier, so as they went along -- Are 18 there other questions or comments? 19 DR. HENGARTNER: Do you also 20 compile complaints? I mean customer 21 satisfaction you know, you don't want to get 22 people mad. You don't want them to complain, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 73 1 I mean as long as they are grumpy that's all 2 right but as long as they don't complain. 3 MS. KIRKENDALL: There is a comment 4 box on the website, that is people are 5 invited, they send in a comment if they want 6 to. 7 DR. HENGARTNER: And is it 8 reviewed? 9 MS. KIRKENDALL: Colleen wasn't 10 reviewing it every month and sends them out 11 to whoever -- those who have seen it. But I 12 think she does you know I think she said 13 that, that's what she does with it. 14 CHAIRMAN BREIDT: Yes. 15 DR. HENGARTNER: There are always 16 going to keep complaining. The idea is to 17 keep the rate as low as possible or to avoid 18 -- notice most things fight in time and 19 that's one way one could actually monitor the 20 ongoing quality. 21 MS. FORSYTH: For expecting contact 22 as well. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 74 1 CHAIRMAN BREIDT: Yes, because you 2 know, people are going to grumble, grumble, 3 grumble -- and then it goes up and that's 4 when you have to be worried about. So let 5 them have the monitoring of the time, which 6 seems to be not too hard in conception -- 7 MS. KIRKENDALL: Well, it's sort of 8 interesting because I think by telling 9 business every month but I don't know that 10 I've seen that summary that she puts out 11 monthly or anything that she would -- and 12 distributes a number of complaints that she 13 gets over time related to the same. 14 MS. FORSYTH: And it might mean 15 that some complaints that are specific to a 16 particular product or a particular, I don't 17 know -- a page or whatever that hearing about 18 them for another page people who manage other 19 contracts or other pages might be able to 20 manage the content -- the distribution of 21 that at least some kind of content might also 22 be -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 75 1 MS. KIRKENDALL: We should have our 2 new web champion to look at it. 3 MS. FORSYTH: Yeah. 4 CHAIRMAN BREIDT: Nagaraj? 5 DR. NEERCHAL: Well, another point 6 he is going to bring up is that you know, one 7 aspect of customer satisfaction is to letting 8 all the customers know that you are improved 9 -- you are trying to improve and I think one 10 thinks that might be useful is based on the 11 survey or even as of your normal way of doing 12 things, you come up with improvement, send a 13 general email to the list saying that you 14 have improved the service. 15 You know, so because many people 16 who are not responding, they may not even 17 have noticed the problem but the fact that 18 you saw a problem and you have improved it, I 19 think that is already kind of improves your 20 scoring in their book. 21 CHAIRMAN BREIDT: Okay, so we are 22 on time or pretty much on time to go to next BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 76 1 break up session. Can we reconvene there? 2 (Recess) 3 MS. SUTHERLAND: Good morning I'm 4 Grace Sutherland welcome to the session on 5 assessing EIA frames and update on EIA wide 6 projects. This will be an update from the 7 spring meeting as to what we've been doing 8 and where we are headed. For a brief 9 background, in the spring EIA presented to 10 the committee the activity this has been 11 involved in up to that point, it included 12 checking respondent list, comparing aggregate 13 data, examining supply to sufficient balances 14 and comparing price data volumes. 15 In response the committee suggested 16 that EIA ask known establishments to identify 17 others within the same market to calculate 18 propensity scores by post stratifying 19 potentially using census data. To apply the 20 principles of dual system estimation to 21 available frame data, to obtain as much 22 information from census without disclosing BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 77 1 sensitive data in order to identify missing 2 respondents. 3 That EIA might forego using 4 balancing items for measuring coverage -- the 5 balancing items with least favorable message 6 of assessing frames. In response to the 7 committee's suggestion with regard to asking 8 establishments to identify others within 9 their same market, EIA had tried that and 10 respondents have been reluctant to provide 11 EIA with their customer's information. 12 However we have had success in obtaining 13 names of the buyers or competitors. For 14 instance our electric power surveys has used 15 this technique. The EIA is not currently 16 preferring the sampling approach but may 17 consider it again in the future. 18 With regards to fine principles of 19 dual system estimation to available frame 20 data, EIA is in favor of this whenever 21 possible and have applied this principle, for 22 example EIA compared the EEIA electricity BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 78 1 renewable frame with the National Renewable 2 Energy Laboratory frame. We also now have an 3 EIA wide interoffice team with evaluating 4 frame sufficiency. Because of the difficulty 5 in coming up with quantitative assessments 6 for frame sufficiency for all surveys, EI is 7 pursuing a qualitative approach. 8 The team is using the information 9 gathered previously on it's survey and is not 10 only gathering additional information and 11 will look at frame stability over a longer 12 period of time. We currently have just the 13 past year or ongoing frame, and we don't know 14 whether or not surveys are using comparable 15 lists and if they are not, why? Whether 16 there reason are such constrains or legal 17 issues? 18 And at the spring meeting in 2005, 19 you will be hearing an update from that 20 team's effort. Up next you will hear Howard 21 Bradsher-Fredrick of the EIA who will 22 summarize the dual system estimation study. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 79 1 The EIA electricity renewable frame and the 2 National Renewable Energy Laboratory Frame. 3 And then Shawana Waugh of the EIA will 4 describe the collaboration between the Census 5 Bureau and EIA regarding the evaluating of 6 several EIA frames and manufacturing sector. 7 And last you will hear from Tom Lorenz of EIA 8 who will describe what he has learnt by using 9 data from EIA's petroleum surveys both 10 monthly and annual to edit MECS data. MECS 11 stands for Manufacturing Energy Consumption 12 Survey. Although Tom's analysis was an 13 editing project, the frames he will be able 14 to use, his analysis in effecting the frames. 15 And at the end of Tom's presentation, the EIA 16 would like a feedback from committee. But 17 first we will hear from Howard 18 Bradsher-Fredrick. 19 MR. BRADSHER-FREDRICK: Okay I 20 think my slides are coming up. Okay, I'm 21 going to discuss the application of the dual 22 system principles to compute a hypothetical BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 80 1 estimate of the number of frame elements 2 missing from both the National Renewable 3 Energy Lab data frame and I mean I will refer 4 to this as NREL and the EIA renewable frame. 5 As Grace mentioned this analysis was 6 requested by the committee at the last 7 meeting, last spring. 8 NREL collects data from secondary 9 sources and generates a nameplate capacity 10 from renewable energy generating plants, the 11 NREL list is publicly available and EIA 12 collects data on renewable plants but it was 13 a 1 megawatt cut off. So we have eliminated 14 the NREL facility below the cut off from the 15 study. This exercise was intended to show 16 how the dual system principles might be 17 applied in a real setting if these 18 assumptions were to hold, I won't get to the 19 assumptions until the end. 20 First I'll go through some of the 21 computations. So excuse me, we are going to 22 proceed to estimate the number of facilities BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 81 1 missing from both frames if the assumptions 2 were to hold. So we look at the total data, 3 you notice that the NREL is just 5,146 4 facilities, the EIA frame had 4,960 and if we 5 break this down into the components, we have 6 in the NREL list and in the EIA frame we 7 4,932 in common and we have in the EIA frame 8 28 that are not in the NREL list and we had 9 214 in the NREL list that were not in the EIA 10 frame. 11 So we had these three sales 12 together. We get a total of 5,174 known 13 facilities. So we want to try to figure out, 14 well, how many are we likely to have missed 15 in both. We can sell for X with this by 16 making a few assumptions. So we assume 17 independence, we have the probability of NREL 18 error times the probability of an EIA error 19 times the total number of facilities, we do 20 the mathematics here, we get 1.2 missing 21 elements. 22 We can also apply that principle to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 82 1 the various fuel strata that also have 2 information on. In this case in fact we 3 looked at all the fuels but all of them came 4 out to be zero except in the case of 5 agricultural residuals. In that case NREL 6 showed 27, EIA showed 33, and you will notice 7 here we have a higher error rate because we 8 have not -- there is a percentage not as many 9 as common, 19 in both, 14 that are not in the 10 NREL, but in the EIA frame, we have 8 that 11 are in NREL and not in the EIA frame and we 12 have X being those that we are trying to 13 figure out which are those that are not in 14 both is an estimate. 15 Again we can do the mathematics 16 here and we get 2.7 missing. Again applying 17 the same assumptions and those assumptions 18 that were essentially using here that there 19 are two necessary ones and I was really 20 seeing that neither of these hold. The 21 frames need to be independently assembled and 22 at times that seems not to be the case BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 83 1 because NREL uses all secondary sources and 2 we know they also use the EIA as a source. 3 And the missing frame information 4 should be random rather than systematic and 5 this appears not to be the case, I mean at 6 least in terms of timber residues. We have a 7 140, other 214 of the missing elements in the 8 EIA frame. The 65% being in all one fuel, so 9 this appears to be system error. 10 MS. SUTHERLAND: Okay, up next it 11 is going to be Shawana Waugh who is going to 12 talk about the collaboration between census 13 and the EIA. 14 MS. WAUGH: Good morning. I will 15 be talking about the five frames that are 16 being evaluated by the Census Bureau and they 17 are all in the manufacturing sector and I'd 18 like to start actually by introducing Rick 19 Hough from Census Bureau. He is the next 20 survey manager and he was actually -- if you 21 like to go ahead and stand up Rick. He is 22 the person who will be working on this BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 84 1 analysis. 2 I'll be covering the purpose of the 3 frames evaluation. I'll be reintroducing you 4 to the five frame surveys here, EIA(?) that 5 is being evaluated and then talking about the 6 methodology that we propose to use as well as 7 applying it to both EIA 5 which is COKE 8 plants and the EIA 63A which is solar thermal 9 panels and then I'll talk a little bit about 10 this schedule in terms of what is next. 11 The purpose of the frame evaluation 12 is basically to look at the coverage and to 13 do this in a quantitative way. Also the 14 surveys from the coal, nuclear, electric and 15 alternative fuel program office, and again 16 there are five of them. And also as part of 17 the analysis we're going to be looking at 18 differences between the EIA and the census 19 frame, as well as the forms that will affect 20 coverage and the way in which we present the 21 coverage. 22 Because you are looking at volume BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 85 1 and so there may be some differences in the 2 question wording on the forms that would 3 impact volumes that are published separately 4 by EIA and census. And also looking at 5 characteristics of establishments that are 6 missing from our census bureau frame. There 7 are two surveys in the coal area, two on the 8 renewable and one in electric power. 9 The two coal surveys, the EIA3 is 10 -- reports consumption of coal by 11 manufacturing plants and the EIA5 reports 12 consumption of coal as well as production of 13 coal, coke by coke plants. And the renewable 14 energy surveys include the 63A, which collect 15 information from manufacturers or photo 16 thermal collectors and the 63B, which 17 collects information from manufacturers who 18 produce photo voltaic cells and modules. And 19 the electric power survey, which actually, 20 they discussed at the previous session, will 21 include the sub set respondent to a report 22 for combined heat and power plants and that's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 86 1 on the aerial frame of the EIA6. 2 This displays the 5 frames again, 3 which are in the left column as well as the 4 census frames that we will be evaluating them 5 with. And it will either be the economic 6 census and the census of manufacturers or it 7 will be the current industrial report and 8 then in each case we are going to be looking 9 at either an input or an output and today I 10 will be talking specifically these solar 11 collectors where we are going to be looking 12 at the dollar value for value of shipments as 13 an output to determine coverage by volume. 14 And again there will be other 15 variables used depending on the key variables 16 of the survey and I've also displayed here 17 the numbers that are on the frame and three 18 of that we are currently working on are 562A 19 and 63B, all have a small population around 20 20 and 21 intake of the coke plant. The 21 manufacturing plants, there are eastward 2002 22 507 and the electric power plants there are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 87 1 actually more on the frame than what I've 2 shown here but we are going to be looking at 3 the 625 combined heat and power plants and 4 then also looking at the independent power 5 producers which are more like an electric 6 utility but may also be manufacturing. 7 There are a number of steps and 8 methodologies. The first one is to actually 9 conduct the matching and then that will be 10 followed by the analysis. And during the 11 matching we've identified the establishment 12 from the EIA frame and also the threshold. 13 So in some cases we may have a smaller 14 population than what the census bureau has 15 because, for example on the electric power 16 surveys we only have people who are producing 17 1 megawatt of electricity to report. Then 18 those establishments will be matched to the 19 economic census bureaus business register. 20 And again we are taking a very 21 broad stroke here by looking at across the 22 economic census which would include both BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 88 1 those establishments whose primary activity 2 is in the manufacturing sector which makes 31 3 to 33 as well as those that might be in 4 another sector. 5 In the third step we are going to 6 be looking at detailed product code 7 information which is actually a 10 digit 8 product code and that information is only 9 available for establishments which are in the 10 manufacturing sector. So when we look at the 11 volume of information that will be for 12 establishments in the manufacturing sector. 13 And when we ascertain establishments that do 14 not match and possibly making some 15 corrections or adjustments. 16 The findings, we'll summarize the 17 results, the tabular results for count as 18 well as volume and I'll be showing you those 19 tables in a moment. And we also hope to 20 summarize some of the differences between the 21 frames including differences in 22 classifications of establishments and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 89 1 boundaries of establishments, and 2 characteristics of missing establishments for 3 example in the count table which we hope to 4 be able to produce -- at the state or 5 regional level we may be able to identify 6 establishments that are missing with, in a 7 specific region or state, the size of an 8 establishment and the type of establishment. 9 And the ability to provide this information 10 will be limited by the census bureaus 11 restrictions on data disclosure. 12 I'm going to walk you through a 13 couple of examples here. The first one is 14 matching for the EIA5 frame, which is the 15 coke plant. And again we've identified the 16 coke plant on the EIA frame and matched those 17 establishments to establishments on the 18 economic census using the business register. 19 And then we have identified for those 20 establishments which produce petroleum coke 21 and cold coke. 22 There's two types of cokes there's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 90 1 petroleum coke and cold coke and we are only 2 interested in the cold coke. And most of 3 these establishments are either in 1 or 2 4 six-digit NAICS codes. They are either in 5 the 324, which is related to petroleum 6 industry, but they are not petroleum industry 7 they are coke plants that produce petroleum 8 or coke products which is related to the 9 petroleum industry. Or they are in the steel 10 and iron mills which is 331111. And then 11 again there are these 10 digit product codes 12 that are associated with the outputs of cold 13 coke and -- 14 For the 63 age cover the solar 15 thermal Collectors we've identified the 16 establishments and also matched those to the 17 business register and again they have a 10 18 digit product code, which is the solar 19 thermal collector and the research has been 20 assigned to identify establishments that do 21 not match and to reconcile those differences. 22 In addition for the 63A we've also BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 91 1 looked at producing tables that would display 2 the results both by count and by volume 3 coverage and this example provides or show if 4 you will for what we intend to produce we've 5 just, -- for each of the states we know the 6 number of establishments that produce solar 7 thermal collectors. We'll identify those 8 that match on the two frames and then 9 individually those that do not match on the 10 economic census separately from those on EIA. 11 And then perhaps the most important 12 one is to identify the percent coverage which 13 will if you'll notice the formula done, it 14 will be the number that match on the economic 15 census divided by the number that are in 16 scope on the economic census. 17 The next table is a volume, is 18 intended to display the volume coverage and 19 again it would be for the 63A and small 20 tables would be done for the other surveys. 21 At this level we anticipate that either we'll 22 be able to produce a national or potentially BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 92 1 a portal and this would be a dollar amount. 2 Again we will be displaying what information 3 we are able to find that matches on both 4 surveys as well as the differences between 5 the two and producing a percent of coverage. 6 The one difference between this 7 table and the previous one is that I 8 mentioned that the 10 digit code the census 9 is, the establishments which are in the 10 manufacturing sector will be able to give a 11 percent which will act, may in some cases be 12 a sub set of those establishments that were 13 actually able to match because if an 14 establishment is outside the manufacturing 15 sector then there would not be 10 digit 16 product code information available for that 17 establishment. 18 The proposed reports done for each 19 of these surveys will be discussion of the 20 matching, the tables that indicate the 21 percent coverage both by count and by volume, 22 and then identifying similar indifferences BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 93 1 between the surveys. And again these 2 differences could be due to the purpose of 3 the survey or the frame, for example there 4 may be differences among how respondents 5 report on joint ventures. 6 On an EIA frame for example there 7 may be actually be 2 respondents for 1 8 establishment and on the census that may be 9 done slightly differently. So there may be 10 reasons for differences in the counts or in 11 the way the volume or information is 12 collected. And also in terms of the 13 questionnaire designed, there may be some 14 reasons for differences in the volumes that 15 are actually collected and we need to look at 16 that a little further. 17 Characteristics of the 18 establishments missing, again we're going to 19 look at location and hopefully size and type 20 of establishment and that will then be 21 helpful in terms of being able to take some 22 action to improve EIA frames. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 94 1 The schedule is that by December of 2 2004 we hope to have the finalized results 3 for the three small surveys, the solar panel 4 and the coke plants that I mentioned earlier 5 as well as the photo voltaic cells and 6 modules and by July of 2005 have the final 7 results for the EIA3 which covers the 8 manufacturing sector -- manufacturing 9 establishments that consume coal and the 10 combined heat and power plants in the 11 electric power survey. 12 SPEAKER: Okay, last up is as we 13 come around we are going to talk about the 14 result in surveys and the MECS data. 15 MR. LORENZ: Good morning everyone. 16 I've been working on a project for the past 17 couple of months now where I've been 18 reconciling EIA 810 and 820 information 2002 19 MECS. I just want to acknowledge a few 20 people Rick, Vikki, Lucy and Eva for helping 21 me out with collecting some of this 22 information. I also want to thank Preston BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 95 1 for doing the slides for me this morning. 2 You know, I just want to give you a quick 3 background on some -- on the three surveys. 4 MECS is a quadrennial survey of manufacturing 5 industries that's based on NAICS and we have 6 3 survey forms one of which is specifically 7 designed for the refineries and we have it 8 setup for the refineries because we don't 9 want any duplication of the other refining 10 surveys such as consumption data of the 11 petroleum based products and for any other 12 sources, with any other, any other energy 13 sources we don't collect -- we collect 14 everything for stock use. 15 The EIA 10 they track the beginning 16 through the ending monthly balances of not 17 only crude oil but also the other petroleum 18 based products that are made at the refinery. 19 The EIA 20 is an annual survey that 20 collects the current and next year 21 predictions of the refining capacities. It 22 also collects previous year purchase fuel BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 96 1 data, which is helpful for us on the MECS. 2 Now you might ask why we are doing this. We 3 are doing this because since the MECS has 4 been fielded since 1985 and there have been 5 differences in the petroleum based fuel 6 consumption data, primarily in the 810 and 7 the MECS. 8 And this year is the first year 9 that we have had an agreement between the EIA 10 and census, initially as Grace mentioned it 11 was for MECS purposes, but we also decided 12 that it was a good time to look out at the 13 micro level data to see if we could find out 14 why there are some of these discrepancies. 15 Another reason that we are doing this is 16 because refining is a very important part of 17 the manufacturing industry, it accounts for 18 approximately 30% of the total energy use in 19 the manufacturing sector in the US. 20 Just I will give a bit about, about 21 what we did on the project. Look at on the 2 22 phases. First phase was an overall BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 97 1 assessment where we compared the active 2 refinery from the MECS to the active and the 3 inactive refineries in the 810 and the 820 4 frame. Phase 2 I mean I call it a 5 discrepancy resolution. We contacted 6 approximately refineries starting with the 7 larger ones and ending with the smaller ones. 8 Once we found a discrepancy we would contact 9 the MECS responded identify the discrepancies 10 to them and have them deal with the 11 discrepancy with their EIA 810 or 820 contact 12 at the refinery internally and then report 13 back to us what the resolution was. 14 I'm just going through results very 15 quickly, in phase 1, 96 refineries out of the 16 116 matched between the 2 groups so that's 17 approximately 83% which isn't too bad. As 18 you noticed there is about 20 refinery 19 difference there and in 6 of those they were 20 in the MECS frame but they were not in the 21 EIA 810 820 frame. 22 One of them was, we identified as BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 98 1 being out of business. The other 5 were 2 identified as wax and lubricating oils were 3 the primary shipments, which is very 4 important because that identifies a certain 5 group, that the 810 and 820 might not 6 necessarily collect for. Two of those 5 7 checked the refining and non-refining 8 operations on our survey. And in phase 2 as 9 you can see the first two bullets, the 10 petroleum based consumption fuels really did 11 not match up all that well and those are the 12 fuels that are on the 810 survey. 13 There's 3 that, there are three 14 petroleum based fuels that matched up fairly 15 well after response errors were corrected. 16 Now those response errors were not only MECS 17 but also response errors that were identified 18 as being response errors on the 810 that had 19 already been corrected since 2002. And the 20 purchase fuel consumption data from the 820 21 matched with the MECS very well so. Just in 22 over all there's I think two different BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 99 1 difference, first the coverage is things that 2 aren't necessarily, that aren't that are okay 3 to have, they are okay to have the coverage 4 differences with the MECS and the 810 820 5 frames. 6 First the kinds of establishments 7 we collect for, such as those I just 8 mentioned a few minutes ago. The waxes and 9 lubricating oils as being a primary 10 shipments. I think a lot of it has to do with 11 the fact that MECS uses the NAICS to fully 12 establish a base. NAICS has a very rigid 13 structure for classifying establishments that 14 the 810 and the 820 don't have. Another 15 coverage difference is that the EIA10 does 16 not cover some of the on site fuel use, 17 that's not used in the refining process and 18 what I'm speaking of is the diesel and motor 19 gas which is used in front side vehicles, 20 used in heavy machinery, pumps, compressors 21 and things of that nature that isn't 22 necessarily used in the refining process. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 100 1 The thing that I think we have to 2 concentrate more on is the data differences, 3 again the next purchase fuel data seems to be 4 okay, it seems to match up fairly well with 5 the 820. However the petroleum based fuel 6 consumption data on the MECS it doesn't match 7 up very well with the 810. And one of the 8 things that we have found was that there is 9 only three refineries that had the same 10 responded between the MECS and the 810, so I 11 think there is some inconsistency and some 12 interpretation when a couple, more than one 13 person is filling out two different surveys, 14 so we have to look into that a little bit 15 more. 16 Now what can we do about it, the 17 potential next step, I mean potential because 18 this all depends upon budgetary provisions 19 and such and if we can do it to not, we still 20 have to look into it. What we'd like is, 21 what we are looking into it is possibly 22 drawing the sample earlier so we can verify BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 101 1 some contact information, possibly sending 2 out you now like a post card to verify that 3 information and get better contacts. Bless 4 you. 5 Another thing that we can do is, is 6 the MECS and the EIA 810 and 820 people can 7 hopefully work together to improve some of 8 the in- consistencies of the data. And what 9 I mean is we can reference each others 10 surveys were appropriate so that we can 11 possibly identify that for example the MECS 12 can say this should be equivalent to the 810 13 survey if that's where it's needed. And 14 again if you have any suggestions, if we can 15 do anything differently I'd be happy to hear 16 it and any questions you might have for me. 17 MS. SUTHERLAND: Thank you, sir. 18 At this time we would appreciate any comments 19 the committee might like to share about our 20 committee's activities. You have any 21 questions. 22 DR. HENGARTNER: I'm glad you are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 102 1 doing the dual system estimation. It's nice 2 to see because I mean it's a good atrocity, 3 it's very dark, always assumptions that go 4 into these things and the independent 5 assumptions as to how saying yours, at most 6 at this tenures. 7 Now what is also true is that very 8 often when the census dept cant know that, 9 when you do the real system estimation is 10 stratified and you do it on the stratified 11 thing and hopefully the assumptions on the 12 stratified might be better. Now the other 13 thing that I just want to point out is that 14 I've tried to do the math myself also and 15 I've got slightly different questions, which 16 is normal, I tend to make mistakes -- 17 MR. BRADSHER-FREDRICK: Well it 18 depends upon the denominator, what one really 19 wants to be used as well what is the total 20 thing and I use the total number of 21 facilities, they I think I just go around and 22 I have used different numbers than previous BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 103 1 to go around, so there may be discrepancy. 2 DR. HENGARTNER: Those small things 3 would address -- if I can sit down with you 4 on a one to one basis so we can go over, it 5 doesn't make a big difference, but if we are 6 going to use the methods we might as well try 7 to be sure that it works. So these were the 8 two comments they had on the dual systems. 9 The other thing that occurred to me 10 as the last presentation on using the census 11 having two forms with different answers, that 12 is always something that I like 13 inconsistencies because it reveals something, 14 it reveals something that mainly we can learn 15 about the errors. 16 I don't know which one of the two 17 is right, but if I learn that you know their 18 inconsistency, people make mistakes that 19 might inform me on knowing on how much faith 20 I should have with the individual numbers on 21 the other forms which I haven't subscripted. 22 Because I mean both people might have done BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 104 1 the same mistake as well. 2 MR. LORENZ: Well we found that 3 too, we found that -- 4 DR. HENGARTNER: So there are ways 5 you can use that to enlighten yourself about 6 the error rate in the response, and although 7 it is a good thing to try to eliminate them 8 afterwards, having at least a way of 9 calculating those error rates is interesting. 10 And it should be taken into account since, in 11 the quality of the data that this we're 12 collecting and that is something that 13 something we don't have the occasion of doing 14 very often, asking the same data twice and 15 getting different answers. So this is 16 interesting in that sense and those are 17 things that you can learn from it. So I -- 18 MR. LORENZ: It's wonderful. 19 DR. HENGARTNER: Okay. 20 DR. FEDER: I just want to make one 21 brief comment on what Nick said that he likes 22 to see inconsistencies I mean that's what's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 105 1 good, I think Barbara probably knows more 2 about the liability studies, when you repeat 3 a survey or a testing instrument specifically 4 to see how consistent the answers would be. 5 So here you have something, which wasn't 6 maybe planned to be this way. But it still 7 gives you data. Of course the instruments 8 are slightly different so one has to try to 9 understand if this is indeed because of 10 uncertainty on behalf of the respondent or 11 may be it's something within the 12 administration of the question that causes 13 the discrepancy. 14 MS. KIRKENDALL: Just a comment on 15 the dual system installation. Everybody 16 knows the replies that Shawna provided on 17 what we're going to get in terms of the count 18 from a comparison -- we should be able to run 19 through that. 20 DR. HENGARTNER: Oh, yes. I mean 21 as I said, these are estimates and like any 22 estimates I mean, it's as good, it's as good BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 106 1 a guess as I could give you otherwise. 2 MS. KIRKENDALL: At least when 3 we've comparing the census bureau referring 4 to the EIA frame was pretty independent. At 5 least I know that first assumption that could 6 be violated. 7 DR. HENGARTNER: Yes. It 8 stratifies then you can even take into 9 account will show you for example it's one 10 industry where everybody is missing, like the 11 lumber industry so that, that's what I 12 think -- if we know something like the 13 different spectral. And that's it. I'm glad 14 you're doing it. 15 MS. KIRKENDALL: Actually what we 16 didn't talk so much about I think, as Grace 17 mentioned it, as we had the EIA team, that is 18 one person from each office and they are 19 charged with going through all of the 20 information that we could find on all of our 21 frames, which is a very good chore to put 22 together and also a big chore to go through BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 107 1 that evaluation. And they are trying to 2 figure out you know what can I say about each 3 frame do they think it looks good and say 4 it's sufficient or whether it's not 5 sufficient or you just not know. 6 We are looking at right now we 7 don't really know. You may you may have some 8 thoughts but you might not choose for the 9 classification done since you know you're 10 going to get answers sometime later, early, 11 more input and this is sometimes very hard to 12 come by. I am hopeful for that exercise 13 because it will be the first time we've ever 14 taken a big broad look at all of our frames, 15 and see what they all look like. 16 DR. HENGARTNER: The other thing I 17 was surprised to learn that it's hard to get 18 information at times, for competitors it's 19 easy. I mean it's not what I expected. Well 20 I mean I didn't think about it. Using that 21 information also to hope to evaluate the 22 frames if that can be done that's good. Now BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 108 1 again I mean I don't know if it's practical 2 yet. 3 MS. KIRKENDALL: Yes, I know it 4 depends on where we find problems in the 5 frames. If we have questions about the 6 quality of the frame or we have some insight 7 to what the issues than that might be or not. 8 That will be something we'll have to look at. 9 DR. FEDER: What is NREL? Who is 10 doing NREL? 11 MR. BRADSHER-FREDRICK: This is The 12 National Renewable Energy Lab and they have 13 some thing on their website that's all 14 generators in the country that operate on 15 renewable fuel and they call them REPIS 16 database. 17 DR. FEDER: Are they government or 18 what are they? 19 MR. HSEN: They are one 20 organization, the DOE. 21 MS. KIRKENDALL: He's not getting 22 that recorded. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 109 1 DR. FEDER: And MECS is done by 2 census right? 3 MR. LORENZ: MECS is done by the 4 EIA. 5 MS. KIRKENDALL: Oh no, MECS is an 6 EIA survey but it is conducted for us by the 7 census bureau. 8 MR. LORENZ: Okay thank you, that I 9 forgot. 10 MS. KIRKENDALL: So we both are 11 very interested in MECS and the quality of 12 MECS. 13 MS. WAUGH: And we have two MECS 14 surveys. 15 MR. BLAIR: I just had a question 16 on the dual system estimation of what 17 variable or criteria that is used to compare 18 an item on one frame to an item, an element 19 on one frame to an element on the other 20 frame? 21 MR. BRADSHER-FREDRICK: Actually it 22 was done visually. I did all of it BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 110 1 personally, it took about 2 months and it was 2 done in the year 2002. And you just had to 3 try to match names. Naming conventions 4 weren't the same, I mean I looked at such 5 things as, when the generator first started 6 operating, you know some of the variables 7 that we have and they have in common. So I 8 think we were pretty accurate. 9 MR. BLAIR: Right. And since you 10 did it personally I'm sure it was -- but in 11 general one think to keep in mind especially 12 if you start doing this with large numbers 13 that you can't do personally by hand, is the 14 possibility of misclassification errors, 15 because this depends on you know getting 16 these things in the right cells. 17 And you know you want to think 18 about sort of what criterion, what criteria 19 you are using to match and if there might be 20 multiple criteria level that will improve 21 your, you know classification. And again 22 whenever we are estimating -- and I think BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 111 1 that's when you don't want to overload. 2 MR. BRADSHER-FREDRICK: Yes, 3 especially if we automate some of it -- 4 MR. BLAIR: Exactly. 5 SPEAKER: Even if it's automated I 6 would encourage you to the ones that are not 7 in common. In one frame but not the other, 8 each one just look at them visually. 9 SPEAKER: Yes. 10 SPEAKER: Because these are the 11 ones that are going to match in terms of 12 errors, and sometimes those naming 13 conventions might change a little bit to 14 know, if all those things might just show 15 little differences, that a human looking at 16 it says Oh No clearly they are distinct. 17 SPEAKER: Yes. 18 SPEAKER: So the ones that are 19 classified probably they match, they are 20 probably correct, they are the ones that I 21 would look at visually. And that would speed 22 up things. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 112 1 MR. BRADSHER-FREDRICK: Yes and 2 there was a visual checking on all of them. 3 MS. KIRKENDALL: Well you'll 4 certainly hear about, I guess we didn't have 5 too many specific questions for you, this is 6 mostly an update session. Probably at the 7 next meeting we can give you a better summary 8 of where we are, we should have heard at 9 least on the small surveys from the census 10 bureau and we should know from our team what 11 the over all assessment looks like. And then 12 we would be trying to figure out what to do 13 with it, or we are going to have a good 14 challenge. 15 MS. WAUGH: I just want to ask for 16 some guidance. If we come up with an 17 increased percentage coverage by volume what 18 would you think would be the position. 19 MS. KIRKENDALL: Good question. 20 MS. FORSYTH: Well a 100. 21 MS. WAUGH: Could we have 20 or 21 22 establishments and then the other two have BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 113 1 like 500 or a 1000. So to the larger thing, 2 is the difference "percentage" that would be 3 commissioned from surveys that only have 20. 4 MR. BLAIR: On the larger ones I 5 would go back to the comment that I made in a 6 different context and that is the 7 stratification to look, to see not just the 8 percentage but sort of why we should be more 9 concerned if you are missing a bunch of 10 people or in a particular type then you will 11 sort of with the overall because the nature 12 of the size or these have been effective 13 estimates and that's what you are interested 14 in. 15 And so you know they are not all 16 created equal, so I don't know, it's just 17 simple as saying what's a good percentage and 18 what's a bad percentage by sort of what's the 19 being typed on your estimation. 20 MS. WAUGH: Okay. 21 MS. KIRKENDALL: At one time Tom 22 said that there are 83% match and that's not BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 114 1 really good. But there were some of those 2 that were really 60% match and those that 3 were really a question that were not online 4 and the others were whether they were 5 inactive or not with this issue. 6 MR. HOUGH: I think the main 7 results of those issues, I don't believe 8 there's anything for designing surveys. The 9 six that we identified were in the realm of 10 refineries, they were small, so even if you 11 make an effort to go out and find them, I 12 don't believe it's going to give that extra 13 data to the point where I don't know if 14 there's a benefit to undertaking that type of 15 exercise. 16 MR. BROOKS: As part of the A20 we 17 don't classify them as refineries, it's 18 really lubrication plants. You may want to 19 add it into your census. 20 MR. HOUGH: They didn't get the 21 definition. 22 MR. BROOKS: From the standard BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 115 1 point of the Department Of Energy, they are 2 not refineries. They take what the 3 refineries produce, which we call as base 4 stocks and purify them or filter them and 5 package them. 6 MR. LORENZ: Which is one of the 7 definitions. 8 MR. BROOKS: But they don't, they 9 don't distill crude oil. 10 MS. KIRKENDALL: I think what it 11 means for the EIA is we just, we may want to 12 be a little more careful of what we say. I 13 don't think it means that we want to add them 14 to our frame but it may mean we want to be 15 more careful in our verbal description of 16 what we had discuss. 17 MR. HOUGH: We are going through 18 the same process what Shawn talked about also 19 and to respond in the same fashion and frame. 20 The ultimate question that I'm asking is, I 21 believe it was discussed a second ago, there 22 are response errors obviously that goes down BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 116 1 to a 10 digit level of detail. The first 2 question that I have to ask when I identified 3 differences in the two frames is if the 4 product value that was reported to the census 5 balanced or is it a response error from a 6 respondent where they are actually 7 manufacturing a product that should not be 8 classified as a solar thermal collector 9 bridge. 10 So there is a kind of a 11 verification going on at first site, I did 12 exactly what I believed -- I looked at the 13 differences and I go to the web and I 14 researched what the company is doing and then 15 look at what the company is telling us they 16 are doing and making sure that that remains 17 accurate because what we found especially in 18 the reconciliation of the MECS data is any 19 time you send a question out, it is open to 20 interpretation from the person who is 21 responding and a lot of the differences in 22 what Tom talked about in my belief goes right BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 117 1 to that interpretation question, and actually 2 some of the respondents actually told us 3 that. 4 They were using, one person used 5 this method of calculation or in context, 6 this person used this method of calculation 7 for content, we need to go talk about it and 8 figure out which one we should be using to 9 report on both. So there are, there is this 10 verification stage I think we have and you 11 know all surveys contain some level of 12 response error and any time you have an 13 ability to compare data, which should be 14 comparable. 15 You need to make sure that you are 16 not comparing apples and oranges and that you 17 are comparing apples and apples. So there 18 is, the benefit is there in both ways here, 19 if there has some response errors that was 20 identified and corrections have been made to 21 the census database and the results that will 22 be given back to the EIA will be from a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 118 1 tabulation that is done after those 2 corrections are made. So the processing is 3 something as we have just suggested that it 4 is on going please. 5 SPEAKER: Looks like we are fairly 6 early but it sounds like we've run out of 7 things to talk about. Summary is supposed to 8 happen at 11:55, so there is another 15 9 minutes. 10 CHAIRMAN BREIDT: Okay, I guess we 11 are ready to summarize our breakout sessions 12 and Barb Forsyth will go first. 13 MS. FORSYTH: I guess I'll begin by 14 summarizing what the session was on, and we 15 talked about frame evaluation and a series of 16 frame evaluation and data comparison test 17 that EIA has been working on, some of it we 18 talked about in our spring meeting and this 19 was an update of some of those or a follow-up 20 to some recommendations that you know of our 21 spring meeting, so we had re-presentation, 22 one on a relatively small dual frame BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 119 1 estimation activity and a second on clearance 2 or working progress comparing three center 3 surveys with five EIA surveys in terms of 4 their coverages and also in terms of the 5 estimates they produced. 6 And then a third comparing the MECS 7 which is a survey of manufacturing industries 8 with two EIA surveys, which I believe focus 9 on refineries and all three of the papers we 10 heard had frame evaluation components and two 11 of them also had some survey comparison data 12 component, so both of those, I think speaks 13 the interest of the folks around the table. 14 In terms of the dual frame estimation paper 15 on -- the dual frame estimation suggested 16 there are relatively few elements missing 17 from the EIA frame or frame, frames and sub- 18 frames, I guess that were compared and so 19 that was promising. 20 EIA noted that there are some 21 relatively strong assumptions required for 22 the dual system estimation and we talked a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 120 1 little bit about that in our follow-up. In 2 terms of the work in progress comparing the 3 three central surveys and the five EIA 4 surveys, again that paper is focusing both on 5 coverage and their plans -- they talk mostly 6 about their plans, are to produce assessments 7 and coverage both in terms of counts of 8 missing elements and also in terms of the 9 volumes of, I guess, products represented 10 volumes of shipments, represented by those 11 missing elements, so that's a nice double 12 look at coverage. 13 And then the third paper look at 14 coverage and got a pretty high percent of 15 matches across the MECS with the two EIA 16 refinery surveys and then they were able to 17 look in more detail at some data differences 18 between the two surveys and some differences 19 between the elements in one set of frames and 20 not in the other, so that they could draw 21 some conclusions about possible definitions 22 or other reasons for the lack of coverage. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 121 1 So, that I think, those are the papers we 2 talked about and then the questions to the 3 committee were in general comments. And, 4 yeah, we have some general comments. 5 First of all there were 6 enthusiastic reception for the dual frame 7 assessment in particular but I think for the 8 task of evaluating frames in general, I think 9 the committee generally was enthusiastic 10 about all three of these activities. And we 11 noted that, especially for the dual frame 12 estimation, some of the assumptions maybe 13 tenuous but there are ways of working around 14 some of those potential risks to the 15 estimation, for example by stratifying and 16 looking at coverage issues within strata in 17 the surveys and that affords several 18 advantages. 19 First, you can determine whether 20 there are systematic holes in any of the 21 frames and those of course are important and 22 under similar point the magnitude of coverage BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 122 1 error maybe less important than the 2 systemeticity of coverage error and 3 stratification can help you assess the latter 4 systemeticity of it and maybe give you a way 5 of interpreting the magnitude of coverage 6 error they would get. 7 So, that the assumptions are 8 tenuous, doesn't really worry us, we can 9 stratify and begin to address it, it worries 10 us of course but there are tools for getting 11 around it and those tools will, will get you 12 other benefits as well. We also noted that 13 there is a real benefit to comparing a range 14 of frames and so the three papers that we 15 talked about looked at frames as very 16 different kinds of different kinds of 17 relationship among themselves and looking at 18 that range of comparison gives you a sense 19 for how the frames fit in together and can 20 give you a tool for looking at, perhaps 21 logical reasons why there are missing 22 elements from some frames, so in particular BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 123 1 the census frame, for example are relatively 2 independent of some of the EIA frames and so 3 they provide a way of perhaps interpreting 4 results you find with other potentially more 5 correlated frame. 6 We raised the question or we talked 7 about matching criteria, it's an important 8 feature of frame evaluation and when the 9 frame evaluation is relatively small, 10 humanized and do some of their matching and 11 we thought there was a real benefit to having 12 experts involved in the matching process. As 13 the process -- it requires more automation, 14 it's important to introduce a lot of matching 15 criteria, that way you can reduce the 16 likelihood of failure to match when there 17 should be a correct match. You can avoid 18 misclassification here, is another word. 19 So, it's important as you automate 20 the processes to make sure you have a lot of 21 matching criteria, so that was one point 22 made. And when you're matching automatically BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 124 1 or when you automated the matching process, 2 it's particularly important to examine those 3 places where you get mismatches and look at 4 those -- if you can involve an expert, that 5 is one place where an expert is likely to be 6 most useful because changes in convention, 7 for example in coding conventions or naming 8 conventions may produce mismatches -- 9 automated mismatches that in fact an expert 10 will be able to determine are not real 11 mismatches but they are in fact 12 classification errors. 13 Let's see -- so I think -- I hope 14 that captures most of what we said about the 15 frame, I was scribbling pretty frantically. 16 I think there was also -- this is kind of 17 interesting, there was also enthusiastic 18 reception for the inconsistent - - 19 inconsistencies in data found across surveys, 20 so we did probe to see some inconsistencies 21 in data and I think the reason is an 22 important one that that inconsistencies can BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 125 1 tell you a lot about your data and I think 2 that's the point that at least some of us 3 made pretty strongly that inconsistencies 4 give you way of evaluating the quality of 5 your data and for tracking down reasons for 6 inconsistencies that are potentially 7 actionable since you might be able to do 8 something about. 9 So if there are data that -- if 10 there are definitions, elements are used in 11 different surveys or in the way they're 12 interpreted by different respondents to 13 different surveys, those are the things that 14 you can act on once you've identified them, 15 so the search for inconsistencies across 16 surveys that should yield similar results, it 17 seems to offer(?) -- is a fruitful activity. 18 So, I don't think there was a whole lot of 19 negative votes although I may have missed 20 something but I think the committee was 21 pretty enthusiastic about all of the work 22 that we were giving. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 126 1 CHAIRMAN BREIDT: Okay, any quick 2 questions or comments? 3 MS. FORSYTH: About addition? 4 CHARIMAN BREIDT: Okay, let's turn 5 it over to Mark to summarize the second 6 phase. 7 DR. BURTON: Okay, we were in a 8 session where we were presented with the -- 9 about the description then and some 10 preliminary results of the multi-regional 11 electricity equilibrium model that Phil and 12 Dave are in the process of developing. This 13 is an extraordinarily ambitious project, the 14 current models are national in scope and 15 don't really allow us to answer a number of 16 other important questions. 17 The models that they were in the 18 process of developing, if is in fact 19 successful will add materially to the ability 20 to evaluate things like summer gasoline 21 prices within specific regions or whether 22 fuel prices within specific regions, so that BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 127 1 it really can't add a great deal to our 2 ability to anticipate what's coming, 3 essentially there is -- it's an equilibrium 4 model with a supply and demand sector, the 5 demand side of the process is estimated on a 6 monthly basis using some time series 7 econometrics techniques that incorporate both 8 economic impacts or economic drivers as well 9 as time series components that they capture a 10 sort of technical nature of electricity 11 demand, it's done for 13 separate regions. 12 We spent a fair amount time talking about why 13 those regional definitions are as they are. 14 The monthly -- the econometric 15 monthly estimates are then allocated to daily 16 and then even hourly demand estimates within 17 each regions, so that what they're trying to 18 do is track the demand for electricity and 19 then the corresponding use over the course of 20 each hour of each day in order to look for a 21 variety of different impacts. 22 So, the supply side of the model is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 128 1 developed based on both base load and 2 available peak load generation, there are 3 certain facilities that they have generated 4 power continuously and then the electricity 5 -- the quantity of electricity available has 6 increased as necessary based on the costs 7 dispatching other facilities and in that way 8 it captures the relative fuel prices because 9 again a facility is dispatched based on it's 10 cost and so that if natural gas is 11 particularly expensive right now, our gas 12 power plant would be more slowly dispatched 13 and plant fueled a different way. 14 It does provide an equilibrium for 15 each region, for each part of the day and it 16 allows the ability to engage in a number of 17 simulations. Now, the first thing that they 18 described was a simulation without any sort 19 of transmission constraint where they simply 20 allow for electricity to flow from one region 21 to another based on the cost of generating 22 that power and the demand within each region. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 129 1 And that then serves as kind of a 2 baseline result -- against which more complex 3 simulations can be compared and by more 4 complex simulations, I prefer to adding in 5 transmission constraints that can at least 6 constraint the amount of power that's traded 7 between one region and another, so that we 8 can -- to the extent that we can identify 9 these transmission constraints and we can 10 also then quantify the impacts that they are 11 having on a variety of different outputs or 12 outcomes. 13 This model is extraordinarily rich 14 in what it may allow to happen, we can look 15 at the equilibrium power supplies and the 16 quantity of electricity that's being produced 17 and is consumed within the overall nation or 18 within the specific regions, we can identify 19 imbalances, we can simulate the effects of 20 changing fuel prices -- we can make the 21 summer warmer or cooler and see how things 22 change. It allows the prediction of imports BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 130 1 and exports within regions of electricity. 2 If complete and functional, this will be an 3 extraordinarily powerful tool. 4 Now, I do this with modeling, not 5 in this context and I know how challenging 6 this is and so it's not surprising that the 7 committee had some question and some things 8 that we wanted more information about. There 9 were some question about how it is that they 10 actually measure and then implement the 11 transmission constraints. I think that 12 that's an area where we all have to admit it 13 right now that there is a great deal of 14 uncertainty. 15 There was some concern that prices 16 fall represented on a monthly basis within 17 the econometric estimation are included or 18 not, they sort of didn't survive the process 19 any further in the daily or hourly demand 20 representation. There were questions about 21 whether or not there had been any work to 22 measure the variability of the outputs, well, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 131 1 and that sort of gets it at, I think one of 2 the most basic issues that we don't really 3 know at this point how well this model 4 performs and moreover it's sort of hard to 5 have the data to make the comparisons to to 6 assess that model performance and that kind 7 of leads us to -- to one of the things that 8 actually I think arose -- one of the issues 9 that arose on the elevator -- 10 Part of the problem is that some of 11 the data necessary to make really good 12 assessments is proprietary data that's not 13 immediately available and the question was 14 raised whether or not if given the potential 15 usefulness of this model, whether or not it 16 might be worth expenditure of EIA resources 17 to go out and actually collect that data 18 directly, if that in fact is doable itself, I 19 will have time between the committee and 20 lunch, so I will stop, if there are other 21 people that were -- that were downstairs and 22 have things to add, I'm sure I didn't capture BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 132 1 everything. 2 DR. FEDER: I don't like to be 3 between the committee and lunch but I would 4 like to add -- started by saying that the 5 model at the baseline is used to evaluate 6 transmission constraints and then it added -- 7 what's so special about the transmission 8 constraints that make it done this way. It's 9 like very hard to model transmission 10 constraints. 11 DR. BURTON: I'm going to let 12 either Phil or Dave -- if he's around to 13 answer that, it's -- 14 MR. TSENG: I think this importance 15 of transmission constraint, actually it's 16 what reflected in the end use electricity 17 prices, so if a region cannot get cheap 18 electricity from other regions, then it would 19 have to be generated within the region like 20 in Florida, for example, if we cannot cheap 21 electricity from other regions, then the 22 price of electricity would be higher. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 133 1 DR. FEDER: My question was about 2 modeling difficulties not transmission, I 3 mean why not just put everything in the model 4 including the transmission constraints. 5 MR. TSENG: Okay, part of the 6 reason is, if we don't solve everything 7 computationally -- that issue and in terms 8 of how do you know a region is experiencing 9 transmission constraints and so we need to 10 actually look at the supply curve within each 11 region and then look at the cost of structure 12 and then see how they would be dispatched and 13 then compare that with the transmission 14 constraint, see if it's binding. 15 DR. BURTON: Well, I think that by 16 doing the unconstrained and constrained gives 17 you some very rough estimate of the potential 18 benefits of additional investments in 19 transmission facilities. 20 DR. FEDER: Good point. 21 CHAIRMAN BREIDT: Okay, so I think 22 now it would be good time to invite -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 134 1 DR. FEDER: We were talking. 2 CHAIRMAN BREIDT: You're still 3 talking, I thought you're finished, I thought 4 I jumped in there, so is there any further 5 public comment. 6 CHAIRMAN BREIDT: Okay, so we're 7 adjourned for lunch and note that from the 8 agenda that when we come back, we come 9 directly back to the breakout sessions. 10 (Recess) 11 MS. KIRKENDALL: Well, I think we 12 should get started. We're five minutes late 13 and we've got a number of people to talk, so 14 why don't I introduce Preston McDowney. This 15 session is about our new natural gas 16 production survey. It's supposed to -- it's 17 been approved by O&B, it's supposed to begin 18 in January and Preston. This is sort of a 19 history and where we are and survey and 20 methodology. 21 MR. McDOWNEY: Good afternoon, I'm 22 Preston McDowney of Statistics and Methods BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 135 1 group and today I'm presenting the simulation 2 results of the probability, sampling for 3 EIA's monthly natural gas production survey. 4 I'm going to start the presentation by 5 recounting Inderjit Kundra's spring 2004 6 presentation today as a committee, I will 7 follow that with the simulation, the steps 8 with the simulation project and then results 9 of the simulation and then I will introduce 10 KMB. 11 At this time I'd like to drive the 12 committee's attention to a hand out that has 13 been placed in front of them, this hand out 14 has supplemental slides to my presentation, 15 more information about some of the slides at 16 the end of presentation. At the last spring 17 meeting -- and to introduce the committee to 18 the 9/14 event, the purpose, if you like, 19 monthly production of natural gas in The 20 United States. We proposed a part budget, 21 pushed aside sample, the goals were to 22 estimate the total US production of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 136 1 co-efficient and variation of the 1% and 2 regional production of CV as 5%. 3 The regions that we were looking at 4 were taxes in the federal gulf, Louisiana, in 5 Mexico, Oklahoma, Wyoming, and other includes 6 all of the other states, excluding Alaska. 7 Presume optimal allocation formula, we used 8 this formula to -- like I have the samples 9 itself. Using that formula we calculated 10 that we needed 176 operators -- natural gas 11 operators to get the 1% CV at the national 12 level. There were 176 operators, 95 13 particular, operators produced 82% of the 14 total natural gas in United States. 15 However to get the 5% CV in other 16 regions, you have to go up to 358 operators, 17 211 of those operators being within a region 18 comprised of a lot of companies that were 19 included 17, in one region and another region 20 and the account promoting state reported 70 21 groups, it reduced survey by about 109 22 operators. And we are extremely happy with BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 137 1 this sample survey size, we actually -- I'll 2 go back -- it actually increased the CV in 3 federal gulf to 1% CV and hand out -- there's 4 -- calculation for how many operators it 5 takes to go from 1% to -- from the 5% to 1% 6 in each region and we had some operators who 7 actually went ahead and did that and the ASA 8 committee didn't have a problem with the 9 approach and so we decided to test the 10 samples out in a simple method, to 11 simulation. 12 Before we just stimulate we have to 13 cerate a pseudo frame and I will talk more 14 about the pseudo frame in the status to 15 follow but after that frame was selected we 16 split certain group for each region 17 individually and then most of the operators 18 were -- remained the same throughout the 19 simulation process. After the group was 20 selected we start to use PPS sampling to 21 sample the remaining operators that were 22 left. After we started 70% of PPS sample BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 138 1 operators, the estimates -- the total natural 2 gas production of the pseudo frame using 3 variations of four different estimation 4 procedures and I'll talk about those 5 estimation procedures later. The pseudo 6 frame consisted of operators that we had from 7 the year 2000 EIA-Sample and on the 2002 8 EIA-23 frame and EIA-23 is the annual natural 9 gas survey. 10 And this was done to adjust, not 11 having a complete frame for the year 2000 and 12 the two year difference reflects the 13 approximate time it takes to prepare a 14 complete frame for a given year, so we go to 15 select sample for the 2005 survey -- the 2004 16 data won't be available yet to at least 2003 17 data to select the sample from. The four 18 estimation procedures, the first was the PPS 19 size estimator and this is used as the 20 inverse of the kind of ---- selected as the 21 weight and to make estimations. 22 Next, regression estimators are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 139 1 provided least square estimator, which is the 2 classic ratio estimator and there's more 3 information about each of the estimation 4 procedures in the supplemental slides. For 5 the weighted least squares and the ordinary 6 least squares estimators, we have two methods 7 to make an estimation. The first used only 8 the non certain companies in estimation 9 procedure, the second method used both the 10 certain and non certain companies in the 11 estimation procedures. 12 So both the big guys and the middle 13 guys were used to estimate for the non sample 14 population. The both methods we tried to do 15 it from variations, the first variation is 16 all of the operators in the given method, the 17 second variation removes out layers in 18 estimation procedure and use internal -- 19 utilize residuals to determine out layers, 20 the third method, remove out layers and 21 overly influential variables and use (?)DFS 22 to determine the influence of variables BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 140 1 observation and information on out layer 2 removal and agent's observations are in 3 subsequent slides. Also the simulation 4 themselves use the program called ARIS(?)to 5 simulate the main samples and also to 6 calculate summary statistics across all of 7 the simulation. 8 We're going to present on example, 9 there was 100,000 samples were simulated. We 10 started with smaller samples of a 100 and a 11 1000 but as you prepare one sample -- one 12 simulation of a 100 samples to another 13 simulation of 100 samples, we got slightly 14 different results for the number of samples 15 to get an equilibrium where you get 16 consistent answers with consistent summaries. 17 Here's just a sample of what the 18 output of the simulations are. We're looking 19 at the percent of the estimated total of the 20 various estimated procedures, all of these 21 you know are using -- all of the operators 22 were not removing any out layers or BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 141 1 influential variables, the PPS estimators 2 assessed gave us the lowest rule average, the 3 lowest, 1% error were given, very large range 4 of errors with 90% of the errors being 5 between negative 8-1/2 and almost 4%. The 6 regression estimators have a much tighter 7 error band but all of the estimators give us 8 a negative bias, the estimators themselves 9 gives us the desired CV of 1% that we're 10 looking for in the beginning of the 11 simulation project. 12 Everything about this is due to the 13 dynamic natural gas industry itself. And in 14 conclusion that the PPS certainly didn't give 15 us the desired accuracy that we're looking 16 for. We didn't get the CV's that we were 17 looking for and the estimation procedures 18 with negative biases and in the second lowest 19 bias there's summary statistics for, we take 20 the out layers out and we take the 21 influential variables out, I didn't put it in 22 the presentation, that doesn't change the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 142 1 results that much at all. 2 This did get the same overall bias 3 and not the CV's, which we are looking for. 4 Because of all this we decided to use a cut 5 off sample, almost 90% coverage at the US 6 level and if there's no burning questions, 7 I'm going to concede the remaining of my time 8 to the distinguished gentleman from Texas to 9 talk about the 90% cut off sample. 10 MR. WOOD: Are there any burning 11 questions? 12 DR. SITTER: I have a couple. In 13 your -- I just want to clarify couple of 14 things if you don't mind. 15 DR. SITTER: Sorry, you used a 16 bunch of different estimators when you did 17 your beta hats, what happened to the unequal 18 probability of selection sampling or did you 19 use different simulations to them? 20 MR. McDOWNEY: We used a different 21 simulation, we do other lot of things to this 22 -- we stimulated a lot of different things, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 143 1 so couldn't present everything that we did, 2 but nothing give us the actual. 3 DR. SITTER: So how did you select 4 your samples when you were using beta hats 5 here as the regression features? 6 MR. McDOWNEY: The sample session, 7 it sounds to me the same because each set -- 8 for each sample we used the different beta 9 hats to -- 10 DR. SITTER: All right. 11 MR. McDOWNEY: Estimate. 12 DR. SITTER: So that means for each 13 sample you selected with unequal probability, 14 PPS? 15 MR. McDOWNEY: Yes. For each 16 sample. 17 DR. SITTER: And what happened to 18 the weights, the unequal probability 19 selection in your beta hats? Were they in 20 there? 21 MR. McDOWNEY: Yes they were in 22 there. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 144 1 MS. KIRKENDALL: No, the weights 2 are not in there the company data were used 3 only, but the probability of selection was 4 not used when computer failed. 5 DR. SITTER: Okay. So these beta 6 hats are not correct. 7 MR. McDOWNEY: It wasn't. 8 DR. SITTER: They are not unbiased, 9 so don't expect them to be unbiased. So if 10 you're going to use an unequal probability of 11 selection PPS and you want to use a 12 regression estimator, you need fill the 13 weights in to your Beta hat. Which can be 14 done with the -- you could've used the ratio 15 estimator for example which is one of the 16 beta hats you used, but you need the -- you 17 need the 1/pi(i)'s and the numerator and 18 denominator for that to be a proper 19 estimator, approximate them by an estimator. 20 MR. McDOWNEY: Okay, your second 21 question? 22 DR. SITTER: Second question is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 145 1 related to -- so this gets worse when you 2 talk about your out layer and influential 3 stuff, because now your actually believing a 4 model where the beta hats is actually not the 5 rating anymore, so I don't think those over 6 influential things are going to be right 7 either. And then when you did and one other 8 question related to that maximum you got for 9 the PPS sample. 10 MR. McDOWNEY: Uh-huh 11 DR. SITTER: You use the Horowitc 12 Thompson estimator, that means you basically 13 use this, your estimator the sum of the 14 y(i)'s/pi(i) or pi(i) is the inclusions of 15 the ---- 16 MR. McDOWNEY: Yes. Something 17 similar to that. 18 DR. SITTER: Something similar to 19 that. So, that can have problems, it's 20 called Famous Bassus elephant example. You 21 probably should instead use a ratio estimator 22 and that maximum would probably disappear, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 146 1 that is, you take the sum of the 2 y(i)'s/pi(i)'s you divide by the sum of the 3 1/pi(i)'s and then you multiply by N, it's a 4 ratio estimator because the kind of bottom is 5 actually an estimate of N and so -- I know it 6 sounds strange, and a lot of people don't 7 realize, but there are situations where 8 estimating a known value is actually better. 9 DR. FEDER: It's one way of 10 calibration. 11 DR. SITTER: Correct, So this is 12 only going to happen in a few cases where you 13 get a bad sample, but you're doing 10,000 14 simulations, you are almost guaranteed to get 15 a few and probably those maximums are 16 therefore not really a problem, in other 17 words they can be fixed. And I think that 18 you make at your some of your CV's, now, of 19 course your sample sides calculation were 20 based on SRS. You're not doing SRS, so you 21 can't really use those of the CV's, so I just 22 want to make sure what's right before you BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 147 1 looked on. I have one other question that 2 it's slipped my mind -- I'm trying to 3 recapture. 4 DR HENGARTNER: Just a 5 clarification when you calculated those 6 things that are the aggression, you did it 7 only -- you excluded the certain percent, 8 correct? 9 MR. McDOWNEY: In one method, yes. 10 MS. KIRKENDALL: In one method -- 11 DR. SITTER: That's the right 12 one(?) we did. 13 MR. McDOWNEY: That's how I did. 14 The first way first. 15 DR. HENGARTNER: So it is, certain 16 example needs to be excluded and -- because 17 you exclude the certainty sample you see the 18 actual CVs larger on the estimated project, 19 but then when you added it again -- 20 DR. SITTER: Then you get the gain. 21 DR. HENGARTNER: Then you get the 22 gain. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 148 1 DR. SITTER: Yes. 2 DR. HENGARTNER: Right -- Okay. 3 DR. SITTER: But one other 4 question, you said you used the random 5 generator to generate samples, was this 6 without replacement, PPS sample you know with 7 the with PPS sample. 8 MR. McDOWNEY: Essentially without 9 replacement with -- you get region 10 individually you know and then we made sure 11 what's the result expected on one region, we 12 removed it from being ---- in the rest of the 13 regions, and then we take their collection 14 especially almost certainly with that. 15 DR. SITTER: So how did you 16 calculate the selection probabilities? 17 MR. McDOWNEY: That question I 18 understand, how do I fit. 19 DR. SITTER: Well -- 20 MR. McDOWNEY: You're trying to 21 place it to each region individually. 22 DR. SITTER: Now- -- but what I BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 149 1 mean is that -- if it's PPS without 2 replacement sample that's not trivial to do. 3 And when I say that it's not the number you 4 divide by is not the PPS anymore. It can't 5 be doubled. 6 MR. McDOWNEY: Yes. 7 DR. SITTER: So did you, were you 8 able to -- 9 MR. McDOWNEY: Yeah we adjusted for 10 the sample itself changing. 11 DR. SITTER: Oh, you did so? 12 MR. McDOWNEY: Yes. 13 DR. SITTER: How did you do that, 14 there's many methods -- different methods of 15 security? 16 MR. McDOWNEY: I essentially 17 physically did it and that, I was modeling it 18 in Excel, it's all in the Excel program. 19 DR. SITTER: Okay, maybe we can 20 talk offline. 21 MR. McDOWNEY: Offline okay. 22 DR. SITTER: Okay, thanks. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 150 1 MR. McDOWNEY: Any other questions? 2 DR. FEDER: I just want to pick up 3 on what Randy said about the Hycar(?) 4 estimator when you divide by sum of The 5 English probability of inclusion which by 6 itself is an estimate of the population size 7 and that's one way of calibration, when you 8 could just caliber to the population size but 9 if you have other information, you can 10 calibrate on this, it will improve the 11 procedures. And I Randy's comment was about 12 difficulty in calculating the pi(i) assuming 13 that -- 14 DR. SITTER: No, I was asking how, 15 right? I mean, I did not understand about -- 16 DR. FEDER: I'm not sure what's the 17 method the PPS use, but if you use, I don't 18 know -- one of this, I think it should stop 19 there. Now has building some of those 20 procedures. I'm just -- I was curious as to 21 what they did. 22 MR. KUNDRA: Oh, I think -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 151 1 DR. SITTER: Systematically. 2 DR. FEDER: So you get the correct 3 inclusion probability to -- 4 DR. SITTER: Right, that's fine, 5 thanks. 6 DR. FEDER: You can also, from the 7 simulation check that you get right inclusive 8 and -- 9 MR. McDOWNEY: Any further 10 questions? Okay, John. 11 MR. WOOD: So after doing all that 12 we decided to use a 90% cut off and SMG 13 recommended it to reserves on production 14 data, we thought it is a hell of a good idea 15 because the difficulty of maintaining and 16 understanding the -- how the frame that you 17 draw all these sample for portable size which 18 in fact our original recommendation is, you 19 know, just a tremendous burden especially as 20 I think our catch phrase now is the Natural 21 Gas industry is very dynamic and the 22 relationship of the large operators to the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 152 1 small operators is always in flux. 2 One of the first line, if you 3 noticed that we in our title you know our 4 plan is to use a 90% cut off, sample as the 5 primary data said that draw of the study. 6 Now, I want to repeat started off with 7 objectives that we needed to improve the 8 quality and timeliness of the EIA's natural 9 gas production estimates, and in fact we 10 wanted to have releasable numbers, 60 days 11 after the end of a production month, and we 12 wanted those to be within 1% of the national 13 level and from 1 to 5% in the six specific 14 areas Texas, federal gulf of Mexico, 15 Oklahoma, Wyoming, New Mexico and Louisiana 16 and those that are rank border and 10%of the 17 US over 48 production. 18 Texas is the largest producing 19 state with about 30% of the US total. And 20 next slide, now -- 21 MR. WOOD: I think that I would 22 like to entertain questions and even maybe BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 153 1 answer them during the talk, because some of 2 this is relatively complex and complicated 3 and it's rather difficult to layout. 4 Basically how we intend to make the estimates 5 in a 15-minute period and when we practice 6 this with the SMG, Nancy had some questions 7 starting about the 3rd slide that she kind of 8 wanted answered then. And I think it's a 9 fairly good way to do it. 10 But in particular we're going to 11 show the data and show it behaves and show 12 how we expect to make the actual estimates 13 from the sample and we going to know what you 14 think about it. In particular, the 15 production distribution by operators is an 16 extremely skewed distribution with the top 5 17 operators having 29 percent of the US 18 production and it's similar in each area. So 19 the impact of the large operators or overtly 20 influential operators or however we chose to 21 identify them is a very important subject. 22 The, again in Texas also the, next slide, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 154 1 what's there -- well we can hardly see it 2 here okay. That was a test. In any case the 3 top 5 operators in Texas have about 25% of 4 the total production in Texas. 5 Then, in fact then, if we have non- 6 response from these large operators or 7 apparent errors from these operators, you 8 know, there are several ways to handle it and 9 we certainly like to have your opinions on 10 how to do so. Or there are some algorithms 11 that we've tested that identify overtly 12 influential operators and how should we deal 13 with these when making estimates. 14 And one of the ways is that if we 15 do not have to use the same group of the 16 sample operators to estimate the non- sample 17 production, in each month. And now we'll go 18 over that several times as we go through the 19 presentation. Well, the next slide. Now 20 this is from our original file where we just 21 took the 300 top operators from 2002. And 22 again the top 5 operators in the lower 48 had BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 155 1 29% of the production. And the top 14 2 operators had 51 percent, the top 53 -- and 3 this is out of 15,000 or so operators, the 4 top 53 had 75% and 223 operators in this 5 particular sample had 9% of the production. 6 And so the other 15000 had 10 percent. 7 Next slide. Now this is about what 8 we intend to do with respect to sample 9 selection. We have built another frame, 10 which is essentially the same operators and 11 essentially the same rank order, but it is 12 built from the thing we're actually going to 13 survey, the gross production that the 14 operators produce. And we've built these 15 from commercially available data sets and 16 therefore we set up a history of the 17 production of every operator, by month, from 18 1997 through 2003 and into 2004. 19 And you get rather similar results 20 if you rank these operators by production 21 size and you pick the first 222, you've got 22 90% of the lower 48 production. If you then BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 156 1 go and look at the individual areas you've 2 got 98% coverage in the Federal offshore Gulf 3 of Mexico, one of the critical areas to 4 monitor. You've got 93% coverage in New 5 Mexico and you have 97% coverage in Wyoming. 6 And that seems like adequate coverage. We 7 then went to Texas and for the rest of the 8 operators holding a cut off at 222, we ranked 9 them by size of production in Texas and you 10 had what, 32 operators and you have 90% 11 coverage in Texas. 12 Then we ranked the Oklahoma 13 production and you have to add 36 operators 14 and you have 90% coverage in Oklahoma. And 15 then ranking the Louisiana operators that 16 haven't already been selected, we ended up 17 adding 4 and we have 90% coverage in 18 Louisiana. And then if you want 90 percent 19 coverage of the other states, excluding 20 Alaska, you have to go all the way up to 337 21 and we don't think that's probably worth the 22 respondent burden or the processing load. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 157 1 And so if we do what we plan, we will pick a 2 sample of approximately 284 operators. And 3 that gives 90% lower 48 coverage and over 90% 4 coverage in the 6 areas that we wanted to 5 specifically monitor. 6 DR. HENGARTNER: John, may I ask a 7 question? 8 MR. WOOD: Yes. 9 DR. HENGARTNER: Can you flashback 10 the -- I just need to understand. So you 11 take the production of every producer, you 12 order them, then you take the sum, correct. 13 MR. WOOD: Right. Rank order by 14 their average annual production during a full 15 calendar year. 16 DR. HENGARTNER: Okay. So this is 17 going to give you a function of that 18 cumulative sum. 19 MR. WOOD: Uh-huh. 20 DR. HENGARTNER: Okay. Now I have 21 some misgivings. So between 222 and 254 I 22 had about 32, right. Between 254 and 280, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 158 1 that's about the same thing, you get 34 more 2 or less, same distance. 3 MR. WOOD: Uh-huh. 4 DR. HENGARTNER: Then I would 5 expect the difference, let's take Oklahoma, 6 the 8317 goes up to 8325. I would have 7 expected that to be larger that difference 8 than between 90 and 83, right. Because if 9 your rank order -- 10 MS. KIRKENDALL: Yes but they rank, 11 in that one they are using the rank in 12 Louisiana. 13 DR. HENGARTNER: For Oklahoma, or 14 -- 15 MR. WOOD: No, in Oklahoma we 16 started here, and you add, you stop and you 17 take the 254 operators. Many of which, in 18 fact most of which operate in every one of 19 these areas. And then you start with the one 20 that would have, you know, the next operator 21 that hasn't already been selected and you 22 rank them by the size of their production in BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 159 1 Oklahoma only -- 2 DR. HENGARTNER: In Oklahoma only, 3 is that -- 4 MR. WOOD: And then you added 36 to 5 get from 83% coverage to 90%. 6 DR. HENGARTNER: But what I'm 7 confused about is if I take the 222 and add 8 32 to get 254 in Oklahoma, those -- these are 9 rank orders in Oklahoma. 10 MS. KIRKENDALL: No, no going -- 11 MR. WOOD: But most of the large 12 operators in Oklahoma have already been 13 selected. 14 DR. HENGARTNER: Okay, then these 15 are nationally ranked. 16 MS. KIRKENDALL: Well no, you start 17 nationally rank and you do a 90% sample. 18 Then you go to the next area you're 19 interested in, you rank that area, and you 20 select the companies that give you the 21 percentage in that area. They also 22 contribute to other areas but you pick them BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 160 1 based on that area. So then you go to 2 another area and you rank the companies that 3 contribute to that area that you picked in. 4 DR. FEDER: Because they operate in 5 more than one region, right. 6 MS. KIRKENDALL: They operate in 7 more than one region. 8 MR. WOOD: I'll tell that. The 9 largest operators in Oklahoma are selected in 10 that first 222. 11 MR. BLAIR: So that column that 12 shows number of operators, that justifies 13 nationally, doesn't apply to each of the -- 14 each of these regions. 15 MR. WOOD: Right, those are -- 16 that's a unique count. 17 MR. BLAIR: Oh, okay. 18 MR. WOOD: That you know, if 19 they're in 5 districts they're not counted 5, 20 that is the national, you know, you may 21 count, you know. 22 MR. BLAIR: Right. But the way the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 161 1 table is set up it would, in first looking at 2 it, you think that the number of operators 3 applies to each of those columns, but it 4 doesn't -- 5 MR. WOOD: It doesn't. 6 SPEAKER: Correct. 7 DR. SITTER: I'm glad I asked the 8 question because I got it all wrong -- 9 MR. WOOD: Well, that was why my 10 sincere invitation that you know, if we are 11 not on the same page here, we're not going to 12 do much better later. 13 DR. SITTER: And you decided this 14 ordering on the report? 15 MR. WOOD: Yes. Now there are few 16 ways to actually get a few less operators 17 that I think - - 18 MS. KIRKENDALL: Importance is 19 based on the size of the region, like for 20 Texas it's one of the -- 21 MR. WOOD: Yeah, this is -- 22 DR. SITTER: Got you. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 162 1 MR. WOOD: But the actual minimum 2 number of people, you know you start out with 3 the one with the least coverage and do it 4 first and rank them back through and it you 5 can do a few, but it's not worth the trouble 6 of explaining it. 7 DR. SITTER: Got you. 8 MR. WOOD: So what we intend to do 9 then is pick these out in a calibration year 10 period and again this is based on each 11 operators annual average production during a 12 year, for example the year of 2000. And then 13 sample those operators and the estimation 14 year would generally have up to a 2- year 15 lag. So we might be sampling those operators 16 and try to estimate the year 2002, from the 17 people we selected in the calibration year of 18 2000. 19 Because there's a long lag between 20 the preliminary data and actually knowing 21 what the final total is. So if we go back 22 now to the year 2000, we know what the total BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 163 1 is and we know what the operation or what the 2 production of each operator in each of those 3 significant estates. And now when you go and 4 look at these, again if you plot by month 5 across the year like 1997, this is the group 6 of operators who had 90% on an annual basis 7 of the 1997 production in Texas. 8 And you notice as soon as you start 9 moving into succeeding years, they start 10 representing a smaller percentage of the 11 total. And in the year 2000, where we have 12 run some detail tests we're going to show in 13 a minute, it's the same thing. There's some 14 significant variation month to month, but 15 this group is picked on the operators that 16 had 90% of the production in the year 2000, 17 on an annualized basis. 18 And again in 2000 as soon as you 19 get into succeeding months, the percentage of 20 the production represented by those operators 21 starts declining, and if you get into the, 22 well let's just finish in January of '03 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 164 1 here, you're down where, you know it's, they 2 represent, they've only got about 87% of the 3 production. So you can't use just a linear 4 multiplier times of production of the sample 5 operators and expect to have anything, but a 6 low biased answer and you saw all the 7 techniques that Preston demonstrated. 8 And again you see the same behavior 9 in 2002. Next slide. This if for the Gulf 10 of Mexico and this group had 95% and you 11 notice that the, you're up over 98% of the 12 survey sample. And again you see the same 13 type of behavior though. After this 14 calibration year of '99, the percentage 15 represented by that group, the 90% group in 16 '99 is you'd unless steadily -- say steadily 17 declining except there's some relatively 18 abrupt changes too. 19 The Gulf of Mexico in particular, 20 large operators can come in and not have it 21 produced the year before and then be a major 22 player, because they bring on a major field BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 165 1 in the off shore Gulf of Mexico. Next slide. 2 Now again, we're going to have a some types 3 of model, where we had picked operators in 4 the calibration year and in Texas in 2002, 5 that would be 203 operators that had 90% of 6 the production. And then this curve, we 7 normalized by dividing by that so that they 8 have a relative 100 in the year 2000. 9 Those operators then have declining 10 production and they only have around 97% of 11 what they started with. They've lost about 12 3%. Those -- and that's 203 operators, the 13 top 90% group. The data that we're trying to 14 estimate is for the people in the 10 percent 15 group. Now that's 5473 operators and they 16 had 10 percent of the production. But their 17 behavior is very different. And in fact they 18 fairly rapidly increased and they had about 19 17% by the end of 2002, 2-year lag to their 20 shares of production. 21 DR. HENGARTNER: Now, you told me 22 the dynamic situations there, operators BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 166 1 coming and going. 2 MR. WOOD: Correct. 3 DR. HENGARTNER: Do you add -- 4 MR. WOOD: And increasing and 5 declining. 6 DR. HENGARTNER: Yes, so in the 7 remaining 10%, do you add a new operator that 8 are generated, that might be very long -- 9 MR. WOOD: That is such an 10 excellent question. Could I have the next 11 slide? 12 SPEAKER: Okay. 13 MR. WOOD: Or maybe go one more 14 forward. One more. You can split them into 15 2 groups. In the bottom 10% there are 16 actually 4,158 operators that were producing 17 in the year 2000. In the next 2 years, they 18 added about 12% to their total, which is 1.2% 19 of the total production. They're also, 20 during the 2 years after 2000, a 139 21 operators, that were not producing at all. 22 And they had at the end of the year 2002, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 167 1 they had gone up to 0.7 percent. Between 2 them they'd gone up to about 2 percent. So 3 the bulk of it was actually from the 4 operators that were in fact producing and you 5 know, since they didn't have as far to go to 6 zero, they preferentially increased, and then 7 there's actually an amazingly complex topic 8 of why the large operators would tend to be 9 declining and the small ones tend to be 10 increasing. But part of it is that is these 11 new companies that are coming on. And you 12 know in part, some of those could be a split 13 of 2 other companies, and we can track that 14 behavior over time although it's rather 15 cumbersome. 16 MS. KIRKENDALL: So of the 139 17 companies, you got them on the list, but are 18 they on your list as -- were they on your 19 list producing zero, are these companies that 20 sort of shows up on the list as a mistake or 21 an error, can you tell? 22 MR. WOOD: You can do a gedanken BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 168 1 experiment and say you didn't know about 2 them. But when you know, it might be 2001 or 3 later when you actually do this calibration 4 for the year 2000. So in all likelihood when 5 we're, let's say sitting out here about 6 January of 2002, we would've known that most 7 of these operators had started production. 8 MS. KIRKENDALL: You would know 9 that they are new names on the list that way 10 you could -- 11 MR. WOOD: Yes. But you know, we 12 wouldn't have known, how much they would, you 13 know they might have been just one month of 14 production let's say, and you wouldn't have 15 you know like annual data for them, or plant 16 history. 17 MS. KIRKENDALL: You might -- you 18 know, how do you find out if they've started 19 producing, these are tiny companies in 2001. 20 MR. WOOD: They appear in a 21 production list of -- usually we go, the 22 states, most of the states, let me just have BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 169 1 it. 2 MS. KIRKENDALL: You don't have 3 2001 data? 4 MR. WOOD: In 2002, in the -- by 5 the, we have probably the majority by far of 6 the production in Texas for 2001, as we start 7 into the first quarter of 2002. It's just 8 not complete. 9 MS. KIRKENDALL: Uh-huh. 10 MR. WOOD: In the first quarter of 11 2001, we'd have excellent estimates of the 12 total and relatively good coverage. 13 MS. KIRKENDALL: But these tend to 14 be little guys at least at the beginning of-- 15 MR. WOOD: Yeah. It's just you 16 don't know it close enough to really 17 calibrate your, you can estimate relatively 18 accurately, but you don't have the data in 19 hand, everyone's actual production report, 20 the full set of all operators to add them all 21 up and say that's the number. You can 22 estimate that number pretty well, and in fact BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 170 1 one of the things we might well want to do is 2 try to use a one-year lag and not a two-year 3 lag. 4 MS. KIRKENDALL: That might be 5 better if you could. 6 MR. WOOD: But, you know again, the 7 people that had zero production in 2000, and 8 there was a 139 of them that appeared in 2001 9 or 2002 and they -- they were pushing toward 10 1% of the total by the end of 2002. 11 DR. SITTER: Did you look to see if 12 it's just some sort of random process, that 13 is, do they disappear and reappear again or 14 are these just brand new people? 15 MR. WOOD: They appear mostly and 16 then stay a while. Some appear and go away. 17 Now by the way it's the oil and gas industry 18 and the reason that there are so many small 19 gas producers is that almost every oil 20 producer also produces gas. It is produced 21 directly with the oil and so they can have a 22 profitable operation producing oil and then BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 171 1 they get some natural gas also which goes 2 into the total. And that type of gas 3 represents about 20% of the U.S production 4 total. 5 DR. SITTER: But you don't have 6 instances where they go away and come back. 7 MR. WOOD: Actually we do intend, 8 and in fact have spent an entertaining 9 afternoon or two putting up all the data of 10 all the operators by month for a 7 year 11 period and in fact watch the many of the 12 large operators declining down and many of 13 the intermediate size and small operators 14 inclining as they become large operators. 15 And so you can in fact there is data to that 16 effect and there is aggregate data to that 17 affect. You want to back up 2 slides. 18 Again, there are four curves here. 19 One of them is the 90% sample and for 2000 in 20 Texas, and you notice it starts losing 21 percentage share as you go forward in time, 22 it's relatively smooth and predictable. This BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 172 1 particular run, this right here, the light 2 blue line is the first quartile of 3 production, which is also the top 5 4 operators. And you notice they went from a 5 relative 100 down to like 92. They lost 6 about 8 percent of their production in a 7 2-year period. 8 And this is one of the reasons why 9 EIA's data was often criticized by energy 10 analysts who look out their window and see 11 the Hudson river. Or even those who'd look 12 out of their window and see the Charles 13 river, that they assume they had some 14 publicly available information on the large 15 operators and if it was declining then all of 16 the production was declining and obviously 17 that isn't necessarily true. Now, this curve 18 here again, is the one we're trying to 19 estimate. 20 The ones that started out on the 21 10% group and it's increasing, the full 90% 22 group is declining, the first quartile was BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 173 1 declining rapidly. This group here, is the 2 90% group with the top 5 operators taken out 3 of it. And that again represents about 65% 4 of the production data in Texas and it's 5 behavior, it increases slightly a percent or 6 two and then is relatively stable across the 7 2-year period. And it's an example of one of 8 the sets of data, if you chose you could use 9 to predict the data's that we're estimating 10 the non-sampled operators. 11 DR. HENGARTNER: Now, just to get 12 this right again, there's 10% of non-samples. 13 MR. WOOD: Correct. 14 DR. HENGARTNER: That has been 15 normalized with 10% of total production. 16 MR. WOOD: During the calibration 17 year. 18 DR. HENGARTNER: Yes. So although 19 it looks like it goes up 15%, it actually 20 goes up only 11/2% of total production. 21 MR. WOOD: Correct yeah, and in 22 fact it does go up -- it increases. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 174 1 DR. FEDER: Yes, that's important. 2 MR. WOOD: But it increases. You 3 know again it increases. It's normalized 4 production. That production increased by 5 15%, which represents 1.5 percent of the 6 total. 7 DR. HENGARTNER: Yes. 8 MR. WOOD: But again, if you know 9 in account for that, and then the stuff from 10 the new operators, you're missing 2% of the 11 data and it's hard to make a 1% accuracy 12 estimate. Now, if we look at the next slide. 13 Again here's the data we're trying to 14 estimate, the yellow line. And in particular 15 the top 15 operators, 50% of the production 16 is the dark line, and again it declines about 17 5% over the two-year period. The top 15 -- 18 the top 15 operators. 19 Now the purple line is, and we 20 started with 203, so the next 188, that we're 21 sampling, is the purple line which, you know, 22 it has more variation in it than the line BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 175 1 which had about a 198 operators in it, you 2 know, 10 more operators. But you know again, 3 at least they didn't decline. And then other 4 groups on the line are none that even got 5 close to replicating the actual behavior of 6 the 10% group. Next, and next. Now, you can 7 write down a relatively general model, that 8 we can always get the total in some month by 9 summing the production from all the 10 operators. 11 In Texas that's, you know, roughly 12 5000. You can break that into 3 groups, and 13 the first sum here is the sum of the -- 14 production sum of the operators used to 15 estimate the non-surveyed operators. The 16 second sum is any of the survey group that 17 you choose not to use their data in the 18 estimating module and the third group is the 19 sum of the non- surveyed operators. Now this 20 is a -- you know a critical concept 21 especially if we're planning to run 22 algorithms to check if they are people that, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 176 1 may be an error, or overtly influential 2 operators or outliners or something and we 3 may need the flexibility of actually changing 4 the group we used to model the non-surveyed 5 operators potentially every month, and you 6 know this general scheme is flexible enough 7 to do that. 8 From those operators that we're 9 going to use to estimate the non- surveyed 10 operators, we're going to multiply them by a 11 function and get an estimate of the 12 non-surveyed operators. Now, in your fantasy 13 world, it would be nice if we could just take 14 your sample which at one point in time 15 represented 90% of the production and divide 16 it by 0.9 and 2 years later get a good 17 estimate for the total. But we've just seen 18 how the data behaves and that will always 19 lead to low biasing. Well, in all the data 20 we looked at it leads to low bias. 21 Next slide. So what we've been 22 tried is you start out with the ratio for the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 177 1 calibration year which is basically 0.11, if 2 you use a 90% sample it's 1/0.9. And then as 3 we saw, that there's a slope to the 4 relationship between the 10 percent group and 5 any of the survey groups, and we -- you know 6 fit and the data for 2 years after the 7 calibration year and determined what the fit 8 parameter Am would be, the next slide. And 9 then you know you -- you multiply by one plus 10 this R and you get the sum of the operators 11 that you surveyed and used in the estimation 12 process. 13 The estimate of the 10 percent you 14 did not sample and the sum of those operators 15 that you sampled, but you did not use them to 16 estimate the 10 percent group. Now this is 17 the errors of the, percent errors in the 18 sample in the calibration run using the, you 19 know, the '99, starting with the '99 20 calibration year. The red bars are plus and 21 minus half a percent. So at least in the 22 calibration fits the errors are rather small. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 178 1 But again these are monthly and so there is 2 some variation month to month. 3 Next slide. And of course it has 4 to get a little more complicated. You 5 noticed in '97, this parameter is 0.0026, and 6 in '98 it's 0.0053, and then 0.0074 and 7 0.0089, 0.0012 basically. So the Am also 8 have the slope over time. Next slide. This 9 is a fit of that. The R2 is 0.9887, you 10 could hope that this is a trend that might be 11 projectable for a year, but not count on. 12 Now if you have no slope component, you even 13 with the Am parameter in there, that adjust 14 roughly for the you know, changing 15 relationship between the 10 percent group and 16 the 90 percent group. 17 It's lagging 2 years, because you 18 calibrated against 2-year-old data and the Am 19 actually had a slope. So you know, that also 20 leads to a bias, although, you know, 21 basically we're in the 1% range for Texas and 22 you'd hope we'd be well in the 1% range for BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 179 1 the nation. If you put in the slope 2 correction, then you know, your, none of the 3 monthly errors are greater than plus or minus 4 5% and they're running effectively without 5 bias. Yes. 6 DR. SITTER: What data do you use 7 to estimate the Am? 8 MR. WOOD: The we had a data set, 9 that is all of the monthly production of all 10 of the operators, starting in January '97 11 through and is complete through 2002. It's 12 relatively complete through 2003 and some 13 2004 data. So this is a history of the 14 actual production as it unfolded. 15 DR. SITTER: Okay. So I'm 16 predicting for January 2003, I have to 17 estimate or which you just taken to be like 18 1/90, and then you got an estimate Am. So 19 what data did they use for the Am, that the 20 estimated got that point through January? 21 MR. WOOD: The one that I got when 22 I went back and did the process 2 years BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 180 1 before. 2 DR. SITTER: So you -- so you took 3 2001 data. 4 MR. WOOD: Yes. 5 DR. SITTER: But. 6 MR. WOOD: We used '99 calibration 7 year, estimated 2001 -- 8 DR. SITTER: Oh I see, I see. 9 MR. WOOD: And then you start with 10 2001 calibration year and use that Am. So it 11 was calibrated on data with almost a 2-year 12 lag. 13 DR. SITTER: All right. I got it. 14 MR. WOOD: Right. 15 DR. SITTER: So you use like '99 to 16 2001 to get your Am -- 17 MR. WOOD: Uh-huh. 18 DR. SITTER: And then took the 19 whole relationship back. 20 MR. WOOD: Correct. Then calibrate 21 it against 2001 and used that Am, the one 22 done with the 1997 calibration year and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 181 1 projected 2003. 2 DR. SITTER: You assumed that the 3 change in this year between the 90% and the 4 10% is sort of stationary over time. They 5 change is stationary. 6 MR. WOOD: Uh-huh. 7 MR. WOOD: And if you want to draw 8 back a slide, you know, that is the change 9 over time, of the Am. This is Am as a 10 function of time starting in well you know, 11 it would be '97 calibration year, '99 12 estimation year and that's the first Am. 13 Could you drop back one more slide and you 14 know, here are the way they are laid out, 15 the, you know, 0.026 and it doubled to 0.53 16 and it's basically going up by that 0.026. 17 So it has a relatively linear slope over 18 those years. 19 Again there's no guarantee that 20 persists into the future. All right. Top 21 four of about three slides. No, one back. 22 This one was done with the first quartile of BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 182 1 production in Texas taken out, the top 5 2 operators taken out. And if you remember 3 from the production plots, the top 5 4 operators were losing the, had the greatest 5 declining slope and they had a, you know, 25% 6 of the production. And so with no 7 correction, this series and the other one 8 went down to I guess, more than 1%. 9 This series is running you know, 10 more like a what, 0.4 percent average low 11 bias and then again you can put a slope 12 component in there and it's the purple line. 13 Now as Nancy pointed out when she looked at 14 this, that led to a slightly larger error 15 here, than you had when you used the full 90% 16 sample. 17 MS. KIRKENDALL: This month -- 18 you're just, you're using the data for those 19 5 companies, it's just you're not using those 20 5 companies to estimate further -- the little 21 guys. 22 MR. WOOD: Correct. And then, you BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 183 1 know again that is, you always use the 2 surveyed data. But you either just add it 3 in, as it'd be nice to know what the 5 4 largest companies actually produce, and 5 secondly you can use any subset of the 6 surveyed operators to calibrate all these 7 parameters and then project which are 10% or 8 what starts out as 10% is the non-surveyed 9 operators, that component that you're not 10 surveying. 11 Now all of these are generally 12 relatively good. You know, it kind of seems 13 like a nice idea if you don't give up a lot 14 of potential large errors to have something 15 that at least, the plan is not to have a 16 biased estimate. Now there are small 17 variations to all the states data, but all of 18 the states tended to have the same type of 19 behavior. You pick a set of operators that 20 represented 90% of the production at a given 21 calibration year and they rapidly or slowly 22 or sometimes radically lose production share BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 184 1 over time. 2 And now the next slide. You know, 3 there's obvious questions here. Do we want 4 to just basically use the full 90%, why does 5 this appear to give better answers than, it's 6 like sampling proportional to size, etcetera. 7 Should we have a biased or picking some group 8 to use and only changing if we have to, like 9 leave the top 5 operators out and unless 10 there's a real problem if you use, and Texas 11 would be the other 188 operators, oh, 198 12 operators. If we don't get some of the very 13 large operators to report, how should we 14 impute for them. And that's potentially the 15 simplest question. Yes. 16 DR. FEDER: Actually I'm just 17 wondering if calculation of standard errors 18 of the estimates won't be hard. Because in 19 addition to the use of the server error and 20 there's going to be a component in the error 21 of the Am and also the R, and I don't know 22 what -- what you do when you publish BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 185 1 estimates, what do you publish, what kind of 2 precision indicators you do for that? I know 3 what EIA is doing and I should've done my 4 homework, which haven't, but it's just 5 another component. I still think that the 6 message makes a lot of sense to me. I'm just 7 wondering what you're going to do about 8 standard errors. 9 MR. WOOD: You want to know -- you 10 know what I'm going to do about it. I'm 11 going to let Nancy worry about it. 12 DR. FEDER: I meant you in the 13 plural sense. 14 MR. WOOD: You'd be -- is that, you 15 know, there is an obvious thing you can do 16 and that's go back and look at a 5 year 17 history and say -- 18 DR. FEDER: Okay, that's a good 19 answer. 20 MR. WOOD: You know, 99 out of a 21 100 times it was less than half a percent 22 error and we don't -- and we'll let you know BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 186 1 if the situation changed a lot. 2 DR. FEDER: Yeah, I mean your 3 method is based on the assumption, which is a 4 logical one, the past is indicative of the 5 future and yeah, you can use that to estimate 6 the server error. 7 MR. WOOD: Yeah, the model be, 8 actual formal estimation of the errors is a 9 much more difficult problem than calibrating 10 the model. And it just -- 11 DR. FEDER: That's always the case. 12 DR. SITTER: I think I read in some 13 of the stuff that we have here that there 14 were more reasons for this approach being 15 attractive than just the simulation study, is 16 that correct? 17 MS. KIRKENDALL: The simulation 18 study made us worry about the frames data, we 19 go in -- the reason why you have that really 20 weird or some of the really weird estimates 21 from the PPS sample, and the PPS estimator 22 was because there are some huge changes that BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 187 1 go on in the industry. Some companies that 2 have a small weight one year and huge -- a 3 small volume one year and huge production the 4 next or vice versa. And the way they do 5 their frames and their update based on the 6 EIA-23 which is a survey they're on, it's an 7 annual survey. They spend a lot of time 8 cleaning the data for the big companies. 9 So the data available within the 10 frame for the big companies are likely to be 11 pretty good. The ones for the smaller 12 companies are fairly not quite as good 13 because they never get the scrutiny that the 14 big ones do, because there's so much more 15 impact on the total. 16 DR. SITTER: Now, you made this 17 claim that stimulation problems were due to 18 some changes in the weights. 19 MS. KIRKENDALL: Well, the changes 20 in the industry, companies that would be huge 21 one year and smaller next or vice versa. 22 Since they wont be -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 188 1 DR. SITTER: Do you think you're 2 able to go in and verify that that's what was 3 causing the PPS to -- 4 MS. KIRKENDALL: A couple of 5 things, once they gave those maximum errors, 6 you could get one or two companies that are 7 in the sample that had big changes over the 8 time. 9 DR. SITTER: Nancy, what this is 10 suggesting is that we should do the PPS and 11 it's by John's method to get the weights to 12 change over time, like the 5 largest ones, 13 their weights or inclusion in the PPS should 14 go down? 15 MS. KIRKENDALL: Well, they would 16 have been in the certainty group for the PPS 17 sample anyway. 18 DR. SITTER: Your point I 19 understand. So you're saying that you think 20 -- you know the weights you selected them 21 with, correct? 22 MS. KIRKENDALL: Well, you know the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 189 1 past data, yeah, you've got our IQ. 2 DR. SITTER: All right. So what 3 you're talking about is you got weights that 4 were large or then you selected the "Y" and 5 the "Y" wasn't as large as you anticipated or 6 vice versa. But that's only going to effect 7 the variability, not the bias. 8 MS. KIRKENDALL: No, but it does 9 make for the occasional large errors. That's 10 what gives you the maximum and minimum error, 11 minimum error, there weren't many minimum 12 errors they were mostly maximum error that's 13 on PPS estimator. 14 DR. SITTER: I'll be curious to see 15 what happens with Hayek estimator as opposed 16 to Horwitz- Thompson in that case. It should 17 be easy enough to calculate. 18 MS. KIRKENDALL: What is the Hayek 19 estimator? 20 DR. SITTER: It's the one where you 21 instead of just -- 22 MS. KIRKENDALL: With the ratio? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 190 1 DR. SITTER: It's basically a ratio 2 estimator. So you use the Horwitz-Thompson 3 estimator, which is what you're already 4 doing. You divide by the sum of the 5 reciprocal weight, reciprocal inclusion 6 probabilities and multiply by the total 7 sample -- by total population, so, so it's a 8 ratio on the population size. 9 MS. KIRKENDALL: Right. 10 DR. SITTER: But it should be less 11 sensitive to that kind of things and that 12 problem may disappear. Now, I think you have 13 other -- 14 MS. KIRKENDALL: I think it would 15 be easy enough to add that to the simulation. 16 DR. SITTER: Yeah, it should be. 17 It's just a different estimator absolutely. 18 Also if you're really putting some emphasis 19 on the simulation results this should really 20 be done right and those datahats are all 21 right. They're not meaningful. 22 MS. KIRKENDALL: You can compute it BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 191 1 in the other way to. 2 DR. SITTER: Yeah, you should 3 compute in the other way and see how it 4 works. Okay, so that's to comment on, 5 because I wasn't sure how much of this is 6 operational. There are operational 7 advantages to not playing with those small 8 ones as well, as you mentioned they're harder 9 to deal with. So I just want to point that 10 if it's two things that you're looking at it, 11 that are influencing what you're deciding to 12 do, you should make sure about simulation 13 results, because I don't think they're really 14 indicative. They may have problems, I don't 15 know, they may even have other problems. 16 MS. KIRKENDALL: Well I mean, the 17 other question about the simulations results, 18 which were used to ----. And now we've got 19 data that we could get from John, that he has 20 with the whole population. 21 MR. WOOD: Correct, and -- 22 MS. KIRKENDALL: 2000, the whole BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 192 1 population in 2002. So we could really do a 2 good simulation with that. 3 DR. SITTER: Now you're still going 4 to run into problems. But I don't know how 5 much you're -- how you're updating your 6 training over the years, because, okay in the 7 cut off sample that property you're seeing 8 there is going to translate into the fact 9 that the guys that are in your certainty -- 10 that are your certainty groups are not going 11 to be the certainty group you would 12 necessarily want over time. 13 MS. KIRKENDALL: I have to select a 14 sample every year. 15 DR. SITTER: That you call your 16 certainty group. 17 MS. KIRKENDALL: Yes. 18 DR. SITTER: And there are 19 operational disadvantages to that. All 20 right, but if you're doing that then they, 21 this -- you see, I think I can explain why 22 this happens. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 193 1 MR. WOOD: Which happens, what 2 happened? 3 DR. SITTER: This property of the 4 90% cut off guys decreasing. 5 MR. WOOD: And what do you -- can 6 you -- 7 DR. SITTER: I'm just 8 hypothesizing, okay. Suppose we have some, 9 we have process that's random, it's 10 fluctuating, that is the big operators are 11 producing, not producing more or less each 12 year. And other operators are coming in and 13 going out, a birth and death process as well, 14 and -- at any given point in time, I take the 15 top group. I guarantee, if I now focus on 16 that top group, it's production will end. 17 DR. HENGARTNER: It's called 18 regression towards mediocrity. 19 DR. SITTER: Regression towards 20 mediocrity. You have to have them. It's not 21 something that's going like this, you've got 22 a group that's going like this and you picked BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 194 1 them. 2 SPEAKER: No. 3 DR. SITTER: Of course, the guys, 4 if you look at who's sitting on top, then the 5 point up there the 90% is staying the same 6 but the players are different. But if you 7 look at a player he's going like this. So at 8 some point, at this point you pick this group 9 of players. So now guaranteed all the guys 10 in those players are now going to go like 11 this, right? So that -- that same group is 12 not going to stay the same group, it's -- it 13 can't, you're picking the maximum, you're 14 picking it when it's at it's peak, it's going 15 to go down. Okay, so. 16 MR. WOOD: This is one of the 17 slides that I think is in your handout. 18 DR. SITTER: Yes. I think I 19 understand it better now. 20 MR. WOOD: And in fact you know, 21 this is let's say the 90% group in the year 22 2001, and you know, it's the same set of BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 195 1 operators, you know who had 86% 4 years 2 before. Their really percent went up and 3 peaked in this year and then came down. 4 DR. SITTER: You're going to see 5 that in every case. 6 MR. WOOD: And you do and in fact 7 that was -- the first, this is the first one 8 we plotted and looked at and said, you know, 9 I can make a general statement about that. 10 DR. SITTER: Yes, I think that's 11 true. I think you could, you know if you -- 12 you know if it were the same model, the 13 random noise that's going on there, then you 14 could probably figure out what that cycle is, 15 I mean or have an idea of what that slope is. 16 What I'm saying is, it's the process of 17 deciding to do a cut off sample in this 18 context, which is causing this pattern that 19 you're seeing. 20 MS. KIRKENDALL: Well, but when we 21 think about doing a cut off sample we will 22 look at it every year and update it. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 196 1 DR. SITTER: That's -- 2 MS. KIRKENDALL: So yes we do have 3 them. 4 DR. SITTER: Yeah, you -- I know 5 you'll update the group but the data you're 6 using to do your estimates is still going to 7 be from 2 years before, because you don't 8 have the data. 9 MS. KIRKENDALL: Right. So we'll 10 be off by about 2 percent. 11 DR. SITTER: Yeah, exactly what 12 John is - - 13 MR. WOOD: The data might be, but 14 the estimate doesn't have to be. 15 MS. KIRKENDALL: Right. 16 DR. SITTER: No I mean in -- I'm 17 sure there are, I mean -- 18 MR. WOOD: I mean as for the 19 estimate. 20 DR. SITTER: No but if the 21 stationarity you're seeing is because of the 22 sort of process -- and you're picking out a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 197 1 maximum group, so that maximum group has some 2 stationarity. 3 MS. KIRKENDALL: Because it's 4 stationary ready. 5 DR. SITTER: What -- what is it any 6 non-stationary process you're going to get a 7 -- let me go. I think that it's stationary, 8 you know, other wise you wouldn't get good 9 estimates. 10 MS. KIRKENDALL: Well the fact that 11 you had the adjustments you made in, the 12 changes over time, which is indicative of the 13 -- 14 MR. WOOD: And if you had great 15 selections at a given time, you'd have to 16 build something like that in or build in the 17 absolute, you have to build in the time 18 change of the distribution. 19 DR. SITTER: What I mean is this 20 that, I think that if you take a stationary 21 process and you pick out a maximum group, 22 that maximum group is going to have a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 198 1 downward trend, it has to. 2 MS. KIRKENDALL: Oh yeah, it has 3 to. 4 DR. SITTER: So I'm not talking 5 about, I'm talking, you're saying that the 6 slope of that line means it's not stationary, 7 yeah, in a certain sense. That's not what I 8 was speaking of actually. 9 MR. WOOD: No, I mean that this is 10 a general phenomenon. 11 DR. SITTER: Yes. 12 MR. WOOD: It's that you take the 13 top group of a -- 14 DR. SITTER: Not sure which would 15 be, I'm not sure whether what, if you sat me 16 down and said should I do something like this 17 or should I do PPS sampling, from what I've 18 seen I can't tell because the PPS isn't 19 correct. 20 MS. KIRKENDALL: But we can always 21 do a better job on simulation. We want to do 22 that in a -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 199 1 DR. SITTER: Yes. 2 MS. KIRKENDALL: And in the 3 estimators have talked about that, given all 4 those. I think the effort is gone into 5 developing the stimulation and this will be 6 an easy ---- come to that. 7 DR. SITTER: But the advantage, the 8 cost of course would be sampling some smaller 9 -- 10 MS. KIRKENDALL: Right. 11 DR. SITTER: Firms. 12 MS. KIRKENDALL: And that's, and I 13 think you would probably end up with a bigger 14 sample presuming it's under that making the 15 sample a bit smaller than we thought we're 16 going to have -- 17 DR. SITTER: I don't know if that's 18 -- 19 MS. KIRKENDALL: We'll get an idea 20 when we do the stimulation. We were thinking 21 closer to 350 companies before we -- because 22 we were sort of moving from the 5% of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 200 1 regional level up to a higher level of 2 accuracy, wanting a higher level of accuracy 3 for each region. So we were at about 250, I 4 don't think anybody was thinking we were 5 going to be able to get much lower than that 6 and with the cut off sample it was like 300, 7 setting on a smaller sample means that we can 8 actually maybe do the work, bigger companies 9 are more cooperative and they are more happy 10 about, less unhappy about sending in the 11 data. 12 MS. NORMAN: Also there is a 13 downward trend that looks like it stayed 14 somewhat relatively stable for at least a 15 year. It will completely plummet the second 16 you hit that 90% group. 17 MR. WOOD: By the way it is a 18 practicable thing to do, to actually go in 19 and project groups of the operators that are 20 on an upward trajectory and those are on a 21 downward trajectory and there are some 22 tremendous things many of which might be BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 201 1 subjects of controversial papers of, why are 2 those large operators declining so fast when 3 a bunch of intermediate size and small size 4 operators are actually increasing their 5 production and it is the behavior of those 6 firms compared to the behavior of other firms 7 which is a different kind of problem but one 8 addressable in policy? 9 MS. KIRKENDALL: But where we are 10 now on this, I think the surveys been 11 approved for January, to begin with January 12 data and we can implement, assuming we have 13 the money to do it but we can implement this 14 methodology, it doesn't mean that we have to 15 stay with it forever but we need something we 16 can do in January. Now we continue to look 17 at it and revise this next time, so, I'm sure 18 you'll hear more about it. 19 MR. WOOD: So yeah, there is a 20 question of is there a sampling procedure 21 which it actually making it more efficient, 22 less operator with the same reliability BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 202 1 survey and we were going to look at that, 2 maybe, could -- 3 MS. KIRKENDALL: I'm not sure you 4 are going to get few operators in a survey, I 5 think if you use random - 6 MR. WOOD: There would actually be 7 more. 8 MS. KIRKENDALL: It could end up 9 being more, because that -- it's certainly 10 something new. 11 MR. WOOD: And then there is the 12 level of effort question, you know, we're 13 dancing around few tens of percent error in 14 individual stake, where we have been, people 15 have in the past said we're off by 4% or 5 or 16 6 or 7 or 8 -- 17 MS. KIRKENDALL: We did that for 18 years. 19 MR. WOOD: And this is well within 20 our target that we started out and it appears 21 to be actually very straight forward in 22 application and relatively easy to explain, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 203 1 and in effect they are, I would suspect that 2 there are the modifications of the sampling 3 proportional to size, would take some fairly 4 experienced people about a year to sort out 5 because now I have questions about them, some 6 of the processes we used also, you know just 7 how, you know, just the way some of the 8 assumptions or approximations were made in 9 the sampling proportional to size and you 10 know, we'd actually use different things, you 11 know, just before -- here we've got a process 12 that is probably getting us and -- 13 MS. KIRKENDALL: We understand it 14 -- 15 MR. WOOD: We could spend a lot of 16 time - 17 MS. KIRKENDALL: We'll see if we 18 can do it. 19 MR. WOOD: No, not that a graduate 20 student couldn't profitable do this but -- 21 DR. SITTER: Well I think there are 22 other things to do just as sensible enough. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 204 1 And one comment is, you know, your hand has a 2 slope and it looks petty -- 3 MR. WOOD: Yes. 4 DR. SITTER: Did you try putting a 5 quadratic term in there because linearing the 6 slope where your slope should really be a 7 quadratic term in the original, like one plus 8 AMP -- 9 MR. WOOD: I can show you these 10 written down in my notes here, yes and 11 exponential and no and this is more, actually 12 I kind of like this, that it's not a 13 constant, in fact the ratio itself is a 14 variable and then the fifth parameter inside 15 the ratio is also a variable. It's a very 16 straightforward way of presenting it, after 17 you've done that, then you can go back and 18 set it with the quadratic and get the same 19 results. And in fact, you get essentially 20 the same thing even if you start fitting the 21 quadratic term. 22 CHAIRMAN BREIDT: Okay, so are we BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 205 1 ready to state the summary. I think you're 2 up Randy. 3 DR. SITTER: That's ridiculous, you 4 know everybody else had twenty minutes to 5 prepare. Give me a break. Every other 6 session finished early and I guess I'm 7 blamed. 8 CHAIRMAN BREIDT: You asked the 9 last question. 10 DR. SITTER: I asked the last 11 question. All right, well I'll give it my 12 best shot. Okay, so this was on monthly 13 natural gas production survey and how to go 14 about doing it. We started with the recap of 15 2004, where the ASA committee proposed using 16 a PPS sample to this situation and some 17 investigation had been done since the 18 stimulation study initially. The stimulation 19 study used psuedo frame based on the EIA-23 20 and we tried using optimum allocation of 21 sample size for stratified SRS as a basis for 22 choosing the sample sizes and then ran a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 206 1 simulation study using the psuedo population 2 to evaluate different estimation procedures. 3 The samples were selected PPS 4 without replacement and different estimators 5 were considered. Some of the estimators, one 6 of the estimators was PPS Horwitz Thompson 7 estimator and I guess -- another were 8 versions of regression estimator regressing 9 on two years previous data essentially, it 10 was regression through the origin I believe 11 just a beta hat multiplier, either with 12 weighted least square or ordinary least 13 squares also considered keeping the certainty 14 units in the calculations of beta hat or not, 15 so there were some questions by the committee 16 on this, I say the committee primary that was 17 me I guess, on the estimators themselves. 18 The results of the stimulations 19 suggested that the regression estimators were 20 negatively biased and the PPS estimators 21 though having no bias and behaving okay 22 didn't have the CV it was supposed to, and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 207 1 had some extreme cases, which would be -- 2 The comment from the committee was 3 that probably the Horwitz Thompson PPS 4 estimator wasn't the best one to look at 5 though, of course it's in the literature and 6 it's unbiased and there's nothing really 7 wrong with it but it does have this property 8 that if you get a, you can have some very 9 badly behaved samples so that might be the 10 result. That might be the cause or problem 11 cases that were observed and that the 12 negative bias of the beta hat is probably due 13 to the fact that the sampling we turned out 14 included in the estimation procedure. 15 When you do do them I suggest that 16 the beta hat or the committee suggested as 17 well that you should not really include 18 certainty in calculation in your beta hat 19 because you don't if the relationship is the 20 same. So the simulation stated, there was 21 some question about it, about some, they can 22 easily be done again to investigate whether BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 208 1 those questions have any bearing on the 2 decisions or any bearing on conclusions about 3 the various estimators, they may or may not. 4 They were other reasons that PPS 5 sampling is not that attractive so it may be 6 that even if committees concerns are 7 realized, they still may decide that this 8 isn't the best, necessarily the best way to 9 go. So then investigation using a 90% cut 10 off sample, that is take the just essentially 11 sample with certainty, the producers which 12 could be attributed to 90% of the production 13 two years previously when looking at today, 14 problem then is that, of course if that 90% 15 production group remains constant over time 16 then in other words they could constantly be 17 attributed to 90% of production then it would 18 be reasonably simple multiplier to take care 19 of the non-sample and in this case a very 20 small number of the five to ten thousand of 21 producers attributed for the 90% of 22 production. So operationally it is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 209 1 attractive. 2 The problem is, is that when you 3 pull the 90% out in say 2001, by 2003 they no 4 longer are attributable to 90% of production. 5 That pattern seems to occur over time and it, 6 the worst comment from the committee that 7 that's probably natural, your picking at a 8 maximum group over a period of time, almost 9 certainly that group will not remain the 10 maximum group because of random perturbation 11 of the process of relying. 12 So I think that the estimation 13 method that was suggested is basically to 14 calibrate that, since there seems to be a 15 general reasonably fit by a linear trend, I 16 guess overtime, the percentage of the 17 production that could be attributed to the 18 maximum, the 90% group over the following two 19 years. One could actually model it and it 20 performs reasonably well. There were some 21 questions asked of the committee, so I've 22 been sort of answering them as we go. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 210 1 Questions were asked about comments for data 2 and approach to modeling or estimating the 3 non-sampled portion. 4 Some of the recommendations on the 5 stimulation study follow that, some questions 6 about the procedure that seems to be the cut 7 off, 90% cut off was how are you going to 8 estimate variation. The answer was by 9 looking at the window of time and seeing how 10 it actually did perform when you get the data 11 in so, that was about the only suggestion. 12 There was a question by the committee about 13 the inclusion probability in the stimulation 14 study and how they were calculated and the 15 answer was that the PPS sample in the 16 stimulation was actually taken systematically 17 and I think there was the other questions 18 were related to outliers and influential 19 observations and the committee did not get to 20 address that issue. 21 CHAIRMAN BREIDT: Thanks Randy, any 22 quick questions or comments. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 211 1 MR. KUNDRA: Can I answer? 2 CHAIRMAN BREIDT: Okay. 3 MR. KUNDRA: I think the problem 4 with PPS was that the frame which was used to 5 select the PPS sample, when they went to the 6 next year, most of the, I don't have any 7 problem with the certain sample, the problem 8 will be non-certain sample because most of 9 the units were not found in the following 10 year. The problem is lying with the frame 11 because the units which have been selected in 12 the using the frame 2000, if you go to 2001 13 or 2002, you don't find those operators over 14 there. 15 So that's one of the problems 16 because that's why they were getting some bad 17 results and so for the variances concerned 18 because you could not substitute, or you 19 could not make any sort of assessment for 20 those varied floors or patterns which are 21 missing from the, for the year which the 22 materials were use. That's one of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 212 1 reasons the frame is not very stable. 2 DR. SITTER: But that wouldn't 3 arise itself in the stimulation study? 4 MR. KUNDRA: I don't know. I'm 5 talking about the PPS sample. 6 DR. SITTER: No, I understand that 7 but that -- 8 MR. KUNDRA: It will rise in the 9 simulation also too. 10 DR. SITTER: It arose in the 11 stimulation and was a cause in the 12 simulation? It's quite a bit different. 13 MS. KIRKENDALL: Well except in a 14 simulation he had, I mean, there were, zeros 15 weren't in there. We took -- 16 DR. SITTER: Yeah, you took out all 17 the unmatched. 18 MR. KUNDRA: We took out all the 19 unmatched. 20 MS. KIRKENDALL: We took the 21 outmatched out, so that -- 22 DR. SITTER: So I think what the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 213 1 committee is saying is, is that the 2 simulation study is one aspect of why you may 3 want to decide on two things. Stimulation 4 study needs to be done, something's need to 5 be done with it, to make sure that you're 6 properly evaluating that aspect. But it's 7 not the only aspect of whether you will or 8 won't decide to go to this sampling frame, 9 and it maybe that the operational things such 10 as a change in frame may overwhelm the 11 efficiency issues. 12 MR. KUNDRA: That is one of the 13 main things. 14 CHAIRMAN BREIDT: Thanks Joe. 15 MR. SEDRANSK: I'll just try to 16 reduce some confusion. There should've been 17 a line on the stimulation results that had a 18 line under the PPS estimator. All of the 19 remaining ones were model- based or were 20 intended to be model-based estimators, so in 21 fact the fact that they didn't do well, just 22 that the model didn't fit terribly well, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 214 1 rather than anything about the selection and 2 that has a lot to do with what Indar just 3 said. 4 I looked with an intern over the 5 summer at these constructed data sets, they 6 do have an underlying linear regression 7 through the origin but they are very bumpy, 8 really bumpy. Ones at the lower end and 9 especially in the middle, which we're really 10 amazed at, so basically what I think we're 11 seeing what the, you know, sort of poor 12 performance for all those weighted least 13 square estimates is in fact, that if you 14 really wanted to perceive that way you need 15 to more, I'll use the word a more 16 sophisticated estimator. 17 Now I wouldn't continue doing this, 18 I was talking to John at lunch and John is 19 concerned about this data set, you know, the 20 one that we can, you know sort of the data 21 that was used for it and I think this is a 22 little bit suspect, in other words, the data BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 215 1 set that was used for the stimulation is 2 questionable but basically a confusion was 3 that the ones below, the reason I said that 4 there should be a line, is all those weighted 5 least squares were intended as model-based 6 estimators and -- 7 DR. SITTER: so there were, their 8 performance was evaluated as model based? 9 MR. SEDRANSK: No, no, no, they 10 were, no they were evaluated as, I mean, they 11 were given a rigorous, a more rigorous test 12 to the sets. But they weren't - 13 DR. SITTER: The thing is that, if 14 you, you know, if you're getting stimulated 15 biases and the stimulation is based on the 16 repeated sample -- 17 MR. SEDRANSK: Sample, that's 18 right, yeah they are. 19 DR. SITTER: Then the bias is, the 20 bias are actually measured in -- 21 MR. SEDRANSK: That's absolutely 22 right. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 216 1 DR. SITTER: Right. 2 MR. SEDRANSK: So that's correct. 3 DR. SITTER: Yeah. In fact and one 4 of the interpretations was that they were no 5 good because they were biased. 6 MR. SEDRANSK: Yeah, the other 7 question which maybe Preston can ask, I'm not 8 too sure we got too many of those really 9 small units, this is related to Indar's 10 comment, I'm not too sure we got too many of 11 those small units. Maybe it's PPS sample, so 12 in fact it wouldn't have fixed this problem 13 up if you know if that group is different 14 than the top. That I can't answer, do you 15 know Preston off hand what percentage, you 16 know, how many small units we got? 17 MR. McDOWNEY: I can't answer that 18 MR. SEDRANSK: Yeah I know, anyway 19 that's all. 20 MS. KIRKENDALL: There probably 21 weren't too many -- 22 MR. McDOWNEY: No. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 217 1 MS. KIRKENDALL: Because it went to 2 other EIA-23 survey, which provided the data 3 for 2000 and we matched to the EIA-23. 4 MR. SEDRANSK: And we matched to 5 the 23 frame in 2002, so I -- 6 MS. KIRKENDALL: The frame would 7 have had more small companies so we matched 8 the data from the EIA-23 and that was more 9 big companies. 10 MR. WOOD: It also reported a 11 couple of thousand. 12 SPEAKER: Let me finish. 13 MS. KIRKENDALL: Okay. 14 CHAIRMAN BREIDT: Let me jump in 15 here now, or we'll go too long, you guys can 16 continue your break out session in the break 17 if you like. So I need to ask those who did 18 not introduce themselves this morning to go 19 ahead and do so now, so Bill will pass around 20 the mike and if you could just give your name 21 and your affiliation please. 22 MR. WEINIG: Might I also add that BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 218 1 if you are sitting under a speaker this thing 2 will squeal like the dickens so, if that's 3 the case, please use the other mike. 4 SPEAKER: That for example is a 5 speak. 6 DR. SITTER: Randy Sitter, Simon 7 Frazer University. 8 MR. WEINIG: Johnny? 9 MR. BLAIR: Johnny Blair, Abt 10 Associates. 11 MR. MERRRIAM: I'm Rob Merriam, 12 from Z INC. 13 MS. COX: Brenda Cox, from 14 Battelle. 15 MR. KUNDRA: Indujit Kundra, from 16 SMG. 17 MR. LANEY: George Laney from 18 Temple. 19 MS. NORMAN: Karen Norman, SMG. 20 MR. WILSON: Nathan Wilson, EIA. 21 MR. TITA: Michael Tita, SAIC. 22 MS. OLSON: Irene Olson, EIA. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 219 1 MS. ULREY: Peri Ulrey, NGSA. 2 MS. CARROLL: Carolyn Carroll, Stat 3 Tech. 4 MR. CASTLEBURY: Jay Castlebury, 5 EIA. 6 MR. LU: Rueypyng Lu, EIA. 7 MR. WOOD: While they are waiting, 8 John Wood. 9 MR. HALE: Doug Hale, EIA. 10 MR. MAGNUS: Howard Magnus, EIA. 11 MR. TRAPPMANN: Bill Trappmann, 12 EIA. 13 MR. KASS: Roy Kass, EIA. 14 MS. GEISERT: Lynn Geisert Z INC. 15 CHAIRMAN BREIDT: Okay, so I think 16 that went around. Now I introduce Neha 17 Khanna to summarize the other breakout 18 session. 19 MS. KHANNA: Thank you. Our 20 session focused on methods to assess the 21 solution that the NEMS models produces and 22 one of the issues of the NEMS is that it BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 220 1 produces an enormous amount of output data so 2 you'll have to, was it understood there are 3 200 papers of results that span several 4 hundred pages and part of the question is, 5 how do you analyze the outputs in a quick but 6 rigorous manner and that was the focus of our 7 session. 8 Professor Lady (?), what he did, 9 was to write -- wrote a graphics interface to 10 facilitate comparisons of solutions and he 11 used three different diagnostic checks as 12 well. These were, what he calls it, his 13 "monotonicity" checks to see if that, the 14 number of types solutions trajectory's 15 actually crossed each other, he used -- he 16 used the forecast error to see for example, 17 if you were forecasting to solve the intent 18 value, if you forecasted it from 1998, you 19 should have much greater variability than if 20 you forecast it from 2004, to see again is 21 NEMS producing you know, as forecast is 22 getting more consistent as you get close to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 221 1 the time and then he also to looks to see, 2 just how variable the forecast were in 3 general and what he found was, after doing 4 that he took a large, he took the NEMS 5 solution and actually tried to approximate 6 the NEMS solution by a simple linear 7 regression model to try and you know, can we 8 get a handle for how the solution have 9 changed over different runs of the NEMS model 10 as there is pieces of assumption of margin 11 for change to what time it's different 12 solutions. 13 And what he found was in fact, that 14 was a reasonable first proclamation of the 15 NEMS solution and that many of the results 16 could be represented by linear 17 approximations, also I was thinking that 18 couple of cases you did use columnar 19 regressions because there was a great 20 non-linearity involved. The basic questions 21 for the committee there was, I think four or 22 five of the questions, we had fixed this, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 222 1 that we decided some of them says the same 2 thing. 3 The basic question is, is this a 4 sensible way of approximating the NEMS 5 solution and I understand, my feeling as the 6 committee said it depends on what, what 7 you're trying to use the solutions for. If 8 you are trying to see are NEMS solutions 9 really changing over different runs of the 10 models yes, it seems to be a good way because 11 it sees some sort of blips in the estimated 12 cooperation of your regression. You could 13 look at why do those blips occur, so if 14 that's what you want to do, this seems to be 15 a sensible approach and we have some common 16 sense on how to improve that and I'll come to 17 those in a minute, and the other side was if 18 you want to see are NEMS results generally 19 sensible, in the sense of you know, all the 20 results that NEMS is providing you, there are 21 comparable with like say results coming out 22 from some other independent models, then you BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 223 1 might want to impose on more economic type 2 structural model on the NEMS solutions to see 3 you know, the kind of results you'd get for 4 example, the price elasticity coming out of 5 NEMS are they or if it is in them, are they 6 really sets of comparable with or consistent 7 with other price elasticity elsewhere? 8 So there were some that directly 9 went to the question of you know, should we 10 look at goodness of fit or should we only 11 look at the possibility of specifications and 12 then there the committee felt it depends on 13 what you are trying to do. As far as trying 14 to improve on some of these, trying to 15 improve upon this approach, at first 16 committee felt that one of the things that we 17 could do was, to develop, develop confidence 18 intervals around predications and see if the 19 new predictions from the NEMS fell within the 20 confidence span then we're pretty much okay, 21 if it fell outside the confidence span then 22 well, this is something interesting and we BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 224 1 need to look into it and I should say 2 actually at the outset, all of this was done 3 to demonstrate it. 4 There is still work in progress and 5 when you find the results that "differ," it 6 should excite interest, it should not 7 necessarily mean that there's something wrong 8 in NEMS because there could be a fundamental 9 change in, a fundamental and reasonable 10 change in an assumption that drives the 11 different results, so this is the just an 12 attempt to find out what's going on and, is 13 something changing? We've also felt that the 14 NEMS approximation should be "stand alone", 15 or free standing so that anyone who is using 16 NEMS and wanted to analyze the results could 17 do it and we didn't need sort of an expert to 18 sit by you to know, how to actually analyze 19 or to approximate the NEMS solution. 20 It was also felt that some of the 21 scenarios that it maybe worthwhile thinking 22 about running some NEMS scenarios, which are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 225 1 maybe now not been run but keeping in mind 2 that we are going to use something like this 3 in the near approximation to analyze the NEMS 4 result in the future. So you know, because 5 of the great heterogeneity and sanguinity 6 issues involved in these models, it may be 7 nice to have certain drives or shift that 8 will help us to identify some of those 9 equations and that maybe something worth 10 considering and we had some discussion about 11 that. 12 We also felt that actual data 13 become available to overlay the NEMS 14 projections with the actual data to see how 15 we were doing. In terms of uncovering 16 programming errors or inappropriate 17 solutions, one member from in audience 18 suggested that we should look at feedback 19 groups and I believe more and more that is 20 actually being done, so that was a good 21 thing. There was one question that we didn't 22 actually address, you all just nodded and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 226 1 then left it, and that was what procedures 2 would be appropriate for using NEMS 3 approximation to facilitate convergence of 4 the NEMS -- and we talked about how the 5 linear approximations might be used to 6 suggest starting value for NEMS but we didn't 7 really say anything beyond that, it's just 8 that I said, "yes, that's good, it seems 9 reasonable". 10 Another thing that came out of the 11 linear approximation to NEMS was, it could be 12 used in fact to partition out the forecasting 13 areas into what part of the those areas are 14 actually due to just differences in forecast 15 and what part of those areas are actually due 16 to just some difference in exogenous 17 variables, so if you have a NEMS projection 18 you use, you actually put in the actual data 19 that came into the regression model and then 20 see how much of it remains unexplained. 21 So you could actually figure out, 22 what was the source of the differences and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 227 1 that would be useful tool to use, I believe 2 that's that the summary to, the kind of free 3 wheeling discussions I'm sure I've left out 4 some pieces here and if somebody wants to add 5 to it, I'll be very happy. 6 CHAIRMAN BREIDT: Maybe just one 7 quick comment on the convergence issue, it's 8 kind of like it wasn't so critical to have -- 9 MS. KHANNA: Oh, Yes. 10 CHAIRMAN BREIDT: Good starting 11 values. 12 MS. KHANNA: Thank you. 13 CHAIRMAN BREIDT: Any questions or 14 comments. Okay so I won't need much time for 15 that. I'll just turn it over to Nancy, who 16 will give an introduction to Program 17 Assessment Rating Tool, PART program 18 evaluation. 19 SPEAKER: Why don't we do this and 20 then we take a break. 21 SPEAKER: Yes. 22 MS. KIRKENDALL: Okay, this is an BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 228 1 introduction basically to the rest of the 2 afternoon and I'm going to give a sort of 3 introduction to the break out sessions that 4 we have scheduled, we've kind of stuck it in 5 because we figured people wouldn't understand 6 what we are really talking about in the break 7 out sessions, if we didn't give you this 8 little introduction. There's a new tool, the 9 Performance Assessment-Rating Tool is a new, 10 well it's three years old. It's used by the 11 Office Management Budget. There're 12 completing it for all federal agencies. We 13 will actually, we, EIA were actually 14 fortunate that we weren't hit with having to 15 do it for the first two years, this is the 16 third year of the program and we did have to 17 do it this year, so this was our first year 18 of introduction to this PART program. 19 Other statistical agencies have 20 been evaluated before we were and of course I 21 wouldn't be too is trying to figure out how 22 to actually implement this, so even though BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 229 1 BOS and BEA have gotten good ratings in the 2 past, not everybody does and most have been 3 for education statistics got a bad rating one 4 year because the person who did the, was 5 interviewed, didn't take it seriously and 6 didn't answer the questions and didn't 7 prepare. But other than being serious about 8 it and they post the results on the website, 9 so you can look and see what everybody, the 10 scores everybody is getting, so it can be 11 viewed in the website there one way. 12 There are four areas in PART, 13 everybody is scored under these four areas 14 and the first one is on program, purpose and 15 design as they show up there, EIA did very 16 well on that so that doesn't identify any 17 issues for us. The second one is, strategic 18 planning and performance measures. We didn't 19 do as well there, partly because we had just 20 changed our performance measures OMB agreed 21 that we had better measures that we were 22 aiming for now, but they couldn't get us full BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 230 1 credit for it because we didn't actually have 2 baseline values and results yet. 3 The third area of program 4 management again, EIA did quite well, so we 5 don't have any particular concerns there and 6 on program results and accountability is an 7 area that we're struggling with, we didn't do 8 very well with that one either. So the 9 challenges I've got listed, none of them is 10 quite as much of a challenge and our managers 11 are new, so we don't have results yet well 12 you know, over time that will change and 13 we'll have results but the last one is, one 14 that's more serious and that is that we don't 15 have any external evaluations. 16 So the evaluation question shows up 17 in two places in PART. In fact it shows up 18 in that first area that we did not do well 19 and in the second area that we did not do 20 well in. In the first area they say they 21 want, regular independent evaluations and 22 they should be used to support program BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 231 1 improvement, assess effectiveness and 2 relevance and when we answered the question 3 requirement to the ASA committee and the fact 4 that we talked to you when we tried to get 5 advice on our technical problems and then we 6 try to use the advice to fix things. 7 We told them about our independent 8 extra review program, we've had lots of 9 things reviewed by independent experts and we 10 try to improve things that way and all of our 11 models are in the public domain and they were 12 unconvinced and they said, "no". So we're 13 actually and well this one lives in the area 14 where performance measures are important too 15 and so they really couldn't give us anything 16 because those are preliminary. 17 In the second one, under program 18 results and accountability, this is a very 19 similar question and they actually refer you 20 back to the first one. They say you want 21 independent evaluations indicate program is 22 effect and it is achieving results. So we BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 232 1 talked about the American Customer 2 Satisfaction Index, we talked to you about 3 that earlier today. We've done small studies 4 of the impact of the EIA's data on markets, 5 you've seen charts where our weekly data are 6 released and the price on the market goes 7 haywire. 8 And so they gave us a small extend 9 on that, so it's better than a "no" but it's 10 not really great. And the big criticism is 11 that, we don't have independent evaluations 12 of our entire program. So this refers back 13 to the question, we'll need to come up with a 14 way of doing independent program evaluation. 15 Independent means mostly outside of EIA. 16 Although you could say that there is some 17 independence in my office if we do an 18 evaluation of a program office, they still 19 might not think that's quite as independent 20 as you like but we can certainly be involved 21 to some extent. 22 And the purpose of these BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 233 1 evaluations should be to identify program 2 improvements. The more defectiveness appears 3 in many places in their, in their discussions 4 so we need to somehow assess effectiveness, 5 we need to assess relevance and we need to 6 identify program impact. On identifying 7 impact, the guidelines suggest that you 8 should do this is the guidelines was prepared 9 by OMB. The only idea they have is to do 10 random controlled trials. I haven't got a 11 clue how you would do that to effect, to 12 estimate the impact of a data program. But 13 they say, you can look for the data from 14 someone, give to others I guess. 15 Anyway the guidance also 16 acknowledges that this is not always feasible 17 but it doesn't really give you much of an 18 effort of a idea of what you could do and so 19 that we have scheduled for you this 20 afternoon, after the break is that, Brenda 21 Cox is working under contract with us and she 22 is trying to develop a way of doing a program BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 234 1 evaluation that perhaps we could give to a 2 contractor or at one point there was a 3 discussion at the interagency council on 4 statistical policy and national agricultural 5 statistic service and the Director suggested 6 that maybe statistical agencies could 7 evaluate each other, so you could have a team 8 of people from agencies that could go and do 9 evaluations for other agencies and the only 10 thing about that is that, you have to have 11 some kind of a template. 12 You'd have to be able to give them, 13 what exactly are you going to do and then 14 maybe they could go off and do it. Sometimes 15 we're still and now this is of interest more 16 than to just EIA because all government 17 agencies have to answer these questions, so 18 we're not the only ones struggling with it. 19 So Brenda is going to talk about her efforts 20 to start looking at developing a template, 21 where an approach that could be used to 22 evaluate survey programs. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 235 1 Then we were thinking in particular 2 about families of survey because in EIA we 3 talk about our petroleum marketing package, 4 it goes out to find out basically about 5 prices in the petroleum area, to evaluate the 6 markets but we have our petroleum supply 7 package. It's a family of survey that 8 captures petroleum supply information. The 9 natural gas program gets information about 10 natural gas. So we have these little you 11 know, groups or packages of programs and it 12 seems like, you know, it might be worthwhile 13 to assess families of programs and you could 14 call that a program evaluation. 15 So that's what Brenda is going to 16 talk about and then Doug Hale is going to be 17 meeting the other session. He's going to be 18 asking about, how can we get extra 19 evaluations for models and analysis and of 20 course he's starting from what we know, we 21 know about the independent expert review 22 program and so on and then afterwards we're BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 236 1 going to come back together and have a more 2 detailed talk to see if there's anyway the 3 that you guys, the energy committee could 4 help us do program evaluation. 5 So if there is something that you 6 could do, that we could say, "see we got this 7 evaluation or this input from our ASA energy 8 committee" and that's a recommendation that 9 we could count as external program evaluation 10 information, so it's a question, how do you 11 phrase the question, what questions do you 12 ask, now truly we have to make it clear to 13 everybody that you're using what you learn, 14 but that's okay. That's what makes it useful 15 to give the advice right. 16 DR. SITTER: You said that BOS and 17 the Bureau of sensors got good scores, what 18 are they doing? 19 MS. KIRKENDALL: Well, they were 20 early in the program and things are changing 21 over time. They are much more stringent now 22 than they were last year and the year before BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 237 1 but one of the things that's been sighted as 2 a good external program evaluation was the 3 Voskin initiative that worked with the CPI, 4 for the bureau of labor statistics. There 5 was, I guess, they were five academics who 6 made some specific recommendations that was 7 charted by the Senate, I guess, to look at 8 the CPI and that's a very important indicator 9 used in many government programs and they 10 made various specific recommendations as to 11 what BLS should do to fix the CPI. 12 And they actually went ahead and 13 they have implemented many of those changes. 14 I think that was, that ended up in 15 recommending that they expand the service 16 sector. I could be wrong about that but -- 17 DR. SITTER: But, I've been here 18 for five years, we make recommendations like 19 that all the time. 20 MS. KIRKENDALL: Well, you know and 21 maybe you do and then the question is, how do 22 we package it. You know because there are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 238 1 other pieces that's just, we are reporting to 2 OMB on this and we were not evidently 3 convincing enough in what we said. 4 DR. NEERCHAL: A quick question, 5 you know, is "program" an official unit or 6 when they did the program evaluation, do they 7 mean like an office by office or department 8 by department, what do they have in mind? 9 MS. KIRKENDALL: That's a good 10 question. I think for their purposes, EIA 11 sells out one program evaluation, with the 12 energy data program. I think for our 13 purposes, we don't have to, I don't think you 14 can get anything useful out of looking at 15 something that big but you can look at the 16 pieces. That's why I thought you know, we 17 could focus on various survey programs or our 18 analysis program or maybe a long-term 19 forecasting program. So, Tom do you have any 20 input, Tom Broene is, was intimately involved 21 in putting together our PART report and 22 working with OMB through this exercise. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 239 1 MR. BROENE: It has to cover the 2 entire agency and so I mean, it's got to be 3 systematic, it's got to be recurring and you 4 have to actually use the results and have 5 proof of that, so I mean they can report on 6 the CPI, this minute things like, how did you 7 go waiting to take our different type of 8 average and everything from -- spend more 9 time pricing, features of cellular phones, 10 lifetime pricing, the characteristics of 11 bananas and other things on the servicing -- 12 they covered everything and it's five guys, 13 did in 18 months. 14 MS. KIRKENDALL: But CPI is not a 15 BOS. 16 MR. BROENE: Right, that's true, I 17 don't know what they did for the rest of BOS. 18 DR. NEERCHAL: The reason I am 19 saying that I think, that many of these 20 academics here will recognize this thing, you 21 know, we have to go through something called 22 program review. External panel will come and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 240 1 look at this statistic, the department, 2 typically department by department and we go 3 through, they have a 7 year cycle office and 4 this being initiated you know, officially 5 this is coming all the way from the board of 6 regions but somebody on campus, from Dean's 7 office is usually in charge of it and I think 8 that independence aspect is really assured 9 because the Dean's office is involved in 10 inviting the outside panel, even though you 11 suggest the panel, they decide and they 12 correspond with them, that, that should, that 13 makes them feel comfortable that they are 14 really independent, even though we know who's 15 coming and so on right, so it seems to me 16 that if, that is my followed question would 17 be, how close the OMB wants to be involved 18 with the evaluation process because if you do 19 an evaluation on your own, like you are doing 20 an intent right now, they don't feel that it 21 is independent enough. 22 It maybe that they wanted more BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 241 1 closer role, you know, and they want to 2 closely monitor it and make sure that it's 3 independent, you know, and that's why if it 4 is and I'm really surprised that they are 5 coming down to the CPI you know, how you 6 know, they are drilling down all the way -- 7 I'm really surprised that they have that kind 8 of time to, you had your programs that is 9 left, surprise. 10 MS. KIRKENDALL: Well now OMB was 11 not involved with that and now it's a long 12 time ago too, but that's what BLS can cite 13 and an external program evaluation you know, 14 and OMB thinks that was fine, because you 15 know, it was chartered by the Senator, 16 whoever, so chartered by an external body. 17 DR. NEERCHAL: I think that the 18 independence, maybe that is the assurance and 19 somehow they want that to be you know, 20 demonstrably independent, right. 21 MS. KIRKENDALL: Uh-huh, Neha. 22 MS. KHANNA: Yes, I think you BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 242 1 actually hit the nail on the head when you 2 talked to him and used the word strategy. I 3 think a lot of what this committee does 4 should constitute an independent review and 5 maybe part of your responsibility should just 6 be, you know, over the twenty or thirty years 7 this committee has been in existence to chart 8 out some of the recommendations, major 9 recommendations that have come out of this 10 committee and actually effected the way a 11 particular program or the EIA as a whole has 12 operated or you know, done data collection or 13 is this modeling everything, what better -- 14 MS. KIRKENDALL: Actually I'm 15 wonder whether Cal's history that he put 16 together might have some information, Cal 17 Kent did that history of the ASA committee 18 and we can take and look and see if there is 19 anything we can steal, yeah and -- 20 DR. HENGARTNER: Los Alamos peer 21 gets reviewed once a year but the one thing 22 that I think, that what you wanted BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 243 1 independent and the one thing that we can 2 always looked at is, essentially publication, 3 the things that re out in the open I mean if 4 you're software's out in the open, if data 5 gets in the open, it gets peer reviewed, 6 that's independence review, if it's wrong, 7 well, the customers are going to let you 8 know. It is true. 9 DR. BURTON: Absolutely. 10 DR. HENGARTNER: And what I don't 11 understand is how OMB doesn't get that. I 12 mean because I think, if everything is in the 13 open you have to still the review 14 publications, that's more than any other 15 places you have and it's mainly a matter of 16 convincing OMB that, this is how things work, 17 more than trying to glamour up something 18 else. 19 MS. KIRKENDALL: Well I agree that 20 especially based on this discussion we can do 21 more on packaging what we have already have 22 and showing it and maybe we'll learn better BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 244 1 how to say, how to describe what we're doing 2 in a right way, that would be a bit more 3 convincing, but there are other things that 4 you know, when I was talking to Brenda about 5 this is external survey evaluation thing, but 6 we've haven't done for example, with our 7 family of surveys, is to really look from an 8 independent point of view and see how they 9 fit together you know, does it make sense 10 that you know, we have exact the same survey 11 goes overtime or the different people answer 12 different questions the same way, is there a 13 better way of structuring the surveys and 14 still capturing the information we need. 15 We do that every three years when 16 we put the forms clearance parents package 17 together but we don't always step back and 18 take a fresh look at it because we know how 19 we've done it and it's hard to see it a 20 different way, when you know this is how 21 you've done it, so it could be that some of, 22 a good outside look at how we operationalize BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 245 1 some of our programs, would be helpful, it 2 might provide some new insights, you know, so 3 that I can see that you might be able to get 4 some benefits from a good outside assessment 5 occasionally, if you can find the right way 6 to ask the questions which is other 7 challenge. 8 MS. KIRKENDALL: Okay. 9 CHAIRMAN BREIDT: We are about 20 10 minutes into fifteen-minute break, so 11 everybody can just grab our snacks and head 12 to the break out session directly. -- 4a 13 MS. KIRKENDALL: Then we need just 14 start the session really 10 minutes late I 15 guess. I'd like to introduce Brenda Cox. 16 She's working with us to come up with a 17 survey program evaluation process. So 18 Brenda. 19 MS. COX: Thank you. I think this 20 is a joint endeavor between myself and Nancy. 21 We've been talking about what do you do to 22 evaluate a survey program. I should say this BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 246 1 research is being conducted under a contract 2 that Battelle has with Z Inc. and we 3 appreciate it, Rob ---- and other Z Inc. 4 people are here as well. This is coming as 5 the performance assessment rating tool, this 6 is my interpretation of what that does. 7 It encourages rigorous assessments 8 of government programs. That doesn't mean, 9 are you doing okay, it means can you do 10 better. That would be the way that I would 11 say. So you may be doing a fine job, but 12 there may be some little way in which you can 13 improve what you're doing. So we continue to 14 boost the quality of government programs, and 15 it provided the impetus to this 16 investigation, in which I'm going to design 17 templates that can be used to evaluate 18 individual surveys, but more importantly 19 government programs that are composed of a 20 family of surveys. 21 EIA does a number of surveys, a 22 large number of surveys. They tend to be BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 247 1 small. It's not like the agencies for 2 Healthcare Research and Quality, which does 3 the math or that cost tens of millions of 4 dollars. Whether they -- their programs tend 5 to be smaller surveys that are, that put 6 together in programs of surveys. Now the 7 objective of what we're going, what I'm going 8 to be doing is first to develop a survey 9 evaluation template and that part has been 10 done, and then go from there to develop a 11 survey program evaluation template. And a 12 survey program is a collection of surveys on 13 a common topic that is conceptually linked. 14 The petroleum marketing survey is one 15 example. 16 And we're going to actually test 17 the concept of the survey evaluation template 18 that's been developed on the 11 surveys that 19 compose the petroleum marketing program. And 20 then the next step will be to test the 21 concept of the survey program evaluation 22 template on that same program. Now, why am I BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 248 1 not feeling anything? 2 SPEAKER: It's once again -- 3 MS. COX: Oh, it's been brought in 4 one at a time. Okay, so the -- the 5 components of the survey evaluation template 6 and I should say the paper that I submitted 7 has the actual template in the appendix. So 8 it's the -- it's the template as it exists 9 now, it is a work in progress. Since we're 10 filling this out almost like a form that 11 would be completed, we start with the name, a 12 brief description of the survey, the survey 13 objectives. Now the survey objectives here 14 have to be what's driving the design and that 15 means you have to say what domains, the 16 domains for which you need separate 17 estimation. 18 If this is a survey that has a 19 strong geographic component and you want 20 separate estimation for each of the regions 21 and divisions of the United States, you have 22 to say so. Because the design will be, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 249 1 that's part of evaluating the design. The 2 target population and this is -- it has to be 3 extremely explicit because you actually 4 review is this an appropriate target 5 population for a study. Some people 6 misunderstand what a target population is, so 7 they define the target population as the 8 population that it's easy to survey as 9 opposed to the population that people are 10 actually going to make inferences about. 11 So for a telephone survey they may 12 define the population to be people who can 13 hear. Well those are who you survey, but 14 that's not the target population because 15 generally you want to make inferences to the 16 entire population of the United States. 17 There's also the sampling frame and it's 18 characteristics. We look at the quality of 19 the frame like, coverage of the target 20 population, whether there is overlap, 21 multiplicity, what's done to really create 22 the sampling frame and make it the best BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 250 1 sampling frame. 2 The sample design it says, I could 3 tell when I came in that you all were looking 4 at, that this group was looking at sample 5 design characteristics particularly for EIA 6 surveys which tend to be economic surveys for 7 which some particular company can drive an 8 estimate because of their very size. And 9 even if they're not driving the estimate for 10 the nation as a whole, they may be driving 11 the estimate for the northeast for instance, 12 because of their size. Other components of 13 the survey evaluation template, the way I've 14 defined it, I should say, this is what I've 15 defined, I think I would look at if I'm 16 evaluating a survey. 17 And I should also say that I tried 18 to think of every possible thing you could 19 look at. Some of them may not be appropriate 20 for every survey, but it was just trying to 21 get everything out there, that you would look 22 at or consider. Sample allocation is another BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 251 1 issue and that you were addressing in fact 2 this morning for the survey. It's very 3 common to create certain selections for very 4 large businesses. Data collection, we need 5 to look at the aspects of data collection to 6 see, if it's both rigorous and efficient in 7 the way it's being done. Editing and 8 implication is another issue for business 9 surveys. 10 There's a feeling that surveys that 11 -- economic surveys added the data too much. 12 Now, I'm not talking EIA here, I'm saying 13 across economic surveys there are people who 14 say, we do too much editing. And for 15 economic surveys the amount of editing that's 16 done is usually a great deal greater than for 17 person surveys. Part of the reason is, the 18 data items tend to be interrelated and you 19 want consistency. 20 If you're doing a balance sheet, 21 you want your assets to be equal to your 22 liabilities plus equity. So there's certain BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 252 1 relationships that need to hold. Weighting, 2 we need to look at the weighting process to 3 see, first is it following the sample design. 4 Secondly, how is it adjusting for 5 non-response, because non-response happens in 6 all of their surveys, and is it appropriately 7 capturing the best, doing the best job of 8 minimizing non-response biased and the 9 effective under-coverage. So we look at 10 things like post-stratification, non-response 11 adjustments etcetera. 12 For the data analysis, this is 13 looking at the way the data are used. And it 14 could be used as input for models, like the 15 NEMS model, it could be used in data reports, 16 etcetera. So looking at the data analysis to 17 say how are the data being used, there might 18 be a question of how they should be used, but 19 also I find the data analysis informs me to 20 as to what the sample design really should 21 be. I looked at a survey recently where, 22 it's like one out of every 2 tables and all BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 253 1 of their reports had a row or a column based 2 upon years since degree or age. In other 3 words two variables that were closely related 4 to the age of the person. This was a 5 population survey. 6 Although age was available, and 7 years since degree were available for use in 8 the sampling process, they weren't used. So 9 that tells you something. Finally, the 10 summary of findings, because you want to do 11 more in the template than just evaluate the 12 survey and say what are you doing. What's 13 good about what you're doing, what should you 14 improve? Now I've emphasized and I think 15 it's very important what's good about what's 16 being done. The people who -- the directors, 17 the managers of these programs, you need to 18 know this is something you're doing right and 19 you need to continue doing it. This is a 20 good idea. 21 They also though need to know areas 22 where improvements could be made. It might BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 254 1 be a redefinition of the stratification 2 variables, extra sorting that might be done 3 to give more control over particular 4 variables. Problems that you're seeing with 5 respect to the weighting procedures, 6 whatever, it needs to be there. And the fact 7 that you're suggesting an area for 8 improvement doesn't mean that the survey is 9 bad. It just means, it's not 100% perfect 10 and there's usually always areas for 11 improvement. So that's kind of, where we're 12 going with the survey evaluation template. 13 Now, when we were discussing how to evaluate 14 a program I said, I don't see how you can 15 evaluate a program until you evaluate the 16 individual surveys that are forming this 17 program. 18 Because you have to see the 19 component parts and then the next step is how 20 do these fit together. So we began with the 21 summary evaluation of each survey and then go 22 to the survey evaluation template, which BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 255 1 would be used to profile each the survey. 2 Now the individual survey template is 3 available in the paper I wrote. The survey 4 evaluation template has not been done yet. 5 Some of the thoughts for the survey 6 evaluate -- for the program evaluation 7 template is to look at issues like, which 8 respondents report to which survey and what 9 data requests are made of them. The reason 10 for doing this is because even though the 11 populations may look really different you 12 could be going to the same companies over and 13 over again and imposing slightly different or 14 maybe even the same questions to them. So 15 you really need to look at the issue of the 16 respondents for each individual survey. 17 And this can be -- this can be a 18 special issue for economic surveys because 19 large businesses tend to be taken for every 20 survey. You want to identify unnecessary 21 burden on survey respondents across survey 22 components and whether specific BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 256 1 subpopulations are overburdened. And you 2 want to determine if the survey program needs 3 restructuring. Restructuring might be needed 4 because you're capturing data in repetitive 5 ways or you're missing some component that 6 you should be collecting data on. Or your 7 time sequence needs improvement. There's 8 different ways that this could happen. 9 Other areas that could be looked 10 at, is to determine how the data across 11 surveys are used in analyses and presented to 12 the public. And again this is to give you a 13 sense for what you should be doing. Not only 14 for data analysis but what you should be 15 doing in your design and reporting. For 16 instance the use made of the data, may be 17 such that it's used for, very timely needs. 18 For that tells you, you need to 19 look at the time dimension of the survey. We 20 want to delaminate the relationship cross 21 design between the data reporting and the 22 sample design. So we need to look just to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 257 1 see what's going on across these surveys. 2 Then there is the issue of synergy across 3 surveys. I'll give you an example from 4 another federal program. This is the 5 scientists and engineers statistical data 6 system, and I've worked on that for a number 7 of years. 8 About 10 years ago, they were 9 evaluated by the committee on national 10 statistics. And one of the recommendations 11 that the committee gave them was you do three 12 surveys of scientists and engineers. You do 13 a survey of doctoral recipients, you do a 14 survey of recent college grads, and you do a 15 survey of college grads that are not so 16 recent. And they said, you don't put these 17 surveys together. You really cover the 18 entire population of scientists and 19 engineers, but you're using different 20 questionnaires and you are not allowing for 21 their integration. So one of their strong 22 recommendations is redesigning surveys so BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 258 1 that they can be integrated to create one 2 unified database. And the National Science 3 Foundation has done that. 4 So that's an example of building 5 synergy across surveys. The system now is so 6 much better than the part because before the 7 college, the doctoral recipients was only US 8 earned doctoral recipients. But by bringing 9 in the other survey of not so recent grads 10 which included foreign graduates, foreign 11 doctorates they could represent the whole 12 doctorate population. I'm sorry. Okay. 13 Another thing that we thought -- I thought of 14 is that some data items maybe being collected 15 repetitively across surveys. We want to look 16 at the relationship between question ready 17 and consistency, from EIA's and the 18 respondent's point of view. 19 The whole point of designing 20 questionnaires is to get as close to the way 21 the respondent is thinking, their records, 22 what they have available to them. Decide if BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 259 1 there is a better design for forms and 2 instructions. I think any economic program 3 has to look at the issue of confidentiality 4 protection, because the data that they're 5 collecting is confidential business 6 information. And in fact the respondent's 7 willingness to respond in part is related to 8 how well confidentiality protection is being 9 applied. And that means looking at the 10 actual data reports too, because some surveys 11 are -- some businesses are so big that if you 12 report for a certain entity, you may be 13 reporting really for them. 14 If you're a rice producer, you may 15 be taking care, you may, could be producing 16 95% of all the rice in Louisiana. So if you 17 report for Louisiana, you're reporting for 18 them and you're revealing their information. 19 We need to look at response rates across 20 surveys and to see are there particular 21 subgroups that are subject greater response, 22 is there a need to work on the issue of BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 260 1 response and what might be done to improve 2 response. We need to determine if the data 3 are reasonably timely and understandable to 4 users. I said recently timely, because users 5 often would like to have the data a day after 6 we collect it. 7 MS. KIRKENDALL: Or before -- 8 MS. COX: Or before -- for the 9 program it is critical that we document the 10 strengths of the program and the desirable 11 data elements already in place. Again this 12 is reinforcing that -- it's reinforcing the 13 behavior you want to see. But truthfully you 14 -- we're dealing with professionals who are 15 trying to do a good job. And you're 16 assessing it and they want to come back and 17 here -- here's all the ways in which you've 18 done a good job. That's a fair thing. We 19 need to specify areas for improvement and 20 specific recommendations for improvement and 21 then suggest enhancements that would increase 22 the utility of the data across surveys. Now BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 261 1 I have questions for the committee. I do 2 have something that I can distribute, that I 3 was looking at, I don't have a slide for 4 this, but I do have copies of the slides as 5 well as notes, but the slide wasn't prepared 6 in time to be distributed. I think there's 7 enough for everyone. I could use one copy of 8 that. 9 I should say that this is very 10 much, this project only started fairly 11 recently and so the first phase is completed 12 which is the survey evaluation template, but 13 not the program evaluation template. But for 14 the program evaluation template, another 15 thing that I started thinking about is the 16 theory on the value of information and that's 17 what I'm -- I gave you, attributes of 18 information value. 19 And I think this really captures a 20 lot than we want to look at, the reporting 21 cycles, in other words, how often are these 22 data collected. Partly how often you need to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 262 1 collect data relates to how swiftly things 2 change as well as how much money you have. 3 Sufficiency, that means are you collecting 4 the data that's needed, are you collecting it 5 from enough people, that you can make 6 meaningful use of these data? 7 Understandability, that's looking 8 at the end products in particular but it 9 could be looking at the questionnaire, saying 10 to what extent are users able to understand, 11 what we're giving them. Freedom from bias, 12 this is pretty critical in many different 13 ways, we need to be concerned about non- 14 response bias, under-coverage bias, 15 measurement bias due to misunderstanding of 16 the questions we're being asked. Reporting 17 delay, that's the delay between when we 18 collect the data versus when we get the data 19 out. And I think every survey needs to look 20 and say, can we get it, how -- what can we do 21 to speed up the process, is there something 22 we can do? Reliability, I would also put BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 263 1 accuracy there too. 2 Part of this is how dependable are 3 the data we're producing. People actually 4 make financial decisions based upon EIA data. 5 Decision relevance, are we giving them the 6 data that they need, in other words, that 7 looks at the content of what's being 8 prepared, cost efficiency. EIA is not alone 9 among federal agencies to have at best a 10 budget that stays flat for several years. 11 And sometimes it's reduced, so we always have 12 to look at the cost efficiency of what we're 13 doing. 14 The comparability in economic 15 surveys and a frequent event is that your 16 users will have access to data sets, that you 17 don't have access, that you may or may not 18 have access to and they'll compare your 19 results to their data set. I was at the USDA 20 during a time when there was a number of 21 complaints about the, the estimates of young 22 calves being born. And their estimate was BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 264 1 way too large. And the way they ascertained 2 it was based upon certain sales and things 3 like that. 4 So they had numbers that should be 5 related to what USDA was reporting and those 6 numbers didn't agree. So there are data sets 7 for which comparisons can be made and the 8 agencies tend to be held accountable for 9 that. Quantitativeness, that relates to the 10 extent to which the data can be quantified 11 and the degree of aggregation. Now against 12 economic surveys we have to look very 13 carefully at aggregation because the identity 14 of the respondent can be revealed through too 15 much aggregation, too little aggregation, 16 excuse me. 17 And in fact there users tend to 18 want a great deal of geographic understanding 19 of the data, which is a very big driver of 20 many aspects. It's the one thing they want 21 that's very difficult to provide, because 22 they want data at a very low level. So BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 265 1 that's just something new that I've been 2 looking at most recently. 3 For the committee with respect to 4 the survey evaluation template which was 5 included as an appendix in the paper to the 6 committee, the questions I have is, are there 7 additional items which should be included in 8 the survey evaluation template? Do some 9 items need to be modified? Is this okay for 10 evaluation of an individual survey and then 11 any suggestion for important components that 12 should be included in the program evaluation 13 template, the template for the entire 14 collection of surveys. And that's it, so I 15 should bring the -- oh, I lost it all. I 16 don't think I can get back. I'm not moving 17 at this. 18 SPEAKER: Okay, so looking at -- 19 MS. COX: Did everyone get a copy 20 of this slide that one in one by the way. 21 DR. FEDER: I think every survey 22 should include some QA procedures that are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 266 1 conducted continuously including some things, 2 I can tell you, I'm not that familiar with 3 how the operation of the EIA service, maybe 4 some of what, I'm going to say it's 5 inappropriate, in this context you have to 6 modify them. But we for instance look for 7 interviewer falsification, verification that 8 the interview took place. I know it's 9 probably inappropriate here. 10 You look at the date, the time to 11 complete a questionnaire, again maybe not 12 appropriate here but you can collect when you 13 make a telephone interview information, how 14 long it takes. You look at outliers, an 15 outlier is both a calling back to check if 16 everything, you know, checks out okay. In 17 addition if the survey is conducted over 18 three months, one month into the survey, I 19 would look at the estimates, partial as they 20 may be, with maybe incomplete waiting just to 21 see is there a trend, am I doing something 22 wrong? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 267 1 We discovered the problem was one 2 of the questions, because it was administered 3 to very young individuals who misinterpreted 4 the question. I can actually, I might as 5 well give you the example the SM, were you 6 enrolled in school last year? And they saw 7 this the action of signing up for school, not 8 being at school. So if that was discovered 9 simply by looking at the estimate. So an 10 ongoing monitoring of the results, 11 verification, you send a call back, thank you 12 very much for your participation. If this is 13 in error lets us know, call this toll free 14 number, is another way of checking that 15 everything works okay. Again I don't know, 16 well I think part of it was covered, but I'm 17 not sure fully if questionnaire testing, 18 we've spoken about here, I'm not sure, but 19 it's something else that gives what to do. 20 MS. COX: Now that is a fair idea, 21 I don't think, it's exclusive there. 22 DR. FEDER: When I was back in BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 268 1 Canada, actually this kind of offers this 2 course and many conferences open to public on 3 questionnaire design and it's astounding how 4 bad the results maybe if the questionnaire is 5 not, because we are professionals. We know 6 exactly what a question means and we expect 7 the answers to match that and often not the 8 case. 9 MS. KIRKENDALL: Johnny? 10 MR. BLAIR: Yeah, first of all, I 11 just want to make the observation that I 12 think that there is a, to use your word, a 13 synergy between this template and EIA 14 procedures, at what they're actually doing. 15 Since I expect you're, not going to put 16 something in the template for evaluation 17 that, you know, that you're not doing. And 18 -- 19 MS. COX: No, I wouldn't say that. 20 MR. BLAIR: Well, I would hope not 21 anyway. But -- 22 MS. KIRKENDALL: Well, your BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 269 1 starting from what you think should be in the 2 template, without looking at any surveys. 3 MR. BLAIR: No, I -- I'm just going 4 mad. 5 MS. KIRKENDALL: It remains to be 6 seen. 7 MR. BLAIR: No, I -- I guess I 8 wasn't clear. But after you see the 9 template, I think that you're probably not 10 going to pass it on into use until you see 11 how you're going to address the things that 12 are in it. But one thing that I see in this, 13 what I see is related to what most of it is 14 some, this notion of quality assurance and so 15 forth, that I think it's important enough to 16 have it as a separate item, a quality 17 assurance plan, that would include, not just 18 -- I mean the kinds of things that were 19 mentioned, some things that you already have 20 in here, for example, the kinds of things you 21 do to check that the weights are correct and 22 so forth. But I think it's important enough BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 270 1 to have as a separate item, as a separate 2 document -- 3 MS. COX: To be evaluated. 4 MR. BLAIR: Right. 5 MS. COX: Yeah. 6 MR. BLAIR: Yes, to be evaluated. 7 And that would include things like, you know, 8 is there a re-interview program or is there 9 something that's done periodically to do 10 validation of some of the data, that is, you 11 know a kind of comprehensive, you know, 12 quality assurance plan. I think that there 13 should be a separate item. Another item that 14 I think and this is kind of related to that 15 but I think it should be there, is something 16 about user documentation. You know, what is 17 it that's provided with the data sets, 18 whether online or in whatever form for the 19 users. Some assessment of that, and the 20 third thing I mentioned that, again it's sort 21 of implied but I think it's not exclusively 22 here, it's sort of the people's side BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 271 1 interviewer training, supervision, those 2 kinds of things, going to give some of that, 3 you know, what overlap it would be in the 4 quality, the quality assurance plan. Now, 5 one other question if I may to follow on, 6 many -- I'm not sure exactly how it's 7 intended, that the template will be used by 8 the outside review or the outside expert. 9 Is this just provided as a document 10 to say you know, there are just some things 11 to keep in mind, or is there something more 12 specific in terms of what instruction is 13 given to the expert reviewers or the outside 14 evaluators or whatever they are called and 15 understanding the summery of findings that 16 you want back, but it's -- it were intended 17 to be as much guidance as is given or is it 18 -- is it intended that it be left fairly open 19 for them to decide what they get back to you 20 or is that something you're going to be doing 21 more of? 22 MS. COX: My intention was that BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 272 1 this would be all different things that you 2 could look at. I was thinking on myself, 3 critically evaluating a survey. And what I 4 would look at and think about, now something 5 are very particular in there, they don't 6 apply to all surveys. Like some are male 7 surveys, they'd have a different way, than a 8 face-to-face survey. But it was all the 9 different things that you would look at if 10 you were concerned about evaluating the 11 quality of the survey. You said something 12 that was interesting. I would hope that this 13 template would be distributed to everyone 14 doing the survey. Because first of all they 15 should be documenting the kind of things that 16 are in that template and I think the very 17 process of documentation can identify 18 problems. 19 MR. BLAIR: Sure. 20 MS. COX: So that I would think 21 that one way the evaluation template can 22 help, is it tells people what to look at and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 273 1 think about and it spotlights the problem 2 even before they're evaluated they can say, 3 oh wait a minute, you know, this is talking 4 about this, I think we've got a little 5 problem here we need to address. So it's 6 almost like giving someone a cheat sheet, but 7 it's a cheat sheet that will allow them to 8 address problems and know that something is a 9 problem even before an outsider has to say it 10 is a problem. 11 MS. KIRKENDALL: The other thing 12 that, thinking about how it might by used is, 13 if we're going to hire somebody to do an 14 external evaluation or somehow otherwise 15 we're talking of doing it. It's going to be 16 cheaper if we can send them documentation 17 that has all the information they would need 18 to do the survey. And I'm not sure EIA 19 documentation is in that stage yet, so we'll 20 find out. The other way they can do it of 21 course is come in and talk to everybody, then 22 try to ferret out the information that way or BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 274 1 some combination of the two. So if it's 2 based on documentation, then you like to have 3 your documentation cover all these pieces. 4 MR. BLAIR: And I guess one piece 5 of that, just responding to that would be a 6 methodology report and which is different 7 from this user documentation and responsive 8 data files that I would have there, I would 9 want an outside reviewer to evaluate. Now, 10 the methodology report has in it what it 11 should have the collection of details. 12 DR. HENGARTNER: Let me start by 13 saying that, in my opinion EIA is ahead of 14 the curve here. Because I mean we've been 15 talking about data quality, data continuity 16 -- 17 SPEAKER: Continuity. 18 DR. HENGARTNER: Thank you, there 19 are some words I just can't say. We've been 20 discussing that I mean for years now, I mean 21 this is really not a new subject then, it is 22 also, I'm a little bit worried about BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 275 1 something. You know, what I'm worried about 2 is that we're going to have these evaluations 3 and we've got to look at the ways of listing 4 them, oh no this data here has a high rate of 5 missing than that one, so that's a bad study. 6 When in fact we know you know that, in some 7 areas the data is much much harder to get 8 than in other areas. And so I -- and the 9 problem is clearly that we don't know what 10 level of achievement we're looking for, 11 right? 12 You say, it is to improve the data 13 collection and so forth, of course, what is 14 achievable? I mean, it's like safety and 15 security, I mean we talked about that thing 16 yearly, all the time. Their incidence all 17 the time is what we're striving for but 18 things happen, and same thing and they say 19 that we're trying to do the best possible but 20 we need to know what is realistic that -- 21 that's been, what are the bench marks? And 22 having a template without the bench marker, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 276 1 knowing what to expect is very hard because 2 it might actually be disheartening to people 3 when they receive the report. And things 4 that are missing on the practical level, we 5 don't talk about customers directly. 6 At EIA, each time we talk about 7 data quality, we always go back, and there's 8 lot of customers, why do we do it in the 9 first place? And yes it's touched a little 10 bit by survey objectives, but I would 11 actually list it, who's using it and why are 12 they using it. There is also something else 13 that I would put in is, we can talk about for 14 example the missingness, I mean to say, to 15 talk about timeliness, missingness and so 16 forth. But the other thing that I would urge 17 you to put into the document is the cost 18 benefit. 19 How much is the cost to reducing 20 the -- to rate of missing it? Yes you can 21 reduce it, but it's going to cost so much 22 that it's not worthwhile. We are going to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 277 1 make a decision based on that. And so having 2 this explicit view will somehow lessen the 3 comparison, because they say we don't know 4 the benchmarks, but knowing how much it 5 costs, that is going to be, the OMB will love 6 that, I think. Yes, there's something else 7 also, yes, you mentioned the overburden, 8 usually when you take a program, like let's 9 say, natural gas or petroleum, the EIA tries 10 to map the whole spectrum from producer to 11 consumer. 12 And because the producers, unless 13 the consumers specifically know the 14 transporters, you don't tend to sample the 15 same people. So you are talking about, 16 within each program to look at redundancy. 17 My fear is that actually the redundancy will 18 come by looking across programs. That means, 19 you look at the producers, right. And you 20 ask, you're going to ask about petroleum and 21 then you're going to knock on their doors for 22 electricity and then you're going to on this, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 278 1 and you see how that, there is going to be 2 the repetition that I think there is, I mean 3 Nancy is going to correct me on this. 4 MS. KIRKENDALL: Actually you're 5 right, and they know it. For example, I mean 6 we had our refinery surveys, we've had a lot 7 of surveys in the filling and supply area, 8 they also respond on the electric power 9 survey as a combined power plant. A 10 different label, but they're in there, at 11 least a lot of them are. 12 DR. HENGARTNER: Yeah, absolutely 13 and so that's where the redundancy would come 14 in. And so by evaluating them program by 15 program, you actually miss exactly what 16 you're trying to review. So -- 17 MS. KIRKENDALL: Brenda has talked 18 about it, we're starting, you look at 19 families are pro- reactive, but once you do 20 that, I think you can start looking at 21 combined programs, so it will be you know -- 22 SPEAKER: Well, the whole EIA. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 279 1 MS. KIRKENDALL: The whole EIA. 2 People say that there are similarities in 3 natural gas and electric power, there should 4 be some commonalities, other than totally 5 different offices, they do nothing the same. 6 And as far, you know, look at those two 7 together, oh they would be some benefits we'd 8 probably find. 9 DR. HENGARTNER: I think the, just 10 last point, I think the confidentiality that 11 shouldn't stand also, I mean for the whole 12 EIA. I mean how do I -- how do we ensure 13 confidentiality, this is so important. And I 14 know you stressed it, but I think this is, 15 that's the reason why I mean, we're getting 16 the data in the first place. And if we 17 cannot guarantee that then we are not doing 18 our jobs. 19 DR. FEDER: And it's the law. 20 DR. HENGARTNER: And it's the law. 21 MS. COX: Yes. 22 DR. HENGARTNER: The bottom line is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 280 1 we've been talking about this all along. So 2 I'm not sure what this is going to bring new 3 to the process and that's an open question, I 4 don't know. 5 DR. FEDER: I wanted to second what 6 Johnny said about implementation and I'd like 7 to recommend that this be an item in every of 8 the evaluation that the level of 9 documentation of the survey is part of the 10 items which should be checked at, and also 11 what really you're talking about, something 12 that that would call data consistency, like 13 the calves and the number of calves actually 14 born versus purchased. And in fact sometimes 15 you get similar estimates from different 16 surveys that don't match and I remember again 17 back in Canada where the ultimate citizen 18 pointed out a difference between two 19 estimates and their follow up of that 20 uncovered a certain estimation issue that was 21 discovered. So I would call it data 22 consistency. I -- and there's been good data BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 281 1 consistency because in surveys I think 2 there's another thing called consistency of 3 data, yeah I guess, that's how we call it. 4 Data consistency, it's at a program level not 5 at the survey level. Of course sometimes 6 within the surveys too. 7 MS. KIRKENDALL: Go ahead. 8 MR. BLAIR: Nicholas mentioned that 9 you don't have, in many cases sort of 10 benchmarks for what you should do to -- what 11 you do have is a baseline and I think it's 12 useful to look at some of these measures, 13 response rate is an obvious one item, you 14 know, response rate of what we see over time. 15 I don't know what the plan is or how 16 frequently these evaluations would be done, 17 but one measure is sort of you know, are you 18 getting better, are you sort of you know, 19 holding your own, are you getting worse, are 20 you holding your own but for higher cost, you 21 know, those kinds of things I think can be 22 looked at. And because you know this is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 282 1 probably, you know, the more elaborate it 2 becomes, the more expensive it's going to 3 become, time consuming. 4 And you may not do this as 5 frequently as you would like, but what might 6 be worth looking into this is sort of partial 7 evaluation. That is if you may, you know, 8 you do this is first time and it's this, you 9 know, massive effort and also you get lots of 10 good information and you identify some things 11 that are you know candidates for improvement 12 to follow up the evaluation, you know, and 13 doesn't have to miss, I mean it can be the 14 whole thing again, but it could also be a 15 follow-up on those particular points that 16 were identified and I think again, that would 17 something, if OMB ever remembers looking at 18 improvement but, you know, but things that 19 could be, you know, like sort of -- 20 MS. KIRKENDALL: And the other 21 thing, just remind you, we talked last time 22 about our survey process that Tom was trying BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 283 1 to measure and Shawna was working on that 2 too. And so that there are lots of different 3 ways of getting evaluations -- about hearing 4 form our program managers what they're doing 5 and then Tom and Shawna went towards them to, 6 sometimes a little more in there but -- so 7 they're very similar and this is the outside 8 view and how do we bring it all together and 9 make sense out of the whole thing. 10 DR. SITTER: I guess we're 11 consistent. I only wrote down three things 12 during the talk, one was cost efficiency, one 13 was document, decisions and trade offs and 14 the other one is which hasn't really been 15 talked about is redundancy, is still very 16 much related to issues, data consistency 17 issue. If you evaluate too much and decide 18 that you're sort of being redundant, you can 19 get rid of all of your redundancy, which is 20 not a good thing. 21 MS. KIRKENDALL: Not really. 22 DR. SITTER: Not necessarily. On BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 284 1 the issue of reducing costs and 2 documentation, documenting, I think the 3 natural gas discussion is a good example that 4 happened this afternoon. The way that this 5 worked in the past may not be -- have been 6 very useful for you. I mean, we make a 7 recommendation you do a simulation and say, 8 we don't like your recommendation, we're 9 going to do something else. We come back and 10 say, they didn't do the simulation very well, 11 then you come back and say well there're 12 other reasons we didn't do it, but 13 documenting that decision, the trade off that 14 you're making is important. 15 That is in the end of the day it 16 comes down to cost, simplicity, data 17 availability, data quality, sometimes, you 18 know, being able to get good quality data, 19 you know, your data quality from the big 20 producers might be much much better and that 21 by using those as an estimate for the guys 22 know you're going to have trouble, I mean, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 285 1 it's a good trade off to me. But that may 2 not appear anyway. Other than the minutes of 3 this meeting it says, it's not, maybe not a 4 good thing for you to do. And I think that 5 that's why, so an external reviewer comes in 6 and you know depending on what they look at, 7 they can say that you're ignoring, but I mean 8 that would be easy enough to say, and that's 9 just not true. 10 MS. KIRKENDALL: Right. 11 DR. SITTER: I mean it's a process, 12 right, we're part of that process. 13 MS. KIRKENDALL: Yes. 14 DR. SITTER: And I think 15 documenting those kind of trade off's is 16 really important. Because when you look at 17 these things sort of first glance, everybody 18 is going to have the same thing, your going 19 to look at, say you decided on this 90% cut 20 on survey, and they're trying to evaluate 21 whether that's a good thing. The things that 22 are going to run through their minds will be BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 286 1 the things running through our minds, why did 2 you do this, why did you do that. 3 If none of that appears there, if 4 we seriously consider doing this and that, 5 and you were even advised to do this and that 6 and you knew the reason that we didn't do 7 that. I think that it doesn't take much, but 8 especially the really obvious one, a report 9 like, you know, why didn't you try to get a 10 better response rate and why you decided to 11 have a outlined this way or you felt it was 12 okay and I'm sure you do many of them but -- 13 MS. KIRKENDALL: Oh, we could 14 definitely get better in documenting and I 15 think the idea -- one of the things that we 16 are doing now that we started doing at least 17 for a couple of years is we do have a summary 18 of each session with the ASA committee and we 19 say what we talked about, what your 20 recommendations were and what we did about 21 it. And it's not very expensive but at least 22 that went on. And maybe that in some BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 287 1 situations we need to be more careful and 2 this natural gas discussion I think it's one. 3 I think we need to fine-tune our simulations, 4 so that we have something else to point to. 5 DR. SITTER: I think that another 6 thing about, despite how much funds we have. 7 I think that over the five years I've been 8 here, the way that it's viewed by you as a -- 9 an external review process has increased. I 10 think that you think of it more that way. 11 But still it's certainly not as formal as a 12 formal review process would be. Now I'm only 13 having a year left. But you know, things 14 like, you're getting -- doing summaries now, 15 they weren't being done before. 16 Writing up documentation and 17 attaching them to the surveys, that are being 18 talked about, so the things that would make, 19 you know, make it a truly more independent 20 external view. I think in some cases it's 21 happening, in other cases not -- over the 22 pipeline. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 288 1 MS. KIRKENDALL: Right. I think 2 you're right that most of our, right now what 3 we're doing is we're bringing things to you 4 in an early stage and we actually, we -- I 5 mean you -- we use your advise, you know, 6 next week. And so there has been an impact 7 on what we do, but it isn't well documented 8 probably because of the fact that it's so 9 early. So maybe one of the things we need to 10 do is pretty close the loop at some point, 11 once we talked about things and we have a 12 final product and then have a discussion of 13 that because we all agree this was a good 14 prospect and we have to get good results, 15 that's something. 16 DR. HENGARTNER: Two meetings ago 17 we had a discussion about how this committee 18 works, namely what mention that the EPA also 19 has advisory committee that they write 20 reports, big reports. And that of course, 21 you can point to that and say, aha, that we 22 have been reviewed, we have a report. It is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 289 1 our fault that, I mean I don't want to write 2 a report. And I'm going to be honest here, 3 but it is a fact that if you would be writing 4 reports, then there would be something to 5 point to in saying, yes this -- we are a body 6 that independently reviewed the EIA on a 7 regular basis. 8 DR. SITTER: There are ways to do 9 that, but for example each one of us 10 summarize -- type that up and send it to the 11 person that will read it, make sure that they 12 tape read what's been said -- you know, you 13 have one person who's identified as the 14 person and so far that can be done reasonably 15 easily and it's not a big onerous deal, all 16 right, it's just an e-mail to read -- you 17 know. 18 MS. KIRKENDALL: Okay, if it's not 19 that long. 20 DR. SITTER: No you just -- it's 21 going to be a page or something, so you can 22 read it, you can see whether you agree and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 290 1 then the other thing is keeping those 2 together. That the other thing I've had 3 difficulty over the years, it's getting 4 better, oh ok so the percent standards(?) we 5 met on this two years ago. 6 MS. KIRKENDALL: I don't remember 7 what that was. 8 DR. SITTER: I don't remember that. 9 Was I here at that -- did I attend that 10 meeting, was that my break out session? This 11 also happens right is that you don't 12 necessarily get assigned to the continuing 13 breakout sessions right, the same group. So 14 a historic -- a history certainly -- 15 DR. HENGARTNER: This is an idea of 16 how to more formally be able to package this, 17 because I mean the fact that we've been 18 talking about all these things for years now 19 and I think I mean you're (inaudible) that is 20 true -- I mean, we talk, we listen, there's a 21 good synergy. We're very happy that you 22 bring up this stuff early on because that's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 291 1 where we have the most impact. 2 MS. KIRKENDALL: That's right. 3 DR. SITTER: And even -- over the 4 years, maybe one of the strongest 5 recommendations, like how this committee 6 should work. I mean Carol put a lot of - did 7 a lot of complaining on our behalf, about how 8 we need the material earlier, we want to be 9 in the process earlier, if you want to -- so 10 I mean it's almost sort of self evaluation of 11 your quality process or is a part of the 12 quality process. 13 MR. BLAIR: When you were giving 14 the overview and the introduction, Randy 15 asked -- I think it might also be useful to 16 find out how they presented it, I mean this 17 motion of packaging that a lot of the things 18 are already there and conveying that you 19 know, that we do have a process in place and 20 that we do have something that's not just 21 when a bunch of people get together every now 22 and then -- you know you have some systematic BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 292 1 way of eliciting information of using it and 2 giving feedback and so forth. So you know -- 3 finding out how some of the other agencies 4 convey what they're doing may also be 5 something that could be done. 6 MS. KIRKENDALL: And also as I said 7 I think that the ones that we've been able to 8 see were the reviews that were done last year 9 and the year before and they are changing, 10 there were things that were agreed as 11 providing outside expert advice last year. So 12 they're getting tougher. 13 DR. HENGARTNER: Yes. 14 MS. KIRKENDALL: But I do think 15 that the packaging is like a lot of us, and 16 so we really need to be. Well, we did this 17 in a fairly short period of time and we 18 probably didn't engage everybody in the 19 agency that could have provided some good 20 input because you know, when you do something 21 in a short period of time. 22 MS. WAUGH: And also since OMB is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 293 1 interested in standards EIA has and standards 2 that relate to the survey process and on 3 survey results that if they could put some 4 kind of matrix, this is what a packaging is 5 for EIA, that would demonstrate that we are 6 meeting our standards whether IT programs 7 that do rounding procedures or our response 8 rates that meets the OMB response rate 9 requirement. I mean there all the surveys 10 meet that requirement but if we could at 11 least demonstrate that we're monitoring that 12 information and that we can identify which 13 surveys meet those standards. That may also 14 be a way of communicating to OMB that we are 15 aware -- both aware we need to do the 16 improvements, but also aware we're either 17 meeting or exceeding their expectations. 18 MS. KIRKENDALL: Well actually that 19 is being done by a different part of OMB than 20 the parts that does the standards, they don't 21 talk to each other. 22 MS. WAUGH: Oh okay. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 294 1 DR. HENGARTNER: Is anyone auditing 2 the OMB? 3 MS. KIRKENDALL: And what they were 4 particularly -- it wasn't so much that they 5 didn't think that we had high quality 6 service. I think they do believe that we do 7 have high quality service, I think they 8 actually think we think pretty highly of OMB. 9 In fact I think our budget examiner tried 10 hard to us good grades on some of those and 11 she'd come and she talked to us and she'd 12 given us information about what she thought 13 and then she'd go back and talk to some 14 mysterious people at OMB and our grade would 15 go lowered. So you know they're working on 16 the standards, they're trying to raise the 17 level across the Federal government and that 18 means they were playing, it was a whisper, 19 where you whisper to me, I whisper in this 20 guys ear and he whispers in the next ear and 21 who knows what 10,000 Johnny finally tells us 22 what we've said. So you know, we put our BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 295 1 ideas together and we talk to Laurie and 2 Laurie goes and talks to whoever and they 3 came out with a grade -- so we need to be 4 able to come up with, sometimes documentation 5 or evidence that is clear to everybody. So 6 even when you whispered around the table you 7 say, oh yeah. Clearly a program evaluation 8 for -- 9 MS. WAUGH: The other part is that 10 you presented earlier on and then Brenda 11 talked about her 3 tiers of program 12 assessment and maybe -- and this is only a 13 disconnect for me, maybe other people saw the 14 connection but you talked -- that OMB wanted 15 to know about relevant effectiveness or 16 efficiency, I can't remember, it might have 17 been both. 18 MR. BLAIR: Effectiveness. 19 MS. WAUGH: Okay, anyway there were 20 five or six things listed, and they've been 21 provided some guidance on these things that 22 we don't think are feasible but I didn't see BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 296 1 the top tier of Brenda's addressing 2 relevance, and to me there needs to be a 4th 3 tier or something that really packages what 4 Brenda's proposing that actually is 5 communicating it in a way that OMB is asking 6 for it. This is the packaging issue. 7 MS. KIRKENDALL: You've got to have 8 relevance and the effectiveness somewhere in 9 there. 10 MS. COX: I'm looking at it as a 11 Statistician and these are -- as a survey 12 statistician -- and these are surveys. So as 13 I'm writing it with more phenomenon, 14 statistical perspective, there's the subject 15 matter perspective in which you say gee we're 16 collecting this, we should be collecting 17 that. 18 MS. KIRKENDALL: And maybe that's 19 another thing that we should be looking at in 20 terms of a program evaluation, instead of 21 having a statistician say are we doing things 22 right, have an outside expert look at what BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 297 1 we're doing and say are we doing the right 2 thing. 3 DR. HENGARTNER: Well but that goes 4 back to the customer. 5 MS. KIRKENDALL: Yes, it's going 6 back to the customer. 7 DR. HENGARTNER: What's the 8 relevance, I mean if you have the customer up 9 front then -- your template, then you have to 10 know them. 11 MS. COX: I think that it's a part 12 of the process you say what are we measuring, 13 how are we measuring it, which is the best 14 way to measure it? Because if you decide you 15 need these changes in what you're measuring 16 -- well it's still evaluated how you've been 17 measuring it and you can make changes there 18 so I don't see the two as being -- I have to 19 have this done before I can do this. 20 DR. SITTER: I think that one 21 aspect of meeting that might want to 22 emphasize the relation with customers, is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 298 1 that it's open. I mean I'm sure a lot of 2 external evaluations are not open. These are 3 public meetings, any of your customers can 4 come and see what you're doing they can even 5 evaluate -- they can criticize they process 6 by which you're trying to make your process 7 better. You're wide open and out in the open 8 and I think that, that's what Nick was saying 9 about your data, I mean it's out there, the 10 public has got it. What better quality check 11 than their response. 12 MS. NORMAN: In fact there was a 13 small point that I was going to bring up. It 14 was actually when you mentioned the 15 documentation and you referenced the examples 16 of natural gas problems that we were talking 17 about earlier and there actually is a mild 18 kind of internal documentation we had set for 19 these things you know, this is why we chose 20 the 90% cut off versus this and we have all 21 this kind of stuff. I think one of the 22 issues that might need to be raised, is maybe BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 299 1 we should be more candid with that 2 information. Maybe that information should 3 be a little bit more readily available, needs 4 to be moved over and maybe formalized a 5 little bit more but maybe we should be 6 sharing that and then only -- I have one 7 other point for your template, but I don't 8 know if it was already suggested or not, I 9 might have missed it. 10 But within the template that you're 11 handing out the independent reviewers maybe 12 one of the things that should be part of the 13 template is asking the independent reviewer, 14 what did we miss. Could you have provided us 15 with any other topics that we should be 16 reviewing that we missed in this template? 17 It's just a quick simple question but they 18 might come back with something that we just 19 haven't thought of. 20 MS. KIRKENDALL: But how we've been 21 doing with the 914, we have done a lot of 22 stuff with the data and one of the things BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 300 1 we're contemplating or I think we would like 2 to do is to put together a topic contributed 3 paper sessions for ASA. Have Preston talk 4 about simulation studies and we'll talk about 5 samples in general and maybe have John talk 6 about the sampling an estimation method and 7 how it's working in practice so. It's 8 because there are a lot of interesting bits 9 and pieces for this question? 10 DR. SITTER: Is there any chance of 11 getting that data? 12 MS. KIRKENDALL: Probably not, I 13 think that's putting together from purchased 14 versions. I wish I could share it because I 15 think we could get a lot of good, really good 16 part with this one. Yeah that's some 17 interesting data there. 18 DR. HENGARTNER: Well there's still 19 a summer internship right Hardy? 20 DR. SITTER: You've got to have a 21 student. 22 MS. KIRKENDALL: Yes there is a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 301 1 summer internship, that's a possibility. 2 There's another proposal that's out there and 3 I haven't heard yet. Joe I don't know if 4 you've talked to John about how he feels 5 about the data that you'd like to use in your 6 -- 7 MR. SEDRANSK: John suggested that 8 it was perhaps easier to get the data 9 statistics he was talking about. 10 MS. KIRKENDALL: Okay that's 11 interesting, so he thinks that you could have 12 access to the data that he's using now? 13 MR. SEDRANSK: Yes that's what he 14 said at lunch. 15 MS. KIRKENDALL: Oh well that's 16 interesting, maybe if you can get it, it can 17 be more broadly shared. We'll have to find 18 out, he's used data sources. 19 DR. SITTER: If you want you can 20 blind him at whatever level. 21 MS. KIRKENDALL: Yes, so there may 22 be something that could be done -- yes. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 302 1 DR. SITTER: But I must admit my 2 interest might be more academic than advisor. 3 MS. KIRKENDALL: It probably was 4 but on the other hand you came up with 5 interesting observations you might very well 6 share them with so. 7 DR. SITTER: I'd have to see if I 8 was allowed to release that information. 9 MS. KIRKENDALL: But anyhow that 10 would be -- that would help us to put 11 together documentation to -- if we wrote a 12 bunch of EIA Topic contributed paper sessions 13 so that it would be easy to take from the 14 package and EIAs documents that describe the 15 work from where we got lurking around out 16 there -- 17 SPEAKER: Okay. Great, so we're a 18 little behind but not too bad, so I guess 19 it's time to go ahead and summarize the 20 breakout session. 21 DR. HENGARTNER: We had a 22 presentation about the idea of designing a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 303 1 template that would essentially guide 2 external reviewers in surveying or evaluating 3 in divisional surveys and in families of 4 surveys and eventually the whole EIA package 5 -- well the whole EIA. We went over 6 essentially what I think we've been 7 discussing over the last two years about data 8 quality, confidentiality and all those other 9 goodies that are in this. There was quite a 10 discussion afterwards beyond just the 11 specifics, an outcome to the specifics that 12 we thought we could supplement to the 13 presentation. 14 There was also some discussion 15 about why we the ASA committee didn't qualify 16 as an external committee and how we could 17 package our work here more efficiently, in 18 particular it all came down to documentation. 19 I think that was one of the late motif of our 20 discussion with documentation, if we could 21 document what we're doing here, if we can 22 document the process by which decisions are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 304 1 made and if that documentation would be 2 transparent then that would provide a good 3 spark in showing how independent we are. 4 About the specifics there's a 5 number of suggestions for example, that this 6 template should include notions of interview 7 falsification, they should look at time to 8 complete the questionnaire that outliers. 9 There was also very good points made about 10 continuous monitoring and I'm -- the ideas is 11 that not only should we look at survey in 12 separate, we should monitor them and try to 13 see that we can anticipate "boo-hoo's" before 14 they happen and Meshe gave us a few good 15 examples on how to do that. 16 There was also the notion that when 17 we talk about making this template, the 18 template was done fairly independently; I 19 hope that's correct. But I there was a 20 synergy to be gained between EIA, which is 21 thinking a lot about data quality and so 22 forth and the template and somehow I'm sure BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 305 1 that will happen as the template will go to 2 EIA and then we are going to look into the, 3 or maybe we should add this, we're already 4 doing it anyway, right. And then update it 5 in that sense. And there was however, some 6 question still about how the template will be 7 used. 8 We didn't know too much about that, 9 other things again, I think everybody talks 10 on touched on this documentation. I mean as 11 long as there's good documentation for the 12 surveys, that's going to go a long way in 13 helping us evaluate those surveys. So, it's 14 not only the template but really gets the 15 documentation in place. The last thing that 16 I'd like to mention is cost, that's also 17 something extremely important when you think 18 about evaluating the survey is cost and cost 19 benefits for example, improving aspects of 20 the survey and that allows us then to say, 21 well this is a thing that we can give in our 22 budget, you give us twice the money we'll do BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 306 1 better. And that might be a good thing to 2 actually have, and so this is a short summary 3 and I'm sure the other members here can chime 4 in and see if I have left out anything else. 5 But that's really it. 6 CHAIRMAN BREIDT: Thank you Nick, 7 any questions or comments? Okay, then I'll 8 go ahead and launch into the other summary. 9 Susan Sereika was going to be here and 10 summarize this session but she has a detached 11 retina and Bill said, well she's got two eyes 12 so what's the problem. He was joking, he's a 13 nice guy. So I'll go ahead and summarize 14 this. 15 Actually Susan was all set up to 16 call in and conduct this by conference call 17 but her doctors' appointment ran late. So 18 there's dedication, a shining example for all 19 committee members here. Okay so let me 20 summarize. This is Doug Hales's break out 21 session, independent evaluation of forecast 22 models and analysis and Joe Sedransk provided BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 307 1 a lot of input as well, in terms of panel 2 discussion and Tom Broene gave us a lot of 3 background on the OMB reporting process. So 4 Doug started with a little history on the 5 independent external reviewer process and he 6 gave us a little bit of background on the 7 PART review that was received from OMB and 8 actually kind of broke down this "No" that we 9 saw on Nancy's slide. 10 So basically there is no recurring 11 independent evaluation of sufficient scope 12 and no independent evaluations of the entire 13 program, these were kind of quotations lifted 14 out of the report. So it's really the scope 15 that seemed to be the issue and it did not 16 seem to be the case that you could just 17 repackage the existing materials and have 18 that fly for OMB. What was really missing 19 was the macro level questions like are these 20 analysis topics appropriate, what's the 21 effectiveness and impact of these analysis, 22 is the program attaining it's objectives and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 308 1 are program resources being allocated 2 effectively. 3 So these are not question that are 4 addressed by the independent expert reviewers 5 who are looking at individual components and 6 they are not addressed by putting your code 7 into the public domain and so on. So these 8 are the kind of macro level questions that 9 were of interest and the committee generally 10 agreed that what was needed here was a 11 high-powered review team. So this is going 12 to be the HPRT from here on. 13 And this could be modeled after a 14 scientific advisory committee or an academic 15 external review team. So these people would 16 need to have some familiarity with EIA 17 products and methods, but no real interest in 18 the outcome, so the standard sort of external 19 review team that you want to put together. 20 The thinking is that EIA would need to do a 21 lot in terms of their current micro level 22 evaluation and so putting together citation BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 309 1 and plan, results from IERs and documenting 2 the source and motivation of all the 3 different analytical products they put 4 together, did this come from legislators, did 5 it come from industry, did it come from 6 Secretary of Energy or whatever. 7 And this is the painful part this 8 is like an academic self study that takes a 9 lot of effort to put together and you piece 10 this thing together. That's what you hand to 11 me high- powered review team, the HPRT and 12 then they come with answers to the macro 13 level questions. One of the things that 14 would also need to be in this micro level 15 review would be a detailed review of some 16 limited number of analytic products, so this 17 might be done on kind of a rotating scheme 18 where you missed out all the analytic 19 products that are put together by EIA. 20 Some of those change relatively 21 slowly in time some are clearly changing more 22 rapidly in response to current needs so you'd BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 310 1 rotate through the list with longer rotation 2 times for the stuff that changes less 3 frequently. But at some point in the review 4 cycle you would need to have a detailed 5 review of these products that again goes into 6 the summary for HPRT to look at. Another 7 comment here was that this entire -- all this 8 stuff put together needs to be absorbable by 9 the HPRT in a relatively short period of 10 time, because they don't have an infinite 11 amount of time to review this stuff. 12 And in putting this together if you 13 look at the model of a departmental external 14 review team, typically the dean is the one 15 who contacts the members of this review team, 16 after the suggestion of names by the 17 department but the -- there needs to be an 18 analogue of the dean in this review process 19 and maybe that would be somebody in OMB or 20 maybe that would be somebody from National 21 Academy of Science and Statistics. 22 Somewhere, someone is involved early in the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 311 1 selection appointment of this review team to 2 immediately establish the independence of 3 this review panel, so they have an 4 opportunity and feedback if necessary, that's 5 not inappropriate to me. 6 So, this team then gets together, 7 evaluates all the micro level stuff that's 8 been put together by EIA and that's what 9 gives you the independent evaluation of 10 sufficient scope. So -- and that should 11 answer OMBs objection, because -- I think all 12 panelists agree that there's a lot of very 13 compelling independent reviews that already 14 being done. There is some additional -- not 15 exactly packaging but a decisional step in 16 the hierarchy that needs to take place in 17 order to get to the level of sufficient 18 scope. 19 So this team could really answer 20 questions like what would happen if this 21 whole program the EIA went away and that 22 seems to be what OMB might be looking for. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 312 1 So I think that's roughly our discussion 2 today, do you guys want to add anything to 3 that or -- 4 DR. NEERCHAL: I think we can 5 probably find out that the OMB review clearly 6 indicates that they have already bought here 7 micro level products. The -- in page 3 of 8 that presentation the bullet clearly says 9 that they value or they have seen the value 10 of your data and analysis. So you have sold 11 the micro level products. 12 MS. KIRKENDALL: Our budget 13 examiner likes us. She really -- she's been 14 on our side, she's been very supportive so, 15 remember when we have an advantage that way, 16 not everybody is so fortunate. So was that 17 modeled after your -- the model that you put 18 together was modeled after the kind of review 19 that were done by the academic department? 20 CHAIRMAN BREIDT: That's right and 21 a typical external review team for an 22 academic department would come in and review BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 313 1 the whole program. So undergraduate and 2 graduate teaching, research and everything. 3 So they would have in front of them the self 4 study report that the department put together 5 and then they would come on campus for a few 6 days and talk to everybody, faculty, staff, 7 students and then probably deans and then 8 they would just complete their report. 9 MS. KIRKENDALL: I wouldn't be 10 surprised if that was the model that OMB was 11 thinking of, naturally now that you describe 12 it and in many ways that's sort of what's the 13 kind of thing that CN Status would do too, 14 except they're a lot more expensive. So 15 expensive you can hardly afford to have them 16 do it. 17 DR. NEERCHAL: I think this was 18 also pointed out that in academic reviews, I 19 usually view it as an opportunity for the 20 department to make a pitch for more 21 resources, more lines, more faculty, more, 22 more, more, more. I think that since you BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 314 1 have not gotten any specific guidelines from 2 OMB, there is nothing wrong in turning this 3 into an opportunity like that. 4 MS. KIRKENDALL: Uh-huh. 5 MS. KHANNA: And I think just to 6 make the process a little more fine tuned, it 7 would be useful -- we talked about how it 8 will be useful to get friends at the AIA. 9 There are people who know the EIA well, it's 10 just not somebody who maybe just knows a 11 particular product that the EIA producers but 12 really doesn't know how the entire agency 13 works, as if would -- that way it's not 14 overwhelming for the review members of the 15 HPRT. 16 MS. KIRKENDALL: Right. 17 DR. BURTON: You don't like that. 18 SPEAKER: Yeah for once we can use 19 an acronym that nobody else does. 20 DR. NEERCHAL: And may I quote Joe 21 on this one, "former members of the committee 22 are okay," like retiring chairmen. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 315 1 CHAIRMAN BREIDT: So are there 2 questions or comments? We're just going into 3 further discussions -- 4 MS. KIRKENDALL: Yeah actually I 5 mean, the rest of it was more discussion 6 about the same and what could the ASA 7 committee do, how can we make use of your 8 results and we talk on it about some of that. 9 We were more down on the weeds I think, I 10 think you're discussion is really -- that 11 would be something we can actually do, we 12 just have to put together the package. 13 DR. SITTER: But it meshes well and 14 we talk mostly about the package. 15 MS. KIRKENDALL: It meshes well. 16 We were talking about what EIA and what's are 17 the kinds of documentation we should put 18 together that might be more convincing. 19 DR. SITTER: But we were thinking 20 of what would an external reviewer want to 21 look at and that's really what Jay was saying 22 -- its really -- Because I know when we do BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 316 1 our -- the last external review that our 2 department had was to become its own 3 department. So it was exactly one of these 4 situations where we have a mandate upon, and 5 we did a tremendous amount of effort into -- 6 you know, putting -- presenting ourselves in 7 a way -- in a certain way and that's easy to 8 do if you're good and I think you're good. 9 So it's not a hard thing to do, it should be 10 viewed as an opportunity and not something to 11 be afraid of. 12 If you're not good then that's a 13 different ball game but if you actually are 14 doing a good game and trying to do a good job 15 which I think is true then, it would be just 16 an opportunity and I think the two 17 discussions seem to have measured really well 18 and that way. We were talking about the same 19 thing, which is good. 20 MS. KIRKENDALL: Yes. 21 DR. FEDER: I'd just like to make a 22 distinction between being good and being good BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 317 1 and improving which is also important, and 2 also I think the goal is not just to make the 3 OMB appreciative but also just to be good. I 4 think -- as a committee member I'd like 5 mostly to think that we are helping the EIA, 6 not only maintain it's very high standard and 7 appreciation in the community but also to 8 continue to improve. 9 MS. KIRKENDALL: Well we've always 10 -- we always do this, the whole point is to 11 improve but - - how we try to use you, that's 12 a different aspect. 13 DR. FEDER: Well certainly. I 14 think since I am sort of half way through my 15 term here I am very impressed with the 16 attention and kind of attitude that I am 17 seeing at the EIA to us not as a nuisance but 18 rather as a tool. It's very -- to me it 19 makes me feel good about investing my time 20 and my employers' time in coming here and 21 making contributions if we can. 22 DR. NEERCHAL: I would like to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 318 1 really second that because you know, I have 2 worked with a number of agencies in DC and I 3 think that I'm really impressed with the kind 4 of products that I've seen and also the -- 5 what Colleen said this morning it was amazing 6 to hear that, you know, from a person who is 7 actually doing this thing. That it's not, 8 you know, when she has a very high score or 9 something, I don't have room to improve 10 anymore. I think that's a fantastic thing to 11 hear and I think that's really, really 12 impressive. 13 So one of the things I'm going to 14 do is to check out that website and see the 15 agencies that's scoring higher than you. I'm 16 going to check it out tonight. 17 DR. FEDER: Are you going to defect 18 to another committee? 19 DR. SITTER: I think you also have 20 to realize that this is tough. Defending 21 yourself publicly and going up early in the 22 process and saying this is what we're trying BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 319 1 to accomplish. Boy, I mean, most people 2 would be pretty tough or organizations would 3 be in a pretty tough place in that situation, 4 that's not easy. You don't have your final 5 product where you thought of everything and 6 you -- and you know, you already made the 7 mistakes and corrected them. You're holding 8 yourself up publicly in front of the worst 9 group here that has no particular -- 10 allegiance to you at all. So I think that's 11 -- 12 MS. KIRKENDALL: Well actually you 13 do, I mean it's not formal but especially 14 after you've been here for a while, I've seen 15 little allegiances. 16 DR. SITTER: Oh I sure hate that I 17 gave you that impression. 18 MS. KIRKENDALL: That doesn't mean 19 you'd always be nice to us. 20 MR. CARUSO: Jay. 21 CHAIRMAN BREIDT: Yes. 22 MR. CARUSO: One thing about the -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 320 1 that I completely understand is opportunity 2 business where you -- you know, you use this 3 as an opportunity for your own agenda, in our 4 case more money. 5 CHAIRMAN BREIDT: Uh-huh. 6 MR. CARUSO: But in the PART, I 7 think success is getting what you ask for. 8 Getting more of what -- really I mean -- I 9 don't know. Anybody who's -- this is 10 relatively new, this is the third year of 11 PART but I don't know anybody who's gotten 12 more than their request by -- so, it's a -- 13 the risk is almost all on the downside. I 14 think that it's not to see with the PART. 15 Most likely, most people get their budget 16 cut, from your request to - - but I also -- 17 CHAIRMAN BREIDT: In an academic 18 setting it usually doesn't work either. 19 DR. NEERCHAL: I've never seen it. 20 MS. KHANNA: Well when you're down 21 in the dumps and you telling you do this or 22 we'll shut the department down, they'll do BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 321 1 it. 2 MS. KIRKENDALL: Well, it sounds 3 like one of the things that maybe we should 4 do is to think about putting together 5 documentation that might be convincing, or 6 one of these outside groups to review and 7 then maybe get you guys to provide us advice 8 on the documentation -- well what would be 9 better, what would make the documentation 10 stronger. 11 DR. NEERCHAL: What were the 12 specific steps involved in the process that 13 you went through last time. Did you put 14 together a document and send it over to OMB? 15 How did it work? 16 MS. KIRKENDALL: Yes. Thomas can 17 tell you about that, he did a lot of it -- 18 MR. BROENE: Well for example we 19 did list all the service reports we had done 20 and we had some examples of where the market 21 had moved. But looking back on it, maybe it 22 wasn't organized well enough maybe it wasn't BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 322 1 packaged quite right. What else can -- 2 MS. KIRKENDALL: Well I mean let's 3 see - - we put together, based on the 4 questionnaire that's on the web. So you 5 could see what they start from. Tom put 6 together answers -- there was a committee, 7 Tom was a key player -- he wasn't the only 8 player, there were a few others and then they 9 went through and talked about assembling the 10 information in response to the question -- 11 what did we think would be convincing or 12 demonstrate that we were good in a certain 13 area. They would screw around and then we'd 14 come down to some of the hard questions and 15 we'd spend more time looking at those. 16 MR. BROENE: I mean we did mention 17 that all the models are in the public domain 18 and we mentioned that data is somewhat 19 internally reviewed because it's used by 20 other people within the agency and you know, 21 maybe I wasn't systematic enough -- I don't 22 know exactly, but it did imply. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 323 1 DR. NEERCHAL: No, I think my 2 impression is that it flew because I think 3 you were actually selling your products, your 4 specific products and I think that clearly we 5 have seen the value of it. I mean, but they 6 were looking for slightly something 7 different, you know, I don't think that they 8 -- and who -- and another question I have is, 9 who read it at OMB who specifically -- you 10 know, which office, which person -- 11 MS. KIRKENDALL: Our budget 12 examiner was the key person who would 13 interface with us and she went through it a 14 couple of times, she came and that was our 15 support. 16 MR. BROENE: At least. 17 MS. KIRKENDALL: At least. And she 18 provided input and then she'd go back and 19 chat with her people and I don't know who in 20 OMB -- probably her. 21 MR. BROENE: I had the impression 22 it was a committee of some type that reviewed BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 324 1 it. 2 MS. KIRKENDALL: Yes, that was 3 probably more than just her office too -- It 4 probably was a broader office, because I 5 think all of the budget examiners are doing 6 this for their agencies and the budget 7 examiners are spread apart -- OMB offices. 8 So it's -- yes it's probably a team at OMB, 9 but it's Tom's office led by a fairly high 10 level of people I think. I don't know the 11 names. 12 DR. NEERCHAL: And that is why I 13 think that that the comment might have come 14 from other people. You're budget examiner is 15 obviously familiar with your products to some 16 whereas the other committee members are not 17 familiar with this, so they were looking for 18 the high level information. 19 CHAIRMAN BREIDT: So if we took 20 something very similar to the OMB report and 21 had be the core of what goes to the HPRT -- 22 MS. KIRKENDALL: Right, organize BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 325 1 our report for the HPRT around what they 2 wanted naturally. 3 DR. NEERCHAL: And probably cover 4 it with the very moderate things -- you know, 5 non-technical you know, write up of al these 6 things that is palatable to a wide -- 7 MR. CARUSO: And the other -- in 8 the Heads of Statistical Agencies meeting 9 that Nancy and I go to on a monthly basis. 10 This topic has been discussed and that was 11 pretty clear from that discussion that there 12 is inconsistencies -- like somebody would 13 say, "oh well we got a good PART score" and 14 another Head would say, "well how did you 15 handle this" and they'd say, "well we did it 16 this way" and they said, "well we did it that 17 way and we got a bad score. So the -- I 18 think part of it is that this is a new 19 process and certainly for our first year so 20 -- But I do think that we clearly need to do 21 this macro on how to deal with this macro 22 issues. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 326 1 MS. KIRKENDALL: And actually I 2 think starting from the package we put 3 together for OMB isn't bad and the questions 4 and then just have -- take a fresh look at 5 some of them and then get an input from a few 6 other folks around and even if we don't share 7 the whole package with you at least you'd 8 share the features that we have. Some 9 questions as to whether they're convincing or 10 not and you may have some clear ideas about 11 it, either how to say it or what are the 12 things to document or -- 13 CHAIRMAN BREIDT: The timeline on 14 the next round -- next summer. 15 MS. KIRKENDALL: It's probably next 16 summer we think. At least we suspect that we 17 will be asking about the things that we 18 didn't do well. They may not ask to see all 19 the stuff they gave us a hundred percent on. 20 I mean, why would they. So that would be my 21 guess that they would be looking at the ones 22 that we might be working on actually. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 327 1 DR. NEERCHAL: I think Randy's 2 right, ours is a very young campus so we have 3 a few departments and we also were required 4 to do this program review for undergraduate 5 only programs only very recently. So we had 6 one program to go through this review for the 7 first time ever -- You know, this is the kind 8 of situation that -- you know, the chair of 9 that program should've called another chair 10 who had gone through it and she did not do 11 that and so she tried to do it alone in some 12 sense and it was not a very good situation 13 because of this. 14 And it was not because they were 15 doing anything wrong, but it was the 16 inexperience of -- so -- I used to see many 17 of them because I am in the APB. The program 18 -- you know -- something -- a local committee 19 that kind of go through some of these 20 documents. So one of the things we 21 recommended is to have a mentor thing you 22 know. As soon as someone is goes through BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 328 1 this, if they have never gone through this 2 they should, you know, hook up with another 3 chair who has gone through this. And that 4 other chair should kind of play the mentoring 5 role on this one because this is one of those 6 things where you have no example and you 7 don't know exactly what your package should 8 look like. 9 DR. FEDER: What Dr. Nagaraj was 10 talking about reminds me that in Britain when 11 my department was -- we were in social 12 statistics department and we were teaching 13 some of the mathematics courses but when they 14 had their research placement exercise and 15 information of the teaching is so on -- first 16 of all they had to hand over the department 17 in another university that got the highest 18 marks, come and tell us about their 19 experience and then we also had a retreat and 20 went to that retreat and got free lunch and 21 some kind of discussions and there was a lot 22 of practice. And what we found out and I BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 329 1 think it's a bit similar to the experience 2 here, those institutions that got -- at the 3 first time when that exercise was done got 4 higher marks and partially it could be -- and 5 by the way we also knew what was the 6 composition of the committee that came to 7 review us and what you could tell about them, 8 it was a retired engineer actually, that 9 headed that committee. 10 And we think that it's also an 11 issue that after the first round we'd done 12 that they would go back to say, you know guys 13 you are too liberal. So there was a time we 14 met -- then of course there's also 15 independent personalities and views that 16 affected this world that were very different. 17 But what we did is, a lot preparation we had 18 the retreat and we had someone come in to 19 tell us like which guy was talking about, 20 talking to people -- how come you did better 21 than us? 22 DR. SITTER: Also what I think was BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 330 1 a major thing -- is for you to hear -- from 2 people to know what is happening in your 3 organization. One of the things I think in 4 our review is that was mentioned -- there's a 5 strength was just sort of the uniformity of 6 what people wanted the department to be, 7 where they wanted it to go, their dedication 8 to certain aspects of it. And its not that 9 you get together as a team and decide that, 10 "oh this is the front we're going to put 11 out." It's not like that, its just -- just, 12 from something like a retreat you actually 13 have discussed it together and you understand 14 that different people in your organization 15 view things differently but they don't -- 16 knowing what the process is about, it's quite 17 different than sort of somebody walking in on 18 the hallway, "Oh yeah, you're there reviewer 19 oh yeah". You know, they don't really 20 understand that there's this big process 21 going on and what it's going to mean and 22 stuff like that. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 331 1 So they may not have thought about 2 it ahead of time and they get very, very much 3 caught off guard and then -- come across is 4 quite different than they really are. So 5 preparation there's just no substitute for 6 preparation and what's part of the 7 preparation is that the people that are going 8 to be interacting with somebody -- external 9 reviewer coming in to talk to them, somebody 10 from OMB, even people preparing stuff, that 11 they understand what's going on, a bigger 12 picture or something like that. 13 CHAIRMAN BREIDT: Are there 14 comments or questions? 15 MS. KIRKENDALL: Those are good 16 ideas, at least we have something we can go 17 on -- a few things, sorry -- more than we 18 have time to address unfortunately. 19 CHAIRMAN BREIDT: Okay, so I think 20 I'll invite public comments. Even then while 21 we adjourn the meeting, see you tomorrow. 22 (Whereupon, at 5:02 p.m., the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 332 1 PROCEEDINGS were continued.) 2 * * * * * 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382