Summary of Comments from the American Statistical Association (ASA)

Committee on Energy Statistics

at a meeting with the

Energy Information Administration (EIA) 

October 18 and 19, 2007

Washington, D.C.

1.  Developing Key Energy Indicators, Janice Lent, (SMG) and Joseph Conklin (OOG), EIA

In June of 2007, an inter-office team of EIA analysts and statisticians was chartered to identify a set of approximately 12 statistics or indicators to be designated Key Energy Indicators.  The selected set of indicators will appear together, possibly in a prominent location, on the EIA website for the purposes of

  1. providing a quick, easily accessible overview—the “big picture”—of the current US energy situation;
  2. highlighting, for the media and the interested public, important changes in the energy situation; and
  3. increasing the accessibility of some of EIA’s most broadly relevant products.

The key indicators will cover a wide range of energy topics, such as consumption, production, source diversity, use of renewable fuels, and environmental impacts.  In this presentation to the ASA Committee on Energy Statistics, we described the inter-office team’s progress to date in identifying indicators and discussed related issues.  We then presented the team’s recommended set of selected energy indicators.

ASA Committee Recommendations:

The general thrust of the Committee’s comments was that EIA should publish Key Energy Indicators (KEI).  Most Committee members advocated developing a beta version of the indicators and putting it online for comment.  Some members also offered to provide testing services for the beta version, using volunteers from university classes to form informal focus groups.  One member pointed out that the KEI could help energy markets work better by helping consumers understand the energy situation.  Members also felt that inclusion of market data among the indicators (e.g., wholesale crude oil prices) was fine, as long as customers were informed that these were not EIA products.

Some Committee members thought that EIA should develop its own energy CPI, while others suggested that EIA simply work with BLS to improve the energy component of the BLS all-items CPI.  Some also remarked that, since different users might want different sets of indicators, EIA might consider publishing two or three sets (e.g., one for industry analysts, another for students).

Other suggestions from Committee members included the following:

  1. Use the indicators to help inform the public about greenhouse gas emissions. 
  2. Be aware that energy intensity and carbon intensity indicators could be misinterpreted.  Some experts disagree on what exactly these measures mean.
  3. When you allow users to drill down to from the current indicators to historical series, give them lots of choices of time intervals.
  4. Consider publishing futures prices as well as spot prices.
  5. Put links to the trade web sites (e.g., Bloomberg), with disclaimers, on the web site with the indicators.
  6. Provide standard errors or other measures of error for the indicators, at least in footnotes.
EIA Intended Response(s):

The KEI Subgroup presented its recommendations, including some of the ASA Committee’s comments, to EIA senior staff on December 6, 2007.  A project plan detailing the steps that would be needed to develop a test version of a web-based presentation of the indicators, has also been drafted and submitted for consideration by senior staff.  The plan includes a testing phase, during which input would be solicited from Committee members willing to gather feedback via student focus groups.

Although it would be ideal to provide different indicator sets for different customers as the Committee suggested, current budget constraints are unlikely to allow development of multiple indicator sets.  With regard to an EIA energy CPI, the KEI Subgroup recommended its independent development.  Improving the energy component of the BLS CPI may not be possible given the tight time frame legislated for its monthly release.

A new KEI Subgroup is now being formed to begin developing the web-based presentation of the indicators.  As resources permit, the new team will consider and discuss the Committee’s other suggestions.

2.  Building An Information Quality Page For EIA’s Web Site, Jacob Bournazian,  Lawrence Stroud and Shawna Waugh,  SMG, EIA

 This break-out session was on a proposal to build an information quality page on EIA’s web site.  The goal of the project was to have a comprehensive location for users to access information relating to various data quality issues.  The proposal arose from a discussion at the October 2005 conference when the committee responded to a presentation on investigating data quality issues by comparing the estimates from different data sources.

ASA Committee Recommendations:

The committee members recommended to clearly define the audience that EIA was targeting for this webpage.   After reviewing two designs for a web page, they expressed a preference for the design with less text and “less clutter.”  The committee recommended learning how people might come to the quality page and what kinds of information they access.  They suggested that users may find this type of information easier if it was included on program office web pages.  They also recommended that some information may be useful to external customers and other information may be too in-depth for a regular person and may be more appropriate for internal EIA staff. 

EIA Intended Response(s):

EIA reviewed the program office home pages for highlighting information on data quality that relates to their specific energy markets or fuel source.  Some reports were posted under the pathway labeled “Analysis” on the respective program office web pages.  EIA is also considering other redesign issues for its Intranet for presenting other information quality reports.  

3.  Industrial Natural Gas Demand, Kobi Platt, EMEU, EIA

Natural gas is an integral production component for many prominent industries in the United States.  In 2006, natural gas deliveries to industrial consumers measured 6.75 trillion cubic feet (Tcf), or more than 30 percent of the natural gas consumed for the year.1  EIA’s Manufacturing Energy Consumption Survey (MECS), which is completed every four years, examines the components of industrial natural gas consumption.  According to the MECS, total industrial natural gas consumption has fallen roughly 13 percent from 1998 to 2002 (from 7.23 Tcf to 6.30 Tcf). 

EIA’s short-term industrial natural gas consumption forecast is based on industrial production indices which are provided by Global Insight and are based on the Federal Reserve’s Industrial Production Database.  The production indices of key natural gas-consuming industries are identified using the MECS.   Taking the share of natural gas use in each industry that is identified, EIA is able to generate a natural-gas weighted industrial production index. 

Since 2002, the trend of industrial production derived from the Global Insight indices has been increasing.  However, over the same period, industrial natural gas consumption in these same industries has been consistently declining.  Analyses of three prominent natural gas consuming industries—petroleum and coal products, agricultural chemicals, and primary metals—indicates that the associated production indices may no longer provide an accurate representation of natural gas use in that industry. 

This presentation introduced EIA’s methodology for constructing a natural gas-weighted industrial production index with relevant background information on the primary natural gas consuming industries that are included.  More importantly, the presentation raised questions about the changing composition of industrial natural gas use in key U.S. industries.  Finally, the presentation concluded with a round-table discussion centered around ways to improve EIA’s short term forecasting of natural gas consumption in the industrial sector. 

ASA Committee Recommendations:

Committee members recommended further research into the price elasticity of demand for natural gas in the industrial sector as well as the effect of environmental regulations.  Also, it was recommended that additional anecdotal research of natural gas consuming industries take place—managers of plants, refineries, and mills.  Finally, members suggested some comparison of other fuel use over the period in which changes in natural gas consumption were observed.

EIA Intended Response(s):

Since the ASA fall 2008 meeting, EIA has conducted anecdotal research into the operations and fuel consumption at domestic petroleum refineries.  In addition, EIA is attempting to coordinate a meeting with Federal Reserve staff to discuss their construction of manufacturing production indices.  Finally, EIA will attempt to examine the issue of price elasticity for specific industries within the industrial sector in order to draw a reference point for basic assumptions about relationship between consumption and price.

4.  Model Based Sampling Methodology for the new Form EIA-923, Joel Douglas CNEAF, EIA

In conjunction with the launch of the new Form EIA-923 ‘Power Plant Operations Report’, the criteria for the monthly sample selection was adjusted to ensure proper coverage ratios by fuel and facility types within fixed geographic regions. The sampling methodology focused on the ability to obtain accurate monthly estimates of electricity generation, fuel consumption, fuel stocks, fuel receipts, and the cost and quality of fuel delivered to electric power plants.

The sample selection methodology placed special emphasis on sample size reduction and the reduction of both respondent and EIA burden.  This was especially true for the commercial and industrial sectors of the electric power industry.

The presentation outlined a five-step process to achieve its stated goals: 1. Utilizes cutoffs based on plant nameplate capacity. 2. Ensure proper coverage of published totals 3. Ensure adequate counts within each estimation strata to ensure accurate imputation for non-sampled facilities. 4. Ensure acceptable relative standard errors for each published total. 5. Add in additional facilities whose exclusion from the monthly sample was determined to be especially detrimental to published totals.

ASA Committee Recommendations:

The cutoff sampling methodology was generally accepted as a good approach to collecting monthly power plant operations data.  No concrete recommendations were given.

EIA Intended Response(s):
EIA is presently assessing the accuracy of the sampling methodology developed using power plant operations data from reporting year 2006 on data currently reporting for 2008. The results are encouraging.

5.  The Quality Control Process for Developing the Annual Energy Outlook, Paul Holtberg, OIAF, EIA

The Energy Information Administration follows a detailed review process prior to release of its major products.  The most extensive review is typically reserved for those EIA products that receive the most visibility.  This presentation discussed the process used for quality control/review for the Annual Energy Outlook (AEO).  The review and quality control process for the AEO begins with reassessment of new historical data to update model relationships and to develop the assumptions that are typically external to the model dynamics such as a world oil prices and other non-U.S. market behavior. Such external assumptions are vetted within OIAF and then the rest of EIA through working groups and a Delphi process. Once the major external model assumptions are developed, model runs are examined carefully within OIAF at all levels of staff and management and discussed at weekly or bi-weekly “run review meetings.” When the major cases of the AEO are considered satisfactory by OIAF management and staff, they are once again vetted through the rest of EIA, including the Administrator and Deputy Administrator, the EIA Offices, and management and analysts in other Offices. The analysis and documentation of the runs and the resulting write-ups follow a similar pattern of review, including both internal Office review and wider EIA review.  After the AEO is released, feedback from the various stakeholders are solicited to provide input to the next production cycle.
Five key questions were posed to the ASA Committee:

  1. Is the external review process sufficient?
  2. Do we need to expand number or type of reviewers included in the process?
  3. Are there ways in which the review process could be streamlined?
  4. Are there other aspects of the AEO that need to be reviewed?
  5. How do we balance need to provide independent projections and analysis with stakeholder inputs we may not agree with?
ASA Committee Recommendations:

Key comments from the Committee include:

  1. It is difficult to provide comments on the process without some data on the “accuracy” of the projections.  Would it possible at a future meeting to look at statistics comparing actual and projected data to see how the AEO is performing.
  2. Maybe there are too many internal reviews or they are performed too frequent.  However, that needs to be balanced by concerns about releasing a projection that includes serious problems that would not be caught by a lesser review.
  3. It comparing results, it is important to know how the projection compares with other projections.
  4. The assumption adopted in the AEO of including no prospective public policy changes makes it difficult to gauge the accuracy of the projections when measured against actual historical data because policy does change.
  5. The number of scenarios completed as part of the AEO is adequate to cover market variation.
  6. To evaluate whether the external review process is sufficient, it is important to understand the type and mix of external reviewers and the questions asked.  However, the current process seems to be very, very extensive.
  7. I would strongly recommend not abridging the internal review process.  While you may shorten the number of hours it takes, it is important to maintain the quality of the process.
  8. The review process itself provides information to the users of the projection and may eliminate questions and concerns raised about the projection.
  9. One way to lessen the work load would be to do less—doing the AEO every other year or only updating parts of the projection—however, this may not be a good time to undertake this approach because of the volatility in energy markets.
EIA Intended Response(s):

EIA would be willing to make a follow up presentation at a future meeting using statistical measures of the projections accuracy and providing a comparison of the AEO with other projections.  We will evaluate the possibility of streamlining the review process, but we understand that that evaluation needs to be balanced to maintain the quality of the process.  While it is not clear if EIA can do the AEO once every two years, we will investigate the possibility of following a different approach.  However, given volatility in energy markets and the importance of energy in public policy right now, we understand that it is unlikely that any changes in the approach can be implemented right now.

6.  Using Regression Analysis for Forecast Evaluation., George Lady, Consultant to EIA

The presentation was an update on the on-going SMG forecast evaluation project. At the April ASA Committee meeting, it was proposed that approximations of National Energy Modeling System (NEMS) supply and demand modules be developed by using regression analyses on data sets assembled from NEMS solutions. Given the regression results, the embodied sensitivities of the endogenous variable to the exogenous variables, e.g., the price elasticity of demand for an energy product, could be used at a future time to determine the degree to which assumed versus actual exogenous variable values would influence forecast accuracy. This particular approach was proposed due to the practical difficulties of retaining operational versions of NEMS, and their corresponding development and operating system environments, for many years into the future. The Committee agreed with this approach, but recommended that the experimental design for the NEMS-solution data sets be fashioned based upon the Latin hypercube, or similar, design.

The Committee’s advice was followed. The AEO2007 version of NEMS was taken from archive, compiled, and used to construct data sets for the regression analyses. The data were formed from multiple runs of NEMS with selected explanatory variables varied according to the Latin hypercube experimental design. Regression analyses were then performed for selected fuels for each of the residential, commercial, industrial, and transportation demand modules of NEMS. These results were presented to the Committee. In general, the results were exceptionally good in terms of statistical fit and significance. The next stage of the project is to use the regression results in assessing the accuracy of NEMS projections. The regression results are to be used to differentially assess the sources of inaccuracy as related to differences between the assumed and actual values for selected explanatory variables.

ASA Committee Recommendations:

The Committee found the purpose of the project to be highly significant and important to an understanding of the accuracy of NEMS projections. The regression results, and the experimental design associated with the underlying data sets were approved. SMG was encouraged to continue the project and undertake the proposed impact analyses of the sources of error for NEMS projections.

EIA Intended Response(s):

Examples of error impact analyses, and methods for presenting the results of the impact analyses, will be developed. An assessment of the short-term projections developed using the Regional Short Term Energy Model (RSTEM) will also be undertaken. It is proposed for RSTEM to develop an impact analysis directly from model runs, for which actual values for explanatory variables will be provided and results with these values compared to projections made with assumed values.

 


1. Energy Information Administration.  Natural Gas Monthly: Annual Natural Gas Consumption by End Usehttp://www.eia.gov/dnav/ng/ng_cons_sum_dcu_nus_m.htm.  April 2007.