ML041910120

LTR-04-0425 - Richard Schneider Ltr Final TVA Control Area Readiness Audit

ML041910120
Person / Time
Site:	Browns Ferry, Watts Bar, Sequoyah   (CPPR-0092, DPR-033, DPR-052, DPR-068, DPR-077, DPR-079, NPF-090)
Issue date:	07/06/2004
From:	Schneider R North American Electric Reliability Council
To:	Needham M Office of Nuclear Reactor Regulation, Tennessee Valley Authority
References
LTR-04-0425
Download: ML041910120 (36)

Text

OFFICE OF TIHE SECRETARY CORRESPONDENCE CONTROL TICKET Dale Printed: Jill 07, 2004 16:48 PAPER NUMBER:

ACTION OFFICE:

AUTIIOR:

AFFILIATION:

ADDRESSEE:

SUBJECT:

ACTION:

DISTRIBUTION:

LETTER DATE:

ACKNOWLEDGED SPECIAL HANDLING:

NOTES:

FILE LOCATION:

LTR-04-0425 EDO LOGGING DATE: 07/07/2004

-10 bbvr, NU ouncl)

(j DEDH DEDM AO DEDR I(V I TVA control area readiness audit Mr. Richard Schneider (N.Amer.Elec.Reli.C NJ Mr. Mitchell Needham Provides letter and report regarding the final Information Chrm, Cmrs, OGC, DOC 07/06/2004 No ADAMS DATE DUE:

DATE SIGNED:

I VYde, S&x-o If

& bbs'I se-CY-1c)

CHAIRMAN - TVA Final Readiness Audit Report Page 1 From:

"Richard Schneider" <Richard.Schneider@nerc.net>

To:

<meneedhamr@tva.gov>

Date:

Tue, Jul 6, 2004 4:41 PM

Subject:

TVA Final Readiness Audit Report To: Mr. Mitchell Needham

Dear Mr. Needham:

For your review, attached is a letter and report regarding the final TVA control area readiness audit.

Please contact me if you have any questions.

Sincerely, Richard Schneider Compliance Assessment Engineer North American Electric Reliability Council 116-390 Village Boulevard Princeton, New Jersey 08540-5731 Phone (609) 452-8060 Fax (609) 452-9550 Richard.Schneider@ NERC.net cc:

Robert K. Harbour, CCC Chair Mark E. Fidrych, OC Chair William F. Reinke, SERC Regional Manager Dick Worthen, SERC Regional Compliance Manager Oswald J. Zeringue, TVA CFO Paul Reber, Audit Team Member Doug Newbauer, Audit Team Member Tom Abrams, Audit Team Member Tom Pruitt, Audit Team Member Bob Berglund, Audit Team Member Mike Flores, Audit Team Member Thanh Luong, Audit Team Member Brendan Kirby, Audit Team Member CC:

<bobharbour prea.com>, <fidrych wapa.gov>, <wreinke sercl.org>,

<dworthen © serc1.org>, <ojzeringue ©tva.gov>, <per@ maininc.org>, <doug.newbauer@ gasoc.com>,

<stabrams @santeecooper.com>, <tvpruitt@ duke-energy.com>, <robertb @ ecar.org>,

<mflores@tep.com>, <Thanh.Luong @ferc.gov>, <kirbybj @ ornl.gov>

NORTH AMERICAN ELECTRIC RELIABILITY COUNCIL Princeton Forrestal Village, 116-390 Village Boulevard, Princeton, New Jersey 08540-5731 July 6, 2004 Mr. Mitchell Needham Tennessee Valley Authority 1 101 Market Street Chattanooga, Tennessee 37402-2801

Dear Mr. Needham:

Tennessee Valley Authority Final Readiness Audit Report Enclosed please find a copy of the audit team's final report for the Tennessee Valley Authority Control Area readiness audit completed April 22, 2004. In that report are the findings and recommendations that were developed as a result of the audit. This report will be posted on the NERC website, http://www.nerc.com/rap/audits.html, in accordance with the Certification and Compliance Committee (CCC) procedures.

That procedure states:

The final audit report vill be posted to the NERC website. Should there be a disagreement regarding the findings and recommendations, the control area may provide a statement in writing to be posted in conjunction ivith final audit report.

Should the control area seek adjudication of the dispute through the dispute resolution process, they shall notify NERC within thirty days from the time the audit is posted and transmitted to the control area.

The audit report includes several recommendations from the audit team. The final procedures for tracking the implementation of these recommendations is being developed by the CCC, however, it is expected that the regions, working with NERC will monitor the implementation.

Phone 609-452-8060 a Fax 609-452-9550 n URL www.nerc.com

Mr. Mitchell Needham July 6, 2004 Page Two Once again I would like to thank you and your staff for all of your support during this audit and would welcome any suggestions you may have to improve the audit process.

Sincerely, David W. Hilt Vice President-Compliance Attachment cc:

Robert K. Harbour, CCC Chair Mark E. Fidrych, OC Chair William F. Reinke, SERC Regional Manager Dick Worthen, SERC Regional Compliance Manager Oswald J. Zeringue, TVA CFO Paul Reber, Audit Team Member Doug Newbauer, Audit Team Member

'Tom Abrams, Audit Team Member "Tom Pruitt, Audit Team Member Bob Berglund, Audit Team Member Mike Flores, Audit Team Member

-Thanh Luong, Audit Team Member

• Brendan Kirby, Audit Team Member

Control Area Readiness Audit Report Tennessee Valley Authority April 21-22, 2004 Chattanooga, TN North American Electric Reliability Council

NERC 2004 Control Area Readiness Audit Table of Contents Preface............................................

3 Introduction and Audit Process...........................................

3 TVA Participants...........................................

4 Auditors............................................

4 Executive Summary............................................

5 Recommendations...........................................

7 Onsite Review Notes............................................

9

1. Criteria and Compliance...........................................

9 1.1. Agreements...........................................

9 1.2. Staff Certification..........................................

10 1.3. Security..........................................

10 1.4. Training..........................................

11

2. Authority..........................................

13

3. Planning Time Frame Responsibilities...........................................

14

4. Real Time Monitoring...........................................

15 4.1. General..........................................

15 4.2. Alarms...........................................

16 4.3. Plans For The Loss Of Control Facilities...........................................

17 4.4. Monitoring Responsibilities..................................

18 4.5. Frequency Monitoring..................................

20 4.6. Voltage Monitoring..................................

21 4.7. Reactive Reserve.................................

21 4.8. Critical Facility Monitoring.................................

22 4.9. Transmission System Congestion..................................

22 4.10.

Load Generation Balance.................................

23 4.11.

Contingency Reserves.................................

23 4.12.

Special Protection Systems 24

5. System Restoration..................................

24

6. Delegation of Reliability Authority Functions.................................

25

7. Outage Coordination.................................

26

8. Transmission and Generation Relaying.................................

26

9. Capacity and Energy Emergency Plan..................................

27

10. Operating Policy/Procedure Changes 29

11. Vegetation Management (Line Clearances)

................................ 30

12. Nuclear Power Plant Requirements 30 Conclusions..................................

32 i

NERC 2004 Control Area Readiness Audit Preface The North American Electric Reliability Council (NERC) has prepared this report. The report represents a review of the readiness of the Tennessee Valley Authority (TVA) to meet its responsibilities as a control area and contains recommendations for follow-up action. It is the responsibility of the TVA to address these areas for improvement and to operate its system in a reliable manner.

Introduction and Audit Process In response to the August 14, 2003 blackout, on February 10, 2004, the NERC Board of Trustees committed to take immediate actions to strengthen the reliability of the North American bulk electric system. Specifically, the board adopted the recommendations of the NERC Steering Group that investigated the August 14, 2003 blackout. These recommendations included:

• A list of specific actions to correct the deficiencies that led to the August 14 blackout;

• Near-term strategic initiatives by NERC and its regional reliability councils to strengthen compliance with existing standards and to formally track completion of recommended actions from August 14 and other significant power system events; and

• Longer-term technical initiatives to prevent or mitigate the impacts of future cascading blackouts.

NERC tasked the readiness audit team with assessing the degree to which the control area meets their responsibilities that are defined as:

"The control area authority is responsible for the safe and reliable operation of their portion of the bulk electric system in cooperation with neighboring control areas and their reliability coordinator."

The audit process includes:

• A self-assessment questionnaire for the control area being audited

• Questionnaires for neighboring control areas

• A questionnaire to the reliability coordinator

• A two day onsite audit by a selected audit team Pre-audit information (responses to the self-assessment questionnaire, a set of questions and guidelines to assist the audit team in the onsite audit, and copies of some of the documentation provided by the control area being audited) was sent to the audit team to assist them in their readiness evaluation. The team met prior to the onsite visit to complete necessary preparations for the audit. This preparation included discussing and reviewing interview assignments, the audit process, interview questions, and questionnaire responses.

NERC 2004 Control Area Readiness Audit TVA Participants Executive Vice President, Transmission/Power Supply Vice President, Electric System Operations (ESO)

Vice President, Transmission Operations and Maintenance (TOM)

General Manager, Transmission Policy Development Chief System Engineer (ESO)

Manager, Operations Management (ESO)

Manager, Real Time Asset Portfolio (ESO)

Senior System Operator, IA Supervisor (ESO)

Manager, Transmission Operations (ESO)

Senior Transmission Operator (2) (ESO)

Manager, Control Systems (ESO)

Manager, SCADA/EMS Manager, Operations Analysis (ESO)

Project Manager, Emergency Preparedness (ESO)

Manager, Compliance and Standards (ESO)

Specialist, NERC/NAESB Policy (ESO)

Project Manager, Compliance (ESO)

Manager, Asset Scheduling (ESO)

Manager, Transmission System Services (ESO)

Reliability Coordinator (ESO)

Specialist, System Reliability (ESO)

Transmission Reliability Engineer Manager, System Applied Maintenance (TOM)

Manager, Transmission Support (TOM)

Manager, Relay and Meter Maintenance (TOM)

Manager, Transmission Planning (Electric System Projects)

Specialist, Technical Training (Chief Operating Officer)

Project Manager (Power Resources and Operations Planning)

(Several System Operators and Engineering Personnel)

Auditors Paul Reber, NERC Doug Newbauer, Georgia Systems Operations Corp.

Tom Abrams, South Carolina Public Service Authority Tom Pruitt, Duke Power Co.

Bob Berglund, ECAR Mike Flores, Tucson Electric Power Thanh Luong, FERC Brendan Kirby, FERC Dick Worthen, SERC NERC 2004 Control Area Readiness Audit Executive Summary The NERC readiness audit team found that TVA has the systems, tools, processes, and personnel to reliably operate its control area. TVA is outstanding in the following areas:

• Training

• Seven shift operation to allow time for training

• Backup facilities that are operating and manned

• Redundant computer hardware and communications facilities Status monitoring of the supporting systems

• Maintaining a wide area view of the TVA and neighboring systems

• Documentation

• Overall organizational engagement with a culture of reliability The readiness of TVA to meet the requirements of control area operation is discussed below. Supporting and backup information is in the body of this report.

1. Agreements and Staffing TVA has the necessary agreements in place to operate as a control area under the direction of its reliability coordinator. The necessary staff is NERC certified (all at the reliability coordinator level),

which was verified by the audit team. It has excellent physical and cyber security systems. TVA has its own federal police force for physical security. TVA training is excellent and the audit team feels it represents a best practice in many respects. For example, TVA uses a seven-team shift rotation that allows for sufficient operator training.

2. Authority The TVA operators have the authority to take whatever actions are necessary to operate the control area in a stable and reliable manner, including the interruption of firm load. This authority is documented in the control room. Management provides the appropriate support for operator actions necessary to maintain a reliable electric system. The audit team feels that TVA operators will take action when needed.

3. Responsibilities in the Planning Time Frame TVA has a very good operational planning process for the daily, monthly, and longer term activities.

It uses both linear and AC analysis. TVA assesses sufficient day-ahead scenarios to insure that it will not be operating in an unstudied, unstable state. TVA is adding an on-shift planner.

4. Real Time Monitoring TVA has a complete monitoring system with redundant computer and communications systems.

TVA also has a hot, manned backup facility. The redundant systems, active backup, and monitoring of the health of the systems gives TVA a 'best practice' in this area. It also has wide-area visibility of the TVA electrical system and the areas around it. TVA has good alarming, voltage monitoring, reactive and real power monitoring. TVA should increase the number and geographic diversity of frequency monitoring points to expand the operators' awareness of the system's operating status. The audit team feels TVA needs to add state estimation and contingency analysis to assist the operator in evaluating the real-time system conditions. It is in the process of adding these functions.

TVA's around the clock monitoring of the computer and communications through its network operations center (NOC) is a 'best practice.'

NERC 2004 Control Area Readiness Audit

5. System Restoration TVA has a documented black start plan on which the operators have been trained. TVA drills on the system restoration twice per year. Black start units are tested as required. TVA is installing a dispatcher training simulator (DTS) that will strengthen its restoration training program.

6. Delegation of the Reliability Coordinator functions TVA is both the reliability coordinator and control area operator. No reliability coordinator or control area operators functions are delegated outside of TVA.

7. Outage Coordination TVA has a process to review, approve, and coordinate outages. TVA uses the NERC system data exchange (SDX) and the Southeastern Electric Reliability Council (SERC) system to keep the industry informed.

8. Relaying Relays on TVA owned facilities and other equipment (not owned by TVA) are tested in accordance with an established maintenance and testing program. The operators know the status of relay equipment and protection systems.

9. Capacity and Energy Emergency Plan TVA has a documented energy emergency plan, and the operators are familiar with it. TVA regularly uses the portions of the plan involving interruptible customers when necessary to respond to system loads.

10. Operating Policy and Procedure Changes TVA has implemented a procedure to insure that its operators are informed of changes in policies and procedures and TVA is currently testing an improved procedure.

11. Vegetation Management TVA has documented vegetation management programs. Vegetation growth causes few transmission line outages.

12. Nuclear Power Plants TVA has special procedures and studies for the nuclear plants. The operators understand these procedures and the requirements for nuclear plants.

NERC 2004 Control Area Readiness Audit Recommendations The NERC TVA readiness audit team makes the following recommendations to TVA.

• TVA should continue with its plans to implement a state estimator and real-time contingency analysis.

The team believes that state estimation and contingency analysis should be in operation for all control areas, but TVA is not currently running them. TVA has purchased both programs from Energy Management System (EMS) vendors and is currently in the process of tuning the state estimator. TVA is using two separate EMS vendors for this project to provide a dual path approach. The team supports this decision and encourages TVA to complete the project.

• TVA should continue with the completion of its dispatcher training simulator (DTS).

A DTS would improve training and TVA has ordered a simulator. The team supports this decision and encourages TVA to complete the project.

• TVA should add time-synchronization to additional disturbance monitors.

TVA time synchronizes about 10% of the disturbance monitors. The team recommends a review of the remaining disturbance monitors to determine where additional time synchronization should be added.

• TVA should increase the number of frequency monitoring points with appropriate geographic distributions.

TVA currently monitors the frequency for automatic generation control (AGC). The team does not believe that this adequately monitors the system when islands may exist. TVA has additional frequency points at several plants and substations. The team recommends that TVA bring these points into a user-friendly display that could be used, in an islanding situation, to determine where the divisions in the TVA system are. The team recognizes that simply adding additional frequency monitors will not be sufficient to diagnose and respond to islanding conditions.

TVA is installing a wide area measurement system (WAMS) that can better determine the separation points of islands. The WAMS will be installed at 17 points and the team believes that is an adequate number of frequency points.

• TVA should include material in its training program to give operators a better understanding of reliability functions and responsibilities for positions other than their own.

The TVA training was excellent but the team found that the operators' knowledge of positions other than their own could be improved. The team recommends that the training include the functions of other positions and put more emphasis on how the positions all fit together. TVA's training program will be a

'best practice' after the addition of the training simulator and the additional operator training for other positions.

• TVA should add a redundant ISN connection.

The team recommends TVA have a redundant NERC ISN connection.

• TVA should improve the reactive reserve display.

TVA monitors reactive reserves but the team recommends that TVA monitor it by area, since reactive power is location dependent. The planning department can determine the critical regions and the EMS can monitor the reactive reserves for those regions.

NERC 2004 Control Area Readiness Audit

• TVA should collect data on generation response to interconnection frequency changes.

TVA does not do regular governor testing on its non-nuclear generation. The team recommends that TVA use its EMS to digitally record the generator output response to interconnection frequency changes to verify that the governors are adequately responding to frequency changes. The existing EMS could be programmed to capture this data and measure the generator response.

NERC 2004 Control Area Readiness Audit Onsite Review Notes TVA serves power loads in parts of seven states through a network of 158 distributors. The service area covers approximately 80,000 square miles, and uses over 17,000 circuit miles of transmission line. Peak load is approximatly 30,000 MW. TVA has 801 primary power facilities including 2,514 circuit breakers, 1,487 power transformers, and over 21,000 protective relays. In addition, TVA has 550 locations with communications gear, over 20,000 communications circuits, 2,200 path-miles of microwave communications curcuits, 2,600 path-miles of fiber optics, and 220 communications towers. TVA has over 58 connection points interconnecting it with 17 other utilities.

TVA commits itself to achieving operational excellence and places a major emphasis on reliability. It has developed four themes to reach its objective:

I. Organizational Engagement How TVA performs affects millions of people.

2. Facilities TVA must have an environment that is geared towards operational excellence. TVA will do what is needed to have a reliable system.

3. Operating System Control All elements necessary to operate the system reliably are under TVA operations control.

4. Coordination and Communication To operate the system reliably and effectively, TVA understands that it must have good communication and information exchange.

As a result of lessons learned from the August 14, 2003, blackout event, TVA's electric system operations has developed a matrix of 35 action items that is updated as items are completed. (This is not the total TVA list. There are many additional action items at the transmission power supply (TPS) level, and others throughout the agency.)

1. Criteria and Compliance 1.1. Agreements The control area must have agreements that establish its authority as a control area. The control area must have agreements that establish the reliability coordinator for its control area.

A udit Notes:

TVA is both the control area operator and the reliability coordinator for its footprint. TVA is also the reliability coordinator for three additional control areas. System Operations Procedure Manual (SOPM), Section 10.08, titled "TVA Sub-region Reliability Plan" establishes the roles of the TVA reliability coordinator and its relationship with the control area. Section 3 defines the role of the reliability coordinator. Section 1.2.2 on page five states, "These roles must be undertaken in the context of the Regional Operations Center (ROC) operating as a NERC reliability coordinator." The responsibilities of the control area and the reliability authority are clearly defined.

The security coordination agreement establishes TVA as the reliability coordinator for the control areas in its reliability footprint. The agreement is signed by all entities for which TVA operates as the reliability coordinator. Section 1.1 states "The parties appoint TVA to act as security coordinator for their transmission systems."

NERC 2004 Control Area Readiness Audit TVA provided correspondence from SERC, dated April 5, 2004 that states that the TVA was certified as a control area prior to the time that certifications were formally documented. TVA successfully completed the last SERC certification audit.

Applicable Documents Reviewed

-I Dated Version l

1 SOPM Sec 10.08 TVA Subregion Reliability Plan 11/05/02 Rev 0000 2

Security Coordination Agreement 10/01/01 3

SERC Correspondence 04/05/04 1.2. Staff Certification Control area operators must be NERC operator certified. The control area must have sufficient NERC operator certified staff for continuous coverage of the control area operating positions.

Audit Notes:

TVA has transmission operators covering four geographic areas of the TVA control area, plus two balancing authorities. All of these positions are covered by NERC certified system operators. An interchange authority works under the balancing authority and is not necessarily NERC certified. All functions in the control center are done under the direction of NERC certified operator.

All TVA operators are certified at the reliability coordinator level, except those in training. TVA provided a list of 72 NERC-certified system operators. The team crosschecked the personnel on the shift schedules for the past two months against the certification list to verify that all were NERC certified.

Applicable Documents Reviewed I

- Dated Version 1

NERC Certified Operators List with numbers 2

March and April Shift Schedule for TO Positions 3

March and April Shift Schedule for BA 1.3. Security Access to the control room must be controlled for security reasons.

Audit Arotes:

Physical Access Control: TVA provided documentation and demonstrations of the physical access controls and the audit team found this to be adequate for the control centers. TVA has its own federal police force to provide physical protection.

Cyber Access Control: TVA provided documentation and demonstrations of the cyber access controls and the audit team found this to be adequate for the control centers.

- Applicable Documents Reviewed I

IDated

'- IVersion 1 SOMP 10.02 SOC Facility Access Control SPP 06/20/02 Rev. 0000 2 SOMP 10.01 ROC Facility Access Control SPP 06/20/02 Rev. 0000 3

SOMP 5.07 Cyber Security Standard 02/20/04 Rev. 0000 I

4 SOMP 5.10 CyberAccess Management 02/20/04 Rev. 0000 5

SOMP 11.09 Threat Alert Levels and Response Guidelines I

NERC 2004 Control Area Readiness Audit 1.4. Training The control area operators must be adequately and effectively trained to perform their roles and responsibilities. The control area must have documents that outline the training plans for the control area operators. The control area must have training records and individual staff training records available for review.

Audit Notes:

TVA's training program is excellent. After TVA installs the training simulator and revises the training program to insure the operators understand the reliability related functions performed by other operators and the reliability coordinator, TVA will have a 'best practice' training program, which includes:

• Adequate time for training

• Excellent resources

• Testing

• Drills

• Individual structured training goals

• Excellent documentation

• Program evaluations TVA is beginning a new training program titled "Excellence in Performance through Continuous Learning." TVA will require employees to answer questions that will be used to provide feedback on training needs. A survey of 25 questions (randomly selected from a database) will be taken by employees at least twice a month. The questions will be appropriate to their individual work processes only. Operator knowledge of NERC policy will be the first topic evaluated in this program.

TVA has two documents that set the policy and procedures for its training program:

• The Electric Systems Operations, System Operator Training Process, ESO-VP-SDP-17.001, "System Operator Training Process" establishes the training program. It describes how Electric System Operations (ESO):

o Implements the requirements of training o

Conducts and documents training activities for system operations personnel o

Trains employees to perform their duties in compliance with regulatory requirements, TVA policy, principles and strategies

• Electric System Operations, Operating Training Program Manual establishes the requirements for the Electric System Operations. It applies to all operators in the ESO. This manual provides the details of the training program, including:

o Initial and continuing curriculum for each operator position o NERC Certification Guidelines o

Instructions for developing, maintaining, and administrating qualification guides o

The technical qualifications training checklists for each operator position TVA uses a seven-team shift rotation that allows for one week of training every seven weeks. This is a best practice. TVA requires that each of its operators receive a minimum of 80 hours9.259259e-4 days <br />0.0222 hours <br />1.322751e-4 weeks <br />3.044e-5 months <br /> of training each year but it is normal for them to have training in excess of the required minimum.

NERC 2004 Control Area Readiness Audit TVA has many training resources using a combination of on-the-job training (for new employees),

subject matter experts, in house trainers, and training modules (with some modules online). TVA has developed some of the training modules in-house and also uses course material developed by outside vendors.

TVA has elective web-based training that meets the TVA required 80 hours9.259259e-4 days <br />0.0222 hours <br />1.322751e-4 weeks <br />3.044e-5 months <br /> of training. Specific classes from TVA University (TVAU), which offers general corporate and industrial training opportunities, are also qualified by TVA to meet its training requirements. This includes the training program developed for system operators. Courses are self-paced, but have a predetermined number of credit hours that are awarded to the operator when s/he demonstrates mastery of the material (usually an exam). Training also includes fieldwork to familiarize operators with their area, new facilities, etc.

TVA offers several training courses to meet the NERC required 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> of emergency training. The modules include contingency reserve requirements, use of interruptible products and loss of EMS/SCADA visibility. TVA provided a training schedule for March 8 to April 28 showing class times for individuals to meet the 40-hour requirement.

TVA provided a core list of training courses of which the team reviewed materials from several, including:

• The participant material for the "use of interruptible products" module. It included chapter exercises and a group exercise at the end to demonstrate understanding and knowledge. The material included course and instructor evaluation forms for participant feedback.

• The handouts for "TVA ESO training, loss or partial loss of SCADA/EMS." This is one of the courses that meets NERC emergency training requirements.

Training includes drills. Recent drill subjects include the following:

• Emergency Evacuation and Transfer of Function

• System Restoration The operator, training department personnel, and the operator's supervisor develop a customized training program for the operator. This is documented in their individual development plan (IDP). The technical training department develops a core set of courses that all operators must complete including the code of conduct, emergency training, evacuation drills, etc. Operators receive annual training on the handling and use of sensitive information and how the standards of conduct affect the real-time operation. The operator can also request training in specific areas of interest and need.

TVA has a training program for new personnel. New control room personnel come from operations and must have 10 years of experience or an engineering degree. The training program usually takes about six months to complete and includes on-the-job training, training modules and testing to demonstrate understanding of the material. TVA also has a module on passing the NERC system operators exam. A mentor is assigned to each new recruit to assist him/her. For each position, TVA has a checklist (task qualification training checklist) of competencies for which the trainee must demonstrate proficiency to his/her mentor.

The team reviewed the TVA electric system operators technical qualifications training check list for new personnel. It shows required areas of necessary knowledge, key points, evaluation method, reference documents, and a signature line for the mentor. It has lines for final manager and employee signatures.

NERC 2004 Control Area Readiness Audit The manager has the final say as to when a person's training is adequate to fulfill an operating position.

The manager uses the completed checklist, scores from testing, and interview results for filling available positions.

The training program documentation was complete and well organized. TVA provided a course catalog titled "courses and ATIS number" which lists most of the courses offered. The automated training information system (ATIS) database documented completed training by employee and course training number. The data can be sorted by course number or individual. TVA can determine which operators participated in a specific drill or course, or list the courses completed by any operator. TVA provided a list of the employees completing the "ESO Spring 2004 SOC and RPDC Evacuation/Transfer of Function Drill" on the audit team request using the ATIS number to demonstrate the flexibility of this system.

TVA evaluates each of its training exercises by participant feedback, follow-up with supervision, and testing. To determine future training needs, TVA is developing a new program entitled excellence in performance (EIP) that will determine the employee's level of knowledge in specific areas. TVA has an operator training committee to review its training program, suggest improvements, and evaluate its overall effectiveness TVA is in the process of adding a dispatcher training simulator. The simulator will greatly aid in training and the team supports this effort. The dispatcher training simulator use will be administered by the training department.

The team found that the operators were not as knowledgeable as the audit team would like about areas of operation other than their own. It is recommended that TVA broaden operator knowledge of the overall operations.

• Applicable Documents Reviewed Dated Version Electric Systems Operations System Operator Training I

Process ESO-VP-SDP-17.001 10102102 Rev 0000 2

ESO Operator Training Program Manual 10/02 Rev 0000 3

Technical Qualifications Training Checklist 4

Use of Interruptible Product participant material 5

March 8 to April 23 Emergency Training Schedule 6

Courses and ATIS Number Catalog_

7 L&K Course List TVA ESO Training Loss for 8

Partial Loss of SCADAJEMS Handouts

2. Authority The control area is responsible for establishing and authorizing the control area operator position that will have the on-shift responsibility for the safe and reliable operation of its portion of the bulk electric system in cooperation with neighboring control areas and its reliability coordinator.

A udit Notes:

The authority for the TVA control area operator is outlined in multiple documents, including a document dated March 14, 2000 signed by the Vice President-Electric System Operations with signature approval of the Executive Vice President-Transmission/Power Supply Group, which clearly states that "TVA system operators are responsible for the reliable operation of TVA's transmission system and control NERC 2004 Control Area Readiness Audit area, and that they have been given the authority to implement the curtailment of interruptible and firm load, generation redispatch, system reconfiguration, and NERC transmission loading relief (TLR)."

Responsibilities are also defined in specific job descriptions as well as in a document titled "TVA system operator responsibility and authority" distributed by the Vice President-Electric System Operations to all NERC-certified operators (reliability, system, transmission) and transmission reliability engineers.

In addition, the TVA sub-region reliability plan and the security coordination agreement clearly define the responsibilities and authority of the reliability coordinator for the TVA sub-region as well as the expectations of the control areas within the sub-region.

In the management interview, a manager stated, "Operators are expected to do everything they can to identify the source of the problem and resolve the problem. As a last resort they will separate. The operators are fully empowered to take whatever action is required."

When the operators were interviewed, they stated that they had the authority to take needed action and that authority was documented in a letter from their vice president and the executive vice president. (Item I below) They also stated "Management does not interfere with operations even if they are in the control room during a developing situation." Operators also stated that the operators understood responsibilities of each operator and the relationship between the control area operator and the reliability coordinator was clear.

Operators referred to an incident in 1999 that involved interrupting load, increasing reactive output, and finally opening tie lines. The operators took the necessary action and management backed their decisions.

_ Applicable Documents Reviewed.- - -

Dated Version :

I Letter to System Operators 02109/04 2 TVA System Operator Responsibility and Authority 10/31/01 3

SOPM Sec 10.08 TVA Subregion Reliability Plan 11/05/02 Rev 0000 4 Sub-region Reliability Plan 11/05/02 5

Security Coordination Agreement 2001

3. Planning Time Frame Responsibilities The control area must have a process for day-ahead planning, as well as a process for longer-term planning, such as week-ahead, year-ahead, etc., for the operation and outage scheduling of transmission facilities and generation and reactive resources.

The control area must have agreements with its reliability coordinator to ensure that day-ahead and longer term plans for the operation and outage scheduling of transmission facilities, generation and reactive resources, will not result in unacceptable bulk electrical system reliability.

Audit Notes:

TVA conducts operational planning activities by addressing multiple planning horizons and by using various study processes and tools. TVA's procedure titled "load flow studies" outlines TVA's study processes addressing the operation planning horizon. Based on this procedure, studies are conducted with a regular frequency and address monthly, weekly, daily, and current day time frames. The frequency at which the studies are conducted and the number of scenarios assessed vary depending on the planning horizon being addressed.

NERC 2004 Control Area Readiness Audit The TVA sub-region reliability plan and the security coordination agreement identify the performance of reliability analysis as a responsibility of the reliability coordinator for the TVA sub-region. Based on interviews with TVA personnel, the current day and next day study processes described in the "load flow studies" procedure are considered reliability coordinator processes. In addition, TVA's procedure titled "transmission outage scheduling and coordination process" places the reliability coordinator in the approval process for requests for clearances.

Both linear and AC analyses are used for current day and next day study processes. In addition, TVA conducts base-transfer sensitivities in its next-day study processes in an effort to ensure that the TVA system will not be operated in an unassessed state. The results of these assessments are shared with real-time control area operations personnel within the sub-region and the reliability coordinator. If during the current day a significant system topology change occurs, the system is reassessed. The system operators, in conjunction with operations planning personnel, determine whether a topology change is significant enough to require a reassessment. TVA is establishing a five-person team (specialists, reliability analysis and operations) to facilitate on-shift work around the clock to conduct real-time assessments.

TVA also conducts seasonal voltage stability and dynamic stability assessments. In addition, through involvement with VACAR-AEP-Southern-TVA-Entergy study group (VAST) and other study group activities, TVA actively participates in seasonal assessments conducted on a regional basis.

_- Applicable Documents Reviewed lDated Iversion.

Il Load Flow Studies 04/16/04 l

_1 2

TVA Sub-region Reliability Plan 11/05/02 l

1 3 l Security Coordination Agreement Oct 2001 l

l 4 l Transmission Outage Scheduling and Coordination Processl 04/05/04 l

l

4. Real Time Monitoring 4.1. General The control area must provide the control area operators with effective, reliable computer and communication facilities for data and status monitoring, and voice communication at both the primary and the backup control facilities.

A udit Notes:

The TVA facilities have the following significant points:

• Current EMS

• No state estimator or real time contingency analysis

• Backup sites

• Redundant hardware and communications 24-hour computer support TVA subscribes to the update service provided by the EMS vendor and as of March 2004, TVA was operating with the most recent release of the EMS. The system monitors over 52,000 status points with 21,000 metered points and the interconnections with 13 neighboring control areas. The system map board has 35,000 addressable lights.

The TVA EMS does not run state estimation or real time contingency analysis for either the control area or the reliability coordinator. TVA has contracts with the EMS vendors to supply state NERC 2004 Control Area Readiness Audit estimation and real time contingency analysis. The state estimator is already installed and is currently being fine-tuned. The audit team believes that these are necessary tools for a system as large as TVA.

TVA is operating two control centers, each backing up the other. The system operation center (SOC) performs the control area functions and the regional operations center (ROC) performs the reliability coordinator functions. Both systems have independent communications and computer systems. Both control centers mirror the data of the other. Each system updates the other so that the functions can be transferred from one to the other.

Each control center has a communication system that operates independently. If one system were to fail, communications would automatically transfer to the other center.

TVA network operations center (NOC) continuously monitors the computers and communications and is staffed 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> each day. The NOC is the first point of contact for the operator having computer or communication problems. TVA has around the clock technical support with assistance on call to resolve problems found by the NOC staff. Over 100 people are available to resolve problems. This system with support is a "best practice system."

The operators state that the EMS and data gathering systems are very good and that they are satisfied with them.

4.2. Alarms The control area operator must have effective and reliable alarming capability. This should be supported in the control area's EMS and/or Supervisory Control and Data Acquisition (SCADA) system by alarm priority.

Audit Notes:

The TVA Alarm system includes:

Alarms on equipment change of state Sixteen levels of priority

• An average of 1.3 million alarms a month

• Alarms on communication system failures The TVA alarm system is a standard vendor supplied package. The system generates alarms when there is an equipment change of status or an overload. The system has 16 priority levels. The higher levels require immediate attention and the lowest levels go immediately to file and are stored there for later analysis. Lower level alarms give the operator an audible alarm that will silence after a predetermined amount of time. Alarms are assigned to areas, such as transmission operations or reliability coordination.

TVA receives about 1.3 million alarms each month. The system can handle tens of thousands of near simultaneous alarms without overloading.

The alarm package is a critical part of the EMS and if the EMS fails, the system will automatically switch to the backup EMS and continue to monitor the system parameters.

The NOC monitoring system alarms on computer or communication system malfunctions.

TVA alarms on first contingency flows with alarm limits determined one day in advance. This is displayed on the TVA flowgate monitor displays. For critical facilities for which a pre-defined NERC 2004 Control Area Readiness Audit contingent element exists, the warning alarm is at 90% of the rating. For the calculated post-contingency flow, a warning alarm is generated if the calculated post-contingent flow is 90% or more.

If the rating is exceeded, then another alarm is generated and a timer begins to record the minutes that the calculated post-contingent flow is above the limit. The TVA reliability coordinator provides this monitoring function, and flowgates are visually displayed in the system operating center.

4.3. Plans For The Loss Of Control Facilities The control area must have a workable plan so it can continue to perform the control area functions required to maintain a reliable bulk electrical system following the sudden catastrophic loss of its primary control facility.

The control area must also have a workable plan to continue to perform the control area functions required to maintain a reliable bulk electrical system following the partial or full failure of its computer facilities or monitoring tools at the primary control facility.

A udit Notes:

TVA has Impressive "best practice" backup facilities Staffed hot backup facility Redundant communication and computer facilities Evacuations plan Complete documentation TVA has a fully functional redundant backup center. Transmission operations are performed at the system operations center (SOC) and reliability coordination is performed at the regional operations center (ROC). Each is a fully functional, staffed center that backs up the other center. In the event that one center goes down, the other keeps operating and additional staff moves to the remaining facility with no reduction in capability.

Both the SOC and the ROC have direct communication links with all RTUs. The communication path is independently routed to each center, so that data continues even if one communication link is lost. Data at the SOC and the ROC is mirrored to the other site periodically, so each will have fresh data if the other system goes down.

TVA operators use their training week to train at the backup center and to test the equipment. TVA provided drill document "ESO Spring 2004 SOC and ROC Evacuation/Transfer of Function Drill" that described the most recent drill and the drill participation list. The drill was repeated four times and included most of the operators.

The backup plans are well documented in the procedures outlined below.

SOPM 11.03 -

system operations center emergency evacuation procedure describes the plans for evacuating the SOC to the ROC. Attachment 3 describes items in the TVA "fly away" kit.

SOPM 11.04 -

regional operations center emergency evacuation procedure describes the plans for evacuating the ROC to the SOC.

SOPM 8.01 -

loss of SCADA and/or EMS describes procedures for partial or full loss of EMS NERC 2004 Control Area Readiness Audit Transmissionlpower supply continuity of operations plan identifies Line of succession

• Essential functions and activities

• Relocation plans Applicable Documents Reviewed Dated

ersion 1

SOPM 11.03 SOC Emergency Evacuation Procedure 06/20/03 Rev 0000 2

SOPM 11.04 ROC Emergency Evacuation Procedure 08/08/03 Rev 0000 3

SOPM 8.01 Loss of SCADA and/or EMS 03/15/04 Rev 0000 4

Transmission/Power Supply Continuity of Operations Plan 11/16/04 5

ESO Spring 2004 SOC and ROC Evacuation/Transfer of Function Drill 04/06/04 6

Drill Participation List 4.4. Monitoring Responsibilities The control area operators must monitor operating data and status in real-time operation, including:

• Multiple frequency monitoring

• Multiple voltage monitoring

• Facility monitoring

• Transmission system congestion

• Load generation balance

• Contingency reserves Special protection systems

• Load tap changing (LTC) settings

• Status of rotating and static reactive resources Audit Notes:

TVA monitors all of the listed parameters. The audit team observed the following in the control room of both the SOC and the ROC:

1. Ace Trend

2. Alarms (look at high and low limits)

Dead band for resets

• Alarm disable Quality codes (out of scan, reasonability limit, etc.)

• Alarm or point out of scan or inhibited

3. Computer Hardware status displays TVA has a group monitoring for the computer equipment and a nice display to show the status

4. Take-over or fail-over displays

5. Capacitor/Reactor Status TVA has displays to show capacitor capacity and voltage and generator Mvar output

6. Company Tie Summary (Actual and Scheduled.)

NERC 2004 Control Area Readiness Audit TVA has display to show actual and scheduled flows with neighboring control areas.

7. Computer Equipment Status

• TVA has a group monitoring for the computer equipment status and a nice display to show the status of the equipments.

8. CPS 1/CPS 2

9. External Generation C/A Generation

10. Frequency Trend

11. Generation Summary

• TVA has displays to show generator output MWV/Mvar by unit

12. Grid One-Lines

13. Transmission Lines with Dynamic Ratings

• TVA equipment ratings can be updated with seasonal ratings, not fully dynamic ratings,

14. Tagging system

• TVA uses the tagging system to checkout interchange net scheduling

15. Regulation Error

16. Spinning Reserve

• From the AGC system summary, the display shows system spinning, primary, and secondary reserve.

17. Reactive Reserves

• TVA has displays to show generator output in Mvar and delta target and capacitor capacity and voltage levels.

18. System Tie Summary

19. Transmission Limitation Summary (Transmission lines and transformers)

20. Voltage Regulator (Status)

21. Time Correction Monitor

22. Contingency Analysis Results

• TVA currently does not have real-time contingency analysis or a state estimator. It is currently testing the State Estimator and advanced network applications.

23. Load Curtailment and Restoration Displays TVA has numerous fault recorders, about 10% of which are synchronized with Global Positioning System (GPS) time. It also has approximately 200 power quality meters, spread throughout the system on large industrial customers delivery points, which also have GPS time synchronism. These meters are located such that they would be very useful in post event analysis. As TVA upgrades its fault monitors, it is prioritizing the order of replacement to maximize the immediate effectiveness of the replacement program. Time synchronization is added to existing equipment if synchronizing time is available and the equipment is time source compatible.

The U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations report noted that "A principal cause of the August 14 blackout was a lack of situational awareness. The need for improved visualization capabilities over a wide geographic area

_19-

NERC 2004 Control Area Readiness Audit has been a recurrent theme in blackout investigations. Improvements in this area will require significant new investments involving existing or emerging technologies." (Pg 159) Recognizing that there is presently no accepted solution for the wide-area-visualization problem, TVA is to be commended for its efforts in this area. TVA is developing operator wide-area regional visibility using vendor provided displays. The current version graphically shows voltage levels for the region with different colors for high and low voltage and varying color intensity proportional to the voltage.

4.5. Frequency Monitoring The control area operator must monitor frequency and direct actions to resolve significant frequency errors, and correct real-time trends that are indicative of potentially developing problems. Frequency monitoring points should be of sufficient number and from several locations with sufficient area coverage to allow the control area operator to effectively monitor the control area, and be able to determine possible islands.

Au dit Notes:

The TVA system has five frequency-monitoring points, two in the SOC, one at the ROC, and two in the field. Other frequency monitoring points are available but are not sufficiently accurate for automatic generation control (AGC) and are not normally observed by the operators. Because of its many strong interconnection points and robust bulk power delivery system, TVA does not foresee island situations, so TVA is not set up to watch for system splits on frequency.

The location of the monitored frequency points are too close together to generally help in determining system separation points. They are set up to automatically switch between the five frequency monitors if a monitor or communication system fails.

The team investigated the additional frequency monitoring points that TVA has at additional facilities and suggests that TVA use these to develop a display to assist in determining system separation locations.

TVA is installing wide area measurement system (WAMS) with 17 points throughout the system (five installed now). This is independent of SCADA and will provide voltage, current, frequency and angle visibility. The WAMS will provide TVA with the data needed to determine separation locations and may fulfill the need, perceived by the team, for a frequency display. The system is expected to be installed by this summer and fully operational by the end of the year.

TVA does not do regular governor testing on its non-nuclear generation. The team recommends that TVA record the generator output response to interconnection frequency changes to verify that the governors are adequately responding to frequency changes. The existing EMS could be programmed to capture this data and measure the generator's response.

TVA has about 40% of its load covered by underfrequency load-shed relays. This load will be dropped in accordance to SERC policy. The operators interviewed were not sure at what frequency these relays operated.

The operators were also unsure at what frequency generator underfrequency relays operate.

Management stated that TVA does not use generator under frequency tripping except on their nuclear plants and at one large fossil fuel plant. TVA relies on its underfrequency load-shedding program for the protection of its generation assets.

NERC 2004 Control Area Readiness Audit The operators stated that they would follow reliability coordinator directives to resolve any unacceptable frequency levels.

4.6. Voltage Monitoring The control area operator must monitor voltage levels, and take appropriate actions to support the bulk electric system voltage if real-time trends are indicative of potentially developing problems.

Voltage measuring points must be of sufficient number and from several locations and voltage levels to allow the control area operator to effectively monitor the voltage profile of their control area.

Audit Notes:

TVA monitors the voltage on all of its approximately 400 high voltage substations and thus, has adequate voltage monitoring points. Automatic voltage regulators (AVR) at the plants are not telemetered into the EMS but the operators get their status daily from the power plant operators. All of the AVRs are in service and are only out of service on an exception basis. All TVA and IPP plants are required to have AVRs.

TVA does not have any power system stabilizers in service.

TVA has voltage schedules set by system planning and the operators maintain these schedules. The minimum voltage is 95% with a 93% post contingency. TVA takes action to maintain the post contingency voltage above 93% prior to the contingency occurring. These voltages are coordinated with neighboring control areas. The voltage schedules are available to the operators on the internal website.

TVA has a procedure for de-energizing high voltage lines to reduce voltage when necessary. This procedure is covered in the 'Electric System Operations Manual, Second Edition.' The procedure has not been used for approximately five years.

TVA does not run online voltage collapse monitoring but does do annual seasonal voltage collapse studies. The planning department determines voltage schedules from studies along with predicted voltage levels after first contingencies. TVA operates to the schedules provided by planning.

4.7. Reactive Reserve The control area must ensure that reactive reserves are available and properly located to satisfy the most severe single contingency.

Audit Notes:

TVA has a policy of maintaining the reactive loading on its generations at or below 30% of capacity in order to maintain 70% reactive reserves on its units. This is accomplished by switching static reserves in and out as needed. This policy is covered in SOMP 3.03 titled "Contingency Reserve Requirements," which "provides guidance for the type and quantity of generation reserves that should be maintained by the balancing authority."

Capacitor banks are put into service based on voltage schedules and the need to maintain reactive reserves on generators.

TVA provides a voltage schedule to all power plants. The interconnection agreement with the IPPs allows TVA to have them operate with a 95% power factor.

NERC 2004 Control Area Readiness Audit TVA has a "voltage page" display on the EMS that shows the status of static capacitors and a "generation page" display that shows the dynamic reserves on generators. There is also a display screen that shows the system reactive reserves and measures that reserve against the TVA policy.

Since the need for reactive reserves is location dependant, the team recommends adding a display screen that shows how TVA is meeting its dynamic reserve policy by region or other logical geographic area.

Presently, the reactive output of some of the generating units in the TVA control area has been tested, but TVA is developing a testing procedure to verify the reactive capacity of all generation units. The audit team encourages TVA to finish and implement this procedure.

I lApplicable Documents Reviewed I

'Dated

- -lVersion

• lIl Contingency Reserve Requirements 103/01/04 l Rev 0000 4.8. Critical Facility Monitoring Monitoring of facilities that are critical to the reliability of the bulk electrical system is a joint responsibility of the control area operators and reliability coordinators.

There must be an established process to determine which facilities will be considered critical to the reliability of the bulk electrical system, and real-time operating information (data and status).

Operating limits for the critical facilities must be provided to the control area operators and the reliability coordinators.

Audit Notes:

TVA has a list of critical facilities that are monitored and meet the interconnected reliability operating limits. The operators stated that they would respond immediately if a limit was exceeded by redispatching, initiating a TLR, using voltage control, or reconfiguring the system. The operators receive alarms at 85% of the actual line ratings. Since it currently does not do real-time contingency analysis, TVA does not alarm on post-contingency flows.

4.9. Transmission System Congestion The control area must monitor transmission flow gates and be prepared to take actions to alleviate congestion in conjunction with and as directed by its reliability coordinator.

Audit Notes:

TVA monitors the current flow on flow gates but does not use real-time contingency analysis. TVA calculates the first contingency total tranasfer capability (FCTTC)for each interface using load flow analysis tools. The lesser of the FCTTC or the contract path limit is used to establish total transfer capability, which is adjusted by transmission reserve margin (TRM) and capacity benefit margin (CBM) to produce available transfer capacity (ATC) scheduling limits for each interface.

Congestion relief actions are predefined for each interface. To relieve transmission congestion, system operators first try redispatch or reconfiguration. If the congestion is not relieved, the operator asks the reliability coordinator to issue a TLR or take other action as necessary. The operators stated that TVA has always responded to TLR curtailments.

NERC 2004 Control Area Readiness Audit All transactions are tagged in, out and through TVA. The only exception would be an emergency transaction that lasted less that one hour. TVA verifies net schedules prior to each hour or prior to any schedule change. After-the-fact interchange checkouts are done three times per day.

4.10.

Load Generation Balance The control area operator must monitor the balance of load, generation and net schedule interchange in his/her control area. The control area operator must take actions to mitigate unacceptable load, generation and net scheduled interchange imbalance.

Audit Notes:

TVA manages its interchange to stay within system requirements including generation ramping capacity. The planning department forecasts the hourly ramping requirement; the balancing authority determines what ramping capability is available on AGC. Hydro, quick start combustion turbines, and pumped storage units are used to make up any deficiency in the AGC and the ramping requirements. The operator checks the net interchange schedule changes from the tagging display and compares that data with current information from planning. The difference is inputted into the EMS so units with AGC will adjust generation.

TVA considers any area control error (ACE) above 131 MW (LI0) to be a large imbalance. The operator has several options to correct generation-load imbalance. These include:

Reduce or increase generation

• Use interruptible load

• Make emergency purchases Use reserves TVA has six steps to load curtailment. (See Section 9 of this report) They are:

In-house energy reduction Public appeals Voltage reduction

• Industrials curtailment

• Residential curtailment Manual load shedding The reliability coordinator is also monitoring the ACE of the control area and wvill call the control area after ten minutes of large load imbalance.

TVA has met CPSI and CPS2 for all months through September 24, 2003.

4.11.

Contingency Reserves The control area operator must monitor the required reserves, and the actual operating reserves in real-time, and must take action to restore acceptable reserve levels when reserve shortages are identified.

Audit Notes:

TVA is not a part of a reserve-sharing group. The largest unit on the system is 1,340 MW. TVA carries about 340 MW of spinning reserve and normally has approximately 3,000 MW of interruptible NERC 2004 Control Area Readiness Audit load that can be dropped within five minutes. In addition to the above 340 MW, there is additional capacity for regulating margin.

TVA has a display that monitors the contingency reserves. Alarms are activated if the operating reserve criteria is insufficient. TVA carries contingency reserves equal to or greater than the largest TVA unit on line and regulating reserves of 1% of the peak load (above the contingency reserves) forecasted for the day.

4.12.

Special Protection Systems The control area operator and the reliability coordinator must be aware of the operational condition of special protection systems (SPS) that may have an effect on the operation of the bulk electrical system.

Audit Notes:

TVA does not have any special protective systems.

Procedure ESO-RA-SPP-10.200 Rev. 0000 dated 11-05-02 Section 3 M states that "No special protection systems are used within the TVA reliability area."

[ lApplicable Documents Reviewed IDated -

Version -

I 1 Procedure ESO-RA-SPP-10.200 11/05/02 IRevO0000

5. System Restoration The control area operator must have a documented system restoration plan that must be provided to the reliability coordinator.

The control area operator must be prepared to restore his/her control area following a partial or total collapse of the system and coordinate system restoration with the neighboring control areas and with the reliability coordinators.

Audit Notes:

TVA system restoration program includes

• Documentation

• Training

• Testing

• Drilling The blackstart procedures and the restoration plan are incorporated by reference into the transmission emergency plan as section seven. The plan is reviewed and updated annually.

The role of the reliability coordinator is defined in the reliability plan section nine. Appendix E refers to the sub-region individual member plans. The reliability coordinator or the individual member initiates the coordination of the restoration with another member.

Fuel resources are as identified by the fossil fuels -

fuel supply summary, which includes the blackstart plants and itemizes inventory for the plants including inventory targets.

NERC 2004 Control Area Readiness Audit Restoration plans include one-line diagrams, instructions and breaker positions for the restoration. Each plant in the plan has communication instructions and procedures.

Based on the blackstart procedure, the power dispatcher, with reliability coordinator approval and coordination, can give direct restoration instructions to all plants and stations during major disturbances.

However, as a result of lack of communications, it may be necessary for individual generators to follow pre-planned procedures. These procedures are sufficiently detailed to restore power to the entire system, if need be.

Restoration plans are on the TVA intranet and available to the operator. In the event the intranet is down, hard copies are available.

Training includes several modules on system restoration, including Synchronization of islands

• Contingency plans for failed resources Restoration drills are conducted twice per year.

Each year, TVA performs load curtailment exercises in the spring and transmission exercises, such as restoration from a major ice storm, in the fall.

TVA has done blackstart tabletop exercises using steam units, and actual blackstart drills on three hydro units. Plants in the plan are started annually to the point of synchronizing to a dead bus. Every hydro and steam plant has a blackstart procedure. All blackstart units are tested every three years.

TVA has not used its restoration under actual blackout conditions.

Applicable Documents Reviewed Dated Version I

Transmission Emergency Plan 10/01/03 Rev 2 2

TVA Sub-region Reliability Plan 11105/02 3

Fuel Supply Summary April 2001

6. Delegation of Reliability Authority Functions Any reliability coordinator functions that have been delegated to a control area operator must be clearly documented. The documentation must recognize that the reliability coordinator continues to be responsible for that function.

Amudit Notes:

TVA is both the reliability coordinator and the control area operator for its footprint. TVA is also the reliability coordinator for three additional control areas. No reliability coordinator or control area functions are delegated outside of the TVA organization.

The procedure ESO-RA-SPP-10.200, Section 3 describes the functions performed by the reliability coordinator.

I Applicable Documents Dated version I IESO-RA-SPP-10.200 111/05/02 JRevO0000 NERC 2004 Control Area Readiness Audit

7. Outage Coordination Planned control area transmission facilities and generating unit outages must be coordinated with the reliability coordinator to ensure thai conflicting outages do not jeopardize the reliability of the bulk electrical system.

Information relative to forced transmission facilities and generating unit outages that may jeopardize the reliability of the bulk electrical system must be shared with affected transmission operators and the control area's reliability coordinator as expeditiously as possible.

A udit Notes:

The TVA sub-region reliability plan requires that operational reliability information is communicated from each operational entity in the sub-region to the reliability coordinator on the SCADA or through the ISN at a five-minute rate or faster.

TVA's procedure titled "transmission outage scheduling and coordination process" places the reliability coordinator in the approval process for planned outages.

In accordance with TVA's security data exchange (SDX) procedures, transmission and generation outage information is posted via SDX. The SDX is updated each hour or when the status of a line changes.

TVA is planning to update the SDX every 20 minutes in the future. In addition, planned generation and transmission outage information is made available to other entities via the SERC outage coordination process.

Transmission scheduled outages require a seven-day notification. Senior transmission operators review and forward outage notifications to system planning for study. Scheduled outages of generating units are reviewed and updated in the asset database. Operations will approve or deny transmission outage requests and input approved outages into the database for coordination studies. Neighboring systems are also included in both transmission and generation outage scheduling. TVA operators have the authority to cancel scheduled outage and have done so.

Applicable Documents Reviewed Dated

-version I

I TVA Sub-region Reliability Plan 11/05/02 2

Transmission Outage Scheduling and Coordination Process 04/05/04

8. Transmission and Generation Relaying Control areas must ensure that transmission and generator relay maintenance is carried out as per control area, Regional, and/or NERC established requirements.

Audit Notes:

TVA has relay maintenance data from both TVA owned plants and IPPs.

TVA's system performance and analysis staff employs electrical engineers who specialize in protective relaying. To ensure proper coordination and optimal protection, these engineers contact adjacent companies' engineers to share settings when work is done on any interconnected line.

IPP interconnection agreements provide for receipt of testing data from generators. Initial information is obtained during the interconnection approval process and is periodically updated. TVA does not currently have independent verification of generator information except to monitor how they follow their voltage schedule. IPPs must meet a 95% power factor requirement and operate to a voltage schedule.

NERC 2004 Control Area Readiness Audit TVA has Operating Committee meetings with its IPPs on a regular basis, as required by the interconnection agreements.

TVA operators are aware of the status of protective relaying and must give permission to take protective relaying systems out of service.

I lAppicabieDocuments Reviewed :

-.Dated I---

Version I I Interconnection Agreement I

I

9. Capacity and Energy Emergency Plan Each control area must have a capacity and energy emergency plan that address the following functions.

(It should be noted that some of the items might not be applicable, as the responsibilities for the item may not rest with the entity being reviewed.)

TVA provided the TVA/Distributor Emergency Load Curtailment Plan (ELCP). This plan covers most of the following functions and the section that pertains to the function is listed below that function. The ELCP covers the reduction of load when supply cannot meet demand.

1. Coordinating functions. The functions to be coordinated with and among neighboring systems.

(The plan should include references to coordination of actions among neighboring systems when the plans are implemented.)

Included in the ELCP Introduction (dated May 17, 2001).

2. Fuel supply. An adequate fuel supply and inventory plan which recognizes reasonable delays or problems in the delivery or production of fuel, fuel switching plans for units for which fuel supply shortages may occur, e.g., gas and light oil, and a plan to optimize all generating sources to optimize the availability of the fuel, if fuel is in short supply.

Fossil fuels fuel supply summary lists the inventory for all TVA fossil plants.

3. Environmental constraints. Plans to seek removal of environmental constraints for generating units and plants.

ELCP Tab M, Power Supply Priorities Item 39 states that the "Senior Vice President of Fossil and Hydro Power makes that decision."

4. System energy use. The reduction of the system's own energy use to a minimum.

ELCP Step 10, Reduction by TVA and Distributors (dated August 1, 2000) states, "TVA and distributors will immediately reduce non-essential, in-house loads in their plants, offices and warehouse."

This issue is also addressed in ELCP Tab M, Step 34 (dated March 23, 2002)

5. Public appeals. Appeals to the public through all media for voluntary load reductions and energy conservation including educational messages on how to accomplish such load reduction and conservation.

Covered in ELCP Step 20, Press Release and ELCP Tab M Step 35 (dated March 23, 2002)

NERC 2004 Control Area Readiness Audit

6. Load management. Implementation of load management and voltage reductions.

Addressed by ELCP Step 30, Voltage Reduction and ELCP Tab M, Step 13 (dated March 23, 2002).

7. Appeals to large customers. Appeals to large industrial and commercial customers to reduce non-essential energy use and start any customer-owned backup generation.

Addressed in ELCP Step 20, Press Releases, the Flowchart "TVA-Distributor Load Curtailment Plan Notification" and ELCP Tab M, Step 35 (dated March 23, 2002).

8. Interruptible and curtailable loads. Use of interruptible and curtailable customer load to reduce capacity requirements or to conserve the fuel in short supply.

Covered in ELCP Tab M, Steps 10, 20, 21, 22, and 24 through 28 (dated March 23, 2002).

9. Maximizing generator output and availability. The operation of all generating sources to maximize output and availability. This should include plans to winterize units and plants during extremely cold weather.

Covered in ELCP Tab M, Steps 2, 3, 8, 12, 23, 30 (dated March 23, 2002).

10. Notifying IPPs. Notification of co-generation and independent power producers to maximize output and availability.

Addressed in ELCP Tab M Step 14 (dated March 23, 2002).

11. Load curtailment. A mandatory load curtailment plan to use as a last resort. This plan should address the needs of critical loads essential to the health, safety, and welfare of the community.

Covered in ELCP Step 40.

12. Notification of government agencies. Notification of appropriate government agencies as the various steps of the emergency plan are implemented Covered in ELCP Tab M dated Step 36 (March 23, 2002) and the Transmission Emergency Plan Section 7.3.State/Federal Coordination

13. Notification to control areas and reliability coordinators. Notification should be made to other control areas and to the reliability coordinator as the steps of the emergency plan are implemented.

This issue is covered in TVA's energy emergency alert procedure.

Audit Notes:

The emergency load curtailment plan outlines steps that cover the above items. The book is distributed within TVA and to the various distribution companies in the TVA control area. The plan "was developed to provide arrangements and contingency plans to meet power system emergencies that might cause NERC 2004 Control Area Readiness Audit temporary shortages of bulk power supply in localized areas of the system or generally throughout the system." The plan is often used since it includes a list of TVA's interruptible customers.

The plan is organized in one binder but its organization was somewhat confusing and plan pages had different revision dates. TVA is aware of this and is in the process of updating and reorganizing the plan.

TVA stated in the self assesment, "TVA's fuel supply is continuously monitored for adequacy by an internal organization. In general, TVA generation facilities maintian approximately 30 days coal supply at each location, but this may vary from time to time. The combustion turbines have dual fuel capability with approximately one week's supply of oil on site." The audit team reviewed the fuel supply summary that showed the inventory and delivery scheduled for all TVA generation to confirm conformance to this policy. The fuel supply summary stated "With dual fuel combustion turbines, if one fuel source is low TVA would switch to the other fuel type."

Applicable Documents Reviewed -

I Dated --- :

Version I

Emergency Load Curtailment Plan Various Various 2

Energy Emergency Alert Procedure l

3 Fuel Supply Summary 04/09/2004

10. Operating Policy/Procedure Changes Control areas must have an established procedure to ensure that control area operators and operations staff are aware of any changes to NERC, Regional and/or local policies or procedures prior to taking over control of a shift position.

Control areas must have shift change procedures for updating incoming shift personnel on the current status of the system.

Audit Notes:

All TVA's documentation is complete and well organized in hard copy and on the TVA internal website.

The team feels TVA's documentation effort is a 'best practice.' TVA converted the documents into a standard format, referred to as the standard processes and procedures (SPP). Many procedures have been consolidated into one manual called the System Operators' Procedures Manual (SOPM). Most policy and procedures are available on the internal website and the operators and management personnel interviewed were able to access these documents easily. Distribution of the SPP and SOPM is controlled. Each is individually numbered and TVA insures that they contained the latest procedures. Each standard processes and procedures has a document reference number and a separate section number in the SOPM.

This gives each document two-reference numbers, which can be confusing at times but does not seem to be an issue with TVA personnel. Each policy, whether in the SOPM or not, has an 'owner' who can be contacted to resolve issues or questions with the policy or procedure.

The TVA, SERC, and NERC policy and procedures are updated on the TVA internal website. Hard copies are kept in the control room in case the website is not available. A secretary updates them immediately after policies are changed. The operators agreed that the book was up to date.

TVA provided two procedures, one for tracking policy and procedures changes and one for informing the next shift.

• SOPM 4.03 North American Electric Reliability Council (NERC) Policy Change Procedure

• SOPM 6.01 Shift Turnover Process NERC 2004 Control Area Readiness Audit TVA sends its personnel emails on policy and procedure changes. The email requires that the employee indicate that s/he has read and understood the change. The electronic copy is color coded to highlight new procedures and changes.

Applicable Documents Reviewed 1 SOPM 4.03 NERC Policy Change Procedure 2 1 SOPM 6.01 Shift Turnover Process Dated 04/12/04 10/10/03 Version Rev 0000 Rev 0000 II

11. Vegetation Management (Line Clearances)

Control areas must have a documented Vegetation Management program.

A udit Notes:

Procedure SAM-SDP-06.001 addresses the vegetation program. The scope of this document includes:

• Inspection program

• Annual maintenance plan

• Documentation procedures a

Clearances for re-clearing undergrowth

• Clearances for removing danger trees Paragraph 5 states, "The span by span inspection results are maintained in the vegetation management organizations data system. The assigned lines are entered into the work management system as individual work orders along with any other emerging work that arises during the work year. Progress is tracked monthly in the work management system."

In accordance with the procedure, "Each year in the late spring -

early summer time frame a span by span helicopter patrol of all EHV lines is performed." Operators confirmed that this procedure was followed.

The operators could not recall any unexplained line trips where vegetation might have been the cause.

They have had vegetation caused trips, but not many and never from overgrown vegetation. While this was a problem 10 years ago, it has been resolved in recent years.

l I,pplicable Documents Reviewed l1lSAM-SDP-06.001 jDated -

Version-l 104/16/04 l Rev 0000 l

12. Nuclear Power Plant Requirements Nuclear power plants have regulatory requirement for voltage and power in both normal and abnormal operating conditions (N-I and system restoration).

Audit Notes:

The agreements with the nuclear stations in the TVA control area, identifying critical transmission line configurations and/or voltages, are listed below:

• Browns Ferry Nuclear Plant -

161 and 500-kV Grid Voltage Schedules and Operating Instructions, Revision I dated September 14, 2001, Sequoyah Nuclear Plant -

Grid Operating Instructions, dated July 13, 2003, NERC 2004 Control Area Readiness Audit

• Watts Bar Nuclear Plant 161 and 500 kV Grid Voltage Schedules and Operating Instructions dated August 31, 2001.

Each plant has different voltage criteria. Transmission planning has a full-time engineer doing studies related to the nuclear plants. While the engineers work day shifts, one is on call 24-hours a day. The operating instructions address shutdown-loads, the 500 kV bus voltage during a loss of coolant accident (LOCA), plus the loss of any single transmission element.

The operators are familiar with the special requirements for nuclear power plants. They stated that restoration of power to nuclear power plants was the highest priority.

Planning staff stated that studies to analyze the nuclear plant voltage limits, considering a trip of the NPP unit and the N-l contingencies, are done once every three years or when required. There is an engineer dedicated full-time to nuclear off-site power supply issues. Planners are aware of voltage criteria of each plant. Planners reference and document plant information in reports and in the studies they perform.

TVA has considered the effects of reduced Mvar capacity from nuclear power uprates.

Applicable Documents Dated Version Browns Ferry Nuclear Plant-161 and 500-kV I

Grid Voltage Schedules and Operating Instructions 09114/01 Rev 1 2

Sequoyah Nuclear Plant - Grid Operating Instructions 07/13/03 Watts Bar Nuclear Plant 161 and 500-kV Grid 3

Voltage Schedules and Operating Instructions 08/31/01 1

NERC 2004 Control Area Readiness Audit Conclusions TVA is very strong in the following areas:

• TVA will have 'best practice' training program when its simulator is installed and the operators receive training concerning reliability functions performed by others.

As discussed in the training section, TVA has a well-organized and documented training program. TVA does an excellent job of evaluating training needs and providing training to meet these needs.

• TVA has 'best practice' backup capabilities.

TVA has two fully functioning control centers; one for the control area operations and one for reliability coordination. Each control center backs up the other. If TVA were to lose the service of one of the control centers, the function of that control center would be transferred automatically to the other control center.

• TVA has 'best practice' redundant computer hardware and communications facilities.

TVA has fully redundant hardware at each control center and independent communication paths at each control center for voice and data. Systems automatically fail-over to the alternative source, making power system monitoring equipment outages very unusual.

• TVA has "best practice" control system monitoring system.

TVA developed a network operations center (NOC) that monitors the status and condition of the computer and communication support systems around the clock. TVA has maintenance personnel on all shifts, with additional people on call across the TVA system.

TVA has a "best practice" wide-area view of the TVA and neighboring electrical systems.

TVA is addressing the recognized industry need for wide-area-visualization by developing operator tools to graphically display voltage levels and power flows across its system.

• TVA has "best practice" documentation.

TVA has done an excellent job of developing and organizing the required documentation. Most of the policies and procedures are available online and TVA continues to expand the web access.

• TVA has a staff meteorologist TVA has a meteorologist on staff to provide and present weather forecasts, in a user-friendly format, to the operators.

The audit team would like to thank the TVA staff for its hospitality and cooperation during the audit process. The audit team trusts that our findings are consistent with the staff's understanding of the control area's role and that the audit team recommendations will be of value.