ML15261A237
| ML15261A237 | |
| Person / Time | |
|---|---|
| Site: | Oconee, Mcguire, Catawba, McGuire  |
| Issue date: | 09/01/1999 |
| From: | Christensen H NRC (Affiliation Not Assigned) |
| To: | Tuckman M DUKE POWER CO. |
| References | |
| NUDOCS 9909150043 | |
| Download: ML15261A237 (89) | |
Text
- -CATEGORY 2.
REGULATORY INFORMATION DISTRIBUTION SYSTEM (RIDS)
ACCESSION NBR:9909150043 DOC.DATE: 99/09/01 NOTARIZED: NO DOCKET #
FACIL:50-269 Oconee Nuclear Station, Unit 1, Duke Power Co.
05000269 50-270 Oconee Nuclear Station, Unit 2, Duke Power Co.
05000270 50-287 Oconee Nuclear Station, Unit 3, Duke Power Co.
05000287 50-369 William B. McGuire Nuclear Station, Unit 1, Duke Powe 05000369 50-370 William B. McGuire Nuclear Station, Unit 2, Duke Powe 05000370 50-413 Catawba Nuclear Station, Unit 1, Duke Power Co.
05000413 50-414 Catawba Nuclear Station, Unit 2, Duke Power Co.
05000414 AUTH.NAME AUTHOR AFFILIATION C
CHRISTENSEN, H.
RECIP.NAME RECIPIENT AFFILIATION A
TUCKMANM.S.
Duke Power Co.0
SUBJECT:
Submits traning manager conference meeting summary of meeting conducted on 990812-13 on recent changes to operator E
licensing program.List of attendees & matl used in presentation enclosed.
G DISTRIBUTION CODE: IE42D COPIES RECEIVED:LTR ENCL SIZE:______
TITLE: Operator Licensing Examination Reports R
NOTES:
Y RECIPIENT COPIES RECIPIENT COPIES ID CODE/NAME LTTR ENCL ID CODE/NAME LTTR ENCL LPD2-1 PD 1
1 LABARGE,D 1
1 2
RINALDI,F 1
1 TAM,P.
1 1
INTERNAL: ACRS 1
1 NT 1
1 NRR/DIPM/OLHP 1
1 RGN2 FILE 01 1
1 D
EXTERNAL: NOAC 1
1 NRC PDR 1
1 0
RLs4PrT
\\
/
c C
z T
NOTE TO ALL "RIDS" RECIPIENTS:
PLEASE HELP US TO REDUCE WASTE.
TO HAVE YOUR NAME OR ORGANIZATION REMOVED FROM DISTRIBUTION LISTS OR REDUCE THE NUMBER OF COPIES RECEIVED BY YOU OR YOUR ORGANIZATION, CONTACT THE DOCUMENT CONTROL DESK (DCD) ON EXTENSION 415-2083 TOTAL NUMBER OF COPIES REQUIRED: LTTR
'16Q ENCL 14
September 1, 1999 Duke Energy Corporation ATTN: Mr. M. S. Tuckman Executive Vice President Nuclear Generation 526 South Church Street P. 0. Box 1006 Charlotte, NC 28201-1006
Dear Mr. Tuckman:
SUBJECT:
TRAINING MANAGERS CONFERENCE MEETING
SUMMARY
On August 12 - 13, 1999, Region II hosted a Training Managers Conference on recent changes to the operator licensing program. The meeting covered changes to the Regulations, the Examination Standards (NUREG 1021), the new inspection program, and other training issues. is the list of attendees and Enclosure 2 is a copy of the slide presentations. is a list of questions received from the participants. These questions will be reviewed and addressed at a future date.
If you have any questions concerning the conference, please contact me at 404-562-4638.
Sincerely, Original signed by Harold 0. Christensen Harold 0. Christensen, Chief Operator Licensing and Human Performance Branch Division of Reactor Safety Docket Nos. 50-413, 50-414, 50-369, 50-370 50-269, 50-270, and 50-287 License Nos. NPF-35, NPF-52, NPF-9, NPF-17 DPR-38, DPR-47, and DPR-55
Enclosures:
- 1.
List of Attendees
- 2.
Licensee Presentation Handouts
- 3.
Participants Questions cc w/encls: (See page 2) 9909150043 990901 PDR ADOCK 05000269 V
- A g
DEC 2
cc w/encls:
L. A. Keller, Manager G. R. Peterson Nuclear Regulatory Licensing Site Vice President Duke Energy Corporation Catawba Site 526 S. Church Street 4800 Concord Road Charlotte, NC 28201-0006 York, SC 29745-9635 H. B. Barron Station Manager Vice President Regulatory Compliance Manager (MNS)
McGuire Site Duke Energy Corporation 12700 Hagers Ferry Road Electronic Mail Distribution Huntersville, NC 28078-8985 Compliance Manager (ONS).
W. R. McCollum Duke Energy Corporation Vice President Electronic Mail Distribution Oconee Site 7800 Rochester Highway Seneca, SC 29672 A. Lindsey Training Manager Catawba Nuclear Station 4850 Concord Road York, SC 29745-9635 Alan Orton, Operation Training Manager McGuire Training & Technology Center 13339 Hagers Ferry Road Huntersville, NC 28078-8985 Bentley K. Jones Training Manager Oconee Nuclear Plant P. 0. Box 1439 Seneca, SC 29679 G. D. Gilbert Regulatory Compliance Manager Electronic Mail Distribution Distribution w/encl: (See page 3)
DEC 3
Distribution w/encl:
P. Tam, NRR F. Rinaldi, NRR D. LaBarge, NRR R. Gallo, NRR D. Trimble, NRR R. Conte, RI D. Hill, RII J. Pellet, RIV PUBLIC OFFICE RII:DRS RII:DRP SIGNATURE
( j I NAME HChristensen COgle DATE 1
/99 8/
/99 8/
/99 8/
/99 8/
/99 8/
/99 COPY?
NO YES NO YES NO YES NO YES NO YES NO YES NO OF AL RECORD COPY DOCUMENT NAME:
G:\\DOCUMENT\\DEC\\DECsumary.wpd
LIST OF ATTENDEES U. S. NUCLEAR REGULATORY COMMISSION Luis A. Reyes, Regional Administrator Victor McCree, Deputy Director, Division of Reactor Safety (DRS)
Harold Christensen, Chief, Operator Licensing & Human Performance Branch, (OLHP), DRS Fred Guenther, Senior Reactor Engineer, Nuclear Reactor Regulator Ronald F. Aiello, Reactor Engineer, OLHP, DRS Richard S. Baldwin, Reactor Engineer, OLHP, DRS Michael E. Ernstes, Reactor Engineer, OLHP, DRS George T. Hopper, Reactor Engineer, OLHP, DRS Larry S. Mellen, Reactor Engineer, OLHP, DRS Beverly Michael, Licensing Assistant, OLHP, DRS Mark S. Miller, Reactor Engineer, OLHP, DRS Charles Payne, Reactor Engineer, OLHP, DRS Marvin Skyes, Reactor Engineer, OLHP, DRS Hironori Peterson, Senior Examiner, Region III LICENSEE CAROLINA POWER AND LIGHT COMPANY Max Herrell, Training Manager, Brunswick Ralph Mullis, Operations Training Superintendent, Brunswick Gregg Ludlam, Supervisor - Operator Continue Training, Brunswick Mark Keef, Training Manager, Harris Thomas Natale, Operations Training Manager, Robinson DUKE ENERGY CORPORATION Al Lindsay, Training Manager, Catawba James Teofilac, Operations Training Manager, Catawba Al Orton, Operations Training Manager, McGuire Ronnie White, Site Training Manager, McGuire Robby Pope, Supervisor of License Requal Training, McGuire Gary Veller, Operations Human Performance Manager, McGuire Tom Coutu, Superintendent of Operations, Oconee.
John Steely, Supervisor Nuclear Operator Training, Oconee Paul Stovall, Manager Operator Training, Oconee Scott Hollingsworth, Operations Training Liaison, Oconee
-Rick Robinson, Operations Training Liaison, Oconee Jack Brission, Operations, Oconee
2 FLORIDA POWER AND LIGHT COMPANY Mark Shepard, Operations Training Supervisor, St. Lucie Jo Magennis, Nuclear Assurance, St. Lucie Maria Lacal, Training Manager, Turkey Point Phillip Finegan, Operations Training Supervisor, Turkey Point Bill Burrow, Online Schedule Supervisor, Turkey Point FLORIDA POWER CORPORATION Ken McCall, Operations Training Manager, Crystal River Frank Dola, Senior Nuclear Operations Specialist, Crystal River Tony Roberts, {FRG Corporation) Representative SOUTHERN NUCLEAR OPERATING COMPANY. INC.
Scott Fulmer, Training Manager, Farley Joe Powell, Nuclear Operations Senior Instructor, Farley Gerry Laska, Nuclear Operations Instructor, Farley Gary O'Hustede, Operation Training Plant Instructor, Farley John Lewis, Training Manager, Hatch Steven Grantham, Operations Training Supervisor, Hatch Robert Brown, Plant Training & Emergency Preparedness Manager, Vogtle SOUTH CAROLINA ELECTRIC AND GAS COMPANY Albert Koon, Operations Training Manager, Summer Perry Ramicone, Lead Instructor Exam Development, Summer James Callicott, Training Evaluation Coordinator, Summer TENNESSEE VALLEY AUTHORITY Daniel Sanchez, Training Manager, Browns Ferry Ardie Champion, Operations Training Manage, Browns Ferry Denny Campbell, Shift Operations Supervisor Instructor, Browns Ferry John Parshall, Shift Operations Supervisor Instructor, Browns Ferry Richared Driscoll, Training Manager, Sequoyah Walt Hunt, Operations Training Manager, Sequoyah John Rodden, Operations Training Manager, Watts Bar Tom Wallace, Operations Superintendent, Watts Bar VIRGINIA ELECTRIC AND POWER COMPANY Joe Scott, Operations Training Supervisor, North Anna Steve Crawford, Senior Instructor Nuclear, North Anna David Llewellyn, Superintendent of Nuclear Training, Surry Harold McCallum, Operations Training Supervisor, Surry Michael Brady, Supervisor of Nuclear Training, Surry OTHERS James Makucin, INPO Bob Post, NEI
R&
CO WELCOME TO U. S. NUCLEAR REGULATORY COMMISSION REGION II TRAINING MANAGER'S CONFERENCE AUGUST 12-13, 1999 SAM NUNN ATLANTA FEDERAL CENTER 61 FORSYTH SW, SUITE 23T85 ATLANTA, GA 30303 e
-ENCLOSURE 2
REGION II TRAINING MANAGER CONFERENCE AGENDA Sam Nunn Atlanta Federal Center (Bridge Conference Room B)
Thursday, August 12, 1999 8:30 - 9:00 a.m.
Introduction L. Reyes V. McCree C. Christensen 9:00 - 10:00 a.m.
Summary of Significant C. Christensen NUREG 1021 Changes 10:00 -10:15 a.m.
Break 10:15 - 11:45 a.m.
ES-200 Series (Exam Process)
M. Emstes 11:45 a.m. - 1:00 p.m.
Lunch 1:00 - 2:00 p.m.
Changes in the NRC B. Boger 2:00 - 2:15 p.m.
Break 2:15 - 3:45 p.m.
ES-300 Series (Operating Test)
R. Aiello 3:45 -*4:00 p.m.
Break 4:00 - 5:00 p.m.
New Inspection Program C. Christensen Friday, August 13, 1999 8:00 - 8:15 a.m.
Day 2 Opening Remarks C. Christensen 8:15 - 9:45 a.m.
ES-400 Series (Written Exams)
R. Baldwin 9:45 - 10:00 a.m.
Break 10:00 - 11:00 a.m.
ES-500 Series C. Payne (Post-Exam Process) 11:00 a.m. - 12:00 p.m.
Recent Training Program Issues G. Hopper 12:00 - 1:00 p.m.
Lunch 1:00 - 2:00 p.m.
ES - 600 Series (Requal Program)
M. Sykes G. Hopper 2:00 - 3:00 p.m.
Closing Remarks L. Reyes V. McCree C. Christensen
TRAINING MANAGERS CONFERENCE August 12 - 13, 1999 TRAINING MANAGERS CONFERENCE INTRODUCTION Luis Reyes Victor McCree Chris Christensen TRAINING MANAGERS CONFERENCE AGENDA - DAY ONE 8:30 a.m.
Introduction
- 9:00 a.m.
NUREG 1021 Changes
-10:00 a.m.
Break 10:15 a.m.
Exam Process
-11:45 a.m.
Lunch 1:00 p.m.
Changes in the NRC 2:00 p.m.
Break 2:15 p.m.
Operating Test 3:45 p.m.
Break 4:00 p.m.
New Inspection Program
- 5:00 p.m.
End Day One
TRAINING MANAGERS CONFERENCE VICTOR M.
McCREE Deputy Director, Division of Reactor Safety Region 11 August 12 -13,1999 REGULATORY TRENDS
- BACKGROUND
- DESCRIBE THE CHALLENGE FACING NRR
- EXPlAIN HOW NRR IS MEETING THE CHALLENGE
- DISCUSS HOW NRR IS DEFINING SUCCESS
- DISCUSS STATUS OF HIGH PRIORITY AREAS FOR ACTION
- THEN, NOW AND THE FUTURE BACKGROUND
- U.S. Nuclear Reasor Industry Average Safety Performance Has Improved Steadily
- Number of Accident Sequence Precursors Declined Significamntly Since 1984
- Five of The NRC Tracked Performance Indicators Show Significant improvement Since 1985 (automatic scrams, safety system actuations, significant events, equipment forced outages and collective radiation exposure)
- Challenge Is to Define Programs (in Rapidly Changing Business And Regulatory Environments) at a Level Which:
- Maintain Safety
- Reduce Unnecessary Regulatory Burden
- Increase Public Confidence
- Improve Efficiency And Effectiveness
AREAS OF NRC STAKEHOLDER CONCERN
- Senate & House Committee Reports on NRC Appropriations -Early June, 1998
- Commission Meeting With Stakeholders -July 17 & Nov. 13,1998
- NRC Oversight Hearing With Senate Subcommittee -July 30, 1998
- Regulatory Framework Needs to Be Predictable, Objective, And Timely
- Concern Exists That Some NRC Regulations And Regulator Practice Pose Unnecessary Burden on Licensees
- In Deregulated Electric Utility Environment, Unnecessary Regulatory Burden is ofSignificant Concern to Licensees
- Need For Continuous Improvement in Regulatory Effectiveness And Efficiency POST-HEARING TASKING MEMO
- Memo From Chairman to EDO *August 7,1998
- Identifies Commission Proposed High Priority Areas For Action
- Tasking Memo Response - August 25, 1998-Contains Short And Long Term Actions (Updated Monthly)
- Maintaining Safety Remains Hithest Priority
- Many Tasks Previously identified And in Operating Plan; Remaining Tasks Added
- Some Existing Tasks May Be Appropriate to Slow, Defer, Cancel, Other to Accelerate
- Challenge Is to Maintain Safety While Reducing Unnecessary Burden V
PERFORMANCE GOALS FOCUS ATTENTION TO MEASURE NUCLEAR REACTOR SAFETY PROGRAM OUTCOMES:
- MAINTAIN SAFETY
- REDUCE UNNECESSARY REGULATORY BURDEN
- INCREASE PUBLIC CONFIDENCE
- INCREASE EFFICIENCYIEFFECTIVENESS OF KEY NRC PROCESSES Determined by NRC Nuclear Reactor Regulation Team Working With Contractor to Define And Implement Planning. Budgeting, And Performance Management Process (PBPM)
PRIMARY AREAS OF AGENCY FOCUS a Risk-Informed And Performance-Based Regulation
- Reactor Inspection And Enforcement
- Reactor Licensee Performance Assessment
- Reactor Licensing And Oversight
- NRC Organizational Structure And Resources
- Other Agency Programs And Areas of Focus (i.e. License Transfers, Dry Cask Storage, Decommissioning)
- Uranium Recovery Issues
- Changes to NRC's Hearing Process REACTOR OVERSIGHT PROCESS IMPROVEMENTS ASSESSMENT. INSPECTION AND ENFORCEMENT
- Suspended SALP Program, Modified Periodic Plant Reviews
- Performance Results Will Be Evaluated to Determine When Enhanced NRC Diagnosis of Licensee Performance Is Warranted. A Risk-informed Baseline inspection Program Will Be Performed For All Sites.
- Framework: Performance Indicators And Risk-Informed Inspection Results Will Be Used to Measure Licensee Safety Performance. Results Will Be Evaluated Using Equivalent Risk-Informed Scales. (Thresholds) a Inspections Will Become More Risk-Inlrined and Results Will Be Evaluated For Their Risk Significance Usin: Rules-Based (Examples)
Scale.
- Assessment: a Streamlined, Structured Review Process Will Be Used. An Action Matrix Will Provide Consistency in Making Response Decisions.
REGULATORY FRAMEWORK Please See Handout
ACTION MATRIX Please See Handout
0 S
TRAINING MANAGERS CONFERENCE AGENDA - DAYTWO
- 8:00 a.m.
Opening Remarks
- 8:15 a.m.
Written Examination a 9:45 a.m.
Break a 10:00 a.m.
Post Exam Process
- 11:00 a.m.
Recent Training Program Issues a12:00 p.m.
Lunch a 1:00 p.m.
Requal Program
- 2:00 p.m.
Closing Remarks Question From the Last Training Managers Conference November 5, 1998
- Need to stay flexible on exam schedule and do not wait until the last minute to accept the criteria from licensee. Suggested method was to accept the criteria early in process of the prep week. Would prefer to have 30-45 days prior to the scheduled Exam.
The Finial Rev 8 Examiner Standard now request that Outlines be sent 75 days prior to the exam date and that the Exam be sent 45 days prior to the exam date. This can be negotiated with the Chief Examiner.
Last Conference Questions
- The NRC should publish the exam schedule early. Suggestion was to publish at least 1 year in advance.
An Operator Licensing WEB page will be coming soon and it will have the exam schedules.
Additionally, we have been sending confirmation letters to each licensee on the exam schedule.
- Recommend the NRC turn over the GFES to the Licensee.
- The near term we plan to go to three GFES exams per year. Long Term we plan to develop a computerized GFES.
Last Conference Questions
- The issue of written exams and limited staff in NRC create a "de facto" situation for licensees to have to write their own exams.
For FY2000 we are writing more exams and Headquarters has requested for additional resourses. However you may be requested to write an exam if you want a given date.
- The issue of tying up exam writers.
The security requirements are like Requal exam requirements now.
Questions for last Conference
- There is inconsistency in the examiner standards and the K/A manual. For example the sampling plan and Part 20 references.
Final Rev 8 issued, however we need more information to better answer questions.
In using the Requal procedure, is the NRC going to inspect manipulations for reactivity levels on an individual basis? Or can the record be maintained as a team.
Records should be individual because the license is issued to an individual.
Questions for last Conference Would like NRC clarification on following the 10 CFR for an SAT program rather than following the guidance in the memo once issued by the Director of NRR.
You can follow your SAT based program, however if you have a Tech Spec or FSAR requirement you need to follow those requirements or get them changed.
Written exams are getting harder due to the raising cognitive levels. Can there be less of acceptance % for the higher level.
- Rev 8 placed a limit on the higher level questions 50 -60% and no more. Additionally, the time limit for the exam has been extended to 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />.
SUMMARY
OF SIGNIFICANT CHANGES OR CLARIFICATIONS Chris Christensen
SUMMARY
OF SIGNIFICANT CHANGES OR CLARIFICATIONS
= Changes to 10 CFR 55
- New 10 CFR 55.40 Exams Prepared Using NUREG-1021
- Licensees may Prepare, Proctor and Grade Written Exam
- Licensees May Prepare Operating Test
- Licensees Shall Establish Procedures to Control Exam Security and Integrity When Preparing Examinations Authorized Representative Shall Approve Exams Before Submittal to NRC Licensees Must Receive NRC Approval of Exam
- NRC shall Prepare, Proctor and Grade Examinations Upon Licensee's Written Request
SUMMARY
OF SIGNIFICANT CHANGES OR CLARIFICATIONS Changes to 10 CFR 55 10 CFR 55.49
- Was Revised to Clarify Compromise and Security Expectations
- Changes to NUREG 1021 ES 200 Series: Examination Process
- Due dates for Exam Outline and Draft Exam Advanced
- Personnel Restrictions are Like Requal
- The Region May Approve Separating the Written Exam and Operating Test By up to 30 Days
SUMMARY
OF SIGNIFICANT CHANGES OR CLARIFICATIONS
- Changes to NUREG 1021 ES 300 Series: Operating Tests
- Dominant Accident Sequences Should Be Considered for Sampling During Operating Test
- Prescripted JPM Questions Deleted
- Can use follow-up questions for Cause
- Alternate path JPMs Increased to 40%
No Reuse of Material on Subsequent Days
- STA Use OK per Licensee Practice
SUMMARY
OF SIGNIFICANT CHANGES OR CLARIFICATIONS
- Changes to NUREG 1021 ES 400 Series: Written Examination Systematic Sampling Required for Outline Construction
- Learning Objectives Not Required
- Higher Cognitive Questions 50 - 60% of Exam
- New and Updated Forms 30 Question Sampling Review
- Exam Time Raised to 5 Hours
- Clean Copy of Answer Sheet Required
- ES 500 Series: Post Examination
- May Hold License for 80-81% Passes
- Administrative Review Process Streamlined
- Licensee May be Requested to Provide Reference Material and Technical Information
SUMMARY
OF SIGNIFICANT CHANGE OR CLARIFICATIONS uChanges to NUREG 1021 ES 600 Series: Requalification Test Item Duplication Expectations Clarified
- Licensed Operators Detailed Off-site Clarified Proficiency Watch Expectations Clarified
-50 Question Written Examination
- Appendices
- Guidance for Developing Multiple Choice Questions in App. B
- App. E - Clarified Making Assumptions
- App. F -Defined:
- Responsible Power Plant Experience
- Technical Specifications as a Reference
Examination Process ES-200 Series Mike Emstes ES-201 Examination Process B. Facility developed eninationst must meet the following:
(1) comply with NUREG-1021 (2) establish. inplement and mantain secuinty procesures (3) exam submittals must be approved by an autnored representative (4) NRC must approve the proposed examinations.
C.I.a Requests for NRC administration of examinations must be in writing in accrdance with 10 CFR 55.40 (c).
Respond to NRC arvaual administrtive letter and keep region appraised of any changes. Conact Region II OLS Branch Chief by phone to negotiate exam dates and development options. Partial development may be negotiated.
ES-201 Examination Process C.1.b The agency enforcement policy applies to exam compromise.
Attachment I has a section called *Other Considerations' which was not in Kiermi Rev. 8 which summarzes some exam securty and irnegrity issues C. I.e The amount of referene material requested from the facility licensee will be dusted based on the NRCs level of involvement in te examination d
lopmtent proes s. The Chief examiner will discuss rference material content and due dates during the phone call prior to me 120-day letter.
The licensee is requested to submit tthree copies of te outlines and examrnattons. Only one cpy of the references is required (Region II mequest)
C If A facility supervisor or manager shall independently review the examination outlines and te proposed exams before they are submited to the NRC.
ES-201 Examination Process C.1.g An authorized representlative of the facility licasso shelf approve the befoe sndig tem o te NRC for review. The authiorized
='ev stive is not the same person as the fm 7;
maltrorizea represeitahive will be the "mne person tat te y letter is menit to. He dloes riot nemed to review the lest items nor be an thie secuity agreemnt.
- ai C.i. sltyr encouraged to conicte sqihirt wocs, with the conent for dhfficulty of the NRC prepared asain of the chaniges rost the NRC has directed for its proposed eamn.
C.I.k Psislity will make ar" necssary changes to tie 9enintuboa as agree upon with the NRC.
iC.? c About four months before the eanticxi the Chief Examner will cll the facility to discjes the nie ilms Wied. Negotiation of delivetry dates may be made to solow most effloent review.
C..
Examiears have the option to riot partiipate i the prep week vist.________________________
ES-201 Examination Process CIA. The wrtt and operatig portions of the exams may be spit by Lip to 30 CIJ. Branch Chief will sign toe OA sheets whte is sastisfieid Out On CV
-Sarnpkfrs eview' of the writen examintion shall be complete within amon week of receiig the examntioni and the belance of tw review caomp;et, in iwo weeks after reosipL.
C.3.j SRO upgrades filling an RO or SOP position do rot need lo be evaluated 2fid "Ifiat.______________________
Examination Process Ol2a Fainlity should linit ecoss to only thoe portions whid, tie irclividual bears responsibil y.
ol.b Gives examples of 'prohibited actiities' for individuas an Me Security Agreement.
Supervisors arnd managers on the Secuity Agreement may contiue the general oversightt of Owe bameg program incudigreviewof examnations and remedial training. They may not prvide idividual applcnt feedback
ES-202 Preparing and Review Operator Licensing Applications C.1.a If more than six months have passed since an applicant's medical examination. the factlty shall canry that the applicant has not developed any reportme nten.
C.2.b The requirement for five significant reacivity manipliations is darified.
D.3 Eligibility Oteria for Limited Senior Reacar Operators moved fron ES-701.
ES-204 Processing Waiver Request 0.1.8 A retake examination must take plaince wii one year of the date on whim tie defeal of tie rginal applcanon becamie iat.
D..g The region may waive th imouiWemts for an examination if the appct was prewoualy licensed at the same faclty Must have terminated participation in requal iess than two years ago D.1.h The region may accept application and administer exams to applicants who have not completed their five rsacrvity manipulations due to extended srsAdown. A cold or refueling license will be issued.
ES-205 Generic Fundamentals Examination Program C 1.a Facility censees should notify tie NRR aperator iiwiaing program ti If they add or delete an individual from their previously submitted rag stration letter for tie GFES before the examination is srirastered.
A third GFES is possible in 2000. Ocober 199. Februry 200. & June 2000 are lkely dates.
The GFES exam date is aivays the Weidnesday after the fist Surday of the month
TRAINING MANAGER CONFERENCE REGION II OFFICES AUGUST 12, 1999 BRUCE A. BOGER, DIRECTOR DIVISION OF INSPECTION PROGRAM MANAGEMENT OFFICE OF NUCLEAR REACTOR REGULATION
Office of Nuclear Reactor Regulation Agency Allegatlons Advisor Director Program Management. Policy Deputy Director Development & Analysis Staff Resource information Management Management Branch Branch Associate Director for Associte Director Project Licensing & Technical Analysis for Inspection & Programs Enforcement o------
-coordinator Division DIrector for Divimton Director for Division Director for Division Director for Inspection Division Director for Systems Safety & Analysis Engineering Licensing and Project Program Management Regulatory Improvement Deputy Director Deprector ector Management Deputy Director Programs Depuntin DrcoMan rjc rga aagement Rgltr mrvmn Deputy Director Deputy Director Reactor Systems Branch Mechanical & CMI PO I inspection Program Branch Tech Specifications Brunch Engineering BranchF Plant Systems Branch Materials & Chemical PD II QA. Vendor inspection.
Engineering Branch Maintenance & Allegations Standardization Branch Branch Prob. Safety Electrical and PD III Operator Licensing. Human Events Assessment, Generic Assessment Branch Instrumentation & Controls Performance & Plant Support Communications &
Branch J
Branch Non-Power Reactors Branch PO IVDecommssioning Generic Issues. Environ..
Finan. & Rulemaking Branch
PERFORMANCE GOALS FU OTC FOCUS ON ACHIEVING OUTCOMES
REGULATORY FRAMEWORK NRC's PUBLIC HEALTH AND SAFETY Overall AS A RESULT OF CIVILIAN Sarety NUCLEAR REACTOR Mission OPERATION S~rteicREACTOR RADIATION SFGAD Performance SAFETY SAFETY Areas INITIATING MITIGATION BARRIER EMERGENCV PULC OCPATIONAl PHYSICAL Cornerstones EVENTS SYSTEMS INTEGRITY PREPAREDNESS PROTECPCION HUMAN
SAFETY CONSCIOUS WORK ----------------
PROBLEM --------
PERFORMANCE ENVIRONMENT IDENTIFICATION AND RESOLUTION
- PERFORMANCE INDICATOR
- INSPECTION
- OTHER INFORMATION SOURCES
- DECISION THRESHOLDS
Tale 1 -
- MA*IU 5 fICO hastasecd Requlmid alaay Regdtoury Rspone Band Repnse Ba hfaleg Evets UrIlamed Scrarm per 7000 Oitical Hmu (akatic arx
>ao
>6.0 marm scrarms during the pevict kxr qirers)
Scramns wth a Enss cf Nana Hea Penovai (over tle
>40
>10.0 previcus 12 qurkis)
U ud Otwc viges per 7000 Otica Has (over
>&0 N/A pevius far qtares)
&attrig Se s Iety Systen Unavaiaiity (SSU)
ABnMs I
I (average cf peicus 12 qiuters) fnergency Powerl
>awi
>5.M1
>2EDGI
>anoi
>10.0 MRI I
I
>41
>120%
IHITS 91
>1.9%9ol
>4.P/9 ROCI
>40%1
>1ZOYOI RIES
>20X1
>5.0011 iMil
>2(JX ITNi
> 2% 9
> 5.%
NO AF"1
>2.%11
>&r I 0
>20%
>5.Oo Safety System FincticiA Failures (omer pv ius faur
>5.0 N/A
~quatr)_
Table 1 - PERFORMANCE INDICATORS Cont'd Cornerstone led4cator Threshokls Increased Required Regulatory Regulatory Response Band Response Band.
Barriers I Reactor Coolant System (RCS) Specific Activity (maximum
>50.0%
>100%
4 Fuel Cladding I monthly values, percent of Tech. Spec limit, during previous four. quarters) 4 fReactor Coolant RCS Identified Leak Rate (maximum monthly values,
>50.0%
>100%
System 1 percent of Tech. Spec. limit, during previous four quarters) 4 Containment Containment Leakage (maximum monthly values,
>60.0%
N/A percentage or LA Over tHe previous four quarters)
Emergency DrillExercise Perrormance (over previouseight quarters)
<90.0%
<70.0%
Preparedness ERO Drill Participation (percentage or Key ERO personnel
<80.0% 1
<60.0% 1 that have participated in a drill or exercise in the previous n
eight quarters)
Alert and Notification System Reliability (percentage
<94.0%
<90.0%
reliability during previous rour quarters) occupational Ioccupational Exposure Control Effectiveness (occurrences >5 Radiation Safety duiaprevious 12 quarters)
PubA Radiation RETS/ODCM Radiological Emuenit Occurrence
>1
>3 Safety (occurrences during previous four quarters)
Physical Protection Protected Area Security Equipment Performance Index
>0.05
>0.15 (over a four quarter period)
Personnel Screening Program Performance (reportable
>2
>5 events during the previous four quarters)
Fitness-for-Duty (FFD)/Personnel Reliability Program
>2
>5 Performance (reportable events during thie previous tour
Plant Oversight Process Management Meeting Licensee Action
- - - - 4NRC Inspection, Regulatory Action Assessment Report Public Assessment Meeting Eaon Evaluation of Findings ski All
ACTION MATRIX LICENSEE PERFORMANCE INCREASING SAF ETY_ SIGNIFICANCE All Assessment One or Two White inputs One Degraded Repetitive Degraded Overall Unacceptable Inputs (Performance (in different Cornerstone (2 White Cornerstone, Multiple Performance; Plants Not Indichtors (Pis) and cornerstones) in a Inputs or I Yellow Input)
Degraded Cornerstones, Permitted to Operate U l Insiection Findings)
Strategic Performance or any 3 White Inputs In Multiple Yellow Inputs, Within this Band, W
Green; Cornerstone Area; Cornerstone a Strategic Performance or I Red Input':
Unacceptable Margin to Objectives Fully Met Objectives Fully Met Area; Cornerstone Cornerstone Objectives Safety Objectives Met with Met with Longstanding Minimal Reduction in Issues or Significant Safety Margin Reduction In Safety Margin Regulatory Routine Senior Branch Chief (BC) or DD or Regional EDO (or Commission)
Commission meeting with Conference Resident Inspector Division Director (DO)
Administrator (RA) Meet Meet with Senior Senior Licensee (SRI) Interaction Meet with Licensee with Licensee Licensee Management' Management Licensee Action Licensee Corrective Licensee Corrective Licensee Self Licensee Performance Action Action with NRC Assessment with NRC Improvement Plan with w
_Oversight Oversight NRC Oversight Z
NRC Inspection Risk-Informed Baseline and Inspection Baseline and Inspection Baseline and Team O
Baseline inspection Follow-up Focused on Cause of Inspection Focused on IL
_U)
Program (Baseline)
Degradation Cause of Degradation t
Regulatory None Document Response to Docket Response to
-10 CFR 2.204 DFI Order to Modify, Suspend, Actions Degrading Area in Degrading Condition
-10 CFR 50.54() Letter or Revoke Licensed j_
ipection Rpoor
-CAL/Order Activities Assessment DD reviewislgn DD reviewisign RA revlewisign RA reviewisign 00 Reotassessment report assessment report assessment report assessment report (wl Inspection plan)
(wl Inspection plan)
(wl inspection plan)
(wl inspection plan)
Commission Informed 2
Public SRI or BC Meet with BC or DD Meet with RA Discuss EDO (or Commission)
Commission Meeting with L
Assessment Licensee Licensee Performance with Discuss Performance Senior Licensee Meeting Licensee with.Senior Licensee anagement.
Management II~-..---Realonui Review I Agency Review ------
Operator/Technician Fatigue "Policy on Factors Causing'Fatigue of Operating Personnel at Nuclear Power Plants" (GL 82-12)
Policy implemented through technical specifications Recent allegations and Congressional interest Use of overtime at some plants not consistent with policy guidelines NRC to reassess policy statement
Reactivity Manipulations Current Rule Requires 5 significant control manipulations that affect reactivity or power to be performed by each license applicant Must be performed on the actual plant Proposed Rule Would continue to accept use of the actual plant -or Would allow use of the simulation facility -if Control manipulations are evolutions that are part of the SAT-based, Commission-approved training program,
-and core and thermal-hydraulic models reflect the actual core that exists or will be loaded at the time of the applicant's operator's license examination, - and - simulator fidelity has been assured by testing Related Regulatory Activity,'
Regulatory Guide 1.149 is being revised to endorse ANSI/ANS 3.5-1998.
10 CFR 55.45 is being revised to reduce unnecessary regulatory burden.
ES-301 Final Rev 8 Change Overview Purpose A" AIt applicants for reactor operator (RO) and senior reactor operator (SRO) licenses at power reactor facilities are required to take an operating test, unless it has been waived in accordancewth 10 CFR 55.47 (refer to ES-204). The specific content of the operating test depends on the type of license for which the applicant has applied.
This standard describes the procedure for developing operating tests that meet the requirements of 10 CFR 55.45, including the use of reactor plant simulation facilities and the conduct of mutti-unit evaluations.
Background 'B*
To the extent applicable, the operating test will require the applicant to demonstrate an understanding of. and the ability to perform, the actions necessary to accomplish a representative sampling from the 13 items identified in 10 CFR 55.45(a) (all 13 iems do not need to be sampled on every operating test).
In addition, the content of the operating test will be identified, in part, from learning objectives contained in the facility licensee's training program and from information in the final safety analysis repot. system description manuals and operating procedures, the facility license and license amendments, licensee event reports. and other materials requested from the facility licensee by the Commission.
Category "B"
Control Room Systems and Facility Walk-through (B.2)
Category B is divided into two subcategories. The fast and larger subcategory (B.1, 'Control Room Systems) focuses on those systems with which licensed operators are most involved (i.e.. those having controls and indications in the main control room). The second subcategory (B.2, 'Facility Walk-Through) ensures that the applicant is familiar with the design and operation of systems located outside the main control room.
The applicant's knowledge and abilities relative to each system are evaluated by administering JPMs and, when necessary, specific follow.
up questions based on the applicant's performance of each JPM.
Category "C"
Integrated Plant Operations (B.3)
Each applicant must demonstrate proficiency on every competency applicable to his or her license level. The onty exception is that SRO Competency Number 5. 'Control Board Operatnfs, is optional for SRO upgrade applicants.
(i.e., SRO.upgrade applicants do not have to fill a position that requires control board operations: however, if they do rotate into such a position, they will be graded on this competency even though they may not be individually observed by an MRC examiner, as discussed in ES-302).
INSTRUCTIONS General Guidelines (D.1.a)
To minimize predictability and maintain test integrity. varied subjects.
systems, and operations shall be evaluated with applicants that are not being examined at the same time, unless measures are taken to preclude interaction among the applicants.
The same JPfs and simulator scenaios shel not be repeated on successive days.
General Guidelines (D.1.a) cont Operating tests written by the facility licensee may not duplicate test items (simulator scenarios or JPMs) from the applicants'sudit fast (or tests If the applicant is retaking the examination) given at or near the end of the license training class. Simulator events and JPMs that are similar to those that were tested on the audit examination are permined provided the actions required to mitigate the transient or complete the task (e.g., using an alternate path as discussed in Appendix C) are significantly different from those required duping the audit examination. The facility licensee shall Identify for the NRC chief examiner those simulator events and JPMs that are similar to those that were tested on the audit examination.
General Guidelines (D.1.d)
When selecting and developing materials (JPMs. scenarios, and questions) for the operating test. ensure that the materials contribute to the test's overall capacity to differentiate between those applicants who are competent to safely operate the plant and those who are not.
Additionally, all of the test items should include the three facets of tast validity (i.e., content. operational, and discrimination) discussed in Appendix A.
Any test items that, when missed, would raise questions regarding adequate justification for denying the applicants license should not be included on the operating test.
General Guidelines (D.1.i)
Every facet of the operating test. Including the prescripted Category A questions and answers, the JPMs for Categories A and B. and the Category C simulator scenarios, should be planned, researched, validated, and documented to the maximum extent possible before the test is administered.
Thetis
-BEFORE-
General Guidelines (D.1.j)
Examiners who will be administering the operating tests but were not involved in their development are expected to research and study the topics and systems to be examined on the operating test so that they are prepared to ask whatever performance-based follow-up questions might be necessary to determine if the applicant is competent in those areas.
As stated in 10 CFR S5.45(a), the operating test requires the applicant to demonstrate an understanding of and the ability to perform the actions necessary to accomplish a representative sample from among 13 Ifems listed in the rule.
General Guidelines (D.1.j) cont f the applicant correctly performs a JPM (including both critical and noncritical steps) and demonstrates familiarity with the equipment and procedures, it is not necessary to ask any follow-up questions.
However, If the applicant fails to accomplish the task standard for the JPMt or demonstrates a lack of understanding regarding the equipment and procedures such as having difficulty #ocating information, control board indications, or controls, the examiner must be prepared to ask performance-based follow-up questions, as necessary, to clarify or confirm the applicants understanding of the system as it relates to the task that was performed.
General Guidelines (D.1.1)
The prescripted questions for Category L and the performance-based follow-up questions for Category B may include a combination of open.
and closed-reference hems. Open-reference items that require applicants to apply their knowledge of the plant to postulated normal. abnormal, and emergency situations are preferred.
Closed-reference items may be used to evaluate the immediate actions of emergency and other procedures. certain automatic actions, operating characteristics, interlocks, set points. and routine administrative activities, as appropriate to the facility.
Refer to Attachment I for more guidance regarding the development and use of prescripted open reference questions for Category A of the walk through test. To the extent possible, the concepts in the attachment should also be applied to performance-based follow-up questions.
Catagory "A"
Administrative Topics (D.2.b)
For each administrative subject. determine the best method for evaluating the applicant's knowledge or ability in that area. Although a performance based evaluation. using a single administrative JPM is generally preferred.
two prescripted questions may be used to conduct the evaluation in each specific subject area selected for evaluation.
The questions may be associated with Category B JPMs or they may be administered separately.
Administrative Topics (D.2.g)
Forward the completed outline to the NRC chief examiner so that it is received by the date agreed upon with the NRC regional office at the time the examination arrangements were confirmed: the outline is normally due approximately 75 days before the scheduled examination date. Refer to ES-201 for additional instructions regarding the review and submittal of the examination outline.
Administrative Topics (D.2.h)
After the NRC chief examiner approves the operating test outline, prepare the final Category A test materials in accordance with the general operating test guidelines in Section D.1. the open-reference question guidelines in Appendix B. and the JPM guidelines in Appendix C.
(i.e., the JPOs, preiscripred questions, and enswers)
Control Room Systems and Facility Walk-Through Specific Instructions For Category 'B" (D.3)
This category of the operating fast evaluates the applicant on systems-related K/AS by having the applicant perform selected tasks and, when necessary, based on the applicant's performance, probing his or her knowledge of the task and Its associated system with specific follow-up questions.
The Category 9 tasks are in addition to and should be diferent fom the events and evolutions conducted during Category C. -integrated Plant Operations.
Specific Instructions For Category '8" (D.3.a) cont The 10 systems and evolutions selected for RO and SRO-I applicants should evaluate at least 7 difterent safety functions. All of the systems and evolutions in each subcategory of the test should be selected from different safety function lists. and the same system or evolution should not be used to evaluate more man one safety function in each subcategory. For PWR operating tests, the primary and secondary systems listed under Safety Function 4, 'Heat Removal From Reactor Core," in Section 1.9 of NUREG.1122 may be treated as separate safety functions; i.e., two systems, one primary and one secondary, may be selected from Safety Function 4.
Specific Instructions For Category 8 (D.3.a) cont The 10 systems and evolutions selected for RO and SROIl applicants should evaluate at least 7 different safety functions. All of the systems and evolutions in each subcategory of the test should be selected from different safety function lists, and the same system or evolution should not be used to evaluate more than one safety function in each subcategory.
For PWR operating tests, the primary and secondary systems listed under Safety Function 4, "Heat Removal From Reactor Core, 'in Section 1.9 of NUREG.1122 may be trested as separate safety functions; i.e.. two systems, one primary and one secondary, may be selected from Safety Function 4.
Specific Instructions For Category 'B* (D.3.b)
For each system selected for evaluation. select from the applicable K/A catalog or the facility licensee's site-specific task list one task for which a JPM exists or can be developed. Review the associated simulator outline if it has already been prepared (refer to Section DA). and avoid those tasks that have already been selected for evaluation on the dynamic simulator test.
The JPMs should, Individually and as a group, have meaningful performance requirements that will provide a legitimate basis for evaluating the applicants understanding of and ability to safely operate the associated systems and the plant (as required by 10 CFR 5s.45).
Specific Instructions For Category 'B* (D.3.b) cont At least one of the tasks shall be related to a shutdown or iow-power condition, and 40 percent of the tasks (i.e., 4/10 for ROs and instant SROs and 2/5 for upgrade SROs) shall require the applicant to execure afternee paths within the faclliWys operating procedures.
In addition. at least one of the tasks conducted in the plant (i.e..
Subcategory 8.2) shall evaluate the applicants ability to implement actions required during an emergency or abnormal condition, and another shall require the applicant to enter the RCA.
This provides an excellent opportunity for the applicant to discuss or demonstrate the radiation control subjects descibed in Administrative ]
Topic A.3.
Specific Instructions For Category '8" (D.3.c)
Forward the completed walk-through test outline to the NRC chief examiner so that it is received by the date agreed upon with the NRC regional office at the time the examination arrangements were confirmed:
the outlines are normally due approximately 75 days before the scheduled examination date. Refer to ES-201 for additional instructions regarding the review and submittal of examination outlines.
Specific Instructions For Category 'B" (D.3.e)
Submit the entire operating test package to the designated facility reviewer or the NRC chief examiner. as appropriate, for review and approval in accordance with Section E. The test must be received by the NRC chief examiner approximately 45 days before the scheduled review date, unless other arrangements have been made.
Integrated Plant Operations Specific Instructions For Category 'C' (D.4.d)
Each scenario set must, at a minimum, require each applicant to respond to the types of evolutions, failures. and transients in the quantities identified for the applicant s license level on Form ES.30t*5, "Transient and Event Checklist. An applicant should only be given credit for those events that require the applicant to perform verifiable actions that provide insight to the applicant's competence.
The required instrument and component failures should normally be completed before starting the major transient: those that are initiated after the major transient should be carefully reviewed because they may require hule applicant action and provide little insight regarding their performance.
Specific Instructions For Category 'C' (D.4.d) cont Each event should only be counted once per applicant.
For example: a power change can be counted as a normal evolution OR as a reactivity manipulation.
Similarly. a component failure that immediately results in a major transient counts as one or the other, but not both.
.0 9
Specific Instructions For Category "C' (D.4.d) cont Any normal evolution, component failure, or abnormal event (other than a reactor trip or other automatic power reduction) that requires the operator to perform a controlled power or reactivity change will satisfy the requirement for a reactivity manipulation.
This Includes events such as an emergency boration, a dropped rod recovery, a significant rod bank realignment, or a manual reactor power reduction in response to a secondary system upset Such events may produce a more timely operator and plant response than a normal power change.
Specific Instructions For Category 'C' (D.4.d) cont tf the facility licensee normally operates with and is required by its technical specifications to have more than two ROs in the control room, the chief examiner may authorize the use of additional surrogates to fill out the crews.
In such cases, take care in planning the scenarios to ensure that the additional operators do not reduce the eraminers' ability to evaluate each applicant on the required number of events and on every competency and rating factor.
Specific Instructions For Category 'C" (D.4.e)
When the proposed simulator operating test outlines are complete. forward them to the NRC chief examiner so they are received by the date agreed upon with the NRC regional office at the time the examination arrangements were confirmed; the outlines are normally due approximately 75 days before the scheduled examination date. Refer to ES-201 for additional instructions regarding the review and submittal of the examination outlines.
Specific Instructions For Category 'C" (D.4.g)
Submit the entire operating test package to the designated facility reviewer or the NRC chief examiner, as appropriate. for review and approval in accordance with Section E.
The test must be received by the NRC chief examiner approximately 45 days before the scheduled administration date. unless other arrangements have been made.
QUALITY REVIEWS Facility Management Review (E.1)
If the operating test was prepared by the facility licensee, the preliminary outline and the proposed test shall be independently reviewed by a supervisor or manager before they are submitted to the NRC regional office for review and approval in accordance with ES-201.
The reviewer should evaluate the outline and test using the criteria on Forms ES-201-2. ES-301-3. and ES-301-4 and include the signed forms (for each different operating test) in the examination package submitted to the NRC in accordance with ES-201.
ATACHMENTS (F)
"Open-Reference Question Guidelines"
Open-Reference Question Guidelines
- 1. The most appropriate format is the short-answer question, which requires the applicant to compose a response rather than select from among a set of alternative responses, as is the case with multiple-choice, matching, and trueffalse questions.
- 2. Provide clear, explicit directions/guidelines for answering the question so that the applicant understands what constitutes a fully Correct response. Choose words carefully to ensure that the stipulations and requirements of the question are appropriately conveyed. Words such as -evaluate,' "outline, and 'erplain," can invite a lot of detail that is not necessarily relevant Open-Reference Question Guidelines (cont)
- 3. Make sure that the expected response matches (and is limited to) the requirements posed in the question. Consider the amount of partial credit to be granted for an incomplete answer. For questions requiring computation, specify the degree of precision expected. Try to make the answer turn out to be whole numbers.
- 4. Avoid giving away part or all of the answer by the way the question is worded. For example: 'if the trdown line became obstructed, could boration of the plant be accomplished shortly after a reactor trip to put the plant in cold shutdown? Itfso, how?- A test-wise applicant can realize that the answer has to be yes, or else the second part of the question would have read something like 'If not, why not?
Open-Reference Question Guidelines (cont)
S. Avoid what could be considered 'trick' questions in which the expected answer does not precisely match the question. For example, asking 'How [do] the SI termination criteria change following an St reinitiation? implies that the termination criteria will change, when in actuality they do not
- 6. Do not use direct look-up questions that only require the applicant to recall where to find the answer to the question. The operational orientation required of questions on the walk-through test and the applicant's access to reference documents, argue against the use of questions that test for recall and memorization. Any questions that do not require any analysis, synthesis, or application of information by the applicant should'be answerable without the aid of reference materials.
Refer to ES-602, Attachment 1. for a more detailed discussion of direct Idook-up questions.
Open-Reference Question Guidelines (cont)
- 7. Questions should also adhere to the generic tem construction principles and guidelines in Appendix B. Moreover, Form ES-602-1, "NRC Checklist for Open-Reference Test tems," contains a list of questions that can be used to evaluate the suitability of the questions for the walk-through portion of the operating test Although the checklist was developed for use in evaluating twqualification written examinations, all ofthe criteria except 9, 10, 11, and the K/A rating on item 7 ore generically applicable.
602-1 Excerpt Items 9, 10. and 11
- 9.
Is the question appropriate for the written examination and the selected format (e.g.. short answer or multiple choice)?
- 10. Do questions in Section A take advantage of the simulator control room setting?
- 11. Does any question have the potential of being a "double-jeoparday question?
ESFORMS ES-301-1, 2, 3, 4, 5, and 6
Examples of LOW discrimination JPMs
- 1.
Reset the Turbine Driven AFW Pump Trip Throttle Valve (PWR)
- 2.
Actuate ADS (single critical step) (BWR)
- 3.
Start the Hydrogen Recombiner (without a fault)
- 4.
Locally borate the RCS with the emergency manual boration valve (PWR)
Examples of DISCRIMINATORY JPMs
++++@o
++++
- 1. Dropped rod recovery (drop a second rod during recovery requiring reactor trip)
- 2.
Local start of equipment with failures requiring the use of altemate procedures.
S.
ES-302 Final Rev 8 Change Overview Purpose 'A' This standard describes how to administer operating tests to initial license applicants in accordance with the requirements of 10 CFR 55.45. It includes policies and guidelines for administering both the walk-through and the integrated plant operations categories of the operating test. It is assumed that the operating test was prepared in accordance with ES-301.
Background "B_
As noted in ES-201. facility licensees will generally prepare proposed operating tests in accordance with ES-301 and submit them to the responsible NRC regional office for review and approval.
Regardless of whether it was prepared by the facility licensee or the NRC.
every operating test will be independently administered and graded by an NRC licensing examiner in accordance with the instructions contained herein and in ES-303.
Responsibilities "C" Facility Licensee (C.1.b)
Safeguard the integrity and security of the operating tests in accordance with facility procedures established pursuant to 10 CFR 55.40b)(2) and the guidelines discussed in Attachment 1 of ES-201.
Responsibilities "C" NRC Regional Office (C.2.a)
Work with the facility contact to coordinate the operating test administration schedule in a manner that maximizes efficiency and maintains security.
Normally, the operating tests should be administered within 30 days before orefter the writen examinations. The region shall obtain concurrence from the NRR operator licensing program office if the examination dates diverge by more than 30 days. (Refer to ES-201 for additional guidance regarding examinations that have to be rescheduled to achieve an acceptable product.)
Test Administration Instructions and Policies (D)
General (D.1.d)
Normally, an NRC examiner wilt be assigned to individually evaluate each applicant during the simulator operating test. However, if a three-person operating crew consists entirely of senior reactor operator (SRO) upgrade applicants (who do not have to be evaluated on the control boards), the chief examiner may assign only two examiners to observe the crew.
Although the applicants in the reactor operator and balance of plant positions may not be individually evaluated, they will be held accountable for any errors that occur asa result of their action(s) or inaction(s) and graded on their ability to 'Operate the Control Boards" (i.e., SRO Competency5). SRO-instantapplicants will always be individually evaluated by an NRC examiner regardless whet operating position they are filling during a given scenario.
General (D.1j)
Although the simulation facility operator will normally assume the role of the other personnel that the applicants direct or notify regarding plant operations, the chief examiner may permit other members of the facility training or operations staff (e.g., a shift technical advisor (STA)) to augment the operating shift team if necessary.
The chief examiner shall fully brief those individuals regarding their responsibilities. reporting requirements, duties, and level of participation before the operating test begins. The examiners must not restrict the surrogate operators activities to such an extent that the applicants being evaluated are required to assume responsibilities beyond the scope of their position.
The surrogate operators will be expected to assume the full responsibilities of the roles they take in the operating test. Consultations with an STA shall be conducted in accordance with the facility licensee's normal control room practice; e.g., an STA shall not be stationed in the simulator if they are on-call at the site.
Walk-Through (Categories A and B)
(D.2.b)
To the extent possible, the examiner should have the applicant perform the control room JPMs on the simulator, rather than asking the applicant to describe how he or she would accomplish the task.
I the examiner observes a discrepancy between the simulator setup and the conditions specified in a JPM. then the examiner shall stop the JPA and correct the situation, as necessary.
Ifthe task can be completed with different values (e.g., wind direction when determining a protective action recommendation during an emergency), then the examiner shall document the differences and coordinate with the facility contact and the NRC chief examiner to validate the applicant's response under the actual conditions.
Walk-Through (Categories A and 8) (D.2.f)
As stated in 10 CFR 55.45(a). the operating test requires the applicant to demonstrate an understanding of and the ability to perform the actions necessary to accomplish a representative sample from among 13 items listed in the rule.
I the applicant correctly performs a JPM (including both critical and noncritical steps) and demonstrates familiarity with the equipment and procedures, the examiner should infer that the applicant's understanding of the system/task is adequate and refrain from asking follow-up questions.
However, ifthe applicant fails to accomplish the task standard for the JPM, exhibits behavior that demonstrates a lack of familiarity with the equipment and procedures, or is unable to locate information, control board indications, or controls, the examiner should ask performance based follow-up questions as necessary to clarify or confirm the applicants understanding of the system as it relates to the task that was performed.
Walk-Through (Categories A and B) (D.2.f) cont Similarty, If the applicant gives an ambiguous answer to a prescripted administrative question in Category A, the examiner is expected to ask probing questions to ensure that the applicant understood the original question and the applicable knowledge or ability.
The examiner shall document all performanceibased questions and enswers for later evaluation.
If an applicant volunteers additional or corrected Information after having completed a task or question, the examiner shall offer the applicant the opportunity to take whatever actions would be required In a similar situation in the plant The examiner will record any revisions to previously performed tasks or answers for consideration when grading the operating test per ES.
303.
Walk-Through (Categories A and B) (D.2.g)
The examiner should practice other good walk-through evaluation techniques as discussed in Section D of Appendix C.
Walk-Through (Categories A and B) (D.2.o)
If the simulation facility should become inoperable and cause excessive delay of the operating tests, the chief examiner should discuss the situation with the facility icensee and the responsible regional supervisor so that management can make a decision regarding the conduct of the operating tests. It may be necessary to reschedule the simulator examinations for a later date.
The simulator should be considered inoperable enderany of the following conditions:
. The simulator exhibits a masslenergy imbalance, erratic logic, or inexplicable panel indications during model execution.
The simulator exhibits unplanned and unexplained events or malfunctions that cause the applicants to divert from the expected responses and success path of the planned scenario.
Walk-Through (Categories A and B) (D.2.o) cont The simulator automatically goes to the 'freeze" state during a scenario or a "beyond simulated limits alarm is received on the instructor station.
- The simulator instructor informs the examination team that a software module has halted or 'kicked out."
Occurrence of any of these abnormal simulator operating conditions during an examination constitutes sufficient cause to stop the scenario. Evaluations of the applicants' performance during any of these simulator malfunction conditions may be unreliable.
When the simulator has been restored to full operability, the chief examiner will determine if the scenario requires replacement may be resumed in progress, or may be restarted from the beginning.
ES-303 Final Rev 8 Change Overview II Purpose A This standard describes the procedures for documenting all categories of the operating test collating the data to arrive at a pass or fail recommendation. and reviewing the documentation to ensure quality.
Background
This standard assumes that the operating test was prepared and administered in accordance with ES-301 and ES-302. respectively.
The procedures contained herein require the examiner to evaluate each applicants performance on the operating test and make a judgement as to whether the applicants level of knowledge and understanding meet the minimum requirements to safely operate the facility for which the license is sought The examiner evaluates each noted deficiency in light of the total breadth of knowledge and ability demonstrated by the applicant in that subject area.
S Responsibilities "C" NRC Examiner of Record (C.2)
As soon as possible after administering the test, the examiner of record shall review, evaluate, and finalize each applicant's operating test documentation in accordance with the instructions in Section D.
If an applicant made an error with serious safety consequences, the examiner may recommend an operating test failure oven if the grading instructions in Section D would normally result in a passing grade. Under such circumstances, the examiner shall thoroughly justify and document the basis for the failure in accordance with Section DJ.b.
Moreover, the NRC regional office shall obtain wrtfren concurrence from the NRR operator licensing program office before completing the licensing action.
Grading and Documentation Instructions "D" Evaluate the Applicants Performance Form ES-303-1, Category B (D.2.b)
To determine a grade for the systems/JPus listed on Form ES-303-1, evaluate each deficiency coded in the rough notes for Category B. If the following criteria are met. assign a satisfactory grade by placing an "S* in the -Evaluation" column for that systemflJPM; otherwise enter a IT:
Tine-critical JPMs must be completed within the allotted time.
Evaluate the Applicants Performance Form ES-303-1, Category B (D.2.b) cont The task standard for the JPM must be accomplished by correctly completing all of the critical steps.
If the applicant initially missed a critical step, but later performed it correctly and accomplished the task standard without degrading the condition of the system or the plant, the applicant's performance on that JPM should be graded as satisfactory.
However, the applicant's error shall be documented in accordance with Section D.3.
Evaluate the Applicant's Performance Form ES-303-1, Category B (D.2.b) cont
- The responses to any performance-based follow-up questions asked pursuant to Section D.2.f of ES-302 must confirm that the applicant's understanding of the systemVJPM is satisfactory.
If the follow-up questions reveal that the applicant's understanding of the sysreniJPM is seriously deficient, the examiner may recommend an unsatisfactory grade for the system even though the applicant successfully completed the task standard for the JPM. The basis for the recommendation shall be thoroughly justiied and documented in accordance with Section D..
Evaluate the Applicant's Performance Form ES-303-1, Category B (D.2.b) cont Conversely, if the applicant did not accomplish the task standard and folow-up questioning revealed that the failure was caused by a deficiency in the procedure or some other factor beyond the applicant's control, the examiner may still recommend a satisfactory grade for the system/JPM.
Once again, the basis for the recommendation shall be thoroughly justified and documented in accordance with Section D.3.
Evaluate the Applicant's Performance Form ES-303-1, Category B (D.2.b) cont After grading the applicants performance with re.specl to all the Category B systems, determine an overall grade for Category B by calculating the percentage of satisfactory system grades.
ff the applicant has an -S-on at least 80 percent of the systems examined, the applicant passes Category B and receives an -S' overall. If the applicant has an 'S-on fewer than 80 percent of the systems, the applicant fails Category B and receives a 'U" overall.
Document the applicants grade by placing an "S-or a 'U-in block B,
'Control Room Systems and Facility Walk-Through,' in the "Operating Test Summary on page l of Form ES-303-1. Enter *N/E" If this category was waived in accordance with ES-204. Document and justify every deficiency in accordance with Section D.3.
Evaluate the Applicant's Performance Form ES-303-1, Category C (D.2.c)
Using Form ES-303-3 or ES-303-4, depending on the applicant's license level. evaluate any deficiencies coded for Category C. Circle the integral rating value (1 through 3) corresponding to the behavioral anchor that most accurately reflects the applicant's performance. A rating of "1' would be justified if the applicant missed a critical task (i.e.. by omission or incorrect performance) or committed multiple errors of lesser significance that have a bearing on the rating factor.
Missuig one or more critical tasks does not necessarily mean that the aplicant will fail the simulator test, nor does success on every critical task prevent the examiner from recommendint a failure if the applicant had other deficiencies that. in the aqqregate. justify the failure based on the co; tnyevaluations. As discussed in &S-301, Competency 5 is pio rRO upgrade applicants. However, the examiner shall evaluate Competency 5 If the applicant rotated into an operating crew position that required the applicant to manipulate the controls.
Document and justify every deficiency in accordance with Section D.3.
Finalize the Documentation (D.3.b)
Deficiencies that do not contribute to an operating test Category failure shall also be documented.
However, a brief statement describing the error and the expected action or response is generally sufficient.
Examiners should keep in mind that their licensing recommendation and the associated documentation are subject to review by the chief examiner and NRC regional office management.
Therefore, the documentation should contain sufficient detail so that the independent reviewer, responsible supervisor, and licensing official can make a logical decision in support of the examiner's recommendation to deny or issue the license.
Finalize the Documentation (D.3.b) cont Retain rough documentation until the chief examiner and NRC regional office management have reviewed the examiners recommendations and concurred in the results (refer to ES-50 1).
Examiners shall retain all applicable notes and documentation associated with proposed denials until the denials become final.
Examiners are advised that such notes would be subject to disclosure If requested under the Freedom of Information Act.
Form ES-303-1 Individual Examination Report ES 303-1__
Appendix C
Final Rev 8 Change Overview Purpose "A"
This Appendix provides a framework for preparing and evaluating job performance measures (JPMs) to ensure they are of appropriate substance and format for initial operator licensing and requalification examinations. The following elements are discussed in detail or attached for information:
- a basic procedure for developing new JPMs (Section B). including forms to document the JPM and to assess the quality of the product (Form ES C-1 and ES-C-2)
- guidelines for the development and use of attemate-path JPMs (Section C)
- a discussion of walk-through evaluation techniques (Section 0)
-THE PRESCRIPTED QUESTION BULLET WAS DELETED Development and Reviewing JPMs "B' ALL PRESCRIPTED QUESTION DEVELOPMENT PARAGRAPHS WERE DELETED
Attachments/Forms ATTACHMENT 1, PRESCRIPTED QUESTION SAMPLES WERE DELETED
TRAINING MANAGERS CONFERENCE NEW INSPECTION PROGRAM NEW INSPECTION PROGRAM
- The New Program Will Establish Baseline Inspections Common to All Plants Inspection Beyond the Baseline Will Be Performed at Plants Below a Specified Threshold
- Predicated on Performance Indicators
- Inspection Findings
- Response to Specific Events or Problems
- Baseline Inspections will Be Grounded on "Cornerstone" Areas Focused on 'Risk Significant" Activities and Systems Focused on How Utilities Find and Fix Problems Focused on How Utilities Accept and Encourage Employees to Raise Safety Issues NEW INSPECTION PROGRAM
- Major Parts of the Baseline Inspection Program
- Inspect Areas Not Covered By Performance Indicators
- Inspect to Verify the Performance Indicators Inspect/Review Effectiveness of Finding and Resolving Problems (Corrective Action Program)
- CORNERSTONES
- Monitor Performance in Three Areas:
- Reactor Safety
- Radiation Safety
- Security
NEW INSPECTION PROGRAM Cornerstones Reactor Safety Initiating Events
-Mitigation Systems
-Barrier Integrity
-Emergency Preparedness
- Radiation Safety
' Plant Worker
' General Public
. Security Physical Protection New Inspection Program Cornerstone Cross-Cutting Elements
. Cross-Cutting Elements
- Element that Effect Each Cornerstone
'Human Performance
-Ability to Raise Safety Issues
-Finding and Fixing Problems NEW INSPECTION PROGRAM Major Elements of the Baseline Program
.The Program is Indicative and Not Diagnostic
'Program Delineates Specific Inspection Activities
'Inspection Findings are Evaluated for Significance
-DiagnosticlRoot Cause Determinations Done By Supplemental Inspection Program
NEW INSPECTION PROGRAM Major Element of Baseline Program Baseline Program is Risk Informed Inspectable Areas Selected based on Significance from a Risk Perspective
- Risk Factor in to Program Four Ways
- Inspectable Areas are Based on Importance to Measuring Comerstone Objective
- Inspection Frequency and Number of Activities and Time Spend Inspecting Based on Risk
- Selection of Inspection Activities in Each Inspectable Area Based on a Risk Matrix Modified by Plant Specific Information
- Inspectors Are Trained in the Use of Risk Information
- Baseline Program is the Minimum Program NEW INSPECTION PROGRAM Cornerstone link to Inspectable Areas NEW INSPECTION PROGRAM PERFORMANCE INDICATORS
- Initiating Events:
-Unplanned Reactor Shutdown
' Loss of Normal Rx Cooling Sys Following Unplanned Shutdown
' Transients - Unplanned Events that result in Rx Power Change
= Mitigating Systems:
'Safety Systems Not Available
- Specific ECCS
- Emergency Power Systems Safety System Failures
NEW INSPECTION PROGRAM Performance Indicators Integrity of Barriers:
Fuel Cladding (coolant activity)
Rx Cooling System Leak Rate Rx Containment Leak Rate Emergency Preparedness:
Emergency Response Organization Drill Performance Readiness of Emergency Response Organization Availability of Notification System NEW INSPECTION PROGRAM Performance Indicators Occupational Radiation Safety:
Compliance with Regulations for Controlling Access to Radiation Areas in Plant Uncontrolled Radiation Exposures to Workers Greater than 10% of Regulatory Limit Public Radiation Safety:
Effluent Releases Requiring Reporting Under NRC Regulations and License Conditions NEW INSPECTION PROGRAM Performance Indicators Physical Protection:
Security System Equipment Availability
. Personnel Screening Program Performance Employee Fitness-for-Duty Program Effectiveness
0 NEW INSPECTION PROGRAM Inspection Findings / Performance Indicator Data Inspection Findings:
- Inspector Observations are Evaluated to Determine Significance Use Defined Process: Significance Determination Process Performance Indicators:
- Thresholds Set Regulatory Response
- Example:
Drill/Exercise Performance - The percentage of all Drill. exercise.
and actual opportunities that were performed in a timely and accurately during the previous eight quarters NEW INSPECTION PROGRAM Regulatory Response
- Characterize the Risk Significance of issue
--Each Issue is evaluated and assigned a color Using a Process Flow Chart
- GREEN: Licensee Response
- White: Increased Regulatory Response
- Yellow Required Regulatory Response
- Red: Plant Not Permitted to Operated within this Band NEW INSPECTION PROCESS PILOT PROGRAM
" Two Pilot Plants Per Region
" Sequoyah
- Harris a Pilot Program Commenced June 1999
- Full Implementation of New Inspection Program by April 2000
CL-pR REGULATOt COMMISSION NNG MANAGERS rCONFERENCE A/C/f B AL Training Managers Conference Changes & Clarifications to ES-401, Part 1 PREPARING INITIAL SITE-SPECIFIC WRITTEN EXAMS
- D.1.b The outline must be systematically selected. Shall not use the site specfic K/A catalog The Plant Wide Generic (PWG) Tier 3 should NOT include system generic tasks. The topics for PWG Tier 1 & 2 and the four KIA categories for Tier 3 Shall be selected from section 2.
Generic Knowledge and Abilities from the K/A catalog.
- 0.1.c 10 site-specic K/As may replace 10 systematic sample items.
for details or issues, with basis and Chief Examiner approval.
- D.1.d Ensure outline samples at least every K/A area twice and the SRO samples topics required by 10 CFR 55.43(b).
SS401 PWR SRO Emrewion irrK FormE S.401-3 anatry Date of Exq arr Exam Level:
9 9
KIA Caftgory Pornts Tier Group pointt K
K K IKj K
K A
AAA G'
Tol 12 3
4-5 6
1 2
3 A
Emergency &
21 Abnortw 6
Evolutions 4
- 2.
1 19 Tar 40 TotatsI
- 3. Gereric Knondesie and Abakes Cat 1 Cat 2 Cat 3 Cat 4 9
17i___________________________
Training Managers Conference Changes & Clarifications to ES-401, Part II
- D.2.a Use existing, modified or new questions. If deviation from submitted sample is necessary discuss with the Chief. Be able to discuss why the change was necessary. Document those reasons.
- D.2.c The written examination MUST be 50-60% higher cognitive order items. (NO more NO lss)
not just questions at the RO level. These should be distributed amongst the 3 tiers.
Training Managers Conference Changes & Clarifications to ES-401. Part Ill
- 0.2.f Retake examinations may NOT have any overlap or reuse items from the failed license exanination.
No overlap between NRC examination and audit unless independently developed. Then only 5 questions allowed.
Repeat of ONLY 25 questions from last 2 NRC examinations and items used in training.
- D.2.g Facility learning objective references are encouraged but not
> required.
If learning objectives are not available, this does not invalidate the question provided it has appropriate K/A and technical references.
Training Managers Conference Changes & Clarifications to ES-401, Part IV
- 0.2.g The draft examination must be received at least 45 days before the examination.
- E.2.a NRC will review and get supervisory review before discussing with licensee.
- E.2.c The NRC WILL perform a 30 question sample review, will review all new plus modified. questions. if required. The sample will include 10 new and 20 modified questions. (All 125 questions will undergo a review.)
Questions previously reviewed and approved by the NRC for that facility will have limited review for unacceptable flaws per ES-401-9.
Training Managers Conference Changes & Clarifications to ES-401, Part V
- E.2.c If the sample shows less than 6 items are unacceptable. detailed review of the rest of the examination will continue. If greater than 6 items are found unacceptable. NRC MAY return the examination or we may complete its review. Review will use ES 401-9.
If the examination is retumed. we expect that the licensee correct the identified flaws and those like kind flaws that were not specfically identified to the rest of the questions.
- E.3.a The NRC Supervisor MUST review and approve all unacceptable item comments.
- E.3.b The NRC supervisor WILL review and approve each comment that would require the licensee to rework a NRC-validated question. (Previously used test items.)
E9-401 wein Exsamnsoir Fom. ES40i-9 Re.w Wokslheeti I l l
I fr.
0 LODS Training Managers Conference Changes Clarifications to ES-401, Part VI
- E4 Final validation of examination after incorporating changes is recommended but NO required.
- Att. I Describes an accptable sampling methodology for systematic selection for the wntten outline.
- Form Modified version of this form provides blanks to record test item 401-7 count for reuse from the last examinations, the source of the questions and the cognitive level for the questions for the examination.
- Form Written examination Review Worksheet. Used to keep track of 401.9 sampled questions.
ES-401-Witten Examnaorti Farm ES-401-7 Otaity CPAstMST ace D0.e of E-am Exam Ler Rr'sRO t -
O.essese wed enser manclyecr a~n soneces to led ~
1
- a. 9 C U-r 1 e se I -
Face y eemer e w
,eee s as WsW4a f e 3* R~f5RO
- ep e as nw than 75 waeei was SRO waeens we aprore p anf.
0.2S o (5.401 No mr Na 21 n
s ae. ielwOfeasefn (pres e**IC O-Vr
- *. n tr.**
oI.e.t -. ntc.ed
..ans, nwthe atl n ae of S e a
at e S
(.o (Les eaw ant eesses eaee na se nW neUt
.00n W...
a"WN B ase aer ts (nomeen 5 0 San doede Me prws from was ni M tan ae r
Crn Cne use seClamessesbenartoeEsca02,ePart*I ADMNISTRN G INITIAL* WRTENEAMNTIN estna ateit e.
Beae l0 ern e0 pen to tenetain e*ery CIA eC.Ie aT seo licensee may se ne w re quredme.nsndnse e
- en.r the cens iuss eSow a
o rad
.2..Du.n..a..en..
se admi..t.
et the..w-,
I I
Training Managers Conference Changesf& Clarifications to ES-402, Part 1 ADMINISTERING INITIAL WRITTEN EXAMINATIONS
- C.1.a The licensee has to maintain security of the examinations.
- C.1i.e The licensee may use machine-gradable sheets but not_________________________
r mquted.
- C.2.a The licensees allowed to administer an NRC developed
- examitnation.
- C.2.b Duting a licensee administered writen exam. the NRC MUST be on site or available by phone.
After NRC approval, the written exam may be administered any time within 30 days of the operating test.
Training Managers Conference Changes & Clarifications to ES-402, Part II ADMINISTERING INITIAL-WITTEN EXAMINATIONS
- D.4.d New time limit for the written is 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />. It can be extended by 30 minute increments, with PRIOR NRC approval. The new time limit should not change the development process.
- E.4 Licensee should submit formal comments within 5 working days after the wrttien examination is administered.
Training Managers Conference Changes & Clarifications to ES-403 GRADING INITIAL SITE-SPECIFIC WRITTEN EXAMS
- C.1.b If NRC developed, licensee has responsitary to subrn f
comments concerning changes to the examination.
- C.2.b NRC may allow the licensee to machine grade a NRC developed written examination.
- D.1.a Do NOT delete any question or change an answer without a valid reference. Unreasonable or unstated assumptions do not justify a change.
- D.2.a Copy each applicant's answer sheet, and set aside. Do NOT mark on the original until all comments are finalized.
- D.2.d if you decide to share PRELIMINARY grades do so with caution.
The NRC MAY NOT accept all the licensee's changes.
Training Managers Conference Changes and Clarifications To Appendix B Written Examination Guidelines, Part I
- C.i.a K/A references ar reguired but Learning Objectives are desired.
This is a check and balance on the facilities training program.
- C.1.b Make sure the question matches the intent of the K/A.
- C.1.C Discrimination validity is defined...the key purpose of any test item is to assess important K/As at a level that distinguishes between safe and unsafe applicants.*
Implementation requires subjective judgement in constructing the stem and distractors.
Training Managers Conference Changes and Clarifications To Appendix B Written Examination Guidelines, Part II
- C.2.a Multiple Choice questions which require the MOST CORRECT answer are NOT allowed. Use a procedural reference!
- C.2.f MI the information in the stem should be relevant. (Don't play find the rock.) Don't add secondary pieces of information in the stem that are not relevant. in order to make the question look more difficult.
- C.2.n Use of geneically correct answers is allowed. but the stem needs to be wrten such that the stem makes them clearly incorrect.
ES-501 INITIAL POST-EXAMINATION ACTIVITIES D. Charles Payne August 13,1999 ES-501
SUMMARY
OF CHANGES w Supervisor or manager shall confirm grading quality and sign QA sheet.
- CE independent regrade for 78-82%.
- Potentially hold passes 80-81%.
- Exam report content more clearly defined.
- PDR records defined.
- New letter for delayed results.
- C.1 No changes to facility requirements.
- Note that clean copy of written exam answer sheet is expected to be provided.
- C.2 No changes to regional requirements.
- Note criteria for determining written exam validity following post-exam comments.
5% changes/deletions -+ facility explanation.
- 10% deletions + evaluate adequate sample.
- D.1 Facility management exam reviews.
. Supervisor or manager shall confirm quality of grading is licensee graded written and sign QA check sheet. Used to be "authorized facility representative."
- Signed QA form represents facility senior I
management concurrence with individual and collective exam results.
- D.2 Chief Examiner reviews.
- No post-exam change will be accepted without a valid plant reference. Uncontrolled lesson plans are not acceptable.
- Verify answer key used as template or to machine grade written exam is accurate.
. Independently review every borderline written exam (78-82%).
. D.3 NRC management review.
. Pass letters for applicants who passed exam but licenses are being withheld.
a If pass written exam with 80-81 % and another applicant fails, will hold license until assured pass/fail decision not affected.
. For delayed licenses, shall ensure still medically fit within last 24 months, not developed permanent physical or mental condition, and up to date in requal.
P If > 3 months pass, advise licensee to properly activate license per 10 CFR 55.53(f).
ES-501 a E.1 and E.2 No significant changes.
. Region still will retain EOPs, AOPs, E-Plan and Tech Specs for incident response.
- E.3 Exam report documentation. Some significant changes.
- Previous revisions stated generic exam report content requirements. Rev. 8 spells out specific types of issues to be included.
- Factual description of test item changes including type and number of psychometric enhancements made.
- Conclusions regarding adequacy of facility proposed exams are not required and should only be considered if have a programmatic issue.
- Any delay in administering the exam and the reason, and any extensions of the written exam time beyond five hours.
- Any exam security issues/incidents.
-All simulator deficiencies encountered while preparing or conducting operating tests will be documented in Simulation Facility Report.
- Generic comments submitted by licensee regarding exams or the process are welcome and will be included in exam report. These do not require regional response or resolution.
- Region will ensure SRO upgrade applicants that fail exam still comply with 10 CFR 55 before resuming RO duties.
.PDR records will include the following:
- exam outlines
- draft and final written
- draft and final operating tests
- associated QA check sheets
- "Other documents"
- Intermediate working copies not needed to be sent to PDR unless provided to facility licensee to facilitate communication.
(II
ES-502 PROCESSING REQUESTS FOR ADMINISTRATIVE REVIEWS AND HEARINGS AFTER INITIAL LICENSE DENIAL D.
Charles Payne August 13, 1999 ES-502
SUMMARY
OF CHANGES
- Reorganized to remove detailed sample letters and administrative review procedures.
- New section noting expectation of facility licensee support during appeals.
- New section to better define NRC responsibilities.
- Administrative review process streamlined.
ES-502 mC.1 Applicant responsibilities. No changes.
- Has 20 days to act on proposed denial. Has 3 options:
- 1. Do nothing.
- 2. Request reconsideration.
- 3. Request a hearing.
- If application denied because of training or experience, can reapply when corrected.
- C.2 Facility licensee responsibilities. New section.
- Facility is expected to provide reference materials and technical support as necessary for NRC to evaluate and resolve concerns raised by applicant.
This includes organizationally supporting the response provided in the answer key.
- C.3 NRC responsibilities. New section but no new responsibilities.
- Splits out our responsibilities from mechanics of the review process.
- Application denials will be processed per D. 1.
- Admin reviews will be processed per D.2.
- Hearings will be conducted per 10 CFR 2, subpart L.
- D.1 Application denial admin review.
.Not many have occurred in Region II. But be cautious of potential outcomes should one be required.
- HQ generally will complete the review within 60 days.
- Since draft applications are not due until 30 days before exam, any issue with eligibility most likely will result in the applicant missing the scheduled exam.
ES-502 0.2 Exam failure. Some minorchanges.
- Detailed administrative review procedures and sample letters have been removed and incorporated into separate internal NRC documents.
- Added option to review the appeal internally atHQ.
- HQ chooses how to process the appeal.
- 1. Can review internally
- 2. Can refer to affected region
- 3. Can convene an appeal panel
THE ISSUE Is it an acceptable praetice to meadminister an identical examination to separate crews that have been separated in time over the finite testing period of the biennial written exam?
MAJOR ISSUES REVOLVNG AROUND REEIIEUSE OF TESTING EfECTTVENESS
>Do you have a testing praetice that measures up to sound and accepted principles of
>Are your exam ination conditions (size scope, discrimination quality) reistively uniform among crews.
DEFINITIONS
- DISCRIINATION VALIDITY
- The ability to discrininate or to make some distinction along a continuum of exaninee perfoniance to detennine whether or not your operators have suficiently 'mastered" the knowledge, skills, abilities, and other attributes to perform theirjobs.
- PREDICTABILT OF A TES
- The ability to forecast or anticipate the test items or topics that will appear on an examination.
- Previously administered test items reduce examination integrity because examination discrimination is reduced.
- When the bank of items is known or anticipated prior to the examination, the exam is highly predictable. Discrimination is reduced because the cognitive level at which the examinees are tested could decrease to the simple recognition level.
- To assume that the capability for items within an examination to discriminate, over time, in the same manner as those items discriminated on the first and second examinations is naive.
QUESTION 0 How does excessive repetitive use of test iterns over a short interval of time affect examination validity ?
If the examinees know or can anticipate the precise and limited pool from which the test items will be drawn, they will tend to only study from that pool and may likely exclude a larger domain of job knowledge.
- When high percentages of test item duplication takes place (e.g., >500/), the discriminant validity of the examination comes under question.
- Successive administrations of the same or closely similar examinations to different crews over the period covering the biennial written exam raises the potential for compromising examination integrity.
GOAL
- Achieve uniform testing conditions among crews as best as can reasonably be achieved so that the exam will be a reliable tool for assessing operator competence.
ACCEPTABLE PRACTICE
) To minimize the potential of reduced discriminant validity, a 50% portion of any readministered examination should consist of a replacement of modified or new items of like-kind content, psychometric attributes, and difficulty levels. Moreover,when items are to be repeated among successive crews, they should be repeated in a distributed manner and approximately equally over all previous exams so as to reduce predictability of a disproportionately large number of item coming form the most recent examination.
SUMMARY
- Successive administations of the same biennial requalification examination to different crews undergoing the same requalification training is considered unacceptable. This would seriously question the discriminant validity of the exam.
- When the content to be tested becomes highly_
predictable, and the boundaries ofwhat will likely be tested are known to the candidate, then the candidate will likely prepare ONLY to that level demanded by the examination.
- Improper testing practices will likely lead to an erosion of knowledge and long term decline in operator performance.
WHY YOU MAY ASK?
> When testing is diminished in level or onmuitted in kind, knowlege degradation occurs.
> The subtle but important coercion implicit in preparation for an examination is lost
> Attention will be focused on what leads to visible success for a candidate.
>- The loss of specific content area study can result in knowlege gaps that cause operator errors.
0 ES-601/602 CONDUCTING NRC REQUALIFICATION EXAMINATIONS
- NO SIGNIFICANT REV. 8 CHANGES
- REACTIVE INSPECTION / EXAMINATION
- SCOPE DETERMINATIONS BASED ON; o
PLANT PERFORMANCE o INSPECTION PROGRAM RESULTS o INITIAL AND REQUAL RESULTS o
OTHER FACTORS ES-601/602 CONDUCTING NRC REQUAL IFICATION EXAMINATIONS
- NRC CONDUCTED OPERATOR REQUAL EXAM COMPOSED OF THREE PARTS:
- TWO SECTION OPEN-REFERENCE WRITTEN 0
PLANT AND CONTROL SYSTEMS o
ADMINISTRATIVE CONTROLS/ PROCEDURAL LIMITS WALK-THROUGH EVALUATION DYNAMIC SIMULATOR ES-6011602 CONDUCTING NRC REQUALIFICATION EXAMINATIONS
- EVALUATION OF REQUAL PROGRAM AND OPERATORS o MINIMUM 12 OPERATORS REQUIRED o 3/4 OF OPERATORS MUST PASS EXAM o 2/3 OF THE CREWS MUST PASS THE SIMULATOR EXAM
Licensed Operator Requalification Inspection Procedure 71111 REACTOR SAFETY-INITIATING EVENTS, MITIGATING SYSTEMS, BARRIER INTEGRITY
. 1 INSPECTION OBJECTIVE To independently gather baseline inspection indicators to determine whether licensee performance meets the following cornerstone objectives:
- Initiating Events: To limit the frequency of those events that upset plant stability and challenge critical safety functions, during shutdown as well as power operations.
- Mitigating Systems: To ensure the availability, reliability, and capability of systems that mitigate initiating events to prevent reactor accidents.
- Barrier Integrity: To ensure that physical barriers protect the public form radionuclide releases caused by accidents.
REQUALIFICATION CORNERSTONES a Mitigating systems (75%)
w Barrier Integrity (25%)
. Emergency Preparedness Inspection Bases
- Inspection supports cornerstones because it can assess operator performance adequacy in responding to events. This inspection evaluates operator performance in mitigating the consequences of events. Poor operator performance results in increase risk due to the human performance factors terms, and assumed operator recovery rates and personnel induced common cause error rates assumed in the facilities IPEs.
INSPECTION AREA VERIFIES:
.Procedure quality and human performance which are both key attributes of the Mitigating Systems cornerstone for which there are no performance indicators.
. Human performance which is also a key attribute of the Barrier Integrity cornerstone for which there are no performance indicators.
PROCESS OUTLINE
=
mm__mlame
FY 2000 REGION II EXAMINATION SCH{EDULE Revised August 6. 1999 Date Facility N umber of Type of Activity Chief Examiner Examination Author
_________Docket No.
Canddates 9P-7/99 Catawba 3 RO Retake R. Baldwin Licensee 50-413 1 SROU 10/4/99 Hatch Initial Prep C. Payne 50-321 10/18/99 Hatch 10 SROI Initial C. Payne NRC ILicensee 11/1/98 50-321 2 SROU 11/8/99 Crystal River Requal Inspection G. Hopper 50-302___________
11/6/99 Browns Perry Requal Inspection.
C. Payne 50-260 11/29/99 Sequoyah Requal Inspection L. Mellen 50-327 110-9/99 Vogtle Initial Prep R. Baldwin 50-424 12/6f99 St. Lucie Requal Inspection G. Hopper 50-335 12/13199 Vogde 1 RO Initial R. Baldwin NRC 50-424 3 SROI 2 S ROU 1/10-2/14/00 Farley Requal Inspection TBD 50-348 1/1-217/00 Turkey Point Requal Inspection TED 50-280 1124/00 St Lucie Initial Prep TED 50-335 1/31/00 Brunswick Initial Prep
2 SROU 2/ -3/00 North Anna Requal Inspection TED 50-338 2114/00 Brunswick 12 RO Initial G. Hopper NRC 2f-)1/00 50-325 4 SROI________
4124/00 Farley Initial Prep M. Ernstes 50-348 4P-4/00 McGuire Initial Prep C. Payne 50-369 5/8/00 Farley 6 RO intinal M. Ernstes Licensee 5r-2/00 50-34 6 SROI 5/8/00 McGuire 6 RO Initial C. Payne Licensee 5r-2/00 50-369 2 SROI
~5 SROU__
5P-9/00 Browns Ferry Initial Prep L. Mellen 5 50259 6/12-7/14/00 McGuire
.Requal Inspection TED
DaeFacility Number of Type of Activity Chief Examiner Examination Author Docket No.
Cniae 6/12/00 Browns Ferry 1IORO Initial L. Mellen NRC 6/-6/00
.50-259 3 SRO1
~3 SROU 6/26/00 Oconee Initial Prep G. Hopper 7/10/00 Oconee 8 RO Initial G. -Hopper Licensee 7/17/00 50-269 2 SROI 3 SROU 7/24/00 Summer Initial Prep M. Ernices
____________ 50-395 7/24/00 Sequoyah Initial Prep C. Payne
____________ 50-327 8/00 Hatch Reaual Inspection TED 817/00 Summner 8 SROU Initial M. Ernaces NRC 50-395 8/7/0 Sequoyah 5 RO Initial C. Payne Licensee /NRC 50-327 4 SROU
________~~~
3SROI 8/14/00 Crystal River Initial Prep TED
_____________ 50-302________
8/28/00 Crystal River 3 RO Initial TED Licensee 9/11/00 50-302 3 SROI
~3 SROU 8/278/00 North Anna Initi Prep R. Aiello
____________ 50-338 8/28/00 Surry Initial Prep R. Baldwin 50-280 9/18/00 North Anna 7 RO Initial R. Aiello Licensee / NRC 9/25/00 50-338 1 SROI
~3 SROU 9/18/00 Surry 8 RO Initial R. Baldwin Licensee / NRC 9/25/00 50-280 2 SROI 3 SROU FY 2001 11/13/00 St. Lucie
-Initial Prep C. Payne 50-335 1.1/13/00 Turkey Point Initial Prep G. Hopper
___________ 50-250 11/27/00 Harris -
Initial Prep M. Ernstes
~50-400_
12/4/00 St. Lucie 4 SROI Initial C. Payne Licensee 50-335 5 SROU 12/4/00 Turkey Point 16 Initial G. Hopper NRC 12/18/00 50-250 12/11/00 Harris 3 RO Initial M. Emsces Licensee 50-400 2 SROl 3 3SROU 32/1 Robinson 7 RD Initial TED
__TED
__ 50-261 5 SROU__
Date Facility Number of Type of Activity Chief Examiner Examninauion Author Docket No.
Cniae 4/01 Vogde 16 Wntial TED NRC 50-424 4/01 Catawba 18 Wntial TED Licensee
___________ 50-41a 5101 Waus Bar 7 RD In1itial TED Licensee 50-390 3 SROI 6 SROU__
TRAINING MANAGERS CONFERENCE August 12-13, 1999 COMMENTS/QUESTIONS A.
Written Examination Questions:
- 1.
The most common issue raised by Hot License Candidates and Requal license holders surround the issue of "trick questions" and operator written exams not being a fair test of operator knowledge.
- 2.
If INPO creates a national initial licensed operator exam bank, will the NRC consider the INPO bank to be current questions that cannot be used as new questions on the exam to be developed?
- 3.
Do not feel that the written exam is a discriminatory tool. How many people do poorly on the written exam that are not weak on the operating test? Let us use our process to take care of the written with our audit exam.
- 4.
The utilities should NOT be the ones to develop the sample plan. This should be developed by the NRC for all examinations administered in the region.
- 5.
NRC needs to understand that increased difficulty of exam process is a negative motivator and could be a distraction to competent board operators. Recommend Survey to Understand Scope and Potential Impact on Safe Plant Operations.
- 6.
Evaluate changing initial exam grading to a curve for pass/fail.
- 7.
Exam difficulty his gone beyond reason and is impacting the requal program.
People are not willing to put up with the hassle and it does not result in better operators. It is impossible to meet question standards and avoid "Tricky" questions, very knowledgeable operators can appear less that competent based on complexity of question rather than a test of knowledge.
- 8.
The NRC exam has become an exercise in exam taking skills instead of a
,knowledge assessment.
- 9.
Would you comment on the following proposal ? Have a "team" from the utility come to the region and work directly with the chief examiner to develop the written exam. I would propose that a team of experienced utility instructions could bring the exam bank and associated reference material and they, with the chief, could produce the written exam in less than 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br />.
Benefits - lower man hours cost, reduced security concerns (less time on site),
fever negative exam report corrects.
2
- 10.
If the utility is producing the written exam, when (how may days/weeks) is your expectation for the chief to get the sample plan to the utility? The point is - getting the sample plan in accordance with NUREG 1021 will not work.
- 11.
In light of the NRC's new goals of reducing unnecessary regulatory burden and increasing efficiency and effectiveness, would it be possible to allow a licensee to build an initial license exam entirely from the bank (rather than 50% new questions), assuming the bank was an appropriate size and security concerns could be solved.
- 12.
The examination process seems to be getting harder as compared to a few years ago.
- 13.
Once we use a comprehensive level question, does it become a knowledge base questions the next time we use it.
- 14.
We may want to have an exam writing workshop.
B.
Operating Examination Questions:
- 1.
The continuous racheting of expectations is bypassing the SAT process. Example Cannot use a high importance JPM because it is perceived to be too easy, and operators are trained and tested on it.
- 2.
Current subjectivity on what is a discriminatory JPM with the removal of the questions.
- 3.
Need region workshops to calibrate us on future JPM direction.
- 4.
Why can't the selection of JPM's for the license exam be driven by the SAT process and K/A value? "Low discriminatory valve" is a euphemism for "too easy" and as a result, the difficulty of the exam is racheting up to an unreasonable level. This is contrary to the NRC stated goals.
- 5.
Open Reference Tech Spec. - it's too complicated to memorize. Tech Specs should be open reference or better yet covered by Operating Exams (JPM). We do not want our Operators to spend valuable time memorizing ITS, nor do we want them to operate from memory.
- 6.
Operating Exam - Section "A" Admin. (Category): This "category" of the new exam process needs to be integrated into the written and JPM (walk thru) segments, and eliminated as a separate entity - only a couple of areas are examined, with no margin for error! An individual can scope high on the written exam, do excellent on the simulator, and pass all of the systems JPMs yet fail to get licensed due to not passing a couple of admin "questions" - the knowledge and/or abilities could easily be included with other exam segments.
3 C.
Licensed Operator Experience Questions:
- 1.
What are experience requirements for SRO/RO?
- 2.
For a SAT-based program, what and where are the requirements for "responsible power plant" experience?
- 3.
Question - Experience Requirements 3 years 1 year 6 months on site What are the real requirements if you have SAT based program?
- 4.
"Responsible" power plant experience
/
This issue needs to be resolved INPO, NRC, NEI need to determine the specifics and let us know.
We need to know without reservation that SRO-instant candidates meet this ambiguous "experience" requirements pgr to them entering a license class.
D.
SAT Program Questions:
- 1.
What is/where do I find my "Commission Approved" Training program?
- 2.
How familiar are, and what kind of training have the examiners received on the SAT process? How familiar (knowledgeable) are the headquarters management on the SAT process? What kind of training have they received?
E.
Reactivity Manipulation Questions:
- 1.
10 CFR 55.59 - the use of SAT based program vice regulatory based programs. Why do you have to track individual control manipulations if you have a SAT based program?
- 2.
"Control Manipulations" in Requal - a prior guidance from previous NRC meeting clearly
- indicated bean counting control manipulation from the Denton letter was a thing of the past - SAT based requal training would naturally contain a large portion of the I
annual/biennial tasks and evolutions, therefore, program participants would be involved during simulator training/evaluation, and/or annual Op. Eval. JPMs; "individuals simulator critical tasks" went away and "crew critical tasks" were required.
4 Teamwork/communications command's control/by the team was the most important.
Bottom line - the implied expectation expressed on 8/12/99 is not congruent with that provided in 1989 by T. Peebles, S. Lawyer, and others who provides us guidance. It appears that we are returning to the middle to early 80's again.
- 3.
Reactivity Manipulations: 1) For ILO training what is the status of allowing simulator manipulations. (when unable to perform in-plant): also, define (What constitutes a control manipulation); why is a rod operability surveillance ok at one plant but not another?
What constitutes a large change? 2) For LOCT - INPO's policy for tracking manipulation seems to be in conflict w/NRC requirement (INPO doesn't require tracking on individual basis).
F.
GFES Questons or Concerns:
- 1.
2000 GFES Dates: Licensee have developed schedules and allocated resources to participate in a April GFES. Changing to a February, June, October schedule would be disruptive, perhaps a April, June schedule for 2000 would allow for a smooth transition (others Licensees made same comment).
- 2.
In order to facilitate transition to administering 3 GFE/year, is it possible to consider administering exams in April, June, and October during year 2000? This would minimize the impact on utilities that already have an exam scheduled. If implementation occurs in FY 2000 and exams are given in February, June, and October (as proposed),
unnecessary burden on these utilities could result.
G.
Miscellaneous Questions:
- 1.
ES-302 - General (D. 1.j) - What determines if a STA is "necessary"?
- 2.
When evaluating SRO success in "Classifying the REP" during the operating exam, what criteria do the examiners use for, when to start the 15 minute clock (expectation)? (15 minute from event to classification)
- 3.
Use of instructors is still an issue. The use of an instructor, who is on the exam security agreement, can't teach candidates attending the Requal program. This is an unnecessary burden on resource restrictions.
- 4.
Is there some way to do a better distribution of clarifications/rulings from one site in the region to another? This would help all of us meet your expectations.
I; a
REQUALIFICATION CORNERSTONES a Mitigating systems (75%)
a Barrier Integrity (25%)
s Emergency Preparedness Inspection Bases
- Inspection supports cornerstones because it can assess operator performance adequacy in responding to events. This inspection evaluates operator performance in mitigating the consequences of events. Poor operator performance results in increase risk due to the human performance factors terms, and assumed operator recovery rates and personnel induced common cause error rates assumed in the facilities IPEs.
INSPECTION AREA VERIFIES:
- Procedure quality and human performance which are both key attributes of the Mitigating Systems cornerstone for which there are no performance indicators.
- Human performance which is also a key attribute of the Barrier Integrity cornerstone for which there are no performance indicators.
IFY 2000 REGION n EXAMINATION SCHEDULE Revised Auzust 6. 1999 Date Facility Number of Type of Activity Chief Examiner Examination Author Docket No.
Candidates
_I 9/27199 Catawba 3 RO Retake R. Baldwin Licensee 50-413 1 SROU 1014199 Hatch Initial Prep C. Payne 50-321 10/18199 Hatch 10 SROI Initial C. Payne NRC Licensee 11/1/98 50-321 2 SROU 11/8/99 Crystal River Requal Inspection G. Hopper 50-302 11/6/99 Browns Ferry Requal Inspection C. Payne 50-260 11/29/99 Sequoyah Requal Inspection L. Mellen 50-327 11/29/99 Vogtle Initial Prep R. Baldwin 50-424 12/6/99 St. Lucie Requal Inspection G. Hopper 50-335 12/13/99 Vogde 1 RO Initial R. Baldwin NRC 50-424 3 SROI 1 2 SROU 1/10-2114100 Farley Requal Inspection TBD 5 0 -34 8 u a_
1/10-2/7/00 Turkey Point Requal Inspection TBD 50-280 124100 St Lucic Initial Prep TBD 50-335 1/31/00 Brunswick Initial Prep G. Hopper 50-325 2/7100 St Lucie 5 RO Retake TBD 50-335 1 SROI 2SROU 2/-3/00 North Anna Requal Inspection TBD 50-338 2/14/00 Brunswick 12 RO Initial G. Hopper NRC 2/21/00 50-325 4 SROI 4/24/00 Farley Initial Prep M. Ernstes 50-348 4124/00 McGuire Initial Prep C. Payne 50-369 518/00 Farley 6 RO Initial M. Ernstes Licensee 5122/00 50-348 6 SROI 5/8/00 McGuire 6 RO Initial C. Payne Licensee 5122/00 50-369 2 SROI 5
SROU 5/29/00 Browns Ferry Initial Prep L. Mellen 50-259 6/12-7/14/00 McGuire Requal Inspection TBD 50-369