ML20129H119

Discusses NEI 961022 Public Workshop for Facility Licensees Plan to Develop Initial Operator Licensing Exams Using Pilot Process Described in GL 95-06, Changes in Operator Licensing Program

ML20129H119
Person / Time
Issue date:	10/29/1996
From:	Richards S NRC (Affiliation Not Assigned)
To:	Boger B NRC (Affiliation Not Assigned)
References
PROJECT-689 GL-95-06, GL-95-6, NUDOCS 9610310060
Download: ML20129H119 (36)

Text

___ . . _ _ _ . . _ _ _ . _ - _ _. _

. e MEMORANDUM TO

Bruca A. Bogir, Diractor October 29, 1996 Division of RIsctor Controls .

! and Human Factors, NRR '

I FROM: Stuart A. Richards, Chief Original signed by:

! Operator Licensing Branch

Division of Reactor Controls

and Human Factors, NRR

]

l

SUBJECT:

NEl PUBLIC WORKSHOP FOR FACILITY LICENSEES

PLANNING TO DEVELOP INITIAL OPERATOR LICENSING

! EXAMINATIONS USING THE PILOT PROCESS DESCRIBED l lN NRC GENERIC LETTER 95-06 4

I On Tuesday, October 22,1996, the staff of the Operator Licensing Branch (HOLB) and

. representatives from each of the NRC Regional Offices participated in a public workshop to j review with facility licensees the pilot examination process described in Generic Letter 95-i 06, " Changes in the Operator Licensing Program." The workshop, which was sponsored by the Nuclear Energy Institute (NEI), was intended ta share lessons learned by previous pilot examination participants with those facility licensees that will be preparing examinations prior to July 1997. NEl developed the agenda for the workshop and prepared the list of registrants: a copy of each document *a attached. The gathering was conducted at the University Club International Center in Washington, DC, and was open to the public.

The NRC staff presented an overview of the pilot examination process, including lessons learned, and assisted facility representatives who had previously conducted pilot examinations in leading discussions of various examination development issues. A copy of the handout that the staff distributed to the attendees is attached, as is a copy of the presentation made by Jim Davis, of NEl, who coordinated the workshop. The staff did not approve nor was it provided with copies of the briefing slides used by the facility discussion leaders.

Overall, I believe the meeting was beneficial because it helped the attendees better understand the NRC's expectations regarding the development of initial licensing examinations by facility licensees. No regulatory decisions or commitments were made during the meeting.

Attachments: As stated

'?

Project No. 689 DISTRIBUTION:

uCentrol Filest "

HOLSRF k\

PUBLIC 31000? p y M)Ef

• See previous concurrence To seeshse o espy of this desunient,Indosee in the bes: 'C' = Copy without attachment / enclosure 'E' = Copy with attachment / enclosure m = Noe.,y OFFICE HOLB/DRCH l DRPM/PDST HOLB/DRCH 6 '

l NAME FGuenther:fg/rc SMagruder SRichards cfff -

DATE 10/25/96* 10/25/96* 10/.4/96 ,

OFFICIAL RECORD COPY pg b0310060961029 PDR PROJ PDR

%(kk h 689

4 MEMORANDUM TO: Bruca A. Bog:r, Director October 29, 1996 lo Division cf Reactor Controls

! 4 and Human Factors, NRR FROM: Stuart A. Richards, Chief 1

' Original signed by:

Operator Licensing Branch Division of Reactor Controls and Human Factors, NRR i

SUBJECT:

NEl PUBLIC WORKSHOP FOR FACILITY LICENSEES PLANNING TO DEVELOP INITIAL OPERATOR LICENSING

! EXAMINATIONS USING THE PILOT PROCESS DESCRIBED

IN NRC GENERIC LETTER 95-03 J

! On Tuesday, October 22,1996, the staff of the Operator Licensing Branch (HOLB) and i representatives from each of the NRC Regional Offices participated in a public workshop to j review with facility licensees the pilot examination process described in Generic Letter 95-l 06, " Changes in the Operator Licensing Program." The workshop, which was sponsored by the Nuclear Energy institute (NEI), was intended to share lessons learned by previous j pilot examination participants with those facility licensees that will be preparing i examinations prior to July 1997. NEl developed the agenda for the workshop and j prepared the list of registrants; a copy of each document is attached. The gathering was

! conducted at the University Club International Center in Washington, DC, and was open to i the public.

i

The NRC staff presented an overview of the pilot examination process, including lessons learned, and assisted facility representatives who had previously conducted pilot examinations in leading discussions of various examination development issues. A copy of

the handout that the staff distributed to the attendees is attached, as is a copy of the

presentation made by Jim Davis, of NEl, who coordinated the workshop. The staff did not

. approve nor was it provided with copies of the briefing slides used by the facility j discussion leaders.

1

Overall, I believe the meeting was beneficial because it helped the attendees better understand the NRC's expectations regarding the development of initial licensing

examinations by facility licensees. No regulatory decisions or commitments were made l during the meeting.

1

,' Attachments: As stated Project No. 689 i

DISTRIBUTION:

' Central Files =

l HOLB RF PUBLIC i *See previous concurrence j To seeshe a espy et tede dosimuss4 Inessee in the ben: *C* = Copy w#thout attach,nont/ enclosure *E* = Copy with attachment / enclosure w = w een, OFFICE HOLB/DRCH DRPM/PDST l HOLB/DRCH g, l J NAME FGuenther:fg/rc SMagruder SRichards c,gr- '

I DATE 10/25/96* 10/25/96* 10/2$/96 OFFICIAL RECORD COPY T

. . , - - . _ . - _ - - . , _ _ - - - - - . .m

p ata ug p k UNITED STATES g

j NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20066-0001 4

9 * * * * * ,o October 29, 1996 MEMORANDUM TO: Bruce A. Boger, Director Division of Reactor Controls and Human Fe s,NRR pC.

FROM: Wrt%. RMrds, Chief Operator Licensing Branch Division of Reector Controls and Human Factors, NRR

SUBJECT:

NEl PUBLIC WORKSHOP FOR FACILITY LICENSEES PLANNING TO DEVELOP INITIAL OPERATOR LICENSING EXAMINATIONS USING THE PILOT PROCESS DESCRIBED IN NRC GENERIC LETTER 95-06 On Tuesday, October 22,1996, the staff of the Operator Licensing Branch (HOLB) and representatives from each of the NRC Regional Offices participated in a public workshop to review with facility licensees the pilot examination process described in Generic Letter 95-06, " Changes in the Operator Licensing Program." The workshop, which was sponsored by the Nuclear Energy Institute (NEI), was intended to share lessons learned by previous pilot examination participants with those facility licensees that will be preparing examinations prior to July 1997. NEl developed the agenda for the workshop and prepared the list of registrants; a copy of each document is attached. The gathering was conducted at the University Club Intemational Center in Washington, DC, and was open to the pubhc.

The NRC staff presented an overview of the pilot examination process, including lessons learned, and assisted facility representatives who had previously conducted pilot examinations in leading discussions of various examination development issues. A copy of the handout that the staff distributed to the attendees is attached, as is a copy of the presentation made by Jim Davis, of NEl, who coordinated the workshop. The staff did not approve nor was it provided with copies of the briefing slides used by the facility discussion leaders.  !

Overall, I believe the meeting was beneficial because it helped the attendees better understand the NRC's expectations regarding the development of initial licensing examinations by facility licensees. No regulatory decisions or commitments were made during the meeting.

Attachments: As stated I Project No. 689 i

---v m -- -myu- -

NEI/NRC WORKSHOP on LICENSEE PREPARED ILO EXAM i

Meeting Agenda i

October 22,1996 \

University Club's International Center 7:30 a.m. Workshop Registration  !

Refreshments 8:30 a.m. Welcome Jim Davis, NEI 8:35 a.m. Overview oiExam process Stu Richards, NRC 9:10 a.m. Summary of Pilot experience Jim Davis, NEl 9:30 a.ai. Panel Discussion oflessons Kurt Rauch, San Onofre learned in preparing pilot exams Keith Link, North Anna Keith Bowden, Brunswick '

Ray Lueneburg, Millstone 10:30 a.m. Break 10:45 a.m. Sample Plan Exam Rob Sandstrom, San Onofre development schedule.

Fred Guenther, NRC staff 11:15 a.m. Written test development.

Ken Rach, Commonwealth Edison Max Bailey, NRC Region III 12:00 p.m. Lunch 1:00 p.m. Written test sample questions Ken Rach, Commonwealth Edison 1:20 p.m.

JPMs and Admin Questions Bob Nunez, Palo Verde Don Florek, NRC Region I 2:00 p.m. Simulator scenario development Keith Link, North Anna Charles Payne, NRC Region II 2:45 p.m. Break 3:00 p.m. Examination validity Rob Sandstrom, San Onofre Review process. Stu Richards, NRC staff 3:30 p.m. Question and Answer period All participant i 4:30 p.m. Adjourn Workshop ATTACHMENT 1

NEl/NRC WORKSHOP on LICENSEE PREPARED ILO EXAM Roger oderson Aaron Brown Operations Training Specialist Virginia Power American Electric Power Service Corporation Terry Brown {

Donnie Ashley Operations Supervisor Operations Training Manager Entergy Operations. Inc.

Tennessee Valley Authority Bruce D. Bryant Bobby Ayers Operations Training Supervisor ,

Nuclear Instructor, License Prep Team Leader Entergy Operations, Inc.

Duke Power Company l David Burns Max Bailey Initial License Class Program Coordinator Nuclear Regulatory Commission Pacific Gas and Electric Company Michael D. Baughman Bruce G. Carlson License Requal Training Section Leader Training Specialist Arizona Public Service Company Wisconsin Electric Power Company David L. Bauguess Timothy C. Cassidy Initial Training Lead Senior Operations Instructor The Cleveland Electric Illuminating Company North Atlantic Energy Service Corp.

Steve Beck John D. Christensen Nuclear Operations Plant Instructor Supervisor, Operations Training Georgia Power Company IES Utilities, Inc.

]

James F. Belzer Joey Clark Training Specialist Entergy Baltimore Gas and Electric Company Gary P. Crone Francis X. Biermann Nuclear Operations Training Supervisor Operating Supervisor, Training Southern Nuclear Operating Company Union Electric Company Mitch Crosby Robert Birley Georgia Power Instructor, LOT PECO Nuclear James Davis Assistant Manager, Operations Training Thomas Blindauer Department Senior Instructor, Operations Initial Training Institute of Nuclear Power Operations Southern Nuclear Operating Company Jim Davis Director, Operations Dennis F. Bonilla Nuclear Energy Institute Director, Utility Services General Physics Corporation Richard W. DeVercelly Control Room Supervisor / Instructor l Keith A. Bowden New York Power Authority Project Specialist. Operator Training i Carolina Power & Light Company David E. Dietzel l Senior Operations Instructor '

Gary T. Box Entergy Operations Inc.  !

License Initial Training Section Leader Arizona Public Service Company ATTACHMENT 2 l

1 .

~ '

Don Duquette Heidi Heaton Vice President, Utility Services Supervisor, Operations initial Training  ;

General Physics Corporation Carolina Power & Light Company )

Mark Elliott Roy Hickok Director, Utility Services North Atlantic General Physics Corporation Steven J. Jobe Charles Embry - Supervisor, Operations Training Programs .

New York Power Authority Nebraska Public Power District l l

James Evans Ed Jones Baltimore Gas & Electric Nuclear Operations Plant Instructor Georgia Power Company Randy H. Evans Licensed Operator Initial Training Program Lead Dennis M. Jones  !

Instructor Qualification Instructor  !

Tennessee Valley Authority The Toledo Edison Company l l

Michael Evringham Derrick Kaopuiki l Operations Traimng Section Head Coordinator Initial License Class  !

Maine Yankee Atomic Power Company Washington Public Power Supply System l l

Frank Fagan Reggie Kimray i Exam Developer Nuclear Station Instructor i Yankee Atomic Electric Company Duke Power Company 1 Carl Fedako Ricky King Nuclear Operations Training Seminar Tennessee Valley Authority Pennsylvania Power & Light Company Jon Kirsch Brian M. Finn Supervisor, Operations Training Training Manager Maine Yankee Atomic Power Company Vermont Yankee Nuclear Power Corporation Tom Kuhas Don Florek Duquesne Light Company Nuclear Regulatory Commission Steven L. Laeng Ronald J. Frigo Qualification Instructor Operations Initial Training Supervisor The Toledo Edison Company Consumers Power Company Ryan Lantz Tom Fueston Nuclear Regulatory Commission New York Power Authority Jim Lash Melissa Gallian Senior Technical Instructor Northern States Power Company I Nuclear Operations Instructor Florida Pow 2r Corporation Keith M. Link William J. Green Instructor  :

Lead Licensed Instructor Virginia Power Nebraska Public Power District Frank L. Maciuska Fred Guenther Supervisor, Operations Training Nuclear Regulatory Commission Rochester Gas and Electric Corporation  !

l 1

Harold McCallum Kurt Rauch Supervisor Operations Training Southern Califorma Edison Virginia Power Michael K. Rasch Senior Operations Instructor Norman Meaker Entergy Operations, Inc.

Senior License Instructor Rochester Gas and Electric Corporation Stu Richards Glenn Meyer Nuclear Regulatory Commission Nuclear Regulatory Commission Joseph S. Romanowski William C. Miller Operations Training Seminar Senior Analyst New York Power Authority Florida Power & Light Company Robert L. Sandstrom Ross Moonitz Manager, Nuclear Training PECO Nuclear Southern California Edison Company John Munro Charles W. Sawyer Nuclear Regulatory Commission Senior Technical Specialist Duke Power Company Walt Nelson Training Supervisor Robert W. Scott American Electric Power Service Corporation Operations Instructor Baltimore Gas and Electric Company Robert E. Niedt elski 8

Supervisor, Initial Training Unit David Seipel Baltimore Gas and Electric Company Training Supervisor l American Electric Power Service Corporation Robert A. Nunez Operations and Engineering Training Leader W. Mike Shelly Arizona Public Service Company Manager, Nuclear Training Entergy Operations, Inc.

Fred Nygard Senior Instructor /LOIT Coordinator Johnie G. Smith Northeast Utilities Nuclear Licensed Operator Training Supervisor Florida Power Corporation Robert L. Parnell Supervisor, Simulator Training Jonathan Sparks GPU Nuclear, Inc. Public Service Electric Mark Parrish Charles Tyner Scientech, Inc. Examiner WD Associates,Inc.

Charles Payne Nuclear Regulatory Commission Arthur Vest Senior Operations Steve Pettinger Entergy Operations, Inc.

Comed Michael T. Wagner Ken Rach Senior Operations Instructor BWR Operations Training Supervisor, Production Entergy Operations. Inc.

Traimng Center Commonwealth Edison Company Gregory P. Young Instructor GPU Nuclear, Inc.

i Keith Young Manager. Nuclear Training IES Utilities. Inc.

Joseph Zerbo Manager, Operations Training Public Service Electric and Gas Company Larry Zilli President Zilli Inc.

i

. . . .. ..- . - - - - . - - . . . - - . ~ - . - . . . . . . . .. . - - _ - . - .. . - . . . .. . .

1. Overview of Pilot Examination 3. Objectives Process o Improve Efficiency and Maintain Effectiveness Eliminate Reliance on NRC Contractors Increase Involvement by Facility Licensees o Aemain Consistent with the Act and Part 55 o Changes Should not Affect the Applicants ,

2. Background 4. Pilot Examination Program o SECY 95-75, SECY 96-123,and SECY 96-206 o Examinations Prepared Per NUREG-1021, NUREG/BR-0122 and the Generic Letter o Generic Letter 95-06 o Generic Letter 95-06 Criteria o Atomic Energy Act of 1954 Ensure Integrity of the Examinations o 10 CFR 55, " Operators' Licenses" -

Maintain Consistency Limit Predictability o NUREG-1021, " Examiner Standards" o 22 Examinations Conducted - All Regions and o NUREG/BR-0122, " Examiners' Handbook" Vendors Represented o 54 R0s and 92 SR0s Tested from 10/95 through 4/96 ATTACHMENT 3

mw a A.=4d- eam 6, d. g.ha--J3.4__,54W.,JJA3>,q,3rds,.

. . 3M C 4 4 3 JMAJ.eJ.,3, 43, AB,NL-s g4 M g 33._ E 4.4. A M4 hag d . E d 4 a.e.44.s M e d h h4__pm.4. ,, h wa. 3& _m.4_ _a_.m_.ha4_A _s A da44 41 hMM%

4f. rag-a i. .e 4mJE t

i g' .

4 ii

.4 e

4 4  ;

+.

b

• E a i j,

1 s

k T

L I

P I

i h

4 4

.-l i

1 i

l l

l 1

i 1

e 1

1 l

._ - - ,_ __, _ _ _ _ _ _ _ ,,___ _ = .,.#--. .. .a . , y - .. _ , , ,-.,..n ,, ,. _ym, --.y_.. -. , ,- ,

5. Industry Coordination 7. Pilot Program Issues / Lessons Learned o Meetings and Workshop o Examination Security and Potential for Compromise o Formal Comment Period 10 CFR 55.49, Integrity of Examinations, states i

" Applicants, licensees, and facility licensees shall not

o. Industry Concerns engage in any activity that compromises the integrity of any application, test, or examination required by this Restrictions on Examination part."

Authors Test Item Histories o ES Guidelines (Revision 7) Remain in Effect and Should be Duplication from Audit Reviewed Examinations Examination Banks o ES and Pilot Program Guidelines Misinterpreted, e.g., -

Site-Specific Task Lists Definition of Component Failure NRC-Prepared Examinations e Facility Learning Curve Steeper Than Expected o inecease Emphasis on Technical Accuracy and Test-Item Quaitty

6. Planning Milestones 8. More Pilot Program Issues / Lessons Learned  !

t o Continue the Revised Process Through o Substantive Chanqes Complete and Exams Finalized Prior to July 1997 Scheduled Adminis+-?*iaa Date - No Substantive Changes  ;

During Exam Week '

o Developed Detailed Rulemaking Plan to Require Licensees to Prepare Exams o ES Target Dates Not Met, e.g., Complete Draft Exam Based on Guidance in NUREG-1021 Submittal Due at -30 Days o Rulemaking Plan also Discusses: o QA Checklists Not Completed Impact on Licensees o Facility Employees with Substantial Training Role Acceptance by Licensees Pros and Cons - Vulnerabilities o Low Level of Knowledge for Written and Operating Exam Exam Results Comparison Test-Items Industry Comments to Draft NUREG-1021 o Test-Item Source for Written Exams and Operating Tests Changes to Process from Pilot and Comments o Duplication of Test-Items from Audit Exams '

i

1. Examin: tion Schedulo 3. Sample Plan Guidelines ,.

and

• Objectives Sample Plan Development -

Promote Diversity and Consistency 10 CFR 55 Sampling Requirements Enable NRC to provide feedback early in  !

the process t

Written Exam Per Handbook Model

• Three Tiers: PWG / Systems / E& APES 1-3 Groups per Tier Multiple K/A Categories 1

• Operating Test Per ES-301 W/T: Safety Groups and Task Lists in K/A Catalogs l Simulator: Select Diverse Events i Covering all Competencies and Rating Factors- '

2. NRC Significant Events 4. Integrated Examination Outline Time Line '

• Written: K/A Record Forms from NUREG/BR-Taraet liik 0122, Rev. 5

- 120 Brief point of contact and issue

• W/T: Fona ES-301-1, " Administrative  ;

notification letter Topics Outline" '

Form ES-301-2, "Indiv. Walk-

- 60 Integrated examination outline due through Test Outline"

- 55 NRC feedback on examination outline

• Simulator: Form ES-301-3, " Scenario Events"  !

- 30 Complete examinations, supporting

• General: Form ES-201-3, " Interim  :

documentation, and references due Examination Outline Quality Assurance Checklist"  ;

- 14 Examination review with facility licensee Form ES-201-2, " Interim (preparation week) Examination Security Agreement"  ;

-7 Examinati( ns approved by NRC

. , _ . _ _ _ _ . . _ _ . . _ _ _ _ . _ _ _ . _ _ _ _ _ _ _ . _ _ _ _ _ _ . _ _ . . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ = _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _

+

1. Written Examination 3. Other Guidance .

Development

• ES-401, Rev. 7, NUREG/BR-0122, Rev. 5, and approved outline 100% multiple choice (no matching)

Level of knowledge Level of difficulty (should discriminate between competent and less than competent operators)

Review and approval process (Interim Form ES-401-2, i Written QA Checklist) l

=

Reference materials sufficient to support why  !

answers are right or wrong l t

2. GL 95-06 Rules 4. Interin Form ES-401-2 '

50 / 40 / 10

1. Questions and answers technically accurate and applicable to facility.  !

Significant modification means a 2. K/As and learning objectives referenced for all change in the stem conditions and at questions.

least one distractor 3. R0/SRO overlap no more than 75%.

New questions should be at the 4. Item duplication from practice exams, quizzes, and '

comprehension or application levels the last two licensing exams is no more than 25%.

Supersedes 10% bank limit in ES-401, 5. No item duplication from the license screening / audit C.I.c exam.

6. Bank use meets limits (50% bank / 40% modified / 10% i

. No more than 25% repetition from quizzes new); with new items at the analysis / comprehension or exams (including last 2 NRC) level. i

7. References / handouts provided do not give away

. N0 repetition from the audit exam answers. i j

8. Question distribution meets Examiners' Handbook and

. Identify the history (i.e., source and proposed examination outline. ,

i date last used) of every question 9. Question psychometric quality and format meet  ;

Examiners' Handbook guidelines (refer to Chapter 4,  !

Appendix A, Sections 1 and 3).

10. The proposed exam contains 100, one-point, multiple choice items. Total correct and corresponds to i value on cover sheet.  !,

I l

5. Pilot Feedback 7. This example is acceptable and was actually used on an NRC pilot exam. t

. Written exams - some exams were administered i with virtually no changes Unit 1 is operating at 100% power. While conducting a system lineup verification, the

. However, a number of exams had many questions operators note that FW-27 was in the OPEN that had - position. The valve checklist called for FW-27 to be in the CLOSED position. There are no i Low level of knowledge or difficulty outstanding R&Rs or Configuration Control Cards on Low operational validity FW-27.

Low discriminatory value l Implausible distractors What action should the operators take? '

a. CLOSE FW-27 as required by the valve checklist  !

and note the change on the valve checklist. t i

b. CLOSE FW-27, report the valve status change to the control room, and prepare a Problem l Investigation Process (PIP) report.

6. WRITTEN EXAMPLES c. Leave the valve OPEN and prepare a Configuration Control Card to document the This is an example of an unacceotable question: deviation.

l If an error is made when taking a log reading, the d.

error should be corrected in which of the following Leave the valve OPEN because the valve is ways: currently in the correct position for the plant i lineup.  !

a. Erase the entry, then write the correct value.

b. A single line through the mistake, initial and i date the correction.  !

c. Line through the mistake several times to cover the mistake, then write the correct value next to the crossed out value. l

d. Use white out, and then when dry, write the correct value, initial and date the correction.

[

- _ _ ..----...-,___.-----._-____-._.-__.-a- - - . - _ - _ _ - - - _ - - - _ - _ _ _ _ _ _ _ _ _ _

-.-- ~ -.- - - - - - - - - - - - - --

1. WALKTHROUGH TEST - JPNs 3.

NUREG-1021 CRITERIA (CON'T)

Alm ADMIN TEST ITEMS o Incorporate Facility-Specific and Industry-Generic Operating Experience JPMs o At least 1 Low Power or Shutdown Condition JPM  !

o 1 or 2 Alternate Path JPMs  !

o At least 1 In-Plant JPM Evaluates Actions i During Emergency or Abnormal Conditions

2. PIL0T CRITERIA j o At least 1 JPM Requires Entry into a RCA l 0 At least 2 (1 for Upgrade SR0s) New or

• Sienificantly Altered JPMs PRE-SCRIPTED QUESTIONS ,

. Significantly Altered - At least One o Open- Reference Items That Demonstrate i Substantive Change that Alters Course of Understanding By Using Knowledge [

Action i o Evaluate PRA/IPE (DAS) for Inclusion i

NUREG-1021 CRITERIA 4. PRE-SCRIPTED QUESTIONS (CON'T) o Differentiate between Competent and Less Than O Ho Direct Look-Up Questions or Simple Memory  !

Competent Operators Questions - Test Problem Solving or Decision Making (NOTE: Simply Eliminating Reference o Evaluate Broad Range of K/As, Systems and Components, Operations and Events Information From Question Stem does H91 Ensure  ;

the Question is Satisfactory Open-Reference  !

Question)  ;

o SR0 Applicants Evaluated at Greater Depth on i Administrative Topics o Memory Questions (Closed Reference) Testing Auto Actions, Set Points, Interlocks, o No Duplication of Material from Written Exam or Immediate Actions are Allowable between Categories of the Operating Test ,

L

5. NGW-EXAMPLE JPM 7. NON-EXAMPLE PRE-SCRIPTED QUESTION

. a LOW LEVEL OF KNOWLEDGE AND DISCRIMINATORY VALUE DIRECT LOOK-UP QUESTION / LOW LEVEL 0F KNOWLEDGE (MEMORY) o TASK - MANUAL INITIATION OF ATWS ARI/RPT o QUESTIDN - RCIC HAS BEEN OPERATING IN THE TEST o INITIATING CUE - SRO DIRECTS YOU TO INITIATE THE MODE FOR AN EXTENDED PERIDO OF TIME WHEN IT IS ATWS ARI/RPT SYSTEM NOTED THAT THE CONTAINMENT OXYGEN LEVEL IS INCREASING. WHY IS THIS HAPPENING 7 o REQUIRED ACTIONS - ARM AND DEPRESS ATWS ARI/RPT PBs o REFERENCE ALLOWED - YES o TIME TO COMPLETE (WITH VERIFICATIONS) - LESS o ANSWER - AIR IN-LEAKAGE FROM THE RCIC TURBINE THAN 1 MINUTE GLAND SEAL SYSTEM.

SUMMARY

EVA.UATION - ALTHOUGH TASK HAS A HIGH K/A, o PRECAUTION IN RCIC OP STATES " EXTENDED IT SHOULO B E SUCCESSFULLY COMPLETED BY 100% OF ALL OPERATION OF THE RCIC SYSTEM CAN CAUSE APPLICANTS INCREASES IN THE PRIMARY CONTAINMENT OXYGEN CONCENTRATION DUE TO AIR IN-LEAKAGE FROM THE RCIC TURBINE GLAND SEAL SYSTEM."

o

SUMMARY

EVALUATION - QUESTION DEMONSTRATES ONLY THAT APPLICANT CAN FIND INFORMATION, NOT PROBLEM SOLVING. ANALYSIS OR UNDERSTANDING

6. EXAMPLE JPM 8. EXAMPLE PRE-SCRIPTED QUESTION

. a COMPREHENSION / UNDERSTANDING LEVEL OF KNOWLEDGE COMPREHENSION /UWERSTANDING LEVEL OF KNOWLEDGE o TASK - INITIATE ALTERNATE COOLANT INJECTION o QUESTION - EXPLAIN THE PURPOSE AND/0R EFFECT FOR W/NUC. SW EACH LOCAL ACTION JUST SIMULATED IN THE RESET OF THE FW FRV.

o INITIATING CUE - SRO DIRECTS YOU TO INITIATE ALTERNATE COOLANT INJECTION W/NUC SW o REFERENCE ALLOWED - YES o

o

SUMMARY

EVALUATION - QUESTION DEMONSTRATES

'B';REQUIREDACTIONS-(1}LINE-UPNUC.SWTORHR (2) ID THAT RHR 'B INJ VALVE WILL NOT OPEN APPLICANT'S UNDERSTANDING OF THE LOCAL ACTIONS THAT (ALT. PATH); (3) OPEN RHR X-CONN ISO. VALVE WERE SIMULATED AS PART OF TASK TO RESET FW FRV.

o TIME TO COMPLETE (WITH VERIFICATIONS) - ~10 MINUTES

SUMMARY

EVALUATION - GOOD USE OF PROCEDURE W/ll i ACTION STEPS (9 CRIT.) AND USE OF ALT. L.U. TO PERFORM TASK

1. SINULATOR SCENARIO DEVELOPMENT 3. NUREG-1021 CRITERIA'(CON'T) o k Repeat of Scenarios on Successive-Days o llg Duplication of Material from Written

xam n between Categories of the Operating Test o Incorporate Facility-Spcific and Industry-Generic Operating Expericnce o Ensure h Significant Expected ,

Operator Action is Documented (Form ES-

2. PIL0T CRITERIA 301 Operator Action Statements such as "E ters AOP XXX.X.X and Performs o At least 1 New or Siend "icantly Altered Scenario Appropriate Actions" are get Sufficient i Er Scenario Set - Ver'"y that M Applicant is Tested Using at least 1 New or Significantly QUANTITATIVE REQUIREMENTS:

Altered Scenario o Minimum Number of Normal Evolutions,

. Si nificantly Altered - At least One Instrument and Component Failures, and Su stantive Change that Alters Course of Action .

Major Transients for Eggh Applicant (Form ES-301-5) o Other Scenarios Altered to Prevent Recognition by o i Applicants Initial Conditions Varied and Include  !

Startup, low Power, and Full Power '

o Evaluate PRA/IPE (DAS) for Inclusion NUREG-1021 CRITERIA i o Differentiate between Competent and Less Than Cogetent Operators - Provides a Comprehensive Evaluation of the Inteerated Plant Knowledges and Skills required of License Applicants i t

o Evaluate Broad Range of K/As, Systems and i Components, Operations and Events 1

I i

i

4. QUALITATIVE REQUIREMENTS: 6. NON-EXAMPLE SCENARIO (CON'T) o Variety of Systems Affected Within Ensh Type of o SUMARY OF R0 EXPECTED ACTIONS:

Event o 1 - INC1 EASE RECIRC FLOW WHILE OBSERVING REACTOR Severity of Events Allow Ensh Applicant to POWER AND TURBINE LOAD Demonstrate Competence Across Range of Conditions 2 - OBSERVE AND REPORT APRM UPSCALE TRIP /INOP ,

ALARM; CHECK OTHER INDICATIONS TO VERIFY FAILURE o Understanding of System and Component AND POWER STABLE; PLACE APRM F IN BYPASS AND RESET Interactions Evaluated by Having 1 Equipment 1/2 SCRAM Failure Exacerbate Another 3 - REPORT INBD RWCU FAILED SHUT AND ATTEMPT TO o Successive Failures Lead to a Gradual REOPEN; SECURE RWCU Deterioration in Plant Status 4 - REPORTS NR LI FAILED HIGH o Scenarios Expose Applicants to Situations That RX VESSEL LVL 8 TRIP"); VERIFIE(ANNUN. S LEVEL STABLE "FW CONTROL Could Cause Plant Degradation or Threaten the Health and Safety of the Public 5 - REPORTS INDICATION OF DECREASING VACUUM REDUCE ,

RXPOWERW/RECIRCFLOW;MANUALLYSCRAMIFVdCUUM o Scenario Sets Enable Applicant to be CANNOT BE MAINTAINED > 25"HG Evaluated on All the ired Competencies and

• ire Unsatisfactor_v ating Evaluations if an 6 - OBSERVES LOSS OF FW; MONITOR LEVEL AND CONTROL icant Performs Poorly (Form ES-301-6) LEVEL BETWEEN LEVELS 2 AND 8 o NOTE: Forms ES-301-5 and 6 Shall be Forwarded  :

to Chief Examiner With ProDosed Scenarios  ;

5. NON-EXAMPLE SCENARIO 7. EXAMPLE SCENARIO I

- LOSS OF CONDENSER VACUUM =

MSL RUPTURE / LOSS OF LEVEL INST 1 o IC: 85% POWER MOL W/ CONDENSATE PUMP IB OOS, HPCS o lC_: 90% POWER E0L W/FLOOO-UP LI AND TBCCW HXER A ,

00S AND TRANSFORMER 111B OOS 00S. TORUS COOLING I/S.

t o EVENTS: o EVENTS: (* - PREINSERT) j 1 - N/R - INCREASE POWER W/RECIRC FLOW l - R - INCREASE POWER W/RECIRC FLOW 2 - I - APRM F FAIL HIGH 2 - N - PERFORM RCIC FULL FLOW SURVEILLANCE 3 - C - RWCU INBOARD VALVE DRIFT SHUT 3 - I - RX PT-404A FAILS TO 0%

4 - I - FAILURE OF THE NON-SELECTED NR LI 4-C

• HPCI TRIP ON INIT. '

5 - M - LOSS OF CONDENSER VACUUM (10 MINUTE RAMP) 5 - M - MSL RUPTURE IN STM TUNNEL AT 100%

6 - C - SLOW TRANS OF SWG 11A 6 - C

• R0D 02-43 DOES NOT SCRAM

1. Examination Validity 3. The Review Process and the . Self-check by author Review Process

• Management review

- Authority to speak on behalf of facility licensee Can be supervisor who did not teach applicants

- NRC expectations

2. Maintaining Examination Validity .

Validate the examination (and Reliability)

- Operating test

. Adhere to external criteria such as - -

Written exam Sampling requirements .

NRC evaluation Limits on bank use 100 question multiple-choice format 10 JPMs with follow-up questions

- Chief examiner review

- Supervisory approval at least 2 simulator scenarios - Preparation week activities

. Don't compromise on internal criteria such as -

~

Level of knowledge (focus on comprehension, application, and analysis questions)

Level of difficulty (the examination must discriminate)

. Verify accuracy and quality i

8. EXAMPLE SCENARIO (CON'T) 10. EXAMPLE SCENARIO (CON'T) 7 - I - RCIC FAILS TO XFER FROM FULL FLOW TO AUTO o

SUMMARY

EVALUATION:

INJECT (1) SCENARIO WHEN COMBINED WITH OTHERS IN SET 8 - C - SRV D STICKS OPEN PROVIDES GOOD COVERAGE OF ALL QUANTITATIVE AND QUALITATIVE CRITERIA 9 - C - RPV "A" INST FAIL TO 100% AT 400# AND RPV "B" FAIL TO 100% AT 300# (2) ALL OF THE MALFUNCTION EVENTS REQUIRE UNDERSTANDING OF PLANT / SYSTEM RESPONSE TO DIAGNOSE o

SUMMARY

OF EXPECTED ACTIONS: AND MANUAL CONTROL BY THE OPERATORS TO MITIGATE 1 - INCREASE RECIRC FLOW WHILE OBSERVING REACTOR (3) GOOD LEVEL OF SR0 COMMAND AND CONTROL, AND POWER AND TURBINE LOAD PROCEDURAL USAGE REQUIRED FOR SCENARIO 2 - PERFORM RCIC FULL FLOW SURVEILLANCE 3 - IDENTIFY RX LP PERM. (<450#) ACTIVATED; ASSESS EFFECTS OF INST FAILURE - "A" LTs SUPPLY LOWER THANACTUALSIGNALTOWRF{l NST AND (2) TAKES ACTIONS IAW TS 3.2 4 - IDENTIFY HPCI AUTO INIT FAILURE, ATTEMPT TO START

9. EXAMPLE SCENARIO (CON'T) 5 - IDENTIFY MSL RUPTURE AND MSIV ISOLATION; CONTROL PRESS W/SRVs; ENTER RPV AND CONTAINMENT CONTROL E0Ps 6 - ID FAILED R0D AND MANUALLY INSERT (02-43) 7 - ID RCIC FAILURE AND TAKE MANUAL-CONTROL (DID NOT AUTO SHIFT FROM TEST LINE-UP) 8 - ID STUCK OPEN SRV AND ATTEMPT TO CLOSE; PLACE RHR IN TORUS COOLING 9 - RECOGNIZE LEVEL DIFFERENCES BETWEEN "A" AND "B" LIs WHEN RX PRESSURE <400# AND <300# (LIs EXPERIENCE " DEGASSING" EFFECTS); IMPLEMENT RX FLOOD-UP E0P AND INJECT W/RHR AND MAINTAIN RX PRESS 60# > TORUS

i, i

i I

l i

i l lNITIAL OPERATOR LICENSING PROGRAM PILOT EXAM PROCESS l NEl/NRC l WORKSHOP l

l October 22,1996 l

James W. Davis Nuclear Energy Institute l

! 202-739-8105 i

f 4 d'

.11 .

l .

i I

PURPOSE l i

i i

! To review " Pilot" process for licensee l prepared initial operator licensing j examinations.

- NRC Generic letter 95-06.  !

j - Use extended by Commission to July 97.

I Share lessons learned from past pilot

examinations.

{ - NRC Headquarters staff.

l - Each NRC Region.

l - Pilot plant participants.

l - Audience.

l NOT a review of NUREG 1021 rev 8.

- Still under NRC review.

2

WORKSHOP SCHEDULE 7:30 a.m. Workshop Registration Refreshments 8:30 a.m. Welcome Jim Davis, NEl 8:35 a.m. Overview of Exam process Stu Richards, NRC staff 9:10 a.m. Summary of Pilot experience Jim Davis, NEl 9:30 a.m. Panel Discussion of lessons Kurt Rauch, San Onofre leamed in preparing pilot exams Keith Link, North Anna Keith Bowden, Brunswick Ray Lueneburg, Millstone 10:30 a.m. Break 10:45 a.m. Sample Plan and Exam Rob Sandstrom, San Onofre development schedule. Fred Guenther, NRC staff 11:15 a.m. Written test development. Ken Rach, Commonwealth Edison Max Bailey, NRC Region 111 12:00 p.m. Lunch 1:00 p.m. Written test sample questions Ken Rach, Commonwealth Edison )

1:20 p.m. JPMs and Admin Questions Bob Nunez, Palo Verde l Don Florek, NRC Region I )

2:00 p.m. Simulator scenario development Keith Link, North Anna Charles Payne, NRC Region ll l 2:45 p.m. Break 3:00 p.m. Examination validity Rob Sandstrom, San Onofre Review process. Stu Richards, NRC staff 3:30 p.m. Question and Answer period All participants 4:30 p.m. Adjourn Workshop l

3 i

5.

l ADMINISTRATIVE I

i Refreshments at breaks.

Lunch on own.

- Food court handy.

Questions.

- Ask during presentations.

- Use cards for questions for the final Q&A period.

Think Posi:ive.

4 M'

i 1

4 1

t i

1 1

l STU RICHARDS

! NRC STAFF i

s 5

! PILOT SCHEDULE l l

! l i

j Month Plant Region l -

October Brunswick ll

) La Salle County 111 Palo Verde IV l

}' -

November Limenck I

Millstone-3 I

Pilgrim I j San Onofre IV December Brunswick II l Fermi Ill

Fort Calhoun IV

) Vogtle 11 January Cook Ill

North Anna ll

! McGuire ll

) Zion Ill j -

February Ginna I

Milstone-2 I i Oconee 111 l Robinson ll j March Crystal River ll j

April Braidwood lll

! s h'

GENERAL OBSERVATIONS Process is more involved than some had programmed for.

- Exam banks not very supportive.

- Understanding of requirements.

No clear pattern of pilot issues.

- Some easily understood and met standards.

- Some exams not judged difficult enough.

More concerns in preparing the written test than other areas.

Industry process can be more efficient.

- Better understanding of requirements.

- Effective use of resources.

7

i .

4 i

l LEVEL OF EFFORT l

Wide range of reported man-hours.

- Should be able to improve efficiency of i process.

l Overall 450 to 1764 hours0.0204 days <br />0.49 hours <br />0.00292 weeks <br />6.71202e-4 months <br />.

l - Hard getting good basis for comparison.

l - High number resulted from extra people on l team for experience.

l Written 250 to 640 hours0.00741 days <br />0.178 hours <br />0.00106 weeks <br />2.4352e-4 months <br />.

- Up to 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> a question?

- Biggest area for most pilots.

l JPMs 26 to 834 hours0.00965 days <br />0.232 hours <br />0.00138 weeks <br />3.17337e-4 months <br />.

- Administrative questions were a problem.

Operating 60 to 420 hours0.00486 days <br />0.117 hours <br />6.944444e-4 weeks <br />1.5981e-4 months <br />.

j - Least difficulty in this area.

l - Had expected to be an area of disagreement j on scenario complexity.

i l

i

}

s

PILOT MAN-HOURS m urs Total Man-hours 1800 1600 1400 1200 1000 i 800 - l 600 i

0 --

i i i i e i i i i i i O k n Written mJPMs O Simulator O Other 9

t i

1 i

i REVISIONS TO EXAM l

1 I

No split between minor and major I

changes.

i l

Written 5 to 80% revised.

j - Expectations on level of difficulty.

l - Psychometrics.

! - Modified bank questions.

i c

JPMs 0 to 100% revised.

j - Significant effort on administrative questions.

I - Issue of open reference instead of direct l look-up i

O aerating 0 to 100% revised.

j - Scenario validation.

! - Level of documentation for scenario.

5 10 i

1

1 -

i -

I WRITTEN EXAMINATION i

Generated most comment and discussion.

- Highest level of effort to prepare.

Most questions were new.  ;

- Adequate exam banks were not available.  ;

- Concern with open and closed exam banks.

Level of difficulty no: well understood  !

by some.

- Need for training in this area. l Psychometric issues.

What must change to stem and at least one distractor accomp ish? l

- Can intent of question be the same?  ;

l l

Pr l

l

l-INDUSTRY CONCERNS i

l l Restrictions on use of instructors to l j prepare examination package. I

- Excluded if " played a substantial role in l training".

i - Exam integrity issue.

I Examination security issues.

! - Highly prescriptive criteria contained in Pilot l Examination Guidelines.

i Memory level vs higher cognitive level

questions.

4

! Need to reduce the amount of

)

reference material.

l Level of effort needed to document l exam question validity. ,

i i

i i

i,

i i

! INDUSTRY " PERCEPTIONS"

}

i l NRC has additional " unwritten" i guidance that has not been shared i with the industry.

i Headquarters involvement evident l during pilot process.  ;

- Shifts in guidance or standards during the ,

process.

- Kept facility off balance.  ;

- Seen in October and November.

Examiners sometimes make

" optional" items mandatory.

- Issue of what is provided to support a written question.

- Additional requirements on format that create added burden.

Test difficulty has been upgraded.

- Previously used, or similar questions, rejected.

13

!- i

- l i

l CONCLUSIONS l

! Early and open communications with

! chief examiner critical.

- Differing expectation levels.

- Well defined review process.

j Need experienced examination j prepara<: ion team.

l - Key factor in effort to prepare.

l - Improves quality of initial product.

The preparation time line must be j

followed.  ;

- Little time to make changes.

j - Can become resource intensive.

j Site areparation visit very beneficial.

1

O CONCLUSIONS (Cont.)

Facility must ensure examinations are technically accurate.

Simulator scenarios must be validated.

Chief Examiner will make final determination on overall level of difficulty of examination package.

is d'

l-l; i

i KEY ISSUES FOR WORKSHOP l l l

Who is restricted from preparing the

! examination? '

i What reference material needs to be l provided the the Chief Examiner?

i j -

Can open reference material be

! used?

l l

How is the required level of difficulty j achieved on written questions? l l -

What is needed to maintain '

l examination security?

I Whic, K&A catalog should you use i and how ':o fit with Examiners Handbook?

J 16 e

PILOT SCHEDULE 4

Month Plant Region October Brunswick 11 C La Salle County 111 @

Palo Verde IV C November Limerick I Millstone-3 I

• i Pilgrim l l San Onofre IV @

December Brunswick ll @

l Fermi Ill

)

Fort Calhoun IV Vogtle ll January Cook Ill North Anna ll @

McGuire ll

Zion Ill February Ginna I Milstone-2 I @

Oconee 111 l Robinson II March Crystal River II April Braidwood lll C

,