ML20210H407
| ML20210H407 | |
| Person / Time | |
|---|---|
| Issue date: | 07/26/1999 |
| From: | Gallo R NRC (Affiliation Not Assigned) |
| To: | Jennifer Davis NUCLEAR ENERGY INSTITUTE (FORMERLY NUCLEAR MGMT & |
| References | |
| NUDOCS 9908030360 | |
| Download: ML20210H407 (9) | |
Text
q
)
-6 Mr. James Davis, Director, Operations July 26, 1999 Nuclear Energy Institute Suite 400
' 1776 Eye Street, NW Washington, D.C. 20006-3708 SUBJECT; INITIAL OPERATOR LICENSING EXAMINATION PERFORMANCE INDICATORS
Dear Mr. Davis:
Thank you for inviting the NRC to comment on I'1e initial operator licensing examination 4
performance indicators being developed by your operator licensing task force. We appreciate the industry's initiative in this area and believe that it may advance the quality and consistency of the initial operator licensing examinations.
We received the enclosed draft performance indicators via electronic mail from Mr. Larry Sanders of the Detroit Edison Company and are responding with this letter, a copy of which will be placed in the public document room. We have solicited comments from the NRC's regional operator licensing staffs. Our consolidated comments and suggestions, which are provided for your consideration and implementation as deemed appropriate, are annotated on the second enclosure. In general, we encourage you to collect as much data as is possible without creating an undue burden on the respondents and to ensure that the survey is unbiased.
Please call me on 301-415-1031 or Fred Guenther on 301-415-1056 if you have any questions.
1 Sincerely, Original signed by:
Robert M. Gallo, Chief Operator Licensing, Human Performance and Plant Support Branch Office of Nuclear Reactor Regulation
Enclosures:
As stated cc: Larry Sanders g g gg g}
f4 DISTRIBUT!ON:
File Center.
IOLB RF PUBLIC i
j
,._) ( b'3 7
BBoger/FGillespie 70437' i
- See previous concurrence FILE NAME: G:/lOHS/GUENTHER/NEl-SURVEY.WPD c. copy..tnout.ti.chrn.nti.ncio.u,.
e. copy wita.it.cnen.ntioncio.ur.
u. No copy OFFICE IOLB/DIPM lC IOLB/DIPM E
lOLB/DIPM L DIPM L
NAME F Guentherfg/rc
- DTrimble
- R Gallo
- B Boger
- DATE 06/25/99 -
06/30 /99 07/22/99 07/26/99 F / F,55 OFF!CIAL RECORD COPY r
N M
l.
MUo g
S UNITED STATES l,
y j
NUCLEAR REGULATORY COMMISSIOlJ t
WASHINGTON, D.C. 20066 0001
+,,,,,o July 26,1999 Mr. James Davis, Director, Operations Nuclear Energy Institute Suite 400 1776 Eye Street, NW Washington, D.C. 20006-3708
SUBJECT:
INITIAL OPERATOR LICENSING EXAMINATION PERFORMANCE INDICATORS
Dear Mr. Davis:
Thank you for inviting the NRC to comment on the initial operator licensing examination pe-formance indicators being developed by your operator licensing task force. We appreciate the industry's initiative in this area and believe that it may advance the quality and consistency of the initial operator licensing examinations.
We received the enclosed draft performance indicators via electronic mail from Mr. Larry l
Sanders of the Detroit Edison Company and are responding with this letter, a copy of which will be placed in the public document room. We have solicited comments from the NRC's regional operator licensing staffs. Our consolidated comments and suggestions, which are provided for your consideration and implementation as deemed appropriate, are annotated on the second enclosure, in general, we encourage you to collect as much data as is possible without creating an undue burden on the respondents and to ensure that the survey is unbiased.
Please call me on 301-415-1031 or Fred Guenther on 301-415-1056 if you have any questions.
Sincerely,
}
L Robert M. Gallo, ief Operator Licensing, Human Performance and Plant Support Branch Office of Nuclear Reactor Regulation
Enclosures:
As stated cc: Larry Sanders l
Il Siegf_ried Guenther_- lLO_ Ex~ m Process P rformance Indicators _
1 Pag 31 l j l
From:
<SANDERSL0dteenergy.com>
To:
OWFN_DO.owf2_po(SXG), GATED.nrcsmtp("porphanos @ peco...
Date:
Thu, Jun 17,199912:18 PM
Subject:
ILO Exam Process Performance indicators Gentlemen, Attached is a survey intended to solicit data from utilities concerning the initial license operator exam process. This survey was developed from our discussions conceming performance indicators during the most recent task force meeting. Please review the documents. Please provide feedback if the aurvey reflects the data required to track the performance for each phase of the initial license exam process. Once we finalize the data collection, I will then develop appropriate charts etc. Please respond by June 30,1999.
- Thanks, Larry Sanders Fermi-2 1-734-586-4121 Email: sandersl@dteenergy.com I
ENCLOSURE 1
[ Siegfried Guenther _- cxamsu-1. doc
' Psg a 1 l Date:
To:
Region (1,2,3, or 4 as appropriate) Ops Training Supervisors / Managers From:
NEIInitial License Exam Task Force
Subject:
Initial License Exam Survey i
Some of us may be aware, a task force has been established by NEI to work with the NRC regarding the initial license exam process. As part of this effort, the task force is establishing performance indicators to monitor both utility and NRC performance implementing the process.
To establish these indicators, the task force needs to establish base line deta. Therefore, we are asking for your help. We need each of the utilities to complete the attached survey for the most recent initial exam administered at their station. Please take a few minutes to complete the i
survey. Once completed, forward it to your training association representative listed below.
With the survey, please state the time frame you plan on administering your next initial exam.
Thank you 4
1
[Siegfried Guenther - rxamsu-1. doc PJge 2l Initial License Examination Performance Data:
l Utility:
NRC Region:
Plant:
Date Exam Administered:
- 1. Number of candidates which entered program:
Number of candidates which passed audit:
Number of candidates which took license exam:
Number of candidates which passed license exam.
- 2. Average score for NRC and audit written exams:
- 3. Exam development & Administrative Timeline:
Was the timeline as described in NUREG 1021 met for the following milestones:
j Yes
_No O
O 120 letter receipt O
O Exam outline submittal (> 75 days)
O O
NRC review of outline @ 5 days) l O
O Exam material submittal 8 45 days)
- 4. How many changes in NRC lead examiners did you experience during your exam process?
5.- Please estimate the amount of Full Time Equivalent (FTE) man hours required to complete the following:
Develop exam outline:
Develop written exam:
7 Validate written exam:
Average time to develop 1 JPM:
Average time to validate 1 JPM:
Average time to develop 1 Scenario:
Average time to validate 1 Scenario:
Suppon NRC dry run activity:
Incorporation of NRC changes:
Implementation of the exam Total Man Hours:
I i
a
l Siegfried Guenther - rx*msu-1. doc
_ Prg3 3l 6.
How many written exam questions were modified or removed by the NRC for the following criteria:
Questions did not comply with NUREG 1021 requirements:
NRC examiner preference:
Increase level of difficulty:
- 7. How many questions on your written exam did 2: 30% of the candidates answer incorrectly?
- 8. Please provide any information which you feel may help us understand the information you provided:
i i
l l
- 9. Please provide any other comments that you think the task force needs to be aware of
)
regarding your examination effort.
- 10. Thank you for taking the time to answer this survey. If you have any questions, please contact your training association member listed below:
MANTAG:
M'NTS:
SSNTA:
WESTRAN:
4 Initial License Examination Performance Data:
Utility:
NRC Region:
Plant:
Date Exam Administered:
- 1. Number of candidates which [that] entered program:
l Number of candidates which [that] passed audit:
Number of candidates which [that] took license exam:
Number of candidates which [that] passed license exam.
[We suggest that the data differentiate between the written exam and operating test.]
- 2. Average score for NRC and audit written exams:
[We suggest that the average score for the audit exam include only the grades for those candidates who went on to take the NRC exam. We also suggest that the focus group consider collecting data on the operating test performance such as which Categories were passed and failed, how many of the JPMs each candidate got correct, and which simu!ato: competencies resulted in failures.]
- 3. Exam development & Administrative Timeline:
Was the timeline as described in NUREG 1021 met for the following milestones:
Yes No O
O 120 letter receipt O
O Exam outline submittal (2:75 days)
O
.O NRC review of outline (s 5 days)
D..
O-Exam material submittal (s 45 days) [This should be 2.]
[The focus group should consider adding other milestones (from Form ES-201-1) covering, for example, NRC review of the proposed exam, facility incorporation of exam changes, and final exam approval. Also,if the survey is intended to be used for NRC-prepared exams as well, there should be a place to note who prepared lwhat parts 'of the exam and the milestone questions should cover both optiodi(e.g., did the licensee provide the reference material 90 days before the exam date?)]
4.
How many changes in NRC lead examiners did you experience during your exam process?
[The focus grouOhould consider asking respondents to explain whether thA changes had any significant impact on the exam development (e.g., changes to test items that the previous examiner had found acceptable).' The focus group should also consider related questions such as whether the facility exam author (s) had ever participated in an NRC exam before and whether the author (s) was (were) changed during the exam process.]
l m
i
c
- 5. Please estimate the amount of roll T;w Lg.. Ani (FTP,) nie. [ staff] hours required to complete the following:
Develop exam outline:
Develop written exam:
[Sub'mit reference material (for NRC-prepared exams):]
Validate written exam:
[Since there could b6iignificant variation in this area, we suggest that this be broken down into, for example, peer technical reviews, editorial myiews, and exam tryout by job incumbents.]
Average time to develop 1 JPM:
Average time to validate 1 JPM:
Average time to develop 1 Scenario:
)
Average time to validate 1 Scenario:
)
[The preceding fouritems should be multiplied by the number of JPMs and scenarios to come up with a total.]
Support NRC dry run activity:
Incorporation of NRC changes:
Implementation of the exam Total Man Hours:
- 6. How many written exam questions were modified or removed by the NRC [or facility licensee in the case of NRC-prepared exams) for the following criteria:
Questions did not comply with NUREG 1021 requirements:
NRC examiner preference:
Increase level of difficulty:
[Rather than these criteria, the focus group.hould consider using the criteria on Form ES-401-9 that NRC examiners will be using to review the exam. Changes should only be made to conform with the NUREG technical and psychometric criteria and proper English. We suggest not using the examiner preference term. Sometimes the examiner may view a change as required to meet the NUREG, but the exam autho_r may not agree and characterize the change as an examiner preference, which is not authorized. If a licensee believes that an examiner is making changes based on personal preference, it should discuss the issue with the examiner's branch chief rather than make the change and complain about it later. Therefore, if the " preference" term is retained, it should be defm' ed or restricted to ensure that it does not become a catch-all for every comment that the author does not agree with. We also believe that it may be useful to collect data on how many questions were taken from the exam bank, how many were modified, how many were new, and how many were repeated from'the last two NRC exams.]
- 7. How many questions on your written exam did 2 30% of the candidates answer incorrectly?
[What is the basis for choosing 30%?_ The focus group should als' consider collecting data regarding o
the number of questions that the licensee recommended deleting or changing and the number the NRC actually deleted or changed after the exam was given. This data might also be broken down with respect to whether the questions were revised during the pre-exam review process. Counting the number of questions that all the candidates got correct might provide insight regarding the exam's ability to discriminate.]
L
p I*
- 8. Please provide any information which you feel may help us understand the information you provided:
- 9. Please provide any other comments that you think the task force needs to be aware of regarding your examination effort.
- 10. 'Ihank you for taking the time to answer this survey. If you have any questions, please contact your training association member lis:ed below:
MANTAG:
MNTS:
SSNTA:
WESTRAN:
4 i
l
.