ML20136F272
| ML20136F272 | |
| Person / Time | |
|---|---|
| Site: | Turkey Point, 05520726  |
| Issue date: | 03/06/1997 |
| From: | Brian Hughes AFFILIATION NOT ASSIGNED |
| To: | |
| Shared Package | |
| ML20136F239 | List: |
| References | |
| 96-721-01-SP, 96-721-1-SP, SP, TAFF-970306, NUDOCS 9703140048 | |
| Download: ML20136F272 (27) | |
Text
_ _ _
I~
DOCKETED l
USNRC UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION W MAR 10 P4 :49 BEFORE THE ATOMIC SAFETY AND LICENSING BOARD uFFICE OF SECRETARY 00CKETING A SEf!VICF l
Before Administrative Judges:
gjwig;
~
Peter B. Bloch, Presiding Officer (Dr. Peter lam, Special Assistant) l i
4 In the Matter of
)
Docket No. 55-20726-SP l
)
RALPH L. TETRICK
)
)
ASLBP No. 96-721-01-SP 1
(Denial of Application for Senior
)
Reactor Operator License)
)
i j
SUPPLEMENTAL AFFIDAVIT OF BRIAN HUGHES
]
Brian Hughes, having first been duly sworn, does hereby state as follows:
1.
I am employed as a Reactor Engineer (Examiner Qualified), in the Operator Licensing Branch, Division of Reactor Controls and Human Factors, Office of l
Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission (NRC), in Washington, D.C.
A statement of my professional qualifications is attached to my Affidavit of January 23,1997, submitted as part of the " Written Presentation of NRC Staff "
4 1
2.
This Supplemental Affidavit is prepared in support of the Staff's motion j
for reconsideration of the Presiding Officer's Initial Decision of February 28,1997, in which the Presiding Officer " rounded up" the grade which he found had been achieved by Ralph L. Tetrick on a written examination administered on June 14,1996, from 79.6% to 80%. With that action, the Presiding Officer found that Mr. Tetrick had passed his written examination and should be awarded a Senior Reactor Operator (SRO) 1 4
9703140048 970310 PDR ADOCK 05000250 0
4 license for use at the Turkey Point Nuclear Generating Plant, Units 3 and 4 (" Turkey j
Point").
?'
3.
As the Presiding Officer noted in his Initial Decision, heretofore the parties have not directed him "to any governing guidance or regulation" concerning the question of whether "it is appropriate to round up the [ grade]... to the nearest integer" i
(Initial Decision, at 16). Such information is provided in this Supplemental Affidavit.
i 4.
As indicated in my Affidavit of January 23, 1997, the Turkey Point written examination administered on June 14,1996 to Mr. Tetrick informed applicants j
i that in order to pass the examination, they "must achieve a grade of 80% or greater" (Affidavit at 3-4; Hearing File Item 9, at 7). Upon reconsideration of Mr. Tetrick's i
I examination answers, the Staff determined that he had correctly answered 78 of 99 valid questions, for a score of 78.8% (or, more precisely, 78.79%), which was below the l
minimum passing grade of 80%; the Staff therefore concluded that Mr. Tetrick had failed the written examination (Affidavit at 4, 6).
1 5.
The Commission's responsibility for preparing, coordinating and grading the written examinations which are administered to candidates for Reactor Operatu and Senior Reactor Operator licenses is performed by the Operator Licensing Branch, Division of Reactor Controls and Human Factors, Office of Nuclear Reactor Regulation (in which I am employed), in conjunction with the NRC Regional Offices. In this capacity, the Staff has routinely reviewed the written examination grades achieved by RO and SRO license applicants, and has awarded such licenses to applicants who achieve a written examination grade of at least 80% and satisfy all other applicable requirements.
~
- i i
j 6.
Following the accident at Three Mile Island, Unit 2 ("TMI") in March 1
1979, the Commission specified that applicants for RO and SRO licenses must achieve
]
a score of at least 80% on their written examinations, thereby raising the minimum passing grade from the previously specified minimum grade of 70%. See letter from j
l Harold R. Denton to "All Power Reactor Applicants and Licensees," dated March 28, i
1980 (Enclosure 1 at 4) (Attachment I hereto). The requirement that applicants "must achieve a grade of 80 percent or greater" on the written examination was incorporated in NUREG-1021, " Operator Licensing Examiner Standards," cited by the Presiding Officer (Initial Decision, at 3). See NUREG-1021, Rev. 7 Supp.1, ES-402, page 5 of 6; Accord, Id., ES-401, page 6 of 7; ES-501, page 3 of 24 (Attachment 2 hereto).
7.
Each year, the NRC administers hundreds of written examinations to applicants for.RO and SRO licenses. The great majority (generally 90% or more) of '
these applicants pass their written examinations with a score of 80% or greater. For example, since 1990, the examination results for SRO applicants have been as follows:
Percentage of SRO Candidates Who Xsg Passed Their Written Examinations 1990 94 %
1991 97 %
1992 98 %
1993 97 %
1994 97 %
1995 93 %
1996 95 %
See SECY 97-033, " Annual Status Report on the Administration of NRC's Requalification Program and Initial Operator Licensing Examinations (WITS 8800098)
(Feb.11,1997) (Attachment 3 hereto), at 4.
l 8.
While most applicants pass their written examinations, those applicants who fail the examinations generally score above 75% - and often achieve grades within one or two points of the minimum passing grade of 80%. Thus, the difference between l
a successful and unsuccessful applicant often involves a difference of one or two incorrect i
i answers on the 100-question examination.
[
9.
Moreover, because the examination process permits applicants to 2
challenge the grading of their examination answers, where the Staff concludes that one
)
l l
or more questions should be deleted from the examination as invalid, fractional scores have resulted (reflecting the number of correct answers as a ratio of the total number of valid questions). In this manner, some applicants have been found to have achieved a j
l final grade of between 79 and 80 percent. Where this has occurred, the Staff's established practice is to deny their license applications for failing to achieve the minimum passing grade of 80% specified in NUREG-1021. Several examples of this established practice, among other such examples which could be cited, are attached hereto, in which the Staff denied the license applications of persons who achieved a final grade of 79.6%, 79.7%, and 79.8%, without rounding up to the next highest integer (see hereto).
10.
The Staff recently brought these matters to the attention of the Commission, in SECY-96-206, "Rulemaking Plan for Amendments to 10 CFR Part 55 to Change Licensed Operator Examination Requirements" (Sept. 25,1996) (Attachment 5 hereto), in which the Staff sought Commission approval of the Staff's plans for operator license rulemaking and of proposed Revision 8 to NUREG-1021. For example, the Staff
_ _ _._ _ ~._ _. _. _ _ _. _ _ _ _ _ _.- _ - _ _
. reported that in a six-month period from October 1,1995 to April 5,1996, a passing grade of at least 80% was achieved by 49 of 54 RO applicants, and by 86 of 92 SRO applicants, on a pilot examination. Further, the Staff noted that "[h]istorically, even those applicants who fail the NRC written examination generally score above 75 percent"
- and "[t]he average grade for the 11 applicants who failed [the] pilot examination was 76.3%." Id., Enclosure 2 at 27,29. With respect to fractional scores between 79 and 80 percent, the Staff stated as follows:
Whenever a grade is deleted from the 100-point written examination, it results in fractional grades. When the grade is above 79.5%, it raises a question regarding the staf's policy on rounding-of because it means the diference between receiving a license or a denial. To address this problem, the staff has revised NUREG-1021 to indicate that the passing grade on the written examination is 80.00 percent.
Id., Enclosure 2 at 24; emphasis added.
11.
In a Staff Requirements Memorandum (SRM) dated December 17,1996, the Commission approved implementation of SECY-96-206, including the proposed issuance of Revision 8 to NUREG-1021 on a voluntary basis, pending completion of the Staff's rulemaking efforts (see Attachment 6 hereto). In doing so, the Commission thereby approved the Staff's proposed clarification, consistent with the Staff's established policy, that the minimum 80% passing grade specified in NUREG-1021 is 80.00 percent.
12.
In accordance with the Commission's SRM, in January 1997 the Staff published Interim Revision 8 to NUREG-1021, which continues to specify a minimum passing grade (" cut score") of 80%, and clarifies that this 80% minimum passing grade requires a minimum grade of 80.00%. See NUREG-1021, Interim Rev. 8, ES-401,
O
- page 4 of 39 (80%), and page 39 of 39 (80.00%);Id., Appendix A, page 6 of 10 (80%),
j page 7 of 10 (80%); Id., Appendix E, page 1 of 5 (80.00%) (see Attachment 7 hereto).
13.
The Staffis aware that written examinations may sometimes be viewed as imprecise.
Nonetheless, the Commission's guidance for preparing written l
examinations helps to assure that the examinations are of good quality; and the.
Commission's post-examination review process, which is available to applicants who are 4
N l
concerned over the grading of their examinations, helps to assure that their concerns are considered fairly and in depth. Further, in accordance with NUREG-1021, the Staff i,
reviews the grading in detail where an applicant scores between 78 and 82 percent (see, e.g., NUREG-1021, Rev. 7, Supp.1, ES-403, page 3 of 5, page 5 of 5 (Attachment 2);
2 NUREG-1021, Rev. 8, ES-403, page 4 of 5, page 5 of 5 (Attachment 7)). This was done in the case of the Turkey Point written examination administered to Mr. Tetrick, a
as well (Hearing File Item 8). Accordingly, the denial of applications for persons, like Mr. Tetrick, who fail to achieve the minimum score of 80% specified in NUREG-1021, is neither unfair nor inappropriate.
14.
Based upon the above, the Staff has concluded that Mr. Tetrick's final I
grade of 79.6%, as modified in the Presiding Officer's Initial Decision, remains below j
the minimum passing grade of 80%. Accordingly, the Applicant has failed the written examination, and the Staff's denial of his application for a SRO license should therefore be sustained.
i e
4
l
=
4 15.
I hereby certify that the foregoing is true and correct to the best of my knowledge, information and belief.
l l
Ai'
)1 Brian'HQh7s Subscribed and sworn to before me this 6th day of March,1997.
.4 Y lAf Q
koilry Public~U Myamminwn L Y
NOTARY FGUC S A.TE OF MARYLAND My Commis: ion Expires
Deardar 1,
1999 i
l a
[
l l
1 l
6 d
4 a---
+,4 4
h.a-A
--4
--A O
e I
a R
ATTACBIENT 1 J
J e
4 1
I 1
. ~. -. _ _ -. -
- -. _ - ~. -.
. _ _.. - - ~.. -.
l g
UNITED STATES i
8 NUCLEAR HEGULATORY COMMIS$10N L
WASHINGTON.D.C.30tes l
M !81380 i
i.
All POWER RFACitR APPLICANIS AND LICENSES i
Centlemen:
i j
SUIRECT: QUALIFICATIONS OF REACItR OPERATORS i
In a letter' dated September 13, 1979, we inforned you of NRR requirenants 1
established as of that date based on our review of the 1MI-2 accident.
} to the letter outlined the staff reccomendations concerning inprovements in the area of operator training for your information.
l Slnce that time, the Ccamission has acted on the staff rec-adations.
i j
It is the purpose of this letter to set forth the revised criteria to be j
used by the staff in evaluating reactor operator training and licensing, that can be inplemented under the current regulations and to establish an i
effective date for their inplementation.
Other criteria that will be established require additional staff work are also addressed. However, i
inplemntation dates cannot be provided at this tina. Conmission review i
in the area of operator training and qualification is continuing and can be expected to result in additional criteria. Finally, requirements will l
be established through rule asking proceedings.
! details the revised criteria and the effective date for their i
inplementation.
Your attention is specifically directed to Sections A,
B and C of Enclosure I since these call out new criteria that will be
{
inplenented in the near futures therefore, your plans regarding training and licensing activities should be prceptly revised to conform to these l
criteria.
Enclosures 2 and 3 provide guidance for establishing training progrars in heat transfer, fluid flow and thernodynamics: and mItipting core damage.
, details control manipulations for requalLfication programs.
j l
Based on our understanding of the industry's reasons for establishing the Institute of Naclear Power Operations and our review of the latest revisions to applicable ANSI standards, we believe you share our desire l
to significantly upgrade the requirements for operations personnel.
i 8004150 3N
!s v
j.
a
?.
y2 4
.e
E ?.;..
2-
\\X:
%erefore, we ur you to effect the knplementation of the criteria as O
soon as practica le rather than wait for the stated inplementation date. Also, we urge you to start plaming for the long range require-rents so that they may be rapidly implemented upon coepletion of the rule making procedure.
ad/ 0 Harold R. Denton, Director Office of Nuclear Reactor Regulation Enclosures.
1.
Requirements for Reactor Operator Training and Licensing 2.
Training in Heat Transfer, R uid M ow and %errodynamics 3.
Training Criteria For Mitigating Core Damage 4.
Control Manipulations e
f t
O e
S
..a :;.y
. ;p. fiy:
~-
. j, y.
i..
T b
ENCIASURE 1
.v' l
CRI7ERIA POR REACIDR OP!RA1DR 7 RAINING AND LICENSING i
i j
A.
Eligibility Requironents to be Administered an Examination.
i 1.
Experienee*
s.
Applicants for* senior operator licenses shall have 4
{
years of responsible power plant experience. Responsible I
power plant experience shculd be that obtained as a I
control rocan operator (fossil or nuclear) or as a 3
l power plant staff engineer involved in the day-to-day
)
activities of the facility, comencing with the final 4
i I
year of construction.
A maximum of 2 years power l
plant experience may be fulfilled by academic or j
\\
j related technical training, on a one-for-one time I
e basis. 1 % years shall be nuclear power plant experience.
i j
At least 6 nonths of the nuclear power plant experience i
shall be at the plant for sich he seeks a license.
w l
l Effective date:
Applications received on or after I
l May 1, 1980.
)
i
}
I i
~
j
- Precritical applicants will be required to 'neet mique qualifications j-designed to acecxmodate the fact that their facility has not yet been in l
operation, j
i
,.a 1
..s
,,y..
b.
Applicants fer senter operater licenns shall have held an operator's license for 1 year.
OI i
i Effective date:
Applications received after Decenber 1, l
1980.
1 2.
Trainina i
m.
Senior operator *: Applicants shall have 3 renths of i
i shift training as an extra man on shift.
f l
'b.
Control room operator *:
Applicants shall have 3 nonths training on shift as an extra person in the control roan.
i i
Effective date:
Applicatioris received after August l
1, 1980.
1 i.
c.
Traini.ng programs shall be codified, as necessary; to l
provide:
l 1)
Training in heat transfer, fluid flow and thermo-l l
dynamics.
2)
Training in the use of installed plant systems i
to control or mitigate an accident in sich the t
core is severely damaged.
I 3)
Increased er:phasis on reactor and plant transients.
l
- Precritical applicants will be required to meet unique qualifications j
designed to accamodate the fact that their facility has not yet been in i
1 i
Awv.s:m
.e #
y 3
Effective date Present prograns have been nodified g4 in response to Bulletins and Orders. Revised programs sho(ild be submitted for OLB review by August 1,1980.
d.
Training center and facility instructors who teach systems, integrated respones transient and sinulator courses shall denenstrate their cccpetence to NRC by-successful conpletion of a senior operator examination.
Effective dates Applications should be submitted no later than August 1,1980 for individuals de do not aircady hold a tenior operator license.
k e.
Instructors shall be enrolled in appropriate requal-ification programs to assure they are cognir. ant of current operating history, proble::n, and changes to procedures and administrative limitations.
Effective date:
Programs should be initiated May 1, 1980. Programs should be stanitted to OLB for review by August 1, 1980.
3.
Facility Certifications Certifications coupleted pursuant to Sections 55.10(a)(6) and 55.33a(4) and (5) of 10 CFR Part 55 shall be signed by the highest level of corporate management for plant operation 3
(for exa::ple, Vice President for Operations).
I 3.-
W mW
y,,.._____
=
=
y, s
.' t
\\
I
- l Effective date Applications received on or after May 1,
.g. ;.
O 1980.
it B.
NRC Examinations 3
1 1.
Increased Scope of Examinations j
a.
A new category shall be added to the operator witten h.,
. examination entitled, " Principles of Heat Transfer and
.(
Fluid Mechanics."
i b.
A new category shall be added to the senior operator witten examination entitled, "Iheory of Fluids and
'Ihentodynmics."
c.
Tine limits shall be inposed for conpletion of the written examinations:
j l
1.
Operator:
9 hours1.041667e-4 days <br />0.0025 hours <br />1.488095e-5 weeks <br />3.4245e-6 months <br />.
s 2.
Senior Operator: 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />.
d.
'the passing grade for the witten examination shall be 80%
l overall,and 70% in each category.
e.
All applicants for senior operator licenses shall be -
required to be administered an operating test as well as l
the written examination.
f.
Applicants will grant permission to NRC to inform their facility managenent regarding the results of the exm-instions for purposes of enrollment in requalification
- j.,
- Prograns. ?
i
.[
{t.
Effective date: Examinations administered on or after May t..
1,1980 for items a. through e.
Applications received on sf or after May 1,1980 for Itan f.
4
<~.
.g, :
i.
l i
/
C.
Requalification Progres l
1.
Content of the licensed operator requalification progres shall be, modified to include instruction in heat transfer, fluid j
flow, thernodynamics and mLtigation of accidents involving a l
degraded core.
i Effective'date: May 1, 1980.
l j
2.
'Ihe criteria for requiring a licensed individual to participate
~
in accelerated requalification shall be nodified to be consistent with the new passing grade for issuance of a licenses 80%
l overall and 70% each category.
t
{
Effective dates Concurrent with the next facility administered annual requalification examination after the issue date of this j
j letter.
t 1
3.
Programs should be modified to require the control nanipulations listed in Enclosure 4.
Normal control manipulations, such as j
l plant or reactor startups, nust be performed.
Control aan-ipulations during abnormal or emergency operations must be walked through with, and evaluated by, a number of the' training staff at a mf.ninurn.
An appropriate sinulator may be used to satisfy the requirements for control manipulations.
Effective date: Programs nodified by August 1, 1980. Renewal applications received after Novenber 1, 1980 cust reflect conpliance with the prograrn.
i
. ', i.
D.
Ione Range Criteria and/or Requiremnts
'lhe following require additional staff work and/or nilemaking prior to their inplemen'tation.
1.
Qualifications a.
Shift supervisors shall have an engineering degree or equivalent qualifications.
b.
Senior operators shall have successfully ca:pleted a course in appropriate engineering and scientific st6 ject equal to 60 credit hours of college level subjects.
2.
Training All applicants shall attend siplator training programs.'
a.
Required control manipulations and exercises to be s
performed shall be the same for " cold" and ' hot" applicants.
4 b.
Eligibility requirements shall be developed for instructors, in addition to that listed in A.2 above.
3.
NRC Examinations a.
NRC shall administer the certification examinations that are presently administered at the conclusion of the off-site portion of the cold training programs.
b.
All applicants shall be required to be administered a simulator stamination in addition to the written exam-inations and plant oral tests.
i c.
NRC shall administer the requalification program annual examination.
s
.s
,,7 j
4 Requalification Programs All licensees shall' participate in sinulator programs as part of the ;
requalification progr as. Control manipulations shall be perfomad pursamme m sh 4 - 4.
4 O
9 e
o e
4 e
ammame e
4 J
l l
ATTACmfEVT 2 I
4 t
Rev.7,Supp.1 Operator Licensing Examiner Standards Manuscript Completed: June 1994 Date Published: June 1994 Division of Reactor Controls and Human Factors Omce of Nuclear Reactor Regulation U.S. Nuclear Regulatory Commission Washington, DC 20555-0001
l i
)
ABSTRACT 4
1 The Operator Licensing Examiner Standards provide policy and guidance to NRC examiners and establish the procedures and practices for examining licensees and applicants for reactor operator ano senior reactor operator licenses at power reactor facilities pursuant to Part 55 of Title 10 of the Code of Federal Reaulations (10 CFR 55). The Examiner Standards are intended to assist NRC examiners and facility licensees to better understand the initial and requalification examination processes and to ensure the equitable and consistent administration of examinations to all applicants. These standards i
are not a substitute for the operator licensing regulations and are subject to revision or other internal operator licensing policy changes.
Revision 7 was published in January 1993 and became effective in August 1993.
Supplement 1 is being issued primarily to implement administrative changes to 4
j the requalification examination program resulting from the amendment to 10 CFR 55 that eliminated the requirement for every licensed operator to pass an NRC-conducted requalification examination as a condition for license renewal. The supplement does not substantially alter either the initial or requalification examination processes and will become effective 30 days after its publication is noticed in the Federal Reaister. The corporate notification letters issued 4
after the effective date will provide facility licensees with at least 90 days notice that the examinations will be administered in accordance with the 1
l 4
revised procedures.
A vertical line in the right namin indicates that material has been added or changed; a vertical line in the left margin indicates that material has been deleted.
4 i
e 4
1 T
i t
Examiner Standards iii Rev. 7, Sup. 1, June 1994
l i
i ES-401 j
PREPARATION OF SITE-SPECIFIC WRITTEN EXANINATIONS FOR POWER REACTORS
~
I A.
PURPOSE This standard specifies the requirements, procedures, and guidelines for the l
preparation of site-specific written examinations for the initial licensing of reactor operator (R0) and senior reactor operator (SR0) applicants at power reactor facilities.
)
B.
BACKGROUND I
The content of the written licensing examinations for R0s and SRos is dictated l
by Sections 55.41 and 55.43, respectively, of Title 10 of the Code of Federal Each examilation shall contain a representative Regulations (10 CFR).
selection of questions on the kncwledge and abilities (K/As) needed to perform duties at the desired licenae level.
The written operator licensing examination is adminir,tered in two sections, a i
generic fundamentals examination (GFE) section and 6 site-specific l
The GFE covers those K/As that do no'. vary significantly among examination.
reactors of the same type (i.e., pressurized or boiling water) and is I
generally administered early in the license training process; refer to ES-205 for a description of the program.
i i
C.
EXAMINATION PREPARATION i
Examiners shall prepare site-specific written operator licensing examinations in accordance with NUREG/BR-0122, " Examiners' Handbook for Developing Operator Licensing Written Examinations," and the guidelines and instructions sunnarized below. The Examiners' Handbook provides specific instructions for preparing content-valid examinations and guidance on writing and formatting The guidelines and various types of written examination questions.
instructions contained herein take precedence over conflicting guidelines in j
the Examiners' Handbook.
i J
1.
General Guidelines 4
The examiner shall prepare the examination so that an applicant l
a.
who is capable of safely operating the plant will be able to j
complete and review the examination within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> and achieve a i
l grade of 80 percent or greater.
When both R0 and SRO examinations are to be given at approximately b.
l the same time, the examiner shall duplicate no more than 75 l
percent of the R0 examination questions on the SRO examination.
j The remaining SR0 questions shall evaluate the additional l
knowledge and abilities required for the higher license level.
The Examiners' Handbook and 10 CFR 55.43(b) provide guidance on 4
l Examiner Standards I of 7 Rev. 7, January 1993 I
4 I
1
~
ES-401 Site-specific Written Examination Form ES-401-1 Cover Sheet U. S. NUCLEAR REGULATORY COMMISSION SITE-SPECIFIC j
WRITTEN EXAMINATION APPLICANT INFORMATION Name:
Region:
I / II / III / IV / V i
Date:
Facility / Unit:
I License Level:
R0 / SR0 Reactor Type:
INSTRUCTIONS j
Use the answer sheets provided to document your answers. Staple this cover sheet on top of the answer sheets.
Points for each question are indicated i
in parentheses after the question. The passing gran requires a final grade of at least 80 percent.
Examination papers will be picked up 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> after the examination starts.
All work done on this examination is my own.
I have neither given nor j
received aid.
Applicant's Signature RESULTS Examination Value Points l
Applicant's Score Points Applicant's Grade Percent 4
4
i 1
! g
^
ES-402 l
ADMINISTERING WRITTEN EXAMINATIONS AT POWER REACTORS 4
l A.
PURPOSE j
This standard specifies the requirements and procedures for administering 1
written examinations at power reactors, including proctoring and facility i
staff review of the examination.
B.
EXAMINATION FACILITIES j
1.
The licensee is responsible for providing facilities suitable for administering the written examination. The examination room and i
associated restroom facilities shall enable the NRC examiners to v
maintain the security and integrity of the written examination.
1, " Guidelines for Administering Written Examinations,"
sumaarizes the NRC's policies regarding written examination facilities and other preparations for administering the written examinations.
2.
An examiner shall evaluate the licensee's examination facilities and other arrangements made in accordance with Attachment 1 to ensure their adequacy. The applicants shall not have access to any reference l
material that was not provided or approved by the chief examiner. The chief examiner shall not begin the examination until he or she is j
satisfied with the arrangements.
C.
PROCTORING THE EXAMINATIONS 1.
The chief examiner shall ensure that the applicants are proctored at all i
times while taking the written examination. The chief examiner shall ensure that the proctor clearly uriderstands his or her responsibilities i
before the examinations are distributed.
The chief examiner should consider the following means to ensure adequate proctoring:
using NRC secretarial help using more than one examiner using resident inspectors i
using other NRC employees The proctor shall give full attention to the applicants taking the examination. The proctor shall not read procedures or other material, grade examinations, or engage in any other activities in a manner that may divert his or her attention from the applicants and possibly cause the examination to be compromised.
2.
At least one examiner shall be available to clarify examination j
questions for the applicants during the examination.
If the examiner who wrote the examination is not available, then the other examiners 3
j Examiner Standards 1 of 6 Rev. 7, January 1993 4
)'
i s
ES-402 Policies and Guidelines 4 \\
for Taking NRC Written Examinations 1.
Cheating on the examination will result in a denial of your application and could result in more severe penalties.
l 2.
After you complete the examination, sign the statement on the cover j
sheet indicating that the work is your own and you have not received or i
given assistance in completing the examination.
i 3.
To pass the examination, you must achieve a grade of 80 percent or j
greater.
.2 4.
The point value for each question is indicated in parentheses after the i
question number.
i 5.
There is a time limit of 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> for completing the examination.
6.
Use only black ink or dark pencil to ensure legible copies.
3 7.
Print your name in the blank provided on the examination cover sheet and l
the answer sheet.
8.
Mark your answers on the answer sheet provided and do not leave any question blank.
1 9.
If the intent of a question is unclear, ask questions of the examiner 4
i only.
10.
Restroom trips are permitted, but only one applicant at a time will be allowed to leave. Avoid all contact with anyone outside the examination j
room to eliminate even the appearance or possibility of cheating.
i 11.
!! hen you complete the examination, assemble a package inc'uding the exarair.ation questions, examination aids, and answer sheets and give it l
to the examiner or proctor.
Remember to sign the statement on the i
examination cover sheet.
12.
After you have turned in your examination, leave the examination area as defined by the examiner.
i 1
Examiner Standards 5 of 6 Rev. 7, January 1993 i
i
./
r-4 e-
i
)
ES-403 GRADING SITE-SPECIFIC WRITTEN EXAMINATIONS AT POWER REACTORS l
A.
PURPOSE 4
i This standard explains the requirements and procedures for resolving the j
facility,11censee's comments on the site-specific written examinations,
{
grading the examinations, and conducting a quality assurance (QA) review of the graded written examinations.
4 8.
RESOLVING FACILITY COMENTS i
1.
The chief examiner shall ensure that a copy of the examination as it was i
administered, with all changes incorporated, is placed in the master examination file. The pen-and-ink corrections made for the applicants while the examination was administered may be changed to typewritten i
l corrections, however, all changes shall be annotated in such a way that they are evident.
)
2.
The chief examiner shall ensure that the master examination is updated to reflect any changes made to the examination and answer key in response to the facility licensee's comments. The chief examiner shall analyze each facility comment and document the reason that the question was changed or the reason that the comment was not accepted. The chief examiner will not change the examination unless the facility licensee submits a valid reference to support its comment. Comments without a i
facility recommendation shall not be accepted but should be noted for reference on future examinations.
3.
If the chief examiner determines that there are two correct answers for a multiple choice question, both answers will be accepted as correct.
l However, if three or more answers could be considered correct, the j
question shall be deleted.
4.
If 10 percent or more of the examination questions are deleted as a l
result of facility comments, the remaining examination must be evaluated to ensure that the test outline requirements in the Examiners' Handbook (NUREG/BR-0122) are still satisfied.
If the content validity of the examination is affected (e.g., several knowledge and ability (K/A) topics are not covered, or the majority of K/As remaining are associated l
with a small number of systems) as a result of deleting questions, the examination may have to be voided. The training and assessment l
specialist on the Operator Licensing Branch staff should be consulted if the validity of the examination is in question.
C.
GRADING THE EXAMINATIONS i
i 1.
The chief examiner will ensure that the examinations are graded I
expeditiously. However, the written examination grades shall not be l
finalized until the facility licensee's comments have been resolved.
f Examiner Standards 1 of 5 Rev.7, January 1993 j
.]
se f'
~.
y
,,w
l i
l ES-403 i
l 2.
The examiner who wrote the examination should normally grade the i
examination. However, if the author of the examination is not available, the number of applicants is unusually large, or the region i
wishes to expedite the grading process, the examination may be graded by l
another examiner.
3.
The grader shall indicate in red pen or pencil the number of points i
given to or deducted for each answer on the applicant's answer sheet.,
4.
If it is necessary to change a grade during the grading process, the i
grader or reviewer shall do it by lining out the original grade in such a way that it remains legible, briefly explaining the reason for the change on the applicant's answer sheet, and initialing the change.
l Under no circumstances will whiteout or other methods that obscure the change be used.
5.
After all the questions are graded, the examiner shall enter the " Test i
Value," " Applicant's Score," and " Applicant's Grade" in the "Results" section of the applicant's written examination cover sheet.
i 6.
After grading the last examination, the examiner shall review the grading in detail and complete Form ES-403-1, " Examination Grading
}
Quality Assurance Checkoff Sheet."
7.
If the operating test results are not available at the same time as the written examination results, the examiner shall fill in the " Written Examination Summary" on page 1 of Form ES-303-1, check the written i
examination " Pass" or " Fail" block in the " Examiner Recommendations" l
section of the form, and sign and date the form as " Grader." The j
examiner should then forward the written examination package, including the master written examination (s) and answer key (s), the applicants' i
examinations, and all associated forms (ES-303-1, ES-401-3, and ES-403-l
- 1) to the designated reviewer for a quality assurance check in j
accordance with Section D.
D.
QUALITY ASSURANCE (QA) REVIEW OF THE GRADED EXAMINATIONS 1
1.
If the written examinations were graded by a contract examiner, the contract manager shall confirm that the quality of the grading was i
checked in detail and sign the bottom of Form ES-403-1 before sending the examinations to the regional office.
2.
The section chief shall ensure that the grading of the written examinations is independently reviewed by a regional examiner. The chief examiner may perform this review in conjunction with his or her normal review specified in Section D.6.
Examiner Standards 2 of 5 Rev.7, January 1993
\\
I, ES-403 i
I 3.
The QA reviews are spot checks, or sampling tests, to' follow up on the detailed review performed by the grading examiner.
If the QA reviews l
indicate significant problems, additional detailed review will be necessary.
l 4.
The reviewer shall use Form ES-403-1 to document the review. At a minimum, the reviewer shall:
Check at least 25 percent of the examinations for math errors in a.
determining the final grade.
b.
Review and recalculate the grades for all borderline cases (i.e.,
80 percent +/- 2 percent overall).
Check all the failing examinations to ensure that the failures are c.
justified. The reviewer shall pay particular attention to the highest failing and the lowest passing examinations to ensure that the examination effectively discriminated between applicants performing above and below the minimum acceptable level of competence.
d.
Review the overall performance on each question to determine if l
there are problems in the facility's training program, in the wording of the questions, or in other areas.
The reviewer should discuss all grading discrepancies with the e.
grader before making any changes. The reviewer shall document all grade changes by carefully lining out the original grade so that it remains legible, entering the revised grade with a brief explanation, and initia111ng the change. Whiteout or other methods that obscure the original grade shall not be used.
f.
When the reviewer is satisfied with the quality of the grading, he or she shall initial and date the items on Form ES-403-1 and sign and date the bottom of the form.
5.
When the regional QA reviewer has completed the review, he or she shall forward the examination package to the chief examiner.
Examiner Standards 3 of 5 Rev.7, January 1993
j ES-403 1
6.
The chief examiner shall review Form ES-403-1 for completeness and review borderline cases in detail to determine if any generic weaknesses exist so that they may be addressed in the examination report. When the chief examiner is satisfied with the examination grading, he or she shall sign and date the form.
i i
The chief examiner shall also ensure that the applicable sections of 1
Form ES-303-1 are properly completed and assemble the final examination 4
package for management review in accordance with ES-501.
If the chief examiner performed the regional QA review, he or she may not perform the 2
management review for the section chief.
ATTACHMENTS / FORMS:
Form ES-403-1,
" Examination Grading Quality Assurance Checkoff Sheet" i
i k
i e
4 i
Examiner Standards 4 of 5 Rev.7, January 1993 d*
3
\\
~
(-
ES-403 Examination Grading Form ES-403-1 Quality Assurance Checkoff Sheet i
Grader (s) Name:
Date of Examination:
l Facility /U~it:
Examination Level:
I Initial & Date item Description Grader Regional Reviewer t
1.
Applicants' scores checked for addition (reviewer check 25% of examinations)
)
2.
Grading for all borderline cases reviewed (80% +/- 2%)
3.
Highest failing and lowest passing i
examinations compared to justify decision i
4.
All other failing examinations checked to ensure grades are justified i
5.
Performance on each question checked for training deficiencies and wording problems i
Signatures Date i
i Grader i
l Contract Manager Regional Reviewer (*)
i Chief Examiner (*)
l Section Chief (*)
(*)
If the chief examiner does the regional review, he or she may not sign i
as section chief.
4 Examiner Standards 5 of 5 Rev.7, January 1993 s
i
/
1
i i
i ES-501 j
i POSTEXAMINATION ACTIVITIES 1
l A.
PURPOSE l
This standard describes the procedures for assembling and reviewing the 1
examination package, notifying the facility and applicants of the examination l
results, preparing the examination report, and evaluating contractors.
\\
B.
BACKGROUND The Operator Licensing Branch's goal is to complete licensing or denial actions within 30 days after the facility licensee's formal written i
examination comments are resolved. The staff should establish its priorities j
and schedules to achieve this goal.
Examiners may use Form ES-501-1, "Postexamination Check Sheet," to track l
completion of the administrative items after the examination is administered.
C.
EXAMINATION REVIEWS 1.
The examiners who participated in the examination assignment shall grade and review the written examinations and cperating tests that they administered in accordance with ES-403 and ES-303, respectively, and forward the examinations and associated documentation to the chief examiner.
2.
The chief examiner or designee shall review the written examination grading to ensure that it meets the requirements of ES-403, then sign and date Form ES-403-1.
The chief examiner or designee shall also review the examiners' operating test documentation to ensure that the test and its grading meet the requirements in ES-301 and ES-303. The reviewer shall ensure that the examiner's operating test comments support the pass or fail recommendation and check for consistency in documentation and grading among the operators tested on the same simulator crew.
If the reviewer is satisfied with the accuracy and completeness of the documentation and concurs with the examiner's reconnendations, he or she shall sign and date the " Final Recommendation" block on Form ES-303-1.
If the reviewer does not agree with the examiner's reconmendation, the reviewer shall discuss the examination findings with the examiner and resolve the disagreement.
If the chief examiner administered the operating test, the section chief should designate another examiner to independently review the documentation and sign the " Final Recommendation" block on page 1 of Form ES-303-1.
l Examiner Standards 1 of 24 Rev. 7, January 1993
i i
i ES-501 l
The applicant must have a grade of 80 percent or greater on the written examination and a grade of " satisfactory" on all three categories of the operating test to qualify for a license.
l 3.
If the management reviewers (i.e., the section chief or licensing do not agree with the final recommendation, the examiner of official)d chief examiner shall be consulted to discuss and resolve any l
record an j
disagreements.
Such disagreements usually arise because of inadequate i
documentation of a denial recommendation.
It is, therefore, very l
important for examiners to be complete and accurate in their grading and j.
comments.
4.
If a recommendation is overturned during the regional management review, the affected summary evaluations will be lined out and initialed by the licensing official. The licensing official will enter the new summary evaluation in the appropriate block and explain the change on Form ES-303-2, " Operating Test Comments," and attachment it to the applicant's Form 303-1.
5.
If an apslicant did not complete the SR0 upgrade training program or failed tte upgrade examination, regional management should ensure that the R0 licensee complies with the requirements of 10 CFR 55.53(e), (f),
and 10 CFR 55.59(a) before resuming active duties as an R0. The and (h)l office should also conduct a case-specific review of the SRO regiona upgrade examination to detemine if the applicant failed as a result of significant deficiencies in R0 knowledge or abilities.
Pursuant to 10 CFR 55.7, the NRC may, by rule, regulation, or order, innose upon any licensee such additional requirements it deems appropriate or necessary to protect the health and to minimize danger to life and property.
If the SR0 upgrade applicant's deficiencies pose such a threat, the NRC may require the facility licensee to provide remedial training and reevaluation and submit evidence of its completion to the NRC.
6.
Regional management should also review the overall examination results and any generic findings, deficiencies, or issues identified in the examination report to determine if any follow-up action is required.
I E.
NOTIFICATION OF RESULTS The regions will notify the facility licensees and applicants of the examination results only after they are reviewed and approved by the licensing official.
1.
The region should normally notify a representative of the facility licensee by telephone and confirm the results by mailing, under a separate cover letter, a copy of Form ES-501-2 and, for each applicant that failed or had significant deficiencies that warrant further evaluation and retraining by the facility licensee, a copy of the applicant's Form ES-303-1 and written examination answer sheet.
Examiner Standards 3 of 24 Rev. 7, Sup. 1, June 1994
4 4
4 ArrACIBENT 3
f l
t 7,
e c<evy l
2 5
l
\\...../
l POLICY ISSUE j
(Information) i i
l I
I February 11, 1997 SECY-97-033 i
EDB:
The Comnissioners fBQM:
Hugh L. Thompson, Jr.
Acting Executive Director for Operations i
SUBJECT:
ANNUAL STATUS REPORT ON THE ADMINISTRATION OF NRC'S REQUALIFICATION PROGRAM AND INITIAL OPERATOR LICENSING
^
i EXAMINATIONS (WITS 8800098)
PURPOSE:
To inform the Comission of the status of the Nuclear Regulatory Commission's (NRC's) licensed operator requalification program and the results of NRC's initial licensing examinations for reactor operator (RO) and senior reactor operator (SRO) applicants.
8ACKGROUND:
The staff issued its first periodic report on the status of NRC's licensed operator requalification program on August 28, 1989, and continued to issue CONTACT:
SECY NOTE:
TO BE MADE PUBLICLY Stuart A. Richards, NRR AVAILABLE IN 5 WORKING DAYS FROM 415-1031 THE DATE OF THIS PAPER
' quarterly or semi-annual status reports through the end of calendar year 199.1.
l In a staff requirements memorandum dated January 8, 1992, the Commission i
directed the staff to henceforth submit an annual report and combine the results of NRC's initial R0 and SRO licensing examinations and requalification examinations. On February 25, 1992, the staff issued its first combined report, SECY-92-066, " Semi-Annual Status Report on the NRC's Requalification Program and Initial Licensed Operator Examinations." On February 5, 1993, the 4
i staff issued its first combined annual report, SECY-93-027, " Annual Status Report on the Administration of the NRC's Requalification Program and Initial i
l Operator Licensing Examinations." On February 18, 1994, the staff issued its 1
annual report, SECY-94-039, " Annual Status Report on the Administration of the NRC's Requalification Program and Initial Operator Licensing Examinations," and discussed its new requalification program inspection process in support of an amendment to Part 55 of Title 10 of the Code of Federal Reaulations (10 CFR) as described in SECY-93-333, " Final Amendments to j
10 CFR Part 55 on Renewal of Licenses and Requalification Requirements for Licensed Operators." On February 17, 1995, the staff issued its annual i.
report, SECY-95-042, " Annual Status Report on the Administration of the NRC's Requalification Program and Initial Operator Licensing Examinations," in which it discussed the results of the elimination of the 10 CFR Part 55 requirement i
tnat each licensed operator pass a comprehensive requalification written examination and an operating test conducted by the NRC during the term of each
]
operator's 6-year license.
On March 24, 1995, the staff issued SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program," in which it stated its intent to revise the manner in which the NRC administers the initial operator licensing program to allow greater participation by facility licensees, to eliminate contractor assistance in this area, and to adjust the degree of NRC involvement in facility requalification examinations. On February 6, 1996, the staff issued its annual report, SECY-96-026, " Annual Status Report on the Administration of the NRC's Requalification Program and Initial Operator Licensing Examinations," in which it discussed the status of ongoing changes to the operator licensing process addressed in SECY-95-075. On Jane 10, 1996, the staff issued SECY-96-123, " Proposed Changes to the NRC Operator Licensing Program," in which it informed the Commission of the results of a pilot program under which facility licensees were permitted, in accordance with the pilot examination guidance in Generic Letter 95-06, " Changes in the Operator Licensing Program," to draft the written examinations and operating tests that the NRC administers to detemine the competence of operator license applicants at nuclear power reactor facilities, and requested the Commission's approval to implement this new examination process on a voluntary basis, while in parallel pursuing mandatory implementation by the industry.
On September 25, 1996, the staff issued SECY-96-206, "Rulemaking Plan for Amendments to 10 CFR Part 55 To Change Licensed Operator Examination Requirements," in which it provided additional information regarding the initial operator licensing examination pilot process and requested the Commission's approval of the rulemaking plan to amend 10 CFR Part 55 to require power reactor facility licensees to prepare the entire initial
i
_3 i
examination.
In a Staff Requirements Memoranduni (SRM) dated December 17, 1996, the Commission approved the staff's rulemaking plan (SECY-96-206) to establish new procedures for the preparation and administration of initial t
examinations for power reactor operators. The Counission also approved implementation of Revision 8 of NUREG-1021, " Operator Licensing Examination Standard for Power Reactors," on a voluntary basis until the rulemaking is complete. The Commission emphasized the importance of the NRC Chief Examiner role in ensuring that examinations developed by facilities meet NRC expectations regarding both format and level of difficulty, and cautioned the 4
staff to budget adequate resources to carry out the task. This change is part l
of NRC's continuing efforts to be consistent with the Administration's initiatives to streamline the functions of the Federal Government and to accommodate anticipated resource reductions, including the eventual l
elimination of contractor support in the operator licensing area.
DISCUSSION:
i NRC Reaualification Examination and Insoection Su==ary for Fiscal Year 1996 4
I During fiscal year 1996 (FY 96) the staff continued to monitor the j
requalification examination process.
It evaluated the licensed operator requalification programs at 41 power reactor facilities during FY 96 to verify a
that the programs could ensure the continued competence of individual licensed operators. All of the programs were evaluated using the process described in Inspection Procedure (IP) 71001, " Licensed Operator Requalification Program Evaluation." The staff uses the IP to periodically evaluate the licensed operator requalification program at 24-month intervals consistent with the facility licensee's requalification examination cycle.
Prior to August 1995, the staff inspected each licensed operator requalification program at least i
once during the facility licensee's Systematic Assessment of Licensee Performance (SALP) cycle. The staff may also conduct requalification examinations, as needed, when it loses confidence in a facility licensee's i
ability to conduct its own examinations or believes that the inspection 1
process will not provide the needed insight; the staff did not conduct any requalification examinations during FY 96. As noted in SECY-95-042 and SECY-96-026, the staff is no longer tracking the number of licensed operators l
and crews who pass or fail their requalification examinations. Based on informal feedback from licensees regarding the successful staff efforts to minimize undue requalification examination stress, the staff considers the issue of undue requalification examination stress closed and will no longer report on this item unless a generic trend warrants closer scrutiny.
The attachment contains detailed results of the requalification program inspections at each facility inspected. A rating of SAT (satisfactory) for
+
4 i
i f
I l
f l the requalification program inspections indicates that the staff did not elect to conduct NRC-administered requalification examinations because of any i
problems that may have been noted. The requalification program results are l
summarized in the f lowing table.
aequetifIcetfan Program Examinstfan and/or Inspection Results for Flecat Yeer 1996 1
j husher Perecnt j
Element Evolunted SAT /UNSAT SAT (Nunse 1021. Rev. D 00 00/0 N/A IRC Program Enaminstlens 1,
IP 71001 Prescas Inspectie,s 41 41/0 100 Totet 41 41/0 100 l
)
Sr ry of Initial Examination Results The staff is continuing to administer initial examinations to applicants for R0 and SRO licenses at power and non-power reactor facilities. The following i
table gives the initial operator licensing examination results for FY 87 i
through FY 96.
l Inittet Enemination Results Percent Peeeed Durine Fiscal Year Emantnetten ear esa ee9
- 90 eet
'92
- 9s
- 94 e9s 96 l
Written 89 90 90 92 93 92 92 92 90 93 i
RO l
Operating 96 95 95 95 96 98 94 95 97 94
}
Written 91 91 94 - 94 97 98 97 97 93 95 i
SRO Operating 91 95 92 93 95 96 95 93 95 93 i
These results show that facility operator training programs continue to i
j produce applicants who pass NRC's licensing examinations at a relatively high j
percentage-rate.
During FY 96, the staff administered approximately 92 site-specific initial licensing examinations to R0 and SRO applicants at power and non-power reactor facilities and approximately 339 generic fundamentals examinations to prospective license applicants at power reactor facilities. This includes the staff's review and approval of 26 operator licensing examinations that had been prepared by facility licensees in accordance with the current examination development guidance (NUREG-1021, Revision 7, Supplement 1) as supplemented by the pilot examination guidance contained in GL 95-06.
Ooerator Licensina Procram Imorovements i
The NRC is continuing its efforts to improve the operator licensing program.
i During FY 96 and FY 97 to date the staff acted on the following initiatives to j
enhance the initial licensing process:
j (1)
Conducted its " Annual Operator Licensing Examiners Training Conference" which provided an opportunity for NRC Certified Operator Licensing l
Examiners to receive training and hear policy direction from senior j
managers as a group, discuss pertinent topics and provide feedback to l
NRR. The conference is an effective tool in promoting unity and j
consistency in administering the operator licensing program.
1 I
(2)
Initiated implementation of Revision 8 of NUREG-1021, " Operator Licensing Examination Standard for Power Reactors," on a voluntary basis until the rulemaking is complete, j
(3)
Issued NRC Generic Letter 95-06, Supplement 1: Changes in the Operator Licensing Program," which informed and notified all holders of operating i
licenses for nuclear power reactors of (1) the results of the pilot program described in GL 95-06 and (2) NRC's decision to change the operator licensing process so that facility licensees may voluntarily 4
prepare the operating tests and prepare, administer, and grade the written examinations that the NRC will review, approve, and use to determine the competence of operator license applicants at power reactor i
facilities, pending rulemaking to require mandatory facility j
participation.
l (4)
Participated in an industry sponsored public workshop for facility l
licensees planning to develop initial operator licensing examinations j
using the pilot process described in NRC Generic Letter 95-06.
(5)
Increased the participation of facility licensees in the development and administration of licensed operator examinations.
(6)
Reduced the participation of NRC contractors in the development and administration of licensed operator examinations.
(7)
Provided guidance through the Acting Director, Office of Nuclear Reactor Regulation, to each Reg'onal Administrator regarding changes in the operator licensing program.
Specific action includes ensuring that NRC examiners are provided adequate time to review licensee drafted examinations, that examination reviews be scheduled sufficiently in advance of the examination date to allow changes to be made to the examination, that examinations meet NRC expectations prior to l
administration, and that NRC management oversight of the examination process is increased during the transition.
u
~
. CONCLUSION:
The staff believes that NRC's initial and requalification examination and inspection programs continue to be effective methods for evaluating the adequacy of facility initial training and requalification programs and for assessing operator competence to perform licensed duties. The staff also believes that its continuing effort to improve these programs are necessary and fully consistent with the Administration's streamlining initiatives.
O N
h L. Thompson, J.
4 Ac Executive D ec or for Operations
Attachment:
Status Report on the NRC Requalification Program - Fiscal Year 1996 DISTRIBUTION:
Commissioners
)
OGC OCAA OIG OPA OCA EDO CFO REGIONS j
SECY
1 STATUS REPORT ON THE NRC REQUALIFICATION PROGRAM FISCAL YEAR 1996 Facility Program Evaluated Inspection Procedure Performed SAT */UNSAT Date byb[brbrbbk iPhibb5 - R$qu$5 Progr$m in$pbckion 5k[
' ib$ b 9
Beaver Valley 1 IP-71;;; - Requal Program Inspection SAT 10-95 Seabrook IP-71001 - Requal Program Inspection SAT 10-95 Millstone 2 IP-71001 - Requal Program Inspection ST 10-95 Surry IP-71001 - Requal Program Inspection SAT 10-95 Big Rock Point IP-71001 - Requal Program Inspection SAT 10-95 i
Prairie Island IP-71001 - Requal Program Inspection SAT 10-95
' Fort Calhoun IP-71nni - Requal Program Inspection SAT 10-95 Grand Gulf IP-71001 - Requal Program Inspection SAT 10-95 Calvert Cliffs IP-71001 - Requal Program Inspection SAT 11-95 Nine Mile 2 IP-71001 - Requal Program Inspection SAT 11-95 Crystal River IP-71001 - Requal Program Inspection SAT 11-95 Watts Bar IP-71001 - Requal Program inspection SAT 11-95 Vogtle IP-71001 - Requal Program Inspection SAT 11-95 Clinton IP-71001 - Requal Program Inspection SAT 11-95 River Bend IP-71001 - Requal Program Inspection SAT 11-95 South Texas IP-71001 - Requal Program Inspection SAT 11-95 San Onofre 2/3 IP-71001 - Requal Program Inspection SAT 11-95 Browns Ferry IP-71001 - Requal Program Inspection SAT 12-95 Sequoyah IP-71001 - Requal Program Inspection SAT 12-95 Dresden IP-71001 - Requal Program Inspection SAT 2-96 TMI 1 IP-71001 - Requal Program Inspection SAT 3-96
- See footnote at end of table.
ATTACHMENT
\\
l '
Facility Program Evaluated Inspection Procedure Performed SAT */UNSAT Date Peach Bottom IP-71001 - Requal Program Inspection SAT 3-96 Vermont Yankee IP-71001 - Requal Program Inspection SAT 4-96 Kewaunee IP-71001 - Requal Program Inspection SAT 4-96 LaSalle IP-71001 - Requal Program Inspection SAT 4-96 Palisades IP-71001 - Requal Program Inspection SAT 4-96 FitzPatrick IP-71001 - Requal Program Inspection SAT 5-96 Maine Yankee IP-71001 - Requal Program Inspection SAT 5-96 Robinson IP-71001 - Requal Program Inspection SAT 5-96 Diablo Canyon IP-71001 - Requal Program Inspection SAT 5-96 Millstone 1-IP-71001 - Requal Program Inspection SAT 6-96 2
Indian Point 3 IP-71001 - Requal Program Inspection SAT 6-96 i
Zion IP-71001 - Requal Program Inspection SAT 6-96 Vogtle IP-71001 - Requal Program Inspection SAT 7-96 l
Brunswick IP-71001 - Requal Program Inspection SAT 7-96 Millstone 3 IP-71001 - Requal Program Inspection SAT 9-90 McGuire IP-71001 - Requal Program Inspection SAT 9-96 i
Summer IP-71001 - Requal Program Inspection SAT 9-96 Waterford 3 IP-71001 - Requal Program Inspection SAT 9-96 Wolf Creek IP-71001 - Requal Program Inspection SAT 9-96
- A program rating of SAT (satisfactory) indicates that the.NRC staff did not elect to conduct NRC administered requalification examinations for cause as a result of any problems that may have
~
been noted.
4 4
1
a m
A w-a 1
l 1
a J
f ArrACBMFXT 4 l
i
\\
1 1
i 1
l l
s
i 3o1 3 Fem IS7 Page 1 of 36 U.S. NUCLEAR REGULATORY COMMISSION OPERATOR LICENSE EXAMINATION REPORT CAn0!DATE'S NAE DOCKET NJOER FACILITT M
M sa0WN'S FERRY FACILITY 0$$CA!Pfl0N EXAMINAlluN ITPE M0T RfACTOR OPERATOR n
SENIOR REACTOR OPERATOR INSTANT g
COLD SEN!0R REACTOR OPERATCA L? GRADE
$fN!0A REACTOR OPERAT04 LIMITED TO FUCL MAh0LIN6 n
POWER i
NON-POWER 5
IIITIAL MULATION FACILITY AVAILABLE RETAC jag yrs NO l
REQUALIF!CAf!0N 1
WRITTEN EXAMINATION
SUMMARY
WRITTEN 81 (PRINT NAME)
DATE CivtN AAAbfD BY (Pt!NT 1
T. n. Hunt //>//,;f,,,4 in / to / So
. T.H.uuntfj/,/gro,:-('
I/4/SENELIC 2/5 3/6 OVERAll CATEGORY 8
A/H/M 8/!/N C/J/0 O/K/P E/L/C F
G (PE st)
OPERATING TEST
SUMMARY
~
ACMINISTEREDBY(PilNT
)
DATE ADMIN!sTEREC T. H. HUNT 12/11/90 AND 12 / 12 / 90 A. ADMINISTRATIVE 40 PICS B. CONTROL ROOM $Y$TEMS/
C. INTEGRATED PLANT OPEAATIONS FACILITY WALKTit0 UGH UNSAT SAT UNSAT EXAMINER REC 0fflENDATIONS PA11 FA!L WAIVE f
ff WRITTFN FIAMINAT10N
$1GN IN[$TETOR OPERATih6 TEST XXX FI E RECOMifMDATION
$1 f{E LICENSE. RECOMMENDATION CATE IWJL LICEN$E
$16NATUR' $ECT C
[/gg/
hdNYLICENSE
,gggy
,b ma saci - 11a - -
u T:ce test-se-ta
CO'd T101 i
8s.202 Operator license Examination Report rom ts us.1 U.S. NUCLEAR REEULATORf ColellS510N OPEAATOR LICfMSE EXAMIllAT!081 REP 0tT AffMCANT'S NAME:
DOCKET NWMDErb 55-M i
R EXAMINATION TYPE (IMTIAL DR RETAr.E) 5ACILITY NAME:
Sawaevdek RFACTOR OPFRATnR 10(
Wo?
SENICR REACTOR QPERATOR (SRO) INSTANT COLD FACILITY
$RO UPGRADE XX SWR DSSCRPTON i
XX SRO LIMITED TO FUEL HANDL;NG PWR WilTTEN EXAM!nAT!0m sisemRT l
WRITTEN SY:
Roh Maler TOTAL EXAMINATION PolhTS:
Et GRADED SY:
Charme Peyne TOTAL APPuCANT PolNT8; 47 DATE ADMINISTERED.
12/S/85 APPLICANT GRADE.
79.7 OPERATING TEST SisetAY ADtAM8Tmt0 BY:
Cnerles Peyne DATE ADMINitTERED:
12/ht$
A. ADMIMSTRATIVE TOPICS S
- 8. CONTROL ROOM SY$7tM$ AND FAClLITY WALK THROUGH U
C. INTEGRATED PLANT OPERATIONS (SIMULATOR TEST)
S DANINER RECaetotDATIONS CHECK BLCCKS PASS FAP.
WAIVE SIGNATURE DATE e
WNTTEN EXAMINATION XX
/
l OPCRATING TEST XX g
f1NAL DECOMMENDATION XX
>-- /
d
= J** 7 4 LICEsist RECOMM018AT10N ISSWE LICENSE SIGNATWm8. SECTION CHIEF DATE
[
k MPNYllCPNAP
/
/
1 r Examiner Standards Rev. 7. January 1H3 4
M*d sac - 118-m ET:s ETem
'. g"'%g
- g g
UNITED STATES B
NUCLEAR REGULATORY COMMISSION E
f WASHINGTON, D.C. 20665-0001
- % *****/
November 19, 1996 i
f 6
M 6
j Dear Mr.6 in response to your letter of October 4,1996, we have reconsidered the proposed denial issued to you on September 16,1996, and reviewed the grading of the written examination administered to you on August 24,1996, in light of the information you supplied. We find that you did not pass the written examination. The result of our review is enclosed.
Consequently, the proposed denial of your license application is sustained. If you accept the proposed denial and decline to request a hearing within 20 days as discussed below, the proposed denial will become a final denial. You may then reapply for a license in accordance with 10 CFR 55.35, subject to the following conditions:
a.
Because this is your second examination failure, you will be required to retake the written examination and the operating test.
b.
You may reapply for a license 6 months from the date of this letter. A reexamination will be scheduled upon request by your facility management.
If you do not accept the proposed denial, you may, within 20 days of the date of this letter, request a hearing pursuant to 10 CFR 2.103(b)(2). Submit your request, in writing, to the Secretary of the Commission, U. S. Nuclear Regulatory Commission, Washington, D. C. 20555, with a copy to the Assistant General Counsel for Hearings and Enforcement, Office of the General Counsel, at the same address.
. Failure on your part to request a hearing within 20 days constitutes a waiver of your right to demand a hearing and, for the purpose of reapplication under 10 CFR 55.35, renders this letter a notice of final denial of your application, effective as of the date of this letter.
if you have any questions, please contact Mr. Stuart A. Richards, at 301-4151031.
l Sincerely, 1
Bruce A. Boger, Director Division of Reactor Controls and Human Factors Office of Nuclear Reactor Regulation i
Docket No. 55-32235
Enclosure:
As stated cc w/ encl.: Mr. Chris Perrson, Operations Training Supervisor, Perry Nuclear Power Plant l
l 2
Failure on your part to request a hearing within 20 days constitutes a waiver of your right to demand a hearing and, for the purpose of reapplication under 10 CFR 55.35, renders this letter a notice of final denial of your application, effective as of the date of this letter.
If you have any questions, please contact Mr. Stuart A. Richards, at 301-4151031.
l Sincerely,
)
l Original signed by:
j Bruce A. Boger, Director Division of Reactor Controls l
and Human Factors j
Office of Nuclear Reactor Regulation I
Docket No. 55-32235
Enclosure:
As stated
{
cc w/ encl.: Mr. Chris Perrson, Operations Training Supervisor, Perry Nuclear Power Plant DISTRIBUTION:
HOLB RF DMcCain MLeach, Rill MABies, Rill TWalker, RI HWilliams, RI d
v2
- w..
,. we. e
. nie.
in m. n. : c. c y wan.ui as hin.nu.ne c.,y
.ncio.
n Nocoev
'0FFICE HOLB/DRCH l4 HOLB/DRCH b
DDIR/DJCFi DIR/$C)ftf l NAME FGuenther:fg/rc SRichards &
LSptDard BBogek DATE 11//B /96 6# A,---
11/ N/96 11//j/96 11/f1/96 0FFICIAL RECORD COPY Document Name:
P:\\ PERRY. R CERTIFIED MAIL RETURN RECEIPT REQUESTED
1 NRC REVIEW FOR M - RO APPLICANT PERRY NUCLEAR POWER PLANT i
4 In response to the applicant's letter of October 4,1996, the NRC reconsidered the proposed denialissued on September 16,1996, and reviewed the grading of question number 89 of the written examination administered to the applicant on August 24,1996.
Giving due consideration to the information supplied by the applicant, the NRC has determined that the applicant's answer to the subject question (choice "a") remains incorrect and that the applicant still fails the written examination with a grade of 79.8 i
percent. The results of the NRC review are outlined below.
}
I REVIEW OF WRITTEN EXAMINATION QUESTIONS
)
EXAMINATION QUESTION:
The plant is in an emergency condition where the PEls have been entered. The Unit l
Supervisor has determined that emergency depressurization (ED) is ANTICIPA TED and has l
directed an operator to open the Main Turb%e Bypass Valves. After the bypass valves l
were opened, plant conditions degraded and the Unit Supervisor directed an immediate ED i
per PEl-B13, "RPV Control (Non-A TWS). " Assuming a normal suppression poollevel,
)
select the required operator action.
a.
Close the bypass valves and open 8 ADS valves.
4 b.
Open 8 ADS valves andleave the bypass valves open.
I c.
Continue depressurization using only bypass valves at greater than 100 1
degrees fahrenheit per hour.
d.
Continue depressurization using bypass valves and any 4 SRVs.
i i
ANSWER: b j
Applicant's Contentions (summary):
The applicant asserts that choices "a" and "d" should also be accepted as correct answers i
because the initial conditions specified in the question are insufficient to eliminate them as totally incorrect answers. The applicent does not dispute that choice "b" is a correct j
answer to the question.
The applicant states that choice "a" is correct because the bypass valves are not required
}
to accomplish the ED once the 8 ADS valves are opened and there is no procedural j
guidance that prohibits shutting the bypass valves. Furthermore, the applicant contends that the initial conditions in the question were not clear and led him to believe that opening the bypass valves had degraded the plant condition and caused the need for the immediate ED. Therefore, he concluded that shutting the valves would improve plant conditions by removing the cause of the degradation. The fact that the two actions stated ENCLOSURE i
i
~
in choice "a" were joined by "and" not "then" prompted him to assume that the actions could be performed in either order, with the bypass valves being closed only after the ADS valves are verified to be open.
I The applicant also believes that choics "d" is correct because the initial conditions did not 2 -
specify how many of the ADS valves and SRVs were operable, and only 4 safety relief valves (SRVs) are required to depressurize the reactor when it is shut down. The continued use of the bypass valves would not be required once the 4 SRVs are verified to be open.
The applicant recommends that he be given credit for selecting choice "a" or that the question be deleted because three choices are correct, i
l NRC Analysis and
Conclusion:
The NRC has reviewed the initial conditions specified in the question and believes that they are clear and adequate, as written. The NRC acknowledges that the question does not i
provide a reason why the ED was anticipated and subsequently required, but it does not consider that information necessary to answer the question.
The question simply asks the applicant to " select the required operator action" to initiate l
an immediate ED per PEl-B13, "RPV Control (Non-ATWS)," when directed to do so by the i
Unit Supervisor. The only pertinent operator action required by the PEl, regardless of the situation that prompted the ED, is to open all (8) ADS valves. The PEl does not require the main turbine bypass valves to be closed. Furthermore, the PEl Bases document for Perry, page 27, states that the decision to utilize other approved procedures during PEI execution 4
rests with the Unit Supervisor. In this case, the Unit Supervisor has directed only an immediate ED per PEl B13, and no other action. Additionally, page 113 of the plant's PEI i
Bases document supports choice "b" in that it recommends that as much energy as j
possible should be rejected to a heat sink other than the suppression pool in order to j
preserve the pool's heat capacity for as long as possible.
{
The NRC also contacted the Perry Nuclear Power Plant operations training staff to discuss the facility licensee's expectations regarding the operation of the main turbine bypass valves in a situation such as the one described in this question. The training staff confirmed that the facility licensee expects and trains its operators to leave the main j
turbine bypass valves open when conducting an ED in accordance with the PEl.
l The applicant's argument that the two independent actions in choice "a" could be done in either order because they were not stated in a sequential manner (i.e., "close the bypass valves, Iban open 8 ADS valves") is not consistent with the facility licensee's guidance n.
Section 6.8.3 of PAP-0201, which states that PEls are developed logically and shall be performed in the order specified. Therefore, in addition to not being an action required by the PEl, closing the bypass valves before opening the ADS valves, as stated in choice "a,"
j would have delayed the ED unnecessarily and resulted in a further degradation of the j
conditions that required the ED in the first place.
i 2
4 In summary, choice "a" is clearly incorrect because the PEI does not require the main turbine bypass valves to be closed, the Unit Supervisor did not direct that the bypass valves be closed, and closing the valves would unnecessarily compromise the heat capacity of the suppression pool, while possibly delaying the opening of the ADS valves, j
Choices "c" and "d" are also incorrect because they do not include opening 8 ADS valves, the only correct action per PEl 813. Although the PEl supports the applicant's contention that 4 SRVs are sufficient to depressurize the reactor, there is no basis for the applicant. to assume that any of the valves are inoperable.
The applicant did not challenge the correctness of choice "b" in his appeal, and it remains l
the only correct answer to this question. The PEI flowchart clearly directs the operator to J
"depressurize the RPV rapidly using the main turbine bypass valves" when ED is anticipated. The PEl never directs that the bypass valves be closed when the ADS valves are subsequently opened to accomplish the immediate ED. " Leaving the bypass valves open" is a cor.tinuation of the last action directed by and completed in accordance with the PEl, as stated in the initial conditions of the question. The fact that the PEl Bases I
document recommends that as much energy as possible be rejected to a heat sink other j
than the suppression pool also supports this action. Furthermore, in additional response to the applicant's contention that there is no procedural guidance in PEl-B13 which prohibits j
shutting the bypass valves, instructing the operator not to undo p:eviously completed actions would be illogical and inconsistent with the procedural philosophy stated in PAP-j 0201.
s t
e 1
i i
i 4
I 3
i 4
va;-G 6.EWU msw Y4 I
Panasonic PPF l
FR0ri :
i E8-303 operator License Examination Report Form 38-303-1 i
W.S. BucLEAR REGULAvoaf ocentIsszou l
ODERATOR LICENSE anuTmTION REPCRT APPLICMIT'8 MAME: 6 DOCERT NUMBER 58 6 I
R EIARIRATICII TYPE: RETh*E FACILITY MAME: FERRr i
IIUCLEAR 70WER PtAlrr J
]
X REACTOR OPERATOR I
BOT SEllIOR REACTOR OPERATOR (830) INSTANT COLD FACILITY
{
CR0 U7 GRADE g
BWR DESCRIPTION sao LIMITso TO FUEL EMfDLIIIG MIR 4
j WRITTEN BIAltIEnf30If 5thSIhltt i,
WRITTEll BYs D. Prawdsik (LITOO) i I
'teTAL EXaMINATIcel POINTas 99.0 GRADED BYt R. Doornbos TOthL APPLICANT P023178 79 0 i
DATE ADNINISTERED: 08/24/96 APPLICAll? GRADE 79.4%
1 1
OPERATINE ESST smedaBT 4
j ADMINISTERED BY n/a DATE ADMINISTERED: n/a l
A.
ADMINISTRATIVE TOPICS i
N/E
_m.
coerrm0L moon sysTExs AnD PacrLITr unLx-Tanosom w/E C.
INTBGRATED PLhilT OPERATICIl8 (SIMUZATOR TEST)
N/E q
i CHECK EW'RE FASS FAIL MA!VE SIGNATU3tB DATE i
f WRITTEN EXAMINAT2001 1
//[hhh /O k#k OPERATING T387 I
B/E
{
FIwhL macciotsuDATIcer A
- [, MM 9/'/M i
LIcaisss a-*Im Issus LIcamas sIonntuna sacTIcel IBF
[
DENY LICE 3185 yde
/4 1
i h
i i
4 d
4 ArrACIBHWT 5 f
em l b; POLICY ISSUE
)..,. !
(Notation Vote) l l
' ?.;
September 25, 1996 SECY-96-206 fE:
The Comissioners t
i fBQtl:
James M. Taylor. Executive Director for Operations
SUBJECT:
RULEMAKING PLAN FOR AMENDMENTS TO 10 CFR PART 55 TO CHANGE LICENSED OPERATOR EXAMINATION REQUIREMENTS PURPOSE:
I i
To provide additional information regarding the initial operator licensing examination pilot process as directed in a Staff Requirements Memorandum (SRM) i of July 23,1996 (M960618A). and to request the Comission's approval of the j
rulemaking plan to amend 10 CFR Part 55 to require power reactor facility licensees to prepare the entire initial examination for reactor operators and l
senior reactor operators and to proctor the written portion of the examination.
t BACKGROUND:
On March 24, 1995. SECY-95-075 " Proposed Changes to the NRC Operator Licensing Program." informed the Consission of the staff's intent to revise 1
the operator licensing program to allow greater participation by facility j
licensees and to eliminate contractor assistance in this area. This action allows the staff to eliminate between $3 million and $4 million in contractor j
support for examination preparation and administration. The FY 1997 and FY 1998 budget request is consistent with this proposal and reflects the i
elimination of contract support for initial examinations. On April 18. 1995.
j the Commission approved the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully I
consider experience from pilot examinations before full implementation of the changes. On August 15. 1995, the staff issued Generic Letter (GL) 95-06.
" Changes in the Operator Licensing Program." outlining the revised examination development process and soliciting volunteers to participate in pilot examinations to evaluate and refine the methodology.
CONTACT:
NOTE: To BE MADE PUBLICLY AVAILABLE imEN THE Stuart Richards. NRR FINAL SRM IS MADE AVAILABLE (301) 415-1031
. ~ -
.. -.... - -....... ~..
~ - - -
w p
.;A 2-i- -
The Comissioners-e%'V Between October 1,1995, and April 5,1996, the staff reviewed and approved 22 operator licensing examinations that had been prepared by facility licensees
+
in accordance with the current examination development guidance (NUREG-1021.
Rev. 7. Supplement 1) as supplemented by the pilot examination guidance t
contained in GL 95-06. These examinations were then used to test 146 reactor operator and senior reactor operator applicants. Facility licensees from each NRC Region participated in the voluntary pilot progrom, including operating l.-
reactors' built by each power reactor vendor.
i The staff described the results of the pilot examinations in SECY-96-123.
" Pro)osed Changes to the NRC Operator Licensing Program." dated June 10. 1996.
i and >riefed the Comission during a public meeting on June 18. 1996. On the basis of the results of the pilot program, the staff recomended that the i
Comission approve the iglementation of the new examination process on a voluntary basis until rulemaking could be cogleted. This rulemaking would i
require all power reactor facility licensees to pre>are the operator licensing i
examinations based on the guidance contained in NUREG-1021. " Operator Licensing Examination Standards for Power Reactors." which has been revised to accomodate the new process. In an SRM dated July 23, 1996, the Comission authorized the staff to continue the pilot process through July 1997 and directed the staff to develop a detailed rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55. The SRM also directed the staff to specifically address several issues regarding the pilot examination program and the proposed changes in the examination process.
i DISCUSSION:
Section 107 of the Atomic Energy Act of 1954, as amended, requires the NRC to determine the qualifications of individuals applying for an operator license.
to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55 to pass an examination satisfying the basic content requirements specified in the regulation. Part 55 does not state who must prepare, administer. or grade the examination: however, the NRC has traditionally performed those tasks itself or through the use of contract examiners.
On the basis of the results of the pilot program as discussed in SECY-%-123.
the staff now believes that 10 CFR Part 55 should be revised to require power reactor facility licensees to >repare their initial operator licensing-examinations. Specifically. tie staff recomends that Subpart E " Written Examinations and Operating Tests." of 10 CFR Part 55 be amended to require all power reactor facility licensees to prepare the entire initial ope.rator licensing examination and to proctor the written sortion of the eximination based on the guidance contained in NUREG-1021. NJREG-1021 has bee) revised (Revision 8) to in)lement the new process. It has been issued for cceent.
however. Revision B has not yet been implemented. The. licensee-pre >ared examinations will be subject to review, modification, and approval
>y NRC examiners before the examinations are administered. The regulation will provide that the NRC may prepare the. examinations in lieu of accepting or modifying an examination prepared by the facility licensee. The changes would not apply to non-power reactor licensees.
e me e me e m
w*****
- 4B> e
- The Comissioners s.z; 7":.
The enclosed rulemaking plan for this proposed change has been developed in accordance with the guidance in NRC Management Directive 6.3. "The Rulemaking Proce;s." As directed in the SRM of July 23. 1996, the rulemaking plan addresses the potential impact of the proposed rule on facility licensees and discusses why this rule will not constitute a backfit.
The second enclosure discusses the following additional items as directed by the SRM:
The pros. cons, and vulnerabilities associated with implementing the e
proposed examination program changes on an industry-wide basis (including a revik of the impact on licensee and NRC resources. the potential effects.on reactor safety, the impact of proposed NUREG-1021 revisions, potential public wrce approach raised by staff memars)ptions and concerns regarding this A discussion of the degree of acceptance of this new approach to e
operator licensing by facility licensees (including a discussion of industry coments received on draft NUREG-1021. Revision 8. and the staff's response to those coments).
An explanation of enhancements to the new licensing examination process e
resulting from the review and feedbac< from the pilot program results (including a justification for subsequent proposed changes to the operator licensing program).
e The results of initial operator license examinations that have been given both under the pilot program and under the current program since SECY-96-123 was prepared (ir.cluding trends observed).
RECOMMENDATION:
That the Cmenission --
1.
Acorove the development of the rule as described in the enclosed Rulemaking Plan.
2.
Acorove the inplementation of Revision 8 of NUREG-1021 on a voluntary basis until the rulemaking is couplete.
J n
f
l The Commissioners + $. r COORDINATION:
The Office of the General Counsel has no legal objection to this initiative.
s J mes M. T lor xeCutive trector for Operations
Enclosures:
1.
Rulemaking Plan for Changes to 10 CFR Part 55 2.
Response to SRM dtd July 23, 1996 (M960618A)
Commissioners' comments or consent should be provided directly to the Office of the Secretary by COB Wednesday, October 9, 1996.
Commission Staff Office comments, if any, should be submitted to the Commissioners NLT october 2,1996, with an information copy to the Office of the Secretary.
If the paper is of such a nature that it requires additional review and comment, the Commissioners and the Secretariat should be apprised of when comments may be expected.
DISTRIBUTION:
Commissioners OGC OCAA OIG OPA OCA ACRS REGIONS EDO SECY
. k,P,s.,?.. s.
- 4
,Jf.k:16.[*;n.', *;
3 s 1.
, s ?. **h r -
e I y {,. '
g f
rQ +
5fi.,;.'g,...
q.,.
$.. t /,4.' i - ' -
5 }.' '.*
s.
e
+
9 4
RULENAKING PLAN FOR CHANGES TO 10 CFR PART 55 l
4 l
ENCLOSURE 1
F,p.:i e..:
RULEMAKING PLAN CHANGES TO 10 CFR 55 AMENDING INITIAL OPERATOR LICENSE EXAMINATION REQUIREMENTS lead Office:
Office of Nuclear Regulatory Research j
Staff
Contact:
H. Tovmassian, RES/RDB Z/f Concurrences:
_a
_f D. Mortison, RE5
'Date /
i LL. tw i (2 7 m r.. < A (h,/tt ( s.. dMd W. Russell, NRR Date g So,.. g L T.L*<Lm.<s V.'s. f~s i fl G $ti (S**
J. Liebarman, DE Date f Tr 4y 4< uf. e1 44 by mm.,s.J.. TISM b'* b W. Olmstead, OGC Date
]
Approval:
f J. T rg
// Date t
a
)
i s
e9
~
e RULEMAKING PLAN Y
i CHANGES TO THE OPERATOR LICENSING PROCESS AT W ER REACTORS (10 CFR PART 55)
I.
Backaround On March 24. 1995. SECY 95 075. " Proposed Changes to the NRC Operator Licensing Program." informed the Commission of the staff's intent to revise t " 9erator licensing program to allow greater participation by facility licensees and to eliminate NRC contractor assistance in this area. This action allows staff to eliminate between 53 million and 54 million in co.ntractor support for examination preparation and administration. The FY 1997 and FY 1998 budget request is consistent with this proposal and reflects the elimination of contract support for initial examinations. C/. (..il 18. 1995. the Comission approved the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full implementation of the changes. On August *15. 1995, the staff 1ssued Generic Letter (GL) 95-
- 06. Changes in the Operator Licensing Program." which outlined the revised examination development process that would permit facility licensees to orepare the initial operator licensing examinations. GL 95-06 a6 solicited volunteers to participate in pilot examinations to evaluate and refine the methodology.
Between October 1. 1995. and April L. 1996, the NRC conducted 22 initial operator licensing examinations using the new methodology. The staff described the results of the allot examinations in SECY-96-123.
"Proposeo Chan June 10. 1996.ges to the NRC Operator Licensing Program." dated and briefed the Comission during a public meeting on June 18.1996. On the basis of the results of the pilot program. the staff recomended that the Commission a) prove the implementation of the new examination process on a voluntary aasis until rulemaking could be completed to require all power reactor facility licensees to prepare the entire initial examination for reactor operators and senior reactor o>erators and to proctor the written tie guidance contained in NUREG 1021. portion of the examination using
" Operator Licensing Examination Standards for Power Reactors." In a staff requirements memorandum dated July 23.1996. the Comission directed the staff to develop a detailed rulemakin Part 55. g plan to justify the changes that may be necessary to 10 CFR II.
Prooosed Chance Revise Subpart E. " Written Examinations and Operating Tests. to include examination development requirements for facility licensees.
Reaulatory Issue' The pilot program demonstrated that the revised examination development 4
process can be both effective and efficient. Coments from the NRC staff and industry personnel who participated.in the pilot examinations were generally favorable., The, quality of the administered examinations 1
O
.P
(as modified by the NRC) and the performance of the individuals who took piq the examinations were comparable to the quality of and the performance e- :
on examinations written by the NRC staff or its contractors. The fact that some of the draft examinations submitted by facility licensees required significant rework illustrated that many facility staffs did not fully understand the criteria for writing an NRC examination. Both the industry and the NRC staff agree that. with training and experience.
the industry should gain proficiency in preparing i.he examination materials.
The staff believes that the new process should be made mandatory for all' power reactor facility licensees. The staff considers iglementation of the new process.on a voluntary basis alone unworkable over the long term. With the elimination of contractor support. the staff will no longer have sufficient examiner resources to write all examinations consistent with the scheduling needs of facility licensees. This resource problem 15 further compounded by the unpredictable nature of the examination workload.
The staff briefed t'he Committee To Review Generic Requirements (CRGR) after the pilot program was complete.
In the minutes of its meeting.
CRGR agreed that. with regard to the technical and safety aspects. the proposed changes represented a reasonable and workable alternatik6 approach. The committee members unanimously endorsed sending the proposal forward to the Executive Director for Operations (ED0) for consideration. However, the CRGR also requested OGC to consider further the procedural question relating the staff's original plan to implement the rensed process by generic letter.
The staff subsequently met with OGC to resolve CRGR's concern and concluded that requiring all facility licensees to prepare the examinations, a task long performed by the NRC. would require rulemaking or the issuance of orders to each facility licensee. Therefore. the staff recomended in SECY-96-123 that the Comission approve the staff's pursuit of rulemaking to require power reactor facility licensees to prepare the operator licensing examinations, while continuing the pilot process on a voluntary basis.
Current Rule Recuirements Section 107 of the Atomic Energy Act of 1954 (AEA). as amended. requires the NRC to determine the qualifications of individuals applying for an operator license. to prescribe uniform conditions for licensing such individuals. and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55 to pass an examination satisfying the basic content requirements that are also specified in the regulation.
Specifically.10 CFR 55.31(a)(3) requires the applicant to submit a written request from an authorized representative of the facility licensee that the written examination and the operating test be administered to the applicant..furthermore. 10 CFR 55.33(a)(2) states
-p #
.i that the Comission will approve an initial application for a license if W@.. d it finds that the applicant has passed the requisite written examination i
and operating test in accordance with Sections 55.41 and 55.45 or 55.43 and 55.45. These examinations and tests determine whether the applicant for an operator's license has learned to operate a facility competently and safely. and additionally. in the case of a senior operator, whether the applicant'has learned to direct 1.ne licensed activities of licensed operators competently and safely.
Subpart E. " Written Examinations and Operating Tests.' includes specific sampling requirements for the content of written examinations (10 CFR 55.41 and 55.43) and operating tests (10 CFR 55.45).
Although Part 55 1s silent regarding who will prepare. administer and grade the writteh examinations and operating tests. the NRC or its contract examiners hNe traditionally performed those tasks.
Preliminary Reaulatory Analysis In light of the fact that the NRC has historically performed the tasks of preparing and administering the initial licensing examinations. a substantial body of guidance has been published in various versions of NUREG-1021 to a1d both NRC and contract examiners. The NRC expects that licensees will use NUREG-1021 to prepare these examinations. as well.
Licensees may propose deviations from specific guidance in NUREG-1021 and the NRC will review and rule or, the acceptability of these deviations. However. the NRC will not approve any deviation which would have the effect of compromising its statutory responsibil1ty of prescribing uniform conditions for these examinations. Examples of unacceptable deviations include, but may not be limited to the use of essay questions in lieu of multiple choice and the administration of open book examinations. The Statements of Consideration for the proposed rule will contain a discussion of the availability and proposed use of NUREG-1021 as guidance for licensee examination preparers.
1.
Draft Rule Lanauace The first sentente in Section 55.33(a)(2) would be revised as follows (new language is underlined):
"The applicant has passed the requisite written examination and operating test in accordance with Sections 55.40. 55.41. and 55.45 or SL40. 55.43. and 55.45.-
Section 55.40. Implementation." would be added to Subpart E as follows:
-(a) Power reactor facility 'icensees shall prepare the required site-specific written examinations and operating tests.
3
+,
i (b) Power reactor facility licensees shall submit the written
$;]
examinations and operating tests to the Comission for review and approval.
1 (c) Power reactor facility licensees shall proctor the site-i specific written examinations.
(d) In lieu of requiring a power reactor facility licensee to
-prepare the examination and tests. and to proctor the site-i specific written examinations. the Commission may elect to perform i
those tasks.
l (e) The Commission will prepare and administer the written examinations and operating tests at non power reactor facilities."
2.
Imoact of 1'he Rule Chance on Facility Licensees The NRC currently depends on NRC employees and contractors to i
prepare and administer the initial operator licensing examinations required by 10 CFR Part 55.
In accordance with 10 CFR 170.12(1).
the cost of NRC time spent and any related contractual costs are i
billed directly to the facility licensees that receive the examination services.
I i
Under the proposed change. each power reactor facility licensee will assume responsibility for preparing the site-specific initial
{
operator licensing examinations at its facilities. thereby allowing the NRC to discontinue the use of contract examiners for that purpose. Facility licensees will be expected to prepare and submit proposed examinations (including the written examination.
i the walk through. and the dynamic simulator tests) to the NRC based on the guidance contained in NUREG-1021. Licensees may propose deviations from the specific guidance contained in NUREG-1021 when submitting exarinations to the staff for approval. The s
NUREG has been edited to accomodate the revised examination process and is the same procedure that the NRC examiners or its I
contractors would use if they were to develop the examinations.
The training staffs at power reactor facilities already have the basic knowledge. skills. and abilities necessary to evaluate j
operator performance and develop test items for the initial licensing examination. During the mid-to-late-1980s. the industry's emphasis in the training area increased significantly.
All power reactor licensees established formal training programs that were based on a systems approach to training (SAT) and accredited by the National Academy for Nuclear Training. Pursuant to 10 CFR 50.120 and 55.4. SAT-based training programs must include the evaluation of the trainae's mastery of training objectives. NRC inspections of licensee requalification programs for l1 censed operators have also found that training staffs generally possess the skills needed to. evaluate the trainee's knowledge.
4
w..,,..
i After the NRC reviews and approves an examination. the facility
- %._. a licensee will proctor and grade the written portion based on the guidance contained in NUREG 1021. The NRC staff will continue to
{
administer and grade. the operating tests. review and approve the written examination results recomended by the facility licensee.
and make the final licensing decisions.
l Feedback from the pilot examination program indicates that the average time spent by a facility licensee to prepare the written examination and operating tests was approximately 600 to 900 3
i staff-hours. A portion of that time (about 200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />) would have j
been spent reviewing and assisting with the aaministration of NRC-J developed examinations under the process now in place and should be subtracted from the total. The resulting average burden of 1
approximat&ly 400 to 600 staff hours was somewhat higher than the i
400 hours0.00463 days <br />0.111 hours <br />6.613757e-4 weeks <br />1.522e-4 months <br /> that NRC contract examiners typically take to prepare an examination. The extra burden is generally attributable to the facility licensees' lack of familiarity with specific NRC examination expectations and to the additional administrative j
requirements. Such as documenting the source of the examination 1.
Questions, that are required to maintain examination integrity.
It is worth noting that some of the facility licensees that participated in the pilot program expended less time than is commonly used by NRC contractors to prepare the examinations.
J' Furthermore. in a few cases, the examinations that facility licensees submitted for revies and approval were. in the judgment.
of NRC chief examiners. as good or better than those prepared by i
an NRC contractor. The staff exoects that most facility licensees will eventually be able to prepare quality examinations in less time than the NRC or a contractor because the facility employees 2
have more detailed knowledge of their facility and easier access to the reference materials required to prepare the examinations.
I Experience during the pilot examination program indicated that the i
volume of reference materials required by the NRC examiners to review and prepare for the examinations was significantly less l-than that required for the NRC staff to write the examinations.
4 The fact that contract examiners will not be used in the revised -
examination process will also eliminate the need for duplicate sets of reference materials to be provided by facility licensees.
Feedback from the industry in response to the staff's solicitation of public coments on the draft revision of NUREG-1021 indicated
~
that facility licensees had been spending from 80 to 160 hours0.00185 days <br />0.0444 hours <br />2.645503e-4 weeks <br />6.088e-5 months <br /> to prepare and ship the reference materials under the existing extmination process.
The additional burden of having to prepare the site-specific initial operator licensing er.liinations should be substantially.
if not totally. offset by reductions in Part 55 review fees billed to the facility licensees pursuant to 10 CFR 170.12(1). Each 4
facility licensee will be billed only for the time that the NRC staff spends to rgvi_ew the' examination prepared by'the facility L
5 i
4 i
j
~. _.
t
licensee and to rework the examination. as necessary to bring it b.%.-
Up to NRC standards: no contractual costs will be accrued.
Although several of the draft pilot examinations were of poor quality and took the NRC staff more time than expected to review and rework. the staff believes that additional cost reductions will be realized as facility licensees gain experience with the NRC examination requirements and the quality of the draft examinations improves.
This rule change will give facility licensees more control over the cost of their examination services because they will be in a position to manage the quality of the product that is submitted to the NRC. The higher the quality of the examination the facility licensee submits, the lower the resulting charges. Under the existing elamination process. facility 11censees are responsible
\\
for the entire cost of preparing the examination. even if the contractor's submittal 1s of poor quality and requires significant rework by the NRC staff before.it can be administered.
Itisalsowdrthnotingthatfacilitylicenseeswillhavethe option of retalning the services of a contractor to prepare the license examination as the NRC often does under the current examination process. The staff understands that the NRC's examination contractors have already expressed an interest in providing their services directly to facility licensees.
It is reasonable to assume that the cost of using this option would be comparable to the contractor service fees that the NRC currently passes on to the facility licensees.
In stenmary. the staff views this process change as resource neutral at worst. and possibly resulting in a resource savings to facility licensees over time.
3.
ImDact of the Rule Chance on 00erator License Acolicants The goal of the rule change is to eliminate the NRC staff's reliance on contractor support for the site-specific initial operator licensing program, while complying with the AEA and maintaining the existing level of effectiveness. To the extent possible. the format. content. length, and level of difficul.ty of the examinations will rema1n unchanged. thereby minimizing the impact of the rule change on the operator license applicants.
NRC examiners will continue to review and approve every written examination and operating test before it is administered. The 1
examiners will work with the facility licensee's staff to modify the draft examinations as necessary to ensure that the guidance contained NUREG-1021 is met ad to maintain consistency with prior licensing examinations at both that facility as well as among all other facilities.
6
If the NRC decides to prepare the examination in lieu of accepting
- d., n.
an examination prepared by the facility licensee, the NRC
' Jd examiners will use the same procedures and guidance (i.e.. NUREG-1021) that the facility licensee would have used to prepare the examination. The operator license applicants should not be able to distinguish between an examination that is prepared by the facility licensee and one that is prepared by an NRC examiner.
4 Alternatives to the Rule Chanae As discussed in the July 16.1996 budget recomendations. contract support for preparing. administering. and grading initial examinations declines beginning in FY 1996 in anticipation of Comission approval to implement the revised operator licensing program.
ff the Comission decides not to amend 10 CFR Part 55 as proposed by the staff. it would require agency resources to be reprogramed to increase the contract support for the operator licensing program or direct examiner resources in each regional y
office to satisfy the demand for initial licensing examinations and to conduct the licensed operator requalification inspections.
III. Other Elements Leaal Analysis by the Office of the General Counsel (OGC)
The proposed rule would change t.1e current practice that the NRC prepare and proctor the initial examination for reactor operators and senior reactor operators and. instead. require the power reactor facility licensee to prepare the entire examination and proctor the written portion of the initial examination. 0GC worked with the staff to develop the backfit analysis set forth below and for the reasons provided in that analysis believes this action does not constitute a
-backfit pursuant to 10 CFR 50.109.
In addition. this action does not require an environmental assessment because it is categorically excluded from that requirement pursuant to 10 CFR 51.22(c)(1). 0GC has not 1 der.uified any Paperwork Reduction Act issues nor does it believe this action constitutes a " major rule pursuant to the Small Business Regulatory Fairness Enforcement Act of 1996.
In all other respects. 0GC has not identified any potential legal complications or known bases for a legal objection to the rulemaking.
Backfit Analysis As discussed in SECY-96-123. 0GC agrees with the staff's position that requiring the facility licensees to prepare the initial operator licensing examinations would not be a backfit pursuant to 10 CFR 50.109.
which states in pertinent part that --
Backfitting is defined as the modification. of or addition to systems, structures, components or design of a facility: or the design approval or manufacturing license for a facility:
or the procedures orprganTzation required to desi'gn'.
7
Construct or operate a facility: any of which may result p
from a new or amended provision in the Corm 11ssion rules or the imposition of a regulatory staff position interpreting the Comission rules that is either new or different from a previously applicable staff position....
The proposed change does not result in a modification of or an addition to systems, structures. components or the design of a facility. The change does not affect the design approval or manufacturing license for a facility. The procedures regt. ired to design or operate a facility will not be affected by the proposed change. The proposed change would require each nuclear power plant licensee to develop the tests that are used to qualify, as meeting the requirements of 10 CFR Part 55. those nuclear power plant operators whom the nuclear power plant licensee wishes to employ *. Development of such tests are not considered to be
" procedures... required to... operate a facility. " The tests are not applied to the facility licensee, tiut rather to the operator license candidates. Further. any procedure necessary to develop the test would not be useful in actually " operating" the facility, even if one broadly interprets " operating" as incit' ding any action necessary to comply with the Corm 11ssion's regulations with respect to operation. The organization required to design or operate a facility will not be affected because all facility licensees already have a training staff to train and evaluate candidates for operator licenses and to train other members of the plant staff. as required by 10 CFR Part 55 and.by 10 CFR 50.120. Therefore an 3rgan12ational change is not required because of this process change.
Sucoortina Documents A regulatory analysis and an Office of Management and Budget (OMB) statement will be prepared. Neither an environmental impact statement nor an environmental assessment will be prepared for this proposed rule.
however. since it is the type of action described as a categorical exclusion in 10 CFR 51.22(c)(1).
Resources Reouired As discussed previously in Section II.2. the rule will require facility licensees to assume responsibility for writing the examinations. a burden currently carried by the NRC. However. the transfer of this responsibility is considered to be resource neutral. and could possibly result in a resource savings to facility licensees.
The rulemaking is expected to take approximately 0.5 staff years to complete. and no contractor support will be needed. However. until the rulemaking is complete and all power reactor facility licensees are raquired to prepare their exami 3tions. the Office of Nuclear Reactor kegulation (NRR) will continue to use contractors. as necessary. to supplement the staff's ability to develop and administer the initial operator licensing examinations and to conduct the requalification
. hen the, rule 'goes into effect, the ~ staff expects inspection program.
W 8
9
1 that the revised examination process and the requalification inspection E#d program can be implemented at all power reactor facilities with the existing NRC resources allotted to the operator licensing program:
i however. during fiscal years 1997 and 1998. the staff intends to dedicate additional resources to the program to allow additional NRC i
examiners to be qualified. as further discussed in Enclosure 2.
" Resources."
1 4
4 l
.1 5
4 1
9 l
e 9-
. _ _ _~
.c.. :
Coanizant Staff / Project Manaaement Concurrino Official i@Sid Office of Nuclear Regulatory Research (Lead):
H. Tovmassian Bill Morris Office of Nuclear Reactor Regulation:
S. Richards Bruce Boger Office of General Counsel:
M. Schwartz Stuart Treby i
Steerina Grouc 3
A steering group %dll not be required for this effort.
1 Enhanced Puolic harticioation This rulemaking udll use electronic bulletin boards. as appropriate. to enhance input from the public.
ED0 or Commission Issuance The proposed and final rule udll be approved by the Comnission.
Schedule Proposed rule published:
3 months after the EDO approves plan Comment period ends:
60 days after the proposed rule is published Final rule published:
6 months after the connent period ends l
k 4
I 1
(
1 a
m 10 e., e.
eo e maaee9 e 9 A m eeO mM **
- * % %9 $
400 e
- m. - -
.:f '1b 975.. ?;.;is...,.
~}
,)j,,'....
' '!c..,4 g f r ;g0.v :4. ;.'.
w,,.,p;e.. g.,2 /y.4.:.' E. t. ' %. f..
- .s. V.,.,; f.
.z: c..c ;.- a n.
. w
,4 ad, k.G^,'/.4 g %).-g.v tg.. e: ~.
, s.:
- .i....
j 39 4
g g.
s.
- 11..:.pc.;
37.%..e,5:. u.
a
','.W. +,v:a.Q. 9,y,t.+
a.t... A:y.. *.
..w:o.....,e..
9,..v..:.,.3
,,,. s < y..
.s J
1 4
1 4
I RESPONSE TO THE STAFF REQUIREMENTS MEMORANDUM 0F J'JLY 23. 1996 (M960618A) 4 I
1 1
1
.J 4
4 e
i i
l
.i i
e DiCL.05URE 2
//
i t
R
~
DISCUSSION OF PROPOSED OPERATOD LICENSING PROGRAM CHANGES
,g l'
~
Backaround
}
The site specific initial operator licensing examination consists of a 100-
~
question written examination and an orerating test that includes a performance demonstration on a dynamic simulator and a plant walk-through. The written examination is in a multiple-choice format, with a passirg grade of 80 percent. The simulator performance demonstration generally consists of two or i
three rcenarios administered to a team of three license applicants. Each applicant is individually evaluated on a range of competencies applicable to i
L the applicant's license level (i.e.. reactor operator (RO).or senior reactor operator (SRO)). The. walk-through portion of the operating test requires each applicant to--1ndividually complete a total of ten tasks, or job performance measures CPMs). in the control room and the plant and to answer specific i
Questions related to the associated systems.
(Note: an R0 license being upgraded to an SRO license only requires 5 JPMs.) The entire site-spec 1fic examination has historically been prepared by NRC license examiners or NRC contractors. As a prerequisite for taking the site-specific examination.
every applicant must pass a separate generic fundamentals examination (GFE).
t consisting of 100 multiple-choice questions covering basic knowledge of
- mechanical components, principles of heat transfer. thermodynamics. fluid mechanics. and fundamentals of reactor theory. An NRC contractor prepares two versions of the GFE (one for boiling water reactors and another for j
pressurized water reactors) twice per year. The GFEs are r2 viewed and approved by *.he Operator Licensing Branch (HOLB) before being mailed to the facility licensees for administration.
l On March 24, 1995. SECY-95-075. " Proposed Changes to the EC Operator Licensing Program." informed the Conunission of the staff's intent to revise the operator licensing program to allow greater participation by facility licensees and to eliminate contractor assistance in this area. On April 18. 1995 the Coninission consented to the staff's proposal to initiate a i
transition process to revise the operator-licensing program and directed the staff to carefully consider experience from pilot examinations before full l
implementation of the changes.
From May to June 1995. the Division of Reactor Controls and Human Factors 4
(DRCH) and HOLB conducted a number of meetings with senior management and i
regional ~ personnel to discuss various options for implementing the proposal outlined in SECY-95 075. On August 15. 1995. after weighing the pros and cons i
of the various options (discussed below). the staff issued Generic Letter (GL) 95-06 " Changes in the Operator Licensing Program." which notified power reactor facility licensees of the NRC's intent to change the operator j
licensing process, outlined a number of criteria to supplement the guidance in NUREG-1021. " Operator Licensing Examiner Standards. and solicited volunteers l
to participate in a 6 month pilot program to evaluate and refine the new examination F.ethodology.
i On June 10. 1996, the. staff issued SECY-96 123. "Froposed Changes to the NRC Operator Licensing Program." which described the pilot examination process and i
criteria. reviewed the examinati,on results. discussed the lessons learned. and i
1
.c
(
l reconnended that the enhanced process be innlemented on a voluntary basis
' 1..73; pending an amendment to 10 CFR Part 55 that woulg require all power reactor i
M.
facility licensees to prepare the entire initial examination for reactor operators and senior reactor operators and to proctor the written portion of the examination. The staff briefed the Commission on June 18. 1996, and in a staff requirements memorandum (SRM) dated July 23.1996, was directed to i
develop a detailed rulemaking plan and to Mdress the following specific issues regarding the proposed changes in the examination process.
Pros and Cons (Vulnerabilities) i In the early stages of developing the pilot examination process. the staff identified several concerns (examination integrity. Quality, and consistency.
j independence.and public perception, examination security, resources requirements, program stability. and examiner proficiency) that have played a i
1 domiriant role in the dqsi examination methodology. gn. innlementation, and refinement of the revised i
The staff evaluated all of the pilot examination criteria specified in GL 95 06 in light of these concerns. The same concerns t'
were also integral in evaluating the post-pilot process modifications that have been incorporated in'the proposed revision of NUREG-1021. Although the staff believes that each concern has been adequately addressed and mitigated.
i industry-wide implementation of the revised examination process entails a i
measure of uncertainty. The following is a discussion of each concern. the measures that have been implemented to mitigate the concern, and the 4
associated vulnerabilities.
4 Integrity. Quality, and Consistency of the Examination j
Section 107 of the Atomic Energy Act of 1954 as amended. requires the Commission to prescribe uniform conditions for licensing individuals as i
operators. determine the qualifications of such individuals. and issue j
licenses to such individuals.
a h
When the staff developed the pilot examination program. it was concerned about whether the existing levels of examinstion uniformity and consistency could be i
maintained. The operator licensing recentralization study (see SECY-93-309.
" Findings of the Operator Licensing Recentralization Study." of November 17. 1993). which had concluded that the level of examination s
uniformity was adequate to ensure that appropriate licensing decisions were being made, highlighted the challenge of controlling the diversity that
+
individual examiners from different regional offices could add to the examination program.. Permitting a substantial number of facility employees having differing levels-of training and experience to prepare the examinations could increase the amount of variation in the levels of knowledge tested and examination difficulty despite the efforts of the NRC to control these factors j
through improved procedures and oversight reviews.
To maintain uniform standards of examination content, format. level of 3
difficulty. and quality. the staff insistri that the participants in the pilot program prepare the written examinations and operating tests in accordance with the procedures and guidelines in Revision 7 of NUREG-1021. NUREG-1021
_contains guidelines and criteria that are adequate to ensure that a licensing examination. independently prepared by.an ;NRC examiner or contractor. will i
2 4
i I
7_
c differentiate between those applicants who have and those was have not 1 g?.il mastered the knowledge and abilities required to safely operate the plant.
7 However, when the staff reviewed those criteria in light of the proposed shift f
in responsibility for writing the examinations. it concluded that some of the criteria were no longer appropriate and that additional criteria were necesf ary to ensure that the integrity of the facility-develJped examinations would remain at acceptable levels. Therefore. the staff promulgated the following supplemental and modified criteria in GL %-06 when it solicited volunteers to participate in the pilot examination orogram:
NUREG 1021 currently allows NRC and contract examiners to use up to 10 e
percent of the written questions. job performance measures, and simulator scenar.10s in a facility licensee's examination bank Wien they 4
develop an initial licensing examination for the facility. This process l
Saved NRC resourees and provided an incentive for facility licensees to i
develop their examination banks because the examiner could conceivably j
extract the entire written examination from a facility licensee's bank.
provided the bank contained 1.000 or more questions that satisfied the 3
psych 0 metric criterja in NUREG/BR-0122. " Examiners
- Handbook for l
Developvig Operator Licensing Written Examinations."
Whereas the NRC's examiners are continually revising old test items and developing new ones. there is currently no requirement for facility licensees to take comparable measures to minimize predictability.
]
Predictable examinations tend not to discriminate between competent and deficient license applicants because what is being tested is simple 4
recognition-of the answer.
- Although studying past examinations can have ll a positive learning value, total predictability of examination coverage through over-reliance upon examination banks reduces examination i
integrity. When the examinees know the precise and limited pool from which test items will be drawn. they will tend only to study from that pool (i.e.. studying to the test) and may exclude from study the larger
)
i domain of job knowledge. When this situation occurs. it decreases confidence in the knowledge inferen:es that are made from performance on the test.
To limit predictability. GL 95-06 revised the limits on the use of i
facility item banks that could have been published. reviewed, or used as a basis for training. GL 95-06 treat ~1 all question banks as if they were open to the applicants because ti.e staff had no basis to regulate the qualities of the item banks and because it would be difficult to 1
confim that a bank was indeed closed to the applicants. The staff attempted to strike a balance between the use of new and existing test items because it appreciated the effort required to develop new test items and understood that existing items are a valuable training i
resource that should not be wasted. On the basis of the results of the generic fundamentals examination (GFE) program. the staff decided to limit the use of existing written q"estions on the site-specific j
examination to no more than 50 percent and to require at least 10 newly develo0ed (i.e.. not previously seen) questions on every examination.
a The rf.maining questions could be generated by.significantly modifying existing bank questions. Each operating test was required-to include at 1
3 3
4
'{
j
. --r
s.
least two JPMs that were either new or significantly altered and at h.;..m least one simulator scenario that was new or significantly altered. The remaining scenarios had to be modifieo to the extent necessary to prevent the applicants from immediately recognizing the scenarios on the basis of the initial conditions or other cues.
NUREG 1021 currently places a 25 percent limit on the number of written e
examination questions that can be repeated from the previous two NRC licensing examinations at the facility. To further limit the predictability of the examinations that would be prepared by the facility licensees. the staff revised this criterion to include all examinations, quizzes. and tests administered to the applicants during their license training class. Furthermore. no questions could be drawn directly from the applicants' audit examination given by the facility licensee at the end of the license training class to determine if the 6pplicant was ready for licensing. This restriction was the subject of industry coments and has been revised as discussed below since completing the pilot program.
NRC and contract examiners are currently required to cite a specific e
facility reference for every question on the written examination.
Questions that were extracted from the facility licensee's question bank would be identified as such.
In order for the NRC chief examiners to monitor the facility licensees
- compliance with the previously stated criteria. GL 95-06 also required the pilot examination participants to state the history (e.g.. bank, rev1:ed, new, and date last used) of each test item on the written examination and operating tests. Although this practice did place an additional burden on the facility licensees. it 1
facilitated the NRC's examination reviews and resulted in lower licensing fees billed to the pilot participants. This requirement was also addressed in the industry comments and has been clarified as discussed below since completing the pilot program.
The Office of Nuclear Reactor Regulation (NRR) also issued supplementar written guidance to the regional offices clarifying the NRC examiners' y responsit,.lities during the pilot examination development process, and it a
conducted a public workshop to discuss the staff's expectations during the pilot program. The written guidance and the minutes of the workshop were placed in the NRC's Public Document Room (PDR) and provided to facility licensees that were scheduled to participate in the pilot examination program.
NRR did not restrict the number or scope of changes that the NRC examiners l;
could ask facility licensees to make in their proposed examinations. The existing quality assurance (0A) checklists in N'JREG-1021 were revised and new checklists were developed to help the examiners identify problems with the preliminary examination outlines and the operating tests that the facility licensee would be required to submit for review.
Because of resource limitations. examiners were instructed to focus primarily
.n the content and construction (i.e.. level of knowledge and difficulty) of the written examination questions. rather than spend their time verifying the detailed technical accuracy (e.g.. procedural and component references) of 4
4 k
e
..--3
i i
l.
every question, as is required for NRC developed examinations. The staff
- h 'g expected the facility licensees to ensure that the examinations were technically sound and instructed the regions to question the. licensees l.
regarding any post examination question deletions and answer key changes.
3 To assess the effectiveness of the pilot examination process. NRR asked the i
NRC examiners who were involved with the examinations to respond to a li questionnaire on various aspects of the new methodology. The feedback
-in@cated that many of the as-written examinations were comparable in quality and level of difficulty.to recent NRC-written examinations. However. the NRC examiners and facility staff had to rework some of the examinations significantly to meet NRC standards. Although the staff had hoped that the j
technical accuracy of the facility licensee's written examinations would be as good as or better than the accuracy of the examinations written by the NRC.
some of the pilot examination results (i.e.. more question deletions and answer key changes than expected) indicated a need for improvement in that area.
1 Despite the challenges of. quality and technical accuracy. the staff believes that each as-given pilot examination provided a valid basis for licensing the j
applicants who passed the examination. The staff expects that the consistency and quality of the draft examinations will improve as facility licensees gain experience and become more familiar with NRC examination requirements. Based on observations during the pilot examinations. the staff has also incorporated in Revision 8 of NUREG-1021 a number of process enhancements that are intended l
.to 1mprove the quality of the examination, developed under the revised process. A rumber of the significant enhancements are discussed below. NRR P
has also instructed the regional examiners and managers not to approve or i
administer any examination that does not adhere to current NRC standards for 5
content, format. quality. level of knowledge. and level of difficulty.
i.
Deviations from the guidance contained in NUREG 1021 will require approval by the Operator Licensing Branch.
If necessary. the examination is to be delayed 4
j or cancelled until the facility licensee and the NRC staff can upgrade or replace the examination.
1 The overall results of the pilot program indicated that the examinations prepared by facility licensees. subject to review. revision where appropriate.
and approval by NRC staff examiners. were generally consistent with examinations prepared by NRC or contract examiners. The pilot examinations
{
also appeared to be equally effective at identifying applicants who had not i
mastered the job requirements well enough to become licensed R0s or SR0s.
Furthermore. the pass rates for R0s and SR0s on the written examinations and operating tests compared favorably with the pass rates for examinations j
prepared by the NRC or its contractors.
I The chief examiners for approximately half of the pilot examinations actually j
considered the submitted examinations to be as good or better than recently I
administered NRC examinations because the 'acility authors were better able to integrate the required site specific knowledge and abilities into j
. operationally oriented questions. All but 5 of the 22 feedback surveys t
completed by the chief examiners indicated that the revised examination process was as effective or. more effective than'the traditional examination 5
3
+
eu m p p se.
e h
.m.m.
t i-
.. j process. Most examiners also agreed that the performance of the facility MQ licensees should improve as they become more familiar with the NRC's i
'~
requirements and expectations.,
i i
in surm1arv procedural. the operator licensing program staff has implemented numerous 1
i measures to maintain and promote the consistency. Quality, and integrity of the initial licensing examinations, h u ver. in practice. It 4
will be the responsibility of individual examiners. v.th appropriate review.
and oversight by their supervisors and the NRR staff. to ensure that the l
licensing proce33 ierreins strong.
It is possible that the pressures to conduct examinations on schedule and the reduced involvement of the NRC in the examination development process could lead to examinations that discriminate i
at a lower level than 1s currently the norm. The revised examination process is much more dependent upon the staff's ability to identify a substandard i
examination and ensure,that the * ' % encies are corrected. If the staff 15 indecisive and the licensee takes advantage of the situation. the examination l
will prcbably not discriminate at the appropriate level.
Independence and Public Perception I
Before 1987, the staff believed that only independently developed and administered examinations could reliably and consistently achieve.the i.
appropriate stanuards of objectivity and discrimination required for the NRC's initial licensing and requalification programs. The staff was concerned that examinations prenared by a facility licensee could be unconsciously biased toward the specific material taught t. the facility and that potential financial or other pressures on the f'acility licensee would create a conflict j
of_1nterest.
l The staff's position changed significantly in 1988 when it revised the requalification examination methodology so as to require each facility licensee to preDare the examination materials and demonstrate its ability to a
evaluate the competence of its licensed operators. From 1989 to 1994. the NRC i
evaluated over 4.000 licensed operators using requalification examinations j-that were prepared by the facility licensees reviewed and approved by the o
NRC. and administered in parallel by NRC examiners and facility personnel.
The examinations appeared to discriminate at the appropriate level, and i
[
approximately 90 percent of the operators passed the examinations on their first attempt.
Despite the success of the requalification examination process. the staff j
continued to indeper.dently develop and administer the initial licensing examinations. However, the overall performance of the license applicants on the initial examinations was not significantly different from the performance i
of the licensed operators on the requalification examinations developed by their facility licensees.
From 1989 to 1994. the NRC administered initial licensing examinations to approximately 4.000 applicants: the pass-rate on L
those exartinations averaged approximat ly 90 oercent.
4 L
In SECY-95-075. the staff proposed to take the next evolutionary step. similar to the Staff's actions with regard to oversight of the requalification program. by changing the guidance;1n.NtREG-1921 tb permit facility licensees 6
i a
~--
J to prepare and. in part. conduct the written and operating examinations for 36]
initial license applicants. The NRC would review the licensees' efforts.
' ~ '
partially participate in the examinations. retain the final pass / fail authority. and continue to issue licenses. This action would allow the staff to perform its licensing functicn in a more efficient manner. while maintaining an effective, independent process upon which to base licensing decisions for operators.
Shortly after SECY 95 075 was issued. The Washington Post reported that consumer advocates were concerned that allowing facility licensees to prepare the examinations might endanger reactor safety because the examinations might be less rigorous than those prepared by the NRC. The information upon which the consumer groups had based their conclusions was preliminary in nature and likely did not consider the level of oversight that the NRC would retain in the examination process or the compensatory measures that the staff would implement to ensure that the examinations retained an appropriate leve.1 of difficulty.
The kasnington Oost article also indicated that the proposal appeared to contradict the staff's earlier decision to reject a request by the Virginia Electric and Power Company (VEPC0) to prepare its own examinations (this topic will be discussed in more detail later) and that a number of mid-level staff members opposed the change. The staff believes that the initial opposition to
,the proposal was a natural reaction to the abrupt change in policy and to the
' fact that many staff members had not been consulted or informed before SECY-95 075 was issued. As promised in the memorandum of April 12.19%. from the Executive Director for Operations. all involved staff members have been well informed of the new policy and have been able to express their views and to have their concerns considered as part of the process to implement the broad goals stated in SECY 95 075.
As noted in SECY-96-123. the results of the pilot program indicated that initial licensing examinations prepared by facility licensees, subject to review and approval by NRC staff examiners in a manner similar to the review and approval of NRC requalification examinations. were as effective as examinations written by contractors in identifying applicants who had not mastered the job requirements for a licensed operator. The pass rates for R0s and SR0s on the written examinations and operating tests compared favorably with the pass rates for examinations prepared by the NRC and its contractors.
To promote examination consistency and minimize the bias toward specific materials taught by the facility licensee. the staff insisted that the pilot examinations be developed in accordance with NUREG-1021 and NUREG/BR-0122.
NUREG/BR-0122 contains explicit guidance for s upling the important safety-related knowledge and abilities (K/As) from NUREG-1122. Knowledge and Abilities Catalog for Nuclear Power Plant Operators:
Pressurized Water Reactors." and NUREG 1123. " Knowledge and Abilities Catalog for Nuclear Power Plant Dorators: Boiling Water React ~s.' companion documents derived from a job-task analysis of the operators' work. The supplementary pilot examination criteria in GL 95-06 required the facility licensee to submit a proposed examination outline to the NRC for review and. approval at least 60 days before the scheduled examination date. This requirement would enable the NRC chief 7
7 --
~
a examiner to assess whether the sampling was appropriate (e.g.. the balance of M"j coverage and importance ratings: duplication and overla between examination 1
sections and from prior examinations: and inclusion of lant specific 3
priorities. such as recent events and risk insights) be ore the facility licensee proceeded with developing the detailed examination materials.
Gl. 95 06 also required facility licensees to submit the proposed examination i
to the NRC for review at least 30 days before the scheduled administration date.
The NRC chief. examiner and the responsible supervisor would work with thE facility licensee to revise the examination as appropriate before authorizing its administration.
4 To minimize personal conflicts of interest. GL %-06 prohibited facility employees who played a substantial role in training the license applicants from becoming involved in preparing the licensing examinations. Although this restriction did lii.tle'to mitigate the fact that everyone who worked for the facility licensee was likely to have a vested interest in the outcome of the i
examinations, it did alleviate some of the concerns regarding personal gains or penalties that the examination author might incur if he or she were directly accountable for.the examination results..This restriction also i.
allayed the staff's concern that an instructor might unconsciously bias the
. examination toward those topics that wre emphasized during the training program. The personnel restrictions were undoubtedly the most controversial i
aspect of the pilot examination process. They were widely discussed among the NRC staff and management before. during. and after the pilot program was implemented. and they were also the principal issue raised by industry regarding the pilot process.
A discussicn of this and a number of other industry concerns follows.
The 1ssues of Independence and public perception also had a significant bearing on the methodology that was selected for adninistering the operating tests. the best measure of the applicants' ability to respond to a plant Although general agreement existed among the staff on most of the event.
pilnt criteria discussed earlier. there was considerable disagreement regarding the level of involvement that NRC examiners should retain in the operating test administration process. The following options were evaluated in depth:
Option 1:
NRC examiners would continue to inoependently administer the simulator scenarios and the plant walk-through portion of the operating test in accordance with current guidance in NUREG-1021.
Involvement by the facility licensees would not change from current practice. and facility employees would not conduct parallel evaluations.
Option 2:
NRC examiners and facility employees would conduct parallel evaluations of each applicant during the dynamic simulator and walk-through portions of the operating test. This is the same methodology th t is used during NRC conducted requalification examinations.
j l
8 i
)
i e
Option 3:
Qg NRC examiners would obscrve facility employees conduct the I
simulator scenarios and a sample of the walk-through tasks for each applicant: however the NRC examiners would not conduct one on-one parallel evaluations of each applicant.
This methodology is similar to that currently used during the requalification inspection program.
After much deliberation. the staff decided to implement Option 1 during the pilot examination program.
The staff believed that Option 1 would have the most favorable public perception and examiner acceptance because it maintains the existing levels of independence and objectivity by relying on NRC examiners to conduct and evaluate the operating tests. Option 2 would have generated more concern regarding conflicts of interest because it would require facility employees to administer and grade the operating tests (to their coworkers and friends) in parallel with NRC examiners. Option 3 would have had the least NRC involvenent and. therefore, the greatest potential impact on independence and public perception. By not directly observing each applicant. the NRC examiner would t.a in a weaker position to judge the performance of the applicant 50 as to datermine which applicants should pass or fall if a crew responded incorrectly d.: ring a simulator scenarlo.
In sumary.111s possible that the pressures to conduct the examinations on schedule and the reduced involvement Uf the NRC in the examination development process could lead to examinations that reflect the biases of the facility licensee and that discriminate at a lower level than is currently the norm.
However. a n: Imber of process requirements have been augmented as necessary to minimize this potential. The staff believes that the revised examination process. which includes restrictions on personnel. independent NRC review and approval of the exdm1 nations. and independent NRC administration of the operating tests, will provide sufficient safeguards to ensure unbiased licensing decisions.
Given the complexity of the proposed change in the examination process. the public may jump to the conclusion that the NRC has transferred all responsibility for examining operator license applicants to the facility licensees.
In actuality. the NRC retains a significant amount of control over the content of the examinations and continues to administer the operating tests.
The challenge may be to explain this fact to the public.
Examination Security In 10 CFR 55.49. applicants. licensees. and facility licensees are prohibited from engaging in any activity that compromises the integrity (security) of any application. test. or examination required by Part 55.
Before 1989, ensuring compliance with the rule was relatively straightforward because facility licensees were not permitted to review the written operator s
licensing exa,11 nation until it was distri*nted to the applicants on examination day. Security compromises ano cheating were rare but not unheard of occurrences.
~
9 m-
The degree of risk to examination security increased significantly in January
,,kfj, 1989 with Revision 5 of NUREG-1021. Revision 5 made the facility licensees largely responsible for preparing the NRC requalification examinations and.
i for the first time, also allowed the facility staff to review the written i
licensing examination up to 2 weeks before its adninistration. The increased risk to security was tolerated because the prereviews helped identify inaccurate questions before the examinatic.u were administered. thereby increasing the content validity of the examinations. minimizing the number of questions from applicants during the examination. and reducing the number of question deletions and changes during the grading process. The graduai shift to multiple-choice questions during the late 1980s and early 1990s alaced a premium on accuracy because the answers to the questions could not )e adjusted after the fact as.they could be with the essay and short answer questions that they replaced.
To maximize security. given the new philosophy. NUREG-1021 prohibits any facility employee w1th specific knowledge of any NRC examination before it is given from comunicating the examination contents to unauthorized individuals and from participating in. any further instruction of the students LCMduled to take the examination.
Before they 3re given access to the examination the facility employees are required to sign a statement acknowledging their understanding of the restrictions and the potential consequences of noncompliance.
They must also sign a post-examination statement certifying that they did not knowingly compromise the examination.
The staff realized that the potential for examination compromise would increase as the facility licensees increased their role in the initial licensing process from one of reviewing the NRC-developed examinations tn that 4
of developing their own examinations.
Therefore the staff attempted to clarify its security expectations in the supplemental pilot examination guidance that was provided to the regional offices (and placed in the PDR) in i
conjunction with GL 95 06. The guldance addressed the restrictions on personnel and a number of physical security precautions', including protecting and mailing the examination materials. The guidance also cautioned examiners 4
to be attentive to examination security measures and required them to review the securi;y requirements with the facility licensee's contact at the time the examination arrangements were confirmed.
The issue of examination security also had a bearing on the methodology that the staff selected for administering the Dilot operating tests. The staff believed that Option 1 would pose the lowest threat to examination security because the smallest number of facility employees would require detailed knowledge of the examination before it was given. Options 2 and 3 both would have required the facility licensee to evaluate the license applicants during the simulator and walk-through tests, which could have required several additional personnel to familiarize themselves with the operating test content. thereby increasing the risk of compromise.
In sumary, the staff expects that the vast majority of facility licensees
.will continue to maintain proper examination security (as they have during the requalification program). Security is enhanced by the fact that the NRC
+
monitorsandoverseesanegensive.rangeofthe'facilitylicensees' activities 10 4
o; e
and is sensitive to allegations and other performance indicators that might
,g.,
expose a problem. Nevertheless. the staff cannot deny that the NRC is now more vulnerable to examination compromise (and the associated public response) than it was before 1989 when it first allowed facility licensees to become involved with the NRC's examination development programs. The staff anticipates that the agency may suffer an occasional lapse in security. For example. unintentional lapses of examination security have occurred at the Byron and Millstone sites during 1996. Another examination compromise at Dresden is under investigation. When a lapse occum. the staff will continue to take pronipt action to cancel the examination, if appropriate. determine the cause of the compromise and implement corrective measures when called for.
Resources As stated in SECY-95-075. the proposed changes were originally conceived as part of the NRC's continuing efforts to streamline the functions of the Federal Government consistent with the Administration *s initiatives and to accormiodate NRC resource reductions. The original plan was to entirely eliminate the use of contractors in the operator licensing area and to conduct all operator licensing program activities. including the gener1c fundamentals i
and initial licensing examinations. and the requalification program. with the existing staff resources. The facility licensees would prepare and. in part.
4 conduct the initial licensing examinations with oversight by the NRC. The NRC's partic1pation would range from conducting part to all of every examination in order to provide an independent basis for making a licensing decision and to maintain examiner skills.
The staff estimated that a qualified facility employee should be able to prepare the site-specific examination based on the guidance contained in NUREG-1021 as supplemented by GL 95-06 in approximately the same amount of time that is currently allocated to a contract examiner.
(See Attachment 1.
the associated rulemaking plan, for a more detailed discussion of the estimated resource burden on facility licensees.) The staff also estimated that an experienced NRC examiner should take approximately two weeks to review the site specific examination (including the written and operating tests) a prepared by a facility licensee.
As previously noted. the staff considered three options for aoministering the operating tests. Assuming a license class consisting of nine applicants (e.g.. six R0s and three SR0s) that could be arranged in three three-person crews. both the first and second options would generally require three NRC 1
examiners to spend a week at the site to administer the tests. The second l
option. which would require the facility licensee to conduct parallel 4
evaluations of each applicant during the simulator scenarios and walk-through tests, would place a si'gnificant additional burden (nominally. three staff-weeks) on the facility licensee. The third option would be equally burdensome to the facility licensee but would wrmit the NRC to reduce (by one) the j
number of examiners dispatched to tie sit 7 The plan to require the facility licensee to ev61uate the applicants during i
the operating tests was perceived as a duplication of effort because the l
l facility licensee had already eva_luated the applicants and found them j
11 l
s - s o
qualified before it submitted the license applications to the NRC. The staff Q"s.,..,i was also concerned that the additional regulatory burden would discourage
'S participation in what was originally envisioned as a voluntary program with no associated rulemaking.
The NRC resource utilization rate during the pilot examinations supports the staff's original estimate that the new examination process and the requalification inspection program could be implemented at all power reactor fac'11 ties w1th the same level of direct NRC resources as is currently allotted to the operator licensing program. As noted in SECY-96 123. the NRC examiners required an average of about 350 hours0.00405 days <br />0.0972 hours <br />5.787037e-4 weeks <br />1.33175e-4 months <br /> to review, prepare for.
i administer, grade. and document each pilot examination. Most of the examinations included an extra visit to the site to review and validate the operating tests.
Powever, with the elimination of contractor support. the total number of examiners available will decline significantly. thereby reducing the ability of the program to have the personnel available to meet peaks in facility licensee requests for examinations. The impact of the loss of examiners due to reassignments. transfers. promotions. and other personnel actions. will be larger without the contract examiners available to compensate for the loss.
To compenstte in this area. NRR intends to require that the regional offices maintain a defined minimum level of examiners. to be defined in each regional operating plan. During fiscal years (FYs) 1997 and 1998. each region may need to qualify additional examiners to meet this expectation. To specifically assist the regions in their efforts to train additional examiners. NRR intends to make available to the regional offices up to 7 full time equivalents (FTE) of inspection support during FY 1997 and up to 12 FTE of inspection support in FY 1998. This inspection support will allow regional personnel. who would normally be assigned to inspection activities. to be assigned to qualify as examiners. By the end of FY 1998. the regions will be expected to maintain the defined level of examiners in order to provide the resource flexibility needed to deal with workload peaks. When examination work is not scheduled, the examiners will be available for inspection activities. A reduced level of contractor support will be required during the transition period.
The ttaff expects that if the quality of an examination prepared by a facility licensee deviates significantly from established norms, it will become increasingly difficult to predict the amount of time necessary to review or rewrvte the examination so as to achieve an acceptable product. This was the case with several of the pilot examinations. and the' staff expects that it will continue to be more of a problem with facility prepared examinations than it hes been with those prepared by an NRC contractor. This added uncertainty in the examination process could increase the risk of broken examination conmitments and lower examination quality: it could also raise costs as a result of inefficient planning.
With the elimination of contractor suppo? ' and the increased uncertainty about the examination quality the staff may no longer have. sufficient examiner resources to write or even to review all exa'inations consistent with the m
scheduling needs of facility licensees. This resource problem 15 further compounded by the unpredictable nature of the examination workload and by 12
other unanticipated demands on the examiner work force. Such as the increase
($";
in the number of examination appeals during the pilot examination program.
Historically the contract examiners have facilitated the staff's ability to respond to unexpected increases in the examination workload and unforeseen departures by staff examiners. Eliminating contractor support will complicate scheduling and limit the staff's ability to conduct retake examinations. which i
could increase the likelihood that an applicant will appeal the results of the i
examination.
It is for these reasons that the staff 1s reconynending to the Comission that 10 CFR Part 55 be amended to require all facility licensees to prepare the licensing examinations.
l With regard to the generic fundamentals examination (GFE). the staff considered the followifig options:
(1) retaining the current format and process under which the examinations are developed by a commercial contractor and (2) revising the stope of the site-specific written examination to sample the generic fundamental knowledge and aoilities. as was done before 1988.
If the examination was revised by adding more questions. it would increase the e
burden of writing and reviewing the examinations (whether it was done by the facility licensee or the NRC).
If the length of the examination was kept the same. inclusion of the generic knowledge and abilities would Waken the evaluation of the site-specific systems and procedures. The staff and the industry generally have agreed that the current program is both efficient and effective and that revising the s.te-specific sample plan would create too many problems. Accordingly the staff has decided to retain the current generic fundamentals examination format and process and to 1mplement the program with contractor assistance at a c?st of about $200.000 per year.
Program Stability From 1988 to 1990. the NRC received numerous complaints from facility licensees and licensed operators that the NRC's requalification examination program was causing undue stress for the operators and the facility employees who were administering the examinatior.s. The Human Factors Assessment Branch conducted a study to address this issue and determined that the frequent changes to the requalification examination process ranked among the most significant sources of undue stress: the rescits of the study are documented in SECY-91-391. "Results of thc Study of Requal111 cation Examination Stress."
One of the corrective actions involved establishing a schedule for periodic revisions of NUREG-1021 and delaying the implementation of each revision for 180 days after it is published.
Although the staff did not survey license applicants during the requalification stress study. it has generally tried to minimize the adverse effect on applicants by avoiding unnecessary changes to the initial examination process. Therefore, when the staff decided to shift responsibility for developing the initial licensing examinations to the facility licensees. it resisted pressure to revise the examination format. To the extent possible. the format. content. snd level of difficulty of the pilot examinations would remain unchanged. therecy making the changes imperceptible to the license applicants. The staff also believed that the facility licensees were generally familiar with the existing requirements in NUREG-1021 13
a s
and that'ar. significant changes in the examination format would confound
- g4 their efforts to assume responsibility for writing the examinations.
The desire to maintain program stability, avoid unnecessary changes, and minimize stress on the applicants also argued against the adoption of Opticas 2 and.3 for adninistering the operating tests. The requirement for facility personnel to evaluate the license applicants would increase the learning curve for the facility licensees and would likely elevate the stress for the applicants because of the double scrutiny and the increased crowding in the simulator.
In sungnary. the staff found no compelling reason to modify the existing examination format. The benefits of retaining the current format far exceed any argument for making changes at the present time.
Examiner Proficiency.
On May 25. 1995. the staff briefed the Comission on the status of the operator licensing programs and the staff's proposal to shift responsibility for preparing the initial operator licensing examinations to the facility licensees. In the ensuing SRM of June 2.1995, the Comission instructed the staff to ensure that the in house capability to administer the examinations is maintained. prbly through the random administration of some examinations.
As noted in SECY-96 123. the staff believes that NRC examiners will maintain the ability to write examinations. if neressary, based on their participation in the new process. Although facility licensees.will prepare the operator licensing examinations based on the guidance contained in NUREG-1021. NRC examiners will continue to review and approve every examination before it is administered to ensure that it conforms to the criteria specified in NUREG-1021 for content. format. quality. and level of-knowledge. Facility licensees will also proctor and grade the written examinations in accordance with NRC procedures. but NRC examiners will continue to independently administer and-grade both the dynamic simulator and the plant walk-through portions of the operating tests.
The NRC will review and approve the assignment of grades for the written examinations. including any changes to the answer keys and the deletion of questions recomended by tne facility licensee. The NRC will also continue to make all licensing decisions and administer the appeals-process for applicants who fall the examination. Additionally. the NRC may elect to prepare an examination, as allowed by the proposed rule, when poor performance by a facility licensee in the training or operations area would jeopardize the validity of the examination.
NRR will continue to require new NRC license examiners to complete a standardized training program before they are certified to conduct licensing examinations. The training program includes classroom and simulator instruction on how to prepare and administer written examinations and operating tests. self-study of operator Heensing procedures and guidance, on-the-job training. and a final practical evaluation by.a certified chief examiner. Once certified. examiners are required to maintain their qualification by conducting examinations and attending refresher training.
NRR is currently revising..the..examinef refresher' training syllabus to focus 14
.~.
^
1 y
on February 22.19%. NEI submitted coments and recomendations on behalf of
/
jdih the nuclear industry, and two facility licensees (VEPC0 and Comonwealth Edison Company) provided additional coments. On April 18. 1996. the staff conducted a public meeting with NEl and other industry representatives to review their comments and recomendations regarding the draft revision of NUREG 1021. The meeting also provided an opportunity to discuss additional 1ssues and proposed changes that had emerged following the issuance of draft Revision 8 of NUREG-1021 for public coment. The significant coments and st!ff responses are as follows:
All three comenters suggested that the personnel restrictions placed on o
individuals involved in writing the examinations should parallel the 1
personnel restrictions of the requalific0 tion program (i.e.. the i
examination auttprs must stop teaching whm examination development begins). The industry believes that fhose restrictions. when combined 4
i with the examination development and review criteria, would provide adequate protection against the unconscious biasing of an examination by
)
an individual.
It is difficult to argue that the instructors would not be the best qualified individuals to prepare the examinations. Many NRC examiners also agreed that the requalification program restrictions would be adequate. However. the staff concluded that independence and conflict of interest were of sufficient concern to require personnel restrictions very similar to those outlined for the pilot process. The staff revised the wording in the draft NUREG to c:arlfy and quantify the amount of involvement that would disqualify an employee from participating in various aspects of examination development. For example. the personnel who prepare the examination outline. Including the selection of knowledge and abilities to be evaluated on the written examination and the JPMs and simulator scenarios to be performed during the operating test. Shall not have had any direct involvement in training the license applicants. Furthermore. only one person with limited involvement (i.e.. who taught more than 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> but who was responsible for no more than 15 percent of the scheduled classroom instruction) may participate in developing the written exart:1 nation questions. and no one may develop questions for the topics they taught. Facility licensees may still propose alternative approaches for consideration on a case-by case basis.
o All three commenters questioned the need for facility licensees to document the history of every test item prox) sed for use on a licensing examination. even if it was drawn from anotler facility licensee's examination bank.
The staff agreed that the wording in the draft NUREG could have resulted in misunderstandings of what was expected.
The staff still believes that it is important to monitor thE Oredictability of the examinations but it lacks the data and the resources to independently identify the source of every test item. This information is readily available to the facility licensee. The staff has clarified the guidance in NUREG 1021 so that test items t. hat..a facility licensee obtains from~another bank 17
..l and deposits into its own bank may be treated as " bank" items. provided-((.i the test items have an equal chance of being selected for use on the examination. Items from another b3nk may be treated as new items if they have not been made available for review and study by the license applicants and there is no basis (e.g.. historical precedent or reciprocal arrangements with the other facility licensee) for the applicants to predict their use on the examination. The staff believes that this revision will-somewhat lessen the burden on facility licensees, while enabling the NRC to maintain an acceptable level of j.
oversight.
All three comenters suggested that NUREG-1021 relax the restrictions on o
the duplication of test items from the applicants' audit examination (a i
practice test given by the facility licensee before the NRC examination)
If the audit exafnination was developed independently of the license examination.
i Under the traditional examination method. the staff does not have access t
to the facility licensee's audit examination. 50 any duplication that occurs between the Edit examination and the licensing examination is unmonitored and coincidental. The staff agreed that the proposal was consistent with currett pr6ctice and changed the NUREG to allow up to five questions to be dJ written independently (plicated at random if the two examinations are 1.0.. no interaction between the examination authors) and the facility licensee identifies the duplicates. Licensing examinations that are not deve'rped independently shall not repeat any questions directly from the audit examination.
All three comenters recomended that NUREG 1021 accomodate facility c
licensees that have " closed" examination banks (banks not available to applicants for study) by. relaxing the limit on the number of test items that can be drawn directly from the bank.
The staff acknowledged that this is a valid issue. but because this issue is so complex. the staff does not intend at this point to establish additional guidelines regarding the maintenance of examination banks. As noted earlier. the staff intends to treat all banks as if they are "open" at this time.
.l c
All three comenters objected to the suggestion in draft NUREG-1021 that facility licensees consider providing an explanation as to why the answer to every written (multiple-choice) question is correct and each g
of the distractors is plausible but incorrect. The industry thought this was a significant adninistrative burden that would. in practice, become a requirement.
Feedback from the NRC chief examiners indicated that some facility
)
licansees adopted this practice vi their own dring the pilot i
exauinations: it proved very beneficial and increased efficiency during the examination reviews. This prompted the staff to include it as an optional activity in draft NUREG-1021. The staff still believes that -
.. ~
1 18
'o
(C e
q
- w the practice has merit but has emphasized in the final NUREG that it is
-.M
- optional, l
o Two comenters recommended that facility licensees be permitted.to use site specific task lists in place of the NRC's K/A catalogs when developing the examination outline. The site-specific lists are used to develop the operator requalification examinations and should also be acceptable for developing content-valid initial licensing examinations.
Since the mid-1980s, the staff has used the NRC's generic K/A catalogs as the basis for ensuring the content validity of the initial licensing examinations. The catalogs are based upon an analysis of the licensed operator's job that was performed by the Institute of Nuclear Power Operations. Many facility licensees have conducted their own job task analyses and developed plant-specific operator task lists for use in their SAT-based training programs. Although the staff does not question i
that the site-specific task lists would generally be acceptable for ensuring the content validity of the examinations. it believes that wholesale substitution of the catalogs could cause.a decline in examination consistency. The staff considers the proposal. appropriate in limited circumstances on a question-by-question basis: therefore. the i
guidance in NUREG-1021 has been revised to permit substitutions and additions of specific knowledge and ability requirements on a case-by-case basis.
o Twc comenters disagreed with a paragraph in Appendix A of the draft NUREG that 1mplied that any Question taken from an open bank would test at the basic memory level. They argued that questions prepared at a higher cognitive level would not be reduced to the basic memory level if the question bank is large. They strongly recomended that the subject-paragraph in Appendix A be deleted Appendix A provides an overview of examination preparation concepts and discusses the basis for some of the NRC's examination criteria.
including the limits on the use of examination banks. The subject paragraph is considered integral to the discussion and was not deleted.
However, the-staff has edited the text to clarify that questions could decrease to the simple recognition level if the item bank is small and available for the examinees to study.
l The staff also clarified one of the criteria in draft NUREG-1021 that.
for purposes of examination quality and consistency, required at least half of the questions on the written examination to be written at the cuiprehension or analysis level. The staff never intended to imply that any question drawn directly from a bank would be counted at anything other than its face value. A question written at the comprehension or analysis level would count as such. even though the cognitive level at whir.h it tests would decline wi*h use and exposure, o
Two comenters recomended that'the facility licensees be allowed more than five days if necessary. to review and grade the written examinations. They.. asserted that relaxing the requirement would only 19 e
"**e e
aan e,e e
.**=e
a
f$
,r h
~
delay the results to the facility licensee and the applicant and that g
the delay would not adversely affect the NRC.
The staff had traditionally allowed the facility licensee five days to coment on the written examination. 50 it decided to retain that time period for the pilot examination program. This was one of the examination criteria on which the chief examiners had been asked to coment. and no coments were made. However. Since the proposed change would not affect the examination outcome. the staff decided to increase the flexibility in the final NUREG 1021 by noting that the facility licensee should submit the graded examinations within five days. If practical One commenter 1[1dicated that NUREG-1021 does not appear to accomodate o
utilities that d6 not want to prepare their own licensing examinations and suggested that utilities be allowed to retain that option.
Because of the unpredictable demand for examinations and the scheduling difficulties that may result from a long-term voluntary program.
SECY 96-123 requested that the Commission approve the staff's pursuit of rulemaking to require that licensees of power reactor facilities prepare the operator licensing examinations in accordance with NUREG-1021. The Comission has delayed making a decision on industry-wide implementation of the revised process until it can review the detailed rulemaking plan (Attachment 1) and the additional issues discussed herein.
One comenter recomended that the facility licensees be held c
responsible for making examination changes "as agreed upon with the NRC" rather than "as directed by the NRC." The comenter added that this alteration would help ensure that the suggested changes are accurate.
Because the staff fully supports the goal of the facility licensee to ensure the accuracy of the examinations. it revised NUREG-1021 as recomended but stipulated that the NRC would retain the final authority to approve the examination.
4 One comenter requested that the NRC clarify which facility staff o
members would be allowed to observe the operating tests in the simulator. NUREG-1021 currently restricts personnel other than the simulator operator from watching the operating test without the approval of the NRC chief examiner. The commenter argued that facility managers and others deemed necessary by the facility should be allowed access to the examination. provided the simulation facility can accomodate them and there is no adverse effect on the applicants.
The staff concluded that this was a reasonable request and revised NUREG 1021 accordingly. Facility management and other personnel deemed necessary by the facility licensee M ll generally be allowed access to tne examination (under security agreements, as appropriate). provided the simulation facility can accomodate them and there is no adverse effect on the applicants.
20
s o
One commenter requested that the NRC clarify the meaning of
@W!
"significantly modify." The facility licensee had been told that the
- 4 f
intent of the question had to be changed, which conflicted with the definition that had been published in GL 95-06 and draft NUREG-1021.
The staff concluded that this was a reasonable request and revised NUREG-1021 to indicate that the intent of the' question does not have to change to qualify as a significant modification.
In sumary the staff believes that the industry generally supports the proposal to shift responsibility for writing the initial licensing examinations to the facility licensees. Some facility licensees actively lobbied for the change.on the basis that it would save them money. Many facility licensees and NEI view the proposed changes as a good first step at increasing the industry's involvement in the examination process and shifting the NRC's role from direct involvemant to oversight. Nevertheless, most facility licensees woufd probably agree that the administrative requirements that were outlined in GL 95-06. refined during the pilot program, and incorporated in Revision 8 of NUREG-1021 have increased the burden of preparing the examination *s.
The issue of industry burden has been a key factor'throughout the pilot program. The staff purposely limited the scope of the original process changes to those measures that would help ensure the integrity. consistency.
Quality. and security of the licensing examinations. The staff was comitted to maintaining an effective and efficient licensing program and realized that the revised process. Which would hava relied on voluntary industry support.
would never succeed if it was unnecessarily burdensome on the facility licensees.
In keeping with its original comitment. the staff believes that it has taken reasonable measures to address the industry's concerns and recomendations without sacrificing the effectiveness of the examination process.
Pilot Enhancements and Justifications To assess the effectiveness of the new examination process. NRR asked the NRC examiners who were involved with the pilot examinations to respond to a questionnaire regarding various aspects of the process. The first and most significant problem observed and reported by the examiners was that the quality and technical accuracy of the examinations submitted by several facility licensees fell short of the NRC's expectations. Moreover. some of the ear'ly pilot examination results (1.e. more question deletions and answer key changes than expected) reflected the need for greater emphasis in that area.
Consequently. when NRR prepared draft Revision 8 of NUREG-1021, it incorporated a number of clarifications and changes that improved the framework established in GL 95 06.
In January IW6. the staff 1ssued a i
prelimrm draft of NUREG-1021 to th( egions for review and coment before soliciting coments from the public and the industry the following month. The draft revision that was issued for public and industry coment contained the following process improvements:,
21 f
4 o.mse
, a,e.
mo.
,.:. m
~
The existing guidance (Revision 7 of NUREG-1021. Revision 5 of NUREG/BR-o em 0122. and GL 95 06) did not specify what percentage of the questions on ba the written examination should test the applicants at the comprehension and analysis levels of knowledge. Many examiners had expressed concern that the lack of guidance in this area would lead to inconsistency and a decrease in the level of examination difficulty. Facility licensees questioned whether examiners had unwritten guidance that required them to make the examination more difficult by revising or replacing simple memory questions.
As noted earlier. NRR had no desire to increase the average difficulty of the examinations. but it also did not want the level of difficulty to diminish.
There. fore to better define the appropriate level of knowledge for the new examination process. the staff reviewed the written examinatton audits performed in the previous two years to determine an aver. age distribution of questions by knowledge level. The staff concluded that approximately half of the questions on the examinations reviewed were written at the comprehension or analysis level and that the remaining questions tested simple memory. On the basis of that information. the staff established a criterion in draft NUREG 1021 that at least half of the examination questions be written at the comprehension or. analysis level.
Tne supplemental guidance that NRR provided to the regions concurrent l
with GL 95 06 included interim 0A checklists to assist the NRC examiners in reviewing and identifying deficiencies in the draft examinations.
The checklists were edited and incorporated in the preliminary draft of NUREG-1021 and edited again in response to regional coments. The draft NUREG-1021 that was issued for public coment included QA checklists for the examination outline. the written examination, both the simulator and walk-through portions of the operating test. and the written examination grading.
The existing guidance (Revision 7 of NUREG-1021 and GL 95-06) did not o
specify a 11mit on the number of JPMs that could be repeated in walk-through tests administered on successive days during the same initial licensing examination visit or from one visit to the next. Many examiners had expressed concern that the lack of guidance in this area could compromise examination integrity and lead to inconsistency. Other examiners argued that no repetitior, should be permitted because it would increase the predictability of the. test. Facility licensees would be tempted to repeat as many JPMs as possible in an effort to save resources.
To strike a balance. the staff concluded that a 30 percent limit would be appropriate. This figure was consistent with the written examination limit of 25-percent duplication of questions from examinations. quizzes.
or tests administered to the license applicants or from the past two NRC lice sing examinations at the facil.:y.
22 o*
6e
,g*
After issuing draft NUREG-1021 for public and industry coment, the staff identified a number of items that merited clarification and improvement.
In addition to reviewing the industry's coments and recomendations (as previously noted) the staff discussed all of the following changes with NEI and other industry representatives during a Jublic meeting held in the NRC's offices on April 18. ).996. Except as noted yelow. the industry representatives did not object to the additional changes.
To limit the number of technical inaccuracies, a problem that affected c
several of the pilot written exa.ninations, the staff revised NUREG-1021 to require the NRC chief examiner or another examiner to independently review and verify the technical accuracy of a sample of the written examination questions. The number of questions verified would depend upon the expected quality of the examination and the time available before the schedeled review with the facility licensee.
As a final check'of the technical accuracy of the written examination.
the staff revised NUREG-1021 to encourage the facility licensee to l
administer the NRC Approved examinat'lon (under security agreements) to one or more licensed personnel who were previously uninvolved in developing the examination. Althougn feedback from the NRC chief examiners indicated that some facility licensees had taken it upon themselves to v611date their examinations in this manner during the pilot program. the facility representatives at the meeting objected to making it a requirement for all facility licensees. Because the staff lacked sufficient information to quantify the value added by this measure it chose to recomend it as a good practice.
C Late in the pilot program an NRC chief examiner who was reviewing a draft examination detected that the facility licensee had used a written examination outline that was identical to one that a sister facility had prepared several weeks earlier. The staff had not anticipated that such a problem might occur but ime(11ately realized that this was a significant loophole in the process. Allowing facility licensees to reuse the same examination outline would definitely improve efficiency, but it would also jeopardize the integrity of the examination process by increasing the predictability of the examinations.
To address this problem. the staff included in NUREG-1021 a caution for examination authors and reviewers to ensure that the outline used during successive audit and licensing examinations does not become repetitive and predictable.
If a facility licensee proposes to use an outline that was previously used at another facility. it shall identify the source of the outline and explain what effect its reuse is expected to have Gn examination integrity, o
NUREG-1021 has for some time included checklists for use by NRC examiners in verifying that their simulator scenarios contained the requirei number of transients and e;during that the scenarios would enable the examiners to evaluate the applicants-on every required competency. There were no specific instructions for filling out the checklists, nor were they required..to be retained in the examination 23 ee, e
=~
e.e en 9-
file. Concerns regarding the quality and consistency of the operating fK" ~.
tests prompted the staff to incorporate instructions for completing the transient and competency checklists and a requirement for the NRC regional offices to retain.the checklists in the examination files.
l NUREG 1021 has always endorsed the goal of having the licensing o
examinations ready to administer approximately a week before they are scheduled to be given. Furthermore. the staff has always encouraged the regions to keep NRR informed if significant problems are encountered with an.....:ination.
The pilot program illustrated and magnified the importance of both concepts. Therefore. the staff clarified the guidance in NUREG-1021 to limit the number of last minute changes and to require the regional office to consult NRR if it appears that the examination cannot be made to conform to the examination standards at least five workidg days befnre the scheduled examination date.
Furthermore, the examinations should be postponed if problems are identifled at the last minute.
c In addition to implementing the pilot examination process. the staff proposes using Revision 8 of NUREG 1021 as a vehicle for activating Revision 1 of NUREG-1122 and NUREG-1123. the NRC's K/A catalogs. which were published in August 1995. The catalogs were revised to make them more currxt and reorganized to make them more consistent with each other and easier to use. As part of the catalog revisions the system-generic K/As in both catalogs were combined with the plant-wide generic K/As. ano new vendor-specific emergeqcy and abnormal plant evolutions were added to the pressurized water reactor catalog (NUREG-1122). These changes would have altered the balance of the written examinations unless the sample plans in NUREG-1021 were also revised to compensate for the changes. Because of an oversight. the staff neglected to enter the changes before issuing the draft NUREG-1021 for public and industry 1
conment.
1 i
When NRR reviewed the regional office operator licensing programs during e
February and March 1996. it discovered that some of the examiners had i
misinterpreted the intent of NUREG-1021 regarding the design of the j
operating tests for SR0 applicants limited to fuel handling. The staff i
discussed the issue with the regions and agreed that strict compliance l
with the existing guidance would overtest those license applicants.
Therefore. the guidance in NUREG 1021 was changed to more closely align the scope of the operating test with the range of responsibilities for l
SR0 fuel handlers i
e Whenever a question is deleted from the 100-point written examination.
it results in fractional grades. When the grade is above 79.5 percent.
it raises a question regarding the staff's policy on rounding-off i
because it means the difference between receiving a license or a denial.
To address this problem. the staff hs revised NUREG-1021 to indicate that the passing grade on the written examination is 80.00 percent.
Although the pass rates of the pilot examinatjons were comparable with o
pass rates for NRC-written examinations. the rate at which applicants j
t 24
-.... ~
..l
~
l requested that the NRC Informally review their examin;. tion failures s
W increased unexpectedly during the pilot program. These requests not only placed a significant burden on the staff to resolve the applicants' issues but also raised concerns regarding the effect that any additional question deletions and answer changes might have on the other licensing decisions that had already been made.
4 E
Consequently the guidance in NUREG 1021 was revised to afford the NRR staff greater discretion in determining the method for resolving 1
applicant appeals (i.e., every appeal does not necessarily have tv ce reviewed by a panel of examiners as long as objectivity and fairness are maintained) and to delay the issuance of licenses to applicants who passed the written examination with insufficient margin.to guarantee that the licensing decision will be sustained if additional questions are deleted or changed upon appeal. Furthermore. to minimize the number of appeals. the staff revised NUREG-1021_to encourage facility licensees to solicit and address concerns from individual license applicants.
during the process of grading the written examinations.
~
Problems encountered while reviewing and resolving the operating test o
appeals that were received during the pilot program prompted the staff to reevaluate the documentation and grading requirements in NUREG 1021.
As a result. the staff has revised NUREG 1021 to permit NRC exarniners to downgrade more than two simulator test rating factors for the same error. provided the error caused serious safety consequences for the plant. The staff also increased the documentation requirements for the simulator test so that the NRC examiners would have to describe any error that prompted them to downgrade the applicant on any rating factor.
During the meeting of April 18.19%. the staff also discussed the possibility of permitting NRC examiners to recomend a failure on the simulator test even if the applicant did not fail the test on the basis of the ccapetency scores. The industry representatives did not believe that such a change would be necessary 1f there were no limit on the number of competencies that t'le examiner could downgrade for a single, serious error: however, they did not object to the change in principle.
Therefore, the staff has included in NUREG-1021 a provision for NRC examiners to recomend such a failure if the applicant comitted an error with serfous safety consequences for the plant or the public. To limit its use. the region must obtain written concurrence from the Chief. HOLB. before completing the licensing action.
The following issues were raised by NRC examiners on the pilot examination feedback surveys but did not result in specific changes to the examination methodology described in NUREG-1021.
o To minimize NRC resource requirements the staff had encouraged the regions to review and prepare for toe pilot examinations without conducting a preparatory visit to the site. However, feedback from those examiners who did not conduct such a visit indicated that it would have facilitated the reyiew and'resulted in fewer problems with the 25 -
i
)$
1 examination. They recomended th.at a preparatory visit be required for i WIN -
every examination. Although NRR has not changed the guidance in M3 NUREG 1021 (i.e., the preparatory visit is optional at the discretion of 3
i j
regional management). It has acknowledged the value of conducting at least part of the examination review and validation at the facility.
The average NRC resource expenditures during the pilot examinations indicated that the trips can be made within the original budget model.
I o
Several NRC examiners recomended that the required due dates for the
}
examination materials from the facility licensees be advanced 50 that i
the examiners have more flexibility in scheduling their time for the 4
examination review. The staff did not change the guidance in NUREG-1021 i
because it already permits the regional offices to work with the facility licensees in determining the appropriate due dates and because a generic change *would increase the scheduling burden on all facility r
licensees.
It would not be appropriate for an examiner to demand the J
examination 60 days in advance because it was required by NUREG 1021.
1 only to have the examination sit untouched in the office for 30 days j
because the examiner was involved with another project.
i b
o When the staff developed the pilot examination process there was some debate as to whether the facility personnel who adninistered the written i
examination should be permitted.to answer the applicants' questions i
regarding the intent of specific questions on the examination. The i
j pilot guidance allowed the facility to answer the applicants' questions j
with caution but required every question and response to be documented i
for subsequent review by the NRC.
The policy was evaluated on the pilot examination questionnaire that every NRC chief examiner was required to L
complete. A minority of the examiners recomended that the policy be i
changed to prohibit the. applicants from asking questions during the written examination. as is the policy on the GFE. However, most-of the examiners concurred with the original pilot policy and none of the coments indicated that the policy had created any problems with
' examination integrity.. Therefore. the staff saw no reason to revise the i
guidance in.NUREG-1021.
J o
Some examiners noted that the pilot program personnel restrictions had hampered the facility licensees
- ability to develop the examinations because they could not assign their best people to the task. This issue was also addressed in the industry coments, as previously discussed.
Examination Results Since Issuance of SECY 96 123 l
The staff completed the last of the pilot examinations on April 5.1996, and i-resumed conducting the licensing examinations using the traditional i
development process. After April 12. 1996, when the Executive Director for Operations informed the Comission of the staff's intent to continue the pilot L
process beyond the period previously discussed with the Comiss1on, the staff proceeded to (ontact those facility licen d s whose examination requests had not yet been assigned to a contractor to determine if the l
' developing the examinations using the pilot methodology. y were interested in The first of those s
I 26 p
1 f'
.............. ~
t 6
o*
---e,.c w
w-
-%e
-+-ee-a-.
_ > _. _ _ _ ~
t s s
j examinations was administered on August 19. 1996, and the results were not l [,]
available in time to report them here.
Since SECY-96 123 was prepared. the staff has used the traditional examination 1
i methodology to prepare licensing examinations for applicants at 14 power reactor facilities. The results of the completed examinations are summarized below: four of the examinations were completed late in August and the results were not available at the time this paper was prepared. The licensing examination results for fiscal year 1995 and the 22 pilot examinations conoucted between Octobcr 1.1995 and April 5.1996, are provided for j
comparison.
Pass Rates' Examinations
%0 R0 R0 SR0 SR0 SR0 Written Operating Total Written Operating Total i
4 39/40 39/39 39/40 45/48' 43/48 41/48' j
Since SECY-96-123 (98%)
(100%)
(98%)
(94%)
(90%)
(85%)
Pilot 49/54 50/54 45/54 86/92 87/91' 83/923 Examinations (91%)-
(93%)
(83%)
(93%)-
(96%)
(90%)
Fiscal Year 1995 94%
98%
92%
9%
9at 92%
1.
Each box indicates the number of applicants who passed that part of the examination over the number of applicants who took it.
2.
One of the SR0 applicants who failed the written examination has filed an appeal that was not yet resolved at the time this paper was prepared.
3.
Two applicants who were reported as preliminary failures in SECY 123 filed appeals that resulted in the failures being overturned.
Susiarv The staff believes that the revised examination process can be implemented at all power reactor facilities with the same level of direct NRC resources as is currently allotted to the operator licensing program. Additional-resource investment will be made during FY97 and FY98 to increase the pool of NRC examiners available. Full implementation of the revised process will enable the staff to eliminate the use of operator licensing contractors (with the exception of those who develop and grade the GFE) that have historically cost the NRC (and the industry) approximately $3 million to 54 million per year.
The staff expects that the change could result in a net resource savings to the facility licensees over time because they will be able to prepare the examinations more efficiently than the NRC or its contractors.
The overall re5ults of the pilot program indicated that the examinations prepared by facility licensees. Subject to review. rewision where appropriate, and approval by NRC staff examiners, were generally of the same caliber as examinations prepared by NRC or contract c aminers. The pilot' examinations 27
.-.- ~
l si i
also appeared to be equally effective at identifying applicants who had not-K" d...
mastered the job requirements well enough to become licensed R0s or SR0s.
y i-!
Furthermore. the pass rates for R0s and SR0s on the written examinations and operating tests compared favorably with the pass rates for examinations
=
prepared by the NRC or its contractors.
i However. despite-the mitigating act1ons previously noted. the revised examination process is much more dependent upon the abilities of the NRC chief i
examiners to identify weak examinations and work with the facility licensee to upgrade the examination. Furthermore it is possible that the presst'res to j
conduct examinations on schedule and the reduced involvement by the NRC in the i
examination development process could lead to examinations that reflect the j
biases of the facility licensees and that discriminate at a lower level than is currently the norm. Moreover. it is reasonable to expect that industry-t wide implementation of*the revised process will increase the chance of future examinction compromises as a result of the larger number of individuals who i
would require access to the examinations and the additional pressures on facility employees to ensure the success of the applicants.
Although a number.of the examinations submitted by the facility licensees did not meet NRC standards for level of knowledge or difficulty. the NRC chief j-examiners effectively worked with the facility licensees to revise the ~
1 examinations 50 that they would discriminate at the appropriate level. To put j
this situation into perspective it has not been too unusual for contractor-i developed examinations to fall short of the NRC's requirements and i
expectations. The chief examiner oftan directs the contract examiner to make I
numerous corrections and changes before reviewing the examination with the j
facility licensee, only to have the facility reviewer identify a host of l-additional errors and inconsistencies.
i The staff has advised facility licensees that the.NRC expects the technical i
quality of examinatlons to remain high and the number of post-examination changes to be low. Similarly. regional examiners and managers have been instructed not to' approve or administer any examination that does not adhere to NRC standards for content. format. Quality. level of knowledge. and level of difficulty. The QA checklists in NUREG 1021 have been modified to mainta k examination consistency and to facilitate the efforts of NRC examiners to i
detect deficient examination materials. The use of any facility prepared examination that is found to be unacceptable will be delayed or cancelled. if necessary. until the facility licensee and the NRC can upgrade or replace the examination.
The supplemental pilot examination guidance that the staff provided to the regional offices (and placed in the PDR) in conjunction with GL %-06 directed the regions to document in the examination report any significant problems i
encountered during the examination development, adninistration and grading.
This guidance has also been included in Revision 8_of NUREG 1021. The regions are expected to address any serious deficiencies in the examinations submitted by the fatility licensees and any sect.. ity concerns cm ing the entire examination process.
If appropriate, the region will request the facility
,[
licensee to describe the actions it will take to improve future performance.
'If the facility licensee 1,s unab.le or 6nwilling to prepare appropriate 28 c
d
+
- m m.
m---
-,---m
,.e
,-r
%%<v
,r
1
.,. 4.. -.-s licensing examinations. the NRC has the responsibility and authority to W y..
develop the examinations in the traditional manner.
Overall the staff does not expect that the proposed revisions to the operator licensing program will significantly increase the risk to the life and health i
of any individual members of the public or that they will be a significant-addition to other societal risks. The staff believes that the licensing decisions that were made on the basis of the pilot examinations were as valid j
as those made using the traditional examination process. Furthermore. the i
i-changes that the staff has made in NUREG-1021 as a result of lessons learned during the pilot program and industry connents submitted in response to the i
staff s solicitation (which were previously discussed in detail) should only i
improve the effectiveness of the examinations if and when the revised process j
j 15 implemented on an industry-wide basis.
i The NRC will continue to set the standard for operator performance by ensuring that the facility licensees maintain appropriately high standards for 4
examination development. The staff will remain actively involved in the
[
operator licensing process by reviewing and approving every written examination and operating test before it is administered. Unacceptable examinations will be revised ano delayed if necessary until the resources are available to upgrade the examinations. Furthermore, the fact that NRC l
examiners will continue to independently administer the operating tests will allay concerns regarding bias and conflicts of interest on that portion of the licensing examination having the most operational validity and requiring the j
most sub.lective judgements.
The safety consequences of occasionally administering a substandard i
examination are probably minimal. By the time the applicant takes the NRC i
licensing examination. he or she will have completed-a comprehensive.
systematically constructed and implemented training program designed to i
maximize the likelihood of success on the examination. The license training programs typically include numerous written examinations and performance demonstrations that culminate in certification by the facility licensee that t
the applicant has successfully completed the facility licensee's requirements i
to be 11:ensed as an RO or ar. SR0. The effectiveness with which the facility j
licensees screen out those trainees that have not mastered the job requirements has been evident in the high pass rates on the licensing examinations.during the past several years.
In most cases. the NRC examination is simply a confirmation of the facility licensee's decision to
+
submit the application.
Historically. even those applicants W10 fail the NRC written examination i
generally score above 75 percent. The average grade for the 11 applicants who failed a pilot examination was 76.3 percent. This statistic indicates that i
the vast majority of ap)licants, including most of those who fail the
}
licensing examination. lave attained an adequate level of knowledge so as not to pose a significant threat to reactre safety. Furthermore. any applicant j
who does become licensed on the basis of an examination that discriminated at a lower than normal level would be backed up by other crew members who could 3
prevent or remedy any operating errors...Although there is nothing to prevent l
a facility licensee from fprminglan operating crew entirely from newly j
29 L
.~
~
3 A
d aa Ju.
ha--
A S _e a& M a
AAno 1-er.&.4-%e 4a-g**
A 44 A
.44-e-f4-*+wMd-.
2.h
-e4 W@-a-ma-.
94 da-J8a "J.
me,A.W4
.a.L-J e-N 9
a 4..
?
t i
a 4
a E
o e
e 4
i I
4 i
t i
I i
I d
i i
e
. g e
- a88" p
g*
m e O
O 30 e
+4 as a g
's-'%ere q444 O
k -
2 k
4 ATTACIBIENT 6 i
t ne UNITED STATES o
y NUCLEAR REGULATORY COMMISSION
- .]
j WASHINGTON, D.C. 20555-0001 u
a December 17, 1996 OFFICE C# THE SECRETARY 4
t MEMORANDUM TO:
James M. Taylor Execu v
Di ector for Operations k
k FROM:
Jo
. Hoyl Secretary
SUBJECT:
ST F REQUIREMENTS - SECY-96-206 - RULEMAKING PLAN FOR AMENDMENTS TO 10 CFR PART 55 TO CHANGE LICENSED OPERATOR EXAMINATION I
REQUIREMENTS 4
The Commission has approved the staff's rulemaking plan to establish new procedures for the preparation and administration of initial examinations for power reactor operatcrs.
The Commission also approved implementation of Revision 8 of NUREG-1021 on a voluntary basis until the rulemaking is complete.
The revised press release, as attached, should be issued to announce i
the agency plans in this matter.
1 The headquarters staff should evaluate, on a continuing basis, a sampling of examinations developed under the new examination procedure and the quality of the Regiunal Operator Licensing Program to ensure that the quality of examinations is maintained.
Further, the Statements of Consideracion for the rule should specifically describe (1) the continuing NRC involvement and oversight in approving site specific initial operator licensing examinations - written, dynamic simulator, and. plant walk.
through;.and (2) the complete NRC control over administration of the dynamic simulator and plant walk through examinations.
The i
press release' should reflect the Commission'~s decision to proceed with rulemaking and should similarly describe the NRC's control over the licensed operator examination process.
The staff should continue to monitor pilot program results and present the most current data to the Commission with the proposed and final rulemaking packages.
If additional' experience gained during the remainder of the voluntary pilot program together with insights gained as a result of the rulemaking process suggest SECY NOTE:
THIS SRM, SECY-96-206, AND THE COMMISSION VOTING RECORD CONTAINING THE VOTE SHEETS OF ALL COMMISSIONERS WILL BE MADE PUBLICLY AVAILABLE 5 WORKING DAYS FROM THE DATE OF THIS SRM.
i a
n 2
that this may not be the appropriate direction to take, the Commission should be informed of such results in order to reconsider its position prior to approving the final rule.
(EDO) (RESARR)
(SECY Suspense:
2/21/97) 9500056 i
The proposed examination process is dependent upon the abilities of the NRC Chief Examiners to identify weak examinations and to work with facility licensees to maintain examination quality.
Maintaining examiner proficiency in these matters is of paramount importance and should be reflected appropriately in the staff l
procedures and by the budgeting of adequate resources every year.
i The staff-should draft at least one initial operator licensing j
examination per region per calendar year to aid in maintaining proficiency in examination writing, to aid in training of new examiners, and to serve as a quality check.
The staff should l
also regularly convene, on intervals not to exceed'24 months, operator licensing examiner conferences and refresher training sessions, to sustain and improve examiner proficiencies in the l
relevant technical and communication skills.
i Once the final rule is in place, the staff should provide the Commission with an annual update of experience with the'new examination procedure and the 1esults of' headquarters oversight of examination development.
(HM}) (NRR)
(SECY Suspense:
Annually, after approval of final rule) 9600191 i
Attachment:
As stated i
cc:
Chairman Jackson Commissioner Rogers l
Commissioner Dicus Commissioner Diaz j
Commissioner McGaffigan OGC OCA
+
OIG Office Directors, Regions, ACRS, ACNW, ASLBP (via E-Mail) i i
4
I COMMISSION APPROVES RULEMAKING PLAN FOR UTILITY ROLE IN INITIAL REACTOR OPERATOR LICENSE EXAMINATIONS The Nuclear Regulatory Commission has authorized the NRC staff to develop a rule requiring that all nuclear power plant licensees prepare - under NRC d rection - initial reactor operator license examinations.
The proposed rule will be prepared for Commission re' view and approval for public comment early next year.
Until a final rule is adopted, applicants for operator licenses will continue to be examined by using either NRC-prepared tests or those prepared by utilities participating voluntarily in an NRC-supervised pilot program begun last year.
The Commission will take the agency's
]
J experience with the pilot program into consideration when it evaluates both the proposed and final rules.
Reactor operator applicants seeking a license to manipulate the controls of a nuclear power plant must pass both a j
comprehensive, multiple-choice written test and a practical, hands-on examination.
The generic fundamentals examination, a secm.J wAitten test that each applicant must pass to be considered for the final license examination, will continue to be written and administered by the NRC.
In the practical phase, i
applicants must demonstrate a m=stery of plant systems and procedures, and the ability to deal with operational events on a control room simulator.
Licenses are issued for a specific facility.
~
e The rulemaking plan includes these previsions:
NRC.will continue to administer and grade the control o
room simulator and plant walk-through examination j
phases, during which each candidate is evaluated on a 1
one-on-one basis.
l All examinations drafted by utilities will be subject e
to review, modification and approval by NRC examiners before the tests are given.
NRC will have the option of preparing a test in lieu of accepting or modifying one pr-pared by a utility.
o To be approved by NRC, examinations must comply with detailed NRC guidance which deals with such matters as appropriate level of difficulty, maintenance of examination security, and restrictions on test preparation by those significantly involved in training license applicants.
The guidance document, " Operator Licensing Examination Standards for Power Reactors" (NUREG-1021), has been revised to implement che new 1
- process, e
once the new licensing examination process has become fully cperational, the NRC staff will prepare at least one examination annually in each of the agency's four regions to ensure that the staff maintains its T
proficiency in examination writing and to serve as a quality check on the process.
Historically, either NRC staff examiners or NRC contractors have prepared and administered all operator license tests.
But in April 1995, the Commission approved a staff proposal that the agency begin evaluating a system wherein nuclear power plant licensees would prepare the tests under NRC oversight.
The Commission took this action to recognize substantial improvements in industry training programs, to make the operator licensing program more efficient, and to realize budgetary savings.
The staff solicited volunteers for a pilot program in a i
letter sent to all nuclear utilities in August of last year, and launched the program two months later.
Between October 1995 and April of this year, the staff reviewed and approved 22 operator licensing examinations prepared by utilities in accordance with published NRC guidance.
These examinations were used to test 146 applicants for reactor operator and senior reactor operator licenses.
After a staff briefing in June, the Commission authorized continaation of the pilot program through July 1997 to provide time for the rulemaking process.
About half the initial reactor operator licensing examina*. ions now being given are prepared, and one of the written examinations is administered, by utilities taking part in the voluntary, pilot program.
\\
J It is esimated that the new license examination process a
l would permit the NRC to save between $3 million and $4 million l
paid annually to contractors for support of operator licensing and requalification inspection programs.
Unaffected by.the new rule is the licensing of operators for non-power reactors, who will continue to be examined by the NRC.
Also unchanged is the present system whereby utilities prepare and administer 1
requalification examinations to licensed operators as part of an NRC-approved training program.
4 4
h a
4 y
e 1
i i
i 4
e
4 I
9 I
ATTACHMENT 7
- (
Interim Rev. 8 e
i Operator Licensing Examination Standards for Power Reactors Manuscript Completed: January 1997 Date Published: January 1997 i
Division of Reactor Controls and Human Factors Omce of Nuclear Reactor Regulation U.S. Nuclear Regulatory Commission Washington, DC 20555-0001 p*~%,,,,
g.....
-. ~.. - _ _ - -.
l 4
(
ABSTRACT l
NUREG-1021, " Operator Licensing Examination Standards for Power Reactors,"
l e.,tablishes the policies, procedures, and practices for examining licensees j
and applicants for reactor operator and senior reactor operator licenses at power reactor facilities pursuant to Title 10, Part 55, of the Code of federa7 4
i Regulations (10 CFR Part 55). The related guidance that was previously j
published in the " Examiners' Handbook for Developing Operator Licensing Written Examinations" (NUREG/BR-0122, Rev. 5, dated Narch 1990) has been incorporated herein. NUREG/BR-0122 is no longer in effect.
1 i
These examination standards are intended to assist NRC examiners and facility licensees to better understand the processes associated with initial and i
requalification examinations. The standards also ensure the equitable and consistent administration of examinations for all applicants. These standards are for guidance purposes and are not a substitute for the operator licensing regulations (i.e.,10 CFR Part 55), and they are subject to revision or other
]
changes in internal operator licensing policy.
This interim revision permits facility licensees to prepare their initial operator licensing examinations on a voluntary basis pending an amendment to 10 CFR Part 55 that will require facility participation. The NRC intends to solicit comments on this revision during the rulemaking process and to issue a j
final Revision 8 in conjunction with the final rule.
l For examinations prepared by the NRC, this revision will become effective 60 i
days after its publication is noted in the federa? Register. The corporate f
notification letters issued after the effective date will give facility i
licensees at least 120 days of advance notice that the examinations will be j
administered in accordance with the revised procedures. Facility licensees that volunteered to prepare their examinations before the date of the Federa7 Register notice (FRN) are expected to prepare the examinations based on the guidance herein or the pilot examination guidance in Generic Letter 95-06, i
" Changes in the Operator Licensing Program," dated August 15, 1995.
Facility i
licensees that volunteer after the date of the FRN are expected to prepare the i
examinations based on the guidance herein.
4 l
l i
NUREG-1021 iii Interim Rev. 8, January 1997 i
f
\\
1 EXECUTIVE Supt %RY i
i ES-303 The applicants' responses to prescripted JPM follow-up questions will be evaluated based primarily on safety-significance.
i ES-401 This standard now includes instructions and fonas for use in i
developing the written examination outline. References to NUREG/BR-0122, " Examiners' Handbook for Developing Operator Licensing Written Examinations," have been deleted.
I References for guidance in developing multiple choice test items i
have been changed from NUREG/BR-0122 to Appendix B, " Written Examination Guidelines."
l Facility licensees shall submit an outline at least 60 days before the examination date, followed by the " ready-to-use" examination at least 30 days before the examination date.
l A site-specific task list may be used to supplement or override, on j
a case-by-case basis, selected individual items in the NRC's knowledge and abilities catalog; a site-specific task list may not j
be used in place of the entire catalog.
This standard now includes several new criteria developed to ensure the integrity of examinations developed by facility licensees.
These criteria include limits on the number of questions that can be taken directly from the facility licensee's item bank or can be repeated from earlier quizzes and examinations.
In an effort to maintain examination quality and consistency, at least 50 percent of the questions on the examination shall test at the comprehension / analysis level.
As a final check for technical accuracy, facility licensees should consider administering the examination to one or more previously uninvolved licensed personnel (under security agreements).
ES-402 Facility licensees will generally administer the written examinations after they are approved by the NRC. The facility licensees will document for subsequent review by the NRC any questions posed by and answers provided to the license applicants during the examination.
If NRC examiners are on site, they may periodically monitor the administration process.
The guidelines for briefing the applicants who will take the examination have been moved to Appendix E.
ES-403 Facility licensees should collect and consider any questions and comments made by the applicants after the examinations are administered. Facility licensees that prepare and administer the NUREG-1021 xi Interim Rev. 8, January 1997
I l
1 EXECUTIVE SUPMARY
~
Appendix A This new appendix discusses the generic examination concepts that play a role in the operator licensing process.
It includes much of the information that was previously i
contained in NUREG/BR-0122 as well as discussions of new j
topics that have a bearing on the level of difficulty of an l.
examination.
l Appendix B This new appendix incorporates the guidance for developing written test questions that was previously contained in NUREG/BR-0122.
It focuses primarily on multiple-choice l
questions, the only type currently permitted on the initial l
operator licensing examination, and includes examples to j
illustrate various psychometric concepts.
l Appendix C This new appendix summarizes the guidelines concerning job i
performance measures that apply to both initial and requalification examinations. Much of this information was j
previcusly contained in Attachments to ES-603. There are no significant policy changes.
Appendix D This new appendix sussiarizes the dynamic simulator scenario guidelines that apply to both the initial and
- [
req'Jalification examination programs. Much of the
)
i information was previously contained in ES-301 and i
Attachments to ES-604.
l Appendix D also describes a number of simulator security i
vulnerabilities (telated to features of the instructor's s,tation, programmet s' tools, and external interconnections) j that NRC examiners and facility personnel should consider when preparing and administering operating tests.
Appendix E This new appendix summarizes all of the policies and j
guidelines applicable to examinees who will be taking an initial or requalification examination. The information was 4
j previously contained in ES-302, ES-402, ES-602, ES-603, and l
ES-604; there are no significant policy changes.
Appendix F This new appendix provides a central location for defining j
terms used throughout this NUREG.
I l
I i
l NUREG-1021 xiv Interim Rev. 8, January 1997 4
A i
ES-401 PREPARING INITIAL SITE-SPECIFIC WRITTEN EXAMINATIONS l
A.
PURPOSE This standard specifies the requirements, procedures, and guidelines for
)
preparing site-specific written examinations for the initial licensing of 5
reactor operator (RO) and senior reactor operator (SRO) applicants at power 1
reactor facilities.
l B.
BACKGROUND The content of the written licensir.g examinations for R0s and SR0s is dictated i
by 10 CFR 55.41 and 55.43, respectival,v. Each examination shall contain a representative selection of questions concerning the knowledge, skills, and
{
abilities (K/As) needed to perform dutias at the desired license level.
i
[
The written operator licensing examination is administered in two sections, including a generic fundamentals examination (GFE) and a site-specific i
examination. The GFE covers those K/As that do not vary significantly among reactors of the same type (i.e., pressurized or boiling water) and is 4
generally administered early in the license training process (refer to ES-205 i
for a description of the program). The instructions in this standard apply i
only to the site-specific examination.
f Except as noted in Section D.1.b, the " Knowledge and Abilities Catalog [s] for Nuclear Power Plant Operators: Pressurized [and Boiling] Water Reactors,"
NUREG-1122 and -1123, respectively, provide the basis for developing content-valid licensing examinations. Each K/A stem statement has been linked to the applicable item number in 10 CFR 55.41 and/or 55.43.
Preparing the license examination using the appropriate K/A catalog, in con. junction with the instructions in this NUREG, will ensure that the examination includes a representative sample of the items specified in the regulations.
C.
RESPONSIBILITIES 1.
Facility Licensee The facility licensee is responsible for the following activities, as a>plicable, depending upon the examination arrangements confirmed with tie NRC regional office (in accordance with ES-201) approximately four months before the scheduled examination date:
a.
Prepare the proposed examination outline (s) in accordance with Section D.1, and submit the outline (s) to the NRC regional office for review and approval in accordance with ES-201.
b.
Submit the reference materials necessary for the NRC regional office to prepare and/or validate the requested examination (s)
(refer to ES-201, Attachment 2).
NUREG-1021 1 of 39 Interin Rev. 8, January 1997
1 1
l ES-401
)
l f.
Review and submit the completed outline to the NRC chief examiner j
for review and approval in accordance with ES-201. Facility-developed outlines shall be independently reviewed and approved by i
an authorized facility representative before being submitted to the NRC regional office. The NRC must receive the outlines by the date agreed upon when the examination arrangements were confirmed l
(normally at least 60 days before the scheduled examination date).
g.
The NRC chief examiner will review the outline within five working i
days and provide comments and recommended changes, as appropriate.
2.
Select and Develoo Questions
\\
l a.
Prepare the site-specific written operator licensing examination i
using a combination of existing, modified, and new questions in i
accordance with the previously approved examination outline (refer to Item D.1 and ES-201) and the criteria summarized below.
t l
If it becomes necessary to deviate from the previously approved examination outline, discuss the proposed deviations with the NRC i
chief examiner and obtain concurrence. Be prepared to explain why l
the original proposal could not be implemented and why the proposed replacement is considered an acceptable substitute.
b.
Take care to ensure that each question is technically accurate and i
free of the following psychometric flaws that could diminish the validity of the examination:
low level of knowledge j
low operational validity j
low discriminatory validity j
implausible distractors i
confusing or ambiguous language l
confusing or inappropriate negatives 1
collection of true/ false statements j
backward logic j
Appendix B provides a detailed discussion and examples of i
questions containing each of these errors. Appendices A and B i
contain more detailed instructions and guidelines for preparing and formatting content-valid examinations and should be referred l
to as necessary while preparing the examination.
l c.
Establish a level of difficulty that will enable an applicant who j
is capable of safely operating the plant to complete and review the examination within four hours and achieve a grade of 80 percent or greater.
In order to maintain examination quality and consistency, at least 50 percent of the questions on the i
i NUREG-1021 4 of 39 Interim Rev. 8, January 1997 4
i
4 i
I i
ES-401 Site-Specific Written Examination Form ES-401-7 Cover Sheet U.S. Nuclear Regulatory Commission Site-Specific i
Written Examination Applicant Information Name:
Region:
I / II / III / IV
{
Date:
Facility / Unit:
License Level:
R0 / SRO Reactor Type:
Finish Time:
I Instructions i
Use the answer sheets provided to document your answers. Staple this cover sheet on top of the answer sheets. The passing grade requires a final i
grade of at least 80.00 percent. Examination papers will be collected four hours after the examination starts.
i i
i I
Applicant Certification All work done on this examination is my own.
I have neither given nor received aid.
Applicant's Signature Results Examination Value Points Applicant's Score Points Applicant's Grade Percent NUREG-102I 39 of 39 Interim Rev. 8, January 1997
i i
i ES-402 ADMINISTERING INITIAL WRITTEN EXANINATIONS l
A.
PURPOSE l
This standard specifies the requirements and procedures for administering written examinations for the initial licensing of reactor operator (RO) and senior reactor operator (SRO) applicants at power reactor facilities. The i
standard includes instructions for proctoring the examinations and conducting post-examination reviews of NRC-developed examinations.
l B.
BACKGROUND As noted in ES-201, facility licensees will generally prepare the written operator licensing examinations, subject to review and approval by the NRC.
j Generally, examinations that are prepared by the facility licensee will also be administered by the facility licensee in accordance with the instructions contained herein.
C.
RESPONSIBILITIES i
1.
Facility Licensee j
a.
The facility licensee shall safeguard the integrity and security j
of the examinations in accordance with ES-201.
j b.
The facility licensee shall provide a single room suitable for j
administering the written examination. To ensure examination j
integrity, the room shall be large enough so that there is only one applicant per table, with a 3-foot space between tables.
The examination room and supporting restroom facilities (i.e., the examination area) shall be located to prevent the applicants from having contact with all other facility and contractor personnel during the written examination.
c.
If desired and compatible with examination security requirements, the facility licensee may arrange for the applicants to have lunch, coffee, or other refreshments during the examination.
d.
Before the scheduled examination date, the facility licensee should familiarize the applicants with the examination policies and guidelines contained in Appendix E.
e.
The facility licensee shall provide the necessary copies of the approved examinations, answer sheets, and handouts (e.g., equation sheets, selected technical specifications, and steam tables) for each applicant, as directed and approved by the NRC chief examiner.
NUREG-1021 1 of 6 Interim Rev. 8, January 1997
YJ i
)
2.
Start the Examination a.
Remind the applicants that they may use calculators to complete the examination, and that only reference materials provided with l
the examination are allowed in the examination area (i.e., the examination room and supporting restrcom facilities).
j b.
Pass out the examinations, blank answer sheets, and all required j
handouts as approved by the NRC chief examiner (e.g., steam 1
tables, equation sheets, and selected technical specifications).
j Instruct the applicants not to review the examination until told l
to do so.
I Provide each applicant with a copy of Appendix E, " Policies and c.
Guidelines for Taking NRC Examinations," and brief the applicants on the rules and guidelines that will be in effect during the i
written examination (i.e., review Parts A and 8 of the appendix).
If time permits and the operating tests have not yet been administered, review those policies and guidelines (i.e., Parts C, D, and E of Appendix E) as well; this will save time later and give the applicants greater opportunity to resolve any questions j
they may have.
d.
Instruct the applicants to verify the completeness of their copies j
by checking each page of the examination.
)
l e.
Answer any questions that the applicants may have regarding the j
examination policies. Start the examination, and record the time.
i 3.
Monitor the Examination a.
The proctor shall give full attention to the applicants taking the i
examination. The proctor shall not read procedures or other material, grade examinations, or engage in any other activities in a manner that may divert his or her attention from the applicants l
and possibly cause the examination to be compromised.
2 b.
Personnel responding to questions raised by the applicants during the examination must be extremely careful not to lead the applicants or give away answers when clarifying questions.
If the j
proctor has any doubt about how to respond to an applicant's question, it is best to withhold additional guidance and instruct the applicant to do his or her best with the information that is provided.
i Any question changes or clarifications shall be made on a chalk j
board or white board, if available, and called to the attention of all the applicants. Changes made to questions during the j
examination should be made in ink on the NRC master copy and on a i
copy that is retained by the facility staff after the examination NUREG-1021 4 of 6 Interim Rev. 8, January 1997 i
1
i l
I i
ES-403 l
j GRADING INITIAL SITE-SPECIFIC WRITTEN EXANINATIONS i
A.
PURPOSE This standard explains the requirements and procedures for grading the site-
)
specific written examinations for the initial licensing of reactor operator j
(RO) and senior reactor operator (SRO) applicants at power reactor facilities.
j The standard includes instructions for evaluating and revising the examinations after they are administered, grading the examinations, and 2
conducting the first quality assurance (QA) review of the graded examinations.
l B.
BACKGROUND As discussed in ES-201, facility licensees will generally develop and administer the initial operator licensing written examinations, subject to review and approval by the NRC.
Facility licensees will also be expected to grade the written examinations, evaluate the outcome, and submit the examination results to their NRC regional office for review, approval, and licensing action in accordance with ES-501.
I C.
' RESPONSIBILITIES 1.
S cility Licensee i
a.
If the facility licensee developed and administered the written 4
examinations, the licensee is also expected to perform the l
following grading activities, as described in Section D:
Review and resolve any questions and comments that arose j
during and after the examination (refer to ES-402).
i Grade the examinations and review the grading using Form ES-403-1, " Written Examination Grading Quality Assurance Checklist."
)
Evaluate the applicants' performance on the examination.
Facility management will review the examination grading based on the guidance in ES-501 and forward the graded examinations and all associated documentation to the NRC chief examiner so that it is received, when practical, within five working days after the j
examination was administered.
4 b.
If the NRC developed the examinations, the facility licensee's responsibility is limited to providing the NRC chief examiner with comments and recommendations regarding question deletions and I
answer key changes. Such comments and recommendations should normally be received within five working days after the exit meeting; any delay in submitting the comments will likely result NUREG-1021 1 of 5 Interim Rev. 8, January 1997
i j
ES-403 in a comparable delay in the final licensing actions.
(Refer to 1
i ES-402 for additional instructions regarding the post-examination j
review and comment process.)
.]
l 2.
LRC Reaional Office j
a.
If the facility licensee grades the examinations, the regional i
office shall provide guidance and assistance, as necessary, to 1
i ensure that the facility licensee complies with the instructions j
in Section D.
b.
If the NRC developed the examinations, the regional office shall grade the examinations in accordance with Section D after l
receiving any comments and recommendations from the facility licensee (refer to ES-402).
c.
After the examinations have been graded (by either the facility licensee or an NRC examiner), the regional office shall review the l
grading, process the documentation, and complete the licensing
[
actions in accordance with ES-501.
l D.
GRADING INSTRUCTIONS The author of the examination should normally grade the examination; however,
+
the examination may be graded by another equally qualified individual if the i
author is not available, the number of applicants is unusually large, or the NRC regional office or facility licensee wishes to expedite the grading process. The examinations shall be graded as expeditiously as possible, in accordance with the following instructions:
1.
Evaluate Questions and Comments i
a.
Evaluate all questions. posed by the applicants during the examination, any pen-and-ink changes made on the master i
examination during its administration, and any post-examination comments or recommendations received from the facility licensee l
and applicants after the examinations were administered.
Determine if any questions should be deleted from the examination, or if any answers need to be changed. Do not delete any question or change any answer unless there is a valid reference to support the change.
l If there is some doubt whether the NRC chief examiner will accept a proposed change, the grader is encouraged to discuss the matter with the chief examiner before proceeding with the grading process. This may help to minimize the need for grading l
corrections during the quality assurance reviews.
1 l
For each comment and recoimeendation, the NRC chief examiner shall document the reason that the question was changed or the comment l
NUREG-1021 2 of 5 Interim Rev. 8, January 1997 4
i 4
l
[;
ES-403 was not accepted; this information will be included in the examination report as discussed in ES-501.
b.
If it is determined that there are two correct answers, both answers will be accepted as correct. However, if three or more i
answers could be considered correct or there is no correct answer, i
the question shall be deleted. Annotate the recommended changes on the master examination and answer key and document the reason i
for every change or deletion.
3 c.
Those applicant questions, facility comments, and recommendations that do not result in answer key changes or question deletions, i
should be evaluated to determine if the associated test questions l
might benefit from editorial changes before they are used on i
another examination.
l 2.
Grade the Examinations l
a.
On each applicant's answer sheet, indicate in red pen or pencU which questions were answered incorrectly, note their correct answers, and indicate which questions (if any) were deleted.
If the answer sheet is more than one page long, it is helpful to note the total number of incorrect answers on each page to aid in i
tabulating the final grade.
If the examinations are graded by machine, attach a copy of each j'
applicant's profile report to his or her answer sheet, or manually j
annotate the answer sheet as noted above.
i b.
If it is necessary to change a grade during the grading process, do so by lining out the original grade in such a way that it i
remains legible.
Briefly explain the reason for the change on the l
applicant's answer sheet, and initial the change. Under no i
circumstances will a grader use " white-out" or other methods that j
obscure the change.
l c.
After grading all the questions, enter the " Examination Value" i
(i.e., the original test point total minus the point value of any deleted questions), the " Applicant's Score," and the " Applicant's Grade" (i.e., the Applicant's Score divided by the Examination l
Value) in the "Results" section of the applicant's written j
examination cover sheet.
)
If a facility chooses to share its preliminary grades with the l-applicants, it should caution them that the outcome may change if the NRC does not accept all of the facility licensee's recommended changes to the examination answer key.
1 I
NUREG-1021 3 of 5 Interim Rev. 8, January 1997 4
I l
i 4
i ES-403 3.
Evaluate and Review the Gradino a.
Evaluate the applicants' performance on each examination question to identify any indications of a problem with the question or a deficiency in the applicants' training program. A table that summarizes the applicants' answers on each question, or a t
computerized item analysis (if the examinations were graded by i
machine) may be used to identify items with which the applicants j
had problems.
l If it appears that a test question was faulty, determine whether the question should be deleted, the answer key should be changed, i
and/or the question should be revised before reuse. Then regrade the examinations as necessary.
If it appears that the training program was deficient, determine j
the need for remedial training and/or a program upgrade.
i l
b.
After evaluating the examinations, review the grading in detaf 7 and complete Form ES-403-1, " Examination Grading Quality Assurance j
Checklist. "
i c.
Forward the examination package (i.e., the master examination and answer key, justification for any examination changes, any ites analysis that was performed, the applicant's examination cover and answer sheets, and Form ES-403-1) to the designated facility l
representative (if applicable) or to the NRC chief examiner for j
quality assurance review in accordance with ES-501.
i E.
ATTACHMENTS / FORMS Form ES-403-1,
" Written Examination Grading Quality Assurance Checklist" i
l l
i i
l i
k NUREG-1021 4 of 5 Interim Rev. 8, January 1997 l
w
2 403 Written Examination Grading Form ES-403-1 i
Quality Assurance Checklist l
Facility:
Date of Exam:
Exam Level: RO/SRO l
Initials l
Item Description a
b c
j I
1.
Answer key changes and question deletions i
justified and documented l
2.
Applicants' scores checked for addition errors l
(reviewers spot check > 25% of examinations) 3.
Grading for all worderline cases (80% +/- 2%)
{
reviewed in detail
'4.
All other failing examinations checked to ensure that grades are justified 4
- 5. -
Performance on missed questions checked for i
training deficiencies and wording problems; j
evaluate validity of questions missed by half or j
more of the applicants Printed Name / Signature Date l
a.
Grader i
j b.
Facility Reviewer (*)
i j
c.
NRC Chief Examiner (*)
l d.
NRC Supervisor (*)
i
(*)
The facility reviewer's signature is not applicable for examinations graded by the NRC; two independent NRC reviews are required.
NUREG-1021 5 of 5 Interim Rev. 8, January 1997
.~.
4.
l APPENDIX A OVERVIEW 0F GENERIC EXANINATION CONCEPTS
]
+
i 1
A.
PURPOSE The purpose of this Appendix is to provide an overview of two fundamental examination concepts, validity and reliability, as they apply to the development of NRC operator licensing and requalification examinations. The following topics are discussed:
l the rationale for providing guidance for the construction, review, and approval of NRC examinations (Section B)
)
the various aspects of validity and how the NRC establishes the j
validity of its examinations (Section C) e the concept of reliability and how it is maintained on NRC i
1 examinations (Section D)
I B.
BACKGROUND
]
The fact that the NRC's operator licensing examinations are prepared and j
administered by many different individuals working in various locations makes it imperative that a defined set of administrative structures and protocols be established and followed to ensure that the examinations are administered l
successfully and consistently.
External structures such as the number and i
kind of items, the length of the examination, security procedures, proctoring j
instructions, and other administrative details are essential to the orderly conduct of an examination. These factors can have a significant effect on the reliability and validity of an examination, the cornerstones that allow the i
NRC to make confident licensing decisions.
In addition to these external structures, the internal attributes of the a
examination, such as its level of knowledge, level of difficulty, and the use i
of item banks, also impact the operational and discriminatory validity of the examination, which, in turn, can affect its consistency and reliability. When the internal and external structures of examinations are allowed to vary significantly, this challenges the unfform condiffons that are required by Section 107 of the Atomic Energy Act of 1954, as amended, and the basis upon j
which the NRC's licensing decisions rest. The NRC must reasonably control and structure the examination processes to ensure the integrity of the licenses it issues.
l Acceptable levels of examination consistency, unifomity,. and fairness would be impossible to achieve without quantitative and qualitative acceptance criteria. While the Examination Standards identify many of the quantitative i
criteria necessary for a well-balanced and consistent examination, those i
criteria can only go so far when it comes to ensuring examination validity and i
reliability. Although the NRC's Knowledge and Abilities Catalogs (NUREG-1122 and -1123) have brought a degree of consistency to the qualitative issue of NUREG-1021 1 of 10 Interin Rev. 8, January 1997 t
i 4
i i
i Appendix A 1
l case.
Refer to Section D for a more detailed discussion of consistency and reliability and to Appendix B for a more detailed discussion of the i
i various levels of knowledge as they relate to the development of written i
test questions.
)
l 3.
Discrimination Validity 1
The third central validity issue concerns the examination's ability to j
discriminate, or to make some distinction along a continuus of examinee l
perfonnance.
In that regard, the primary objective of the NRC examinations is to determine whether or not the examinees have sufficiently " mastered" the knowledge, skills, abilities, and other 1
attributes to perform the job of reactor operator (RO) or senior reactor 1
operator (SRO) at a specific plant. The NRC examinations are not intended to distinguish among levels of competency or to identify the i
most qualified individuals, but to make reliable and valid distinctions at the minimum level of competency,that the agency has selected in the interests of public protection.
a.
Criterion-referenced Testing l
The NRC's initial and requalification examinations, like most j
licensinq examinations, are criterion-rather than norm-referenced
)l j
tests. "his means that there is a pass-fail or minimal cut score or a
grade that the examinee must achieve to demonstrate sufficient l
knowledge and ability to safely operate the power plant.
If the examination does not iritend to discriminate at an agreed-upon minimal measure of knowledge or performance, then there is little reason to give the examination. For a criterion-referenced test to i
be effective, both the individual test items and the examination in i
total must discriminate between safe and unsafe operator performance.
l b.
Cut Scores l
In some commercially-developed examinations, the system for i
establishing cut scores assumes a fixed test content and variable j
cut scores.
In such a case, the problem is one of finding the 1
optimal value of the cut score variable. However, for NRC examinations the situation is just the reverse. The cut scores (on i
the written examination and JPMs) are fixed at 80 percent; it is the i
content of the examination which varies from occasion to occasion j
because of the plant-specific character of the test material. As is discussed below, there are several reasons why the cut score must be i
{
fixed, including the uniqueness of each examination, consistency, i
and public confidence.
i NUREG-1021 6 of 10 Interim Rev. 8, January 1997 i
l I
I
Appendix A 4
I In the writing, reviewing, setting of scoring standards, and grading of any particular NRC examination, both the examination author and the reviewer are well aware of the NRC-established passing score of j
80 percent. They may also have knowledge of how prior examinees have done on questions similar to the ones being used on the i
examination under construction and expectations as to how a qualified or an unqualified applicant should do on the examination.-
They must use this knowledge to control the nature and difficulty of the examination such that an examinee who is deemed to be qualified i
scores above the passing grade and an examinee who 10 deemed to be j
unqualified scores below this grade.
The traditional cut score on the examination should not be viewed as arbitrary. Rather, it reflects a point on the test at which author i
and reviewer judgment separates the qualified from the unqualified.
l To be sure, this judgment is implicit rather than explicit.
Nonetheless, the judgment is probably similar to other methodologies for determining passing test scores. For example, rather than j'
exp1fef tly judging the probability that a minimally qualified applicant will pass an item, the author is fspf fef tly being asked to write an examination on which, in the author's judgment, the minimally qualified applicant will obtain a score of at least 80 percent. Achieving this objective requires the author and reviewer j
to integrate their content and process skills.
\\
2 c.
Cut Scores and the Level of Difficulty For the cut score of 80 percent to be meaningful requires that individual test items be written "near" that level; a target range of 70 to 90 percent level of difficulty is recommended. Test items that are so difficult that few if any of the examinees are expected to answer correctly do not discriminate and should not be used on an NRC examination. Test items that are so easy or fundamental that even those examinees who are known to have perfomance problems will be able to answer correctly should be used with discretion. Despite the best intentions, it is expected that every examination will contain some test items that all or most of the examinees will answer correctly or incorrectly. This does not necessarily mean that the test items or the examination are invalid.
It should be stressed that the intent is not for everyone to get a.
score of 80 percent. In fact, historically over 90 percent of examinees score 80 percent or above on the NRC examinations. A score of 80 percent is the minimal pass score that the author and reviewer must keep in mind as a functional level of discrimination for setting item difficulty.
In order to achieve this, the test author must keep in mind and integrate the following concepts:
NUREG-1021 7 of 10 Interim Rev. 8, January 1997
l 1
i J
APPENDIX E POLICIES AND GUIDELINES FOR TAKING NRC EXAMINATIONS i
Each examinee shall be briefed on the policies and guidelines applicable to the examination category (written and/or operating test) being administered.
f The applicants may be briefed individually or as a group.
Facility licensees are encouraged to distribute a' copy of this appendix to every examinee before the examinations begin. All items apply to both initial and requalification examinations, except as noted.
PART A - GENERAL GUIDELINES i
i 1.
[#ead Fer6atisJ Cheating on any part of the examination will result in a denial of your application and/or action against your license, j
i 2.
If you have any questions concerning the administration of any part of the examination, do not hesitate asking them before starting that part i
of the test.
4.
SRO applicants will be tested at the level of responsibility of the i
senior licensed shift position (i.e., shift supervisor, senior shift supervisor, or whatever the title of the position may be).
5.
You must pass every part of the examination to receive a license or to continue perfonsing license duties. Applicants for an SRO-upgrade license may require remedial training in order to continue their R0 duties if the examination reveals deficiencies in the required knowledge and abilities.
4 6.
The NRC examiner is not allowed to reveal the results of any part of the i
examination until they have been reviewed and approved by NRC i
management. Grades provided by the facility licensee are preliminary j
until approved by the NRC. You will be informed of the official examination results about 30 days after all the examinations are complete.
I PART B - WRITTEN EXAMINATION GUIDELINES i
j 1.
[Aead Fer6atis] After you complete the examination, sign the statement j
on the cover sheet indicating that the work is your own and you have not j
received or given assistance in completing the examination.
2.
To pass the examination, you must achieve a grade of 80.00 percent or greater.
Every question is worth one point.
3.
For an initial. examination, the time limit for completing the examination is four hours.
For a requalification examination, the time limit for completing both sections of the examination is three hours.
If both sections are NUREG-1021 1 of 5 Interim Rev. 8, January 1997
3 I
i' i
j Appendix E e
administered in the simulator during h single three-hour period, you may return to a section of the examination that was alreartv completed or retain both sections of the examination unM1 the allotted time has i
expired.
i i
4.
You may bring pens and calculators into the examination room. Use only black ink to ensure legible copies.
l 5.
Print your name in the blank provided on the examination cover sheet and i
the answer sheet. You may be asked to provide the examiner with some form of positive identification.
i i
6.
Mark your answers on the answer sheet provided and do not leave any question-blank. Use only the paper provided and do not write on the i
back side of the pages.
If you decide to change your original answer, draw a single line through the acror, enter the desired answer, and 4
i initial the change.
J 7.
If the intent of a question is unclear, ask questions of the NRC examiner or the designated facility instructor only.
i l
8.
Restroom trips are permitted, but only one applicant at a time will be j
allowed to leave. Avoid all contact with anyone outside the examination j
room to eliminate even the appearance or possibility of cheating.
9.
When you complete the examination, assemble a package including the 2
examination questions, examination aids, answer sheets, and scrap paper and give it to the NRC examiner or proctor. Remember to sign the statement on the examination cover sheet indicating that the work is 1
your own and that you have neither given nor received assistance in i
completing the examination. The scrap paper will be disposed of j
immediately after the examination.
10.
After you have turned in your examination, leave the examination area as defined by the proctor or NRC examiner.
If you are found in this area while the examination is still in progress, your license may ba denied or revoked.
11.
Do you have any questions?
PART C - GENERIC OPERATING TEST GUIDELINES (CATEGORIES A. B. AND Q l.
If you are asked a question or directed to per form a task that is unclear, you should not hesitate to ask for clarification.
2.
The examiner will take notes throughout the test to document your performance, and sometimes the examiner may take a short break for this reason. The amount of note-taking does not reflect your level of NUREG-1021 2 of 5 Interim Rev. 8, January 1997
i i
4 i
APPENDIX F j
GLOSSARY j
Achievement test: An instrument designed to measure a trainee's grasp of some i
body of knowledge or skill proficiency.
i j
8Danti:
In most instances, a period of time equal to 365 days reckoned from j
any point in a calendar year to the same point in time in the following i
calendar year. However, annual requirements in successive years can reach a j
period of nearly two years. Annual could encompass a range extending to 729 days depending on when an event occurred in the first calendar year and viewing December 31 of the following calendar year as meeting the annual l
requirement.
I Anolicant: Any individual who has submitted an NRC Fom 398 in pursuit of an RO or SRO license. For purposes of this and the other Examination Standards, it is synonymous with " candidate."
Annlicant license level: The level of operator license (i.e., R0 or SRO) for which the applicant has applied.
Antitude test: An instrument designed to assess an individual's potential for performing some task or skill area.
Averace: A score that provides an indication of the typical performance of a group of scores. The mean, median, and mode of a distribution of scores are all commonly used as averages.
Biennial:
In most instances, a period of time equal to 730 days and synonymous with the term "two years." Biennial requirements can extend beyond 730 days if the requirement is met during the anniversary month of the second year. For example, a biennial medical examination last performed on January 10, 1995, would be due again by January 31, 1997. January is seen as the anniversary month, the period of time between the two examinations is longer than 730 days, but the biennial requirement is satisfied.
Bloom's Taxonomy: A classification system that depicts knowledge and information processing of knowledge in a Merarchy from lowest to highest as follows: fundamental knowledge, comprehension, analysis, synthesis, and evaluation.
Calendar auarter: One of four parts of a calendar year, each consisting of a 3-month segment.
In any calendar year, the first quarter is from the first day of January to the last day of Narch, the second quarter is from the first day of April to the last day of June, the third quarter is from the first day of July to the last day of September, and the fourth quarter is from the first day of October to the last day of December.
Cateaory: One of 3 major subdivisions of related subjects on the operating test. Refer to Section D of ES-301 for a description of and detailed instructions for developing each operating test category.
NUREG-1021 1 of 6 Interim Rev. 8, January 1997
i i
Appendix F l
1 Central tendency: A term referring to the most typical performance of a group of individuals; generally the mean, median, or mode Coanitive: Aspects of a person or test level that refer to knowledge or j
understandino.
)
Content validitv: The degree to which a test measures the specific objectives or content of that test.
l Correlation coefficient: A numerical value ranging from -1 to +1 that j
indicates the relationship between two sets of scores or other measures of each individual in a group. A value of 0 indicates no relationship; +1 or -1 indicates a perfect relationship, either positive or negative.
Criterion: A characteristic or combination of characteristics used as the i
basis for judging a performance.
l Criterion-referenced test: An examination based upon mastery of objectives of
(
content that was or should have been taught and mastered and one that uses an established standard or cutoff score as a measure of acceptable performance.
Cut score: The score at which a trainee is deemed to have met the criteria on an exam.
l Diaanostic test: An instrument that is designed to identify the strengths and weaknesses of an individual for a given content area.
Difficulty index: A numerical index ranging from 0.00 to 1.00 that indicates the percentage of trainees who answer a test item correctly. An index of 0.00 indicates that no one answered the test item correctly while an index of 1.00 indicates that all individuals answered the item correctly.
Discrimination index: A measure of a test item's ability to differentiate between good and poor trainees. A high discrimination index indicates that more high performers than low performers answered the item correctly (high and low are typically determined by overall test scores but may also be established by external criteria).
Discrimination validity: Setting the ites difficulty at an estimated level around the cut score.
Distractor: An incorrect alternative among the choices of a test item.
Error of measurement: Any difference between an obtained score and a true score on a test is referred to as error of measurement. The actual error of measurement can only be estimated since it is impossible to know what the true score is.
I NUREG-1021 2 of 6 Interim Rev. 8, January 1997
i i
UNITED STATES OF AMERICA 00C ETED NUCLEAR REGULATORY COMMISSION 0
pC l
BEFORE THE ATOMF SAFETY AND LICENSING BOARD 97 MAR 10 P4 :49 i
In the Matter of
)
)
Docket No. 55-20726-SfFFIC RALPH L. TETRICK
)
)
BRANCH (Denial of Senior Reactor
)
i Operator License)
)
CERTIFICATE OF SERVICE i
i I hereby certify that copies of (1)
"NRC STAFF'S MOTION FOR RECONSIDERATION" and (2) "NRC STAFF'S REQUEST FOR ISSUANCE OF AN ORDER STAYING THE EFFECTIVENESS OF THE PRESIDING OFFICER'S INITIAL DECISION (LBP-97-2)" in the above-captioned proceeding have been served on the following by deposit in the United States mail, first class, or as indicated by an j
l asterisk through deposit in the Nuclear Regulatory Commission's internal mail system, 4
this 10th day of March 1997.
4 Peter B. Bloch, Presiding Officer
- Adjudicatory File * (2)
Administrative Judge Atomic Safety and Licensing Board Atomic Safety and Licensing Board Panel Mail Stop: T-3 F23 Mail Stop: T-3 F23 U.S. Nuclear Regulatory Commission U.S. Nuclear Regulatory Commission i
Washington, D. C. 20555 Washington, D. C. 20555 j
Fax: 301-415-5599 Atomic Safety and Licensing Board ~
Dr. Peter S. Lam
- Panel
- Administrative Judge Mail Stop: T-3 F23 Atomic Safety c.nd Licensing Board U.S. Nuclear Regulatory Commission Mail Stop: T-3 F23 Washington, D. C. 20555 U.S. Nuclear Regulatory Commission Washington, D. C. 20555 Office of the Secretary * (2)
Attn: Docketing and Service Branch Mr. Ralph L. Tetrick Mail Stop: OWFN-16/G15 18990 SW 270 Street U.S. Nuclear Regulatory Commission Homestead, FL 33031 Washington, D. C. 20555 Office of Commission Appellate Adjudication *
/
Mail Stop: OWFN-16/G15 6/C y
U.S. Nuclear negulatory Commission Washington, D. C. 20555 Sherwin E. Turk Counsel for NRC Staff et