ML20216F809
| ML20216F809 | |
| Person / Time | |
|---|---|
| Issue date: | 08/30/1996 |
| From: | Russell W NRC (Affiliation Not Assigned) |
| To: | Morrison D NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES) |
| Shared Package | |
| ML20216F483 | List: |
| References | |
| FRN-62FR42426, RULE-PR-55 AF62-1-026, AF62-1-26, NUDOCS 9709120113 | |
| Download: ML20216F809 (1) | |
Text
)
A C & 2. - I f
- osauq g'
t UNITED STATES ppg s
j NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20565-0001
,s.,...../
August 30, 1996 MEMORANDUM 10:
David L. Morrison, Director Office of Nuclear Regulatory Research FROM:
William T. Russell, Director gA///f4-
}
Office of Nuclear Reactor Regulation
SUBJECT:
RULEMAKlHG PLAN - CHANCES 10 10 CFR 55 -
AMENDING INITIAL OPERATOR LICENSING EXAMINATION REQUIREMENTS 3
}
As requested by your memorandum dated August 26, 1996, NRR has reviewed the subject rulemaking plan and concurs with the plan as revised in the attached markup.
The most significant revision is to delete reference to NUREG-1021 in the
}-
wording of the draft rule. This revision was based on a discussion of the issue between staff from OGC, RES, and NRR.
If there are questions, please contact Stuart Richards, Chief, Operator
$g Licensing Branch, NRR at 415-1031.
t
Attachment:
As stated t
g4*.g113970504 SS 62FR42426 PDR j
m.
.m*
4%a 1
-e "ln Clo
[
}*,
UNITED STATES y%
j j
NUCLEAR REGULATORY COMMISSION y
t WASHINGTON, o.o. 20060 0001
/
August 26, 1996 MEMORANDUM TO:
James Lieberman, Director Office of Enforcement William J. Olmstead, Associate General Counsel for Licensing and Regulation Office of the General Counsel William T. Russell, Director Office of Nuclear Reactor Regulation FROM:
David L. Morrison, Director Office of Nuclear Regulatory Research -- gg pp
SUBJECT:
RULEMAKING PLAN - CHANGES TO 10 CFR 55 - AMENDING INITIAL OPERATOR LICENSE EXAMINATION REQUIREMENTS Your concurrence is requested on the attached rulemaking plan package. This action is on the Chairman's tracking list with an aggressive schedule, therefore, please respond promptly.
Backaround:
Section 107 of the Atoirf: Energy Act of 1954, as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Current regulations in Part 55 dealing with initial license examinations for reactor operators (RO) and senior reactor operators (SRO) do not state who will write, administer, or grade the examination; however, the NRC has traditionally performed those tasks itself or through contract examiners. On March 24, 1995, the staff, in SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program," informed the Commission of its intent-to revise the aperator licensing program to allow greater participation
-by facility licensees and to eliminate contractor assistance in this area.
This action will allow staff to elimini..e between 33M and $4M annually on contractor support for examination preparation and administration. On April 18, 1995, the Commission approved the staff's proposal to initiate a transi-tion process to revise the operator licensing program, and directed the staff to carefully consider experience from pilot examinations before full implemen-tation. On August 15, 1995, the staff issued Generic Letter (GL) 95-06,
" Changes in the Operator Licensing Program," outlining the revised examination development process and soliciting volunteers to participate in pilot examint.-
tions to. evaluate ant refine the methodology.
\\
4 9
e f m e9ohgg.g*-
6 3 gejam43.-
eaes.
i ff C
M - [G l Yfh _
4
'J. 1.ieberman et at.- ',
Et Between October 1, 1995 and April 5, 1996, the NRC conducted a pilot program in which it reviewed and approved 22 operator licensing examinations that~ had been written by facility licensees in accordance with the current examination development guidance (NUREG-1021, Rev. 7, Sup?lement 1), as supplemented by the' pilot examination guidance contained in G 95-06. The staff described the results of the pilot examinations in SECY-96-123, " Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996, and briefed the Comission during a public meeting on June 18, 1996. Based on the results of the pilot program, the staff recommended that the Comission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to draft the operator licensing examinations in accordance with NUREG-1021, " Operator-Licensing Examination Standards for Power Reactors." In a staff requirements memorandum (SRM) dated July 23, 1996, the Comission directed the staff to develoo a detailed rulemaking plan to justify the changes that may be neces-sary to 10 CFR Part 55. The SRM also directed the staff to specifically address-several issues about the pilot examination program and the proposed changes in the examination process.
The rulemaking plan and the response to the SRM's specific requests, which were prepared by the Office of Nuclear Reactor Regulation and reviewed and approved by the Office of Nuclear Regulatory Research, are attached to the enclosed Comission Paper One issue I would appreciate specific coment on is that of whether the rule should contain an explicit reference to NUREG-1021.or alternately require an operator licensing program acceptable to the NRC.
If the former course is taken that will have the effect of codifying the NUREG and if we wish to change it'later based on experience with implementing the new rule a rulemaking would be raquired, if the later course is taken a regulatory guide would be issued referencing the NUREG and it would only be necessary to change the regulatory guide.
The following is a sumary of this request:
1.
Title:
Rulemaking Plan, " Changes to the Operator Licensing Process at Power Reactors (Part 55)."
3 2.
Task leader:
Harry S. Tovmassian (415-6231) 3.
Steerina Group Members: None 4.
Enhanced Public Participation:
This rulemaking will use electronic balletin boards, as appropriate to
' enhance input from the public.
5.
Level 1 or 2 Item of Compatibility for Aareement States: No 6.
Reouested Action: Office Concurrence y-
I l
J. Lieberman et at. l 7.
Reauested Completion Date:
August 30, 1996 8.
Resources and Coordination:
Estimated resources to develop this rulemaking are.5 FTE. No contractor support is needed. A copy of this concurrence package has been forwarded to the Office of the Controller for coordination of resource issues and to the IG for information.
Attachment:
Memorandum to Taylor from Morrison w/enclr.
cc w/ attachment:
R. Scroggins, OC H. T. Bell, OIG G. Cranford, IRM D. Meyer, ADM i
w l
- f un 1
UNITED STATES NUCLEAR REGULATORY COMMISSION 3
- 'l waswinotou o.c. mwoooi
%.....)
MEMORANDUM 10:
James M. Taylor Executive Director for Operations fROM:
David L. Morrison, Director-Office of Nuclear Regulatory Research
SUBJECT:
RULEMAK'NG PLAN - CHANGES T0 10 CFR PART 55 - AMENDING INITIAL OPERATOR LICENSE EXAMINATION REQUIREMENTS Attached for your signature is a Comission paper which requests the Commission's approval of the staff's rulemaking plan for requiring power
. reactor licensees to prepare and administer initial reactor operator (RO) and senior reactor operator-(SRO) licensing examinations. Section 107 of the Atomic Energy Act of 1954, as amended, requires the NRC to determine the qualifications. f individuals applying for an operator license, to prescribe uniform contlitions for licensing such individuals, and to issue licenses as appropriate. Current regulations in Part 55 dealing with initial license examinations for R0s and SR0s do not state who will prepare, administer, or
-grade the examination; however, the NRC.has traditionally performed those tasks itself or through contract examiners. On March 24, 1995, SECY-95-075,
" Proposed Changes to the NRC Operator Licensing Program," informed the Commission of the staff's intent to revise the operator licensing program to cllow greater participation by facility licensees and to eliminate contractor assistance in tht; area. This action will allow staff to eliminate between
$3M and $4M annually on contractor support for examination preparation and administration. On April 18, 1995, the Commission approved the r+aff's proposal to initiate a transition process to revise tne operator licensing program, and directed-the staff to carefully consider experience from pilot examinations before full implementation. On August 15, 1995, the staff issued Generic letter (GL) 95-06, " Changes in the Operator Licensing Program,"
outlining the revised examination development process and soliciting volunteers to participate in pilot examinations to evaluate and refine the methodology.
Between October 1, 1995 and April 5, 1996, the NRC conducted a pilot program in which it reviewed and approved 22 operator licensing examinations that had been prepared by facility licensees in accordance with the current examination development guidance (NUREG-1021. Rev. 7, Supplement 1), as supplemented by the' pilot examination guidance contained in GL 95-06. The staff described the results of the pilot examinations in UY-96-123, " Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996, and briefed tnb Commission during a public meeting on June 18, 1996. Based on the results of the pilot program, the staff recommended that the Commission approve the implementation of. the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to prepare the operator licensing examinations in accordance with FUREG-1021, " Operator
+
f :.
ll J. M. Taylor Licensing Examination Standards for: Power Reactors." In a Staff Requirements-
- Memorandum (SRM) dated July 23, 1996, the Commission directed the staff to develop.a detailed-rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55. The SRM also directed _the staff to specifically address several issues about the pilot-examination program and the proposed
- changes in the examination process. The attached Commission Paper contains both the staff's rulemaking plan and a response to the issues raised in the SRM.
Attachment:
Commission Paper w/encls.
I w--
J. M. Taylor 2-Licensing Examination Standards for Power Reactors."
In a Staff Requirements
~ Memorandum (SRM) dated July 23, 1996, the Comission directed the staff to develop a detailed rulemaking plan to justify the changes that may be-necessary to 10 CFR Part 55. The SRM also directed the staff to specifically address several issues about the pilot examination program and the proposed changes in the examination process. The attached Commission Paper contains both the staff's rulemaking plan and a response to the issues raised in the SRM.
Attachment:
Comission Paper w/encls.
DISTRIBUTION:
Central f/c RDB r/f
-RAuluck DMendiola CGallagher LRiani DOCUMENT NAME:0:\\TOVMASSI\\TRANSMEMO.EDO
/
I kkhW(
D:RES [i OFC RDB:DRI RDB:DRA n _h D
f NAME HTovmassian:ayw SBahadur h BMorris DMorrkon!
9 / 6/96 I
[/M/96 7
/ d/96 4/9y96 DATE
/
0FFICE RECORD COPY RES FILE CODE:
c-
+.
--A.mi.-..,wJ-ss.s a
e a
.4.a.a.,,ua 4
a a
A.,.,
,.s,g4
., y a
a pn__.ausaw,_
n_,.-,---y.___.-
_s 3
1 4
T COMMISSION PAPER i
i t
i 4
o i.
e
c f.QB:
_The Commissioners i
faQM:
James M. Taylor, Executive Director.for Operations
-l
SUBJECT:
RULEMAKING PLAN FOR AMENDMENTS TO 10 CFR PART 55 TO CHANGE LICENSED OPERATOR EXAMINATION REQUIREMENTS RUREQlf:
in pc4 preAce Ae op hae c+bM h _
'rmation regarding the initia' operator licensing To provide additional inf o examination pilnt process as directed in a Staff Requirements Memorandum (SRM) of July 23, 1996 (M96061EA), and to request the Commiss ion's approval of the rulemaking plan to amend 10 CFR Part 55 to require pows r reactor facility licensees to. prepare and adminhter in' accordance with NUREG-1021, " Operator Licensing Examination Standards for Power Reactors," the written examinations i
and operating tests-that the NRC uses to determine the ccmpetence of operator license applicants.-
BACKGROUND:
On March 24, 1995, SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program," informed the Commission of the staff's intent to revise the operator licensing program to allow greater participation by facility licensees and to eliminate contractor assistance in this area. This action-
.g * -
will allowistaff to eliminate between-53M and.54M on contractor support for-examination preparation and administratiM. On April 18, 1995, the Commission approved the staff's proposal to initiate a transition process to revise the 1
operator licensing program and directed-the staff to carefully consider
- experience from pilot examinations before full implementation of the changes.
On August 15, 1995, the staff issued Generic letter (GL) 95-06, " Changes in the Operator Licensing Program," outlining the revised examination development procels and soliciting _ volunteers to participate in pilot examinations to evaluate and refine the methodology.
~
CONTACT:
Stuart-Richards,-NRR (301) 415-1031
~.. -
i-g1 k-4 The Comissioners SM-Between October 1,1995, and ' April 5,1996, the #RC-reviewed and approved -22
-operator licensing examinations that had-been prepared by facility licensees in-accordance-with the current examination development guidance (NUREG-1021 Rev. 7 _ Supplement 1) as supplemented by the pilot' examination guidance contained in GL 95-06. These examinatiu s were then used to test 146 reactor operator and senior reactor operator applicants.
Facility licensees-from each NRC Region, operating reactors built by all power reactor vendors, participated in;the voluntary pilot program.
The staf f described the results of the pilot examinations in-SECY-96-123,
" Pro)osed thanges to the NRC Operator Licensing Program," dated June 10, 1996, and 3riefed the Comission during a public meeting on June 18, 1996. On the basis of the results of the pilot program,- the staff recomended that the Comission approve the implementation of the new examinatic:1 process on a voluntary basis until rulemaking could be completed to require all power
- pbc or__ facility licensees to prepare the operator licensina examinations in
_ react _ dance wFtnl NUREG-1021
" Operator Licensing Examination Standards for ve* 4 -
accor
' m Power Reactors," which has been revised to-accomodate the new process.
In an SRM dated July 23, 1996, the Commission authorized the staff to continue the pilot process through July 1997 and directed the staff to develop a detailed rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55. The SRM also directed the staff to specifically address several issues regarding the pilot examination program and the proposed changes in the examination process.
g g g j3,m y.g DISCUSSION:
T^ "# "
- F$ A~ ~ ~
y s wn gee Section 107 of the Atomic Energy Act of 1954 as amended, requires the hRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55-to pass an examination satis "ying the basic content requirements specified in the regulation.
) art 55 does not state who must
. prepare, administer, or grade the examinati )n: however, the NRC has traditionally performed those tasks itself
)r through contract examiners.
On the basis of the results of the pilot pr
> gram as discussed in SECY-96-123, the staff now believes that 10_CfR Pa-55 ihould be revised to require power reactor facility licensees to prepare their initial operator licensing examinations. Specifically, the staff recommends that Subpart E, " Written Examinations _and Operating Tests," of 10 CFR Part 55 be amended to require all
- 3 * * ~powtr reactor facility licensees to prepare ithe initial operator _ licensing u>d--
examtiatlons in icTo7dlinM wHh1 NUREG-1021." The licensee-prepared
~
examinations will be subject to review, modification, and approval by NRC examiners before the examinations are administered. The regulation-will provide that:the NRC may prepare the examinations in lieu-of-accepting or modifying an examination prepared by the. facility licensee. The changes would not apply to non-power reactor licensees, Theh-nlemakinq plan for this proposed shangs-has-beEn'd'Teloped in i
e
[accordancewith
" N i -- ?
.3, "The Rulemaking Q ian p
Process " E
_ he SRM of July 23, i
I i
The Commissioners.
The enclosed rulemaking plan for this proposed change has been developed in accordance with the guidance in NRC Management Directive 6.3, "The ".ulemaking Process." As directed in the SRM of July 23, 1996, the rulemaking plan addresses the potential impact of the proposed rule on facility licensees and discusses why this rule will not constitute a backfit.
The second enclosure discusses the fellowing additional items as directed by the SRM:
The pros, cons, and vulnerabilities associated with implementing the e
proposed examination program changes on an industry-wide basis (including a review of the impact on licensee and NRC resources, the potential effects on reactor safety, the impact of proposed NUREG-1021 revisions, potential public perceptions, and concerns regarding this approach raised by staff members).
A discussion of the degree of acceptance of this new approach to operator licensing by facility licensees (including a discussion of industry comments received on draft NUREG-1021 and an explanation of the staff's response to those comments),
e An explanation of enhancements to the new licensing examination process resulting from the review and feedback from the pilot prograrr results (including a justification for subsequent proposed changes to the operator licensing program).
The results of initial operator license examinations that have been given both under the pilot program and under the current program since SECY-96-123 was prepared (including trends observed).
RECOMMENDATION:
That the Commission --
1.
Approve the development of the rule as described in the enclosed Rulemaking Plan.
2.
Approve the implementation of Revision 8 of NUREG-1021 on a voluntary basis until the rulemaking is complete.
I The Commissioners.
C00RDINATIQ!i:
L ngo :u,*,g 3
The Office of the deneral Counsel has no legal objection to this initiative, and the Offices of Nuclear Reactor Regulation and Nuclear Regulatory Research concur in this rulemaking plan.
James M. Taylor Executive Director for Operations
Enclosures:
1.
Rulemaking Plan for Changes to 10 CFR Part 55 2.
Response to SRM dtd July 23, 1996 (M960618A)
)
4 5
.The Ct.mmissioners.
-q.
"'I COORDINATION:
The Office of the General Counsel has no legal objection to this initiative, and the Offices of Nuclear Reactor Regulation and Nuclear Regulatory Research concur in this rulemaking plan.
James M. Taylor Executive Director for Operations
Enclosures:
1.
Rulemaking Plan for Changes to 10 CFR Part 55
- 2. -Response to SRM dtd July 23, 1996 (M960618A) f 1
Distribution:
RDB r/f Central file E00 r/f LRiani CGallagher DOCUMENT NAME: 0:\\T0VMASSI\\RULEPLAN.FNL T.
. 4.
w.
. m in wm e. m 4.n-pm.%A<tw.A. E. co v.,m.n-sm.neww.or.
u - wo co.,
l D RES
////J __j EDO l
OFFeCE RDSDRA Mf f' l
DRA:RDS C
QQM V~
l waw srovm. n.v.
e $hwur m,n. -
w,=on f 57/
aT.vio, Datt irvas g pee 7thes
( ge v
i ese OFFICIAL RECORD COPY
h
(
- j 9
RULEMAKING PLAN FOR CHANGES TO 10 CFR PART 55 ENCLOSURE 1
- ~ ~ - - - - - - - - - _ _ _. _ _ _ _ _ _ _ _ _. _ _ _ _ _ _ _
i RULEMAKING PLAN CHANGES 10 10 CFR 55 AMENDING INITIAL OPERATOR LICENSE EXAMINATION REQUIREMENTS Lead Office:
Offic, of Nuclear Regulatory Research Staff
Contact:
H. Tovmassian, RES/RDB 2[f Concurrences:
./fr/
//W
' Dale /
D. MoVPison, RES W. Russell, NRR Date J. Lieberman, OE Date R-Olmstead, OGC Date Approval:
J. Taylor Date
d RVLEMAKING PLAN CHANGES TO THE OPERATOR LICENSIEG PROCESS AT POWER REACTORS (10 CFR PART 551 1.
Backaround On March 24, 1995, SECY-95-075, " Proposed Changes to the N RC Operator Licensing Program," informed the Commission of the staff's intent to revise the operator licensing program to allow greater parjicipation by facility licensees and to eliminate NRC contractor assistance in this area. This action will allow staff to eliminate between $3! and $4M on contractor support for examination preparation and administration. On April 18, 1995, the Co,wission ap poved the staff's proposal to initiatt.
a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full implementation of the changes. On August 15, 1995, the staff issued Generic Letter (GL) 95-06, " Changes in the Operator Licensing Program," which outlined the revised examination development process that would permit facility licensees to prepare the initial operator licensing examinations. GL 95-06 also solicited volunteers to participate in pilot examinations to evaluate and refine the methodology.
p yhg Between October 1, 1995, and April 5, 1996, the NR: conducted 22 initial operator licensing examinations using the new methadology. The staff described the results of the pilot examinations in SECY-96-123,
" Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996, and briefed the Commission during a public meeting on June 18, 1996. On the basis of the results of the pilot program, the staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be ensees to prepare the completed to require all power reactor facility li[NUREG-1021, " Operator operator licensing examinations in accordance with Licensing Examination Standards for Power Reactors." In a staff requirements memorandum dated July 23, 1996, the Commission directed the staff to develop a detailed rulemaking plan to justify the changes that may be r.ecessary to 10 CFR Part 55.
II.
Prooosed Change Revise Subpart E " Written Examinations and Operating Tests," to include examination development requirements for facility licensees.
Reaul atory Issue The pilot program demonstrated that the revised examination development process can be both effective and efficient. Comments from the NRC staff and industry personnel who participated in the pilet examinations were generally favorable. The quality of the administered examinations (as modified by the NRC) and the performance of the individuals who took the examinations were comparable to the quality of and the performance 1
on examinations written by the NRC staff or its contractors. The fact that some of the draft examinations submitted by facility licensees required significant rework illustrated that many facility staffs did not fully understand the criteria for writing an NRC examination. Both the industry and the NRC staff agree that, with training and exoerience, the industry should gain proficiency in preparing the examination materials.
The staff believes that the new process should be made mandatory for all power reactor facility licensees. The staff considers implementation of the new process on a voluntary basis alone unworkable over the long term. With the elimination of contractor support, the staff will no longer have sufficient m?iminer resources to write all examinations consistent with the schN aling needs of facility licensees. This resource problem is furtter compounded by the unpredictable nature of the examination workload.
The staff briefed the Committee To Review Generic Requirements (CRGR) af ter the pilot program was complete.
In the minutes of its meeting, CRGR agreed that, with regard to the technical and safety aspects, the-proposed changes represented a reasonable and workable alternative approach. The committee members unanimously endorsed sending the proposal forward to the Executive Director for Operatiom (E00) for consideration. However, the CRGR also requested OGC to consider further the procedural question relating the staff's original plan to implement the revised process by generic letter.
The staff subsequently met with OGC to resolve CRGR's concern and concluded that requiring all facility licensees to prepare the examinations, a task long performed by the NRC, would require rulemaking or the issuance of orders to each facility licensee. Therefore, the staff recommended in SECY-96-123 that the Commission approve the staff's pursuit of rulemaking to require power reactor facility licensees to prepare the operator licensing examinations-in-accordance-win-X.
NUREG-10M, while continuing the pilot process on a voluntary basis.
Current Rule Reouirements Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe un crm conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55 to pass an examination satisfying the basic content requirements that are also specified in the regulation.
Specifically,10 CFR 55.31(a)(3) requires the applicant to submit a written reque? from an authorized representative of the facility licensee that the written examination and the operating test be administered to the applicant. Furthermore, 10 CIR 55.33(a)(2) states that the Commission will approve an initial application for a license if it finds that the applicant has passed the requisite written examination 2
' 7 and operating test in accordance with Sections 55.41 and 55.45 or 55.43 and 55.45. These examinations and tests determine whether the applicant for an operator's license has learned to operate a facility competently and safely, and additionally, in the case of a senior operator, whether the applicant has learned to direct the licensed activities of licensed operators competently and safely.
Subpart E, " Written Examinations and Operating _ Tests," includes specific sampling requirements for the content of written examinations (10 CFR 55.41 and 55.43) and operating tests (10 CFR 55.45).
Although Part 55 is silent regarding who will prepare, administer, and grade the written examinations and operating tests, the NRC or its contract examiners have traditionally performed those tasks.
Preliminary Reaulatory Analysis 1.
Draft Rule Lanauaae The first sentence in Section 55.33(a)(2) would be revised as follows (new langtage is underlined):
"The applicant has passed the requisite written examination and operating test in accordance with Sections 55.40. 55.41, and 55.45 or 55.40. 55.43, and 55.45.
Section 55.40, " Implementation," would be added to Subpart E as follows:
y.L e 6<
1-7pA pn',.oenad kn.
"(a) Power reactor facility licensees shall prepare 2nd :d+! ster...'
therequiredsite-specificwrittenexaminationandoperatingtestsM in :::crdance-with-the volicies-and-guidelines-in-NUREG-1021,
' Operator Meensing-Examination-Standards for-power--Reactors,".
(b) The Caissien W14-review,-modify-as-necesscryrand-approve-in acc0rd:nt: 4th 4WREbl 021r-t he-wr i t t en-e x ami na t i on s - and -ope ra t i ng 4e:ts prepared by the f4c444ty44censees.
In 14eu-of-accepting the w4 tan-examinatten and eparating-tests-prepared by the facility licen:::, th: C:=!:: ten ny prepare-the-examinat4on-and-tests.
C'
) The Commission will prepare and administer the written examinations and operating tests at nor-power reactor facilities."
2.
Imoact of the Rule Chance on Facility Licensees N
The NRC currently depends on NRC employees and contractors to prepare and administer the initial operator licensing examinations required _by 10 CFR Part 55.
In accordance with 10 CFR 170.12(i),
the cost of NRC time spent and any related contractual costs are billed directly to the facility licensees that receive the examination services.
1 3
n (b) b.ar - tmW (.w.% Msen skJ sht & wrmen eummh.s an A ogem.h{ds
% v.s :m..a u.
e w % ~.,
m m p
.a h w s.4 % m,nm %m in.h.3 ed o ped'; rg h3 S.
- * ^
i'^
4 L heo er re.cp,ciny o ye r%Yor C.\\ity\\k " D f*F"C r
tk e.
C.c:ntM.isc,n fn<,,7
\\es t-
- [- W (+'$ c.rd -tm yaJ+ c %si'5tangedW we;thn capin mA',on,
yo - g u r. (sm $he> v._ +O,
{cpwNnc) ewurdf.d',ersa-rd M b
{
D' O
900.E(,,48% hdeskCe*JE ConMo b g wic\\ e,rw e:,, on 8
w e,& o.
)
\\eys M N e
7
-Under the proposed change, each power reactor factaity-licensee will x
assume responsibility for prepari ng the site-specific initial operator licensing examinations at its facilities, thereby-allowing theNRC:todiscontinuetheuseof{contractexaminersforthat facility licensees will prepare and submit pro:osed
- purpose, examinations (including the written examination, the wal(-througN.
h and the dynamic simulator tests) to the NRC in accordance with the cy c.Ak ed intettetion in NUREG-1021. The NUREG has been edited to accommodate the revised examination process and is the same procedure that the NRC examiners or its centractors would use if they were to develop the examinations.
In conformance with Section 107 of the AEA, which requires the Commission to prescribe uniform conditions for licensing operators, the NRC will not c4MMer qM
- alternative testing methodologiespear Wu-> 9 (rom M 4,'o05 6 %.,pk y.ba udaind m 6 4.6 %64.n3 W e,a*'m % h sM M The training staffs at power reactor facilities already have the ogrd.
basic knowledge, skills, and abilities necessary to evaluate operator performance and develop test items for the initial licensing examination.- During the mid-to-late-1980s, the industry's-emphasis in the training-area increased significantly. All power reactor licensees established formal training programs that were based on a systems approach to training (SAT) and accredited by the National Academy for Nuclear Training. Pursuant to 10 CFR 50.120 and 55.4, SAT-based training programs must include the evaluation of the trainee's mastery of training objectives. NRC inspections of licensee requalification programs for licensed operators have also found that training staffs generally possess the skills needed to evaluate the trainee's knowledge.
4 pree After the NRC\\ reviews and approves an examination, the facility licensee williadministee and grade the written portion vr-peesce4bd u
w o h e Q' % in NUREG-1021. The NRC staff will continue to administer and grade 9""
the operating tests, review and approve the written examination results recommended by the facility licensee, and make the final licensing decisions.
Feedback from the pilot examination program indicates that the average time spent by a facility licensee to prepare the written examination and operating tests was approximately 600 to 800 staff-hours. A portion of-that time (about 200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />) would have been spent reviewing and assisting with the administration of NRC-developed examinations under the process now in place and should be subtracted from the total. 'The resulting average burden of approximately 400 to 600 staff-hours was somewhat higher than the 400 hours0.00463 days <br />0.111 hours <br />6.613757e-4 weeks <br />1.522e-4 months <br /> that NRC contract examiners typically take to prepare an examination. The extra burden is generally attributable to the facility licensees' lack of familiarity with ' specific NRC examination expectations and to the additional administrative ree.uirements, such as documenting the source of the examination questions, that are required to maintain examination integrity.
It is worth noting that some of the facility licensees that
-participated in the pilot program expended less time than is 4
i
l
^M commonly used by NRC contractors to prepare the examinations.
Furthermore, in a few cases, the examinations that facility licensees submitted for review and approval were, in the judgment of NRC chief examiners, as good-or better than those prepared by an NRC contractor. The staff expects that most facility licensees will eventually be able to prepare quality examinations in less time than the NRC or a contractor because the facility employees have more-detailed knowledge of their facility and easier access to the l
reference materials required to prepare the examinations.
Experience during the pilot examination program indicated that the-volume of reference materials required by the NRC examiners to review and prepare for the examinations w.s significantly less than that required fci the NRC staff to write the examinations. The f.ct that contract examiners will not be used in the revised examination process will also eliminate the need for duplicate sets of reference materials-to be provided by facility licensees. Feedback from the industry in response-to the staff's solicitation of public comments on the draft revision of NUREG-1021 indicated that facility licensees had been spending from 80 to 160 hours0.00185 days <br />0.0444 hours <br />2.645503e-4 weeks <br />6.088e-5 months <br /> to prepare and ship the reference materials under the existing examination process.
The additional burden of having to prepare the site-specific initial operator licensing examinations should be substantially, if not t
totally, offset by reductions in Part 55 review fees billed to the facility licensees pursuant to 10 CFR 170.12(i). -Each facility licensee will be billed only for the time that the NRC staff spends to review the examination prepared by the facility licensee and to rework the examination, as necessary, to bring it up to NRC standards; no contractual costs will be accrued. Although several of the draft pilot examinations were of poor quality and took the NRC staff more time than expected to review and rework, the staff believes that additional cost reductions will be realized as facility licensees gain experience with the NRC examination requirements and the quality of the draft examinations improves.
This rule change will give facility licensees more control over the cost of their examination-services because they will be in a puttion to manage the quality of the product that is submitted to the NRC. The higher the quality of the examination the facility licensee submits, the lower the resulting charges. Under the existing examination process, facility licensees are responsible for the entire cost of preparing the examination, even if the contractor's submittal is of poor quality and requires significant rework by the NRC staff before it can be administered.
It is also worth noting that facility licensees will have-the option of. retaining the services of a contractor to prepare the license examination as the NRC often does under the current examination process. The staff understands that the NRC's examination contractors have already expressed an interest in providing their services directly to facility licensees.
It-is reasonable to assume 5
4
i x
IY that-the cost of using this-option-would be comparable to the contractor service-fees.that the'NRC currently passes on to the facility licensees.
In summary, the staff views this process change-as resource neutral
.at worst and possibly resulting in a-resource savings-to facility licensees over time.
3.-
Imoact of the Rule Chanae on Operator License Aeolicants The goal of the rule change is to eliminate the NRC staff's reliance on contractor support for the site-specific initial operator licensing program, while complying with the AEA and maintaining the existing level of effectiveness. To the extent possible, the -
format, content, length, and~ level of difficulty of the examinations-will remain unchanged, thereby minimizing the impact of the rule change on'the operator license applicants.
us m
NRC examiners will continue to review and approve every written examination and operating test before it-is administered.,
The examiners will work with the facility licensee's staff to modify the draft examinations as necessary to e a r c^-M iance with'NUREG-1021 b m*Y and to maintain consistency with prior licensing examinations at that facility and among facilities,gAce if the NRC decides to prepare thr examination in lieu of accepting an examination prepared by the freility licensee, the NRC examiners will use the same procedures and criteria (i.e., NUREG-1021)-that
-the facility licensee would have'used to prepare the examination.
The operator license applicants should not be-able to distinguish between an-examination that is prepared by the' facility licensee and one that is prepared by an NRC examiner.
4.
Alternatives to the Rule Chanog If the Commission decides not to amend 10 CFR Part 55 as proposed by the staff, it would require the restoration of contractor funding-for the operator licensing program or the allocation of sufficient direct examiner resources in each regional office to satisfy the demand for initial licensino examinations and to conduct the licensed operator requalificition inspections. Historically, the NRC has spent approximately $3 million to $4 million per year on contractor support funding for the operator licensing program.
111.. Other Elementi
-Leaal Analysis by the Office of-the General Counsel (OGC)
'0GC has: reviewed this rulemaking plan and does not believe this action constitutes a backfit pursuant to 10 CFR 50.109.
In addition, OGC has-concluded that this' action does not constitute a major rule pursuant-to 6
.5
L 1 49 l
l hwh the Small Business Regulatory fairness Act of 1996. Finally, at this time, OGC is not aware of any reason for a legal objection to this rulemaking plan.
3ackfit Analysis As discussed in SECY-96-123, OGC agrees with the staff's position that requiring the facility licensees to prepare the initial operator licensing examinations would not be a backfit pursuant to 10 CFR 50.109, which states in pertinent part that --
Backfitting is defined as the modification of or addition to systems, structures, components, or design of a facility; or the design approval or manufacturing license for a facility; or the procedures or organization required to design, construct or operate a facility; any of which may result from a new or amended provision in the Commission rules or the imposition of a regulatory staff position interpreting the Commission rules that is either new or different from a previously applicable staff position....
The proposed change does not result in a modification of or an addition to systems, structures, components, or the design of a facility. The change does not affect the design approval or manufacturing itcense for a facility. The procedures required to design or operate a facility will not be affected by the proposed change. The proposed change would require each nuclear power plant licensee to develop the tests that are used to qualify, as meeting the requirements of 10 CFR Part 55, those nuclear power plant operators whom the nuclear power plant licensee wishes to employ. Development of such tests are not considered to be
" procedures... required to... operate a facility." The tests are not applied to the facility licensee, but rather to the operator license candidates. Further, any procedure necessary to develop the test would not be useful in actually " operating" the facility, even if one broadly interprets " operating" as including any action necessary to comply with the Commission's regulations with respect to operation. The organization required to design or operate a facility will not be affected because all facility licensees already have a training staff to train and evaluate candidates for operator licenses and to train other members of the plant staff, as required by 10 CFR Part 55 and by 10 CFR 50.120. Therefore an organizational change is not required because of this process change.
Suecortino Documents A regulatory analysis and an Office of Management and Budget (GMB) statement will be prepared. Neither an environmental impact statement nor an environmental assessment will be prepared for this proposed rule, however, since it is the type of action described as a categorical exclusion-in 10 CFR 51.22(c)(1).
7
e-
$
- 4*
3
'd>[4*
/////
O4p IMAGE EVAL.UATION TEST TARGET (MT-3) e
+
1.0 lg au m 5!!!E i,l
[m RM l.25 1.4 1.6
~ ~
4 150mm 4
6" 4
- D;f&
I
' xxxxf 4
+
4 p
PHOTOGRAPHIC SCIENCES CORPORATION P.O. BOX 338
,g;%
WEBSTER, NEW YORK 14580 4
(716) 265-1600
/
sq>%+k) e)
/////
gb 9
IMAGE EVALUATION 4
g TEST TARGET (MT-3) 6 e!g 1.0 lf W Bla "ljIL??
i i,i
[u HM J1
~7 1.25 l.4 1.6 4
150mm 4
6" 4
4k?%,,
/<!b
% g y p,,,/,
_ _ =,,,, _ %,,%.3,
3 WEBSTER NEV/ YORK 14580 (716) 265 1600
h
- 4) +N+&
3 5' h
(+b IMAGE EVALUATION 4
g res11Amee1<m1.a>
1.0 Em W pljE4 l'i E im H M 1
1 I
.8 I.25 1.4 1.6
~ ~ ' '
4 150mm 4
6"
/A ii
+++gt <
+<%& $)F o
,,0.
P.O. BOX 338 WEBSTER, NEW YORK 14580 4
(716) 265 1600
s-h 9 *$r
////
($ k g t 'A IMAGE EVALUATION
/
gf TEST TARGET (MT-3) k u l.0 Em M y @ Ela 1.1
[" EE l.8 l=
' l.25 1.4 1.6
==
==
4 150mm 6"
4 d#%
- 4Nhp
%,Q;/O
=,,u - -
WEBSTER hE N YORK 14580 (716) 265 1600
4# Q pp*,,g@ *sh>
b
((p
' g k, IMAGE EVALUATION
// 4,
"[ f[4'
- g "9
TEST TARGET (MT-3) g9 g777p
+
r I.0 lt M BLA
- sljDE E in R M l_4l l=_
Lg
'v-i.25 u
t6 4
150mm 4
6"
[%*%
/+'O
+$1 %,,,
+ Wb
'N o *[b4 <>p PHOTOGRAPHIC SCIENCES CORPORATION o
,,ee,m eO o P.O. BOX 338 4'
WEBSTER, NEW YORK 14580 4
(716) 265 1600
'N--
b;s)He ed Ptss,o sh.6 Menasb J A.a Mim d
- e. so m.3 bh py" %
du dd.\\ '.ond
@t %unee s 4e b e. 3d ke3, b h h m i sn eno.sete. 2 ' a n 3 m c,3"
1 aced a-premium on accuracy because the answers to the questions could not >e adjusted after the fact as they could be with the essay and short answer nuestions that
- they replaced.
To maximize security, given the new philosophy, NUREG-1021 prohibits.ny facility employee with specific knowledge of any NRC examination before it--is -
given from communicating the examination contents to unauthorized individuals and from participating in any further instruction of the students scheduled to take the examination. Before they are given access to the examination, the facility employees are required to sign a~ statement acknouledging their-understanding of the restrictions and the potential consequences of noncompliance. They must-also sign a post-examination statement certifying that they did not knowingly compromise the examination.
The staff realized that the potential for examination compromise would l
increase as the facility licensees increased their role in the initial licensing process from one of reviewing the NRC-developed examinations to that of developing their own examinations. Therefore, the staff attempted to clarify its security expectations in the supplemental pilot examination guidance that was provided to the regional offices (and placed in the POR) in conjunction with GL 95-06. The guidance addressed the' restrictions on personnel and a number of physical securf ty precautions, including protecting and mailing the examination materials. Ine guidance also cautioned examiners to be attentive to examination security measures and required them to review the security requirements with the facility licensee's contact at the time the examination arrangements were confirmed.
The issue of examination security also had a bearing on the methodology that the staff selected for administering the pilot operating tests. The staff believed that Option I would pose the lowest threat to examination security because the smallest number of facility employees would require detailed knowledge of the examination before it was given. Options 2 and 3 both would have required the facility licensee to evaluate the license applicants during the _ simulator and walk-through tests, which could have required several additional personnel to familiarize themselves with the operating test content, thereby increasing the risk of compromise.
In summary, the staff expects that the vast majority of facility licensees will continue.to maintain proper examination security (as they have during the requalification program). -Security is enhanced by the f act that the NRC monitors and oversees an extensive range of the facility licensees' activities and is sensitive to allegations and other performance indicators that might expose a problem. Nevertheless, the staff cannot deny that the NRC is now more vulnerable to examination compromise (and the associated public response) 10
N e q \\e.,on, h M k g d eu,@ed ha $ m rh j W <-, c'49t t *d e A-N b r in 6*q,,
k.ibroPdb
[ber.sl(N\\u,ba.Get G em c
weeb < c.qrm cd h4 t-i a en w \\,e y e before 1989 when it first-allowed facility licensees to become
-than it was
~ q The staff
-involved wl;h the NRC's. examination development pro rams.
anticipates that the agency may suffer an occasiona lapseinsecurity.IWhen a lapse occ ars, the staff will continue to take prompt action to cancel the examination, determine the cause of the compromise, and implement corrective measuresC.de<a d O Resourcos As stated in SECY-95-075, the proposed changes wer: criginally conceived as part of the NRC's continuing efforts to ttreamline the functions of the Federal Government consistent with the Administration's initiatives and to accommodate NRC resource reductions. The original plan was to entirely eliminate the use of.ontractors in the operator licensing area-and to conduct all operator licensing program activities including the generic fundamentals
\\
and initial licensing examinations, and the requalification program, with the i
existing staff resources. The facility licensees would prepare and, in part, conduct the initial licensing examinations with oversight by the NRC. The NRC's participation would range from conducting part to all of every examination in order to provide an independent basis for making a licensing decision and to maintain examiner skills.
,g The staff estimated that a qualified facility employee should be able to prepare the site-specific examination in accordance withlNUREG-1021, as supplemented by Gl. 95-06, in approximately the same amount of time that is currently allocated to a contract examiner.
(See Attachment 1, the associated rulemaking plan, for a more detailed discussion of the estimated resource burden on facility licensees.) The staff also estimated that an experienced NRC examiner should take approximately two weeks to review the site-specific examination (including the written and operating tests) prepared by a facility licensee.
As previously noted, the staff considered three options for administering the operating tests. Assuming a license class consisting of nine applicants (e.g., six R0s and three SR0s) that could be arranged in three, three-person crews, both the first and second options would generally require three NRC examiners to spend a week at the site to administer the tests. The second option, which would require the facility licensee to conduct parallel evaluations of each applicant during the simulator scenarios and walk-through tests, would place a significant additional burden (nominally, three staff-weeks) on ihe facility licensee. The third option would be equally burdensome to the facility licensee but would permit.the NRC to reduce (by one) the number of examiners dispatched to the site.
The plan to require the facility licensee to evaluate the applicants during the operating tests was perceived as a duplication of effort because the facility licensee had already evaluated the applicants and found them qualified before it submitted the license applications to the NRC. The staff was also concerned that the additional regulatory burden would discourage participation in what was originally envisioned as a voluntary program with no associated rulemaking.
11
't
-w T
w 7
y vi--
f-w
The NRC resource utilization rate during the pilot examinations supports the staff's original estimate that the new examination process and the requalification inspection program could be implemented at all power reactor facilities with the same level of direct NRC resources as is currently allotted to the operator licensing program. As noted in SECY-96-123, the NRC examiners required an average of about 350 hours0.00405 days <br />0.0972 hours <br />5.787037e-4 weeks <br />1.33175e-4 months <br /> to review, prepare for, administer, grade, and document each pilot examination. Most of the examinations included an extra visit to the site to review and validate the cperating tests.
However, with the elimination of contractor support, the total number of examiners available will decline significantly, thereby reducing the ability of the program to have the personnel available to meet peaks in facility licensee requests for exan.; nations. The tropact of the loss of examiners du; to reassignments, transfers, promotions, and other personnel actions, will be larger without the contract examiners available to compensate for the loss.
To compensate in this area, NRR intends to require that the regional offices maintain a defined minimum level of examiners, to be defined in each regional operating plan. During fiscal years (FYs) 1997 and 1998, each region may need to qualify additional examiners to meet this expectation. To specifically assist the regions in their efforts to train additional examiners, NRR intends to make available to the regional offices up to 7 full time equivalents (FTE) of inspection support during FY 1997 and up to 12 FTE of inspection support in FY 1998. This inspection support will allow regional personnel, who would normally be assigned to inspection activities, to be assigned to qualify as examiners. By the end of FY 1998, the regions will be expected to maintain the defined level of examiners in order to provide the resource flexibility nceded to deal with workload peaks. When examination work is not scheduled, the examiners will be available for inspection activities. A reduced level of contractor support will be required during the transition period.
The staff expects that if the quality of an examination prepared by a facility licensee deviates significantly from established norms, it will become increasingly difficult to predict the amount of time necessary to review or rewrite the axamination so as to achieve an acceptable product. This was the case with several of the pilot examinations, and the staff expects that it will continue to be more of a problem with facility-prepared examinations than it has been with those prepared by an NRC contractor. This added uncertainty in the examination process could increase the risk of broken examination commitments and lower exauination quality; it could also raise costs as a result of inefficient planning.
With the elimination of contractor support and the increased uncertainty about the examination quality, the staff may no longer have sufficient examiner resources to write or even to review all examinations consistent with the scheduling needs of facility licensees. This resource problem is further compounded by the unpredictable nature of the examination workload and by other unanticipated demands on the examiner work force, such as the increase in the number of examination appeals during the pilot examination program.
Historically, the contract examiners have facilitated the staff's ability to respond to unexpected increases in the examination workload and unforeseen departures by staff examiners. Eliminating contractor support will complicate 12
scheduling and limit the staff's ability to conduct retake examinations, which could increase the likelihood that an applicant will appeal the re sults of the examination.
It is for these reasons that the staff is recommending to the Commission that 10 CFR Part 55 be amended to require all facility licensees to prepare the licensing examinations.
With regard to the generic fundamentals examination (GFE), the staff considered the following options:
(1) retaining the current format and process under which the examinations are developed by a commercial contractor and (2) revising the scope of the site-specific written examination to sample If the generic fundamental knowledge and abilities, as was done before 1988.
the examination was revised by adding more questions, it would increase the burden of writing and reviewing the examinations (whether it was done by the facility licensee or the NRC).
If the length of the examination was kept the same, inclusion of the generic knowledge and abilities would weaken the evaluation of the site-specific systems and procedures. The staff and the indestry generally have agreed that the current program is both efficient and effective and that revising the site-specific sample plan would create too many problems. Accordingly, the staff has decided to retain the current examination format and process and to implement the program with contractor assistance at a cost of about $200,000 per year.
Program Stability from 1988 to 1990, the NRC received numerous complaints from facility licensees and licensed operators that the NRC's requalification examination program was causing undue stress for the operators and the facility employees who were administering the examinations. The Human factors Assessment Branch conducted a study to address this issue and determined that the frequent changes to the requalification examinat,on process ranked among the most significant sources of undue stress; the results of the study are documented in SECY-91-391, 'Results of the Study of Requalification Examination Stress."
One of the corrective actions involved establishing a schedule for periodic revisions of NUREG-1021 and delaying the implementation of each revision for 180 days after it is published.
Although the staff did not survey license applicants during the requalification stress study, it has generally tried to minimize the adverse effect on applicants by avoiding unnecessary changes to the initial examination process. Therefore, when the staff decided to shift responsibility for developing the int. al licensing examinations to the facility licensees, it resisted pressure to revise the examination format. To the extent possible, the format, content, and level of difficulty of the pilot examinations would remain unchanged, tt.ereby making the changes imperceptible to the license applicants. The staff also believed that the facility licensees were generally familiar with the existing requirements in NUREG-1021 and that any significant changes in the examination format would confound their efforts to assume responsibility for writing the examinations.
The desire to maintain program stability, avoid unnecessary changes, and minimize stress on the applicants also argued against the adoption of Options 2 and 3 for administering the operating tests. The requirement for facility 13
=__
personnel to evaluate the license applicants would increase the learning curve for the facility licensees and would likely elevate the stress for the applicants because of the double scrutiny and the increased crowding in the simulator.
In summary, the staff found no compelling reason to modify the existing examination format.
The benefits of retaining the current format far exceed any argument for making changes at the present time.
Examiner Proficiency On May 25, 1995, the staff briefed the Commission on the status of the operator licensing programs and tue staff's proposal to shift respons1'ility o
for preparing the initial operator licensing examinations to the facility licensees, in the ensuing SRM of June 2,1995, the Commission instructed the staff to ensure that the in-house capability to administer the examinations is maintained, possibly through the random addnistration of some examinations.
kw wwA
.n se 9 As noted in SECY-96-?,23, the staff beliEEiI~that NRC examiners will inaintain the ability to write examinations, if necessary, based on their participation in the new process. Although facility licensees will prepare the operator licensing examinations in accordance withiNUREG-1021, NRC examiners will continue to review and approve every examination before it is administered to ensure that it conforms to the criteria specified in NUREG-102i for content, format, quality, and level of knowledge, f acility licensees will also administer. and grade the written examinations in accordance with NRC ge proccdures, but NRC examiners will cor.tinue to independently administer and grade bott 'he dynamic simulator and the plant walk-through portions of the The NRC will review and approve the assignment of grades for operating tests.
the written examinations, including any changes to the answer keys and the deletion of questions recommended by the facility licensee. The NRC will also continue to make all licensing decisions and administer the appeals process for applicants who fail the examination. MAN %,
- A m M
- F(" ""h p,c A, % _ % c W h b nsee n c
, a, - s A A y h pr % a c A,4., t<.NRR will continue to require new NRC license examiners to comp
% ~pc m ao standardized training program before they are certified to conduct licensing WW examinations. The training program includes classroom and simulator instruction on how to prepare and administer written examinations and operating tests, self-study of operator licensing procedures and guidance, on-the-job training, and a final practical evaluation by a certified chief examiner. Once certified, examiners are required to maintain their qualification by conducting examinations and attending refresher training.
NRR is currently revising the examiner refresher training syllaLds to focus more attention on reviewing, critiquing, and approving written examination questions. Examiners are also required to attend the operator licensing examiners' training conference, which is periodically convened by NRR to provide guidance and exchange information.
To ensure that the on-site examination activities are being performed as intended, supervisors are required to periodically evaluate every certified examiner during the conduct of an operating test. Furthermore, to verify compl4ance-with NUREG-1021 and sess the uniformity with which the g
wu gg 14 A na un w ea rn
l 2
examinations are administered within and among the regions, the NRR staff periodically audits the administration of operator licensing examinations at-selected facilities and annually reviews the~ implementation of the operator licensing. function in each region'. The NRR audits and reviews include an' assessment of. the level of knowledge and the level-of difficulty of the written examinations and the operating tests; each audit and review is
- documented in a report to the responsible regional Olvision Director.
ladustry' Acceptance and Response In the-summer of 1994, VEPC0 recommended to the NRC that the initial licensing examination process be changed to make-facility licensees responsible for developing the examinations subject to review and approval by the NRC chief examiner. The NRC wotJ.d continue to administer and grade the examinations.
The proposal, which was submitted as part of VEPCO's cost-beneficial licensing action program, estimated that the utility could save an average of $67,500 per station for each initial license class by implementing the proposed approach. The staff reviewed and denied the proposal because of concerns regarding independence and objectivity and the potential for increased variability in the level of knowledge tested and examination difficulty.
The staff reconsidered its positim early in 1995 when the NRC's fiscal expectations changed significantly. The staff decided to explore the possibility of eliminating the use of contractors in the operator licensing r
program by allowing facility licensees to develop the examinations, essentially just as VEPC0 had proposed.
It was estimated that such a change would save the NRC from $3 million to $4 million per year in contractor support funding and that the reduction in recovered fees would largely or 4
completely compensate the facility licensees for their additional expenses.
i i
While evaluating various options for implementing the changes proposed in SECY-95-075, the staff obtained informal feedback from the Nuclear Energy Institute (NEI) regarding the industry's level of involvement in administering the operating tests. NE! had polled a number of facility licensees and determined that there was no support for parallel grading (Option 2 previously i
discussed) because of concerns regarding applicant stress, the additional resource burden, and simulator crowding. The majority of those surveyed expressed cautious support for Option 3, under which NRC examiners-would have observed the facility licensee administer the operating tests. However, those surveyed wanted the NRC cxaminers to focus only on the performance of the applicants ad not on the examination practices of the facility licensee, Option 3 would shift the most responsibility to the facility licensees, but it i
was viewed as the most' difficult option to implement. Facility licensees perceived that Option 1, under which NRC examiners independently administer the operating tests, would be the least. demanding on facility licensees and the easiest to implement and would retain the greatest level of independence in the licensing decisions. NE! indicated ~that all of the facility licensees surveyed were satisfied with. the GFE and thought it should be continued in the current manner.
The staff believes that the industry's response to Gl. 95-Ofs which described the pilot process and solicited volunteers to participate in a six-month 15 i
i
~,
-,~w
- c. -.. -
~
program to evaluate its feasibility. was generally favorable. Of the 35 facility licensees that had requested: examinations during the pilot period, 22 agreed to prepare their own examinations in accordance with GL 95-06. Most of the facility licensees that declined to participate cited as their reason
- their inability to reschedule the required resources on such short notice because of the restrictions on the use of personnel who h6d been involved in training the license applicants.
In September 1995, shortly before beginning the pilot examinations, the staff convened a public workshop to discuss the NRC's expectations regarding industry participation in the pilot program for the operator l hensing examination. The workshop also gave facility representatives an opportunity to ask questions of the staff and to submit written comments regarding the pilot examination criteria outlined in GL 95-06. Most of the questions and comments related to the restrictions on which personnel the facility licensee could use to prepare the examination and the requirement to document the history of the examination materials. Some facility licensees questioned the amount of reference materials that the NRC would need to review knd prepare for the examination.
The staff challenged the pilot examination participants who expected to have problems satisfying the personnel restrictions to suggest alternative ways to achieve the same objective. The staff comitted to review the alternatives on a case-by-case basis and to propose a final position in Revision 8 of NUREG-1021, which would be publisbd for coment near the end of the pilot program. The staff also made it clear from the beginning that facility licensees could use contractors to develop the examinations.
Feedback from NRC examinere, who participated in the pilot exa.1 nations and industry representatives who attended a public progress meetirq sponsored by NEl in January 1996 confirmed that the personnel restrictions were the most troublesome aspect of the pilot process. Many facilities, particularly those with smaller training staffs, contended that the quality of the examinations was suffering becausc most of their best qualified personnel were involved in teaching and were ineligible to prepare the examinations. Those attending the meeting generally agreed that the industry and the NRC had learned a great deal from the pilot program, that most of the issues could likely be resolved through training and improved communication, and that the pilot process should be fully implemented, in february 1996, the staff issued draft Revision 8 of NUREG-1021 that incorporated both the )ilot process outlined in GL 95-06 and preliminary lessons learned from tic early pilot examinations. The draft revision was sent to the PDR and placed on the NRC's home page on the World Wide Web, and a federal Register notice requesting industry and public comments was published on February 22, 1996. NEI submitted comments and recommendations on behalf of the nuclear industry, and two facility licensees (VEPCO and Commonwealth Edison Company) provided additional comments. On April 18, 1996, the staff conducted a public meeting with NEl and otbcr industry representatives to review their comments and recommendations regarding the draft revision of NUREG-1021.
The meeting also provided an opportunity to discuss additional issues and-proposed changes that had emerged following the issuance of draft
- Revision 8 of NbREG-1021 for public comment. The significant comments and staff responses are as follows:
16 3
e n
we w
v
=
t l
All three commenters suggested that the personnel restrictions placed on
}
o individuals involved in writing the examinations should parallel the personnel restrictions of the requalif_ication program (i.e., the examination authors must stop teaching when examination development begins). The industry believes that those restrictions, when combined with the examination development and review criteria, would provide adequate protection against the unconscious biasing of an examination by an individual.
It is difficult to argue that the instructors would not be the best qualified individuals to prepare the examinations. Many NRC examiners also agreed that the requalification program restrictions would be adequate. However, the staff concluded that independence and conflict of interest were of sufficient concern to require personnel restrictions very similar to tho,e outlined for the pilot process. The staff revised the wording in the draft NUREG to clarify and quantify the asunt of involvement that would disqualify an employee from participating in various aspects of examination development, for example, the personnel-wht prepare the examination outline, including the selection of knowledge and abilities to be evaluated on the written examination and the JPMs and simulator scenarios to be performed during the operating test, shall not have had any direct involvement in training the license applicants, furthermore, only one person with limited involvement (i.e., who taught more than 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> but who was responsible for no more than 15 percent of the schedulad classroom instruction) may participate in developing the written examination questions, and no one may develop questions for the topics they taught, fa W ity licensees may still propose alternative approaches for considera> ion on a case-by-case
- basis, o
All three commenters questioned the need for facility licensees to document the history of every test item proposed for use on a licensing examination, even if it was drawn from another facility licensee's examination bank.
The staff agreed that the wording in the draft NUREG could have resulted in misunderstandings of what was expected. The staff still believes that it is important to monitor the predictability of the examinations but it lacks the data and the resources to independently identify the source of every test item. This information is readily available to the fu.ility licensee. The staff h6 clarified the guidance in NUREG-1021 so that test items that-a facility _ licensee obtains from another bank and deposits into its own bank may be treated as " bank" items, provided the test items have an equal chance of being selected for use on the examination.
Items from another bank may be treated as new items if they have not been made aval~ table for review and study by the license applicants and there is no basis (e.g., historical precedent or reciprocal arrangements with the other f acility licensee) for the applicants to predict their use on the examination. The staff believes that this revision will somewhat lessen the burden on facility 17 3
y u+
w w-y w.-
T
^
- 4
~ licensees,- while enabling 'the' NRC to maintain an acceptable level of oversight, o-
- All three commenters suggested that NUREG-1021 relax the restrictions on-the duplication of test items from the applicants' audit-examination (a practice test given by the facility licensee before the NRC examination) if the audit examination was developed independently of the license examination, Under the traditional examination method, the-staff does-not have access to-the facility licensee's audit examination, so any duplication that occurs between the audit examination and the licensing examination is unmonitored and coincidental. The staff agreed that-the proposal was consistent with current practice and changed the NUREG to allow up to five questions to be duplicated at random if the two examinations are written independently (i.e., no interaction between the examination authors) and the facility licensee identifies the duplicates. Licensing.
examinations that are not. developed independently shall not rereat any questions-directly from the audit examination.
All three commenters recommended that NUREG-1021 accommodate-facility o
licensees that have " closed" examination banks (banks i,ct available to.
applicants for study) by relaxing the limit on the number of test items that can be drawn directly from the bank.
The staff acknowledged that this is a valid issue, but because tbs issue is so complex, the staff does not intend at this point to establish additional guidelines regarding the maintenana of examination c
banks. As noted earlier, the staff intends to treat all buks as if they are "open" at this time, All three commenters ob,jected to the suggestion in draft NUREG-1021 that o
facility licensees consider providing an explanation as to why the ansar to every written (multiple-choice) question is correct and each of the distractors is plausible but incorrect. The industry thought
'his was a significant administrative burden that would, in practice, become a requirement.
Feedback from the NRC chief examiners indicated that some facility licensees adopted this practice on their own during the pilot examinations; it proved very beneficial and increased efficiency during the examination reviews. This prompted the staff-to include it as an optional activity in drait NUREG-1021. The staff still believes that
- the practice has merit but has emphasized in the final NUREG that it is
- optional, o
Two commenters recommended that facility licensees be permitted to use site-specific task lists in place of the NRC's K/A catalogs when developing the examination outline. The site-specific lists are used to i
develop the operator requalification examinations and should also be acceptable for developing content-valid initial licensing examinations.
18 g
e-,.,
m m
, + +
w w
s, e
-_n,--
-c,-~,~
r N.+
rr-r.r+.
N. v-
--e
- - - +-- -
w m-v
Since the mid-1980s, the staff has used the NRC's generic K/A catalogs as the basis for ensuring the content validity of the initial licensing examinations. The catalogs are based upon an analysis of the licensed operator's job that was performed by the Institute of Nuclear Power Operations. Many facility licensees have conducted their own job task 1
analyses and osveloped plant-specific operator task lists for use in their SAT-based training programs. Although the staff does not question that the site-specific task lists would generally be acceptable for ensuring the content validity of the examinations, _it believes that wholesale substitution of the catalogs could cause a decline in examination consistency. The staff considers the proposal appropriate in limited circumstances on a question-by-question basis; therefore, the guidance in NUREG-1021 has been revised to permit substitutions and additions of sp..elfic knowledge and ability requirements on a case-by-case basis,
.Two commenters disagreed with a paragraph in Appendix A of the draft o
NUREG that implied that.any question taken from an open bank would test at the basic memory level. They argued that questions prepared at a higher cognitive level would not be reduced to the basic memory level if 4
the question bank is large. They strongly recommended that the subject paragraph in Appendix A be deleted.
Appendix A provides an overview of examination preparation concepts and discusses the basis for some of the NRC's examination criteria, including the limits on the use of examination banks. The subject paragraph is considered integral to the discussion and was not deleted.
However, the staff has edited the text to clarify that questions could decrease to the simple recognition level if the item bank is small and available for the examinees to study.
The staff also clarified one of the criteria in draft NUREG-1021 that, for purposes of examination quality and consistency, required at least half of the' questions on the written examination to be written at the comprehension or analysis level. The staff never intraded to imply that any question drawn directly from a bank would be counted at anything other than its face value. A question written at the comprehension or analysis level would count as such, even though the cognitive level at which it tests would decline with use and exposure, o-Two ummente-s recomended that the facility licensees be allowed more than five days, if necessary, to review and grade the written examinations. They asserted that relaxing the requirement would only delay the results to the facility licensee and the applicant and that the delay would not adversely affect the NRC.
The staff had traditionally allowed the facility licensee five days to comment on the written examination, so it decided to retain that time period for the pilot examination program. This was one of the examination criteria on which the chief examiners had been asked to comment, and no comments were made. However, since the proposed change 19
+
would not affect the examination outcome, the staff decided to increase the flexibility in the final NUREG-1021 by noting that the facility licensee should submit the graded examinations within five days, if practical, o
One commenter indicated that NUREG-1021 does not appear to accomodate utilities that do not want to prepare their own licensing examinations and suggested that utilities be allowed to retain that option.
Because of the unpredictable demand for examinations and the scheduling difficulties that may result from a long-term voluntary program, SECY-96-123 requested that the Commission approve the staff's pursuit of rulemaking to require that licensees of power reactor facilities prepare the operator licensi..g examinations in accordance with NUREG-1021. Tt.a Commission has delayed making a decision on industry-wide implemcntation of the revised process until it can review the detailed rulemaking plan (Attachment 1) and the additional issues discussed herein, o
One comenter recomended that the facility licensees be held responsible for making examination changes "as agreed upon with the NRC" rather than "as directed by the NRC."
The commenter added that this alteration would help ensure that the suggested changes are accurate.
Because the staff fully supports the goal of the facility licensee to ensure the accuracy of the examinations, it revised NUREG-1021 as recomended but stipulated that the NRC would retain the final authority to approve the examination, o
One commenter requested that the NRC clarify which facility staff members would be allowed to observe the operating tests in the simulator. NUREG-1021 currently restricts personnel other than the simulator operator from watching the operating test without the approval of the NRC chief examiner. The commenter argued that facility managers and others deemed necessary by the facility should be allowed access to the examination, provided the simulation facility can accommodate them and there is no adverse effect on the applicants.
The staff concluded that this was a reasonable request and revised NUREG-1021 accordingly. Facility management and other personnel deemed necessary by the facility licensee will generally be allowed access to the examination (Jnder security agreements, as appropriate), provided the simulation facility can accommodate them and there is no adverse effect on the applicants, o
One comenter requested that the NRC clarify the meaning of "significantly modify." The facility licensee had been told that the intent of the question had to be changed, which conflicted with the definition that had been published in GL 95-06 and draft NUREG-1021.
20
4 I
The staff concluded that this was a reasonable request and revised-t NUREG-1021 to-indicate that the intent of the question does not have to change to quali_fy as t significant modificaticn.
In summary, the staff believes that the-industry generally supports the-proposal-to shift responsibility for writing the initial licensing examinations to the facility licensees. Some facility licensees actively i
4 lobbied for the change on the basis that it would save them money. Many facility licensees and NEl view the proposed changes.as a-good first-step at increasing the industry's involvement in the examination process and shifting the NRC's role from direct involvement to oversight. Neverthel.ts., most facility licensees would probably agree that the administrative requirements that were outlined in GL 95-06, refined during the pilot program, and incorporated in Revision 8 of NUREG-1021 have increased the burden of preparing the examinations.
- The issue of industry burden has been a key factor throughout the pilot program. The staff purposely limited the scope of the original process changes to those measures that would help ensure the integrity, consistency, quality, and security of the licensing examinations. The ctaff was comitted to maintaining an effective and efficient licensing program and realized that the revised process, which would have relied on voluntary industry support.
would never succeed if it was unnecessarily burdensome on-the facility licensees.
In keeping with its original commitment, the staff believes that it has taken reasonable measures to address the industry's concerns and recommendations without sacrificing the effectiveness of the examination
- process.
Pilot Enhancements and Justifications To assess the effectiveness of the new examination process, NRR asked the NRC i
= examiners who were involved with the pilot examinations to respond to a questionnaire regarding various aspects of the process. The first and most significant problem observed and reported by the examiners was that the quality and technical accuracy of the examinations submitted by several facility licensees fell short of the NRC's expectations. Moreover, some of the errly pilot examination results (i.e., more question deletions and answer key changes than expected) reflected the need for greater emphasis in that
- area.
Consequently, when NRR prepared draft cvision 8 of NUREG-1021, it incorporated a number of clarifications and changes that improved the framework established in GL 95-06, in January 1996, the staff issued a preliminary draft of NUREG-1021 to the regions-for review and comment before soliciting comments from the public and the industry the following month. The draft revision that was issued for public and industry comment contained the following process improvements:
-The existing guidance (Revision 7 of NUREG-1021, Revision 5 of NUREG/BR-o
-0122, and GL 95-06) did not specify what percentage of the questions on the written examination should test the appilcants at the comprehension 21
and analysis levels of knowledge. Many examiners had expressed concern u
that the lack of guidance in this area would lead to inconsistency and a decrease in the level of examination difficulty.
Facility licensees questioned wh % r examiners had unwritten guidance that required them to make the ex.nnation more difficult by revising or replacing simple memory questions.
As noted earlier, NRR had no desire to increase the average difficulty of the examinations, but it also did not want the level of difficulty to diminish. Therefore, to better define the appropriate level of knowledge for the new examination process, the staff reviewed the written examination audits performed in the previous two years to determine an average distribution of questions by knowledge level. The staff concluded that approximately half of the questions on the examinations reviewed were written at the comprehension or analysis level and that the remaining questions tested simple memory. On the basis of that information, the staff established a criterion in draft NUREG-1021 that at least half of the examination questions be written at the comprehension or analysis level, The supplemental guidance that NRR provided to the regions concurrent o
with CL 95-06 included interim QA checklists to assist the i4RC examiners in reviewing and identifying deficiencies in the draft examinations.
The checklists were edited and incorporated in the prelim. nary draft of NUREG-1021 and edited again in response to regional comments. The draft NUREG-1021 that was issued for public comment includeri QA checklists for the examination outline, the written examination, both the simulator and walk-through portions of the operating test, and the wri' ten examination
- grading, The existing guidance (Revision 7 of NUREG-1021 and GL 95-06) did not o
specify a limit on the number of JPMs that could be repeated in walk-through tests administered on successive days during the same initial licensing examination visit or from one visit to the next. Many examiners had expressed concern that the lack of guidance in this area could compromise examination integrity and lead to inconsistency. Other examiners argued that no repetition should be permitted because it would increase the predictability of the test. Facility licensees would be tempted to repeat as many JPMs as possible in an effort to save resources.
To strike a balance, the staff concluded that a 30-percent limit would be appropriate. This figure was consistent with the written examination limit of 25-percent duplication of questions from examinations, quizzes, or tests administered to the license applicants or from the past two NRC licensing examinations at the facility.
After issuing draft HUREG-1021 for public and industry comment, the staff identified a number of items that merited clarification and improvement.
In addition to reviewing the industry's comments and recommendations (as previously noted), the staff discussed all of the following changes with NE!
22
and other industry representatives during a oublic meeting held in the NRC's offices on April 18, 1996. Except as noted selow, the industry representatives did not object to the additional changes.
lo limit the number of technical inaccuracies, a problem that affected o
several of the pilot written examinations, the staff revised NUREG-1021 to require the NRC chief examiner or another examiner to independently review and verify the technical accuracy of a sample of the written examination questions. The number of questions verified would depend upon the expected quality of the examination and the time available before the scheduled review with the facility licensee.
As a final check of the technical accuracy af the written examination, the staff revis,d NbREG-1021 to encourage the facility licensee to administer the NRC-approved examination (under security agreements) to one or more licensed personnel who were previously uninvolved in developing the examination. Although feedback from the NRC chief examiners indicated that some facility licensees had taken it upon themselves to validate their examir.ations in this manner during the pilot program, the facility representatives at the meetir.g objected to making it a requirement for all f acility licensees. Because the staff lacked sufficient informatM to quantify the value added by this measure, it chose to recommer.4 it as a good practice, Late in the pilot program, an NRC chief examiner who was reviewing a o
draft examination detected that the facility licens:3 had used a written examination outline that was identical to one that a sister facility had prepared several weeks earlier. The staff had not anticipated that such a problem might occur but immediately realized that this was a significant loophole in the process. Allowing facility licensees to reuse the same examination outline would definitely improve efficiency, but it would also jeopardize the integrity of the examination process by increasing the predictability of the examinations.
To address this problem, the staff included in NUREG-1021 a caution for examination authors and reviewers to ensure that the outline ust.d during huccessive audit and licensing examinations does not become repetitive and predictable, if a facility licensee proposes to use an outline that was previously used at another facility, it shall identify the source of the outline and explain what effect its reuse is expected to have on examiaation integrity.
o NUREG-1021 has for some time included checklists for use by NRC examiners in verifying that their simulator scenarios contained the required number of transients and ensuring that the scenarios would enable the examiners to evaluate the applicants on every required competency. There were no specific instructions for filling out the checklists, nor were they required to be retained in the examination file. Concerns regarding the quality and consistency of the operating tests prompted the staff to incorporate instructions for completing the 23
. - ~. -
'4 transient and competency checklists and a requirement for the NRC-regional offices to retain the checklists in the examination files.
o NUREC-1021 has always endorsed the goal of having the licensing examinations ready to administer approximately a week before they are
' scheduled to be given. - furthermore, the staff has always encouraged the regions to keep NRR informed if significant problems are encountered--
with an examination. The pilot program illustrated and magnified the importance of both cancepts. Therefore, the staff clarified the guidance in NUREG-1021 to limit the number of last-minute changes and to require the regional office to consult NRR if it appears that the examination cannot be made to conform to the examination standare at least five working days before the scheduled examination date.
Furthermore, th+ exa 4tnations should be postponed if problems are identified at the last minute, in addition to in.plementing the pilot examination process,- the staff o
proposes using Revision 8 of-NUREG-1021 as a vehicle for activating Revision 1 of NUREG-ll22 and NUREG-1123, the NRC's K/A catalogs, which were published in August 1995. The catalogs were revised to make them more current and reorganized to make them more consistent with each other and easier to use. As part of the catalog revisions, the system-generic K/As in both catalogs were combined with the plant-wide generic K/As, and new vendor-specific emergency and abnormal plant evolutions were added to the pressurized water reactor catalog (NUREG-ll22). These changes would have altered the balance of the written. examinations unless the sample plans in NUREG-1021 were also revised to compensate for the changes. Because of an oversight, the staff neglected to enter the changes t;efore issuing the draf t NUREG-1021 for public and industry
- comment, When NRR reviewed the regional office operator licensing programs during o
February and March 1996, it discovered that some of the examiners had misinterpreted the intent of NUREG-1021 regarding the design of the operating tests for SRO applicants limited to fuel handling. The staff discussed the issue with the regions and agreed that strict compliance with the existing guidance would overtest those license applicants.
Therefore, the guidance in NUREG-1021 was changed to more closely align the scope of the operating test with the range of responsibilities for SRO-fuel handlers.
Whenever a question is deleted from the 100-point written examination,
-o
'it results in fractional grades. When the grade is above-79.5 percent, it raises a question regarding the staff's policy on rounding-off because it means the difference between receiving a license or a denial.
To address this problem, the staff has revised NUREG-1021 to indicate that the passing grade on the written examination is 80.00 percent.
Although the pass rates of the pilot examinations were comparable with o
pass rates for NRC-written examinations, the rate at which applicants requested that the NRC informally review their examination failures
-24
increased _ unexpectedly during the pilot program. These requests not only placed a significant burden on the staff to resolve the applicants' issues but also raised concerns regarding the etfect that any additional cuestion deletions and answer changes might have on the other licensing cecisions that had already been made.
Consequently, the guidance in NUREG-1021 was revised to afford the NRR staff greater discretion in determining the niethod for resolving applicant appeals (i.e., every appeal does not necessarily have to be reviewed by a panel of examiners as long as objectivity and fairness are maintained) and to delay the issuance of licenses to applicants who passed the written examination with insufficient margin to guarantee that the licensing decision will be sustained if additional questions are deleted or changed upon appeal. Furthermore, to minimize the number of appeals, the staff revised NUREG-1021 to encourage facility licensees to solicit and address concerns from individual license applicants during the process of grading the written examinations.
o Problems encountered while reviewing and resolving the operating test appeals that were received during the pilot program prompted the staff to reevaluate the documentation and grading requirements in NUREG-1021.
As a result, the staff has revised NUREG-1021 to permit NRC examiners to downgrade more than two simulator test rating factcrs for the same error, provided the error caused serious safety consequences for the plant. The staff also increased the documentation requirements for the simulator test so that the NRC examiners would have to describe any error that prompted them to downgrade the applicant on any rating factor.
During the meeting of April 18, 1996, the staff also discussed the possibility of permitting NRC examiners to recommend a failure on the simulator test even if the applicant did not fail the test on the basis of the competency scores. The industry representatives did not believe that such a change would be necessary if there were no limit on the number of competencies that the examiner could downgrade for a single, serious error; however, they did not object to the change in principle.
Therefore, the staff has included in NUREG-1021 a provision for NRC examiners to recommend such a failure if the applicant committed an error with serious safety consequences for the plant or the public. To limit its use, the region must obtain written concurrence from the Chief, HOLB, before completing
.a licensing action.
The (cllowing issues were raised by NRC examiners on the pilot examination feedback surveys but did not result in specific chang to the examination methodology described in NUREG-1021.
o To minimize NRC resource requirements, the staff had encouraged the regions to review and prepare for the pilot examinations without conducting a preparatory visit to the site. However, feedback from those examiners who did not conduct such a visit indicated that it would have facilitated the review and resulted in fewer problems with the 25
examination. They recommended that a preparatory visit be required for every examination. Although NRR has not changed the guidance in NUREG-1021 (i.e., the preparatory visit is optional at the discretion of regional management), it has acknowledged the value of conducting at least part of the examination review and validation at the facility.
The average NRC resource expenditures during the pilot examinations indicated that the trips can be made within the original budget model, o
Several NRC examiners recommended that the required due dates for the axamination materials from the facility licensees be advanced so that the examiners have more flexibility in scheduling their time for the examination review. The staff did not change the guidance in NUT.EG-1021 because it already permits the regional offices te work with the facility licensees in determining the appropriate due dates and because a generic change would increase the scheduling burden on all facility licensees, it would not be appropriate for an examiner to demand the examination 60 days in advance because it was required by NUREG-1021, only to have the examination sit untouched in the office for 30 days because the examiner was involved wi_th another project, o
When the staff developed the pilot examination process, there was some debate as to whether the facility personnel who administered the written examination should be permitted to answer the applicants' questions regarding the intent of specific qustions on the examination. The pilot guidance allowed the facility to answer the applicants' questions with caution but required every question and response to be documented for subsequent review by the NRC. The policy was evaluated on the pilot examination questionnaire that every NRC chief examiner was required to complete. A minority of the examiners recommended that the policy be changed to prohibit the applicants from asking questions during the written examination, as is the policy on the GFE. However, most of the examiners concurred with the original pilot policy, and none of the comments indicated that the policy had created any problems with examination integrity. Therefore, the staff saw no reason to revise the guidance in NUREG-1021.
Some examiners noted that the pilot program personnel restrictions had o
hampered the facility licensees' abil ty to develop the examinations because they could not assign their best people to the task. This issue was also addressed in the industry comments, as previously discussed.
Examination Results Since Issuance of SECY-96-123 The staff completed the last of the pilot examinations on April 5, 1996, and resumed conducting the licensing examinations using the traditional development process. After April 12, 1996, when the Executive Director for Operations informed the Commission of the staff's intent to continue the pilot process beyond the period previously discussed with the Commission, the staff proceeded to contact those facility licensees whose examination requests had not yet been assigned to a cnntractor to detennine if they were interested in developing the examinations using the pilot methodology. The first of those 26
examinations was administered on August 19, 1996, and the susults were not i
available in time to report them here.
Since SECY-96-123 was prepared, the staff has used the traditional examination methodology to prepare licensing examinations for applicants at 14 power reactor facilities.
The results of the completed examinations are summarized i'
below; four of the ensinations were completed late in August, and the results were not available at the time this paper was prepared. The licensing examination results for fiscal year 1995 and the 22 pilot examinations conducted between October 1, 1995 and April 5. 19t'5, are provided for comparison.
Pass Rates' R0 R0 R0 SRO.
SRO SRO Examinations Written Operating Total Written Operating Total 39/40 39/39 39/40 45/48' 43/48 41/48' Since SECV-96-123 (98%)
(100%)
(98%)
(94%)
(90%)
(85%)
Pilot 49/54 50/54 45/54 86/92 87/918 83/92*
Examinations (91%)
(93%)
(83%)
(93%)
(96%)
(90%)
Fiscal Year 1995 94%
98%
92%
95%
95%
92%
1.
Each box indicates the number of applicants who passed that part of the examir.ation over the number of applicants who took it.
2.
One of the SRO applie nts who fedled the written examination has filed an aop.1 that was not yet resolved at the time this paper was pre,."id.
3.
Tw. 9plicants who were reported as preliminary failures in SECY-96-123 filed' appeals that resulted in the failures being overturned.
ilheitX The staff believes that the revised examination process can be implemented at all power reactor facilities with the same level of direct NRC resources as is currently allotted to the operator licensing progrcm., Full implementation of the revised process will enable the staff to eliminate the use of operator licensing.ontractors (with the exception of those'whc develop and grade the GFE) that have hictorically cost the NRC (and the indu stry) approximately
$3 million to $4 million per year. The staff expects that the change could result in a net resource savings to the facility licen sees over time because they will be abis to prepare the examinations more efficiently than the NRC or its contractors.
The overall results.of the pflot program indicated that the examinations prepared by facility licensees, subject to review, rev ision where ap)ropriate, and approval by NRC staff examiners, were generally of the same caliser as examinations prrpared by NRC or contract examiners. l ie pilot examinations also appeared to be equally effective at identifying applicants who had not-4 6
k$N k
kh g 1-Thd4 4 h %& dv.3 yO and FMS V
i -
b Wro se. % g.u\\ch 9M v.w m nets av d\\ d.t.
l l-
,,,,,,.--, e,. -,,, -.
,.,,-.n,
. - -. - - - - -. =, - -,
.-~
1 mastereothejobrequirementswellenoughtobecomelicensedR0sorSR0s.
furthermore, the pass rates for R0s and SR0s on the written examinations and operating tests compared favorably with the pass rates for examinations prepared by the NRC or its contractors.
j However, despite the mitigating actions previously noted, the revised examination process is much more dependent upon the abilities of the NRC chief examiners to identify weak examinations and work with the facility licensee to upgrade the examination.
Furthermore, it is possible that the pressures to conduct examinations on schedule and the reduced involvement by the NRC in the examination development process could lead to examinations that reflect the biases of the facility licensees and that discriminate at a lower level than is currently the norm. Moreover, it is reasonable to expect that industry-wide implementation of tht revised process will increase the chance of future examination compromises as a result of the larger number of individuals who would require access to the examinations and the additional pressures c.':
facility employees to ensure the success of the applicants.
Although a number of the examinations submitted by the facility licensees did not meet NRC standards for level of knowledge or difficulty, the NRC chief examiners effectively worked with the facility licensees to revise the examinations so that they would discriminate at the appropriate level. To put this situation into perspective, it has not been too unusuLl for contractor-developed examinations to fall short of the NRC's requirements and expectations. The chief examiner often directs the contract examiner to make numerous corrections and changes before reviewing the examination with the facility licensee, only to have the facility reviewer identify a host of additional errors and inconsistencies.
The staff has advised facility licensees that the NRC expects the technical quality of examinations to remain high and the number of post-examination changes to be low. Similarly, regional examiners and managers have been instructed not to approve or administer any examination that does not adhere to NRC standards for content, format, quality, level of knowledge, and level of dif ficulty. The QA checklists in NUREG-1021 have been modified to maintain examination consistency and to facilitate the efforts of NRC examiners to detect deficient examination materials. The use of any facility-prepared examination that is found to be unacceptable will be delayed or cancelled, if necessary, until the facility Itcensee and the NRC can upgrade or replace tne examination.
The supplemental pilot examination guidance that the staff provided to the regional offices (and placed in the PDR) in conjunction with GL 95-06 directed the regions to document in the examination report any significant problems c p C _ encopntered during the examination development, administration, and grading.
This requirement-has also been included in Revision 8 of NUREG-1021. The regions are expected to address any serious deficiencies in the examinations submitted by the facility licensees and any security concerns during the entire examination process, if appropriate, the region will request the facility licensee to describe the actions it will take to improve future performance, if the facility licensee is unable or unwilling to prepare 28
appropriate licensing examinations, the NRC has the responsibility and authority to develop the examinations in the traditional manner.
Overall, the staff does not expect that the pro)osed revisions to the o>erator licensing program will significantly increase tie risk to the life and iealth of any individual members of the public or that they wt11 be a significant addition to other societal risks. The staff believes that the licensing 4
decisions that were made on the basis of the pilot examinations were as valid as those made using the traditional examination process, furthermore, the changes that the staff has made in NUREG-102) as a result of lessons learned duringthepilotprogramandindustrycommentssubmittedinresponsetothe staff s solicitation (which were previo:tsly discussed in detail) should only improve the effectiveness of the examinations if and when the revised process is implemented on an industry-wide basis.
The NRC will continue to set the standard for operator performance by ensuring that the facility licensees maintain approprtately high standards for examination development. The staff will remain actively involved in the operator licensing process by reviewing and approving every written examination and operating test before it is administered. Unacceptable examinations will be revised and delayed, if necessary, until the resources are available to upgrade the examinations. Furthermore, the fact that NRC examinerswillcontinuetoindependentlyadministertheoperatingtes(swill allay concerns regarding bias and conflicts of interest on thet M portion of K
the licensing examination having the most operational validitf-and-requiring the most subjective judgements.
The safety consequences of oc:asionally administering a substandard examination are probably minimal. By the time the applicant takes the NRC ticensing examination, he or she will have completed a comprehensive.
systematically constructed and implemented training program designed to
~
maximize the likelihood of success on the examination.
The itcense training programs typically include numerous written examinations and performance demonstrations that culminate in certification by the facility licensee that the applicant has successfully completed the facility licensee's requirements to be licensed as an R0 or an SRO. The effectiveness with which the factitty licensees screen out those trainees that have not mastered the job requirements has been evident-in the high pass rates on the licensing examinations during the past several years.
In most cases, the NRC examination is simply a conitrmation of the facility licensee's decision to submit the. application.
Historically, even those applicants who fati the NRC written examination generally score above 75 percent. The average grade for the 11 applicants who failed a pilot examination was 76.3 percent. This statistic indicates that the vast majority of applicants, including most of those who fail the licensing examination, have attained an adequate level of knowledge so as not to pose a significant threat to reactor safety. Furthermore, any applicant who does become licensed on the basis of an examination that discriminated at a lower than normal level would be backed up by other crew members who could prevent or remedy any operating errors. Although there is nothing to prevent a facility licensee from forming an operating crew entirely from newly.
29
. ~.
licensed operators, most facility licensees integrate their new operators with e
existing crews who possess significant operating experience, r
i 4
30 s
__ _.,.