ML20216F809

Informs That NRR Has Reviewed Subj Rulemaking Plan & Concurs W/Plan as Revised in Attached Markup

ML20216F809
Person / Time
Issue date:	08/30/1996
From:	Russell W NRC (Affiliation Not Assigned)
To:	Morrison D NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES)
Shared Package
ML20216F483	List: ML20216F480 ML20216F496 ML20216F508 ML20216F518 ML20216F534 ML20216F542 ML20216F561 ML20216F582 ML20216F587 ML20216F606 ML20216F627 ML20216F726 ML20216F751 ML20216F759 ML20216F776 ML20216F809 ML20216F867 ML20216F949
References
FRN-62FR42426, RULE-PR-55 AF62-1-026, AF62-1-26, NUDOCS 9709120113
Download: ML20216F809 (1)

Text

)

1. osa uq A C & 2. - I f g' t UNITED STATES ppg s

j* NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20565-0001 s.,...../

August 30, 1996 MEMORANDUM 10: David L. Morrison, Director Office of Nuclear Regulatory Research FROM: William T. Russell, Director gA///f4-Office of Nuclear Reactor Regulation

}

SUBJECT:

RULEMAKlHG PLAN - CHANCES 10 10 CFR 55 -

AMENDING INITIAL OPERATOR LICENSING EXAMINATION REQUIREMENTS 3

As requested by your memorandum dated August 26, 1996, NRR has reviewed the

} subject rulemaking plan and concurs with the plan as revised in the attached markup.

The most significant revision is to delete reference to NUREG-1021 in the wording of the draft rule. This revision was based on a discussion of the issue between staff from OGC, RES, and NRR.

}-

If there are questions, please contact Stuart Richards, Chief, Operator Licensing Branch, NRR at 415-1031.

$gt

Attachment:

As stated t

g4*.g113970504 SS 62FR42426 PDR j

m.

1 .m* 4%a

-e "ln Clo

[ }*, UNITED STATES y%y j j t

NUCLEAR REGULATORY COMMISSION WASHINGTON, o.o. 20060 0001 '

' %, /

• • • • • August 26, 1996 MEMORANDUM TO: James Lieberman, Director -

Office of Enforcement William J. Olmstead, Associate General Counsel for Licensing and Regulation Office of the General Counsel William T. Russell, Director Office of Nuclear Reactor Regulation FROM: David L. Morrison, Director . ,

Office of Nuclear Regulatory Research -- gg ,

pp

SUBJECT:

RULEMAKING PLAN - CHANGES TO 10 CFR 55 - AMENDING INITIAL OPERATOR LICENSE EXAMINATION REQUIREMENTS Your concurrence is requested on the attached rulemaking plan package. This action is on the Chairman's tracking list with an aggressive schedule, therefore, please respond promptly.

Backaround:

Section 107 of the Atoirf: Energy Act of 1954, as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Current regulations in Part 55 dealing with initial license examinations for reactor operators (RO) and senior reactor operators (SRO) do not state who will write, administer, or grade the examination; however, the NRC has traditionally performed those tasks itself or through contract examiners. On March 24, 1995, the staff, in SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program," informed the Commission of its

. intent-to revise the aperator licensing program to allow greater participation

-by facility licensees and to eliminate contractor assistance in this area.

This action will allow staff to elimini..e between 33M and $4M annually on contractor support for examination preparation and administration. On April 18, 1995, the Commission approved the staff's proposal to initiate a transi-tion process to revise the operator licensing program, and directed the staff to carefully consider experience from pilot examinations before full implemen-tation. On August 15, 1995, the staff issued Generic Letter (GL) 95-06,

" Changes in the Operator Licensing Program," outlining the revised examination development process and soliciting volunteers to participate in pilot examint.-

tions to. evaluate ant refine the methodology.  ;

\

4 e f m e9ohgg.g*- -

6 3 gejam43.- eaes.

9 i

ff C M - [G l Yfh _

- . 4

'J. 1.ieberman et at.- ',

Et Between October 1, 1995 and April 5, 1996, the NRC conducted a pilot program in which it reviewed and approved 22 operator licensing examinations that~ had been written by facility licensees in accordance with the current examination development guidance (NUREG-1021, Rev. 7, Sup?lement 1), as supplemented by the' pilot examination guidance contained in G 95-06. The staff described the results of the pilot examinations in SECY-96-123, " Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996, and briefed the Comission during a public meeting on June 18, 1996. Based on the results of the pilot program, the staff recommended that the Comission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to draft the operator licensing examinations in accordance with NUREG-1021, " Operator-Licensing Examination Standards for Power Reactors." In a staff requirements memorandum (SRM) dated July 23, 1996, the Comission directed the staff to develoo a detailed rulemaking plan to justify the changes that may be neces-sary to 10 CFR Part 55. The SRM also directed the staff to specifically address-several issues about the pilot examination program and the proposed changes in the examination process.

The rulemaking plan and the response to the SRM's specific requests, which were prepared by the Office of Nuclear Reactor Regulation and reviewed and approved by the Office of Nuclear Regulatory Research, are attached to the enclosed Comission Paper One issue I would appreciate specific coment on is that of whether the rule should contain an explicit reference to NUREG-1021.or alternately require an operator licensing program acceptable to the NRC. If the former course is taken that will have the effect of codifying the NUREG and if we wish to change it'later based on experience with implementing the new rule a rulemaking would be raquired, if the later course is taken a regulatory guide would be issued referencing the NUREG and it would only be necessary to change the regulatory guide.

The following is a sumary of this request:

1.

Title:

Rulemaking Plan, " Changes to the Operator Licensing Process at Power Reactors (Part 55)."

3

2. Task leader: Harry S. Tovmassian (415-6231)

3. Steerina Group Members: None

4. Enhanced Public Participation:

This rulemaking will use electronic balletin boards, as appropriate to

' enhance input from the public.

5. Level 1 or 2 Item of Compatibility for Aareement States: No

6. Reouested Action: Office Concurrence y- - __

I l

J. Lieberman et at. l 7. Reauested Completion Date: August 30, 1996

8. Resources and Coordination:

Estimated resources to develop this rulemaking are .5 FTE. No contractor support is needed. A copy of this concurrence package has been forwarded to the Office of the Controller for coordination of resource issues and to the IG for information.

Attachment:

Memorandum to Taylor from Morrison w/enclr.

cc w/ attachment:

R. Scroggins, OC H. T. Bell, OIG G. Cranford, IRM D. Meyer, ADM i

w l

• f un 1

UNITED STATES NUCLEAR REGULATORY COMMISSION 3 <* waswinotou o.c. mwoooi

- 'l

%.....)

MEMORANDUM 10: James M. Taylor Executive Director for Operations fROM: David L. Morrison, Director-Office of Nuclear Regulatory Research

SUBJECT:

RULEMAK'NG PLAN - CHANGES T0 10 CFR PART 55 - AMENDING INITIAL OPERATOR LICENSE EXAMINATION REQUIREMENTS Attached for your signature is a Comission paper which requests the Commission's approval of the staff's rulemaking plan for requiring power

. reactor licensees to prepare and administer initial reactor operator (RO) and senior reactor operator-(SRO) licensing examinations. Section 107 of the Atomic Energy Act of 1954, as amended, requires the NRC to determine the qualifications . f individuals applying for an operator license, to prescribe uniform contlitions for licensing such individuals, and to issue licenses as appropriate. Current regulations in Part 55 dealing with initial license examinations for R0s and SR0s do not state who will prepare, administer, or

-grade the examination; however, the NRC.has traditionally performed those tasks itself or through contract examiners. On March 24, 1995, SECY-95-075,

" Proposed Changes to the NRC Operator Licensing Program," informed the Commission of the staff's intent to revise the operator licensing program to cllow greater participation by facility licensees and to eliminate contractor assistance in tht; area. This action will allow staff to eliminate between

$3M and $4M annually on contractor support for examination preparation and administration. On April 18, 1995, the Commission approved the r+aff's proposal to initiate a transition process to revise tne operator licensing program, and directed-the staff to carefully consider experience from pilot examinations before full implementation. On August 15, 1995, the staff issued Generic letter (GL) 95-06, " Changes in the Operator Licensing Program,"

outlining the revised examination development process and soliciting volunteers to participate in pilot examinations to evaluate and refine the methodology.

Between October 1, 1995 and April 5, 1996, the NRC conducted a pilot program in which it reviewed and approved 22 operator licensing examinations that had been prepared by facility licensees in accordance with the current examination development guidance (NUREG-1021. Rev. 7, Supplement 1), as supplemented by the' pilot examination guidance contained in GL 95-06. The staff described the results of the pilot examinations in UY-96-123, " Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996, and briefed tnb Commission during a public meeting on June 18, 1996. Based on the results of the pilot program, the staff recommended that the Commission approve the implementation of. the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to prepare the operator licensing examinations in accordance with FUREG-1021, " Operator

. +

f :.

J. M. Taylor ll Licensing Examination Standards for: Power Reactors." In a Staff Requirements-

- Memorandum (SRM) dated July 23, 1996, the Commission directed the staff to develop.a detailed-rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55. The SRM also directed _the staff to specifically address several issues about the pilot-examination program and the proposed

- changes in the examination process. The attached Commission Paper contains both the staff's rulemaking plan and a response to the issues raised in the SRM.

Attachment:

Commission Paper w/encls.

I w-- - -.

J. M. Taylor 2-Licensing Examination Standards for Power Reactors." In a Staff Requirements

~ Memorandum (SRM) dated July 23, 1996, the Comission directed the staff to develop a detailed rulemaking plan to justify the changes that may be-necessary to 10 CFR Part 55. The SRM also directed the staff to specifically address several issues about the pilot examination program and the proposed changes in the examination process. The attached Commission Paper contains both the staff's rulemaking plan and a response to the issues raised in the SRM.

Attachment:

Comission Paper w/encls.

DISTRIBUTION:

Central f/c RDB r/f

-RAuluck DMendiola CGallagher LRiani DOCUMENT NAME:0:\TOVMASSI\TRANSMEMO.EDO

/ .

OFC RDB:DRI RDB:DRA n _h I DkkhW( D:RES [if NAME HTovmassian:ayw SBahadur h BMorris DMorrkon!

DATE / d/96

/ 4/9y96 9 / 76/96 I [/M/96 0FFICE RECORD COPY ,

RES FILE CODE:

l c-

+. . . .- - . _-

--A.mi.-..,wJ-ss.s a e a .4.a.a.,,ua 4 a a , A.,., ,.s,g4 ., y a a pn__.ausaw,_ - - - . # .- - - - - - ._ n_,.-,---y.___.-

_s ._.

3 1

4 T

COMMISSION PAPER i

i t

i 4 o i.

e

c .

f.QB: _The Commissioners  ;

i faQM: James M. Taylor, Executive Director.for Operations -l

SUBJECT:

- RULEMAKING PLAN FOR AMENDMENTS TO 10 CFR PART 55 TO CHANGE LICENSED OPERATOR EXAMINATION REQUIREMENTS RUREQlf: in pc4 preAce Ae op hae c+bM h _

To provide additional inf o'rmation regarding the initia' operator licensing examination pilnt process as directed in a Staff Requirements Memorandum (SRM) of July 23, 1996 (M96061EA), and to request the Commiss ion's approval of the rulemaking plan to amend 10 CFR Part 55 to require pows r reactor facility licensees to. prepare and adminhter in' accordance with NUREG-1021, " Operator

- Licensing Examination Standards for Power Reactors," the written examinations i

and operating tests-that the NRC uses to determine the ccmpetence of operator license applicants.-

BACKGROUND:

On March 24, 1995, SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program," informed the Commission of the staff's intent to revise the operator licensing program to allow greater participation by facility

.g * - licensees and to eliminate contractor assistance in this area. This action-will allowistaff to eliminate between-53M and.54M on contractor support for-examination preparation and administratiM. On April 18, 1995, the Commission approved the staff's proposal to initiate a transition process to revise the 1

operator licensing program and directed-the staff to carefully consider

experience from pilot examinations before full implementation of the changes.

On August 15, 1995, the staff issued Generic letter (GL) 95-06, " Changes in the Operator Licensing Program," outlining the revised examination development procels and soliciting _ volunteers to participate in pilot examinations to evaluate and refine the methodology.

~

CONTACT:

Stuart-Richards,-NRR (301) 415-1031

~ .. - .

i- .

g1 -

k-4 The Comissioners SM-Between October 1,1995, and ' April 5,1996, the #RC- reviewed and approved -22

-operator licensing examinations that had-been prepared by facility licensees in-accordance-with the current examination development guidance (NUREG-1021 Rev. 7 _ Supplement 1) as supplemented by the pilot' examination guidance contained in GL 95-06. These examinatiu s were then used to test 146 reactor operator and senior reactor operator applicants. Facility licensees-from each NRC Region, operating reactors built by all power reactor vendors, participated in;the voluntary pilot program.

The staf f described the results of the pilot examinations in-SECY-96-123,

" Pro)osed thanges to the NRC Operator Licensing Program," dated June 10, 1996, and 3riefed the Comission during a public meeting on June 18, 1996. On the basis of the results of the pilot program,- the staff recomended that the Comission approve the implementation of the new examinatic:1 process on a voluntary basis until rulemaking could be completed to require all power

• pbc or__ facility licensees to prepare the operator licensina examinations in

'- ve* 4 - _ accorreact _ dance wFtnl NUREG-1021" Operator Licensing Examination Standards for

'm Power Reactors," which has been revised to-accomodate the new process. In an SRM dated July 23, 1996, the Commission authorized the staff to continue the pilot process through July 1997 and directed the staff to develop a detailed rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55. The SRM also directed the staff to specifically address several issues regarding the pilot examination program and the proposed changes in the examination process. , g g g j3,m y .g DISCUSSION: T^s "#

y wn "gee

• F$ A~ ~ ~

Section 107 of the Atomic Energy Act of 1954 as amended, requires the hRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55-to pass an examination satis "ying the basic content requirements specified in the regulation. ) art 55 does not state who must

. prepare, administer, or grade the examinati )n: however, the NRC has traditionally performed those tasks itself )r through contract examiners.

On the basis of the results of the pilot pr > gram as discussed in SECY-96-123, the staff now believes that 10_CfR Pa- 55 ihould be revised to require power reactor facility licensees to prepare their initial operator licensing examinations. Specifically, the staff recommends that Subpart E, " Written Examinations _and Operating Tests," of 10 CFR Part 55 be amended to require all

• 3 * * ~powtr reactor facility licensees to prepare ithe initial operator _ licensing u>d-- ~

examtiatlons in icTo7dlinM wHh1 NUREG-1021." The licensee-prepared

-* examinations will be subject to review, modification, and approval by NRC examiners before the examinations are administered. The regulation-will provide that:the NRC may prepare the examinations in lieu-of-accepting or modifying an examination prepared by the. facility licensee. The changes would not apply to non-power reactor licensees, i

Theh-nlemakinq plan for this proposed shangs-has-beEn'd'Teloped e in

" N i -- ? .3, "The Rulemaking

[accordancewith _ he SRM of July 23, Process " E .

Q ian p i

I i The Commissioners .

The enclosed rulemaking plan for this proposed change has been developed in accordance with the guidance in NRC Management Directive 6.3, "The ".ulemaking Process." As directed in the SRM of July 23, 1996, the rulemaking plan addresses the potential impact of the proposed rule on facility licensees and discusses why this rule will not constitute a backfit.

The second enclosure discusses the fellowing additional items as directed by the SRM:

e The pros, cons, and vulnerabilities associated with implementing the proposed examination program changes on an industry-wide basis (including a review of the impact on licensee and NRC resources, the potential effects on reactor safety, the impact of proposed NUREG-1021 revisions, potential public perceptions, and concerns regarding this approach raised by staff members).

• A discussion of the degree of acceptance of this new approach to operator licensing by facility licensees (including a discussion of industry comments received on draft NUREG-1021 and an explanation of the staff's response to those comments),

e An explanation of enhancements to the new licensing examination process resulting from the review and feedback from the pilot prograrr results (including a justification for subsequent proposed changes to the operator licensing program).

• The results of initial operator license examinations that have been given both under the pilot program and under the current program since SECY-96-123 was prepared (including trends observed).

RECOMMENDATION:

That the Commission --

1. Approve the development of the rule as described in the enclosed Rulemaking Plan.

2. Approve the implementation of Revision 8 of NUREG-1021 on a voluntary basis until the rulemaking is complete.

I

. The Commissioners C00RDINATIQ!i: L ngo :u ,* ,g 3 The Office of the deneral Counsel has no legal objection to this initiative, and the Offices of Nuclear Reactor Regulation and Nuclear Regulatory Research concur in this rulemaking plan.

James M. Taylor Executive Director for Operations

Enclosures:

1. Rulemaking Plan for Changes to 10 CFR Part 55

2. Response to SRM dtd July 23, 1996 (M960618A)

)

4 5

.The Ct.mmissioners -q. .

"'I -

COORDINATION:

The Office of the General Counsel has no legal objection to this initiative, and the Offices of Nuclear Reactor Regulation and Nuclear Regulatory Research concur in this rulemaking plan.

James M. Taylor Executive Director for Operations

Enclosures:

1. Rulemaking Plan for Changes to 10 CFR Part 55

2. -Response to SRM dtd July 23, 1996 (M960618A) f 1

Distribution:

RDB r/f Central file E00 r/f LRiani CGallagher DOCUMENT NAME: 0:\T0VMASSI\RULEPLAN.FNL ,

T. . 4 . . w. . m in wm e . m 4 .n-pm.%A<tw.A. E . co v .,m .n-sm.neww.or. u - wo co.,

V~

OFFeCE RDSDRA Mf f' l DRA:RDS C

, QQM l l D RES ////J __j EDO l waw srovm . n .v. e $hwur m,n. - w,=on f 57/ aT.vio, g pee ( ge v Datt irvas 7thes i ese OFFICIAL RECORD COPY

h

( .

j 9

RULEMAKING PLAN FOR CHANGES TO 10 CFR PART 55 ENCLOSURE 1

- ~ ~ - - - - - - - - - _ _ _ . _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _

i RULEMAKING PLAN CHANGES 10 10 CFR 55 AMENDING INITIAL OPERATOR LICENSE EXAMINATION REQUIREMENTS Lead Office: Offic, of Nuclear Regulatory Research Staff

Contact:

H. Tovmassian, RES/RDB Concurrences: ./fr/

D. MoVPison, RES

//W 2[f

' Dale /

W. Russell, NRR Date J. Lieberman, OE Date R- Olmstead, OGC Date Approval:

J. Taylor Date

RVLEMAKING PLAN d

CHANGES TO THE OPERATOR LICENSIEG PROCESS AT POWER REACTORS (10 CFR PART 551

1. Backaround On March 24, 1995, SECY-95-075, " Proposed Changes to the N RC Operator Licensing Program," informed the Commission of the staff's intent to revise the operator licensing program to allow greater parjicipation by facility licensees and to eliminate NRC contractor assistance in this area. This action will allow staff to eliminate between $3! and $4M on contractor support for examination preparation and administration. On April 18, 1995, the Co,wission ap poved the staff's proposal to initiatt.

a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full implementation of the changes. On August 15, 1995, the staff issued Generic Letter (GL) 95-06, " Changes in the Operator Licensing Program," which outlined the revised examination development process that would permit facility licensees to prepare the initial operator licensing examinations. GL 95-06 also solicited volunteers to participate in pilot examinations to evaluate and refine the methodology. p yhg Between October 1, 1995, and April 5, 1996, the NR: conducted 22 initial operator licensing examinations using the new methadology. The staff described the results of the pilot examinations in SECY-96-123,

" Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996, and briefed the Commission during a public meeting on June 18, 1996. On the basis of the results of the pilot program, the staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be ensees to prepare the completed to require all power reactor facility operator licensing examinations in accordance with li[NUREG-1021, " Operator Licensing Examination Standards for Power Reactors." In a staff requirements memorandum dated July 23, 1996, the Commission directed the staff to develop a detailed rulemaking plan to justify the changes that may be r.ecessary to 10 CFR Part 55.

II. Prooosed Change Revise Subpart E " Written Examinations and Operating Tests," to include examination development requirements for facility licensees.

Reaul atory Issue The pilot program demonstrated that the revised examination development process can be both effective and efficient. Comments from the NRC staff and industry personnel who participated in the pilet examinations were generally favorable. The quality of the administered examinations (as modified by the NRC) and the performance of the individuals who took the examinations were comparable to the quality of and the performance 1

on examinations written by the NRC staff or its contractors. The fact that some of the draft examinations submitted by facility licensees required significant rework illustrated that many facility staffs did not fully understand the criteria for writing an NRC examination. Both the industry and the NRC staff agree that, with training and exoerience, the industry should gain proficiency in preparing the examination materials.

The staff believes that the new process should be made mandatory for all power reactor facility licensees. The staff considers implementation of the new process on a voluntary basis alone unworkable over the long term. With the elimination of contractor support, the staff will no longer have sufficient m?iminer resources to write all examinations consistent with the schN aling needs of facility licensees. This resource problem is furtter compounded by the unpredictable nature of the examination workload.

The staff briefed the Committee To Review Generic Requirements (CRGR) af ter the pilot program was complete. In the minutes of its meeting, CRGR agreed that, with regard to the technical and safety aspects, the-proposed changes represented a reasonable and workable alternative approach. The committee members unanimously endorsed sending the proposal forward to the Executive Director for Operatiom (E00) for consideration. However, the CRGR also requested OGC to consider further the procedural question relating the staff's original plan to implement the revised process by generic letter.

The staff subsequently met with OGC to resolve CRGR's concern and concluded that requiring all facility licensees to prepare the examinations, a task long performed by the NRC, would require rulemaking or the issuance of orders to each facility licensee. Therefore, the staff recommended in SECY-96-123 that the Commission approve the staff's pursuit of rulemaking to require power reactor facility licensees to

- prepare the operator licensing examinations-in-accordance-win- X.

NUREG-10M, while continuing the pilot process on a voluntary basis.

Current Rule Reouirements Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe un crm conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55 to pass an examination satisfying the basic content requirements that are also specified in the regulation.

Specifically,10 CFR 55.31(a)(3) requires the applicant to submit a written reque? from an authorized representative of the facility licensee that the written examination and the operating test be administered to the applicant. Furthermore, 10 CIR 55.33(a)(2) states that the Commission will approve an initial application for a license if it finds that the applicant has passed the requisite written examination 2

' 7 and operating test in accordance with Sections 55.41 and 55.45 or 55.43 and 55.45. These examinations and tests determine whether the applicant for an operator's license has learned to operate a facility competently and safely, and additionally, in the case of a senior operator, whether the applicant has learned to direct the licensed activities of licensed operators competently and safely.

Subpart E, " Written Examinations and Operating _ Tests," includes specific sampling requirements for the content of written examinations (10 CFR 55.41 and 55.43) and operating tests (10 CFR 55.45).

Although Part 55 is silent regarding who will prepare, administer, and grade the written examinations and operating tests, the NRC or its contract examiners have traditionally performed those tasks.

Preliminary Reaulatory Analysis

1. Draft Rule Lanauaae The first sentence in Section 55.33(a)(2) would be revised as follows (new langtage is underlined):

"The applicant has passed the requisite written examination and operating test in accordance with Sections 55.40. 55.41, and 55.45 or 55.40. 55.43, and 55.45.

Section 55.40, " Implementation," would be added to Subpart E as follows: y.L e 6< 1-7pA -

pn',.oenad kn.

"(a) Power reactor facility licensees shall prepare 2nd :d+! ster...'

therequiredsite-specificwrittenexaminationandoperatingtestsM in :::crdance-with-the volicies-and-guidelines-in-NUREG-1021,

' Operator Meensing-Examination-Standards for-power--Reactors,".

(b) The Caissien W14-review,-modify-as-necesscryrand-approve-in acc0rd:nt: 4th 4WREbl 021r-t he- wr i t t en-e x ami na t i on s - and -ope ra t i ng 4e:ts prepared by the f4c444ty44censees. In 14eu-of-accepting the w4 tan-examinatten and eparating-tests-prepared by the facility licen:::, th: C:=!:: ten ny prepare-the-examinat4on-and-tests.

C' ) The Commission will prepare and administer the written examinations and operating tests at nor-power reactor facilities."

2. Imoact of the Rule Chance on Facility Licensees N The NRC currently depends on NRC employees and contractors to prepare and administer the initial operator licensing examinations required _by 10 CFR Part 55. In accordance with 10 CFR 170.12(i),

the cost of NRC time spent and any related contractual costs are billed directly to the facility licensees that receive the examination services.

1 3

n (b) b.ar - tmW (.w.% Msen skJ sht & wrmen eummh.s an A ogem.h{ds

.a h w s.4 % m,nm %m in .h.3

% v.s :m..a u . e w % ~., m m p ed o ped'; rg h3 S.

• * ^ i'^

r 4 L heo er re.cp,ciny o ye r%Yor C.\ity\k " D f*F"C C.c:ntM .isc,n fn<,,7 \es t-

[- W (+'$ c.rd -tm yaJ+ c %si'5tangedW we;thn capin mA',on , tk e.

yo - g u r. (sm $he> v._ +O ,

900.E(,,48% hdeskCe*JE {cpwNnc) ewurdf.d',ersa-rd M b { o .O D' l

$ ConMo b g wic\ e,rw e:,, on 8 w e,& )

• \eys M N e -

7

-Under the proposed change, each power reactor factaity-licensee will x

assume responsibility for prepari ng the site-specific initial operator licensing examinations at its facilities, thereby-allowing theNRC:todiscontinuetheuseof{contractexaminersforthat purpose, facility licensees will prepare and submit pro:osed examinations (including the written examination, the wal(-througN.

and the dynamic simulator tests) to the NRC in accordance with the cy h c.Ak ed intettetion in NUREG-1021. The NUREG has been edited to accommodate the revised examination process and is the same procedure that the NRC examiners or its centractors would use if they were to develop the examinations. In conformance with Section '

107 of the AEA, which requires the Commission to prescribe uniform conditions for licensing operators, the NRC will not c4MMer qM

- alternative testing methodologiespear Wu-> 9 (rom M 4,'o05 6 % .,pk y.baThe udaind m 6 4 .6 %64.n3 W e,a*'m % h sM M training staffs at power reactor facilities already have the ogrd .

basic knowledge, skills, and abilities necessary to evaluate operator performance and develop test items for the initial licensing examination.- During the mid-to-late-1980s, the industry's-emphasis in the training-area increased significantly. All power reactor licensees established formal training programs that were based on a systems approach to training (SAT) and accredited by the National Academy for Nuclear Training. Pursuant to 10 CFR 50.120 and 55.4, SAT-based training programs must include the evaluation of the trainee's mastery of training objectives. NRC inspections of licensee requalification programs for licensed operators have also found that training staffs generally possess the skills needed to evaluate the trainee's knowledge. 4 pree After the NRC\ reviews and approves an examination, the facility licensee williadministee and grade the written portion vr-peesce4bd u wo h e Q' % in NUREG-1021. The NRC staff will continue to administer and grade 9"" the operating tests, review and approve the written examination results recommended by the facility licensee, and make the final licensing decisions.

Feedback from the pilot examination program indicates that the average time spent by a facility licensee to prepare the written examination and operating tests was approximately 600 to 800 staff-hours. A portion of-that time (about 200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />) would have been spent reviewing and assisting with the administration of NRC-developed examinations under the process now in place and should be subtracted from the total. 'The resulting average burden of ,

approximately 400 to 600 staff-hours was somewhat higher than the 400 hours0.00463 days <br />0.111 hours <br />6.613757e-4 weeks <br />1.522e-4 months <br /> that NRC contract examiners typically take to prepare an examination. The extra burden is generally attributable to the facility licensees' lack of familiarity with ' specific NRC examination expectations and to the additional administrative ree.uirements, such as documenting the source of the examination questions, that are required to maintain examination integrity. It is worth noting that some of the facility licensees that

-participated in the pilot program expended less time than is 4

i

l ,

^M commonly used by NRC contractors to prepare the examinations.

Furthermore, in a few cases, the examinations that facility licensees submitted for review and approval were, in the judgment of NRC chief examiners, as good-or better than those prepared by an NRC contractor. The staff expects that most facility licensees will eventually be able to prepare quality examinations in less time than the NRC or a contractor because the facility employees have more-detailed knowledge of their facility and easier access to the l reference materials required to prepare the examinations.

Experience during the pilot examination program indicated that the-volume of reference materials required by the NRC examiners to review and prepare for the examinations w.s significantly less than that required fci the NRC staff to write the examinations. The f.ct that contract examiners will not be used in the revised examination process will also eliminate the need for duplicate sets of reference materials-to be provided by facility licensees. Feedback from the industry in response-to the staff's solicitation of public comments on the draft revision of NUREG-1021 indicated that facility licensees had been spending from 80 to 160 hours0.00185 days <br />0.0444 hours <br />2.645503e-4 weeks <br />6.088e-5 months <br /> to prepare and ship the reference materials under the existing examination process.

The additional burden of having to prepare the site-specific initial operator licensing examinations should be substantially, if not t

. totally, offset by reductions in Part 55 review fees billed to the facility licensees pursuant to 10 CFR 170.12(i). -Each facility licensee will be billed only for the time that the NRC staff spends to review the examination prepared by the facility licensee and to rework the examination, as necessary, to bring it up to NRC standards; no contractual costs will be accrued. Although several of the draft pilot examinations were of poor quality and took the NRC staff more time than expected to review and rework, the staff believes that additional cost reductions will be realized as facility licensees gain experience with the NRC examination requirements and the quality of the draft examinations improves.

This rule change will give facility licensees more control over the cost of their examination-services because they will be in a puttion to manage the quality of the product that is submitted to the NRC. The higher the quality of the examination the facility licensee submits, the lower the resulting charges. Under the existing examination process, facility licensees are responsible for the entire cost of preparing the examination, even if the contractor's submittal is of poor quality and requires significant rework by the NRC staff before it can be administered.

It is also worth noting that facility licensees will have-the option of. retaining the services of a contractor to prepare the license examination as the NRC often does under the current examination process. The staff understands that the NRC's examination contractors have already expressed an interest in providing their services directly to facility licensees. It-is reasonable to assume 5

4

i

- ,: x IY that-the cost of using this-option-would be comparable to the contractor service-fees.that the'NRC currently passes on to the facility licensees.

In summary, the staff views this process change-as resource neutral

.at worst and possibly resulting in a-resource savings-to facility ,

licensees over time.

3.- Imoact of the Rule Chanae on Operator License Aeolicants The goal of the rule change is to eliminate the NRC staff's reliance on contractor support for the site-specific initial operator licensing program, while complying with the AEA and maintaining the existing level of effectiveness. To the extent possible, the -

format, content, length, and~ level of difficulty of the examinations-will remain unchanged, thereby minimizing the impact of the rule change on'the operator license applicants. ,

m us NRC examiners will continue to review and approve every written examination and operating test before it-is administered., The examiners will work with the facility licensee's staff to modify the draft examinations as necessary to e a r c^-M iance with'NUREG-1021 b m*Y and to maintain consistency with prior licensing examinations at that facility and among facilities, gAce if the NRC decides to prepare thr examination in lieu of accepting an examination prepared by the freility licensee, the NRC examiners will use the same procedures and criteria (i.e., NUREG-1021)-that

-the facility licensee would have'used to prepare the examination.

The operator license applicants should not be-able to distinguish between an-examination that is prepared by the' facility licensee and .

one that is prepared by an NRC examiner.

4. Alternatives to the Rule Chanog If the Commission decides not to amend 10 CFR Part 55 as proposed by the staff, it would require the restoration of contractor funding-for the operator licensing program or the allocation of sufficient direct examiner resources in each regional office to satisfy the demand for initial licensino examinations and to conduct the licensed operator requalificition inspections. Historically, the NRC has spent approximately $3 million to $4 million per year on contractor support funding for the operator licensing program.

111. . Other Elementi

-Leaal Analysis by the Office of- the General Counsel (OGC)

'0GC has: reviewed this rulemaking plan and does not believe this action constitutes a backfit pursuant to 10 CFR 50.109. In addition, OGC has-concluded that this' action does not constitute a major rule pursuant- to 6

.5

L 1 49 l

l .

hwh the Small Business Regulatory fairness Act of 1996. Finally, at this time, OGC is not aware of any reason for a legal objection to this rulemaking plan.

3ackfit Analysis As discussed in SECY-96-123, OGC agrees with the staff's position that requiring the facility licensees to prepare the initial operator licensing examinations would not be a backfit pursuant to 10 CFR 50.109, which states in pertinent part that --

Backfitting is defined as the modification of or addition to systems, structures, components, or design of a facility; or the design approval or manufacturing license for a facility; or the procedures or organization required to design, construct or operate a facility; any of which may result from a new or amended provision in the Commission rules or the imposition of a ,

regulatory staff position interpreting the Commission rules that  :

is either new or different from a previously applicable staff position....

The proposed change does not result in a modification of or an addition to systems, structures, components, or the design of a facility. The change does not affect the design approval or manufacturing itcense for a facility. The procedures required to design or operate a facility will not be affected by the proposed change. The proposed change would require each nuclear power plant licensee to develop the tests that are used to qualify, as meeting the requirements of 10 CFR Part 55, those nuclear power plant operators whom the nuclear power plant licensee wishes to employ. Development of such tests are not considered to be

" procedures... required to... operate a facility." The tests are not applied to the facility licensee, but rather to the operator license candidates. Further, any procedure necessary to develop the test would not be useful in actually " operating" the facility, even if one broadly interprets " operating" as including any action necessary to comply with the Commission's regulations with respect to operation. The organization required to design or operate a facility will not be affected because all facility licensees already have a training staff to train and evaluate candidates for operator licenses and to train other members of the plant staff, as required by 10 CFR Part 55 and by 10 CFR 50.120. Therefore an organizational change is not required because of this process change.

Suecortino Documents A regulatory analysis and an Office of Management and Budget (GMB) statement will be prepared. Neither an environmental impact statement nor an environmental assessment will be prepared for this proposed rule, however, since it is the type of action described as a categorical exclusion-in 10 CFR 51.22(c)(1).

7

e-

$3* 4*

IMAGE EVAL.UATION O4p

/////

'd>[4*

. TEST TARGET (MT-3) e

<> +

1.0 lg au m 5!!!E i,l [m RM

~ ~ -

l.25 1.4 1.6 4 150mm >

4 6" >

4 <$

+

I '

xxxxf PHOTOGRAPHIC SCIENCES CORPORATION P.O. BOX 338 WEBSTER, NEW YORK 14580

1. D;f&

4 4

,g;%

4 p

(716) 265-1600

/

e)9 gb4 sq>%+k) g IMAGE EVALUATION TEST TARGET (MT-3)

///// 6 '

e!g 1.0 lf W Bla i

"ljIL??

i,i [u HM J1

~7 1.25 l.4 1.6 4 150mm >

4 6" >

4 $

4k?%,, /<!b

% g3 y p,,,/, _ _ = ,,,, _  % ,,%.3 ,

WEBSTER NEV/ YORK 14580

, (716) 265 1600

h 5' 3 h IMAGE EVALUATION 4

4) g

+N+& res11Amee1<m1.a>

(+b ,

1.0 Em W pljE4 l'i E im H M 1

1.8 I

~ ~ ' '

I.25 1.4 1.6 4 150mm >

4 6" >

$,% ii /A

+<%& o

$)F ___,_.,___

,,0 . . -

P.O. BOX 338 WEBSTER, NEW YORK 14580

+++gt <

4

, (716) 265 1600

s-h 9 *$r t 'A IMAGE EVALUATION ($ k g

////

/ gf TEST TARGET (MT-3) ,

k u

l.0 Em M y @ Ela 1.1 [" EE '

l.8 l=

l.25 1.4 1.6

4 150mm >

• 6" >

4 $

d#% #<#

• 4Nhp .

- = ,,u - -  %,Q;/O .

,,,< WEBSTER hE N YORK 14580

, (716) 265 1600

4# Q IMAGE EVALUATION ((p ' gbk, pp*,,g@ "9 *sh> TEST TARGET (MT-3)

"[ f[4'

1. g g777p

@ // 4 ,

, g9

+ r I.0 lt M BLA sljDE

! E in R M l _4l l=_ Lg

'v- i.25 u t6 4 150mm >

4 6" >

[%*% /+'O

+ Wb

+$1 %,,, 'N PHOTOGRAPHIC SCIENCES CORPORATION o ,,ee ,m eO o P.O. BOX 338 o4'*[b4 <>p WEBSTER, NEW YORK 14580 4

, (716) 265 1600

'N-- b;s)He ed Ptss,o sh.6 Menasb J A.a Mim d e. so m.3 bh py" %

du dd.\ '.ond @t %unee s 4e b e. 3d ke3,*

<u b h h m i sn eno.sete. 2 ' a n 3 m c,3" 4 \ Resources Reaqjrff As discussed previously in Section 11.2, the rule will require facility licensees to assume responsibility for writing the examinations, a '

burden currently carried by the NRC, However, the transfer of this responsibility is considered to be resource neutral, and could possibly result in a resource savings to facility licensees.

The rulemaking is expected to take approximately 0.5 staff years to complete, and no contractor support will be needed. However, until the rulemaking is complete and all power reactor facility licensees are required to prepare their examinations, the Office of Nuclear Reactor Regulation (NRR) will rontinue to use contiactors, as necasary, to supplement the =taff's ability to develop and administer the initial operator. licensing examinations and to conduct the requalification inspection program. When the rule goes into effect, the staff expects that the revised examination process and the requalification inspection program can be implemented at all power reactor facilities with the existing NRC resources allotted to the operator licensing programy J Coanizant Staff / Pro.iect Manaaement Concurrina Official Office of' Nuclear Regulatory Research (Lead): H. Tovmassian Bill Morris Office of Nuclear Reactor Regulation: S. Richards Bruce Boger Office of General Counsel: M. Schwartz Stuart Treby Steerina Group A steering group will not be required for this effort.

Enhanc.t.d_.Public Participat ion This rulemaking will use electronic bulletin boards, as appropriate, to enhance input from the public.

EDO or Commission Issuance The proposed and final rule sill be approved by the Commission.

Schedule Proposed rule published: 3 months after the E00 approves plan Comment period ends: 60 days after the proposed rule is publishea Final rule published: 6 months after the comment period ends 8

t II RESPONSF TO THE STAFF REQUIREMENTS MEMORANDUM 0F JULY 23, 1996 (M960618A)

ENCLOSURE 2

t DISCUS $10N OF PROPOSED OPERATOR LICENSING PROGRAM CHANGES

, y N3 % qpAd % un m ue w e4 p%e. 5 Tim, Backaround' L The sita-specific initial operator licensing examinat ion consists of a 100-question written examination and an operating test that includes a performance demonstration on a dynamic sipulator and-a-plant walkhthrough.- The written

) forinat, with a passing grade examination is multiple-choice ud clend of 00 percent.- The simulator lierformancedemons i::h j)tration generally consists of two or three scenarios administered to 7m of thren license applicants.

Each applicant is individually evaluateo .., a range o" competencies applicable ^

to the applicant's license level (i.e., reactor operator (RO) or senior reactor operator (SRO)). The walk-through portion of the operatirtg test T T W h requires each applichnt to individually complete a to al of to tasks,_or_jou performance measures (JPMs), in the control room and he plant and to answer The entire site-specificquestionsrelatedtotheassociatedsystems$yNRClicenseexaminers specific examination has historically been prepared or NRC contractors. As a prerequisite for taking the site-specific examination, every applicant must pass a separate generic fundamentals examination (GFE), consisting of 100 multiple-choice questions covering mechanical components, principles of heat transfer, thermodynamics, fluid mechanics, and fundamentals of reactor theory. An NRC contractor prepares two versions of the GFE (one for boiling water reactors and another for pressurized water reactors) twice per year. The GFEs are reviewed and 4

approved by the Operator Licensing Branch (HOLB) before being mailed to the facility licensees for administration.

On March 24, 1995, SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program," informed the Commission of the staff's intent to revise the operator licensing program to allow greater participation by facility licensees and to eliminate contractor assistance in this area. On April 18, 1995, the Commission consented to the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full implementation of the changes.

i From May to June 1995, the Division of Reactor Controls and Human f actors (DRCH) and HOLB conducted a number of meetings with senior management and regional personnel to discuss various options for implementing the proposal outlined in SECY-95-075. On August 15, 1995, after weighing the pros and cons of the various options (discussed bel 4), the staff issued Generic letter (GL) 95-06, " Changes in the Operator Licensing Program," which notified power reactor facility licensees of the NRC's intent to change the operator licensing process, outlined a number of criteria to supplement the guidance in NUREG-1021, " Operator Licensing Examiner Standards," and solicited volunteers to participate in a 6-month pilot program to evaluate and refine ti"e new examination methodology.

On June 10, 1996, the staff issued SECY 46-123, " Proposed Changes to the NRC Operator Licensing Program," which described the pilot examination process and ,

criteria, reviewed the examination results, discussed the lessons learned -and recommended that the enhanced process be implemented on a voluntary basfs I

s

pending an amendment to 10 CFR Part 55 that would require all power reactor facility licensees to prepare their licensing examinations. The staff briefed the Commission on June IJ, 1996, and in a staff requirements memorandum (SRM) dated July 23, 1996, was directed to develop a detailed rulemaking plan and to address the following specific issues regarding the proposed changes in the examination process.

Pros and Cons (Vulnerabilities)

In the early stages of developing the pilot examination process, the staff identified several concerns (examination integrity, quality, and consistency, independence and public perception, examination security, resources requirements, program stability, and examiner proficiency) that have played a dominant role in the design, implementation, and refinement of the revised examination methodology. The staff evaluated all of the pilot examination criteria specified in GL 95-06 in light of these coi. trolling concerns. The same concerns were also integral in evaluating the post-pilot process modifications that have been incorporated in the proposed revision of NUREG-102). Although the staff believes that each concern has been adequately addressed and mitigated, it would be disingenuous to assert that industry-wide implementation of the revised examination process does not entail a measure of uncertainty. The following is a discussion of each concern, the measures that have been implemented to mitigate the concern, and the associated vulnerabilities.

Integrity, Quality, and Consistency of the Examination Section 107 of the Atomic Energy Act of 1954, as amended, requires the Commission to prescribe uniform conditions for licensing individuals is operators, determine the qualifications of such individuals, and issue licenses to such individuals in such form as the Commission may prescribe, When the staff developed the pilot examination program, it was concerned about whether ihe existing levels of examination uniformity and consistency could be maintained. The operator licensing recentralization study (see SECY-93-309,

" Findings of the Operator Licensing Recentralization Study," of November 17, 1993), which had concluded that the level of examination uniformity was adequate to ensure that appropriate licensing decisions were being made, highlighted the challenge of controlling thE diversity that individual examiners from different regional offices could add to the examination program. Permitting a substantial number of facility employees having differing levels of training and experience to prepare the examinations could increase the amount of variation in the levels of knowledge tested and exmination difficulty despite the efforts of the NRC to control these factors through improved procedures and oversight reviews.

To maintain uniform standards of examination content, format, level of difficulty, and quality, the staff insisted that the participants in the pilot program prepare the written examinations and operating tests in accordance with the procedures and guidelines in Revision 7 of NUREG-1021. NUREG-1021 contains guidelines and criteria that are adequate to ensure that a licensing examination, independently prepared by an NRC examiner or contractor, will 2

4 1 differentiate between those applicants who have and those w..o have not i mastered the knowledge and abilities required to safelf operate he plant, flowever, when the staff reviewed those criteria in light of the proposed shift in responsibility for writing the examinations, it concluded that some of the criteria were no longer apprnpriate and that additional criteria were necessary to ensure that the integrity of the facility-developed examinations would remain at acceptable levels. Therefore, the staff promulgated the following supplemental and modified criteria in GL 95-06 when it solicited volunteers to participate in the pilot examination program:

o NUREG-1021 currently allows NRC and contract examiners to use up to 10 percent of the written questions, job performance measures, and simulator scenarios in a facil:ty licensee's examination bank when they develop an init:al licensing examination for the facility. This process saved NRC resources and provided an incentive for facility licensees to develop their examination banks because the examiner could conceivably extract the entire written examination from a facility licensee's bank, provided the bank contained 1,000 or more questions that satisfied the psychome*ric criteria in NUREG/BR-0122, " Examiners' Handbook for Developing Operator Licensing Written Examinations."

Whereas the NRC's examiners are continually revising old test items and developing new ones, there is currently no requirement for facility licensees to take comparable measures to minimize predictability.

Predictable examinations tend not to discriminate between competent and deficient Itcense appitcants because what is being tested is simple recognition of the answer. Although studying past examinations can have a positive learning value, total predictability of examination coverage through over-reliance upon examination banks reduces examination integrity. When the examinees know the precise and limited pool from which test items will be drawn, they will tend only to study from that pool (i.e., studying to the test) and may exclude from study the larger domain of job knowledge. When this situation occurs, it decreases confidence in the knowledge inferences that are made from performance on the test.

To limit predictability, GL 95-06 revised the limits on the use of facility item banks that could have been published, reviewed, or used as a basis for training. GL 95-06 treated all question banks as if they were open to the applicants because the staff had no basis to regulate the uualitiee. of the item banks and because it would be difficult to confirm that a bank was indeed closed to the applicants. The staff attempted to strike a balance between the use of new and existing test items because it appreciated the effort required to develop new test items and understood that existing items are a valuable training resource that should not be wasted. On the basis of the results of the generic fundamentals examination (GFE) program, the staff decided to limit the use of existing written questions on the site-specific examination to no more than 50 percent and to require at least 10 newly asveloped (i.e., not previously seen) questions on every examination.

The remaining questions could be generated by significantly modifying 3

existing bank questions. Each operating test was required to include at least two JPMs that were eitbar new or significantly altered and at least one simulator scenario that was new or significantly altered. The remaining scena tos had to be modified to the extent necessary to prevent the applicants from immediately recognizing the scenarios on the basis of the initial conditions or other cues.

o NUREG-1021 currently places a 25-percent limit on the number of written examination questions that can be repeated from the previous two NRC licensing examinations at the facility. To further limit the predictability of the examinations that would be prepared by the facility licensees, the staff revised this criterion to include all examinations, quizzes, and tests administered to the applicants during their license trainly class. Furthermore, no questions could be draw..

directly from the applicants' audit examination given by the fact!ity licensee at the end of the license training class to determine if the applicant was ready for licensing. This restriction was the subject of industry comments and has been revised as discussed below since completing the pilot program, o NRC and contract examiners are currently required to cite a specific facility reference for every question on the written examination.

Questions that were extracted from the facility licensee's question bank would be identified as such, in order for the NRC chief examiners to monitor the facility licensees' compliance with the previously stated criteria, GL 95-06 also required the pilot examination participants to state the history (e.g., bank, revised, new, and date last used) of each test item on the written examination and operating tests. Although this practice did place an additional burden on the facility licensees, it facilitated the NRC's examination reviews and resulted in lower licensing fees billed to the pilot participants. This requirement was also addressed in the industry comments and has been clarified as discussed below since completing the pilot program.

The Office of Nuclear Reactor Regulation (NRR) also issued supplementary written guidance to the regional offices clarifying the NRC examiners' responsibilities during the pilot examination development process, and it conducted a public workshop to discuss the staff's expectations during the pilot program. The written guidance and the minutes of the workshop were placed in the NRC's Public Document Room (POR) and provided to f acility licensees that were scheduled to participate in the pilot examination program.

NRR did not restrict the number or scope of changes that the NRC examiners could ask facility licensees to make in their proposed examinations. The existing quality assurance (QA) checklists in NUREG-1021 were revised and new checklists were developed to help the examiners identify problems with the preliminary examination outlines and the operating tests that the facility licensee would be required to submit for review.

Because of resource limitations, examiners were instructed to focus primarily on the content and construction (i.e., level of knowledge and difficulty) of 4

MA,,% b h sph m b ed m wts acu JS repe quA % %.rau- l cqe

s.rD,&.m Qr A 1 \

the written examination questions, rather than spend their time verifying the  !

detailed technical accuracy (e.g., procedural and component references) of i every question, as is required for NRC-developed examinations. The staff-expected the facility licensees to ensure that the examinations were technically sound and instructed the regions to question the licensees regarding any post-examination question deletions and answer key changes.

To assess the effectiveness of the pilot examination process, NRR asked the NRC examiners who were involved with the examinations to respond to a questionnaire on various aspects of the new methodology. The feedback indicated that many of the as-written examinations were comparable in quality l and level of difficulty to recent NRC-written examinations. However, the NRC t examiners and facility staff had to rework some of the examinations significantly to meet NRC standards. Although the staff had hoped that the technical accuracy of the facility licensee's written examinations would be as good as or better than the accuracy of the examinations written by the NRC, some of the pilot examination results (i.e., more question deletions and answer key changes than expected) indicated a need for improvement in that area.

Despite the challenges of quality and technical accuracy, the staff believes that each as-given pilot examination provided a valid basis for licensing the applicants who passed the examination, The staff expects that the consistency and quality of the draft examinations will impF

• as facility licensees gain experience and become more familiar with NRC e. m .ation requirements. Based on observations during the pilot examinations, w staff has also incorporated in Revision 8 of NUREG-1021 a number of process enhancements that are intended to improve the quality of the examinations developed under the revised process. A number of the significant enhancements are discussed below. NRR has also instructed the regional examiners and managers not to approve or administer any examination that does not adhere to current NRC standar.ds for content, format, quality, level of knowledge, and level of difficulty.' If necessary, the examination is to be delayed or cancelled until the facility licensee and the NRC staff can upgrade or replace the examination.

The overall results of the pilot program indicated that the examinations prepared by facility licensees, subject to review, revision where appropriate, and approval by NRC staff examiners, were generally consisteat with examinations prepared by NRC or contract examiners. The pilot examinations also appeared to be equally effective at identifying applicants who had not mastered the job requirements wel Pe w gh to become licensed R0s or SR0s.

Furthermore, the pass rates for RM a'no SR0s on the written examinations and A operating tests compared favor hiy-sith the pass rates for examinations prepared by the NRC or its contractors.

The chief examiners for approximately half of the pilot examinations actually considered the submitted examinations to be as good or cetter than recently administered NRC examinations because the facility authors were better able to integrate the required site-specific knowledge and abilities into operationally oriented questions. All but 5 of the 22 feedback surveys completed by the chief examiners indicated that the revised examination process was as ef fective or more ef fective than the traditional examination 5

process. Most examiners also agreed that the performance of the facility licensees should improve as they become more familiar with the NRC's requirements and expectations.

In summary, the operator licensing program staff has irc.plemented numerous procedural measures to maintain and promote the consistency, quality, and integrity of the initial licensing examinations . However, in practice, it will be the responsibility of individual examiners, with appropriate review and oversight by their supervisors and the NRR staff, to ensure that the licensing process remains strong, it is possible that the pressures to conduct examinations on schedule and the reduced involvement of the NRC in the examination development process could lead to examinations that discriminate at a lower level than is currently the norm. The revised examination process is much more dependent upon the staff's ability to identify a substandard examination and ensure that the deficiencies are corrected. If the staff is indecisive and the licensee takes advantage of the situation, the examination will probably not discriminate at the appropriate level.

Independence and Public Perception Before 1987, the staff believed that only independent!y developed and administered examinations could reliably and con +)stently achieve the appropriate standards of objectivity and discrimination required for the NRC's initial licensing and requalification programs. The staff was concerned that examinations prepared by a facility licensee could be unconsciously biased toward the specific material taught by the facility and that potential financial or other pressures on the facility licensee would create a conflict of interest.

The staff's position changed significantly in 1988 when it revised the requalification examination methodology so as to require each facility licensee to prepare the examination materials and demonstrate its ability to evaluate the competence of its licensed operators. From 1989 to 1994, the NRC evaluated over 4,000 licensed operators using requalification examinations that were prepared by the facility licensees, reviewed and approved by the NRC, and administered in parallel by NRC examiners and facility personnel.

The examinations appeared to discriminate at the appropriate level, and approximately 90 percent of the operators passed the examinations on their first attempt.

Despite the success of the requalification examination process, the staff continued to independently develop and administer the initial licensing examinations. However, the overall performance of the license applicants on the initial examinations was not significantly different from the performance of the licensed operators on the requalification examinations developed by their facility licensees. From 1989 to 1994, the NRC administered initial licensing examinations to approximately 4,000 applicants; the pass-rate on those examinations averaged approximately 90 percent.

In SECY-95-075, the staff proposed to take the next evolutionary step, similar to the staff's actions with regard to oversight of the requalification program, by changing the guidance in NUREG-1021 to permit facility licensees 6

to prepare and, in part, conduct the written and operating examinations for initial license applicants. The NRC would review the licensees' efforts, partially participate in the examinations, retain the final pass / fail authority, and continue to issue licenses. This action would allow the staff to perform its licensing function in a more efficient manner, while maintaining an effective, independent process upon which to base licensing decisions for operators, Shortly after SECY-95-075 was issued, The Washington Post reported that consumer advocates were concerned that allowing facility licensees to prepare the examinations might endanger reactor safety because the examinations might be less rigorous than those prepared by the NRC. The information upon which the consumer groups had based their conclusions was preliminary in nature and likely did not consid e the level of oversight that the NRC Nould retain in the examination process or the compensatory measures that the staff would implement to ensure that the examinations retained an appropriate level of difficulty.

The h shington Post article also indicated that the proposal appeared to contr. dict the staff's earlier decision to reject a request by the Virginia Electric and Power Company (VEPCO) to prepare its own examinations (this topic will be discussed in more detail iner) and that a number of mid-level staff members opposed the change. The staff believes that the initial opposition to the proposal was a natural reaction to the abrupt change in policy and to the fact that many staff members had not been consulted or informed before SECY-95-075 was issued. As promised in the memorandum of April 12, 1995, from the Executive Director for Operations, all involved staff members have been well informed of the new policy and have been able to express their views and to have their concerns considered as part of the process to implement the broad goals stated in SECY-95-075.

As noted in SECY-96-123, the results of the pilot program indicated that initial licensing examinations prepared by facility licensees, subject to review and approval by NRC staff examiners in a manner similar to the review and approval of NRC requalification examinations, were as effective as examinations written by contractors in identifying applicants who had not /

mastered the job requirements for a licensed operator. The pass rates for Rgs , x and SR0s on the writtan examinations and operating tests compared favorkbly with the pass rates for examinations prepared by the NRC and its contratfor,s. -

To promote examinrtion consistency and minimize the bias toward specific materials taught by the facility licensee, the staff insisted that the pilot examinations be developed in accordance with NUREG-1021 and NUREG/BR-0122.

NUREG/BR-0122 contains explicit guidance for sampling the important, safety-related knowledge and abilities (K/As) from NUREG-ll22, " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors," and NUREG-ll23, " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors," companion documents derived from a job-task analysis of the operators' work. The supplementary pilot examination criteria in GL 95-06 required the facility licensee to submit a proposed examination outline tc the NRC for review and approval at least 60 days before the scheduled examination date. This requirement would enable the NRC chief 7

4

- examiner to assess whether the sampling was appropriate (e.g.,-the balance of  !

coverage and importance ratings; duplication and overlap between-examination sections and from prior examinations; and inclusion of plant-specific priorities, such as recent events and risk insights) before the facility. .

licensee proceeded with developing the detailed examination materials.

GL 95-06 also required facility licensees to submit the proposed examination to the NRC for review at least 30 days before the scheduled administration date. The NRC chief examiner and the responsible supervisor would work with the-facility licensee to revise the examination as appropriate before

- authorizing its administration.

To minimize personal conflicts of interest, GL 95-06 )rohibited facility employees who played a substantial role in training tie license applicants from becoming involved in reparing the licensing examinations. Although this restriction did little to mitigate the fact that everyone who worked for the facility licensee was likely to have a vested interest in the outcome of the examinations, it did alleviate some of the concerns regarding personal gains or penalties that the examination author might incur if he or she were directly accountable for the examination results. This restriction also allayed the staff's concern that an instructor might unconsciously bias the '

examination toward those topics that were emphasized during the training program. The personnel restrictions were undoubtedly the most controversial

aspect of the pilot examination process. They were widely discussed among the ,

' NRC staff and management before, during, and after the pilot program was implemented, and they were also the principal issue raised by industry

regarding the pilot process. A discussion of this and a number of other <

industry concerns follows.

The issues of independence and public perception also had a significant bearing on the methodology that was selected for administering the operating tests, the best measure of the applicants' ability to respond to a plant event.- Although general agreement existed among the staff on rost of the pilot criteria discussed earlier, there was considerable disagreement

regarding the level of involvement that NRC examiners should retain in the operating test administration process. The following options were evaluated in depth

Option 1: NRC examiners would continue to independently administer the simulator scenarios and the plant walk-through portion of the operating test in accordance with current guidance in NUREG -1021. Involvemen' by the facility licensees would not change from current practice, and facility employees would not conduct paralle? evaluations.

Option 2: NRC examiners and facility employees would conduct parallel evaluations of each applicant during the dynamic. simulator and walk-through portions of the operating test. This is the same methodology that is used during NRC-conducted requalification examinations.

NRC examiners would observe facility employees conduct the

. Option 3:

simulator scenarios and a sample of the walk-through tasks 8

e-y y y .ps - --s w - -e . - p y g ,

- . ~ . _ . -_ . - - . .

' for each applicant; however, the NRC examiners would not conduct one-on-one parallel evaluations of each applicant.

This methodology is similar to that currently used during the requalification inspection program.

After much deliberation, the staff decided to implement Option 1 during the pilot examination program. The staff believed that Option I would have the most favorable public perception and examiner acceMance because it maintains the existing levels of independence and objectivit; ay relying on NRC examiners to conduct and evaluate the operating tests. Option 2 would have generated more concern regarding conflicts of interest because it would require facility employees to administer and grade the operating tests (to

-their coworkers and friends) in parallel with NRC examiners. Option 3 would have had the least NRC invohement and, therefore, the greatest potential impact on independence and public perception. By not directly observing each applicant,-the NRC examiner would be in a weaker position to judge the performance of the applicant so as to determine which applicants should pass or fail if a crew responded incorrectly during a simulator scenario.

In summary, it is possible that the pressures to conduct the examinations on schedule and the reduced involvement of the NRC in the examination development process could lead to examinations that reflect the biases of the facility licensee and that discriminate at a lower level than is currently the norm.

However, a number of process requirements have been augmented as necessary to minimize'this potential. The staff believes that the revised examination process, which includes restrictions on personnel, independent NRC review'and approval of the examinations, and independent NRC administration of tne operating tests, will provide sufficient safeguards to ensure unbiasei licensing decisions.

Given the complexity of the proposed change in the examination process, the public may jump to the conclusion that the NRC has transferred all responsibility for examining operator license applicants to the facility licensees, in actuality, the NRC retains a significant amount of control over the content of the examinations and continues to administer the operating tests. The challenge may be to explain this fact to the public.

Examination Security

• In 10 CFR 55.49, applicants, licensees, and facility licensees are prohibited from engaging in any activity that cor..comises the integrity (security) of any -

application, test, or examination required by Part 55.

Before 1989, ensuring compliance with the rule was relatively straightforward because f acility licensees were not permitted to review the written operator licensing examination until it was distributed to the applicants on examination day. Security compromises-and cheating were rare but not unheard of.

The degree of risk to examination security increased significantly in January 1989 with Revision 5 of NUREG-1021. Revision 5 made the facility licensees largely responsible for preparing the NRC requalification examinations and, 9

6

- . ~ . -- --- - - - - - - - .. - - . _ .- _ . - . -

4 j

V - for the first time, also allowed the facility staff to review the written .

licensing examination up to 2 weeks before its administration. The increased

~'

risk to security was' tolerated because the prereviews helped identify inaccurate questions before the examinations were administered,-thereby-_

increasing the content validity of the examinations, minimizing the number of questions from applicants during the examination, and reducing the number of -

o question deletions and changes during the grading process. The gradual shift to multiple-choice questions during the late 1980s and early 1990s >1 aced a-premium on accuracy because the answers to the questions could not >e adjusted after the fact as they could be with the essay and short answer nuestions that

they replaced.

To maximize security, given the new philosophy, NUREG-1021 prohibits .ny facility employee with specific knowledge of any NRC examination before it--is -

given from communicating the examination contents to unauthorized individuals and from participating in any further instruction of the students scheduled to take the examination. Before they are given access to the examination, the facility employees are required to sign a~ statement acknouledging their-understanding of the restrictions and the potential consequences of noncompliance. They must-also sign a post-examination statement certifying that they did not knowingly compromise the examination.

The staff realized that the potential for examination compromise would l

increase as the facility licensees increased their role in the initial licensing process from one of reviewing the NRC-developed examinations to that of developing their own examinations. Therefore, the staff attempted to clarify its security expectations in the supplemental pilot examination guidance that was provided to the regional offices (and placed in the POR) in conjunction with GL 95-06. The guidance addressed the' restrictions on personnel and a number of physical securf ty precautions, including protecting and mailing the examination materials. Ine guidance also cautioned examiners to be attentive to examination security measures and required them to review the security requirements with the facility licensee's contact at the time the examination arrangements were confirmed.

The issue of examination security also had a bearing on the methodology that the staff selected for administering the pilot operating tests. The staff believed that Option I would pose the lowest threat to examination security because the smallest number of facility employees would require detailed knowledge of the examination before it was given. Options 2 and 3 both would have required the facility licensee to evaluate the license applicants during the _ simulator and walk-through tests, which could have required several additional personnel to familiarize themselves with the operating test content, thereby increasing the risk of compromise.

In summary, the staff expects that the vast majority of facility licensees will continue.to maintain proper examination security (as they have during the requalification program). -Security is enhanced by the f act that the NRC monitors and oversees an extensive range of the facility licensees' activities and is sensitive to allegations and other performance indicators that might expose a problem. Nevertheless, the staff cannot deny that the NRC is now more vulnerable to examination compromise (and the associated public response) 10

• N e q \e.,on, h M k g d eu,@ed ha $ m rh j W <-, c'49t t *d e A- N G em c b r in 6*q, ,

t- ,

i k.ibroPdb [ber.sl(N\u,ba.Get weeb < c.qrm cd h4 '

a en w \,e y e before 1989 when it first-allowed facility licensees to become

~q -than it was

-involved wl;h the NRC's. examination development pro rams. The staff anticipates that the agency may suffer an occasiona lapseinsecurity.IWhen a lapse occ ars, the staff will continue to take prompt action to cancel the examination, determine the cause of the compromise, and implement corrective measures C.de<a d O Resourcos As stated in SECY-95-075, the proposed changes wer: criginally conceived as part of the NRC's continuing efforts to ttreamline the functions of the

. Federal Government consistent with the Administration's initiatives and to accommodate NRC resource reductions. The original plan was to entirely eliminate the use of .ontractors in the operator licensing area-and to conduct

\i all operator licensing program activities including the generic fundamentals and initial licensing examinations, and the requalification program, with the

' existing staff resources. The facility licensees would prepare and, in part, conduct the initial licensing examinations with oversight by the NRC. The NRC's participation would range from conducting part to all of every examination in order to provide an independent basis for making a licensing decision and to maintain examiner skills. ,g The staff estimated that a qualified facility employee should be able to prepare the site-specific examination in accordance withlNUREG-1021, as supplemented by Gl. 95-06, in approximately the same amount of time that is currently allocated to a contract examiner. (See Attachment 1, the associated rulemaking plan, for a more detailed discussion of the estimated resource burden on facility licensees.) The staff also estimated that an experienced NRC examiner should take approximately two weeks to review the site-specific examination (including the written and operating tests) prepared by a facility licensee.

As previously noted, the staff considered three options for administering the

' operating tests. Assuming a license class consisting of nine applicants (e.g., six R0s and three SR0s) that could be arranged in three, three-person crews, both the first and second options would generally require three NRC examiners to spend a week at the site to administer the tests. The second option, which would require the facility licensee to conduct parallel evaluations of each applicant during the simulator scenarios and walk-through tests, would place a significant additional burden (nominally, three staff-weeks) on ihe facility licensee. The third option would be equally burdensome to the facility licensee but would permit.the NRC to reduce (by one) the

number of examiners dispatched to the site.

The plan to require the facility licensee to evaluate the applicants during the operating tests was perceived as a duplication of effort because the facility licensee had already evaluated the applicants and found them qualified before it submitted the license applications to the NRC. The staff was also concerned that the additional regulatory burden would discourage participation in what was originally envisioned as a voluntary program with no associated rulemaking.

11

-w w y vi-- f- w

The NRC resource utilization rate during the pilot examinations supports the staff's original estimate that the new examination process and the requalification inspection program could be implemented at all power reactor facilities with the same level of direct NRC resources as is currently allotted to the operator licensing program. As noted in SECY-96-123, the NRC examiners required an average of about 350 hours0.00405 days <br />0.0972 hours <br />5.787037e-4 weeks <br />1.33175e-4 months <br /> to review, prepare for, administer, grade, and document each pilot examination. Most of the examinations included an extra visit to the site to review and validate the cperating tests.

However, with the elimination of contractor support, the total number of examiners available will decline significantly, thereby reducing the ability of the program to have the personnel available to meet peaks in facility licensee requests for exan.; nations. The tropact of the loss of examiners du; to reassignments, transfers, promotions, and other personnel actions, will be larger without the contract examiners available to compensate for the loss.

To compensate in this area, NRR intends to require that the regional offices maintain a defined minimum level of examiners, to be defined in each regional operating plan. During fiscal years (FYs) 1997 and 1998, each region may need to qualify additional examiners to meet this expectation. To specifically assist the regions in their efforts to train additional examiners, NRR intends to make available to the regional offices up to 7 full time equivalents (FTE) of inspection support during FY 1997 and up to 12 FTE of inspection support in FY 1998. This inspection support will allow regional personnel, who would normally be assigned to inspection activities, to be assigned to qualify as examiners. By the end of FY 1998, the regions will be expected to maintain the defined level of examiners in order to provide the resource flexibility nceded to deal with workload peaks. When examination work is not scheduled, the examiners will be available for inspection activities. A reduced level of contractor support will be required during the transition period.

The staff expects that if the quality of an examination prepared by a facility licensee deviates significantly from established norms, it will become increasingly difficult to predict the amount of time necessary to review or rewrite the axamination so as to achieve an acceptable product. This was the case with several of the pilot examinations, and the staff expects that it will continue to be more of a problem with facility-prepared examinations than it has been with those prepared by an NRC contractor. This added uncertainty in the examination process could increase the risk of broken examination commitments and lower exauination quality; it could also raise costs as a result of inefficient planning.

With the elimination of contractor support and the increased uncertainty about the examination quality, the staff may no longer have sufficient examiner resources to write or even to review all examinations consistent with the scheduling needs of facility licensees. This resource problem is further compounded by the unpredictable nature of the examination workload and by other unanticipated demands on the examiner work force, such as the increase in the number of examination appeals during the pilot examination program.

Historically, the contract examiners have facilitated the staff's ability to respond to unexpected increases in the examination workload and unforeseen departures by staff examiners. Eliminating contractor support will complicate 12

scheduling and limit the staff's ability to conduct retake examinations, which could increase the likelihood that an applicant will appeal the re sults of the examination. It is for these reasons that the staff is recommending to the Commission that 10 CFR Part 55 be amended to require all facility licensees to prepare the licensing examinations.

With regard to the generic fundamentals examination (GFE), the staff considered the following options: (1) retaining the current format and process under which the examinations are developed by a commercial contractor and (2) revising the scope of the site-specific written examination to sample the generic fundamental knowledge and abilities, as was done before 1988. If the examination was revised by adding more questions, it would increase the burden of writing and reviewing the examinations (whether it was done by the facility licensee or the NRC). If the length of the examination was kept the same, inclusion of the generic knowledge and abilities would weaken the evaluation of the site-specific systems and procedures. The staff and the indestry generally have agreed that the current program is both efficient and effective and that revising the site-specific sample plan would create too many problems. Accordingly, the staff has decided to retain the current examination format and process and to implement the program with contractor assistance at a cost of about $200,000 per year.

Program Stability from 1988 to 1990, the NRC received numerous complaints from facility licensees and licensed operators that the NRC's requalification examination program was causing undue stress for the operators and the facility employees who were administering the examinations. The Human factors Assessment Branch conducted a study to address this issue and determined that the frequent changes to the requalification examinat,on process ranked among the most significant sources of undue stress; the results of the study are documented in SECY-91-391, 'Results of the Study of Requalification Examination Stress."

One of the corrective actions involved establishing a schedule for periodic revisions of NUREG-1021 and delaying the implementation of each revision for 180 days after it is published.

Although the staff did not survey license applicants during the requalification stress study, it has generally tried to minimize the adverse effect on applicants by avoiding unnecessary changes to the initial examination process. Therefore, when the staff decided to shift responsibility for developing the int. al licensing examinations to the facility licensees, it resisted pressure to revise the examination format. To the extent possible, the format, content, and level of difficulty of the pilot examinations would remain unchanged, tt.ereby making the changes imperceptible to the license applicants. The staff also believed that the facility licensees were generally familiar with the existing requirements in NUREG-1021 and that any significant changes in the examination format would confound their efforts to assume responsibility for writing the examinations.

The desire to maintain program stability, avoid unnecessary changes, and minimize stress on the applicants also argued against the adoption of Options 2 and 3 for administering the operating tests. The requirement for facility 13

. =__ ..

personnel to evaluate the license applicants would increase the learning curve for the facility licensees and would likely elevate the stress for the applicants because of the double scrutiny and the increased crowding in the simulator.

In summary, the staff found no compelling reason to modify the existing examination format. The benefits of retaining the current format far exceed any argument for making changes at the present time.

Examiner Proficiency On May 25, 1995, the staff briefed the Commission on the status of the o

operator licensing programs and tue staff's proposal to shift respons1'ility for preparing the initial operator licensing examinations to the facility licensees, in the ensuing SRM of June 2,1995, the Commission instructed the staff to ensure that the in-house capability to administer the examinations is maintained, possibly through the random addnistration of some examinations.

se 9 kw wwA .n As noted in SECY-96-?,23, the staff beliEEiI~that NRC examiners will inaintain the ability to write examinations, if necessary, based on their participation in the new process. Although facility licensees will prepare the operator licensing examinations in accordance withiNUREG-1021, NRC examiners will continue to review and approve every examination before it is administered to ensure that it conforms to the criteria specified in NUREG-102i for content, format, quality, and level of knowledge, f acility licensees will also administer. and grade the written examinations in accordance with NRC ge proccdures, but NRC examiners will cor.tinue to independently administer and grade bott 'he dynamic simulator and the plant walk-through portions of the operating tests. The NRC will review and approve the assignment of grades for the written examinations, including any changes to the answer keys and the deletion of questions recommended by the facility licensee. The NRC will also continue to make all licensing decisions and administer the appeals process for applicants who fail the examination. p,c AMAN, % _% % ,c*W A hmbMnsee

• F("n ""h c

m ao

, a , - s A A y h pr % a c A ,4., t<.NRR will continue to require new NRC license examiners

% ~pc to comp standardized training program before they are certified to conduct licensing "- WW examinations. The training program includes classroom and simulator instruction on how to prepare and administer written examinations and operating tests, self-study of operator licensing procedures and guidance, on-the-job training, and a final practical evaluation by a certified chief examiner. Once certified, examiners are required to maintain their qualification by conducting examinations and attending refresher training.

NRR is currently revising the examiner refresher training syllaLds to focus more attention on reviewing, critiquing, and approving written examination questions. Examiners are also required to attend the operator licensing examiners' training conference, which is periodically convened by NRR to provide guidance and exchange information.

To ensure that the on-site examination activities are being performed as intended, supervisors are required to periodically evaluate every certified examiner during the conduct of an operating test. Furthermore, to verify compl4ance-with NUREG-1021gand sess the uniformity with which the wu gg  %

14 A na un w ea rn

l 2 examinations are administered within and among the regions, the NRR staff periodically audits the administration of operator licensing examinations at-selected facilities and annually reviews the~ implementation of the operator licensing. function in each region'. The NRR audits and reviews include an' assessment of. the level of knowledge and the level-of difficulty of the written examinations and the operating tests; each audit and review is '

- documented in a report to the responsible regional Olvision Director.

ladustry' Acceptance and Response In the-summer of 1994, VEPC0 recommended to the NRC that the initial licensing examination process be changed to make-facility licensees responsible for developing the examinations subject to review and approval by the NRC chief examiner. The NRC wotJ.d continue to administer and grade the examinations.

The proposal, which was submitted as part of VEPCO's cost-beneficial licensing action program, estimated that the utility could save an average of $67,500 per station for each initial license class by implementing the proposed approach. The staff reviewed and denied the proposal because of concerns regarding independence and objectivity and the potential for increased variability in the level of knowledge tested and examination difficulty.

The staff reconsidered its positim early in 1995 when the NRC's fiscal expectations changed significantly. The staff decided to explore the r possibility of eliminating the use of contractors in the operator licensing program by allowing facility licensees to develop the examinations, essentially just as VEPC0 had proposed. It was estimated that such a change would save the NRC from $3 million to $4 million per year in contractor 4

support funding and that the reduction in recovered fees would largely or completely compensate the facility licensees for their additional expenses. i i

While evaluating various options for implementing the changes proposed in SECY-95-075, the staff obtained informal feedback from the Nuclear Energy Institute (NEI) regarding the industry's level of involvement in administering the operating tests. NE! had polled a number of facility licensees and determined that there was no support for parallel grading (Option 2 previously i discussed) because of concerns regarding applicant stress, the additional resource burden, and simulator crowding. The majority of those surveyed expressed cautious support for Option 3, under which NRC examiners-would have observed the facility licensee administer the operating tests. However, those surveyed wanted the NRC cxaminers to focus only on the performance of the applicants ad not on the examination practices of the facility licensee, i Option 3 would shift the most responsibility to the facility licensees, but it was viewed as the most' difficult option to implement. Facility licensees perceived that Option 1, under which NRC examiners independently administer the operating tests, would be the least. demanding on facility licensees and the easiest to implement and would retain the greatest level of independence

. in the licensing decisions. NE! indicated ~that all of the facility licensees surveyed were satisfied with. the GFE and thought it should be continued in the current manner.

The staff believes that the industry's response to Gl. 95-Ofs which described the pilot process and solicited volunteers to participate in a six-month

' 15 i

i

, ~, -,~w , , -- - c. - . . - . - . - - .-

~

program to evaluate its feasibility. was generally favorable. Of the 35 facility licensees that had requested: examinations during the pilot period, 22 agreed to prepare their own examinations in accordance with GL 95-06. Most of the facility licensees that declined to participate cited as their reason

- their inability to reschedule the required resources on such short notice because of the restrictions on the use of personnel who h6d been involved in training the license applicants.

In September 1995, shortly before beginning the pilot examinations, the staff convened a public workshop to discuss the NRC's expectations regarding industry participation in the pilot program for the operator l hensing examination. The workshop also gave facility representatives an opportunity to ask questions of the staff and to submit written comments regarding the pilot examination criteria outlined in GL 95-06. Most of the questions and comments related to the restrictions on which personnel the facility licensee could use to prepare the examination and the requirement to document the history of the examination materials. Some facility licensees questioned the amount of reference materials that the NRC would need to review knd prepare for the examination. The staff challenged the pilot examination participants who expected to have problems satisfying the personnel restrictions to suggest alternative ways to achieve the same objective. The staff comitted to review the alternatives on a case-by-case basis and to propose a final position in Revision 8 of NUREG-1021, which would be publisbd for coment near the end of the pilot program. The staff also made it clear from the beginning that facility licensees could use contractors to develop the examinations.

Feedback from NRC examinere, who participated in the pilot exa.1 nations and industry representatives who attended a public progress meetirq sponsored by NEl in January 1996 confirmed that the personnel restrictions were the most troublesome aspect of the pilot process. Many facilities, particularly those with smaller training staffs, contended that the quality of the examinations was suffering becausc most of their best qualified personnel were involved in teaching and were ineligible to prepare the examinations. Those attending the meeting generally agreed that the industry and the NRC had learned a great deal from the pilot program, that most of the issues could likely be resolved through training and improved communication, and that the pilot process should be fully implemented, in february 1996, the staff issued draft Revision 8 of NUREG-1021 that incorporated both the )ilot process outlined in GL 95-06 and preliminary lessons learned from tic early pilot examinations. The draft revision was sent to the PDR and placed on the NRC's home page on the World Wide Web, and a federal Register notice requesting industry and public comments was published on February 22, 1996. NEI submitted comments and recommendations on behalf of the nuclear industry, and two facility licensees (VEPCO and Commonwealth Edison Company) provided additional comments. On April 18, 1996, the staff conducted a public meeting with NEl and otbcr industry representatives to review their comments and recommendations regarding the draft revision of NUREG-1021. The meeting also provided an opportunity to discuss additional issues and-proposed changes that had emerged following the issuance of draft

- Revision 8 of NbREG-1021 for public comment. The significant comments and staff responses are as follows:

16 3

e , n we w v

-. = - . . -. _- - . - .-

t l

} o All three commenters suggested that the personnel restrictions placed on  ;

individuals involved in writing the examinations should parallel the personnel restrictions of the requalif_ication program (i.e., the examination authors must stop teaching when examination development begins). The industry believes that those restrictions, when combined with the examination development and review criteria, would provide .

adequate protection against the unconscious biasing of an examination by '

an individual.

It is difficult to argue that the instructors would not be the best '

qualified individuals to prepare the examinations. Many NRC examiners also agreed that the requalification program restrictions would be adequate. However, the staff concluded that independence and conflict of interest were of sufficient concern to require personnel restrictions very similar to tho,e outlined for the pilot process. The staff revised the wording in the draft NUREG to clarify and quantify the asunt of involvement that would disqualify an employee from participating in various aspects of examination development, for example, the personnel-wht prepare the examination outline, including the selection of knowledge and abilities to be evaluated on the written examination and the JPMs and simulator scenarios to be performed during the operating test, shall not have had any direct involvement in training the license applicants, furthermore, only one person with limited involvement (i.e., who taught more than 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> but who was responsible for no more than 15 percent of the schedulad classroom instruction) may participate in developing the written examination questions, and no one may develop questions for the topics they taught, fa W ity licensees may still propose alternative approaches for considera> ion on a case-by-case basis, o All three commenters questioned the need for facility licensees to document the history of every test item proposed for use on a licensing examination, even if it was drawn from another facility licensee's examination bank.

The staff agreed that the wording in the draft NUREG could have resulted in misunderstandings of what was expected. The staff still believes that it is important to monitor the predictability of the examinations but it lacks the data and the resources to independently identify the source of every test item. This information is readily available to the fu.ility licensee. The staff h6 clarified the guidance in NUREG-1021 so that test items that-a facility _ licensee obtains from another bank and deposits into its own bank may be treated as " bank" items, provided the test items have an equal chance of being selected for use on the examination. Items from another bank may be treated as new items if they have not been made aval~ table for review and study by the license applicants and there is no basis (e.g., historical precedent or reciprocal arrangements with the other f acility licensee) for the applicants to predict their use on the examination. The staff believes that this revision will somewhat lessen the burden on facility 17 y u+ w w.-

T

^

• 4 ~ licensees,- while enabling 'the' NRC to maintain an acceptable level of oversight, o- - All three commenters suggested that NUREG-1021 relax the restrictions on-

- the duplication of test items from the applicants' audit-examination (a --

practice test given by the facility licensee before the NRC examination) if the audit examination was developed independently of the license .

examination, Under the traditional examination method, the-staff does-not have access to-the facility licensee's audit examination, so any duplication that

occurs between the audit examination and the licensing examination is unmonitored and coincidental. The staff agreed that-the proposal was consistent with current practice and changed the NUREG to allow up to five questions to be duplicated at random if the two examinations are written independently (i.e., no interaction between the examination authors) and the facility licensee identifies the duplicates. Licensing.

examinations that are not. developed independently shall not rereat any questions-directly from the audit examination.

! o All three commenters recommended that NUREG-1021 accommodate-facility

' licensees that have " closed" examination banks (banks i,ct available to.

applicants for study) by relaxing the limit on the number of test items that can be drawn directly from the bank.

The staff acknowledged that this is a valid issue, but because tbs >

issue is so complex, the staff does not intend at this point to establish additional guidelines regarding the maintenana of examination '

c banks. As noted earlier, the staff intends to treat all buks as if they are "open" at this time, o All three commenters ob,jected to the suggestion in draft NUREG-1021 that facility licensees consider providing an explanation as to why the ansar to every written (multiple-choice) question is correct and each of the distractors is plausible but incorrect. The industry thought

' 'his was a significant administrative burden that would, in practice, become a requirement.

Feedback from the NRC chief examiners indicated that some facility licensees adopted this practice on their own during the pilot examinations; it proved very beneficial and increased efficiency during the examination reviews. This prompted the staff-to include it as an optional activity in drait NUREG-1021. The staff still believes that

- the practice has merit but has emphasized in the final NUREG that it is

- optional, o Two commenters recommended that facility licensees be permitted to use site-specific task lists in place of the NRC's K/A catalogs when i developing the examination outline. The site-specific lists are used to develop the operator requalification examinations and should also be acceptable for developing content-valid initial licensing examinations.

18

_ .-% g e- , . , m m , + + w w s, -c,-~,~ N.+ N . v-

- Since the mid-1980s, the staff has used the NRC's generic K/A catalogs as the basis for ensuring the content validity of the initial licensing examinations. The catalogs are based upon an analysis of the licensed operator's job that was performed by the Institute of Nuclear Power Operations. Many facility licensees have conducted their own job task 1 analyses and osveloped plant-specific operator task lists for use in their SAT-based training programs. Although the staff does not question that the site-specific task lists would generally be acceptable for ensuring the content validity of the examinations, _it believes that wholesale substitution of the catalogs could cause a decline in examination consistency. The staff considers the proposal appropriate in limited circumstances on a question-by-question basis; therefore, the guidance in NUREG-1021 has been revised to permit substitutions and additions of sp..elfic knowledge and ability requirements on a case-by-case basis, o .Two commenters disagreed with a paragraph in Appendix A of the draft NUREG that implied that.any question taken from an open bank would test at the basic memory level. They argued that questions prepared at a 4 higher cognitive level would not be reduced to the basic memory level if the question bank is large. They strongly recommended that the subject paragraph in Appendix A be deleted.

Appendix A provides an overview of examination preparation concepts and discusses the basis for some of the NRC's examination criteria, including the limits on the use of examination banks. The subject paragraph is considered integral to the discussion and was not deleted.

However, the staff has edited the text to clarify that questions could decrease to the simple recognition level if the item bank is small and available for the examinees to study.

The staff also clarified one of the criteria in draft NUREG-1021 that, for purposes of examination quality and consistency, required at least half of the' questions on the written examination to be written at the

, comprehension or analysis level. The staff never intraded to imply that any question drawn directly from a bank would be counted at anything other than its face value. A question written at the comprehension or analysis level would count as such, even though the cognitive level at which it tests would decline with use and exposure, o- Two ummente-s recomended that the facility licensees be allowed more than five days, if necessary, to review and grade the written examinations. They asserted that relaxing the requirement would only delay the results to the facility licensee and the applicant and that the delay would not adversely affect the NRC.

The staff had traditionally allowed the facility licensee five days to comment on the written examination, so it decided to retain that time period for the pilot examination program. This was one of the examination criteria on which the chief examiners had been asked to comment, and no comments were made. However, since the proposed change 19

+

would not affect the examination outcome, the staff decided to increase the flexibility in the final NUREG-1021 by noting that the facility licensee should submit the graded examinations within five days, if practical, o One commenter indicated that NUREG-1021 does not appear to accomodate utilities that do not want to prepare their own licensing examinations and suggested that utilities be allowed to retain that option.

Because of the unpredictable demand for examinations and the scheduling

' difficulties that may result from a long-term voluntary program, SECY-96-123 requested that the Commission approve the staff's pursuit of rulemaking to require that licensees of power reactor facilities prepare the operator licensi..g examinations in accordance with NUREG-1021. Tt.a Commission has delayed making a decision on industry-wide implemcntation of the revised process until it can review the detailed rulemaking plan (Attachment 1) and the additional issues discussed herein, o One comenter recomended that the facility licensees be held responsible for making examination changes "as agreed upon with the NRC" rather than "as directed by the NRC." The commenter added that this alteration would help ensure that the suggested changes are accurate.

Because the staff fully supports the goal of the facility licensee to ensure the accuracy of the examinations, it revised NUREG-1021 as recomended but stipulated that the NRC would retain the final authority to approve the examination, o One commenter requested that the NRC clarify which facility staff members would be allowed to observe the operating tests in the simulator. NUREG-1021 currently restricts personnel other than the simulator operator from watching the operating test without the approval of the NRC chief examiner. The commenter argued that facility managers and others deemed necessary by the facility should be allowed access to the examination, provided the simulation facility can accommodate them and there is no adverse effect on the applicants.

The staff concluded that this was a reasonable request and revised NUREG-1021 accordingly. Facility management and other personnel deemed necessary by the facility licensee will generally be allowed access to the examination (Jnder security agreements, as appropriate), provided the simulation facility can accommodate them and there is no adverse effect on the applicants, o One comenter requested that the NRC clarify the meaning of "significantly modify." The facility licensee had been told that the intent of the question had to be changed, which conflicted with the definition that had been published in GL 95-06 and draft NUREG-1021.

20 1

4 I

The staff concluded that this was a reasonable request and revised- t

%" NUREG-1021 to-indicate that the intent of the question does not have to change to quali_fy as t significant modificaticn.

In summary, the staff believes that the-industry generally supports the-proposal-to shift responsibility for writing the initial licensing examinations to the facility licensees. Some facility licensees actively i 4

lobbied for the change on the basis that it would save them money. Many facility licensees and NEl view the proposed changes.as a-good first-step at increasing the industry's involvement in the examination process and shifting the NRC's role from direct involvement to oversight. Neverthel.ts., most facility licensees would probably agree that the administrative requirements that were outlined in GL 95-06, refined during the pilot program, and incorporated in Revision 8 of NUREG-1021 have increased the burden of preparing the examinations. ,

- The issue of industry burden has been a key factor throughout the pilot program. The staff purposely limited the scope of the original process changes to those measures that would help ensure the integrity, consistency, >

quality, and security of the licensing examinations. The ctaff was comitted to maintaining an effective and efficient licensing program and realized that the revised process, which would have relied on voluntary industry support.

would never succeed if it was unnecessarily burdensome on-the facility licensees. In keeping with its original commitment, the staff believes that it has taken reasonable measures to address the industry's concerns and recommendations without sacrificing the effectiveness of the examination

- process.

, Pilot Enhancements and Justifications To assess the effectiveness of the new examination process, NRR asked the NRC

= examiners who were involved with the pilot examinations to respond to a i questionnaire regarding various aspects of the process. The first and most '

significant problem observed and reported by the examiners was that the quality and technical accuracy of the examinations submitted by several facility licensees fell short of the NRC's expectations. Moreover, some of the errly pilot examination results (i.e., more question deletions and answer key changes than expected) reflected the need for greater emphasis in that

- area.

Consequently, when NRR prepared draft cvision 8 of NUREG-1021, it incorporated a number of clarifications and changes that improved the framework established in GL 95-06, in January 1996, the staff issued a preliminary draft of NUREG-1021 to the regions-for review and comment before soliciting comments from the public and the industry the following month. The draft revision that was issued for public and industry comment contained the following process improvements:

o -The existing guidance (Revision 7 of NUREG-1021, Revision 5 of NUREG/BR-

-0122, and GL 95-06) did not specify what percentage of the questions on

' the written examination should test the appilcants at the comprehension 21

u and analysis levels of knowledge. Many examiners had expressed concern that the lack of guidance in this area would lead to inconsistency and a decrease in the level of examination difficulty. Facility licensees questioned wh % r examiners had unwritten guidance that required them to make the ex .nnation more difficult by revising or replacing simple memory questions.

As noted earlier, NRR had no desire to increase the average difficulty of the examinations, but it also did not want the level of difficulty to diminish. Therefore, to better define the appropriate level of knowledge for the new examination process, the staff reviewed the written examination audits performed in the previous two years to determine an average distribution of questions by knowledge level. The staff concluded that approximately half of the questions on the examinations reviewed were written at the comprehension or analysis level and that the remaining questions tested simple memory. On the basis of that information, the staff established a criterion in draft NUREG-1021 that at least half of the examination questions be written at the comprehension or analysis level, o The supplemental guidance that NRR provided to the regions concurrent with CL 95-06 included interim QA checklists to assist the i4RC examiners in reviewing and identifying deficiencies in the draft examinations.

The checklists were edited and incorporated in the prelim. nary draft of NUREG-1021 and edited again in response to regional comments. The draft NUREG-1021 that was issued for public comment includeri QA checklists for the examination outline, the written examination, both the simulator and walk-through portions of the operating test, and the wri' ten examination grading, o The existing guidance (Revision 7 of NUREG-1021 and GL 95-06) did not specify a limit on the number of JPMs that could be repeated in walk-through tests administered on successive days during the same initial licensing examination visit or from one visit to the next. Many examiners had expressed concern that the lack of guidance in this area could compromise examination integrity and lead to inconsistency. Other examiners argued that no repetition should be permitted because it would increase the predictability of the test. Facility licensees would be tempted to repeat as many JPMs as possible in an effort to save resources.

To strike a balance, the staff concluded that a 30-percent limit would be appropriate. This figure was consistent with the written examination limit of 25-percent duplication of questions from examinations, quizzes, or tests administered to the license applicants or from the past two NRC licensing examinations at the facility.

After issuing draft HUREG-1021 for public and industry comment, the staff identified a number of items that merited clarification and improvement. In addition to reviewing the industry's comments and recommendations (as previously noted), the staff discussed all of the following changes with NE!

22

and other industry representatives during a oublic meeting held in the NRC's offices on April 18, 1996. Except as noted selow, the industry representatives did not object to the additional changes.

o lo limit the number of technical inaccuracies, a problem that affected several of the pilot written examinations, the staff revised NUREG-1021 to require the NRC chief examiner or another examiner to independently review and verify the technical accuracy of a sample of the written examination questions. The number of questions verified would depend upon the expected quality of the examination and the time available before the scheduled review with the facility licensee.

As a final check of the technical accuracy af the written examination, the staff revis,d NbREG-1021 to encourage the facility licensee to administer the NRC-approved examination (under security agreements) to one or more licensed personnel who were previously uninvolved in developing the examination. Although feedback from the NRC chief examiners indicated that some facility licensees had taken it upon themselves to validate their examir.ations in this manner during the pilot program, the facility representatives at the meetir.g objected to making it a requirement for all f acility licensees. Because the staff lacked sufficient informatM to quantify the value added by this measure, it chose to recommer.4 it as a good practice, o Late in the pilot program, an NRC chief examiner who was reviewing a draft examination detected that the facility licens:3 had used a written examination outline that was identical to one that a sister facility had prepared several weeks earlier. The staff had not anticipated that such a problem might occur but immediately realized that this was a significant loophole in the process. Allowing facility licensees to reuse the same examination outline would definitely improve efficiency, but it would also jeopardize the integrity of the examination process by increasing the predictability of the examinations.

To address this problem, the staff included in NUREG-1021 a caution for examination authors and reviewers to ensure that the outline ust.d during huccessive audit and licensing examinations does not become repetitive and predictable, if a facility licensee proposes to use an outline that was previously used at another facility, it shall identify the source of the outline and explain what effect its reuse is expected to have on examiaation integrity.

o NUREG-1021 has for some time included checklists for use by NRC examiners in verifying that their simulator scenarios contained the required number of transients and ensuring that the scenarios would enable the examiners to evaluate the applicants on every required competency. There were no specific instructions for filling out the checklists, nor were they required to be retained in the examination file. Concerns regarding the quality and consistency of the operating tests prompted the staff to incorporate instructions for completing the 23

- . - . . .- -- . - ~ . - .- - _ - . - - - - - -

'4 transient and competency checklists and a requirement for the NRC-regional offices to retain the checklists in the examination files. ,

o NUREC-1021 has always endorsed the goal of having the licensing examinations ready to administer approximately a week before they are

' scheduled to be given. - furthermore, the staff has always encouraged the regions to keep NRR informed if significant problems are encountered-- .

with an examination. The pilot program illustrated and magnified the importance of both cancepts. Therefore, the staff clarified the

• guidance in NUREG-1021 to limit the number of last-minute changes and to require the regional office to consult NRR if it appears that the examination cannot be made to conform to the examination standare at least five working days before the scheduled examination date.

Furthermore, th+ exa 4tnations should be postponed if problems are identified at the last minute, o in addition to in.plementing the pilot examination process,- the staff proposes using Revision 8 of-NUREG-1021 as a vehicle for activating Revision 1 of NUREG-ll22 and NUREG-1123, the NRC's K/A catalogs, which were published in August 1995. The catalogs were revised to make them more current and reorganized to make them more consistent with each other and easier to use. As part of the catalog revisions, the system-  ;

generic K/As in both catalogs were combined with the plant-wide generic K/As, and new vendor-specific emergency and abnormal plant evolutions were added to the pressurized water reactor catalog (NUREG-ll22). These changes would have altered the balance of the written. examinations unless the sample plans in NUREG-1021 were also revised to compensate for the changes. Because of an oversight, the staff neglected to enter the changes t;efore issuing the draf t NUREG-1021 for public and industry ,

comment, o When NRR reviewed the regional office operator licensing programs during February and March 1996, it discovered that some of the examiners had misinterpreted the intent of NUREG-1021 regarding the design of the operating tests for SRO applicants limited to fuel handling. The staff discussed the issue with the regions and agreed that strict compliance with the existing guidance would overtest those license applicants.

Therefore, the guidance in NUREG-1021 was changed to more closely align the scope of the operating test with the range of responsibilities for SRO-fuel handlers.

-o Whenever a question is deleted from the 100-point written examination,

'it results in fractional grades. When the grade is above-79.5 percent, it raises a question regarding the staff's policy on rounding-off because it means the difference between receiving a license or a denial.

To address this problem, the staff has revised NUREG-1021 to indicate that the passing grade on the written examination is 80.00 percent.

o Although the pass rates of the pilot examinations were comparable with pass rates for NRC-written examinations, the rate at which applicants requested that the NRC informally review their examination failures

-24

increased _ unexpectedly during the pilot program. These requests not only placed a significant burden on the staff to resolve the applicants' issues but also raised concerns regarding the etfect that any additional cuestion deletions and answer changes might have on the other licensing cecisions that had already been made.

Consequently, the guidance in NUREG-1021 was revised to afford the NRR staff greater discretion in determining the niethod for resolving applicant appeals (i.e., every appeal does not necessarily have to be reviewed by a panel of examiners as long as objectivity and fairness are maintained) and to delay the issuance of licenses to applicants who passed the written examination with insufficient margin to guarantee that the licensing decision will be sustained if additional questions are deleted or changed upon appeal. Furthermore, to minimize the number of appeals, the staff revised NUREG-1021 to encourage facility licensees to solicit and address concerns from individual license applicants during the process of grading the written examinations.

o Problems encountered while reviewing and resolving the operating test appeals that were received during the pilot program prompted the staff to reevaluate the documentation and grading requirements in NUREG-1021.

As a result, the staff has revised NUREG-1021 to permit NRC examiners to downgrade more than two simulator test rating factcrs for the same error, provided the error caused serious safety consequences for the plant. The staff also increased the documentation requirements for the simulator test so that the NRC examiners would have to describe any error that prompted them to downgrade the applicant on any rating factor.

During the meeting of April 18, 1996, the staff also discussed the possibility of permitting NRC examiners to recommend a failure on the simulator test even if the applicant did not fail the test on the basis of the competency scores. The industry representatives did not believe that such a change would be necessary if there were no limit on the number of competencies that the examiner could downgrade for a single, serious error; however, they did not object to the change in principle.

Therefore, the staff has included in NUREG-1021 a provision for NRC examiners to recommend such a failure if the applicant committed an error with serious safety consequences for the plant or the public. To limit its use, the region must obtain written concurrence from the Chief, HOLB, before completing .a licensing action.

The (cllowing issues were raised by NRC examiners on the pilot examination feedback surveys but did not result in specific chang to the examination methodology described in NUREG-1021.

o To minimize NRC resource requirements, the staff had encouraged the regions to review and prepare for the pilot examinations without conducting a preparatory visit to the site. However, feedback from those examiners who did not conduct such a visit indicated that it would have facilitated the review and resulted in fewer problems with the 25

examination. They recommended that a preparatory visit be required for every examination. Although NRR has not changed the guidance in NUREG-1021 (i.e., the preparatory visit is optional at the discretion of regional management), it has acknowledged the value of conducting at least part of the examination review and validation at the facility.

The average NRC resource expenditures during the pilot examinations indicated that the trips can be made within the original budget model, o Several NRC examiners recommended that the required due dates for the axamination materials from the facility licensees be advanced so that the examiners have more flexibility in scheduling their time for the examination review. The staff did not change the guidance in NUT.EG-1021 because it already permits the regional offices te work with the facility licensees in determining the appropriate due dates and because a generic change would increase the scheduling burden on all facility licensees, it would not be appropriate for an examiner to demand the examination 60 days in advance because it was required by NUREG-1021, only to have the examination sit untouched in the office for 30 days because the examiner was involved wi_th another project, o When the staff developed the pilot examination process, there was some debate as to whether the facility personnel who administered the written examination should be permitted to answer the applicants' questions regarding the intent of specific qustions on the examination. The pilot guidance allowed the facility to answer the applicants' questions with caution but required every question and response to be documented for subsequent review by the NRC. The policy was evaluated on the pilot examination questionnaire that every NRC chief examiner was required to complete. A minority of the examiners recommended that the policy be changed to prohibit the applicants from asking questions during the written examination, as is the policy on the GFE. However, most of the examiners concurred with the original pilot policy, and none of the comments indicated that the policy had created any problems with examination integrity. Therefore, the staff saw no reason to revise the guidance in NUREG-1021.

o Some examiners noted that the pilot program personnel restrictions had hampered the facility licensees' abil ty to develop the examinations because they could not assign their best people to the task. This issue was also addressed in the industry comments, as previously discussed.

Examination Results Since Issuance of SECY-96-123 The staff completed the last of the pilot examinations on April 5, 1996, and resumed conducting the licensing examinations using the traditional development process. After April 12, 1996, when the Executive Director for Operations informed the Commission of the staff's intent to continue the pilot process beyond the period previously discussed with the Commission, the staff proceeded to contact those facility licensees whose examination requests had not yet been assigned to a cnntractor to detennine if they were interested in developing the examinations using the pilot methodology. The first of those 26

• examinations was administered on August 19, 1996, and the susults were not i available in time to report them here.

Since SECY-96-123 was prepared, the staff has used the traditional examination ,

methodology to prepare licensing examinations for applicants at 14 power reactor facilities. The results of the completed examinations are summarized i' below; four of the ensinations were completed late in August, and the results were not available at the time this paper was prepared. The licensing examination results for fiscal year 1995 and the 22 pilot examinations conducted between October 1, 1995 and April 5. 19t'5, are provided for comparison. ,

Pass Rates' R0 R0 R0 SRO. SRO SRO Examinations Written Operating Total Written Operating Total 39/40 39/39 39/40 45/48' 43/48 41/48' Since SECV-96-123 (98%) (100%) (98%) (94%) (90%) (85%)

Pilot _

49/54 50/54 45/54 86/92 87/918 83/92*

Examinations (91%) (93%) (83%) (93%) (96%) (90%)

Fiscal Year 1995 94% 98% 92% 95% 95% 92%

1. Each box indicates the number of applicants who passed that part of the examir.ation over the number of applicants who took it.

2. One of the SRO applie nts who fedled the written examination has filed an aop .1 that was not yet resolved at the time this paper was pre,."id.

3. Tw. 9plicants who were reported as preliminary failures in SECY-96-123 filed' appeals that resulted in the failures being overturned. ,

ilheitX The staff believes that the revised examination process can be implemented at all power reactor facilities with the same level of direct NRC resources as is >

currently allotted to the operator licensing progrcm., Full implementation of the revised process will enable the staff to eliminate the use of operator licensing .ontractors (with the exception of those'whc develop and grade the

• GFE) that have hictorically cost the NRC (and the indu stry) approximately

$3 million to $4 million per year. The staff expects that the change could result in a net resource savings to the facility licen sees over time because they will be abis to prepare the examinations more efficiently than the NRC or its contractors.

The overall results.of the pflot program indicated that the examinations prepared by facility licensees, subject to review, rev ision where ap)ropriate, <

and approval by NRC staff examiners, were generally of the same caliser as examinations prrpared by NRC or contract examiners. l ie pilot examinations also appeared to be equally effective at identifying applicants who had not-4 k kh g 1- Thd4 6 k$N i-4 h %& dv.3 yOVand FMS b Wro se. % g.u\ch 9M v.w m nets av d\ d .t.

l l-

., , - , - - - - , , ,,,,,,.-- , e ,. - , , , - . - ,.,,-.n, , -, - - - . . - - . - - - - - . = , - - ,

1 l

mastereothejobrequirementswellenoughtobecomelicensedR0sorSR0s.

furthermore, the pass rates for R0s and SR0s on the written examinations and operating tests compared favorably with the pass rates for examinations prepared by the NRC or its contractors. j However, despite the mitigating actions previously noted, the revised .

examination process is much more dependent upon the abilities of the NRC chief examiners to identify weak examinations and work with the facility licensee to I upgrade the examination. Furthermore, it is possible that the pressures to  !

conduct examinations on schedule and the reduced involvement by the NRC in the !

examination development process could lead to examinations that reflect the ,

biases of the facility licensees and that discriminate at a lower level than is currently the norm. Moreover, it is reasonable to expect that industry-wide implementation of tht revised process will increase the chance of future examination compromises as a result of the larger number of individuals who would require access to the examinations and the additional pressures c.':

facility employees to ensure the success of the applicants.

Although a number of the examinations submitted by the facility licensees did not meet NRC standards for level of knowledge or difficulty, the NRC chief examiners effectively worked with the facility licensees to revise the examinations so that they would discriminate at the appropriate level. To put this situation into perspective, it has not been too unusuLl for contractor-developed examinations to fall short of the NRC's requirements and expectations. The chief examiner often directs the contract examiner to make numerous corrections and changes before reviewing the examination with the facility licensee, only to have the facility reviewer identify a host of additional errors and inconsistencies.

The staff has advised facility licensees that the NRC expects the technical quality of examinations to remain high and the number of post-examination changes to be low. Similarly, regional examiners and managers have been instructed not to approve or administer any examination that does not adhere to NRC standards for content, format, quality, level of knowledge, and level of dif ficulty. The QA checklists in NUREG-1021 have been modified to maintain examination consistency and to facilitate the efforts of NRC examiners to detect deficient examination materials. The use of any facility-prepared examination that is found to be unacceptable will be delayed or cancelled, if necessary, until the facility Itcensee and the NRC can upgrade or replace tne examination.

The supplemental pilot examination guidance that the staff provided to the regional offices (and placed in the PDR) in conjunction with GL 95-06 directed the regions to document in the examination report any significant problems c p C _ encopntered during the examination development, administration, and grading.

This requirement- has also been included in Revision 8 of NUREG-1021. The regions are expected to address any serious deficiencies in the examinations submitted by the facility licensees and any security concerns during the entire examination process, if appropriate, the region will request the facility licensee to describe the actions it will take to improve future performance, if the facility licensee is unable or unwilling to prepare 28

appropriate licensing examinations, the NRC has the responsibility and authority to develop the examinations in the traditional manner.

Overall, the staff does not expect that the pro)osed revisions to the o>erator licensing program will significantly increase tie risk to the life and iealth >

of any individual members of the public or that they wt11 be a significant 4 addition to other societal risks. The staff believes that the licensing decisions that were made on the basis of the pilot examinations were as valid as those made using the traditional examination process, furthermore, the changes that the staff has made in NUREG-102) as a result of lessons learned duringthepilotprogramandindustrycommentssubmittedinresponsetothe staff s solicitation (which were previo:tsly discussed in detail) should only ,

improve the effectiveness of the examinations if and when the revised process is implemented on an industry-wide basis.

The NRC will continue to set the standard for operator performance by ensuring that the facility licensees maintain approprtately high standards for examination development. The staff will remain actively involved in the operator licensing process by reviewing and approving every written examination and operating test before it is administered. Unacceptable examinations will be revised and delayed, if necessary, until the resources are available to upgrade the examinations. Furthermore, the fact that NRC examinerswillcontinuetoindependentlyadministertheoperatingtes(swill allay concerns regarding bias and conflicts of interest on thet M portion of K the licensing examination having the most operational validitf-and-requiring the most subjective judgements. ,

The safety consequences of oc:asionally administering a substandard examination are probably minimal. By the time the applicant takes the NRC  ;

ticensing examination, he or she will have completed a comprehensive.

systematically constructed and implemented training program designed to  ;

maximize the likelihood of success on the examination. The itcense training

, ~

programs typically include numerous written examinations and performance '

demonstrations that culminate in certification by the facility licensee that the applicant has successfully completed the facility licensee's requirements to be licensed as an R0 or an SRO. The effectiveness with which the factitty licensees screen out those trainees that have not mastered the job requirements has been evident-in the high pass rates on the licensing examinations during the past several years. In most cases, the NRC ,

examination is simply a conitrmation of the facility licensee's decision to submit the. application.

Historically, even those applicants who fati the NRC written examination generally score above 75 percent. The average grade for the 11 applicants who failed a pilot examination was 76.3 percent. This statistic indicates that the vast majority of applicants, including most of those who fail the licensing examination, have attained an adequate level of knowledge so as not to pose a significant threat to reactor safety. Furthermore, any applicant who does become licensed on the basis of an examination that discriminated at a lower than normal level would be backed up by other crew members who could prevent or remedy any operating errors. Although there is nothing to prevent a facility licensee from forming an operating crew entirely from newly.

29

. . . _ _ _ _ . . _ _ . . . . . , _.-__ _ . _ . _ . _ _ . _ . . .. . _ . ~ . _ . _ . . _ . . . - _ , _ . . _ _ . . . . . -

e licensed operators, most facility licensees integrate their new operators with existing crews who possess significant operating experience, r

i 4

30 s

.. . __ _ . , .