Text

Informs That Review of 10CFR55, Operators Licenses, Commission Paper & Rulemaking Plan to Change Licensed Operator Exam Requirements Has Been Completed.Nrc Recommends Addition of Stated Paragraph
	ML20216F949
Person / Time
Issue date:	09/03/1996
From:	Treby S; NRC OFFICE OF THE GENERAL COUNSEL (OGC)
To:	Morrison D; NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES)
Shared Package
ML20216F483	List: ML20216F480; ML20216F496; ML20216F508; ML20216F518; ML20216F534; ML20216F542; ML20216F561; ML20216F582; ML20216F587; ML20216F606; ML20216F627; ML20216F726; ML20216F751; ML20216F759; ML20216F776; ML20216F809; ML20216F867; ML20216F949;
References
	FRN-62FR42426, RULE-PR-55 AF62-1-028, AF62-1-28, NUDOCS 9709120142
	Download: ML20216F949 (1)
	v • d • e

p me:u,_ UNITED STATES Ag4g g

.rf

~

'), NUCLE AR REEULATORY CCMMISSION

, ,g WASHINGTON. D C #06t 5-0001 y September 3, 1996 g

%, . . . . . /

Of IfC[ Of 14 4[

Gt NlitAL COUNLIL MEMORANDUM TO: David Morrison, Director f ,

Office of Nuclear Rea of Resea )

FROM: Stuart A. Treby jg//i>/ #f t <

Anniotant General Counsel for Rulemaking and Fuel Cycle SUBJECTt NO LEGAL ODJECTION TO RULEMAKING PLAN FOR AMENTHENT OF 10 CFR PART 55 TO CHANGE LICENSED OPERATOR EXAMINATION REQUIREMENTS We have completed our review of the 10 CFR Part 55, " Operators' Licennen," Commission paper and rulemaking plan to change licensed operator examination requirements. We recommend the addition of the following paragraph in the section of the rule plan labeled "OGC Analysis."

The proposed rule would change the current requirement that the NRC prepare and proctor the initial examination for Reactor Operators and Senior Reactor Operators and, instead, require the licensee to prepare and proctor the written portion of the initial examination. The Office of the General Counsel (OGC) worked with the utaff to develop the backfit analysia net forth below and for the reasono provided in that analysia believea. thin action does not constitute a backfit pursuant to 10 CFR 50.109. In addition, thin action does not require an environmental assesament because it is categorically excluded from that requi rement })ursuant to 10 CFR 51.22 (c) (1) . OGC has not identified any Paperwork Reduction Act issues nor doen it believes thin action constitutes a " major rule" pursuant to the Small Busineca Regulatory Fairneso Enforcement Act. of 1996. In all other respecto, OGC has not identified any potential legal complications or known bases for a legal objection to the rulemaking.

Subject to the inclusion of thin paragraph and the revisiono noted at red tabs on the draft document (attached), OGC does not have any legal objection to this rulemaking plan. OGC believes that all rulemaking plano should be forwarded to the Commission ao negative conaent papers rather than as informational papers, cc: Stuart A. Richards, HOLB, NRR CONTACT: Maria Schwartz, OGC 41b1888 9709120142 97C904 PDR PR SS 62FR42426 PDR

~

-- _ __ -_ 4

T i.f

} q g\ umTEo STATES t < .f . . NUCLEAR REOULATORY COMMISSION l 5 wAsNINoToN o.C. 80eeHm01 y

- "' August 26, 1996 MEMORANDUM T0: James Lieberman, Director Office of Enforcement William J. Olmstead, Associate General Counsel d

- for Licensing and Regulation Office of the General Counsel William.T. Russell, Director Office of Nuclear Reactor Regulation FROM: David L. Morrison, Director . .

Office of Nuclear Regulatory Research y ,

y

SUBJECT:

RULEMAKING PLAN - CHANGES 10 10 CFR 55 - ANENDING INITIAL

, OPERATOR LICENSE EXAMINAil0N REQUIREMENT!

i Your concurrence is requested on the attached rulemaking plan packa e. This

! action is on the Chairman's tracking list with an aggressive schedu e,

therefore, please respond promptly.

Ettkaround:

4 Section 107 of the Atomic Energy Act of 1954, as amended, requires the NRC to determine the qualifications of individuals applying for an operator licenso, to prescribe uniform conditions for licensing such individuals, and to issue

licenses as appropriate. Current regulations in Part 55 dealing with initial I license examinations for reactor operators (RO) and senior reactor operators (SRO) do not state who will write, administer, or grade the examination;.

. however, the NRC has traditionally performed those tasks itself or through

' contract examiners. On March 24, 1995, the staff, in SECY-95-075, ' Proposed Changes to the NRC Operator Licensing Program," informed the Consiission of its intent-to revise the operator licensing program to allow greater >articipation by factitty licensees and to eliminate contractor assistance in tits area.

This action will allow staff to elimini..e between $3M and $4M annually on

! contractor support for examination 1 reparation and administration. On April

} 18, 1995, the Commission approved tie staff's proposal to initiate a transi-tion process to revise the operator licensing program, and directed the staff to carefully consider experience from pilot examinations before full implemen-tation. On August 15, 1995, the staff issued Generic Letter (GL) 95-06,

' Changes in the Operator Licensing Program,' outlining the revised examination development process and soliciting volunteers to participate in pilot examina-

tions to evaluate and refine the methodology.

l 1

I i

, . . . , _ . .. . ~ . . -

L 41 J. Lieberman et at. 1 Between October 1, 1995 and April 5, 1996, the NRC conducted a pilot program I in which it reviewed and approved 22 operator licensing examinations that had been written by facility licensees in accordance with the curren+. examination

{ development guidance (NUREG-1021. Rev. 7. Sup)1ement 1), as supplemented by the pilot examination guidance contained in G. 95-06. The staff described the results of the pilot examinations in SECY-96-123, " Proposed Changes to the NRC 1 Operator Licensing Program," dated June 10, 1996, and briefed the Comission j during a public meeting on June 18, 1996. Based on the results of the pilot program, the staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to draft the operator licensing examinations:in accordance with NUREG-1021, ' Operator Licensing Examination Standards for Power Reactors.' in a staff requirements memorandum (SRM) dated July 23, 1996, the Commission directed the staff to develop a detailed rulemaking plan to justify the changes that may be neces-sary to 10 CFR Part 55. The SRM also directed the staff to specifically address several issues about the pilot examination program and the proposed changes in the examination process.

The rulemaking plan and the response to the SRM's specific requests, which were prepared by the Office of Nuclear Reactor Regulation and reviewed and approved by the Office of Nuclear Regulatory Research, are attached to the d ,pt enclosed Commission Paper g One issue I would appreciate specific comment on is that of whether the ru e should contain an explicit reference to NUREG-1021 or alternately require an operator licensing program acceptable to the NRC. If the former course is taken that will have the effect of codifying the NUREG and if we wish tc change it later based on experience with implementing the new rule a rulemaking would be required, if the later course is taken a regulatory guide would be issued referencing the NUREG and it would only be necessary to change the regulatory guide.

The following is a summary of this request:

1. lilh: Rulemaking Plan, " Changes to the Operator Licensing Process at Power Reactors (Part 55)."

2. Task leader: Harry S. Tovmassian (415-6231)

3. Steerino GrouD Members: None

4. Enhanced Public Participatign:

This rulemaking will use electronic bulletin boards, as appropriate to enhance input from the public.

5. ' Level 1 or 2 Item of Compatibility for Aareement States: No

6. Be_qunted Action: Office Concurrence r . ___

J

1

. s s

J. Lieberman et at. '

Nfl 1. Reouested Completion Date: August 30, 1996

8. Resources and Coordination:

Estimated resources to develop this rulemaking are .5 Fi[. No contractor support is needed. A copy of this concurrence package i has been forwarded to the Office of the Controller for coordination of resource issues and to the IG for information.

Attachmentt Memorandum to Taylor from Morrison w/encis, i cc tt/ attachment:

l R. Scroggins, OC l H. T. Bell, OlG

.i G. Cranford, IRM D. Meyer, ADM

+

a- niw 4. ..wm__ _ - - ..

ppa ,, , _ __

k *

/*

[

a UNITED STATES j p, NUCLEAR REGULATORY COMMISSION

,[$

{ ,

WADHINGToN, o.C. MM

'( .

H k MEMORANDUM 10: James M. Taylor Executive Director for Operations fROM: David L. Morrison, Director Office of Nuclear Regulatory Research i

4

SUBJECT:

RULEMAK"lG PLAN - CHANGES TO 10 CFR PART 55 - AMENDING INITIAL OPERATOR LICENSE EXANINATION REQUIREMENTS Ctached for your signature is a Comission paper which requests the s.ommission's approval of the staff's rulemaking plan for requiring power reactor licensees to prepare and administer initial reactor operator (RO) and senior reactor operator LSRO) licenstr. Section 107 of the Atomic Energy Act of 1954, as amended,g examinations.

requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Current regulations in Part 55 dealing with initial license examinations for R0s and SR0s do not state who will prepare, administer, or grade the examination; however, the NRC has traditionally performed those tasks itself or through contract examiners. On March 24. 1995, SECY-95-075,

" Proposed Changes to tne NRC Operator Licensing Program " informed the Comission of the staff's intent to revise the operator licensing program to allow greater participation by facility Itcensees and to eliminate contractor assistance in this area. This action will allow staff to eliminate between

$3M and $4M annually on contractor su) port for examination preparation and i

administration. On April 18, 1995, t1e Commission ap) roved the staff's proposal to initiate a transition process to revise tie operator licensing program, and directed the staff to carefully consider experience from pilot examinations before full implementation. On August 15, 1995, the staff issued Generic Letter (GL) 95-06, ' Changes in the Operator Licensing Program,"

outlining the revised examination development process and soliciting volunteers to participate in pilot examinations to evaluate and refine the methodology.

Between October 1,1995 and April 5,1996, the NRC conducted a pilot program in which it reviewed and approved 22 operator licensing examinations that had been prepared by facility licensees in accordance with the current examination development guidance (NUREG-1021, Rev. 7, Sup)lement 1), as supplemented by the pilot examination guidance contained in G 95-06. The staff described the results of the pilot examinations in SECY-96-123, " Pro)osed Changes to the NRC Operator Licensing Program," dated June 10.-1996, and ortefed th6 Commisstor, during a public meeting on June 18, 1996.- Based on the results of the pilot program, the staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to prepare the operator licensing examinations in accordance with FUREG-1021, " Operator r- , ,

I 4 -

lj J. M. Taylor Licensing Examination Standards for Power Reactors." In a Staff Requirements Mesorandum ($RM) dated July 23. 1996, the Commission directed the staff to develop a detailed rulemaking plan to justify the changes that may be _

pecessary to 10 CFR Part 55. The SRM also directed the staff to specifically address several issues about the pilot examination program and the proposed changes in the examination process, The attached Commission Paper contains both the staff's rulemaking plan and a response to the issues raised in the SRM.

Attachment:

j Commission Paper w/encIs.

I

+

$-9 *'$' .++o.4,Me.- _ _ _

f b

e jy J. M. Taylor - 2-Licensing Examination Standards for Power Reactors ' In a Staff Requirements Memorandum (SRM) dated July 23. 1996, the Commission directed the staff to develop a detailed rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55, the SRN also directed the staff to specifically address several issues about the pilot examination program and the proposed changes in the examination process. The attached Commission Paper contains

- both the staff's rulemaking plan and a response to the issues raised in the ,

SRM.

Attachment:

Commission Paper w/encls.

01110 Mill 0N:

Central f/c RDS r/f RAuluck DMondiola CGallagher LRiani DOCUMENT NAME 0:\T0VMA551\TRAN5 MEMO.EDO

/ ,

0FC RDB:0Rd RDB:0RA n & I ONW 5. D:RES [ >

NAME HTovmassiantayw SRahadur [A Ihrris DNorrko[

DATE / d/96

/ 4/ W 96 9 8 ( /96 [/)//96 0FFICE RECORD COPY ,

RES FILE CODE:

N i-Weepassue ** ww - 9.*w e s..v -.e um wpe e = m - ap = v 4

.,. 2 m

( .

q .

d

-l .

lh ,

f i

COMMISSION PAPER 4 e

,_, , ..,n.- a, wn ** _

->=%<ww ** *

M

~

7 . .

{

mYa~

~ LQRt The Commissioners LBM: James M. Taylor, Executive Director for Operations

SUBJECT:

RULEMAKING PLAN FOR AMENONENTS TO 10 CFR PART 55 TO CHANGE j LICENSED OPERATOR EXAMINAil0N REQUIREMENTS 7,

PURPOSE: g p,4 p,,ay ,,g, y we , ,4,w.a Q To provide additional infornation regarding the initta' operator licensing examination pilot procesi as directed in a Staff Requii ementsMemorandum(SRN) of July 23, 1996 (M960(ll A), and to request the Commiss ion's approval of the rulemaking plan to amend 10 CFR Pargg.gguire powt r reactor facility licensees to prepare and adokitter .. .... .. :: e!!h NUREG-1021, ' Operator Licensing Examination Standards for Power Reactors,' the written examinations and operating tests that the NRC uses to determine the competence of operator Itcense applicants.

BACKGROUND:

On March 24, 1995, SECY 95-075, " Proposed Changer *.'. U.e NRC Operator Licensing Program,' informed the Commission of ti. staff's intent to revise the operator licensing program to allow greater participation by facility y* .~ Id(pnsees and to eliminate contractor assistance in this area. This action w< 11 allowistaff to eliminate between $3M and $4M on contractor support for examination preparation and administration. On April 18, 1995, the Commission approved the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full implementation of the changes.

On August 15, 1995, the staff issued Generic letter (GL) 95-06, ' Changes in-I the Operator Licensing Program,' outlining the revised examination development s

process and soliciting volunteers to participate in pilot examinations to evaluate and refine the methodology, CONTACT:

Stuart Richards, NRR (301) 415-1031 MM = %'**w4-W- _--me.qm_v- _

_ _ _ _ ~ __

it U The Comissioners -

2-s%d Betnen October 1,1995, and April 5,1996, the #RC- reviewed and approved 22 operator licensing examinations that had been prepared by facility licensees in accordance with the current examination development guidance (NUREG-1021.

Rev. 7 Supplement 1) as supplemented by the pilot examination guidance contained in GL 95-06. These examinativies were then used to test 146 reactor

operator and senior reactor operator applicants, facility licensees from each NRC Region, operating reactors built by all power reactor vendors, participated in the voluntary pilot program.

The staff described the results of the pilot examinations in SECY-96-123,

Proposed Changes to the NRC Operator Licensing Program,' dated June 10, 1996, and briefed the Comission during a public meeting on June 18, 1996. On the basis of the results of the pilot program, the staff recomended that the Comission approve the implementation of the new examination process on a 4 voluntary basis until rulemaking co d be completed to require all power

'pW ..rnctor factitty licensees to prepare the operator licensing examinations brl>*d CO wwA accordance witP NUREG-1021, " Operator Licensing Examination Standards for m Power Reactors," which has been revised to accommodate the new process, in an SRM dated July 23, 1996, the Comission authorized the staff to continue the pilot process through July 1997 and directed the staff to develop a detailed rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55. The SRM also directed the staff to spei.ifically address several issues regarding the pilot examination program and the proposed changes in the examination process. % g ,a g g w Q g n

EE. q%

y Ma t uonn u proWe # A.g W ce ruv 5*d a.w mJN b Section 107 of the Atomic Energy Act of 195 i'as amended, requires the NftC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license ipplicants are required by 10 CFR Part 55 to pass an examination satis ying the basic content r

requirements specified in the regulation. Part 55 does not state who must prepare, administer, or grade the examinati )n; however, the NRC has traditionally performed those tasks itself )r through contract examiners.

On the basis of the results of the pilot pr ) gram as discussed in SECY-96-123, the staff now believes that 10 CFR Pa. 55 Lhould be revised to require power reactor facility licensees to )repare their initial operator licensing

. examinations. Specifically, tie staff recomends that Subpart E, ' Written

. Examinations and Operating Tests," of 10 CF t Part 55 be amended to require all OS 7' powgr__ reactor facility licensee > to prepare the initial operator licensing

~

%'""f~ examinations 4*-eeeerdeW-WTIB NUREG-1021.C The licensee-prepared

-examtWalTons w!11 be subject to review, modification, and approval by NRC examiners before the examinations are administered. The regulation will o provide that the NRC may prepare the examinations in lieu of actepting or p\ c modifying an examination prepared by the facility licensee. The changes would V not apply to non-power reactor licensees.

The'e ETUsed-Niaadino plan for this proposed artsi~ife%1oped in accordance with the guidanc h N N n_ n Directive 6.3, 'The Rulemaking ProcessJ As.-directeo in the SRM of July 23, , making plan 4

% re- w - we, . m, a

____7 we wm

{ ,

}

I h the Commissioners i The enclosed rulemaking plan for this proposed change has been developed in accordance with the guidance in NRC Management Directive 6.3, 'The Rulemaking .

Process." As directed in the SRM of July 23, 1996, the rulemaking plan !

" addresses the potential impact of the proposed rule on facility licensees and !

discusses why this rule will not constitute a backfit. l

' The second enclosure discusses the following additional items as directed by the SRH: '

1 e lhe pros, cons, t.nd vulnerabilities associated with implementing the i proposed examination program changes on an industry-wide basis i (including a review of the impact on licensee and NRC resources, the potential effects on reactor safety, the impact of proposed NUREG-1021 ;

revisions, potential public serceptions, and concerns regarding this :

approach raised by staff mem>ers).

e A discussion of the degree of acceptance of this new approach to operator licensing by facility licensees (including a discussion of .

industry comments received on draft NUREG-1021 and an explanation of the staff's response to those comments). ;

e An explanation of enhancements to the new licensing examination process

resulting from the review and feedback from the pilot program results (including a justification for subsequent proposed changes to the operator licensing program),

e The results of initial operator license examinations that have been given both under the pilot program and under the current prog am since

$ECY-96-123 was prepared (including trends observed).

((KO_gMNOATION:

That the Commission --

1. 6ftrove the development of the rule as described in the enclosed Rulemaking plan.

2. Approve the implementation of Revision 8 of NUREC-1021 on a voluntary basis until the rulemaking is complete.

-~

4 _,,,_,,,p_

, .m.

, i s .

i .

The Cosnissioners ' If'].

C00MINATION: t Sonneg,

.r W 1he Office of the General Creasel has no legal objection to this initiative, and the Offices of Nuclear heactor Regulation and Nuclear Regulatory Research l I

l concur in this rulemaking plan.

s! 4 i

'e i

James M. Taylor Executive Olrector '

1 for Operations l Enclosures

~ 1. Rulemaking Plan for Changes to 10 CFR Part 55

4. i

2. Response to SRM did July 23, 1996 (M960618A) 1 g 8 i

i i

1 i

'**%W'** '

-n+s,e_ _

.w- c-- - --mv,,-----,v,.-- r ----,,-.-v,y ,% -- , - +, - - , .--. , , m, ,,c a ,,,,,,,-n, ,,e.. oc,,.yr,,,.yw....-- ..v--,,,,,.n-e--*.m----.,,wy,,+-+--

The Commissioner * - 4-COORDINATION:

1he and the OfficeOffices of the GeneralReactor of Nuclear Counsel has no legal Regulation and Nuclear objection Reguto htory this initiative,h Researc concur in this rulemaking plan.

James H. Taylor Executive Director for Operations j

Enclosures:

1. Rulemaking Plan for Changes to 10 CFR Part 55

2. Resp"onse to SRM dtd July 23, 1996 (r)60618A) 4 f

(

Distribution:

RDB r/f Central file EDO r/f

, LRiani CGillaglier DOCUMENT NAME: 0:\T0VMASSl\RULEPLAN.fNL ,

t. .%.eime . m w. e.em J>.n< n _ t.,.,c,, ,n, % . ....%,,

omca mot osa fjf f l onA not C

, 974 ,_v I o att f // R l 800 I wains .41. :- - -

.v. mSn L-Ar Man ~ . Des @= a I h7I JT.

v Da tt gggoe Kglte ;'6 tee {W I !**

omcat ne como copy

~ ~ - - _.

e -> - , - -,v-- w m, . - . , , . , - - , -- . . , -

.7

! l k-RULEMAKING PLAN FOR CHANGES TO 10 CFR PART 55 ENCLOSURE 1

, , _ -T*- = c

v -

l n.

tW(4 i J

RULEMAKING PLAN ;

CHANGES 10 10 CFR $$

/, MEN 0 LNG INITIAL OPERATOR LICENSE i i EXAMINATION REQUIREM[NTS

'l' l 1

Lead Office Office of Nuclear Regulatory Research .

Staff Contacts H. Tovmassian, RES/RDB

(

Concurrences * #v * ~

D. Rortisoh, RL5 a (W ]

'JaT4 f/

I f

t W. Russell, NRR Date P

J. Lieberman, DE Date i

W. Olmstead, DGC Date t

' Approvals t J. Taylor- Date ;

i 4

5 i

?7x'l

- N w, __ -

4 e .;

j -;

, , - , . . ~ _ , , , . . , _ . _ _ _ . , _ . . . . . , , , , . . , . , , _ . . . . , . - - - - . ,

_._.g, . . - - _ . . - _ , , , _

P l

RhipKING. PLAN

' CHAKES 10 THE OPEM10R LICENSING,.PRKill A1. POWER REAC10R$ f10 CfR PAR 1 $$)

I l.. Backaround On March 24, 1995, $[CY-95-0?$. " Proposed Changes to the N tc Operator Licensing Program,' informed the Commission of the staff's intent to revise the operator licensing program to allow greater par icipation by y

+ facilityItcenseesandtoeliminateNRCcontractorassista6ceinthis

? area. This action will allow staff to eliminate between $7 and $4M on contractor support for examination preparation and administration. On !

April 18, 1996, the Coastssion a> proved the staff's proposal to initiatt a transition process to revise tae operator licensing program and directed the staff to carefully consider experience from pilot ,

examinations before full implementation of the changes. On August 15. 1995, the staff issued Generic Letter (Gl) 95-06, ' Changes in the Operator Licensing Program,' which outlined the revised examination development process that would permit facility licensees to prepare the initial operator licensing examinations. GL 95-06 also solicited volunteers to participate in pliot examinations to evaluate and refine the methodology. g g A4n Between October 1,1995, and April 5,1996, the NR: cond ated 22 initial operator Itcensing examinations using the new methadology. Tne 614ff described the results of the pilot examinations in SECY-96-123,

' Proposed Changes to the NRC Operator Licensing Pr agram ' dated June 10, 1996, and briefed the Commission during a pub 1lcmeetingon June 18, 1996.. On the basis of the results of tne pilot program, the !

staff recomended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor fact 11ty 11 :ensees to prepare the i i

operator (d*j LicensingItcensingExamination examinations Standards for M-ecceAc. witMUREG-102L.ff" Power Reactors.' In a sta Operator

requirements memorandum dated July 23, 1996, the Commission directed the 1

staff to develop a detailed rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55.

11. Proonsed Changt Revise Subpart E " Written Examinations and Operating Tests,' to include examinationdevelopmentrequirementsforfacilityItcensees, honulatory . lssue The pilot program demonstrated that the revised examination development process can be both effective and efficient. Comments from the NRC staff and industry persennel who participated in the pilot examinations.

3:

were generally favorable. The quality of lint.inistered examinations (as modified by the NRC) and the performance of the individuals who took the examinations were comparable to the quality of and the performance -

1 l 1*tO h -,P 7 % dFf M -- - --

..,,m., pgy%..,_ . , , , -,__,..,,,.,,,r._yx__,, ,,ym ,,.,y,. , , . ,_ry.,, ,,.r ,_,,,,..y.,y, ,,_ mm,.,,mm,,w,.w

?/ on examinations written by the NRC staff or tts contractors. The fact i that some of the draft examinations submitted by facility licensees h' required significant rework illustrated that many facility staffs did !

not fully understand the criteria for writing an NRC examination. Both the industry and the NRC staff agree that, with training and experience, the industry should gain proficiency in preparing the examination materials.

The staff believes that the new process should be made mandatory for all power reactor facility licensees. The staff considers implementation of the new process on a voluntary basis alone unworkable over the long ters. With the elimination of contractor support, the staff will no i

longer have sufficient examiner resources to write all examinations consistent with the scheduling needs of facility licensees. This resource problem is further compounded by the unpredictable nature of the examination workload.

The staff briefed the Comittee To Review Generic Requirements (CRCR) after the pilot program was complete. In the minutes of its meeting, CRGR agreed that, with regard to the technical and safety aspects, the proposed changes represented a reasonable tid workable alternative approach. The comittee members unanimously endorsed sending the proposal forward to the Executiva Director for Operations (EDO) for consideration. However, the CRGR also requested OGC to consider further the procedural question relating the staff's original plan to implement the revised process by generic letter.

The staff subsequently met with OGC to resolve CRGR's concern and concluded that requiring all facility licensees to prepare the examinations, a task long performed by the NRC, would require rulemaking or the issuance of orders to each facility licensee. Therefore, the staff recommended in SECY-96-123 that the Comission approve the staff's pursuit of rulemaking to require power reactor facility licensees to prepare the operator licensing examinations 4&m,or4mse-w44- X-WRRr-lefi, while continuing the pilot process on a voluntary basis.

Cyr. rent Rule Reguirements Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe un crm conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CfR Part 55 to pass an examination satisfying the basic content requirements that are also specified in the regulation.

Specifically, 10 CfR 55.31(a)(3) requires the applicant to submit a written request from an authorized representative of the facility licensee that the written examination and the operating test be administered to the applicant, furthermore, 10 CFR 55.33(a)(2) states that the Comitston will approve an initial application for a license if it finds that the applicant has passed the requisite written examination 2

~ . _

l ;

test in accordance with Sections 55.41 and 55.45 or $5.43 l Q" ' endoperatin$heseexaminationsandtestsdeterminewhethertheapplicant and 55.45.

for an operator's license has learned to operste a factitty competently and safely, and additionally, in the case of a senior operator, whether the :pplicant has learned to direct the licensed activities of Itcensed operators competently and safely.

Subpart E, ' Written Examinations and Operating Tests,' includes specific

' sampling requirements for the content of written examinations

(10 CFR 55.41 and 55.43) and operating tests (10 CFR 55.45).

Although Part 55 is slient regarding who will prepare, administer, and grade the written examinations and operating tests, the NRC or its contract examiners have traditionally performed those tasks.

Pre 11m[ nary Reaulatory Analysis

1. Draft Rule Langgggt The first sentence in Section 55.33(a)(2) would be revised as follows (new language is underlined):

'The applicant has passed the requisite written examination and operating test in accordance with Sections 55.40;--55.41, and 55.45 or 55J0. 55.43, and 55.45.

Section 55.40, ' implementation

would be added to Subpart E as follows: , n3 p h @ qs pf

,.w. w .m u % n. 1

~

"(a) Power reactor facility Itcensees shall prepare ed & '-!:ter the required site-specific written examination and operating tests,(

in :::;rd: :: wHh-the-peHef ee-end-toideHees-in-WREG-let h Spreter licensing-f**mination-StandarMor Power ReactorAl (b)- - ik C:-- ',;;ica aiH-reviewr-modHy-as n;;;;terfr-6hd-*PPreve-in

rdr:: eith ".MC-1431r-the-we44 ten-examinat4 ens-and-operating teet: ; n; r:d-by.t'e fact 44ty !!ce teet. != lieu ef ettepti=a the w&4en-emam4aatton rd Opretig teett preparM hy the -f ac!!.t[y M : = ;;;, t h Cx--ic:i;; ::y F:p r: th :=:incti= =d t::th

) The Commission will prepare and administer the written examinations and operating tests at non-power reactor facilities.'

2. Imoact of the Rule Chanae on facility Licenseet N The NRC currently depends on NRC employees and contractors to prepare and administer the initial operator licensing examinations required by 10 CFR Part 55, in accordance with 10 CFR 170.12(1),

the cost of NRC time spent and any related contractual costs are billed directly to the facility licensees that receive the examination services.

4 3

(k ber ee nW k%h bsm Au sam.t u.e weh e,c.mm% u A opra.63 t=Sh h

.. .m..a v .,g5 m <-m , ,_a % s m som s e n ed e ge r I i rg te s .

M " M n E *^ "od

@ Tn lieu t:4 te E.uf m $'re % P4"t i

kd Te h }and % y ring %=. 'ste

y~e5 ped re tw-rede 4rh writha ensows.4*,onr , M. Comm\nkn (ca.y

%u w Am %s gTcfrin ou' p$ler4 M kwhIS' exk fb'd t 9 9 054l, _ % 8l\ ( hted' f *eN*M C.4MA) g f d Ar t e, on e,se.w e .

, ~. _ , , .

. _ , _- _ . _ . , . , , . _._-__.-,y__ _ _ _ , < , - . . _ _ _ _ - . . . , . . , .

1 : :

I eMod b, Under the proposed change, each power reactor facielty licensee will

+

f hh '

assume responsibility for prepart ag the site-specific initial o>erator licensing examinations at its facilities, thereby allowing 3 tie NRC to discontinue the use of I contract examiners for that /

purpose. Facility licensees will prepare and submit prososed examinations (including the written examination, the wal(-throug ( ,

and the dynamic simulator tests) to the NRC fn-4ccordahtt_Wtth the y i t. h 3 instruction & in NURfG-1021. The NURIG has been edited to accommodate the revised examination process and is the same procedure that the NRC examiners or its centractors would use if they were to develop the examinations, in conformance with Sectici 107 of the ACA, which requires the Commission to prescribe uniform

'; conditions for licensing operators, the NRC will not sentider attrou- d

Vi A,c M alternativs testing methodologieq % e m W s* 4 ( 9 (ru m *# ^ H k esc-W Au 4M %4d ha 3 % %=

6 %e ..g.dh pwna uAisJ inThe training staffs at power reactor factifties already haveeptro the d basic knowledge, skills, and abilities necessary to evaluate operator performance and develop test items for the initial licensing examination. During the mid-to-late-1980s, the industry's

~

emphasis in the training area increased significantly. All power reactor Itcensees established formal training programs that were based on a systems approach to training (SA1) and accredited by the National Academy for Nuclear Training. Pursuant to 10 CFR 50.120 and 55.4, SAT-based training programs must include the evaluation of the traince's mastery of training objectives. NRC inspections of Itcensee requalification programs for licensed operators have also found that training staffs generally possess the skills needed to evaluate the trainee's knowledge.

. prodtf After the NRC\ reviews and approves an examination, the facility pel en licensee willladminister and grade the written portion to-peeeeH h t 1 in-omwhet" M in NUREG 1021. The NRC staff will continue to administer and grade M"' " the operating tests, review and approve the written examination results recommended by the facility Itcensee, and make the final licensing decisions, feedback from the pilot examination program indicates that the average time spent by a facility licensee to prepare the written examination and operating tests was approximately 600 to 800 staff-hours. A portion of that time (about 200 hours0.00231 days 0.0556 hours 3.306878e-4 weeks 7.61e-5 months ) would have been spent rrviewing and assisting with the administration of NRC-developed examinations under the process now in place and should be subtracted from the total. The.resulting average burden of approximately 400 to 600 staff-hours was somewhat higher than the 400 hours0.00463 days 0.111 hours 6.613757e-4 weeks 1.522e-4 months that NRC contract examiners typically take to prep re aa.

examination. The extra burden is generally attributable to the fnility licensecs' lack of familiarity with specific NRC examination expectations and to the additional administrative requirements, such as documenting the source of the examinatibn questions, that are required to maintain examination integrity. It is worth noting that some of the facility licensees that participated in the pilot program expended less time than is 4

~ . , .- _ .__ _ - _. _ _ _- ._ _

m-r.

)

k ,Q comonly used by NRC contractors to prepare the examinations.

furthermore, in a few cases, the examinations that facility

% licensees submitted for review and approval were, in the judgment of NRC chief examiners, as good or better than those prepared by an NRC contractor. The staff expects that most facility Itcensees will eventually be able to prepare quality examinations in less time than the NRC or a contractor because the facility employees have more

. detailed knowledge of their factitty and easier access to the reference materials required to prepare the examinations.

Experience dur"ng the pilot examination program indicated that the volume of reference materials required by the NRC examiners to review and prepare for the examinations was significantly less than

' that required (c. the NRC staff to write the examinations. The feet that contract examiners will not be used in the revised examination process will also eliminate the need for duplicate sets of reference materials to be provided by facility licensees. Feedback from the j industry in response to the staff's solicitation of public comments on the draft revision of NUREG-1021 indicated that facility licensees had been spending from 80 to 160 hours0.00185 days 0.0444 hours 2.645503e-4 weeks 6.088e-5 months to prepare and ship the reference materials under the existing examination process.

The additional burden of having to prepare the site-specific initial operator licensing examinations should be substantially, if not totally, offset by reductions in Part 55 review fees billed to the facility licensees pursuant to 10 CfR 170.12(1). E4ch facility licensee will be billed only for the time that the NRC staff spends to review the examination prepared by the facility licensee and to rework the examination, as necessary, to bring it up to NRC standards; no contractual costs will be accrued. Although several of the draft pliot examinations were of poor quality and took the NRC staff more time than expected to review and rework, the staff believes that additional cost reductions will be realized as facility licensees gain experience with the NRC examination requirements and the quality of the draf t examinations improves.

' >> c),

This rule change will give factitty licensees more control over the 1 cost of their examination services because they will be in a e position to manage the quality of the product that is submitted to the NRC. The higher the quality of the examination the facility (I)d

'hy Itcensee subtrits, the lower the resulting charges. Under the existing examination process, facility Itcensees are responsible for (

the entire cost of preparing the examination, even if the +

i 4*

contractor's submittal is of poor quality and requires significant rework by the NRC staff before it can be administered. -fQA It is also worth noting that facility licensees will have the option of retaining the services of a contractor to prepare the license examination as the NRC often does under the current examination process. The staff understands that the NRC's examination contractors have already expressed an interest in providing their services directly to facility licensees. It is reasonable to assume 5

. w ..

E "m that the cost of using this option would be comparable to the I contractor service fees that the NRC currently passes on to the facility licensees.

l In sumary, the staff views this process change as resource neutral at worst, and possibly resulting in a resource savings to facility

< licensees over time.

3. Impact of the Rule Chance onJ)perator License Apolicants The goal of the rule change is to eitminate the NRC staff's reliance on contractor support for the site-specific initial operator

}

licensing program, while complying with the AEA and maintaining the existing level of effectiveness. To the extent possible, the I format, content, length, and level of difficulty of the examinations will remain unchanged, thereby minimizing the impact of the rule change on the operator license app 1tcants. ,,g g , i n NRC examiners will continue to review and approve every written examination and operating test before it is administered. The examiners will work with the facility licensee's staff to modify the draf t examinations as necessary to envre :W!me with 'NUREG-1021 o m*p and to maintain consistency with prior licensing examinations at that f acility and among f act11 ties, Awa g

If the NRC decides to prepare thr examination in lieu of accepting an examination prepared by the fa cility licensee, the NRC examiners will use the same procedures and crtier_ta (i.e., NVREG-1021) that the facility licensee would have used to prepare the exaraination.

The operator license applicants should not be able to distinguish between an examination that is prepared by the facility Itcensee and one that is prepared by an NRC examiner.

4. Alternatives to the Rule Chanap if the Commission decides not to amend 10 CFR Part 55 as proposed by the staff, it would require the restoration of contractor funding for the operator licensing program or the allocation of sufficient direct examiner resources in each regional office to satisfy the demand for initial licensina examinations and to conduct the licensed operator requalif t: alon inspections. Historically, the NRC has spent approximately $3 million to $4 million per year on contractor support funding for the operator licensing program.

111. Other Elements Leoal Analysis by the Office of the General Counsel (OGC)

OGC has reviewed this rulemaking plan and does not believd this action constitutes a backfit pursuant to 10 CFR 50.109. In addition, OGC has concluded that this action does not constitute a major rule pursuant to

.. 6 0

~

%%.nk A

qS theSaallBusinessRegulatory'lIstrnessActof1996. finally, at this i time OGC is not aware of any reason for a legal objection to this rulemaking plan.

BAckfit Analys.11 As discussed in SECY-96-123, OGC aarees with the staff's position that

requiring the facility licensees to prepare the initial operator licensing examinations would not be a backfit pursuant to 10 CFR 50.109, which r,tates in pertinent part that --

Backfitting is defined as the modification of or addition to systems, structures, components, or design of a facility; or the design approval or manufacturing license for a facility; or the procedures or organization required to design, construct or

operate a faellity; any of which may result from a new or amended provision in the Comission rules or the imposition of a regulatory staff position interpreting the Commission rules that is either new or different from a previously appItcable staff position....

The proposed change does not result in a modification of or an addition to systems, structures, components, or the design of a facility. The change does not affect the design approval or manufacturing license for a facility. The procedures required to design or operate a facility will not be affected by the proposed change. The proposed change would require each nuclear power plant licensee to develop the tests that are used to qualify, as meeting the requirements of 10 CFR Part 55, those nuclear power plant operators whom the nuclear power plant licensee wishes to employ. Development of such tests are not considered to be

' procedures... required to... operate a facility.' The tests are not applied to the facility licensee, but rather to the operator license candidates. Further, any procedure necessary to develop the test would .

not be useful in actually " operating' the facility, even if one broadly interprets ' operating" as including any action necessary to comply with the Comission's regulations with respect to operation. The organization required to design or operate a facility will not be affected because all facility licensees already have a training staff to train and evaluate candidates for operator licenses and to train other members of the plant staff, as required by 10 CFR Part 55 and by 10 CFR 50.120. Therefore an organizational change is not required because of this process change.

Suncortina Documents A regulatory analysis and an Office of Management and Budget (OM6) statement will be prepared. Neither an envircnmental impact statement nor an environmental assessment will be prepared for this proposed rule, however, since it is the type of action described as a categorical exclusion in 10 CFR 51.22(c)(1).

7 "1 '## ' ' ' .-

c < pp 4+ 1% , _ . , .(%,,, ,, g _ .,

y.__,.g.,.__ ,__y,_..._.-_4-. 3_. . y,q - p . e ,

hee.g.ig FM Wt. 71%, w dw% Men 4 b

) * ;

aa 4A.44 a a.s...a ..- s. % py- % r

' aN H N v % , e s W b <. Sud&4, n u 4.ror Meow.i a oa .a c 1, % m n. 3,' 1 ea Resources Renuired ,

j A2 discussed previously in Section !!.2, the rule will require facilit/

licensees to assume responsibility for writing the examinations, a burden currently carried by the NRC. However, the transfer of this responsibility is considered to be resource neutral, and could possibly 1 1 result in a resource savings to facility Itcensees.- !

I The rulemaking is expected to take approximately 0.5 staff years to

complete, and no contractor support wl11 be needed. However, until the l rulemaking is complete and all power reactor facility licensees are i required to prepare their examinations, the Office of Nuclear Reactor

Regulation (NRR) will continue to use contractors as necessary to supplement the taff's ability to develop and administer the in\tlal operator licensing examinations and to conduct the requalification inspection program. When the rule goes into effect the staff expects thattherevisedexaminationprocessandtherequalIficationinsection program can be implemented at all power reactor facilities with existing NRC resources allotted to the operator licensing progr -

Coanizant Staff / Pro.iect Manaaement Concurrina Official Of fice of' Nuclear Regulatory Research (Lead): H. Tovmassian Bill Morris Office of Nuclear Reactor Regulation: S. Richards Bruce Boger Office of General Counsel; M. Schwartz Stuart Treby Steerina Group A steering group will not be required for this effort.

Enhanced Public Particioation This rulemaking will use slectronic bulletin boards, as appropriate, to enhance input from the pubite.

[00 or Commission Issuance

] The proposed and final rule will be approved by the Commission.

Schedule Proposed rule published: 3 months after the EDO approves plan Comment period ends: 60 days after the proposed rule is.

published Final rule published: -6 months af ter the comment period ends 8

~ .~

~ _ - , . . - . - - , ,. , _ , . ., _

-, .. -., ..- . .- . . - - - - _ - _ , - - . - - . --.-. -.- - . . , ~

A sa n#... ma.p 46 M,. a. .a .a - - - . o'u 8 '; _ s

.t i

, [

flg@,4 [

m, ;

4 t t

i !,

,3 i

.set ; ,

h. f f ~I i

k ,

I

'i e l

-I

?

RESPONSE TO THE STAFF REQUIREMENTS MEMORANDUM OF JULY 23, 1996 (M960618A) !

t h

e l

i t

1' f

+ -

h 1

f i

1 5

ENCLOSURE 2 ,

i j

k

'[

l

- - , . - * +

c .4 - ,_+ . . . . .. I I

., r.. - w,._r w - _m , _ . y. . _ . . , . -

3.g 5 **

4

. . . _ . . , _ _. , . . , .. ....,.,.,,..,._,_,m...-_ , , ....__--.i. , , , . _ , - , . - - , - _ . . , _ - . . . . , . . - . . . _ . .. . _ . ,. ,, - ... _, _ ,.,=.,,,.,_ _ - ,..., , . _ , ,

. ,.,e

'[

DISCUS $10N OF PROPOSEQ.0PERATOR LICENSING PR0fiRAM CHAiGES 4% g/h.q q a d h a i Backaround &

% h 4 p.w , s m. j The site-specificfinitial operator licensing examinat ion consists of a 100-question writtenjtxamination and an operating test that includes a performance demonstration on a dynamic sipulator and-e-plant walk-through. Tha written examination is/ multiple-choic% end-ehnd h:h 4m for nat, with a passing grade /

of 80 percent. The simulator )Prformance-dem06stratt >n generally consists of two or three scenarios administered to a team of thre n license applicants.

Each applicant is individually evaluated on a range o ' competencies applicable

~

to the applicant's license level (i.e., reactor opera' or (RO) or s Ator reactor operator CSRO)). The walk-through portion of the operati test ty requires each app'icant to individually complete a tot al of ten ta ks or_Jo%

b performance measures (JPMs), in the control room and the plant and to answer specific que..lons re ated to the associated systems 1 The entire site.

specificexaminationhashistoricallybeenprepareddyNRClicenseexaminers or .C contractors. As a prerequisite for taking the site-specific ensination, every applicant must pass a separate generic fundamentals examination (GFE), consisting of 100 multiple-choice questions covering mechanical components, principles of heat transfer, thermodynamics, fluid mechanics, and fundamentals of reactor theory. An NRC contractor prepares two versions of the GFE (one for boiling water reactors and another for pressurized water reactors) twice per year. The GfEs are reviewed and approved by the Operator Licensing Branch (HOLB) before being mailed to the facility licensees for administration.

On March 24, 1995, SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program,' informed the Commission of the staff's intent to revise the operator licensing program to allow greater pa.ticipation by facility licensees and to eliminate contractor assistance in this area. On April 18, 1995, the Commission consented to the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full implementation of the changes.

From May to June 1995, the Division of Reactor Controls and Human f actors (DRCH) and HOLB conducted a number of meetings with senior management and regional personnel to discuss various options for implementing the proposal outlined in SECY-95-075. On August 15, 1995, after weighing the pros and cons of the various options (dl uussed bel 4), the staff issued Generic letter (GL) 95-06, ' Changes in the Operator Licensing Program," which notified power reactor facility licensees of the NRC's intent to change the operator licensing process, outlinsd a number of criteria to supplement the guidance in NUREG-1021, ' Operator Licensing Examiner Standards," and solicited volunteers to participate in a 6-month pilot program to evaluate and refine the new examination methodology.

On June 10, 1996, the staff issued SECY-96-123, ' Proposed Chant es to the NRC j

Operator Licensing Program," which described the pilot examin*, tion process and criteria, reviewed the examination results, discussed the le, sons learned, and recommended that the enhanced process be implemented on a vtsluntary basis 1

pending an amendment to 10 CFR P . 55 that wNid require all power reactor m

q factitty licensees to prepare '.netr Itcensipj examinations. 14e staff briefed the Commission on June 18, 1916, and in a n aff requirements memorandum (SRM) dated July 23, 1996, was direc,ed to deve'op a detailed rulemaking plan and to address the following specific issues ro'jarding the proposed changes in the 1

examination process.

1 Eros and Cons (Vgingsbilitiuj

' In the early stages of developing the pilot examination process, the staff identified sever:.1 concerns (examination integrity, quality, and consistency.

- Independence and public perception, examination security, resources requirements, program stability, and examiner proficiency) that have played i dominant role in the design, implementation, and refinement of the revised examination methodology. The staf f evaluated all of the pilot examination criteria specified in GL 95-06 in 'ight of these contri,11tng concerns. The same concerns were also integral in evaluating the post-pilot process rodifications that have been incorporated in the proposed revision of NUREG-1021. Although the staff believes that each concern has been adequately addressed and mitigated, it would be disingenuous to assert that industry-wide implementation of the revised examination process does not entail a measure of uncertainty. The following is a discussion of each concern, the measures that have been implemented to mitigate the concern, and the associated vulnerabilities.

Integrity, Quality, and Consistency of the Examination Section 107 of the Atomic fnergy Act of 1954, as amended, requires the Commission to prescribe uniform conditions for licensing individuals as operators, determine the qualifications of such individuals, and issue licenses to such individuals in such form as the Commission may prescribe.

When the staff developed the pilot examination program, it was concerned about wi.S her the existing levels of examination uniformity and consistency could be maintained. The operator licensing recentralization study (see SECY-93-309,

" findings of the Operator Licensing Recentralization Study,' of November 17,1993), which had concluded that the level of examination uniformity was adequate to ensure that appropriate licensing decisions were being made, highlighted the challenge of controlling the diversity that i

individual examiners from dferent regional offices could add to the

examination program. permitting a substantial numbsr of facility employees l having differing levels of training and experience to prepare the examinations could increase the amount of variation in the levels of knowledge tested and examination difficulty despite the efforts of the NRC to control these factors through improved pry edt.res and oversight reviews.

To maintain uniform standards of examination content, format, evel of difficulty, and quality, the staff insisted that the participa nts in the pilot program prepare the written examinations and operating tests in accordance with the procedures and guidelines in Revision 7 of NUREG-1021. NUREG-1021 contains guidelines and criteria that are adequate to ensure that a licensing examination, independently prepared by an NRC examiner or contractor, will 2

" differentiate between those applicants who have and those w..o have not li mastered the knowledge and abilities required to safely operate the plant.

However, when the staff reviewed those criteria in light of the proposed shift in responsibility for writing the examinations, it concluded that some of the criteria were no longer appropriate and that additional criteria were developed examinations

necessary to ensure that the integrity of the facilityff promulgated the j would remain at acceptable levels. Tharefore, the sta following supplemental and modified cr.teria in GL 95-06 when it solicited volunteers to participate in the pilot examination program

o NUREG-1021 currently allows NRC and contract examiners to use up to 10 percent of the written questions, job performance measures, and simulator scenarios in a facility licensee's examination bank when they develop an init:al licensing examination for the f.-ility. This process saved NRC resources and provided an incentive foi a tility licensees to develop their examination banks because the examine could conceivably extract the entire written examination from a facility licensee's bank, provided the bank contained 1,000 or more questions that satisfied the psychometric criteria in NUREG/BR-0122, "Exami.wrs' Handbook for Developing Operator Licensing Written Examinations."

Whereas the NRC's examiners are continually revising old test items and developing new ones, theie :s currently no requirement for facility licensees to take comparable measures to minimize predictability.

Predictable examinations tend not to discriminate between competent and deficient license applicants because what is being tested is simple recognition of the answer. Although studying past examinations can have a positive leaaing value, total predictability of examination coverage through over-reliance upon examination banks reduces examination integrity. When the examinees know the precise and limited pool from which test items will be drawn, they will tend only to study from that pool (i.e., studying to the test) and may exclude from study the larger domain of job knowledge. When this situation occurs, it decreases confidence in the knowledge inferences that are made from performance on the test.

To limit predictability, GL 95-06 revised the limits on the use of 1 facility item banks that could have been published, reviewed, or used as a basis for training. GL 95-06 treated all question banks as if they were tpen to the applicants because the staff had no basis to regulate the ou.9 d ies of the item banks and because it would be difficult to confirm that a bank was indeed closed to the applicants. The staff attempted to strike a balance between the use of new and existing test items because it appreciated the effort required to develop new test items and understood that existing items are a valuable training ret . et . hat should not be wasted. On the basis of the results of the gW:q %ndamentals examination (GFE) program, the staff decided to lien F.e un of existing written questions on the site-specific examinettors to no more than 50 percent and to require at least 10 newly developed (i.e., not previously seen) questions on every examination.

The remaining questions could be generated by ::1gnificantly modifying 3

i

- 4 e -q p w,-r ,

r

L <

6 existing bank questions. Each operating test was required to include at least two JPMs that were either new or significantly altered and at

&j least one simulator scenario that was new or significantly altered. The

' remaining scenarios had to be modified to the extent necessary to prevent the applicants from imediately recognizing the scenarios on the 3

basis of the initial conditions or other cues.

4 '

o NUREG-1021 currently places a 25-percent limit on the number of written examination questions that can be repeated from the previous two NRC I licensing examinations at the facility. To furtlier limit the a predictability of the examinations that would .n prepared by the facility licensees, the staff revised tnis critorion to include all examinations, quizzes, and tests adminiv ered '.u the applicants during their license trainl y class. Furthermore, t.o questions could be draw..

directly from the applicants' audit examination given by the factitty licensee at the end of the license training class to determine if the applicant was ready for licensing. This rastriction was the_ subject of industry comments and has been revised as discussed below since completing the pilot program.

o NRC and contract examiners are currently required to cite a specific facility reference for every question on the written examination.

Questions that were extracted from the facility licensee's question bank would be identified as such. In order for the NRC chief examiners.to monitor the facility licensees' compliance with the previously stated criteria, GL 95-06 also required the pilot examination participants to state the history (e.g., bank, revised, new, and date last ustd) of each test item on the written exainination and operating tests. Although this practice did place an additional burden on the facility licensees, it facilitated the NRC's examination reviews and resulted in lower licensing fees billed to the pilot participants. This requirement was

~

also addressed in the industry comments and has been clarified as discussed below since completing the pilot program.

The Office of Nuclear Reactor Regulatien (NRR) also issued supplementary written guidance to the regional offices clarifying the NRC examiners' responsibilities during the pilot examination development process, and it conducted a public workshop to discuss the staff's expectations during the pilot program. The written guidance and the minutes of the workshop were

,i placed in the NRC's Public Document Room (POR) and provided to facility licensees that were scheduled to participate in the pilot examination program, l' NRR did not restrict the number or scope of changes that the NRC examiners could ask facility licensees to make in their proposed examinations. The existing quality assurance (QA) checklists in NUREG-1021 were revised and new checklists were developed to help the examiners identify problems with the preliminary examination outlines and the operating tests that the facility licensee would be required to submit for review.

'l Because of resource limitations, examiners were instructed to focus primarily on the content and corstruction (i.e., level of knowledge and difficulty) of 4

i L

1 hwh b h y.A e.u=. udgae) m 4

pxas .c a t e ae. qc d bg %. prc.

% OQctbr %5) t!" As the written examination questions, rather than spend their time verifying the

[ g~' detailed technical accuracy (e.g., procedural and component references) of every question, as is required for NRC-developed examinations. The staff expected the facility licensees to ensure that the examinations were j technically sound and instructed the regions to question the licensees I regarding any post-examination question deletions and answer key changes.

To assess the effectiveness of the pilot examination process, NRR asked the NRC examiners who were involved with the exar.inations to respond to a questionnaire on various aspects of the new methodology. The feedback indicated that many of the as-written exam rations were comparable in quality and level of difficulty to recent NRC-written examinations. However, the NRC examiners and facility staff had to rework some of the examinations i

significantly to meet NRC standards. Although the staff had hoped that the technical accuracy of the facility licensee's written examinations would be as good as or better than the accuracy of the examinations written by the NRC, some of the pilot examination results (i.e., more question deletions and answer key changes than expected) indicated a need for improvement in that area.

Despite the challenges of quality and technical accuracy, the staff believes that each as-given pilot examination provided a valid basis for licensing the applicants who passed the examination. The staff expects that the consistency and quality of the draft examinations will improve as facility licensees gain experience and become more familiar with NRC examination requirements. Based on observations during the pliot examinations, the staff has also incorporated in Revision 8 of NUREG-1021 a number of process enhancements that are intended to improve the quality of the examinations developed under the revised

)rocess. A number of the significant enhancements are discussed below. NRR 1as also instructed the regional examiners and managers not to approve or administer any examination that does not adhere to current NRC standards for content, format, quality, level of knowledge, and level of difficulty.1 If necessary, the examination is to be delayed or cancelled until the facility licensee and the NRC staff can upgrade or replace the examination.

The overall results of the pilot program indicated that the examinations prepared by facility licensees, subject to review, revision where appropriate, and approval by NRC staff examiners, were generally consistent with examinations prepared by NRC or contract examiners. The pilot examinations also appeared to be equally effective at identifying applicants who had not mastered the job requirements wd 7np.gh to become licensed R0s or SR0s.

Furthermore,thepassrates(of pnd SR0s on the written examinations and )C operating tests compared favorb 4tth the pass rates for examinations prepared by the NRC or its contractors.

The chief examiners for approximately half of the pilot examinations actually considered the submitted examinations to be as good or better than recently administered NRC examinations because the facility authors were better able to integrate the required site-specific knowledge and abilities into operationally oriented questions. All but 5 of the 22 feedback surveys completed by the chief examiners indicated that the revised examination process was as effective or more effective than the traditional examination 5

h

> y process. Most examiners also agreed that the performanca of the facility i licensees should improve as they become more familiar with the NRC's requirements and expectations.

in summary, the operator licensing program staff has implemented numerous 4 procedural measures to maintain and promote the consistency, quality, and-integrity of the initial licensing examinations. However, in practice, it will-be the responsibility of individual examiners, with appropriate review l

and oversight by their supervisors and the NRR staff, to ensure that_the licensing process remains strong, it is possible that the pressures to

conduct examinations on schedule and the reduced involvement of the NRC in the j examination development process could_ lead to examinations that discriminate at a lower level than is currently the norm. The revised examination process is much more dependent u>on the staff's ability to identify a substandard examination and ensure tiat the deficiencies are corrected. If the staff is indecisive and the licensee takes advantage of the situation, the examination-will probably not discriminate at the appropriate level.

Independence and Public Perception before 1987, the staff believed that only independently developed and administered examinations could reliably and consistently achieve the appropriate standards of objectivity and discrimination required for.the NRC's initial licensing and requalification programs. The staff was concerned that examinations prepared by a facility licensee could be unconsciously biased toward the specific material taught by the facility and that potential financial or other pressures on the f a:llity licensee would create a conflict of interest.

The staff's position changed significantly in 1988 when it revised the requalification examination methodolony so as to require each facility licensee to prepare the examination materials and demonstrate its ability to evaluate the competence of its licensed operators. From 1989 to 1994, the NRC evaluated over 4,000 licensed operators using requalification examinations that were prepared by the facility licensees, reviewed and approved by the NRC, and administered in parallel by NRC examiners and facility personnel.

The examinations appeared to discriminate at the appropriate level, and approximately 90 percent of the operators passed the examinations on their first attempt.

Despite the success of the requalification examination process,_the staff a continued to independently develop and administer the initial licensing 1 examinations. However, the overall performance of the license applicants on the initial examinations was not significantly different from the performance of the licensed operators on the requalification examinations developed by their facility licensees. From 1989 to 1994, the NRC administered initial licensing examinations to approximately 4,000 applicants; the pass-rate on those examinations averaged approximately 90 percent, i In SECY-95-075, the staff proposed to take the next evolutionary step, similar to the-staff's actions with regard to oversight of the requalification program, by changing the guidance in NUREG-1021 to permit facility. licensees 6

W ie MTVWy g%es . ,Ae*w -v *'PM

l s' to prepare and, in part, conduct the written and ooerating examinations for initial license applicants. The NRC would review the licensees' efforts,

-I partially participate in the examinations, retain the final pass / fail '

authority, and continue to issue licenses. This action would allow the staff j to perform its licensing function in a more efficient manner, while maintaining an effective, independent process upon which to base licensing decisions for operators.

Shortly after SECY-95-075 was issued, The Washington Post reported that

consumer advocates were concerned that allowing 4cility licensees to prepare the examinations might endanger reactor safety because the examinations might be less rigorous than those prepared by the NRC. The information upon which the consumer groups had based their conclusions was preliminary in nature and likely did not consid.c the level of oversight that the NRC would retain in the examination process or the compensatory measures that the staff would implement to ensure that the examinations retained an appropriate level of difficulty, i

The Washington Post article also indicated that the proposal appeared to contradict the staff's 6arlier decision to reject a request by the Virginia Electric and Power Company (VEPCO) to prepare its own examinations (this topic will be discussed in more detail l uer) and that a number of mid-level staff members opposed the change. The staff believes that the initial opposition to the proposal was a natural reaction to the abrupt change in policy and to the fact that many staff members had not been consulted or informed before SECY-95-075 was issued. As promised in the memorandum of April 12, 1995, from the Executive Director for Operations, all involved staff members have been well informed of the new policy and have been able to express their views and to have their concerns considered as part of the process to implement the broad goak stated in SECY-95-075.

As noted in SECY-96-123, the results of the pilot program indicated that initial licensing examinations prepared by facility licensees, subject to review and approval by NRC staff examiners in a manner similar to the review and approval of NRC requalification examinations, were as effective as examinations written by contractors in identifying applicants who had noty ,

mastered the job requirements for a licensed operator. The pass rates orRfs x and SR0s on the written examinations and operating tests compared favor bl with the pass rates for examinations prepared by the NRC and its contractors.

, To promote examinction consistency and minimize the bias teward specific j materials taught by the facility licensee, the staff insisted that the pilot examinations be developed in accordance with NUREG-1021 and NUREG/BR-0122.

NUREG/BR-0122 contains explicit guidance for sampling the important, safety-related knowledge and abilities (K/As) from NUREG-ll22 " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors " and NUREG-ll23, " Knowledge and Abilities Catalog for Nuclear Power

, Plant Operators: Boiling Water Reactors," companion documents derived from a job-task analysis of the operators' work. The supplementary pilot examination criteria in GL 95-06 required the facility licensee to submit a proposed examination outline to the NRC for review and approval at least 60 days before the scheduled examination date. This requirement would enable the NRC chief 7

weespgng --- wa ww , .,-. e.yymm-

l <

a . .

4 I f

examiner to assess whether the sampling was appropriate (e.g., the balance of .

}i coverage and importance ratings; duplication and overlap between examination sections and from pnor examinations; and inclusion of plant-si,ecific priorities, such n recent events and risk insights) before the facility

itcensee proceehd with developing the detailed examination materials.

GL 95-06 alsn required facility licensees to submit the proposed examination to the NR0 for review at least 30 days before the scheduled administration i date. The NRC chief examiner and the responsible supervisor would work with l

the facility licensee to revise the examination as appropriate before 4 authorizing its administration, i To ministi.e personal conflicts of interest, GL 95-06 prohibited facility l employees who played a substantial role in training the license applicants .

from becoming involved in .,reparing the licensing examinations. Although this restriction did little to mitigate the fact that everyone who worked for the factitty licensee was likely to have a vested interest in the outcome of the examinations, it did alleviate some of the concerns regarding personal gains or penalties that the examination author might incur if he or she were directly accountable for the examination results. This restriction also allayed the staff's concern that an instructor might unconsciously bias the examination toward those topics that were emphastred during the training program. The personnel restrictions were undoubtedly the most controversial aspect of the pilot examination process. They were widely discussed among the NRC staff and management before, during, and after the pilot program was implemented, and they were also the principal issue raised by industry regarding the pilot process. A discussion of this and a number of other industry concerns follow:.

The issues of independence and public perception also had a significant bearing on the methodology that was selected for administering the operating tests, the best scasure of the applicants' ability to respond to a plant event. Although general agreement existed among the staff on most of th.!

pilot criteria discussed earlier, there was considerable disagreement regarding the level of involvement that NRC examiners should retain in the operating test administration process. The following options were evaluated in depth:

Option 1: NRC examiners would continue to independently administer the simulator scenarios and the plant walk-through portion of the operating test in accordance with current guidance in NUREG 1021. Involvement by the facility licensees would not change from current practice, and facility employees would not conduct parallel evaluations.

Option 2: NRC examiners and facility employees would conduct parallel evaluations of each applicant during the dynamic simulator and walk-through portions of the operating test. This is the same methodology that is used during NRC-conducted requalification examinations.

Option 3: NRC examiners would observe facility employees conduct the simulator scenarios and a sample of the walk-through tasks 8

w- ,.;m.. ..-. m.,g---

- -ye m aw,e.- w.+w,.cs.- .

f' M for each applicant; however, the NRC exantrcrs would not conduct one-one ne parallel evaluations of each applicant.

A This methodology is similar to that currently used during the requalification inspection program.

Af ter much deliberation, the staff decided to implement Option 1 during the

' pilot examination program. The staff believed that Option I would have'the most favorable public perception and examiner acceptance because it maintains the existing levels of independence and objectivity by relying on NRC-

j examiners to conduct and evaluate the operating tests. Option 2 would have 8 generated more concern regarding conflicts of interest because it would require facility employees to administer and grade the operating tests (to their coworkers and friends) in parallel with NRC examiners. Option 3 would have had the least NRC invoh eecnt and, therefore, the greatest >otential impact on independence and pubile perception. By not directly o> serving each applicant, the NRC examinar would be in a weaker position to judge the performance of the applicant so as to determine which applicants should pass or fall if a crew responded incorrectly during a simulator scenario.

In summary, it is possible that the pressures to conduct the examinations on schedule and the reduced involvement of the NRC in the examination development process could lead to examinations that reflect the biases of the facility Itcensee and that discriminate at a lower level than is currently the norm.

However, a number of process requirements have been augmented as necessary to minimize this potential. The staff believes that the revised examination process, which includes restrictions on personnel, independent NRC review and approval of the examinations, and independent NRC administration of the operating tests, will provide sufficient safeguards to ensure unbiased licensing decisions.

- Given the complexity of the proposed change in the examination process, the public may jump to the conclusion that the NRC has transferred all responsibility for examining operator license applicants to the facility licensees. In actuality, the NRC retains a significant amount of control over the content of the examinations and continues to administer the operating tests. The challenge may be to explain this fact to the public.

I:xamination Security in 10 CfR 55.49, applicants, licensees, and facility licensees are prohibited from engaging in any activity that cor,comises the integrity (security) of any appilcation, tast, or examination required by Part 55.

Before 1989, ensuring compliance with the rule war relktively straightforward because facility licensees were not permitted to review the written operator licensing examination until it was distributed to the applicants on examination day. Security compromises and cheating were rare but not unheard of. ..

The degree of risk to examination security increased significantly in January 1989 with Revision 5 of NUREG-1021. Revision 5 made the facility licensees

- largely responsible for preparing the NRC requalification examinations and, 9

A

_s., . . _ *-*n *b

,w

J ,

i n

n M -

for the first time, also allowed the facility staff to review the written licensing examination up to 2 weeks before its administration. The increased risk to security was tolerated because the prereviews helped identify 1

inaccurate questions before the examinations were administered, thereby j increasing the content validity of the examinations, minimizing the number of 4 questions from applicants during the examination, and reducing the number of 1 -question deletions and changes during the grading process. The gradual shift i to multiple-choice questions during the late 1980s and early 1990s ) laced a premium on accuracy because the answers to the questions could not >e adjusted

,? :fter the fact as they could be with the essay and short answer questions that they replaced.

To maximize security, given the new philosophy, NUREG-1021 prohibits eny facility employee with specific knowledge of any NRC examination before it is given from communicating the examination contents to unauthorized individuals and from participating in any further instruction of the students scheduled to take the examination. Before they are given access to the examination, the facility employees are required to sign a statement acknowledging their understanding of the restrictions and the potential consequences of noncompliance. They must also sign a post-examination statement certifying that they did not knowingly compromise the examination.

The staff realized that the potential for examination compromise would increase as the facility licensees increased their role in the initial licensing process from one of reviewing the NRC-developed examinations to that of developing their own examinations. Therefore, the staff attempted to clarify its security expectations in the supplemental pilot examination guidance that was provided to the regional offices (and placed in the POR) in conjunction with GL 95-06. The guidance addressed the restrictions on personnel and a number of physical security precautions, including protecting and mailing the examination materials. The guidance also cautioned examiners i to be attentive to examination security measures and required them to review

' the security requirements with the facility licensee's contact at the time the examination arrangements were confirmed.

The issue of examination security also had a bearing on the methodology that the staff selected for administering the pliot operating tests. The steff believed that Option I would pose the lowest threat to examination security because the smallest number of facility employees would require detailed knowledge of the examination before it was given. Options 2 and 3 both would have required the facility licensee to evaluate the license applicants during the simulator and walk-through tests, which could have required several

additional personnel to familiarize themselves with the operating test content, thereby increasing the risk of compromise.

In su m ary, the staff expects that the vast majority of facility licensees will continue to maintain proper examination security (as they have during the requalification program). Security is enhanced by the fact that the NRC monitors and oversees an extensive range of the facility licensees' activities and is sensitive to allegations and other performance indicators that might expose a problem. Nevertheless, the staff cannot deny that the NRC is now more vulnerable to examination compromise (and the associated public response) 10 3 sum .n+is -

p a 4-:,,;. ----

% w p. , m ..s uws \ p A

( ' %,nicM M S ou r'A y h as t o%tecdak O.

~', 6m ed @ @ 4bt 40% durW 6%

hW:e waenadwa c.m po eda hdi %da.-

%mPf'*dL _

L1 4

n a. m.ua..

$ than it was before 1989 when it first allowed facility licedees to become involved wi ;h the NRC's examination development pro rams. The staff

\

anticipates that the agency may suffer an occasiona lapseinsecurity.IWhen

}; a lapse oce 4rs, the staff will continue to take prompt action to cancel the j examination, determine the cause of the compromise, and implement corrective

$ measures l

L6 e.d b Resources As stated in SECY-95-075, the proposed changes wer: originally conceived as part of the NRC's continuing efforts to streamline the functions of the Federal Government consistent with the Administration's initiatives and to accommodate NRC resource reductions. The original plan was to entirely eliminate the use of .ontractors in the operator licensing area and to conduct

\ all operator licensing program activities, including the generic fundamentals i and initial licensing examinations, and the requalification program, with the

' existing staff resources. The facility licensees would prepare and, in part, i conduct the initial licensing examinations with oversight by the NRC. The NRC's participation would range from conducting part to all of every examination in order to provide an independent basis for making a licensing decision and to maintain examiner skills, pd gn j g The staff estimated that a qualified fa ity employee should be able to prepare the site-specific examination ( H ocordance-withlNUREG-1021, as supplemented by Gl. 95 06, in approximately the same amount of time that is currently allocated to a contract examiner. (See Attachment 1, the associated rulemaking plan, for a more detailed discussion of the estimated resource burden on facility licensees.) The staff also estimated that an experienced NRC examiner should take approximately two weeks to review the site-specific examination (including the written and operating tests) prepared by a facility licensee.

As previously noted, the staff considered three options for administering the operating tests. Assuming a license class consisting of nine applicants (e.g., six R0s and three SR0s) that could be arranged in three, three-person crews, both the first and second options would generally require three NRC examiners to spend a week at the site to administer the tests. The second option, which would require the facility licensee to conduct parallel evaluations of each applicant during the simulator scenarios and walk-through tests, would place a significant additional burden (nominally, three staff-weeks) on ihe facility licensee. The third option would be equally burdensome to the facility licensee but would permit the NRC to reduce (by one) the number of examiners dispatched to the site.

The plan to require the facility licensee to evaluate the applicants during the operating tests was perceived as a duplication of effort because the facility licensee had already evaluated the applicants and found them qualified before it submitted the license applications to the NRC. The staff was also concerned that the additional regulatory burden would discourage participation in what was originally envisioned as a voluntary program with no associated rulemaking, 11

lj.- ,

c 5 -s .

l k The NRC resource utilization rate during the pilot examinations supports the

. staff's~ original estimate that the new examination process and the=

. requalifica". ton inspection program could be implemented at all power reactor facilities with the same level of direct NRC resources as-is currently

' allotted to the operator Itcensing program. As noted in SECY-96-Its, the NRC j

examiners required an average of about 350 hours0.00405 days 0.0972 hours 5.787037e-4 weeks 1.33175e-4 months to review, prepare for,.

administer, grade, and document each pilot examination. .Most of the examinations included an extra visit to the site to review and validate the-

, . operating tests. [

+

3 However, with tha elimination of contractor support, the total number of 3

i examiners available will decline significantly, thereby reducing the ability _ .

T/ of the program to have the personnel available to meet peaks in facility-licensee request ( for exas.; nations. The impact of the loss of examiners du:

to reassignments, transfers, promotions,- and other personnel actions, will be larger without the contract examiners available to compensate for the loss.

To compensate in this area, NRR intends to require-that-the regional offices maintain a defined minimum level of examiners, to be defined in each regional-operating plan. During fiscal years-(FYs) 1997 and 1998, each region may need to qualify additional examiners to meet this expectation. To specifically assist the regions in their efforts to train additional examiners, NRR intends to make available to the regional offices up to 7 full time equivalents-(FTE) of inspection support during FY 1997 and up to 12 FTE of inspection support in ,

FY 1998. This inspection support will allow regional personnel, who would '

normally be assigned to inspection activities, to be assigned to qualify as examiners. By the end of FY 1998, the regions will be expected to maintain the defined level of examiners in order to provide the resource flexibility needed to deal with workload peaks. When examination work is not scheduled, the examiners will be available for inspection activities. A reduced level of

+ contractor support will be required during the transition period.

,[

The staff expects that if the quality of an examination prepared by a facility Itcensee deviates significantly from established norms, it will become increasingly difficult to predict the amount of time necessary to review or rewrite the examination so as to achieve an acceptable product. This was the

case with several of the pilot examinations, and the staff expects-that-it will continue to be more of a problem with facility-prepared examinations than

,; it has been 6 h those prepared by an NRC contractor. This-added uncertainty -

4- in the examu / 'on process could increase the risk of broken examination

commitments a " lower examination quality; it could also raise costs as a

. result of inee .cient rianning.

1 - With the elimination of contractor support and the increased uncertainty about i the examination quality, the staff may no longer have sufficient examiner--

resources to write or even to review all examinations consistent with the-scheduling needs of facility Itcensees. This-resource problem is further-

- compounded by the unpredictable nature of ~ the examination workload and by other unanticipated demands-on the examiner work force, such as the increase in the number of examination appeals during the pilot examination program.

Historically, the contract examiners have facilitated-the staff's ability to i respond to unexpectediincreases in the examination workload and unforeseen

{ - departures by_ staff examiners. Eliminating contractor support will complicate 1

12

, i ,

e i.

' #4 m T-

-,----w e - - _ g 9- , ,.w.,- - , _ . , ,

-,--.--# . .- -- - . . - - , . . w-> -r, m. ,w,-,,--.c w- ,,w.c -

(

scheduling and limit the staff's ability to conduct retake examinations, which

- h' could increase the likelihood that an applicant will appeal the results of the i examination. it is for these reasons that the staff is recommending to the Commission that 10 CFR Part 55 be amended to require all facility licensees to prepare the licensing examinations.

! With regard to the generic fundamentals examination (GFE), the staff considered the following options: (1) retaining the current format and process under which the examinations are developed by a commercial contractor and (2) revising the scope of the site-specific written examination to sample the generic fundamental knowledge and abilities, as was done before 1988, if the examination was revised by adding more questions, it would increase the j burden of writing and reviewing the examinations (whether it was done by the facility licensee or the NRC). If the length of the examination was kept the

! sam., inclusion of the generic knowledge and abilities would weaken the evaluation of the site-specific systems and procedures. The staff and the industry generally have agreed that the current program is both efficient and i

effective and that revising the site-specific sample plan would create too

! many problems. Accordingly, the staff has decided to retain the current examination format and process and to implement the program with contractor assistance at a cost of about $200,000 per year.

Program Stability From 1988 to 1990, the NRC received numerous complaints from facility licensees and licensed operators that the NRC's requalification examination program was causing undue stress for the operators and the facility employees who v.ere administering the examinations. The Human Factors Assessment Branch conducted a study to address this issue and determined that the frequent changes to the requalification examination process ranked among the most significant sources of undue stress; the results of the study are documented in SECY-91-391, "Results of the Study of Requalification Examination Stress."

One of the corrective actions invcived estabitshing a schedule for periodic revisions of NUREG-1021 and delaying the implementation of each revision for 180 days after it is published.

Although the staff did not survey license applicants during the requalification stress study, it has generally tried to minimize the adverse effect on applicants by avoiding unnecessary changes to the initial examination process. Therefore, when the staff decided to shift responsibility for developing the int al licensing examinations to the facility licensees, it resisted pressure to revise the examination format. To the extent possible, the format, content, and level of difficulty of the pilot examinations would remain unchanged, thereby making the changes imperceptible to the license applicants. The staff also believed that the facility licensees were generally familiar with the existing requirements in NUREG-1021 and that any significant changes in the examination format would confound their efforts to assume responsibility for writing the examinations.

The desire to maintain program stability, avoid unnecessary changes, and minimize stress on the applicants also argued ago nst the adoption of Options 2 and 3 for administering the operating tests. The requirement for facility 13

p .

(

dhi personnel to evaluate the license applicants would increase-the learning curve for the facility licensees and would likely elevate the stress for the applicants because of the double scrutiny and the increased crowding in the simulator.

S In-summary, the staff found no compelling reason to modify the existing 4

1 examination format. The benefits of retaining the current format far exceed i any argument for making changes at the present time. l l

d Examiner Proficiency On May 25r 1995, the staff briefed the Commission on the status of the 4 operator licensing programs and the staff's proposal to shift responsibility for preparing the initial operator licensing examinations to the facility-

, licensees. In the ensuing SRM of June 2, 1995, the Commission instructed the staff to ensure that the in-house capability to administer the examinations is maintained, possibly through the random administration of some examinations.

lu 4 v. M e cd au=A m AsnotedinSECY-96-123,Me.admstaffbelievesfhat-NRCexaminerswillmaintain the ability to write exaWinations, if necossary, based on their participation in the new process. Although facility licensees will prepare the operator l licensing examinations # uenr M e-with NUREG-1021, NRC examiners will i continue to review and approve every examination before it is administered to ensure that it conforms to the criteria specified in NUREG-1021 for content, format, quality, and level of knowledge, facility licensees will also cb r- M M e" and grade the written examinations in accordance with NRC procedures, but NRC examiners will continue to independently administer and grade both the dynamic simulator and the plant walk-through pcrtions of the operating tests. The NRC will review and approve the assignment of grades for the written examinations, including any changes to the answer keys and the deletion of questions recommended by the facility licensee. _ The NRC will also continue to make all licensing decisions and. administer the appeals process for applicants who fail the examin tion, ed&%dy,w. A my ded % geetee e e am,.w, e .9.A b %. f prom. A r% 6+ Ehu ear e.4e'ee b a.Cuy homue. w pr b %'-

NRR will continue to require new NRC license examiners to comp ete a standardized training program before they are certified to conduct licensing creh examinations. The training program includes classroom and simulator *%J'26'.

instruction on how to prepare and administer written examinations and M o>erating tests, self-study of operator licensing procedures and guidance, on '** N d:i l tie-job training, and a final practical evaluation by a certified chief N ~

examiner. Once certified, examiners are required to maintain their l

i qualification by conducting examinations and attending refresher training. "'M J' NRR is currently revising the examinor refresher training syllabus to focus more attention on reviewing, critiquing, and approving written examin2 tion questions. Examiners are also required to attend the operator licensing _

examiners' training conference, which is periodically convened by NRR to provide guidance and exchange information.

1 -

To ensure that the on-site examination activities are being performed as intended, supervisors are required to periodically evaluate every certified examiner during the conduct of an operating test. Furthermore, to verify, k p g n-1t O5 s the uniformity with which the o f- 1 !

l-

! M =4-

. 1 4*phg) c

" ' ~-

14 wN

,1 _ 'nt . .

Its 9-4

* *

n. a~ wwem. _ ._

g 9w 9- -g,--- q e y- w ,-,-w 't*-'"w'-"-$ W r'---~9

M'-"*

e-'" "t-- ' - - ' = ----W*"- e ' ' ' - 9' **= N--*

4 5 e l'

M' examinations are administered within and among the regions, the NRR staff periodically audits the administration of operator licensing examinations at selected f acilities and annually reviews the implementation of the operator licensing function in each region. The NRR audits and reviews include an assessment of the level of knowledge and the level of difficulty of the written examinations and the operating tests; each audit and review is documented in a report to the responsible regional Olvision Of rector.

Jadntry Acceptance and Resoonse in the summer of 1994 VEPCO recomended to the NRC that the initial licensing examination >rocess be changed to make facility licensees responsible for developing tae examinations subject to review and approval by the NRC chief

.. examiner. The NRC wod d continue to administer and grade the examinations.

2 The proposal, which was submitted as part of VEPCO's cost-beneficial Itcensing action program, estimated that the utility could save an average of $67,500 per station for each initial license class by implementing the proposed j approach. The staff reviewed and denied the proposal because of concerns regarding independence and objectivity and the potential for increased variability in the level of > owledge tested and examination difficulty.

The staff reconsidered its positi m early in 1995 when the NRC's fiscal expectations changed significantly, ihe staff decided to explore the possibility of eliminating the use of contractors in the operator licensing program by allowing facility licensees to develop the examinations, essentially just as VEPC0 had proposed. It was estimated that such a change would save the NRC from $3 million to $4 million per year in contractor support funding and that the reduction in recovered fees would largely or completely compensate the f acility licensees for their additional expenses.

While evaluating various options for implementing the changes proposed in SECY-95-075, the staff obtained informal feedback from the Nuclear Energy Institute (NEI) regarding the industry's level of involvement in administering the operating tests. NEl had polled a number of facility licensees and determined that there was no support for parallel grading (Option 2 previously discussed) because of concerns regarding applicant stress, the additional resource burden, and simulator crowding. The majority of those surveyed expressed cautious support for Option 3, under which NRC examiners would have observed the facility licensee administer the operating tests. However, those surveyed wanted the NRC examiners to focus only on the performance of the applicants .ad not on the examination practices of the facility licensee.

Option 3 would shift the most responsibility to the facility licensees, but it was viewed as the most difficult option to implement. Facility licensees perceived that Option 1, under which NRC examiners independently administer the operating tests, would be the least demanding on facility licensees and the easiest to implement and would retain the greatest level of ind?pendence in the licensing decisions. NEl indicated that all of the facility licensees surveyed were satisfied with the GFE and thought it should be continued in the current manner.

' The staff believes that the industry's response to GL 95-06, which described the pilot process and solicited volunteers to participate in a six-month 15 p weq +=ta - - - - - , = - - -

p,mu 't ml-

k >

Y ..

t V '

f program to evaluate its feasibility, was generally favorable. Of the 35 facilit> licensees that had requested examinations during the pilot period, 22

.)' agreed to prepare their own examinations in accordance with GL 95-06. Most of the facility licensees th.t declined to participate cited as their reason their inability to reschedule the required resources on such short notice because of the restrictions on the use of personnel who had been involved in training the Itcense applicants.

In September 1995, shortly before beginning the pilot examinations, the staff

! convened a public workshop to discuss the NRC's expectations regarding

! industry participation in the pilot program for the operator licensing examination. The workshop also gave facility representatives an opportunity to ask questions _of the staff and to submit written coments regarding the pilot examination criteria outlined in GL 95-06. Most of the questions and commnts related to the restrictions on which personnel the facility licensee could use to prepare the examination and the requirement to document the history of the examination materials. Some facility licensees questioned the amount of reference materials that the NRC would need to review and prepare for the examination. The staff challenged the pilot examination participants who expected to have problems satisfying the personnel restrictions to suggest alternative ways to achieve the same objective. The staff comitted to review the alternatives on a case-by-case basis and to propose a final position in Revision 8 of NUREG-1021, which would be published for coment near the end of the pilot program. Tne staff also made it clear from the beginning that facility licensees could use contractors to develop the examinations, feedback from NRC oxaminers who participated in the pilot examinations and industry representatives who attended a public progress meeting sponsored by NE! in January 1996 confirmed that the personnel restrictions were the most troublesome aspect of the pilot process. Many facilities, particularly those with smaller training staffs, contended that the quality of the examinations was suffering because most of their best qualified personnel were involved in teaching and were ir, eligible to prepare the examinations. Those attending the meeting generally agreed that the industry and the NRC had learned a great deal from the pilot program, that most of the issues could likely be resolved through training and improved comunication, and that the pilot process should be fully implemented.

In Febrnry 1996, the staff issued draft Revision 8 of NUREG-1021 that incorporated both the )tiot process outlined in GL 95-06 and preliminary lessons learned from tic early pilot exami.iations. .The draft revision was sent to the PDR and placed on the NRC's home page on the World Wide Web, and a 3

federal Register notice requesting industry and public comments was published on February 22, 1996. NEl submitted coments and recommendations on behalf of the nuclear industry, and two facility licensees (VEPCO and Cosmonwealth Edison Company) provided additional coments. On April 18, 1996, the staff conducted a public meeting with NEI and other industry representatives to review their coaments and recomendations regarding the draft revision 'of NUREG-1021. The meeting also provided an opportunity to discuss additional issues and proposed changes that had emerged following the issuance of draft Revision 8 of NUREG-1021 for public coment. The significant comments and

.[ staff responses are as follows:

16 1

we-~=- - - .% __

. - . _ - . . -= -

x 4'

. t y n:

All three commenters suggested that the personnel rutrictions placed on

{

$ 0 individuals involved in writing the examinations should parallel the personnel restrictions of the requalification program (i.e., the examination authors must stop teaching when examination development begins). The industry believes that those restrictions, when combined with the examination development and review criteria, would provide j adequate protection against th? unconscious blasing of an examination by an individual.

It is difficult to argue that the instructnrs would not be the best qualified individuals to preps.re the examinations. Many NRC examiners also cgreed that the requalification program restrictions would be LT adequate. However, the staff concluded that independence and conflict of interest were of sufficient concern to require personnel-restrictions very similar to those outlined for the pilot process. The staff revised the wording in the draft NUREG to clarify and quantify the amount of involvement that would disqualify an employee from participating in various aspects of examination development. For example, the personnel who prepare the examination outline, including the selection of knowledge and abilities to be evaluated on the written examination and the JPMs and simulator scenarios to be performed during the operating test, shall not have had any direct involvement in training tie license applicants. Furthermore, only one person with limited involvement

,~

(i.e., who taught more than 40 hours4.62963e-4 days 0.0111 hours 6.613757e-5 weeks 1.522e-5 months but who was responsible for no more than 15 percent of the scheduled classroom instruction) may participate in developing the written examination questions, and no one may develop questions for the topics they taught. Facility licensees may still propose alternative approaches for consideration on a case-by-case basis.

o All three commenters questioned the need for facility licensees to

! document the history of every test item proposed for use on a licensing examination, even if it was drawn from another facility licensee's examination bank.

The staff agreed that the wording in the draft NUREG could have resulted in misunderstandings of what was expected. The staff still believes that it is important to monitor the predictability of the examinations but it lacks the data and the resou ces to independently identify the

+ source of every test item. This information is readily available to the facility licensee. The staff ha clarified the guidance in NUREG-1021

so-that test items that a facility licensee obtains from another bank and deposits into its own bank may be treated as " Lank" items, provided the test items have an equal chance of being selected for use on the examination. Items from another bank may be treated as new items if they have not been made available for review and study by the license applicants and there-is no basis (e.g., historical precedent or reciprocal arrangements with the other facility licensee) for the

.asplicants to predict their use on the examinati on.. The staff believes t1at this revision will somewhat lesten the burden on' facility j i

17 l

< -.y mW'da w-e. -.M-Weseg

k. A i ,

f W m licensees, while enabling the NRC to maintain an acceptable level of '

oversight.

) -

o All three'commenters saggested that NUREG-1021 relax the restrictions on !

the-duplication of test items from the appilcants' audit examination-(a- .

practice test given by the facility licensee before the NRC examination) i _

if the audit examination was developed independently of the license i examination.

I Under the tradittorial examination method, the staff does not ha$e access.

to the facility licensee's audit examination, so any duplication that 1 occurs between the audit examination and.the licensing examination is

} unmonitored and coincidental. The staff agreed that the proposal was i

i consistent with current practice and changed the NUREG to. allow up to five questions to be duplicated at random if the two examinations are written-independently (i.e., no interaction between the examination authors) and the facility licensee identifies the duplicates. Licenstrg examinations that are not developed independently shall not repeat any-questions directly from the audit examination.

o All three commenters recommended that NUREG-1021 accommodate facility i

1 licensees that have " closed" examination banks (banks not available to applicants for study) by relaxing the limit on the number of test items that can be drawn directly from the bank.

The staff acknowledged that this is a valid issue, but because this issue is so complex, the staff does not intend at this point to establish additionel guidelines regarding the maintenance of examination banks. As noted earlier, the staff intends to treat all banks as if they are "open" at this time, o All three commenters objected to the suggestion in draft NUREG-1021 that facility licensees consider providing an explanation as to why the answer to every written (multiple-choice) question is correct and each of the distractors is plausible but incorrect. The industry thought

! this was a significant administrative burden that would, in practice, become 4 requirement.

f Feedback frc.a the NRC chief examiners indicated that some facility licensees adopted this practice on their own during the pilot examinations; it proved very beneficial and increased efficiency during the examination reviews. This prompted the staff to include it-as-an-

~

o)tional activity in draft NUREG-1021. The staff still believes that-

-tie practice has merit but has emphasized in the final NUREG that it is optional.

o Two commenters recons, ended that facility licensees be perinitted to use-

[ site-specific task lists in place of the NRC's K/A catalogs when j developing the examination outline. The site-specific-lists are used to develop the operator requalification examinations and should also.be-ll. acceptable for developing content-valid initial licensing examinations.

1

i '

18

't il

+

"7 * ' ' " ' ' ' -rey v *e- ,

4 X y, .

y e

44F -

W' c;pp -

Since the mid-1980s, the staff h'as used the'NRC's generic K/A catalogs:

as the basis for ensuring the content validity of the initial licensing h .

- examinations.. The catalogs are based upon an analysis of the licensed il operator's job that was performed by the Institute of Nuclear Power

- Operations.- Many facility licensees have conducted their own' job task analyses and developed plant-specific o>erator task itsts for use in their SAT-based training programs. Altiough the staff does not question :

fj that the site-specific task lists would generally be acceptable for ensuring the content validity of the examinations, it believes that- .

i wholesale substitution of the catalogs could cause a decline in s- examination consistency. The staff considers the proposal appropriate in limited circumstances on a question-by-question basis; therefore, the- ;

] guidance in NUREG-1021 has been revised to permit substitutions and i j additions of sp._cific knowledge and ability requirements on a case-by- !

j case basis.

-o Two commenters disagreed with a-paragra>h in Appendix A of the draft NUREG that implied that any question tacen from an open bank would test at the basic memory level. They argued that questions prepared at a l higher cognitive level would not-be reduced to the basic memory level if the question bank is large. They strongly recommended that the subject 4 _ paragraph in Appendix A be deleted.

- Appendix A provides an overview of examination preparation concepts and discusses the basis for some of the NRC's examination criteria,.

including the limits on the use of examination banks. The subject paragraph is considered integral to the discussion and was not deleted.

However, the staff has edited the text to clarify that questions could decrease to the simple recognition level if the item bank is'small and c available for the eraminees to study.

4 The staff also clarified one of the criteria in draft NUREG-1021 that, for purposes of ex3mination quality and consistency, required at least' half of the questic as m the written examination to be written at the comprehension or annysis level. The staff never intended to imply any question drawn directly from a bank would be counted at anythingth other than its face value. A question written at the comprehension or analysis level would count as such, even though the cognitive level at which it tests would decline with use and exposure.

. , . o Two commente*s recommended that the facility licensees be allowed more 4I :

than five days, if necessary, to review and grade the written examinations. They asserted that relaxing the requirement would only

% delay.the results to the facility licensee and the applicant and that

+ the delay would not adversely affect the NRC.

The staff had traditionally allowed the: facility licensee.five days to

. comment on the written examination, so it-decided to retain that time *

~

period-for the pilot examination program. This was one of the examination criteria on which the chief. examiners had been asked to

, comment, and no comments were made. However, since the proposed change ii-19 i

&~-- re --

7

. ~ , - .;. -, ,+ ,..m--....m. -

- , .-,n . - - . - . - . - ~ _ - - . , ~ , . . . . - . -, . . . . - .

b ,

1

< NI would not affect the examination outcome, the staff decided to increase b the flexibility in the final NUREG-1021 by noting that the facility licensee should submit the graded examinations within five days, if practical.

i o One comenter indicated that NUREG-1021 does not appear to accomodate utilities that do not want to prepare their own licensing examinations and suggested that utilities be allowed to retain that option.

Because of the unpredictable demand for examinations and the scheduling difficult;es that may result from a long term voluntary program, SECY-96-123 requested that the Comission approve the staff's pursuit of rulemaking to require that licensees of power reactor facilities prepare the operator licensl..g examinations in accordance with NUREG-1021. it.a Comission has delayed making a decision on industry-wi'e implementation l of the revised process until it can review the detailed rulemaking plan

! (Attachment 1) and the additional issues discussed herein.

1 I o One comenter recommended that the facility licensees be held responsible for making examination changes "as agreed upon with the NRC" rather than "as directed by the NRC." lhe comenter added that this alteration would help ensure that the suggested changes are accurate.

Because the staff fully supports the goal of the facility licensee to ensure the accuracy of the examinations, it revised NUREG-1021 as recommended but stipulated that the NRC would retain th3 final authority to approve the examination.

o One comenter requested that the NRC clarify which facility staff members would be allowed to observe the operating tests in the simulator. NUREG-1021 currently restricts personnel other than the simulator operator from watching the operating test without the approval of the NRC chief examiner. The commenter argued that facility managers and others deemed necessary by the facility should be allowed access to the examination, provided the simulation facility can accomodate them and there is no adverse effect on the applicants.

The staff concluded that this was a reasonable request and revised NUREG-1021 accordingly, facility management and other personnel deemed necessary by the facility licensee will generally be allowed access to the examination (under security agreements, as appropriate), provided the simulation facility can accomodate them and there is no adverse effect on the applicants.

o One comenter requested that the NRC clarify the meaning of "significantly modify." The facility licensee had been told that the intent of the question had to be changed, which conflicted with the definition that had been published in GL 95-06 and drtft NUREG-1021.

20 s

" "" - ~ .-. ., _ - _ _ _ , _ _

a

) ,

The staff concluded that this was a reasonable request and revised i

1 M' *fi NUREG-1021 to indicate that the intent of the question does not have to change to qualify as a significant modification.

in summary, the staff believes that the industry generally supports the 3 proposal to shift responsiblitty for writing the initial licensing examinations to the facility licensees. Some facility licensees actively lobbled for the change on the basis that it would save them .,oney. Many facility licensees and NEl view the proposed changes as a good first step at increasing the industry's involvement in the examination process and shifting the NRC's role from direct involvement to oversight. Nevertheless, most facility licensees would probably agree that the administrative requirements that were outlined in GL 95-06, refined during the pilot program, and incorporated in Revision 8 of NUREG-1021 have increased the burden of preparing the examinations.

The issue of industry burden has been a key factor throughout the pilot program. The staff purposely limited the scope of the original process changes to those measures that would help ensure the integrity, consistency, quality, and security of the licensing examinations. The staff was comitted to maintaining an effective and efficient licensing program and realized that the revised process, which would have relied on voluntary industry support, would never succeed if it was unnecessarily burdensome on the facility licensees. In keeping with its original commitment, the staff believes that it has taken reasonable measures to address the industry's concerns and recommendations without sacrificing the effectiveness of the examination process.

Pilot Enhancements and Justification 1 To assess the effectiver. ass of the new examination process, NRR asked the NRC examiners who were involved with the pilot examinations to respond to a questionnaire regarding various aspects of the process. The first and most significant problem observed and reported by the examiners was that the quality and technical accuracy of the examinations submitted by several fa':llity licensees fell short of the NRC's expectations. Moreover, some of the early pilot examination results (i.e., more question deletions and answer key changes than expected) reflected the need for greater emphasis in that area.

Consequently, when NRR prepared draft cvision 8 of NUREG-In21, it incorporated a number of clarifications and changes that improved the framework established in GL 95-06. In January 1996, the staff issued a preliminary draft of NUREG-1021 to the regions for review and comment before soliciting comments ' rom the public and the industry the fo? lowing month. The draft revision that was issued for public and industry comment contained the following process improvements:

o The existing guidance (Revision 7 of NUREG-1021. Revision 5 of NUREG/BR-0122, and GL 95-06) did not specify what percentage of the questions on the written examination should test the applicants at the comprehension 21 l

I

e .

/

KW' and analysis levels of knowledge. Many examiners had expressed concern that the lack of guidance in this area would lead to inconsistency and a decrease.in the level of examination difficulty. Facility-licensees

) questioned whether examiners had unwritten guidance that required then 4

to make the examination more difficult by revising or replacing simple memory questions.

}

As noted earlier, NRR haa no desire to increase the average difficulty l of the examinations, but it also did not want-the level of difficulty to diminish. Therefore, to better define the appropriate level of knowledge for the new examination process, the staff reviewed the written examination audits performed in the previous two years to determine an average distribution of questions by knowledge level. The staff concluded that approximately half of the questions on the examinations reviewed were written at the comprehension or analysis level and that the remaining questions tested simple memory. On the

)l basis of that information, the staff established a criterion in draft

NUREG-1021 that at least half of the examination questions be writt6n at the comprehension or analysis level, o The supplemental guidance that NRR provided to the regions concurrent with GL 95-06 included interim QA checklists to assist the NRC examiners in reviewing and identifying deficiencies in the draft examinations.

The checklists were edited and incorporated in the preliminary draft of NUREG-1021 and edited again in response to regional comments. The draft NUREG-1021 that was issued for public comment included QA checklists for the examination outline, the written examination, both the simulatcr and walk-through portions of the operating test, and the written examination grading.

'I o The existing guidance (Revision 7 of NUREG-1021 and GL 95-06) did not specify a limit on the number of JPMs that could be repeated in walk-through tests administered on successive days during the same initial licensing examination visit or from one visit to the-next. Many examiners had expressed concern that the lack of guidance in this area could compromise examination integrity and lead to inconsistency. Other examiners argued that no repetition should be permitted because it would increase the predictability of the test. Facility Itcensees would be I tempted to repeat as many JPMs as possible in an effort to save resources, i

r To strike a balante, the staff concluded that a 30-percent limit would 1j be appropriate. This figure was consistent with the written examination limit of 25-percent duplication of questions from examinations, quizzes, or tests administered to the license applicants or from the past two NRC licensing examinations at the facility.

After issuing draft NUREG-1021 for public and industry comment, the staff I identified a number of items that merited clarification and improvement. In 1 addition to reviewing the industry's comments and recommendations-(as j previously noted), the staff discussed all of the following changes with NE!

i a

22

- W"" "VW.M,b g _

,r-,--

) ..

[ ,

M, and other industry representatives dur ig a )ublic meeting held in the NRC's offices on April 18, 1996. Except as noted >elow, the industry representatives did not object to the additional changes, o To limit the number of technical inaccuracles, a problem that affected several of the pilot written examinations, the staff revised NUREG-1021 to requit'e the NRC chief examiner or another examiner to independently review ant' verify the technical accuracy of a sample of the written examinatio,' questions. The number of questions verified would depend upon the e>oected quality of the examination and the time available before the scheduled review with the facility licensee.

As a final check of the technical accuracy of the written examination, the staff revised NUREG-1021 to encourage the facility licensee to administer the NRC-approved examination (under security agreements) to Y' one or more licensed personnel who were previously uninvolved in j developing the examination. Although feedback from the NRC chief i

examiners indicated that some facility licensees had taken it upc, themselves to validate their examinations in this manner during the pilot program, the facility representatives at the meeting objected to making it a requirement for all f acility licensees. Because the staff lacked sufficient informat'n to quantify the value added by this measure, it chose to recommend it as a good practice, o Late in the pilot program, an NRC chief examiner who was reviewing a draft examination detected that the facility licensee had used a written examination outline that was identical to one that a sister facility had prepared several waeks earlier. The staff had not anticipated that such a problem might occur but immediately realized that this was a significant loophole in the process. Allowing facility licensees to reuse the same examination outline would definitely improve efficiency, but it would also jeopardize the integrity of the examination process by increasing the predictability of the examinations.

To address this problem, the staff included in NUREG-1021 a caution for examination authors and reviewers to ensure that the outline used during successive audit and licensing examinations does not become repetitive and predictable. If a facility licensee proposes to use an outline that was previously used at another facility, it shall identify the source of the outline and explain what effect its reuse is expected to have on examiitation integrity.

O NUREG-1021 has for some time included checklists for use by NRC examiners in verifying that their simulator scenarlos contained the required number of transients and ensuring that the scenarios would enable the examiners to evaluate the applicants on every required competency. There were no specific instructions for filling out the checklists, nor were they required to be retained in the examination file. Concerns regarding the quality and consistency of the operating tests prompted the staff to incorporate instructions for completing the 23

t . . .

f . . ,

9 transient and competency checklists and a requirement for the NRC regionul offices to retain the checklists in the examination filss, o NUREG-1021 has always endorsed the goal of having the licensing e

examinations ready to administer approximately a week before they are scheduled to be given. Furthermore, the staff has always encouraged the regions to keep NRR informed if significant problems are encountered with an examination. The pilot program illustrated and magnified the importance of both concepts. Therefore, the staff clarified the j guidance in NUREG-1021 to limit the number of last-minute changes and to require the regional office to consult NRR if it appears that the examination cannot be made to conform to the examination standards at least five working days before the scheduled examination date, l

i Furthermore, the exa.inations should be postponed if problems are identified at the last minute, o in addition to implementing the pilot examination process, the staff proposes using Revision 8 of NUREG-1021 as a vehicle for activating Revision 1 of NUREG-ll22 and NUREG-ll23, the NRC's K/A catalogs, which were published in August 1995. The catalogs were revised to make them more current and reorganized to make them more consistent with each other and easier to use. As part of the catalog revisions, the system-generic K/As in both catalogs were combined with the plant-wide generic K/As, and new vendor-specific emergency and abnormal plant evolutions were added to the pressurized water reactor catalog (NUREG-il22). These changes would have altered the balance of the written examinations unless the sample plans in NUREG-1021 were also revised to compensate for the changes. Because of an oversight, the staff neglected to enter the char.ges before issuing the draft NVREG-1021 for public and industry comment.

1 o When NRR reviewed the regional office operator licensing programs during February and March 1996, it discovered that some of the examiners had misinterpreted the intent of NUREG-1021 regarding the design of the operating tests for SR0 applicants limited to fuel handling. The staff discussed the issue with the regions and agreed that strict compliance with the existing guidance would overtest those license applicants.

Therefore, the guidance in NUREG-1021 was changed to more closely align the scope of the operating test with the range of responsibilities for SRO fuel handlers, o Whenever a question is deleted from the 100-point written examination, it results in fractional grades. When the grade is above 79.5 percent, it raises a question regarding the staff's policy on rounding-off because it means the difference between receiving a license or a denial.

To address this problem, the staff has revised NUREG-1021 to indicate that the passing grade on the written examination is 80.00 percent.

o Although the pass rates of the pilot examinations were comparable with pass rates for NRC-written examinations, the rate at which applicants requested that the NRC informally review their examination failures 24

j ,

'h increased unexpectedly during the pilot program. ihese requests not only-placed a significant burden on the staff to resolve the applicants' j issues but also raised concerns regarding the effect that any additional question deletions and answer changes might have on the other licensing decisions that had already been made.

1 Consequently, the guidance in NUREG-1021 was revised to afford the NRR staff greater discretion in determining the method for resolving appiteant appeals (i.e., every appeal does not necessarily have to be reviewed by a panel of examiners as long s objectivity and fairness are maintained) and to delay the issuance of licenses to applicants who passed the written examination with insufficient margin to guarantee that the licensing decision will be sustained if additional questions are deleted or changed upon appeal, furthermore, to ministre the number of appeals, the staff revised NUREG-1021 to encourage facility licensees to solicit and address concerns from individual license applicants during the process of grading the written examinations, i

o Problems encountered while reviewing and resolving the operating test appeals that were received during the pilot program prompted the staff to reevaluate the documentation and grading requirements in NUREG-1021.

As a result, the staff has revised NUREG-1021 to permit NRC examiners to downgrade more than two simulator test rating factors for the same error, provided the error caused serious safety consequences for the plant. The staff also increased the documentation requirements for the simulator test so that the NRC examiners would have to describe any error that prompted them to downgrade the applicant on any rating factor.

During the meeting of April 18, 1996, the staff also discussed the possibility of permitting NRC examiners to recommend a failure on the simulator test even if the appitcant did not fail the test on the basis of the competency scores. The industry representatives did not believe

'that such a change would be necessary if there were no limit on the number of competencies that the examiner could downgrade for a single, serious error; however, they did not object to the change in principle.

Therefore, the staff has included in NUREG-1021 a provision for NRC examiners to recommend such a failure if the applicant committed an error with serious safety consequences for the plant or the public. To limit its use, the region must obtain written concurrence from the Chief, HOLB, before completing .a licensing action.

The following issues were raised by NRC examiners on the pilot examination feedback surveys but did not result in specific changes to the examination methodology described in NUREG-1021.

o To minimize NRC resource requirements, the staff had encouraged the regions to review and prepare for the pilot examinations without conducting a preparatory visit to the site. However, feedback from those examiners who did not conduct such a visit indicated that it would have facilitated the review and resulted in fewer problems with the 25 P

~

" = = - ' _ , , , , . _

{ ..

[

~

w i f

O examination.. They recomended that a preparatory visit be required for every examination. Although hRK has not changed the guidance in i NUREr,-1021 (i.e., the. preparatory visit is optional at the discretion of regional management), it has acknowledged the value of conducting at f .

least part of the examination review and validation at the facility.

) The average NRC resource expenditures during the pilot examinations indicated that the trips an be made within the original budget model.

o Several NRC examiners recormended that the required due dates for the j examination materials from the facility licensees be advanced so that the examiners have more flexibility in scheduling their time for the j

1 examination review. The staff did not change the guidance in NUREG-1021 1 because it already permits the regional offices to work with the facility licensees in determining the appropriate due dates and because a generic change would increase the scheduling burden on all facility licensees. It would not be appropriate for an examiner to demand the examination 60 days in advance because it was required by NUREG-1021, only to have the examination sit untouched in the office for 30 days because the examiner was involved wlth another project, o When the staff developed the pilot examination process, there was some debate as to whether the facility personnel who administered the written examination should be permitted to answer the applicants' questions regarding the intent of specific questions on the examination. The pilot guidance allowed the facility to answer the applicants' questions with caution but required every question and response to be documented for subsequent review by the NRC. The policy was evaluated on the pilot examination questionnaire that every NRC chief examiner was reouired to complete. A minority of the examiners recomended that the policy be changed to prohibit the applicants from asking questions during the written examination, as is the policy on the GFE. However, most of the examiners concurred with the original pilot policy, and none of the comments indicated that the policy had created any problems with examination integrity. Therefore, the staff saw no reason to revise the guidance in NUREG-1021.

o Some examiners noted that the pilot program personnel restrictions had hampered the facility licensees' ability to develop the examinations because they could not assign their best people to the task. This issue ,

was also addressed in the industry comments, as previously discussed.

Examination Results Since Issuance of SECY-96-lu The staff completed the last of the pilot examinations on April 5,1996, and resumed conducting the licensing examinations using the traditional development process. After April 12, 1996, when the Executive Director for Operations informed the Comission of the staff's intent to continue the pilot process beyond the period previously discussed with the Comission, the staff j proceeded to contact those facility licensees whose examination requests had not yet been assigned to a contractor to determine if they were interested in developing the examinations using the pilot methodology. The first of those 26 1

)

" - - , ..%i.-- _

y,, - ,_

f l s VM examinations was administered on August 19, 1996, and the results were not available in time to report them here.

h Since SECY-96-123 was prepared, the staff has used the traditional examination methodology to prepare licensing examinations for applicants at 14 power reactor facilities. The results of the completed 6xaminations are summarized below; four of the examinations were completed late in August, and the results were not available at the time this paper was prepared. 1he licensing examination results for fiscal year 1995 and the 22 pilot examinations conducted between October 1, 1995 and April 5, 1905, are provided for comparison, j- Pass Rates' R0 R0 R0 SRO SR0 SRO Examinations Written Operating Total Written Operating Total 39/40 39/39 39/40 45/48' 43/48 41/48' Since SECY-96-123 (98%) (100%) (98%) (94%) -(90%) (85%)

Pilot 49/54 50/54 45/54 86/92 87/91' 83/92' Examinations (91%) (93%) (83%) (93%) (96%) (90%)

Fiscal Year 1995 94% 98% 92% 95% 95% 92%

1. Each box indicates the number of applicants who passed that part of the examination over the number of applicants who took it.

2. One of the SRO applicants who failed the written examination has filed an appeal that was not yet resolved at the time this paper was prepared.

3. Two applicants who were reported as preliminary failures in SECY-96-123 filed appeals that resulted in the failures being overturned.

Suminary The staff believes that the revised examination process can be implemented at all power reactor facilities with the same level of direct NRC resources as is currently allotted to the operator licensing program.; Full implementation of the revised process will enable the staff to eliminate the use of operator licensing .ontractors (witn the exception of those whc develop and grade the GFE) that have hi:torically cost the NRC (and the indu stry) approximately

$3 million to $4 million per year. The staff expects that the change could result in a net resource savings to the facility licen sees over time because they will be able to prepare the examinations more efficiently than the NRC or its contractors.

The overall results of the pilot program indicated that the examinations prepared by facility licensees, subject to review, revision where appropriate, and approval by NRC staff examiners, were generally of the same caliber as examinations prepared by NRC or contract examiners. T 1e pilot examinations also appeared to be equally effective at identifying applicants who had not 27 b cM.%d re s=rce, %%4r w e W m,c k. Jor' $ VW1 and M8 5 Wres3e. h pd & Wu_

GLwa.m sner$ av nd t d e. .

~~.- ---n-

1 . . . , ,

I M~'

masterea the job requirements well enough to become licensed R0s or SR0s.

furthermore, the pass rates for R0s and SR0s on the written examinations and

\ operating tests compared favorably with the pass rates for examinations prepared by the NRC or its contractors.

However, despite the mitigating actions previously noted, the revised examination process is much more dependent upon the abilities of the NRC chief examiners to identify weak examinations and work with the facility licensee to upgrade the examination. Furthermore, it is possible that the pressures to conduct examinations on schedule and the reduced involvement by the NRC in the 8 examination development process could lead to examinations that reflect the i biases of the facility licensees and that discriminate at a lower level than is currently the norm. Moreover, it is reasonable to expect that industry-wide implementation of tht. revised process will increase the chance of future examination compromises as a result of the larger number of individuals who would require access to the examinatiuns and the additional pressures on facility employees to ensure the success of the applicants.

Although a number of the examinations submitted by the facility licensees did not meet NRC ttandards for level of knowledge or difficulty, the NRC chief examiners effectively worked with the facility licensees to revise the examinations so that they would discriminate at the appropriate level. To put this situation into perspective, it has not been too unusual for contractor-developed examinations to fall short of the NRC's requirements and expectations. The chief examiner often directs the cor.:,ract examiner to make numerous corrections and changes before reviewing the examination with the facility licensee, only to have the facility reviewer identify a host of additional errors and inconsistencies.

The staff has advised facility licensees that the NRC expects the technical quality of examinations to remain high and the number of post-examination changes to be low. Similarly, regional examiners and managers have been instructed not to approve or administer any examination that does not adhere to NRC standards for content, format, quality, level of knowledge, and level of difficulty. The QA checklists in NUREG-1021 have been modified to maintain examination consistency and to facilitate the efforts of NRC examiners to detect deficient examination materials. The use of any facility-prepared examination that is found to be unacceptable will be delayed or cancelled, if necessary, until the facility licensee and the NRC can upgrade or replace the examination.

The supplemental pilot examination guidance that the staff provided to the regional offices (and placed in the PDR) in conjunction with GL 95-06 ofrected the regions to document in the examination report any significant problems

-Jacepntered during the examination development, administration, and grading.

O02* + This'respirement- has also been included in Revhion 8 of NUREG-1021. The regions are expected to address any serious deficier.cies in the eyaminations submitted by the facility licensees and any security concerns during the entire examination process. If appropriate, the region will request the facility licensee to describe the actions it will take to improve future performance, if the facility licensee is unable or unwilling to prepare 28

~- _ _ ._

, .. n d ; .y .;,(,, .-

- appropriate licensing examinations, the NRC has-the res onsibility and '

authority to develop the examinations in the-traditiona manner.-

L Overall, the staff does'not' expect that the pro >osed revisions to the ourator d' licensing program will-significantly increase tie risk to the life and sealth

- of any individual members of the pub 1'ic of that they will be a significant i

addition to other societal risks. The staff believes that the licensing 3 decisions that were made on the basis of the pliot examinations were as valid O as those made using the traditional examination process. Furthermore, the i O changes that the staff has made in NUREG-1021 as a result of lessons learned 7 d..Inythepilotprogramandindustrycommentssubmittedinresponsetothe staff s solicitation (which were previously discussed in detail) should only laprove the e'fectiveness of the examinations if and when the revised process is-implemented 9n_an industry + 1de basis. 3

- The NRC will w d inue to set the standard for operator performance by ensuring that the facilitv licensees maintain appropriately high standards for.

examination development. . The staff will remain actively involved in the operator licensing process by reviewing and approving every written -

4 examination and operating test before it is administered. Unacceptable examinations will be revised and delayed, if necessary, until the resources are available to upgrade the exrmations. Furthermore, the fact that NRC examiners will continue to independently administer the operat s will allay concerns regarding b;as and conflicts of interest on t rtion of quiring K

the licensing eramination having the most operational validit the most subjective judgements.

The safety consequences of occasionally administering a substandard examination are probably minimal. By the time the-applicant takes the NRC-licensing examination, he or she will have completed a comprehensive, ,

systematically constructed and implemented training program designed to maximize the itkelihood of success on-the examination. The license training programs typically include numerous written examinations and performance demonstrations that culminate in certification by the facility licensee that e the applicant has successfully completed the facility licensee's requirements

- to be licensed as an RO or an SRO. The effectiveness with which the facility licensees screen out those trainees that have not mastered the job requirements has been evident.in the high. pass rates on the Itcensing.

examinations during the past several years, in most cases, the NRC- '

4 . examination is simply a confirmation of the facility licensee's decision to subalt-the-application.

Historically, even those applicants who fail.the NRC written examination-generally score above 75 percent. The average grade for the 11 applicants who failed a pilot examination was 76.3 percent. This statistic indicates that.

- the vast majority of ap)1icants, including most of those who fall the-licensing examination,-. Save attained an adequate level of knowledge so as not to pose a significant threat to reactor safety. Furthermore, any applicant

_who does become-licensed on-the basis of an examination that discriminated at a lower than normal-level would be backed up by other crew members who could

- prevent or remedy any operating errors. Although there is=nothing to prevent

, a factitty licensee from forming an operating crew entirely from newly 29 L l l

)-

- . . - ____-,--4 -

.., ,s<- , , ,, , .-. . ,_m., _,,.,.m. .m .~.,m__,,, ,, m, , - y. ,,-,,..-xm.g._ ,,.m-,,,.,

n. . ,

q

,.p ..

fj H

[

_4,-

$(hg.-~p licensed operators,'most-facility licensees-integrate their new operators with

. existing crews'who. possess significant operating experience.

.4 h

i

,4 ,

l' A

-c t h

).

.4 f I

i e

30 !

s

.,v: ,

-- _ M -"C

ML20216F949

Contents

Text

SUBJECT:

SUBJECT:

Attachment:

Attachment:

SUBJECT:

Enclosures:

Navigation menu

ML20216F949

Text

SUBJECT:

SUBJECT:

Attachment:

Attachment:

SUBJECT:

Enclosures:

Navigation menu

Search