ML20134C900

Recommends That Commission Approve Development of Rule as Described in Encl Rulemaking Plan

ML20134C900
Person / Time
Issue date:	09/25/1996
From:	Taylor J NRC OFFICE OF THE EXECUTIVE DIRECTOR FOR OPERATIONS (EDO)
To:
References
SECY-96-206, SECY-96-206-R, NUDOCS 9610100219
Download: ML20134C900 (47)

Text

__ _ ________ ______ - -

" oooo..... .............. i RELEASED TO THE PDR  !

efguo%n  ; .

i -l

• ululu seu  ;

p .e e data initds

...e s e s e e e >e e e ee es e >

I POLICY ISSUE (Notation Vote)

September 25, 1996 SECY-96-206 FOR: The Commissioners FROM: James M. Taylor. Executive Director for Operations

SUBJECT:

RULEMAKING PLAN FOR AMENDMENTS TO 10 CFR PART 55 TO CHANGE LICENSED OPERATOR EXAMINATION REQUIREMENTS PURPOSE:

To provide additional information regarding the initial operator licensing examination pilot process as directed in a Staff Requirements Memorandum (SRM) of Jul 23. 1996 (M960618A), and to request the Commission's approval of the rulema ing plan to amend 10 CFR Part 55 to require power reactor facility licensees to prepare the entire initial examination for reactor operators and senior reactor operators and to proctor the written portion of the examination.

BACKGROUND:

On March 24, 1995. SECY-95-075. " Proposed Changes to the NRC Operator Licensing Program." informed the Commission of the staff's intent to revise the operator licensing program to allow greater participation by facility licensees and to eliminate contractor assistance in this area. This action allows the staff to eliminate between $3 million and $4 million in contractor support for examination preparation and administration. The FY 1997 and FY 1998 budget request is consistent with this proposal and reflects the elimination of contract support for initial examinations. On April 18, 1995, the Comission approved the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full implementation of the ,

changes. On August 15, 1995, the staff issued Generic Letter (GL) 95-06. ({p '

" Changes in the Operator Licensing Program." outlining the revised examinatio '

development process and soliciting volunteers to participate in pilot g examinations to evaluate and refine the methodology. O CONTACT:

NOTE: TO BE MADE PUBLICLY AVAILABLE WHEN THE Stuart Richards. NRR FINAL sRM IS MADE AVAILABLE (301) 415-1031 i

g y m' a -

"40128 '

b,l0'0 @ l

~

l 9 k g p,4 cc-

The Commissioner: Between October 1,1995, and April 5.1996, the staff reviewed and approved 22 operator licensing examinations that had been prepared by facility licensees in accordance with the current examination development guidance (NUREG-1021.

Rev. 7. Supplement 1) as supplemented by the pilot examination guidance contained in GL 95-06. These examinations were then used to test 146 reactor operator and senior reactor operator applicants. Facility licensees from each NRC Region participated in the voluntary pilot program, including operating reactors' built by each power reactor vendor.

The staff described the results of the pilot examinations in SECY-96-123.

" Proposed Changes to the NRC Operator Licensing Program." dated June 10. 1996, and ariefed the Commission during a public meeting on June 18, 1996. On the basis of the results of the pilot program, the staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed. This rulemaking would j require all power reactor facility licensees to pre)are the operator licensing examinations based on the guidance contained in NUREG-1021. " Operator Licensing Examination Standards for Power Reactors." which has been revised to accommodate the new process. In an SRM dated July 23, 1996, the Commission i authorized the staff to continue the pilot process through July 1997 and l directed the staff to develop a detailed rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55. The SRM also directed the staff to specifically address several issues regarding the pilot examination l program and the proposed changes in the examination process.

DISCUSSION:

Section 107 of the Atomic Energy Act of 1954, as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55 to pass an examination satisfying the basic content requirements specified in the regulation. Part 55 does not state who must prepare, administer, or grade the examination; however, the NRC has traditionally performed those tasks itself or through the use of contract examiners.

On the basis of the results of the )ilot program as discussed in SECY-96-123.

the staff now believes that 10 CFR 3 art 55 should be revised to require power reactor facility licensees to 3repare their initial operator licensing examinations. Specifically, tie staff recommends that Subpart E. " Written Examinations and Operating Tests." of 10 CFR Part 55 be amended to require all  ;

ower reactor facility licensees to prepare the entire initial operator icensing examination and to proctor the written portion of the examination based on the guidance contained in NUREG-1021. NUREG-1021 has been revised (Revision 8) to implement the new process. It has been issued for comment, however. Revision 8 has not yet been implemented. The licensee-pre)ared examinations will be subject to review, modification, and approval )y NRC examiners before the examinations are administered. The regulation will provide that the NRC may prepare the examinations in lieu of accepting or modifying an examination prepared by the facility licensee. The changes would not apply to non-power reactor licensees.

C D I The Commissioners ,

The enclosed rulemaking plan for this proposed change has been developed in

. accordance with the' guidance in NRC Managerent Directive 6.3. "The Rulemaking Process." As directed in the SRM of July 23, 1996, the rulemaking plan addresses the potential igact of the proposed rule on facility licensees and discusses why this rule will not constitute a backfit. *

~

The second enclosure discusses the following additional items as directed by the SRM:

e The pros cons, and vulnerabilities associated with implementing the ,

proposed examination program changes on an industry-wide basis >

(including a review of the impact on licensee and NRC resources, the potential effects on reactor safety, the impact of proposed NUREG-1021 revisions, potential public yercepticns, and concerns regarding this approach raised by staff mem3ers).  !

- e A discussion of the degree of acceptance of this new approhch to operator licensing by facility licensees (including a discussion of industry comments received on draft NUREG-1021. Revision 8, and the staff's response to those comments).

i o An explanation of enhancements to the new licensing examination process resulting from the review and feedback from the pilot program results (including a justification for subsequent proposed changes to the operator licensing program).

e The results of initial operator license examinations that have been given both under the pilot program and under the current program since ,

3 SECY-96-123 was prepared (including trends observed). l RECOMMENDATION:

That the Commission -- -

t 1. Acorove the development of the rule as described in the enclosed Rulemaking Plan.

! 2. Accrove the implementation of Revision 8 of NUREG-1021 on a voluntary

basis until the rulemaking is complete.

f

~ _ _ _ . . _ _ .

The Commissioners i i

C00RDINATION:

The Office of the General Counsel.has no legal objection to this initiative. .

.- l J mes M. T lor xeCutive Director t for Operations l

?

Enclosures:

l

1. Rulemaking Plan for Changes to 10 CFR Part 55 .

2. Res>onse to SRM dtd July 23, 1996  :

(1960618A)  ;

Commissioners' comments or consent should be provided directly to the Office .

of the Secretary by COB Wednesday, October 9, 1996.  !

Commission Staff Office commencs, if any, should be submitted to the j Commissioners WLT October 2, 1996, with an information copy to the Office of }

the Secretary. If the paper is of such a nature that it requires additional i review and comment, the Commissioners and the Secretariat should be apprised i of when comments may be expected.

DISTRIBUTION:

Commissioners OGC {

OCAA [

OIG [

OPA OCA ACRS {

REGIONS [

EDO  ;

SECY j I

l l

[

l.

t h

i l o

- s l

i

\

i 1

1 l

l

\

l 1 l

1 i

i l

l l

l RULEMAKING PLAN FOR CHANGES TO  ;

10 CFR PART 55 l l

1 i

i I

I 1

1 I

1 l

l i

l ENCLOSURE 1 r

l

L i

RULEMAKING PLAN ,

CHANGES TO 10 CFR 55-AMENDING INITIAL OPERATOR LICENSE EXAMINATION REQUIREMENTS Lead Office: Offic: of Nuclear Regulatory Research Staff

Contact:

H. Tovmassian, RES/RDB Concurrences: .u/

D. Mortison, RES

.e

1. lff

'Date u.. iW l b y m e-- A th</74 (s* AL d j W. Russell, NRR Date sT k mns k T. L** Lan~ <~ V.'s- G*" I thil ti ( 'H"'" }

j J. Lieberman, OE Date V TesLy(.e gl.ol 4 <si by me m.J. 4ls W II'* S l W. Olmstead, OGC Date I i

Approval: , f g J. I r // Date

i e

. RULEMAKING PLAN CHANGES TO THE OPERATOR LICENSINJ PROCESS AT POWER REACTORS (10 CFR PART 55)

1. Backaround On. March 24. 1995. SECY-95-075. " Proposed Changes to the NRC Operator Licensing Program." informed the Comission of the staff's intent to revise the operator licensing program to allow greater participation by facilit licensees and to eliminate NRC contractor assistance in this j area. his action allows staff to eliminate between 53 million and

$4 million in contractor support for examination preparation and  ;

administration. The FY 1997 and FY 1998 budget request is consistent  !

with this proposal and reflects the elimination of contract support for i initial examinations. On April 18, 1995. the Comission approved the  !

staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully consider j

experience from pilot examinations before full implementation of the ,

changes. On August 15, 1995, the staff issued Generic Letter (GL) 95-

06. " Changes in the Operator Licensing Program." which outlined the revised examination development process that would permit facility licensees to prepare the initial operator licensing examinations. GL 95-06 also solicited volunteers to participate in pilot examinations to evaluate and refine the methodology.

Between October 1. 1995, and April 5. 1996, the NRC conducted 22 initial operator licensing examinations using the new methodology. The staff described the results of the 3110t examinations in SECY-96-123.

" Proposed Chan June 10. 1996 ges andtobriefed the NRC theOperator Licensing Commission duringProgram." dated on a public meeting June 18. 1996. On the basis of the results of the pilot program. the staff recomended that the Comission a) prove the implementation of the new examination process on a voluntary ) asis until rulemaking could be completed to require all power reactor facility licensees to prepare the entire initial examination for reactor operators and senior reactor i o)erators and to proctor the written portion of the examination using l t1e guidance contained in NUREG-1021. " Operator Licensing Examination l Standards for Power Reactors." In a staff requirements memorandum dated l July 23,1996, the Comission directed the staff to develop a detailed '

rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55.

II. ProDosed Chanae i i

Revise Subpart E. " Written Examinations and Operating Tests." to include examination development requirements for facility licensees. ,

Reaulatory Issue The pilot program demonstrated that the revised examination development process can be both effective and efficient. Comments from the NRC staff and industry personnel who participated in the pilot examinations were generally favorable. The quality of the administered examinations 1

o l l  :

l (as modified by the NRC) and the performance of the individuals who took i the examinations were comparable to the quality of and the performance on examinations written by the NRC staff or its contractors. The fact that some of the draft examinations submitted by facility licensees  !

required significant rework illustrated that many facility staffs did ,

not fully understand the criteria for writing an NRC examination. Both 1 the industry and the NRC staff agree that, with training and experience.

the industry should gain proficiency in preparing the examination l materials.

l The staff believes that the new process should be made mandatory for all l power reactor facility licensees. The staff considers implementation of l the new process on a voluntary basis alone unworkable over the long-l term. With the elimination of contractor support. the staff will no longer have sufficient examiner ret. irces to write all examinations consistent with the scheduling neea of facility licensees. This resource problem is further compour~ed by the unpredictable nature of the examination workload.

The staff briefed the Committee To Review Generic Requirements (CRGR) after the pilot program was complete. In the minutes of its meeting.

CRGR agreed that, with regard to the technical and safety aspects, the proposed changes represented a reasonable and workable alternative approach. The committee members unanimously endorsed sending the proposal forward to the Executive Director for Operations (EDO) for consideration. However, the CRGR also requested OGC to consider further the procedural question relating the staff's original plan to implement the revised process by generic letter.

The staff subsequently met with OGC to resolve CRGR's concern and concluded that requiring all facility licensees to prepare the examinations, a task long performed by the NRC. would require rulemaking or the issuance of orders to each facility licensee. Therefore. the staff reconmended in SECY-96-123 that the Commission approve the staff's pursuit of rulemaking to require power reactor facility licensees to prepare the operator licensing examinations, while continuing the pilot process on a voluntary basis.

Current Rule Reouirements j Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires  !

the NRC to determine the qualifications of individuals applying for an  !

operator license, to prescribe uniform conditions for licensing such i individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55 to pass an examination satisfying the basic content requirements that are also specified in the ]

regulation. i Specifically.10 CFR 55.31(a)(3) requires the applicant to submit a written request from an authorized representative of the facility l licensee that the written examinatior and the operating test be l

administered to the applicant. Furthermore 10 CFR 55.33(a)(2) states l 2

l

. .. . - - . .= _. -. . .- - -. .- .

that the Commission will approve an initial application for a license if it finds that the applicant has passed the requisite written examination and operating test in accordance with Sections 55.41 and 55.45 or 55.43 and 55.45. These examinations and tests determine whether the applicant for an operator's license has learned to operate a facility competently and safely, and additionally in the case of a senior operator, whether the applicant has learned to direct the licensed activities of licensed operators competently and safely.

Subpart E. " Written Examinations and Operating Tests." includes specific 2

sampling requirements for the content of written examinations (10 CFR 55.41 and 55.43) and operating tests (10 CFR 55.45).

4 Although Part 55 is silent regarding who will prepare, administer, and grade the written examinations and operating tests, the NRC or its

,4 contract examiners have traditionally performed those tasks.

Preliminary Reaulatory Analysis In light of the fact that the NRC has historically performed the tasks

, of preparing and administering the initial licensing examinations. a substantial body of guidance has been published in various versions of NUREG-1021 to aid both NRC and contract examiners. The NRC expects that licensees will use NUREG-1021 to prepare these examinations, as well.

Licensees may propose deviations from specific guidance in NUREG-1021 and the NRC will review and rule on the acceptability of these

, deviations. However the NRC will not approve any deviation which would have the effect of compromising its statutory responsibility of prescribing uniform conditions for these examinations. Examples of i unacceptable deviations include, but may not be limited to, the use of essay questions in lieu of multiple choice and the administration of open book examinations. The Statements of Consideration for the proposed rule will contain a discussion of the availability and proposed use of NUREG-1021 as guidance for licensee examination preparers.

1. Draft Rule Lanouaae i The first sentence in Section 55.33(a)(2) would be revised as follows (new language is underlined):

"The applicant has passed the requisite written examination and  !

operating test in accordance with Sections 55.40. 55.41. and 55.45 l or 55.40. 55.43. and 55.45."  !

Section 55.40. " Implementation." would be added to Subpart E as follows:

i l

"(a) Power reactor facility licensees shall prepare the required I site-specific written examinations and operating tests.

i I

3

l*

I (b) Fower reactor facility licensees shall submit the written examinations and operating tests to the Commission for review and j approval.

(c) Power reactor facility licensees shall proctor the site- I specific written examinations.

(d) In lieu of requiring a power reactor facility licensee to prepare the examination and tests. and to proctor the site-s)ecific written examinations. the Commission may elect to perform i

t10se tasks.

, (e) The Commission will prepare and administer the written I

exaiainations and operating tests at non-power reactor facilities."

2. Imoact of the Rule Chance on Facility Licensees The NRC currently depends on NRC employees and contractors to prepare and administer the initial operator licensing examinations required by 10 CFR Part 55. In accordance with 10 CFR 170.12(i).

the cost of NiC time spent and any related contractual costs are billed directiy to the facility licensees that receive the examination se' vices.

Under the proposed change. each power reactor facility licensee l l will assume responsibility for preparing the site-specific initial operator licensing examinations at its facilities. thereby allowing the NRC to discontinue the use of contract examiners for i that purpose. Facility licensees will be expected to prepare and l submit pro)osed examinations (including the written examination.

l the walk-t1 rough. and the dynamic simulator tests) to the NRC based on the guidance contained in NUREG-1021. Licensees may l propose deviations from the specific guidance contained in NUREG-1021 when submitting examinations to the staff for approval. The NUREG has been edited to accommodate the revised examination l process and is the same procedure that the NRC examiners or its contractors would use if they were to develop the examinations.

The training staffs at power reactor facilities already have the i

basic knowledge, skills. and abilities necessary to evaluate i

operator performance and develop test items for the initial licensing examination. During the mid-to-late-1980s the industry's emphasis in the training area increased significantly.

All power reactor licensees established formal training programs that were based on a systems approach to training (SAT) and accredited by the National Academy for Nuclear Training. Pursuant to 10 CFR 50.120 and 55.4. SAT-based training programs must include the evaluation of the trainee's mastery of training objectives. NRC inspections of licensee requalification programs for licensed operators have also found that training staf fs

generally possess the skills needed to evaluate the trainee's knowledge.

4 i

.o e

! After the NRC reviews and approves an examination. the facility licensee will proctor and grade the written portion based on the guidance contained in NUREG-1021. The NRC staff will continue to I administer and grade the operating tests review and approve the

written examination results recommended by the facility licensee.

and make the final licensing decisions.

Feedback from the pilot examination program indicates that the average time spent by a facility licensee to prepare the written

, examination and operating tests was approximately 600 to 800 l staff-hours. A portion of that time (about 200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />) would have i been spent reviewing and assisting with the administration of NRC-developed examinations under the process now in place and should be subtracted from the total. The resulting average burden of approximately 400 to 600 staff-hours was somewhat higher than the l 400 hours0.00463 days <br />0.111 hours <br />6.613757e-4 weeks <br />1.522e-4 months <br /> that NRC contract examiners typically take to prepare an examina:lon. The extra burden is generally attributable to the facility licensees' lack of familiarity with specific NRC examination expectations and to the additional administrative requirements. such as documenting the source of the examination questions, that are requireo to maintain examination integrity.

l It is worth noting that some of the facility licensees that participated in the pilot program expended less time than is commonly used by NRC contractors to prepare the exam 1 nations.

Furthermore, in a few cases. the examinations that facility l licensees submitted for review and approval were. in the judgment ,

l of NRC chief examiners, as good or better than those prepared by l an NRC contractor. The staff expects that most facility licensees will eventually be able to prepare quality examinations in less time than the NRC cr a contractor because the facility employees have more detailed knowledge of their facility and easier access to the reference materials required to prepare the examinations. 1 Experience during the pilot examination program indicated that the volume of reference materials required by the NRC examiners to review and prepare for the examinations was significantly less than that required for the NRC staff to write the examinations.

The fact that contract examiners will not be used in the revised examination process will also eliminate the need for duplicate sets of reference materials to be provided by facility lice 1 sees.

Feedback from the industry in response to the staff's solicitation of public coments on the draft revision of NUREG-1021 indicated that facility licensees had been spending from 80 to 160 hours0.00185 days <br />0.0444 hours <br />2.645503e-4 weeks <br />6.088e-5 months <br /> to prepare and ship the reference materials under the existing examination process.

The additional burden of having to prepare the site-specific

, initial operator licensing examinations should be substantially.

l if not totally offset by reductions in Part 55 review fees billed to the facility licensees pursuant to 10 CFR 170.12(1). Each facility licensee will be billed only for the time that the NRC staff spends to review the examination prepared by the facility 5

1

licensee and to rework the examination as necessary. to bring it j up to NRC standards: no contractual costs will be accrued.

Although several of the draft pilot examinations were of poor j

-quality and took the NRC staff more time than expected to review i and rework, the staff believes that additional cost reductions I will be realized as facility. licensees gain experience with the .l NRC examination requirements and the quality of the draft examinations improves.

j This rule change will give facility licensees more control over  ;

the cost of their examination services because they will be in a  :

position to manage the quality of the product that is submitted to  !

the NRC. The higher the quality of the examination the facility 1 licensee submits. the lower the resulting charges. Under the  :

existing examination process, facility licensees are responsible  :

for the entire cost of preparing _the examination, even if the i contractor's submittal is of poor quality and requires significant i rework by the NRC staff before it can be administered. l It is.also Worth noting that facility licensees will have the option of retaining the services of a contractor to prepare the I license examination as the NRC often does under the current examinction process. The staff understands that the NRC's examination contractors have already expressed an interest in providing their services directly to facility licensees. It is reasonable to assume that the cost of using this option would be comparable to the contractor service fees that the NRC currently oasses on to the facility licensees.

In summary. the staff views this process change as resource neutral at worst..and possibly resulting in a resource savings to  :

facility licensees over time.

3. Imoact of the Rule Chanae on Ooerator License Aoolicants The goal of the rule change is to eliminate the NRC staff's >

., reliance on contractor support for. the site-specific initial operator licensing program, while complying with the AEA and maintaining the existing level of effectiveness. .To the extent possible. the format. content. length, and level of difficulty of the examinations will remain unchanged. thereby minimizing the impact of the rule change on the operator license applicants.

NRC eA : .sers will continue to review and approve every written ere tion and operating test before it is administered. The e e n trs will work with the facility licensee's staff to modify the dr3ft examinations as necessary to ensure that the guidance i contained NUREG-10211s met and to maintain consistency with prior licensing examinations at both that facility as well as amona all F

other facilities.

6 1 .--

i

.If'the NRC decides to prepare the examination in lieu of accepting I an examination prepared by the facility licensee, the NRC ~!

' examiners will use the same procedures and guidance (i.e.. NUREG- '

1021) that the facility licensee would have used to prepare the examination. The operator license applicants should not be able  :

to distinguish between an examination that is prepared by the l facility licensee and one that is prepared by an NRC examiner.

4. Alternatives to the Rule Change  ;

As discussed in the July 16, 1996 budget recommendations. contract support for preparing, administering, and grading initial examinations declines beginning in FY 1996 in anticipation of l Commission approval to implement the revised operator licensing i program. If the Commission decides not to amend 10 CFR Part 55 as  !

proposed by the staff, it would require agency resources to be i reprogrammed to increase the contract support for the operator  ;

licensing program or direct examiner resources in each regional  ;

office to satisfy the demand for initial licensing examinations i and to conduct the licensed operator requalification inspections.

III. Other Elements

'Leoal Analysis by the Office of the General Counsel (0GC) J The proposed rule would change the current practice that the NRC prepare  !

and proctor the initial examination for reactor operators and senior '

reactor operators and, instead require the power reactor facility licensee to 3repare the entire examination and proctor the written portion of t1e initial examination. 0GC worked with the staff to develop the backfit analysis set forth below and for the reasons 3rovided in that analysis believes this action does not constitute a Jackfit pursuant to 10 CFR 50.109. In addition, this action does not require an environmental assessment because it is categorically excluded from that requirement pursuant to 10 CFR 51.22(c)(1). 0GC has not identified any Paperwork Reduction Act issues nor does it believe this action constitutes a " major rule" pursuant to the Small Business Regulatory Fairness Enforcement Act of 1996. In all other res)ects. 0GC has not identified any potential legal complications or known )ases for a legal objection to the rulemaking.

Backfit Analysis  !

As discussed in SECY-96-123. 0GC agrees with the' staff's position that requiring the facility licensees to prepare the initial ~ operator

' licensing examinations would not be a backfit pursuant to 10 CFR 50.109, which states in pertinent part that --

Backfitting is defined as the modification of or addition to systems, structures. compcnents, or design of a facility; or ,

the design approval or manufacturing license for a facility: l or the procedures or organization required to design, '

7

.. - - ~ _ _ . -. . .. .. . -- _ - . . . .

construct or operate a facility: any of which may result  !

from a new or amended provision in the Commission rules or the imposition of a regulatory staff position interpreting '

the Commission rules that is either new or different from a l previously applicable staff position. ..

The proposed change does not result in a modification of or an addition  !

to systems, structures, components. or the design of a facility. The l change does not affect the design approval or manufacturing 1icense for

~

a facility. The procedures required to design or operate a facility l will not be affected by the proposed change. The proposed change would j require each nuclear power plant licensee to develop the tests that are i used to qualify. as meeting the requirements of 10 CFR Part 55, those  !

nuclear power plant operators whom the nuclear power plant licensee l wishes to employ. Development of such tests are not considered to be

" procedures.. required to... operate a facility." The tests are not i applied to the facility licensee, but rather to the operator license candidates. Further, any procedure necessary to develop the test would not be useful in actually " operating" the facility, even if one broadly interprets " operating" as including any action necessary to comply with the Commission's-regulations with respect to operation. The 1 organization required to design or operate a facility will not be  !

affected because all facility licensees already have a training staff to i train and evaluate candidates for operator licenses and to train other members of the plant staff, as required by 10 CFR Part 55 and by 10 CFR 50.120. Therefore an organizational change is not required because of this process change.

Suonortino Documents A regulatory analysis and an Office of Management and Budget (OMB) l statement will be prepared. Neither an environmental impact statement nor an environmental assessment will be prepared for this proposed rule.

however, since it is the type of action described as a categorical ,

exclusion in 10 CFR 51.22(c)(1).

l Resources Reouired As discussed previously in Section 11.2. the rule will require facility licensees to assume responsibility for writing the examinations. a burden currently carried by the NRC. However, the transfer of this l responsibility is considered to be resource neutral. and could possibly l result in a resource savings to facility licensees.

l The rulemaking is expected to take approximately 0.5 staff years to complete, and no contractor support will be needed. However, until the rulemaking is complete and all power reactor facility licensees are ,

l required to prepare their examinations, the Office of Nuclear Reactor Regulation (NRR) will continue to use contractors, as necessary. to supplement the staff's ability to develop and administer the initial 4 operator licensing examinations and to conduct the requalification inspection program. When the rule goes into effect. the staff expects '

8

that the revised examination process and the requalification inspection program can be implemented at all power reactor facilities with the existing NRC resources allotted to the operator licensing program:

however, during fiscal years 1997 and 1998, the staff intends to dedicate additional resources to the program to allow additional NRC examiners to be qualified, as further discussed in Enclosure 2.

" Resources."

l l

l l

1 J

l l

i i

l 1

l L l' l

i i

9 I

i

Coanizant Staff / Proiect Manaaement Concurrino Official L

Office of Nuclear Regulatory Research (Lead): H. Tovmassian Bill Morris  :

Office of Nuclear Reactor Regulation: S. Richards Bruce Boger i Office of General Counsel: M. Schwartz Stuart Treby Steerina Groun A steering group kdll not be required for this effort.

Enhanced Public Particioation l

This rulemaking will use electronic bulletin boards, as appropriate, to enhance input from the public.

EDO or Commission Issuance The proposed and final rule udll be approved by the Commiss on. I Schedule Proposed rule published: 3 months after the EDO approves plan Comment period ends: 60 days after the proposed rule is published Final rule published: 6 months after the comment period ends l

i

~

10

i l

i 1

i RESPONSE TO THE STAFF REQUIREMENTS MEMORANDUM OF JULY 23. 1996 (M960618A) i i

1 ENCLOSURE 2 i l

l

e l l

l DISCUSSION OF PROPOSED OPERATOR LICENSING PROGRAM CHANGES i Backaround l The site-specific initial operator licensing examination consists of a 100-L cuestion written examination and an operating test that includes a performance l cemonstration on a dynamic simulator and a plant walk-through. The written

! examination is in a multiple-choice format, with a passing grade of 80 l percent. The simulator performance demonstration generally consists of two or  ;

three scenarios administered to a team of three license applicants. Each i applicant is individually evaluated on a range of competencies applicable to the applicant's license level (i.e., reactor operator (RO) or senior reactor operator (SRO)). The walk-through portion of the operating test requires each l l a plicant to individually complete a total of ten tasks or job performance measures (JPMs). in the control room and the plant and to answer specific

! questions related to the associated systems. (Note: an R0 license being upgraded to an SR0 license only requires 5 JPMs.) The entire site-specific t examination has historically been prepared by NRC license examiners or NRC  ;

contractors. As a prerequisite for taking the site-specific examination. j every applicant must pass a separate generic fundamentals examination (GFE). l consisting of 100 multiple-choice questions covering basic knowledge of '

mechanical components, principles of heat transfer. thermodynamics. fluid mechanics and fundamentals of reactor theory. An NRC contractor prepares two versions of the GFE (one for boiling water reactors and another for

, pressurized water reactors) twice per year. The GFEs are reviewed and l approved by the Operator Licensing Branch (HOLB) before being mailed to the  !

facility licensees for administration.

On March 24. 1995. SECY-95-075. " Proposed Changes to the NRC Operator i

Licensing Program." informed the Commission of the staff's intent to revise  ;

the operator licensing program to allow greater participation by facility j licensees and to eliminate contractor assistance in this area. On i April 18. 1995, the Commission consented to the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full  !

1mplementation of the changes.

, From May to June 1995, the Division of Reactor Controls and Human Factors

-(DRCH) and HOLB conducted a number of meetings with senior management and i regional personnel to discuss various options for implementing the proposal l outlined in SECY-95-075. On August 15. 1995. after weighing the pros and cons of the various options (discussed below). the staff issued Generic Letter (GL) 95-06. " Changes in the Operator Licensing Program." which notified power reactor facility licensees of the NRC's intent to change the operator licensing process. outlined a number of criteria to supplement the guidance in i NUREG-1021. " Operator Licensing Examiner Standards." and solicited volunteers i to participate in a 6-month pilot program to evaluate and refine the new l examination methodology. l l

On June 10. 1996, the staff issued SECY-96-123. " Proposed Changes to the NRC

' Operator Licensing Program." which described the pilot examination process and criteria, reviewed the examination results, discussed the lessons learned, and l

1 l

l <

l l

l

recommended that the enhanced process be implemented on a voluntary basis pending an amendment to 10 CFR Part 55 that would require all power reactor facility licensees to prepare the entire initial examination for reactor o)erators and senior reactor operators and to proctor the written portion of t1e examination. The staff briefed the Commission on June 18. 1996, and in a staff requirements memorandum (SRM) dated July 23, 1996. was directed to develop a detailed rulemaking plan and to address the following specific issues regarding the proposed changes in the examination process.

Pros and Cons (Vulnerabilities)

In the early stages of developing the pilot examination process, the staff identified several concerns (examination integrity. quality and consistency.

independence and public perception, examination security, resources requirements, program stability, and examiner proficiency) that have played a dominant role in the design, implementation, and refinement of the revised examination methodology. The staff evaluated all of the pilot examination criteria specified in GL 95-06 in light of these concerns. The same concerns were also integral in evaluating the post-pilot process modifications that have been incorporated in the proposed revision of NUREG-1021. Although the staff believes that each concern has been adequately addressed and mitigated.

industry-wide implementation of the revised examination process entails a measure of uncertainty. The following is a discussion of each concern the measures that have been implemented to mitigate the concern, and the i associated vulnerabilities.  !

Integrity, Quality, and Consistency of the Examination Section 107 of the Atomic Energy Act of 1954. as amended. requires the Commission to prescribe uniform conditions for licensing individuals as operators, determine the qualifications of such individuals, and issue licenses to such individuals.

l When the staff developed the pilot examination program, it was concerned about I whether th0 existing levels of examination uniformity and consistency could be  ;

maintained. The operator licensing recentralization study (see SECY-93-309. l

" Findings of the Operator Licensing Recentralization Study." of November 17, 1993), which had concluded that the level of examination uniformity was adequate to ensure that appropriate licensing decisions were being made. highlighted the challenge of controlling the diversity that individual examiners from different regional offices could add to the examination program. Permitting a substantial number of facility employees having differing levels of training and experience to prepare the examinations could increase the amount of variation in the levels of knowledge tested and examination difficulty despite the efforts of the NRC to control these factors l

through improved procedures and oversight reviews.

To maintain uniform standards of examination content, format, level of difficulty, and quality. the staff insisted that the participants in the pilot program prepare the written examinations and operating tests in accordance with the procedures and guidelines in Revision 7 of NUREG-1021. NUREG-1021 4

contains guidelines and criteria that are adequate to ensure that a licensing examination, independently prepared by an NRC examiner or contractor, will

, 2

I l

differentiate between those applicants who have and those who have not mastered the knowledge and abilities required to safely operate the plant.

However. when the staff reviewed those criteria in light of the proposed shift in responsibility for writing the examinations. it concluded that some of the criteria were no longer appropriate and that additional criteria were necessary to ensure that tne integrity of the facility-developed examinations l would remain at acceptable levels. Therefore, the staff promulgated the following supplemental and modified criteria in GL 95-06 when it solicited volunteers to participate in the pilot examination program:

e NUREG-1021 currently allows NRC and contract examiners to use up to 10 I percent of the written questions. job performance measures, and simulator scenarios in a facility licensee's examination bank when they develop an initial licensing examination for the facility. This process saved NRC resources and provided an incentive for facility licensees to develop their examination banks because the examiner could conceivably extract the entire written examination from a facility licensee's bank.

provided the bank contained 1.000 or more questions that satisfied the psychometric criteria in NUREG/BR-0122. " Examiners' Handbook for Developing Operator Licensing Written Examinations."

Whereas the NRC's examiners are continually revising old test items and developing new ones. there is currently no requirement for facility licensees to take comparable measures to minimize predictability.

Predictable examinations tend not to discriminate between competent and deficient license applicants because what is being tested is simple recognition of the answer. Although studying past examinations can have a positive learning value, total predictability of examination coverage through over-reliance upon examination banks reduces examination integrity. When the examinees know the precise and limited pool from which test items will be drawn, they will tend only to study from that pool (i.e.. studying to the test) and may exclude from study the larger domain of job knowledge. When this situation occurs. it decreases confidence in the knowledge inferences that are made from performance on the test.

To limit predictability. GL 95-06 revised the limits on the use of facility item banks that could have been published. reviewed, or used as a basis for training. GL 95-06 treated all question banks as if they were open to the applicants because the staff had no basis to regulate the qualities of the item banks and because it would be difficult to confirm that a bank was indeed closed to the applicants. The staff attempted to strike a balance between the use of new and existing test items because it appreciated the effort required to develop new test items and understood that existing items are a valuable training resource that should not be wasted. On the basis of the results of the generic fundamentals examination (GFE) program. the staff decided to limit the use of existing written questions on the site-specific examination to no more than 50 percent and to require at least 10 newly developed (i.e.. not previously seen) questions on every examination.

The remaining questions could be generated by significantly modifying existing bank questions. Each operating test was required to include at 3

l

least two JPMs that were either new or significantly altered and at least one simulator scenario that was new or significantly altered. The remaining scenarios had to be modified to the extent necessary to 3revent the applicants from immediately recognizing the scenarios on the Jasis of the initial conditions or other cues.

e NUREG-1021 currently places a 25-percent limit on the number of written examination questions that can be repeated from the previous two NRC licensing examinations at the facility. To further limit the predictability of the examinations that would be prepared by the facility licensees. the staff resised this criterion to include all examinations, quizzes. and tests administered to the applicants during their license training class. Furthermore. no questions could be drawn directly from the applicants' audit examination given by the facility licensee at the end of the license training class to determine if the applicant was ready for licensing. This restriction was the subject of industry comments and has been revised as discussed below since completing the pilot program.

o NRC and contract examiners are currently required to cite a specific facility reference for every question on the written examination.

Questions that were extracted from the facility licensee's question bank would be identified as such. ra order for the NRC chief examiners to monitor the facility licensees compliance with the previously stated criteria, GL 95-06 also required the pilot examination participants to state the history (e.g.. bank. revised, new, and date last used) of each test item on the written examination and operating tests. Although this practice did ) lace an additional burden on the facility licensees, it facilitated tie NRC's examination reviews and resulted in lower licensing fees billed to the pilot participants. This requirement was also addressed in the industry comments and has been clarified as discussed below since completing the pilot program.

The Office of Nuclear Reactor Regulation (NRR) also issued supplementary written guidance to the regional offices clarifying the NRC examiners' responsibilities during the pilot examination development process, and it conducted a public workshop to discuss the staff's expectations during the l pilot program. The written guidance and the minutes of the workshop were placed in the NRC's Public Document Room (POR) and provided to facility licensees that were scheduled to participate in the pilot examination program.

i NRR did not restrict the number or scope of changes that the NRC examiners could ask facility licensees to make in their proposed examinations. The existing quality assurance (QA) checklists in NUREG-10?1 were revised and new checklists were developed to help the examiners identify problems with the preliminary examination outlines and the operating tests that the facility licensee would be required to submit for review.

Because of resource limitations, examiners were instructed to focus primarily on the content and construction (i.e.. level of knowledge and difficulty) of l the written examination questions rather thr _ pend their time verifying the detailed technical accuracy (e.g. procedural and component references) of 4 1 l

every question. as is required for NRC-developed examinations. The staff expected the facility licensees to ensure that the examinations were technically sound and instructed the regions to question the licensees regarding any post-examination question deletions and answer key changes. i To assess the effectiveness of the )ilot examination process. NRR asked the

NRC examiners who were involved wit 1 the examinations to respond to a questionnaire on various aspects of the new methodology. The feedback indicated that many of the as-written examinations were comparable in quality and level of difficulty to recent NRC-written examinations. However, the NRC examiners and facility staff had to rework some of the examinations significantly to meet NRC standards. Although the staff had hoped that the technical accuracy of the facility licensee's written examinations would be as good as or better than the accuracy of the examinations written by the NRC.

some of the pilot examination results (i.e.. more question deletions and

, answer key changes than expected) indicated a need for improvement in that j area.

i Despite the challenges of quality and technical accuracy, the staff believes that each as-given pilot examination provided a valid basis for licensing the applicants who passed the examination. The staff expects that the consistency

and quality of the draft examinations will improve cs facility licensees gain experience and become more familiar with NRC examination requirements. Based 4 on observations during the pilot examinations. the staff has also incorporated a

in Revision 8 of NUREG-1021 a number of process enhancements that are intended to improve the quality of the examinations developed under the revised

3rocess. A number of the significant enhancements are discussed below. NRR las also instructed the regional examiners and managers not to approve or administer any examination that does not adhere to current NRC standards for content format. quality, level of knowledge. and level of difficulty.

Deviations from the guidance contained in NUREG-1021 will require ap3roval by the Operator Licensing Branch. If necessary the examination is.to )e delayed or cancelled until the facility licensee and the NRC staff can upgrade or

replace the examination.

1 The overall results of the pilot program indicated that the examinations I l

prepared by facility licensees. subject to review. revision where appropriate.

! and approval by NRC staff examiners, were generally consistent with examinations prepared by NRC or contract examiners. The pilot examinations e

also appeared to be equally effective at identifying applicants who had not mastered the job requirements well enough to become licensed R0s or SR0s.

Furthermore, the pass rates for R0s and SR0s on the written examinations and operating tests compared favorably with the pss rates for examinations prepared by the NRC or its contractors.

The chief examiners for approximately half of the pilot examinations actually considered the submitted examinations to be as good or better than recently .

ddministered NRC examinations because the facility authors were better able to i integrate the required site-specific knowledge and abilities into operationally oriented questions. All but 5 of the 22 feedback surveys completed by the chief examiners indicated that the revised examination process was as effective or more effective than the traditional examination 5

t i

I process. Most examiners also agreed that the performance of the facility i

. licensees should improve as. they become more familiar with the NRC's  !

requirements and expectations. i In summary, the operator licensing program staff has implemented numerous  !

procedural measures to maintain and promote the consistency.. quality. and i integrity of the initial licensing examinations. However in practice. it i will be the responsibility of individual examiners. with appropriate review i and oversight by their supervisors and the NRR staff, to ensure that the  !

. licensing process remains strong. It is possible that the pressures to {

conduct examinations on schedule and the reduced involvement of the NRC in the l examination develo) ment process could lead to examinations that discriminate i at a lower level tlan is currently the norm. The revised examination process  !

is much more dependent u)on the staff's ability to identify a substandard examination and ensure tlat the deficiencies are corrected. If the staff is-indecisive and the licensee takes advantage of the situation, the examination will probably not discriminate at the appropriate level.

Independence and Public Perception i Before 1987. the staff believed that only independently developed and administered examinations could reliably and consistently achieve the appropriate standards of objectivity and discrimination required for the NRC's initial licensing and requalification programs.. The staff was concerned that  !

examinations prepared by a facility licensee could be unconsciously biased i toward the specific material taught by the facility and that potential  !

financial or other pressures on the facility licensee would create a conflict  !

of interest.

l l The staff's position changed significantly in 1988 when it revised the requalification examination methodology so as to require each facility licensee to prepare the examination materials and demonstrate its ability to l

evaluate the competence of.its licensed operators. From 1989 to 1994, the NRC evaluated over-4.000 licensed operators using requalification examinations that were prepared by the facility licensees. reviewed and approved by the NRC. and administered in parallel by NRC examiners and facility personnel.  :

l The examinations appeared to. discriminate at the a)propriate level. and l

approximately 90 percent of the operators passed t1e examinations on their l first attempt.

~

Despite the success of the requalification examination process. the staff continued to independently develop and administer the initial licensing I examinations. However the overall performance of the license applicants on  !

L the initial examinations was not significantly different from the performance l l- -of the licensed operators on the requalification examinations developed by  !

their facility licensees. From 1989 to 1994. the NRC administered initial licensing examinations to approximately 4.000 applicants: the pass-rate on those examinations averaged approximately 90 percent.

i' In SECY-95-075 the staff 3roposed to take the next evolutionary step, similar

[ to the staff's actions wit 1 regard to oversight of the requalification L program, by changing the guidance in NUREG-1021 to permit facility licensees 6

l

_ _ __ _ . . ___,_ _ _, ~, _ _ . ,__ _._

I to prepare and, in part, conduct the written and operating examinations for  !'

initial license applicants. The NRC would review the licensees' efforts.

partially participate in the examinations, retain the final pass / fail authority, and continue to issue licenses. This action would allow the staff to perform its licensing function in a more efficient manner. while  !

maintaining an effective. independent process upon which to base licensing decisions for operators.

Shortly after SECY-95-075 was issued. The Washington Post reported that l consumer advocates were concerned that allowing facility licensees to prepare the examinations might endanger reactor safety because the examinations might .

be less rigorous than those prepared by the NRC. The information upon which  !

the consumer groups had based their conclusions was preliminary in nature and t likely did not consider the level of oversight that the NRC would retain in the examination process or the compensatory measures that the staff would implement to ensure that the examinations retained an appropriate level of difficulty.

The Washfngton Post article also indicated that the proposal appeared to contradict the staff's earlier decision to reject a request by the Virginia i

Electric and Power Company (VEPC0) to prepare its own examinations (this topic  :

will be discussed in more detail later) and that a number of mid-level staff  !

members opposed the change. The staff believes that the initial opposition to the proposal was a natural reaction to the abrupt change in policy and to the ,

fact that many staff members had not been consulted or informed before I SECY-95-075 was issued. As promised in the memorandum of April 12. 1995, from l the Executive Director for Operations all involved staff members have been well informed of the new policy and have been able to express their views and i to have their concerns considered as part of the process to implement the 1

broad goals stated in SECY-95-075. 1 As noted in SECY-96-123. the results of the pilot program indicated that

initial licensing examinations prepared by facility licensees. subject to review and approval by NRC staff examiners in a manner similar to the review i

and approval of NRC requalification examinations, were as effective as

'; examinations written by contractors in identifying applicants who had not mastered the job requirements for a licensed operator. The pass rates for R0s

- and SR0s on the written examinations and operating tests compared favorably with the pass rates for examinations prepared by the NRC and its contractors.

To promote examination consistency and minimize the bias toward specific materials taught by the facility licensee, the staff insisted that the pilot 4 - examinations be developed in accordance with NUREG-1021 and NUREG/BR-0122.

NUREG/BR-0122 contains explicit guidance for sampling the important, safety-related knowledge and abilities (K/As) from NUREG-1122. " Knowledge and Abilities Catalog for Nuclear Power Plant 03erators: Pressurized Water i

l Reactors." and NUREG-1123. " Knowledge and A)ilities Catalog for Nuclear Power '

Plant 0)erators: Boiling Water Reactors," companion documents derived from a job-tasc analysis of the operators' work. The supplementary pilot examination criteria in GL 95-06 required the facility licensee to submit a proposed examination outline to the NRC for review and approval at least 60 days before the scheduled examination date. This requirement would enable the NRC chief l

7

1 examiner to assess whether the sampling was appropriate (e.g., the balance of coverage and importance ratings: duplication and overlap between examination sections and from prior examinations: and inclusion of plant-specific priorities such as recent events and risk insights) before the facility licensee proceeded with developing the detailed examination materials. i GL 95-06 also required facility licensees to submit the proposed examination 1 to the NRC for review at least 30 days before the scheduled administration i date. The NRC chief examiner and the responsible supervisor would work with the facility licensee to revise the examination as appropriate before authorizing its administration.

To minimize personal conflicts of interest, GL 95-06 3rohibited facility employees who played a substantial role in training t1e license applicants i from becoming involved in preparing the licensing examinations. Although this restriction did little to mitigate the fact that everyone who worked for the l i

facility licensee was likely to have a vested interest in the outcome of the examinations. it did alleviate some of the concerns regarding personal gains or penalties that the examination author might incur if he or she were directly accountable for the examination results. This restriction also allayed the staff's concern that an instructor might unconsciously bias the examination toward those topics that were emphasized during the training program. The personnel restrictions were undoubtedly the most controversial aspect of the pilot examination process. They were widely discussed among the .

NRC staff and management before, during, and after the pilot program was 1 implemented. and they were also the principal issue raised by industry i regarding the pilot process. A discussion of this and a number of other i industry concerns follows.

The issues of independence and public perception also had a significant bearing on the methodology that was selected for administering the operating tests, the best measure of the applicants' ability to respond to a plant event. Although general agreement existed among the staff on most of the pilot criteria discussed earlier. there was considerable disagreement regarding the level of involvement that NRC examiners should retain in the operating test administration process. The following options were evaluated in depth:

Option 1: NRC examiners would continue to independently administer the simulator scenarios and the plant walk-through portion of the operating test in accordance with current guidance in NUREG-1021. Involvement by the facility licensees would not l change from current practice, and facility employees would not conduct parallel evaluations. j Option 2: NRC examiners and facility employees would conduct parallel l evaluations of each applicant during the dynamic simulator j and walk-through portions of the operating test. This is l the same methodology that is used during NRC-conducted i requalification examinations.

8 l

l l

- - . l

l Option 3: NRC examiners would observe facility employees conduct the simulator scenarios and a sample of the walk-through tasks for each applicant; however, the NRC examiners would not l

conduct one-on-one parallel evaluations of each applicant.

This methodology is similar to that currently used durino the requalification inspection program.

After much deliberation. the staff decided to im)1ement Option 1 during the pilot examination program. The staff believed t1at Option 1 would have the most favorable public perception and examiner acceptance because it maintains the existing levels of independence and objectivity by relying on NRC examiners to conduct and evaluate the operating tests. Option 2 would have generated more concern regarding conflicts of interest because it would require facility employees to administer and grade the operating tests (to their coworkers and friends) in parallel with NRC examiners. Option 3 would have had the least NRC involvement and. therefore, the greatest )otential impact on independence and public 3erception. By not directly o) serving each applicant, the NRC examiner would 3e in a weaker Josition to judge the performance of the applicant so as to determine w1ich applicants should pass or fail if a crew responded incorrectly during a simulator scenario.

In summary, it is possible that the pressures to conduct the examinations on schedule and the reduced involvement of the NRC in the examination development process could lead to examinations that reflect the biases of the facility licensee and that discriminate at a lower level than is currently the norm.

However, a number of process requirements have been augmented as necessary to minimize this potential. The staff believes that the revised examination process, which includes restrictions on personnel, independent NRC review and approval of the examinations. and independent NRC administration of the operating tests, will provide sufficient safeguards to ensure unbiased licensing decisions.

Given the complexity of the proposed change in the examination process. the public may jump to the conclusion that the NRC has transferred all responsibility for examining operator license applicants to the facility  !

licensees. In actuality, the NRC retains a significant amount of control over l the content of the examinations and continues to administer the operating tests. The challenge may be to explain this fact to the public.

Examination Security In 10 CFR 55.49. applicants, licensees, and facility licensees are prohibited from engaging in any activity that compromises the integrity (security) of any application. test or examination required by Part 55.

Before 1989, ensuring compliance with the rule was relatively straightforward ,

because facility licensees were not permitted to review the written operator i licensing examination until it was distributed to the applicants on i examination day. Security compromises and cheating were rare but not unheard i of occurrences.

9

l l

The degree of risk to examination security increased significantly in January 1989 with Revision 5 of NUREG-1021. Revision 5 made the facility licensees l largely responsible for preparing the NRC requalification examinations and. l for the first time, also allowed the facility staff to review the written licensing examination up to 2 weeks before its administration. The increased risk to security was tolerated because the prereviews helped identify inaccurate questions before the examinations were administered, thereby increasing the content validity of the examinations, minimizing the number of i questions from applicants during the examination. and reducing the number of  !

question deletions and changes during the grading process. The gradual shift to multiple-choice questions during the late 1980s and early 1990s ) laced a premium on accuracy because the answers to the questions could not )e adjusted after the fact as they could be with the essay and short answer questions that they. replaced.

(

l To maximize security, given the new philosophy. NUREG-1021 prohibits any

, facility employee with specific knowledge of any NRC examination before it is

! given from comunicating the examination contents to unauthorized individuals and from participating in any further instruction of the students scheduled to take the examination. Before they are given access to the examination, the facility employees are required to sign a statement acknowledging their l

understanding of the restrictions and the potential consequences of noncom)liance. They must also sign a post-examination statement certifying

that t1ey did not knowingly compromise the examination.

The staff realized that the potential for examination compromise would i increase as the facility licensees increased their role in the initial l licensing process from one of reviewing the NRC-developed examinations to that l

l of developing their own examinations. Therefore. the staff attempted to clarify its security expectations in the supplemental pilot examination guidance that was provided to the regional offices (and placed in the PDR) in i

conjunction with GL 95-06. The guidance addressed the restrictions on personnel and a number of physical security precautions. including protecting and mailing the examination materials. The guidance also cautioned examiners to be attentive to examination security measures and required them to review the security requirements with the facility licensee's contact at the time the examination arrangements were confirmed.

The issue of examination security also had a bearing on the methodology that the staff selected for administering the pilot operating tests. The staff believed that Option 1 would pose the lowest threat to examination security because the smallest number of facility employees would require detailed knowledge of the examination before it was given. Options 2 and 3 both would have required the facility licensee to evaluate the license applicants during the simulator and walk-through tests which could have required several additional personnel to familiarize themselves with the operating test content. thereby increasing the risk of compromise.

l I In summary. the staff expects that the vast majority of facility licensees I

will continue to maintain proper examination security (as they have during the requalification program). Security is enhanced by the fact that the NRC monitors and oversees an extensive range of the facility licensees' activities l 10

and is sensitive to allegations and other performance indicators that might '

expose a problem. Nevertheless, the staff cannot deny that the NRC is now more vulnerable to examination compromise (and the associated public response) than it was before 1989 when it first allowed facility licensees to become involved with the NRC's examination development programs. The staff anticipates that the agency may suffer an occasional lapse in security. For example, unintentional lapses of examination security have occurred at the Byron and Millstone sites during 1996. Another examination compromise at Dresden is under investigation. When a lapse occurs. the staff will continue to take prompt action to cancel the examination. if appropriate, determine the cause of the compromise and implement corrective measures when called for.

Resources As stated in SECY-95-075. the proposed changes were originally conceived as part of the NRC's continuing efforts to streamline the functions of the Federal Government consistent with the Administration's initiatives and to accommodate NRC resource reductions. The original plan was to entirely eliminate the use of contractors in the operator licensing area and to conduct all operator licensing program activities, including the generic fundamentals and initial licensing examinations. and the requalification program, with the existing staff resources. The facility licensees would prepare and. in aart.

conduct the initial licensing examinations with oversight by the NRC. T1e NRC's participation would range from conducting part to all of every examination in order to provide an independent basis for making a licensing l

decision and to maintain examiner skills.

l The staff estimated that a qualified facility em)loyee should be able to prepare the site-specific examination based on t1e guidance contained in l NUREG-1021. as supplemented by GL 95-06. in approximately the same amount of

time that is currently allocated to a contract examiner. (See Attachment 1.

the associated rulemaking plan, for a more detailed discussion of the estimated resource burden on facility licensees.) The staff also estimated that an experienced NRC examiner should take approximately two weeks to review the site-specific examination (including the written and operating tests) prepared by a facility licensee.

As previously noted, the staff considered three options for administering the operating tests. Assuming a license class consisting of nine applicants (e.g., six R0s and three SR0s) that could be arranged in three. three-person crews. both the first and second options would generally require three NRC examiners to spend a week at the site to administer the tests. The second option. which would require the facility licensee to conduct parallel

, evaluations of each applicant during the simulator scenarios and walk-through l

tests. would place a significant additional burden (nominally three staff-4 weeks) on the facility licensee. The third option would be equally burdensome to the facility licensee but would )ermit the NRC to reduce (by one) the number of examiners dispatched to t1e site.

The plan to require the facility licensee to evaluate the applicants during the operating tests was perceived as a du)lication of effort because the facility licensee had already evaluated t1e applicants and found them 11 4

9

1 qualified before it submitted the license applications to the NRC. The staff was also concerned that the additional regulatory burden would discourage participation in what was originally envisioned as a voluntary program with no associated rulemaking.

The NRC resource utilization rate during the pilot examinations supports the staff's original estimate that the new examination process and the requalification inspection program could be implemented at all power reactor facilities with the same level of direct NRC resources as is currently allotted'to the operator licensing program. As noted in SECY-96-123 the NRC examiners required an average of about 350 hours0.00405 days <br />0.0972 hours <br />5.787037e-4 weeks <br />1.33175e-4 months <br /> to review, prepare for, administer, grade, and document each pilot examination. Most of the examinations included an extra visit to the site to review and validate the operating tests.

However, with the elimination of contractor support. the total number of examiners available will decline significantly. thereby reducing the ability of the program to have the personnel available to meet peaks in facility licensee requests for examinations. The impact of the loss of examiners due to reassignments. transfers, promotions, and other personnel actions will be larger without the contract examiners available to compensate for the loss.

To compensate in this area. NRR intends to require that the regional offices maintain a defined minimum level of examiners. to be defined in each regional operating plan. During fiscal years (FYs) 1997 and 1998, each region may need-to qualify additional examiners to meet this expectation. To specifically assist the regions in their efforts to train additional examiners. NRR intends to make available to the regional offices up to 7 full time equivalents (FTE) of inspection support during FY 1997 and up to 12 FTE of inspection support in FY 1998. This inspection support will allow regional personnel, who would l normally be assigned to inspection activities, to be assigned to qualify as examiners. By the end of FY 1998, the regions will be expected to maintain the defined level of examiners in order to provide the resource flexibility needed to deal with workload peaks. When examination work is not scheduled.

the examiners will be available for inspection activities. A reduced level of

! contractor support will be required during the transition period.

The staff expects that if the quality of an examination prepared by a facility l licensee deviates significantly from established norms. it will become l increasingly difficult to predict the amount of time necessary to review or l rewrite the examination so as to achieve an acceptable product. This was the l

case with several of the pilot examinations. and the staff expects that it will continue to be more of a problem with facility-prepared examinations than l it has been with those prepared by an NRC contractor. This added uncertainty l in the examination process could increase the risk of broken examination

! ' commitments and lower examination quality: it could also raise costs as a j result of inefficient planning.

With the elimination of contractor support and the increased uncertainty about the examination quality, the staff may no longer have sufficient examiner resources to write or even to review all examinations consistent with the scheduling needs of facility licensees. This resource problem is further compounded by the unpredictable nature of the examination workload and by 12

._- . -.. - -_- - - -. .- --.---- - - - - - . ~ .

l.

Other unanticipated demands'on the examiner work force. such as the increase i in the number of examination appeals during the pilot examination program. ,

i Historically, the contract examiners have facilitated the staff's ability to i respond to unexpected increases in.the examination workload and unforeseen- '

2 de]artures by staff examiners. Eliminating contractor support will complicate scleduling and limit the staff's ability to conduct retake examinations, which

could increase the likelihood that an a3plicant will appeal the results of the examination. It is for these reasons tlat the staff is recomending to the Commission that 10 CFR Part 55 be amended to require all facility licensees to

, prepare the licensing examinations.

I

'With regard to the generic fundamentals examination (GFE). the. staff ,

i considered the following options: (1) retaining the current format and i _ process under which the examinations are developed by a comercial contractor and (2) revising the sco)e of.the site-specific written examination to sample the generic fundamental (nowledge and abilities, as was done before 1988. If the examination was revised by adding more questions, it would increase the ,

burden of writing and reviewing the examinations (whether it was done by the facility licensee or the NRC). If the length of the examination was kept the same.. inclusion of the generic knowledge and abilities would weaken the '

evaluation of the site-specific systems and procedures. The staff and the

. industry generally have agreed that the current program is both efficient and effective and that revising the site-specific sample plan would create too many problems. Accordingly, the staff has decided to retain the current -

generic fundamentals examination format and process and to implement the program with contractor assistance at a cost of about $200.000 per year. '

Program Stability From 1988 to 1990, the NRC received numerous complaints from facility licensees and licensed op.erators that the NRC's requalification examination program was causing undue stress for the operators and the facility employees who were administering the examinations. The Human Factors Assessment Branch conducted a study to address this issue and determined that the frequent -i changes to the requalification examination process ranked among the most significant sources of undue stress: the results of the study are documented in SECY-91-391. "Results of the Study of Requalification Examination Stress."

One of the corrective actions involved establishing a schedule for periodic l revisions of NUREG-1021 and delaying the implementation of each revision for 180 days after it is published.

Although the staff did not survey license applicants during the  !

requalification stress study. it has generally tried to minimize the adverse l effect on applicants by avoiding unnecessary changes to the initial l examination process. Therefore. when the staff decided to shift responsibility for developing the initial licensing examinations to the facility licensees. it resisted pressure to revise the examination format. To the extent possible the format, content. -and level of difficulty of the pilot examinations would remain unchanged, thereby making the changes imperceptible to the license applicants. The staff also believed that the facility ,

licensees were generally familiar with the existing requirements in NUREG-1021 1 13 3

i i

and that any significant changes in the examination format would confound l their efforts to assume responsibility for writing the examinations. j The desire i.o maintain program !tability, avoid unnecessary changes and minimize stress on the applicants also argued against the adoption of Options 2 and 3 for administering the operating tests. The requirement for facility personnel to evaluate the license applicants would increase the learning curve for the facility licensees and would likely elevate the stress for the applicants because of the double scrutiny and the increased crowding in the simulator.  ;

In summary, the staff found no compelling reason to modify the existing i examination format. The benefits of retaining the current format far exceed any argument for making changes at the present time.

Examiner Proficiency On May 25. 1995, the staff briefed the Commission on the status of the  !

operator licensing programs and the staff's proposal to shift responsibility i for preparing the initial operator licensing examinations to the facility licensees. In the ensuing SRM of June 2. 1995, the Commission instructed the i

staff to ensure that the in-house capability to administer the examinations is maintained. possibly through the random administration of some examinations. ,

1 As noted in SECY-96-123. the staff believes that NRC examiners will maintain  !

the ability to write examinations, if necessary, based on their participation I in the new process. Although facility licensees will prepare the operator licensing examinations based on the guidance contained in NUREG-1021. NRC i examiners will continue to review and approve every examination before it is administered to ensure that it conforms to the criteria specified in NUREG- 1 i 1021 for content. format quality, and level of knowledge. Facility licensees  :

. will also proctor and grade the written examinations in accordance with NRC 4 procedures, but NRC examiners will continue to independently administer and grade both the dynamic simulator and the plant walk-through portions of the .

. o)erating tests. The NRC will review and approve the assignment of grades for i t1e written examinations, including any changes to the answer keys and the deletion of questions recommended by the facility licensee. The NRC will also continue to make all licensing decisions and administer the appeals process for applicants who fail the examination. Additionally. the NRC may elect to i

)repare an examination, as allowed by the proposed rule, when poor performance  ;

ay a facility licensee in the training or operations area would jeopardize the validity of the examination.

NRR will continue to require new NRC license examiners to complete a t

standardized training program before they are certified to conduct licensing examinations. The training program includes classroom and simulator instruction on how to prepare and administer written examinations and operating tests, self-study of operator licensing procedures and guidance, on-the-job training, and a final practical evaluation by a certified chief examiner. Once certified, examiners are required to maintain their qualification by conducting examinations and attending refresher training.

NRR is currently revising the examiner refresher training syllabus to focus

14

. _ ~ __ - -- .- . - .- -- -. _- -

! more attention on reviewing. critiquing, and approving written examination l

questions. Examiners are also required to attend the operstor licensing -

examiners' training conference. which is periodically convened by NRR to provide guidance and exchange information.

To ensure that the on-site examination activities are being performed as intended. supervisors are required to periodically evaluate every certified examiner during the conduct of an operating test. Furthermore, to verify the quality of the examination process and to assess the uniformity with which the examinations are administered within and among the regions the NRR staff periodically audits the administration of operator licensing examinations at selected facilities and annually reviews the implementation of the operator licensing function in each region. The NRR audits and reviews include an assessment of the level of knowledge and the level of difficulty of the ritten examinations and the operating tests: each audit and review is documented in a report to the responsible regional Division Director.

Industry Acceotance and ResDonse

! In the sumer of 1994. VEPC0 recommended to the NRC that the initial licensing examination 3rocess be changed to make facility licensees responsible for developing t1e examinath as subject to review and approval by the NRC chief l examiner. The NRC would continue to administer and grade the examinations.

l The proposal, which was submitted as part of VEPCO's cost-beneficial licensing-action program, estimated that the utility could save an average of 567.500 per station for each initial license class by implementing the proposed approach. The staff reviewed and denied the proposil because of concerns I i

regarding independence and objectivity and the potential for increased variability in the level of knowledge tested and examination difficulty.

The staff reconsidered its position early in 1995 when the NRC's fiscal I expectations changed significantly. The staff decided to explore the l possibility of eliminating the use of contractors in the operator licensing

program by allowing facility licensees to develop the examinations.

essentially just as VEPC0 had proposed. It was estimated that such a change )

would save the NRC from $3 million to $4 million per year in contractor support funding and that the reduction in recovered fees would largely or l

completely compensate the facility licensees for their additional expenses.  ;

l While evaluating various options for implementing the changes proposed in )

SECY-95-075. the staff obtained informal feedback from the Nuclear Energy Institute (NEI) regarding the industry's level of involvement in administering the operating tests. NEI had polled a number of facility licensees and determined that there was no support for parallel grading (0] tion 2 previously discussed) because of concerns regarding applicant stress, t1e additional resource burden, and simulator crowding. The majority of those surveyed i expressed cautious support for Option 3. under which NRC examiners would have i observed the facility licensee administer the operating tests. However, those I surveyed wanted the NRC examiners to focus only on the performance of the applicants and not on the examination practices of the facility licensee.

Option 3 would shift the most responsibility to the facility licensees. but it was viewed as the most difficult option to implement. Facility licensees 15 i

1

perceived that Option 1. under which NRC examiners independently administer the operating tests. would be the least demanding on facility licensees and  !

the easiest to implement and would retain the greatest level of independence  ;

in the licensing decisions. NEI indicated that all of the facility licensees l surveyed were satisfied with the GFE and thought it should be continued in the current manner.

The staff believes that the industry's response to GL 95-06. which described I

.the pilot process and solicited volunteers to participate in a six-month <

program to evaluate its feasibility was generally favorable. Of the 35 i facility licensees that had requested examinations during the pilot period. 22 agreed to prepare their own examinations in accordance with GL 95-06. Most of the facility licensees that declined to participate cited as their reason their inability to reschedule the required resources on such short notice because of the restrictions on the use of personnel who had been involved in training the license applicants.

In September 1995. shortly before beginning the pilot examinations, the staff convened a public workshop to discuss the NRC's expectations regarding  !

industry participation in the pilot program for the operator licensing examination. The workshop also gave facility representatives an opportunity  :

to ask questions of the staff and to submit Written comments regarding the pilot examination criteria outlined in GL 95-06. Most of the questions and i coments related to the restrictions on which personnel the facility licensee could use to prepare the examination and the requirement to document the history of the examination materials. Some facility licensees questioned the ,

amount of reference materials that the NRC would need to review and prepare )

for the examination. The staff challenged the pilot examination participants  ;

who expected to have problems satisfying the personnel restrictions to suggest )

alternative ways to achieve the same objective. The staff committed to review  !

the alternatives on a case-by-case basis and to propose a final position in ,

Revision 8 of NUREG-1021. which would be published for comment near the end of i the pilot program. The staff also made it clear from the beginning that 1 facility licensees could use contractors to develop the examinations.

Feedback from NRC examiners who participated in the pilot examinations and  ;

industry representatives who attended a public progress meeting sponsored by '

NEI in January 1996 confirmed that the personnel restrictions were the most f

troublesome aspect of the pilot process. Many facilities. particularly those l

with smaller training staffs contended that the cuality of the examinations was suffering because most of their best qualifiec personnel were involved in teaching and were ineligible to prepare the examinations. Those attending the meeting generally agreed that the industry and the NRC had learned a great deal from the pilot program, that most of the issues could likely be resolved through training and improved communication. and that the pilot process should be fully implemented.

In February 1996 the staff issued draft Revision 8 of NUREG-1021 that incorporated both the pilot process outlined in GL 95-06 and preliminary lessons learned from the early Jilot examinations. The draft revision was sent to the PDR and placed on t1e NRC's home page on the World Wide Web. and a Federal Regfster notice requesting industry and public comments was published

. 16

- -- -. . -. - - - - ~ - - - - . - --

l on February 22. 1996. NEI submitted comments and recommendations on behalf of the nuclear industry, and two facility licensees (VEPC0 and Commonwealth Edison Company) provided additional comments. On April 18. 1996, the staff conducted a public meeting with NEI and other industry representatives to

review their comments and recommendations regarding the draft revision of

NUREG-1021. The meeting also provided an opportunity to discuss additional issues and proposed changes that had emerged following the issuance of draft 4

Revision 8 of NUREG-1021 for public comment. The significant comments and 1 staff responses are as follows:

1 4 o All three commenters suggested that the personnel restrictions placed on individuals involved in writing the examinations should parallel the personnel restrictions of the requalification program (i.e., the i examination authors must stop teaching when examination development begins). The industry believes that those restrictions, when combined with the examination development and review criteria, would provide adequate protection against the unconscious biasing of an examination by an individual.

It is difficult to argue that the instructors would not be the best l qualified individuals to prepare the examinations. Many NRC examiners i also agreed that the requalification program restrictions would be  :

adequate. However. the staff concluded that independence and conflict i of interest were of sufficient concern to require personnel restrictions i very similar to those outlined for the pilot process. The staff revised ,

the wording in the draft NUREG to clarify and cjuantify the amount of  :

involvement that would disqualify an employee irom participating in '

various aspects of examination development. For example. the personnel who prepare the examination outline including the election of knowledge and abilities to be evaluated on the written examination and the JPMs and simulator scenarios to be performed during the o)erating l test. shall not have had any direct involvement in training t1e license '

applicants. Furthermore, only one person with limited involvement (i.e.. who taught more than 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> but who was responsible for no more than 15 percent of the scheduled classroom instruction) may participate in developing the written examination questions. and no one may develop questions for the topics they taught. Facility licensees may still 3ropose alternative approaches for consideration on a case-by-case

) asis.

o All three commenters questioned the need for facility licensees to document the history of every test item pro)osed for use on a licensing examination. even if it was drawn from anotler facility licensee's examination bank.

The staff agreed that the wording in the draft NUREG could have resulted in misunderstandings of what was expected. The staff still believes that it is important to monitor the predictability of the examinations but it lacks the data and the resources to indepenaently 1dentify the source of every test item. This information is readily available to the facility licensee. The staff has clarified the guidance in NUREG-1021 so that test items that a facility licensee obtains from another bank 17

.- ., . . .- . . . . - - - - - - - - - - =

l

! b l

and deposits into its own bank may be treated as " bank" items. provided I l the test items have an equal chance of being selected for use on the i examination. Items from another bank may be treated as new items if i

, they have not been made available for review and study by the license  !

l applicants and there is no basis (e.g.. historical precedent or reciprocal arrangements with the other facility licensee) for the  !

a]plicants to predict their use on the examination. The staff believes  !

tlat this revision will somewhat lessen the burden on facility  !'

! licensees. while enabling the NRC to maintain an acceptable level of oversight.

l  !

i o All three commenters suggested that NUREG-1021 relax the restrictions on l l the duplication of test items from the applicants' audit examination (a practice test given by the facility licensee before the NRC examination) )

if the audit examination was developed independently of the license  !

examination.

Under the traditional examination method, the staff does not have access to the facility licensee's audit examination, so any duplication that occurs between the audit examination and the licensing examination is unmonitored and coincidental. The staff agreed that the proposal was consistent with current practice and changed the NUREG to allow up to five questions to be du written independently (plicated at random i.e.. no interaction if the two between theexaminations examination are l authors) and the facility licensee identifies the duplicates. Licensing examinations that are not developed independently shall not repeat any j

-questions directly from the audit examination l

o All three commenters recommended that NUREG-1021 accommodate facility licensees that have " closed" examination banks (banks not available to a]plicants for study) by relaxing the limit on the number of test items tlat can be drawn directly from the bank.

The staff acknowledged that this is a valid issue, but because this issue is so-complex. the staff does not intend at this point to establish additional guidelines regarding the maintenance of examination banks. As noted earlier, the staff intends to treat all banks as if they are "open" at this time.

o All three comenters objected to the suggestion in draft NUREG-1021 that l facility licensees consider providing an explanation as to why the 4 answer to every written (multiple-choice) question is correct and each of the distractors is plausible but incorrect. The industry thought this was a significant administrative burden that would, in practice.

, become a requirement.

Feedback from the NRC chief examiners indicated that some facility ,

licensees adopted this practice on their own during the pilot examinations: it proved very beneficial and increased efficiency during

the examination reviews. This prompted the staff to include it as an optional activity in draft NUREG-1021. The staff still believes that i

18  !

i L

l I

l the practice has merit but has emphasized in the final NUREG that it is  !

l optional.

l l .o Two commenters recommended that facility 11censees be permitted to use l site-specific task lists in place of the NRC's K/A catalogs when  !

developing the examination outline. The site-specific lists are used to I develop the operator requalification examinations and should also be j acceptable for developing content-valid initial licensing examinations.

Since the mid-1980s, the staff has used the NRC's generic K/A catalogs i

! as the basis for ensuring the content validity of the initial licen' sing i i examinations. The catalogs are based upon an analysis of the licensed I operator's job that was performed by the Institute of Nuclear Power  ;

l Operations. Many facility licensees have conducted their own job task l analyses and developed plant-specific 03erator task lists for use in their SAT-based training programs. Altlough the staff does not uestion that the site-specific task lists would generally be acceptable or ensuring the content validity of the examinations, it believes that i wholesale substitution of the catalogs could cause a decline in l examination consistency. The staff considers the proposal appropriate in limited circumstances on a question-by-question basis; therefore, the guidance in NUREG-1021 has been revised to permit substitutions and additions of specific knowledge and ability requirements on a case-by-case basis.

l

! o Two commenters disagreed with a paragra)h in Appendix A of the draft l NUREG that implied that any question tacen from an open bank would test l at the basic memory level. They argued that questions prepared at a higher cognitive level would not be reduced to the basic memory level if the question bank is large. They strongly recommended that the subject paragraph in Appendix A be deleted.

! Appendix A 3rovides an overview of examination preparation concepts and discusses t1e basis for some of the NRC's examination criteria.

l including the limits on the use of examination banks. The subject l paragraph is considered integral to the discussion and was not deleted.

However. the staff has edited the text to clarify that questions could decrease to the simple recognition level if the item bank is small and available for the examinees to study.

The staff also clarified one of the criteria in draft NUREG-1021 that.

for purposes of examination quality and consistency, required at least 4 half of the questions on the written examination to be written at the  !

comprehension or analysis level. The staff never intended to im)1y that l any question drawn directly from a bank would be counted at anytling other than its face value. A question written at the comprehension or analysis level would count as such. even though the cognitive level at  !

which it tests would decline with use and exposure.

o Two commeaters recommended that the facility licensees be allowed more l than five Gys, if necessary. to review and grade the written examinations. They asserted that relaxing the requirement would only  ;

- l 19 i

i

i e

delay the results to the facility licensee and the applicant and that  !

the delay would not adversely affect the NRC.

]

The staff had traditionally allowed the facility licensee five days to i comment on the written examination, so it decided to retain that time period for the pilot examination program. This was one of the examination criteria on which the chief examiners had been asked to coment. and no comments were made. However, since the proposed change would not affect the examination outcome, the staff decided to increase '

the flexibility in the final NUREG-1021 by noting that the facility licensee should submit the graded examinations within five days, if i practical. l o One commenter indicated that NUREG-1021 does not appear to accommodate utilities that do not want to prepare their own licensing examinations and suggested that utilities be allowed to retain that option.

Because of the unpredictable demand for examinations and the scheduling difficulties that may result from a long-term voluntary program.

SECY-96-123 requested that the Comission approve the staff s pursuit of rulemaking to require that licensees of power reactor facilities prepare the operator licensing examinations in accordance with NUREG-1021. The Commission has delayed making a decision on industry wide implementation of the revised process until it can review the detailed rulemaking plan (Attachment 1) and the additional issues discussed herein.

o One comenter recommended that the facility licensees be held responsible for making examination changes "as agreed upon with the NRC" rather than "as directed by the NRC." The commenter added that this alteration would help ensure that the suggested changes are accurate.

Because the staff fully supports the goal of the facility licensee to ensure the accuracy of the examinations. 1t revised NUREG-1021 as recommended but stipulated that the NRC would retain the final authority to approve the examination.

o One commenter requested that the NRC clarify which facility staff i members would be allowed to observe the operating tests in the '

simulator. NUREG-1021 currently restricts personnel other than the simulator operator from watching the operating test without the approval of the NRC chief examiner. The commenter argued that facility managers i and others deemed necessary by the facility should be allowed access to the examination provided the simulation facility can accommodate them and there is no adverse effect on the applicants.

The staff concluded that this was a reasonable request and revised NUREG-1021 accordingly. Facility management and other personnel deemed necessary by the facility licensee will generally be allowed access to the examination (under security agreements. as appropriate). provided the simulation facility can accommodate them and there is no adverse effect on the applicants.

20

l l o One commenter requested that the NRC clarify the meaning of "significantly modify." The facility licensee had been told that the  :

intent of the question had to be changed. which conflicted with the definition that had been published in GL 95-06 and draft NUREG-1021.  ;

1 The staff concluded that this was a reasonable request and revised  !

NUREG-1021 to indicate that the intent of the question does not have to  :

change to qualify es a significant modification.  !

In summary. the staff believes that the industry generally supports the proposal to shift responsibility for writing the initial licensing examinations to the facility licensees. Some facility licensees actively l lobbied for the change on the basis that it would save them money. Many  !

facility licensees and NEI view the proposed changes as a good first step at  !

increasing the industry's involvement in the examination process and shifting the NRC's role from direct involvement to oversight. Nevertheless, most facility licensees would probably agree that the administrative requirements that were outlined in GL 95-06. refined during the pilot 3rogram, and incorporated in Revision 8 of NUREG-1021 have increased t1e burden of preparing the examinations.

l The issue of industry burden has been a key factor throughout the pilot program. The staff purposely limited the scope of the original process changes to those measures that would help ensure the integrity, consistency.

quality, and security of the licensing examinations. The staff was committed to maintaining an effective and efficient licensing program and realized that 1 the revised process, which would have relied on voluntary industry support.  !

l would never succeed if it was unnecessarily burdensome on the facility licensees. In keeping with its original commitment, the staff believes that l

it has taken reasonable measures to address the industry's concerns and recormiendations without sacrificing the effectiveness of the examination l process. ,

L Pilot Enhancements and Justifications l To assess the effectiveness of the new examination process. NRR asked the NRC l examiners who were involved with the pilot examinations to respond to a questionnaire regarding various aspects of the process. The first and most l

significant problem observed and re)orted by the exam 1ners was that the quality and technical accuracy of t1e examinations submitted by several

! facility licensees fell short of the NRC's expectations. Moreover some of

the early pilot examination results (i.e. more question deletions and answer l key changes than expected) reflected the need for greater emphasis in that l area.

I Consequently, when NRR prepared draft Revision 8 of NUREG-1021, it incorporated a number of clarifications and changes that improved the framework established in GL 95-06. In January 1996, the staff issued a

preliminary draft of NUREG-1021 to the regions for review and comment before soliciting comments from the public and the industry the following month. The draft revision that was issued for public and industry comment contained the following process improvements

21

\

l l

q l

o~ The existing guidance (Revision 7 of NUREG-1021. Revision 5 of NUREG/BR- 1 0122, and GL 95-06) did not specify what percentage of the questions on I the written examination should test-the applicants at the comprehension 1

. and analysis levels of knowledge. Many_ examiners had expressed concern  ;

that the lack of guidance in this area would lead to inconsistency and a -]

decrease in the level of examination difficulty. Facility licensees  ;

questioned whether examiners had unwritten ~ guidance that required them i to make the examination more difficult by revising or replacing simple I memory questions. )

As noted earlier. NRR had no desire to increase the average difficulty of the examinations, but it also did not want the level of difficulty to  ;

diminish. Therefore, to better define the.aapropriate level of l knowledge for the new examination process. t1e staff reviewed the '

written examination audits aerformed in the previous two years to determine an average distri)ution of questions by knowledge level. The

. staff cmcluded that approximately half of the questions on the examinations reviewed were written at the comprehension or analysis level and that the remaining questions tested simple memory. On the basis of that information, the staff established a criterion in draft NUREG-1021 that at least half of the examination questions be written at the comprehension or analysis level.

o The supplemental guidance that NRR 3rovided to the regions concurrent with GL 95-06 included interim QA clecklists to assist the NRC examiners in reviewing and: identifying deficiencies in the draft examinations.

The checklists were edited and incorporated in the preliminary draft of NUREG-1021 and edited again in response to regional comments. The draft NUREG-1021 that was issued for public comment included QA checklists for the examination outline, the written examination, both the simulator and walk-through portions of the operating test, and the written examination

_ grading.

o -The existing guidance (Revision 7 of NUREG-1021 and GL 95-06) did not saecify a limit on the number of JPMs that could be repeated in walk- l t1 rough tests administered on successive days during the same initial licensing examination visit or from one visit to the next. Many examiners had expressed concern that the lack of guidance in this area could compromise examination integrity and lead to inconsistency. Other ,

examiners argued that no repetition should be permitted because it would l increase the predictability of the test. Facility licensees would be  ;

tempted to repeat as many JPMs as possible in an effort to save  !

resources.

y:

To strike a balance. the staff concluded that a 30-percent limit would i l be appropriate. This figure was consistent with the written examination i

limit of 25-percent duplication of questions from examinations, quizzes.  !

[ or tests administered to the license applicants or from the past two NRC l licensing examinations at the facility.

l i

i  !

j 22  !

i j

l , . . - - - . -

]

After issuing draft NUREG-1021 for public and industry comment, the staff 4 identified a number of items that merited clarification and improvement. In addition to reviewing the industry's ccmments and recommendations (as previously noted). the staff discussed all of the following changes with NEI and other industry representatives during a public meeting held in the NRC's offices on April 18, 1996. Except as noted below. the industry l representatives did not object to the additional changes.

i o To limit the number of technical inaccuracies a problem that affected several of the pilot written examinations. the staff revised NUREG-1021

to require the NRC chief examiner or another examiner to independently review and verify the technical accuracy of a sample of the written examination questions. The number of questions verified would depend

upon the expected quality of the examination and the time available l before the scheduled review with the facility licensee.

' As a final check of the technical accuracy of the written examination.

the staff revised NUREG-1021 to encourage the facility licensee to

administer the NP.C-approved examination (under security agreements) to one or more licensed personnel who were previously uninvolved in developing the examination. Although feedback from the NRC chief examiners indicated that some facility licensees had taken it upon themselves to validate their examinations in this manner during the pilot program, the facility representatives at the meeting objected to making it a requirement for all facility licensees. Because the staff lacked sufficient information to cuantify the value added by this measure, it chose to recommend it as a good practice.

o Late in the pilot program. an NRC chief examiner who was reviewing a draft examination detected that the facility licensee had used a written examination outline that was identical to one that a sister facility had prepared several weeks earlier. The staff had not anticipated that such a problem might occur but immediately realized that this was a significant loophole in the process. Allowing facility licensees to ,

reuse the same examination outline would definitely improve efficiency. '

but it would also jeopardize the integrity-of the examination process by increasing the predictability of the examinations.

To address this problem. the staff included in NUREG-1021 a caution for examination authors and reviewers to ensure that the outline used during l successive audit and licensing examinations does not become repetitive  ;

and predictable. If a facility licensee proposes to use an outline that  :

was previously used at another facility. it shall identify the source of l the outline and explain what effect its reuse is expected to have on l examination integrity.  ;

o NUREG-1021 has for some time included checklists for use by NRC l examiners in verifying that their simulator scenarios contained the ,

required number of transients and ensuring that the scenarios would '

enable the examiners to evaluate the applicants on every required competency. There were no specific instructions for filling out the checklists. nor were they required to be retained in the examination 23

file. Concerns regarding the quality and consistency of the operating tests prompted the staff to incorporate instructions for completing the transient and competency checklists and a requirement for the NRC regional offices to retain the checklists in the examination files.

o NUREG-1021 has alwayr endorsed the goal of having the licensing examinations ready to administer approximately a week before they are scheduled to be given. Furthermore, the staff has always encouraged the regions to keep NRR informed if significant problems are encountered with an examination. - The pilot program illustrated and magnified the importance of both concepts. Therefore. the staff clarified the guidance in NUREG-1021 to limit the number of last-minute changes and to require the regional office to consult NRR if it appears that the examination cannot be made to conform to the examination standards at least five working days before the scheduled examination date.

Furthermore, the examinations should be postponed if problems are identified at the last minute.

o In addition to implementing the pilot examination process. the staff proposes using Revision 8 of NUREG-1021 as a vehicle for activating Revision 1 of NUREG-1122 and NUREG-1123. the NRC's K/A catalogs which were published in August 1995. The catalogs were revised to make them more current and reorganized to make them more consistent with each other and easier to use. As part of the catalog revisions, the system-generic K/As in both catalogs were combined with the plant-wide generic K/As. and new vendor-specific emergency and abnormal plant evolutions were added to the pressurized water reactor catalog (NUREG-1122). These changes would have altered the balance of the written examinations unless the sample plans in NUREG-1021 were also revised to compensate for the changes. Because of an oversight. the staff neglected to enter the changes before issuing the draft NUREG-1021 for public and . industry comment.

o- When NRR reviewed the regional office operator licensing programs during February and March 1996, it discovered that some of the examiners had misinterpreted the intent of NUREG-1021 regarding the design of the operating tests for SR0 ap)licants limited to fuel handling. The staff discussed the issue with t1e regions and agreed that strict compliance with the existing guidance would overtest those license applicants.

Therefore, the guidance in NUREG-1021 was changed to more closely align thescogeoftheoperatingtestwiththerangeofresponsibilitiesfor SRO fue handlers o Whenever a question is deleted from the 100-point written examination, it results in fractional grades. When the grade is above 79.5 percent.

it raises a question regarding the staff's policy on rounding-off because it means the difference between receiving a license or a denial.

To address this problem. the staff has revised NUREG-1021 to indicate that the passing grade on the written examination is 80.00 percent.

o Although the pass rates of the pilot examinations were comparable with pass rates for NRC-written examinations, the rate at which applicants 24

requested that the NRC informally review their examination failures increased unexpectedly during the pilot program. These requests not only placed a significant burden on the staff to resolve the applicants' issues but also raised concerns regarding the effect that any additional cuestion deletions and answer changes might have on the other licensing cecisions that had already been made. 1 Consequently, the guidance in NUREG-1021 was revised to afford the NRR staff greater discretion in determining the method for resolving applicant appeals (i.e., every appeal does not necessarily have to be reviewed by a panel of examiners as long as objectivity and fairness are maintained) and to delay the issuance of licenses to applicants who passed the written examination with insufficient margin to guarantee l that the licensing decision will be sustained if additional questions '

are deleted or changed upon appeal. Furthermore. to minimize the number of appeals. the staff revised NUREG-1021 to encourage facility licensees l

to solicit and address concerns from individual license applicants during the process of grading the written examinations.

, o Problems encountered while reviewing and resolving the operating test

appeals that were received during the pilot program prompted the staff l to reevaluate the documentation and grading requirements in NUREG-1021.

l As a result. the staff has revised NUREG-1021 to permit NRC examiners to i downgrade more than two simulator test rating factors for the same error, provided the error caused serious safety consequences for the i plant. The staff also increased the documentation requirements for the l simulator test so that the NRC examiners would have to describe any

! error that prompted them to downgrade the applicant on any rating factor.

During the meeting of April 18, 1996, the staff also discussed the possibility of permitting NRC examiners to recommend a failure on the simulator test even if the a)plicant did not fail the test on the basis of the competency scores. T1e industry re)resentatives did not believe that such a change would be necessary if t1ere were no limit on the number of competencies that the examiner could downgrade for a single, serious error; however. they did not object to the change in principle.

Therefore, the staff has included in NUREG-1021 a provision for NRC i

examiners to recommend such a failure if the applicant committed an error with serious safety consequences for the plant or the public. To limit its use. the region must obtain written concurrence from the Chief. HOLB, before completing the licensing action.

l l The following issues were raised by NRC examiners on the pilot examination

feedback surveys but did not result in specific changes to the examination l methodology described in NUREG-1021.

. 9 To minimize NRC resource requirements the staff had encouraged the regions to review and prepare for the pilot examinations without conducting a preparatory visit to the site. However, feedback from those examiners who did not conduct such a visit indicated that it would have facilitated the review and resulted in fewer problems with the 25

- . . - - _ -. = - . - - _- -

• l I

4

, examination. They recommended that a preparatory visit be required for  !

every examination. Although NRR has not changed the guidance in  !

NUREG-1021 (i.e.. the pre]aratory visit is optional at the discretion of I regional management), it 1as acknowledged the value of conducting at l

least part of the examination review and validation at the facility.

The average NRC resource expenditures during the pilot examinations I indicated that the trips can be made within the original budget model.

4 i

i o Several ARC examiners recommended that the required due dates for the examination materials from the facility licensees be advanced so that

! the examiners have more flexibility in scheduling their time for the examination review. The staff did not change the guidance in NUREG-1021 because it already permits the regional offices to work with the facility licennes in determining the appropriate due dates and because a generic change would increase the scheduling burden on all facility

licensees. It would not be appropriate for an examiner to demand the examination 60 days in advance because it was required by NUREG-1021.,

only to have the examination sit untouched in the office for 30 days because the examiner was involved with another project.

o When the staff developed the pilot examination process. there was some debate as to whether the facility personnel who administered the written examination should be permitted to answer the a)plicants' questions regarding the intent of specific questions on t1e examination. The pilot guidance allowed the facility to answer the applicants' questions.

with caution but required every question and response to be documented for subsequent review by the NRC. The policy was evaluated on the pilot examination questionnaire that every NRC chief examiner was required to

complete. A minority of the examiners recomended that the policy be changed to prohibit the applicants from asking que<,tions during the written examination..as is the policy on the GFE. However, most of the i examiners concurred with the original pilot policy, and none of the comments indicated that the policy had created any problems with examination integrity. Therefore, the staff saw no reason to revise the guidance in NUREG-1021.

4 o Some examiners noted that the pilot program personnel restrictions had hampered the facility licensees' ability to develop the examinations

. because they could not assign their best people to the task. This issue was also addressed in the industry comments. as previously discussed.

Examination Results Since Issuance of SECY-96-123 The staff completed the last of the pilot examinations on April 5,1996, and resumed conducting the licensing examinations using the traditional 1 development process. After April 12. 1996. when the Executive Director for i Operations informed the Commission of the staff's intent to continue the pilot I process beyond the period previously discussed with the Commission, the staff proceeded to contact those facility licensees whose examination requests had not yet been assigned to a contractor to determine if they were interested in developing the examinations using the pilot methodology. The first of those 26

. - _ _ - - = - .- . -_. - - . .

examinations was administered on August 19. 1996, and the results were not available in time to report them here.

Since SECY-96-123 was prepared. the staff has used the traditional examination methodology to prepare licensing examinations for applicants at 14 power reactor facilities. The results of the completed examinations are sumarized below: four of the examinations were completed late in August. and the results were not available at the time this paper was ]repared. The licensing examination results for fiscal year 1995 and t1e 22 pilot examinations conducted between October 1.1995 and April 5,1996, are provided for comparison.

Pass Rates' R0 R0 R0 SR0 SR0 SRO Examinations Written Operating TotJ Written Operating Total 39/40 39/39 39/40 45/48' 43/48 41/48' Since SECY-96-123 (98%) (100%) (98%) (94%) (90%) (85%)

Pilot 49/54 50/54 45/54 86/92 87/91 3 83/923 Examinations (91%) (93%) (83%) (93%) (96%) (90%)  ;

Fiscal Year 1995 94% 98% 92% 95% 95% 92%

1. Each box indicates the numbe of applicants who passed that part of the examination over the number of applicants who took it.

2. One of the SR0 applicants who failed the written examination has filed an appeal that was not yet resolved at the time this paper was prepared.

3. Two applicants who were reported as areliminary failures in SECY 123 filed appeals that resulted in t1e failures being overturned.

Summary The staff believes that the revised examination process can be implemented at all power reactor facilities with the same level of direct NRC resources as is currently allotted to the operator licensing program. Additional resource investment will be made during FY97 and FY98 to increase the pool of NRC examiners available. Full implementation of the revised process will enable the staff to eliminate the use of operator licensing contractors (with the exception of those who develop and grade the GFE) that have historically cost the NRC (and the industry) approximately $3 million to 54 million per year.

The Staff expects that the change could result in a net resource savings to the facility licensees over time because they will be able to prepare the l examinations more efficiently than the NRC or its contractors.

l l The overall results of the pilot program indicated that the examinations prepared by facility licensees. subject to review, revision where appropriate, and approval by NRC staff examiners, were generally of the same caliber as examinations prepared by NRC or contract examiners. The pilot examinations 27 l

1 also appeared to be equally effective at identifying applicants who had not mastered the job requirements well enough to become licensed R0s or SR0s.  :

Furthermore, the pass rates for R0s and SR0s on the written examinations and

' operating tests compared favorably with the pass rates for examinations prepared by the NRC or.its contractors.

t However, despite the mitigating actions previously noted the revised  ;

. examination process is much more dependent upon the abilities of the NRC chief i examiners to identify weak examinations and work with the facility licensee to  ;

upgrade the examination. Furthermore, it is possible that the pressures to . 1 conduct examinations on schedule and the reduced involvement by the NRC in the .

examination development process could lead to examinations that reflect the >

biases of the facility licensees and that discriminate at a lower level than is currently the norm. Moreover, it is reasonable to expect that industry- 1 wide implementation of the revised process will increase the chance of future  :

examination compromises as a result of the larger number of individuals who would require access to the examinations and the additional pressures on facility employees to ensure the success of the applicants.

Although a number of the examinations submitted by the facility licensees did not meet NRC standards for level of knowledge or difficulty, the NRC chief examiners effectively worked with the facility licensees to revise the examinations so that they would discriminate at the appropriate level. To put '

this situation into perspective. it has not been too unusual for contractor- i developed examinations to fall short of the NRC's requirements and expectations. The chief examiner often directs the contract examiner to make i numerous corrections and changes before reviewing the examination with the i facility licensee, only to have the facility reviewer identify a host of additional errors and inconsistencies.

The staff has advised facility licensees that the NRC expects the technical quality of examinations to remain high and the number of post-examination changes to be low. Similarly, regional examiners and managers have been instructed not to approve or administer any examination that does not adhere to NRC standards for content, format. quality. level of knowledge. and level of difficulty. The 0A checklists in NUREG-1021 have been modified to maintain examination consistency and to facilitate the efforts of NRC examiners to detect deficient examination materials. The use of any facility-prepared examination that is found to be unacceptable will be delayed or cancelled, if necessary, until the facility licensee and the NRC can upgrade or replace the examination.

l The supplemental pilot examination guidance that the staff 3rovided to the regional offices (and placed in the PDR) in conjunction wit 1 GL 95-06 directed the regions to document in the examination report any significant problems encountered during the examination development, administration, and grading.

This guidance has also been included in Revision 8 of NUREG-1021. The regions are expected to address any serious deficiencies in the examinations submitted by the facility licensees and any security concerns during the entire

! examination process. If appropriate, the region will request the facility l licensee to describe the actions it will take to improve future performance.

! If the facility licensee is unable or unwilling to prepare appropriate 28

I 1

licensing examinations, the NRC has the responsibility and authority to develop the examinations in the traditional manner.

Overall. the staff does not expect that the pro)osed revisions to the o)erator l licensing program will significantly increase t1e risk to the life and lealth

of.any' individual members.of the public or that they will be a significant addition to other societal risks. The staff believes that the licensing decisions that were made on the basis of-the pilot examinations were as valid as those made using the traditional examination process. Furthermore. the i

changes that the staff has made in NUREG-1021 as a result of lessons learned L

during the pilot program and industry comments submitted in response to the staff's sol 1 citation (which were previously discussed in detail) should only improve the effectiveness of the examinations if and when the revised process is implemented on an industry-wide basis.

The NRC will continue to set the standard for operator performance by ensuring thatthefacilitylicenseesmaintainappropriatelyhighstandardsfor l examination deveiopment. The staff will remain actively involved in the i L operator licensing process by reviewing and approving every written i examination and operating test before it is administered. Unacceptable examinations will be revised and delayed if necessary, until the resources are available to upgrade the examinations. Furthermore, the fact that NRC examiners will continue to independently administer the operating tests will allay concerns regarding bias and conflicts of interest on that portion of the. .

licensing examination having the most operational validity and requiring the-  !

most subjective judgements. l 1

The safety consequences of occasionally administering a substandard l examination are probably minimal. By the time the applicant takes the NRC l licensing examination, he or she will have completed a comprehensive, systematically constructed and implemented training program designed to maximize the likelihood of success on the examination. The license training )

programs typically include nun rous written examinations and performance  !

demonstrations that culminate in certification by the facility licensee that '

the applicant has successfully completed the facility licensee's requirements to be licensed as an R0 or an SRO. The effectiveness with which the facility licensees screen out those trainees that have not mastered the job requirements has been evident in'the high pass rates on the licensing examinations during the past several years. In most cases. the NRC examination is simply a confirmation of the facility licensee's decision to submit the application.

Historically, even those applicants who fail the NRC written examination generally score above 75 percent. The average grade for the 11 applicants who failed a pilot examination was 76.3 percent. This statistic indicates that

the vast majority of ap)licants, including most of those who fail the

! licensing examination. lave attained an adequate level of knowledge so as not t

to pose a significant threat to reactor safety. Furthermore, any applicant

( who does become licensed or the basis of an examination that discriminated at

! a lower than normal level would be backed up by other crew members who could 4 prevent or remedy any operating errors. Although there is nothing to prevent j a facility licensee from forming an operating crew entirely from newly 29 4

i i

l i

I

l b

. E licensed operators. most facility licensees integrate their new operators with existing crews who possess significant operating experience.  ;

i e

i i

1 l

v i

i l

}

t I 1 i

l 1

l i i I

l

)

i 1

I t

l l

4 30 ,

1 i

1