ML20216F776

Requests Concurrence on Attached Rulemaking Plan Package for Amend to 10CFR55 to Change Licensed Operator Exam Requirements

ML20216F776
Person / Time
Issue date:	08/26/1996
From:	Morrison D NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES)
To:	Lieberman J, Olmstead W 8, Russell W NRC (Affiliation Not Assigned), NRC OFFICE OF ENFORCEMENT (OE), NRC OFFICE OF THE GENERAL COUNSEL (OGC)
Shared Package
ML20216F483	List: ML20216F480 ML20216F496 ML20216F508 ML20216F518 ML20216F534 ML20216F542 ML20216F561 ML20216F582 ML20216F587 ML20216F606 ML20216F627 ML20216F726 ML20216F751 ML20216F759 ML20216F776 ML20216F809 ML20216F867 ML20216F949
References
FRN-62FR42426, RULE-PR-55 AF62-1-025, AF62-1-25, NUDOCS 9709120101
Download: ML20216F776 (46)

Text

.

s

=4

• • • vy A P 4 '2. 1 y* k_ UNITED STATES -

Q j

NUCLEAR REGULATORY COMM!SS!ON WASHINGTON, D.C. 20066-0001

1. 1. 4

• • • • • - August-26, 1996 MEMORANDUM TO: James Lieberman, Director >

Office of Enforcement William J. Olmstead, Associate General Counsel for Licensing and Regulation Office of the General. Counsel William T. Russell, Director Office of Nuclear Reactor Regulation FROM: David L.'Morrison, Director . -

Office of Nuclear Regulatory Research ,

p -'

SUBJECT:

RULEMAKING PLAN - CHANGES 10 10 CFR 55 - AMENDING INITIAL OPERATOR LICENSE EXAMINATION REQUIREMENTS Your concurrence is requested on the attached rulemaking plan package. This action is on the Chairman's tracking list with an aggressive schedule, therefore, please respond promptly.

Backaround:

Section 107 of the Atomic Energy Act of 1954, as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Current regulations in Part 55 dealing with initial license examinations for reactor operators (RO) and senior reactor operators (SRO) do not state who will write, administer, or grade the examination; however, the NRC has traditionally performed those-tasks itself or through contract examiners. On March 24, 1995, the staff, in SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program," informed the Commission of its intent to revise the operator licensing program to allow greater participation by facility licensees and to eliminate contractor assistance in this area.

This action will allow staff to eliminate between $3M and $4M annually on contractor support for examination preparation and administration. On April  ;

18, 1995, the Ccmmission approved the staff's oroposal- to initiate a transi-tion process to revise the operator licensing program, and directed the staff to carefully consider experience from pilot examinations before full implemen-tation. On August 15, 1995, the staff issued Generic Letter (GL) 95-06,

" Changes in the Operator Licensing Program," outlining the revised examination development process and soliciting volunteers to participate in pilot examina-tions to_ evaluate and refine the methodology.

9709120101 970904 62 42426 PDR

. _ _ _ -_________a

s c.

J. Licberiaan et at. Between October 1, 1995 and April 5, 1996, the NRC conducted a pilot program in which it reviewed and approved 22 operator licensing examinations that had been written by facility licensees in accordance with the current examination development guidance (NUREG-1021. Rev.-7, Supplement 1), as supplemented by the pilot examination guidance contained in GL 95-06. The staff described the results of-the pilot examinations in SECY-96-123, " Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996, and briefed the Commission during a public meeting on June 18, 1996. Based on the results of the pilot program, the staff recommended that the Commission approve the implementation of the-new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to draft the operator licensing examinations in accordance with NUREG-1021, " Operator Licensing Examination Standards for Power Reactors." In a staff requirements memorandum (SRM) dated July 23, 1996, the Commission directed the staff to develop a detailed rulemaking plan to justify the changes that may be neces-sary to 10 CFR Part 55. The SRM also directed the staff to specifically.

address several issues about the pilot examination program and the proposed changes in the examination process.

The rulemaking plan and the response to the SRM's specific requests, which were prepared by the Office of Nuclear Reactor Regulation and reviewed and approved by the Office of Nuclear Regulatory Research, are attached to the enclosed Commission Paper One issue I would appreciate specific comment on is that of whether the rule should contain an explicit reference to NUREG-1021 or alternately require an operator licensing program acceptable to the NRC. If the former course is taken that will have the effect of codifying the NUREG and if we wish to change it later based on experience with implementing the new rule a rulemaking would be required. If the later course is taken a regulatory guide would be issued referencing the NUREG and it would only be necessary to change the regulatory guide.

The following is a summary of this request:

1.

Title:

Rulemaking Plan, " Changes to the Operator Licensing Process at Power Reactors (Part 55)."

2. Task leader: Harry S. Tovmassian (415-6231)

3. Steerina Group Members: None

4. Enhanced Public Participation:

This rulemaking will use electronic bulletin boards, as appropriate to enhance input from the public.

5. Level 1 or 2 Item of Compatibility for Aareement States: No

6. Reauested Action:-Office Concurrence J

b. c o

J. l.ieberman et at. 7. Reauested Completion Date: August 30, 1996

8. Resources'and Coordination:

Estimated resources to develop 'this rulemaking are .5-FTE. No contractor. support is needed. A copy of this concurrence package has been forwarded to the Office of the Controller for coordination of resource issues and to the IG-for information.

Attachment:

Memorandum to Taylor from Morrison w/ enc 1s.

cc w/ attachment:

R. Scroggins, OC H. T. Bell, 0lG G. Cranford, IRM 0, Meyer,'ADM

.(*

i

.b

. c:

~~

~

. 1

1. ')

4 e

EDB:~ The Commissioners

EBQM

' James M.. Taylor, Executive Director for Operations ~

SUBJECT:

RULEMAKING PLAN FOR AMENDMENTS TO 10 CFR PART 55.TO CHANGE LICENSED OPERATOR EXAMINATION-REQUIREMENTS PURPOSE:

ETo provide additional information regarding the initial operator licensing examination pilot process as directed in a Staff Requirements Memorandum:(SRM) e of July 23,:1996 (M960618A),"and to request.the Commission's approval of the rulemaking plan to amend 10 CFR Part 55 to require power reactor facility 4

. licensees to prepare and administer in accordance with NUREG-1021,." Operator.

Licensing Examination Standards for Power Reactors," the written examinations

, and operating tests that the NRC uses to determine the competence of operator

. license applicants.

BACKGROUND:-

On March 24, 1995, SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program,"-informed the Commission of the staff's intent to revise.

the operator licensing program to~ allow greater participation by facility licensees and to eliminate contractor assistance in this area. :This action will- allow staff. to eliminate between $3M and $4M on contractor support- for

examination preparation and administration. On April 18, 1995, the Commission approved the staff's proposal to initiate a transition process to revise the-

. operator licensing program and directed the staff to carefully consider

experience from pilot examinations before full-implementation of the changes..

On August 15, 1995,:the staff issued Generic Letter (GL) 95-06, " Changes 11n *

the Operator Licensing Program," outlining the revised examination development 1 process.and soliciting volunteers to participate in pilot examinations to - ,

.  ; evaluate;and. refine the methodology.

ICONTACT:- -

Stuart Richards, NRR-(301)415-1031 ,

g I 4

~~

v ?m - , v- - - .w,- - e-,

e The Commissioners <

Between October 1, 1995, and April 5, 1996, the NRC reviewed and approved 22 operator licensing examinations that had been prepared by facility licensees in accordance with the current examination development guidance (NUREG-1021, Rev. 7, Supplement 1) as supplemented by the pilot examination guidance contained in GL 95-06. These examinations were then used to test 146 reactor operator and senior reactor operator applicants. Facility licensees from each NRC Region, operating reactors built by- all power reactor vendors, participated in the voluntary pilot program.

The staff described the results of the pilot examinations in SECY-96-123,

" Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996, and briefed the Commission during a public meeting on June 18, 1996. On the basis of the results of the pilot program, the staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to prepare the operator licensing examinations in accordance with NUREG-1021, " Operator Licensing Examination Standards for Power Reactors," which has been revised to accommodate the new process. In an SRM dated July 23, 1996, the Commission authorized the staff to continue the pilot process through July 1997 and directed the staff to develop a detailed rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55. The SRM also directed the staff to specifically address several issues regarding the pilot examination program and the proposed changes in the examination process.

~

DISCUSSION:

Section 107 of the Atomic Energy Act of 1954 as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55 to pass an examination satisfying the basic content requirements specified in the regulation. Part 55 does not state who must prepare, administer, or grade the examination; however, the NRC has traditionally performed those tasks itself or through contract examiners.

On the basis of the results of the pilot program as discussed in SECY-96-123, the' staff now believes that 10 CFR Part 55 should be revised to require power reactor facility licensees to prepare their initial operator licensing examinations. Specifically, the staff recommends that Subpart E, " Written Examinations and Operating Tests," of 10 CFR Part 55 be amended to require all power reactor facility licensees to prepare the initial operator licensing examinations in accordance with NUREG-1021. The licensee-prepared examinations will be subject to review, modification, and approval by NRC examiners before the examinations are administered. The regulation will provide that the NRC may prepare the examinations in lieu of accepting or-modifying an examination prepared by the facility licensee. The changes would not apply to non-power reactor licensees.

The enclosed rulemaking plan for this proposed change has been developed in accordance with the guidance in NRC Management Directive 6.3, "The Rulemaking Process." As directed in the SRM of July 23, 1996, the rulemaking plan

i The Commissioners -

3-The enclosed rulemaking plan for this proposed change has been developed in accordance with the guidance in NRC Management Directive 6.3, "The Rulemaking Process." As directed in the SRM of July 23, 1996, the rulemaking plan addresses the potential impact of the proposed rule on facility licensees and discusses why this rule will not constitute a backfit.

The second enclosure discusses the following additional items as directed by the SRM:

e The pros, cons, and vulnerabilities associated with implementing the proposed examination program changes on an industry-wide basis (including a review of the impact on licensee and NRC resources, the

, potential effects on reactor safety, the impact of proposed NUREG-1021 revisions, potential public aerceptions, and concerns regarding this approach raised by staff mem)ers),

e A discussion of the degree of acceptance of this new approach to operator licensing by facility licensees (including a discussion of industry comments received on draft NUREG-1021 and an explanation of the staff's response to those comments).

l e An explanation of enhancements to the new licensing examination process resulting from the review and feedback from the pilot program results (including a justification for subsequent proposed changes to the operator . licensing program),

e. The results of initial operator license examinations that have been given both under the pilot program and under the current program since SECY-96-123 was prepared (including trends observed).

RECOMMENDATION:

That the Commission --

1. Approve the development of the rule as described in the enclosed-Rulemaking Plan.

2. Approve the implementation of- Revision 8 of NUREG-1021 on a voluntary basis until the rulemaking is complete.

i t w e

i lThe Commissioners - COORDINATION:- --

- The Office of the General Counsel has no-legal objection to this initiative, and the Offices of Nuclear Reactor Regulation and Nuclear Regulatory Research concur in this rulemaking plan, i

Janes M. Taylor Executive Director for Operations

Enclosures:

1. Rulemaking Plan for Change.: to 10 CFR'Part 55

2. - Response to SRM dtd July 23, 1996 (M960618A)

_ _ _ _ _ _ _ _ _ _ = _ _

RULEMAKING PLAN CHANGES TO THE OPERATOR LICENSING PROCESS AT POWER REACTORS (10 CFR PART 55)

1. Rukaround )

On ilarch 24, 1995,.SECY-95-075,-" Proposed Changes to the NRC Operator Licensing Program," informed the Commission of the staff's intent to revise the operator licensing program to allow greater participation by l facility licensees and to eliminate NRC contractor assistance in this area. This action will allow staff to eliminate between $3 and $4M on '

contractor support for examination preparation and administration. On April 18, 1995, the Commission approved the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full implementation of the changes. On August 15, 1995, the staff issued Generic Letter (GL) 95-06, " Changes in the Operator Licensing Program," which outlined the revised examination development process that would permit facility licensees to prepare the initial operator licensing examinations. GL 95-06 also solicited volunteers to participate in pilot examinations to evaluate and refine the methodology.

Between October 1, 1995, and April 5, 1996, the NRC conducted 22 initial operator licensing examinations using the new methodology. The staff j described the results of the pilot examinations in SECV-96-123

" Proposed Changes to the NRC Operator Licensing Progran.," dated June 10, 1996, and briefed the Commission during a public meeting on June 18, 1996. On the basis of the results of the pilot program, the staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to prepare the operator licensing examinations in accordance with NUREG-1021, " Operator

- Licensing Examination Standards for Power Reactors." In a staff requirements memorandum dated July 23, 1996, the Commission directed the staff to develop a detailed rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55.

11. Proposed Chanae

Revise Subpart E, " Written Examinations and Operating Tests," to include examination development requirements for facility lictnsee

:.

Reculatory Issue The pilot program demonstrated that the revised examination development process can be both effective and efficient. Comments from the NRC staff and industry personnel who participated in the pilot examinations were generally favorable. The quality of the administered examinations (as modified by the NRC) and the performance of the individuals who took the examinations were comparable to the quality of and the performance 1

on examinations written by the NRC staff or its contractors. - The fact that some.of the draft examinations submitted by facility licensees required significant rework illustrated that many facility staffs did

- not-fully understand the criteria for writing an NRC examination. Both the--industry and'the NRC staff agree that,~with trajning and experience, the industry should gain proficiency in preparing the examination materials, c The staff believes-that the new process should be made mandatory for all power reactor. facility licensees. The staff considers implementation of the new process on a voluntary basis alone unworkable over the long

. term. With the elimination of contractor support, the staff will no

longer have sufficient examiner resources to write all examinations consistent with the scheduling needs of facility licensees. This resource problem is further compounded by the unpredictable nature of 2

.the examination workload.

The staff briefed the Committee To Review Generic Requirements (CRGR) after the pilot program was conplete.- In the minutes of its meeting,

. CRGR-agreed that, with regard to the technical and safety aspects, the proposed changes represented a reason nie and workable alternative approach. The committee members unanimously endorsed sending the proposal forward to the Executive Director _ for Operations (EDO) for consideration. However, the CRGR also requested OGC to consider further the procedural question relating the staff's original plan to implement

, the revised process by generic letter.

The staff subsequently met with OGC to resolve CRGR's concern and

concluded that requiring all facility licensees to prepare the examinations, a task long performed by the NRC, would require rulemaking s

or the issuance of orders to each facility licensee. Therefore, the staff recommended in SECY-96-123 that the Commission approve the staff's

[~ pursuit of rulemaking to require power reactor facility licensees to L prepare the operator licensing examinations in accordance with NUREG-1021,_while continuing the pilot process on a voluntary basis.

Current Rule Reauirements Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications-of individuals applying for an-operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are. required by 10 CFR Part 55 to pass an examination satisfying the basic content requirements that are also specified in-the

-regulation.

Specifically. 10 CFR 55.31(a)(3) requires the applicant to submit a written request from an authorized representative of the facility icensee that the written examination and the operating test be _

_ administered to:the applicant. Furthermore, 10 CFR 55.33(a)(2) states '

that the Commission will approve an initial application for a license if it finds that the applicant has passed the requisite written examination 2

i

- . - . - . . - -- ._ , - . ~ . - - - - - . - . _ . - - -

and' operating" test-in accordance with SectionsL55.41-and 55.45 or 55.43 and 55.45. These examinations and tests determine whether the applicant for. an operator's license has learned to operate a facility competently

and-safely,:and additionally.-in the case of a senior operator, whether the appil: ant has learned to direct the-licensed activities of. licensed 2 operators competently and safely.

! Subpa: t E ." Written Examinations and Operating Tests," includes specific -

[

-sampling requirements for the content of written examinations (10 CFR 55.41 and 55.43) and operating tests (10 CFR 55.45).

- Although Part 55 is ' silent regarding who will- prepare, administer, and grade the written examinations and operating tests, the NRC or-its e contract examiners have traditionally performed those tasks.

Preliminary Reaulatory Analysis

. 1. Draft Rule Lanaunae The first sentence in Section 55.33(a)(2) would be revised as follows (new language is underlined):

"The applicant.has passed:the requisite written examination and  :

, ' operating test in accordance with Sections 55.40. 55.41, and 55.45 or 55.40. 55.43, and 55.45.

Section 55.40, " Implementation," would be added to Subpart E as follows: ,

"(a) Power reactor facility licensees shall prepare and administer the required site-specific written examination and operating tests in accordance with the policies and guidelines in NUREG-1021,

' Operator Licensing Examination Standards for Power Reactors."

l -(b)- The Commission will review, modify as--necessary, and approve in h accordance with NUREG-1021, the written examinations and operating tests-prepared by the facility licensees. In lieu of accepting the written examination and operating tests prepared by the facility i licensee, the Commission may prepare the examination and-tests.

(c) The Commission will prepare and administer the written examinations and operating tests at non-power reactor facilities." .

2. Imoact of the= Rule Chanae on Facility Licensees The NRC currently. depends on NRC employees and contractors to prepare and administer the initial operator licensing examinations.

- required by 10 CFR Part 55. :In accordance with 10 CFR 170.12(i),

' the cost of NRC time spent.and any related contractual costs are ,

billed directly'to the facility licensees that receive.the

' examination services.

.- 3

~ , e y-& -~n n v g -- --s, + gs,s-- a

.4 w , , ~<,n-

Under the proposed change, each power reactor facility licensee will assume responsibility for preparing the site-specific initial o)erator licensing examinations at its facilities, thereby allowing tie NRC to discontinue the use of contract examiners for that purpose. Facility licensees will prepare and submit proposed examinations (including the written examination, the walk-through, and the dynamic simulator tests) to the NRC in accordance with the instructions in NUREG-1021. The NUREG has been edited to accommodate the revised examination process and is the same procedure that the NRC examiners or its contractors would use if they were to develop the examinations. In conformance with Section 107 of the AEA, which requires the Commission to prescribe uniform conditions for licensing operators, the NRC will not consider alternative testing methodologies.

The training staffs at power reactor facilities already have the basic knowledge, skills, and abilities necessary to evaluate operator performance and develop test items for the initial licensing examination. During the mid-to-late-1980s, the industry's emphasis in the training area increased significantly. All power reactor licensees established formal training programs that were based on a systems approach to training (SAT) and accredited by the National Academy for Nuclear Training. Pursuant to 10 CFR 50.120 and 55.4, SAT-based training programs must include the evaluation of the trainee's mastery of training objectives. NRC inspections of licensee requalification programs for licensed operators have also found that training staffs generally possess the skills needed to evaluate the trainee's knowledge.

After the NRC reviews and approves an examination, the facility licensee will administer and grade the written portion as prescribed in NUREG-1021. The NRC staff will continue to administer and grade the operating tests, review and approve the written examination results recommended by the facility licensee, and make the final licensing decisions, feedback from the pilot examination program indicates that the average time spent by a facility licensee to prepare the written examination and operating tests was approximately 600 to 800 staff-hours. A portion of that time (about 200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />) would have been spent reviewing and assisting with the administration of NRC-developed examinations under the process now in place and should be subtracted from the total. The resulting average burden of approximately 400 to 600 staff-hours was somewhat higher than the 400 hours0.00463 days <br />0.111 hours <br />6.613757e-4 weeks <br />1.522e-4 months <br /> that NRC contract examiners typically take to prepare an examination. The extra burden is generally attributable to the facility licensees' lack of familiarity mth specific NRC examination expectations and to the additional administrative requirements, such as documenting the source of the examination questions, that are required to maintain examination integrity. It is worth noting that some of the facility licensees that participated in the pilot program expended less time than is 4

commonly used by NRC contractors to prepare the examinations.

Furthermore, in a few cases, the examinations that facility licensees submitted for review and approval were, in the judgment of NRC chief examiners, as good or better than those prepared by an NRC contractor. The staff expects that most facility licensees will eventually be able to prepare quality examinations in less time than the NRC or a contractor because the facility employees have more detailed knowledge of their facility and easier access to the reference materials required to prepare the examinations.

Experience during the pilot examination program indicated that the volume of reference materials required by the NRC examiners to review and prepare for the examinations was significantly less than that required for the NRC staff to write the examinations. The fact that contract examiners will not be used in the revised examination process will also eliminate the need for duplicate sets of reference materials to be provided by facility licensees. Feedback from the industry in response to the staff's solicitation of public comments on the draft revision of NUREG-1021 indicated that facility licensees had been spending from 80 to 160 hours0.00185 days <br />0.0444 hours <br />2.645503e-4 weeks <br />6.088e-5 months <br /> to prepare and ship the reference materials under the existing examination process.

The additional burden of having to prepare the site-specific initial operator licensing examinations should be substantially, if not totally, offset by reductions in Part 55 review fees billed to the facility licensees pursuant to 10 CFR 170.12(i). Each facility licensee will be billed only for the time that the NRC staff spends to review the examination prepared by the facility licensee and to rework the examination, as necessary, to bring it up to NRC standards; no contractual costs will be accrued. Although several of the draft pilot examinations were of poor quality and took the NRC staff more time than expected to review and rework, the staff believes that additional cost reductions will be realized as facility licensees gain experience with the NRC examination requirements and the qual Ry of the draft examinations improves.

This rule change will give facility licensees more control over the cost of their examination services because they will be in a position to manage the quality of the product i. hat is submitted to the NRC. The higher the quality of the examination the facility licensee submits, the lower the resulting charges. Under the existing examination process, facility licensees are responsible for the entire cost of preparing the examination, even if the contractor's submittal is of poor quality and requires significant rework by the NRC staff before it can be administered.

It is also worth noting that facility licensees will have the option of retaining the services of a contractor to prepare the license examination as the NRC often does under the current examination process. The staff understands that the NRC's examination contractors have already expressed an interest in providing their services directly to facility licensees. It is reasonable to assume 5

that the cost of using this option would be comparable to the contractor service fees that the NRC currently passes on to the facility licensees.

In sumary, the staff views this process change as resource neutral at worst, and possibly resulting in a resource savings to facility licensees over time.

3. Imoact of the Rule Chance on Operator License Apolicants The goal of the rule change is to eliminate the NRC staff's reliance on contractor support for the site-specific initial operator licensing program, while complying with the AEA and maintaining the existing level of effectiveness. To the extent possible, the format, content, length, and level of difficulty of the examinations will remain unchanged, thereby min nizing the impact of the rule d

change on the operator license applicants.

NRC examiners will continue to review and approve every written examination and operating test before it is administered. The examiners will work with the facility licensee's staff to modify the draft examinations as necessary to ensure compliance with NUREG-1021 and to maintain consistency with prior licensing examinations at that facility and among facilities, if the NRC decides to prepare the examination in lieu of accepting an examination prepared by the facility licensee, the NRC examiners will use the same procedures and criteria (i.e., NUREG-1021) that the facility licensee would have used to prepare the examination.

The operator license applicants should not be able to distinguish between an examination that is prepared by the facility licensee and one that is prepared by an NRC examiner.

4. Alternatives to the Rule Chance If the Commission decides not to amend 10 CFR Part 55 as proposed by the staff, it would require the restoration of contractor funding for the operator licensing program or the allocation of sufficient direct examiner resources in each regional office to satisfy the demand for initial licensing examinations and to conduct the licensed operator requalification inspections. Historically, the NRC has spent approximately $3 million to $4 million per year on contractor support funding for the operator licensing program.

!!I. Other Elements Leaal Analysis by the Office of the General Counsel (0GC)

OGC has reviewed this rulemaking plan and does not believe this action

< m titutes a backfit pursuant to 10 CFR 50.109. In addition, OGC has concluded that this action does not constitute a major rule pursuant to 6

}

the Sns11 Business Regulatory Fairness Act of 1996. Fina11y, at this time, OGC is not aware of any reason for a legal objection to this rulemaking plan.

. Backfit Analysis As discussed in SECY-96-123,-OGC agrees with the staff's position that requiring the facility licensees to prepare the initial operator licensing examinations would not be a backfit pursuant to 10 CFR 50.109, which states in pertinent part that --

Backfitting is defined as the modification of or addition to systems, structures, components, or design of a facility; or the design approval or manufacturing license for a facility; or the procedures or organization required to design, construct or operate a facility; any of which may result from a new or amended provision in the Commission rules or the imposition of a regulatory staff position interpreting the Commission rules that is either new or different from a previously applicable staff position....

The proposed change does not result in a modification of or an addition to systems, structures, components, or the design of a facility. The change does not affect the design approval or manufacturing license for a facility. The procedures required to design or operate a facility will not be affected by the proposed change. The proposed change would require each nuclear power plant licensee to develop the tests that are used to qualify, as meeting the requirements of 10 CFR Part 55, those nuclear power plant operators whom the nuclear power plant licensee wishes to employ. Development of such tests are not considered to be

" procedures... required to... operate a facility." The tests are not applied to the facility licensee, but rather to the operator licene candidates. Further, any procedure necessary to develop the test would not be useful in actually " operating" the facility, even if one broadly interprets " operating" as including any action necessary to comply with the Commission's regulations with' respect to operation. The organization required to design or operate a facility will not be affected because all facility licensees already have a training staff to train and evaluate candidates for operator licenses and to train other 2 members of the plant staff, as required by 10 CFR Part 55 and by 10 CFR 50.120. Therefore an organizational change is not required because of this process change.

Suonortina Documents A regulatory analysis and an Office of Management and Budget (0MB) statement will be prepared. Neither an environmental impact statement nor an environmental assessment will be prepared for this proposed rule, however, since it is the type of action described as a categorical exclusion.in 10 CFR 51.22(c)(1).

7

Resources Reautred As discussed previously in Section 11.2, the rule will require facility licensees to assume responsibility for writing the examinations, a burden currently carried by the NRC. However, the transfer of this responsibility is considered to be resource neutral, and could possibly result in a resource savings to facility licensees.

The rulemaking is expected to take approximately 0.5 staff years to complete, and no contractor support will be needed. However, until the rulemaking is complete and all power reactor facility licensees are required to prepare their examinations, the Office of Nuclear Reactor Regulation (NRR) will continue to use contractors, as necessary, to supplement the staff's ability to develop and administer the initial operator licensing examinations and to conduct the requalification inspection program. When the rule goes into effect, the staff expects that the revised examination process and the requalification inspection program can be implemented at all power reactor facilities with the existing NRC resources allotted to the operator licensing prog m .

Coanizant Staff / Proiect Manaaement Concurrina Official Office of Nuclear Regulatory Research (Lead): H. Tovmassian Bill Horris Office of Nuclear Reactor Regulation: S. Richards Bruce Boger Office of General Counsel: M. Schwartz Stuart Treby Steerina Groun A steering group wiil not be required for this effort.

Enhanced Public Particioation This rulemaking will use electronic bulletin boards, as appropriate, to enhance input from the public.

EDO or Commission Issuance The proposed and final rule will be approved by the Commission.

Schedule Proposed rule published: 3 months after the ED0 approves plan Comment period ends: 60 days after the proposed rule is published Final rule published: 6 months after the comment period ends 8

. . . , - . - .. - . . _ . ~ . . , .-- . . . - - . . . _ . . . - . . . - - . . . . . . . . .

+

j ..

T 4 j  :

. RESPONSE TO THE STAFF REQUIREMENTS' MEMORANDUM 0F b JULY 23, 1996'(M960618A)-

I 9

1 ENCLOSURE-2

_ _ 1 w >

T + 1

-'. f

. . . _ _ - . _ - - - . . . . . ~ ~ . . - _ . _ . . - - - . - - . . . . - . . . , . . - . , . - . . . . - . . -

. . . . . .._, _- - . - - . . . - . . +_.m

DISCUSSION OF PROPOSED OPERATOR LICENSING PROGRAM CHANGES Backaround The site-specific initial operator licensing examination consists of a 100-

. question written examination and an operating test that includes a performance demonstration on a dynamic simulator and a plant walk-through. The written examination is multiple-choice and closed-book in format, with a passing grade of 80 percent. The simulator performance demonstration generally consists of two or three scenarios administered to a team of three license applicants.

Each applicant is individually evaluated on a range of competencies applicable to the applicant's license level (i.e., reactor operator (RO) or senior reactor operator (SRO)). The walk-through portion of the operating test requires each applicant to individually complete a total of ten tasks, or job performance measures (JPMs), in the control room and the plant and to answer specific questions related to the associated systems. The entire site-

. specific examination has historically been prepared by NRC license examiners or NRC contractors. As a prerequisite for taking the site-specific examination, every applicant must pass a separate generic fundamentals examination (GFE), consisting of 100 multiple-choice questions covering

' mechanical components, principles of heat transfer, thermodynamics, fluid mechanics, and fundamentals of reactor theory. An NRC contractor prepares two versions of the GFE (one for boiling water reactors and another for pressurized water reactors) twice per year. The GFEs are reviewed and approved by the Operator Licensing Branch (HOLB) before being mailed to the facility licensees for administration.

• On March 24, 1995, SECY-95-075, " Proposed Changes to the NRC Operator Licensing Program," informed the Commission of the staff's intent to revise the operator licensing program to allow greater participation by facility licensees and to eliminate contractor assistance in this area. On April 18, 1995, the Commission consented to the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to carefully consider experience from pilot examinations before full implementation of the changes.

From May to June 1995, the Division of Reactor Controls and Human Factors (DRCH) and HOLB conducted a number of meetings with senior management and regional personnel to discuss various options for implementing the proposal outilned in SECV-95-075. On August 15, 1995, after weighing the pros and cons of the various options (discussed below), the staff issued Generic letter (GL) 95-06, " Changes in the Operator Licensing Program," which notified power reactor facility licensees of the NRC's Intent to change the operator licensing process, outlined a number of criteria to supplement the guidance in NUREG-1021, " Operator Licensing Examiner Standards," and solicited volunteers to participate in a 6-month pilot program to evaluate and refine the new examination methodology.

On June 10, 1996, the staff issued SECY-96-123, " Proposed Changes to the NRC Operator Licensing Program," which described the pilot examination process and criteria, reviewed the examination results, discussed the lessons learned, and recommended that the enhanced process be implemented on a voluntary basis 1

pending an amendment to 10 CFR Part 55 that would require all sower reactor facility licensees to prepare their licensing examinations. 11e staff briefed the Commission on June 18, 1996, and in a staff requirements memorandum (SRM) dated July 23, 1996, was directed to develop a detailed rulemaking plan and to address the following specific issues regarding the proposed changes in the examination process.

Erm .and Cons (Vulnerabilities)

In the early stages of rieveloping the pilot examination process, the staff identified several concerns (examination integrity, quality, and consistency, ind6pendence and public perception, examination security, resources requirements, program stability, and examiner proficiency) that have played a dominant role in the design, implementation, and refinement of the revised examination methodology. The staff evaluated all of the pilot examination criteria specified in GL 95-06 in light of these controlling concerns. The same concerns were also integral in evaluating the post-pilot process modifications that have been incorporated in the )roposed revision of NUREG-1021. Although the staff believes that eac1 concern has been adequately i addressed and mitigated, it would be disingenuous to assert that industry-wide implementation of the revised examination process does not entail a measure of uncertaint). The following is a discussion of each concern, the measures that have been inplemented to mitigate the concern, and the associated vulnerabilities, t

Integrity, Quality, and Consistency of the Examination Section 107 of the Atomic Energy Act of 1954, as amended, requires the Commission to prescribe uniform conditions for licensing individuals as operators, determine the qualifications of such individuals, and issue licenses to such individuals in such form as the Commission may prescribe.

When the staff developed the pilot examination program, it was concerned about whether the existing levels of examination uniformity and consistency could be maintained. The operator licensing recentralization study (see SECY-93-309,

' Findings of the Operator Licensing Recentralization Study," of Ncvembe 17,1993), which had concluded that the level of examination uniformity was adequate to ensure that appropriate licensing decisions were being made, highlighted the challenge of controlling the diversity that individual examiners from different regional offices could add to the examination program. Permitting a substantial number of facility employees having differing levels of training and experitnce to prepare the examinations could increase the amount of variation in the levels of knowledge tested and examination <tifficulty despite the efforts of the NRC to control these factors through improved procedures and oversight reviews.

To maintain uniform standards of examination content, format, level of

- difficulty, and quality, the staff insisted that the participants in the pilot program prepare the written examinations and operating tests in accordance with the procedures and guidelines in Revision 7 of NUREG-1021. NUREG-1021 contains guidelines and criteria that are adequate to ensure that a licensing examination, independently prepared by an NRC examiner or contractor, will 2

- w E- y-- -- -

t differentiate between those applicants who have and those who have not mastered the knowledge and abilities required to safely operate the plant.

-However, when the staff reviewed those criteria in light of the proposed shift

-in responsibility for writing the examinations, it concluded that some of the criteria were no longer ap)ropriate and that additional criteria were necessary to ensure that tie integrity of the facility-developed examinations would remain at acceptable levels.- Therefore, the staff promulgated the following supplemental and modified criteria in GL 95-06 when it solicited volunteers to participate in the pilot exwnination programt o NUREG-1021 currently allows NRC and contract examiners to use up to 10 percent of the written questions, job performance measures, and simulator scenarios in a facility licensee's examinstion bank when they develop an initial licensing examination for the facility. This process saved NRC resources and provided an incentive for facility licensees to develop their examination banks because the examiner could conceivably extract the entire written examination from a facility licensee's bank,  ;

provided the bank contained 1,000 or more questions that satisfied the

, psychometric criteria in NUREG/BR-0122, " Examiners' Handbook for Developing Operator Licensing Written Examinations."

Whereas the NRC's examiners are continually revising old test items and developing new ones, there is currently no requirement for facility licensees to take comparable measures to minimize predictability.

Predictable examinations tend not to discriminate between competent and deficient Itcense applicants because what is being tested is simple recognition of the answer. Although studying past examinations can have a positive learning value, total predictability of examination coverage through over-reliance upon examination banks reduces examination integrity. When the examinees know the precise and limited pool from which test items will be drawn, they will tend only to study from that i pool (i.e., studying to the test) and may exclude from study the larger domain of job knowledge. When this situation occurs, it decreases confidence in the knowledge inferences that are made from performance on the test.

To limit predictability, GL 95-06 revised the limits on the use of facility item banks that could have been published, reviewed, or used as a basis for training. GL 95-Ofi treated all question banks as if they were open to the a?plicants because the staff had no basis to regulate the qualities of tile item banks and because it would be difficult to confirm that e bank was indeed closed to the applicants. The staff attempted to strike a balance between the use of new and existing test items because it appreciated the effort required to develop new test items and understood that existing items are a valuable training resource that should not be wasted. On the basis of the results of the generic fundamentals examination (GFE) program, the staff decided to limit-the use of existing written questions on the site-specific 4 examination to no more than 50 percent and-to require at least 10 newly developed [i.e., not previously seen) questions on every examination.

The remain 4ng questions could be generated by significantly modifying 3

existing bank questions. Each operating test was required to include at least two JPMs that were either new or significantly altered and at least one simulator scenario that was new or significantly altered. The remaining scenarios had to be modified to the extent necessary to prevent the applicants from immediately recognizing the scenarios on the basis of the initial conditions or other cues.

O NUREG-1021 currently places a 25-percent limit on the number of written examination questions that can be repeated from the previous two NRC licensing examinations at the facility. To further limit the predictability of the examinations that would be prepared by the facility licensees, the staff revised this criterion to include all examinations, quizzes, and tests administered to the applicants during their license training class. furthermore, no questions could be drawn directly from the applicants' audit examination given by the facility Itcensee at the end of the license training class to determine if the applicant was ready for licensing. This restriction was the subject of industry comments and has been revised as discussed below since completing the pilot program, o NRC and contract examiners are currently required to cite a specific facility reference for every question on the written examination.

Questions that were extracted from the facility licensee's question bank would be identified as such, in order for the NRC chief examiners to monitor the facility licensees' compliance with the previously stated criteria, GL 95-06 also required the pilot examination participants to state the history (e.g., bank, revised, new, and date last used) of each test item on the written examination and operating tests. Although this practice did 11 ace an additional burden on the facility licensees, it facilitated t1e NRC's examination reviews and resulted in lower licensing fees billed to the pilot participants. This requirement was also addressed in the industry comments and has been clarified as discussed below since completing the pilot program.

The Office of Nuclear Reactor Regulation (NRR) also issued supplementary written guidance to the regional offices clarifying the NRC examiners' responsibilities during the pilot examination development process, and it conducted a pubite workshop to discuss the staff's expectations during the pilot program. The written guidance and the minutes of the workshop were placed in the NRC's Public Document Room (POR) and provided to facility licensees that were scheduled to participate in the pilot examination program.

NRR did not restrict the number or sc9pe of changes th:t the NRC examiners could ask facility licensees to make in their proposed examinations. The existing quality assurance (QA) checklists in NUREG-1021 were revised and new checklists were developed to help the examiners identify problems with the preliminary examination outlines and the operating tests that the facility licensee would be required to submit for review.

Because of resource limitations, examiners were instructed to focus primarily on the content and construction (i.e., level of knowledge and difficulty) of 4

the written examination questions, rather than spend their time verifying the detailed technical accuracy (e.g., procedural and component references) of every question, as is required for NRC-developed examinations. The staff expected the facility licensees to ensure that the examinations were technically sound and instructed the regions to question the licensees regarding any post-examination question deletions and answer key changes.

To assess the effectiveness of the allot examination process, NRR asked the NRC examiners who were involved wit 1 the examinations to respond to a questionnaire on various aspects of the new methodology. The feedback indicated that many of the as-written examinations were comparable in quality and level of difficulty to recent NRC-written examinations. However, the NRC examiners and facility staff had to rework some of the examinations significantly to meet NRC standards. Although the staff had hoped that the technical accuracy of the facility licensee's written examinations would be as good as or better than the accuracy of the examinations written by the NRC, some of the )ilot examination results (i.e., more question deletions and answer key c1anges than expected) indicated a need for improvement in that area.

Despite the challenges of quality and technical accuracy, the staff believes that each as-given pilot examination provided a valid basis for licensing the applicants who passed the examination. The staff expects that the consistency and quality of the draft examinations will improve as facility licensees gain experience and become more familiar with NRC examination requirements. Based on observations during the pilot examinations, the staff has also incorporated in Revision 8 of NUREG-1021 a number of process enhancements that are intended to improve the quality of the examinations developed under the revised

)rocess. A number of the significant enhancements are discussed below. NRR 1as also instructed the regional examiners and managers not to approve or administer any examination that does not adhere to current NRC standards for content, format, quality, level of knowledge, and level of difficulty. If necessary, the examination is to be delayed or cancelled until the facility licensee and the NRC staff can upgrade or replace the examination.

The overall results of the pilot program indic ated that the examinations prepared by facility licensees, subject to review, revision where appropriate, and approval by NRC staff examiners, were generally consistent with examinations prepared by NRC or contract examiners. The pilot examinations also appeared to be equally effective at identifying applicants who had not mastered the job requirements well enough to become licensed R0s or SR0s.

Furthermore, the pass rates for Ros and SR0s on the written examinations and operating tests compared favorably with the pass rates for examinations prepared by the NRC cr its contractors.

The chief examiners for approximately half of the pilot examinations actually considered the submitted examinations to be as good or better than recently administered NRC examinations because the facility authors were better able to integrate the required site-specific knowledge and abilities into operationally oriented questions. All but 5 of the 22 feedback surveys completed by the chief examiners indicated that the revised examination process was as effective or more effective than the traditional examination 5

process. Most examiners (' M agreed that the performance of the facility licensees should improve - sney become more familiar with the NRC's requirements and expectat # ,.

In summary, the operator licensing program staff has implemented numerous procedural measures to maintain and promote the consistency, quality, and integrity of the initial licensing examinations. However, in practice, it will be the responsibility of individual examiners, with appropriate review and oversight by their supervisors and the NRR staff, to ensure that the licensing process remains strong. It is possible that the pressures to conduct examinations on schedule and the reduced involvement of the NRC in the examination develo) ment process could lead to examinations that discriminate at a lower level t1an is currently the norm. The revised examination process is much more dependent u)on the staff's ability to identify a substandard examination and ensure t1at the deficiencies are corrected. If the staff is indecisive and the licensee takes advantage of the situation, the examination will probably not discriminate at the appropriate level.

Independence and Public Perception Before 1987, the staff believed that only independently developed and administered examinations could reliably and consistently achieve the appropriate standards of objectivity and discrimination required for the NRC's initial licensig and requalification programs. The staff was concerned that examinations prepared by a facility licensee could be unconsciously biased toward the specific material taught by the facility and that potential financial or other pressures on the facility licensee would create a conflict of interest.

The staff's position changed significantly in 1988 when it revised the requalification examination methodology so as to require each facility licensen to prepare the examination materials and demonstrate its ability to evaluate the competence of its licensed operators. From 1989 to 1994, the NRC evaluated over 4,000 licensed operators using requalification examinations that were prepared by the facility licensees, reviewed and approved by the NRC, and administered in parallel by NRC examiners and facility personnel.

The examinations appeared to discriminate at the a3propriate level, and approximately 90 percent of the operators passed tie examinations on their first attempt.

Despite the success of the requalification examination process, the staff continued to independently develop and administer the initial licensing examinations. However, the overall performance of the license applicants on the initial examinations was not significantly different from the performance of the licensed operators on the requalification examinations developed by their facility licensees. From 1989 to 1994, the NRC administered initial licensing examinations to approximately 4,000 applicants; the pass-rate on those examinations averaged approximately 90 percent, in SECY-95-075, the staff )roposed to take the next evolutionary step, similar to the staff's actions witi regard to oversight of the requalification program, by changing the guidance in NUREG-1021 to permit facility licensees 6

- . = - - -_ --. - - - - - - -- . --- .-- _---

to prepare and, in part, conduct the written and operating examinations for initial license applicants. The NRC would review the licensees' efforts, partially participate in the examinations, retain the final pass / fail authority, and continue to issue licenses. This action would allow the staff to perform its licensing function in a more efficient manner, while maintaining an effective, independent process upon which to base licensing decisions for operators.

Shortly after SECY-95-075 was issued, The Washington Post reported that consumer advocates were concerned that allowing facility licensees to prepare the examinations might endanger reactor safety because the examinations might be less rigorous than those pre)ared by the NRC. The information upon which the consumer groups had based t1eir conclusions was preliminary in nature and likely did not consider the level of oversight that the NRC would retain in the examination process or the compensatory measures that the staff would implement to ensure that the examinations retained an appropriate level of difficulty.

The Washington Post article also indicated that the proposal appeared to contradict the staff's earlier decision to reject a request by the Virginia Electric and Power Company (VEPCO) to prepare its own examinations (this topic will be discussed in more detail later) and that a number of mid-level staff members opposed the change. The staff believes that the initial opposition to the proposal was a natural reaction to the abrupt change in policy and to the fact that many staff members hed not been consulted or informed before SECY-95-075 was issued. As promised in the memorandum of April 12, 1995, from the Executive Director for Operations, all involved staff members have been well informed of the new policy and have been able to express their views and to have their concerns considered as part of the process to implement the broad goals stated in SECY-95-075.

As noted in SECY-96-123, the results of the pilot program indicated that initial licensing examinations prepared by facility licensees, subject to review and approval by NRC staff examiners in a manner similar to the review and approval of NRC requalification examinations, were as effective as examinations written by contractors in identifying applicants who had not mastered the job requirements for a licensed operator. The pass rates for Ros and SR0s on the written examinations and operating tests compared favorably with the pass rates for examinaitons prepared by the NRC and its contractors.

To promote examination consistency and minimize the bias toward specific materials taught by the facility licensee, the staff insisted that the pilot examinations be developed in accordanw with NUREG-1021 and NUREG/BR-0122.

NUREG/BR-0122 contains explicit guidance for sampling the important, safety-related knowledge and abilities (K/As) from NUREG-ll22 " Knowledge and Abilities Catalog for Nuclear Power Plant 0)erators: Pressurized Water Reactors," and NUREG-ll23, " Knowledge and A)ilities Catalog for Nuclear Power Plant 0)erators: Boiling Water Reactors," companion documents derived from a job-tas( analysis of the operators' work. The supplementary pilot examination criteria in GL 95-06 required the facility licensee to submit a proposed examination outline to the NRC for review and approval at least 60 days before the scheduled examination date. This requirement would enable the NRC chief 7

examiner to assess whether the sampling was appropriate (e.g., the balance of coverage and importance ratings; duplication and overlap between examination sections and from prior examinations; and inclusion of plant-specific priorities, such as recent events and risk insights) before the facility licensee proceeded with developing the detailed examination materials.

GL 95-06 also required facility licensees to submit the proposed examination to the NRC for review at least 30 days before the scheduled administration date. The NRC chief examiner and the responsible supervisor would work with the facility licensee to revise the examination as appropriate before authorizing its administration.

To minimize personal conflicts of interest, GL 95-06 )rohibited facility employees who played a substantial role in training tie license applicants from becoming involved in preparing the licensing examinations. Although this restriction did little to mitigate the fact that everyone who worked for the facility licensee was likely to have a vested interest in the outcome of the examinations, it did alleviate some of the concerns regarding personal gains or penalties that the examination author might incur if he or she were directly accountable for the examination results. This restriction also allayed the staff's concern that an instructor might unconsciously bias the examination toward those topics that were emphasized during the training program. The personnel restrictions were undoubtedly the most controversial aspect of the pilot examination process. They were widely discussed among the NRC staff and management before, during, and after the pilot program was implemented, and they were also the principal issue raised by industry regarding the pilot process. A discussion of this and a number of other industry concerns follows.

The issues of independence and public perception also had a significant bearing on the metiodology that was selected for administering the operating tests, the best measure of the applicants' ability to respond to a plant event. Although general agreement existed among the staff on most of the pilot criteria discussed earlier, there was considerable disagreement regarding the level of involvement that NRC examiners should retain in the operating test administration process. The following options were evaluated in depth:

Option 1: NRC examiners would continue to independently administer the simulator scenarios and the plant walk-through portion of the operating test in accordance with current guidance in NUREG-102). Involvement by the facility licensees would not change from current practice, and facility employees would not conduct parallel evaluations.

Option 2: NRC examiners and facility' employees would conduct parallel evaluations of each applicant during the dynamic simulator and walk-through portions of the operating test. This is the same methodology that is used during NRC-conducted requalification examinations.

Option 3: NRC examiners would observe facility employees conduct the simulator scenarios and a sample of the walk-through tasks 8

.- _-- .- --~ , _ - . - - - . _ ._-

i for each applicant: however, the NRC examiners would not conduct one-on-one parallel evaluations of each applicant.

This methodology is similar to that currently used during the requalification inspection program.

After much deliberation, the staff decided to im)1ement Option 1 during the pilot examination program. The staff believed t1at Option I would have the most favorable public perception and examiner acceptance because it maintains the existing levels of independence and objectivity by relying on NRC examiners to conduct and evaluate the operating tests. Option 2 would have ,

generated more concern regarding conflicts of interest because it would require facility amployees to administer and grade the operating tests (to their coworkers and friends) in parallel with NRC examiners. Option 3 would have had the least NRC involvement and, therefore, the greatest potential impact on independence and public perception. By not directly observing each applicant, the NRC examiner would be in a weaker )osition to judge the performance of the applicant so as to determine witch applicants should pass or fail-if a crew responded incorrectly during a simulator scenario.

In summary, it is possible that the pressures to conduct the examinations on schedule and the reduced involvement of the NRC in the examination development process could lead to examinations that reflect the biases of the facility licensee and that discriminate at a lower level than is currently the norm.

Ilowever, a number of process requirements have been augmented as necessary to minimize this potential. The staff believes that the revised examination

, process, which includes restrictions on personnel, independent NRC review and approval of the examinations, and independent NRC administration of the operating tests, will provide sufficient safeguards to ensure unbiased licensing decisions.

Given the complexity of the proposed change in the examination process, the public may jump to the conclusion that the NRC has transferred all responsibility for examining o)erator license applicants to the facility licensees, in actuality, the NRC retains a significant amount of control over the content of tha examinations and continues to administer the operating tests. The challenge may be to explain this fact to the public.

Examination Security in 10 CFR 55.49, applicants, licensees, and facility licensees are prohibited from engaging in any activity that compromises the integrity (security) of any application, test, or examination required by Part 55.

Before 1989, ensuring compliance with the rule was relatively straightforward because facility Itcensees were not permitted to review the written operator licensing examination until it was distributed to the applicants on examination day. Security compromises and cheating were rare but not unheard of.

The degree of risk to examination security increased significantly in January 1989 with Revision 5 of NUREG-1021. Revision 5 made the facility licensees largely responsible for preparing the NRC requalification examinations and, 9

for the first time, also allowed the facility staff to review the written Itcensing examination up to 2 weeks before its administration. The increased risk to security was tolerated because the prereviews helped identify inaccurate questions before the examinations were administered, thereby increasing the content validity of the examinations, minimizing the number of questions from applicants during the examination, and reducing the number of question deletions and changes during the grading process. The gradual shift to multiple-choice questions during the late 1980s and early 1990s ) laced a premiun on accuracy because the answers to the questions could not se adjusted after the fact as they could be with the essay and short answer questions that they replaced.

To maximize security, given the new philosophy, NUREG-1021 prohibits any facility employee with specific knowledge of any NRC examination before it is given from communicating the examination contents to unauthorized individuals and from participating in any further instruction of the students scheduled to take the examination. Before they are given access to the examination, the T facility employees are required to sign a statement acknowledging their understanding of the restrictions and the potential consequences of noncom)11ance. They must also sign a post-examination statement certifying that t1ey did not knowingly compromise the examination.

The staff realized that the potential for examination compromise would increase as the facility licensees increased their role in the initial licensing process from one of reviewing the NRC-developed examinations to that of developing their own examinations. Therefore, the staff attempted to clarify its security expectations in the supplemental pilot examination guidance that was )rovided to the regional offices (and placed in the PDR) in conjunction with G 95-06. The guidance addressed the restrictions on personnel and a number of physical security precautions, including protecting and mailing the examination materials. The guidance also cautioned examiners to be attentive to examination security measures and required thein to review the security requirements with the facility licensee's contact at the time the examination arrangements were confirmed.

The issue of examination security also had a bearing on the methodology that the staff selected for administering the pilot operating tests. The staff believed that Option 1 would pose the lowest threat to examination security because the smallest number of facility employees would require detailed knowledge of the examination before it was given. Options 2 and 3 both would have required the facility licensco to evaluate the license applicants during the simulator and walk-through tests, which could have required several additional personnel to familiarize themselves with the operating test content, thereby increasing the risk of compromise.

In summary, the staff expects that the vast majority of facility licensees will continue to maintain proper examination security (as they have during the requalification program). Security is enhanced by the fact that the NRC monitors and oversees an extensive range of the facility licensees' activities and is sensitive to allegations and other performance indicators that might expose a problem. Nevertheless, the staff m nnot deny that the NRC is now more vulnerable to examination compromise (w the associated public response) 10

than it was before 1989 when it first allowed facility licensees to become involved with the NRC's examination development programs. The staff anticipates that the agency may suffer an occasional lapse in security. When a lapse occurs, the staff will continue to take prompt action to cancel the examination, determine the cause of the compromise, and implement corrective measures.

Resources As stated in SECY-95-075, the proposed changes were originally conceived as part of the NRC's continuing efforts to streamline the functions of the Federal Government consistent with the Administration's initiatives and to accommodate NRC resource reductions. The original plan was to entirely eliminate the use of contractors in the operator licensing aret and to conduct all operator licensing program activities, including the generic fundamentals and the requalification program, with the and initial existing licensing staff exam' The resot..*ces. nations,lity faci licensees would prepare and, in part, conduct the initial licensing-examinations with= oversight by the NRC. T se -

NRC's participation would range from conducting part to all of every examination in order to provide an independent basis for making a licensing decision and to maintain examiner skills.

The staff estimated that a qualified facility employee should be able to prepare the site-specific examination in accordance with NUREG-1021, as .

supplemented by GL 95-06, in approximately the same amount of time that is currently allocated to a contract examiner. (See Attachment 1, the associated rulemaking plan, for a more detailed discussion of the estimated resource burden on facility licensees.) The staff also estimated that an experienced NRC examiner should take approximately two weeks to review the site-specific examination (including the written and operating tests) prepared by a facility licensee.-

As previously noted, the staff considered three options for administering the operating tests. Assuming a license class consisting of nine applicants

-(e.g., six R0s and three SR0s) that could be arranged in three, three-person crews, both the first and second options would generally require three NRC examiners to spend a week at the site to administer the tests. The second option, which would require the facility licensee to co.iduct parallel evaluations of each applicant during the simulator scenarios and walk-through tests, would place a significant additional burden (nominally, three staff-weeks) on the facility licensee. The third option would be equally burdensome to the facility licensee but would aermit the NRC to reduce (by one) the number of examiners dispatched to tie site.

The plan to require the~ facility licensee to evaluate the applicants during the operating _ tests was perceived as a du)11 cation of effort because the facility-licensee had already evaluated tie applicants and found them qualified before it submitted the license applications to the NRC. The staff

-was alsu concerned-that the additional regulatory burden would discourage participation in what was originally envisioned as a voluntary program with no

associated rulemaking..

11-

The NRC resource utilization rate during the pilot examinations supports the staff's original estimate that the new examination process and the requalification inspection program could be implemented at all power reactor facilities with the same level of direct NRC resources as is currently allotted to the operator licensing program. As noted in SECY-96 123, the NRC examiners required an average of about 350 hours0.00405 days <br />0.0972 hours <br />5.787037e-4 weeks <br />1.33175e-4 months <br /> to review, prepare for, administer, grade, and document each pilot examination. Most of the examinations included an extra visit to the site to review and validate the operating tests.

However, with the elimination of contractor support, the total number of examiners avsilable will decline significantly, thereby reducing the ability of the program to have the personnel available to meet peaks in facility licensee requests for examinations. The impact of the loss of examiners due to reassignments, transfers, promotions, and other personnel actions, will be larger without the contract examiners available to compensate for the loss.

To compensate in this area, NpR intends to require that the regional offices maintain a defined minimum level of examiners, to be defined in each regional operatin plan. During fiscal years (fYs) 1997 and 1998, each region may need to qua11.y additional examiners to meet this expectation. To specifically assist the regions in their efforts to train additional examiners, NRR intends to make available to the regional offices up to 7 full time equivalents (FTE) of inspection support during FY 1997 and up to 12 FTE of inspection support in FY 1998. This inspection support will allow regional personnel, who would normally be assigned to inspection activities, to be assigned to qualify as examiners. By the end of FY 1998, the regions will be expected to maintain the defined level of examiners in order to provide the resource flexibility needed to deal with ' orkload peaks. When examination work is not scheduled, the examiners will be available for inspection activities. A reduced level of contractor support will be required during the transition period.

The staff expects that if the quality of an examination prepared by a facility licensee deviates significantly from established norms, it will become increasingly difficult to predict the amount of time necessary to review or rewrite the examination so as to achieve an acceptable product. This was the case with several of the pilot examinations, and the staff expects that it will continue to be more of a problem with facility-prepared examinations than it has been with those prepared by an NRC contractor. This added uncertainty in the examination process could increase the risk of broken examination commitments and lower examination quality; it could also raise costs as a result of inefficient planning.

With the elimination of contractor support and the increased uncertainty about the examination quality, the staff may no longer have sufficient examiner resources to write or even to review all examinations consistent with the scheduling needs of facility licensees. This resource problem is further compounded by the unpredictable nature of the examination workload and by other unanticipated demands on the examiner work force, such as the increase in the number of examination appeals during the pliot examination program.

Historically, the contract examiners have facilitated the staff's ability to respond to unexpected increases in the examination workload and unforeseen departures by staff examiners. Eliminating contractor support will complicate 12

i scheduling and limit the staff's ability to conduct retake examinations, which could increase the likelihood that an a)plicant will appeal the results of the examination. It is for these reasons t1at the staff is recommending to the Commission that 10 CFR Part 55 be amended to require all facility licensees to prepare the licensing examinations.

With regard to the generic fundamentals examination (GFE), the staff considered the following options: (1) retaining the current format and process ur. der which the examinations are developed by a commercial contractor and (2) revising the sco)e of the site-specific written examination to sample the generic fundamental (nowledge and abilities, as was done before 1988. If the examination was revised by adding more questions, it would increase the burden of writing and reviewing the examinations (whether it was done by the facility licensee or the NRC). If the length of the examination was kept the same, inclusion of the generic knowledge and abilities would weaken the evaluation of the site-specific systems and procedures. The staff and the industry generally have agreed that the current program is both efficient and effective and that revising the site-specific sample plan would create too many problems. Accordingly, the staff has decided to retain the current examination format and process and to implement the program with contractor assistance at a cost of about $200,000 per year.

Program Stability from 1988 to 1990, the NRC received numerous complaints from facility licensees and licensed operators that the NRC's requalification examination program was causing undue stress for the operators and the facility employees who were administering the examinations. The Human factors Assessment Branch conducted a study to address this issue and determined that the frequent changes to the requalification examination process ranked among the most significant sources of undue stress; the results of the study are documented in SECY-91-391 "Results of the Study of Requalification Examination Stress."

One of the corrective action:: involved establishing a schedule for periodic revisions of NUREG-1021 and delaying the implementation of each revision for 180 days after it is published.

Although the staff did not survey license applicants during the requalification stress study, it has generally tried to minimize the adverse effect on applicants by avoiding unnecessary changes to the initial examination process. Therefore, when the staff decided to shift responsibility for developing the initial licensing examination: to the facility licensees, it resisted pressure to revise the examination format. To the extent possible, the format, content, and level of difficulty of the pilot examinations would remain unchanged, thereby making the changes imperceptibit to the license applicants. The staff also believed that the facility licensees were generally familiar with the existing requirements in NUREG-1021 and that any significant changes in the examination format would confound their efforts to assume responsibility for writing the examinations.

The desire to maintain program stability, avoid unnecessary char.ges, and minimize stress on the applicants also argued against the adoption of Options 2 and 3 for administering the operating tests. The requirement for facility 13

i personnel to evaluate the license applicants would increase the learning curve for the facility licensees and would likely elevate the stress for the applicants because of the double scrutiny and the increased crowding in the simulator.

1 In sumary, the staff found no compelling reason to modify the existing examination format. The benefits of retaining the current format far exceed any argument for making changes at the present time.

Examiner Proficiency On May 25, 1995, the staff briefed the Commission on the status of the operator licensing programs and the staff's proposal to shift responsibility for preparing the initial operator licensing examinations to the facility licensees. In the ensuing SRM of June 2, 1995, the Commission instructed the staff to ensure that the in-house capability to administer the examinations is i maintained, possibly throunh the random administration of some examinations.

As noted in SECY-96-123, the staff believes that NRC exam bers will maintain the ability to write examinations, if necessary, based on their participation in the new process. Although facility licensees will prepare the operator licensing examinatians in accordance with NUREG-1021, NRC examiners will continue to review and approve every examination before it is administered to ensure that it conforms to the criteria specified in NUREG-1021 for content, format, quality, and level of knowledge. Facility licensees will also administer and grade the written examinations in accordance with NRC

- procedures, but NRC examiners will continue to independently administer and grade both the dynamic simulator and the plant walk-through portions of the c)erating tests. The NRC will review and approve the assignment of grades for tie written examinations, including any changes to the answer keys and the deletion of questions recommended by the facility licensee. The NRC will also continue to make all licensing decisions and administer the appeals process for applicants who fail the examination.

NRR will continue to require new NRC license examiners to complete a standardized training program before they are certified to conduct licensing examinations. The training program includes classroom and simulator instruction on how to prepare and administer written examinations and o)erating tests, self-study of operator licensing procedures and guidance, on-tie-job training, and a final practical evaluation by a certified chief examiner. Once certified, examiners are required to maintain their qualification by conducting examinations and attending refresher training. .

NRR is currently revising the examiner refresher training syllabus to focus more attention on reviewing, critiquing, and approving written examination questions. Examiners are also required to attend the operator licensing examiners' training conference, which is-periodically convened by NRR to provide guidance and exchange informati.in.

To ensure that the on-site examination activities are being performed as intend 60, supervisors are required to periodically evaluate every certified examiner during the conduct of an operating test. Furthermore, to verify ce gliance with NUREG-1021 and assess the uniformity with which the 14 I

.,~- , _ . _ , _ , . . . , . . , - - . , ,m

examinations are administered within and among the regions, the NRR staff periodically audits the administration of operator licensing examinations at selected facilities and annually reviews the impicmentation of the operator licensing function in each region. The NRR audits and reviews include an assessment of the level of knowledge and the level of difficulty of the written examinations and the operating tests; each audit and review is documented in a report to the responsible regional Division Director.

Industry Acceptance and Response in the summer of 1994, VEPC0 recommended to the NRC that the initial licensing examination )rocess be changed to make facility licensees responsible for developing tie examinations subject to review and approval by the NRC chief examiner. The NRC would continue to administer and grade the examinations.

The proposal, which was submitted as part of VEPCO's cost-beneficial licensing action program, estimated that the utility could save an average of $67,500 per station for each initial license class by implementing the proposed approach. The staff reviewed and denied the proposal because of concerns regarding independence and objectivity and the potential for increased variability in the level of knowledge tested and examination difficulty.

The staff reconsidered its position early in 1995 when the NRC's fiscal expectations changed significantly. The staff decided to explore the possibility of eliminating the use of contractors in the operator licensing program by allowing facility licensees to develop the examinations, essentially just as VEPC0 had proposed, it was estimated that such a change would save the NRC from $3 million to $4 million per year in contractor support funding and that the reduction in recovered fees would largely or completely compensate the facility licensees for their additional expenses.

While evaluating various options for implementing the changes proposed in SECY-95-075, the staff obtained informal feedbac( from the Nuclear Energy Institute (NEI) regarding the industry's level of involvement in administering the operating tests. NEl had polled a number of facility licensees and determined that there was no support for parallel grading (Option 2 previously discussed) because of concerns regarding ap)licant stress, the additional resource burder, and simulator crowding. Tie majority of those surveyed expressed caut ous support for Option 3, under which NRC examiners would have observed the fdcility licensee administer the operating tests. However, those surveyed wanted the NRC examiners to focus only on the performance of the applicants and not on the examination 3ractices of the facility licensee.

Option 3 would shift the most responsi)ility to the facility licensees, but it was viewed as the most difficult option to implement. Facility licensees perceived that Option 1, under which NRC examiners independently administer the operating tests, would be the least demanding on facility licensees and the easiest to implement and would retain the greatest level of independence in the licensing decisions. NE! indicated that all of the facility licensees surveyed were satisfied with the GFE and thought it should be continued in the current manner.

The staff believes that the industry's response to GL 95-06, which described the pilot process and solicited volunteers to participate in a six-month 15

l program to evaluate its feasibility, was generally favorable. Of the 35 factitty licensees that had requestod examinations during the pilot period, 22 agreed to prepare their own examinations in accordance with GL 95-06. Most of the facility licensees that declined to participate cited as their reason their inability to reschedule the required resources on such short notice because of the restrictions on the use of personnel who had been involved in training the Itcense applicants.

In September 1995, shortly before beginning the pilot examinations, the staff convened a public workshop to discuss the NRC's expectations regarding industry participation in the pilot program for the operator licensing examination. The workshop also gave facility representatives an opportunity ,

to ask questions of the staff and to submit written comments regarding the I pilot examinatien criteria outlined in GL 95-06. Most of the questions and <

coments related to the restrictions on which personnel the facility licensee could use to prepare the examination and the requirement to document the history of the examination materials. Some facility licensees questioned the amount of reference materials that the NRC would need to review and prepare for the examination. The staff challenged the pilot examination participants who expected to have problems satisfying the personnel restrictions to suggest alternative ways to achieve the same objective. The staff comitted to review the alternatives on a case-by-case basis and to propose a final position in Revision 8 of NUREG-1021, which would oe published for comment near the end of the pilot program. The staff also made it clear from the beginning that facility licensees could use contractors to develop the examinations.

Feedback from NRC examiners who participated in the pilot examinations and industry representatives who attended a public progress meeting sponsored by NEl in January 1996 confirmed that the personnel restrictions were the most troublesome aspect of the pilot process. Many facilities, particularly those with smaller training staffs, contended that the quality of the examinations was suffering because most of their best qualified personnel were involved in teaching and were ineligible to prepare the examinations. Those attending the meeting generally agreed that the industry and the NRC had learned a great deal from the pilot program, that most of the issues could likely be resolved through training and improved communication, and that the pilot process should be fully implemented.

In February 1996, the staff issued draft Revision 8 of NUREG-1021 that incorporated both the )ilot process outlined in GL 95-06 and preliminary lessons learned from tie early )ilot examinations. The draft revision was sent to the PDR and placed on tie NRC's home page on the World Wide Web, and a federal Register notice requesting industry and public coments was published on February 22, 1996. NEl submitted comments and recommendations on behalf of the nuclear industry, and two facility licensees (VEPC0 and Commonwealth Edison Company) provided additional comments. On April 18, 1996, the staff conducted a public meeting with NEl and other industry representatives to review their comments and recommendations regarding the draft revision of NUREG-1021. The meeting also provided an opportunity to discuss additional issues and proposed changes that had emerged following the issuance of draft Revision 8 of NUREG-1021 for public comment. The significant comments and staff responses are as follows:

16

., ~ _ _ _ . _ ___ _ _ _ _ _ _ __ _ _ _ , ._

i l

l 1

o All three commenters suggested that the personnel restrictions placed on l individuals involved in writing the examinations should parallel the '

personr.el restrictions of the requalification p.*ogram (i.e., the examination authors must stop teaching when examination development begins). The industry believes that those restrictions, when combined l with the examination development and review criteria, would provide I adequate protection against the unconscious biasing of an examination by an individual, it is difficult to argue that the instructors would not be the best qualified individuals to prepare the examinations. Many NRC examiners also agreed that the requalification program restrictions would be adequate, llowever, the staff concluded that independence and conflict of interest were of sufficient concern to require personnel restrictions very similar to those outlined for the pilot process. The staff revised the wording in the draft NUREG to clarify and quantify the amount of i involvement that would disqualify an employee from participating in various aspects of examination development. For example, the personnel who prepare the examination outline, including the selection of knowleige and abilities to be evaluated on the written examination and the JMs and simulator scenarios to be performed during the o)erating test, shall not have had any direct involvement in training tie license applicants, furthermore, only one person with limited involvement (i.e., who taught more than 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> but who was responsible for no more than 15 percent of the scheduled classroom instruction) may participate in developing the written examination questions, and no one may develop questions for the topics they taught. Facility licensees may still

)ropose alternative approaches for consideration on a case-by-case

) asis.

o All three commenters questioned the need for facility licensees to document the history of every test item proposed for use on a licensing examination, even if it was drawn from another facility licensee's examination bank.

The staff agreed that the wording in the draft NUREG could have resulted in misunderstandings of what was expected. The staff still believes that it is important to monitor the predictability of the examinations but it lacks the data and the resources to independently identify the source of every test item. This information is readily available to the facility licensee. The staff has clarified the guidance in NUREG-1021 so that test items that a facility licensee obtains from another bank and deposits into its own bank may be treated as " bank" items, provided the test items have an equal chance of being selected for use on the examination. Items from another bank may be treated as new items if they have not been made available for review and study by the license applicants and there is no basis (e.g., historical precedent or reciprocal arrangements with the other facility licensee) for the applicants to predict their use on the examination. The staff believes that this revision will somewhat lessen the burden on facility 17

Itcensees, while enabling the NRC to maintain an acceptable level of oversight, o All three commenters suggested that NUREG-3021 relax the restrictions on the duplication of test items from the applicants' audit examination (a practice test given by the facility licensee before the NRC examination) if the audit examination was developed independently of the license examination.

Under the traditional examination method, the staff does not have access to the facility licensee's audit examination, so any duplication that occurs between the audit examination and the licensing examination is unmonitored and coincidental. The staff agreed that the proposal was consistent with current practice and changed the NUREG to allow up to five questions to be duplicated at random if_the two examinations are written independently (i.e., no interaction between the examination authors) and the facility licensee identifies the duplicates. Licensing examinations that are not developed independently shall not repeat any questions directly from the audit examination, o All three commenters recommended that NUREG-1021 accommodate facility licensees that have " closed" examination banks (banks not available to a)plicants for study) by relaxing the limit on the number of test items t1at can be drawn directly from the bank.

The staff acknowledged that this is a valid issue, but because this issue is so complex, the staff does not intend at this point to establish additional guidelines regarding the maintenance of examination banks. As noted earlier, the staff intends to treat all banks as if they are "open" at this time, o All three commenters objected to the suggestion in draft NUREG-1021 that facility Itcensees consider providing an explanation as to why the answer to every written (multiple-choice) question is correct and each of the distractors is plausible but incorrect. The industry thought this was a significant administrative burden that would, in practice, become a requirement, feedback from the NRC chief examiners indicated that some facility licensees adopted this practice on their own during the pilot examinations; it proved very beneficial and increased efficiency during the examination reviews. This prompted the staff to include it as an o)tional activity in draf t NVREG-1021. The staff still believes that tie practice has merit but has emphasized in the final NUREG that it is optional, o Two commenters recommended that facility licensees be permitted to use site-specific task lists in place of the NRC's K/A catalogs when developing the examination outline. The site-spec?fic lists are used to develop the operator requalification examinations and should also be acceptable for developing content-valid initial licensing examinations.

18

i Since the mid-1980s, the staff has used the NRC's generic K/A catalogs i as the basis for ensuring the content validity of the initial licensing i examinations. The catalogs are based upon an analysis of the licensed operator's job that was performed by the Institute of Nuclear Power .

Operations. Mt.ny facility Itcensees have conducted their own job task t

analyses and developed plant-specific o)erator task lists for use in their SAT-based training programs. Altlough the staff does not question  :

that the site-specific task lists would generally be acceptable for ensuring the content validity of the examinations, it believes that wholesale substitution of the catalogs could cause a decline in examination consistency. The staff considers the proposal appropriate -

in limited circumstances on a question-by-question basis; therefore, the guidance in NUREG-1021 has been revised to permit substitutions and additions of specific knowledge and ability requirements on a case-by-case basis. ,

o Two commenters disagreed with a paragra?h in Appendix A of the draft NUREG that implied that any question tacen from an open bank would test at the basic memory level. They argued that questions prepared at a higher cognitive level would not be reduced to the basic memory level if the question bank is large. They strongly recommended that the subject paragraph in Appendix A be deleted.

Appendix A 3rovides an overview of examination preparation concepts and discusses tie basis for some of the NRC's examination criteria, including the limits on the use of examination banks. The subject paragrap:1 is considered integral to the discussion and was not deleted. '

However, the staff has edited the text to clarify that questions could decrease to the simple recognition level if the item bank is small and available for the examinees to study.

The staff also clarified one of the criteria in draft NUREG-1021 that, for purposes of examination quality and consistency, required at least half of the questions on the written examination to be written at the comprehension or analysis level. The staff never intended to imply that any question drawn directly from a bank would be counted at anytling other than its face value. A question written at the comprehension or analysis level would count as such, even though the cognitive level at which it tests would decline with use and exposure.

o Two commenters recommended that the facility licensees be allowed more than five days, if necessary, to review and grade the written examinations. They asserted that relaxing the requirement would only delay the results to the facility licensee and the applicant and that the delay would not adversely affect the NRC.

The staff had traditionally allowed the facility licensee five days to

. comment on the written examination, so it decided to retain that time period for the pilot examination program. This was one of the examination criteria on which the chief examiners had been asked to comment, and no comments were made. However, since the proposed change 19

l would not affect the examination outcome, the staff decided to increase the flexibility in the final NUREG-1021 by noting that the facility licensee should submit the graded examinations within five days, if practical, o One commenter indicated that NUREG-1021 does not appear to accommodate utilities that do not want to prepare their own licensing examinations and suggested that utilities ba allowed to retain that option.

Because of the unpredictable demand for examinations and the scheduling difficulties that may result from a long-term voluntary program, SECY-96-123 requested that the Commission approve the staff s pursuit of l rulemaking to require that licensees of power reactor facilities prepare the operator licensing examinations in accordance with NUREG-1021. The Commission has delayed making a decision on industry-wide implementation of the revised process until it can review the detailed rulemaking plan (Attachment 1) and the additional issues discussed herein.

o One commenter recommended that the facility licensees be held res m sible for making examination changes "as agreed upon with the NRC" ratier than "as directed by the NRC." The commenter added that this alteration would help ensure that the suggested changes are accurate.

Because the staff fully supports the goal of the facility licensee to ensure the accuracy of the examinations, it revised NUREG-1021 as recommended but stipulated that the NRC would retain the final authority to approve the examination.

o One commenter requested that the NRC clariff which facility staff members would be allowed to observe the operating tests in the simulator. NUREG-1021 currently restricts personnel other than the simulator operator from watching the operating test without the approval of the NRC chief examiner. The commenter argued that facility managers and others deemed necessary by the facility should be allowed access to the examination, provided the simulation facility can accommodate them and there is no adverse effect on the applicants.

The staff concluded that this was a reasonable request and revised NUREG-1021 accordingly, facility management and other personnel deemed necessary by the facility licensee will generally be allowed access to the examination (under security agreements, as appropriate), provided the simulation facility can accommodate them and there is no adverse effect on the applicants, o One commenter requested that the NRC clarify the meaning of "significantly modify." The facility licensee had been told that the intent of the question had to be changed, which conflicted with the definition that had been published in GL 95-06 and draft NUREG-1021.

20

The staff concluded that this was a reasonable request and revised NUREG-1021 to indicate that the intent of the question does not have to change to qualify as a significant modification.

In summary, the staff believes that the industry generally supports the proposal to shift responsibility for writing the initial licensing examinations to the facility licensees. Some facility licensees actively lobbied for the change on the basis that it would save them money. Many facility licensees and NEl view the proposed changes as a good first step at increasing the industry's involvement in the examination process and shifting the NRC's role from direct involvement to oversight. Nevertheless, most facility licensees would probably agree that the administrative requirements that were outlined in GL 95-06, refined during the pilot )rogram, and incorporated in Revision 8 of NVREG-1021 have increased tle burden of preparing the examinations.

The issue of industry burden has been a key factor throughout the pilot program. The staff purposely limited the scope of the original process changes to those measures that would help ensure the integrity, consistency, quality, and security of the licensing examinations. The staff was committed to maintaining an effective and efficient licensing program and realized that the revised process, which would have relied on voluntary industry support, would never succeed if it was unnecessarily burdensome on the facility licensees. In keeping with its original commitment, the staff believes that it has taken reasonable measures to address the industry's concerns and recomi.tendations without sacrificing the effectiveness of the examination process.

Pilot Enhancements and Justifications To asse'ss the effectiveness of the new examination process, NRR asked the NRC examiners who were involved with the pilot examinations to respond to a questionnaire regarding various aspects of the process. The first and most significant problem observed and reported by the examiners was that the quality and technical accuracy of the examinations submitted by several facility Itcensees fell short of the NRC's expectations. Moreover, some of the early pilot examination results (i.e., more question deletions and answer key changes than expected) reflected the need for greater emphasis in that area.

Consequently, when NRR prepared draft Revision 8 of NUREG-1021, it incorporsted a number of clarifications and changes that improved the framework established in GL 95-06, in January 1996, the staff issued a preliminary draft of NURE0t-1021 to the regions for review and comment before  !

soliciting comments from the public and the industry the following moi h. The draft revision that was issued for public and industry comment containtd the i fc110 wing process improvements:

o The existing guidance (Revision 7 of NUREG-1021, Revision 5 of NUREG/BR- ,

0122, and GL 95-06) did not specify what percentage of the questions on l the written examination should test the applicants at the comprehension l 21 l

l 1

and analysis levels of knowledge, Many examiners had expressed concern that the lack of guidance in tliis area would lead to inconsistency and a l decrease in the level of examination difficulty. Facility licensees i questioned whether examiners had unwritten guidance that required them to make the examination more difficult by revising or replacing simple memory questions, j As noted earlier, NRR had no desire to increase the average difficulty I of the examinations, but it also did not want the level of difficulty to diminish. Therefore, to better define the a>propriate level of knowledge for the new examination process, tie staff reviewed the written examination audits )erformed in the previous two years to determine an average distri aution of questions by knowledge level. The staff concluded that approximately half of the questions on the examinations reviewed were written at the comprehension or analysis level and that the remaining questions tested simple memory. On the basis of that information, the staff established a criterion in draft NUREG-1021 that at least half of the examination questions be written at the comprehension or analysis level, o The supplemental guidance that NRR arovided to the regions concurrent .

with GL 95-06 included interim QA ciecklists to assist the NRC examiners in reviewing and identifying deficiencies in the draf t examinations.

The checklists were edited and incorporated in the preliminary draft of NUREG-1021 and edited again in response to regional comments. The draft NUREG-1021 that was issued for public comment included QA checklists for the examinat!vn outline, the written examination, both the simulator and walk-through portions of the operating test, and the written examination grading, o did not The existing s)ecify a limitguidance (Revision on the number of 7JPMs of NUREG-1021 that could be and GL 95-06)in walk-repeated tirough tests administered on successive days during tie same initial licensing examination visit or from one visit to the next. Many examiners had expressed concern that the lack of guidance in this area could compromise examination integrity and lead to incoasistency. Other examiners argued that no repetition should be permitted because it would increase the predictability of the test. facility licensees would be tempted to repeat as many JPMs as possible in an effort to save resources.

To strike a balance, the staff concluded that a 30-percent limit would be appropriate. This figure was consistent with the written examination limit of 25-percent duplication of questions from examinations, quizzes, or tests administered to the license applicants or from the past two NRC licensing examinations at the facility.

After issuing draft NUREG-1021 for public and industry comment, the staff identified a number of items that merited clarification and improvement, in addition to reviewing the industry's coments and recommendations (as previously noted), the staff discussed all of the following changes with NEI 22 l

and other industry representatives during a aublic meeting held in the NRC's offices on April 18, 1996. Except as noted >elow, the industry representatives did not object to the additional changes, o To limit the number of technical inaccuracies, a problem that affected l several of the allot written examinations, the staff revised NVREG-1021 l to require the iRC chief examiner or another examiner to independently i review and verify the technical accuracy of a sample of the written examination questions. The number of questions verified would depend upon the expected quality of the examination and the time available before the scheduled review with the facility licensee.

i As a final check of the technical accuracy of the written examination, 1 the staff revised NUREG-1021 to encourage the facility licensee to administer the NRC-approved examination (under security agreements) to one or more licensed personnel who were previously uninvolved in developing the examination. Although feedback from the NRC chief examiners indicated that some facility licensees had taken it upon themselves to validate their examinations in this manner during the pilot program, the facility representatives at the meeting objected to making it a requirement for all facility licensees. Because the staff lacked sufficient information to quantify the value added by this ,

measure, it chose to recommend it as a good practice. l i

o 1. ate in the pilot program, an NRC chief examiner who was reviewing a draft examination detected that the facility licensee had used a written .

examination outline that was identical to one that a sister facility had '

prepared several weeks earlier. The staff had not anticipated that such a problem might occur but immediately realized that this was a significant loophole in the process. Allowing facility licensees to reuse the same examination outline would definitely improve efficiency, but it would also jeopardize the integrity of the examination process by increasing the predictability of the examinations.

To address this problem, the staff included in NUREG-1021 a caution for examination authors and reviewers to ensure that the outline used during succassive audjt and licensing examinations does not become-repetitive and predictable. If a facility licensee proposes to use an outline that was previously used at another facility, it shall identify the source of the outline and explain what effect its reuse is expected to have on examination integrity, o NUREG-1021 has for some time included checklists for use by NRC examiners in verifying that their simulator scenarios contained the required number of transients and ensuring that the scenarios would enable the examiners to evaluate the applicants on every required competency. There were no specific instructions for filling out-the checklists, nor were they required to be retained in the examination file. Concerns regarding the quality and consistency of the operating tests prompted the staff to incorporate instructions for completing the 23

transient and competency checklists and a requirement for the NRC regional offices to retain the checklists in the examination files.

o NUREG-1021 has always endorsed the goal of having the licensing examinations ready to administer approximately a week before they are scheduled to be given. Furthermore, the staff has always encouraged the regions to keep NRR informed if significant problems are encountered with an examination. The pilot program illustrated and magnified the importance of both concepts. Therefore, the staff clarified the guidance in NUREG-1021 to limit the number of last-minute changes and to require the regional office to consult NRR if it appears that the examination cannot be made to conform to the examination standards at least five working days before the scheduled examination date.

Furthermore, the examinations should be postponed if problems are identified at the last minute.

o in addition to implementing the pilot examination process, the staff proposes using Revision 8 of NUREG-1021 as a vehicle for activating Revision 1 of NUREG-ll22 and NUREG-ll23, the NRC's K/A catalogs, which were published in August 1995. The catalogs were revised to make them more current and reorganized to make them more consistent with each other and easier to use. As part of the catalog revisions, the system-generic K/As in both catalogs were combined with the plant-wide generic K/As, and new vendor-specific emergency and abnormal plant evolutions were added to the pressurized water reactor catalog (NUREG-ll22). These changes would have altered the balance of the written examinations unless the sample plans in NUREG-1021 were also revised to compensate for the changes. Because of an oversight, the staff neglected to enter the changes before issuing the draft NUREG-1021 for public and industry comment, o When NRR reviewed the regional office operator licensing programs during February and March 1996, it discovered that some of the examiners had misinterpreted the intent of NUREG-1021 regarding the design of the operating tests for SRO applicants limited to fuel handling. The staff discussed the issue with the regions and agreed that strict compliance with the existing guidance would overtest those license applicants.

Therefore, the guidance in NUREG-1021 was changed to more closely align the scope of the operating test with the range of responsibilities for SR0 fuel handlers.

o Whenever a question is deleted from the 100-point written examination, it results in fractional grades. When the grade is above 79.5 percent, it raises a question regarding the staff's policy on rounding-off because it means the difference between receiving a license or a denial.

To address this problem, the staff has revised NUREG-1021 to indicate that the passing grade on the written examination is 80.00 percent, o Although the pass rates of the pilot examinations were corrparable with pass rates for NRC-written examinations, the rate at which applicants requested that the NRC informally review their examination failures 24

increased unexpectedly during the pilot program. These requests not only placed a significant burden on the staff to resolve the appitcants' issues but also raised concerns regarding the effect that any additional question deletions and answer changes might have on the other licensing decisions that had already been made.

Consequently, the guidance in NUREG-1021 was revised to afford the NRR staff greater discretion in determining the method for resolving appitcant appeals (i.e., every appeal does not necessarily have to be reviewed by a pane' of examiners as long as objectivity and fairness are maintained) and to delay the issuance of licenses to applicants who passed the written examination with insufficient margin to guarantee that the licensing decision will be sustained if additional questions are deleted or changed upon appos1. Furthermore, to minimize the number of appeals, the staff revised NUREG-1021 to encourage facility licensees to solicit and address concerns _from individual license applicants during the process of grading the written examinations, o Problems encountered while reviewing and resolving the operating test

-appeals that were received during the pliot program prompted the staff to reevaluate the documentation and grading requirements in NUREG-1021.

As a result, the staff has v vised NUREG-1021 to permit NRC examiners to downgrade more than two simulator test rating factors for the same error, provided the error caused serious safety consequences for the plant. The staff also increased the documentation requirements for the simulator test so that the NRC examiners would have to describe any error that prompted them to downgrade the applicant on any rating factor.

During the meeting of April 18, 1996, the staff also discussed the possibility of permitting NRC examiners to recommend a failure on the simulator test even if the a>plicant did not fail the test on the basis of the competency scores. 11e industry re)resentatives did not believe that such a change would be necessary if tiere were no limit on the number of competencies that the examiner could downgrade for a single, serious error; however, they did not object to the change in principle.

Therefore, the staff has included in NUREG-1021 a provision for NRC examiners to recommend such a failure if the appitcant committed an error with serious safety consequences for the plant or the public. To limit its use, the region must obtain written concurrence from the Chief. HOLB, before completing the Itcensing action.

The following issues were raised by NRC examiners on the pilot examination feedback surveys but did not result in specific changes to the examination methodology described in NVREG-1021.

o To minimize NRC resource requirements, the staff had encouraged the regions to review and prepare for the pilot examinations without conducting a preparatory visit to the site. However, feedback from those examiners who did not conduct such a visit indicated that it would have facilitated the review and resulted in fewer problems with the 25

l examination. They recommended that a preparatory visit be required for every examination. Althou NUREG-1021 (i.e., the pre)gh NRR has not changed the guidance inaratory visit is option regional management), it 1as acknowledged the value of conducting at least part of the examination review and validation at the facility.

The average NRC resource expenditures during the pilot examinations indicated that the trips can be made within the original budget model, o Several NRC examiners recommended that the required due dates for the examination materials from the facility licensees be advanced so that the examiners have more flexibility in scheduling their time for the examination review. The staff did not change the guidance in NUREG-1021 because it already permits the regional offices to work with the facility licensees in determining the appropriate due dates and because a generic change would increase the scheduling burden on all facility licensees. It would not be appropriate for an examiner to demand the examination 60 days in advance because it was required by NUREG-1021, only to have the examination sit untouched in the office for 30 days s

because the examiner was involved with another project.

o When the staff developed the pilot examination process, there was some debate as to whether the facility personnel who administered the written examination should be permitted to answer the a)plicants' questions regarding the intent of specific questions on tie examination. The pilot guidance allowed the facility to answer the applicants' questions with caution but required every question and response to be documented for subsequent review by the NRC. The policy was evaluated on the pilot examination questionnaire that every NRC chief examiner was required to complete. A minority of the examiners recommended that the policy be chagedtoprohibittheapplicantsfromaskingquestionsduringthe writ s examination, as is the policy on the dFE. However, most of the examiwrs concurred with the original pilot policy, and none of the comments indicated that the policy had created any problems with examination integrity. Therefore, the staff saw no reason to revise the guidance in NUREG-1021.

O Some examiners noted that the pilot program personnel restrictions had hampered the facility licensees' ability to develop the examinations because they could not assign their best people to the task. This issue was also addressed in the industry comments, as previously discussed.

[xaminatian_R uults Since issuance of SECY-96-123 The staff completed the last of the pilot examinations on April 5,1996, and resumed conducting the licensing examinations using the traditional development process. After April 12, 1996, when the Executive Director for Operations informed the Commission of the staff's intent to continue the pilot process beyond the period previously discussed with the Commission, the staff proceeded to contact those facility licensees whose examination requests had not yet been assigned to a contractor to determine if they were interested in developing the examinations using the pilot methodology. The first of those 26

I examinations was administered on August 19, 1996, and the results were not available in time to report them here.

Since SECY-96-123 was prepared, the staff has used the traditional examination '

methodology to prepare licensing examinations for applicants at 14 power reactor facilities. The results of the completed examinaticas are summarized below; four of the examinations were completed late in August and the results were not available at the time this paper was prepared. Thelicensing examination results for fiscal year 1995 and the 22 pilot exaatnations conducted between October 1, 1995 and April 5, 1996, are provided for comparison.

Pass Rates' R0 R0 R0 SRO SRO SRO Examinations Operating Written Operating Total Written Total 39/40 39/39 39/40 45/48' 43/48 41/48' Since SECY-96-123 (98%) (100%) (98%) (94%) (90%) (85%)

Pilot 49/54 50/54 45/54 86/92 87/918 83/92 8

Examinations (91%) (93%) (83%) (93%) (96%) (90%)

Fiscal Year 1995 94% 98% 92% 95% 95% 92%

1. Each box indicates the number of applicants who passed that part of the examination over the number of applicants who took it.

2. One of the SRO applicants who failed the written examination has filed an appeal that was not yet resolved at the time this paper was prepared.

3. Two applicants who were reported as preliminary failures in SECV-96-123 filed appeals that resulted in the failures being overturned.

lunjagy The staff believes that the revised examination process can be implemented at all power reactor facilities with the same level of direct NRC resources as is currently allotted to the operator licensing program. Full implementation of the revised process will enable the staff to eliminate the use of operator licensing contractors (with the exception of those who develop and grade the GFE) that have historically cost the NRC (and the industry) approximately

$3 million to $4 million per year. The staff expects that the change could result in a net resource savings to the facility licensees over time because they will be able to prepare the examinations more efficiently than the NRC or its contractors.

The overall results of the pilot program indicated that the examinations prepared by facility licensees, subject to review, revision where ap)ropriate, and approval by NRC staff examiners, were generally of the same caliser as examinations prepared by NRC or contract examiners. The pilot examinations also appeared to be equally effective at identifying applicants who had not 27

mastered the job requirements well enough to become licensed R0s or SR0s.

Furthermore, the pass rates for R0s and SR0s on the written examinations and operating tests compared favorably with the pass rates for examinations prepared by the NRC or its contractors.

However, despite the mitigating actions previously noted, the revised examination process is much more dependent upon the abilities of the NRC chief examiners to identify weak examinations and work with the facility licensee to upgrade the examination. Furthermore, it is possible that the pressures to conduct examinations on schedule and the reduced involvement by the NRC in the examination development process could lead to examinations that reflect the biases of the facility licensees and that discriminate at a lower level than is currently the norm. Moreover, it is reasonable to expect that industry-wide implementation of the revised process will increase the chance of future examination compromises as a result of the larger number of individuals who would require access to the examinations and the additional pressures on facility employees to ensure the success of the applicants.

Although a number of the examinations submitted by the facility licensees did not meet NRC standards for level of knowledge or difficulty, the NRC chief examiners effectively worked with the facility licensees to revise the examinations so that they would discriminate at the appropriate level. To put this situation into perspective, it has not been too unusual for contractor-developed examinations to fall short of the NRC's requirements and expectations. The chief examiner often directs the contract examiner to make numerous corrections and changes before reviewing the examination with the facility licensen, only to have the facility wiewer identify a host of additional errors and inconsistencies.

The staff has advised facility licensees that the NRC expects the technical quality of examinations to remain high and the number of post-examination changes to be low. Similarly, regional examiners and managers have been instructed not to approve or administer any examination that does not adhere to NRC standards for content, format, quality, level of knowledge, and level of difficulty. The QA checklists in NUREG-1021 have been modified to maintain examination consistency and to facilitate the efforts of NRC examiners to detect deficient examination materials. The use of any facility-prepared examination that is found to be unacceptable will be delayed or cancelled, if necessary, until the facility licensee and the NRC can upgrade or replace the examination.

The supplemental pilot examination guidance that the staff arovided to the regional offices (and placed in the PDR) in conjunction witi GL 95-06 directed the regions to document in the examination report any significant problems encountered during the examination development, administration, and grading.

This requirement has also been included in Revision 8 of NUREG-1021. The regions are ex)ected to address any serious deficiencies in the examinations submitted by tie facility licensees and any security concerns during the entire examination process. If appropriate, the region will request the facility licensee to describe the actions it will take to improve future performance, if the facility licensee is unable or unwilling to prepare 28 l

E appropriate licensing examinations, the NRC has the responsibility and authority to develop the examinations in the traditional manner.

Overall, the staff does not expect that the pro >osed revisions to the o>erator licensing program will significantly increase tie risk to the life and 1ealth of any individual members of the public or that they will be a significant addition to other societal risks. The staff believes that the licensing decisions that were made on the basis of the pilot examinations were as valid as those made using the traditional examination process. Furthermore, the changes that the staff has made in NUREG-1021 as a result of lessons learned duringthepilotprogramandindustrycommentssubmittedinresponsetothe staff s solicitation (which were previously discussed in detail) should only improve the effectiveness of the examinations if and when the revised process is implemented on an industry-wide basis.

The NRC will continue to set the standard for operator performance by ensuring that the facility licensees maintain appropriately high standards for examination development. The staff will remain actively involved in the operator licensing process by reviewing and approving every written examination and operating test before it is administered. Unacceptable examinations will be revised and delayed, if necessary, until the resources are available to upgrade the examinations. Furthermore, the fact that NRC examiners will continue to independently administer the operating tests will allay concerns regarding bias and conflicts of interest on that the portion of the licensing examination having the most operational validity and requiring the most sub;ective judgements.

The safety consequences of occasionally administering a substandard examination are probably minimal. By the time the applicant takes the NRC licensing examination, he or she will have completed a comprehensive, tystematically constructed and implemented training program designed to maximize the likelihood of success on the examination. The license training programs typically include numerous written examinations and performance demonstrations that culminate in certification by the facility licensee that the applicant has successfully completed the facility licensee's requirements to be licensed as an RO or an SRO. The effectiveness with which the facility licensees screen out those trainees that have not mastered the job requirements has been evident in the high pass rates on the licensing examinations during the past several years. In most cases, the NRC examination is simply a confirmation of the facility licensee's decision to submit the application.

Historically, even those applicants who fail the NRC written examination generally score above 75 percent. The average grade for the 11 applicants who failed a pilot examination was 76.3 percent. This statistic indicates that the vast majority of ap)1icants, including most of those who fail the licensing examination, save attained an adequate level of knowledge so as not

'to pose a significant threat to reactor safety. Furthermore, any appitcant who does become. licensed on the basis of an examination that discriminated at a lower than normal level would be backed up by other crew n,emers who could prevent or remedy any operating errors. Although there is nothing to prevent a facility licensee from forming an operating crew entirely from newly 29

licensed operators, most facility licensees integrate their new operators with existing crews who possess significant operating experience.

e z

n Y_

'l f

M 30 E

_ _ _ _ _ _ _ _ _ _ _ _ _ . _ . _ _ _ _.