ML20195G579
| ML20195G579 | |
| Person / Time | |
|---|---|
| Issue date: | 05/13/1998 |
| From: | Roe J NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES) |
| To: | Funches J, Gray J, Lieberman J, Paperiello C, Shelton B NRC OFFICE OF ENFORCEMENT (OE), NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM), NRC OFFICE OF NUCLEAR MATERIAL SAFETY & SAFEGUARDS (NMSS), NRC OFFICE OF THE CONTROLLER, NRC OFFICE OF THE GENERAL COUNSEL (OGC) |
| Shared Package | |
| ML20195E260 | List: |
| References | |
| FRN-64FR19868, RULE-PR-55 AF62-2, AF62-2-006, AF62-2-6, NUDOCS 9906160108 | |
| Download: ML20195G579 (92) | |
Text
i-0 f?
QFg z
't y
'k UNITED STATES g
j NUCLEAR REGULATORY CCMMISSION Pon f
WAliilNOTON, D.C. 2004&4001
. o, s, *****/
May 13, 1998 j
MEMORANDUM TO:
Carl J. Paperiello, Director Office of Nuclear Material Safety and Safeguards i
Joseph R. Gray, Associate General Counsel for Licensing and Regulation Office of the General Counsel Jesse L. Funches Chief Financial Officer Brenda Jo. Shelton, Chief Records Management Branch Information Management Division Office of the Chief Information Officer James Lieberman, Director Office of Enforcement David L. Meyer, Chief Rules and Directives Branch Division of Administrative Services Office of Administration h
FROD:
Jack W. Roe, Acting Directo Division of Reactor Program anagement Office of Nuclear Regulatory Research
SUBJECT:
FINAL RULE -INITIAL LICENSED OPERATOR EXAMINATION I
REQUIREMENTS, AMENDMENTS TO 10 CFR PART 55
,Y s
Your concurrence is requested on the attached final rule.
1 Background This final rule is being published to notify the public of NRC's decision to require l
Its power reactor facility licensees to prepare and administer the initial operator and senior operator licensing examinations. Section 107 of the Atomic Energy Act of 1954 (AEA), as I
amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55,
" Operators' Licenses," to pass an examination satisfying the basic content requirements that are also specified in that regulation. Traditionally, the NRC has prepared these examinations through its staff or contractor examiners. This rule, ;f promulgated, will require the facility licensees to prepare these examinations and proctor and grade the written portion.
E i 2.]
~
~y' k3.UO[
9906160108 990608 PDR PR 55 64FR19868 PDR j
j hoc / ccsor
~~
~~~
g.
(-
' C. J. Paperiello, et al~.
I j
The notice of proposed rulemaking was published in the Feders/ Register (tii2 FR 42426) on
" August 7,- 1997, for a 75-day comment period. The NRC received a total of 13 comment letters on the proposed rule; two of the letters arrived after the comment period closed. The respondents included three NRC examiners, one contract examiner, five nuclear utilities and one utility employee, one nonpower reactor facility licensee, the State of Illinois, and the Nuclear Energy Institute (NEI), which submitted its comments on behalf of the nuclear power industry.
The staff has carefully considered the public comments received and has made modifications to the proposed rule and the guidance in Revision 8 of NUREG-1021, " Operator Licensing Examination Standards for Power Reactors," as appropriate. The staff is now proposing to publish the final rule in the Feders/ Register.
q i
The following is a summary of this request:
]
1.
Iltla: Final Rule, " Initial Licensed Operator Examination Requireme nts" 2.
Task Leader: Harry S. Tovmasslan,' NRR/DRPM (415-6231)
I 3.
Working Group Members: Siegfried Guenther, NRR/DRCH (415-1056) 4.
Steering Group Members No 5.
Enhanced Public Particination: No-6.
Level 1 or 2 Item of Comoatibility for AareemfM.Stataa No 7.
Reauested Action: Office Concurrence j
8.-
Reauested Comoletion Date: Two weeks from receipt of memorandum 9.
Resources and Coordination Estimated resources to develop this rulemaking are 0.5 FTEs.- Copies of this concurrence package have been forwarded to the OPA, CRGR, ACRS, and OlG for information.
Attachment:
As stated
)
cc w/ attachment:
H. Bell, OlG W. Beecher, OPA.
J. Larkins, ACRS T. Martin, AEOD (CRGR) 4
)
[
^
p r
L A
r '-
l'
- +_
[.-
(
l s
4
^
{
EQB::
The Commissioners,
l t
L EBQM: J L. Joseph Callan ;.
L Executive Director for Operations l
SUBJECT:
FINAL RULE'- REQUIREMENTS FOR INITIAL OPERATOR LICENS!NG l
EXAMINATIONS j
PURPOSE:
l To obtain Commission approval to publish a final amendment to 10 CFR Part 55 dealing with tiie preparation of operator licensing examinations by facility licensees.
1 BACKGROUND:
On March 24,1995, the staff informed the Commission of its intent to revise the operator licensing program to allow greater participation by facility licensees and to eliminate contractor assistance in this area (SECY-96-075, " Proposed Changes to the NRC Operator Licensing Program").' In a staff requirements memorandum (SRM) of April 18,1995, the Commission approved the staff's proposal to initiate a transition process to revise the operator licensing.
' program and directed the staff to carefully consider experience from pilot examinations before fully implementing the changes. On August 15,1995, the NRC staff issued Generic Letter (GL) i
- 95-06, " Changes in the Operator Licensing Program," outlining the revised process for developing examinations, and soliciting volunteers to participate in pilot examinaticns to evaluate and refine the methodology.
Between October 1,1995, and April 5,1996, the NRC staff administered 22 operator licensing examinations using the revised process. The staff documented the results of the pilot '
u examinations in SECY-96-123, " Proposed Changes to the NRC Operator Licensing _ Program,"
.m and briefed the Commission on June 18,1996.= In an SRM of July 23,1996, the Commission directed the staff to prepare a rulemaking plan to just4y the changes to 10 CFR Part 55 and to J present additional information on a number of issues related to the revised examination process. The SRM also authorized the staff to continue the pilot program on a voluntary basis pending a final Commission decision on the revised examination process.
CONTACT:
Rooert Gallo, NRR 301-415-1031.
e
The Commissioners On September 25,1996, the NRC staff issued SECY 96-206, "Rulemaking Plan for Amendments to 10 CFR Part 55 To Change Licensed Operator Examination Requircrnents."
Upon Commission approval of the rulemaking plan in an SRM of December 17,1996, the staff prepared the proposed rulemaking (SECY-97-79, dated April 8,1997), which was approved and modified in accordance with an SRM of June 26,1997, and published in the Federal Register on August 7,1997 (62 FR 42426). When the comment period expired on October 21,.1997,11 comment letters had been received and 2 more arrived after the expiration date. The comments from all 13 letters are summarized, and NRC's responses are provided in the Statement of Consideration for the Final Rule (Enclosure 2).
DISCUSSION-From the time the pilot program began in October 1995 through the end of September 1997, the NRC staff reviewed, approved, and adminlaitered a total of 68 examinations that were voluntarily developed by facility licensees in accordance with the NRC's examination criteria and guidance.
Facility licensees prepared the written examinations and the operating tests, proctored the written examinations, and graded the written examinations using the guidance provided by the NRC in GL 95-06 during the early stages of the pilot program, and subsequently in interim Revision 8 of NUREG-1021, " Operator Licensing Examination Standards for Power Reactors."
NRC examiners thoroughly reviewed the examinations and tests to determine if they were consistent with NRC standards, directed facility licensees to make whatever changes were necessary to achieve NRC standards, and approved the examinations and tests before they were administered.~ NRC examiners independently administered all of the operating tests, reviewed the written examination grading, and made the final licensing recommendations for approval by NRC management pursuant to 10 CFR 55.51.
The revised examination process makes the operator licensing program more consistent with the NRC's other oversight programs because it requires NRC examiners to review materials prepared by the facility licensees and holds the facility licensees responsible for the quality of those materials. The revised process has enabled NRC examiners to focus more on the psychometric quality of the examinations (e.g., the cognitive level at which the questions are written and the plausibility of the distractors or wrong answer choices) prepared by the facility licensees than on the technical accuracy of the examinations, which was their primary focus when the examinations were prepared by the NRC. This shift in the NRC examiners' focus, whic is made possible by the revised examination development process, has the potential to improve the overall quality of the licensing examinations.
The NRC reviews of the 68 examinations showed that the quality of the licensee-proposed examinations varied widely; essentially all of the proposed examinations required some changes and many needed significant changes to achieve NRC standards for quality and level of difficulty. In some cases, the examinations were delayed for a period of tima in order for the NRC staff to evaluate the examinations and review the facility licensees' changes. Feedback from the NRC examiners who were involved with the pilot examinations indicates that the facility-prepared examinations that were finally approved by the NRC were comparable, in
' terms of quality and level of difficulty, to examinations prepared by the NRC before and since
~,;
e
~
i e
The Commissioners beginning the pilot program. Although the staff expected that the quality of the proposted examinations would improve as the industry became familiar with NRC examination criteria and expectations and gained experience in preparing the examinations, some of the proposed examinations continue to reqdire more changes than anticipated. The staff's expectations regarding the quclity and level of difficulty of the proposed examinations have not yet been achieved. (Enclosure 1 provides a discussion of this issue and actions that are being taken to address it.)
The average passing rates for reactor operators (ROs) and senior reactor operators (SROs) on both the written and operating portions of the 68 facility-prepared examinations administered through the end of fiscal year (FY) 1997 were somewhat lower than the corresponding passing rates on the examinations administered during FY 1995, the last year in which all of the examinations were prepared by the NRC or its contractors. However, the average passing rates were not significantly below the range of passing rates for FYs 1990 through 1994. This supports the NRC staff's conclusion that the facility-prepared examinations, revised as directed by the NRC, are effective. The examinations are discriminating at a conservative and acceptable level (i.e., the average level of difficulty has not declined). (Enclosure 1 includes a detailed summary and analyr,is of the examination results.)
With regard to the efficiency of the revised examination process, the pilot examination experience to date suggests that the average industry cost will not be significantly different from
' what it was when the NRC prepared all of the licensing examinations. Feedback from the industry reflects that the cost for some faci 2y licensees was higher than it was for an NRC-prepared examination; other licensees, howew, were able to prepare acceptable quality examinations at te lower cost than if the examinatios had been prepared by the NRC. The industry generally attributed the higher cost to the revised examination and administrative criteria under the pilot examination process. Although the NRC staff acknowledges that the revised criteria contribute to the elevated cost, many of the variables that affect the final cost of the examination (e.g., its initial quality) are now under the control of facility licensees. Those facility licensees that submit acceptable quality examinations are likely to save resources despite the additional administrative criteria that the NRC considers necessary under the revised examination process. However, those facility licensees that submit examinations that require many changes because they do not meet NRC standards, may have increased costs.
The higher cost of revising a low quality examination should provide an incentive for facility licensees to submit acceptable quality examinations. (Enclosure 3 provides an analysis of the industry resource burden.)
Comments from the NRC chief examiners who worked on the pilot examinations indicate that the average amount of time spent reviewing and revising the facility prepared examinations was generally consistent with the estim.ates developed before starting the pilot program. Although it took longer than expected to reviso a number of the examinations to meet NRC standards for quality, tha resource burden was generally offset by other examinations that required less effort to review and revise. Therefore, the NRC staff has concluded that the original objective of the pilot examination and transition program has been achieved. The program has demonstrated that the NRC can eliminate the use of contractors in the operator licensing program (except for i
the generic fundamentals examination) and can conduct ti,e initial operator licensing and a
4
r-j The Commissioners 3 requalification irispection programs with the existing NRC staff. (Enclosures 1 and 3 provide an analysis of the NRC resource burden.)
As discussed in SECY-96-206 and the proposed rulemaking, the NRC staff is focusing additional attention on examinati6n security. Therefore, the staff has amended 10 CFR 55.49 in the Final Rule to ensure that applicants, licensees, and facility licensees understand the scope and intent of the regulation and to require facility licensees to establish procedures to control examination security and integrity. The staff has also strengthened the discussion of examination securhy in final Revision 8 of NUREG-1021 and modified NUREG-1600, " General
- Statement of Policy and Procedures for NRC Enforcement Actions," to address enforcement
- action relative to the requirements in 10 CFR 55.49. As a separate action, the NRC will delay any examination that may have been compromised until the scope of the potential compromise is determined and measures can be taken to address the integrity and validity of the examination.
In summary, the NRC staff has concluded that the revised examination process is both effective and efficient. The staff has reviewed the vulnerabilities discussed in SECY-96-206 (i.e., quality i
L and consistency; independence and public perception; examination security; NRC resources; j
- program stability; and examiner proficiency) and evaluated the measures that have been taken to mitigate the vulnerabilities. (Enclosure 1 includes a summary analysis of each vulnerability and the potential effects on reactor safety.) The revised program provides the staff the ability to i
maintain acceptable standards of performance in each of these areas, and the pilot program has demonstrated that it is capable of doing so. NRC license examiners will continue to review each examination to ensure that it conforms to the guidelines in NUREG-1021 regarding content, format, quality, and levels of knowledge and difficulty. The NRC will direct facility licensees to revise the examinations, as necessary, and will approve every examination before it is given. The NRC will delay those examinations that do not meet NRC standards by the scheduled administration date and, to the extent that facility employees do not follow the examination criteria in NUREG-1021, address deficiencies in the submitted examinations as licensee performance issues. Moreover, NRC examiners will directly observe and evaluate the performance of every license applicant on both the dynamic simulator and walk-through portions of the operating test. The staff believes that this willimprove the NRC's ability to focus on operator performance issues and enable NRC examiners to accrue more experience in a shorter period of time.
COORDINATION:
The Office of the General Counsel (OGC) has no legal objection to this final rule. As discussed in the Federa/ Reg / ster notice (Enclosure 2), OGC has determined that the backfit rule (10 CFR 50.109) does not apply. The Office of the Chief Financial Officer has reviewed this Commission paper for resource impacts and has no objections. This final rule has also been reviewed by the Advisory Committee on Reactor Safeguards (ACRS) and the Committee to Review Generic Requirements (CRGR).
-j c
t
t The Commissioners RESOURCES:
This action has allowed the staff to discontinue the use of contractor support for the i
development and administration of site-specific initial operator licensing examinations. Before
. initiating the pilot examination and transition process at the beginning of Fiscal Year (FY) 1996,
. the NRC spent approximdely $3 million per year on contractor assistance for initial examinations and requalification inspections. During FY 1996, when approximately 40 percent of the examinations were voluntarily prepared by facility licensees, the NRC spent j
approximately $1.6 million for contractor assistance (to prepare and administer initial examinations and assist with requalification inspections). In FY 1997, facility participation
~ increased to include approximately 75 percent of the examinations, and the NRC's spendir,g on
. contractor assistanc+ for the licensing examinations and requalification inspections decreased to approximately $0.5 million.
The FY i998 and FY 1999 budgets are consistent with this proposal and reflect the elimination of contractor support for the operator licensing program (with the exception of the generic fundamentals examination). if the Commission decides not to amend 10 CFR Part 55 as recommended by the ' RC staff, other measures will be needed to ensure that the facility N
licensees' requirements for licensed operators are satisfied. Such measures might include (1) examination length or format changes that will enable the existing staff to conduct the examination and inspection programs, (2) scheduling the examinations based on the NRC resources available (which may require the ' staff to prioritize its activities based on the facility licensees' needs), (3) an increase in direct examiner resources in each regional office to satisfy the demand for initial licensing examinations, or (4) the restoration of contractor support for the operator licensing program.
RECOMMENDATIONS:
That the Commission
'1.
Approve the notice of final rulemaking (Enclosure 2) for publication in the Federal Register.
2.
Certsfy that this rule, if promulgated, would not have a significant economic impact on a substantial number of small entities to satisfy the requirements of the Regulatory Flexibility Act,5 U.S.C. 605(b).
3.
Determine that the backfit rule,10 CFR 50.109, does not apply to this rule.
'4.
Determine that neither an environmental impact statement nor an environmental assessment hae been prepared because this proposed rule is eligible for a categorical exclusion as defined in 10 CFR 51.22(c)(1).
The Commissioners ' -
- 5. ~
hinig that a.
The Chief Counsel for Advocacy of the Small Business Administration will be informed of the certification regarding economic impact on small entities and the reasons for it as required by the Regulatory Flexibility Act.
b.
The final rule contains information collection requirements that have been reviewed and approved by the Office of Management and Budget (OMB).
l c.
' A regulatory analysis has been prepared (Enclosure 3),
d.-
A revision to the enforcement policy is being prepared arid will be issued separately.
e.
' This is not a " major" rule as defined in the Small Business Regulatory Enforcement Fairness Act of 1996,5 U.S.C. 804(2) f.
The appropriate Congressional committees will be informed (Enclosure 4).
g.
A public announcement will be issued when the final rulemaking is filed with the Office of the Federal Register (Enclosure 5).
h.
Copies of the notice of final rulemaking end final Revision 8 of NUREG-1021,
" Operator Licensing Examination Standards for Power Reactors," will be distributed to all facility licensees.-
L. Joseph Callan 4
Executive Director for Operations
Enclosures:
As sieted (5) l l
The Commissioners ' 5.
Main that a.
The Chief Counsel for Advocacy of the Small Business Administration will be informed of the certification regarding economic impact on small entities and the reasons for it as required by the Regulatory Flexibility Act.
b.
The final rule contains information collection requirements that have been reviewed and approved by the Office of Management and Budget (OMB).
c.
A regulatory analysis has been prepared (Enclosure 3).
d.
A revision to the enforcement policy is being prepared and will be issued separately.
e.
This is not a " major" rule as defined in the Small Business Regulatory Enforcement Faimess Act of 1996, 5 U.S.C. 804(2).
f.
The appropriate Congressional committees will be informed (Enclosure 4).
i g.
A public announcement will be issued when the final rulemaking is filed with the Office of the Federal Register (Enclosure 5).
h.
Copies of the notice of final rulemaking and final Revision 8 of NUREG-1021,
" Operator Licensing Examination Standards for Power Reactors," will be distrDuted to all facility licensees.
L. Joseph Callan i
Executive Director for Operations
Enclosures:
As stated (5)
DISTRIBUTION:
. Commissioners i
OGC OPA OlG CIO EDO ACRS OCCA OCA CFO SECY REGIONS To recehre a com' of thee document, indicate in the box: "C" e Copy w/o attachment / enclosure "E" = Copy w/ sttachme,nt/ enclosure "N" = Dio copy l OFFICE NMSS l
OGC l
OE I
ADM l
OCFO l
l NAME CJ Paperiello JR Gray J L4berman DL Meyer JL Funches lDATE
/ /98
/ /98
/ 198
/ /98
/ /98 IOFFICE OClo l
[_
EDO l
l l
NAME BJ Shelton S Cothns LJ CaHan DATE
/ /98
/ /98
/ /98
/ /98
/ /98 OFFICIAL RECORD COPY
j Pilot Program Results, Changes, and Vulnerabilities i
PILOT PROGRAM RESULTS, CHANGES, AND VULNERABILITIES
1 Background
In a staff requirements memorandum (SRM) of April 18,1995, the Commission approved the NRC staff's proposal to initiate a transition process to revise the operator licensing program to require power reactor licensees to prepare the initial licensing examinations for their own
' facilities and directed the NRC staff (hereafter staff) to carefully consider experience gained from a pilot examination program before fully implementing the changes. With the Commission's approval, the staff issued Generic Le'ter (GL) 95-06, " Changes in the Operator Licensing Program," on August 15,1995, outlining the revised examination development process and soliciting volunteers to participate in a pilot examination program to evaluate and refine the methodology.
)
Between October 1,1995, and April 5,1996, the staff reviewed and approved 22 operator licensing examinations prepared by facility licensees within the framework of a pilot program.
These r 'sminations were prepared using the guidance in Revision 7 (Supplement 1) of NURE11021, " Operator Licensing Examiner Standards," and the additional guidance in j
. GL 95-06. The results of the pilot examinations were discussed in SECY-96-123, " Proposed Changes to the NRC Operator Licensing Program," dated June 10,1996, and lessons leamed
, from those examinations were incorporated in interim Revision 8 of NUREG-1021, which was retitled " Operator Licensing Examination Standards for Power Reactora." Based on the results of the pilot examinations, the staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to prepare the initial licensing examinations and to proctor and grade the written portion of the examinations.
In an SRM of July 23,1996, the Commission authorized the staff to continue the pilot examination process on a voluntary basis and directed the staff to develop a detailed
- rulemaking plan to justify the changes that may be necessary to 10 CFR Part 55. The i
Commission also directed the staff to address the pros, cons, and vulnerabilities associated
. with the revised examination process to fscilitate a Commission decision on whetNu to implement the revised process on an industrywide basis. With the Commission's approval, the staff resumed the pilot program in August 1996 (after having completed the original 22 examinations in April 1996), and by the end of Fiscal Year (FY) 1997 had reviewed, approved,
- and administered 46 additional examinations developed by facility licensees in accordance with i
NRC guidance, thereby raising the total number of pilot-style examinations to 68. The results of those examinations are discussed in Section l DN.
On September 25,1996, the staff forwarded the requested rulemaking plan and a discussion of the pros, cons, and vulnerabilities of the pilot process (including a detailed discussion of the measures taken to mitigate the vulnerabilit'es) to the Commission in SECY-96-206, "Rulemaking Plan for Amendments to 10 CFR Part 55 to Change Licensed Operator Examination Requirements." in an SRM of December 17,1996, the Commission directed the staff to implement interim Revision 8 of NUREG-1021 on a voluntary basis ard to proceed with
' the proposed rulemaking. The staff forwarded the proposed rule (SECY-97-079," Proposed Rule - Initial Licensed Operator Examination Requirements") to the Commission on 1
y i
- April 8,'1997,' and on June 26,1997, the Commission approved its publication in the Federal Registerfor a 75-day comment period. Tne notice of proposed rulemaking (62 FR 42426) dated August 8,- 1997, also invited public comments on interim Revision 8 of NUREG-1021.
~
l Public comments related to the regulation are addressed in the Federal Register notice (FRN)
< (Enclosure 2), and significant comments regarding interim Revision 8 of NUREG-1021 are i
discussed in Section ll below (the staff also addressed several editorial comments during the revision process)..
The staff has reviewed the vulnerabilities discussed in SECY-96-206 (i.e., quality and consistency, independence and public perception, examination security, NRC resources,.
program stability, and examiner proficiency) in light of the additional experience with the revised
.~ examination process since that paper was issued on September 25,1996. The status of each -
vulnerability, including an overview of the compensatory measures, and the potential effects that the revised process might have on reactor safety are summarized in Section lll below.
Pilot Prooram Overview i-From the time the pilot program began in October 1995 through the end of September 1997, the NRC staff reviewed, approved, and' administered a total of 68 examinations that were voluntarily developed by facility licensees under the pilot examination and transition program.
l L
' Facility licensees prepared the written examinations and the operating tests, proctored the -
written examinations, and graded the written examinations using the guidance provided by the NRC in GL 95-06 during the early stages'of the pilot program, and subsequently in interim Revision 8 of NUREG-1021, " Operator Licensing Examination Standards for Power Reactors."
NRC examiners thoroughly reviewed the examinations and tests to determine if they were consistent with NRC standards, directed facility. licensees to make whatever changes were necessary to achieve NRC standards if the submitted examinations and tests were deficient, and approved the examinations and tests before they were administered. NRC examiners independently administered all of the operating tests, reviewed the written examination grading, and made the final licensing recommendations for approval by NRC management.
.i i
i 2
SECTION I PILOT PROGRAM RESULTS The SRM for SECY-96-206 directed the staff to (1) continue monitoring the results of the pilot examinations for indications that the revised examination development process may not be appropriate and (2) to present the current data to the Commission along with the proposed and
' final rulemaking packages.' Accordingly, Table 1.1 summarizes the results of the 68 pilot examinations conducted between October 1995 (the beginning of the pilot program) and the C) end of September 1997. For purposes of comparison, Table 1.1 also includes the examination results for FYs 1990 through 1995 (the last year in which the NRC prepared all of the examinations) and the examinations prepared by the NRC since the pilot program was initiated (i.e., for FYs 1996 and 1997). In conducting its analysis, the staff has focused primarily on the.
annual examination passing rates because the small number of applicants on most
- examinations makes it ddficult to draw programmatic conclusions from the individual examination results.
Table 1.1 - Comparison of Examinaten Passing Rates' Exams RO Written RO Operating SRO Written SRO Operating FY 1990- 1994 Range 92 - 95 %
95 98 %
94 - 08 %
93 - 96 %
FY 1995 (Last year all NRC) 94 %
98 %
96 %
95%
Facility (27) 93%
94 %
94 %
95%
FY 1996 NRC (41) 98 %
94 %
96 %
92 %
l Total (68) 96 %
94 %
96 %
94 %
i Facility (41) 89%
94 %
93%
92%
FY 1997 NRC (13)2 96 %
93 %
918 %
84'%
Total (54) 91 %
93 %
93%
91 %
Facility Total (68) 91 %
94 %
93 %
94 %
(FY 1996 and FY 1997)
- 1. The results indicate the percentage of applicants who passed the examination or test.
- 2. One additional NRC-prepared SRO examination was administered, but the facility licensee subsequently withdrew the applications and the results were nullified. If those results are included, the written and operating test passing rates would be 82 and 76 percent, respectively.-
The average passing rates for reactor operators (ROs) and senior reactor operators (SROs) on both the written and operating portions of the 68 facility-prepared examinations administered through the end of FY 1997 were only slightly lower than the corresponding passing rates on the NRC-prepared examinations administered during FY 1995, the last year in which all of the examinations were prepared by the NRC or its contractors. However, as shown in Table 1.1, the average passing rates tend to fluctuate, and the passing rates on the facility-prepared examinations were generally consistent with the historical range. The fluctuations in the 3
i i
i
)
passing rates from year to year and between the examinations prepared by the NRC and facility
' licensees could be caused by a number of factors including variations in the average level of experience of the license applicants, changes in the quality of the training or the facility -
licensee's threshold for screening its applicants before they take the licensing examination, or variations in the average level of difficulty of the examinations. Even if it is assumed that all of the variation in the passing rates on the facility-prepared examinations has been caused by the changes in the examinations, the magnitude and direction of the variation supports the staff's conclusion that, on the average, the facility-prepared examinations, revised as directed by the NRC, are effective. The examinations are discriminating at a conservative and acceptable level (i.e., the average level of difficulty has not declined and any possible increase is not considered to be of practical significance),'and the safe operation of the facilities is not being compromised.
i If the lower passing rate on the facility-prepared written examinations is indeed an indication
- that the level of difficulty of the examinations has increased, it may be a reflection of the NRC examiners' concentration on the psychometric quality (e.g., the cognitive level at which the
-. questions are written and the plausibility of the distractors - those answer choices that should appear reasonable but, nevertheless, are wrong) in addition to the technical quality of the examinations. Although the staff did not intend for the level of difficulty or the failure rate on the examinations to increase, the examiners' efforts to achieve NRC standards regarding the
. cognitive level of questions and to improve the plausibility of the distractors may have improved the discrimination validity of the examinations. Consequently, those applicants who may have passed an examination containing lower cognitive level questions on which some of the distractors could be eliminated as implausible are now having more difficulty selectird the correct answers. The NRC examiners' efforts to improve the plausibility of the distractors has
- reduced the applicants' chances of passing the examination by simply guessing the corr 6ct answers. However, considering the historical fluctuation iu the average examination passing rates and the other factors (e.g., training deficiencies or less stringent screening of applicants by facility licensees) that could be responsible for some or all of the observed decline, the staff has concluded that any increase in the level of difficulty is not significant. (The level of difficulty of the written examinationc is further discussed below in connection with the review of pilot examination quality.) -
The RO passing rates on the facility-prepared operating tests during FYs 1996 and 1997 were consistent with the passing rates on the tests prepared by the NRC during those came years.
The 98 percent RO operating test passing rate during FY 1995 was unusually high compared to prior yearsc Although the passing rate on the facility-prepared SRO operating tests during FY 1997 was slightly lower than the FY 1995 baseline, it was the same as the passing rate on the NRC-prepared operating tests during FY 1996c The SRO' passing rates on the NRC-prepared written examinations and operating tests during j
FY 1997 were lower than the FY 1995 baseline, the historical ranges, and the facility-prepared j
results during FY 1997. The low passing rates on the NRC-prepared examinations are in large paa attributable to two examinations on which the applicants' performance was unusually poor j
for reascns that may include variations in the quality of the applicants' training and the difficulty of the examinations. This result illustrates that the applicants' performance on the examinations can vary from year to year regardless of whether the examinations are prepared by the NRC or by facility licensees.
1 i
)
4 I
i l
i
y, Another indicator of the relative consistency in the level of difficulty of the NRC-prepared and facility-prepared examinations is the amount of variation in the applicants' average grades on the written portion of the licensing examinations.'. Table 1.2 summarizes the average RO and SRO grades for FYs 1995 through 1997. The data are consistent with the passing rates summarized in Table 1.1 and support the staff's conclusion f. hat the facility-prepared.
examinations'are discriminating at a conservative and apprcpriate level. The staff expects that
, minimally qualified applicants will be able to achieve a score of at least 80 percent on an NRC written' licensing examination.
' Table 1.2. Comparison of Average Written Examination Gradas
. Exams -
-SRO-FY 1995 (M NRC) '
87.0 %
88.5 %
Facility ~
87.1 %
88.5 %
F!1996 NRC.
' 87.7%
88 4 %
Total 87.5%
88.5 %
Facility 84.4 %
86 4 %
FY 1997 NRC' 86.6%.
86.8 %
Total -
84.9% '
86.5 %
Examination Quality
[Although t'he overall examination results indicate that the pilot-style examinations are effective at discriminating between those applicants who have mastered the job requirements and those who have not, the NRC reviews showed that essentially all of the facility-prepared examinations required some changes. According to questionnaires completed by the NRC chief examiners responsible for the pilot examinations, the average facility-prepared examination required
' approximately 10 to 20 changes, which is consistent with the number of changes often required on examinations prepared by NRC contmct examiners. However, a number (about 40 percent in FY 1997) of the facility-prepared examinations required significant changes to achieve NRC standards for quality and level of difficulty, and, in some cases, the examinations were delayed oin order for the staff to fully evaluate the examination and allow time for the facility personnel to
- change the examinations. In FY 1997, about 10 percent of the examinations had to be delayed
- some for only a week and others for up to a month - until the changes could be made. The written examinations generally required the most changes. The majority of the changes were necessary to improve the plausibility of the distractors. Both the walk-through and dynamic simulator portions of the operating tests required changes to replace or enhance
- nondiscriminatory tasks,' events, and questions.
The s' taff evaluated comments made by the NRC chief examiners and facility licensees involved with the examinations to gain an understanding of why so many of the examinations have required significant changes. Four possible explanations have been considered, but their order 5
and level of significance is uncertain: (1) the examination criteria and guidance in interim Revision 8 of NUREG-1021 may have been unclear; (2) the facility personnel assigned to
.,1
- prepare the licensing examinations may not have had the expertise to construct examinations meeting NRC standards of quality; (3) the facility personnel assigned to prepare the examinations may have been given insufficient time or other resources to prepare an adequate 1
' examination; and (4) it is possible that the facility personnel assigned to prepare the examinations may not have followed the guidance in NUREG-1021 because of a bias toward writing examinations that were less difficult. Each of these possibilities is discussed below.
With regard to the examination criteria and guidance in interim Revision 8 of NUREG-1021, the
- NRC chief examiners who responded to surveys on thel. experiences during the pilot-style examinations reported that the instructions were clear. They recommended some clarifications -
l
- in the guidance, but no major changes to the examination criteria that have a direct bearing on examination quality. As discussed in SECY-96-206, the staff solicited industry comments on a draft of Revision 8 of NUREG-1021 in February 1996, but none of the respondents (which included the Nuclear Energy Institute (NEI) and two facility licensees) questioned the clarity of the guidance? When SECY-97-079 solicited comments on interim Revision 8 of NUREG-1021, one facility licensee (of the five that responded) commented that the guidance on level of difficulty needs to be clarified to enable weility licensees to satisfy the NRC's expectations in this area and to minimize the NRC examiners' subjective judgment. (Both the NRC examiners' and the facility licensees' comments on interim Revision 8 of NUREG-1021 are summarized in Section ll below.) Based on the information available, the staff has concluded that, in general, the guidance in interim Revision 8 of NUREG-1021 is sufficiently detailed to yield examinations of acceptable quality, difficulty, and. consistency. NUREG-1021 discusses and illustrates the guidelines for constructing questions and the psychometric principles that justify most of the changes that were made in the facility-prepared examinations. To further ensure that facility licensees understand the guidelines in NUREG-1021, the staff has conducted or participated in a number of public meetings and workshops in an effort to communicate its expectations to the facility employees who will prepare the examinations. Additional NRC and industry workshops are being planned to address examination quality and solicit industry feedback on NUREG-1021. The staff will continue to accept and review questions and comments on the NUREG and expects that additional clarifications and changes will become necessary in response to unanticipated issues.-
Although it was not reflected in their public comments, some facility managers have since commented during meetings with the staff (e.g., a January 27 - 29,1998, Region 11 public workshop to review test-writing techniques and other examination issues) that the NRC has raised its standard with regard to level of difficulty (particularly that of the written examinations) since beginning the pilot program. The staff did not intend to raise its standards during the pilot program; however, it is possible that the NRC exarniners' efforts to maintain the effectiveness of the operator licensing process have raised the discrimination validity and level of difficulty of the examinations.
The NRC has always expected its licensing examinations to discriminate between applicants who have and have not mastered the knowledge, skills, and abilities required to perform licensed RO m1d SRO duties. To accomplish that objective, the NRC established a number of criteria against which its examiners are expected to evaluate test items in order to achieve an 6
,a
,t k
u
.L acceptable examination in termiof quality,' consistency, level of knowledge and level of difficulty.;ln addition to being technically accurate, the NRC expects written examination questions to be written at the highest level of knowledge appropriate for the topics selected for
. the examination and the distractors in multiple choice questions to be plausible (i.e., they
- should appear reasonable but, nevertheless, be wrong). When the NRC or a contractor prepared an examination, the.tephnical accuracy of the questions generally took precedence
- over their psychometric qualityc However, the staff believes that the overall quality of the questions has improved over time as a result of repetitive use and revision and would have continued to improve even if the.NRC had retained responsibility for developing the licensing examinations. Now that facility licensees are being held responsible for the technical accuracy of the questions, NRC examiners are able to devote more effort to ensuring that the distractors are plausible and the questions are written at the appropriate level of knowledge.
In the late 1980s, when the NRC shifted the format of the written examination from essay and short answer to multiple choice questions, it took time to develop the NRC's bank of questions, particularly those that tested at the higher cognitive levels because they are more challenging and costly to write.L By the time the NRC initiated the pilot program in FY 1996, the average written examination (based on audits performed by the staff) contained approximately 50
. percent comprehension or analysis questions. To ensure that the examinations would remain effective, the staff revised interim Revision 8 of NUREG-1021 to establish a 50 percent lower
' limit on the higher cognitive level questions. The staff has concluded that the lack of an upper limit may have contributed to the increased variability in the level of difficulty of the examinations. The staff has also noted that some examiners may hnve suggested that facility licensees submit more than the minimum percentage of comprehension-level questions, thinking that the examination authors are more likely to over-estimate than under-estimate the cognitive levels of the questions. - As discussed in Section il below in connection with the resolution of puolic comments on interim Revision 8 of NUREG-1021, the staff has clarified final Revision 8 by specifying a range of higher cognitive level questions, instead of a lower limit.
With regard to the expertise of the facility personnel assigned to prepare the examinations (i.e.,
the second possible explanation for the number of examination changes), the preliminary and proposed regulatory analyses for this rulemaking expressed the staff's expectation that the
- training personnel at power reactor facilities already possess the basic expertise needed to develop test items for the initiallicensing examinations. The staff believed that many of the problems with examination quality and level of difficulty could be attributed to the facility -
licensees' lack of familiarity with the NRC's requirements and expectations and that, with experience, the industry would gain proficiency in preparing the examinations. However, the overall quality of the examinations submitted to the NRC during the pilot program did not improve as expected over time. Although approximately half of the 17 facility licensees that had '
i prepared more than one examination by the end of FY 1997 did maintain or improve the quality of their second or third examinations, the quality of the other facility licensees
- second or thir?
examinations was lower. However, the pilot examination program has demonstrated that the NRC's review and approval process is ensuring that the quality and validity of the examinations that are administered to the license applicants, and upon which the NRC's licensing decisions l
are based, meet NRC standards.
Although it is unclear to what extent the problems with examination quality and level of difficulty 7
J
are determined by the qualifications and training of the examination writers, the staff has asked
- the industry to address this area. Some facility licensees and NRC examiners have asserted that the personnel restrictions in NUREG-1021 could impede training or diminish examination quality because facility licensees must choose where to assign their best-qualified personnel and whether to hire contractors ito write the examinations. Although the staff still believes that -
. the restrictions are needed to limit the potential for bias, it has relaxed the personnel restrictions in final Revision 8 of NUREG-1021 to allow facility licensees greater flexibility regarding which instructors can prepare the examinations (as explained in Section 11 below in connection with the staffs response to the public comments on interim Revision 8 of NUREG-1021 and the review of concems regarding independence and public perception in Section 111).
~ With regard to th' level of facility resources required for preparing a licensing examination (the e
third possible explanation for the number of examination changes noted above), NRC examiners have seen instances where the personnel on some facility training staffs have been challenged to prepare the examinations without additional assistance. Moreover, NEl publicly q
commented that the personnel restrictions in NUREG-1021 have made it more difficult for facility licensees with small training staffs to prepare the examinations. Therefore, some facility licensees may have difficulty allocating the resources necessary for their training staffs to prepare good quality licensing examinations, while continuing to train the license applicants.
The fact that some facility licensees have had to hire contractors to prepare their licensing examinations may also have had an effect on the technical quality of their examinations and the
- efficiency with which they were prepared..(Issues regarding the facility resource burden are further discussed in the regulatory analysis (Enclosure 3), and the NEl comment regarding personnel restrictions, including the staffs rationale for relaxing those restrictions, are discussed in Section il below.)~
As noted above, the fourth possible explanation for the number of examination changes is that facility personnel assigned to prepare the examinations may not have followed the guidance in NUREG-1021 because they may have been biased toward writing examinations that were less difficult. Some NRC examiners have reported a reluctance on the part of facility personnel to make NRC-directed changes that might increase the level of difficulty of the examination and expressed concem (in their public comments on the rulemaking) that the NRC's review and approval process may not adequately compensate for the conflict of interest inherent in the revised examination method. Two NRC examiners commented that the revised process puts training managers in a no-win situation and that some managers will prepare the easiest
- possible examination to minimize the risk of their applicants failing the examination. (A training manager made such a comment to an NRC examiner, and the region addressed the issue in a
. management meeting with the facility licensee.) The staff agrees that some facility personnel could prepare an examination that will maximize their applicants' chances for success despite the criterie and guidance in NUREG-1021 and their own expertise in testing and measurement techniques. The issues of independence and conflict of interest, as well as the controls that the staff has implemented to address the issues, are discussed in Section lli below in connection with the review'of the vulnerabilities associated with the revised examination process and in the i
FRN in connection 'with the resolution of public comments on the rule, in summary, the staff has concluded that, d6 spite its concems regarding the quality of the
. facility-prepared examinations as submitted to the NRC, the revised examination process offers n,
8 y
o an effective attemative to examinations prepared by the NRC or an NRC contractor. The pilot program has demonstrated that NRC examiners and their supervisors can and will ensure that facility licensees revise the examinations as necessary to arrive at acceptable levels of quality and difficulty. Moreover, the staff will delay those examinations that do not meet NRC standards by the scheduled administration date and, wWn appropriate, address significant deficiencies in the submitted examinations as 5%qee performance issues in the examination reports.
Pilot Process Efficiency With regard to the efficiency of the revised examination process, experience with the pilot examinations to date suggests that the average cost to facility licensees will not differ significantly from the cost of the NRC or its contractors preparing all of the licensing examinations. Comments from the industry reflect that it cost some facility licensees more to prepare their own examinations than they paid the NRC to prepare them; other licensees, however, were able to prepare good quality examinations at a lower cost than if the examinations were prepared by the NRC.
The industry attributed the higher cost to the revised examination and administrative criteria (e.g., documenting the source of test questions) under the pilot examination process. Although the staff acknowledges that the revised criteria contribute somewhat to the increased cost, facility licensees are now controlling many of the variables that affect the quality and, consequently, the final cost of an examination. For example, facility licensees can manage the size and quality of their examination banks and, to a large extent, the training and experience of the people they assign to write their licensing examinations. The revised examination process puts facility licensees in control of developing the examinations and holds them responsible for their quality. If a facility licensee submits a good quality examination, it is likely to save resources despite the additional administrative requirements; however, if the facility licensee submits an examination that requires many changes, it is likely to cost more than if the NRC had prepared the examination.
Comments from the NRC chief examiners who worked on the pilot examinations indicate that the average amount of time NRC staff spent reviewing the facility-prepared examinations and directing changes was generally consistent with the estimates developed before starting the pilot program. Although about 20 percent of the examinations reviewed in FY 1997 required significantly more effort than expected to achieve the NRC's standards, the burden was generally offset by other examinations that required less effort to review and revise. Therefore, the staff has concluded that the original objective of the pilot examination and transition program has been achieved. The program has demonstrated that the NRC can eliminah the use of contractors in the operator licensing program (except for the generic fundamentals examination) and can conduct the initial operstar licensing and requalification inspection programs with existing NRC staff.
9
r 1
SECTION 11 STAFF RESPONSES TO COMMENTS ON INTERIM REVISION 8 OF NUREG-1021 i
Public Comments As noted above, since beginning the pilot program in October 1995, the NRC staff has given the industry and public two opportunities to formally comment on the guidance in Revision 8 of NUREG-1021. The staff has also participated in a number of public meetings and workshops that have given the industry an altemative means to provide feedback or to raise questions regarding the guidance in NUREG-1021. The staff has considered all of the comments and questions and made changes or clarifications, as appropriate. The staff will continue to accept and review questions and comments as they arise and expects that additional clarifications and changes will become necessary in response to unanticipated issues.
Of the 13 individuals and organizations that responded to the proposed rule published in the Federa/ Register on August 7,1997 (62 FR 42426),7 also commented on interim Revision 8 of NUREG-1021. The staff considered all of the comments and revised intenm Revision 8 of NUREG-1021 as appropriate; the significant comments and the staff's responses are summarized below. A copy of final Revision 8 of NUREG-1021 will be sent to each facility licensee upon Commission approval of the final rule. Copies of NUREG-1021, Revision 8, will
. also be available in the NRC Public Document Room (PDR),2120 L Street, NW (Lower Level),
Washington, DC, and on the intemet at http://www.nrc. gov for a limited period of time.
Comment: Seven respondents recommended changes to the restrictions that interim Revision 8 of NUREG-1021 places on facility personnel who can participate in developing the licensing examinations. The respondents from the nuclear industry generally believe that the restrictions place an undue burden on facility licensees with no apparent benefit and that the personnel who played an active role in training the license applicants should not be restricted from preparing the licensing examinations. The industry strongly endorsed adopting the NRC's requalification examination personnel restrictions so that any instructor could prepare the initial licensing examination as long as he or she refrained from teaching the applicants after starting work on their licensing examination. Some of the industry respondents asserted that the guidance regarding supervisory involvement in the examination development process needs to be clarified. One NRC license examiner recommended that the restrictions be clarified to prevent the same person from preparing both the audit examination that the facility licensee uses to confirm the applicants' readiness and the NRC licensing examination.
Response: The staff acknowledges that the personnel restrictions may have increased the burden for some facility licensees with smaller training staffs, and that the quality of the training could suffer if personnel are diverted from training assignments to prepare the examinations.
Therefore, ths NRC has revised the personnel restrictions in final Revision 8 of NUREG-1021 in a manner that the staff believes will reduce the burden on facility licensees without substantially increasing the potential for conflicts of interest that could bias the examination process.
Specifically, final Revision 8 will allow facility instructors to prepare the written examination questions without regard to the amount of time they spent training the license applicants, (as is the case on requalification examinations).but it will still preclude instructors from preparing questions related to those topics on which they provided instruction (which remains somewhat 10
1 L
more restrictive than the requalification criteria) and from instructing the applicants once they
'begin working on the licensing examination (as was the case in interim Revision 8 and on requalification examinations). As a result of this change, a facility licensee that has two instructors who equally shared the responsibility for training the license applicants, can use both instructors to prepare examination que3tions related to the topics they did not teach, whereas, they previously would both have been restricted from any examination development. Facility licensees may still propose for staff consideration, on a case-by-case basis, other approaches to address the NRC's concem regarding conflict-of-interest. The vulnerability discussion regarding independence and public perception (in Section lil) prm% additional justification for this change and an overview of the controls that the staff has implemented to address this j
issue.
i With regard to facility management involvement in the examination development and review process, the staff has revised the guidance in final Revision 8 of NUREG-1021 so that it does
. not unnecessarily restrict management activities or authority.
Comment: One facility licensee commented that the guidance on leve! of di#iculty for both the written examination and the operating tests needs to be clarified so that licensees can better satisfy the NRC's expectations on their first try. According to the licensee, the current guidance relies too heavily on the subjective judgment of the chief examiner, which fosters inconsistency.
L Response: The staff agrees that assessing the level of difficulty of test items does involve some level of subjectivity. Writers of facility examinations and NRC examiners who are trained i
in the subject matter, measurement pinciples, and psychometrics, and who have general knowledge of operator and trainee performance on similar test items, can make informed judgments on level of difficulty based on the guidance in NUREG-1021.
Interim Revision 8 of NUREG-1021 discusses the concepts of discrimination validity and i
examination difficulty and includes a number of criteria directed at limiting the amount of variation in the difficulty of both the written examinations and the operating tests. For example, i
NUREG-1021, Appendix D, " Simulator Testing Guidelines," discusses the quantitative and
{
qualitative attributes of simulator scenarios, and Section ES-301, " Preparing initial Operating Tests," of NUREG-1021 provides a specific target range for each of the quantitative attributes l
applicable to the initiallicensing examination. Similarly, NUREG-1021, Appendix B, " Written
)
l Examination Guidelines," discusses the principles of level of knowledge and level of difficulty, l
and ES-401,
- Preparing Initial Site-Specific Written Examinations," specifies an upper limit on
. the number of questions based on fundamental knowledge or memory. However, it has come to'the staff's attention that ES-401 does not specify an upper limit on the number of comprehension-level or analysis-level questions and that this may have contributed to j
variations in the level of difficulty of some examinations. Therefore, tha staff has revised the I
criteria in interim Revision 8 of NUREG-1021 to specify ranges for memory and comprehension /
analysis-level questions.
Other measures that help minimizc variation in the level of examination difficulty include (1)
NRC and industry workshops designed to help facility licensees better understand the NRC's standards (as stated in NUREG-1021) and expectations with regard to level of difficulty, (2)
NRC supervisory approval of every examination, and (3) regional oversight activities by the I
11 t.
o
i Office of Nuclear Reactor Regulation (NRR). The NRC staff has conducted and participated in a number of meetings and workshops since beginning the pilot program, and additional workshops are being planned. Secondly, an NRC supervisor is required to review and approve every examination, with pmticular emphasis on the changes recommended by the chief examiner, before the eF2mination is administered. Moreover, NRR periodically conducts examination reviews, which include an assessment of their level of difficulty, to ensure l
l consistent regio 1alimplementation of the NRC's standards and expectations.
Comment: An NRC license examiner recommended that any question taken directly from a
)
bank of questions that is available to the applicants for study should be considered a memory-
{
level question regardless of the cognitive level at which it is written.
Response: Although this is a valid concem, making such a change would likely increase the burden on the facility licensees (because of the increased challenge of writing additional -
questions at the higher cognitive levels) and increase the level of difficulty of the examinations.
]
Therefore, the staff has made no changes to interim Revision 8 of NUREG-1021 in response to this comment.
Comment: An NRC license examiner recommended that NUREG-1021 should contain a detailed list of minimum physical security precautions and that facility licensees should be required to list the steps they have taken to ensure examination security, including measures to control any contractors who prepare the examinations.
Response: NUREG-1021 is a guidance document intended to help NRC examiners and facility licensees gain a better understanding of the procedures for, and NRC management's expectations regarding, preparation of the initial opert
- ensing and requalification examinations. The staff has clarified the examination security expectations in final. Revision 8 of NUREG-1021, including the control of licensee contractors. In a separate action, the staff issued an Information Notice (lN) to advise power reactor facility licensees of the NRC's 1
perspective and expectations regarding the integrity of examinations developed by the facility licensees' employees and representatives.
l Furthermore, based on several security incidents that occurred since beginning the pilot
. examination program, the staff has concluded that applicants, licensees (operators), and facility licensees may not understand what is meant by the term " compromise," as used in 10 CFR
~
55.49. Therefore, the staff has concluded that it would be beneficial to clarify 10 CFR 55.49 in the final rule so that facility licensees understand their responsibility for maintaining control over the examination process and establish procedures for that purpose.
i i
Comment: Two facility licensees suggested that applicants should be allowed to perform some
)
or all of the fee required reactivity manipulations on the simulator rather than on the actual reactor.
Response: The requirement for five significant control manipulations is addressed in 10 CFR 55.31(a)(5), and such a change is beyond the scope of this rulemaking. Therefore, the staff will i
address this comment separately.
12 I
Comment: An NRC license examiner recommended deleting the provision in interim Revision l
8 of NUREG-1021 that allows up to 30 percent of the job performance measures (JPMS) on the walk-through portion of an operating test to be repeated on walk-through tests given on a later day.
Response: The issue of repeating JPMS during successive walk-through operating tests was discussed in SECY-96-206, and the 30 percent limit was included in interim Revision 8 of NUREG-1021 because no limit had existed previously when the NRC was preparing the examinations. The staff has revised its position on this issue in favor of the approach recommended by the examiner because it will improve examination security and be consistent with the policy on repeating simulator scenarios. The staff believes that the impact of the change will be minimal.
Comment: One nuclear industry employee commented that requiring facility licensees to prepare and submit an outline of the written examination and operating test in advance of the actual examination is an unnecessary administrative burden. He suggested that the due date
- for the examination itself be advanced and the requirement for a separate outline be eliminated.
Response: Preparing the outline, particularly for the written portion of the examination, in accordance with the guidance NUREG-1021 ensures that the required knowledge and abilities are systematically sampled and evaluated on the licensing examination. Submitting the outline to the NRC for review and approval before preparing the examination itself, enables the NRC to comment and the facility licensee to make the required changes before significant licensee resources are expended in developing the final examination materials.
In response to this comment and to comments (regarding the submittal of examination materials) from NRC examiners involved with the pilot examinations, the staff is changing the
.' guidance in interim Revision 8 of NUREG-1021 by advancing the due dates for the examination outline and the proposed examination in order to provide additional time for facility licensees to resolve any concems the NRC has with those submittals.
Comment: An NRC license examiner recommended that NUREG-1021 should contain guidance to ensure that the written examination outlines are systematically developed. He argued that this will minimize the possibility that facility licensees will bias the selection of topics toward those that were emphasized during the applicants' license training program. (Another NRC examiner submitted a differing professional view (DPV) regarding the potential for bias in the selection of topics for the examination and recommended that the NRC should continue to prepare the examination outlines.)
Response: The staff agrees that the specific written examination sampling guidance in interim Revision 8 of NUREG-1021 does not ensure that the written examination is free of biases that could affect its validity. Therefore, the staff has revised final Revision 8 of NUREG-1021 to reduce the risk of bias by ensuring that facility licensees prepare the examination outlines in a systematic manner. (The panel that reviewed the DPV concluded that this was an acceptable resolution of the issue.)
13
Comment: 'NEl and a nuclear industry employee recommended that facility licensees be allowed to use their site-specific task lists instead of NUREG-1122,
- Knowledge and Abilities
- [K/A) Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors," or NUREG-L 1123, " Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water l
Reactors," to establish the content validity of the examinations.
L I
Response:' The staff believes that allowing substitution of the NRC's K/A catalogs (which were j
derived from an industry job-task analysis and evaluated by a panel of subject matter experts representing the nuclear industry and the NRC) would be inappropriate and inconsistent with
' the regulations. Although the Part 55 rulemaking of 1987 generally endorsed the " systems approach to training," Sections 55.41(a), 55.43(a), and 55.45(a) all state that the knowledge, skills, and abilities selected for evaluation on a written examination and an operating test will be i
identified, in part (emphasis added), from leaming objectives derived from a systematic analysis of licensed RO and SRO duties performed by each facility licensee. While NUREG-1262, j
l-
" Answers to Questions at Public Meetings Regarding implementation of Title 10, Code of J
Federal Regulations, Part 55 on Operators' Licenses," confirmed the NRC's intent that the training program's leaming objectives would become the major source of the licensing examination, it also cautioned that the NRC would not be limited to those leaming objectives.
Moreover, since 1986, NUREG-1021 has encouraged examiners to use the facility licensees' leaming objectives as a source of topics for the written examination, and it has always required examiners to reference the facility licensees' training materials when developing a licensing examination.
i The systematic sampling procedures for preparing the written and walk-through examination outlines per interim Revision 8 of NUREG-1021 are designed around the structure of the NRC's K/A catalogs and may not be compatible with the facility-specific task lists. As discussed in i
SECY-96-206, interim Revision 8 of NUREG-1021 contains provisions for facility licensees to add, substitute, or delete specific knowledge and ability requirements on a case-by-case basis.
Allowing facility licensees to substitute their entire site-specific task lists for the NRC's K/A catalogs could decrease the level of examination consistency, increase the potential for biasing
)
the sample of material covered on the licensing examinations, and increase the level of effort for_NRC examiners to review the proposed outlines to ensure adequate sampling per Part 55.
The staff has concluded that the current approach of requiring facility licensees to explain j
deviations from the NRC's K/A catalogs is conservative, consistent, and efficient. Therefore, no changes were made to Revision 8 of NUREG-1021 in response to this comment.
)
i Comment: NEl recommended that the time limit for taking the written examination should be i
increased from 4 to 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> because of the increased percentage of higher cognitive-level
. questions on the examinations.
i Response: Feedback regarding the pilot examinations has confirmed that a number of applicants could not complete the examination in the 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> currently specified in NUREG-1021. The staff believes that increasing the time limit to 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> may be necessary because of the greater focus on improving the plausibility of the test question distractors (i.e., the wrong-j answer choices) and the required range of higher cognitive-level questions.
I 14
g The staff believes that the time limit can be increased to 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> without affecting the discrimination validity of the examinations. The nature of the NRC licensing examination it such that allowing sufficient time to demonstrate knowledge is of primary concem.
J Comment: One facility' licensee and a facility employee suggested that the simulator operating test administration procedure in NUREG-iO21 should allow other operating assistants, such as a shift technical advisor (STA) ar.d shift manager, to help the applicants during critical evolutions on the simulator when required by the plant's technical specifications and procedures.
l Response: Because the operating tests are administered one-on-one (i.e., with one NRC
~
i examiner assigned to each applicant on the crew), the number of applicants on the crew can normally not exceed the number of NRC examiners dispatched to the site, interim Revision 8 of NUREG-1021 already makes allowances for the operating crews to be supplemented with additional surrogate personnel up to the minimum RO staffing level required by the facility
)
licensee's technical specifications, but it does not address the use of STAS or additional shift managers (i.e., senior operators). Therefore, in final Revision 8 of NUREG-1021, the staff will allow the use of STAS in accordance with the facility licensee's normal staffing practice for that position. However, the staff does not intend to permit more than one person to fill a senior operator position during the simulator test because the principal duties of the shift manager
_ position (i.e., assuming the role of the emergency director, performing emergency classifications, and making protective action recommendations) are normally a part of the operating test for senior operator applicants.
' Comment: An NRC license examiner suggested that the requalflication examination procedures in NUREG-1021 should include limits on test item duplication between successive examinations.
Response: The staff's expectations regarding this issue were previously communicated to the
. NRC regional offices, and the staff agrees that the same information should be incorporated in final Revision 8 of NUREG-1021. Specifically, final Revision 8 will limit test item duplication between successive NRC requalification examinations to no more than 50 percent.
Comments From NRC Staff in addition to the public comments discussed above, the staff considered a number of comments, recommendations, and questions from NRC examiners, supervisors, and managers involved in the operator licensing program. Although most of the issues related specifically to the' pilot process, some applied to other aspects of the operator licensing program. A number of the issues were previously addressed in response to the public comments; the remaining significant comments and responses are summarized below. Several additional minor changes are summarized, as appropriate, in the Executive Summary section of final Revision 8 of NUREG-1021, Comment: A senior manager suggested that NUREG-1021 should provide facility management with the opportunity to appeal to NRC management if the facility has significant concems regarding the staff's application of the examination criteria in NUREG-1021, 15 x.
Response: The staff agrees that such a provision is appropriate and consistent with the NRC's policy of minimizing regulatory impact when appropriate. ES-201, " Initial Operator Licensing Examination Process," of final Revision 8 of NUREG-1021 has been edited to incorporate such a practice.
4 Comment: A regional supervisor suggested that ES-301, " Preparing Initial Operating Tests," of NUREG-1021 should better define the NRC's expectations regarding the permissible number of closed-reference questions on the walk-through portion of the operating test.
Response: The staff agrees that the guidance in interim Revision 8 of NUREG-1021 lacks specificity and may contribute to variations in the quality and level of difficulty of the operating tests. Final Revision 8 of NUREG-1021 has been clarified to address the issue.
J I
i 1
16 l
t
SECTION lli VULNERABILITY REVIEW
~ At the Commission's direction, SECY-96-206, the rulemaking plan for amendments to the examination requirements in 10 CFR Part 55, discussed the pros, cons, and vulnerabilities of proceeding with the revised examination process on an industrywide basis. The SRM that later approved the rulemaking plan directed the staff to monitor the pilot program results and inform l
the Commission if the staff gained any insights that suggest that the revised examination process should not be pursued. In response to this directive, the staff has reviewed the i
vulnerabilities (i.e., quality and consistency, independence and public perception, examination security, NRC resources, program stability, and examiner proficiency) that were discussed in the rulemaking plan in light of the staffs experience with the pilot process since SECY-96-206 was issued on September 25,1996. The staff has also reviewed the potential effects on reactor safety.
For ease of review, the staff has categorized and addressed the public comments on the proposed rulemaking with respect to the same vulnerabilities. Appropriate cross-references to the FRN for the final rule are provided to minimize the need for repetition.
i Quality and Consistency of the Examinations The staffs expectations regarding the quality and level of difficulty of the proposed examinations have not yet been achieved. As discussed above, the staff has considered a l
number of possible reasons for the problems that have been observed, but no definitive 1
l explanations have been identified. Additional public comments related to this vulnerability are i
addressed in the associated FRN.
1 To maintain acceptable uniformity of examination content, format, level of difficulty, and quality, the staff expects facility licensees to prepare the written examinations and operating tests in accordance with the procedures and guidelines in NUREG-1021. A number of administrative criteria and controls have been incorporated in Revision 8 of NUREG-1021 to implement the new examination development process. Those revisions include: limits on the number of test items (e.g., written questions and JPMs) that can be taken directly from the facility licensees' l
examination banks, the number of test items that can be developed by modifying existing bank items, and the number of newly-developed test items; limits on the repetition of test items from l
previous quizzes and examinations given at the facility; guidelines regarding the cognitive level of written examination questions; and improved quality assurance checklists. Revision 8 of NUREG-1021 discourages facility licensees from using testing methodologies that do not j
conform with NUREG-1021 but allows them to propose attematives for NRC review.
The significant changes required on many of the proposed examinations suggests that the guidance in interim Revision 8 of NUREG-1021 may not have been understood. However, as noted above, the staff believes that the guidance in interim Revision 8 of NUREG-1021 was i
sufficiently clear to enable facility licensees to prepare examinations of acceptable quality and l
consistency, and, therefore, has made only minor revisions to NUREG-1021 since issuing SECY-96-206. The staff has conducted and participated in a number of public meetings and workshops in an effort to communicate its expectations to the facility employees who will 17
prepare the examinations and to obtain feedback regarding the guidance in NUREG-1021.
- Additional NRC and industry workshops are being planned to address continued concerns regarding the quality of the submitted exarr,inations. Moreover, as noted earlier, the staff has asked the industry to address the issue of oxamination quality.
The revised examination process is depe',wient upon the ability and judgment of the NRC chief examiners to identify weak test items and io communicate the problems to the facility representatives so that the required corractions are understood and implemented. Although a number of the pilot examinations submitted by the facility licensees have not met NRC standards for quality, level of knowledge, or level of difficulty, the NRC chief examiners have directed the facility licensees to revise the examinations so that they would discriminate at the appropriate level. The staff acknowledges that some of the facility-prepared examinations will be more difficult than others (yet within an acceptable range of variation) and that some of the -
variation can likely be traced to individual differences among the NRC examiners who review
- and approve the examinations. Such was also the case when the NRC or its contractors wrote the examinations. However, the staff believes that the variation in the level of difficulty can be
- maintained within acceptable limits and would be even greater if NRC examiners did not take action to ensure that the submitted examinations are consistent with NRC standards.
To ensure consistent implementation of the guidance in NUREG-1021, an NRC supervisor is required to review and approve every examination, with particular emphasis on the changes recommended by the chief examiner, before the examination is administered. Supervisors are also required to periodically evaluate every certified examiner during the conduct of an operating test to ensure that the on-site' examination activities are being performed as intended.
Furthermore, the NRR staff periodically reviews the licensing examinations given et selected facilities and the implementation of the operator licensing function in each region to verify the quality of the examination process and to ensure the consistent implementation of the NRC's standards and expectations. The NRR reviews include an assessment of the level of knowledge and level of difficulty of the written examinations and operating tests; each review is documented in a report to the responsible regional Division Director.
NRR has instructed regional managers and examiners not to approve or administer any examination that does not adhere to current NRC standards for content, format, quality, level of knowledge, and level of difficulty, if a facility licensee does not prepare appropriate licensing examinations, the revised process provides that the NRC, which has the responsibility to ensure that public health and safety are maintained, will develop the examinations, independence and Public Perception The staff implemented a number of measures during the pilot examination process to ensure that an adequate level of independence would be maintained and to minimize the potential for bias during the examination development. GL 95-06, which outlined the pilot examination process in August 1995, restricted all facility employees who played a " substantial" role in training the license applicants from participating in the examination development. Questions regarding the meaning of " substantial" prompted the staff to clarify the personnel restrictions in interim Revision 8 of NUREG-1021, which incorporated lessons leamed during the first 22 pilot examinations and was issued for use in February 1997. Interim Revision 8 prohibited facility 18 l
l instructors from participating in the examination development process if they had spent more
- than a specified amount of time training the applicants who would be taking the NRC examination and prohibited any instructor from preparing written examination questions related
[
to those topics in which the instructor had provided training. The personnel restrictions in the GL and interim Revision 8 supplemented a related provision, which, for purposes of maintaining examination security, barred anyone who had knowledge of the contents of the NRC examination from further training-related activities involving the license applicants. The latter provision had been adopted from the requalification examination program but did not address the separate but equally important perception that the people who were responsible for training the applicants, if allowed to prepare the examination, could bias its content or level of difficulty.
The nuclear power industry has argued that the applicants' instructors should be permitted to i
prepare the licensing examinations, just as the instructors who perform the requalification training are permitted to prepare the requalification examinations required by 10 CFR 55.59(a).
As noted in the public comments above and in the associated FRN, the industry has indicated i
that the additional personnel restrictions implemented during the pilot examination process have unnecesorily increased the facility licensees' administrative burden in preparing the examinations ano forced facility licensees to divert their best-qualified instructors from their primary duty of training the Scense applicants or to use less-qualified instructors to prepare the licensing examination, with a possible decrease in quality.
The staff has evaluated the industry's concems in light of the other public comments related to
)
- this vulnerability and the other changes that are being made in final Revision 8 of NUREG-1021. Therefore, as noted above in response to the public comments, the staff has removed l
the restriction that prohibits instructors from preparing any written examination questions based
. solely on the amount of time they spent training the license applicants. This change should provide licensees with increased flexibility in managing their resources and possibly reduce their costs without adversely affecting the independence of the examination process. For example, if a facility licensee has two instructors who equally shared the responsibility for training the license applicants, they will now be permitted to prepare the examination questions related to the topics they did not teach, whereas, they previously would both have been restricted from any examination development. As discussed above in response to the public comments on the NUREG, final Revision 8 will also include explicit guidance for systematically sampling the important, safety-related K/As from NUREG-1122 or NUREG-1123 when preparing the written examination outline. This change should reduce the chances that the written examination might be biased in favor of those topics that were emphasized during the training program and away from other important topics that received less attention. The operating test outline will continue to be independently developed by an individual who was not involved in the applicants' license training. These measures should provide sufficient assurance that the examinations are prepared in an impartial manner.
The staff maintains, for the reasons that follow, that the revised personnel restrictions are l
needed to limit the potential for bias in the examination development process.
l l
The personnel restrictions were an important component of the pilot examination process. The conclusions of the acceptability of the pilot process, such as the comparability of examination passing rates and test scores, are linked to the 19 Lu
n l
independence of the test developer. To eliminate controls on who can write examination material at this point in the transition process could undermine the staff's conclusions regarding the effectiveness of the pilot process.
The personnel restrictions maintain greater consistency with other professional licensing
, and certification examinations (e.g., medical, legal, aircraft, and engineering), which are prepared by an independent body.
The initial licensing examinations are considered "NRC" examinations upon approval, even if they were prepared by the facility licensee, whereas, the requalification examinations required by 10 CFR 55.59(a) are classified as " facility" examinations.
Unlike the requalification examinations, the NRC's initial licensing examinations are not a part of the facility licensee's training program. The initial examination is the comerstone of the NRC's licensing decision.
On June 18,1996, the staff briefed the Commission on the results of the pilot program, noted its concems regarding the potential for facility employees to bias the examinations, and discussed the personnel restrictions that the staff had implemented 1
to mitigate those concems. The Commission acknowledged the vulnerability in this area, including the concem that instructors could teach to the test.
Eliminating the personnel restrictions could affect the public perception regarding the revised process and indicate that the NRC is not concemed about the potential for examination bias.
With regard to the issue of public perception, The Washington Post reported, shortly after the j
staff disclosed its intent to revise the examination process in March 1995, that consumer advocates were concemed that allowing facility licensees to prepare the examinations might endanger reactor safety because the examinations might be less rigorous than those prepared by the NRC. However, in the three years since the original proposal, the NRC has afforded the public a number of opportunities to comment on the revised examination process, but none have been received. For example, Generic Letter 95-06 and its supplement announced and later summarized the results of the pilot examination program; comments were invited, but none were received. When a Federal Register notice (61 FR 6869) dated February 22,1996, announced the availability of draft Revision 8 of NUREG-1021 and solicited comments, only NEl and two facility licensees responded. And, when the proposed rule was published in the Federal Register (62 FR 42426) on August 7,1997, the 13 respondents included NEl, five nuclear utilities and one employee, four NRC or contract examiners, one non-power reactor facility licensee, and the State of Illinois, but no consumer advocates or members of the general public.
Concems regarding the potential for conflict of interest have also prompted the NRC to review the clarity of 10 CFR 55.49; The regulation encompasses not only activities like cheating and lapses in security but also activities that compromise the integrity or validity of the examination (e.g., noncompliance with the criteria designed to limit the potential for bias in the selection of topics to be evaluated on the written examination). Therefore, the NRC has concluded that it would be beneficial to amend 10 CFR 55.49 in the final rule to ensure that applicants, 20 l
i l
I -
licensees, and facility licensees, understand the scope of the regulation, if the NRC determines that a facility licensee has biased the scope, content, or level of difficulty of an examination to enhance the chances that its applicants would pass the examination, the NRC will utilize its i
enforcement authority including, as warranted, civil penalties, orders against the individuals involved, and, charging the individuals involved with deliberate misconduct pursuant to l
- 10 CFR 50.5.
l
. In summary, the pressures to conduct the examinations on schedule and the reduced involvement of the NRC in the examination development process could lead to examinations that reflect the biases of the facility licensee and that discriminate at a lower level than is currently the norm. However, a number of process requirements are in place to reduce the biases: the revised examination process includes restrictions on personnel, the systematic selection of topics to be tested on the written examination, independent NRC review, NRC-
- directed revisions, NRC approval of the examinations, and independent NRC administration of the operating tests that should provide sufficient safeguards to reduce the possibility of biased licensing decisions. The pilot examination results, to date, suggest that these measures, in
- combination with the NRC examiners' increased focus on the psychometric quality of the written j
questions, will be effective at maintaining the discrimination validity of the licensing examinations.
. A summary of the' public comments related to this vulnerability and the staff's responses are
. included in the associated FRN.
j Examination Security As noted in SECY-96-206, the stan expects that the majority of facility licensees will continue to maintain proper examination security. Nevertheless, the revised examination development process may provide greater opportunities for examination compromise (as that term is clarified in an amendment to 10 CFR 55.49) than the traditional process in which fewer facility employees had knowledge of and access to the examination for a shorter period of time. Since beginning the pilot examination program, there have been a number of instances in which facility licensees have (1) failed to exercise positive control of the examinations by not locking the room or container in which the examinations were being prepared or stored; (2) failed to a
control the activities of personnel involved in preparing or reviewing the examinations, as required by the signed security agreements, or to restrict unauthorized personnel from gaining access to the' examinations; and (3) lost control of the examinations while printing or making copies.
NRC examiners are required by Revision 8 of NUREG-1021 to be attentive to examination
)
l-security measures, to review the NRC's security expectations with the facility licensee at the I
time the examination arrangements are confirmed, and to report any security concems to NRC management. If the NRC determines during its preparation that an examination may have been compromised, it will delay the examination until the scope of thu potential compromise is determined and measures can be taken to address the integrity and validity of the examination.
If the compromise is discovered after the examination has been administered, the NRC will delay the licensing action for the affected applicants until the staff can make a determination regarding the validity of the examination. If the compromise is not discovered until after the 21 l
l i
L
E E
licensing action is complete, the NRC will reevaluate the licensing decision pursue.t to 10 CFR j
55.61(b)(2) if it determines that the original licensing decision was based on an invalid j
. examination. As a separate measure, the NRC will take enforcement action, when appropriate, j
and consult with the Office of Investigations in cases involving the possibility of personal wrongdoing.
The staff has taken a number of actions since the issuance of SECY-96-206 to increase the
' NRC examiners' and the facility licensees' sensitivity to examination security issues and to l
mitigate the vulnerability in this area. Those actions include (1) a revision of NUREG-1600, i
l
" General Statement of Policy and Procedure for NRC Enforcement Actions," to address j
l enforcement action against the license applicants, Part 55 licensees, and Part 50 licensees that l
are subject to the requirements in 10 CFR 55.49; (2) a change in the final rule, based on I
l comments on the proposed rule, to clarify 10 CFR 55.49 and to require facility licensees to establish procedures to control examination security and integrity; (3) further clarification and elaboration on the guidance in NUREG-1021 about examination security; and (4) the issuance of an IN to advise power reactor facility licensees of the NRC's perspective and expectations
.I regarding the security of examinations developed by the facility licensees' employees and representatives.
NRC Resources When the staff developed the pilot examination process, it estimated that it would take approximately 370 examiner-hours to review and otherwise prepare for, administer, grade, and document an average operator licensing examination. Although a number of the proposed examinations consumed more NRC resources than expected to achieve NRC's standards, the i
burden was generally offset by other examinations that required less effort to review and revise.
Therefore, the NRC staff continues to believe that the use of contractors in the operator licensing program (except for the generic fundamentals examination) can be eliminated and that the revised initial operator licensing and recualification inspection programs can be implemented with the existing NRC staff. Before initiating the pilot examination and transition program at the beginning of FY 1996, the NRC spent approximately $3 million per year for j
contractor assistance to prepare and administer initial examinations and assist with requalification inspections. During FY 1996, when approximately 40 percent (27/68) of the examinations were voluntarily prepared by facility licensees, the NRC spent approximately $1.6 1
'million for contractor assistance in those areas. In FY 1997, facility participation increased to i
include approximately 75 percent (41/55) of the examinations, and the NRC's spending on contractor assistance for the licensing examinations and requalification inspections decreased to approximately 50.5 million (approximately 3 full-time equivalents (FTEs)). During FY 1997, the NRC expended approximately 16.8 staff FTEs to conduct all the site-specific initial operator licensing examinations, plus 3.8 staff FTEs for generic operator licensing activities. The licensed operator requalification inspection program consumed approximately 4.5 additional staff FTEs (plus a minimal level of contractor support), bringing the total resources used for the operator licensing program in FY 1997 to approximately 28.1 FTEs. This actual resource burden is generally consistent with the budget estimates prepared before beginning the pilot examination program and supports the staff's conclusion that the revised examination process is an efficient altemative to the traditional NRC-prepared examinations. The NRC resource burden should decrease further if and when the rule is implemented (which would cause the 22
number of NRC-prepared examinations to decrease) and the quality of the facility-prepared examinations improves.
However,' experience during the pilot program has shown that the quality of the facility-prepared examinations can vary widely, making it difficult to predict the amount of time necessary for the NRC to review and revise any particular examination in order to meet NRC standards.
Although the staff had expected the quality of the examinations to improve over time, some of the proposed examinations continue to require more changes than anticipated. The added examiner workload could increase the possibility of lower examination quality, raise the NRC's cost, and affect the staff's ability to satisfy the facility licensees' needs for licensing examinations. As noted in SECY-g7-079, NRR has issued a memorandum to the regional administrators emphasizing the importance of (1) assigning adequate resources to carry out the operator licensing task, (2) completing a thorough review of every facility-prepared examination, and (3) not administering any examination that does not meet NRC standards for quality and level of difficulty. Furthermore, as noted above, the staff has asked the industry to address the issue of examination quality, which should reduce the need for changes and the cost of preparing the examinations.
The resource burden on facility licensees is addressed in the associated FRN and regulatory analysis.
Program Stability As noted in SECY-96-206, when it started the pilot program, the staff intended that the format, content, and level of difficulty of the facility-prepared examinations would remain consistent with previous NRC-prepared examinations so that the transition from NRC-prepared to facility-prepared examinations would be imperceptible to the license applicants. Although the staff has made a number of changes in the examination criteria and guidelines in Revision 8 of NUREG-1021, it has made every effort to maintain consistency. Of the three parameters, level of difficulty is the most subjective and most complex to define and control. However, based on the overall examination results (i.e., the slight decline in the passing percentage and average written examination grades discussed above) since starting the transition process, it appears that the level of difficulty of the facility-prepared examinations, revised as directed by the NRC, is consistent with those prepared by the NRC. Considering the historical fluctuation in the average examination passing rates and the other factors (e.g., training program quality and screening of applicants by facility licensees) that could be responsible for some or all of the observed decline, the staff has concluded that any increase in the level of difficulty is not significant..
Examiner Proficiency As noted in the responses to the public comments summarized in the FRN, the staff believes I
that by participating in the revised examination process, NRC examiners will maintain their proficiency. An NRC examiner will review and approve every examination before it is administered to ensure that it conforms to the guidelines in NUREG-1021 regarding content, format, quality, and levels of knowledge and difficulty. NRC examiners will also continue to i
independently administer and grade both the dynamic simulator and the plant walk-throu; 23
1 I
I portions of the operating tests. Because NRC examiners will be administering all of the operating tests, the revised process will enable the examiners to accrue more experience in a
)
shorter period of time and to maintain their proficiency. The smaller pool of all NRC examiners i
- may also improve the consistency of the operating test evaluations and the licensing decisions.
New NRC license examiners will still be required to complete a standardized training program,
)
including the development of a complete written examination and operating test, as part of their certification process. Once certified, e.xaminers are required to maintain their qualification by periodically conducting examinations and attending refresher training. NRR has revised the refresher training syllabus to focus more attention on reviewing, critiquing, and approving written examination questions. As directed by the Commission in the SRM of December 17, 1996, the NRR staff will ensure that each region writes at least one initial operator licensing examination per calendar year. Furthermore, as directed by the SRM, the staff intends to conduct examiner training conferences at intervals not to exceed 24 months; the last such conference was convened in October 1997.
l Routine management oversight activities performed by the regions and NRR should provide additional assurance that examiner proficiency is being maintained. Regional supervisors are required to review and approve the examination changes recommended by their chief examiners and they periodically evaluate every certified examiner during the conduct of an i
operating test. Moreover, the NRR staff periodically reviews the licensing examinations given.
at. selected facilities and the implementation of the operator licensing function in each region to I
verify the quality of the examination process and to ensure the consistent implementation of the NRC's standards and expectations.
Reactor Safety As noted above and in the responses to the public comments summarized in the FRN, the staff j
believes that the revised examination process will improve the staff's ability to focus on operator performance because NRC examiners will be administering all of the operating tests. The staff acknowledges that the total number of facility operating procedures reviewed in the process of examination development may decrease under the revised method. However, NRC examiners are still expected to review and identify discrepancies in the procedures that will be exercised during the walk-through JPMs and the simulator scenarios. Because only staff examiners will be administering the operating tests and reviewing all of the applicable procedures, the revised process will enable the examiners to accrue more experience in a shorter period of time.
The staff also acknowledges that the revised examination process places an extra burden on the facility licensees' training staffs and may divert resources from their primary training activities so that they can prepare the licensing examinations. If a facility licensee places insufficient resources on either training or testing, the quality of its proposed licensing examinations or the passing rate on those examinations would most likely suffer. Although many of the facility-prepared examinations have required numerous revisions to achieve NRC quality standards, the examination results, to date, are consistent with the results on previous NRC-prepared examinations, suggesting that the quality of the facdity licensees' training programs has not been affecteo.
24
l The NRC will continue to set the standard for operator performance by ensuring that facility licensees maintain appropriate standards for examination quality and level of difficulty. The J
staff will remain actively involved in the operator licensing process by reviewing and approving every written examination and operating test before it is administered. Unacceptable examinations have been and will continue to be revised and delayed, if necessary, until the examinations are upgraded.
The overall results of the facility-prepared examinations support the staff's conclusion that the licensing decisions that have been made on the basis of those examinations are as valid and conservative as those made using the traditional examination process. The fact that the facility licensees are preparing the examinations with NRC review and approval, should have no negative effect on the safe operation of the plants.
i l
l 25
z-Federal Register Notice (including Statement of Consideration)
)
i 1
l 1
)
[7590-01-P]
[
NUCLEAR REGULATORY COMMISSION 10 CFR Part 55 RIN 3150-AF62 Initial Licensed Operator Examination Requirements AGENCY: Nuclear Regulatory Commission.
ACTION: Final rule.
SUMMARY
- The Nuclear Regulatory Commission (NRC) is amending its regulations to require all nuclear power facility licensees to prepare, proctor, and grade the written examinations and prepare the operating tests that the NRC uses to evaluate the competence of individuals applying for operator licenses at those plants. The amendment requires facility licensees to l
submit, upon approval by an authorized representative of the facility licensee, each examination l
and test to the NRC for review and approval. The amendment preserves the NRC's authority to prepare the examinations and tests, as necessary, if the NRC has reason to question a facility licensee's ability to prepare adequate examinations. In addition, the NRC will periodically invoke this authority in order to maintain the proficiency of its own license examiners. The l
l Commission is concerned with examination integrity; therefore,10 CFR Part 55.49 has been amended to clarify the Commission's intent that facility licensees establish procedures to control examination security and integrity.
EFFECTIVE DATE: This final rule is effective on (insert the date 180 days from date of f
l:
i 2
_ publication 'in the Federal Register).
i T
.)
FOR FURTHER INFORMATION CONTACT: Siegfried Guenther, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, telephone:
ll 301-415-1056; e-mail: sxg@nrc. gov.
i i_
l i
SUPPLEMENTARY INFORMATION:
i
\\
i i
i
Background
l c
q Section 107 of the Atomic Energy Act (AEA) of 1954, as amended, requires the NRC to r
u l
determine the qualifications of individuals applying for an operator's license, to prescribe l
L uniform conditions for licensing such individuals, and to issue licenses as appropriate. Pursuant j
. to the AEA,10 CFR Part 55 requires applicants for operators' licenses to pass an examination l-j that satisfies the basic content requirements specified in the regulation. Although neither the l
~
AEA nor Part 55 specifies who must prepare, proctor, or grade these examinations, the NRC has traditionally performed those tasks itself or through its contract examiners. The NRC and its contract examiners have used the guidance in NUREG-1021, " Operator Licensing Examination Standards for Power Reactors," once titled " Operator Licensing Examiner l
Standards," to prepare the initial operator licensing examinations. This document has been revised as experience has been acquired in preparing the examinations. The current version is
)
i designated Revision 8.'
- 1. Copies'are available for inspection or copying for a fee from the NRC Public Document Room (POR) at 2120 L Street NW, Washington, DC 20555-0001; the PDR's mailing address is i
l l
l f
3 in accordance with 10 CFR 170.12 (i), the NRC's staff and contractual costs are recovered from facility licensees that receive examination services. In Fiscal Year (FY) 1995, the NRC spent approximately $3 million on contractor support for the preparation and administration of the initial operator licensing examinations and for support of requalification program inspections. On March 24,1995, in SECY-95-075, " Proposed Changes to the NRC a
Operator Licensing Program," the staff advised the Commission of its intent to eliminate the use of contractors by allcwing facility licensees to prepare the examinations.
On April 18,1995, the Commission approved the NRC staff's proposal to initiate a transition process to revise the operator licensing program and directed the NRC staff to carefully consider experience from pilot examinations before fully implementing the changes.
On August 15,1995, the NRC issued Generic Letter (GL) 95-06, " Changes in the Operator Licensing Program,"' outlining the revised examination development process and soliciting volunteers to participate in pilot examinations to evaluate and refine the methodology.
Between October 1,1995, and April 5,1996, the NRC reviewed and approved 22 operator licensing examinations, including both the written examinations and the operating i
tests, prepared by facility licensees as part of a pilot program. These examinations were prepared using the guidance in Revision 7 (Supplement 1) of NUREG-1021' and the additional guidance in GL 95-06.
The results of the pilot examinations were discussed in SECY-96-123, " Proposed l
Changes to the NRC Operator Licensing Program," dated June 10,1996. Based on the results l
of the pilot program, the NRC staff recommended that the Commission approve the I
Mail Stop LL-6; telephone is 202-634-3273; fax is 202-634-3343. Revision 8 of NUREG-1021 is also available for downloading from the intemet at http://www.nrc. gov for a limited period of time.
L
p 4
implementation of the new examination process on a voluntary basis until rulemaking could be L
completed to require all power reactor facility licensees to prepare the entire initial operator licensing examination and to proctor and grade the written portion of the examination. On July 23,1996, the Commission authorized the staff to continue the pilot examination process on a voluntary basis and directed the staff to develop a rulemaking plan to justify the changes that would be necessary to 10 CFR Part 55. The Commission also directed the staff to address a number of additional items (e.g., pros, cons, and vulnerabilities) regarding the revised examination process to facilitate a Commission decision on whether to implement the revised process on an industrywide basis.
l With Commission approval, the NRC staff resumed conducting pilot-style examinations on August 19,1996, and by the end of FY 1997 had reviewed, approved, and administered 42 additional examinations that were developed by facility licensees. This raised the total number of examinations completed using the pilot process to 68.
i On September 25,1996, the NRC staff forwarded the rulemaking plan and a response to the additional items to the Commission in SECY-96-206, "Rulemaking Plan for Amendments to 10 CFR Part 55 to Change Licensed Operator Examination Requirements." SECY-96-206 addressed a number of areas (i.e., quality and consistency, independence and public j
perception, examination security, NRC resources, program stability, and examiner proficiency) l In which the NRC could be more vulnerable under the revised examination process and the measures that the NRC has taken to manage the vulnerabilities. On December 17,1996, the Commission directed the staff to proceed with the proposed rulemaking. The staff forwarded the proposed rule (SECY-97-079, " Proposed Rule - Initial Licensed Operator Examination i
i Requirements *) to the Commission on April 8,1997, and on June 26,1997, the Commission j
approved publication of the proposed rule for a 75-day comment period. The proposed rule L
)
m i
I p
l 5
was published in the Federal Register (62 FR 42426) on August 7,1997, and, when the public comment period expired on October 21,1997,11 comment letters had been received. Two additiunal comment letters arrived after the expiration date but were also considered in the
' development of the final rulemaking.
i Discussion The pilot examinations demonstrated that the revised process, under which facility licensees prepare the written examinations and operating tests, is generally effective and l
' efficient. From the time the pilot program began in October 1995 through the end of September I
1 1997, the NRC staff reviewed, approved, and administered a total of 68 examinations that were voluntarily developed by facility licensees under the pilot examination and transition program.
~
Facility licensees prepared the written examinations and the operating tests, proctored the written examinations, and graded the written examinations using the guidance provided by the NRC in GL 95-06 during the early stages of the pilot program, and subsequently in interim 1
- Revision 8 of NUREG-1021, " Operator Licensing Examination Standards for Power Reactors."
l NRC examiners thoroughly reviewed the examinations and tests to determine if they were consistent with NRC standards, directed facility licensees to make whatever changes were necessary to achieve NRC standards if the submitted examinations and tests were deficient, t
and approved the examinations and tests before they were administered. NRC examiners independently administered all of the operating tests, reviewed the written examination grading, and made the final licensing recommendations for approval by NRC management.
Comments from the NRC chief examiners who evaluated the pilot examinations indicate
- that the quality and level of difficulty of the licensee-prepared examinations (when modified as L
5.
i-6
)
. directed by the NRC) are generally comparable to the examinations prepared by the NRC (i.e.,
J by the staff or NRC contractors). The passing rate on the 68 pilot-style examinations administered through the end of FY 1997 was only slightly lower than the passing rate on the power reactor licensing examinations administered during FY 1995, the last year in which all examinations were prepared by the NRC. However, considering the historical fluctuation in the average examination passing rates and the other factors (e.g., training program quality and screening of applicants by facility licensees) that could be responsible for some or all of the observed difference, the Commission has concluded that the observed change in the passing rates is not significant. The average grades on the facility-prepared, NRC-approved written examinations were also comparable if slightly lower than the grades on examinations prepared by the NRC during FY 1995. These data support the conclusion that the facility-prepared examinations are discriminating at a conservative and acceptable level and that the revised examination process is effective. Therefore, the fact that the facility licensees are' preparing the examinations with NRC review and approval, should have no negative effect on the safe operation of the plants.-
Although the NRC-approved examinations were comparable to NRC-prepared examinations, essentially all of the examinations prepared by facility licensees required some changes subsequent to NRC review, and many of the examinations required significant rework.
The NRC had originally believed that, with training and ' experience, the industry would quickly gain proficiency in preparing the examinations, but the overall quality of the examinations
)
submitted to the NRC during the pilot program did not improve as expected over time. Although approximately half of the 17 facility licensees that had prepared more than one examination by the end of FY 1997 did maintain or improve the quality of their second or third examination submittals, the quality of the other facility licensees'second or third examinations was lower.
g7 7
. Consequently, the NRC has asked the industry to address the issue of examination quality and determine the need for additional training on examination development. The NRC will continue to (1) direct facility licensees to revise the examinations as necessary to achieve an acceptable i
level of quality and discrimination, (2) delay those examinations that do not meet NRC standards by the scheduled ' administration date, (3) oversee the regional implementation of the j
operator licensing process to ensure consistency, (4) address significant deficiencies in the l
submitted examinations as licensee performance issues in the examination reports, as appropriate, and (5) conduct or participate in workshops, as necessary, to ensure that facility licensees understand the NRC's examination criteria, i
~
With regard to the efficiency of the revised examination process, the pilot examination experience to date suppcrts the conclusion that the average industry cost will not differ significant8y from the cost of NRC-prepared examinations. Comments from the industry reflect that the cost for some facility licensees to prepare the examinations was higher than it was for l
NRC-prepared examinations; however, other licensees prepared good quality examinations at l
lower cost than the NRC. The industry generally attributed the higher cost to the revised examination and administrative criteria under the pilot examination process. Although the NRC acknowledges that the revised criteria contribute somewhat to the elevated cost, many of the variables that affect the quality and, consequently, the cost of the examination will be under the facility licensees' control and can present an opportunity for cost savings. For example, facility u
licensees will be able to manage the size and quality of their examination banks and the training and experience of the personnel they select to write their licensing examinations. The revised examination process puts facility licensees in control of developing the examinations and holds them responsible for their quality, if a facility licensee submits a good quality examination, it is likely to save resources despite the additional administrative criteria; however, if the facility l
8 i
licensee submits an examination that requires many changes, it will likely cost more than if the
- NRC had prepared the examination.
Comments from the NRC chief examiners who worked on the pilot examinations indicate that the average amount of time spent reviewing and revising the facility-prepared examinations was generally consistent with the estimates developed before starting the pilot program. Although a number (approximately 20 percent in FY 1997) of the examinations required significantly more NRC effort than originally anticipated to bring them up to the NRC's standards, the resource burden was generally offset by other examinations that required less effort to review and revise. Therefore, the Commission believes that the original objective of the pilot examination and transition program has been achieved. The program has demonstrated that the NRC can eliminate its use of contractors in the operator licensing program (except for the generic fundamentals examination) and can conduct the initial operator licensing and requalification inspection programs with the existing NRC staff.
- Wdh regard to examination security, applicants, licensees (operators) and facility i
licensees, are reminded that 10 CFR 55.49 prohibits them from engaging in any activity that compromises the integrity of any application, tests, or examination required by 1 - CFR 55. The Commission considers an examination to be " compromised" if any activity occurs that could
- affect the equitable and consistent administration of the examination, regardless of whether the activity takes place before, during, or after the examination is administered. Based on several security incidents that occurred since beginning the pilot examination program, the NRC has concluded that applicants, licensees, and facility licensees may not understand that the provisions of 10 CFR 55.49 pertain to more than just those activities involving the physical administration of an examination. Therefore, the Commission has determined that it would be i
beneficial to amend 10 CFR 55.49 in the final rule to clearly state the expectation that facility i
1
9 l
!!canwees establish procedures to control examination security and integrity. Revision 8 of NUREG-1021 identifies a number of security provisions that facility licensees should consider when establishing their procedures.
Consistent with NUREG-1021, facility employees with specific knowledge of any NRC examination before it is given may not communicate the examination contents to unauthorized i
individuals and may not participate in any further instruction of the students scheduled to take the examination. Before they are given access to the examination, the facility employees are expected to sign a statement acknowledging their understanding of the restrictions and the potential consequences of noncompliance. When the examinations are complete, the same employees are expected to sign a post-examination statement certifying that they have not knowingly compromised the examination. In addition to the restrictions on personnel, NUREG-1021 also discusses a number of physical security precautions, including simulator considerations and protecting and mailing the examination materials.
NRC examiners are expected to be attentive to examination security measures, to review the security expectations with the facility licensee at the time the examination arrangements are confirmed, and to report any security concems to NRC management. If the NRC determines during its preparation that an examination may have been compromised, it will delay the examination until the scope of the potential compromise is determined and measures can bs taken to address the integrity and validity of the examination. Pursuant to 10 CFR 55.51, the NRC must make a determination prior to issuing a license that the test or examination is valid, meeting the requirements of the Atomic Energy Act and the Commission's regulations. If the compromise is discovered after the examination has been administered, the NRC will delay the licensing action for the affected applicants until the staff can make a determination regarding the validity of the examination. If the compromise is not discovered
q.
10 until after the licensing actlon is complete, the'NRC will reevaluate the licensing decision, if it
~
determines tha' the originallicensing decision was based on an invalid examination, and take appropriate action pursuant to 10 CFR 55.61(b)(2).
As a separate measure, the NRC will evaluate every potential examination compromise to determine if a violation of 10 CFR 55.49 has occurred. The Commission is modifying NUREG 1600," General Statement of Policy and Procedures for NRC Enforcement Actions,"
(Enforcement Policy) to cover instances where a compromise occurs but measures are taken to
-remedy the situation that created the compromise before the examination is adnunistered. A compromise that is discovered after un examination is given would be considered a significant regulatory concem and categorized at least at Severity Level 111. The NRC intends to utilize its,
full enforcement authority including, as wanranted, civil penalties and orders against persons who (1) compromised examination integrity in violation of 10 CFR 55.49, (2) committed I
deliberate misconduct in violation of 10 CFR 50.5, or (3) provided incomplete or inaccurate information to the NRC in violation of 10 CFR 50.9. In addition, cases involving willful violations
- will be referred to the Department of Justice.
The Commission has reviewed the vulnerabilities and costs associated with the revised examination process and considered the measures that the NRC staff has taken to mitigate the vulnerabilities. With regard to examination quality and level of difficulty, the Commission acknowledges that the effectiveness of the revised examination process is contingent on the NRC staffs review of the facility-proposed examinations to ensure that NRC standards are achieved. The Commission has concluded, based on the results of the pilot examination program, that the controls implemented by the NRC staff will provide reasonable assurance that the examinations that are administered to the license applicants will provide a valid and consistent basis upon which to'make the licensing decisions. The Commission also realizes h
11 that the frequency of examination security incidents and the risk of undetected compromises may increase. !1owever, the Commission is confident that the compensatory measures discussed above will sufficiently control the vulnerability in this area. The Commission is aware that the staff's original expectation that facility licensees would eventually realize cost savings p
under the revised process as they gain proficiency in preparing the examinations has not yet been realized. However, the Commission has concluded that neither the increased -
vulnerabilities nor the absence of clear industry cost benefit provides sufficient basis for discontinuing the revised examination process. The Commission also finds that the revised examination process is more consistent with the NRC's other oversight programs because it requires NRC examiners to review materials prepared by the facility licensees. The revised process enables NRC examiners to focus more on the psychometric quality of examinations (e.g., the cognitive level at which the questions are written and the plausibility of the distractors or wrong answer choices) prepaied by the facility licensees than on the technical accuracy of the examinations, which was their primary focus when the examinations were prepared by NRC contractors. This shift in the NRC examiners' focus, which is made possible by the revised examination development process, has the potential to improve the overall quality of the licensing examinations.
Summary of Public Comments The 75-day public comment period began when the notice of proposed rulemaking was published in the Federa/ Register (62 FR 42426) on August 7,1997, and closed on October 21, 1997. The notice (FRN) requested public comment on the proposed lule, on the L..........
e i
L 12 implementation guidance in interim Revision 8 of NUREG-1021, and on the folloiving two I
questions:
1 l-I
- 1. Are there portions of the operator exams that are common to all licensees, and would, therefore, be more efficiently developed by the NRC7 l
- 2. Is the conclusion in the regulatory analysis correct that it would be less costly for each licensee to prepare its own initial operator examinations to be reviewed, revised, and l
l administered by the NRC, than to have one NRC contractor prepare these exams for all licensed operators with the costs to be reimbursed by licensee fees?
The NRC received 13 comment letters on the proposed rule; two of the letters arrived after the comment period closed, but they were considered nonetheless. The respondents 1
included three NRC examiners, one contract examiner, five nuclear utilities and one utility employee, one nonpower reactor facility licensee, the State of lilinois, and the Nuclear Energy 1
Institute (NEI), which submitted its comments on behalf of the nuclear power industry. Copies of the public comrnents are available in the NRC Public Document Room,2120 L Street, NW l
(Lower Level), Washington, DC, and on the internet at "http://ruleforum. lint. gov /cgi-bin /
1 rulemake? source =OE PRULE".
1 Seven of the respondents (three NRC examiners, one contract examiner, one utility employee, one nonpower facility licensee, and the State of Illinois) recommended that the rule change be disapproved. Five of the industry respondents (NEl and four utilities) supported the i
rule change; however, one utility endorsed NEl's comments but stated that it did not agree with i
the proposed rule in its present form. NEl and two of the utilities stated that they would rather 1
continue with a voluntary program, but they would support mandatory participation with the rule l
change rather than return to the previous process under which NRC contractors wrote most of the examinations.
T l
13 Those comments related to the two specific questions raised in the proposed l
rulemaking, and those that have a direct bearing on the rule are discussed below. The comments are categorized as they relate to the vulnerabilities discussed in SECY-96-206 (i.e.,
l l
- quality and consistency; independence and public perception; security; NRC resources;and i
examiner proficiency) and reactor safety; there were no comments related to program stability.
i One NRC examiner, NEl, four of the utilities, and the utility employee also provided specific comments and recommendations regarding the implementation guidance in interim Revision 8 of NUREG-1021. Those comments are addressed in Enclosure 1 of the Commission (SECY) paper associated with this rulemaking. A copy of the SECY is available in the NRC Public Document Room, on the internet at http://www.nrc. gov, or from Siegfried Guenther, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, at 301-j 415-1056 or e-mail at sxg@nrc. gov.
i Comment: With regard to the first specific question included in the proposed rulemaking,2 of the 13 respondents (NEl and one utility) stated that all of the common material is already included in the generic fundamentals examination (GFE) and that the remaining elements are best covered as part of the site-specific examination.
Response' It appears that the current allocation of topics between the GFE and site-specific written examinations is generally perceived to be an efficient method of covering the topics required by 10 CFR 55.41 and 55.43. Therefore, the Commission finds no basis for changing the process to have the NRC separately develop portions of the initial examination that would be common to all facilities.
I Comment: Seven of the 13 respondents (NEl, two utilities, a utility employee, and three
14 examiners) directly or indirectly addressed the second specific question in their letters. NEl and one utility stated that the revised examination criteria in interim Revision 8 of NUREG-1021 have increased the level of effort and will result in higher licensing fees regardless of who prepares the examinations. However, NEl and another utility agreed that comparing the cost of utility-prepared examinations to those prepared by the NRC is difficult, but they concluded that it should be less costly for utilities to prepare the examinations than to have the NRC prepare them under the same criteria.
NEl also stated that the relative cost of the two examination processes should not be the only factor in deciding whether to proceed with the rulemaking. NEl indicated that preparing higher cognitive level questions requires detailed plant knowledge, better provided by facility licensees, and that the revised process (which has eliminated the use of NRC contractors to administer the operating tests) will allow NRC staff to evaluate each applicant without relying on third-party observers.
Two NRC examiners, one contract examiner, and a utility employee asserted that the facility licensees' cost has gone up under the revised examination process. They cited various reasons for the increased cost, including training personnel to write the examinations and then restricting them from training the applicants, and upgrading equipment to maintain examination security. The NRC examiners based their comments on feedback from facility training personnel; one examiner indicated that it took facility licensees an average of 700 hours0.0081 days <br />0.194 hours <br />0.00116 weeks <br />2.6635e-4 months <br /> to prepare each examination. The utility employee stated that the rule change will simply transfer the cost of contractors from the NRC to the utilities.
Response The NRC acknowledges that the pilot examination administrative criteria in particular (e.g., the restrictions on facility training personnel and the need to document the source of the test items) have probably caused the cost of preparing the examinations to be
15 so;newhat higher than it would have been if facility licensees had been allowed to prepare the l
l-examinations using the same criteria that applied to the NRC and its contractors before starting
)
the pilot program. However, when the NRC first developed the revised examination process, with its additional administrative criteria, the NRC still believed that the cost for facility licensees to prepare the examinations would be offset by the reduction in the licensing fees and that a cost savings could be realized as facility licensees gained experience with the process. Many of the facility licensees that participated in the pilot program demonstrated that it is possible to prepare a good quality examination at the same or lower cost than the NRC or its contractors could prepare a comparable examination. The fact that a number of facility licensees did not prepare acceptable examinations may be as much an indication of the licensees' inefficiency and inexperience as it is a symptom of deficiencies in the examination criteria. Those facility licensees that did not initially submit acceptable examinations, eventually paid more in fees because of the additional effort required for the NRC to review, and the licensees' staffs to rewrite, the examinations. Finally, it is possible that the magnitude of the increase in effort and cost may be perceived to be higher than it actually is because the industry had originally expected to save money if the NRC would have a!! owed facility licensees to prepare the examinations using the version of NUREG-1021 that was in effect before beginning the pilot program.
With regard to the additional security costs cited by the examiners, the Commission has i
stressed the importance of maintaining examination security, but the NRC has not required facility licensees to invest in additional physical security systems. However, the frequency of security incidents since beginning the pilot examination program has prompted the NRC to clarify the intent of 10 CFR 55.49 in the final rule and to include additional security guidance in final Revision 8 of NUREG-1021. This action will help ensure, among other things, that facility
1 16 I-licensees understand their responsibility for maintaining control over the examination process.
The pilot examinations demonstrated that some of the people assigned by the facility to
- develop the examinations did not have sufficient expertise required to prepare good quality examination materials consistent with NRC standards. As noted earlier, the NRC has asked the industry to address the issue of examination quality and the need for additional training on examination development.' The t' RC acknowledges that the restrictions on the use of instructors to prepare the licensing examinations may be partially responsible for limiting the availability of qualified examination preparers. Therefore, the staff has revised the personnel restrictions in final Revision 8 of NUREG-1021 in an effort to reduce the burden on facility licensees. Specifically, instructors will be allowed to prepare the written examination questions without regard to the amount of time they spent training the license applicants; however, the instructors will still be precluded from preparing questions related to those topics on which they provided instruction and from instructing the applicants once they begin working on the i
licensing examination (as was the case in interim Revision 8). This change should provide licensees with increased flexibility in managing their resources and possibly reduce their costs without adversely affecting the independence of the examination process.
The NRC has revised the regulatory analysis in response to the public comments and lessons learned from the pilot program. The NRC has also reevaluated the additional administrative criteria in interim Revision 8 of NUREG-1021 and considers them reasonable and essential to mitigate the vulnerabilities (e.g., quality, security, and conflict of interest) of the new examination process and to facilitate the NRC staffs review of the proposed examinations.
Comment Two NRC examiners with pilot examination experience reported that the quality of the simulator and walk-through tests has decreased significantly and that, in most
. cases, the quality and difficulty of the submitted examinations have been below NRC standards.
All four examiners cited various reasons why the quality and difficulty of the facility-prepared examinations might be lower than examinations prepared by the NRC or its contract examiners including: (1) the facility licensees' tendency to narrow the scope of the operating test to those procedures that the facility thinks are important (and emphasized in the training program) and (2) the belief that most facility training personnel do not have the expertise to develop valid test items. Two NRC examiners stated that the quality of the examinations has not improved during the pilot program and is not likely to improve because there is nothing to prevent licensees from
. using different people to develop successive examinations. A utility employee asserted that the utilities' limited contact with the process by preparing an examination once every 18 to 24 months will not foster consistency or develop skilled examination writers.
Two NRC examiners stated that the elimination of NRC contract examiners who participated in examinations across the four NRC regions will be detrimental to examination consistency. One NRC examiner stated that the guidance in interim Revision 8 of NUREG-1021 is not sufficiently prescriptive to ensure nationwide consistency in the level of knowledge tested and the level of difficulty of the examinations and that several specific changes should be
' included in NUREG-1021 to address his concems.
The State of Illinois asserted that the quality and consistency of the written examination questions can be maintained because the NRC can change and approve the questions before they are used. However, the State also recommended that the NRC should compile the examination questions and proctor the examinations (refer to the conflict-of-interest discussion below).
According to NEl, the recent facility-prepared examinations were of higher quality than the examinations prepared by the NRC before the pilot program started. Many of the NRC-
i i
18 prepared examinations had to be revised in response to the facility licensees' technical reviews.
Resoonse: Essentially all of the facility-prepared examinations required some changes j
(
and many required significant changes to make them conform to the NRC's standards for
]
i quality and level of difficulty. According to the questionnaires completed by the NRC chief examiners responsible for the pilot examinations, the average facility-prepared written examination required approximately 10 to 20 changes, which is consistent with the number of changes often required on examinations prepared by NRC contract examiners. Most NRC chief examiners judged the final examinations, with the NRC's changes incorporated, to be comparable to recent NRC-prepared examinations in terms of quality and level of difficulty.
Moreover, the fact that the passing rate on the facility-prepared examinations is generally consistent with the historical passing rate on examinations prepared by the NRC suggests that the NRC-approved examinations have discriminated at an acceptable level and that they have provided an adequate basis for licensing the applicants at those facilities.
Although the NRC expected that the proposed examination quality would improve as facility licensees gained experience and familiarity with the NRC's requirements and expectations, the overall quality of examinations submitted to the NRC during the transition process did not improve appreciably over time. Although approximately half of the 17 facility licensees that had prepared more " 'n one examination by the end of FY 1997 did maintain or improve the quality of their secorm wr third examination submittals, the quality of the other l
facility licensees'second or third examinations was lower. Although it is unclear to what extent the problems with proposed examination quality and difficulty have been caused by a lack of sufficient expertise on the part of the examination writers, the NRC has asked the industry to address this issue. Furthermore, the NRC staff has conducted and participated in a number of public meetings and workshops in an effort to communicate its expectations to the facility
i I
L 19 l
. cmployees who will be preparing the examinations. Additional NRC and industry workshops l
l
- will be conducted to address examination quality and solicit industry feedback.
In SECY-96-206, the NRC staff discussed the issues of examination quality and consistency and how they might be affected when a large number of facility employees assume l
the role that had been filled by a smaller number of axperienced NRC and contract examiners.
The NRC staff's comprehensive examination reviews versus the examination criteria in NUREG-1021, in combination with supervisory reviews and the examination oversight activities conducted by the Office of Nuclear Reactor Regulation, should mitigate the vulnerability in this area. Moreover, the industry and staff initiatives to improve the expertise of the examination writers should eventually enhance examination quality and consistency.
Comment: All four examiners who submitted comments, a nonpower reactor facility licensee, and the State of Illinois asserted that allowing the facility licensees to prepare the operator licensing examinations decreases the level of independence and creates a conflict of
' interest for facility personnel having responsibility for training and licensing the operators. Their letters maintained that the new process makes it possible for the utilities to " teach the examination," to test applicants only on what was taught, or to avoid testing in areas with known difficulties. One NRC examiner noted that the new process places training managers in a no-win situation because if applicants fail the examination, the managers look like poor trainers, and if the examination is too easy, the NRC gives them a bad report. He and another NRC euminer reported, based on their experience during the pilot examinations, that some facility personnel openly admit that they would develop the easiest possible examination to ensure that all their applicants would pass.
One NRC examiner noted that the NRC review and approval process cannot adequately
20 compensate for the conflict-of-interest problems inherent in this approach and recommended a change to interim Revision 8 of NUREG-1021 that would limit the licensees' latitude in selecting topics for the examination outline. The State of Illinois suggested that the NRC should compile the questions and proctor the examination to maintain more of the checks and balances that existed under the old process.
The nonpower reactor facility licensee noted that most professional licensing examinations are developed by independent agencies, and that this fosters a sense of professionalism in the license applicants.
Response The NRC agrees that the revised examination process decreases the level i
of independence in the licensing process and may create a potential conflict of interest for i
facility personnel involved in preparing the examination. As discussed in SECY-96-206, the NRC included a number of measures in Revision 8 of NUREG-1021, including restrictions on which facility employees can develop the examination and a comprehensive review of every examination by an NRC examiner, to ensure that an adequate level of independence is maintained and to minimize the potential for bias in the examination development process.
Although the NRC wi!! no longer prohibit instructors from participating in the examination development based solely on the amount of time they spent training the license applicants (as discussed above in response to comments concerning the industry burden under the revised examination process), this change is not expected to substantially increase the potential for bias in the operator licensing process because instructors will still be prohibited from writing questions related to the topics in which they provided instruction. Moreover, as a separate measure, the NRC has amended final Revision 8 of NUREG-1021 to include an expectation that facility licensees will use an objective, systematic process for preparing the written examination outline. This process enhancement should further limit the potential for bias in the
I 21 selection of topics to be evaluated on the written examination.
The Commission has concluded that the amended personnel restrictions, in combination I
with the systematic selection of topics for the written examination and the continued independent selection of topics for the operating test, will establish a sufficient level of independence to address the vainerability associated with the conflict of interest inherent in the revised examination process. However, if the NRC determines that a facility licensee has intentionally biased the scope, content, or level of difficulty of an examination (i.e.,
compromised its integrity contrary to 10 CFR 55.49) to enhance the chances that its applicants 1
would pass the examination, the NRC will utilize its enforcercent authority including, as warranted, civil penalties, orders against the individuals involved, and, charging the individuals involved with deliberate misconduct pursuant to 10 CFR 50.5.
Concems regarding the potential for conflict of interest and the frequency of security incidents since beginning the pilot examination program have prompted the NRC to review the clarity of 10 CFR 55.49. The regulation encompasses not only activities like cheating and lapses in security I ut also activities that compromise the integrity or validity of the examination itself (e.g., noncompliance with the criteria designed to limit the potential for bias in the selection of topics to be evaluated on the written examination). Therefore, the NRC has concluded that it would be beneficial to amend 10 CFR 55.49 to clearly state the expectation that facility licensees establish procedures to control examination security and integrity.
Comment Three NRC examiners and the State of Illinois asserted that the revised examination process increases the threat to examination security. One examiner noted that the examination is onsite for a longer period of time, thereby proportionally increasing the risk of being compromised. Another examiner cited the fact that a number of examination reports l
i
22
~
have documented problems with security.
Response: As discussed in SECY-96-206 and in the proposed rulemaking, the 1
Commission is aware of the vulnerability in this area because several security incidents have occurred since beginning the pilot examination program. Therefore, based on the comments received and the experience with security incidents,.the NRC has clarified 10 CFR 55.49 in the final rule to ensure that applicants, licensees, and facility licensees understand the scope and intent of the regulation. The NRC has also strengthened the discussion of examination security in final Revision 8 of NUREG-1021 and modifieKi NUREG-1600, " General Statement of Policy i
1 1
and Procedures for NRC Enforcement Actions," to address enforcement action against parties subject to the requirements in 10 CFR 55.49. NRC examiners are expected to review the i
NRC's physical security guidelines and the facility licensee's specific plans for ensuring examination security at the time the examination arrangements are confirmed with the designated facility contact. Furthermore, the NRC has issued an information notice to advise power reactor facility licensees of the NRC's perspective and expectations regarding the integrity of examinations developed by the facility licensees' employees and representatives, and it has asked NEl to take the initiative in developing a model for securing examinations.
As a separate action, the NRC will delay any examination that may have been compromised until the scope of the potential compromise is determined and measures can be taken to address the integrity and validity of the examination. If the compromise is discovered
- after the examination has been administered, the NRC will delay the licensing action for the affected applicants until the staff can make a determination regarding the impact that the i
compromise has had on the examination process. If the compromise is not discovered until after the licensing action is complete, the NRC will reevaluate the licensing decision pursuant to i
I
1 l
l 23 1
l 10 CFR 55.61(b)(2) if it determines that the original licensing decision was based on an invalid l
examination.
Comment One NRC examiner disagreed with the conclusion in the proposed rulemaking that the facility-prepared examination process is an efficient use of NRC resources when compared to the NRC-prepared or contractor-prepared examinations. He noted that, in most cases, the quality and difficulty of the proposed examinations have been below NRC standards (as discussed above) and that it has taken a significant effort on the part of the NRC chief examiner to achieve an acceptable product.
An NRC contract examiner asseded that NRC cost-saving is a poor reason for changing the rule, since the utilities pay for the examinations anyway. He noted that the pilot examination process has led to a loss of certified examiners and contends that those NRC examiners who are left will become more dissatisfied with theirjobs and will leave because they will be required to travel more to compensate for the loss of contractors.
I Response: The NRC acknowledges that many of the facility-prepared examinations (about 20 percent in FY 1997) required significantly more NRC examiner time than desired or planned in order to achieve NRC quality standards. However, questionnaires filled out by NRC chief examiners for the pilot examinations indicate that the average amount of time spent on reviewing and upgrading the examinations is generally consistent with the estimates developed before starting the pilot program (i.e. approximately 170 examiner-hours). As noted in SECY-97-079, the NRC has issued a meinorandum to the regional administrators emphasizing the importance of (1) assigning adequate resources to carry out the operator licensing task, (2) completing a thorough review of every facility-prepared examination, a id (3) not administering any examination that fails to meet NRC standards for quality and level of difficulty.
l
24 Furthermore, all the time that NRC examiners spend reviewing an examination and bringing it up to standards is ultimately billed to the facility licensee.
The Commission acknowledges that facility licensees bear the cost of preparing the licensing examinations whether or not the NRC performs this function. However, this comment does not take into account the NRC's internal accounting p ocedures. As a result of this process change, the NRC has been able to reprogram the funds that were saved to perform otherfunctions.
The NRC's budget cuts have necessitated agency-wide downsizing, which can be expected to result in increased travel for many NRC employses, not just the operator licensing examiners. The number of NRC full-time equivalent (FTE) license examiners has remained essentially constant throughout the pilot program and, aside from normal attrition and staff turnover, the loss of certified examiners has been limited to NRC contractors.
Comment: Two NRC examiners expressed concem that examinerproficiency will decrease as a result of implementing the revised examination process. One of the examiners stated that examination reviewers will not maintain the same base of knowledge as examination writers maintained and that they will lose their familiarity with plant operating procedures.
Resoonse: The Commission has concluded that the revised examination process affords sufficient NRC staff involvement that NRC examiners will maintain an acceptable level of proficiency. An NRC examiner will review and approve every examination before it is administered to ensure that it conforms to the criteria specified in NUREG-1021 for content, format, quality, and level of knowledge and difficulty. NRC examiners will also continue to independently administer and grade both the dynamic simulator and the plant walk-through portions of the operating tests. Because NRC examiners will be administering all of the
p L
25-operating tests, the Commission believes that the revised process will enable the examiners to accrue more experience in a shorter period of time and to maintain their proficiency. New NRC license examiners will still be required to complete a standardized training program, including i
. the development of a complete written examination and operating test, as part of their qualification process. Moreover, the NRC will ensure that the in-house capability to prepare the examinations is maintained by (1) requiring each region to write at least one initial operator licensing examination per calendar year, (2) requiring a regional supervisor to review and approve every examination and the Office of Nuclear Reactor Regulation to conduct periodic examination reviews, (3) conducting examiner refresher training, and (4) convening an operator licensing examiners' training conference at intervals not to exceed 24 months.
Comment:' A utility employee asserted that the revised examination process will not enhance the competency of the operators or reactor safety because the facilities' training resources will be diverted from their primary purpose (i.e., training the applicants) as much as six months before the examination date. Three NRC examiners also took issue with the
. conclusion in the proposed rulemaking that the NRC staff's focus on operator performance and its core of experience will improve under the pilot examination process because contractors will no longer be used to administer the operating tests. Two of the examiners asserted that the reduction in the amount of procedural research by examiners will result in the identification and correction of fewer procedural problems. Two of the examiners also stated that the contract examiners help maintain examination consistency' across the NRC regions and that their
- contribution to the operator licensing program goes beyond simple task completion.
Response The Commission expects that those training departments that cannot readily and safely absorb the examination development work will use the funds that they were 1
26' previously paying to the NRC through the fee recovery program to secure the additional personnel to do the extra work. If a facility licensee places insufficient resources on either training or testing, the quality of its proposed licensing examinations or the passing rate on those examinations would most likely suffer. Although many of the facility-prepared examinations have required significant changes to achieve NRC quality standards, the examination results, to date, are generally consistent with the results on previous NRC-prepared examinations, suggesting that the quality of the facility licensees' training programs has not been affected. Therefore, the fact that facility licensees will be preparing the examinations is not expected to have a negative effect on reactor safety.
The NRC acknowledges that the contract examiners identified procedural and training problems in addition to their primary responsibility for preparing and administering the licensing examinations, and that they helped maintain examination consistency by working on examinations in each of the NRC's regions. As noted in connection with the discussion of examination quality, the Commission realizes that the revised examination process increases the possibility of inconsistency, but it believes that the examination criteria in final Revision 8 of NUREG-1021, in combination with the NRC's examination oversight programs, will minimize such inconsistencies so that they remain within acceptable limits.
When the NRC initiated the pilot program, its goal was to eliminate the need for NRC contract examiners without compromising the existing levels of reactor safety. Because NRC examiners will be administering all of the operating tests, the revised process will enable the NRC examiners to accrue more experience in a shorter period of time and may improve the consistency of the operating test evaluations and the licensing decisions. Although the total i
number of procedmes reviewed in the process of developing examinations may be fewer under the revised method, NRC examiners will still be expected to review and identify discrepancies in
{
I
I l
27 the procedures that will be exercised during the walk-through portion of the operating test and during the simulator scenarios.
Other Comments
(
l Since beginning the pilot examination program, the Commission has sought to obtain up-to-date insights regarding the effectiveness of the revised examination process based on the i
staff's growing body of experience in reviewing the facility-prepared examinations. Many of the j
i staff comments received have paralleled the public comments and require no further attention in this notice. However, one recommendation to amend the actual wording of the proposed regulation is considered worthy of discussion and incorporation. Specifically, it was recommended that the rule should indicate that a key manager would be responsible for submitting the examination because that individual would be in a position to ensure that the facility licensee's operations and training departments apply sufficient resources to prepare a quality examination. The Commission finds that the recommendation is consistent with normal NRC practice and the analogous regulatory requirement in $55.31(a)(3), which requires "...an authorized representative of the facility licensee by which the applicant will be employed..." to submit a written request that examinations be administered to the applicant. Therefore, the wording of $55.40(a)(2) has been amended to require an authorized representative of the l
facility licensee to approve the written examinations and operating tests before they are l
submitted to the NRC for review and approval.
l l
l t
28 Availability of Guidance Document for Preparing Operator Licensing Examinations As a consequence of preparing and administering the initial operator licensing examinations over a number of years, the NRC has developed a substantial body of guidance to aid its examiners. That guidance has been published in various versions of NUREG 1021, the latest version of which (final Revision 8) incorporates lessons leamed since interim Revision 8 was published in February 1997 as well as refir ements prompted by the comments submitted in response to the FRN of August 7,1997 (62 FR 42426), which solicited pubhc comments in conjunction with the proposed rulemaking. A copy of final Revision 8 of NUREG 1021 will be mailed to each facility licensee. Copies may be inspected and/or copied for a fee at the NRC's Publ6c Document Room 2120 L Street NW (Lower Level) Washington. DC. Final Revision 8 of NUREG 1021 is also electron 6cally available for downloading from the intemet at "http //www.ntc. gov,"
The NRC plans to prepare, admir.! ster, and grade initial operator licensing examinations at least four times a year, using NUREG 1021 as gu6 dance. Licensees will also be expected to use the guldance in NUREG 1021 to prepare the 16 censing examinat6ons The NRC staff will review and approve any sitematives that do not conform with this guidance The NRC will not approve any sitemative that would compromise its statutory responsitslity of prescribing uniform conditP>ns for the operator 16 censing examinat6ons. Examples of unacceptable attematives include, but are not limited to, the use of essay questions in place of muffiple choice questions and the administration of open book examinations
s
'29 Final Rule j
I This regulation adds a new section, $55.40, " Implementation," to Subpart E of 10 CFR 1
Part 55, which requires power reactor facility licensees to prepare the written examinations and
)
operating tests, to ' submit the examinations that have been approved by an authorized 1
representative of the facility licensee to the NRC for review and approval, and to proctor and grade the written examinations. These requirements are contained in $$55.40(a)(1), (2), and (3), respectively.-
1 Each power reactor facility licensee is expected to prepare and submit the proposed j
examinations (including the written examination, the walk-through, and the dynamic simulator i
- tests) to the NRC consistent with the guidance in NUREG-1021. An authorized representative I
. of the facility licensee shall approve the written examinations and operating tests before they j
i are submitted to the NRC. The NRC staff will review the entire examination and direct whatever changes are necessary to ensure that adequate levels of quality, difficulty, and consistency are maintained. After the NRC staff reviews and approves an examination, the
. facility licensee will proctor and grade the wntten portion consistent with the guidance in i
NUREG-1021. The NRC staff will continue to independently administer and grade the operating tests, review and approve the written examination results, and make the final licensing decisions. The facility licensee will not conduct parallel operator evaluations during i
i the dynamic simulator or the walk-through tests.
l Pursuant to the requirements in $55.40(b), the NRC staff maintains the authority to prepare the examinations and tests and to proctor and grade the site-specific written examinations. This allows the NRC to maintain its staff proficiency to perform these activities, i
I 30 Also, if the NRC has reason to question a licensee's ability to prepare an acceptable examination, $55.40 (b) gives the NRC authority to prepare and administer the examinations i
and tests.
Paragraph (c) of $55.40 reasserts that the NRC will continue to prepare and administer the written examinations and operating tests at non-power reactor facilities. The NRC has taken this position because the non-power reactor community does not have an accreditation process for training and qualification or the resources to prepare the examinations.
This regulation also amends $55.49 because the NRC has determined, since the proposed rule was published, that applicants, licensees, and facility licensees may be interpreting $55.49 too narrowly by limiting it to actual cases of cheating. The amendment clarifies that the regulation pertains to all activities that could affect the equitable and consistent administration of the examination, including activities prior to, during, and after the examination is administered. The amendment also states the NRC's expectation that facility licensees will establish procedures to control examination security and integrity.
Environmental impact: Categorical Exclusion i
The NRC has determined that this rule is the type of action described as a categorical exclusion in 10 CFR 51.22(c)(1). Therefore, neither an environmentalimpact statement nor an environmental assessment has been prepared for this regulation.
31 l
Paperwork Reduction Act Statement This final rule amends information co!Iection requirements that are subject to the Paperwork Reduction Act of 1995 (44 U.S.C. 3501 et seq). These requirements were approved by the Office of Management and Budget (OMB), approval number 3150-0101. The additional i
public reporting burden for this collection of information is estimated to average 500 hours0.00579 days <br />0.139 hours <br />8.267196e-4 weeks <br />1.9025e-4 months <br /> per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information (i.e., preparing the examinations). Send comments on any aspect of this collection of information, including suggestions for reducing the burden, to the Information and Records Management Branch (T-6F33), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by intemet electronic mail to bjs1@nrc. gov, and to the Desk Officer, Office of Information and Regulatory Affairs, NEOB-10202, (3150-0101), Office of Management and Budget, Washington, DC 20503.
Public Protection Notification i
1 The NRC may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.
i Regulatory Analysis The Commission has prepared a regulatory analysis on this regulation. The analysis examines the costs and benefits of the attematives considered by the Commission. The
i-32 regulatory analysis is available for inspection in the NRC Public Document Room, 2120 L Street NW (Lower Level), Washington, DC. Single copies of the analysis may be obtained from Siegfried Guenther, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, at 301-415-1056 or by e-mail at sxg@nrc. gov.
Regulatory Flexibility Certification i
in accordance with the Regulatory Flexibility Act of 1980, (5 U.S.C. 605(b)), the i
i Commission certifies that this rule does not have a significant economic impact on a substantial number of small entities. This rule affects only the licensing and operation of nuclear power plants. ' The companies that own these plants do not fall within the scope of the definition of "small entities" described in the Regulatory Flexibility Act or the Small Business Size Standards i
stated in regulations issued by the Small Business Administration at 13 CFR Part 121.
Backfit Analysis The pertinent part of 10 CFR 50.109(a)(1) defines backfitting as "the modification of or j
l addition to... the procedures or organization required to... operate a facility; any of which may
. result from a new or amended provision in the Commissihcrules or the imposition of a regulatory staff position interpreting the Commissbrules that is either new or different from a previously applicable staff position...." Part 55 addresses the qualifications and requirements for operators' licenses and changes are not per se subject to the backfit rule in Part 50.
l Changes to these requirements could be included within the backfit definition of " procedures or organization required to... operate a facility;" however, in this case, the shift of responsibility l
- y
33
. from the NRC staff (or its contrectors) to the facility licensee for developing and administering the initial written operator license examination, developing the initial operating tests, and establishing procedures to control examination security and integrity would not constitute a
" modification of the procedures required to operate a facility" within the scope of the backfit rule.
Therefore, no backfit analysis is necessary.
This rule does not affect the basic procedures for operator license qualification, i.e., the required training programs, the required testing, the content and format of the exams, the grading of the exams, or the basis for issuing an operator license. The shift in responsibility for preparation of the initial licensing examination does not affect the content or format of the examination. The rule is designed to ensure that the format, content, and quality of the initial written examination will not be modified. The rule requires the NRC to provide oversight of facility licensees' development and administration of initial written examinations and the development of the operating tests. The NRC also retains its discretion to determine whether to administer the initial written examination itself, as well as continuing to determine whether to grant or deny an application for a reactor operator or senior reactor operator license, and consider candidates' appeals.
The licensee's organizational structure required to operate the facility will not be modified. All reactor licensees have a training component as part of their organizational
- structure, and this rule does not alter that organizational structure. Although, the rule could have an "effect" on the licensee's organization, it does not require any modification to the
' organizational structure.
L..
i 34 Enforcement Policy in conjunction with this final rule, the Commission is separately publishing modifications to NUREG-1600, " General Statement of Policy and Procedure for NRC Enforcement Actions,"
to address enforcement action against parties subject to the requirements in 10 CFR 55.49 (i.e., Part 55 license applicants / licensees and Part 50 licensees).
List of Subjects in 10 CFR Part 55 Criminal penalties, Manpower training programs, Nuclear power plants and reactors, Reporting and recordkeeping requirements.
For the reasons given in the preamble and under the authority of the Atomic Energy Act of 1954, as amended; the Energy Reorganization Act of 1974, as amended; and 5 U.S.C. 553; the NRC adopts the following amendments to 10 CFR Part 55.
1 PART 55 - OPERATORS' LICENSES 1.
The authority citation for Part 55 continues to read as follows:
AUTHORITY: Secs. 107,161,182,68 Stat. 939,948,953, as amended, sec. 234,83 Stat. 444, as amended (42 U.S.C. 2137,2201,2232,2282); secs. 201, as amended,202,88 Stat.1242, as amended, 1244 (42 U.S.C. 5841, 5842).
E 35 Sections 55.41,55.43, 55.45, and 55.59 also issued under sec. 306, Pub. L.97-425,96 Stat. 2262 (42 U.S.C.10226). Section 55.61 also issued under secs. 168,187,68 Stat. 955 (42 U.S.C. 2236, 2237).
)
- 2. In 955.8, paragraph (b) is revised to read as follows:
655.8 information Collection Reauirements: OMB Anoroval.
i (b) The approved information collection requirements contained in this p art appear in
$955.31, 55.40, 55.45, 55.53, and 55.59.
- 3. A new $55.40 is added to read as follows:
655.40 Imolementation (a) Power reactor facility licensees shall -
(1) Prepare the required site-specific written examinations and operating tests; (2) Upon approval by an authorized representative of the facility licensee, submit the written examinations and operating tests to the Commission for review and approval; and (3) Proctor and grade the NRC-approved site-specific written examinations.
(b) In lieu of requiring a specific power reactor facility licensee to prepare the examinations and tests or to proctor and grade the site-specific written examinations, the Commission may elect to perform those tasks.
(c) The Commission will prepare and administer the written examinations and operating tests at non-power reactor facilities.
t
36
- 4. $55.49 la revised to read as follows:
$55.49 Intearity of examinations and tests.
Appilcants, licensees, and facility licensees shall not engage in any activity that compromises the integrity of any application, test, or examination required by this part, and facility licensees shall establish, implement, and maintain procedures to prevent compromises.
An examination or test is considered compromised if any activity, regardless of intent, affected, or but for detection would have affected, the equitable and consistent administration of the test or examination. This includes activities related to the preparation and certification of the license applications and all activities related to the preparation, administration, and grading of the required tests and examinations.
Dated at Rockville, Maryland, this day of 1998.
For the Nuclear Regulatory Commission.
John C. Hoyle, Secretary of the Commission.
i l
Regulatory Analysis
REGULATORY ANALYSIS FOR RULEMAKING ON REQUIREMENTS FOR INITIAL LICENSED OPERATOR EXAMINATIONS 1.
Statement of Problem and Objective Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate.
To implement this statutory mandate, operator license applicants are required by 10 CFR Part 55, " Operators' Licenses," to pass a written examination and an operating test. The written examination and operating test must satisfy the basic content requirements that are specified in the regulation. Although neither the AEA nor Part 55 specifies who must prepare, administer, and grade these examinations, the NRC has traditionally performed those tasks itself or through its contract examiners. Because this has been a costly process in terms of NRC personnel and contractual support, the NRC staff has evaluated an attemative approach - one that will require nuclear power plant licensees to prepare the examinations and submit them to the NRC for review and approval. This approach has been tested and assessed through a voluntary pilot program and has been deemed by the NRC staff to be feasible. The monitoring and assessment of this voluntary pilot program has demonstrated that examinations prepared by facility licensees and modified as directed by the NRC are comparable in terms of their quality and discrimination validity to those prepared by the NRC and its contract examiners under the existing process; therefore, the safe operation of the nuclear power plants is not compromised by using the altemative approach. Thus, the NRC is amending 10 CFR Part 55 to require nuclear power plant licensees to prepare these examinations and has published a final version of Revision 8 of NUREG-1021,
" Operator Licensing Examination Standards for Power Reactors," as guidance. This action will allow the NRC to eliminate the need for contractor support for the operator licensing program (with the exception of the generic fundamentals examination). The fiscal year (FY) 1998 and FY 1999 budgets are consistent with this proposal and already reflect the elimination of contract examiner support made possible under the voluntary pilot and transition program.
The NRC staff's primary objective in shifting responsibility for preparing the initial operator licensing examinations to the power reactor facility licensees is to reduce the amount of NRC resources used in this area. This change in policy is part of the NRC's continuing effort to streamline the functions of the Federal Govemment consistent with the Administration's initiatives and to accommodate NRC resource reductions. Given the limited resources available, the NRC has judged that the operator licensing contractor funds could be better spent on other work rather than paying for examinations that the facility licensees are generally qualified and motivated to prepare. Pursuant to the provisions of the AEA, the NRC will ensure that the quality of the operator licensing examinations and the effectiveness of the operator licensing program are maintained.
These changes are not intended to affect the format, content, scope, quality and level of difficulty of the examinations, thereby minimizing the impact of the rule change on the operator license applicants.
1
2.
Backgrc,und 10 CFR 55.31(a)(3) requires the applicant for an operator's license to submit a written request from an authorized representative of the facility licensee that the written examination and the operating test be administered to the applicant. Furthermore, 10 CFR 55.33(a)(2) states that the Commission will approve an initial application fo, a license if it finds that the applicant has passed the requisite written examination and operating test in accordance with 9955.41 and 55.45 for reactor operators, or $$55.43 and 55.45 for senior operators. These written examinations and operating tests determine whether the applicant for an operator's license has leamed to operate a facility competently and safely, and additionally, in the case of a senior operator, whether the applicant has leamed to direct the licensed activities of licensed operators competently and safely.
As stated above, the NRC or its contract examiners have traditionally prepared, administered, and graded the written examinations and operating tests. Before beginning the pilot program, the NRC spent approximately $3 million per year to retain contractor support for the operator licensing programs (including initial licensing examinations and requalification inspections), in accordance with 10 CFR 170.12 (i),
the NRC staff and contractual costs are recovered from the facility licensees that receive examination services.
This rule will change the current practice in which the NRC prepares and proctors the initial examination for reactor operators and senior operators and, instead, will require each power reactor facility licensee to prepare the entire examination and proctor and grade the written portion of the initial examination. This action does not modify the procedures or organization required to operate the plant, and, therefore, does not constitute a backfit pursuant to 10 CFR 50.109.
3.
Identification and Analysis of Three Alternative Approaches 3.1 Alternative 1 -Take No Action As discussed above, the budget for FYs 1998 and 1999 is consistent with this action and reflects the elimination of contract support. If 10 CFR Part 55 is not amended, it may require actions such as (1) the restoration of contractor support for the operator licensing program, (2) an increase in the direct examiner resources in each regional office to satisfy the demand for initial licensing examinations and inspections, (3) the implementation of other measures (e.g., changing the format or length of the examinations) that will enable the existing NRC staff to conduct the examination and inspection programs, or (4) scheduling the examinations based on the NRC resources available (which may require the staff to prioritize its activities based on the facility licensees' needs).
2
3.2 Alternative 2 - Provide Regulatory Guidance This alternative was rejected because the NRC staff considers implementation of the new process on a voluntary basis alone unworkable over the long term. If the NRC does not require facility licensees to prepare the initial operator licensing examinations, there will be no guarantee that each licensee would elect to prepare these examinations.
With the elimination of contractor support, the NRC staff may no longer have sufficient examiner resources to prepare examinations consistent with the scheduling needs of facility licensees. This resource problem is further compounded by the unpredictable nature of the examination workload and by other unanticipated demands on the examiner work force.
l 3.3 Alternative 3 - Amend 10 CFR Part 55 l
This alternative requires every power reactor facility licensee to prepare the initial operator licensing examinations and to proctor and grade the written portion of the examination. This enables the NRC to eliminate the use of contractors in the operator licensing program (with the exception of the generic fundamentals examinations).
Before initiating the pilot examination and transition program at the beginning of FY 1996, the NRC spent approximately $3 million per year on contrarW support for licensing examinations and requalification inspections. Under this dtornative, the NRC staff will perform those tasks that were previously performed by contract examiners, including examination administration and inspections of licensee requalification programs. The funds will be spent on other, high priority NRC work, rather than paying for contractors to prepare examinations that the facility licensees are generally qualified to prepare. As discussed above, the budget for FYs 1998 and 1999 is consistent with this action and reflects the elimination of contract support.
4.
Regulatory impact - Qualitative Costs and Benefits 1
Facility Licensees 1
Before beginning the pilot examination program, the NRC depended on NRC employees and contractors to prepare and administer the initial operator licensing examinations required by 10 CFR Part 55. NRC contractors also assisted in the inspection of requalification programs administered by facility licensees. In accordance with 10 CFR 170.12 (i), the cost of NRC time spent and any related contractual costs were (and are) billed directly to the facility licensees that receive (d) the examination services.
Under the revised examination process, each power reactor facility licensee will assume responsibility for preparing the site-specific initial operator licensing examinations at its l
facilities. This will allow the NRC to discontinue the use of contract examiners for both the initial licensing and requalification inspection programs. Facility licensees will be 3
expected to prepare the proposed examinations (including the written examination, the I
walk through, and the dynamic simulator tests) based on the guidance in NUREG 1021 and to submit the examinations to the NRC for review and approval. Facility licensees 3
I l
will have the option to hire a contractor to prepare the license examination (as the NRC often did before starting the pilot process).
This rule change gives facility licensees more control over the cost of their examination services because it puts them in a position to manage the quality of the product that is submitted to the NRC. The higher the quality of the examination that the facility licensee submits, the lower the resulting NRC charges. If the NRC or one of its contractors writes an examination, the facility licensee is responsible for the entire cost of prepring the examination.
The NRC will thoroughly review the examinations prepared by the facility licensees consistent with the guidance in NUREG-1021 and will direct the facility training staffs to make whatever changes are necessary to achieve a product that meets the NRC's standards for quality and level of difficulty. Each facility licensee will be billed only for the time that the NRC staff spends to review the examination prepared by the facility licensee, to direct the revisions that are necessary for the examination to meet NRC standards, and to administer and grade the operating tests. The NRC will not approve any examination that fails to meet its standards or that compromises the NRC's statutory responsibility for prescribing uniform conditions for the licensing of operators pursuant to the AEA. Examples of unacceptable deviations include, but may not be limited to, the use of essay questions in lieu of multiple choice questions and the administration of open-book examinations.
After the NRC reviews and approves an examination, the facility licensee will proctor and grade the written portion according to the guidance in NUREG-1021. The NRC staff will continue to administer and grade the operating tests, review and approve the written examination results recommended by the facility licensee, and make the final licensing decisions.
Based on lessons learned and feedback from the pilot examinations completed at the time, the proposed regulatory analysis predicted that the average time spent by a facility licensee to prepare the written examination and operating tests would be approximately 600 to 800 staff-hours. Because a portion of that time (about 200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />) would be spent reviewing and assisting with the administration of NRC-developed examinations under the traditional examination process,200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br /> should be subtracted from the total. The resulting average burden of approximately 400 to 600 staff-hours was somewhat higher than the 400 hours0.00463 days <br />0.111 hours <br />6.613757e-4 weeks <br />1.522e-4 months <br /> that the NRC staff or its contract examiners typically take to prepare an examination. The extra burden was generally attributable to the facility licensees' unfamiliarity with specific NRC examination expectations and to the additional administrative tasks, such as documenting the source of the examination questions, required to maintain examination integrity. The NRC staff believed that the facility training staffs already had the basic knowledge, skills, and abilities necessary to evaluate operator performance and develop test items for the initial licensing examination and expected that the burden would diminish as facility training staffs gained familiarity and experience with the NRC's spacific examination requirements and expectations. Moreover, the NRC staff expected that the facility employees' more detailed knowledge of their facility and easier access to the reference materials required 4
to prepare the examinations would eventually enable them to prepare acceptable quality examinations in less time than the NRC or a contractor could.
In an effort to quantify the potential cost savings, the proposed regulatory analysis assumed that the facility licensees would eventually prepare the examinations in the same amount of time as the NRC allotted its contract examiners to perform the task (i.e., approximately 400 hours0.00463 days <br />0.111 hours <br />6.613757e-4 weeks <br />1.522e-4 months <br />), and that the time would be equally distributed between l
4 contractors and the licensee's own in-house staff at a cost of $120 per hour and $60 per hour, respectively. This translated into an industrywide burden of $2.16 million, assuming 60 site-specific licensing examinations per year, and was roughly the same as the cost would be if the NRC were to prepare the examinations relying equally on in-house and contractor staff efforts. The NRC staff had no basis for estimating the cost savings that might be gained if facility licensees availed themselves of the aforementioned efficiencies, but a 10 percent reduction in the burden would result in a small industrywide savings of about $220,000 per year.
The proposed rulemaking that was published in the Federal Register (62 FR 42426) on August 7,1997, specifically requested public comments on the NRC staff's conclusion in the proposed regulatory analysis that it would eventually be less costly for each facility licenseo to prepare its own initial operator examinations rather than to have an NRC contractor prepare the examinations for all licensed operators with the costs reimbursed by licensee fees, The proposed rulemaking also invited public comments on interim Revision 8 of NUREG-1021, which defines the NRC's examination criteria and expectations. Seven respondents directly or indirectly addressed the resource question j
in their comment letters, and seven respondents submitted specific comments and recommendations related to NUREG-1021.
The Nuclear Energy Institute (NEl) and one utility stated that the revised examination criteria in interim Revision 8 of NUREG-1021 have increased the level of effort and will result in higher licensing fees regardless of who prepares the examinations. However, NEl and another utility agreed that comparing the cost of utility-prepared examinations to those prepared by the NRC is difficult. They concluded that it should be less costly for utilities to prepare the examinations than to have the NRC prepare them, provided the preparers are using exactly the same criteria. NEl also stated that the relative cost i
of the two examination processes should not be the only factor in deciding whether to proceed with the rulemaking. NEl indicated that preparing higher-cognitive level questicns requires detailed plant knowledge better known to licensees and that the revised process will allow NRC staff to concentrate on directly evaluating each applicant without relying on third-party observers.
Two NRC examiners, one contract examiner, and a utility employee asserted that the facility licensees' cost has increased under the revised examination process. They cited various reasons for the increased cost, including training personnel to write the examinations and then restricting them from training the applicants and upgrading equipment to maintain examination security. The NRC examiners based their opinions on informal feedback from facility training personnel. One examiner indicated that it was taking facility licensees an average of 700 hours0.0081 days <br />0.194 hours <br />0.00116 weeks <br />2.6635e-4 months <br /> to prepare each examination. The 5
1
p i
l l
l utility employee stated that the rule will simply transfer the cost of contractors from the l
NRC to the utilities.
j l
The NRC staff acknowledges that the revised examination and administrative criteria in l
Interim Revision 8 of NUREG-1021 have probably caused the cost of the examinations l
to be somewhat higher than it would have been if facility licensees had been allowed to prepare the examinations using the same criteria that applied to the NRC and its contractors before starting the pilot program. The staff has reevaluated the revised I
criteria in light of the public comments and lessons leamed during the pilot examination program and concluded that the criteria remain necessary to mitigate the vulnerabilities (e.g., quality, security, and conflict of interest) inherent in the revised examination process and to facilitate the NRC staff's review of the proposed examinations. The staff has incorporated several minor changes (e.g., allowing instructors to prepare written examination questions on subjects they did not teach, without regard to the amount of time they spent training the applicants) and clarifications in the final version of Revision 8 of NUREG-1021. However, the changes are not expected to significantly affect the i
cost of preparing an examination.
t Despite the revised examination and administrative criteria, many of the facility licensees that participated in the pilot program demonstrated that it is possible to prepare an acceptable quality examination at the same or lower cost than the NRC or its j
contractors could prepare a comparable examination. However, approximately 40 percent of proposed pilot examinations have not met NRC standards for quality and have required significant changes. The fact that the lower quality examinations take longer to review and require significant rework by the facility licensees has driven up their total cost and caused the staff to question the validity of its original expectation that i
the industry may realize additional cost reductions as it gains experience with the NRC's examination requirements.
As noted earlier, the proposed regulatory analysis assumed that the training personnel l
at power reactor facilities already have the basic expertise necessary to develop test items for the initial licensing examination. During the second half of the 1980s, the industry increased its emphasis in the training area, and all power reactor licensees established formal training programs that were based on a systems approach to training (SAT) and were accredited by the National Academy for Nuclear Training. Moreover, pursuant to 10 CFR 50.120 and 55.4, SAT-based training programs must evaluate the trainee's mestery of training objectives, and NRC inspections of licensee requalification programs for licensed operators confirmed that training staffs generally possessed the skills needed to evaluate the trainees' knowledge.
However, the public comments on the proposed rule, comments from NRC examiners l
involved with the pilot examinations, and the number of significant changes required on many of the pilot examinations have raised concems about the capability of some of the facilities to write the licensing examinations. The NRC staff realizes that the restrictions l'
on the use of instructors to prepare the licensing examinations may have prompted some facility licensees having smaller training staffs to use less-qualified personnel to prepare the examinations. And although the NRC continues to believe that personnel l
6
restrictions are necessary to address the potential for conflicts of interest in the -
examination development process, it has reconsidered the scope of the restrictions in interim Revision 8 of NUREG-1021 in an effort to reduce the potential burden on facility licensees. Consequently, final Revision 8 will allow facility instructors to prepare the
. written examination questions without regard to the amount of time they spent training the license applicants. However, final Revision 8 will still preclude instructors from preparing questions related to those topics on which they provided instruction and from instructing the applicants once they begin working on the licensing examination (as was the case in interim Revision 8).
i lt is unclear to what extent the examination quality has been affected by the training and qualification of the examination writers, but the NRC staff has asked the industry to address this issue and to determine the need for additional standards for training on testing and measurement principles and examination development techniques. The cost of such an initiative is uncertain at this time, but the benefits would be realized throughout the facility licensees' training programs and not just in the initial operator licensing program.
With legard to the additional security costs cited in the public comments, it is important to note that although the NRC has stressed the seriousness of maintaining examination security, the staff has not required that facility licensees invest in additional physical security systems. The NRC's expectations regarding the facility licensees' responsibility for maintaining examination security were outlined in draft Revision 8 of NUREG-1021, which was issued for public comment in February 1996, and subsequently clarified in interim Revision 8 of NUREG-1021, which was issued for use in February 1997. The security incidents that occurred during the pilot program and the public comments on the proposed rule prompted the NRC to include additional security guidelines in final Revision 8 of NUREG-1021; however, the guidelines are instructional in nature and do not require any security system improvements. The security incidents that occurred during the pilot program has also prompted the NRC to conclude that applicants, licensees, and facility licensees may not understand the provisions of $55.49, " Integrity of Examinations and Tests." Therefore, the NRC has determined that it would be beneficial to amend $55.49 in the final rule to clearly state the expectation that facility licensees establish and maintain procedures to control examination security and integrity. The NRC staff believes that most facility licensees have already established examination security policies and practices, so the additional cost of documenting those policies and practices in formal procedures should not be si nificant. Establishing 0
formal security procedures should raise facility employees' awareness of this important issue and could avert future security incidents that might lequire examinations (or portions of the examinations) to be delayed and replaced at significant expense to the affected facility licensees.
As noted in the proposed regulatory analysis, the revised examination process will eliminate the need for facility licensees to duplicate and ship multiple sets of reference materials because NRC contract examiners (who would have required a separate copy of the reference material) will not be used to prepare the examinations. Feedback from the industry in response to the NRC staffs solicitation of public comments on the draft
]
7 i
i w
e revision of NUREG-1021 (refer to 01 FR 6869, of February 22,1996) indicated that i
facility licensees had been spending an additional 80 to 160 hours0.00185 days <br />0.0444 hours <br />2.645503e-4 weeks <br />6.088e-5 months <br /> to prepare and ship the reference materials under the existing examination process. Under the revised process, the facility licensees will generally submit only materials that are needed for the NRC chief examiner to verify the accuracy of the examination questions. This is a reduction of resources, but it has not beer' quantified in this analysis, l
in summary, before beginning the pilot program, the NRC relied on its own staff and contractors to write the site-specific operator licensing examinations, and the cost was billed to the facility licensees in accordance with 10 CFR 170.12 (i). To appropriately prepare and administer such examinations, the NRC staff and contractors had to learn i
the details of the operation of each specific plant. In effect, this necessitated that, to prepare the examinations, the examining staff duplicated technical expertise already present at each site. This rule will eliminate such inefficiency by placing the responsibility for preparing the examinations upon each power reactor facility licensee.
I Efficiency should be gained if the NRC focuses its efforts on maintaining the appropriate scope, depth, and quality of the examinations, leaving the preparation of the detailed examination materials to the facility licensees.
The NRC expects that the initial period of inefficiency will continue until facility licensees learn the process for preparing the examinations, expand and improve their examination question banks, and upgrade the capabilities of their training staffs, as necessary.
Experience during the pilot examination program has demonstrated that the quality of the examinations and the efficiency of the process will be slow to improve.' It may take considerable time and effort to develop a bank of written examination questions appropriate for use on the initial licensing examination, but such a bank should improve the quality of future examinations and should be more efficient. The NRC still expects that with experience and training most facility licensees will be able to prepare acceptable quality examinations that require less NRC review resources and provide larger potential savings to the licensees. Those facility licensees that submit acceptable quality examinations are likely to save resources despite the additional administrative criteria that the NRC considers necessary under the revised examination process.
However, those facility licensees that submit examinations that require many changes because they do not meet NRC quality standards, will likely end up with a more costly examination than if it had been prepared by the NRC.
Qoerator License Anolicants To the extent possible, the format, content, quality, and level of difficulty of the examinations should remain unchanged, thereby minimizing the impact of the rule change on the operator license applicants. NRC examiners will continue to review, to direct revisions where necessary, and approve every written examination and operating test before it is administered. The NRC acknowledges that some of the facility-prepared examinations will be more difficu". than others and that some of the variation can likely be traced to individual differences among the NRC examiners who review and approve the examinations. Such was also the case when the NRC or its contractors wrote the examinations. However, the NRC believes that the variation in the level of difficulty can 8
1L
be maintained within acceptable limits and would be teven greater if NRC examiners do not take action to ensure that the submitted examinations are consistent with NRC standards.
The average reactor operator (RO) and senior operator (SRO) passing rates on both the
]
written and operating portions of the 68 facility-prepared examinations administered i
through the end of FY 1997 were only slightly lower than the corresponding passing rates on the NRC-prepared examinations administered during FY 1995, the last year in which all of the examinations were prepared by the NRC or its contractors. However, the average passing rates tend to fluctuate, and the pilot examination results were comparable to the range of passing rates for FYs 1990 through 1994. The average passing rates for ROs on the facility-prepared operating tests during FYs 1996 and 1997 were consistent with the passing rates on the tests prepared by the NRC during those i
same years. Although the passing rate on the facility-prepared SRO operating tests during FY 1997 was slightly lower than in FY 1995, it was the same as the passing rate on the NRC-prepared operating tests during FY 1996, i
i The NRC staff believes that the lower passing rate on the facility-prepared written i
examinations for reactor operators and senior operators may be a reflection of the NRC 1
examiners' increased concentration on the psychometric quality (e.g., the cognitive level at which the questions are written and the plausibility of the distractors or wrong answer choices)in addition to the technical quality of the examinations. Although the NRC did not intend for the level of difficulty or the failure rate of the examinations to increase, it is l
possibie that the NRC examiners' efforts to achieve NRC standards regarding the l
cognitive level of the questions and the plausibility of the distractors have affected the l
discrimination validity of the examinations. Consequently, those marginal applicants l
who'may have passed an examination on which two of the distractors could often be elimhated as implausible are now having more difficulty. However, considering the histor: cal fluctuation in the average examination passing rates and the other factors (e.g., training program quality and screening of applicants by facility licensees) that l
could be msponsible for some or all of the observed decline, the staff has concluded that any increase in the level of difficulty is not significant.
l If the NRC decides to prepare the examination in lieu of accepting an examination l
prepared by the facility licensee, the NRC examiners will use the same procedures and guidance (i.e., NUREG-1021) that the facilRy licensees would una to prepare the i
examinations.
NRC Staff l
When the staff developed the pilot examination process, it estimated that it would take approximately 370 examiner-hours to review and otherwise prepare for, administer, grade, and document an average operator licensing examination. Although a number of i
the proposed examinations consumed more NRC resources than expected to achieve NRC's standards, the resource burden was generally offset by other examinations that required less effort to review and revise. Therefore, the NRC staff continues to believe that the use of contractors in the operator licensing program (except for the generic 9
fundamentals examination) can be eliminated and that the revised initial operator licensing and requalification inspection programs can be implemented with the existing NRC staff. Before initiating the pilot examination and transition process at the beginning of FY 1996, the NRC spent approximately $3 million per year for contractor assistance to prepare and administer initial examinations and assist with requalification inspections.
D' ring FY 1996, when approximately 40 percent (27/68) of the examinations were u
voluntarily prepared by facility licensees, the NRC spent approximately $1.6 million for contractor assistance in those areas, in FY 1997, facility participation increased to include approximately 75 percent (41/55) of the examinations, and the NRC's spending on contractor assistance for the licensing examinations and requalification inspections decreased to approximately $0.5 million (approximately 3 full-time equivalents (FTEs)).
During FY 1997, the NRC expended approximately 16.8 staff FTEs to conduct eli the site-specific initial operator licensing examinations, plus 3.8 staff FTEs for generic operator licensing activities. The licensed operator requalification inspection program consumed approximately 4.5 additional staff FTEs (plus a minimal level of contractor support), bringing the total resources used for the operator licensing program in FY 1997 to approximately 28.1 FTEs. This actual resource burden is generally consistent with the budget estimates prepared before beginnines the pilot examination program and supports the staffs conclusion that the revised examination process is an efficient attemative to the traditional NRC-prepared examinations. The NRC resource burden anould decrease further if and when the rule is implemented (which would cause the number of NRC-prepared examinations to decrease) and the quality of the facility-prepared examinations improves.
However, experience during the pilot program has shown that the quality of the facility-prepared examinations can vary widely, making it difficult to predict the amount of time necessary for the NRC to review and revise any particular examination so as to meet NRC standards. Although the staff had expected the quality of the examinations to improve over time, some of the proposed examinations continue to require more changes than anticipated. The added examiner workload could increase the possibility
)
of lower examiradan quality, raise the NRC's cost, and affect the staffs ability to satisfy the facility licensees' needs for licensing examinations. As noted in SECY-97-079, the Office of Nuclear Reactor Regulation (NRR) has issued a memorandum to the regional administrators emphasizing the importance of (1) assigning adequate resources to carry out the operator licensing task, (2) completing a thorough review of every facility-prepared examination, and (3) not administering any examination that does not meet NRC standards for quality and level of difficulty.
I 5.
Decision Rationale The proposed amendments to 10 CFR Part 55 will require all power teactor facility licensees to prepare the entire initial operator licensing examinations and to proctor and grade the written portion of the examinations. The qualitative assessment of costs and l
benefits discussed above, leads the NRC to conclude that the overall impact of the rulemaking would not significantly increase licensee costs and could eventually result in a savings to licensees as they become more familiar with the NRC examination guidelines. Any improvements in efficiency would likely be due, in part, to the facility 10 l
t
l E
employees' better understanding of the plant design and operating characteristics and their ready access to the reference materials required to prepare and validate the examinations. The apparent need of some facilities to improve their examination development capabilities in order to achieve the expected standards for examination quality may delay the realization of cost savings for some facility licensees and may limit the overall industry cost benefit. Facility licensees that do not prepare examinations that meet the NRC's quality standards will likely derive no cost benefit from this action because of the additional effort required to review and rewrite the examinations.
i The voluntary pilot program has demonstrated that the revised examination prccess is effer'ive as well as efficient. The pilot examinations, revised as directed by the NRC, were comparable in terms of their quality to examinations prepared by the NRC and its contract examiners. The overall passing rate and average written examination grades on the 68 pilot examinations administered during FYs 1996 and 1997 were the same as, or slightly lower than, those on the power reactor licensing examinations during FY 1995, when all the examinations were prepared by the NRC or its contractors.
Therefore, the safe operation of the facilities was in no way compromised.
l 1
The NRC staff believes that its ability to focus on operator performance and its core of i
experience may improve if the pilot process is implemented on an industrywide basis because every applicant will be directly observed by an NRC employee. Before beginning the transition process, contract examiners administered about half of the operating tests and collected the observations that formed the basis for the NRC's licensing actions. The revised process will enable the NRC examiners to accrue more experience in a shorter period of time and may improve the consistency of the operating l
test evaluations and the licensing decisions. Although the total number of procedures l
reviewed in the process of examination development may decrease under the revised method, NRC examiners are still expected to review and identify discrepancies in the l
procedures that will be exercised during the plant walk-through test and the dynamic simulator scenarios.
Attemative 3 was selected as the preferred attemative because it has the potential to save the facility licensees money as they become proficient in preparing the i
examinations, has negligible impact upon operator license applicants, provides a substantial cost savings to the NRC, and has the potential for an improvement in the staff's experience and ability to focus on operator performance.
6.
Implementation Schedule The rule will be implemented 180 days after the date on which it is published in the l
FederalRegister. No effect on other schedules is anticipated.
I 11
I Congressional Letters i
7 c
l u
The Honorable Dan Schaefer, Chairman:
Subcommittee on Energy and Power
. Committee on Commerce l
Unded States House of Representatives L
Wesh;ngton, DC 20515
Dear Mr. Chairman:
1 The NRC has sont the enclosed amendments to the Commission's rules in 10 CFR Part 55 to i
the Office of the Federal Register for publication. ' This rulemaking will require facility licensees to prepare the entire initial examination for reactor operators and senior reactor operators and to proctor and grade the written portion of the examination.1 Facility licensees will be required to -
submit the examinations and tests to the NRC for review and approval. This rulemaking preserves the NRC's authority to prepare the examinations and tests, as necessary, and enables the NRC to eliminate the use of contractors for the operator licensing program.
Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform
- conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55, " Operators' Licenses," to pass an examination satisfying the basic content requirements that are also specified in the regulation. Although the
' AEA is not specific as to who will prepare and administer the examination, the NRC has traditionally performed these functions through its staff or contract examiners. -This final rule requires facility licensees to prepare and administer the site-specific written examinations in recognition of the improvements in industry training programs,' to make the operator licensing program more efficient, and to realize budgetary savings for the Federal Govemment.
?
Historically, the NRC has spent approximately $3 million annually on contractor support for operator licensing examinations and requalification inspections.
The NRC will continue to (1) prepare and administer the generic fundamentals examination to operator license applicants, (2) thoroughly review every site-specific examination, direct changes as required to achieve NRC standards, and approve the examination before it is given, (3) administer and grade the operating test portion of the site specific examination, meluding a
-. control room simulator examination and a one-on-one demonstration of specific operating tasks, and (4) prepare and administer at least one site specific examination annually in each of
{
the NRC's four regions.'
i Sincerely, 1
Dennis K. Rathbun, Director i
Office of Congressional Affairs
Enclosure:
Federal Register Notice I
cc: Representative Ralph Hall I
I i
I-1
]
m
..LTha Han:r:bla D:n Schxfir, Chrirman J
Subcommittee on Entrgy cnd Power e
- Committee on Commerce.
United States House of Representatives Washington, DC 20515
Dear Mr. Chairman:
The NRC has sent the enclosed amendments to the Commission's rules in 10 CFR Part 55 to the Office of the Federal Register for publication. This rulemaking will require facility licensees
. to prepare the entire initial examination for reactor operators and senior reactor operators and to proctor and grade the written portion of the examination. Facility licensees will be required to submit the examinations and tests to the NRC for review and approval. This rulemaking preserves the NRC's authority to prepare the examinations and tests, as necessary, and enables the NRC to eliminate the use of contractors for the operator licensing program.
Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to
)
determine the qualifications of individuals applying for an operator license, to prescribe uniform
)
conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55, " Operators' Licenses," to pass an examination satisfying the basic content requirements that are also specified in the regulation. Although the AEA is not specific as to who will prepare and administer the examination, the NRC has traditionally performed these functions through its staff or contract examiners. This final rule requires facility licensees to prepare and administer the site-specific written examinations in -
- recognition of the improvements in industry training programs, to make the operator licensing program more efficient, and to realize budgetary savings for the Federal Govemment.
Historically, the NRC has spent approximately $3 million annually on contractor support for operator licensing examinations and requalification inspections.
The NRC will continue to (1) prepare and administer the generic fundamentals examination to operator license applicants, (2) thoroughly review every site-specific examination, direct i
changes as required to achieve NRC standards, and approve the examination before it is given,
)
(3) administer and grade the operating test portion of the site-specific examination, including a control room simulator examination and a one-on-one demonstration of specific operating tasks, and (4) prepare and administer at least one site-specific examination annually in each of I
the NRC's four regions.
Sincerely, Dennis K. Rathbun, Director Office of Congressional Affairs
Enclosure:
i Federal Register Notice
-l l
cc: Representative Ralph Hall Distnbution Subj/ central, EDO R/F DOCUMENT NAME:[G:\\GUENTHER\\PT55FNL.LTR)
To receive a copy of this document. Iruhoose in the bor *C" = Copy udthout ettechment/ enclosure "E"e Copy udth attachment / enclosure "N" - No copo OFFICE OCA NAME DRathbun j
DATE-
/ /98
/ /98
/ /98-
/ /98
/ /98
]
i
- The Honorable James M. Inhofe, Chairman Subcommittee on Clean Air, Wetlands, Private Property and Nuclear Safety Committee on Environment and Public Works United States Senate Washington, DC 20510 -
Dear Mr. Chairman:
The NRC has sent the enclosed amendments to the Commission's rules in 10 CFR Part 55 to the Office of the Federal Register for publication.' This rulemaking will require facility licensees to prepare the entire initial examination for reactor operators and senior reactor operators and to proctor and grade the written portion of the examination. ' Facility licensees will be required to submit the examinations and tests to the NRC for review and approval. This rulemaking preserves the NRC's authority to prepare the examinations and tests, as necessary, and enables the NRC to eliminate the use of contractors for the operator licensing program.
Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as apprcpriate. Operator license applicants are required by 10 CFR Part 55, " Operators' Licenses," to pass an examination
. satisfying the basic content requirements that are also specified in the regulation. Although the
' AEA is not specific as to who will prepare and administer the examination, the NRC has traditionally performed these functions through its staff or contract examiners. This final rule requires facility licensees to prepare and administer the site-specific written examinations in recognition of the improvements in industry training programs, to make the operator licensing
' program more efficient, and to realize budgetary savings for the Federal Govemment.
Historically, the NRC has spent approximately $3 million annually on contractor support for operator licensing examinations and requalification inspections.
The NRC will continue to (1) prepare and administer the generic fundamentals examination to operator license applicants, (2) thoroughly review every site-specific examination, direct changes as required to achieve NRC standards, and approve the examination before it is given, (3) administer and grade the operating test portion of the site-specific examination, including a control room simulator examination and a one-on-one demonstration of specific operating tasks, and (4) prepare and administer at least one site specific examination annually in each of the NRC's four regions.
Sincerely, Dennis K. Rathbun, Director Office of Congressional Affairs
Enclosure:
Federal Register Notice cc:. Senator Bob Graham E
Tha H:norabia J mes M. Inhofa, Chairman Subcommittee on CI:cn Air, Witt:nds, Priv:ta Property and Nuclear Safety Committee on Environment and Public Works United States Senate Washington, DC 20510
Dear Mr. Chairman:
The NRC has sent the enclosed amendments to the Commission's rules in 10 CFR Part 55 to the Office of the Federal Register for publication. This rulemaking will require facility licensees to prepare the entire initial examination for reactor operators and senior reactor operators and to proctor and grade the written portion of the examination. Facility licensees will be required to submit the examinations and tests to the NRC for review and approval. This rulemaking preserves the NRC's authority to prepare the examinations and tests, as necessary, and enables the NRC to eliminate the use of contractors for the operator licensing program.
Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55, " Operators' Licenses," to pass an examination satisfying the basic content requirements that are also specified in the regulation. Although the AEA is not specific as to who will prepare and administer the examination, the NRC has traditionally performed these functions through its staff or contract examiners. This final rule requires facility licensees to prepare and administer the site-specific written examinations in recognition of the improvements in industry training programs, to make the operator licensing program more efficient, and to realize budgetary savings for the Federal Govemment.
Historically, the NRC has spent approximately $3 million annually on contractor support for operator licensing examinations and requalification inspections.
The NRC will continue to (1) prepare and administer the generic fundamentals examination to operator license applicants, (2) thoroughly review every site-specific examination, direct changes as required to achieve NRC standards, and approve the examination before it is given, (3) administer and grade the operating test portion of the site-specific examination, including a control room simulator examination and a one-on-one demonstration of specific operating tasks, and (4) prepare and administer at least one site-specific examination annually in each of the NRC's four regions.
Sincerely, Dennis K. Rathbun, Director Office of Congressional Affairs
Enclosure:
Federal Register Notice cc: Senator Bob Graham p
Distribution: Subj/ central, EDO R/F 5
DOCUMENT NAME:[G:\\GUENTHER\\PT55FNL.LTR) 7t receive e copy of this document. Indicato in the bor: "C"a Copy udthout ollachmenWenolosure T e Copy udth allachmenuanolosure "N" a No copo OFFICE -
OCA NAME DRathbun DATE
/ /98
/ /98
/ /98
/ /98
/ /98
I.
Public Notice l
l t
l i
I 1
I 1
e i
j i
COMMISSION PUBLISHES FINAL RULE FOR UTILITY ROLE IN INITIAL REACTOR OPERATOR LICENSE EXAMINATIONS The Nuclear Regulatory Commission has amended its regulations to require that all nuclear power plant licensees prepare, proctor, ar d grade their initial reactor operator license written exarr,.:stions and prepare their operating tests - all subject to NRC approval.
\\
?
Reactor operator applicants seeking a license to manipulate the controls of a nuclear 3
f power plant must pass both a comprehensive, multiple-choice written test and a practical, I
b hands-on examination. The generic fundamentals examination, a separate written test that each applicant must pass to be considered for the final, site-specific license examination, will i
continue to be written and administered by the NRC.
Historically, either NRC staff examiners or NRC contractors have prepared and 1
administered all operator license tests. In April 1995, the Commission approved a staff proposal to begin a pilot program in which nuclear power plant licensees would prepare the j
i tests under NRC oversight. The Commission took this action in recognition of the improvements in industry training programs, to make the operator licensing program more
)
efficient, and to realize budgetary savings.
All examinations and tests written by utilities and tailored to their specific plants are subject to review, modification, and approval by NRC examiners before the examinations and tests are given. If the submitted examination or test fails to meet NRO's quality standards, the NRC has the option of preparing the test or examination in lieu of accepting or modifying one prepared by a utility. The NRC will contir,ue to administer and grade the hands-on portion of the test, including a control room simulator examination and a one-on-one demonstration of specific operating tasks.
1 l
a
f:
NUREG-1021, " Operator Licensing Examination Standards for Power Reactors," has
.been revised to implement the new process; it provides detailed NRC guidance on matters such as the appropriate level of knowledge and difficulty, maintenance of examination security, and restrictions on test preparation by those significantly involved in training the license applicants.
The NRC staff will prepare at least one examination annually in each of the agency's four regions to ensure that the NRC staff retains proficiency in examination writing and to serve
.j as a quality check on the process.
l l-The NRC staff solicited volunteers for a pilot program in August 1995, and launched the l
program 2 months later. Between October 1995 and the end of September 1997, the staff reviewed and approved 68 operator licensing examinations prepared by utilities in accordance 1
l with published NRC guidance. The results of the examinations have been consistent with the L
results on examinations prepared by the NRC.
L Historically, the NRC has spent approximately $3 million annually on contractor support V
i
)
for the operator licensing program. Licensing of operators for research and test reactors who
. will continue to be examined by the NRC and will be unaffected by this rule. Also unchanged is the present system whereby utilities prepare and administer requalification examinations to licensed operators as part of an NRC-approved training program.
Full details of the final rule are included in a notice published in the i
edition of the FederalRegister.
\\
ll' 2
L j