ML12249A237

Fairness Review (Fje 31AUG12)

ML12249A237
Person / Time
Site:	Vogtle
Issue date:	09/04/2012
From:	Frank Ehrhardt Division Reactor Projects II
To:	Mchale J Division of Inspection and Regional Support
Linda K. Gruhler 404-997-4633
References
Download: ML12249A237 (11)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION REGION II 245 PEACHTREE CENTER AVENUE NE, SUITE 1200 ATLANTA, GEORGIA 30303-1257 September 4, 2012 MEMORANDUM TO: John McHale, Branch Chief Operator Licensing Division of Inspection and Regional Support FROM: Frank Ehrhardt, Branch Chief

/RA/ Division of Reactor Projects

SUBJECT:

2012 VOGTLE INITIAL LICENSING EXAMINATION

Enclosed is the independent review, conducted by a Region II Manager, of contentions

raised by applicant C. Smith concerning the 2012 Vogtle initial licensing examination.

Enclosure:

As stated CONTACT: Frank Ehrhardt, RII, DRP 404-997-4611

ML12249A237 X SUNSI REVIEW COMPLETE X FORM 665 ATTACHED OFFICE RII:DRP/Br2 SIGNATURE FJE NAME F. Ehrhardt DATE 9/ 4 /2012 E-MAIL COPY YES NO Enclosure

SUMMARY

An applicant failed the written portion of the Vogtle 2011 initial license examination but passed the operating test. The facility did not request a waiver of the operating test and the applicant was subsequently reexamined in 2012. The applicant failed the operating test portion of the Vogtle 2012 examination but passed the written examination. After receiving a license denial letter, the applicant contacted the Director, DIRS, contending bias on the part of the examiners. An independent Region II manager was unable to substantiate the applicant's contentions.

BACKGROUND The Senior Reactor Operator - Instant (SRO-I) applicant, C. Smith, passed the operating test portion of an initial license examination administered in March, 2011 at Vogtle, but failed the written test.

The licensee did not request a waiver of the operating test on behalf of the applicant when they submitted the applicant's Form 398 in preparation for the 2012 initial license examination.

Examiners administered the operating test portion of an initial license examination to the applicant in March and April of 2012. The applicant failed the operating test and was notified of the test results in May 2012. In June 2012, the Director, Division of Inspection and Regional Support, received correspondence from the applicant regarding regional examiners' conduct of the examination. In this correspondence, the applicant raised contentions concerning 1) the manner in which examiners conducting the 2012 examinations applied the guidelines of NUREG-1021 concerning operating test waivers and 2) examiner bias. Specific issues raised by the applicant in her correspondence are paraphrased and listed below:

• The applicant does not understand why she had to retake the operating test for the 2012 examination when she passed the 2011 operating test. (Re-take of 2012 operating test)

• The applicant contends that the NRC examiner s on the 2012 examination team strongly discouraged the licensee from submitting a waiver of the 2012 operating test on her behalf. (Waiver discussions between examiners and licensee)

• The applicant contends that the examiner s on the 2012 examination team were biased and held her to a higher standard of performance as compared to other applicants. (Potential examiner bias)

In response to the applicant's contentions, Region II management decided to conduct an independent management review of the applicant's contentions of bias. The management review was performed by Frank Ehrhardt, an independent Region II manager who has prior operator licensing experience as a Chief Examiner for Westinghouse pressurized water reactors. This manager was also from the Division of Reactor Projects, a division separate from

2 the operator licensing (examiner) branch. The manager performed the following activities in conducting the review:

• Interviewed examiners that conducted the 2011 and 2012 Vogtle initial license examinations.

• Obtained and evaluated independent written statements from the examiners that conducted the 2011 and 2012 Vogtle initial license examinations.

• Assessed-mail correspondence between the examiners and the licensee regarding potential waivers for applicants that took the 2012 Vogtle examination.

• Reviewed the operator licensing docket file maintained in Region II for C. Smith. This file includes the applicant's NRC Form 398, "Personal Qualifications Statement: Licensee", Individual Examination Reports (Form ES-301-1) of the applicant's 2011 and 2012 Vogtle initial license examinations, and correspondence between the applicant and the NRC.

• Reviewed Individual Examination Reports of other applicants who took the Vogtle 2012 initial license examination.

Re-take of 2012 operating test The applicant passed the operating test portion, but failed the written portion, of the 2011 initial license examination. NUREG-1021, ES-204, Proc essing Waivers Requested by Reactor Operator and Senior Reactor Operator Applicants, paragraph C.1.a states that an applicant may

request a waiver of a license requirement by checking the appropriate block in Item 4.f on NRC Form 398, "Personal Qualifications Statement: Licensee" and the applicant should also explain the basis for requesting the waiver in Item 17, "Comments." ES-204, paragraph C.2 states that the facility licensee's senior management representative on site must certify the final license application, thereby substantiating the basis for the applicant's waiver request.

The manager reviewed the NRC Form 398 that the licensee submitted on behalf of the applicant in March of 2012. The manager determined that Item 4.f on the applicant's Form 398 was not marked to indicate that a waiver was being requested for an operating test and that no supporting information had been entered for Item 17 to provide justification for a waiver of the operating test. Thus, the reason the applicant was required to retake the operating test in 2012 is because a waiver of the operating test portion of the 2012 examination was not requested in accordance with NUREG-1021.

Waiver discussions between examiners and licensee

ES-204, D.1.a states that "if an applicant failed only one portion of the site-specific initial licensing examination (i.e., either the written examination overall,-..), the region may (emphasis added) waive those examination areas that were passed. NUREG-1021 does not provide a standard for examiners to use in making this decision. Thus, examiners are not required to grant a waiver request and their decision is based on judgment.

3 As described in NUREG-1021, the operating test consists of two portions: 1) a simulator operating test, and 2) a walk-through test comprised of administrative, (simulated) control room, and (simulated) in-plant job performance measures (JPMs). Per NUREG-1021, the walk-through test for an SRO-I applicant consists of 5 administrative, 7 control room, and 3 in-plant JPMs. As described in NUREG-1021, an applicant will receiv e a competency grade of 3.00 if he or she makes no errors related to the Rating Factors within the Competency. In order to pass the simulator portion of the operating test, an applicant must achieve the following minimum Competency Grades:

In accordance with ES-303, Section D.2, the applicant's performance will be graded as satisfactory if:

• The grade for all competencies is greater than 1.8, or

• The individual grades for all competencies other than Communications are 2.0 or greater and the grade for Communications is less than or equal to 1.8 but greater than 1.0 The applicant's performance will be graded as unsatisfactory if the grade for Communications is 1.0 or the grade for any other competency is 1.8 or less.

The examiner of record graded the applicant on the simulator portion of the 2011 operating test as summarized in the following table.

Competency Competency Grade 1. Interpretation/Diagnosis 2.50 2. Procedures 2.20 3. Control Board Operations 2.33 4. Communications 2.00 5. Directing Operations 2.80 6. Technical Specifications 3.00 The applicant received an overall grade of satisfactory on the walk-through portion of the 2011 operating test. The applicant received satisfacto ry grades on all administrative, control room, and in plant JPMs. In order to receive a satisfactory grade on a JPM, the applicant must successfully complete all critical steps required to complete the task, as indicated by the documented performance measures contained in t he JPM. Per NUREG-1021, ES- 303, D.3.c, examiners are required to document deficiencies that do not contribute to an operating test failure, for example, weaknesses or errors displayed by the applicant in performing non-critical steps of JPMs. Examiners documented weaknesses in the applicant's performance on 6 of the

15 JPMs.

4 The manager determined that the Chief Examiner and Examiner of Record for the applicant's 2011 examination assessed the applicants' performance as marginal or borderline and informed the Operator Licensing Branch Chief, shortly after the 2011 examination, that the applicant was not a good candidate to receive a waiver of any future operating test. The Chief Examiner stated that, when assessing whether or not to recommend a waiver of the operating test, he weighed the applicant's performance on the simulator portion of the operating test more heavily than performance on the walkthrough (JPM) portion. His stated rational for doing so was that the simulator test most closely mimicked the changing, complex situations that the licensed operator would face when acting to protect public health and safety. This reasoning is consistent with ES-301, B.3, "Simulator Operating Test, which states, "This is the most performance-based aspect of the operating test and is used to evaluate the applicant's ability to safely operate the plant's systems under dynamic, integrated conditions." Additionally, although the walk-through and simulator portions of the operating test are grades separately, the Chief Examiner stated he noted similarities in the types of errors made by the applicant when performing both portions of the operating test.

To gauge whether the Chief Examiner's assessment was consistent with Region II norms, the manager reviewed data drawn from the Operator Licensing Tracking System database. The manager did not find any instances within Region II of an operating test waiver being requested and denied within the last three years (1/1/09 - 6/30/12). The manager did find 33 instances over the last three years when the region granted waivers of the operating test (11 SRO and 22 RO). The manager compared the simulator test competency scores for those individuals who were granted waivers to those of the applicant and determined that the applicant's simulator test competency scores were, overall, significantly lower than any of the scores for those individuals who were granted waivers. The manager also performed an independent review of comments on the applicant's 2011 walk-through (JPM) test. The manager determined that, while the applicant's performance on each JPM was graded as satisfactory, the weaknesses and errors demonstrated by the applicant supported the Chief Examiner's overall impression that the applicant's performance was borderline with respect to her competency as a potentially licensed operator.

The manager concluded that the Chief Examiner's assessment of the applicants' performance with respect to suitability for a waiver was consistent with the guidelines contained in NUREG-1021 as well as past practice within Region II.

The manager determined that the licensee had asked the Chief Examiner for the 2012 examination if Region II would approve an operating test waiver, if one was submitted on behalf of the applicant. The licensee made this inquiry, via e-mail, before submitting the applicant's NRC Form 398 in March of 2012. The manager determined that the Chief Examiner, after consultation with the Operator Licensing Branch Chief, informed the licensee that Region II "would likely deny a waiver request on behalf of the applicant for the operating test portion of the examination." The Chief Examiner stated that, because he was not involved in the 2011 examination, he simply followed the recommendation of the 2011 Chief Examiner and Examiner of Record when responding to the licensee's question. The Chief Examiner's motivation for 5 telling the licensee that the applicant would likely be denied a waiver was so that the licensee could arrange for any necessary remedial training as early as possible if they decided to include the applicant in a future license class.

The Chief Examiner for the 2012 examination stated that the licensee had submitted a preliminary (draft) Form 398 for the applicant, after the e-mail conversations described above, and before submitting the final Form 398 on behalf of the applicant. Additionally, the licensee had checked Item 4.f requesting a waiver of the operating test for the applicant. The Chief Examiner stated that he contacted the licensee and asked if they had changed their mind and intended to submit a request for a waiver of the operating test on behalf of the applicant. The licensee representative responded that they had checked Item 4.f. in error and did not intend to request a waiver of the operating test when they submitted the final Form 398.

The manager determined that none of the examiners conducting the 2011 or 2012 examinations told the licensee they could not or should not submit a waiver request on behalf of the applicant and never communicated directly with the applicant regarding a potential waiver. Additionally, the manager determined that the licensee did not contact the Region II Operator Licensing Branch Chief in order to clarify the reasoning behind, or informally appeal, the Chief Examiners assessment regarding the likely outcome of a request for a waiver.

The manager concluded that the communication between the Chief Examiner and the licensee regarding a potential waiver request was consistent with the NRC Value of openness and Principles of Good Regulation of efficiency.

Potential examiner bias

The primary objective of the NRC's examinations is to determine whether or not the examinees have sufficiently mastered the knowledge, skills, abilities, and other attributes to perform the job of an RO or an SRO at a specific plant. The NRC's examinations are not intended to distinguish among levels of competency or to identify the most qualified individuals, but to make reliable and valid distinctions at the minimum level of competency that the agency has selected in the interests of public protection. Thus, the NRC's initial licensing examinations are criterion-referenced test rather than norm-referenced tests.

As described in NUREG-1021, Appendix A, Overview of Generic Examination Concepts, acceptable levels of examination consistency, uniformity, and fairness would be impossible to achieve without quantitative and qualitative acceptance criteria. The examination material used by examiners during licensing examinations is designed to conform to the examination standards outlined in NUREG-1021. The examination standards include quantitative and qualitative criteria necessary for a well-balanced and consistent examination. Conformance to these standards decreases the likelihood that inconsistencies among examination content and administration will jeopardize the val idity of the NRC's licensing decision.

6 Thus, initial examinations are highly pre-scripted, with critical tasks, steps, and key performance elements explicitly determined and documented in advance in consultation with facility training representatives. Therefore, there is little, if any, opportunity to hold any applicant to a higher standard of performance than another.

2012 Exam Administration

The examiners comprising the 2011 Vogtle examination team, their roles, and the portions of the operating test each examiner administered to the applicant are outlined in the table below.

Examiner Role Portions of Test Administered J. Hopkins Examiner of Record (All) Simulator Scenarios (pass - 12 total comments) Seven JPMs (all graded sat, 2 with comments) P. Capehart Chief Examiner Six JPMs (all graded sat, 3 with comments) M. Meeks Examiner Two JPMs (both graded sat, one with comments)

The examiners comprising the 2012 Vogtle examination team, their roles, and the portions of the operating test each examiner administered to the applicant are outlined in the table below.

Examiner Role Portions of Test Administered M. Bates Chief Examiner / Examiner of Record (All) Simulator Scenarios (fail - 18 total comments)

Five JPMs (all graded sat, one with comments) P. Capehart Examiner Two JPMs (both graded sat, no comments) M. Meeks Examiner Eight JPMs (all graded sat, three with comments)

The above tables show that the applicant's examiner of record for the 2012 examination did not participate in the 2011 examination and would have had no opportunity to be biased through direct observation of her performance on the 2011 examination. Put another way, the examiner who evaluated the applicant's performance on the 2012 simulator test as unsatisfactory was the one member of the examination team who had no opportunity to observe the applicant's performance on the 2011 examination.

The tables also show that the Chief Examiner for the 2011 examination, who directly observed her performance on six JPMs during the 2011 examination and assessed the applicant's performance relative to a request for a waiv er, played a relatively small role in examining the applicant in 2012. Finally, the other examiner who participated in the 2012 examination played a small role in examining the applicant in 2011, directly observing the applicant's performance on two JPMs.

The manager also interviewed the Chief Examiner for the 2012 examination, as well as the other examiner who administered the bulk of the applicant's walk-through exam, to determine if they had obtained detailed knowledge of the applicant's performance on the 2011 examination prior to administering the 2012 examination. The manager determined that the Chief Examiner/Examiner of Record for the 2012 examination did not review or read the applicant's 2011 Individual Examination Report. The Chief Examiner stated that it was not necessary for 7 him to review the applicant's 2011 examination results to respond to the licensee's inquiry regarding a potential waiver because he followed the recommendation of the 2011 Chief Examiner. The Chief Examiner also stated that he purposely did not review the applicant's 2011 examination results because he wanted to ensure he remained objective when examining the applicant.

The manager also determined that M. Meeks did not review the applicant's 2011 Individual Examination Report before administering the 2012 examination in order to ensure he remained unbiased. The examiner stated that his expectation regarding the applicant's performance on the 2012 operating test was that she would likely pass because the licensee had recommended her as ready for a license and because she had received additional training and practice on simulator scenarios and JPMs.

In developing a plan to administer the simulator operating test, the Chief Examiner stated that his goals were to minimize the number of scenarios that needed to be developed and administered, minimize the number of scenarios that would require use of a surrogate operator, and, to the extent practicable, ensure that any necessary surrogates were used in the Balance of Plant Position. These goals are consistent with the guidance contained in ES-301. ES-301 provides guidance that permits the examiners to examine an applicant on one additional scenario above the minimum number required if doing so will reduce the use of surrogate operators. When the Chief Examiner applied these guidelines to developing a detailed plan, he determined that the applicant would be evaluated during three simulator scenarios.

The Chief Examiner stated that he assigned applicants to examiners to ensure that an examiner who had no detailed knowledge of the applicant's performance on the 2011 operating test was assigned as her examiner of record for the 2012 examination. He stated he did this specifically to ensure the applicant would receive an unbiased evaluation. This action is consistent with the guidance contained in NUREG-1021, ES-201, D.1.a concerning personnel restrictions for NRC examiners. ES-201, D.1.a states "The regional office shall not assign an examiner who failed an applicant on an operating test to administer any part of that applicant's retake operating test. Because the applicant did not fail the 2011 operating test, it would have been acceptable to assign any one of the 2012 examiners as the applicant's examiner of record. By assigning himself, the Chief Examiner ensured, to the maximum extent possible, that the applicant received an unbiased evaluation.

The manager concluded that the Chief Examiner of the 2012 examination followed the guidance contained in NUREG-1021 when determining the number of scenarios to administer to the applicant and took measures beyond the guidance specified in NUREG-1021 when assigning himself as the applicant's examiner of record.

2012 Operating Test Grading NUREG-1021, ES-303, D.2 states "-..depending on the applicant's lice nse level, and the following generic guidance, evaluate any deficiencies coded for the simulator test to determine a 8 grade for every available rating factor (RF) and co mpetency. Keep in mind that the simulator test is generally graded based on competencies ra ther than consequences; every error that reflects on an operator's competence is considered equal unless it is related to the performance of a critical task.

The manager compared the simulator test comments documented in the applicant's Individual Examination Report to comments documented in the Individual Examination Reports of other applicants that took the Vogtle 2012 operating test. The manager observed evidence, based on numerous examples, that examiners applied a sim ilar threshold for documenting behavior that was considered to constitute a weakness or error across all applicants. In many cases, the same weakness or error exhibited by multiple applicants was categorized in the same rating factor in their Individual Examination Reports. The manager observed differences in describing or categorizing some errors, but the differences can be explained either by differences in the root cause for making the error or because applicants were filling different shift positions (e.g. RO vs. BOP) when making the errors.

NUREG-1021, ES-303, D.1.d states, in part "Whenever possible, attempt to identify the root cause of the applicant's deficiencies and code each deficiency with no more than two different rating factors. However, one significant deficiency may be coded with additional rating factors if the error can be shown, consistent with the criteria in D.3.b, to be relevant to each of the cited rating factors." The examiners administering the 2012 examination stated that in applying this guidance in evaluating all of the applicants taking the 2012 examination they decided to assign each deficiency to only the single rating factor that was most closely related to the root cause of the error, even though the guidance allowed for assigning an additional rating factor under normal circumstances.

Prior to issuing the license denial to the applicant, the Chief Examiner obtained independent reviews of the applicant's Individual Examination Report from two senior examiners and one examiner. The purpose of these reviews was to ensure that the applicant's performance was accurately described and characterized and that the examiner's assessment of the applicant's performance complied with the guidelines contained in NUREG-1021. The two senior examiners previously held SRO licenses and also had significant experience developing, administering, and evaluating operating tests as members of licensee training staffs. The other examiner previously held an SRO license. The Chief Examiner stated that he incorporated the majority of the comments made by the reviewers. The manager determined that none of the independent reviewers raised issues related to ex cessively 'strict' assessments of the applicant's performance or potential bias.

The manager concluded that examiners applied the same threshold for evaluating the applicant's performance, and documenting the applicant's errors, as for other individuals taking the exam. The manager was unable to detect a pattern of bias based on a review of Individual Examination Reports, statements from examiners, or interviews. The manager concluded that the applicant received lower documented scores on the simulator portion of the operating test, 9 in comparison to other individuals taking the test, because the applicant simply made more errors during the simulator scenarios.

The applicant passed all JPMs administered in 2011 as well as all JPMs administered in 2012. However, in an attempt to uncover any potential bias concerning the administration of the walkthrough test, the manager compared the number of comments made regarding the applicant's performance on JPMs by the examiners conducting both the 2011 and 2012 walk-through examinations. This comparison is summarized in the table below.

Total Administered Total Passed Total Comments Capehart/Meeks -

2011JPMs 8 8 4 Capehart/Meeks -

2012 JPMs 10 10 3 Because the applicant received a satisfactory grade on all 2012 JPMs, and the examiners made fewer comments concerning the applicant's performance in 2012, the manager determined that reviewing the 2012 JPM comments would not provide additional information regarding potential bias. The manager concluded that these examiners did not exhibit bias in grading the applicant's 2012 walk-through exam.

Conclusion The manager, after conducting an independent review of the applicant's contentions, reached the following conclusions:

• The applicant did not receive a waiver for the operating test portion of the Vogtle 2012 initial license examination because the facility licensee did not request a waiver on behalf of the applicant.

• The applicant's contention that examiners discouraged the facility licensee from requesting a waiver of the operating test portion of the Vogtle 2012 initial license examination is unsubstantiated.

• The applicant's contention of bias by examiners in administering or evaluating her operating test is unsubstantiated.

Furthermore, the manager determined that the applicant was examined fairly, in a normal fashion, in accordance with the guidelines contained in NUREG-0121.