IR 05000416/2003301
| ML032660087 | |
| Person / Time | |
|---|---|
| Site: | Grand Gulf  |
| Issue date: | 09/18/2003 |
| From: | Gody A Operations Branch IV |
| To: | Forbes J Entergy Operations |
| References | |
| EA-03-116 ER-03-301 | |
| Download: ML032660087 (20) | |
Text
ber 18, 2003
SUBJECT:
GRAND GULF NUCLEAR STATION - NRC EXAMINATION REPORT 05000416/2003-301
Dear Mr. Forbes:
On September 17, 2003, the NRC completed a review of examination development practices at your Grand Gulf Nuclear Station. The enclosed report documents the examination development findings, which were discussed on September 16, 2003, with Jerry Roberts and members of your staff.
The review examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your licenses. Within these areas, the review consisted of selected examination of procedures and representative records, observations of activities, and interviews with personnel. It was determined that a number of examinations developed by your staff did not meet the requirements in 10 CFR 55.40(b)(1). Part 55.40(b)(1) of Title 10 of the Code of Federal Regulations requires licensees to prepare examinations and tests in accordance with NUREG-1021, "Operator Licensing Examination Standards for Power Reactors." The examiners concluded that examinations administered in December 2000, June 2001, and August 2002 were not developed in accordance with NUREG-1021, and that the bias introduced by that improper development method combined with some candidate foreknowledge of questions on their examination affected their equitable and consistent administration.
Nevertheless, we determined that the NRC operator licensing decisions made following the subject examinations were appropriate. This conclusion was based on the following facts:
(1) the resultant examinations, after conservatively removing all bank questions that were provided to the candidates prior to their examination, met the general sampling requirements of 10 CFR 55.41 and 55.43, albeit, significantly reduced in coverage depth and not developed using the appropriate random and systematic selection process; (2) the remaining questions contained sufficient discriminatory validity; (3) candidate performance on the remaining NRC written examination questions was satisfactory; and (4) your written requalification examination program was not susceptible to the introduction of bias and was challenging to licensed operators. We should emphasize that a significantly different outcome could easily have been
Entergy Operations, Inc. -2-determined if the examination development and administration issues were even slightly more extensive.
Based on the results of this review, the NRC has determined that a Severity Level IV violation of NRC requirements occurred. This violation is being treated as a noncited violation, consistent with Section VI.A of the Enforcement Policy. This noncited violation is described in the subject examination report. If you contest the violation or significance of this noncited violation, you should provide a response within 30 days of the date of this examination report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington DC 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 611 Ryan Plaza Drive, Suite 400, Arlington, Texas 76011; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington DC 20555-0001; and the NRC Resident Inspector at the Grand Gulf Nuclear Station.
In accordance with 10 CFR 2.790 of the NRCs "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
If you have any questions please contact me at 817.860.8159.
Sincerely,
/RA/
Anthony T. Gody, Chief Operations Branch Division of Reactor Safety Docket: 50-416 License: NPF-29
Enclosures:
NRC Examination Report 05000416/2003-301
REGION IV==
Docket: 50-416 License: NPF-29 Report No.: 05000416/2003-301 Licensee: Entergy Operations, Inc.
Facility: Grand Gulf Nuclear Station Location: Waterloo Road Port Gibson, Mississippi Dates: April 21 - 23, 2003 (onsite)
June 16, 2003 August 18 - 29, 2003 Examiners: G. E. Werner, Senior Operations Engineer D. S. Muller, Senior Reactor Engineer Approved By: Anthony T. Gody, Chief Operations Branch Division of Reactor Safety
-2-SUMMARY OF FINDINGS ER 05000416/03-301, 4/21-9/11/2003; Grand Gulf Nuclear Station; Initial Licensed Operator Examination Process Review.
NRC examiners evaluated the written examination development process for the December 2000, June 2001, and August 2002 NRC initial operator license examinations.
Cornerstones: Not applicable The NRC identified that the licensee had not developed the December 2000, June 2001, and August 2002 written examinations for the NRC initial operator license program in accordance with the requirements of 10 CFR 55.40 and NUREG-1021, "Operator Licensing Examination Standards for Power Reactors," Revision 8, Supplement 1, and its draft during a pilot evaluation period.
The NRC also identified that the examination author and facility reviewer initialed Step 1.b of Form ES-201-2, indicating that the examination outlines were systematically and randomly prepared, when, in fact, the examination outlines were not systematically and randomly prepared in accordance with Examiner Standard 401, Section D.1. This misrepresentation impacted the NRCs ability to perform its regulatory function. In addition, both content and discrimination validity were affected by the licensees practice of developing examinations and, as a result, affected the equitable and consistent administration of the examination. This violated 10 CFR 50.9, 55.40, and 55.49. This Severity Level IV violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy. (Section .2.2)
Report Details 4. OTHER ACTIVITIES (OA)
4OA4 License Examination
.2 Initial Licensing Examination Development and Administration The licensees process for developing NRC initial operator license examinations was reviewed because on April 3, 2003, the NRC identified that a large number of knowledge and ability statements were repeated on the outlines for the NRC initial operator license written examinations (examinations) administered on August 22, 2002, from the June 2001 initial operator license examination. This observation was not indicative of a random and systematic examination outline development as required by NUREG-1021, Operator Licensing Examination Standards for Power Reactors, Revision 8, Supplement 1, dated April 2001.
.2.1 Examination Development a. Inspection Scope A review was conducted to determine if the licensees examination development process satisfied the requirements contained in NUREG-1021, Revision 8, Supplement 1 (or its draft version utilized in an earlier pilot effort). The licensee authored examinations covered by this review were the examinations administered in August 2002, June 2001, and December 2000. The purpose of this review was to ascertain if the facility licensee's author introduced bias and whether the bias affected the equitable and consistent administration of the examination.
The facility licensees requalification examination development process was also reviewed to ascertain if any bias was introduced and whether the bias, if present, affected the equitable and consistent administration of the NRC biennial requalification examinations.
The licensees actions to resolve examination development issues once identified by the NRC were also reviewed.
b. Findings Introduction. Title 10, Part 55, of the Code of Federal Regulations (10 CFR Part 55)
establishes procedures and criteria for the issuance, maintenance, and renewal of licenses to reactor operators and senior operators. Subpart E (10 CFR 55.40 through 55.49) of 10 CFR Part 55 establishes requirements for the development and administration of written examinations and operating tests.
-2-Part 55.40(b) of Title 10 of the Code of Federal Regulations indicates, in part, that power reactor facility licensees may prepare, proctor, and grade the written examinations required by 10 CFR 55.41 and 55.43 and may prepare the operating tests required by 10 CFR 55.45, subject to the following conditions:
(1) Power reactor facility licensees shall prepare the required examinations and tests in accordance with the criteria in NUREG-1021; and (2) Pursuant to 10 CFR 55.49, power reactor facility licensees shall establish, implement, and maintain procedures to control examination security and integrity.
Description. The reviewers found that the licensee failed to prepare the August 2002 examination in accordance with the criteria contained in NUREG-1021, Revision 8, Supplement 1, dated April 2001. Specifically, NUREG-1021, Revision 8, Supplement 1, Examiner Standard (ES)-401.D.1.b. states, in part, that each examination outline be developed by systematically and randomly selecting specific knowledge and ability statements to complete each of the three tiers of the examination outline. This failure to develop the examination in accordance with NUREG-1021 resulted in the introduction of bias and affected the equitable and consistent administration of the examination. The methodology used by the facility licensees examination preparer was documented on an informal white paper. Taken on its own merit, the process described by the white paper provided little detail and did not clearly describe a process that was systematic and random down to the knowledge and ability statement level. Once the examination preparer demonstrated how the process described in the white paper was utilized to develop an examination, it became apparent that the process did not yield an examination outline that was free of bias and did not adhere to an acceptable examination development model. The model failed to randomly sample at the knowledge and ability statement level. While systems and topics were selected in an appropriate systematic and random manner, the statement selection was dependent on the available questions for selection in the licensees examination bank1 and was not random as required by NUREG-1021. This ultimately resulted in an NRC initial operator license examination containing numerous questions selected from the licensees examination bank1.
NUREG-1021, Revision 8, Supplement 1 (both draft and final forms), specifically reemphasized the need for a systematic and random process for selecting topics and questions. The following is an excerpt from the abstract of Supplement 1.
Supplement 1 to Revision 8 is being issued to update and clarify the NRCs guidelines regarding: (1) the systematic and random selection of topics and questions for the written examination, including limits on question usage; (2) the training and qualification
In this case, the term examination bank refers to an electronic collection of examination questions previously used on examinations. Many of these questions are public records and are often made available to operator license candidates for their review prior to their examination. As of April 2003, the licensees examination bank consisted of approximately 750 questions.
-3-of operator license applicants; (3) the documentation of NRC staff concerns related to draft examination quality; and (4) a number of other minor issues.
NUREG-1021, Revision 8, Supplement 1 (both in its draft and final forms), increased the number of questions that could be selected from a bank from 50 to 75. However, the acceptability of this increase was predicated on an assumption that the knowledge and ability statements were selected in a random and systematic manner while developing the examination outline. It also assumed that the bank questions would be randomly selected if the bank contained more than one question for a specific knowledge and ability statement. These random and systematic processes would minimize the probability that a large number of questions were selected for an examination developed using a relatively small examination bank available for candidates review. For the given size of the licensees bank1 the examiners determined, with the assistance of a risk analyst, that the most probable number of bank questions that should have been selected using a random and systematic outline development process would be in the range of 7 to 12 questions.
A review of the examinations administered in June 2001 and December 2000, revealed that they had also been developed in the same manner as the August 2002 examination. A discussion of the impact of this examination development process had on the validity of the subject examinations follows.
A review of the licensees requalification program examination development process revealed that it was not susceptible to the same introduction of bias due to a prescribed method of sampling.
Once informed of the examination development issue, the licensee conducted an effective self-assessment and implemented corrective actions that resulted in a revision to Licensee Procedure TQ-105, NRC Initial License Examination Development, Validation, and Administration. If implemented properly, the revision would result in proper examination development. Licensee corrective actions were found to be appropriate.
Analysis. As discussed above, the examinations administered in August 2002, June 2001, and December 2000, were developed in a manner that introduced bias and did not adhere to an acceptable examination development model by failing to randomly sample at the knowledge and ability statement level. Specifically, the examination author selected a large number of bank1 questions during the examination development phase. Most examination bank1 questions were from the previous examinations.
In this case, the term examination bank refers to an electronic collection of examination questions previously used on examinations. Many of these questions are public records and are often made available to operator license candidates for their review prior to their examination. As of April 2003, the licensees examination bank consisted of approximately 750 questions.
-4-Examination Validity NUREG-1021, Appendix A, provides an overview of generic examination concepts, one of which is examination validity. Examination validity is maintained when an examination is demonstrated to measure what it is intended to measure. In the case of the NRC examinations, the intent is to measure the candidates knowledge and ability, such that, those who pass will be able to safely perform the duties of reactor operator and senior operator. Examination validity is comprised of three basic parts: (1) content validity, (2) operational validity, and (3) discrimination validity.
Content validity involves the concepts of linkage to specific job tasks of the reactor operator and senior operator and the use of a sample plan, which results in adequate examination coverage that does not introduce bias. In general, the examination questions developed by the licensee had adequate linkage to specific job tasks. To assess coverage of the August 2002, June 2001, and December 2000 examinations, the NRC conservatively eliminated all unmodified questions previously used on examinations and found that both the resultant reactor operator and senior operator outlines had minimal topic coverage within each tier and group with two exceptions. The first exception occurred on the resultant August 2002 reactor operator examination, which had no statement remaining for Tier 3, Category 2, Equipment Control. The second exception occurred on the resultant June 2001 reactor operator examination, which had no statement remaining for Tier 3, Category 3, Radiation Control.
Operational validity ensures that the test items address an actual or conceivable mental or psychomotor activity performed on the job. A review of the subject examinations indicated that examination questions were developed in an operationally valid manner.
Discrimination validity involves the examinations ability to determine if the candidates have sufficiently mastered the knowledge, skills, abilities, and other attributes to perform the job of reactor operator or senior operator. Because the Grand Gulf Nuclear Station examinations were developed with such a large number of questions that the candidates reviewed prior to taking the examination, discrimination on those questions was reduced because the cognitive level at which the candidates were tested decreased to varying degrees of simple recognition. This effect on examination discrimination capability becomes more apparent when the test bank size available for review is small. In the case of the August 2002, June 2001, and December 2000 examinations, the candidates had a limited number of examinations available to them for review prior to their examination. This was similar to having a relatively small examination bank available for review prior to an examination, which was constructed significantly from that examination bank. A number of operators licensed following the August 2002 examination were interviewed by the examiners and each operator interviewed acknowledged that they had recognized a number of questions on their examination as being similar to ones they previously reviewed. Most candidates interviewed indicated that the review of previous examinations helped them get prepared for the NRC style of asking questions and that the review of previous examination questions may have
-5-helped them eliminate the wrong answers. With respect to examination development and the level of difficulty, an analysis of the remaining questions once the unmodified questions previously used on examinations were removed, indicated that 50-60 percent of the questions was at the comprehension/analysis level as required by NUREG-1021.
Clearly, the operator candidates performed better on those questions that they reviewed prior to their examination. A more detailed review of candidate performance is conducted in the following section.
The NRC determined that both content and discrimination validity were affected by the licensees practice of developing examinations and, as a result, affected the equitable and consistent administration of the examination. If the inadequate examination development method been identified by the NRC examiners during the examination review process, the examination would not have been approved by the NRC for administration to operator license candidates. Likewise, had the NRC examiners identified the inadequate examination development method following examination administration, but before license issuance, no licenses would have been issued without conducting a detailed review of the impact. After considering the impact of the inadequate examination development methodology and conservatively removing questions, which were reviewed by the candidates prior to examination administration, the resultant examinations were deficient with respect to the examination implementation requirements of 10 CFR 55.40. Nevertheless, the resultant examinations were found to have met the more general sampling requirements of 10 CFR 55.41 and 55.43.
Enforcement. The review of licensee development of NRC initial operator license examinations found that the licensee failed to develop NRC written examinations in a manner free of bias by failing to randomly sample at the knowledge and ability statement level. In addition, examination development methods used by the licensee were not formally controlled and subsequently did not receive adequate management review for subsequent revisions to NUREG-1021. Further discussion on enforcement is in Section .2.2.b of this report.
.2.2 Examination Administration a. Inspection Scope The licensees examination development issues were reviewed to determine if they affected candidate performance on the August 2002, June 2001, and December 2000 examinations.
The licensees requalification program was reviewed to determine its effectiveness at identifying and correcting licensed operator knowledge and ability deficiencies.
b. Findings Introduction. As discussed above, Title 10, Part 55, of the Code of Federal Regulations (10 CFR Part 55) establishes procedures and criteria for the issuance, maintenance, and renewal of licenses to reactor operators and senior operators. Part 55, Subpart E (10 CFR 55.40 through 55.49) of Title 10 of the Code of Federal Regulations
-6-establishes requirements for the development and administration of written examinations and operating tests.
Part 55.40(b) of Title 10 of the Code of Federal Regulations indicates, in part, that power reactor facility licensees may prepare, proctor, and grade the written examinations required by 10 CFR 55.41 and 55.43, and may prepare the operating tests required by 10 CFR 55.45, subject to the following conditions:
(1) Power reactor facility licensees shall prepare the required examinations and tests in accordance with the criteria in NUREG-1021; (2) Pursuant to 10 CFR 55.49, power reactor facility licensees shall establish, implement, and maintain procedures to control examination security and integrity.
Part 55.49, Integrity of examinations and tests, of Title 10 of the Code of Federal Regulations states, in part, Applicants, licensees, and facility licensees shall not engage in any activity that compromises the integrity of any application, test, or examination required by this part. The integrity of a test or examination is considered compromised if any activity, regardless of intent, affected, or, but for detection, would have affected the equitable and consistent administration of the test or examination.
This includes activities related to the preparation and certification of license applications and all activities related to the preparation, administration, and grading of the tests and examinations required by this part.
Description. The examiners found that the discrimination validity of the August 2002, June 2001, and December 2000 examinations was affected by the licensees practice of developing examinations with a large number of questions that the candidates were exposed to just prior to taking their examination. This exposure just prior to their examination affected the equitable and consistent administration of the examination by providing initial operator license candidates an advantage over their counterparts across the nation.
Analysis. As discussed in Section .2.1 above, the NRC initial operator license examinations administered in August 2002, June 2001, and December 2000, were developed in a manner that introduced bias and did not adhere to an acceptable examination development model by failing to randomly sample at the knowledge and ability statement level. Specifically, the examination author selected a large number of bank1 questions during the examination development phase. Most examination bank1 questions were from the previous NRC initial operator license examinations. The operator candidates performed better on those questions that they had reviewed prior to their examination.
In this case, the term examination bank refers to an electronic collection of examination questions previously used on examinations. Many of these questions are public records and are often made available to operator license candidates for their review prior to their examination. As of April 2003, the licensees examination bank consisted of approximately 750 questions.
-7-August 2002 NRC Initial Operator License Examination The NRC initial operator license examination administered in August 2002 had at least 11 questions from the March 1998, 10 questions from the April 2000, 11 questions from the December 2000, and 24 questions from the June 2001 examinations, and at least 9 other bank questions. The inspection revealed that the operator license candidates were provided copies of the June 2001, December 2000, and April 2000 examinations within 1 month of their examination and that a number of other questions were available to the candidates prior to examination administration. The examiners conducted an analysis of the examination questions that had been provided to the candidates and conservatively removed 48 and 46 of the questions contained on the August 2002 senior operator and reactor operator examinations, respectively. An analysis of operator performance on the August 2002 examination revealed that, on average, candidates scored 99 percent on those questions they had access to prior to their examination and scored 92 percent on those questions they had not seen prior to the examination.
The NRC conducted an assessment of how well the August 2002 examination measured a candidates knowledge and abilities. This assessment included a review of each candidates performance on initial license class examinations, the audit examinations, any practice examinations, subsequent requalification program performance, and performance history in the plant since licensing. The assessment concluded that each candidate had performed in an acceptable manner since licensing and that their NRC written examination grades on the unaffected questions were similar to their performance both before and after the NRC written examination.
The NRC concluded that the licensing decisions made following the August 2002 NRC examinations were appropriate. This conclusion was based on the following facts:
(1) the resultant examination, after conservatively removing all bank questions provided to candidates prior to their examination, met the general sampling requirements of 10 CFR 55.41 and 55.43, albeit significantly reduced in coverage depth and not developed using the appropriate random and systematic selection process.
(Section .2.1); (2) the remaining questions contained sufficient discriminatory validity (Section .2.1); (3) candidate performance on the remaining NRC written examination questions averaged 92 percent, which was indicative of both past and current performances as candidates and operators; and (4) the licensees written requalification examination program was not susceptible to the introduction of bias and was challenging to licensed operators.
June 2001 NRC Initial Operator License Examination The NRC initial operator license examination administered in June 2001 had at least 11 questions from the March 1998, 8 questions from the April 2000, and 10 questions from the December 2000 examinations, and at least 5 other bank questions. The inspection revealed that the operator license candidates were provided copies of the December 2000 (comprehensive, NRC, and audit) examinations within 2 months of their examination and that a number of other questions were available to the candidates prior to examination administration. The examiners conducted an analysis of the examination questions that the candidates had been exposed to and conservatively removed 44 of
-8-the questions contained on the June 2001 reactor operator examination. An analysis of operator performance on the June 2001 examination revealed that, on average, candidates scored approximately 96.5 percent on those questions they had access to prior to their examination and scored 82 percent on those questions they had not seen prior to the examination.
The NRC conducted an assessment of how well the June 2001 examination measured a candidates knowledge and abilities. This assessment included a review of each candidates performance on initial license class examinations, the audit examinations, any practice examinations, subsequent requalification program performance, and performance history in the plant since licensing. The assessment concluded that each candidate had performed in an acceptable manner since licensing and that their NRC written examination grades on the unaffected questions were similar to their performance both before and after the NRC written examination.
Because of the significant difference between the original scores and the conservative scoring, which removed any question that was exposed to the candidate prior to taking their examination and the number of borderline resultant grades, an independent review of the June 2001 examination was conducted. This review supported the conclusion that, while marginal, all candidates passed the resultant examination. In addition, each candidate had performed in an acceptable manner since licensing and that their NRC written examination grades on the unaffected questions were similar to their performance both before and after the NRC written examination.
The NRC concluded that the licensing decisions made following the June 2001 NRC examinations were appropriate. This conclusion was based on the following facts:
(1) the resultant examination, after conservatively removing all bank questions provided to candidates prior to their examination, met the general sampling requirements of 10 CFR 55.41 and 55.43, albeit, significantly reduced in coverage depth and not developed using the appropriate random and systematic selection process (Section .2.1); (2) the remaining questions contained sufficient discriminatory validity (Section .2.1); (3) candidate performance on the remaining NRC written examination questions averaged 82 percent, which was indicative of both past and current performances as candidates and operators; and (4) the licensees written requalification examination program was not susceptible to the introduction of bias and was challenging to licensed operators.
December 2000 NRC Initial Operator License Examination The NRC initial operator license examination administered in December 2000 had at least 17 questions from the March 1998, and 20 questions from the April 2000 examinations and at least 6 other bank questions. The inspection could not identify which previous examinations were provided to the December 2000 candidates prior to examination administration. However, the December 2000 examination was a retake examination for candidates who did not pass the April 2000 examination. The examiners conducted an analysis of the examination questions that had been exposed to the candidates and conservatively removed 37 of the questions contained on the December 2000 senior operator retake examination. An analysis of operator
-9-performance on the December 2000 examination revealed that, on average, candidates scored approximately 96 percent on those questions they had access to prior to their examination and scored 91 percent on those questions they had not seen prior to the examination.
The NRC conducted an assessment of how well the December 2000 examination measured a candidates knowledge and abilities. This assessment included a review of each candidates performance on initial license class examinations, the audit examinations, any practice examinations, subsequent requalification program performance, and performance history in the plant since licensing. The assessment concluded that each candidate had performed in a acceptable manner since licensing and that their NRC written examination grades on the unaffected questions were similar to their performance both before and after the NRC written examination.
The NRC concluded that the licensing decisions made following the December 2000 NRC examinations were appropriate. This conclusion was based on the following facts:
(1) the resultant examination after conservatively removing all bank questions provided to candidates prior to their examination met the general sampling requirements of 10 CFR 55.41 and 55.43, albeit significantly reduced in coverage depth and not developed using the appropriate random and systematic selection process (Section .2.1), (2) the remaining questions contained sufficient discriminatory validity (Section .2.1), (3) candidate performance on the remaining NRC written examination questions averaged 91 percent, which was indicative of both past and current performances as candidates and operators, and (4) the licensees written requalification examination program was not susceptible to the introduction of bias and was challenging to licensed operators.
Enforcement. The review of licensee development of NRC initial operator license examinations found that the licensee failed to develop NRC written examinations in a manner free of bias by failing to randomly sample at the knowledge and ability statement level. In addition, examination development methods used by the licensee were not formally controlled and subsequently did not receive adequate management review for subsequent revisions to NUREG-1021. This combined with the level of exposure initial operator license candidates had on actual examination content affected the equitable and consistent administration of the test or examination.
Part 55.40 of Title 10 of the Code of Federal Regulations states, in part, Power reactor facility licensees may prepare, proctor, and grade the written examinations required by 10 CFR 55.41 and 55.43 and may prepare the operating tests required by 10 CFR 55.45, subject to the following conditions:
(1) Power reactor facility licensees shall prepare the required examinations and tests in accordance with the criteria in NUREG-1021 as described in paragraph (a) of this section; and (2) Pursuant to 10 CFR 55.49, power reactor facility licensees shall establish, implement, and maintain procedures to control examination security and integrity.
-10-Accordingly, NUREG-1021, ES-401 Section D.1.b. states, in part, that each examination outline be developed by systematically and randomly selecting specific knowledge and ability statements to complete each of the three tiers of the examination outline.
Attachment 1 to ES-401 provides one acceptable method for randomly selecting knowledge and ability statements. Examiner Standard-401 Section D.1.b. further states, in part, that other methodologies may be used provided they are reproducible and scrutable and yield an examination outline that is free of bias, adhere to the applicable examination model, and samples at the specific knowledge and ability statement level.
Part 50.9 of Title 10 of the Code of Federal Regulations states, in part, that information provided to the Commission by a licensee shall be complete and accurate in all material respects. NUREG-1021, ES-201, Section C.1.f, states, in part, that when a licensee chooses to write its own examination, it shall develop the outlines and examinations in accordance with ES-301, ES-401, and ES-701. Examiner Standard-401, Section D.1.b.,
states, in part, that specific knowledge and ability statements be systematically and randomly selected in a manner that produces an examination free of bias. Additionally, ES-201, Section C.1.f, states, in part, that the facility shall use Form ES-201-2, Examination Outline Quality Checklist. Form ES-201-2, Step 1.b., states Assess whether the outline was systematically and randomly prepared in accordance with Section D.1 of ES-401 and whether all K/A categories are appropriately sampled.
When the examination is developed by the licensee, this step is required to be initialed by both the examination author and facility reviewer.
Part 55.49, Integrity of examinations and tests, of Title 10 of the Code of Federal Regulations states, applicants, licensees, and facility licensees shall not engage in any activity that compromises the integrity of any application, test, or examination required by this part. The integrity of a test or examination is considered compromised if any activity, regardless of intent, affected, or, but for detection, would have affected the equitable and consistent administration of the test or examination. This includes activities related to the preparation and certification of license applications and all activities related to the preparation, administration, and grading of the tests and examinations required by this part.
Contrary to the above, on December 18, 2000, June 1, 2001, and August 23, 2002, the licensee administered NRC examinations and tests, which were not developed in accordance with the criteria contained in NUREG-1021, ES-401, Section D.1.b., as required by 10 CFR 55.40. In addition, the examination author and facility reviewer initialed Step 1.b of Form ES-201-2, indicating that the examination outlines were systematically and randomly prepared, when, in fact, the examination outlines were not systematically and randomly prepared in accordance with ES-401, Section D.1. The licensees process to systematically and randomly develop an examination outline did not ensure that the examination outline was free of bias by failing to randomly sample at the knowledge and ability statement level. A result of this development method was that examinations were administered that had a large number of repeat questions from previous examinations administered at the Grand Gulf Nuclear Station. These questions were reviewed by the candidates shortly prior to their NRC initial operator license examination. The large number of repeat questions combined with the availability of previously administered NRC initial operator license written examinations (as public documents) affected the equitable and consistent administration of the test or
-11-examination. This was evidenced by the improved candidate performance on the previously administered examination questions when compared to the candidate performance on the new or modified examination questions. This Severity Level IV violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy (05000416/2003301-001). The licensee entered this issue into their corrective action program as CR-GGN-2003-1270.
4OA6 Meetings, including Exit Exit Meeting Summary The examiner presented the inspection results to members of the licensees staff on April 22, 2003, June 12, 2003, and September 16, 2003.
During all meetings, licensee management acknowledged the inspection findings and stated that none of the material examined during the inspection was considered proprietary.
ATTACHMENT KEY POINTS OF CONTACT Licensee Jeff S. Forbes, Vice President - Operations George A. Williams, Acting Vice President - Operations Jerry C. Roberts, Director - Nuclear Safety Gregory Sparks, Manager - Operations Ron Barnes, Manager - Training and Development J. Lee Robertson, Manager - Quality Assurance Charles Ellsaesser, Manager - Corrective Action Program Thomas McIntyre, Supervisor - Operations Training G. Alan Middlebrooks, Supervisor - Operations Training Michael Rasch, Senior Operations Instructor LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED Opened and Closed 05000416/2003301-001 NCV Failure to develop NRC initial operator license examinations administered in August 2002, June 2001, and December 2000, in accordance with NUREG-1021.
Failure to ensure equitable and consistent administration of NRC initial operator license examinations as required by 10 CFR55.49. (Section 4OA4.2.2)
LIST OF DOCUMENTS REVIEWED Condition Reports CR-GGN-1999-00418 4/7/1999 Investigation into training materials used in the September 1999 License Operator Class CR-GGN-2000-00776 5/31/2000 Investigation into NRC initial examination failures on the May 19, 2000 License Operator Class CR-GGN-2003-01270 4/17/2003 Investigation into NRC examination development methodology following NRC verbal communication of possible development concerns.
-2-Procedures GCD-RO-CRO01.02 Control Room Operator Training Program Course Description 01-S-04-2 Licensed Operator Requalification Training TQ-105, Revision 2 NRC Initial License Examination Development, Validation, and Administration Other Documents No document ID License Operating Training schedule for the August 2002 License Operator Class No document ID License Operating Training schedule for the June 2001 License Operator Class No document ID License Operating Training schedule for the December 2000 License Operator Class No document ID Operator performance data for the licensed operator requalification program Accession Number ML003762429 Publicly available examination material package for the December 2000 NRC initial license examination Accession Number ML010180604 Publicly available examination material package for the June 2001 NRC initial license examination Accession Number ML020570217 Publicly available examination material package for the August 2002 NRC initial license examination