ML030370307

From kanterella
Jump to navigation Jump to search
Audit - August 2002 Exam
ML030370307
Person / Time
Site: Arkansas Nuclear Entergy icon.png
Issue date: 02/19/2003
From: Quay T
NRC/NRR/DIPM/EHPB
To: Chamberlain D
Division of Reactor Safety IV
References
Download: ML030370307 (22)
v • d • e


Text

February 19, 2003 MEMORANDUM TO: Dwight D. Chamberlain, Director Division of Reactor Safety, RIV

/RA/

FROM: Theodore R. Quay, Chief Equipment and Human Performance Branch Division of Inspection Program Management, NRR

SUBJECT:

ARKANSAS NUCLEAR ONE, UNIT 1, EXAMINATION AUDIT This memorandum refers to the audit conducted by David Muller of my staff of the initial examinations administered by the NRC at Arkansas Nuclear One, Unit 1, during the week of August 22, 2002. This audit included on-site observations of the Region IV exam team, and reviews of examination material associated with the facility authored, NRC approved exam.

The chief examiner, Thomas Stetka, and the rest of the exam team conducted all examination activities in a highly professional manner.

In general, the examination was determined to be in compliance with Revision 8, Supplement 1 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors. However, as a result the audit, three items were identified where increased regional attention appears warranted: (1) the entry of examination materials into ADAMs, (2) the level of difficulty and effectiveness of operating test job performance measures (JPMs), and (3) scenario content and crew rotation, such that each applicant is evaluated on the required number and type simulator scenario events.

I have attached the applicable checklists (Attachments 1 and 2) from IOLB Manual Chapter 320, IOLB Program Review of Initial Examinations Administered at Licensee Facilities, which includes further information. If you have any questions, please contact David Trimble, Chief, Operator Licensing and Human Performance Section, at (301)415-2942.

Attachments: As stated

February 19, 2003 MEMORANDUM TO: Dwight D. Chamberlain, Director Division of Reactor Safety, RIV

/RA/

FROM: Theodore R. Quay, Chief Equipment and Human Performance Branch Division of Inspection Program Management, NRR

SUBJECT:

ARKANSAS NUCLEAR ONE, UNIT 1, EXAMINATION AUDIT This memorandum refers to the audit conducted by David Muller of my staff of the initial examinations administered by the NRC at Arkansas Nuclear One, Unit 1, during the week of August 22, 2002. This audit included on-site observations of the Region IV exam team, and reviews of examination material associated with the facility authored, NRC approved exam.

The chief examiner, Thomas Stetka, and the rest of the exam team conducted all examination activities in a highly professional manner.

In general, the examination was determined to be in compliance with Revision 8, Supplement 1 of NUREG-1021, Operator Licensing Examination Standards for Power Reactors. However, as a result the audit, three items were identified where increased regional attention appears warranted: (1) the entry of examination materials into ADAMs, (2) the level of difficulty and effectiveness of operating test job performance measures (JPMs), and (3) scenario content and crew rotation, such that each applicant is evaluated on the required number and type simulator scenario events.

I have attached the applicable checklists (Attachments 1 and 2) from IOLB Manual Chapter 320, IOLB Program Review of Initial Examinations Administered at Licensee Facilities, which includes further information. If you have any questions, please contact David Trimble, Chief, Operator Licensing and Human Performance Section, at (301)415-2942.

Attachments: As stated (2)

DISTRIBUTION:

TGody, RIV DTrimble, IOHS EBarnhill, IOHS TStetka, RIV JMunro, IOHS IOLB R/F ADAMs Accession #ML030370307 To receive a copy of this document, indicate in the box: "C" = Copy without attachment/enclosure "E" = Copy with attachment/enclosure "N" = No copy OFFICE DIPM: IOHS DIPM: IOHS:SC DIPM: IEHB:BC NAME DMuller DTrimble TQuay DATE 2/12/03 2/14/03 2/19/03 02/ /03 OFFICIAL RECORD COPY

IOLB-MC-320 11/15/99 Review of the Written Examination Region: IV Chief Examiner: T. Stetka Facility: ANO Unit 1 Prep Week Dates: Week of July 22, 2002 Exam Week Dates: Written: August 16, 2002; Operating Test: August 19-22, 2002 A. Examination Preparation Activities

1. Written operator licensing examinations prepared in accordance with Appendix B and ES-401.D.1 through D.4 of NUREG-1021, "Operator Licensing Examination Standards for Power Reactors.

See comments for A.3, A.4, and A.12 YES U* NO

2. The examination outline K/As were identified using a systematic process.

(ES-401 D.1.b)

  • YES U NO
3. Perform an independent review on a sampling (~25%) of the written examination questions and ensure that the knowledge tested by the questions is consistent with its referenced K/A.

YES U* NO 2 Qs out of 46 Q sample (4.3%) were not consistent with their K/As.

4. If both RO and SRO examinations were given at the same time, check that no more than 75 percent of the RO examination questions were duplicated on the SRO examination, and that the remaining SRO questions evaluate the additional knowledge and abilities required for the higher license level consistent with the guidance provided in the Examination Standards and 10 CFR 55.43(b).

Questions related to 10 CFR 55.41(b) topics may be appropriate SRO-level questions if they evaluate knowledge and abilities at a level unique to the SRO position. (ES-401 D.2.d)

YES U* NO RO/SRO overlap OK. 2 SRO only questions did not appear to test at the SRO level.

5. The examination(s) repeat no more than 25 percent of the questions from examinations, quizzes, or tests administered to the license applicants during their last license class (for facility-developed examinations) or from the past two licensing examinations at the facility. (ES-401 D.2.f)

N/A, systematic random K/As *YES NO

6. Facility-written examinations repeat no questions from the applicants audit examination unless the two examinations are independently developed. (Five questions may be duplicated if independently developed.) (ES-401 D.2.f)
  • YES U NO
7. Bank use meets limits -- no more than 75% from the bank, at least 10% new, and the remainder modified. Perform a sampling check of the modified questions (~25%) and confirm that the pertinent conditions in the stem and at least one distractor have been significantly modified. Confirm that a technical reference and cross-reference to the facilitys examination bank is noted as applicable for each question. (ES-401 D.2.f & g)

YES U NO

8. All test questions are in the multiple choice format and worth one point. (ES-401 D.2.e)

YES U NO

9. A copy of the "Policies and Guidelines for Taking NRC Examinations," Appendix E, Parts A and B (General and Written Examination Guidelines) was provided to each applicant and the applicants were briefed on the rules and guidelines in effect for the written examination. Verify the guidelines are consistent with the revised version in NUREG-1021, Final Revision 8.

Not reviewed YES NO

10. Examination items listed on Form ES-401-7, "Written Examination Quality Assurance Checklist," were checked by the examination author and all reviewers, including chief examiner and branch chief. The regional reviewer selectively reviewed and verified the accuracy of the examination item checks listed on Form ES-401-7 if the facility wrote the examination. (ES-401 E.2.c)

Not reviewed, see comments on ADAMs. YES NO

11. Perform an independent review on a sampling (~25%) of the written examination questions using Form ES-401-7 and verify that the results agree with the regions Form ES-401-7.

See items A.3, A.4, and A.12 YES U* NO

12. The regional office assessed the psychometric quality of the facility proposed examination using Form ES-401-9, Written Examination Review Worksheet.

Verify that feedback to the facility for unacceptable questions explains how the Appendix B attributes are not being met and the 30 question sampling review is completed one week after the Region receives the examination. Also, confirm that any comments that would require the facility to rework an NRC-validated question are reviewed and approved by the responsible supervisor. (ES-401 E.3.b)

Note: ES-201 C.3.f requires the chief examiner to note any necessary changes for review and comment by the responsible supervisor before reviewing with the facility author or contact.

401-9s not reviewed, see comments on ADAMs. YES NO

 Analyze the sampled questions and determine the level of knowledge required to answer. (ES-401 and Appendix B of NUREG-1021)

- Knowledge Fundamentals - 61 %

- Higher Cognitive - 39 %

 Rate each sampled question using a five-point difficulty rating scale.

Provide a one or two sentence explanation that describes the basis for the rating. The rating scale is as follows:

DIFFICULTY RATING 1 2 3 4 5 Low XAve/Med High

- Questions in the mid-range, ranging in scores between 2.0 - 4.0 are acceptable models for inclusion in the examination.

- Questions at the extreme scores (1.0 - 1.99 and 4.01 - 5.0) are either too easy or too difficult and therefore have reduced validity and may be non-discriminatory.

No question rated a 1" or a 5". Average difficulty of 46 Q sample was 3.0.

 Evaluate the sampled questions for any psychometric or content flaws and determine if the explanations are clear and based on Appendix B.

3 Qs out of 46 Q sample (6.5%) contained content flaws.

13. No more than 50-60% of the questions on the examination are written at the comprehension/analysis level. Verify the number by reviewing the knowledge classification provided by the facility for each question on the administered examination, i.e., evaluate the facilitys view regarding compliance with this guideline. Also, note the number of higher cognitive level questions as determined by the facility and if this number is greater than 60%. Explain.

YES U NO

14. The branch chief reviewed the entire examination before authorizing the chief examiner to proceed with the facility prereview per ES-201. (ES-401 E.3.a)

N/A - facility developed exam. YES NO

15. The NRC and/or facility review comments were appropriately resolved and the responsible supervisor signed Form ES-401-7. (ES-401 E.3.d)

Not reviewed, see comments on ADAMs. YES NO B. Examination Administration Activities Not Observed

1. The chief examiner has reviewed the facility examination administration policies for compliance with the guidelines in ES-402, C.1 and D.

Not reviewed YES NO

2. The NRC chief examiner inspected the examination facilities to ensure their adequacy and, if on-site during examination administration, periodically monitored the facilitys examination administration. (ES-402 C.2.b)

Not reviewed YES NO

3. No more than 30 days elapsed between the written examination and operating tests without obtaining NRR program office concurrence. (ES-402 C.2.b)

YES U NO

4. All applicant questions regarding specific written examination test items and all statements of clarification are documented (verbatim if possible) for reference by the NRC in resolving grading conflicts. (ES-402 C.3.b)

Not reviewed YES NO

5. Each examinee was briefed on the policies and guidelines for taking NRC written examinations according to Appendix E. The first two guidelines for Parts A and B were read verbatim. For NRC-proctored examinations, the NRC verified each applicants identity and examination level against the examination assignment sheet (ES-201, Attachment 4, "Examination Assignment Sheet"). (ES-402 D.1)

YES U NO

6. The time allowed to complete the examination was not extended without prior approval of the NRC regional office. (ES-402 D.4.d)

Not reviewed YES NO C. Post-Examination Reviews

1. After the completion of the written examination, the master copy of the examination was annotated to reflect all changes made to questions during the administration of the examination. A copy was given to the facility or NRC for review, as appropriate. (ES-402 D.3.b and E.1)

Not reviewed. YES NO

2. The facility pre-reviewers signed the post-examination security statement (Form ES-201-3) after the examinations were completed. (ES-402 E. 3)

Not reviewed. See comments on ADAMs. YES NO

3. Facility submitted formal comments within 5 working days (or as arranged by regional management) after the examination is administered. The comments are signed by an authorized facility representative. (ES-402 E.4, E.6)

No post-exam comments YES U NO D. Grading and Documentation

1. The facility licensees comments and recommendations were properly resolved, examination item analysis performed, all questions posed by the applicants during the examination were evaluated and the master examination revised as necessary. (ES-403 D.1 & D.3) NOTE: Verify that facility comments on the written examination, specific NRC resolution for each facility comment, including a precise explanation for accepting or rejecting each facility comment, and a specific justification for every additional item deletion or change are properly documented in the examination report. (ES-501 E.3.b & Attachment 1)

YES U NO

2. Each applicants answer sheet was copied before beginning the grading process and set aside for later use.

YES U NO

3. Each applicants original answer sheet is graded in red pen or pencil and annotated as necessary according to ES-403 D.2.b.

Electronic grading acceptable. YES U NO

4. Grading items listed on Form ES-403-1, "Written Examination Grading Quality Checklist," were checked by the examination grader and all reviewers, including NRC chief examiner. The responsible supervisor reviewed and approved the completed checklist ensuring all requirements were completed. (ES-403 D.3.c)

YES U NO

5. Perform an independent review of the written examination grading using Form ES-403-1 and verify that the results agree with the region's Form ES-403-1.

YES U NO

6. Generic strengths and weaknesses identified as a result of the grading and reviewing the written examinations, and any significant grading deficiencies are documented in the examination report. (ES-501 E.3.a)

YES U NO E. Comments and Notes on the Written Examination Process A.3 K/A mismatches ES Appendix B, section C.1.b states that exam authors should ensure that questions match the intent of their K/As. ES-401, section E.2.c, requires Regional NRC exam reviewers to sample the exam questions for conformance with the referenced K/As. For the most part, this Regional sampling appears to have been properly performed. However, during this review, two questions were identified that did not appear to match the intent of their referenced K/A:

Question Subject of Question Referenced K/A Mismatch?

No.

SRO #5 Given a fire in the plant, 2.4.30 - Knowledge of WHICH Given in stem that will (QID 0411) what is the time requirement [emphasis added] events related to report to NRC; doesnt for notifying the NRC? system status should be reported to test WHICH events outside agencies. should be reported.

SRO #65 Given plant conditions, A01AK3.2 - Knowledge of the Asks for correct (QID 0062) including a condensate REASONS FOR [emphasis added] response, does not test pump trip [causes a the following responses as they the REASONS FOR the runback], what is the correct apply to Plant Runback: Normal, correct response.

response? abnormal, and emergency operating procedures.

A.4 SRO only level questions ES-401 D.2.d states that the 25 SRO-level questions shall evaluate the additional knowledge and abilities required per the higher license level per 10 CFR 55.43(b) or the facility licensees learning objectives. Questions related to 10 CFR 55.41(b) topics may also be appropriate SRO-level questions if they evaluate knowledge and abilities at a level that is unique to the SRO job position. This audit examined all 25 ANO SRO-only level questions, and determined that 2 of these questions did not appear to meet these requirements (SRO questions #12 [QID 0032] and #41 [QID 0169]):

6 Both of these questions appeared to test RO systems level knowledge and not knowledge unique to the SRO job position;

- Neither of these questions K/As were linked to 10 CFR 55.43(b); and

- Neither of these questions tested topics contained in 10 CFR 55.43(b).

A.12 Question content flaws ES-401 D.2.b states that questions shall be free of psychometric flaws. However, during a review of the 46 sampled questions, 3 questions were identified as containing psychometric flaws:

Question No. Nature of the Flaw RO #88 Lack of stem/question objectivity. Question asks for what EOP will BEST mitigate the given (QID 0126) event.

RO #92 More than one correct answer. Basis for choice c as incorrect is that service water would (QID 0205) be affected, but service water conditions not mentioned in stem - cant rule out choice c.

RO #93 All answers possibly correct - the correct answer also includes all the supposed incorrect (QID 0232) answers. (If minimum conditions for shift relief are > 1% power or a SDM of 1.5%, then wouldnt a higher power or larger SDM also allow shift relief?)

ADAMs Retention Comments ES 501F.1 states that the original or a copy of examination materials be retained in the facilitys master examination file or be available via ADAMs. By convention, and to ensure public availability, examination materials should be available via ADAMs. However, for this examination, only 4 records were available via ADAMs (at roughly 4 months after the exam): 1) the corporate notification letter, 2) the examination report, 3) the final written exam, and 4) the written exam grading analysis. Notable items not available via ADAMs included: 1) the operating test, and 2) all of the QA forms (ES-201-2, ES-301-3, ES-301-4, ES-301-5, ES-301-6, ES-401-7, and the ES-401-9 forms).

F. Summary - Examination Performance Measures **

1. Timeliness of Reviews and Feedback
  • NRC Comments on Outline - days following NRC receipt
  • NRC Comments on Examination - days following NRC receipt
  • NRC Examination Management Approval - days prior to exam admin.
  • Post exam comments resolved - days following exam admin.
2. Examination Content and Changes
  • # of Higher Cognitive Questions - NRC / Fac.
  • # of Bank/Mod/New Questions - NRC / Fac.
  • # of Questions Repeated from NRC Exams - NRC / Fac.
  • # of Questions Repeated from Quizzes/Audit Exams - NRC / Fac.
  • # of Proposed Questions Changed by NRC - Unacc. / Minor*
  • # of NRC-validated Questions Changed by NRC - *
  • - explain reason(s) for changes
3. Resources
  • # of Facility Hours Developing Proposed Exam - Hrs
  • # of NRC Hours reviewing Proposed Exam - Hrs
  • # of Facility Hours Resolving NRC Comments - Hrs Notes: * - These ES criteria may be answered in the affirmative based on review of examination reference materials, discussion with either the facility author(s) and/or NRC examiners, or reliance on facility and NRC reviewers initials. If the criteria are answered negatively then an explanation for the negative response will be provided in Section E including its basis.
    • - Since these measures may not be addressed by any specific ES guideline, the reviewer should as a minimum provide the information source and clarifying information as appropriate.

IOLB-MC-320 11/15/99 Review of the Operating Test Region: IV Chief Examiner: T. Stetka Facility: ANO Unit 1 Prep Week Dates: Week of July 22, 2002 Exam Week Dates: Written: August 16, 2002; Operating Test: August 19-22, 2002 A. Examination Preparation Activities

1. The chief examiner scheduled the same examiner to administer all three operating test categories to an applicant unless otherwise authorized by the responsible regional supervisor. (ES-201 C.3.j & ES-302 D.1.d)

NOTE: Under certain circumstances the responsible regional supervisor may authorize the chief examiner to divide the operating test categories and subcategories among different examiners (simulator operating tests consisting of multiple scenarios shall not be divided).

YES U NO

2. No NRC examiner was scheduled to administer more than four operating tests in any one week. (ES-201 C.2.e)

YES U NO

3. Coverage of the administrative topics (Category A) conform to the guidance provided in ES-301 D.2.a through D.2.d:

 Each candidate was evaluated on the required number of subjects listed below during the operating test:

(ES-301 D.2.a)

Attachment 2

Topic Number of Subjects A.1, "Conduct of Operations" 2 A.2, "Equipment Control" 1 A.3, "Radiation Control" 1 A.4, "Emergency Plan" 1 YES U NO

 SRO applicants evaluated in greater depth than RO applicants. (ES-301 D.2.c)

(NOTE: RO applicants need only understand the mechanics and intent of the related subjects as they pertain to his or her tasks at the facility.)

YES U NO

 Form ES-301-1, "Administrative Topics Outline," adequately describes the administrative subjects selected for evaluation and the method(s) by which each subject will be evaluated. The method of evaluation should include the title of any JPMs and a brief summary of the questions. (ES-301 D.2.f)

YES U NO

4. Coverage of the control room systems and facility walk-through topics (Category B) conform to the guidance provided in ES-301 D.3.a and b:

 Based on the applicants license level, each candidate was evaluated on the required number of systems from the safety function groupings identified in the applicable K/A Catalog (BWR or PWR) listed below during the test: (ES-301 D.3.a)

License Level Subcategory B.1 Subcategory B.2 Total RO 7 3 10 SRO-instant(I) 7 3 10 SRO-upgrade(U) 2 or 3 3 or 2 5 NOTES: The 10 systems and evolutions should evaluate at least 7 different safety functions (5 for SRO-U). One of the control room systems or evolutions must be an ESF, and the same system or evolution should not be used to evaluate more than one safety function. For PWR operating tests, the primary and secondary systems listed under Safety Function 4 may be treated as separate safety functions.

YES U NO

5. No JPM tasks replicate tasks that have been selected for evaluation on the dynamic simulator test. (ES-301 D.3.b)

YES U NO

6. All the JPMs, individually and as a group, have meaningful performance requirements that provide a basis for evaluating the applicants understanding of and ability to operate the associated systems. (ES-301 D.3.b)

YES U NO

7. No more than 80% of any applicants walk-through test is taken from the facilitys bank without significant modification. (ES-301 D.3.b)

YES U NO

 At least one JPM is related to a shutdown or low power condition and no more than 40% of the JPMs require the applicants to execute alternate paths within the procedures. (ES-301 D.3.b)

YES U NO

 At least one of the JPMs conducted in the plant evaluates the applicant's ability to implement actions required during an emergency or abnormal condition. (ES-301 D.3.b)

YES U NO

 At least one of the JPMs requires the applicant to escort the examiner into the radiologically controlled area. (ES-301 D.3.b)

YES U NO

 Independently rate each JPM using a five-point difficulty rating scale.

Provide a one or two sentence explanation that describes the basis for the rating. The rating scale is as follows:

DIFFICULTY RATING 1 2 X 3 4 5 Low Ave/Med High

- JPMs in the mid-range, ranging in scores between 2.0 - 4.0 are acceptable models for inclusion in the examination.

- JPMs at the extreme scores (1.0 - 1.99 and 4.01 - 5.0) are either too easy or too difficult and therefore have reduced validity and may be non-discriminatory.

JPM Level of Difficulty could be improved. See Section D, Comments and Notes on the Operating Test.

Auditor Assessed Level of Difficulty Ratings (1 = Low Difficulty 5 = High Difficulty)

Admin (RO) Admin (SRO) Systems (RO and SRO-I) Systems (SRO-U)

A.1.a = 3.0 A.1.a = 2.0 B.1.a = 3.0 B.1.f = 1.5 B.1.a = 3.0 A.1.b = 2.5 A.1.b = 2.5 B.1.b = 2.5 B.1.g = 3.0 B.1.c = 2.5 A.2 = 3.0 A.2 = 2.5 B.1.c = 2.5 B.2.a = 2.5 B.1.d = 1.5 A.3 = 2.0 A.3 = 3.0 B.1.d = 1.5 B.2.b = 2.0 B.2.b = 2.0 A.4 = 2.5 A.4 = 3.0 B.1.e = 2.0 B.2.c = 2.0 B.2.c = 2.0 Ave. = 2.6 Ave. = 2.6 Ave. = 2.25 Ave. = 2.20

8. Applicants at a facility qualifying for a dual or multi-unit licenses are properly tested on the different systems, control board layouts, and any other differences between the units during the walk-through portion of the operating test. (ES-301 D.1.g)

N/A

9. Prescripted questions developed for Category A of the operating test comply with the criteria provided in Attachment 1 of ES-301 and ES-602 for the development and review of open-reference questions. (ES-301 D.1.l)

YES U NO

10. Coverage of the operator knowledge and abilities in the simulator examination (Category C) conform to the guidance provided in ES-301 D.4.a through D.4.d:

 Each RO and SRO-I applicant are required to perform a direct reactivity manipulation. (Form ES-301-5)

YES U NO

 Initial conditions are varied among the scenarios and included startup, low power, and full power situations. (ES-301 D.4.c)

YES U NO

 Each scenario set exercises each applicant on the types and quantities of evolutions, failures, and transients, identified for the applicants license level, as described on Form ES-301-5, "Transient and Event Checklist,"

with each event only counted once per applicant. (ES-301 D.4.d)

YES NO U

 For each event listed on Form ES-D-1, a Form ES-D-2, "Operator Actions," that describes the expected communications, actions, and reference material to be used by each operating position on the crew, is prepared. (ES-301 D.4.d)

YES U NO

 Using Forms ES-301-5 and ES-301-6, review each scenario set to ensure that each competency and rating factor is adequately covered and to ensure that every applicant will have the opportunity to perform or respond to the required numbers and types of evolutions and events.

(ES-301 D.4.d)

YES NO U

 Every operator is evaluated using at least one new or significantly modified scenario and all other scenarios have been altered according to ES-301 D.4. (ES-301 D.4.b)

YES U NO

 Scenario events do not duplicate operations that will be tested during the walk-through portion of the operating test. (ES-301 D.4.c)

YES U NO

 Independently rate each scenario using a five-point difficulty rating scale.

Provide a one or two sentence explanation that describes the basis for the rating. The rating scale is as follows:

DIFFICULTY RATING 1 2 X3 4 5 Low Ave/Med High

- Scenarios in the mid-range, ranging in scores between 2.0 - 4.0 are acceptable models for inclusion in the examination.

- Scenarios at the extreme scores (1.0 - 1.99 and 4.01 - 5.0) are either too easy or too difficult and therefore have reduced validity and may be non-discriminatory.

Scenario Ratings - Scenarios were of average difficulty. Number of events met targets of Form ES-301-4; each scenario took about one and a half hours to run. (Tech specs were typically looked at after scenario completion, which saved some time.) Number of manipulations and level of analysis was about average.

11. The three categories of the operating test are not redundant, nor should they duplicate material that is covered on the written examination. (ES-301 D.1.h)

YES U NO

12. Operating tests written by the facility licensee do not duplicate test items (scenarios or JPMs) from the applicants audit test given at or near the end of the license training class. (ES-301D.1.a)

Not reviewed

13. The chief examiner received supervisory approval and reviewed the proposed operating tests with the facility licensee and Form ES-301-3 is reviewed and signed after the pre-examination review changes are incorporated. (ES-301 E.2.e & g, ES-201 C.2.i)

Not reviewed - form not in ADAMs. YES NO B. Examination Administration Activities

1. The Region obtained NRR program office concurrence if the dates of the operating tests and written examination diverged by more than 30 days.

(ES-302 2.a)

YES NO N/A U

2. An NRC examiner briefed all of the applicant(s) in accordance with parts A, C, D, and E of Appendix E before beginning the operating test.

(ES-302 D.1)

YES U NO

3. Any SRO upgrade applicants not being individually evaluated while in the reactor operator or balance of plant positions were graded on their ability to Operate the Control Boards - SRO Competency 5. (ES-302 D.1.d)

N/A

4. The same simulator scenarios were not repeated during successive days.

(ES-302 D.1.f)

YES U NO

5. Except for the simulation facility operators, no other member of the facility's staff observed an operating test without the chief examiner's permission. No applicant was allowed to witness any other applicants operating test. (ES-302 D.1.j)

YES U NO

6. The chief examiner confirmed with the facility licensee that the instructors station, programmers tools, and external connections can not compromise operating test security during administration. (ES-302 D.1.k)

YES U NO

7. The chief examiner arranged for any NRC examiners not familiar with the facility to receive a plant tour before administering any operating tests.

(ES-302 D.1.l)

Not reviewed

8. Any JPMs not validated by the NRC or facility licensee during the preparatory site visit are validated before use on the operating test.

YES U NO

9. The simulator setup matches the conditions specified for each JPM. No JPM is administered without initial conditions discrepancies corrected. (ES-302 D.2.b)

YES U NO

10. The examiners collect any applicant sketches, flow paths, or other illustrations made by the applicant in answer to any questions. (ES-302 D.2.c)

YES U NO

11. The examiners ask follow-up questions during categories A and B JPMs only if the applicant fails to accomplish the task standard, demonstrates a lack of familiarity with equipment and procedures, or is unable to locate information, indications or controls. (ES-302 D.2.f)

YES U NO

12. Each scenario was validated on the simulator before it was administered to the applicants. (ES-302 D.3.a)

YES U NO

13. Forms ES-D-1 and ES-D-2 reflect any changes made to the scenario events or the expected operator actions as a result of scenario validation runs. (ES-302 D.3.c)

YES U NO

14. The examiners reviewed and discussed the scenarios together and immediately before beginning the simulator tests, the examiners should review the scenario events with the simulator operator and provided him or her with a copy of Form ES-D-1. (ES-302 D.3.e)

YES U NO

15. The examiners who administered the simulator test confer immediately after completing the scenario set to compare notes and to verify that each examiner observed his or her applicant perform the required number of transients and events in a manner sufficient to justify an evaluation of all the required competencies. (ES-302 D.3.m)

YES U NO

16. All of the examiners observations are consistent and mutually supportive. A performance deficiency shared by more than one applicant is appropriately documented for those applicants. (ES-302 D.3.m)

YES U NO

17. The examiners ask the simulator operator to record selected parameters on the facilitys safety parameter display system. The chief examiner retains the recordings as backup documentation. (ES-302 D.3.f)

YES U NO

18. The examiner in charge of each scenario arranged a communication system with the simulator operator to insert event malfunctions without cuing the applicants.

(ES-302 D.3.g)

YES U NO

19. The examiners have the simulator operator advance and mark with the date, time and initials any control room strip chart recorders useful in recreating the sequence of events. (ES-302 D.3.h)

YES U NO

20. Applicants are informed during the operating test briefing if time compression will be used during the simulator test. (ES-302 D.3.g)

Time compression not used. N/A

21. The examiners monitor conversations between the simulator operator and the applicants. (ES-302 D.3.j)

YES U NO

22. The examiners ask the simulator operator to provide and retain copies of the logs, charts, and other materials that can assist in documenting the applicants performance if they do not perform as expected. (ES-302 D.3.n)

YES U NO

23. The applicants were given sufficient time (normally about 5 minutes) to familiarize themselves with plant conditions before starting each simulator scenario. (Appendix E, Part E)

YES U NO C. Operating Test Documentation and Grading Activities Most Items Not Observed NOTE: Perform an independent review on a sampling of the grading of the operating tests and verify that the examiner's comments, if required by ES-303 D.3.b, appropriately support his or her recommendation.

1. The examiners met after completion of the simulator scenarios and compared notes to ensure documentation for applicants on the same operating crew is consistent. Operating errors, that involved more than one applicant in an operating crew, are noted by all involved evaluating examiners. (ES-303 C.3.a &

D.1.e)

YES NO

2. Any applicant generated or used material that contributes to an unsatisfactory performance evaluation is cross-referenced to the applicable deficiency and attached to the examination package. (ES-303 D.1.a)

N/A - all applicants passed.

3. The validity and technical accuracy of any performance-based questions that were not prescripted and any simulator test unexpected events or actions have been verified in accordance with the criteria of ES-301 D. (ES-303 D.1.b)

YES NO

4. Unsatisfactory JPM grading evaluations based solely on follow-up questions are documented in accordance with ES-303 D.3. (ES-303 D.3.b)

N/A - all applicants passed.

5. Review each documented simulator operating test performance deficiency and verify that unsatisfactory grades are, when practicable, assigned to no more than two rating factors for the same performance deficiency. (ES-303 D.1.e)

NOTE: While ES-303 D.1.e indicates that an applicants deficiency should not be assigned to more than two rating factors, a significant deficiency can be coded with additional rating factors if the error is relevant to each of the rating factors according to the criteria in ES-303 D.3.b.

YES U NO

6. Every unsatisfactory grade is supported with detailed documentation as discussed in ES-303. (ES-303 D.3.b)

N/A - all applicants passed.

7. An independent pass or fail recommendation is made by the chief examiner or designee and reviewed by the responsible regional supervisor. The responsible supervisor concurs on any recommendation to overturn the examiners results, and the specific reasons for this action are explained on Form ES-303-2. (ES-303 C.3.d)

YES U NO

8. All applicable examiner notes and documentation associated with the proposed denial are retained by the examiner until any denial becomes final. (ES-303 D.3.b)

N/A - all applicants passed.

D. Comments and Notes on the Operating Test A.7 JPM Level of Difficulty ES-301D.3.b states that Category B JPMs should, individually and as a group, have meaningful performance requirements that will provide a legitimate basis for evaluating the applicants understanding of and ability to safely operate the associated systems and the plant. Although assessing the level of difficulty is subjective, it appeared to this auditor that 2 Category B JPMs on the ANO exam only tested the applicants system-related abilities and knowledge at a minimum level:

  • ANO JPM B.1.d (Loss of DHR - Perform Gravity Feed to the RCS from BWST) only had 4 steps, two switch manipulations, and was completed by all applicants in 5 minutes or less.
  • ANO JPM B.1.f (Energize Bus A2 from Bus A4") only had 4 steps, three switch manipulations, and was completed by all applicants in 5 minutes or less.

Although these two JPMs were rated by this auditor as level of difficulty = 1.5, the Chief Examiner did not agree with this comment and considered these two JPMs to have a difficulty level of 2.0. In addition to these two JPMs , the bulk of the remaining Category B JPMs also appeared to be of below average difficulty. The level of difficulty of ANOs Category B JPMs could have been improved.

A.10 Number and Types of Scenario Events ES-301 D.4.d states that each scenario set must, at a minimum, require each applicant to respond to the types of evolutions, failures, and transients in the quantities identified for the applicants license level on Form ES-301-5, Transient and Event Checklist. One of the requirements of Form ES-301-5 is that each RO applicant perform a normal evolution. Although each ANO scenario set contained at least two normal evolutions, two RO applicants only acted in an assisting role during the normal evolutions, due to the crew rotation used and the design of the scenarios. This observation (made by the auditor after the exam) was acknowledged by the Chief Examiner, who agreed that the scenario sets/crew rotations should have been designed better, such that each RO was the primary performer of at least one normal evolution. Although two ROs were affected, this auditor and the Chief Examiner agreed that the scenarios, as administered, were sufficient to make appropriate licensing decisions.

E. Summary - Examination Performance Measures **

  • Timeliness of Reviews and Feedback
  • NRC Comments on Outline - days following NRC receipt
  • NRC Comments on Oper. Tests - days following NRC receipt
  • NRC Examination Management Approval - days prior to exam admin.
  • Examination Content and Changes
  • # of Alternate Path JPMs - NRC / Fac.
  • # of Bank/Mod/New JPMs & Scenarios - NRC / Fac.
  • # of Proposed Oper. Test Items Changed by NRC - Unacc. / Minor*
  • # of NRC-validated Test Items Changed by NRC - *
  • - explain reason(s) for any changes
3. Resources
  • # of Facility Hours Developing Proposed Oper. Tests - Hrs
  • # of NRC Hours reviewing Proposed Oper. Tests - Hrs
  • # of NRC Preparation Week Review Hours - Hrs
  • # of Facility Preparation Week Review Hours - Hrs
  • # of Facility Hours Resolving NRC Comments - Hrs Notes: * - This ES criteria may be answered in the affirmative based on review of examination reference materials, discussion with either the facility author(s) and/or NRC examiners, or reliance on facility and NRC reviewers initials. If the criteria are answered negatively then an explanation for the negative response will be provided in Section E including its basis.
    • - Since these measures may not be addressed by any specific ES guideline, the reviewer should as a minimum provide the information source and clarifying information as appropriate.
Retrieved from "https://kanterella.com/w/index.php?title=ML030370307&oldid=2676106"