ML093340500

Comments on Exam - NRC Comments on Draft Licensee-Developed Exam (Written & Operating Tests)

ML093340500
Person / Time
Site:	Peach Bottom
Issue date:	01/18/2010
From:	Fish T H Operations Branch I
To:	Exelon Generation Co
Hansell S
Shared Package
ML091380320	List: ML092090318 ML093340500 ML093370113 ML093620008 ML093620012 ML093620016 ML093620021 ML100500538 ML100810008
References
50-277/09-302, 50-278/09-302, TAC U01792
Download: ML093340500 (9)

Text

Peach Bottom License No.: DPR-44, DPR-56 Docket No.: 50-277, 50-278 Operator Licensing Exam Dates: 12/07-12/18/09 NRC Comments on Draft Licensee-Developed Exam (Written & Operating Tests) (Folder 2) Chief Examiner:

T. Fish TAC No. U01792 Report No.: 50-277/50-278/09-302 Public and Non-Sensitive NRR-079, SUNSI Review Complete ADAMS Package Accession No. ML091380320 G:\DRS\OPERATIONS BRANCH\BIXLER\EXAMMASTERFOLDERS\FY1 0\ 12-07 -09PB.DOC Rev. 10106/09 Peach Bottom License No.: DPR-44, DPR-56 Docket No.: 50-277, 50-278 Operator Licensing Exam Dates: 12/07-12/18/09 NRC Comments on Draft Licensee-Developed Exam (Written & Operating Tests) (Folder 2) Chief Examiner:

T. Fish TAC No. U01792 Report No.: 50-277/50-278/09-302 Public and Non-Sensitive NRR-079, SUNSI Review Complete ADAMS Package Accession No. ML091380320 G:\DRS\OPERATIONS BRANCH\BIXLER\EXAMMASTERFOLDERS\FY1 0\ 12-07 -09PB.DOC Rev. 10106/09 ES-401 Written Examination Review Worksheet Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job-Minutia #1 Back-Q= SRO B/M/N UtE/S Explanation Focus Dis!. Link units ward KIA Only 1 H 3 Y N S 2 F 3 Y N S 3 F 2 Y N S 4 H 3 Y B S 5 H 4 Y B S 6 H 3 Y N S 7 H 2 Y M S 8 F 2 Y N S Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 -5 (easy -difficult) rating scale (questions in the 2 -4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

• The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• The stem or distractors contain cues (I.e., clues, specific determiners, phrasing, length, etc).

• The answer choices are a collection of unrelated true/false statements.

• The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:

• The question is not linked to the job requirements (I.e., the question has a valid KIA but, as written, is not operational in content).

• The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).

• The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-<Jnly (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met). ES-401 Written Examination Review Worksheet Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) St,m H Pa"a' Job-Minutia #/ Back-Q= SRO B/M/N utE/S Explanation Focus . Link units ward KIA Only 1 H 3 I y

• 2 F 3 Y 3 F 2 E y N S 4 H 3 Y B S 5 H 4 Y B S 6 H 3 Y N S 7 H 2 Y M S 8 I F 2 Y N S Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 -5 (easy -difficult) rating scale (questions in the 2 -4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

• The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• The stem or distractors contain cues (I.e., clues, specific determiners, phrasing, length, etc).

• The answer choices are a collection of unrelated true/false statements.

• The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:

• The question is not linked to the job requirements (I.e., the question has a valid KIA but, as written, is not operational in content).

• The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).

• The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-<Jnly (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job-Minutia #1 Back-Q: SRO B/M/N utE IS Explanation Focus Dist. Link units ward KIA Only 9 H 3 Y N S 10 H 3 Y B S 11 H 3 Y N S 12 F 3 Y N S 13 H 2 Y M S 14 F 2 Y N S 15 H 3 Y B S 16 H 3 Y N S 17 H 3 Y B S 18 F 3 Y N S 19 H 3 Y M S 20 H 3 Y N S 21 F 2 Y N S 22 F 3 Y N S 23 H 3 Y M S 24 H 3 Y M S 25 F 3 Y B S 26 H 2 Y B S 27 F 3 Y N S 28 H 3 Y N S ES-401 2 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job-Minutia #1: I Q= SRO B/M/N utE IS Explanation Focus Dist. Link units KIA Only 9 H 3 Y N S 10 H 3 Y B S 11 H 3 Y N S 12 F 3 Y N S ltilij 2 Y M S 2 Y N S 3 B S =t 16 H 3 Y N S 17 H 3 Y B S 18 F 3 Y N S 19 H 3 Y M S 20 H 3 = Y N S jti F 2 Y N S F 3 Y N S R 23 Y M S 24 Y M 25 F 3 Y B S 26 H 2 Y B S 27 F 3 Y N S 28 H 3 Y N S

---

ES-401 3 Form ES*401*9 1. 2. LOK LOD Q# (F/H) (1-5) 29 H 2 2 30 F 31 F 3 F 32 3 33 H 2 H 34 2 35 H 2 36 F 3 37 4 H 38 H 4 H 39 3 40 F 2 41 H 2 42 H 2 F 43 3 44 H 3 45 F 2 H 46 2 47 F 3 48 H 3 3. Psychometric Flaws Partial Stem Cues T/F Credo Focus Dist. 4. Job Content Flaws Back-Job-Minutia #1 Link units ward 5. Other 6. 7. Q= SRO B/M/N UtE/S KIA Only Y B S Y M S Y B S Y M S Y B S Y N S Y B S Y B S Y M S Y M S Y B S Y B S Y N S Y N S Y M S Y N S Y N S Y B S Y N S Y B S 8. Explanation KIA met: Temperature is based on protecting various internal components, not based on limitation of containment shell KIA met. Valves listed are PCIS valves; PCIS is an ESF ES-401 3 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job-Minutia #1 Back-Q= SRO B/M/N UtE/S Explanation Focus Dist. Link units ward KIA Only 2 Y B E 30 F 2 Y M S 31 F 3 Y B S 32 F 3 Y M S 33 H 2 Y B S 34 H 2 Y N S 35 H tij Y B S 36 F Y B S 37 H 4 Y S 38 I H 4 Y M S I 39 H 3 Y B S 40 F 2 Y B S I 41 H 2 Y N S 42 H 2 Y N S 43 F 3 Y M S KIA met: Temperature is based on protecting various internal components, not based on limitation of containment shell 44 3 Y N S 45 F 2 I y N S H 2 Y B S F 3 I y N S H 3 Y B S KIA met. Valves listed are PCIS valves; PCIS is an ESF ES-401 4 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job-Minutia #1 Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 49 H 3 Y N S 50 F 2 Y B S 51 F 2 Y N S KIA met. Interpreting DW pressure includes knowing consequences of high DW pressure 52 H 2 Y B S 53 H 3 Y B S 54 H 3 Y N S 55 H 2 Y N S 56 H 2 Y N S 57 F 2 Y N S 58 H 3 Y B S 59 H 3 Y M S 60 H 3 Y B S 61 F 2 Y M S 62 H 2 Y N S 63 F 2 Y M S 64 F 2 Y B S 65 F 3 Y B S 66 H 2 Y B S 67 F 2 Y B S 68 F 3 Y N S ES-401 4 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5)

Partial Job-Minutia -Q= SRO B/M/N U/E/S Explanation Focus . Link KIA Only G:G 3 Y ,0 F 2 Y 1 F 2 Y N S KIA met. Interpreting DW pressure includes knowing consequences of high DW pressure 52 I H 2 B S 53 3 B I S 3 N S 2 N S 2 Y N S 57 2 Y N S 58 H 3 Y B S 59 H 3 Y M S 60 H Y B S 61 F I M S 62 H t+/- N S 63 Y M S 64 F 2 Y B S 65 F 3 Y B S 66 H 2 Y B S 67 F 2 Y B S 68 F 3 Y ON S ES-401 5 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job-Minutia #! Back-Q= SRO B/M/N UtE/S Explanation Focus Dist. Link units ward KIA Only 69 F 2 Y B S 70 F 3 X Y B E Added bullet to stem that no dose extension is required 71 H 2 Y B S 72 H 3 Y B S 73 F 3 Y N S 74 H 2 Y B S KIA met. Operators use the monitors to confirm CREV response.

75 F 2 Y N S 76 H 3 Y Y N S 77 H 3 Y Y N S 78 F 2 Y Y N S 79 H 2 Y Y N S 80 H 2 Y Y B S 81 H 3 Y Y M S 82 H 3 Y Y M S 83 H 2 Y Y N S 84 H 2 Y Y N S 85 F 2 Y N B U Not SRO-only because original part (2) answer choices were lAs for ROs. Revised procedural actions in (2) to be SRO specific.

86 H 2 Y Y B S 87 H 2 Y Y B S 88 H 2 Y Y N S ES-401 5 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job-Minutia #! Back-Q= SRO B/M/N UtE/S Explanation Focus Dist. Link units ward KIA Only 69 F 2 Y B S 70 F 3 X Y B E Added bullet to stem that no dose extension is required IR* 2 Y S 3 Y S 3 Y S 74 H 2 Y B KIA met. Operators use the monitors to confirm CREV response.

75 F 2 Y N S 76 H 3 Y Y 77 H 3 Y Y Y Y N Y Y N Y Y B 81 H 3 Y Y M S 82 H 3 Y Y M S 83 H 2 Y Y N S 84 H 2 jj Y Y N S j 85 F R y N B U Not SRO-only because original part (2) answer choices were lAs for ROs. Revised procedural actions in (2) to be SRO specific.

86 H Y Y B S 87 H 2 Y Y B S 88 H 2 Y Y N S ES-401 6 Form ES-401-9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Credo Dist. Partial Job-Link Minutia #1 units ward Q= SRO KIA Only 89 H 2 Y Y M S 90 H 3 Y Y B S 91 H 3 Y Y M S Part B of KIA met 92 H 3 Y Y M S 93 H 2 Y Y M S 94 H 2 Y Y B S 95 F 2 Y Y M E Modified*Justification" section of 401-5 form to explain plausibility of Cand D 96 H 2 Y N M U Not SRO: Discharge valve response is RO system level knowledge.

Replaced Q 97 H 2 Y Y M S 98 F 2 Y Y N S 99 F 2 Y Y N S 100 H 3 Y Y M S ES-401 6 Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job-Minutia #1 Back-Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward KIA Only 89 H 2 I y y M S 90 H 3 Y Y B S 91 H 3 Y Y M S Part B of KIA met 92 H 3 Y Y M S 93 H 2 Y Y M S 94 H 2 Y Y B S 95 F 2 Y Y M E Modified*

Justification" section of 401-5 form to explain plausibility of Cand D 96 H 2 Y N M U Not SRO: Discharge valve response is RO system level knowledge.

Replaced Q 97 H 2 y Y M S 98 F 2 Y Y N S 99 F 2 Y Y N S 100 H 3 Y Y M S To: Todd Fish, Chief Examiner From: Fred Bruns, Peach Bottom Regulatory Exam Author

Subject:

Revisions to Peach Bottom Operating Exam Materials for the December 2009 Initial NRC License Examination Date: November, 16,2009 The following materials were revised based on comments from the NRC Examination Team during the on-site exam preparation held November 2 -4,2009. A. Administrative JPMs: Modified critical versus non-critical steps for 1 RO and 1 SRO Conduct of Operations JPM. Clarified time-critical criteria for the RO Emergency Plan JPM. Modified the associated KiA for the SRO Radiation Control JPM (on the JPM and Form ES-301-1). Made additional editorial changes to various JPMs based on NRC examiner comments.

B. Control Room JPMs: Modified critical versus non-critical steps for 1 JPM. Made editorial changes to various JPMs based on NRC examiner comments.

C. In-plant JPMs: Expanded the scope of 1 JPM such that the task is performed on 2 components vice 1. Made editorial changes to various JPMs based on NRC examiner comments. -1 To: Todd Fish, Chief Examiner From: Fred Bruns, Peach Bottom Regulatory Exam Author

Subject:

Revisions to Peach Bottom Operating Exam Materials for the December 2009 Initial NRC License Examination Date: November, 16,2009 The following materials were revised based on comments from the NRC Examination Team during the on-site exam preparation held November 2 -4,2009. A. Administrative JPMs: 1. Modified critical versus non-critical steps for 1 RO and 1 SRO Conduct of Operations JPM. 2. Clarified time-critical criteria for the RO Emergency Plan JPM. 3. Modified the associated KiA for the SRO Radiation Control JPM (on the JPM and Form ES-301-1).

4. Made additional editorial changes to various JPMs based on NRC examiner comments.

B. Control Room JPMs: 1. Modified critical versus non-critical steps for 1 JPM. 2. Made editorial changes to various JPMs based on NRC examiner comments.

C. In-plant JPMs: 1. Expanded the scope of 1 JPM such that the task is performed on 2 components vice 1. 2. Made editorial changes to various JPMs based on NRC examiner comments.

D. Simulator scenarios: Scenario 1 -minor changes to Simulator Operator instructions. Scenario 2 minor scripting addition to Event 6. Scenario 3: Modified the setup conditions such that a system is available at the beginning of the scenario, but is unavailable later in the scenario (removed during Event 7). Modified scripting for Event 7 (see above). Scenario 4 -none. Revised Form ES-D-1 for scenario 3. D. Simulator scenarios:

1. Scenario 1 -minor changes to Simulator Operator instructions.

2. Scenario 2 minor scripting addition to Event 6. 3. Scenario 3: a. Modified the setup conditions such that a system is available at the beginning of the scenario, but is unavailable later in the scenario (removed during Event 7). b. Modified scripting for Event 7 (see above). 4. Scenario 4 -none. 5. Revised Form ES-D-1 for scenario 3.