ML102150100

From kanterella
Jump to navigation Jump to search

and Peach Bottom Lsro - NRC Comments on Draft Licensee-Developed Exam (Written & Operating Tests) (Folder 2)
ML102150100
Person / Time
Site: Peach Bottom, Limerick  Constellation icon.png
Issue date: 05/12/2010
From:
Exelon Generation Co
To:
Operations Branch I
HANSELL S
Shared Package
ML092470056 List:
References
TAC U01795, TAC U01796
Download: ML102150100 (10)
v • d • e


Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #/ Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 1 h 2 s 2 h 2 s/e Will this be noticeable after several days sId?

3 h 2 s 4 h 2 s 5 f 2 s 6 h 2 s 7 f 2 s 8 f 2 s Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
  • The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).
    • The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

4.

  • One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

Check the appropriate box if a job content error is identified:

    • The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

5.

  • The question requires reverse logic or application compared to the job requirements.

Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer'S judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

H

8. At a minimum, explain any "U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Oist. Link units ward KIA Only 9 f 2 s 10 h 2 s 11 h 2 s 12 h 3 s 13 h 2 x E Notify CR of loss of power to RMS, assembly move may continue 14 h 2 s

\~':\t 15 f 2 s 16 f 2 s Instructions I

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

4. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
5. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
6. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

4.

  • One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

Check the appropriate box if a job content error is identified:

    • The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

  • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

5.

  • The question requires reverse logic or application compared to the job requirements.

Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section O.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any 'U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOO (F/H) (1-5) Stem Cues TlF Credo Partial Job- Minutia #I Back- 0= SRO B/M/N utEtS Explanation Focus Oist. Link units ward KIA Only 17 f 2 X ~ Relevance to LSRO?

18 f 2 s 19 h 2 s 20 h 2 s 21 h 2 X U A,O too implausible. Maybe hoist motion interlock? RWCU isolate?

22 h 3 s 23 h 2 s 24 f 2 s Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

7. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

S. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 4 range are acceptable).

9. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
    • The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
    • The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

    • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (Blank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
7. Based on the reviewer's jUdgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfaclory?

n

8. At a minimum, explain any *U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES*401 Written Examination Review Worksheet Form ES*401*9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 25 f 2 s 26 h 2 s 27 h 2 s 28 h 2 s 29 h 2 s 30 h 2 s 31 f 2 s Bes1e illdbSb iel S8~Q*" "'-'a Q:;Ig~ .99W' Re,*, te ePl~ pcsitlcl1 32 f 1- x

$ FeSJ'eRSQ &9 'ili.",ep.t:u;hiii , Q'"

1\

Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

10. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
11. Enter the level of difficulty (LOD) of each question using a 1 ~ 5 (easy difficult) rating scale (questions in the 2 4 range are acceptable).
12. Check the appropriate box if a psychometric flaw is identified:
  • The stem lacks sufficient focus to elicit the correct answer (e.g .* unclear intent, more information is needed, or too much needless information).
    • The stem or distractors contain cues (Le., clues, specific determiners. phrasing, length. etc).

The answer choices are a collection of unrelated truelfalse statements.

    • The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:
    • The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).

    • The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified. or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer'S judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

B. At a minimum, explain any *un ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

r---

Q# LOK LOD (F/H) (1-5) Stem Cues TfF Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N UlE/S Explanation Focus Oist. link units ward KIA Only 33 h 2 s 34 f 2 s 35 f 1 u/c This one is basic rad worker training 36 h 2 s 37 f 2 s 38 h 2 s fLe c' 39 h 2 s 40 h 2 s Instructions

[Refer to Section D of ES-4Q1 and Appendix B for additional information regarding each of the following concepts.]

13. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
14. Enter the level of difficulty (LOD) of each question using a 1 5 (easy - difficult) rating scale (questions in the 2 4 range are acceptable).
15. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, speCific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

  • The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
  • One or more distractors is (are) partially correct (e.g .. if the applicant can make unstated assumptions that are not contradicted by stem).
4. Check the appropriate box if a job content error is identified:
    • The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (Bjank, (M)odified, or (N)ew. Check that (M}odified questions meet criteria of ES-401 Section 0.2.f.
7. Based on the reviewer's jUdgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E}ditorial enhancement, or (S)atisfactQry?

B. At a minimum, explain any "U* ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES*401 Written Examination Review Worksheet Form ES*401*9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. B.

r Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N UlE/S Explanation Focus Dis\' Link units ward KIA Only 41 h 2 s 42 h 3 x x U Do not give LCO in stem, change "B" to~ SDC sUbsytem7'

.j;\lolliHEji.* f"t .., '1 d~ . > .,. Yl-ti")

43 h 2 x oJ Relevance for LSRO 44 h 2 s 45 h 2 s 46 h 2 s 47 h 3 s 48 h 2 s

Instructions

[Refer to Section D of ES*401 and Appendix B for additional information regarding each of the following concepts.}

16. Enter the ievel of knowiedge (LOK) of each question as either (F)undamentai or (H)igher cognitive level.
17. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
18. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible: single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (Le., the question has a valid KiA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KiA and those that are designated SRO-only (KiA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E}ditorial enhancement, or (S}atisfactory?
8. At a minimum. explain any *U" ratings (e.g .. how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9 8.

Link I

Partial I JOb-I Minutia #/

units Explanation 50 2 s

6/23/2010 LGS 1 PBAPS LSRO NRC PREP WEEK ISSUES AND RESOLUTION ITEM ISSUE RESOLUTION I

WRITTEN EXAM 0#2 Discussion regarding would change be sufficient No Change: After discussion 10# 23088 that candidate would not pick stays the same? question accepted as written 0#13 NRC Suggested changing answer include action Enhancement: Change made 10 # 23104 to notify MCR and accepted 0#17 NRC considered not relevant to LSRO (indication No Change: Reviewed ST-6-047 10# 23062 of CR WID in MCR) 370-1 step 4.5.7 which identifies LSRO duty in MCR relatl3d to question. After discussion question accepted as written 0#21 Two Oistractors not consider plausible: Originally UNSAT: Change made 10# 23095

  • Fuel Pool water level will lower and accepted
  • WRNM indication will go up not NRC Suggested changes to distractors:
  • Fuel Pool System will isolate
  • Refueling bridge main hoist will be inoperable 0#32 Considered related Basic Electrical Safety worker No Change: After discussion, 10 # 23111 requirements accepted as written 0# 35 Considered L01, Basis Rad Worker topic Originally UNSAT: Question

. 10 # 23107 rewritten to include second Column, and additional differentiation, Change made and accepted 0#42 Last line in Stem considered leading. Distractor Originally UNSAT: Change made 10 # 23092 not plausible: and accepted

  • RHR HXCHRS not part of SOC not NRC Suggested:
  • Take out last line of stem.
  • Change distractor to "SOC may be out of service for up to 12 hrs" I

I Q#43 Not operationally Valid. Originally UNSAT: Change made

  • 10 # 23115
  • conditions are not refueling related and accepted

. Suggested Change conditions to refuel related I

lof 3

LGS JPMS 2084 FHD turnover: Contained Typographical error fixed typographical error 2701 Calc stay time: NRC suggested Enhancement: Values for "worker 2" challged change values for "worker 2" to to more restrictive changed more restrictive 2035 Hoist Jam: Discussed adding Enhancement: Cover updated "LSRO performs as RPO to cover" 2083 Hoist Tube Hang Up: NRC Enhancement: Changes made suggested:

  • Add direction for evaluation to show candidate attachments at appropriate steps
  • Clearly note when changing between RPO and FHD roles 2029 Rod Test Box: Discussed adding Enhancement: Change made note allowing JPM to start in AER (Evaluator would ask candidate where they would get Box) 2082 FP makeup with Fire Water: Typographical error fixed Contained typographical error 2 of 3

PBAPS JPMS 3064 Fuel Movement, Fire: Will a Enhancement: Clarified the cueing d

candidate decide to take the Fuel announcement to be given AFTER Assembly back to the original

  • located above the 'MOVED TO' location for the

~~~

  • ____~~I~oc_a~t~io~n_in_s_t_ea_d~cl move? __f_in~is~h_i_ng__

th~e____~f=u_e~I.----

3065 I Fuel Support Piece Removal, Enhancement: Clarified 'STEPS' and Fault Lockout: Make it clear for 'STANDARDS' to make it clear that the FHD each step that the applicant is was directing the RPO to perform the required directing the RPO

  • actions.

~~~-----,~~~------~----~~~--~~~~

3066 Fuel Movement, Loss of SBGT: Enhancement:

Change JPM from a 'simulate' to a -Changed the position of the candidate from

'perform'. FHD to RPO (will actually move the Refuel Platform).

-Changed the critical steps to be the RPO steps (not the FH D actions).

-Changed the Fuel move from Fuel Pool to Fuel Pool (instead of core); RPO will grapple Dummy Fuel Bundle and raise the bundle (then lower and re-seat the bundle).

-FHD will ask the RPO for recommendation on what to do once the report of the loss SBGT is made.

-Changed Task reference to reflect the change in position from FHD to RPO.

Allow follow-up questions to be Enhancement: ..IPM set up to give follow-up given AFTER leaving the Refueling questions at any time AFTER the performance Platform. portion (as long as candidate is sequestered).

3068 Drywell to Reactor Well Bellows Enhancement: Added the alarm horn noise as Failure: Include alarm horn noise as part of the cue.

part of cue.

  • How do you expect the applicant to Enhancement: Added the initiating condition of contact the control room, etc. having radio contact with the control room/NMD Reactor Services Foreman.

Change Cues on Fuel Pool level to Enhancement: Changed cues on Fuel Pool be given when applicant asks level to be given ONLY after candidate asks for for/checks Fuel Pool Level (note / checks the water level in the Fuel Pool.

this is Fuel Pool Level I not Skimmer Surge Tank level) 3 of 3

Retrieved from "https://kanterella.com/w/index.php?title=ML102150100&oldid=2556561"