ML062900360

From kanterella
Jump to navigation Jump to search
10-2006 - NRC Draft Exam Comments
ML062900360
Person / Time
Site: Cooper Entergy icon.png
Issue date: 11/10/2006
From:
NRC Region 4
To:
References
ES-401, ES-401-9
Download: ML062900360 (15)
v • d • e


Text

ES-401 COOPER NUCLEAR STATION-10-2006 Written Examination Form ES-401-9 Review Worksheet (password: lewisandclark) (TAC#: X02344 )

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 Y S (M) 2 F 2 X Y E (N) CONSIDER DELETING DISTRACTOR A AND REPLACING THE DISTRACTOR WITH ANOTHER DISTRACTOR THAT INVOLVES A SYSTEM ENGINEER. Changed 3 H 3 Y S (N) 4 F 2 US (N) GFES QUESTION OK Licensee Explained 5 F 2 Y S (B) 6 F 3 X Y E (B) CONSIDER REWRITING THE DISTRACTORS TO READ AS FOLLOWS : A. BOTH LOW-LOW SET VALVES CONTINUE TO CYCLE; B. BOTH LOW-LOW SET VALUES STOP CYCLING; C. THE LOW-LOW SET VALVE (71F) , WHICH CAN BE CONTROLLED FROM THE ADS ROOM CONTINUES TO CYCLE; D. THE LOW-LOW SET VALVE (71D) WHICH CANNOT BE CONTROLLED FROM THE ASD ROOM CONTINUES TO CYCLE. Changes Made 7 F 2 Y S (N) 8 F 2 X Y US (N) EITHER DISTRACTORS A OR D COULD BE SELECTED. THE EXPLANATION FOR DISTRACTOR B DISCUSSES A VALVE MV-81 WHICH IS NOT IN THE QUESTION. Distractor B changed 9 F 3 Y S (B)

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ One or more distractors is (are) not credible.

@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only

6. Based on the reviewer=s judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 10 F 2 Y S (N) 11 F 2 X Y US (B) DISTRACTOR B IS NOT CREDABLE A&C changed 12 H 3 Y S (N) 13 H 4 Y E (?) THE QUESTION HAS TWO K/As, WHICH ONE IS APPLICABLE? B THE QUESTION DOES NOT LIST ITS SOURCE, i.e., BANK, NEW, ETC.

14 H 3 Y S (N) 15 H 3 X Y US (N) DISTRACTORS A & C CAN BOTH BE CONSIDERED TO BE A CORRECT ANSWER. A changed 16 F 2 Y S (N) 17 F 2 Y S (N) 18 F 2 Y US (B) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE (EOP7A) IS DELETED OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION AND A POTENTIAL CUE FOR OTHER QUESTIONS. Revised Notes,Cautions, Entry Conds. Deleted 19 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)N 20 F 3 Y S (B) 21 H 2 Y S (N)

Y (B) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE 22 F 3 US (EOP3A) IS DELETED OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION AND A POTENTIAL CUE FOR OTHER QUESTIONS. Ref. Removed 23 F 3 Y S (N) 24 H 3 Y S (N) 25 F 3 Y S (N) 26 F 2 Y S (B) 27 F 2 Y US (N) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE (EOP5A) IS DELETED OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION. Discussed-Multiple Alt. Paths 28 F 2 Y S (N)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 29 H 3 Y S (N)

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 30 F 2 Y Y US (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED).

SRO ONLY QUESTION. N-Ros trained to recognize LCOs 31 H 3 X N US (N) THE QUESTION DOES NOT MATCH THE K/A. PART B OF THE K/A, USE OF PROCEDURES, IS NOT COVERED. QUESTION STEM PROVIDES AN ANSWER CUE. Rewritten N

32 F 2 Y E

(?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED) 33 F 3  ? E (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED).

WHY ARE THERE 2 K/AS??B 211000.K2.01 34 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)B 35 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)N 36 F 3 Y S (B) 37 F 3 Y S (B) 38 H 3 Y S (N) 39 F 2 Y S (N) 40 H 3 Y S (B) 41 F 2 Y S (B) 42 F 2 Y S (B)

Y 43 H 2 S (N) 44 F 2 Y S (B) 45 H 3 Y S (B) 46 H 2 Y Y US (B) SRO ONLY QUESTION Modified Stem 47 F 3 Y S (B) 48 F 2 Y S (B)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 49 F 2 Y S (B) 50 F 2 Y S (N) 51 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN B 52 F 3 N US (N) THE QUESTION DOES NOT MATCH THE K/A. Discussed OK 53 H 3 N US (N) THE QUESTION DOES NOT MATCH THE K/A. Discussed OK 54 F 2 Y S (N) 55 F 2 Y S (B) 56 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 57 H 3 N US (N) THE QUESTION DOES NOT MATCH THE K/A IN THAT THE PROCEDURES TO CORRECT, CONTROL, AND MITIGATE THE IMPACT ARE NOT IDENTIFIED.SAT after discussed w/ lic.

58 F 2 Y S (B) 59 H 2 N US (B) THE QUESTION DOES NOT MATCH THE K/A. CHANGE THE SENTENCE IN THE STEM TO READ: WITHOUT REFERENCE TO A PROCEDURE, WHICH OF THE FOLLOWING ACTIONS WOULD YOU, THE OPERATOR, TAKE TO PREVENT OVERPRESSURIZATION OF THE REACTOR WATER CLEANUP SYSTEM PIPING? Chgd Stem 60 F 2 X X Y US (B) DISTRACTORS B THROUGH D ARE NOT CREDIBLE Chgd Stem 61 H 3 Y Y US (N) SRO ONLY QUESTION. OK RO activity @ CNS 62 H 3 Y S (B) 63 H 2 Y S (N) 64 H 3 Y S (N) 65 H 3 Y E (N) THE EXPLANATION FOR DISTRACTOR C STATES THAT IT IS INCORRECT. Corrected 66 H 3 Y S (M) 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 67 H 4 Y ?S (B) DO NOT UNDERSTAND THIS QUESTION - NEED TO HAVE EXPLAINED BY LICENSEE. DO NOT UNDERSTAND RELATIONSHIP WITH EOP-3A. Removed Ref Mtl 68 H 3 Y S (B) 69 F 3 Y S (B) 70 F 3 Y S (B) 71 F 3 Y S (B) 72 F 2 Y S (B) 73 F 2 X Y US (?) SOURCE OF QUESTION IS UNKNOWN. IT APPEARS THAT DISTRACTORS B AND D COULD BE CORRECT. B Rewritten 74 H 3 Y S (M) 75 H 3 Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 76 H 3 Y Y S (N) 77 H 3 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE TWO K/As - WHICH ONE IS APPLICABLE? B 2.1.33 78 F 3 Y Y S (N) 79 F 2 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 80 H 3 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 81 H 2 Y Y S (B) 82 F 2 Y Y S (N) 83 H 3 Y Y S (N) 84 F 2 Y Y S (N) 85 F 3 Y Y S (N) 86 F 2 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Ste Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation m Dist. Link units ward K/A Only Foc us 87 H 3 X N Y US (B) THE QUESTION DOES NOT MATCH THE K/A IN THAT THE PROCEDURES TO CORRECT, CONTROL, AND MITIGATE THE IMPACT ARE NOT IDENTIFIED. IT APPEARS THAT DISTRACTORS C AND D COULD BE CORRECT. Chgd Stem 88 H 3 Y Y S (B) 89 H 2 Y Y S (N) 90 H 3 Y Y S (M) 91 F 2 Y Y US (N) QUESTION IS A SIMPLE LOOK-UP QUESTION WHEN TS 5.5.8 IS PROVIDED TO APPLICANT. Rewritten 92 H 2 X Y Y US (B) DISTRACTORS A AND B COULD BOTH BE CORRECT. WHAT DOES NON-TRM MEAN? Discussed OK 93 H 4 Y Y S (N) 94 H 2 X Y Y US (?) SOURCE OF QUESTION IS UNKNOWN. IT APPEARS THAT DISTRACTORS A AND C COULD BOTH BE CORRECT. N Rewritten 95 H 3 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE TWO K/As ? WHICH ONE IS APPLICABLE? B 2.2.18

(?) SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE THREE 96 F 3 Y Y E K/As ? WHICH ONE IS APPLICABLE? B 2.2.32 97 F 2 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 98 F 2 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 99 F 3 Y N US (?) SOURCE OF QUESTION IS UNKNOWN. NOT A SRO ONLY QUESTION. N OK 10CFR55.43 (2) & (5) 100 H 3 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B SRO TOTALS

Sat: 11 Unsat: 5 Ed: 9 H: 14 (56%)

F: 11 (44%)

Bank: 4 13 Mod: 1 New: 9 11 Unk: 11 0 RO TOTALS Sat: 45 Unsat: 17 Ed: 13 H: 27 (36%)

F: 48 (64%)

Bank: 29 37 Mod: 3 New: 31 35 Unk: 12 0

S:\DRS\OB\OB-Only\EXAMS\CNS\Exam Review\APPENDIX E ADMIN.wpd APPENDIX E - REGION IV OPERATING TEST ADMINISTRATIVE JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link SA1 S 3 S SA2 S 2 X S ONLY ONE CRITICAL STEP.

SA3 S 2 S SA4 S 3 X S ONLY ONE CRITICAL STEP SA5 S 2 X S ONLY ONE CRITICAL STEP RA1 S 2 S RA2 S 2 E STEP THREE OF THE PERFORMANCE CHECKLIST SAYS TO DETERMINE IF THE DELTA T VALUES ARE WITHIN + 5 DEGREES F. THE PUMP START CHECKLIST STEPS 10 THROUGH 14 HAVE A TOLERANCE OF + 4 DEGREES F. WHICH IS CORRECT?

RA3 S 2 U ATTACHMENT 1 HPCI DATA SHEET SHOWS HPCI-MO-19 INITIAL STROKE TIME AS 18.7 SEC AND THE RETEST STROKE TIME AS 16.4 SEC. THESE VALUES ARE WITHIN THE IST ACCEPTABILITY RANGE. THEREFORE, THIS VALVE IS OPERABLE. VALVE HPCI-MO-58 EXCEEDED THE IST ACCEPTABLE RANGE ON BOTH STROKES WHICH MAKES THIS VALVE INOPERABLE. THIS IS INCONSISTENT WITH THE GOAL OF THE JPM AND NEEDS TO BE RESOLVED.

RA4 S 2 U THE JPM'S ATTACHMENT 3 SAYS THAT THE ATTACHMENT MAY BE GIVEN TO THE CANDIDATE, HOWEVER, ATTACHMENT 3 HAS THE ANSWERS DOCUMENTED ON IT.

- Page 1 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 2 -

S:\DRS\OB\OB-Only\EXAMS\CNS\Exam Review\APPENDIX E JPMs.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link a D 2 S PROCEDURE 2.2.74 UNAVAILABLE FOR REVIEW.

b D 3 S PROCEDURE 6.HPCI.103 UNAVAILABLE FOR REVIEW.

c D 3 E THE JPM TITLE IS ADMINISTRATIVE JPM FOR OPERATIONS, BUT THIS IS NOT AN ADMINISTRATIVE JPM. STEP 10 HAS AN INCOMPLETE NUMBER FOR THE FLOW CONTROLLER. INSTEAD OF "RCIC-FIC", IT SHOULD BE "RCIC-FIC-91". ADDITIONALLY, STEP 10 HAS A "NOTE" THAT DISCUSSES A PROBLEM WITH THE RCIC CONTROLLER, IS THIS THE ALTERNATE PATH?

d D 2 S PROCEDURE 2.2.77 UNAVAILABLE FOR REVIEW.

e D E THE JPM TITLE IS ADMINISTRATIVE JPM FOR OPERATIONS, BUT THIS IS NOT AN ADMINISTRATIVE JP

M. PROCEDURE

2.2.69.3 UNAVAILABLE FOR REVIEW. iT IS NOT CLEAR WHERE THE ALTERNATE PATH IS SINCE THE TITLE INCLUDES A "FAILED 2/3 CORE COVERAGE PERMISSIVE". STEP 6 HAS NUMBER OF QUESTION MARKS - WHY? STEP 8 HAS THE CANDIDATE START THE RHR PUMP, BUT ONLY CHECK THAT THE RED LIGHT IS ON -

WHAT ABOUT OBSERVING MOTOR CURRENT? STEP 9 HAS A TERM MISSING AS EVIDENCED BY QUESTION MARKS.

f D 2 S PROCEDURE 2.2.14 UNAVAILABLE FOR REVIEW. SIMPLE JPM.

g D 3 E THE JPM TITLE IS IN PLANT JPM FOR OPERATIONS, BUT THIS IS NOT AN IN PLANT JPM.

STEP 10 IS INCONSISTENT WITH STEP 4.13 BECAUSE IT DOES MENTION THE PRESSING OF FUNCTION KEY F-4. THE JPM HAS THE CANDIDATE COMPLETE STEPS 4.5 THROUGH 4.21 OF THE PROCEDURE, HOWEVER, STEPS 22 AND 23 OF THE JPM ARE INCONSISTENT WITH PROCEDURE STEP 4.21.

h D 3 U JPM STEPS 2, 3, AND 4 HAS THE OPERATOR PERFORMING DUTIES THAT ARE NOT IN PROCEDURE 5.4FIRE. IN GENERAL, THE JPM IS INCONSISTENT WITH THE PROCEDURE.

i S 3 S PROCEDURE 5.3ALT-STRATEGY COULD NOT BE OPENED BECAUSE IT HAD A SECURITY HOLD ON IT.

J D 3 E iIT IS NOT CLEAR HOW YOU GO FROM STEP 9 IN THE JPM TO STEP 11. THE PROCEDURE GOES FROM STEP 5.1.11 TO STEP 7 AND DOESN'T RETEST THE SCRAM UNTIL STEP 7.3.9.

HOW YOU GO FROM JPM STEP 10 TO PROCEDURE STEP 7.3.9 NEEDS CLARIFICATION.

k S 2 S

- Page 1 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 2 -

S:\DRS\OB\OB-Only\EXAMS\CNS\Exam Review\APPEND IX N.wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen 1 2 3 4 5 6 7 8 9 10 Explanation (See below for instructions)

Set ES TS Crit IC Pred TL L/C Eff U/E/S X X U NEEDS TIME LINE AND IDENTIFICATION OF CRITICAL STEPS.

1 2 X X U NEEDS TIME LINE AND IDENTIFICATION OF CRITICAL STEPS.

3 X S NEEDS TIME LINE 4 X S NEEDS TIME LINE Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. ES: ES-301 checklists 4, 5, & 6 satisfied.
2. TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.
3. Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
4. IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
5. Pred: Scenario sequence and other factors avoid predictability issues.
6. TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
7. L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
8. Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
9. Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
10. Provide a brief description of problem in the explanation column

- Page 1 -

Retrieved from "https://kanterella.com/w/index.php?title=ML062900360&oldid=2185213"