ML062900360: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
 
(Created page by program invented by StriderTol)
 
(One intermediate revision by the same user not shown)
Line 2: Line 2:
| number = ML062900360
| number = ML062900360
| issue date = 11/10/2006
| issue date = 11/10/2006
| title = Cooper Nuclear Station - 10-2006 - NRC Draft Exam Comments
| title = 10-2006 - NRC Draft Exam Comments
| author name =  
| author name =  
| author affiliation = NRC/RGN-IV
| author affiliation = NRC/RGN-IV
Line 16: Line 16:


=Text=
=Text=
{{#Wiki_filter:ES-401 COOPER NUCLEAR STATION-10-2006 Written Examination Form ES-401-9 Review Worksheet (password: lewisandclark) (TAC#: X02344 )  
{{#Wiki_filter:ES-401                                               COOPER NUCLEAR STATION-10-2006 Written Examination                                                                 Form ES-401-9 Review Worksheet (password: lewisandclark) (TAC#: X02344 )
: 3. Psychometric Flaws  
: 1. 2.            3. Psychometric Flaws               4. Job Content Flaws         5. Other     6.                                        7.
: 4. Job Content Flaws  
Q#     LOK LOD (F/H) (1-5)     Stem Cues         T/F  Cred. Partial Job- Minutia #/ Back-               Q= SRO U/E/S                                       Explanation Focus                    Dist.           Link            units ward      K/A Only 1       H       3                                                                             Y             S   (M) 2       F       2                               X                                             Y             E   (N) CONSIDER DELETING DISTRACTOR A AND REPLACING THE DISTRACTOR WITH ANOTHER DISTRACTOR THAT INVOLVES A SYSTEM ENGINEER. Changed 3       H       3                                                                             Y             S   (N) 4       F       2                                                                                           US  (N) GFES QUESTION OK Licensee Explained 5       F       2                                                                             Y             S   (B) 6       F       3                               X                                             Y             E   (B) CONSIDER REWRITING THE DISTRACTORS TO READ AS FOLLOWS : A. BOTH LOW-LOW SET VALVES CONTINUE TO CYCLE; B. BOTH LOW-LOW SET VALUES STOP CYCLING; C. THE LOW-LOW SET VALVE (71F) , WHICH CAN BE CONTROLLED FROM THE ADS ROOM CONTINUES TO CYCLE; D. THE LOW-LOW SET VALVE (71D) WHICH CANNOT BE CONTROLLED FROM THE ASD ROOM CONTINUES TO CYCLE. Changes Made 7      F      2                                                                              Y              S  (N) 8      F      2                              X                                              Y            US  (N) EITHER DISTRACTORS A OR D COULD BE SELECTED. THE EXPLANATION FOR DISTRACTOR B DISCUSSES A VALVE MV-81 WHICH IS NOT IN THE QUESTION. Distractor B changed 9      F      3                                                                              Y              S  (B)
: 5. Other   Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus  Cues T/F  Cred.Dist. Partial Job-Link Minutia #/ units Back-ward  Q= K/A SROOnly 6. U/E/S 7. Explanation 1 H 3                   Y   S (M) 2 F 2       X           Y   E (N) CONSIDER DELETING DISTRACTOR A AND REPLACING THE DISTRACTOR WITH ANOTHER DISTRACTOR THAT INVOLVES A SYSTEM ENGINEER.
Instructions
Changed 3 H 3                   Y   S (N)   4 F 2                       U S  (N) GFES QUESTION OK Licensee Explained 5 F 2                   Y   S (B) 6 F 3       X           Y   E (B) CONSIDER REWRITING THE DISTRACTORS TO READ AS FOLLOWS : A.
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
BOTH LOW-LOW SET VALVES CONTINUE TO CYCLE; B.
: 1.        Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
BOTH LOW-LOW SET VALUES STOP CYCLING; C. THE LOW-LOW SET VALVE (71F) , WHICH CAN BE CONTROLLED FROM THE ADS ROOM CONTINUES TO CYCLE; D. THE LOW-LOW SET  
: 2.        Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
: 3.        Check the appropriate box if a psychometric flaw is identified:
            @          The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
            @          The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
            @          The answer choices are a collection of unrelated true/false statements.
            @          One or more distractors is (are) not credible.
            @          One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
: 4.        Check the appropriate box if a job content error is identified:
            @          The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
            @          The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
            @          The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
            @          The question requires reverse logic or application compared to the job requirements.
: 5.        Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
: 1. 2.        3. Psychometric Flaws          4. Job Content Flaws  5. Other 6.                  7.
Q# LOK LOD (F/H) (1-5) Stem Cues    T/F    Cred. Partial Job- Minutia #/ Back-    Q= SRO U/E/S              Explanation Focus                Dist.        Link          units ward K/A Only
: 6. Based on the reviewer=s judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 7. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).


VALVE (71D) WHICH CANNOT BE CONTROLLED FROM THE ASD ROOM CONTINUES TO CYCLE.
ES-401                                                                          2                                                            Form ES-401-9
Changes Made
: 1. 2.      3. Psychometric Flaws          4. Job Content Flaws  5. Other 6.                              7.
Q#  LOK LOD (F/H) (1-5) Stem Cues T/F      Cred. Partial Job- Minutia #/ Back-    Q= SRO U/E/S                          Explanation Focus              Dist.        Link          units ward K/A Only 10    F    2                                                              Y        S (N) 11 F    2                        X                                    Y        US (B) DISTRACTOR B IS NOT CREDABLE A&C changed 12    H    3                                                              Y        S (N) 13    H    4                                                              Y        E (?) THE QUESTION HAS TWO K/As, WHICH ONE IS APPLICABLE? B THE QUESTION DOES NOT LIST ITS SOURCE, i.e., BANK, NEW, ETC.
14    H    3                                                              Y        S (N) 15    H    3                        X                                    Y        US (N) DISTRACTORS A & C CAN BOTH BE CONSIDERED TO BE A CORRECT ANSWER. A changed 16    F    2                                                              Y        S (N) 17    F    2                                                              Y        S (N) 18    F    2                                                              Y        US (B) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE (EOP7A) IS DELETED OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION AND A POTENTIAL CUE FOR OTHER QUESTIONS. Revised Notes,Cautions, Entry Conds. Deleted 19    F    2                                                              Y        E (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)N 20    F    3                                                              Y        S (B) 21    H    2                                                              Y        S (N)
Y          (B) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE 22    F    3                                                                      US (EOP3A) IS DELETED OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION AND A POTENTIAL CUE FOR OTHER QUESTIONS. Ref. Removed 23    F    3                                                              Y        S (N) 24    H    3                                                              Y        S (N) 25    F    3                                                              Y        S (N) 26    F    2                                                              Y        S (B) 27    F    2                                                              Y        US (N) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE (EOP5A) IS DELETED OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION. Discussed-Multiple Alt. Paths 28    F    2                                                              Y        S (N)
: 1. 2.      3. Psychometric Flaws          4. Job Content Flaws  5. Other 6.        7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F      Cred. Partial Job- Minutia #/ Back-    Q= SRO U/E/S    Explanation Focus              Dist.        Link          units ward K/A Only 29  H    3                                                              Y        S  (N)


7 F 2                   Y   S (N) 8  F 2       X            Y   U S (N) EITHER DISTRACTOR'S "A" OR "D" COULD BE SELECTED. THE EXPLANATION FOR DISTRACTOR "B" DISCUSSES A VALVE MV-81 
ES-401                                                                          2                                                            Form ES-401-9
: 1. 2.      3. Psychometric Flaws          4. Job Content Flaws  5. Other  6.                              7.
Q#  LOK LOD (F/H) (1-5) Stem Cues T/F      Cred. Partial Job- Minutia #/ Back-    Q= SRO    U/E/S                        Explanation Focus              Dist.        Link          units ward K/A Only 30    F    2                                                              Y    Y  US  (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED).
SRO ONLY QUESTION. N-Ros trained to recognize LCOs 31    H    3            X                                                N        US  (N) THE QUESTION DOES NOT MATCH THE K/A. PART B OF THE K/A, USE OF PROCEDURES, IS NOT COVERED. QUESTION STEM PROVIDES AN ANSWER CUE. Rewritten N
32    F     2                                                             Y         E
(?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED) 33    F    3                                                              ?          E  (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED).
WHY ARE THERE 2 K/AS??B 211000.K2.01 34    F    2                                                              Y          E  (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)B 35    F    2                                                              Y          E  (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)N 36    F    3                                                              Y          S  (B) 37    F    3                                                              Y          S  (B) 38    H    3                                                              Y          S  (N) 39    F    2                                                              Y          S   (N) 40    H    3                                                              Y          S  (B) 41    F     2                                                             Y          S  (B) 42    F    2                                                              Y          S  (B)
Y 43    H    2                                                                        S   (N) 44    F    2                                                              Y          S  (B) 45    H    3                                                              Y          S   (B) 46    H    2                                                              Y    Y  US  (B) SRO ONLY QUESTION Modified Stem 47    F    3                                                              Y          S  (B) 48    F    2                                                              Y          S  (B)
: 1. 2.      3. Psychometric Flaws          4. Job Content Flaws  5. Other  6.      7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F      Cred. Partial Job- Minutia #/ Back-    Q= SRO    U/E/S Explanation Focus              Dist.         Link          units ward K/A Only 34 of 34                  NUREG-1021, Draft Revision 9


WHICH IS NOT IN THE QUESTION.
ES-401                                                                         2                                                             Form ES-401-9
Distractor B changed 9  F  3                    Y    S  (B)  Instructions
: 1. 2.       3. Psychometric Flaws         4. Job Content Flaws   5. Other 6.                            7.
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
Q#   LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO U/E/S                        Explanation Focus              Dist.        Link          units ward K/A Only 49    F    2                                                              Y        S (B) 50    F    2                                                              Y        S (N) 51    F    2                                                              Y        E (?) SOURCE OF QUESTION IS UNKNOWN B 52    F    3                                                              N        US (N) THE QUESTION DOES NOT MATCH THE K/A. Discussed OK 53    H    3                                                              N        US (N) THE QUESTION DOES NOT MATCH THE K/A. Discussed OK 54    F    2                                                              Y        S (N) 55    F    2                                                              Y        S (B) 56    F    2                                                              Y        E (?) SOURCE OF QUESTION IS UNKNOWN. B 57    H    3                                                              N        US (N) THE QUESTION DOES NOT MATCH THE K/A IN THAT THE PROCEDURES TO CORRECT, CONTROL, AND MITIGATE THE IMPACT ARE NOT IDENTIFIED.SAT after discussed w/ lic.
: 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
58    F    2                                                              Y        S (B) 59    H    2                                                              N        US (B) THE QUESTION DOES NOT MATCH THE K/A. CHANGE THE SENTENCE IN THE STEM TO READ: WITHOUT REFERENCE TO A PROCEDURE, WHICH OF THE FOLLOWING ACTIONS WOULD YOU, THE OPERATOR, TAKE TO PREVENT OVERPRESSURIZATION OF THE REACTOR WATER CLEANUP SYSTEM PIPING? Chgd Stem 60    F    2            X          X                                    Y        US (B) DISTRACTORS B THROUGH D ARE NOT CREDIBLE Chgd Stem 61 H    3                                                              Y      Y US (N) SRO ONLY QUESTION. OK RO activity @ CNS 62    H    3                                                              Y        S  (B) 63    H    2                                                              Y        S  (N) 64    H    3                                                              Y        S  (N) 65    H    3                                                              Y        E  (N) THE EXPLANATION FOR DISTRACTOR C STATES THAT IT IS INCORRECT. Corrected 66    H    3                                                              Y        S (M) 34 of 34                                        NUREG-1021, Draft Revision 9
: 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating sca le (questions in the 2 - 4 range are acceptable).
: 3. Check the appropriate box if a psychometric flaw is identified:  The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information). The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc). The answer choices are a collection of unrelated true/false statements. One or more distractors is (are) not credible. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
: 4. Check the appropriate box if a job content error is identified:  The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content). The question requires the recall of knowledge that is too specific for the closed refe rence test mode (i.e., it is not required to be known from memory). The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons). The question requires reverse logic or applic ation compared to the job requirements.
: 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
: 3. Psychometric Flaws  
: 4. Job Content Flaws  
: 5. Other   Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus  Cues T/F Cred.Dist. Partial Job-Link Minutia #/ units Back-ward  Q= K/A SROOnly 6. U/E/S 7. Explanation  6. Based on the reviewer
=s judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 7. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).   


ES-401 2 Form ES-401-9  
ES-401                                                                         2                                                           Form ES-401-9
: 3. Psychometric Flaws  
: 1. 2.      3. Psychometric Flaws         4. Job Content Flaws   5. Other 6.                              7.
: 4. Job Content Flaws  
Q#   LOK LOD (F/H) (1-5) Stem Cues T/F     Cred. Partial Job- Minutia #/ Back-   Q= SRO U/E/S                           Explanation Focus              Dist.        Link          units ward K/A Only 67    H    4                                                              Y        ?S (B) DO NOT UNDERSTAND THIS QUESTION - NEED TO HAVE EXPLAINED BY LICENSEE. DO NOT UNDERSTAND RELATIONSHIP WITH EOP-3A. Removed Ref Mtl 68    H    3                                                              Y        S (B) 69    F    3                                                              Y        S (B) 70    F    3                                                              Y        S (B) 71    F    3                                                              Y        S (B) 72    F     2                                                             Y         S (B) 73    F     2                             X                              Y       US (?) SOURCE OF QUESTION IS UNKNOWN. IT APPEARS THAT DISTRACTORS B AND D COULD BE CORRECT. B Rewritten 74    H    3                                                              Y         S (M) 75    H    3                                                              Y        E (?) SOURCE OF QUESTION IS UNKNOWN. B 76    H     3                                                             Y     Y  S (N) 77    H     3                                                              Y      Y E (?) SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE TWO K/As - WHICH ONE IS APPLICABLE? B 2.1.33 78    F    3                                                             Y     Y  S (N) 79    F    2                                                              Y      Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 80    H     3                                                             Y      Y  E (?) SOURCE OF QUESTION IS UNKNOWN. B 81    H    2                                                              Y      Y S (B) 82    F     2                                                             Y     Y  S (N) 83    H    3                                                              Y      Y S (N) 84    F     2                                                             Y     Y  S (N) 85    F     3                                                              Y      Y S (N) 86    F    2                                                              Y      Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 34 of 34                                          NUREG-1021, Draft Revision 9
: 5. Other   Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus  Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward  Q= K/A SROOnly 6. U/E/S 7. Explanation 10  F 2                   Y   S (N) 11  F 2        X            Y   U S (B) DISTRACTOR "B" IS NOT CREDABLE A&C changed 12  H 3                   Y   S (N) 13  H  4                    Y    E (?) THE QUESTION HAS TWO K/As, WHICH ONE IS APPLICABLE?
B THE QUESTION DOES NOT LIST IT
'S SOURCE, i.e., BANK, NEW, ETC. 14  H  3                   Y   S (N)  15  H 3       X            Y   U S (N) DISTRACTORS "A"  & "C" CAN BOTH BE CONSIDERED TO BE A CORRECT ANSWER.
A changed  16 F 2                   Y   S  (N) 17  F 2                   Y   S (N) 18  F  2                    Y    U S (B)  QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE (EOP7A) IS DELETED  OTHERWISE THE QUESTION IS A SIMPLE


LOOKUP QUESTION AND A POTENTIAL CUE FOR OTHER
ES-401                                                                      2                                                        Form ES-401-9
: 1. 2.      3. Psychometric Flaws        4. Job Content Flaws  5. Other 6.                              7.
Q#  LOK  LOD (F/H) (1-5) Ste Cues T/F    Cred. Partial Job- Minutia #/ Back-    Q= SRO U/E/S                          Explanation m              Dist.        Link          units ward K/A Only Foc us 87    H    3                            X                            N    Y  US (B) THE QUESTION DOES NOT MATCH THE K/A IN THAT THE PROCEDURES TO CORRECT, CONTROL, AND MITIGATE THE IMPACT ARE NOT IDENTIFIED. IT APPEARS THAT DISTRACTORS C AND D  COULD BE CORRECT. Chgd Stem 88    H    3                                                          Y    Y  S (B) 89    H    2                                                          Y    Y  S (N) 90    H    3                                                          Y    Y  S (M) 91    F    2                                                          Y    Y  US (N) QUESTION IS A SIMPLE LOOK-UP QUESTION WHEN TS 5.5.8 IS PROVIDED TO APPLICANT. Rewritten 92    H    2                            X                            Y    Y  US (B) DISTRACTORS A AND B COULD BOTH BE CORRECT. WHAT DOES NON-TRM MEAN? Discussed OK 93    H    4                                                          Y    Y  S (N) 94    H    2                            X                            Y    Y  US (?) SOURCE OF QUESTION IS UNKNOWN. IT APPEARS THAT DISTRACTORS A AND C COULD BOTH BE CORRECT. N Rewritten 95    H    3                                                          Y    Y  E (?) SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE TWO K/As ? WHICH ONE IS APPLICABLE? B 2.2.18
(?) SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE THREE 96    F    3                                                          Y    Y  E K/As ? WHICH ONE IS APPLICABLE? B 2.2.32 97    F    2                                                          Y    Y  E (?) SOURCE OF QUESTION IS UNKNOWN. B 98    F    2                                                          Y    Y  E (?) SOURCE OF QUESTION IS UNKNOWN. B 99    F    3                                                          Y    N  US (?) SOURCE OF QUESTION IS UNKNOWN. NOT A SRO ONLY QUESTION. N OK 10CFR55.43 (2) & (5) 100    H    3                                                          Y    Y  E (?) SOURCE OF QUESTION IS UNKNOWN. B SRO TOTALS


QUESTIONS. Revised  Notes,Cautions, Entry Conds. Deleted 19  F  2                    Y   E  (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)
Sat: 11 Unsat: 5 Ed:   9 H: 14 (56%)
N  20 F 3                    Y   S  (B)  21  H 2                    Y    S  (N) 22  F 3                  Y    U S (B) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE (EOP3A) IS DELETED  OTHERWISE THE QUESTION IS A SIMPLE
F: 11 (44%)
Bank: 4 13 Mod:  1 New:  9 11 Unk: 11 0 RO TOTALS Sat: 45 Unsat: 17 Ed:   13 H: 27 (36%)
F: 48 (64%)
Bank: 29 37 Mod:    3 New: 31 35 Unk: 12 0


LOOKUP QUESTION AND A POTENTIAL CUE FOR OTHER QUESTIONS.
S:\DRS\OB\OB-Only\EXAMS\CNS\Exam Review\APPENDIX E ADMIN.wpd APPENDIX E - REGION IV OPERATING TEST ADMINISTRATIVE JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX
Ref. Removed 23  F  3                    Y      S  (N)  24  H  3                    Y    S  (N)  25  F  3                    Y    S  (N)  26  F  2                   Y    S  (B)  27  F  2                    Y    U S  (N) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE (EOP5A) IS DELETED  OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION.
: 1. 2.             3. Attributes          4. Job Content   5.                                             6.
Discussed-Multiple Alt. Paths 28  F  2                    Y    S  (N)                 
JPM#  Dyn LOD                                         Errors    U/E/S                                       Explanation (D/S) (1-5)                                                                                       (See below for instructions)
: 3. Psychometric Flaws
IC  Cues   Critical Scope Over- Job-   Minutia Focus        Steps (N/B)    lap Link SA1  S     3                                                     S SA2  S     2                   X                                  S     ONLY ONE CRITICAL STEP.
: 4. Job Content Flaws
SA3  S     2                                                     S SA4  S     3                   X                                  S    ONLY ONE CRITICAL STEP SA5   S     2                   X                                  S     ONLY ONE CRITICAL STEP RA1  S     2                                                     S RA2   S     2                                                      E    STEP THREE OF THE PERFORMANCE CHECKLIST SAYS TO DETERMINE IF THE DELTA T VALUES ARE WITHIN + 5 DEGREES F. THE PUMP START CHECKLIST STEPS 10 THROUGH 14 HAVE A TOLERANCE OF + 4 DEGREES F. WHICH IS CORRECT?
: 5. Other  Q#  1. LOK (F/H) 2. LOD (1-5)  Stem Focus  Cues  T/F  Cred. Dist. Partial Job-Link Minutia #/ units Back-ward  Q= K/A SROOnly 6. U/E/S 7. Explanation 29 H 3          Y  S (N)
RA3   S    2                                                      U    ATTACHMENT 1 HPCI DATA SHEET SHOWS HPCI-MO-19 INITIAL STROKE TIME AS 18.7 SEC AND THE RETEST STROKE TIME AS 16.4 SEC. THESE VALUES ARE WITHIN THE IST ACCEPTABILITY RANGE. THEREFORE, THIS VALVE IS OPERABLE. VALVE HPCI-MO-58 EXCEEDED THE IST ACCEPTABLE RANGE ON BOTH STROKES WHICH MAKES THIS VALVE INOPERABLE. THIS IS INCONSISTENT WITH THE GOAL OF THE JPM AND NEEDS TO BE RESOLVED.
ES-401 2 Form ES-401-9
RA4  S    2                                                      U     THE JPM'S ATTACHMENT 3 SAYS THAT THE ATTACHMENT MAY BE GIVEN TO THE CANDIDATE, HOWEVER, ATTACHMENT 3 HAS THE ANSWERS DOCUMENTED ON IT.
: 3. Psychometric Flaws
                                                                      - Page 1 -
: 4. Job Content Flaws
: 5. Other  Q#  1. LOK (F/H) 2. LOD (1-5) Stem Focus  Cues T/F  Cred. Dist. Partial Job-Link Minutia #/ units Back-ward  Q= K/A SROOnly 6. U/E/S  7. Explanation 30  F  2                    Y  Y  U S  (?)  SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED). SRO ONLY QUESTION.
N-Ros trained to recognize LCOs 31  H  3    X                N    U S  (N)  THE QUESTION DOES NOT MATCH THE K/A. PART "B" OF THE K/A, USE OF PROCEDURES, IS NOT COVERED. QUESTION STEM PROVIDES AN ANSWER CUE. Rewritten 32  F  2                    Y    E N (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED) 33  F  3                    ?   E (?)  SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED).
WHY ARE THERE 2 K/AS??
B 211000.K2.01 34  F  2                    Y    E  (?)  SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)
B 35  F  2                    Y    E  (?)  SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)
N 36  F  3                    Y    S (B)  37  F  3                   Y    S (B)  38  H  3                    Y    S (N)  39  F  2                   Y    S (N)  40  H  3                    Y    S (B)  41  F  2                   Y    S (B)  42  F  2                    Y    S (B)  43  H  2                   Y   S (N)  44  F  2                   Y    S (B)  45  H  3                    Y    S (B)  46  H  2                   Y  Y  U S (B)  SRO ONLY QUESTION Modified Stem 47   F  3                    Y    S (B)  48  F 2                    Y    S  (B)                                                 
: 3. Psychometric Flaws
: 4. Job Content Flaws
: 5. Other   Q#  1. LOK (F/H)  2. LOD (1-5)  Stem Focus  Cues  T/F  Cred. Dist. Partial Job-Link Minutia #/ units Back-ward  Q= K/A SROOnly 6. U/E/S 7. Explanation


34 of 34 NUREG-1021, Draft Revision 9 ES-401 2 Form ES-401-9
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
: 3. Psychometric Flaws
: 1.       Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
: 4. Job Content Flaws
: 2.       Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
: 5. Other  Q#  1. LOK (F/H) 2. LOD (1-5) Stem Focus  Cues  T/F  Cred. Dist. Partial Job-Link Minutia #/ units Back-ward  Q= K/A SROOnly 6. U/E/S  7. Explanation 49  F  2                    Y    S  (B) 50  F  2                    Y    S  (N) 51  F  2                    Y    E  (?)  SOURCE OF QUESTION IS UNKNOWN B  52  F  3                   N    U S  (N) THE QUESTION DOES NOT MATCH THE K/A. Discussed  OK 53  H  3                    N    U S  (N) THE QUESTION DOES NOT MATCH THE K/A. Discussed  OK 54  F  2                    Y    S  (N) 55  F  2                    Y    S  (B) 56  F  2                    Y    E  (?)  SOURCE OF QUESTION IS UNKNOWN.
: 3.        Check the appropriate box when an attribute weakness is identified:
B  57  H  3                    N    U S  (N)  THE QUESTION DOES NOT MATCH THE K/A IN THAT THE PROCEDURES TO CORRECT, CONTROL, AND MITIGATE THE IMPACT ARE NOT IDENTIFIED.
* The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
SAT after discussed w/ lic.
* The JPM does not contain sufficient cues that are objective (not leading).
58  F  2                    Y    S  (B) 59  H  2                    N    U S (B) THE QUESTION DOES NOT MATCH THE K/A. CHANGE THE
* All critical steps (elements) have not been properly identified.
* Scope of the task is either too narrow (N) or too broad (B).
* Excessive overlap with other part of operating test or written examination.
: 4.        Check the appropriate box when a job content error is identified:
* Topics not linked to job content (e.g., disguised task, not required in real job).
* Task is trivial and without safety significance.
: 5.        Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 6.        Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).
                                                                                          - Page 2 -


SENTENCE IN THE STEM TO READ: "WITHOUT REFERENCE TO A PROCEDURE, WHICH OF THE FOLLOWING ACTIONS WOULD YOU, THE OPERATOR, TAKE TO PREVENT OVERPRESSURIZATION OF THE REACTOR WATER CLEANUP SYSTEM PIPING?"
S:\DRS\OB\OB-Only\EXAMS\CNS\Exam Review\APPENDIX E JPMs.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX
Chg'd Stem 60  F  2    X    X            Y    U S  (B)  DISTRACTORS "B" THROUGH "D  ARE NOT CREDIBLE Chg'd Stem 61  H  3                    Y  Y  U S  (N)  SRO ONLY QUESTION. OK  RO activity @ CNS 62  H  3                    Y    S  (B)  63  H  2                    Y    S  (N)  64  H  3                    Y    S  (N)  65  H  3                    Y    E (N)  THE EXPLANATION FOR DISTRACTOR C STATES THAT IT IS INCORRECT. Corrected 66  H  3                    Y    S  (M)  34 of 34 NUREG-1021, Draft Revision 9 ES-401 2 Form ES-401-9
: 1. 2.            3. Attributes          4. Job Content   5.                                               6.
: 3. Psychometric Flaws
JPM#  Dyn  LOD                                        Errors    U/E/S                                        Explanation (D/S) (1-5)                                                                                         (See below for instructions)
: 4. Job Content Flaws
IC  Cues   Critical Scope Over- Job-   Minutia Focus        Steps (N/B)    lap Link a    D    2                                                      S     PROCEDURE 2.2.74 UNAVAILABLE FOR REVIEW.
: 5. Other  Q#  1. LOK (F/H) 2. LOD (1-5) Stem Focus  Cues T/F  Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SROOnly 6. U/E/S 7. Explanation 67  H  4                    Y    ?S  (B)  DO NOT UNDERSTAND THIS QUESTION - NEED TO HAVE EXPLAINED BY LICENSEE. DO NOT UNDERSTAND RELATIONSHIP WITH EOP-3A.
b   D    3                                                     S     PROCEDURE 6.HPCI.103 UNAVAILABLE FOR REVIEW.
Removed Ref Mtl 68  H  3                    Y   S  (B)  69  F  3                   Y    S (B)  70  F  3                    Y   S  (B)  71  F  3                   Y   S (B)  72  F  2                   Y    S  (B)  73  F  2         X          Y   U S  (?)  SOURCE OF QUESTION IS UNKNOWN. IT APPEARS THAT
c   D    3                                                     E    THE JPM TITLE IS ADMINISTRATIVE JPM FOR OPERATIONS, BUT THIS IS NOT AN ADMINISTRATIVE JPM. STEP 10 HAS AN INCOMPLETE NUMBER FOR THE FLOW CONTROLLER. INSTEAD OF "RCIC-FIC", IT SHOULD BE "RCIC-FIC-91". ADDITIONALLY, STEP 10 HAS A "NOTE" THAT DISCUSSES A PROBLEM WITH THE RCIC CONTROLLER, IS THIS THE ALTERNATE PATH?
d   D    2                                                      S     PROCEDURE 2.2.77 UNAVAILABLE FOR REVIEW.
e   D                                                            E    THE JPM TITLE IS ADMINISTRATIVE JPM FOR OPERATIONS, BUT THIS IS NOT AN ADMINISTRATIVE JP


DISTRACTORS "B" AND "D" COULD BE CORRECT. B  Rewritten 74  H  3                    Y    S  (M)  75  H  3                    Y    E  (?)  SOURCE OF QUESTION IS UNKNOWN.
==M. PROCEDURE==
B  76  H  3                    Y  Y  S  (N)  77  H  3                    Y  Y  E  (?)  SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE TWO K/As - WHICH ONE IS APPLICABLE?
2.2.69.3 UNAVAILABLE FOR REVIEW. iT IS NOT CLEAR WHERE THE ALTERNATE PATH IS SINCE THE TITLE INCLUDES A "FAILED 2/3 CORE COVERAGE PERMISSIVE". STEP 6 HAS NUMBER OF QUESTION MARKS - WHY? STEP 8 HAS THE CANDIDATE START THE RHR PUMP, BUT ONLY CHECK THAT THE RED LIGHT IS ON -
B  2.1.33 78  F  3                   Y  Y  S  (N)  79  F  2                   Y  Y  E  (?)  SOURCE OF QUESTION IS UNKNOWN.
WHAT ABOUT OBSERVING MOTOR CURRENT? STEP 9 HAS A TERM MISSING AS EVIDENCED BY QUESTION MARKS.
B  80  H  3                    Y  Y  E  (?)  SOURCE OF QUESTION IS UNKNOWN.
f    D    2                                                     S     PROCEDURE 2.2.14 UNAVAILABLE FOR REVIEW. SIMPLE JPM.
B  81  H  2                   Y  Y  S (B)  82  F  2                   Y  Y  S  (N)  83  H  3                    Y  Y  S  (N)  84  F  2                   Y  Y  S  (N)  85  F  3                   Y  Y  S  (N)  86  F  2                    Y  Y  E (?)  SOURCE OF QUESTION IS UNKNOWN.
g    D    3                                                     E     THE JPM TITLE IS IN PLANT JPM FOR OPERATIONS, BUT THIS IS NOT AN IN PLANT JPM.
B  34 of 34 NUREG-1021, Draft Revision 9 ES-401 2 Form ES-401-9
STEP 10 IS INCONSISTENT WITH STEP 4.13 BECAUSE IT DOES MENTION THE PRESSING OF FUNCTION KEY F-4. THE JPM HAS THE CANDIDATE COMPLETE STEPS 4.5 THROUGH 4.21 OF THE PROCEDURE, HOWEVER, STEPS 22 AND 23 OF THE JPM ARE INCONSISTENT WITH PROCEDURE STEP 4.21.
: 3. Psychometric Flaws
h    D    3                                                      U    JPM STEPS 2, 3, AND 4 HAS THE OPERATOR PERFORMING DUTIES THAT ARE NOT IN PROCEDURE 5.4FIRE. IN GENERAL, THE JPM IS INCONSISTENT WITH THE PROCEDURE.
: 4. Job Content Flaws
i    S     3                                                     S     PROCEDURE 5.3ALT-STRATEGY COULD NOT BE OPENED BECAUSE IT HAD A SECURITY HOLD ON IT.
: 5. Other  Q#  1. LOK (F/H)  2. LOD (1-5)  Ste m Foc us  Cues  T/F  Cred. Dist. Partial Job-Link Minutia #/ units Back-ward  Q= K/A SROOnly 6. U/E/S 7. Explanation 87  H  3         X          N  Y  U S (B)  THE QUESTION DOES NOT MATCH THE K/A IN THAT THE PROCEDURES TO CORRECT, CONTROL, AND MITIGATE THE IMPACT ARE NOT IDENTIFIED. IT APPEARS THAT DISTRACTORS
J    D    3                                                      E    iIT IS NOT CLEAR HOW YOU GO FROM STEP 9 IN THE JPM TO STEP 11. THE PROCEDURE GOES FROM STEP 5.1.11 TO STEP 7 AND DOESN'T RETEST THE SCRAM UNTIL STEP 7.3.9.
HOW YOU GO FROM JPM STEP 10 TO PROCEDURE STEP 7.3.9 NEEDS CLARIFICATION.
k    S    2                                                      S
                                                                      - Page 1 -


"C" AND" D " COULD BE CORRECT.
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
Chg'd Stem 88  H  3                    Y  Y  S (B) 89  H  2                    Y  Y  S  (N) 90  H  3                   Y  Y  S  (M) 91  F  2                    Y  Y  U S  (N) QUESTION IS A SIMPLE LOOK-UP QUESTION WHEN TS 5.5.8 IS PROVIDED TO APPLICANT. Rewritten 92  H  2          X          Y  Y  U S (B) DISTRACTORS "A" AND "B" COULD BOTH BE CORRECT. WHAT
: 1.        Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
: 2.        Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
: 3.        Check the appropriate box when an attribute weakness is identified:
* The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
* The JPM does not contain sufficient cues that are objective (not leading).
* All critical steps (elements) have not been properly identified.
* Scope of the task is either too narrow (N) or too broad (B).
* Excessive overlap with other part of operating test or written examination.
: 4.        Check the appropriate box when a job content error is identified:
* Topics not linked to job content (e.g., disguised task, not required in real job).
* Task is trivial and without safety significance.
: 5.       Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
: 6.        Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).
                                                                                          - Page 2 -


DOES NON-TRM MEAN?  Discussed  OK 93  H  4                    Y  Y  S  (N)  94  H  2          X          Y  Y  U S  (?)  SOURCE OF QUESTION IS UNKNOWN. IT APPEARS THAT
S:\DRS\OB\OB-Only\EXAMS\CNS\Exam Review\APPEND IX N.wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen      1     2     3       4     5       6     7       8     9                 10 Explanation (See below for instructions)
 
Set     ES     TS     Crit   IC   Pred   TL     L/C     Eff U/E/S X                   X                     U       NEEDS TIME LINE AND IDENTIFICATION OF CRITICAL STEPS.
DISTRACTORS A AND C COULD BOTH BE CORRECT. N  Rewritten 95  H  3                    Y  Y  E  (?)  SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE TWO K/As ?  WHICH ONE IS APPLICABLE?
1 2                     X                   X                     U       NEEDS TIME LINE AND IDENTIFICATION OF CRITICAL STEPS.
B  2.2.18 96  F  3                    Y  Y  E (?)  SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE THREE K/As ?  WHICH ONE IS APPLICABLE?
3                                           X                     S       NEEDS TIME LINE 4                                           X                     S       NEEDS TIME LINE Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
B  2.2.32 97  F  2                    Y  Y  E  (?)  SOURCE OF QUESTION IS UNKNOWN.
: 1.     ES: ES-301 checklists 4, 5, & 6 satisfied.
B  98  F  2                    Y  Y  E  (?)  SOURCE OF QUESTION IS UNKNOWN.
: 2.     TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.
B  99  F  3                    Y  N  U S  (?)  SOURCE OF QUESTION IS UNKNOWN. NOT A SRO ONLY QUESTION.
: 3.     Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
N    OK  10CFR55.43 (2) & (5) 100  H  3                    Y  Y  E  (?)  SOURCE OF QUESTION IS UNKNOWN.
: 4.     IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
B                                                                                    SRO  TOTALS
: 5.     Pred: Scenario sequence and other factors avoid predictability issues.
 
: 6.     TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
Sat:  11 Unsat: 5 Ed:      9 H:  14 (56%)
: 7.     L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
F:  11 (44%)
: 8.     Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
Bank:  4 13 Mod:    1 New:    9 11 Unk:  11 0  RO TOTALS  Sat:    45 Unsat: 17 Ed:      13 H:  27 (36%)
: 9.     Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
F:  48 (64%)
: 10. Provide a brief description of problem in the explanation column
Bank:  29 37 Mod:      3 New:  31 35 Unk:  12 0
                                                                          - Page 1 -}}
S:\DRS\OB\OB-Only\EX AMS\CNS\Exam Rev iew\APPENDIX E ADMIN.wp d- Page 1 -
AP PENDIX E - REGION IV OPER ATING TEST AD MINISTR AT IVE JOB PERF ORMAN CE MEAS URE QU ALITY R EVIEW MATRIX JPM#1.Dyn (D/S)2.LOD (1-5)3. Attributes
: 4. Job Conten t Errors 5.U/E/S 6.Explanatio n (See b elow for i nstruct ions)IC Focus Cues Critical Steps Scope (N/B)Over-lap Job-Link Minutia SA1 S 3 S SA2 S 2 X S ONLY O NE CRI TICAL S TEP.SA3 S 2 S SA4 S 3 X S ONLY O NE CRI TICAL S TEP SA5 S 2 X S ONLY O NE CRI TICAL S TEP RA1 S 2 S RA2 S 2 E STEP THR EE OF TH E PERFOR MANCE CHECKL IST SAYS T O DETER MINE I F THE D ELTA T VALUES ARE W ITHIN + 5 DEGREES F. TH E PUMP START CHEC KLIST STEPS 10 THROU GH 14 HAVE A TOLERANCE OF
+ 4 DEGREES F. WHICH IS COR RECT?RA3 S 2 U ATTACHM ENT 1 H PCI DAT A SHEET S HOWS HPCI-MO-1 9 INITI AL STRO KE TIME AS  18.7 SEC AND TH E RETEST STROKE TIME AS 16.4 S EC. TH ESE VALU ES ARE WITHIN THE IST ACCEPTABILITY RANGE. THER EFORE, THIS VALVE IS OPERABLE.
VALVE HPCI-MO-58 EXCEE DED TH E IST AC CEPTABLE RANGE ON BOTH STROKES WHICH MAK ES THIS VALVE INOPERA BLE. TH IS IS INC ONSISTEN T WITH THE G OAL OF THE JPM AND N EEDS TO BE RESOLVED.RA4 S 2 U THE JPM'S ATTACHME NT 3 SAYS THAT THE ATT ACHMENT MAY BE GIVEN TO THE CANDID ATE, HOWEVER, ATTACH MENT 3 HAS TH E ANSWERS  DO CUMEN TED ON IT. 
- Page 2 -
Instructions for Completing M atrix This form is no t contained in or required by N UREG-1021.
Utilities are not required or encouraged to use it. The pur pose of this form is to enhance regio nal consistency in reviewing operating tests.
Additional infor mation on these are as may be found in Examinat ion Good Pract ices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space p rovided.1.Determine w hether the task is dynamic (D) o r static (S).
A dynamic task is one that inv olves continuous mon itoring and respon se to varyi ng parameters. A static task is basically a sy stem reconfiguratio n or realignment.
2.Determine leve l of difficulty (LOD) using established 1-5 rat ing scale. Level s 1 and 5 represen t inappropriate (low or hi gh) discriminatory level for th e license being tested.3.Check the appropr iate box w hen an attribute weakness is ide ntified:*The initiating cue is not sufficien tly clear to e nsure the operato r understands the task and how to begin.*The JPM does not contain su fficient cues that are objective (not leading).
*All critical steps (elements) have n ot been properly identified.
*Scope of the task is either too nar row (N) or too broad (B).*Excessive overlap w ith other part of operating test or writt en examinati on.4.Check the appropr iate box w hen a job content error is identif ied:*Topics not linked to job content (e.g., disguised task, not required in real job).
*Task is triv ial and witho ut safety signif icance.5.Based o n the revi ewer's ju dgmen t, is the JPM as written (U)nacc eptabl e (requi ring rep air or rep lacem ent), in need of (E)ditori al enha nceme nt, or (S) atisfa ctory?6.Provi de a b rief desc ript ion o f pro blem in th e expl anat ion c olum n. Pro vide conc lusi on on wheth er JP M SET criteria satisfied (i.e., number/
distribution of saf ety functions, A.3 and A.4 int egrated wit h parts B/C, Admin topics per sect ion meet ES).
S:\DRS\OB\OB-Only\EX AMS\CNS\Exam Rev iew\APPENDIX E JPMs.wpd- Page 1 -
AP PENDIX E - REG ION IV O PERATIN G TEST JO B PERFOR MANCE MEASUR E  QUAL ITY REVI EW MATRIX JPM#1.Dyn (D/S)2.LOD (1-5)3. Attributes
: 4. Job Conten t Errors 5.U/E/S 6.Explanatio n (See b elow for i nstruct ions)IC Focus Cues Critical Steps Scope (N/B)Over-lap Job-Link Minutia a D 2 S PR OC ED UR E 2.2.7 4 U NA VA IL AB LE FO R R EV IE W.b D 3 S PR OC ED UR E 6.H PC I.10 3 U NA VA IL AB LE FO R R EV IE W.c D 3 E THE JPM TITLE I S ADMINISTRATIVE JPM FOR OPERATIONS , BUT TH IS IS NO T AN ADMINISTRATIVE JPM. STEP 1 0  HAS AN INCO MPLETE NUMBER FOR THE FLOW CONTR OLLER. INSTEA D OF "R CIC-F IC", IT SHOUL D BE "R CIC-F IC-91". ADD ITIONAL LY, STEP 10 HAS A "NOTE" THAT DISCUS SES A PROBLEM W ITH THE RCIC CONTROLLER, IS THIS THE ALTERNATE PATH?
d D 2 S PR OC ED UR E 2.2.7 7 U NA VA IL AB LE FO R R EV IE W.e D E THE JPM TITLE I S ADMINISTRATIVE JPM FOR OPERATIONS , BUT TH IS IS NO T AN ADMIN ISTRATIV E JPM. PROCED URE 2.2.69.3 UNAVAIL ABLE FOR REVIEW. iT IS NOT CLEAR WHERE THE ALTERNATE PAT H IS SINCE THE TITLE INCLUDE S A "FAILED 2/3 CORE COVERAG E PERMI SSIVE". S TEP 6 H AS NUM BER OF Q UESTION MARKS - WHY?  STEP 8 HAS THE CANDIDATE STA RT THE RHR PUMP, BUT ONLY CHECK THAT THE RED LIGHT IS ON -
WHAT ABOUT O BSERVING MOTOR CURR ENT?  S TEP 9 H AS A TERM MISSIN G AS EVIDENC ED BY QU ESTION M ARKS.f D 2 S PROCEDURE 2.2.14 UNAVA ILABLE FOR REVIE W. SIMP LE JPM.g D 3 E THE JPM TITLE I S IN PLANT JPM FOR OPERATIONS , BUT TH IS IS NO T AN IN PLANT JPM. STEP 10 IS INCONSISTENT WITH STEP 4.13 BECAUSE IT DOES MENT ION THE PRESSING OF FUNCTION KEY F-4. THE JPM HAS TH E CANDIDATE COM PLETE STEPS 4.5 THROUG H 4.21 OF THE PROCEDUR E, HOWEVER, STEPS 22 AND 23 OF T HE JPM ARE INCON SISTENT WITH PROCEDURE STEP 4.21
.h D 3 U JPM STEP S 2, 3, AND 4 HAS TH E OPERAT OR PERF ORMIN G DUTI ES THAT ARE NOT IN PROCED URE 5.4FIRE. IN GEN ERAL, T HE JPM IS INCO NSISTEN T WITH THE P ROCEDU RE. i S 3 S PROCEDURE 5.3ALT-STRATEGY COULD NOT BE OPENED BECAUSE IT HAD A  SECURITY HOLD ON IT.
J D 3 E iIT IS NOT CLEAR HOW YOU GO FROM STEP 9 IN THE JPM TO STEP
: 11. THE PROCEDURE GOES FR OM STEP 5.1.1 1 TO ST EP 7 AND DOESN'T RETEST THE SC RAM U NTIL ST EP 7.3.9. HOW YOU GO FROM JP M STEP 10 TO PROCEDURE STEP
 
====7.3.9 NEEDS====
CLARIFICA TION.k S 2 S
- Page 2 -
Instructions for Completing M atrix This form is no t contained in or required by N UREG-1021.
Utilities are not required or encouraged to use it. The pur pose of this form is to enhance regio nal consistency in reviewing operating tests.
Additional infor mation on these are as may be found in Examinat ion Good Pract ices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space p rovided.1.Determine w hether the task is dynamic (D) o r static (S).
A dynamic task is one that inv olves continuous mon itoring and respon se to varyi ng parameters. A static task is basically a sy stem reconfiguratio n or realignment.
2.Determine leve l of difficulty (LOD) using established 1-5 rat ing scale. Level s 1 and 5 represen t inappropriate (low or hi gh) discriminatory level for th e license being tested.3.Check the appropr iate box w hen an attribute weakness is ide ntified:*The initiating cue is not sufficien tly clear to e nsure the operato r understands the task and how to begin.*The JPM does not contain su fficient cues that are objective (not leading).
*All critical steps (elements) have n ot been properly identified.
*Scope of the task is either too nar row (N) or too broad (B).*Excessive overlap w ith other part of operating test or writt en examinati on.4.Check the appropr iate box w hen a job content error is identif ied:*Topics not linked to job content (e.g., disguised task, not required in real job).
*Task is triv ial and witho ut safety signif icance.5.Based o n the revi ewer's ju dgmen t, is the JPM as written (U)nacc eptabl e (requi ring rep air or rep lacem ent), in need of (E)ditori al enha nceme nt, or (S) atisfa ctory?6.Provi de a b rief desc ript ion o f pro blem in th e expl anat ion c olum n. Pro vide conc lusi on on wheth er JP M SET criteria satisfied (i.e., number/
distribution of saf ety functions, A.3 and A.4 int egrated wit h parts B/C, Admin topics per sect ion meet ES).
S:\DRS\OB\OB-Only
\EXAMS\CNS\Exam Review\APPEND IX N.wpd- Page 1 -
APPENDIX N - REGIO N IV OP ERATIN G TEST SC ENAR IO REV IEW MATRIX Scen Set 1 ES 2 TS 3 Crit 4 IC 5 Pred 6 TL 7 L/C 8 Eff 9 U/E/S 10 Expla nation (See below for i nstructions) 1 X X U NEEDS TIME LINE AND IDENTIFICATION O F CRITICAL STEPS.
2 X X U NEEDS TIME LINE AND IDENTIFICATION O F CRITICAL STEPS.
3 X S NEEDS TIME LINE 4 X S NEEDS TIME LINE Instructions f or Completing Matrix This form is no t conta ined in or require d by NURE G-1021. Util ities are no t requi red or encou raged to use i t. The p urpose of this form is to enhance region al consiste ncy in re viewi ng operating test sce nario sets. A dditional information on th ese areas may be found in E xamination Good Practices App endix D. Check or mark any item(s) requiring co mment and expl ain the is sue in the space provi ded.1.ES: ES-301 c hecklists 4, 5, & 6 satisfied.
2.TS: Set inclu des SRO TS acti ons for each SRO, w ith required ac tions exp licitly detailed.
3.Crit: Each mani pulation o r evoluti on has ex plicit suc cess criteria documented in Form ES-D-2.
4.IC: Out of serv ice e quipmen t and o ther i nitia l con ditio ns rea sonab ly c onsis tent be twee n scen arios and n ot pred ictiv e of sce nario even ts and actions.
5.Pred: Scenari o sequence and other factors avoi d predictabi lity is sues.6.TL: Ti me l ine con stru cted , in clu din g ev ent a nd p roce ss tr igger ed c ondi tion s, su ch th at sc enar io c an ru n w itho ut ro utin e ex amin er cu ing.7.L/C: Len gth and complexit y for eac h scenar io in the set is reason able fo r the cr ew mix being e xamined, such tha t all applic ants have reasonably similar ex posure and ev ents are neede d for evaluati on purposes.
8.Eff: Sequence of events i s reasonably efficient for examin ation purpose s, especial ly wi th respect to l ong delays or interaction s.9.Based on the review er's judgment, is th e scenario s et as writte n (U)nacceptabl e (requiring repai r or replacemen t), in need o f (E)ditorial enhance ment, or (S)at isfac tory?10.Provi de a b rief de script ion o f probl em in t he ex plana tion c olumn}}

Latest revision as of 13:06, 23 November 2019

10-2006 - NRC Draft Exam Comments
ML062900360
Person / Time
Site: Cooper Entergy icon.png
Issue date: 11/10/2006
From:
NRC Region 4
To:
References
ES-401, ES-401-9
Download: ML062900360 (15)
v • d • e


Text

ES-401 COOPER NUCLEAR STATION-10-2006 Written Examination Form ES-401-9 Review Worksheet (password: lewisandclark) (TAC#: X02344 )

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 Y S (M) 2 F 2 X Y E (N) CONSIDER DELETING DISTRACTOR A AND REPLACING THE DISTRACTOR WITH ANOTHER DISTRACTOR THAT INVOLVES A SYSTEM ENGINEER. Changed 3 H 3 Y S (N) 4 F 2 US (N) GFES QUESTION OK Licensee Explained 5 F 2 Y S (B) 6 F 3 X Y E (B) CONSIDER REWRITING THE DISTRACTORS TO READ AS FOLLOWS : A. BOTH LOW-LOW SET VALVES CONTINUE TO CYCLE; B. BOTH LOW-LOW SET VALUES STOP CYCLING; C. THE LOW-LOW SET VALVE (71F) , WHICH CAN BE CONTROLLED FROM THE ADS ROOM CONTINUES TO CYCLE; D. THE LOW-LOW SET VALVE (71D) WHICH CANNOT BE CONTROLLED FROM THE ASD ROOM CONTINUES TO CYCLE. Changes Made 7 F 2 Y S (N) 8 F 2 X Y US (N) EITHER DISTRACTORS A OR D COULD BE SELECTED. THE EXPLANATION FOR DISTRACTOR B DISCUSSES A VALVE MV-81 WHICH IS NOT IN THE QUESTION. Distractor B changed 9 F 3 Y S (B)

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ One or more distractors is (are) not credible.

@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only

6. Based on the reviewer=s judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 10 F 2 Y S (N) 11 F 2 X Y US (B) DISTRACTOR B IS NOT CREDABLE A&C changed 12 H 3 Y S (N) 13 H 4 Y E (?) THE QUESTION HAS TWO K/As, WHICH ONE IS APPLICABLE? B THE QUESTION DOES NOT LIST ITS SOURCE, i.e., BANK, NEW, ETC.

14 H 3 Y S (N) 15 H 3 X Y US (N) DISTRACTORS A & C CAN BOTH BE CONSIDERED TO BE A CORRECT ANSWER. A changed 16 F 2 Y S (N) 17 F 2 Y S (N) 18 F 2 Y US (B) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE (EOP7A) IS DELETED OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION AND A POTENTIAL CUE FOR OTHER QUESTIONS. Revised Notes,Cautions, Entry Conds. Deleted 19 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)N 20 F 3 Y S (B) 21 H 2 Y S (N)

Y (B) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE 22 F 3 US (EOP3A) IS DELETED OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION AND A POTENTIAL CUE FOR OTHER QUESTIONS. Ref. Removed 23 F 3 Y S (N) 24 H 3 Y S (N) 25 F 3 Y S (N) 26 F 2 Y S (B) 27 F 2 Y US (N) QUESTION IS ACCEPTABLE IF THE HANDOUT REFERENCE (EOP5A) IS DELETED OTHERWISE THE QUESTION IS A SIMPLE LOOKUP QUESTION. Discussed-Multiple Alt. Paths 28 F 2 Y S (N)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 29 H 3 Y S (N)

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 30 F 2 Y Y US (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED).

SRO ONLY QUESTION. N-Ros trained to recognize LCOs 31 H 3 X N US (N) THE QUESTION DOES NOT MATCH THE K/A. PART B OF THE K/A, USE OF PROCEDURES, IS NOT COVERED. QUESTION STEM PROVIDES AN ANSWER CUE. Rewritten N

32 F 2 Y E

(?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED) 33 F 3  ? E (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED).

WHY ARE THERE 2 K/AS??B 211000.K2.01 34 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)B 35 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN (i.e., BANK, MODIFIED)N 36 F 3 Y S (B) 37 F 3 Y S (B) 38 H 3 Y S (N) 39 F 2 Y S (N) 40 H 3 Y S (B) 41 F 2 Y S (B) 42 F 2 Y S (B)

Y 43 H 2 S (N) 44 F 2 Y S (B) 45 H 3 Y S (B) 46 H 2 Y Y US (B) SRO ONLY QUESTION Modified Stem 47 F 3 Y S (B) 48 F 2 Y S (B)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 49 F 2 Y S (B) 50 F 2 Y S (N) 51 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN B 52 F 3 N US (N) THE QUESTION DOES NOT MATCH THE K/A. Discussed OK 53 H 3 N US (N) THE QUESTION DOES NOT MATCH THE K/A. Discussed OK 54 F 2 Y S (N) 55 F 2 Y S (B) 56 F 2 Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 57 H 3 N US (N) THE QUESTION DOES NOT MATCH THE K/A IN THAT THE PROCEDURES TO CORRECT, CONTROL, AND MITIGATE THE IMPACT ARE NOT IDENTIFIED.SAT after discussed w/ lic.

58 F 2 Y S (B) 59 H 2 N US (B) THE QUESTION DOES NOT MATCH THE K/A. CHANGE THE SENTENCE IN THE STEM TO READ: WITHOUT REFERENCE TO A PROCEDURE, WHICH OF THE FOLLOWING ACTIONS WOULD YOU, THE OPERATOR, TAKE TO PREVENT OVERPRESSURIZATION OF THE REACTOR WATER CLEANUP SYSTEM PIPING? Chgd Stem 60 F 2 X X Y US (B) DISTRACTORS B THROUGH D ARE NOT CREDIBLE Chgd Stem 61 H 3 Y Y US (N) SRO ONLY QUESTION. OK RO activity @ CNS 62 H 3 Y S (B) 63 H 2 Y S (N) 64 H 3 Y S (N) 65 H 3 Y E (N) THE EXPLANATION FOR DISTRACTOR C STATES THAT IT IS INCORRECT. Corrected 66 H 3 Y S (M) 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 67 H 4 Y ?S (B) DO NOT UNDERSTAND THIS QUESTION - NEED TO HAVE EXPLAINED BY LICENSEE. DO NOT UNDERSTAND RELATIONSHIP WITH EOP-3A. Removed Ref Mtl 68 H 3 Y S (B) 69 F 3 Y S (B) 70 F 3 Y S (B) 71 F 3 Y S (B) 72 F 2 Y S (B) 73 F 2 X Y US (?) SOURCE OF QUESTION IS UNKNOWN. IT APPEARS THAT DISTRACTORS B AND D COULD BE CORRECT. B Rewritten 74 H 3 Y S (M) 75 H 3 Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 76 H 3 Y Y S (N) 77 H 3 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE TWO K/As - WHICH ONE IS APPLICABLE? B 2.1.33 78 F 3 Y Y S (N) 79 F 2 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 80 H 3 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 81 H 2 Y Y S (B) 82 F 2 Y Y S (N) 83 H 3 Y Y S (N) 84 F 2 Y Y S (N) 85 F 3 Y Y S (N) 86 F 2 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Ste Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation m Dist. Link units ward K/A Only Foc us 87 H 3 X N Y US (B) THE QUESTION DOES NOT MATCH THE K/A IN THAT THE PROCEDURES TO CORRECT, CONTROL, AND MITIGATE THE IMPACT ARE NOT IDENTIFIED. IT APPEARS THAT DISTRACTORS C AND D COULD BE CORRECT. Chgd Stem 88 H 3 Y Y S (B) 89 H 2 Y Y S (N) 90 H 3 Y Y S (M) 91 F 2 Y Y US (N) QUESTION IS A SIMPLE LOOK-UP QUESTION WHEN TS 5.5.8 IS PROVIDED TO APPLICANT. Rewritten 92 H 2 X Y Y US (B) DISTRACTORS A AND B COULD BOTH BE CORRECT. WHAT DOES NON-TRM MEAN? Discussed OK 93 H 4 Y Y S (N) 94 H 2 X Y Y US (?) SOURCE OF QUESTION IS UNKNOWN. IT APPEARS THAT DISTRACTORS A AND C COULD BOTH BE CORRECT. N Rewritten 95 H 3 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE TWO K/As ? WHICH ONE IS APPLICABLE? B 2.2.18

(?) SOURCE OF QUESTION IS UNKNOWN. WHY ARE THERE THREE 96 F 3 Y Y E K/As ? WHICH ONE IS APPLICABLE? B 2.2.32 97 F 2 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 98 F 2 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B 99 F 3 Y N US (?) SOURCE OF QUESTION IS UNKNOWN. NOT A SRO ONLY QUESTION. N OK 10CFR55.43 (2) & (5) 100 H 3 Y Y E (?) SOURCE OF QUESTION IS UNKNOWN. B SRO TOTALS

Sat: 11 Unsat: 5 Ed: 9 H: 14 (56%)

F: 11 (44%)

Bank: 4 13 Mod: 1 New: 9 11 Unk: 11 0 RO TOTALS Sat: 45 Unsat: 17 Ed: 13 H: 27 (36%)

F: 48 (64%)

Bank: 29 37 Mod: 3 New: 31 35 Unk: 12 0

S:\DRS\OB\OB-Only\EXAMS\CNS\Exam Review\APPENDIX E ADMIN.wpd APPENDIX E - REGION IV OPERATING TEST ADMINISTRATIVE JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link SA1 S 3 S SA2 S 2 X S ONLY ONE CRITICAL STEP.

SA3 S 2 S SA4 S 3 X S ONLY ONE CRITICAL STEP SA5 S 2 X S ONLY ONE CRITICAL STEP RA1 S 2 S RA2 S 2 E STEP THREE OF THE PERFORMANCE CHECKLIST SAYS TO DETERMINE IF THE DELTA T VALUES ARE WITHIN + 5 DEGREES F. THE PUMP START CHECKLIST STEPS 10 THROUGH 14 HAVE A TOLERANCE OF + 4 DEGREES F. WHICH IS CORRECT?

RA3 S 2 U ATTACHMENT 1 HPCI DATA SHEET SHOWS HPCI-MO-19 INITIAL STROKE TIME AS 18.7 SEC AND THE RETEST STROKE TIME AS 16.4 SEC. THESE VALUES ARE WITHIN THE IST ACCEPTABILITY RANGE. THEREFORE, THIS VALVE IS OPERABLE. VALVE HPCI-MO-58 EXCEEDED THE IST ACCEPTABLE RANGE ON BOTH STROKES WHICH MAKES THIS VALVE INOPERABLE. THIS IS INCONSISTENT WITH THE GOAL OF THE JPM AND NEEDS TO BE RESOLVED.

RA4 S 2 U THE JPM'S ATTACHMENT 3 SAYS THAT THE ATTACHMENT MAY BE GIVEN TO THE CANDIDATE, HOWEVER, ATTACHMENT 3 HAS THE ANSWERS DOCUMENTED ON IT.

- Page 1 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 2 -

S:\DRS\OB\OB-Only\EXAMS\CNS\Exam Review\APPENDIX E JPMs.wpd APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX

1. 2. 3. Attributes 4. Job Content 5. 6.

JPM# Dyn LOD Errors U/E/S Explanation (D/S) (1-5) (See below for instructions)

IC Cues Critical Scope Over- Job- Minutia Focus Steps (N/B) lap Link a D 2 S PROCEDURE 2.2.74 UNAVAILABLE FOR REVIEW.

b D 3 S PROCEDURE 6.HPCI.103 UNAVAILABLE FOR REVIEW.

c D 3 E THE JPM TITLE IS ADMINISTRATIVE JPM FOR OPERATIONS, BUT THIS IS NOT AN ADMINISTRATIVE JPM. STEP 10 HAS AN INCOMPLETE NUMBER FOR THE FLOW CONTROLLER. INSTEAD OF "RCIC-FIC", IT SHOULD BE "RCIC-FIC-91". ADDITIONALLY, STEP 10 HAS A "NOTE" THAT DISCUSSES A PROBLEM WITH THE RCIC CONTROLLER, IS THIS THE ALTERNATE PATH?

d D 2 S PROCEDURE 2.2.77 UNAVAILABLE FOR REVIEW.

e D E THE JPM TITLE IS ADMINISTRATIVE JPM FOR OPERATIONS, BUT THIS IS NOT AN ADMINISTRATIVE JP

M. PROCEDURE

2.2.69.3 UNAVAILABLE FOR REVIEW. iT IS NOT CLEAR WHERE THE ALTERNATE PATH IS SINCE THE TITLE INCLUDES A "FAILED 2/3 CORE COVERAGE PERMISSIVE". STEP 6 HAS NUMBER OF QUESTION MARKS - WHY? STEP 8 HAS THE CANDIDATE START THE RHR PUMP, BUT ONLY CHECK THAT THE RED LIGHT IS ON -

WHAT ABOUT OBSERVING MOTOR CURRENT? STEP 9 HAS A TERM MISSING AS EVIDENCED BY QUESTION MARKS.

f D 2 S PROCEDURE 2.2.14 UNAVAILABLE FOR REVIEW. SIMPLE JPM.

g D 3 E THE JPM TITLE IS IN PLANT JPM FOR OPERATIONS, BUT THIS IS NOT AN IN PLANT JPM.

STEP 10 IS INCONSISTENT WITH STEP 4.13 BECAUSE IT DOES MENTION THE PRESSING OF FUNCTION KEY F-4. THE JPM HAS THE CANDIDATE COMPLETE STEPS 4.5 THROUGH 4.21 OF THE PROCEDURE, HOWEVER, STEPS 22 AND 23 OF THE JPM ARE INCONSISTENT WITH PROCEDURE STEP 4.21.

h D 3 U JPM STEPS 2, 3, AND 4 HAS THE OPERATOR PERFORMING DUTIES THAT ARE NOT IN PROCEDURE 5.4FIRE. IN GENERAL, THE JPM IS INCONSISTENT WITH THE PROCEDURE.

i S 3 S PROCEDURE 5.3ALT-STRATEGY COULD NOT BE OPENED BECAUSE IT HAD A SECURITY HOLD ON IT.

J D 3 E iIT IS NOT CLEAR HOW YOU GO FROM STEP 9 IN THE JPM TO STEP 11. THE PROCEDURE GOES FROM STEP 5.1.11 TO STEP 7 AND DOESN'T RETEST THE SCRAM UNTIL STEP 7.3.9.

HOW YOU GO FROM JPM STEP 10 TO PROCEDURE STEP 7.3.9 NEEDS CLARIFICATION.

k S 2 S

- Page 1 -

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 2 -

S:\DRS\OB\OB-Only\EXAMS\CNS\Exam Review\APPEND IX N.wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen 1 2 3 4 5 6 7 8 9 10 Explanation (See below for instructions)

Set ES TS Crit IC Pred TL L/C Eff U/E/S X X U NEEDS TIME LINE AND IDENTIFICATION OF CRITICAL STEPS.

1 2 X X U NEEDS TIME LINE AND IDENTIFICATION OF CRITICAL STEPS.

3 X S NEEDS TIME LINE 4 X S NEEDS TIME LINE Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. ES: ES-301 checklists 4, 5, & 6 satisfied.
2. TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.
3. Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
4. IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
5. Pred: Scenario sequence and other factors avoid predictability issues.
6. TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
7. L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
8. Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
9. Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
10. Provide a brief description of problem in the explanation column

- Page 1 -

Retrieved from "https://kanterella.com/w/index.php?title=ML062900360&oldid=2185213"