ML060870645

From kanterella
Jump to navigation Jump to search
10-2005 - NRC Draft Exam Comments
ML060870645
Person / Time
Site: San Onofre  Southern California Edison icon.png
Issue date: 10/28/2005
From: Gody A
Operations Branch IV
To: Rosenblum R
Southern California Edison Co
References
50-361/05-302, 50-362/05-302, ES-401-9, SONGS-10-2005 50-361/05-302, 50-362/05-302
Download: ML060870645 (16)
v • d • e


Text

ES-401 SONGS-10-2005 Written Examination Form ES-401-9 Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= S R O U/E /S Explan ation Focus Dis t. Link units ward K/A O nly 1B H 2 Y N S A-3, CR 2M F 3 Y N S C-3, CR 3N F 3 Y N S C-6, CR 4B H 2 Y N S A-3, CR 5B F 2 Y N S A-2, CR 6B F 2 Y N S C-5, CR 7B H 3 X Y N U C-10, OR Distractor A could be argued correct because increasing AFW flow would increase subcooling.

8B H 3 Y N S C-5, OR 9N H 3 X X Y N U B-10, CR - Based on the scenario, the crew would be in the EOIs and the correct answer (B) is the only distractor that says continue in the EOIs.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= S R O U/E /S Explan ation Focus Dis t. Link units ward K/A O nly Instructions

[Ref er to Sec tion D of E S-4 01 an d A ppen dix B f or additional inf ormation regard ing eac h of th e following c onc epts .]

1. En ter th e level of kn owled ge (L O K) of eac h q ues tion as either ( F)un dam ental or ( H )igh er c ogn itive level.
2. Enter the level of difficulty (LO D) of each ques tion using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

@ Th e stem lacks su fficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless inform ation).

@ Th e stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ O ne or m ore distr actors is (are) n ot cred ible.

@ On e or more distractors is (are) partially correct (e.g., if the applicant can m ake unstated assu mptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ Th e question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ Th e question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from mem ory).

@ Th e question contains data with an unrealistic level of accu racy or incons istent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Ch eck q ues tions th at are sam pled for c onform anc e with the ap proved K /A an d thos e that are des igna ted S R O -only (K/A and license level mismatches are unacceptable).
6. Based on the review ers jud gm ent, is the qu estion as written (U )nacceptable ( req uiring rep air or rep lac em ent), in need of ( E)ditorial enh anc em ent, or (S)atis fac tory?
7. At a m inimu m, explain an y U ratings (e.g., how the A ppen dix B p syc hom etric attribu tes are n ot being m et).

ES-401 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly 10B H 2 X Y N E C-5, OR - The distractors should be formatted the same (e.g. A should read 0 to 15 minutes) 11B F 2 N Y U A-7, CR - The KA is focused on LCO/Safety Limits and the question is how RPS is affected by a loss of pressurizer narrow range pressure instrumentation.

This is an SRO Only question.

12M F 3 Y N S A-3, CR.

13M H 3 Y N S A-6, CR.

14N H 3 Y N S C-5, CR.

15B H 3 Y N S B-4, CR.

16N H 3 Y N S B-4. CR.

17N F 3 Y N S C-4, CR.

18B H 2 Y N S A-4, CR 19B H 3 Y N S A-4, CR.

20B F 3 X Y N U B-4, CR - According to the referenced EOP, the first action is to initiate attachment 6 and this is not listed as an answer. Therefore, it could be argued there is no correct answer.

21B H 3 Y N S D-4, CR, 4/05.

22B F 3 Y N S B-4, CR 23B H 3 Y N S D-11, CR.

24B H 3 Y N S B-11, CR 25B H 3 Y N S B-4, CR.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly 26N H 3 X Y Y U D-43.7, CR - Because this question tests on fuel handling procedures, it is a 10CFR55.43 (SRO) question. There is no job link between the control room RO and the refueling spent fuel pool gate seals.

27B F 3 Y N S D-4, CR.

28B F 3 Y N S B-9, CR.

29N H 3 Y N S B-6, CR.

30B H 3 N N U A-3, CR. The question asks what action is to be taken while the KA asks why the action is taken.

31B F 2 N N U C-10, CR - This is a fundamental question, not analysis. This is an A2 KA and requires testing both the prediction of what bad will happen and what procedures are used to mitigate the potential consequences. This question does not do this.

32B H 3 X X Y N U A-2, CR - The stem of this question asks what action is required and only one of the distractors (the correct one) actually requires an action to take the channel out of service.

33B H 3 Y N S A-3, CR.

34N H 3 Y N S A-9, CR 35B H 3 N N U A-10, CR - This is an A2 KA and the same comment applies as in question 31. Consider adding a procedure to each of the distractors, e.g., use SO23-13-24 for one or two distractors and use another procedure in the other distractors.

36N H 3 Y N S D-14, OR.

37N H 3 Y N S A-4, CR.

38M H 3 Y N S D-5, CR.

39M H 3 Y N S B-6, CR.

40N H 4 Y N S B-14, CR.

41B F 2 Y N S C-8, CR.

42M H 2 X Y N U D-3, CR - The procedure provided states the RCP temperature must be maintained below 225 degrees. The correct answer would mean RCP temperature is not less than 225 degrees. It could be argued answer C is correct.

43B H 3 N N U C-10, CR - The KA is about the inlet/outlet temperatures for the RHR heat exchanger. The question does not address this.

44B H 3 Y N S B-4, CR.

45B H 2 Y N S C-7, CR.

46N F 3 Y N S B-10, CR.

47B H 3 Y N S B-10, CR.

48N H 3 Y N S A-10, OR.

49N H 3 Y N S C-5, CR 50N F 3 X Y N U B-10, CR - Technically, answer C is correct since this is done at less than 108 VDC per the procedure versus at 108 VDC. Thus it can be argued there is more than one correct answer. Also the procedure states all the 1E buses have to be less than 108 VDC versus just one as asked in the question.

51N H 3 Y N S C-4, CR.

52N F 3 Y N S D-10, CR.

53N H 3 Y N S D-10, CR.

54B H 3 N N U B-10, CR - The question does not test on the alarms being consistent with plant conditions as required by the KA.

55B F 3 Y N S B-10, CR.

56B H 3 Y N S B-10, CR.

57B H 3 Y N S A-6, CR.

58B H 3 Y N S A-7, CR.

59N F 3 X Y Y U B-10, CR - On the surface, this appears to be an SRO only question. There isnt any apparent job link to the RO unless he does the notification.

60N H 3 N N U D-13, CR - The KA is focused on an accidental release whereas the question is about a planned release.

61N F 2 Y N S B-11, CR.

62B H 2 Y N S C-14, OR 63B H 3 Y N S C-10, CR.

64B H 3 Y N S C-10, CR.

65N H 3 N N U C-4, CR - The KA is on RCS leakage and heat removal systems. The question is on RCS leakage calculations.

66N F 2 Y N S D-2, CR.

NUREG-1021, Draft Revision 9

ES-401 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly 67B F 2 Y N S D-10, CR.

68B F 2 Y N S C-5, CR.

69B F 2 Y N S A-10, CR.

70B H 3 X N Y U B-10, CR - The refueling floor actions required are not directed or performed by the RO. This question appears to be SRO only and there is no job-link to the RO. Also the KA is on Refueling Process and the question is on an abnormal refueling floor event.

71B H 3 Y Y U A-43.2, OR - The interpretation of tech specs falls into the SRO only exam per 10CFR55.43.2.

72M H 2 X Y N U C-12, CR - The way the stem is worded, what is the maximum time there is no correct answer because the maximum time is actually 11 hours1.273148e-4 days <br />0.00306 hours <br />1.818783e-5 weeks <br />4.1855e-6 months <br />, not 10.

Need to reword the stem or provide the correct answer.

73B H 2 Y N S B-12, CR.

74B F 3 Y N S C-10, CR.

75B F 3 Y N S A-3, CR.

RO RO EXAM TOTALS 45 B B = 60%

7 M M = 9.3% Answer Dist. : A = 20 Open Reference = 7 23 N N = 30.7% B = 22 Previous NRC Qs = 2 C = 21 F = 25 F = 33% D = 12 H = 50 H = 67%

10CFR55.41 Distribution LOD = 2.76 41.1 = 0 41.8 = 1 41.2 = 3 41.9 = 2 41.3 = 8 41.10 = 21 41.4 = 13 41.11 = 3 41.5 = 7 41.12 = 2 41.6 = 5 41.13 = 1 41.7 = 3 41.14 = 3

GENERAL COMMENT

S:

1. The answer distribution is skewed away from D. Recommend increasing the number of correct answers using distractor D.
2. Five questions require the use of steam tables to answer Tsat - Psat type questions. This seems like a high number and may be argued as double-jeopardy.
3. Initially, 22 questions were reviewed as U. Most of these will be easily rectified.
4. The number of comprehension/analysis questions is 67% (50 questions).

The limit is 60% (45 questions) per the quality checklist. All other metrics are within those allowed by NUREG 1021.

5. The 10CFR55.41 distribution is random and appears to be normal compared to other exams.

NOTE: Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

ES-401 SONGS 2005 Written Examination Form ES-401-9 Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= S R O U/E /S Explan ation Focus Dis t. Link units ward K/A O nly 76B H 3 Y Y S A-5, CR.

77B H 3 Y N U D-5, CR - This is not an SRO only as the first part of the question is system diagnosis and the second part is the RO immediate actions.

78N H 3 X Y Y E B-5, CR - Need to change procedure in Distractor "A" to match that in Distractor "D"to make the distractors more credible.

79B H 3 Y N E D-5, CR - This is not an SRO only question as it tests on EOP entry conditions which is RO knowledge. The question should ask what procedure(s) the SRO should transition to for this scenario.

80M H 3 Y Y S D-5, CR.

81N H 3 Y N U A-41.10, CR - This is not an SRO only as it tests system knowledge and followup actions.

82B H 3 Y Y S A-41.5, CR 83N H 3 Y N E C-41.5, CR - This is RO knowledge.

84N H 3 Y N U C-41.10, OR - This is RO knowledge.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= S R O U/E /S Explan ation Focus Dis t. Link units ward K/A O nly Instructions

[Ref er to Sec tion D of E S-4 01 an d A ppen dix B f or additional inf ormation regard ing eac h of th e following c onc epts .]

1. En ter th e level of kn owled ge (L O K) of eac h q ues tion as either ( F)un dam ental or ( H )igh er c ogn itive level.
2. Enter the level of difficulty (LO D) of each ques tion using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

@ Th e stem lacks su fficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless inform ation).

@ Th e stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ O ne or m ore distr actors is (are) n ot cred ible.

@ On e or more distractors is (are) partially correct (e.g., if the applicant can m ake unstated assu mptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ Th e question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ Th e question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from mem ory).

@ Th e question contains data with an unrealistic level of accu racy or incons istent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Ch eck q ues tions th at are sam pled for c onform anc e with the ap proved K /A an d thos e that are des igna ted S R O -only (K/A and license level mismatches are unacceptable).
6. Based on the review ers jud gm ent, is the qu estion as written (U )nacceptable ( req uiring rep air or rep lac em ent), in need of ( E)ditorial enh anc em ent, or (S)atis fac tory?
7. At a m inimu m, explain an y U ratings (e.g., how the A ppen dix B p syc hom etric attribu tes are n ot being m et).

ES-401 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly 85B H 3 Y Y S A-5, CR.

86N H 3 Y N U D-10, CR - System diagnosis and knowing actions is RO knowledge. Question does not address what procedures would be used.

87B H 3 N Y U A-5, CR - The question does not address predict the impact as required by the KA.

88N F 2 Y Y S B-2, CR.

89B F 2 N Y U B-2, CR - This is a good SRO question but the KA is focused on system status criteria requiring notification of plant personnel. The question does not address notification of plant personnel.

90B F 3 Y Y S A-2, CR

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly 91N H 2 Y Y S C-2, CR 92N F 4 Y Y S D-2, CR.

93N H 3 Y Y S A-6, OR.

94B F 2 Y Y S A-2, CR.

95M H 2 Y Y S B-2, CR.

96B F 2 X Y Y U A-7, CR - This is not a discriminatory question. It is not clear what is being done that could be considered a core alteration (moving a light bulb?).

Also it is not clear on how the CRS or RE would provided oversight from the control room as stated in distractors B and C.

97N F 2 Y Y S B-5, CR.

98B F 3 Y Y S D-1, CR.

99B H 3 Y Y S B-5, OR.

100 F 3 Y Y S C-5, CR.

B NUREG-1021, Draft Revision 9

ES-401 Form ES-401-9 SRO 13 B B = 52% LOD = 2.72 U = 13 SRO TOTALS 2 M M = 8% E=0 10 N N = 40% LOK S = 12 Answer Dist: A=9 Open Reference = 4 H = 16 B=6 Previous NRC Qs = 0 H = 64% C=4 D=6 F=9 F = 36% 10CFR55.43 Distribution 43.1 = 1 43.5 = 10 43.2 = 7 43.6 = 1 43.3 = 0 43.7 = 1 43.4 = 4 43.8 = 0

GENERAL COMMENT

S:

1. Many of the questions are unsatisfactory because they are not SRO only.
2. All metrics are within those permitted by NUREG 1021, Rev 9.
3. There are no questions that, in the reviewer opinion, tests applicant ability to apply tech specs.

NOTE: Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

S:\DRS\OB \EXAMS\SO NG S\SON GS -EXAM 10-05 ADA MS LO AD\SO NG S-10-2005-NRC D RAFT EXAM C OM MEN TS\SO NG S-10-2005-DRAFT JPM C OM MEN TS.wpd APPENDIX E - R EG IO N IV O PE RA T ING T ES T JO B P ER FO RM AN CE ME AS UR E Q UA LIT Y R EV IEW MA T RIX

1. 2. 3. A ttributes 4. Job Content 5. 6.

JPM# D yn LOD Errors U/E /S Explan ation (D/S) (1-5) (See below for instructions)

IC Cues Cr itical Scope Over- Job- Minut ia Focus Steps (N/B) lap Link a D 3 E B E This JPM looks very long and could require an excessive amount of time to complete. Confirm the validation time during validation week. Also, see general comments below.

b D 3 E E See general comments below.

c D 2 E E See general comments below.

d D 3 S See general comments below.

e D 2 E E See general comments below.

f D 2 S See general comments below.

g D 3 S See general comments below.

h D 3 E X U This JPM exercises the standard post trip actions (SPT A)he JPM. The STPA s are a major part of the scenarios. This question needs to be rewritten. Also, see general comments below.

i D 3 S See general comments below.

j D 3 S See general comments below.

k D 3 S See general comments below.

G eneral C omm ent 1: P ut the J PM num ber on th e outline.

General Comment 2: It is not clear from the JPMs reviewed what is to be provided to the candidate in the way of an in itial briefing s heet.

RO Admin S 3 E E See general comment below.

A.1 .a A.1 .b S 3 S See general comment below.

A.3 S 3 S See general comment below.

A.4 S 3 S See general comment below.

SR O Ad min S 3 E E See general comment below.

A.1 .a A.1 .b S 3 E E See general comment below.

A.2 S 3 S See general comment below.

A.3 S 3 S See general comment below.

- Page 1 -

JPM# D yn LOD Errors U/E /S Explan ation (D/S) (1-5) (See below for instructions)

IC Cues Cr itical Scope Over- Job- Minut ia Focus Steps (N/B) lap Link A.4 S 3 U E T he J PM Inform ation Sh eet task d oes not m atch the JP M ou tline. Th e outline states the task is to classify an EP event and the JPM task is to determine the NRC reportability time. See general comment below.

General Comm ent 1: The administrative JPM s need an answer key (e.g. a completed form).

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically a system reconfiguration or realignment.
2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3. Check the appropriate box when an attribute weakness is identified:
  • The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
  • The JPM does not contain sufficient cues that are objective (not leading).
  • All critical steps (elements) have not been properly identified.
  • Scope of the task is either too narrow (N) or too broad (B).
  • Excessive overlap with other part of operating test or written examination.
4. Check the appropriate box when a job content error is identified:
  • Topics not linked to job content (e.g., disguised task, not required in real job).
  • Task is trivial and without safety significance.
5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 2 -

S:\DRS\OB\EXAMS\SONG S\SONG S-EXAM 10-05 ADAMS LOAD\SONG S-10-2005-NRC DRAFT EXAM COMMEN TS\SON GS-10-2005-DRAFT SIMULATO R CO MMENT S.wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen 1 2 3 4 5 6 7 8 9 10 Explanation (See below for instructions)

Set ES TS Crit IC Pred TL L/C Eff U/E/S 1 E E Suggest adding an instructor cue for the Ops Manager to direct a plant shutdown following the inoperability determination for the MSIVs and FWIVs (Event 4). This cue would be used should the SRO decide to wait for management approval/direction to shut the unit down.

2 S 3 S 4 S Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. ES: ES-301 checklists 4, 5, & 6 satisfied.
2. TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.
3. Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
4. IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
5. Pred: Scenario sequence and other factors avoid predictability issues.
6. TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
7. L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
8. Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
9. Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
10. Provide a brief description of problem in the explanation column

- Page 1 -

ADDITIONAL COMMENTS

  • Are the scenarios new? Yes.
  • Remove machine operator instructions from the front of the scenarios, but retain the "Indications available".
  • Include some blank paper space for our note taking.
  • The outline for Scenario 3 and the title for Event 4 in Scenario 3, states "Letdown Pressure Instrument fails Low", however, the machine directions and the response steps for this event respond to a temperature instrument failure.
  • The title for Event 5 in Scenario 3 states that it involves a steam flow transmitter failure whereas the response steps are for a feed flow transmitter

- Page 2 -

Retrieved from "https://kanterella.com/w/index.php?title=ML060870645&oldid=2194202"