ML093200602

2009-10-Draft Written Exam Comments

ML093200602
Person / Time
Site:	San Onofre
Issue date:	11/16/2009
From:	Apger G NRC Region 4
To:	Southern California Edison Co
References
50-361/09-302, 50-362/09-302, ES-401, ES-401-9 50-361/09-302, 50-362/09-302
Download: ML093200602 (9)

Text

ES-401 CG-2009-01 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 3 B S A, CR, 41.10 2 H 2 N S B, CR, 41.5 3 H 3 N S B, CR, 41.6 A, CR, 41.1 - Recommend changing core reload in progress to core reload has been completed Otherwise if it is assumed it is early in the reload process, it 4 H 2 E E N E could be argued there are no correct answers. Also, the reference for the question is the TS bases document. Need to change the reference as TS Bases is an SRO only document. Agree. Fixed.

5 H 3 N S D, CR, 41.7 6 F 3 B S A, CR, 41.5 7 F 3 B S A, CR, 41.3 8 H 3 N S D, CR, 41.10 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 7 OBDI 202 - IOLE Process

ES-401 SO-2009-10 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 9 H 3 B S B, CR, 41.4 10 H 3 B S B, CR, 41.6 D, CR, 41.6 - This is a KA mismatch because the question does not test on an RPS U power supply. Actually, this does test on RPS since the applicant needs to know RPS 11 F 3 U B S power supplies to know that a reactor trip will not occur. A partial trip signal is present. OK as written.

12 H 3 N S C, CR, 41.7 13 H 3 N S A, CR, 41.4 14 H 3 B S D, CR, 41.7 15 H 3 N S C, CR, 41.10 A, CR, 41.7 Need to change the valves and distractors. #2 is not discriminated since 16 H 3 B E all distractors have CLOSED for this valve. Why choose E-089 MSIV bypass vice the MSIV? Distractors have been changed.

S B, CR, 41.7 A and B are the same answer, since by the reference (LP 2XIR06),

17 F 3 U N minimum speed is 3600rpm, making A and B correct. Change A. Distractors U changed.

18 H 3 B S C, CR, 41.7 19 H 3 N S A, CR, 41.10 20 H 3 B S C, CR, 41.7 21 F 3 N S B, CR, 41.7 22 F 3 N S C, CR, 41.7 B, CR, 41.5 - Recommend changing the word significantly in the stem. If voltage is significantly higher, the EDG output breaker would trip on over current making no 23 H 3 E N E correct answer. Change significantly to some value greater than bus voltage.

Change C to RAISE to establish a negative VAR load Changed distractors.

24 H 3 N S B, CR, 41.11 25 F 3 B S C, CR, 41.7 26 H 3 N S C, OR, 41.10 27 H 3 M S A, CR, 41.4 Page 2 of 7 OBDI 202 - IOLE Process

ES-401 SO-2009-10 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 28 F 3 N S B, CR, 41.9 B, CR, 41.14 - CAF to ensure SG pressure rises due to improved thermal heat 29 H 4 N E transfer resulting in less governor valve travel for the same power level.

30 H 3 N S C, CR, 41.10 31 F 3 N E D, CR, 41.7 Add is after Pressurizer level in last part of stem. Done.

32 H 2 B S D, CR, 41.9 B, CR, 41.7 Remove if Narrow Range Level Transmitter from stem or remove 33 H 3 N E second bullet. Done.

34 H 3 N S C, CR, 41.5 35 H 3 N S D, CR, 41.13 36 H 3 N S A, CR, 41.7 37 F 3 B S D, CR, 41.11 38 F 3 N S D, CR, 41.4 39 H 3 B S B, CR, 41.6 40 H 3 M S C, OR, 41.14 Formula sheets are the reference.

41 F 3 B S B, CR, 41.5 42 H 3 N S D, CR, 41.7 43 H 3 B S A, CR, 41.3 44 H 4 N S D, CR, 41.7 Capitalize maximum in stem. Done.

45 F 3 N S B, CR, 41.8 Change Isolation to Actuation Done.

46 H 3 B S D, CR, 41.5 47 F 3 B S B, CR, 41.6 48 H 3 M S C, OR, 41.14 Steam tables are the reference.

49 H 4 N S C, CR, 41.10 50 H 3 N S D, CR, 41.5 51 H 3 M S A, CR, 41.10 Page 3 of 7 OBDI 202 - IOLE Process

ES-401 SO-2009-10 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 52 H 3 B S D, CR, 41.7 53 F 3 N S D, CR, 41.4 54 H 3 N E A, CR, 41.4 Indent the last five bullets. Done.

E C, CR, 41.10 - It is not clear from the documentation provided how the loss of bus 55 H 3 E N 2A08 is related to the KA (Grid Disturbances) 2A08 is powered from output of main S generator. Question OK as written.

56 F 3 B S D, CR, 41.10 57 F 3 N S A, CR, 41.13 C, CR, 41.5 - This questions KA is the reason for loss of steam dumps given lowering vacuum. This question can be answered without knowing the reason, just the setpoint. Recommend deleting the setpoint portion of the distractors. Change 58 F 3 E B E stem to following describes the reason why SBCS valves close when the SBCS LVI setpoint is reached? Remove the setpoints from the distractors. Question changed.

D, CR, 41.6 Channel designators start with JE in the stem; however, the references 59 H 3 N E all start with JI. Which one is correct? Changed to JI.

60 F 4 N S C, CR, 41.5 61 H 3 N S A, CR, 41.11 62 F 3 N S B, CR, 41.4 63 H 3 N S B, CR, 41.10 D, CR, 41.9 - Three of the four distractors, including the correct answer, are true /

U false statements. Add in this condition? at the end of the stem. For distractors A 64 F 3 U N E and B, change first part of sentence to A pump For distractors C and D, change The Containment Sump Pumps to They. Changed distractors and stem.

65 F 3 N S D, CR, 41.10 66 F 3 N S A, CR, 41.10 D, CR, 41.10 - Recommend adding a procedure reference to the stem to preclude 67 F 3 E B E the potential of more than one correct answer. Add In accordance with SO23-3-3, Title, to the stem. Done.

68 F 3 B S B, CR, 41.4 Page 4 of 7 OBDI 202 - IOLE Process

ES-401 SO-2009-10 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 69 F 4 M S C, CR, 41.10 70 F 3 B S D, CR, 41.10 B, OR, 41.7 - Could not verify accuracy of the distractors due to not having the entire 71 H 4 E B E drawing 30103. Check this. Question OK.

A, CR, 41.12 Weak tie to KA; however, blocking access to containment during a 72 F 3 N S security event could be considered radiological safety principle.

73 H 3 N S A, CR, 41.12 74 F 3 N S B, CR, 41.10 75 F 3 N S C, CR, 41.10 76 H 3 N S C, CR, 43.5 77 H 3 N S B, CR, 43.5 C, CR, 43.2 - Typically we do not expect candidates to have LCOs greater than one 78 H 3 E N E hour memorized. Consider deleting the time element from the distractors. Agree, remove the time requirements. Revised question.

79 H 3 B S A, CR, 43.5 80 H 3 N S C, CR, 43.5 81 H 3 N S D, CR, 43.5 82 H 3 N S C, CR, 43.2 C, OR, 45.5 - The stem asks about NRC notification and the distractors address EP 83 H 3 E N E classifications. Recommend revising the stem. The stem has a typo (he)

Agree. Fixed.

84 H 3 N S D, CR, 43.5 85 H 3 N S D, CR, 45.2 86 H 4 N E A, CR, 43.2 Can the completion times be removed? Yes, Done.

87 H 3 B S B, CR, 43.5 88 F 3 N S A, CR, 43.2 th 89 H 3 N E D, CR, 43.2 Remove It has been determined that from the 4 bullet. Done. Page 5 of 7 OBDI 202 - IOLE Process

ES-401 SO-2009-10 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 90 H 3 N S A, CR, 43.5 91 H 3 N S B, CR, 43.5 92 F 2 B S A, CR, 43.1 93 F 2 N S B, CR, 43.7 94 F 2 B S A, CR, 43.1 95 H 3 B S C, CR, 43.5 U C, CR, 41.10 - The procedure change process is considered RO knowledge. This is 96 F 2 U B S a procedure used by SORs only, and meets their JTA. OK as written.

97 F 3 N S B, CR, 43.2 98 F 3 N S C, CR, 43.4 99 H 3 B S A, CR, 43.5 100 H 3 B S A, CR, 43.5 B= 27 F= 30 (40%) E= 7 12 Additional Notes: After review by chief examiner, there was 1 unsat question on the RO TOTALS: M= 5 H= 45 (60%) U= 3 1 RO exam.

N= 43 S= 65 62 B= 8 F= 6 (24%) E= 2 4 SRO TOTALS: M= 0 H= 19 (76%) U= 1 Additional Notes:

N= 17 S= 22 20

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is __3.04_ on the RO exam and 2.84 on the SRO exam. The RO average difficulty is higher than normal and the SRO average difficulty is a typical value for this reviewer.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 1 43.1 = 2 Page 6 of 7 OBDI 202 - IOLE Process

ES-401 SO-2009-10 7 Form ES-401-9 41.2 = 0 43.2 = 8 41.3 = 2 43.3 = 0 41.4 = 8 43.4 = 1 41.5 = 9 43.5 = 12 41.6 = 6 43.6 = 0 41.7 = 17 43.7 = 1 41.8 = 1 41.9 = 3 41.10 = 18 41.11 = 3 41.12 = 2 41.13 = 2 41.14 = 3

5. The answer distribution is: RO / SRO A= 17 / 8 B= 20 / 5 C= 17 / 7 D= 21 / 5

6. There are 4 / 1 (RO/SRO) questions with attachments provided.

7. This review was performed by S. Garchow the week of 8/17/2009. Page 7 of 7 OBDI 202 - IOLE Process

SONGS October 2009 Synopsis of Written Exam Changes (These Comments are from additional OPS validation following the Rev. 0 NRC Exam Submittal)

1. #11 - Change wording of Pressurizer Heater from Control to Proportional in Distractor A.

2. #13 - Modified Distractors C and D based on Validation Exam results.

3. #15 - Removed last portion of Distractor C to avoid confusion over required actions.

4. #16 - Changed psig to psia in Stem.

5. #17 - Modified Distractor B to better reflect actual condition of Main Feedwater Pump post-trip.

6. #23 - Changed wording in Part 2 of all Distractors to read "ADJUST Voltage Regulator Control Switch to establish... to eliminate Distractor imbalance.

7. #29 - Changed wording on Distractors C and D from Reactor Coolant System... to Steam Generator... to be consistent with the heat transfer equation being used.

8. #31 - Modified wording in Stem to reflect title used in the Operating Instruction and made minor Distractor changes to address control of Pressurizer Level setpoint as opposed to Pressurizer Level.

9. #32 - Changed last bullet in the Stem to 75°F and modified unobserved parameters from 15 minutes to five minutes. Removed the word foggy as humid conditions nominally exist at SONGS.

10. #33 - Added "...off-scale high" at the end of the Stem to ensure the examinee knows the extent of the failure.

11. #35 - Added a bullet to the Stem to identify a Blowdown Radiation Monitor in alarm to enhance correctness of question.

12. #38 - Changed wording in Stem to ...Fire Detection System failure?

13. #39 - Changed wording of Item #1 in Stem to read "MG Set Output Contactor RED Lights as only the output contactors are affected. Added Reference Drawing from System Description SD-SO23-520, Figure 2 to Technical References on Worksheet.

14. #42 - Changed Distractor D from There is NO boration flowpath aligned... to There is NO boric acid flow... as this better defines the condition that exists. This was necessary because the gravity flow path is aligned; however, there is no boric acid flow.

15. #45 - Added ...required to... to Stem per NRC review.

16. #46 - Changed last bullet in Stem to from 50 to 75 psia to raise resulting pressure greater than 2300 psia.

17. #47 - Added 3rd bullet to ensure that examinee knows the CEAs have not inserted.

18. #49 - Added the words ...and lower REPCET - THOT T to Distractor C to better reflect all the Natural Circulation Criteria that must be met.

19. #54 - Added bullet to the Stem to provide information on flow indication for each Unit.

20. #55 - Added due to" statements to Distractors B and C to eliminate Distractor imbalance.

21. #57 - Changed wording of Distractors A & D from SECURE... to LOSS of...

22. #66 - Changed color of paper in Distractor A from yellow copy to goldenrod controlled copy to reflect actual color of paper and differentiate it from Distractor B.

23. #67 - Removed the Second Point of Control Cabinet reference number 2A040 from the 2nd bullet in the Stem to eliminate confusion with the breaker number being operated.

24. #69 - Change Distractor D from 532°F to 535°F as some copied versions of the exam looked like 522°F.

25. #70 - Removed the word MINIMUM from the Stem to avoid confusion with the Containment Structure Equipment Hatch Shield Doors which is a recent Technical Specification change.

26. #72 - Added the reason for the Security Code Red Event classification to Distractor A to better define what was occurring at the plant. Changed Distractor B from HI HI alarm to HI alarm to eliminate the possibility that the cause of the alarm was a Refueling Cavity Seal rupture.

Modified Distractor C to eliminate the implication that a continuous dilution event was occurring.

SONGS October 2009 Synopsis of Written Exam Changes (These Comments are from additional OPS validation following the Rev. 0 NRC Exam Submittal)

27. #77 - Changed last word in the 2nd bullet of the Stem from ...diagnosed. to ...completed.

SO23-12-2 must be entered in order for the correct diagnostic tool to be used. Changed THOT from 525°F to 522°F since Reactor Coolant Pumps are running.

28. #78 - Modified the location of the leak and added a high vibration condition in the Stem to clarify the cause of the leakage based on Validation Exam results. Modified Stem conditions to be consistent with leak location.

29. #79 - Removed ...on Unit 2/3 designation from the end of Distractors A and C as it does not materially affect selection of answer. Added bullet to Stem reporting Steam Generator levels.

Modified Distractor C as it seemed plausible to enter the Functional Recovery Procedure based on Validation Exam results.

30. #80 - Reworded 1st bullet and removed 3rd bullet to avoid confusion over actual Reactor Coolant System level.

31. #81 - Specified the time after the large break LOCA in the Stem and removed the statement

...earlier than specified. from Distractor D. Raised RWST concentration from 2850 ppm to 2875 ppm as some copied versions of the exam looked like 2650 ppm.

32. #82 - Reconfigured Stem and Distractors based on Validation Exam results.

33. #85 - Modified Distractors A, B, and C based on Validation Exam results. Added ...cross-tying AFW Trains... to Distractor D.

34. #86 - Added Annunciator conditions to Stem in order to differentiate between a channel failure and low pressure condition. Added Annunciator Technical References. Modified Distractors based on Validation Exam results.

35. #87 - Changed CONSTANT to RISING in the 2nd from last bullet in the Stem. Added a bullet to differentiate between the responses of the Main Feedwater Pumps. Modified Stem and Distractors to better describe how the entire Feedwater Control System was functioning.

36. #88 - Modified Distractors B, C and D due to Validation Exam results.

37. #89 - Changed order of the Distractors for better readability and modified Distractor B based on Validation Exam results.

38. #90 - Modified Stem and Distractors due to Validation Exam results. Modified Technical References as required.

39. #91 - Changed wording in Distractor D from "... reduced..." to ...elevated... as this answer could be correct as written.

40. #93 - Changed the wording in Distractor B to reflect actual area that should be monitored vice room number (Fuel Transfer Tube Area = Penetration Room 111). Changed Distractor D to equalize Distractor length between A & C and B & D.

41. #95 - Minor wording changes in Stem and Distractors B and C.

42. #97 - Changed wording in the Stem to reflect actual condition of the Main Steam Safety Valves (they are gagged vice isolated). Added BOC condition to the Stem. Removed first three words from Distractor C for Distractor balance and changed peak centerline temperature to reflect an EOC value. Modified Distractors A and D based on Validation Exam results.

43. #99 - Changed 2nd bullet in Stem to reflect that Pressurizer Safety Valve is still open. Added last bullet to Stem to reflect current condition of RCS subcooling. Modified wording in Distractor B to differentiate it from Distractor A.

44. #100 - Modified procedure entry for Distractor B due to Validation Exam results. Modified information for Pressurizer level, pressure and SG level in the Stem.