ML110200472

From kanterella
Jump to navigation Jump to search

NRC Comments on Draft Licensee-Developed Exam (Written & Operating Tests) (Folder 2)
ML110200472
Person / Time
Site: Susquehanna  Talen Energy icon.png
Issue date: 11/01/2010
From: Caruso J
Operations Branch I
To:
Susquehanna
Hansell S
Shared Package
ML102180025 List:
References
TAC U01793
Download: ML110200472 (27)
v • d • e


Text

Draft Operating Test Comments Scenarios General Comment - Need more instrument failures currently mostly component malfunctions also scenarios lacking in major integrated electrical plant challenges.

Scenario No.1 (Re-designated Spare Scenario):

1. State in turnover target for raising power.
2. Event number 10 overlaps with spare scenario.
3. Re-designate this scenario the spare.

Scenario NO.2:

1. Replaced simple component failure with an instrument bus failure.
2. Changed order of last malfunctions to improve flow of scenario.
3. Typo in Scenario Summary page 3, says 100MWe decrease when it should read 50 MWe.
4. Typo on page 28," ... trip both RRPs "one vice once" at a time ... "
5. Deleted critical task for SBLC since no longer critical. Made proposed failure of SBLC more serious such that no longer a success path which then required the crew to intentonially lower level to control power during the ATWS. This change ensured that the planned critical task for lowering level to control power was indeed critical.
6. Added new critical task to Emergency Depressurize the RPV due to exceeding Heat Capacity Temperature Limit (HCTL).
7. Added classify the event for the SRO applicant following the scenario.

Draft Spare (Re-designated Scenario No.1):

1. Scenario was recommended as spare (re-designated this scenario number 1).
2. Changed order of malfunction no. 5 moved up to event no. 2 to improve the flow of the scenario.
3. Deleted malfunction no. 4 which was a simple component malfunction Le., pump trip and start standby pump.
4. Added failure of an auxiliary buss which a major electrical plant fault adding complexity and diversity to the scenario set.
5. Deleted event no. 10 which was redundant left this event in new spare scenario originally designated as Scenario no.1).
6. Deleted Inadvertent HPCI start as a Critical Task - exam team determined did not meet criteria for a CT.

SRO Admin JPMs

1. A.1.1, Conduct of Operations, "Authorize Bypass of Rod Block Monitor during Startup" Task appeared to be equivalent to direct look-up. Replaced task.
2. A.1.2, Conduct of Operations, Implement ON-145-04, "Reactor Water Anomaly" - step 8 Evaluator note does not agree with standard. Note removed.
3. A.2 - ok
4. A.3, Radiation Control, Respond to SGTS Rad Monitor "A" failure While Purging Primary

- LOD borderline too easy. The procedure tE~lIs you the affected TRM and the TRM tells you the affected TS. The Task was revised to increase LOD. Provided indications of actual Hi SGTS Exhaust Radiation Monitors. The applicant needs to identify SGTS failure to isolate.

5. A.4, Emergency Plan, provide keys for Emergency Notification and PAR forms.

RO Admin JPMs

1. A.1.1, Conduct of Ops, Evaluate Overtime Request with Respect to Work Hour limits, originally proposed task too easy. Increased LOD by adding a second week of work history data that needs to be evaluated and must also identify that does not meet 34 hour3.935185e-4 days <br />0.00944 hours <br />5.621693e-5 weeks <br />1.2937e-5 months <br /> break in a 9 day period.
2. Note: Other RO Admin JPMs were acceptable as proposed.

Simulator and In-Plant "IPMs

1. "IPM C, Restore DW Cooling was designated as alternate path. This JPM was determined to be acceptable as a non alternate path JPM. The JPM was just implementing the procedure as written and was not exercising an alternate procedural path.
2. "IPM D, improved alternate path - changed alternate path from EDG output breaker failing to close automatically to recognizing that the D ESW pump failed to start in Auto and manually starting the ESW pump.
3. "IPM F, prepare reactivity manipulation sheet and control rod movement sheet.
4. "IPM H. determined to not meet alternate path criteria. "IPM replaced with new alternate path "IPM.

Draft Operating Test Comments Scenarios General Comment - Need more instrument failures currently mostly component malfunctions also scenarios lacking in major integrated electrical plant challenges.

Scenario No.1 (Re-designated Spare Scenario):

1. State in turnover target for raising power.
2. Event number 10 overlaps with spare scenario.
3. Re-designate this scenario the spare.

Scenario NO.2:

1. Replaced simple component failure with an instrument bus failure.
2. Changed order of last malfunctions to improve flow of scenario.
3. Typo in Scenario Summary page 3, says 100MWe decrease when it should read SO MWe.
4. Typo on page 28," ... trip both RRPs "one vice once" at a time ... "

S. Deleted critical task for SBlC since no longer critical. Made proposed failure of SBlC more serious such that no longer a success path which then required the crew to intentonially lower level to control power during the ATWS. This change ensured that the planned critical task for lowering level to control power was indeed critical.

6. Added new critical task to Emergency DeprE~ssurize the RPV due to exceeding Heat Capacity Temperature Limit (HCTl).
7. Added classify the event for the SRO applicant following the scenario.

Draft Spare (Re-designated Scenario No.1):

1. Scenario was recommended as spare (re-designated this scenario number 1).
2. Changed order of malfunction no. S moved up to event no. 2 to improve the flow of the scenario.
3. Deleted malfunction no. 4 which was a simple component malfunction i.e., pump trip and start standby pump.
4. Added failure of an auxiliary buss which a major electrical plant fault adding complexity and diversity to the scenario set.

S. Deleted event no. 10 which was redundant left this event in new spare scenario originally designated as Scenario no.1).

6. Deleted Inadvertent HPCI start as a Critical Task - exam team determined did not meet criteria for a CT.

SRO Admin JPMs

1. A.1.1, Conduct of Operations, "Authorize Bypass of Rod Block Monitor during Startup" Task appeared to be equivalent to direct look-up. Replaced task.
2. A.1.2, Conduct of Operations, Implement ON-145-04, "Reactor Water Anomaly" - step 8 Evaluator note does not agree with standard. Note removed.
3. A.2 - ok
4. A.3, Radiation Control, Respond to SGTS Rad Monitor "Au failure While Purging Primary

- LOD borderline too easy. The procedure tells you the affected TRM and the TRM tells you the affected TS. The Task was revised to increase LOD. Provided indications of actual Hi SGTS Exhaust Radiation Monitors. The applicant needs to identify SGTS failure to isolate.

5. A.4, Emergency Plan, provide keys for Emergency Notification and PAR forms.

RO Admin JPMs

1. A.1.1, Conduct of Ops, Evaluate Overtime Request with Respect to Work Hour limits, originally proposed task too easy. Increased LOD by adding a second week of work history data that needs to be evaluated and must also identify that does not meet 34 hour3.935185e-4 days <br />0.00944 hours <br />5.621693e-5 weeks <br />1.2937e-5 months <br /> break in a 9 day period.
2. Note: Other RO Admin JPMs were acceptable as proposed.

Simulator and In-Plant JPMs

1. JPM C, Restore OW Cooling was designated as alternate path. This JPM was determined to be acceptable as a non alternate path JPM. The JPM was just implementing the procedure as written and was not exercising an alternate procedural path.
2. JPM 0, improved alternate path - changed alternate path from EDG output breaker failing to close automatically to recognizing that the 0 ESW pump failed to start in Auto and manually starting the ESW pump.
3. JPM F, prepare reactivity manipulation sheet and control rod movement sheet.
4. JPM H. determined to not meet alternate path criteria. JPM replaced with new alternate path JPM.

NRC Comments on Walkthrough JPMs 12-15-10 Sim JPM 'a'. Emergency Cond Pump Ops (44.0N.1792.1Ol):

  • Need to add a reference for possibly using the OP to shut down the Condensate Pump.

Proposed Resolution: Added an Evaluator Note that the student may refer to OP-144-001 to shut down the Condensate Pump. RMP 12-21-10 Sim JPM 'b', Post Scram Recovery of RRP (64.0P.4841.1Ol):

  • Step for taking the Controller for RRP A to 0% was not critical.

Proposed Resolution: Made Step 10 a critical step and put an Evaluator Note describing where on the HMI screen the controller is located.

Also, added annunciator window numbers for alarms received in steps 12 and 17, along with an Evaluator Cue after Step 18 that another Operator will log the Recirc temperatures into eSOMS. RMP 12-21-10 Sim JPM 'c', Reset! restore DW Cooling (34.EO.1619.101):

  • Need to change this JPM from an Alternate Path to a non-Alternate Path.

Proposed Resolution: Renamed JPMfrom 34. EO. 1619.151 to 34. EO. 1619. 101 and removed the Alternate Path steps.

Also, added an additional cue to the cue sheet that CIG has not been restored iaw ES-184 002 to remove any confusion as to whether or not the student should close the CIG valves in Step 7 and restore the CIG compressors in Step 16.

Re-snapped the IC with U1IA cross-tied to U2 to remove the alarm for IA low pressure.

Filled out and approvedfor use a copy ofES-134-001 up to step 4.3. RMP 12-21-10 Sim JPM 'd', Energize a Dead 4KV ESS Bus (Alt Path) (04.0N.1203.251):

  • Is there a time delay for the auto start of the D ESW Pump?

Proposed Resolution: Yes, there is a time delay for the auto start ofthe D ESW Pump even with a manual start ofthe D EDG. However, there is a malfunction on D ESW Pump so that the auto logic fails. Added a fault statement prior to Step 18 to ensure the evaluator is aware that the student needs to manually start the D ESW Pump. RMP 12-21-10 Sim JPM 'e'. RHRSW to Fuel Pool (35.0N.1662.1OU:

  • ON-135-00l needs to show the steps as complete that are described in the cue sheet.

Proposed Resolution: Added cueing to ON-135-001 to show the steps complete as described in the cue sheet.

Also, added an Evaluator Cue after Step 2 that the TS actions in Step 3.2 ofthe procedure are being complied with.

Added an Evaluator Cue after Step 4 that the valves that are normally closed and de energized were determined to be closed by the current surveillance.

Changed the Standard in Step 6 to reflect that the LOCA override light will only momentarily light up and that it will go out when the key is released. Also, that the annunciator will not come in due to no LOCA signal being present. Added an Evaluator Cue to question the student as to whether or not they would expect to have LOCA isolations based on initial conditions.

Terminated the JPM before the step to shut offthe RHRSW Pump and close the cross-tie valves. RMP 12-21-10 Page 1 of 2

NRC Comments on Walkthrough JPMs 12-15-10 Sim JPM 'f, Respond to a Stuck Rod (Alt Path) (55.0N.2000.152):

)

  • Should remove the stuck rod malfunction when student reaches 300 psid instead of 350 psid.

Proposed Resolution: Changed JPM to remove the stuck rod malfunction at 300 psid.

Added an Evaluator Note to not provide the student with ON-155-001 until requested by the student.

Added an Evaluator Note that the student should not use the full-inlfull-out display to detennine the rod position in Step 6. RMP 12-21-10 Sim JPM 'g', Insert SLO Setpoints (78.0P.3680.1O 1):

  • The Booth Operator should place the APRM Bypass switch in the appropriate position instead of the student as the student will be in the Lower Relay Room.

Proposed Resolution: Changed Step 5 to reflect that the Booth Operator will place the APRM Bypass switch to APRM 3.

Added an Evaluator Note up front to remind us to reset the triplinop memory on the APRM Voter prior to starting the JPM.

Added Steps 22, 23, and 24 for the student to perfonn instead of having another Operator do them. Also, added Step 26 for the student to cOfifinn that the joystick was taken out of bypass. RMP 12-21-10 Sim JPM 'h', Perform Alternate RD Using Bypass Valves (Alt Path) (93.EO.1129.151):

  • Substitute Perform Alternate RD Using Bypass Valves (Alt Path) (93.EO.1129.151) instead of -38" Isolation Verification (alt path) (59.0N.2084.151).

Proposed Resolution: Substituted 93. EO. 1129. 151 for 59. ON. 2084. 151.

Allowed one SRV (G) to be opened by the student.

Added Evaluator Cueing and student responses to mimic the EO-100-112flow chart starting at Step RD-8 through Step RD-13.

Changed the initiating cue sheet #2 to be for ROs only. The SROs will now use a marked up EO-100-112 board. RMP 12-21-10 Page 2 of2

) NRC Comments on Walkthrough JPMs 12-7-10 SimJPM 'd', Energize a Dead 4KV ESS Bus (All Path) (04.0N.1203.251):

  • Alternate path actions not sufficient.

Proposed Resolution: No changes made. After phone consultation with Chief Examiner, and further review ofalternate path actions, it was agreed that no changes were required and actions were sufficient. TN 12-7-10 Sim JPM 'c', Reset / restore OW Cooling (Alt Path) (34.EO.1619.151 ):

  • Alternate path actions not sufficient.

Proposed Resolution: Changed initial conditions to make CIG "NOT restored" and removed reference to CIG status from "Task Conditions". The JPM now requires candidates to evaluate the status of CIG (a support system for this task), recognize that it has not been restored, and utilize a separate procedure section to restore CIG prior to resetting and restoring DW cooling. TN 12-4-10

.~ Sim JPM 'h', -38" Isolation Verification (alt path) (59.0N.2084.151) Safety Function 5:

  • Alternate Path actions not sufficient.

Proposed Resolution: Delete JPM and revise RO walkthrough outline with newly developed alt path JPM for safety function 3. TN 12-3-10.

Perform Alternate RD Using BPVs (93. EO. 1129. 151 ) Safety Function 3:

Candidates are directed to perform rapid depressurization using ADS. They must recognize that ADS and SRVs fail. They are then directed to assist the CRS in evaluating systems available for alternate rapid depressurization. Following successful system status evaluation, they are directed to perform alternate RD using bypass valves (provided they determine that BPVs are available).

ES*401 Written Examination Review Worksheet Form ES*401*9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (FtH) (1-5) Stem Cues TtF Credo Partial Job- Minutia #I Back- Q= SRO BtM/N U/E/S Explanation Focus Dist. Link units ward KIA Only 1 H 3 N S 2 H 3 B S 3 F 3 B S Ok direct power supply Q in limited numbers 4 F 2 X N E 1) Concern that 1Y218 powers IRM recorders. B may be partially correct in that IRMs will not accurately indicate on recorders on 1C652. Added to the stem, "on ANY control room indicator" to clarify and make B incorrect.

2) Another simple power supply Q.

5 H 2 N S 6 H 2 N S 7 F 2 N S 8 H 3 X N E Appears B&C are not plausible - are there any safety related pumps that flow set is a combination of injection and min flow? If not then appears not plausible.

Revised stem and distractors to improve plausibility.

Note: Written Exam Draft Received 12/2. Review completed 12/9 and comments on the SRO portion of written provided 12/10. RO comments provided on site Monday, 12/13.

Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional infonnation regarding each of the following concepts.]

1. Enter the level of knowiedge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2. Enter the level of difficulty (LaD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 4 range are acceptable).
3. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more infonnation is needed, or too much needless infnnn"tinn\

The stem or distractors contain cues (i.e., clues, specific detenniners, phrasing. length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowiedge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.1.
7. Based on the reviewer's jUdgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES*401 Written Examination Review Worksheet Form ES-401*9

2. 5. Other 8.

~1~~ IFocus Stem Icue~ I BaCk-I Q=

ward KJA Explanation 9 H 2 B S 10 F 3 x N u A&B do not appear to be plausible are there any standby filters installed in your power plant that auto shift? If not distracters don't appear plausible.

Revised stem and distracters to remove filter swap concept 11 F 3 B S H I 3 13 H 3 Distractor B needs to be more dear as to what is happening with the N2 supply open why would that cause 02 to lower but at a slower rate. Revised "8" justification to elaborate on plausibility.

14 F 3 Revise the stem condition from 475# to 440# to make it closer to actual limit to increase plausibility of A&D choices. Done 15 H 3 16 H 3

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues TIF Creel. Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 17 H 3 N E Technically is it correct that at 35# that all the ADS valves would still be expected to be open? Is this really operationally oriented?

Revised stem and distractors to make more operationally oriented and ED is relevant.

18 H 3 N S 19 H 2 X N U 1) A&B do not appear to be plausible. You would not design a system for pressure suppression that would result in containment pressure rapidly increase on an ED- your obviously bypassing the suppression function and

2) Similar to a simulator JPM.

Randomly selected new KIA and replaced with a bank Q.

20 H 3 N S For answer "A" reference material does not mention F01SA closing does this occur? Reference provided.

21 F 3 N S 22 H 3 N E Don't like the term LCO met or not met. The 'A' manual initiation is inoperable and the required action needs to be taken. In that sense the LeO is met. If the question asked the required actions for these conditions then it would be straight forward. Suggest changing it to required action statements.

Clarified wording of answer choices using tennino/ogy used at SSES regarding TS Action statements.

23 F 1 N E Simple power supply question. Already several other direct power supply questions. It is desirable rather than directly asking power supplies to pose scenarios that involve loss of power and require operators to determine plant impact - more operationally oriented.

Revised Q and distracters.

24 H 2 M S

ES*401 Written Examination Review Worksheet Form ES-401*9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist Link units ward KIA Only 25 H 3 N S 26 H 2 B S 27 F 2 X N U Distractor plausibility unless RHR can be X-connected with SW or Fire Pumps then the distracters are not plausible - replace Q and reselect a random KIA.

Randomly selected new KIA and replaced with a bank Q.

28 H 3 N S 1) Typo in stem add "ing to word "follow"

2) Don't understand explanation of correct answer. Power lost to MCC 1B237 affects SP cooling valve F024A but motor operated should stay open- smoke screen? The correct answer should include FO-28A closing on HI OW.
3) KIA match is this Q testing logic on Hi OW with SP cooling in service or SP cooling valve power supplies?

Maybe change one or more of the distracters to show position of F024A.

4) This is Q is H not F Changes made to address comments.

29 H 2 N S 30 H 2 X N E 1) Delete first sentence in stem. This is providing information that the applicant should know and will make LOD higher.

Change made 31 H 3 N S Is this req'd RO knowledge from memory? Do the ROs need to be qualified on the Refuel Bridge?

Facility stated there ;s a valid LO for this information.

5. Other 6. 7. 8.

Stem ICues Focus I

Back-I Q= SRO I B/MIN I UtE/S ward KiA Only Explanation 32 H 2 x N U I Distractors don't appear to be plausible. Unless CRD suction can be realigned to other sources no! plausible - replace Q and reselect a random KiA.

Randomly selected new KIA and developed new Q.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

10. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
11. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
12. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (I.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (I.e., the question has a valid KiA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic ievel of accuracy or inconsistent units (e.g., panei meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KiA and those that are designated SRO-only (KiA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "un ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 33 H 3 N S 34 H 3 M S 35 F 1-2 X B U 1) LOD=1

2) A&C not plausible very basic system knowledge is that get insert signal first;
3) Recommend drawing anoiher KiA ai random and replacing Q.

Randomly selected new KJA and wrote new Q.

36 F 2 X N E A&D Distractors not plausible - no flow in recirc loops makes A implausible and FW return temp provides no indication of RPV temp Changed stem of Q so that applicant has to determine status of recirc pumps which makes "A" plausible and changed "D" distractor to make more plausible.

37 F 2 N E Revise B&D to 30% vice 20% which is limiter #1 set-point improves plausibility. Change made 38 F 3 B S 39 F 3 E Water level must be greater than or equal to +45". Change level in stem to +50" in stem closer to required level makes it a little more challenging.

Change made also revised stem conditions to make +50" plausible.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws I 5. Other I 6. 8.

Q# I LOK LOD (F/H) (1-5) Stem Cues Explanation Focus 40 I H I 2 I S I Borderline LOD. Maybe the Q could be improved by changing from a half SCRAM to having a channel failed such that they need to diagnose a half scram first.

Revised as suggested.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

13. Enter the level of knowiedge (LOK) of each question as either mO;:Jmp.nt;:J1 or (H}igher cognitive level.
14. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy- scale (questions in the 2 - 4 range are acceptable).
15. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g .. unclear intent. more information is needed. or too much needless information).

The stem or distractors contain cues (I.e .* clues. specific determiners. phrasing. length. etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired. more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (I.e .* the question has a valid KIA but. as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e.* it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g .* panei meter in percent with question in gaiions).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (Blank. (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2,f.
7. Based on the reviewer's jUdgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U' ratings (e.g .* how the Appendix B psychometric attributes are not being met).

ES*401 Written Examination Review Worksheet Form ES-401*9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N UlE/S Explanation Focus Dist. Link units ward KIA Only 41 F 3 N S 42 H 2 X N E 1) The cause of the failure to intiate a RPS scram from the TCV fast closure may make an applicant think that RPS is failed and they will eliminate B. Wording in stem may need to be more clear as to what the failure is. Maybe, you should consider wording the first bullet to TCV Fast closure input to RPS failed.

2) LOD borderline - not remembering HP SRAM set point is only thing that would make C&D plausible?

Stem revised and C&D revised to improve plausibility and increase LOD.

43 H 3 N S 44 H 3 N S 45 F 3 B S

4. Job Content Flaws 8.

Partial I Job- I Minutia #1 I Back-I Q= Explanation Link units ward KiA x x 1) Distractors A & B are vague and could be considered partially correct depending on what "requirements" means.

2) Don't like the Q it is not very operationally oriented achieving safe SID inside or outside CR is like a trick Q.
3) Recommend Q more operationally oriented.
4) Furthermore Q does not appear to meet KiA actions contained in AOPs Developed new Q using same KIA.

47 48 B E With low level in SP "D" kind of cues correct answer.

Recommend revising A to read "SPOTMOS Div 2 Lower RTDs ONLY" gives more balance to answer choices.

Comment incorporated.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.J

16. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
17. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2.- 4 range are acceptable).
18. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (Le., the question has a valid KiA but, as written, is not operational in content).

The question requires the recall of knowledge that is too speCific for the ciosed reference test mode (Le., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KiA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.1.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 49 H 3 N S 50 F 2 B S Technical information does not show the difference between zones.

51 H 3 N E UN maybe enhanced by listing in second part RPV level is below min steam cooling level. SSES doesn't use Min steam cooling term but -205" added to itA" which is top ofjet pump riser.

52 F 3 N S 53 H 2 N S 54 H 2 N S 55 H 2 X N U 1) C&D are not plausible since still have battery power to power bus.

2) B is true. The panel is always energized from all sources, battery chargers and battery. Technically ask which is carrying the load is the battery charger. Note that info supplied shows that battery charger is limited to 40 amps.

If exceeded 40 amps the battery would supply the load.

Replaced with Q from 2004 NRC exam testing same KIA.

56 H 2 N S

ES*401 Written Examination Review Worksheet Form ES*401*9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (FtH) (1-5) Stem Cues TtF Credo Partial Job- Minutia #1 Back- Q= SRO StM/N utEtS Explanation Focus Dist. Link units ward KIA Only 57 F 1 X N U 1) Distractors not plausible no loss of offsite power - no reason for power to transferred

2) Simple recall of actions on control room abandonment.
3) No conditions stated in the stem which would make more challenging. E.g. If partial loss of offsite power and not all ES buses powered.

Significantly Revised Q including a LOOP.

58 H 4 N S 59 H 3 B S 60 H 2 S S 61 F 2 X N U A & B not plausible. RWM normally does not enforce any blocks above LPSP and LPAP.

Revised stem & disfiacters.

62 H 2 X N U Plausibility of S, C, and D. Knowing 2 facts TCVs close on a load reject and bypass valve capability 25% can easily pick correct answer and eliminate B, C, D.

Replaced with Bank Q testing same KIA.

63 F 3 X S E Why is A plausible. Turbine building vent exhaust, should not be connected to RB stack monitor panel.

Revised "A" to make more plausible 64 H 2 X N U A&B only logical if don't remember DW cooling Hi DW pressure greater than 1.72 psig Randomly selected new KIA and developed new Q.

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

00 LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KiA Only 65 H 3 N S 66 F 2 B E Typo A and D "complete" not "completed" Fixed typo and clarified stem that you are the designated A TC RO.

67 F 2 X B S 1) Didn't include next page of reference is there any other methods listed in the procedure?

2) Should list the procedure in the stem OP-000-002 to tighten stem

, Listed procedure in stem and verified no other methods listed in procedure.

68 F 3 B E 1) Make "A" power closer to limit. i.e. 21 % and make action of D the same for symmetry.

2) Also make "D" thermal power 24% closer to limit.

Currently if only know correct actions can pick D without knowing which limit is violated.

Revised as suggested.

69 H 3 X N E 1) C does not appear plausible since alarm does not affect pump operation.

2) Also print needs to be listed as reference required. Can not answer without the print.
3) Technically the training material is wrong. It shows the LT upstream of PCV-14811C. However on M-148 it is shown correctly.

Revised "C"; listed print needed on references required; will revise training material after exam.

70 H 3 B E The correct answer D has an additional failure built in. Seems like you are asking what could give these indications.

Revised wording of "D" plus justification.

71 F 2 N 5

1. 2. 3. Psychometric Flaws 4. Job Content Flaws I 5. Other I 6. I 7. I 8.

Q# I LOK LOD (FfH) (1-5) Stem Cues TfF Explanation Focus 72 F 1-2 X 8 U IWhat makes A & D plausible.

I Revised distraciers to improve plausibility. '

Instructions

[Refer to Section D of ES-401 and Appendix 8 for additional information regarding each of the following concepts.)

25. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
26. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
27. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated trueffalse statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (Le., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are deSignated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (8)ank, (M}odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
7. Based on the reviewer's jUdgment, is the question as written (U}nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any *U* ratings (e.g., how the Appendix 8 psychometric attributes are not being met).

ES*401 Written Examination Review Worksheet Form ES-401*9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back Q= SRO B/MiN UlE/S Explanation Focus Dist. Link units ward KIA Only 73 H 2 N S 74 F 3 N S 75 H 3 N S 76 H 2 X M U 1) C&D do not appear to be plausible at 30% with "A" scoop tube de-energized.

2) Redact P/F map regions.

Selected and revised bank Q testing same KIA. PIF map regions will be redacted.

77 F 2 M E 1) Memory not H;

2) *C" obvious answer;
3) B&D not very plausible;
4) LOD borderline
5) If you decide to fix and retain Q answer C(2) is confusing needs clarification at to what systems.

Revised Q and distractors 78 F 2 N E "A" does not appear plausible with a slight smell of smoke?

Revised "A" to improve plausibility.

79 H 3 N E "D" not plausible to inject poison with power subcritical and only 2 rods stuck at 48.

Revised stem slightly and "0" to make more plausible.

I 1. I 2.

3. Psychometric Flaws 4. Job Content Flaws I 5. Other I 6. I 7. I 8.

Q# LOK (F/H)

LOO (1-5) I C~ed'l Partial I J?b-I Minutia I OIS1. Lmk

  1. ! I I I BaCk-I Q= I SRO BtMlN utEtS I units ward KIA Only Explanation I M I S Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

28. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
29. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
30. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent. more information is needed, or too much needless inform::ltion\

The stem or distractors contain cues (Le., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g .. ifthe applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e.. it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
6. Enter question source: (B)ank. (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
7. Based on the reviewer's jUdgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
8. At a minimum, explain any "un ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 81 H 1 X N U 1) KIA match not testing ATWS and knowledge of LCO instrument failure not actual ATWS?

2) 2) Looks like a borderline DLU with TS provided.

Randomly selected new KIA and wrote new Q.

82 H 3 N S 83 H 2 B S 84 H 3 N S 85 F 3 N S 86 H 3 M S 87 H 1-2 X X M U 1) Distractors do not appear to be plausible. You have an ATWS in progress with SBLC injecting it makes no sense to either secure SBLC and or start RCIC and inject with SBLC which is an alternate method of SBLC injection.

2) I wouldn't consider this a good KIA Match for abnormal system flow SBLC is injecting and power is decreasing the fact that one pump is unavailable does not constitute a flow problem?

Revised Q and distractors

,,,88 H 2 N S

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Credo Partial Job- Minutia #/ Back- Q= SRO B/M/N utEtS Explanation Focus Dist. Link units ward KIA Only 89 H 3 B S Note: in this case not selecting correct procedure but the Q does require diagnosis and direction of correct procedural action so it fits 55.43 b(5).

90 H 1-2? N E 1) With the TS provided this is a borderline direct look-up Q?

The only understanding tested is the understanding that loss of T -10 is a loss of one offsite supply and doing the math to arrive at the correct answer.

2) For this Q and the next Q it might be better not to include the T5 in the stem and just provide several additional specs and let the applicants figure out which T5s apply as part of the task.

Revised Q and distracters.

91 H 1-2? M E 1) With the TS provided this is a borderline direct look-up Q?

2) For ihis Q and ihe next Q it might be better not to include the TS in the stem and just provide several additional specs and let the applicants figure out which TSs apply as part of the task.
3) Prefer more challenging T5 calls for an SRO involving integrated plant multiple specs Q and distraciers revised to make increase LOD 92 H 1-2? N E 1) Why is it necessary to state in the stem that "These switches provide High Steam Dome Pressure Scram signal."?
2) With the TS provided this is a borderline direct look-up Q?
3) For this Q it might be better not to include the TS in the stem and just provide several additional specs and let the applicants figure out which TSs apply as part of the task.
4) Prefer more challenging TS calls for an SRO involving integrated plant multiple specs Q and distracters revised to make increase LOD 93 H 3 B 5
1. 2. 3. Psychometric Flaws 4. Job Content Flaws I 5. Other I 6. I 7. 8.

Q# LOK LOO (F/H) (1-5) Explanation 94 F B S 1 95 H 2 X B E 1) "A" is the obvious answer in that it is the lease intrusive extemal to the system.

2) In contrast the distracters are fairly obviously need Temp Engineering Change Revised Q and distracters to increase LOD.

96 I F 1-2 x N U 1) "0" is the obvious answer for failure of RCS boundary Containment Rad monitors and for failure of containment Rx Bldg Rad monitors are needed.

2) Oistractors are not plausible.

Replaced with bank Q testing same KIA b"

Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

34. Enter the level of knowiedge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
35. Enter the level of difficulty (LOO) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
36. Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., undear intent, more information is needed, or too much needless information).

The stem or distradors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (I.e., the question has a valid KJA but, as written. is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KJA and those that are deSignated SRO-only (KJA and license level mismatches are unacceptable).
6. Enter question source: (B)ank, (M)odified. or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section 0.2.f.
7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement). in need of (E}ditorial enhancement. or (S)atisfactory?
8. At a minimum. explain any *U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (FtH) (1-5) Stem Cues TtF Credo Partial Job- Minutia #1 Back- Q= SRO B/M/N utE/S Explanation Focus Dist. Link units ward KIA Only 97 H 2 X B U 1) For SRO Qs in particular, I'd prefer not mentioning the procedure and Attachment K in the stem just include it in a list of references provided for the exam. In that way they then need to be smart enough to use the document in answering the question.

2) "A&B" do not appear to be plausible it should be obvious that you got a SCRAM or should have from the 8" level and *C&D" highlight that fact.
3) You should probably consider revising the Q to a more involved scenario.
4) You don't have Memory or Comprehension/Analysis marked. Looks like higher order.

Revised stem and distracters removed reference in stem.

98 F 2 X B U KIA match - knowledge of post maintenance operability requirements. I don't believe that knowing this definition demonstrates knowledge of post maintenance operability requirements. What does it really prove?

Randomly selected new KIA and wrote a new Q 99 H 1 X M U Replace this Q way too easy equivalent to DLU. SRO TS questions should be complex involving multiple plant challenges - integrated plant.

Replaced with a bank Q testing the same KIA.

100 F 2 N E For SRO Qs in particular, I'd prefer not mentioning the procedure in the stem just include it in a list of references provided for the exam.

Procedure removed from stem.

Retrieved from "https://kanterella.com/w/index.php?title=ML110200472&oldid=2552414"