ML16321A479

2016-10 Draft Written Exam Comments

ML16321A479
Person / Time
Site:	Palo Verde
Issue date:	11/03/2016
From:	Vincent Gaddy Operations Branch IV
To:	Arizona Public Service Co
GADDY V
References
50-528/16-10, 50-529/16-10, 50-530/16-10 50-528/OL-16, 50-529/OL-16, 50-530/OL-16
Download: ML16321A479 (20)

Text

ES-401 PV-2016-10 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 M S

-Simplify stem: and the reason if not.

Changed stem as recommended.

-Distractor B - lower the RVUH level to make the distractor more plausible. Using the 1-2 detectors uncovered level (41%) would provide the applicant a more plausible distractor to consider.

E Changed from 67% in the head to 2 HJTCs uncovered. This is how QSPDS 2 H 3 X N actually displays the status of fluid in the head/vessel.

S

-Stem: Do control room QSPDS indications provide just the number of detectors uncovered (and the crew has to convert it to %), or does it actually provide percent?

The question stem should match the indications given by QSPDS - i.e. if the percent level is not shown on the display, then the question stem should provide only the number of detectors/zones uncovered. Is the conversion of detectors uncovered to percent level required applicant knowledge?

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). _Rev 2 Page 1 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Changed from 67% in the head to 2 HJTCs uncovered. This is how QSPDS actually displays the status of fluid in the head/vessel.

-This may be an enhancement to consider for Standard Appendix 2: provide the number of detectors uncovered in the HPSI Throttle Criteria instead of requiring user to convert detectors uncovered to percent (RVLMS indicates RVUH level is greater than or equal to 16% [1-3 detectors uncovered]).

Added suggestion to the list of CRs to be generated following the exam.

Added the number of HJTCs uncovered and the % inventory in the vessel head since this is what is indicated on QSPDS in the control room.

Stem: Is 50% WR realistic for a SG that has not been fed during SPTAs after a trip from 100% power? Is level increasing because AFW started, or would it make more sense to have level going down to improve the distractors?

3 H 3 X N S Given these conditions, there is no reason that the crew would not be feeding the SGs, therefore the conditions listed indicate that feed is in progress. The reason for the different levels is that an economizer closed in one feed line and lends credibility to the possibility of an ESD.

4 F 2 N S Stem can be improved to eliminate run-on sentence. Suggest the following:

Unit 3 was operating at 100% when RCP 1A experienced a sheared shaft. Based on these conditions, the first RPS trip will be ___.

5 H 3 X N S The crew trips RCP 1A and RCS pressure control is challenged during SPTAs (adequate NPSH remains). The crew should also trip ___.

Broke up into two parts instead of one long sentence per recommendation.

Change low RCP flow to low RCS flow / SG DP low - Changed to Low RC Flow SG 1 to match the B05 RPS alarm indication.

Agreed on high cog Question is a fundamental/memory question, not higher cognitive.

We believe question is high cog since the examinee would need to know what 6 H 3 N S the controller setpoints would be at MOC in order to determine whether or not a high failure of the controller output would change the flow by > 10 gpm and actuate the interlock. We can discuss futher. _Rev 1 Page 2 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Agreed on High Cog

-Asking how the crew should restore SDC leaves some room for interpretation.

Restoring the B train would not necessarily be a wrong action on its own. Rephrase the stem to state, in accordance with 40AO-9ZZ17, Inadvertent PPS-ESFAS Actuations, what actions should be taken to restore SDC? Then remove the procedure references from the answer choices.

Made suggested changes.

7 H 2 X N S -Explanation for C and D states that A LPSI pump will cavitate since sump suction valve opens; but, pump trips immediately on RAS correct? So it would never get a chance to cavitate because it trips before sump suction opens. Correct explanation.

The explanation says that it is plausible that the pump would cavitate since the pump suction is realigned to the sump, which has no water in it. The pump does not actually cavitate though because the LPSI Pump trips on the RAS signal.

Confirm that the RCP CLR OUTLET FLOW LO alarm actuates and resets at the same setpoint, 450gpm.

There are two flow transmitters which will alarm for low NC flow to each of the 4 RCPs. One has an alarm value of 450 gpm with a reset value of 459 gpm and the other has an alarm value of 475 gpm with a reset value of 480.6 gpm. The acceptable range for each instrument is ~7.5 gpm. Since the procedure directs 8 F 3 N S raising cooling flow until all of the low flow alarms are clear, the best number to use would be 491 gpm. Therefore, changed A1 and B1 from 450 gpm to 480 gpm and added the supporting documentation.

Look for engineering data for the 1600 gpm on the normal chiller to get a number that is not ending in 00. Changed 1600 gpm out for 163 gpm which is the minimum required flow for the RCP HP Coolers, Thrust Bearing Lube Oil Coolers and Seal Coolers for each RCP.

Stem states: RCN-PT-100X, Pressurizer Control Channel Y, failed to 100%.

Supposed to be channel X correct?

Yes, changed to X.

E The references describe an auto permissive signal generated for PT-100Y failed 9 H 3 X X M high, and an auto modulation signal generated for PT-100X high, but dont see an S

auto modulate permissive discussed (distractors A and C). Should A and C be Auto Permissive Light will illuminate? clarify.

Both. An auto permissive signal will be generated and this is indicated by the Auto Permissive Light illuminating onB06.

10 F 2 N S _Rev 1 Page 3 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 11 F 4 N S NRC 2013

-Rearrange and rephrase stem for clarity:

• Unit 1 has tripped from

• AFA-P01 and AFB-P01 are unavailable for use.

• The CO manually starts AFN-P01 to feed both SGs per CRS direction.

• AFN-P01 suction valves CTN-HV-1 and -4 are both open.

• An inadvertent Train A SIAS subsequently occurs.

E 12 H 3 X X B Ordered bullets as recommended. Deleted the bullet regarding the status of S CTN-HV-1 and 4 since they must be open for AFN to be running.

-Rephrase Dist D: is NOT running. The pump can be restarted by taking the handswitch directly to START.

Changed as recommended

- Distractors A&B - if the pump started, would the SGs be fed or not? What is the technical basis that makes the other plausible?

Added information to the plausibility statements to clarify.

-Distractors A&C: Does not seem plausible to believe that it could be necessary to bring offsite power onto a vital bus from anywhere other than the control room. What examples of these evolutions are there, as mentioned in Dist A and C explanations?

8 of the 24 validators selected either A or C (answers which say the parallel is performed at the SBOG). Seems to be a strong case for plausibility of this action. I agree with you point that it seems odd to do this, but all controls for U

the SBOG are outside the control room so it is plausible that paralleling the E SBOG with offsite power locally would occur there as well. I will work on either 13 F 2 X X N S writing a replacement question or finding one in the bank if needed.

Consider changing to ask which bus is preferred to align the SBOG to and figure out a second part.

Replaced with new question asking about which class bus the SBOG is aligned to in a blackout and the coping time as discussed in the region.

Split stem into 2 sentences. Add hours after part 2.

14 F 2 N S Explanations dont match choices.

15 H 3 N S Question was changed and this was overlooked. Updated so the explanations match the choices. _Rev 1 Page 4 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 16 H 3 N S NRC 2015 Is not part 2 of A and C also correct for the same reason SDCHX is correct, i.e., the isolation ensures adequate cooling flow to the SDCHX during accident conditions, which cools containment spray during RAS and therefore ensures containment integrity during accident?

Interesting point. In reality, RAS would result in the SDCHX acting as a cooling medium for CS, however prior to getting to a RAS, a large part of the heat E inside containment would be removed using water from the RWT (which is 17 H 2 X B already cold) and thus would not have much impact. If you want, we can S

change part 2 of A and C to say, To ensure Containment Integrity is maintained in the event of an ESD inside containment. The concept is similar, however you would never receive a RAS on an ESD so the issue of the SDCHX cooling CS is avoided altogether.

Changed part 1 of A and C from To ensure containment integrity to To ensure containment isolation since the auto closure of the cross tie valves could be argued to maintain containment integrity, however it does not ensure containment isolation.

Question is not targeting the K/A precisely - its not testing on actual failure modes.

Pick one of the components and rephrase to On a loss of instrument air, Component X fails [closed/open] at [setpoints]

Replaced question. New question gives a loss of IA and asks for the failure position of ADVs as well as where ADVs can be operated from following a loss of IA.

E New question was not completely technically correct. Changed new question 18 F 3 X N to address technical inaccuracies.

S Split stem into 2 statements.

Due to the loss of Instrument Air, ADVs will _____(1)_____.

When backup nitrogen is aligned, the ADVs can be operated for a MINIMUM of

_____(2)_____ .

Distractor B: It is not plausible to choose insertion of a CEA in order to comply with a U limit on CEA insertion.

19 H 1 X N E S LOD-1: Elementary knowledge that a misaligned rod is addressed by the CEA alignment TS not by rod insertion limits. The decision to move the one rod in, rather than 7 rods out is also elementary considering basic understanding of flux distribution _Rev 1 Page 5 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only and reactivity control. Consider working in the various position indications and which ones apply for TS compliance or CEA withdrawal prohibits.

We understand the comments, however dont necessarily agree. Knowledge of what CEA Insertion Limits is not completely intuitive as implied in the comments. An operator would need to know if the limit is based on a fixed rod height in the core or based on how far a CEA or group of CEAs are inserted relative to other CEAs. If thought that one CEA being higher than the rest was an issue due to some CEAs being inserted further than the one which is 7.5 higher, 3.1.7 is a viable option. Based on validation data, 3 of 20 licensed operators missed the question, putting this particular question at an 85% pass rate, which indicates an appropriate LOD. All 3 who missed the question selected an answer with LCO 3.1.7.

Evaluate another option instead of TS - something like the recommended rod control interlocks.

Question was modified to enhance plausibility of distractors. Did not use rod control interlocks as we could not create a plausible 3rd distractor when using interlocks or rod position indications.

Modify B.2 and D.2 to no more than two hours after CEA 18.

20 H 2 M S 21 H 2 N S Distractor A: Can also be considered a potential correct answer depending on the rate of degradation.

Changed bullet to say slowly degrading to make A clearly incorrect.

Distractor D: ability or Inability of rx power cutback system?

Correctly says ability. If RPCB system was in service, the plant could handle a main turbine trip, however it is not in service at 50% power. Also, the RCS U and the two atmospheric SBCS valves can accommodate ~ 30% load rejection 22 H 2 X X N (7-10% per SBCS valve and ~ 10% in the RCS), therefore only tripping the main S

turbine is plausible.

Cueing: The correct answer is the only choice that references SBCS which is referenced in the annunciator in the stem. Consider just stating the current vacuum level, not which annunciator has alarmed. Some conceptual overlap with question #1 exists.

Changed the stem to say 7.0 inches HgA and slowly degrading to remove cueing in the stem. _Rev 1 Page 6 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only We believe this is a high cog question since the student has to analyze the 23 H 2 N S location of the leak, and determine where that radiation will be monitored as well as how it will be filtered.

24 H 3 N S Distractors A and B not plausible for an applicant to think that Nuclear Cooling Water rad monitor is the primary indication of high RCS activity.

U Changed from the NC Radiation Monitor to the Containment High Range RMs 25 F 2 X M S since they are used to determine the status of the fuel cladding barrier.

You had this listed as high cog but we feel that it is probably two low cog parts which keep it as low cog.

26 H 3 N S 27 F 3 N S 28 H 3 B S PVNGS Bank 29 F 2 B S PVNGS Bank 30 H 3 N S 31 H 2 M S Enhance discriminatory value of the question by providing an RCS temp, pressure, and LTOP status in stem, requiring applicant to interpret what MODE the unit is in, rather than giving them MODE 4 (unless LTOP is inherent to having SDC in service, in which case dont provide that info). Simply stating that the plant is in Mode 4 appears to cue the correct answer is lower mode functional recovery.

E Changed as recommended. Did not include LTOP status in stem as this is 32 H 2 X X N implied due to being on SDC.

S Changed from 250°F and 200 psig to 275°F and 225 psig since students could easily transpose the 200 and 250 which are the entry conditions for using CS for SDC.

Are there any examples of actual ECCS components where one train is AC and the E redundant train component is DC?

33 F 2 X N S Yes, if I understand what youre asking, the long term HPSI recirc valves in each train have two valves in series, one is AC and one is DC. _Rev 1 Page 7 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 8 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only More discriminatory to test two components on the same train that use diverse power supplies, e.g., LPSI B Flow Control Valve to RC Loop 2B is Class 480VAC, but hot leg injection valve on that train is 125VDC.

Changed per recommendationhowever these valves are not automatically repositioned on an ESFAS signal so the KA match may be looser than before.

These valves are operated manually when transitioning to long term cooling (hot leg injection). Update reference on worksheet.

Also test on the actual bus number itself.

I think if we do that we are cueing since the buses are PHA (AC train A) and PKC (DC train C) and the valve numbers indicate that. The valves are SIA-HV-604 and SIC-HV-321. I think it would be more discriminating to maintain the answers as is.

Take a look at the RAS valves which reposition and ask an AC and DC valve. -

Changed question to ask about a HPSI injection valve power supply (actuated on SIAS - AC powered) and a combined recirc valve to the RWT (actuated on a RAS - DC powered) as discussed in the region.

34 F 3 N S Distractors C&D: It is not plausible to consider that the fuel pool heat exchangers would be affected by an isolation of NCW to containment. Keep the answer choices and distractors as-is, but change components in stem to the following:

U 35 H 2 X N 1. CEDM S 2. Nuclear sample coolers

3. LDHX Changed per recommendation.

Choices A and B, provide voltage rating for consistency with C and D.

36 F 4 N S Changed per recommendation.

Distractor A: it is not plausible that RPS would allow two channels to be bypassed at once. The existence of some kind of interlock is elementary knowledge.

Distractor D: It is implausible that by attempting to bypass RPS A, it would prevent RPS A bypass and then take the one step further to un-bypass RPS B.

U Modified question to enhance the plausibility of A and D by saying one or the 37 H 2 X N S other will go into trip.

We believe this is a high cog question due to the fact that you have know the interlock exists and apply the interlock based on which transmitter was already in bypass. If A was in bypass and B was attempted to be placed in bypass, the answer would be different (A would remain, B would not go into bypass). _Rev 1 Page 8 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 9 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 38 F 2 N S 39 F 3 N S 40 F 3 N S Modify to test on:

1) CS pump can be aligned to cold leg only / hot and cold legs, and

2) whether CS pump is / is not procedurally allowed to inject into both cold leg and E spray header simultaneously 41 F 2 X N S Changed per recommendation.

Add reference to the relevant procedure in the stem, to preclude there being some obscure SAMG which would allow injection into cold leg and spray header.

K/A - Question does not test the definition (what is the mechanism for water hammer) or operational implication (for example, how do you operate the MSIV bypass and/or steam drains/traps in order to prevent water/steam hammer? Leave main steam drain valves open during warmup or cycle open and closed?) - the operational implication is given in the stem.

Also Stem focus - too much verbiage, would be better simplified to What is the primary reason for doing this?

1 U This KA was on the 2015 NRC RO exam and the question was asked almost 42 F X N 2 S exactly as you suggested. Replaced with question from 2015 exam (Q43),

which puts us at the limit of 4 from the past two exams.

Reworded second part to say draining Main Steam lines by manually bypassing steam traps to the condenser for operational relevancy.

NRC 2015 memory/fundamental, not higher cognitive. Recall of basic AFW system operation.

AFN-P01 has no auto-start feature - this part is borderline LOD-1. Suggest modify part 1 to test on the AFN-P01 interlock which prevents manual start with SIAS-A 2 E signal present. Modify stem to include feedline break inside containment which 43 H X N actuates SIAS-B only, and ask if manually taking AFN-P01 handswitch to START will 3 S start pump or not or some version of that.

Question replaced with new question (changes from modified to new). Did not use the recommended question as it would be very close to question 12. Also felt that it was implausible to have a valid SIAS-A with no SIAS-B. Eliminated _Rev 1 Page 9 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 10 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only portion of question asking which pumps start on an AFAS to maintain LOD at the appropriate level. Question is now high cog.

Simplify stem to:

A Reactor Power Cutback occurs in response to a loss of a Main Feedwater Pump.

44 F 2 X N S The cutback signal is generated by MFW pump ____ and the turbine setback will lower turbine load ___

Changed per recommendation.

Distractors: 12% is pretty high is there a lower level with plausibility - 7%?

In the MODE 2 to MODE 1 procedure, the only other power level which could E lend plausibility is 5%. I think 12% works better since this is the point at which 45 F 2 X N the main generator is synched to the grid so you would obviously need S additional feedwater at that time. If you really want to use 5%, we can make the change.

Question changed to use 5% as the distractor instead of 12%.

Cueing: There is only one procedure that you would implement FIRST over any of the others. The other three would be implemented concurrently if no trip occurred. Also, E only 1 procedure is an EOP - makes it stick out. Just ask the power supply for a 46 F 2 X N major AC load.

S Changed question to ask which RCPs are powered from NAN-S01. Question changed from high to low cog.

47 H 3 N S What is the bank source - previous NRC exam? Clarify for all bank questions.

PV Bank - 2010 RO Exam 48 H 3 B S Added information about bank information including all past NRC exams for all bank questions.

49 F 3 N S 50 H 3 B S PV Bank - 2010 RO Exam 51 H 3 B S PV Bank - 2012 RO Exam 52 H 3 N S U Distractors A&C LOD-1: Not plausible to believe that safety-related EW is cooled by 53 H 2 X M non-safety PW. Beginner-level systems knowledge, not sufficiently discriminatory for S license exam. _Rev 1 Page 10 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 11 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only The knowledge check in choices B & D is sound however.

Changed into two questions. Maintained the knowledge check from original B&D and added a second part about the backup cooling source for SDCHXs in the event of a loss of both EW pumps. Question is now a modified question.

Previous bank question was not used on previous PV NRC Exam.

54 H 2 B S PV Bank - 2009 RO Exam Distractor C: Not plausible to believe that you would just override and open UV-516 after a letdown isolation. Can you provide example of when this practice would be allowed? A controlled restoration of letdown will have been practiced routinely.

E Several valves are overridden and opened following an inadvertent actuation, 55 H 2 X N S however your point is valid. Changed the two answers related to restoring letdown to ask whether or not the CIAS would have to be reset prior to restoring letdown - it does not need to be reset prior to restoring letdown due to the override features of the letdown valves.

Dont use the term design basis since the answer is coming from Tech Spec Bases, just use the term basis. Id typically expect a design basis to be found in the FSAR.

That said, tech spec basis information is typically SRO-only knowledge. Can you E confirm this is in fact expected to be RO knowledge at PVNGS?

56 F 3 X X N S Changed per recommendation. TS Bases are not considered to be RO knowledge, however the reason for certain control bands is. The band of 33-53% in the pressurizer while at power is not a TS basis, so it is fair game for the ROs.

57 H 3 N S PVNGS Bank. Really its a combination of two bank questions as the distractors for the purpose of the charcoal filters were poor and I couldnt 58 F 2 M S come up with decent replacements. Neither bank question was previously used on an NRC Exam.

Changed to modified - added the two bank question to the worksheet Make alignment match other questions.

Not sure what youre asking for here. Im sure its an easy fix, just dont know 59 F 4 B S what youre getting at.

PV Bank - 2010 RO Exam Edit stem question to Which one of the following would cause the lower level 60 H 3 N S Changed per recommendation. _Rev 1 Page 11 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 12 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Clarify in explanation for answer B how the tripped refueling purge supply fan affects containment parameters, which in turn affects SFP level, to demonstrate the link to the K/A.

Added explanation for correct answer.

Distractors A&B part 1: Not plausible to believe that a SG pressure instrument has failed high when it is reading normal SG pressure for the given conditions - just testing on whether applicant knows normal SG pressure. Possibly make pressure E read higher than normal.

61 H 3 X N S Changed the value of SGN-PI-1024 from 1025 psia to 1050 psia to be 25 psia higher than normal.

Also, changed A and C part 2 to say, Insert a manual permissive as needed since multiple valves cannot procedurally have manual permissives in AUTO.

Which one of the following conditions would cause this response?

62 H 2 B S Changed per recommendation.

PV Bank - 2010 RO Exam 63 F 3 N S 64 F 3 B S NRC 2015 Are you certain that there is no way that a fire truck could take a suction on the water rec reservoirs? Change second half of question to: If needed, a Fire Truck can E utilize connections provided at the _____ to pump to the fire water system.

65 F 2 X N [Assuming the truck is actually hooking up to a connection at the circ water bay, and S

there is no connection at the reservoir]

Changed per recommendation.

66 F 3 N S 67 F 2 N S Cueing: The phrase with a corresponding rise in cation conductivity is included in the stem and two of the answer choices, including the correct answer.

E Simplify stem to Which of the following requires the CRS to direct a manual reactor 68 F 3 X N S trip?

Changed per recommendation and added, Per 40AO-9ZZ10, Condenser Tube Rupture, E Make phrasing between B and C consistent.

69 F 2 X B S Changed per recommendation. _Rev 1 Page 12 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 13 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Revise D: Consider Initiate a controlled plant shutdown, or Withdraw Regulating Group CEAs above PDIL within [plausible time period].

Changed distractor D per recommendation.

PV Bank - 2010 RO Exam Distractor D is weak, as you cant ensure air wont be lost to the valve. Rephrase to:

permitted only if the air line to the valve actuator is tagged to ensure air cannot be E isolated to the valve.

70 F 3 X B S Changed per recommendation.

PVNGS Bank.

71 F 2 N S Missing explanation section.

72 H 2 N S Added answer explanations.

73 F 2 N S Labels on instruments in red boxes difficult to read.

Weve had no feedback about that during validations but I can try to change to 74 F 2 N S white lettering if you want.

Take a picture in the simulator and use that.

Used an actual photograph of the transmitters on the exam, looks a little better.

75 F 3 N S Distractor A is a subset of C fundamentally the same.

E 76 H 3 X N Changed C to to minimize contamination levels in the Main Steam and S

Feedwater systems Distractors A&B part 1: Not credible to cross-connect EW to an NC system that has an unisolable leak in containment. There is no reason to consider that a leak in containment NC would be isolated by connecting to EW. [peer review identified]

E 77 H 2 X B Couldnt come up with a different distractor for A & B part 1. Replaced with Q S 76 from the 2015 SRO exam which had the exam same K/A. This makes one question on the SRO exam from the past two NRC exams at PVNGS.

NRC 2015 78 F 2 N S RO level knowledge. Question asks for an aspect of the sequence of steps in the 79 H 3 X N U EOP and use of an RNO step. There is no procedure selection/transition. _Rev 1 Page 13 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 14 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S Replaced with new question. New question places the CRS in a situation in which procedural direction is conflicting and the CRS has to use guidance in the technical background document to determine how to proceed.

Remove since the FR will not restore from both distractors. Statement also lacks any procedure transition/selection. Replace with remain in 40EP-9EO08 and perform E

80 H 3 X N Appendix 53, Align De-energized Buses, in anticipation of the restoration offsite S power.

Changed per recommendation.

-Choice B actually appears to be a true statement - TSB states If the required capability in Condition G is not met, the effects of an AOO or DBA could cause further depression of the voltage at the ESF bus and actuation of the degraded voltage relays. Blocking Fast Bus Transfer does prevent spurious actuation of loss of voltage relays. 2 correct answers.

Changed question to only have one correct answer.

-TSB states that both Fast Bus Transfer must be blocked AND associated tie breaker

[NAN-S03B or NAN-S04B] to house load buses [NAN-S01 or NAN-S02] are OPEN.

Confirm that it is inherent to the conditions given that NAN-S03B and NAN-S04B are already open.

Yes, in order to block FBT, NAN-S03B and NAN-S04B must be open.

U 81 H 3 X N -Explanation states that Condition 2 is the preferred option (it is the option given in S

the correct answer). However TSB states that Condition 1 is the preferred option, and with only 1 unit operating, Condition 1 is met in the following manner: switchyard voltage is increased by any number of methods implemented by the ECC while maintaining the generator gross MVAR output of the Palo Verde unit to <= 0. Either leave question as-is but clarify in the explanation that the answer is not the preferred option but is the only allowed option given, or modify question to test on both correct AND preferred option - may even be able to have 4 distinct (not 2 by 2) choices in this case.

Changed question to align with these recommendations.

Split stem into two sentences for clarity.

82 F 2 N S Handouts provided.

E 83 H 2 X N S Include fuel building radiation monitor readings as attachments to question 83, not in the reference packet. _Rev 1 Page 14 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 15 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Will do when the exam is finished being modified.

The RU-146 Ch 1 plot is very close to the alert value of 1.13E-01 make it a little higher so its not a question.

I dont see the need to make RU-146 higher since the RA2.2 is met as of ~ 1050 and would have to be classified NLT 1105. RA1.1 may be met but the threshold isnt met until ~ 1105 and then has to be in alarm for 15 minutes before RA1.1 is met so the magnitude of RU-145 seems less important. Maybe I dont really understand why you want the level raised but we can discuss and I can change if needed.

Doesnt seem necessary to provide Release Evaluation Flowchart -- SRO should know that a high alarm on any effluent monitor = a release that exceeds federal limits.

We disagree. The applicant would have to know the criteria in the top box from memory and the process of the flowchart. It is not reasonable to expect someone to memorize the release flowchart to answer this question. Simply having a high alarm on an effluent monitor does not necessarily mean the release exceeds limits. We can discuss further.

Consider a different second part to the question - is a PAR evaluation required.

Updated question to asking about the requirement for a PAR instead of the status of the release.

Handouts provided.

84 H 3 N S 85 H 3 N S Question worksheet does not designate whether handouts are to be provided or not 86 H 2 N S No reference provided. Updated worksheet.

87 H 3 N S Added LCO 3.7.5 (first two pages only) to the SRO exam handout.

88 H 3 N S SRO: Question tests above the line TS knowledge for TS 3.4.16, therefore is RO level.

U 89 F 2 X N Created new question. New question provides a loss of a fuel building S

ventilation RM and asks what is needed to comply with the TLCO for FBEVAS and the ODCM.

E Handouts provided.

90 H 2 X N S Q80 already tests the CTPC safety function (part 1) - significant overlap. _Rev 1 Page 15 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 16 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Changed question to replace the part asking about the CTPC SF and asked about the time to make notifications.

K/A is focused on EAL thresholds - take part 2 a step farther and apply to an EAL.

Changed question to utilize the criteria for one train of CS flow to apply it to an EAL as recommended.

Will the fuel cladding section of the EAL chart be blacked out on EAL chart handout?

E If not, direct lookup.

91 F 2 N S Yes, the fuel cladding section will be blacked out. See the provided references, this is the electronic file which will be printed and given to the examinees.

Distractors C&D Part 1: Not credible that the fuel handling machine would analyze the direction that the machine was going in and stop x-y vector motion only in that direction upon mast contact - moving in any direction after contact risks further damage. Non-existent function. Choose another interlocks function and use it as E

92 F 2 X N the distractor - something that actually exists and provides a legitimate safety S purpose.

Modified the question to ask whether or not the manual handwheel can be used to move the bridge and trolley prior to overriding the mast bumper interlock, enhancing the plausibility of the distractors.

93 F 2 M S

-The last reference highlighted (4.17 Standing Orders) states that a Standing Order is used to disseminate information related to operator actions required by Operability Determination contingency plans. In this question however an Operability Determination is the wrong answer since the affected component is found in the TRM, not TS. Is Functional Assessment interchangeable with Operability Determination in this context (40DP-9OP02 4.17)?

Yes, in this context, OD and FA are interchangeable. For added clarity, modified question to indicate the initial work window for the pump is during the E outage scheduled for 45 days from now.

94 F 3 X N -Its also not clear to me why communicating compensatory measures via Night S

Order initially is explicitly wrong. Question does not say anything about how long the condition is expected to last / next opportunity to repair (30 day cancellation of night orders is used as justification why Night Order is wrong). Conceivably a Night Order would be issued first (lower level of approval needed?) followed up by a more detailed Standing Order.

Modified stem slightly to ensure clarity of which answer is clearly right and wrong. The question clearly says, per 40DP-9OP26 and this procedure directs communication of compensatory actions via a standing order. So while Conduct of Shift Operations does not explicitly prohibit initially issuing a night _Rev 1 Page 16 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 17 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only order, it is not the correct answer on shift and is not the correct answer per the stem of the question.

Question worksheet does not designate whether handouts are to be provided or not.

95 F 3 M S No reference provided. Updated worksheet.

Distractor A: Modify to OPERABLE, but the mode change is allowed only after performing a risk assessment per SR 3.0.4 Changed per recommendation.

-A risk assessment is NOT required in this case, is that correct? Explain better why.

E 96 H 3 X B The note above the double line in LCO 3.7.5 states that LCO 3.0.4 B is not S applicable.

Distractors C and D: the mode change can be made Changed from completed to made PVNGS Bank.

97 H 2 B S LOD-1/Distractors B,C,&D: Engineers and coordinators do not approve releases, especially during the EOPs - not plausible that an applicant would consider these positions. Replace with Plant Manager, TSC Director, Emergency Director (non Shift Manager).

U Changed per recommendation, however substituted OSC Manager for Plant 98 F 1 X B Manager since Plant Manager is not an ERO position, and TSC Emergency S

Coordinator for TSC Director to match the PVNGS ERO titles.

PV Bank - 2008 & 2012 SRO Exam Changed TSC Emergency Coordinator to TSC Operations Manager as discussed.

99 F 2 N S Part 1 has overlap with Q32, testing on knowledge of Mode 4 with SDC in service impact on EOP response. Deconflict.

E Modified the question to ask whether or not you would transition from the FR 100 H 2 N S to the LMFRP when a subsequent event occurs in MODE 4 with LTOP in service. This is a different procedure use question than Q32. Kept the EAL classification portion of the question. _Rev 1 Page 17 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 Written Examination Review Worksheet Form ES-401-9 B = 15 13 F = 40 37 E = 18 Additional Notes:

Updated totals in red RO TOTALS: M= 57 H = 35 38 U= 8 N = 55 55 S = 49 B= 34 F= 10 15 E= 10 Additional Notes:

Updated totals in red SRO TOTALS: M= 22 H= 15 15 U= 4 N= 20 19 S= 11

GENERAL COMMENT

S

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N

2. Chief Examiner original comments are in black. Licensee provide follow-up comments in a different color.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). _Rev 2 Page 18 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 19 Form ES-401-9

3. Average difficulty is _2.52 on the RO exam and 2.4 on the SRO exam.

Overall, lower than average level of difficulty exam.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 0 43.1 4 41.2 2 43.2 9 41.3 4 43.3 1 41.4 13 43.4 2 41.5 2 43.5 6 41.6 3 43.6 0 41.7 21 43.7 3 41.8 4 41.9 2 41.10 14 41.11 4 41.12 1 41.13 3 41.14 2

5. The answer distribution is: RO / SRO A = 18 (24%) / 10 (40%) 17 (23%) / 7 (28%)

B = 17 (23%) / 6 (24%) 20 (27%) / 7 (28%)

C = 20 (27%) / 5 (20%) 22 (29%) / 4 (16%)

D = 20 (27%) / 4 (16%) 16 (21%) / 7 (28%)

6. There are _0___ RO questions with handouts provided and __3___ SRO questions with handouts provided.

GENERIC Comments: _Rev 1 Page 19 of 20 OBDI 202 - IOLE Process

ES-401 PV-2016-10 20 Form ES-401-9

-For some questions it would be helpful to briefly explain why the correct answer is actually correct, similar to the explanations of why the distractors are incorrect. Added an explanation as to what makes the right answer right if not clearly explained in the explanations for the distractors.

-For all bank questions, clarify the source. Previous NRC exam or facility bank? Updated the 401-9 and the worksheets with the requested information.

-40% of SRO questions are answer A. Shuffle some of these to answer D (16%) Multiple questions answers changed due to resolving/replacing questions on the SRO portion. New answer breakdown for both the RO and SRO is listed above in red.

-By my assessment, less than half the RO questions are Higher Cognitive Level (35 = 47%). Edit at least 3 Fundamental questions to require some level of higher cognitive assessment, to achieve > 50%. Questions were modified to now have 38 high cog questions on the exam for a total of 51%. _Rev 1 Page 20 of 20 OBDI 202 - IOLE Process