ML091811184

CW-2009-06-DRAFT Written Comments

ML091811184
Person / Time
Site:	Callaway
Issue date:	06/30/2009
From:	Apger G NRC Region 4
To:
References
Download: ML091811184 (11)

Text

ES-401 CW-2009-06 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only As the question is modified from the Bank question, it does not meet the definition of M E significant modification needed to classify this as a Modified Bank question (see 1 H 3 B S NUREG-1021, Section ES-401, Sub 2.f). Either it needs to be revised to meet this definition, or classified as a Bank question. Done.

First, the plant conditions give indications that may be conflicting:

It says that a SBLOCA has occurred with NO operator action, yet it says that the reactor was tripped and ECCS is operating.

E 2 F 3 X X N Second, the typical two methods of decay heat removal are the S/Gs and High Head S

Injection/SI using feed and bleed if S/Gs are unavailable. It is unclear how the accumulators or the RCPs could plausibly be used for long term cooling.

Editorial made. Question is OK.

E Distractor answer B is related to a plant alarm that doesnt exist. This doesnt appear 3 H 3 X N S to be a plausible distractor. B was changed. Question is OK as written.

4 H 3 X N U Distractors B and D are not based on any sort of level of significance in Pressurizer Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S level or Letdown isolation control. They are not plausible distractors. Question is OK as written.

E Mismatch between title of procedure stated in question and procedure reference 5 H 4 N provided needs to be resolved (Loss of RHR Flow versus Loss of RHR with Level S Greater than 94). Question is OK as written.

U Distractors C and D can both be eliminated if the applicant knows the one piece of 6 F 3 X N information about whether ESW provides auto makeup or not. A different plausible S distractor should be added. Question was revised.

U The question asks what the response of the Pressurizer Pressure Control System is 7 F 3 X N due to the conditions, but the correct response is associated with the Pressurizer S Level Control System. OK as written.

8 H 3 N S E In the justifications for the answers, both events is referred to several times. It is 9 F 3 N S assumed that this means LOCA and SLR, but it should be clarified. OK as written E The attached procedure section showing the correct actions is incomplete. Provide 10 H 3 N S the additional page(s). OK as written U The question is asking what the correct actions are for loss of a vital AC electrical 11 H 3 X N instrument bus. The K/A statement requires demonstrated knowledge of the reasons S why those actions are taken. OK as wiritten With the information given, it would appear that this is testing the applicants ability to operate only. If this is the case, then it needs to be verified that these are physical actions that an RO would take.

E The attached procedure section showing the correct actions is incomplete. Provide 12 F 2 N the additional page(s).

S This question is classified as a Comprehension/Analysis type. If the applicant is given the details on what has happened without having to discern whether there is an issue or not, and then is asked to provide a list of actions, this would be better classified as a Fundamental Knowledge (F) question. Question verified OK.

The question is written about knowledge of abnormal operating procedures for high U turbine vibration. The K/A is tied to emergency actions for a loss of Nuclear Service 13 F 2 X N S Water, which is in no way implied in the question. This is a K/A mismatch. Question and KA replaced.

U The K/A has two parts: ability to see control room indications to determine system 14 F 3 X B S status, and show understanding of how actions/directives affect the system. As Page 2 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only written, this appears to be addressing the understanding of actions, but not the ability to determine system status based on indications. The question initial conditions say that there is a loss of instrument air, instead of providing indications to see if the applicant can determine there is a loss of instrument air in progress.

Editorial: The Importance Ratings need to be shown on the question. Also, this K/A is associated CFR 43.5 as well. Question OK as written Editorial: Fix the K/A number and Importance Ratings on the question: it is associated E with AA2.09, not AA2.08.

15 H 3 N S It is assumed that the designator NE01 is the same as DG A based on the question text. This needs to be clarified. Changes made E

16 H 3 B Editorial: The K/A is related to CFR 55.41.7, versus 55.41.8 and .10. OK as is.

S A question on determining the CSF path color designations while in an emergency U and the transition points is normally a SRO only type question. Also, for the SRO, it 17 F 4 X N S would be expected that a reference would need to be available to determine the CSF color designations to set plant priorities. Question was revised.

Without providing a reference to the applicant with this question, they could easily U Answer B or C and argue they are correct as well because the CSF in red drives you 18 F 4 X N to use the FR procedure as the controlling procedure, and establishing any S Containment Spray flow at this point is greater than having none at all, which is the current condition. Question is OK as written The K/A is associated with knowledge of interrelations between Pressurizer Level U Control malfunctions and controllers/positioners. The question and the correct 19 F 3 X N S answer provided are directed towards Pressurizer Pressure Control malfunctions.

This is a K/A mismatch. Ok as written.

20 F 4 N S The K/A says that the applicant needs to be show the ability to determine and interpret S/G pressure related to the situation. The question is testing whether the 0

applicant has knowledge of the S/G pressure set point associated with 557 F Tc.

U 21 F 3 X N Something about what the indications show outside of the control room and S interpreting whether Tc is being maintained appropriately needs to be addressed.

Editorial: Last bulleted initial condition should say RCS cold leg temperature. OK as written 22 F 4 N S 23 H 4 N S Page 3 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 24 H 4 B S E Editorial: The K/A addresses 10 CFR 55 content for 41.5 and 43.5. Fixed backward 25 F 4 N S logic 26 F N S As the question is modified from the Bank, it does not meet the definition of significant modification needed to classify this as a Modified Bank question (see NUREG-1021, Section ES-401, Sub 2.f). The wording of the question has changed but the conditions of the question have not changed. Either it needs to be revised to meet this definition, or classified as a Bank question.

E Also, the applicant may argue that Answer D is partially incorrect because an 27 F B S engineering analysis may result in return to normal values after adverse containment is declared based on high radiation.

In the Question History section, it says N/A for Last NRC Exam. Does this mean that the review of the past two exams was completed, and the question wasnt found there, or does it mean that a review of past exams is not applicable here? Changed source. Question is OK.

28 F 2 N S 29 F N S 30 F N S The initial conditions provide a RWST Boron concentration that is within specification E the attached reference material (2350-2500 ppm). Having adequate concentration 31 H 3 X N S there does not result in the need for a minimum BAST concentration. Question was revised.

32 H 3 N S 33 F 3 N S 34 F 2 N S The K/A requires the applicant to show ability to predict/monitor changes in U parameters (including CCW flow rate) associated with CCWS control operation. The 35 F 2 X B S question is written to test the knowledge of what inputs cause Radwaste CCW valves to close. Question OK as written.

36 F 2 N S 37 F 4 N S Page 4 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E The explanation provided for the answer doesnt match the answers given backed up 38 F 4 N S by the logic diagram provided. Revise. OK as written 39 H 3 N S 40 H 3 B S The scenario in the question is a complete loss of Containment Air Cooling fans, E failure of Core Spray, with a Large Break LOCA. The justification documentation 41 H 3 N provided assumes two Containment Air Cooling fans fail, Core Spray Starts, and S there is a Steam Line Break. What is provided does not justify the answers given.

Question verified correct.

42 H 3 N S 43 H 2 N S 44 H 3 N S 45 F 2 N S 46 H 3 B S 47 H 3 B S 48 H 2 N S E Editorial: The justifications for the answer reference a backup motor that is not 49 F 2 B mentioned in the question. It is mentioned in the printout of the bank question. This S should be revised for consistency. Justification changed. Ok question.

U This question addresses portion (a) of the K/A statement, but not portion (b).

50 F 3 X B S Editorial: The Importance Rating for the SRO is 3.2. Question OK as written.

This is denoted as a New question, but a related question is attached. Either it needs E

51 H 3 B to be marked as Bank or Modified if one of the distractors is changed.

S Editorial: This addresses 10 CFR 55.41.5 and 43.5. Bank question.

52 H 3 M S The valves in the procedure appear to have a four letter identifier on them (KA FV, U GS HV). If it is not common to have system valves with three letter identifiers such 53 H 3 X N S as in two of the distractors (KA V), this would automatically make these not credible distractors. OK as written.

54 H 4 B S Page 5 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E Editorial: Need to revise the K/A # and Importance Ratings to reflect the K/A selected 55 F 3 B S after the initial rejection. Fixed.

56 H 2 N S E In order for this to be considered a measure of plant specific knowledge, the 57 H 2 N indications given need to be tied specific instrumentation at the plant. Revised stem S to make it plant specific 58 H 3 N S Spelled out GT-TSH-0019.

59 F 3 N S Added bullet to status off-site power.

The K/A tests the ability of the applicant to monitor operation of the Containment U Purge System including the case when an isolation signal is received. The question 60 F 3 X N S is written to test knowledge only of what causes a system stop signal. Question was revised.

61 H 3 N S 62 H 3 N S E

63 H 2 N Editorial: Add the Importance Rating values to the question. Done S

64 H 3 N S 65 H 3 N S 66 H 3 B S 67 F 3 N S The K/A tests the ability to explain AND apply precautions/limitations. The question U

68 H 3 X B only addresses the ability to apply limitations.

S OK as written 69 H 2 B S The K/A addresses the ability to analyze the effect of maintenance activities on safety U system operability (status of LCOs). The question test analysis of what would cause 70 H 4 X B S certain changes in the electrical plant, but does not address operability implications of anything. Question replaced.

71 H 2 N S Page 6 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 72 H 2 N S 73 H 3 N S The question requires the applicant to select the appropriate abnormal procedure to U enter. Assessment of the situation and selection of the correct procedures to enter is 74 F X B S the responsibility of the SRO per 10 CFR 55.43.5. Question replaced. This is a fundamental knowledge question.

75 H 2 B S E Unless this type of accident is taught as being a large vapor space LOCA, it would 76 H 3 N be advised to change the description. It is related to a Pressurizer vapor space S LOCA or stuck open relief valve. OK as written.

77 H 3 N S U

78 H 4 N RO knowledge. Question was replaced.

S The K/A for the question is asking for plant/crew response and what to do about the U situation, not the selection or application of a procedure (10 CFR 55.43.5). An 79 H 2 X N S applicant can answer the question using integrated plant and system knowledge, which is not unique to the SRO. Answers changed.

The K/A and 10 CFR 55.43.5 require the SRO to be able to use the indications and make an assessment of what is going on, and to determine which procedure to enter.

In this case, something about what items supplied by DC battery power is needed.

U 80 H 1 X N The present question tells the applicant that there is a Station Blackout (a loss of S

offsite power and no EDGs supplying load). This doesnt show a plant specific scenario where the SRO would have to determine they are in a SBO, and indicate a procedure entry. Question rewritten.

E The question has to associate E-Plan and PAR knowledge with a Steam Line 81 H 3 X N Rupture/Excessive Heat Transfer/Depressurization of all S/Gs. In the written S question, it is based on a Station Blackout. Question revised.

The question addresses knowledge of which RPS trip is expected the basis of the U applicable reactor trip set point, but it does not text the knowledge of operating 82 H 3 X X N S limitations in the technical specifications (LCOs and Safety Limits) and their bases.

Question was rewritten.

U 1) Procedure OTO-ZZ-00001, Page 1 (Rev. 31), indicates that when the 83 H 3 X M Control Room is evacuated, CCW pump B or D is running. Answers B, C, S

and D indicate that CCW pump D is available. What if CCW pump B was Page 7 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 8 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only started instead of D prior to exiting the Control Room? Couldnt the applicant argue that the correct answer is only partially correct?

2) There is a note that says that the K/A was changed and provides basis in the question document, but it doesnt show up on Form ES-401-4.

Question OK as written.

E There is a note that says that the K/A was changed and provides basis in the 84 H 3 M S question document, but it doesnt show up on Form ES-401-4. Question OK.

The K/A for the question is asking for a determination/interpretation of adherence to U procedures and limitations stated in the plant licensing basis. Assessment of the 85 H 3 X B situation with selection of the appropriate procedure is needed (10 CFR 55.43.5). An S applicant can answer the question using integrated plant and system knowledge, which is not unique to the SRO. Question OK as written.

1) To answer the given question requires only knowledge of immediate actions, which is not unique to the SRO.

2) The present question doesnt require a determination of what procedure would be entered based on the situation, which doesnt address the K/A U statement and 10 CFR 55.43.5.

86 H 2 X X N S 3) Notes say that this was associated with the 2005 NRC exam. Not sure if it is Bank or Modified yet. Could be issue with requirement to not repeat questions from last 2 exams.

4) Editorial: On Form ES-401-2, the importance rating indicated is that for the RO vice the SRO (3.2 vice 3.7). Question rewritten.

U 87 H 3 N RO knowledge. Question rewritten.

S

1) Answers B and D are related hydrogen concentration (the effect statement says pressure, but based on the context and Justification text, it is believed to be concentration) and temperature. It is noted that U

88 H 3 X X N temperature increases have no effect on hydrogen concentration, so if it is S not physically possible, it is hard to define these as plausible distractors.

2) The present question does not test the applicants ability to show selection of the appropriate procedure (10 CFR 55.43.5). Question rewritten.

For this to be a SRO only question, assessment of the situation with selection of the U appropriate procedure is needed (10 CFR 55.43.5). An applicant can answer the 89 H 2 X N S question using integrated plant and system knowledge, which is not unique to the SRO. Question rewritten. Page 8 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 9 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only

1) This question can be answered correctly by evaluating the Applicability statement of the correct LCO, which is not unique to the SRO.

U 2) In developing this as a SRO only question, it needs to be shown how the 90 F 2 X B content of the question addresses part of 10 CFR 55.43.

S

3) Verify nomenclature to see if N/A in Question History for Bank questions means that they were not used in the last 2 NRC exams, or what meaning of N/A is. Question OK as written.

91 H 3 B S E The actions described in the answer reflect knowledge/action that would be expected 92 H 3 X N of a RO in the control room during such operations. This does not appear to be a S SRO-only question. Changed distracters.

The K/A statement tests the knowledge coordinating EOP implementation in U coordination with other procedures. There is nothing in the question that addresses 93 H 3 X N S evaluation of EOP implementation versus abnormal procedure implementation.

Question OK as written.

94 H 3 B S 95 H 3 B S 96 H 4 B S

1) The question is written around ventilation changes in the Fuel Building based on high radiation indications. The correct answer says that control building supply will be secured, but this is driven by different ventilation radiation monitors (RE-22 or -33 versus RE-27 or -28). This would appear to make Answer D correct.

U 97 H 3 X X B 2) The content of the question is focused on system response to alarms and S

immediate actions related to an unexplained unplanned event. These on the surface are not SRO only specific knowledge items. Having details on the event and making a decision on procedure entry to justify the appropriate action would discern this as a SRO only question. Question rewritten.

The content of the question is focused on system knowledge and immediate actions U related to a fuel drop event. These on the surface are not SRO only specific 98 H 3 X N knowledge items. Having details on the event and making a decision on procedure S entry to justify the appropriate action would discern this as a SRO only question.

Question OK as written.

99 H 4 N S Page 9 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 10 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only

1) The question as written doesnt show the applicants ability to identify and check other plant indications that confirm or not what has happened.

E 2) The question gives multiple indications that support each other to show 100 H 3 X X N what the cause of alarm is, and asks for immediate action. This is not S SRO-only application. The indications should be evaluated to determine what the cause is, and entry into the correct procedure(s) should be shown.

Question revised.

B= 21 F= 33 (44%) E=

RO TOTALS: M= 1 H= 42 (56%) U= 4 Additional Notes:

N= 53 B= 7 F= 1 (4%) E= Additional Notes: The 32 percent SRO unsat performance was due to the exam SRO TOTALS: M= 3 H= 24 (96%) U= 8 writer assuming a correct methodology from RII exam writers would be satisfactory in RIV.

N= 15

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is __ 3___ on the RO exam and _3_ on the SRO exam.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 0/0 (GFE) 43.1 = 0/0 (Nothing here, could be an issue) 41.2 = 3/1 43.2 = 0/2 41.3 = 4/1 (GFE) 43.3 = 0/1 41.4 = 3/1 43.4 = 0/3 41.5 = 24/6 43.5 = 8/17 41.6 = 3/1 43.6 = 0/0 (Nothing here, could be an issue) 41.7 = 31/3 43.7 = 0/0 (Nothing here, could be an issue) 41.8 = 8/1 41.9 = 3/1 41.10 = 23/9 41.11 = 0/0 (Nothing here, could be an issue) 41.12 = 2/4 41.13 = 0/0 (Nothing here, could be an issue) 41.14 = 0/0 (GFE) Page 10 of 11 OBDI 202 - IOLE Process

ES-401 CW-2009-06 11 Form ES-401-9

5. The answer distribution is: RO / SRO A = 19 (25%) / 6 (24%)

B = 19 (25%) / 7 (28%)

C = 20 (27%) / 6 (24%)

D = 17 (23%) / 6 (24%)

6. There are 2 (RO) questions with attachments provided. Page 11 of 11 OBDI 202 - IOLE Process