ML053010343

ES-401-9, Written Examination Review Worksheet, for the Prairie Island Initial Examination - August 2005

ML053010343
Person / Time
Site:	Prairie Island
Issue date:	08/08/2005
From:	Mcneil D NRC/RGN-III/DRS/OLB
To:	Nuclear Management Co
References
50-282/05-301, 50-306/05-301
Download: ML053010343 (5)

Text

ES-401, Rev. 9 Written Examination Review Worksheet Form ES-401-9 All questions were reviewed for psychometric flaws and K/A match. The last 25 questions are the SRO questions.

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1N H 2.5 S 2M H 2.0 S Procedure indicates the RCP should be tripped when the Rx is SD. Note 3N F 2.0 SU modifies the step to allow continued operation of the RCP. Left as is.

4N F 2.0 S Check terminology seal leakoff vs seal return flow Changed to seal 5B H 2.0 E leakoff flow.

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

• The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

• The answer choices are a collection of unrelated true/false statements.

• The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

• The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

• The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

• The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401, Rev. 9 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 6N H 2.5 U K/A is not correct. Question replaced.

Capitalize the word PREFERRED in the question stem. Capitalized &

7B F 2.5 E underlined.

Can distractor c. be correct if the applicant interpolates? If so, then two 8M H 2.5 SU correct answers. Procedure does not allow interpolation. Question is ok as is.

Sixth bullet - verb/tense error, pressures is. Fix verb/tense error.

9M H 2.0 E Fixed typo.

10M F 2.0 S 11N F 2.0 S 12B F 2.0 S 13N H 2.5 S Suggest providing a power level in the given conditions Added power 14B F 2.5 E level..

15N F 2.5 S 16B F 2.0 S 17M H 2.5 S 18M H 2.5 S 19M H 2.0 S 20N F 2.0 S 21B H 2.0 S Is this material required from memory? Procedure steps. Required from 22M F 2.0 S memory. Operators should know this from log taking and system training.

ES-401, Rev. 9 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only Memorizing sequences of steps is not required. Containment spray is only shut off in accordance with the correct answer. SI pumps are never 23B H 2.5 SU shut off until another injection source is provided. Although they are sequential steps in a procedure, a trained operator should be able to determine the correct answer with plant/system knowledge.

24M F 2.5 S Is the steam flow provided in the initial conditions excessive? Value 25M H 3.5 E modified to conform with plant conditions.

26M F 2.5 S May not be required from memory. Requires specific knowledge of an 27M H 2.5 SU RNO to get to the right answer. Question is a procedure basis question that an RO is expected to know from memory. OK as is.

28N F 2.0 S LOD may be too low - justify. Calculation is required to determine 29N H S correct answer. Not simple question.

30N H 2.5 S 31N H 3.0 S This appears to be more than memory. it requires a comparison of plant 32N F 3.0 U conditions with requirements and then making a decision. Is this required RO knowledge/skill? Higher skill. RO required knowledge.

33M H 2.5 S Suggest changing the words: were NOT returned to their normal 34N F 2.6 E position to: left in the withdrawn position to prevent incorrect assumptions. Change made.

35M H 3.0 S 36B H 2.0 S 37M H 2.5 S

ES-401, Rev. 9 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 38N H 2.5 S 39N H 2.5 S 40N H 2.5 S 41B H 2.5 S Can the applicant make a reasonable assumption that the low lube oil pressure trip has failed, the pump seized and answer d. is correct? This 42N H 2.0 S assumption would be incorrect. answer d. is not correct. Question is ok as is.

43N F 2.0 S Verify distractor b. is incorrect. DGs usually start on undervoltage. The explanation/justification may be incorrect. If not incorrect, then 2 correct answers provided. Can distractors c. & d. be reworked? If this is a vital 44B F 2.5 SU bus, there may be no lockout for the bus making the distractors implausible. Distractor b. is incorrect. Reworked c. & d. Distractors were weak but still plausible as written. Q was not unsat as submitted, but needed enhancement.

45M H 2.0 S Distractor d. is not plausible. Question is not higher skill, only memory.

46M H 2.0 U Agreed. Question changed to memory skill.

See if we can change ONLY. It doesnt complete the stem correctly.

47M F 3.5 E From memory? Question is ok as-is.

48N H 2.0 E Distractor b. is not plausible. Enhancement made.

Not required for ROs from memory. ROs are required to recognize entry conditions for tech specs, not memorize action statements. Change 49N F 2.0 SU action(s) is/are to ensure single and plural actions are evaluated.

Immediate action required from tech specs. Operators are required to know this from memory.

Second part of the stem is not necessary, only a repeat of the question.

50M H 2.5 E Agreed, modified question stem.

ES-401, Rev. 9 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 51N H 2.5 S 52N F 2.0 S 53N F 2.5 S 54M F 2.0 S 55M H 2.5 S 56N F 2.5 S 57N H 3.0 S 58B F 2.0 S This appears to be a fundamental memory question. agreed. Changed 59B H 2.0 U to memory level question.

60N H 2.0 S 61B H 2.0 S Distractor d. is also correct. Sequence in supplementary actions is not required to be memorized, hence, this step is a correct action to 62N H 2.5 U execute, although maybe not the next action. Agreed, modified distractor d.

63M H 2.5 U This material is memory. Agreed, changed to memory.

64N F 2.5 S 65N F 2.5 S 66N F 2.5 S Question does not discriminate between competent and less-than 67M F 1.0 U competent operator. Knowledge is required of non-licensed personnel.

Agreed, replaced question.

68B F 2.0 S

ES-401, Rev. 9 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 69N F 2.0 S 70N H 2.5 S 71N F 2.5 S 72N F 2.5 S 73B F 2.5 S 74N F 2.5 S This is a higher skill question based on where the break might be and 75N F 2.5 U the response of the fire protection equipment. Agreed, changed to higher skill.

76M H 3.0 U S 77N H 3.0 U S 78N H 3.0 U S 79N H 2.5 U S 80N H 2.0 U S 81N H 2.5 U S 82N H 3.5 U S 83N F 2.5 U S Make sure distractor a. is incorrect. Not required to re-insert bundle.

Suggest adding the following bullets: Unit 2 is at 100% power. and, All 84N F 2.5 U E other equipment is in normal operation. This will put bounds on any assumptions the applicants may make. Agreed, modified.

Verify in distractor (d.) that the containment cleanup fans will start with containment pressure at 12 psig. Check this answer is correct since 85N H 2.5 U S entry into YELLOW path is not required. Cleanup fans will start, answer is correct, entry into YELLOW path is required.

ES-401, Rev. 9 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 86N H 2.5 U S 87N H 3.0 U S 88N F 2.5 U S 89N H 2.5 U S 90N H 2.0 U S 91N F 2.0 U S 92N H 2.0 U S 93B F 2.0 U S 94B H 2.0 U S 95B F 2.0 U S 96B F 2.0 U S 97N F 2.0 U S 98B F 2.0 U S 99M F 2.0 U S Has to decide that an ATWS is occurring based on indications provided.

100 Suggest inserting some LOCA procedures to mix up the procedures.

F 2.0 U U M Agreed, changed to higher skill. LOCA would be incorrect and not plausible under the given conditions.