ML16280A462

2016-09 Draft Written Comments

ML16280A462
Person / Time
Site:	River Bend
Issue date:	09/30/2016
From:	Vincent Gaddy Operations Branch IV
To:	Entergy Operations
References
50-458/OL-16, ES-401-9
Download: ML16280A462 (12)

Text

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 to 5 (easy to difficult) rating scale (questions in the 2 to 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

a. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

b. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

c. The answer choices are a collection of unrelated true/false statements.

d. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

e. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

a. The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

b. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

c. The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

d. The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

Place the answer letter here Place the 55.41 or 55.43 item here

8. At a minimum, explain any Unsat ratings (e.g., how the Appendix B psychometric attributes are not being met).

Ans

1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 CFR 8 Letter stem cred job #/ back- SRO- A/B/C/ 55.41x55 Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only D .43y 1 F 2 N S D 41.8 2 H 3 N S C 41.8 3 F 3 N S C 41.7 4 H 2 N S B 41.14 Is C also correct? Val results seem to indicate this. NRC corrected by adding incorrect sp 5 for the 15 valve in distracter C and new more credible distracter for H 3 N S B 41.7 D.

6 H 3 N S C 41.5 7 H 3 N S D 41.5 8 H 3 N S A 41.7 RBS wrote new question, sat.

9 H 2 N S A 41.2 10 H 3 N S D 41.6

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 11 F 3 N S C 41.7 NRC 2003 Exam (Q14)

12. Rev 5 - NRC concern is that If all RHR pump are running and RPV pressure is 200 psig then all RHR pumps should also be injecting at near rated flow of

~5000gpm. With ~15000 gpm going in an applicant could argue that while level is currently -150 inches, level would be going up very rapidly since BWR vessels are ~200 gal/in. Therefore, level would recover before the ADS timer times out making C also 12 H 3 B S A 41.8 correct. It seems that maybe the real question is can you have a leak in the drywell that exceeds RHR capacity with 3 pumps running so that water level cant recover and truly stabilizes at -150 inches as given in the stem? We just gave them one RHR pump in the stem to fix this concern. RBS okay with this fix.

There doesn't seem to be any discriminatory value. B is always 13 H 2 N S B 41.7 correct. Need to fix this question.

NRC Wrote new question and is now sat.

14 F 3 N S B 41.7 15 F 2 N S D 41.7 16 H 2 N S C 41.7 17 H 2 N S B 41.7 18 H 3 N S C 41.7

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 19 F 3 N S A 41.7 wrote new question-needs validation. STM incorrect?? New 20 F 3 N S B 41.7 question written by RBS and is now Sat.

21 F 4 N S D 41.7 PS is NJS-SWG1U1 was not 22 F 3 N S B 41.7 correct-NRC corrected it.

23 H 3 N S A 41.7 NRC changed PS for A and B 24 F 3 N S D 41.7 after RBS changed them to safety-related MCCs, which was not credible. Question is now Sat.

25 H 3 B S A 41.4 NRC 2012 (Q47)

NNS switchgears don't seem plausible because they are 4160 switchgears and not 480. NRC 26 F 2 N S D 41.7 changed all distracters. NJS-SWG1B and NJS-SWG1D were used to fix this issue. It is now Sat.

27 H 3 N S D 41.6 28 F 3 N S C 41.6 NRC Fixed distracter D issues-too 29 H 3 N S C 41.5 long, too much info compared to others question is now Sat.

30 F 3 M S C 41.7 31 F 3 N S A 41.7 We change distractor D to EHS-MCC2H for consistency and 32 F 4 N S B 41.7 credibility. NRRC agrees and is now Sat.

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 Can eliminate two of the temp aspects from Q33 with info from Q51, questions are very similar.

33 F 2 N S D 41.9 NRC took temps out of Q51 and used four reasons (1 X 4 question) to fix this issue. RB agreed and question is now Sat.

34 H 3 N S B 41.5 35 F 2 N S A 41.5 36 H 3 N S C 41.7 37 H 3 N S B 41.9 38 F 3 N S A 41.10 39 H 3 N S D 41.5 C and D are not credible for this 40 F 3 N S A 41.8 question.NRC repaired and question is now Sat.

This question was replaced by RBS but not finished and also has 41 F 2 N S C 41.10 unbalanced distracter D. NRC repaired both of these items and question is now Sat.

RBS modified the question significantly but did not fix the 42 H 3 N S D 41.4 explanation section. NRC completed this work - sat.

43 H 3 N S A 41.7 44 F 3 N S D 41.10 NRC edited explanation items and 45 H 2 N S B 41.7 changed from H2 to F3. Item was Sat.

46 F 2 N S D 41.7 47 F 2 B S C 41.10 2010 NRC

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 Rev5- question did not address the second part of KA, enclosure 12 could be used to bypass the 48 H 3 N S C 41.7 trip, and stem had cueing challenges with it, so a new question was written that addresses both parts of the KA.

Question is now Sat.

49 H 3 B S D 41.5 2010 NRC (Q11) 50 F 4 N S B 41.9 51 H 3 B S A 41.9 2008 Audit Exam 52 F 2 N S C 41.7 2010 NRC 53 H 3 B S A 41.10 Reference Required to answer question.

54 H 3 N S B 41.7 55 H 3 N S C 41.8 Rev 5- NRC changed it back to the originally submitted question.

Distracters C and D are not credible. You cant start an event with an SAE or GE before you melt the fuel, etc, which takes time. The design of the plant 56 F 4 N S A 41.10 makes these not credible. Also, for the KA, an inter-relation between offsite release rate and the e-plan for an RO is what instrument is used for the EOP entry, which is an RO task.

Question is now sat.

NRC Edited stem to be correct 57 H 3 N S D 41.5 and edited the distracters (part 2 of B and C) for balance-question is now sat.

58 H 3 N S A 41.10

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 Replaced distracter C since it was 59 H 3 N S D 41.5 not credible without a cooldown in progress-sat.

60 F 3 N S A 41.7 NRC repaired question-sat.

61 H 3 N S B 41.7 62 H 4 N S D 41.2 NRC wrote new question-sat.

63 H 3 B S C 41.13 2003 NRC RBS wrote new question for this topic but the distracters are not balanced. You can answer the question with knowledge of 64 H 3 N S D 41.5 second part only and first part is the KA. I corrected this issues by modifying the distracters into a proper 2 X 2. RBS repaired and question is now Sat.

RBS modified question to add groups (GRP2, 3, etc) and broke question due to KA mismatch-took reason for actions out of 65 H 3 N S B 41.7 selections- NRC put this back into the selections to repair the question-now sat.

66 F 2 N S B 41.10 Negatively focused stem and NUREG discourages this due to confusion-there are several of 67 H 3 N S C 41.10 these on the exam so removing this question and one other one.

NRC wrote new question and it is now sat.

Need explanation section 68 F 2 B S B 41.7 completed for substituted question made by RBS. Corrected in rev6 submittal and is now Sat.

69 F 3 N S A 41.10

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 NRC changed answer letter for 70 F 4 B S C 41.10 better distribution-sat.

71 F 3 B S D 41.10 2015 NRC 72 F 2 N S A 41.12 73 F 2 N S D 41.12 74 F 3 N S C 41.10 Rev5- NRC - Needed a different question-negatively focused stems are not preferred. NRC pulled bank question from 2003 NRC exam, Q77. Updated for 75 H 3 B S A 41.7 distracter wording but all choices remain the same, answer verified through updated EPTSG-2-now sat.

June 2007 NRC 76 F 3 B S D 43.2 77 H 3 N S A 43.5 2014-03 NRC 78 H 3 B S A 43.4 Reference Required to answer question.

79 H 3 N S D 43.2 80 H 3 N S B 43.5 81 H 3 N S A 43.5 82 F 3 N S D 43.4 83 H 3 N S B 43.2 84 H 3 N S C 43.2 85 H 3 B S B 43.5 2014-12 NRC

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 86 F 3 N S D 43.2 Reference Required to answer 87 H 3 N S C 43.5 question.

88 F 3 N S C 43.2 Reference Required to answer 89 H 3 N S C 43.5 question.

90 H 2 N S A 43.5 91 H 3 N S C 43.2 Reference Required to answer question.

Most systems in tech specs are designed for the design basis LOCA. This is systems 92 F 3 N S B 43.2 knowledge. K/A mismatch.

Doesn't test on cause of high radiation. Not really a Tier 1 question either. Wrote a new question and is now sat.

93 H 3 N S A 43.5 94 F 2 N S D 43.5 95 H 3 N S A 43.5 96 H 3 N S C 43.5 Reference Required to answer 97 H 3 B S C 43.5 question.

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 NRC 2012 Pulled a bank question for this one. The question we wrote is really not SRO only even though procedure selection is there, it is embedded in procedure hierarchy to enter AOP first, then SOP. On 98 H 3 B S A 43.4 the pulled bank question, we modified the distracters slightly for balance and better credibility but the question is the same and is now sat.

Reference Required to answer 99 F 2 N S B 43.5 question.

Reference Required to answer 100 F 3 N S D 43.1 question.

Results Table RO LOK -H 40 Avg RO LOD 2.83 Flaws 10 CFR Distribution RO LOK-F 35 AVG SRO LOD 2.88 Stem focus 0 41.1 13 43.1 1 SRO LOK - H 17 Overall LOD 2.84 Cues 0 41.2 2 43.2 8 SRO LOK - F 8 T/F 0 41.3 0 43.3 0

%  % Cred Dist 0 41.4 2 43.4 3 RO Bank 11 14.67 SRO Bank 5 20 Partial 0 41.5 10 43.5 13 RO Mod 1 1.333 SRO Mod 0 0 job link 0 41.6 3 43.6 0 RO New 63 84 SRO New 20 80 units 0 41.7 32 43.7 0

% minutia 0 41.8 5 Total Bank 16 16 backward 0 41.9 4 Total Mod 1 1 KA 0 41.10 13 Total New 83 83 SRO-only 0 41.11 0

%  % LOD = 1 0 41.12 2 RO Sat 75 100 SRO Sat 25 100 41.13 1 RO Unsat 0 0 SRO Unsat 0 0 Answer Dist ( in %) 41.14 1 RO Edit 0 0 SRO Edit 0 0 RO-A 19 SRO-A 7

%  % RO-B 17 SRO-B 5 Total Sat 100 100 Total Unsat 0 0 RO-C 19 SRO-C 7 Total Edit 0 0 RO-D 20 SRO-D 6

ES-401 ST-2016-05 Written Examination Review Worksheet Form ES-401-9 Question Overlap with the Previous Two NRC Exams QA form ES-401-6 item 4 requires 4 or less questions on RO exam from previous two NRC exams, SRO portion two or less questions from previous two NRC exams*

• if this is exceeded, the region shall call NRR to explain why/reasons.

RO overlap from previous two NRC exams 2013 exam - Q1 2014 exam - Q32, Q39 SRO overlap from the previous two NRC exams 2014 exams - Q78, Q85

Questions Requiring References to Answer 6-10 questions rule of thumb on SRO portion of exam are allowed to be open reference, while the RO exam is generally none RO Questions that require hand-outs 53 EOP-00001, Fig 2 (HCTL curve)

SRO Questions that require hand-outs 78 needs TS3.8.7, TS 3.8.9, STP-302-0102 87 needs EAL table only 89 needs 3.3.3.1 PAM Inst 92 Table SC-2 97 needs TRM 3.3.11.3 99 needs EAL table only 100 needs EAL table only