ML16125A075
| ML16125A075 | |
| Person / Time | |
|---|---|
| Site: | Robinson  |
| Issue date: | 04/27/2016 |
| From: | Division of Reactor Safety II |
| To: | Duke Energy Progress |
| References | |
| 50-261/OL-16-301 | |
| Download: ML16125A075 (4) | |
Text
Christopher McComber NRC exam author NRC exam analysis and comparison to Validation results The NRC exam had a total of 5 high missed questions, defined by greater than 40% of the candidates missing that particular question. They were R05, ROb, SRO$3, SR087 and 5R090. I will discuss each of these question individually in the following paragraphs.
R05 (53% missed): This question remained unchanged through the entire validation and approval with the exception of a valve nomenclature issue fCVC vs LCV) in one of the answer choices. The first round validation had 5 validators (2 RO/3 SRO), and 4 out of 5 (1 SRO missed it) answered this correctly. The second round validation contained 3 validators (2 RO/l SRO), ALL answered the question correctly.
After the NRC comment resolution, again this question remained unchanged, the test was validated by 2 RO and 2 SROs. Only 1 of the 4 got the question correct (1 RO). We looked at the fact that this question throughout the process had not changed, and was evaluated by a total of 12 validators with a 66.7%
success rate. None of the validators ever made comment to the question being unfair or not operationally valid.
Post exam review showed that our candidates that missed this question (2 RO and 6 SRO) all chose distractor D. The reason that they did not choose B, the correct answer, was that they in their minds convinced themselves that HCV-121 fails open, which is true on a loss of air to the valve. However the answer choice was that the valve failed closed. It did not imply that this failure was due to a loss of air, but each candidate that got this wrong, thought that it was trick that the valve didnt fail closed. When developing the question, we simply stated the valve failed in the closed position, and was due to control instrumentation to the positioner. The candidates made an incorrect assumption.
RO1O (67% missed): This question did change following the NRC review and the 401-9 comment resolution. The original question was rejected as an EDITORIAL for being GFES and not matching the KA on both parts. Therefore, this version of the question was only validated by the 4 validators (2 RO/2 SRO) post NRC comment resolution. It was missed by 2 (both SROs), so it had a 50% success rate. The AD-TQ-ALL-0220 requires that all questions with >50% miss rate are evaluated. The exam team (myself, Exam team supervisor and OPS SRO) still evaluated this question and determined that it was SAT and operationally valid. None of the validators made comment on the question.
Post exam review showed that the candidates that missed this question (4 RO and 6 SRO) did so for 2 reasons. The more common reason was that they used the incorrect curve (by temperature) on the curve provided. They chose the temperature that they were at, not where they were going to. The other reason that some chose the incorrect boron concentration was because they believed that we were asking for a difference in boron required. The stem contained information that the RCS was at 100 ppm already. The candidate in this case had to assume that there was a small difference in the math, because the choice picked when added to 100 ppm would put them at 780-790 ppm, which is above the
correct answer choice given by the correct curve, 760-770 ppm. Additionally the wording in the stem was as such, raise the RCS boron concentration to a MINIMUM of which should have clearly indicated that we were looking for a final value, NOT a difference to get to that value. There was one candidate that got the question wrong due to the other part, they just did not remember if the reactivity effects from cool down were worse at beginning or end of life.
5R083 (45% missed): This question also changed following NRC ES-401-9 comments. The NRC evaluated our initial submission as UNSAT. Of note, this was one of our early submission items, however due to the schedule of validation, both the 1st and 2 round validations occurred before we received the UNSAT and made changes to the question to fix that issue. The NRC stated it was NOT SRO only. Therefore, this version of the question was only validated by the 2 SRO validators post NRC comment resolution. Both of them answered it correctly and there were no comments.
Post exam reviewed showed that the SRO candidates that answered this incorrectly did so based on the fact that they assumed that something else had to be going on to drive backpressure into the restricted region and that it would prudent to have an AO dispatched to perform the Attachment 1, even though not procedurally directed. One candidate got both parts wrong, indicating an overall weakness in the mitigation strategies of AOP-012.
SROS7 (45% missed): This question was conceptually the same throughout validation (1st round 75%, 2 round 100%), but did have some changes in the stem (information placed into a table format vs bullets) to clean the question presentation up, and the way the second question statement asked for the material we were testing. The NRC comments had us make changes that created a situation in which there was no correct answer, and then we fixed that by adding an additional bullet in the stem. That last change (bullet concerning RWST level) was what led one of our 2 SRO validators to jump on the wrong answer immediately (50% success as the other SRO answered correctly). He recommended a change, which the exam team discussed with and had the NRC accept following the last round (post NRC comment) of validation. The edited question was then administered for validation to 3 SRO5, 2 of which had never seen any version of the question. The third was our second round validator and had previously answered it correctly. ALL 3 of these SROs answered the question correctly.
Post exam review showed that the SRO candidates that missed this did so for the same reasons. They either did not recognize that the C SG was both faulted and ruptured, and determined that it needed to be fed to greater than 9% NR level (SGTR mitigation strategy) or they did recognize that it was both faulted and ruptured, and incorrectly believed that the mitigation strategy still applied (feed to >9%).
One candidate got both parts incorrect, indicating that they did not realize that EOP-ECA-3.1 is required to be entered first, and that you cannot direct enter EOP-ECA-3.2 from EOP-E-3.
SRO9O (55% missed): This question was UNSAT on initial submission, and had a KA change. The new question written to the new KA was also deemed UNSAT, due to being LOD=1. A new question was written which only the last 2 SRO validators saw. Both of those validators answered the question correctly and there were no comments.
Post exam review showed that every one of the candidates that answered this wrong picked the B answer choice. This was because they did not recognize that they did not have a spray flow path. This was due to the Sl-8SOC valve not having power, which the candidates would have needed to evaluate from the given information that was in the stem.
Christopher McComber NRC exam author NRC Post Exam Comments Following the exam analysis and review, H.B. Robinson has no Post Exam Comments to formally make on any questions contained within the ILC-16 NRC Written Examination.
Christopher McComber NRC exam author NRC exam analysis and comparison to Validation results The NRC exam had a total of 5 high missed questions, defined by greater than 40% of the candidates missing that particular question. They were R05, ROb, SRO$3, SR087 and 5R090. I will discuss each of these question individually in the following paragraphs.
R05 (53% missed): This question remained unchanged through the entire validation and approval with the exception of a valve nomenclature issue fCVC vs LCV) in one of the answer choices. The first round validation had 5 validators (2 RO/3 SRO), and 4 out of 5 (1 SRO missed it) answered this correctly. The second round validation contained 3 validators (2 RO/l SRO), ALL answered the question correctly.
After the NRC comment resolution, again this question remained unchanged, the test was validated by 2 RO and 2 SROs. Only 1 of the 4 got the question correct (1 RO). We looked at the fact that this question throughout the process had not changed, and was evaluated by a total of 12 validators with a 66.7%
success rate. None of the validators ever made comment to the question being unfair or not operationally valid.
Post exam review showed that our candidates that missed this question (2 RO and 6 SRO) all chose distractor D. The reason that they did not choose B, the correct answer, was that they in their minds convinced themselves that HCV-121 fails open, which is true on a loss of air to the valve. However the answer choice was that the valve failed closed. It did not imply that this failure was due to a loss of air, but each candidate that got this wrong, thought that it was trick that the valve didnt fail closed. When developing the question, we simply stated the valve failed in the closed position, and was due to control instrumentation to the positioner. The candidates made an incorrect assumption.
RO1O (67% missed): This question did change following the NRC review and the 401-9 comment resolution. The original question was rejected as an EDITORIAL for being GFES and not matching the KA on both parts. Therefore, this version of the question was only validated by the 4 validators (2 RO/2 SRO) post NRC comment resolution. It was missed by 2 (both SROs), so it had a 50% success rate. The AD-TQ-ALL-0220 requires that all questions with >50% miss rate are evaluated. The exam team (myself, Exam team supervisor and OPS SRO) still evaluated this question and determined that it was SAT and operationally valid. None of the validators made comment on the question.
Post exam review showed that the candidates that missed this question (4 RO and 6 SRO) did so for 2 reasons. The more common reason was that they used the incorrect curve (by temperature) on the curve provided. They chose the temperature that they were at, not where they were going to. The other reason that some chose the incorrect boron concentration was because they believed that we were asking for a difference in boron required. The stem contained information that the RCS was at 100 ppm already. The candidate in this case had to assume that there was a small difference in the math, because the choice picked when added to 100 ppm would put them at 780-790 ppm, which is above the
correct answer choice given by the correct curve, 760-770 ppm. Additionally the wording in the stem was as such, raise the RCS boron concentration to a MINIMUM of which should have clearly indicated that we were looking for a final value, NOT a difference to get to that value. There was one candidate that got the question wrong due to the other part, they just did not remember if the reactivity effects from cool down were worse at beginning or end of life.
5R083 (45% missed): This question also changed following NRC ES-401-9 comments. The NRC evaluated our initial submission as UNSAT. Of note, this was one of our early submission items, however due to the schedule of validation, both the 1st and 2 round validations occurred before we received the UNSAT and made changes to the question to fix that issue. The NRC stated it was NOT SRO only. Therefore, this version of the question was only validated by the 2 SRO validators post NRC comment resolution. Both of them answered it correctly and there were no comments.
Post exam reviewed showed that the SRO candidates that answered this incorrectly did so based on the fact that they assumed that something else had to be going on to drive backpressure into the restricted region and that it would prudent to have an AO dispatched to perform the Attachment 1, even though not procedurally directed. One candidate got both parts wrong, indicating an overall weakness in the mitigation strategies of AOP-012.
SROS7 (45% missed): This question was conceptually the same throughout validation (1st round 75%, 2 round 100%), but did have some changes in the stem (information placed into a table format vs bullets) to clean the question presentation up, and the way the second question statement asked for the material we were testing. The NRC comments had us make changes that created a situation in which there was no correct answer, and then we fixed that by adding an additional bullet in the stem. That last change (bullet concerning RWST level) was what led one of our 2 SRO validators to jump on the wrong answer immediately (50% success as the other SRO answered correctly). He recommended a change, which the exam team discussed with and had the NRC accept following the last round (post NRC comment) of validation. The edited question was then administered for validation to 3 SRO5, 2 of which had never seen any version of the question. The third was our second round validator and had previously answered it correctly. ALL 3 of these SROs answered the question correctly.
Post exam review showed that the SRO candidates that missed this did so for the same reasons. They either did not recognize that the C SG was both faulted and ruptured, and determined that it needed to be fed to greater than 9% NR level (SGTR mitigation strategy) or they did recognize that it was both faulted and ruptured, and incorrectly believed that the mitigation strategy still applied (feed to >9%).
One candidate got both parts incorrect, indicating that they did not realize that EOP-ECA-3.1 is required to be entered first, and that you cannot direct enter EOP-ECA-3.2 from EOP-E-3.
SRO9O (55% missed): This question was UNSAT on initial submission, and had a KA change. The new question written to the new KA was also deemed UNSAT, due to being LOD=1. A new question was written which only the last 2 SRO validators saw. Both of those validators answered the question correctly and there were no comments.
Post exam review showed that every one of the candidates that answered this wrong picked the B answer choice. This was because they did not recognize that they did not have a spray flow path. This was due to the Sl-8SOC valve not having power, which the candidates would have needed to evaluate from the given information that was in the stem.
Christopher McComber NRC exam author NRC Post Exam Comments Following the exam analysis and review, H.B. Robinson has no Post Exam Comments to formally make on any questions contained within the ILC-16 NRC Written Examination.