ML15272A137
| ML15272A137 | |
| Person / Time | |
|---|---|
| Site: | Harris  |
| Issue date: | 09/28/2015 |
| From: | Office of Nuclear Reactor Regulation |
| To: | |
| Scot Sloan 301-415-1619 | |
| Shared Package | |
| ML15272A103 | List: |
| References | |
| Download: ML15272A137 (2) | |
Text
Page 1 of 2 During the Plant Harris 71111.11 inspection the week of 8/3/15 the site was informed of a potential written exam item flawed rate of >20%. This greatly exceeded what was found during multiple independent reviews during the inspection readiness performance. The discrepancy between the two evaluation results was surprising because the Duke Energy Fleet participates in NRC exam workshops, performs benchmarking and assessments across the Regions to determine evaluation methods, and follows prescribed processes to ensure exam quality. The implementation of the comprehensive exam process determines areas in which retraining is needed to upgrade licensed operator and senior operator knowledge.
The driver for the difference in evaluation results centered around a previously unused method for determining Level of Difficulty(LOD) for exam items. The new evaluation was based on the use of electronic search functions of available reference material that supports the open reference format. This new expectation would result in a fundamental change across the industry in exam question design and challenges an accepted industry practice of using electronic reference materials.
The Inspection Procedure is clear on determining plausibility for distractors from ATTACHMENT 71111.11 page 29of 47; Two or more distractors are not credible. For open-reference questions, distractors should be judged as if the question was closed-reference This statement is the basis for how we choose plausible distractors during development we analyze for items such as a controllers normal and emergency set point. Both numbers are used lending to the plausibility of the answer choice.
The new basis for calling a question flawed due to low LOD, is because using search functions on distractors may allow them to be easily eliminated, thereby reducing difficulty level to the point of not being a discriminatory question.
The Duke Energy position is that plausibility and LOD with regards to distractor analysis cannot be mutually exclusive. If a licensed operator is presented with a correct answer and three plausible distractors the question will be discriminatory.
The use of electronic references with search capabilities during examinations has been a standard practice for years and we know of no issues raised for questionable LOD determinations because of the ability to use electronic search functions.
While the electronic search feature issue raised by the lead NRC inspector may need further evaluation, the industry and Duke uses question validations and actual exam results to verify LOD. Therefore we do not support any questions being challenged under the new search capability evaluation method used for the 2014 Harris exam questions that were reviewed and said to meet this new rejection criteria.
Our standards in development align with the industry. We have no reason to believe our written exams are not valid and discriminatory. If it is necessary to mitigate the word search exam taking strategy we will need regulatory and industry involvement to
Page 2 of 2 determine an appropriate course of action to meet the new expectation that prevents unintended consequences while maintaining an operationally valid exam.