ML20032B170
| ML20032B170 | |
| Person / Time | |
|---|---|
| Issue date: | 10/06/1981 |
| From: | NRC |
| To: | |
| Shared Package | |
| ML20030B696 | List: |
| References | |
| NUDOCS 8111050026 | |
| Download: ML20032B170 (8) | |
Text
.
3 OPERATOR LICENSING EXAMINATIONS ACTION PLAN A wide ranging discussion of the operator licensing program was conducted on August 4,1981 by the Director, and Deputy Director, Division oi Human Factors Safety with members of the Operator Licensing Branch, including consultants, a former member of the branch, and a representative from Analysis and Technology who had been the project manager for a study of the operator licensing program (NRC-03-80-ll7). The meeting was prompted by the recent investigation into cheating on examinations that revealed inadequate proctoring of examinations.
The agenda called for a detailed discussion of present practices in examination procedures, including development of the examination, administration at the facility, grading of the writtens, and orals and feedback to licensees.
A critical examination of the above was undertaken, followed by a discussion of methods to improve the effectiveness and efficiency of the examination process.
Subsequent to the meeting, we revised the form of the operator and senior operator written examination.
Listed below are the items of concern, conclusions, and the recommended actions.
4 Actions are addressed as short term and long term. Some actions directed toward eliminating cheating have already been implemented; most of the rest are short term and can become effective with approval of the action plan.
(
0111050026 811006 hR Df E PDR
A.
Preparation of Examinations a.
Members of the Branch indicated that, at times, they were having difficulty obtaining material necessary for preparing the examina-tions in the proper form and in a timely manner.
We have developed a form letter with appropriate attachments, addressed to corporate management that sets forth the utility's responsibility regarding material to be furnished, and other administrative items in order to assure proper preparation of the examinations and administration thereof.
b.
Although examiners submit examinations for review to the Section Leaders and/or Branch Chief, there is no formal QA plan for review of the examinations.
We have developed an Exaniners Standard (ES-108) that details a QA program for examination review prior to their administration.
c.
Prearrangenents with facility management do not detail. space requirements and instructions regarding required conduct of candidates during the administration of examinations.
\\
We have prepared a revision to Examiners' Standard,
~
ES-201 that will provide instructions to the examiners
~
regarding pre-arrangements, administration of the examinations at the facility, instructions to the candidates and proctoring requirements.
B.
Administration of Examinations a.
Written Examinations (1) There is a lack of detailed instructions to the examiners regarding proctoring, instructions to candidate, reference mdterial provisions, control of paper, allowing people to leavetheroom,andotheradministrativema}ters.
We have incorporated the instructions into Examiners' Standard ES-201.
~
(2) There is no detailed procedure for facility management review of the examination and concurrence in the applicability c the examination questions and answer key.
l l
We have incorporated the instructions into Examiners' Standard ES-201.
B.
Oral Examinations i
(1) There is a lack of uniformity in recording the results of the oral test and a lack of quantitativ,e criteria for evaluating the candidate's performance.
3-y yi e
_<------t,,9 gy.--
y
-,y-n 2-
For a short term solution, we will request all the examiners to review Examiners Standard ES-305, Instructions for Completing Oral Examination Notes and to submit their recommendations for improvement to the appropriate Section Leaders. The Branch Chief and Section Leaders will develop additional instructions for completing the notes.
(2) There are no Examiners Standards that address the simulator portion of the oral test.
Prepare letter for distribution to all' licensees that scopes simulator examinations for both plant specific and non-plant specific simulators.
The guidelines in this letter will be the basis for the first several sets of simulator examinations.
Based on the above examination experience, we will develop an Examiners' Standard that addresses simulator examinations.
C.
Grading and Final Review a.
OLB has consistently honored requests for examinations, including those for reexaminations for individuals who
s e *,
=
failed their original examination.
In order to accomodate these requests, OLB has deferred grading of other examinations until several sets of examinations have been administered.
This has on occasion resulted in long delays between administra-tion and final action on an application.
We will review the previous twelve month work load to determine the number of assignments that might have been avoided throug;i better scheduling.
Based on the review, we will develop a position that will minimize the number of trips to a facility in a twelve month period, so as to provide time to grade examinations immediately following their administration.
b.
Graded examinations are reviewed by the Section Leaders and Branch Chief prior to issuance of licenses or denial letters However, this is no formal QA procedure that indicates the scope of the review.
We have developed an Examiners' Standard' (ES-109) that details a QA program for the review of graded examinations.
5-i i
I
D.
Improve Format and Relevance of'the Written Examinations NUREG/CR-1750, Analysis, Conclusions and Recommendations Regarding Operator Licensing, comments from industry, and our own observa-l tions have indicated that the written examinations should be more relevent to the job responsibilities of operators and senior operators.
There are several studies underway that will use job task analyses and other techniques to develop more appropriate examinations.
For the short term, we have developed an examination that changes the format and content and reduces the time required to complete the written examination.
E.
Long Term Action Plans a.
Preparation of Examinations (1) OLB attempts to generate an original examination each time a set of examinations is prepared for a facility.
Of necessity, questions are repeated with, or wither'. slight variations.
Considerable time is required to put an examination together, including rewriting previously used questions.
Insufficient time is available to review facility operating history, LERs, abnormal occurences and IE
- 6' -
,n
Inspection Reports. Requirements for and development of a totally new examination appears to be needed and LQB of DHFS has the lead with coordination from OLB.
In the interim, we plan to develop a compre-hensive data bank of generic and plant specific questions and answer keys to facilitate more
. uniform and rapid preparation of examinations.
Examiner preparation time then can be concentrated on review of procedures, operating history, LERS, and IE reports.
(2) Many facilities are very similar; i e., BWR 4s and Westinghouse four loops.
Howeve.r they are treated as separate, different plants when scheduling examinations and preparing the examina-tions.
This adds to the time required for overall administration of examinations.
We plan to investigate the feasibility of identical /
l l
similar written examinations being administered concurrently for similar facilities at the same time to avoid compromising the examinations.
l l
s
m r
b.
Administration of Examinations (1)
Examiner Standards exist for administering the oral test. However, there are no specific criteria that assure a uniform, quantitative evaluation of candidates.
lle plan to develop a revised Examiners Standard that specifies minimum acceptable performance and knowledge in each area that is avaluated during the oral test and simulator test that will assure a more uniform quantitative evaluation of candidates.
(2) Simulator examinations administered by OLB,have been conducted at plant specific simulators in conjunction with,the oral test.-
The simulators and plant oral tests are normally. administered by the same examiner. With the advent of non-plant specific simulator examinations, there will be many occasions when different examiners administer the simulator cnd plant oral tests for the same candidate.
We plan to develop Examiner Notes for the Administra-tion of the simulator examination and oral test that clearly indicate those areas examined, so as to
' void duplication.
a i l l
,,