ML20069L741
| ML20069L741 | |
| Person / Time | |
|---|---|
| Site: | Millstone  |
| Issue date: | 06/30/1994 |
| From: | Ruth B, James Smith NORTHEAST NUCLEAR ENERGY CO. |
| To: | |
| Shared Package | |
| ML20069L739 | List: |
| References | |
| NUDOCS 9406200203 | |
| Download: ML20069L741 (10) | |
Text
i f
I MILLSTONE UNIT 3 POST NRC RO EXAMINATION EVALUATION JUNE 1994 W s3 Prepared by:
J\\ce Bradley W. Ruth, S or Operator Trairiing - MP3 M
I ThM.
Accepted by:
V Jeffrey F. Smith, Manager Operator Training - MP3 & CY l
i 9406200203 940610 PDR ADOCK 0500 s
V
MILLSTONE UNIT 3 POST NRC RO EXAMINATION EVALUATION INTRODUCTION During the week of February 14,1994, NRC representatives and Millstone Unit 3 (MP3)
Training and Operations representatives, under examination security agreement, reviewed all portions (written, operating, and job performance measures) of the Reactor Operator (RO) examinations to be administered during the week of February 28,1994. Many successful examination reviews had been conducted by Millstone personnelin the past, but a complete changeout in MP3 Operator Training management resulted in a break in those successful practices. Specifically, past practice had been to include a supervisor and several experienced reviewers on the facility examination review team. In this instance, no supervisory personnel were included on the facility's team, and only two reviewers were provided for the written exam.
Additionally, the position of MP3 Operations Liaison had not yet been filled. Thus, the Training Department requested the assistance of an on-shift Senior Reactor Operator for the exam review, rather than using the Operations Liaison -- a Shift Supervisor level individual assigned to training support and oversight on a full-time basis by Operations Management.
The February 1994 examination review team consisted of:
o Paul Bissett NRC Chief Examiner for the MP3 RO examination i
o Joe D' Antonio NRC Examiner o
Bill Hemming NRC Contract Examiner (examination author) o Joe Arsenault MP3 LOIT Program Coordinator o
Mike Brewster MP3 Supervising Control Operator o
Bili Landon MP3 Simulator Operations Assistant (simulator operator) o Bob Royce MP3 NLO Coordinator (in-plant JPM escort)
The written examination was administered to five (5) RO license candidates on February 28,1994. Although student performance on the operating test and job performance measures was very good, the reconstructed results of the written examination indicated that the candidates had significant difficulty with this portion. During the March 4,1994 i
examination exit meeting, the NRC commented on the ineffectiveness of the preexamination review, and the low quality of the training materials provided to the NRC for examination development.
A detailed post examination review, conducted by MP3 training supervision and staff, resulted in formal written comments to the NRC on 15 of the written examination MP3 Post Examination Evaluation Page 1 of 6
1 questions. The comments resulted in six (6) question deletions and six (6) answer key modifications (two correct answers). After the modifications, all of the candidates passed the examination with greater than 80%.
SCOPE The scope of this evaluation includes; 1) a review of the preexamination review process, and 2) the quality of the submitted materials for use as an examination development tool.
This scope was chosen for the following reasons:
o The large number of r "'en examination comments (fifteen) that were submitted to the NRC frur;I the post examination review, and the NRC's subsequent disposition of those comments; o
The NRC's comments during the preexamination review week and at the NRC examination exit meeting stating that materials provided for examination development (specifically lesson plans) required significant revision; o
Good student performance on both MP3's final examinations, and the NRC's Operating and the Job Performance Measures examinations; o
An independent assessment of each student's readiness for the Operating and Job Performance Measures por'J:ns of the examination was conducted by senior members of the North Atlantic Energy Service Company Operator Training staff prior to the NRC examination.
Their results were positive and consistent with those of the MP3 training and operations staff; and o
Good student performance on the MP3 Licensed Operator Initial Training (LOIT) written progress examinations.
The questions used on these progress examinations were considered to be of consistent quality, and at the same cognitive level, as questions used by the NRC during recently administered examinations at other utilities.
MP3 Post Examination Evaiestion Page 2 of 6
ANALYSIS and CONCLUSIONS J
The analysis and conclusions are categorized into the two (2) specific evaluation areas identified in the Scope.
CONCLUSION - Preexamination Review Process i
NNECO's Preexamination Review Process did not ensure an accurate wdtten examination.
ANALYSIS At the time of the MP3 LOIT examination, past successful examination reviews had been based on lessons learned by experienced reviewers, rather than a documented method or guideline. Also, due to recent management changes within the Nuclear Training Department, none of the supervisory personnel in the MP3 chain of comrrand through the department Director had been directly involved in a complete initial examination review. The lack of prior, direct experience, and lack of written guidance, resulted in decisions that compromised the quality of the preexamination review process. These decisions were:
o Supervision was not dimetty involved in the preexamination review process The preexamination review process did not directly include training supervision.
The decision was based on the following items.
1)
The MP3 Supervisor, Operator Training was rewly assigned to the unit.
Based on a presumed deficit in unit-specific technical knowledge, this individual was not placed on the exam review team. This meant he was not on the security agreement, and was available to oversee the final preparation of the exaniinee teams on the simulator. In retrospect, his involvement in the exam review effort would have significantly improved NU's overall performance on this occ5sion.
2)
The MP3 Assistant Supervisor Operator Training was assigned to be the SRO " role player" for the NRC examinations.
This precluded his involvement in the preexamination review process.
Failure to include management individuals in the review process placed the MP3 reviewers in the position of making judgements and decisions without the benefit of managementinput. The full burden of examination acceptability was placed on the reviewers. In retrospect, supervisory involvement in the review process would MP3 Post Examination Evaluation Page 3 of 6
likely have led to a written examination that would not have required significant post-examination changes.
i o
An insufficient number of Individuals were involved in the preexamination review process The preexamination review process did not includo a sufficient number of people to conduct a complete and accurate assessment of the written examination. Four (4) individuals were assigned to the preexamination review process, but only two (2) of these individuals were assigned to assess the written examination content.
A third individual was assigned to operate the simulator and ensure operating examination / job performance measure scenarios did not exceed the simulator certification limits. The fourth individual was assigned only to escort the NRC examiners forin-plantjob performance measure validations. This level of support was not sufficient to research al] aspects of the written examination and simultaneously validate the operating and job performance measure portions of the NRC prepared RO examinations in the allotted time period.
CONCLUSION - Trainina Materials MP3 Training Materials provided to the NRC did not readily support Examination Development needs.
ANALYSIS During the 1990 and 1991 time period, the Millstone Unit 3 Operator Training programs experienced a major tumover of staff. The subsequent hiring of new staff members, concurrent with the associated responsibility of Licensing / Certifying these individuals, limited the ability of the Training Department to totally revise training materials. Pen and ink changes were made to a significant number of documents to accurately reflect the on-going modifications to plant design and procedures. With the full restoration of the staff, an on-going program to go back and revise the corrected materials was implemented. This progress is currently on schedule with the majority of the NSSS system being fully revised by this fall.
Included in the revision process is:
o A revalidation of all operator task lists.
o Upgrading of the cognitive level of objectives (an MP1 LORT '91 and '92 Lesson Leamed) 4 MP3 Post Examination Evaluation Page 4 of 6
o Revision of text and lesson materials to include pen and ink corrections and revised objectives to reflect the higher cognitive level.
o incorporation of consistent instructional strategies into training materials.
The impact of degraded quality of the materials on the NRC's examination development, and pre-examination review is summarized below.
o The MP3 examination bank provided to the NRC was difficult to use j
The examination bank provided to the NRC was difficult to use for examination development. The examination bank items submitted to the NRC were provided in numerical order versus sorted by system orlesson plan. Also, the examination items were not tied to leaming objectives or the Knowledge and Ability Catalog (K&A). These shortcomings resulted from:
1)
The conversion from the old Training Management System (TMS) to a newly implemented, improved system (Taskmaster) was incomplete.
Specific examination items were not yet linked to learning objectives or lesson modules.
2)
The limited sorting capabilities of the TMS.
The NRC examination writer stated during the preexamination review week that it was easier to generate examination items "from scratch" than it was to select and use questions from the MP3 examination bank. Thus, inadequately organized examination bank materials placed the full burden of examination development on the NRC's examination writer.
o LOIT oblectives are written to lower coonitive level in 1993, in response to the 1991-1992 MP1 Requal failures, a new model for establishing higher cognitive level questions was established. Unfortunately at the time of the exam there was not a sufficient number of exam questions at the new level. It is NU's belief, had the upgrade effort of the objectives and exam bank i
been further advanced,it would have facilitated the development of an exam more
)
focused on system interrelations and integrated knowledge vice very special detailed information.
Lesson plans and texts had many " pen and ink" cha nces that were difficult to readl o
Lesson plans and texts provided to the NRC had many " pen and ink" changes that were difficult to read. This was because the " pen and ink" changes did not always MP3 Post Examination Evaluation Page 5 of 6
copy well, and were in some cases, unreadable. There were also some instances where numerous " pen and ink" changes to the same document made it difficult to follow and use for examination development. " Pen and ink" changes have been used as an allowable means ofincorporating "non-intent" (learning objective intent not impacted) changes into text and lesson materials in lieu of making full revisions requiring unneeded reviews and approval. However, the illegible " pen and ink" changes could have limited the NRC examination writer's ability to use a more professionally based material in developing the exam.
CORRECTIVE ACTIONS We believe two major activities will prevent recurrence in future NRC examinations.
These activities are:
1)
Retum to our past practice of having an appropdate number of reviewem, including supervision.
2)
Completion of the MP3 Training Matedal Upgrade Project.
MP3 Post Examination Evaluation Page 6 of 6
- ~ _. -... -.. _... _... -
o A
Docket'No. 50-423 B14876 Millstone Nuclear Power Station, Unit No. 3 Response to Reactor Operator Examination Report No. 50-423/94 :
. Memorandum from D.
B.
Miller, Jr. to Millstone Lilensed Operators and Operator Instructors June 1994
e O
Northeast Utilities System Memo June 10, 1994 MP-94-396 TO:
Millstone Licensed Operato 'ar'd Operator Instructors e
FROM:
D. B. Miller, Jr. O
~
Senior Vice PresTde Millstone Station Ext. 4300
SUBJECT:
CONDUCT OF OPERATIONS During recent NRC licensing examinations, a contradiction between station procedures i
and management philosophy was identified. Station procedures state that Reactor Operators (RO) have the authority and responsibility to initiate Reactor Protection System (RPS) functions anytime they determine such action is necessary to maintain the reactor plant in a safe operating condition. On the_ other hand, department management stated that they preferred ROs to receive direction from the Shift Supervisor or Supervising Control Operator prior to manually initiating RPS functions.
A broad range of answers on an examination question relating to this topic showed the RO candidates were confused as to what they could and could not do.-
I have reviewed our Millstone procedures for this situation and believe that proper, and conservative guidance is contained in the Millstone Conduct of Operations, OP 276/2276/3276. Section 6.1.4.5 of this procedure states:
"If at any time a Control Operator believes that a manual scram / trip or ESF/ safeguards actuation is necessary they shall announce the pertinent plant conditions and their recommendation or intention and then perform the required action. No response is required from the SCO or SS."
i This direction is correct and proper, and it is my expectation that each licensed operator will take such action, when appropriate, without hesitation. Teamwork and good communications demand that the operator announce both the conditions and the actions he is initiating. However, the announcement, or lack of an acknowledgement from the SS or SCO, should not delay the required action.
Feel free to discuss this issue with your unit's management, or me directly, if you desire additional information or clarification.
m*0 REV 094
- -v
.n
i D. B. Miller, Jr.
Conduct of Operations Page 2 of 2 cc:
S. E. Scace J. P. Stetz T. C. Feigenbaum H. F. Haynes G. H. Bouchard 4
F. R. Dacimo M. B. Brown M. H. Brothers J. F. Smith 1
R. W. Heidecker B. W. Ruth D. J. Meekhoff 1
.