ML20135E322

Independent Review Team Rept 1996 MP -1 Lout NRC Exam Failures

ML20135E322
Person / Time
Site:	Millstone, Haddam Neck  File:Connecticut Yankee Atomic Power Co icon.png
Issue date:	02/13/1997
From:	Barron W, Dooley L, Frye S CONNECTICUT YANKEE ATOMIC POWER CO.
To:
Shared Package
ML20057G416	List: ML20135E315 ML20135E322
References
NUDOCS 9703060432
Download: ML20135E322 (83)

Text

. - - - . _ - - - - - _ . _ . - . - - .

-~

- Northeast UtilitiesSystem

/[//[\h Independent Review Team Report .

1996 MP -1 LOUT NRC Examination Failures t

l l

PREPARED BY:

a a

khJH# Ann -

1/r>M L. 51. Dooley Cf' ream Member l W. H. Barron Team Member i Mb v Y Nam Member

\

i S. R. Frye Team Member T. P. GilbeVt ,

hdt$ h /w femm Member C. W. Sawyer Ln w #!'#/SA K. D. Mu(cino Team Lead Date REVIEWED BY:

LLM~w /~ 4de

. Ferguson / Date

~

ACCEPTED BY:

aM L' 9l/Sk7

/ bate J. Kyayer V

18882 318883 PDR;,g 5

. . j i

l 1

EXECUTWE

SUMMARY

Backaround Early in 1996, Unit i Operations management requested a Licensed Operator Upgrade Traimng (LOUT) program be developed and presented. The original candidates for this program were to be NRC licensed reactor operators (ROs), who were to receive the additional trauung required to attain NRC senior reactor operator (SRO) licenses.

, The Nuclear Training Department (NTD) was requested to deliver this program startmg in June 1996, with a projected NRC exam date in the latter part of the fourth quarter 1996.

Using these criteria, (candidate background and available time) the Unit 1 Operator Trammg Branch (OTB),

revised the accredited Licensed Ope.ator Initial Training (LOIT) program to accommodate these parameters. This I is the first time a specific program for " upgrade" candidates was used at Unit 1. l In March 1996, Unit 1 Operations management informed NTD that in addition to the RO candidates previously )

mentioned, other candidates would be included in the program as well. These candidates were unlicensed and  !

would pursue SRO status as " Instant SROs" Individual trauung plans were developed for these candidates, however no significant modiEcation to the LOUT trauung program was made to accommodate the background and skills of these individuals.

The training program began in June 1996 and continued through the NRC exam in December 1996. From the start  !

of the program design through its completion, every individual with programmatic responsibilities was either removed or replaced due to job transfers or promotion. During the first week of the program one student withdrew, After the company final exam was given in November 1996, another candidate refused remediation and withdrew from the program ,

From November 19 through November 21,1996, the Unit 1 Operations Training Branch (OTB) and the line

organization conducted a review and validation of the NRC exam. As a result of the initial exam review approximately 70% of the exam was rewritten. On November 22, the exammation was revalidated by two Unit 1 SROs.

d On December 2,1996, the candidates took the written portion of the exam. At the three hour point, the hTC exammer authorized an additional 30 minutes to the original four hour time limit due to many candidates taking longer than anticipated. Later in the day, Unit 1 OTB conducted a prelimmary scoring of the written exam and estimated that only one candidate out of seven passed. The candidates were informed of these results.

, On December 6,1996, the candidates completed the operations portion of the exam.

At the NRC exit meeting on Dec. 6,1996, questions were raised regarding the competency of the licensed RO to l SRO candidates to perform certain RO tasks.

j The results of this exam are as follows (see Attachment VIII):

• One candidate passed both the written and operations exammations s e One candidate passed the written exam only

• One candidate passed the operations exam only Both the operations and written exams must be passed to obtain an SRO license. Only one candidate out of seven successfullypassedthe exam.

This Independent Review Team (IRT) was established to inw :tigate the circumstances surrounding these exam failures.

4 4

4

e J

EXECUTIVE

SUMMARY

(cont.)

Root Cause A lack of management oversight and accountability by management, supenision, and staff coupled with the ineffectiveness of the corrective action program to remedy previously identified, long standing issues led to

the degradation of training program quality.

4 Additional aspects of the root cause are discussed below.

Accountability and Resnonsibility Operations and Training management both missed numerous indications and warnings concenung the progress and quality of the LOUT program. This information was available prior to and during the conduct of the program.

This event could have been prevented had appropriate actions been taken to resolve long standing issues. Trauung staff and supervision contributed to this event by ineffectively applying procedures and accreditation criteria in the design and conduct of the program.

Breakdowns in the Levels of Defense in Ouality l Failures in the first three levels of the defense of quality were observed. Some ' procedures were not followed or were used ineff:ctively. Supervision did not provide adequate guidance to the mstructors and failed to act on observed problems. Management failed to provide in-depth oversight and follow through. The oversight organization (QA) failed to identify some of the precursor conditions during audits and surveillances. This event would have been prevented or mitigated had these lower levels of defense been effective. This event was stopped by the NRC exam, the 4th and last barrier in the defense ofquality 1 1

Ineffective Corrective Action Program '

While some aspects of this event were unique due to this being the first LOUT program for MP-1, many of the basic causal factors contributing to the unsatisfactory performance on this exammation were found to be similar to .

previously identified problems' across the three Millstone units. These unresolved weaknesses in operator trauung

l have been identified in numerous self-assessments, QA audits, INPO evaluations, and NRC inspections (see i

Attachment I) and should have been viewed as precursors to the current MP-1 event. The corrective action process has not been given priority nor used effectively to resolve previously identified weaknesses. Had these previously identified issues been remedied in a timely manner, the negative outcome of the LOUT class might have been prevented. Significant examples include

i e A self assessment process that does not systematically review past trammg problems to identify the continuing success of remedial actions.

• Corrective action plans not being developed for issues identified in previous self assessments.

• Issues identified at one unit not being reviewed for applicability at other units.

• Exam banks did not meet industry and regulator expectations.

• Incumbent task lists were not updated as required.

• Plant and industry events were not consistently included in trammg.

• Simulator training was not effectively observed by line and training management.

1 8

1989 MP 3 LORT failure; 1991 & 1992 MP 1 LORT failures; 1992 NRC NUREG-1220 inspection; 1994 MP-1 LOIT company exam failures; 1996 MP-2 LOIT company exam failures.

Fmal DraA Il February.1997 a 1996 MP.I LOUT NRC faanunaton Faihses IRT U

I l 1 ,

1 l

EXECUTIVE

SUMMARY

(cont.)

Lack of Mananement Ow.sht and Accountability 4

! In addition to the previously unresolved weaknesses in operator trauung, management oversight was ineffective in

} identifying and correcting cmerging challenges and deficiencies associated with the 1996 LOUT program Management was not cognizant of and did not respond adequately to several factors that contributed to this event.

. The magmtude of changes incorporated into the new LOUT program (e.g.,13 weeks of "on-shift" y

component reduced to 4 weeks,31 systems selected for self-study with no formal instruction).

. Deficiencies in student and program records regarding prerequisites, class attendance, on-the-job (OJT) j trauung time, "as taught" schedules, program change documentation (Adverse Condition Report - ACR-

{ Ml-97-0117 initiated by Operator Trauung supervision during IRT investigation).

. The cumulative effect of personnel changes on the experience level of the program staff.

3 1

i . Failure to follow procedures and the Systematic Approach to Trauung (SAT) 1 l - In developing the LOUT program -

j - To upgrade curriculum for plant design changes (PDCRs), procedure changes, etc.

1 - To update task lists and exam banks

?

t Experience And Staffine j The experience level of the key individuals associated with the LOUT program played a significant role in this

event. As a result of frequent personnel changes at all levels during the conduct of the program, the staff i experience level was diluted.

Systematic Annroach to Trainine (SAT) 3 j A lack of utilization and undmi.nding of the SAT led to errors in judgment in program design and implementation.

Deviations from or failure to follow established procedures contributed to a significant breakdown in the SAT j process.

Candidate Selection Process

]

j Due to an informal candidate selection process, with undefined selection criteria, some of the candidates selected for j the 1996 LOUT program may have been marginally motivated to obtain an NRC license. One candidate left the  ;

, program within the first week, another candidate refused remediation after failing the company final exam.

s Exam Review and Validation

The process to review and validate the exam was ineffective. This was evidenced by the need to challenge 10 questions after the exam was admmistered. Additionally, this process was not formalized or proceduralized. It depended on the experience of the staff, of which three out of four reviewers had not presiously resiewed an NRC exam.

FRC Written Exam The difEculty of the NRC written exam was evaluated by the IRT and an independent consultant (Nuclear Trauung and Educational Services, Inc. - Attwhment III). It was detemuned that the exam was lengthy and difficult but within NRC guidelines. In light of the weak performance on the simulator and JPM portions of the exam, the difficulty of the written portion was not viewed by the IRT as a signi6 cant factor.

Fen! Drat - Il Fahruary.1991 a-1996 MP-l 14X.fT NRC Eamusanam Fakses IRT E

EXECUTIVE

SUMMARY

(cont.)

Generic 1==!!e=*ians The focus of this investigation centered on Unit 1 Operations Trammg, and the circumstances surroundmg the LOUT program. While a review of the other programs was not conducted , evidence suggests that certam related problems may exist in trammg programs for the other units. Further in-depth review of these programs is warranted. For example, ACR M2-96-0657 was issued on Nov. 3,1996 for Unit 2 Operations Trammg failing to follow the SAT process. 'Ihe issues identified are similar to those found by this IRT on Unit 1. The prelinunary investigation of the MP-2 issues indiemtaA that other trammg programs are at risk or have simdar problems.

Sienificance and Conseauences Concern over th: ability of MP-1 Operations Trammg to adequately prepare individuals for NRC license exams may suggest other training related problems and lead to NRC attention and action. Problems concerning ,

compliance with the National Academy for Nuclear Trammg accreditation criteria can result in actions by the l accrediting board. 1 Recommendations Numerous remedial corrective actions are tw.. _.=-vi in this report to addr$s the root cause and various contributing factors. Among the most urgent of these are the following interim compensatory measures: l

1. Draw upon INPO loanees or industry experts to temporarily function in the MP-1 Operator Tram.ng I Branch until the experience level of the incumbents can be raised to an appropriate level. i i

2. Conduct in-depth reviews for applicability ofissues from this investigation to other trauung programs for I all Millstone Units. Consider temporary "stop-work" orders for those programs determmed to be unacceptable.

3. Evaluate competencies of the licensed RO to SRO emndid*s based on exammation results. Evaluate how I identified weaknesses of the candidates pertain to the rernmindar of the licensed staff.

4. Conduct reviews of previously identified problem arms and expedite close-out of those relating to ,

accreditation, regulations, and trammg quality. This should include re evaluating priorities and resource l commitments agamst other Nuclear Trammg Doartment excellence initiatives. )

5. Implement an effective corrective action program which includes accountability for closure in a timely and effective manner.

6. Senior management should clarify and reinforce expectations of Trainu.& nd Operations management regarding their responsibilities for oversight of trammg Implement methoo and measures to ensure that oversight is efrective.

I 1

l I

I Uw%S$ULear IV

TABLE OF CONTENTS Page Executive Summary i Table of Contents 1 1.0 Introduction 2 2.0 References 6 1

i 3.0 Individuals Involved 6 '

4.0 Analysis 7 l l

~

4.1 Management Oversight 9 4.2 Self Assessment and Corrective Action 11 i 4.? Systematic Approach to Training 15 l 4.4 Experience and Staffing 19 4.5 Candidate Selection Criteria 24 4.6 Training Records 26 4.7 Training Materials 27 4.8 NRC Exam, Review, and Validation 28 5.0 Event Consequences 30 6.0 Root Cause, Lessons Learned and Conclusions 31 7.0 Summary of Recommendations 34 8.0 Attachments 36 I Summary of Previously Identified Issues II References III NETS Review of NRC Examination IV Evaluation of LOUTIssues Against NRC & INPO Requirements V Analysis of NTD Procedures VI Barrier and Failure Mode Analysis VII Conclusion and Causal Factor to Recommendation Matrices VIII Previous Pass Rates MP-1 LOIT & 1996 Eram Results IX IRT Charter X Accredited Program Sequence l

996 M N Faihees LILT

i i

i

1.0 INTRODUCTION

An Independent Review Team (IRT), which included utility peers and indapandam consultants, was chartered to assess the collective events leadmg to and including the Senior Reactor Operator (SRO) exammation failures of six l

out of seven candidates (Attachment IX). The events reviewed included training program revision and design, ,

program implementation, exam review and validation, and the examination itself.

Analysis methods employed in this review include the Phoenix Handbook" from NSRC, and concepts from the '

Failure Prevention Incorporated (FPI) International root cause literature to supplement the company's Root Cause Procedure NGP 3.15. ,

i 1.1 RESPONSE TO THE EVENT Based on the prelimmary wTitten exam scores and NRC comments about simulator performance the Unit 1 Operator Training Branch (OTB) initiated ACR M1-96-1024 and ACR M1-96-1025 that documented weaknesses in both LOLTT written exams performance and LOUT simulator and JPM performance. Additionally, senior management commissioned the formation of an IRT to investigate and determine the root cause and casual factors l of this event.

I 1.2 POST EVENT On January 12,1997, the IRT convened at the Millstone Trauung Center. During the next two weeks, the team <

l conducted interviews and reviewed documentation relating to this event. The major post event findmgs, conclusions and recommendations are included in the body of this report and summarized in the Executive Summary. During the team investigation ACR M1-97-019 and ACR M1-97-Oll7 were initiated by Unit 1 OTB documenting weaknesses in the application of Systems Approach to Trammg (SAT), and inability to document completion of LOUT trammg.

1.3 COMPARATIVE TIMELINE A comparative time of the event and precursors is found on the following pages. This timelin: reviews the actual )'

occurrences of the event, an analysis of how and where the occurrences departed from what should have happened, and a discussion of the significance of each occurrence to the event as a whole.

l l

l l

1 5wYDOna m 2 l

l

. ~ _ - ~ - . . . . . -- - . . - - .- --.. - - - .- . .. -.-. .. . - - - -. . = -

1 1996 LOUT Program Failures -Timeline of' Events

@ O '88 9 )

mg mg w.I %+ n.,s M M W c-- ~

~

- ~ ;u .

.. th M hm happsed ~wet , . .

v. -

men ,a t-~ <

W g~ % >my <

,,g -

Pre 93 The Spitematic Approach to Trainmg (SAT) tweaks down. Anfierence to SAT should have been maintained. 11ms sets the stage for future problems.

(precursor) 93-94 Symptomatic fines applied to vmble SAT deficiencies. Operations Root cause to SAT proldems should have been identified and The illusion that "mit is well" was generated.

training programs re-accredited. actions taken. (precursor) 10/95 ASOT (Assistand Supervisor of Operator Trauung) accepts position m Extensive tumover should have occurred or ASOT position One barrier in the 2nd Icvel defense of quahty removed.  !

operations. No replacement is made due to reesanization. filled.

I/96 IIXJr program requested to be developed by lune organization.

1/96 New 1AXTT proyam coordmater aansgned. Enhancedm. 6y oversight ofinexpenenced coordmator. A barrier in the 1st level of defense of quakty is challeged by placmg e First tune ._ ._

an L,m.J individual in the position. Given the importance of a No significant tumover or oversight evident. ' 11XTT, . _/ should have recognued this potential weakness and-

^

t/96 IIXTT propam designed without reference to accredited program or Accreditation requirements and procedures should have been 1he first level in the defense of quehty failent 11ee basis for all NTD procedures followed. operator training resulting in a new or first time license was the Ixensed Operator Initial Training Program. (IDIT). This proyam had been reviewed by INPO and accredited by the National Academy Failing to consider the programs requirements severely damaged the revised lIXTr pmyam nliddy.

1/96-2/96 IIXIT program reviewed by Ops traniing supervisor and pisnt Acuedstation requirements and NTD procedures should have lhe second level in the defense of quality farled. Reviews by been recognized. supesvesson and the line should have revealed deficiencies in this area. 1 The inabihty to recognize these problems suggests weakness si line and trainina supervision understanding of the tramuns process.

1196-2/96 IIXTT program revision not renewed by Ops Trng Mgr. Program should have been reviewed. The second level in the defense of quality failed. Management review ,

ofmajor program changes is vital, t 2/96 Planned self-study- ,

^

not ', ,- Os candidates were not Program structure reviewed to ensure that original design 1he first and second levels in the defense ofquahey failed. Systems  !

known. No program modifications made. . -- remained valid. traemns represents a large percentage of the knowledge that naast be mastered by the candidate. The original plan to -- . ' some of this training by self-study before the start of fornini clamarcom could have been an efficient unethod to reduce adual classroom l ours. The training stafIshould have recognized the impact of adding a significant self-study component to an already diallenging classroom schedule.

3/96 New Operations Manager for Urut I appointed. Detailed orientation of traming.y lities. . The second level in the defense of quehty was challenged. Anewiv  !

I appointed snanager is often inundated with details and on thejob lem Plantandtreuung _

^ should have paid close attentian to those aspects relating to trauung to ensure expectations were being niet.

4/96 Operations Tranung Supervisor takes on sde position. Replaced by Enhanced management oversight of 4 id supervisor The second level in the defense of quehty was challenged. A new incumbeid senior instructor in " acting status 5 supervisor is allen inundated with details end on thejob leannng.

Training - . should have assisted the new supervisor to ensure espectations were beine niet.

6/96 Tranung Readuiess meetmg not held Meetmg between plant and training to review current status and This nieetmg would have been another opportumty to re-check determine readmess ofpmgram. assumptions and plans.

Feel rheit . II Feteuery.19e7 tve6 hip.I t.OUT NRC Enemmenon Feemes IRT 3

i 1996 LOUT Prograsu Failures -Tieneline of" Events (continued)  !

P4j Dussy sMs > i@%j94d% ,4y,?, jjgp pM9w:^ g,psMhaveW gggg s " gjp ,

,W xjjggsy > o.g.z.g.g g; cyy , - < n ,s i 4 m .

a .

~~ ., ,

x < .

6/96 Casuhdates are aresounced "only a few days" before start of program. SuWecient tune should have been available to conglete training As prevsously mentioned, not announnng cavdulates preveseed l Some candidstes are selected the day before program stest. equivalesicy assesunsuits.. , _. of the self. study - , "^ Additionally,thiswas i not enougli time for die training statto , , review die }

backpound and knowiecke level of these seest e==Iman p 6/96 One studest withdraws in first week of the prograrn De nietsvation and abehty of the condulates should have been License condulates represent a sigpufocant invesensent ofutility

===-t gerior to class stast. resources. " ""'- ",,the license hold o:cupies a posiG m of trust i with regard to maintaining the plant. O tie selection process should be rigorous enough to screen out those candidsles who are not strongly  !

motivated.

6/96 Plant ensnagesnent appeares ShsR Manager as" mentor

• to class. Mentor Frequess' and fruitful interaction with condulates and fee Riack De second level in the defense of speahly was ineffective. De only eneels with class once during the entire program. provided to plast and tranung. Mentor had the potential of being an valuable source of fecesch [

retarden the quality and conduct chhe pregam.

7/96 IIRJT Coordmater accepts ddierent position. Replaced by incumbent Coresnaty and coordmation of program. De first level un the defense of quahty was challengest A new i instructor employee is oRen inundated widi details and on the job learneig.

Training superviseen should have assessed See siew coordmater to ensure _ , were being met 7/96-8/96 Two caN attend GFE classes in parallel with classronm, resulting GFE trammg should be completed prior to starting lacerne Candidmees were started in die IJMTI program who had not smet their in niissed classrooat traenug. program. entrylevelpreresynntes. AdditionallyunreasoneMeespectations  !

placed on those condulates who were al endy coping with a self-study ,

and an angessive classroom schedule. I 7/96 Exam bank queshans provided to NRC do not nieet NRC guidehnes Iligh quahty exam bank provided to and u!ied by NRC. De i " , exam banit drove the NRC to author many of the test i forcing all epsesteens to be developed by NRC questsons instead ofusing the utility test bank. t 7/96 Candidates' fee &ack to trasneng that they do not have tune to read all Oiven the ningrntude of the changes in this progrant, close Osven the fact that this was the first time nt Unit I were a 1 .

self. study texts. Training acknowledges issue but is constramed by attention should have been paid to student comments. licensed class was given, strong cananderetum to the studeed comments schedule. should have been given. i 8/96 ASOT position filled but; Experienced ASOT available to provide oversight and gendance A second barrier me the defense of quehty was not available. .

• Amqpied to temporary project.

. Not SRO certified.  !

9/96 Operations Traming Manager removed, suo replacement snade. An . , " Ops Trng Mgr. appoeted and assipied. A second level in the es efesse ofquehty was removed. At the tune sie manager was removed, both the training supervisor and coorduister had been on thejob for only a few months.

W96 IJMTr coordniator reepseats additiosial resnurces to support pregam M _ assessment of risk in not provulms resnurces Another indication of problems in Sie prossess was sus aded on .

6/96 - Program status nieetings not held. Regular status smetags to review progress and adpust plan e Adddional opportumises for ensurmg the quahey of training were  ;

11/96 needed. t missed.

l1/96 Company written exam given;.

e Averagescore 86.3%

e I f'ailure 11/13/96 11Rrr Progrant Coorihmeter authors memo requesteg academic review Acadenne review board meets and appropr ate oversight given. De coordinator had reservations about one cand=Ima= readmess for i board for one candedste based on peut -.i - A board was not the exam. His was an appregnate action for the coordinator to take.

convened. k was also a formal action and should have warranted a formal response .

I t/96 Candedste who failed company exam withdrawn froen program citing De motivation and shihty of the candidates should have been license condulates represent a sipuficant invesensent of utshly personal reasons assessed prior to class start. resources. Addstsonally, the license holder occupies a position of trust with regard to nimintaining the plant. De selection process should be k ngerous crungle to screen out those candidates who are not strongly Issotivated. i r.d o a n r ,,im 1996 MP-I LOUT NItC r Faemes titi 4

- t 1996 LOUT Program Failures -Timeline of' Events (continued) 4What seemsey hoppassed ,.,f". ~ , @hmt shouMhave happeinsem ,  ;.qg gc, ;;;.y ezu; '. un 4 p h. v-g' g, e. y ,my

. L n yDelet; aw,.-

3 b.g: ~, en

. em , , 4.w _

4 .4 9 . n ,

IIII8/96 Practice feel exam given.

• Test scores similar to company final. 11ns test was presented as a practice exam test after the corricany

. Average score 78.7% e Management consideration ofresults exam. The results should have raised questions regerihng the e 3 failures. student's readmess for the NRC exam.

No action taken by training or Operations .._ m..-..". ,

II/19/96 Inexpenenced exam reviewers assigned. Expenenced ind viduals or appropriate oversight provided. The brst and second levels in the defense of quahty failed 1he eGects of this inexperience resulted in an ineffective review and validation of the NRC exam.

II/19/96 Exam review and validation occur concurrently. Exam review should have Ixen completed sufreciently in Combing the review and vahdation phases reduced the utilities alnhty advance of and :. 4.. ..; from validation. to evaluate the exam construction. Typically reviews are done in advance of the validation. This provides sufficient time for dialog and consensus between the NRC and utility.

11/19/96- Exam review results in rewntmg over 70% of exant Degree ofchanges requwed should have tnggered questionmg While exam revision is normal, the degree of revnion seen in this It/20/96 attitude over the suitability of the exam. exam seemed hirli.  ;

11/20/96 Exam revahdated toy non representative SROs. Exam takes 3.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> to Vahdaten should be performed by individuals representative of If more representative SROs had twen used, better data about the  ;

l complete. the expected knowledge and skill level of the candidates. Exam length of the exam would have been evident.

should take ,,,m-- di 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />.

12/2/96 Wnteen exant taken .

• Emant should have been able to be completed in 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />. A less than effective exarn review resulted in a challengmg but vahd ,

e NRC exansiner allows candidates additional halfhour at die 3 e 100 % pass rate. exant hour point.

e 6 out of 7 fail (, --- j results) 12/2/96 Candidates informed of r J... . y results. Prelumnary results should be released after candidates complete Informmg the candidates of prelunmary results may have distracted all portions of the exam. and de-motivated them 12/3/96 Sunulator and JPM portion of exam taken. NRC indicates weak 100 % pass rate. At the NRC exit meetmg, questions were rais.ed regarthng the performance.  % L.-y of the hcerned RO candidates to perform certain RO }

tasks. I 12/8/96 Urut i OTU initiated ACR MI-96-1024 and ACR Mi-96-1025 for weaknesses in ifMTT exam and simulator and JPM i,d .. .

12/13/96 letter sent to NRC contamung comments on 10 exam questions. An effective review and validstron would enr ail goal of 0 The number cf comments sent to the NRC reflects poorly on the challenges to NRC, quality of the exam review. The purpose of the exam review is to prevent any -

12M6- IRT commissioned. Scr ior Management recognition of potential seriousness of event.

t/97 1/15/97 ACR Ml-97e109 imtiated to desenhe weaknesses in SAT for thist I Ops traimng recosmtion of SAT issue.

Ops training  ;

I/17/97 ACR Ml-97-0117 iratiated desenbmg inabihty to document completion Ops treming recognition ei records issue.  ;

ofIJXTItraining.

I I

i l

l I

1 Fed Dren - Il Fearuary, r?97

+

tves MP-I IDUr ffRC Exammenen Fehses IRT l

l

. - - - _ . -._ ~, - .- - - . .

2.0 REFERENCES

Documents reviewed by this IRT are listed in Attachment II.

3.0 INDIVIDUALS INVOLVED 3.1 INDEPENDENT REVIEW TEAM MEMBERS e

K. C. Muccino (Team Leader)

e W. H. Barron e L. H. Dooley

. S. R. Frye e T. P. Gilbert

• C. W. Sawyer 3.2 PERSONS INTERVIEWED OR CONTACTED Name Department / Position SRO Uceme Can&date

] SRO Ucense Can&date

, MP 1 Operator Trauung Instructor MP-1 Assistant Operabons Supemsor SRO Ucense Can&date MP 1 Shift Manager SRO Ucense Can&date Previous Dir. Nuc. Trauung Mar, Eng Support & Prof Devel (hTD)

MP.1 QAS (Contractor)

MP 1 Operator Trauung Instructor (Contractor)

Director, Nuclear Tramuna Prenous MP-1 Manager Operator Tr-me MP ! D: rector, Uvut Operauons Currera LOUT Program Coordinator SRO License Can&date Performance Development Instructor MP-1 Assistant Operator Trammg Supemsor Previous LOLT Program Coordmator Nuclear Trauung Sernees MP 1 Supee.isor, Operator Traming MP 1 Operatons Manager Previous MP-1 Opermuons Manager i MP-1 Operator Trauung Instructor MP-1 Operator Trauung Instructor Manaaer. Operator Trauung

, MP 1 O wrator Trainme Instructor (Contractor)

V' , astant Operauons Manager b ,:ense Can&date SRO Ucense Can&date Mgr. QAS Asse= aments

' NETS

• Consultant - NRC E.xammation Renew 1

Fmal Dndt - 11 February.1997 land MP 1 LOUT NILC s- Fatsus IIt?

l i

4

l l

l 4.0 ANALYSIS His event is the result of a combination of factors that: ,

e Made the situation vulnerable and setup the event l

• Triggered the event

• Aggravated the event already in motion  !

e Mitigated the event i' This section examines the above four factors, followed by a dciled evaluation and discussion of the major issues identified.

l 4.0.1 EVENT SETUP (What allowed the event to happen). l The Unit 1 Operator Trauung Branch (OTB) madequately redesigned the Licensed Operator Initial Trauung  ;

(LOIT) program to meet Unit 1 Operations management nerds for additional SRO licensed personnel. The l following factors set the utility up for the event. 1 i

4.0.1.A Prev lous Weaknesses in tite Systems Approach to Training (SAT):. The IRT deterrmned through 1 interviews with vadous members of NTD that the SAT process had been breakmg down for a period of time. His J was manifested by not following established procedures and misundere'=ading the concepts of SAT.

i 4.0.1.B OrganisationalPressures: The original candidates proposed by the plant in early 1996 were currently l licensed ROs pursuing upgrades to SROs. He LOUT program redesign was based to a large extent on the time the candidates would available for trammg He trauung staff was given a start date by the plant and evpa*ad to develop a program that would fit within this available time. This method contrasts sharply with a synunatic 1 approach, that considers first what must be taught, the entry skills of the candidates, and then da'arminac arp~4ad l course length.

4.0.1.C. InexperiencedProgram Designer (LOUTCoordinator): The coordmatar had not previously participated in a major program design or modification., He was unaware of the requirements of the accredited LOIT program, j and did not use available procedures when modifying the accredited program j 4.0.1.D Selection Criteria - Entry Level Assumptions: The original proposed candidates consisted of currently licensed ROs pursuing upgrades to SROs. He program modifications considered certam entry level knowledge and skills. As the course grew nearer to startmg, mstant SRO candidates were added Neither NTD staff or 1 supervision questioned the appropriateness of combining these two distinctly different audiences in an already l significantly modified program. Additionally, the original. revised program included a four month self-study component. The intent was for the candidates to self-study 31 selected systems and to receive periodic tests on the topics studied. These systems were selected based on the probability of not being on the NRC exam. This self-study component was included to a-q+==te for the relatively short time available for formal trammg However, due to delays in the selection process, some candidatas were not announced until May 1996, resulting in the self-study component not being implemented as planned. This change to the program design was not considered by NTD staff to be significant.

ImYSYir n iar 7

4.0.LE. Ineffective Management Oversight, involvement and Accountability: Prior to the start of the class, numerous opportunities for management intervention were missed. Management (plant and training) was more focused on the larger issues facing the Millstone site than on specifics. This coupled with an overconfidence in the subordinates' skills and abilities led to a " hands-off" approach to program design by trammg and line management.

This lack of oversight resulted in several non conservative decisions on the part of the trammg staff including l reducing the on-shift component from 13 weeks to 4 weeks, and requiring some candidates to attend both General  !

Fundamentals Exam (GFE) training and LOUT at the same time.

1 4.0.2 EVENT TRIGGER (What set the event in motion). l The start of the LOUT program in June of 1996 was the trigger for the event. Up until this point, reconsideration I of all decisions and assumptions regarding the program were possible without significant wasted resources. After the program started, with candidates and staff assigned, the simulator schedule and test date established, the l difficulty in effecting major programmatic changes was increased.

4.0.3 AGGRAVATING FACTORS (What made the event worse than it could have been).

4.0.3.A Self-Study: Some systems were selected to be presented by self-study instead ofin classroom. The bulk of this self-study was to be done prior to the start of formal trauung. However this self-study material was not given to the candidates until the start of the class. This significantly increased the burden on the candidates especially when considering the already shortened program. 1 4.0.3.B GFE Attendance: Two candidates had not completed the NRC admmistered GFE prior to the start of the program. These candidates were required to attend selected portions of GFE concurrently with LOUT attendance, again increasing the individuals burden. Neither the candidates, mstructor or training supervisor questioned the appropriateness of this and the self-study expectations. This resulted, in some cases, of candidates missing LOUT training to attend GFE trammg.

4.0.3.C IneffectiveManagementInvolvement: Numerous examples ofineffective involvement were evident during the conduct of the program. The Shift Manager Mentor, assigned by the operations department, met only once with the candidates during the entire program. This valuable source of feedback to management, and resource to the candidates was missing. Additionally, management failed to recognize and or act on several indicators from both the candidates and coordinator of potential trouble. For example, the coordinator requested an academic rew hoard for one student based on the readiness of the candidate to sit for the NRC exam. This board was not cce.ened.

4.0.3.D Resource Issues: In some cases trammg material was not updated as a result of procedure and plant modifications. At times, test questions were given on plant changes that were not referenced in the training materials. Additionally, on-shift time with the exception of prescribing certain OJT cards to complete was unstmetured. While the need for additional mstructional assistance was requested by the coordinator, no action was taken by management.

4.0.3.E Exam Review and Validation: The overall experience of validators was low with 3 of 4 having never validated an NRC exam prior to this exam. This process is very dependent on staff experience to provide an adequate review and validation. After review and validation, questions about the length, scope, and difficulty of the exam remamed, but the staff was uncertam of what actions could be taken to address the situation. Additionally, the review and validation took place concurrently during the same week thereby reducing the effectiveness of the process. After the exam,10 exam questions ( 10% of exam) were challenged by the unit. This indicates that the review of exam questions was incomplete and that the validators may not have been representative of the candidate population.

Fmal Drd 11 February,1997 1996 MP.1 LCOT NkC IJumananan Failures !RT g

l 1,

j 4.0.4 MITIGATING FACTORS (What prevented the event from W'ning worse).

4

4.0.4.A Management Actions

NTD and Operations management realized the severity of this event by the i completion of the NRC exam week. ACR's were written, and the IRT was commissioned. Addihal ACR's were

] written as necessary during the course of the IRT.

4.0.5 ISSUES In reviewing the casual factors and elements of this event, the team identified the fellowing eight issues. These issues are discussed in detail in the remainder of this section of the report.

• Management Oversight, Involvement and Accountability e Self Assessment and Corrective Action e Systematic Approach to Training

. Experience and Staffing

. Candidate Selection Criteria e Trauung Records

~

e Trauung Materials e NRC Exam, Review, and Validation 4.1 MANAGEMENT OVERSIGHT, INVOLVEMENT AND ACCOUNTABILITY 4.1.1 Management oversight of the 1996 LOUT program was ineffective in identifymg and correctmg challenges and deficiencies in presenting quality traming Throughout the condact of the program, decisions were made and actions taken without consultation or review by management. Procedures and policies relatmg to oversight activities were not effective. For example:

• Corrective Actions for previously identified issues were not implemented.

• Trammg management did not reahze the inexperience level of the ' original LOUT program coordmator. This.. I coordinator did not utilize either procedures or the accredited trammg requirements in the design of the LOUT program These omissions were not re-ni-A by ====p=

. Operations Trammg management was not cogmzant of the magnitude of changes made to the accredited program For example the Operator Training Manager was under the impression that "on-shift" a m-=t was reduced to 8 weeks, not the 4 weeks actually presented during trammg e Required assessments of trammg by Operations and Operations Trammg management were not conducted. )

e The Operations Department Mentor (Shift Manager) ceAM only one observation of trammg during the entire program Other than this one observation the mentor had no association with the class.

. The Operations Liaison position was open throughout the time of the class. As a result trammg material and exams were not reviewed by a plant subject matter expert.

• Senior management failed to recogmze the cumulative effect of personnel changes of the individuals associated ;

with this program Every position that held programmatic responsibilities for LOUT was replaced or i reassigned during the time frame of the program J e- Indications that progrewaatic difficulties were being encountered was provided by both the candidates and the second LOUT coordinator. No apparent action wm taken based on this feedback.

• Management was not effective in communicating and following up its expectation that procedural compliance and adherence to SAT were a high priority. Instead, many staff have interpreted recent management changes and comments to imply that the first priority of training is to meet the customer's needs FmalDrne . Ilfstruary.1997 1996 MP4 LOUTNR.C 8 ==-a-iFeihees IRT g

. - . . - . -. -- . _ __ - .~ - - - . - - _ . - .. . - . - - . ..

4 g

. The LOUT program coordinator formally requested an memdamic review board for one student based on that

, candidates documented performance. This academic resiew d'd not occur.

Training supervision and management were unaware that deficiencies were evident in student trauung records until notified by the IRT.

Issues relating to the ineffective management oversight have been identised in numerous other inspections, audits and assessments. (Refer to Attachment I)

Examples of previous reports indicating weaknesses in Management Oversight

• ACR 11037 Adverse Trend in Unit 1 Simulator Evaluations Root Cause Analysis (October 1996)

I noted that a lack of ownership by both the Shift Managers and Trauung Mentors contributed to degraded crew performance.

j e 1995/1996 MP-2 LOIT Class Comoanv Final Examination Root Cause Evaluatinn (September 1996) indicated insufficient oversight by Unit 2 trauung management was a casual factor to the event.

• OAS Audit Reoort #A21071 " Unit Staff Trainino"(April 1995) noted that managers and supervisors were not conducting trammg observations as required by procedures. -

4.1.2 Concimions

4.1.2.A Management oversight of the 1996 LOUT program was ineffective in identifying and correcting challenges and de6ciencies in the presenting o r oaality trauung Management missed numerous opportunities to ensure that tran mg delivery was of the highest quality.

4.1.2.B Management failed to consider previously identified oversight issues (across all units) as " warning flags" and adjust its actions appropriately.

4.1.3 Recommendations

4.1.3.A Review industry guidance and INPO identified "best perfonners" with respect to management oversight. Adopt best practices into oversight process.

4.1.3.B Reinforce with Trammg and Operations management their responsibilities regarding oversight of trauung 4.1.3.C Establish high expectations regarding oversight and implement methods to ensure that oversight is documented, frequent, and useful.

4.1.3.D Ensure that oversight feedback is communicated and addressed consistently across all units.

4 N f& N

. . ~ - - . - - _ _ _ _ - . . - - - - - . - - -..-- ...- - -.-.-

l l 4.2 SELF ASSESSMENT AND CORRECTIVE ACTION PROGRAM

4.2.1 There were numerous previously identified, long-=tanding trammg issues and problems. These areas were identified in NRC inspections, INPO evaluations, QA audits and internal self assessments The reports evaluated 3

covered trauung programs at all Millstone units. In many cases, these identtSed issues 'were the same as those i issues identified as casual factors for the LOUT exam failures. In short, if these issues had been acted on in a timely manner the negative outcome of the LOUT class might have been prevented. Attachment I contains a 1 summary of trammg issues from 14 past inspections, audits and assessments. A review of these items indicates that l tra nmg management had sufficient warmng that trammg programs were degrading. I i

The following examples contain illustrative comments from past evaluations, inspections and assessments.

Management 1996 Program Coordinator Assessment of Trainina Prc-roms indicated that attendance

- Oversight problems at trammg has been tolerated by both Trauung and Line Management.

, -l SystematicApproach NRC Insocction 93-80 indicated that TPCC committee size limits the committees' l to Training ability to provide checks and balances among all aspects of the trammg program l i

Experience and The Team Asssist of 1995 Nuclear Trainino Deoartment Activities indicated that Staffng _ instruction is skewed to the instructor's expertise, resultirig in incomplete leammg.

Training Records Surveillance A-324 " Verification of LORT Trainino Attandance (March 1992) j indicated that attendance records were found to be of poor quality.

Selection Criteria 1996 Procram Coordmators As=>ument ofTraininn Pronrams indicates that the Exam l validation process is weak due reputedly due to resource limitations. Several program l l coordinators believe that their exams do not disenmmate between competent and non-l competent personnel. 1 4

Quality ofTraining Team Assessment of 1995 Nuclear Trainino Denartment Activities indicated that  !

l Materials trauung focuses on passing tests, not on operating the plant.

) l i

j 4.2.1.1 Effectiveness of self assessment in generating and ensuring change NTD utilizes a multi-level self assessment process that considers many elements of trammg In some cases these

)

, self assessments were in-depth and self critical. Unit 1 Operator Trammg Branch (OTB) conducted a program by 1 program assessment in late 1995. 'Ihis assessment noted the following weaknesses:

l

• The self assessment process did not systematically review past trammg problems to identify the continuing success of remedial actions.

l i e The assessments included line management only as interviewees, not participants.

e No corrective action plan was developed for those issues identified in the self assessment.

. Unit 1 OTB was unable to provide any examples of how student feedback or supervisor assessments recommended improvements in trammg The lack of such feedback indicates a potential weakness in the

evaluation phase of trauung This was not noted as an area in need ofimprovement j e Unit 1 OTB was unable to offer any examples of how continuing trauung and plant upgrade topics were

included in the LOUT program, despite this being a clear commitment to provide such trairdng to those candidates holding RO licenses.

FmalDran 11 February.1997 19e6 MP.I Lot # NP.C Emanunsean Fedmus litT 3g l

4 e ,

e Unit 1 OTB offered only r'ummal feedback in Objective 3 " Development and Qualification of Staff." ne staff did not effectively evaluate this objective and failed to identify the lowering of group experience within Unit 1 OTB.-

i e

l Issues identified at one unit are not always reviewed for applicability at other units. For example Adverse  ;

Condition Report (ACR) M2-96-0657 was issued for Unit 2 Operations trauung failing to follow the SAT

process on 11/3/96. Unit I did not issue a similar ACR until the IRT review. Rose items listed in this later 1 ACR were in existence at the time of the Unit 2 ACR.

! e items identified in self assessment are not always assigned appropriate corrective action and tracked to completion. This issue was also noted in ACR M1-%-0339.

l 4.2.1.2 Urgency in resolving corrective actions The team did not sense any significant urgency on the part of trammg or line management in resolving and closing

out training deficiencies. For example,.

e i Task list for Unit I licensed operators has not been reviewed and updated since-1993. By NTD procedure this 4

action is required every two years. Despite being overdue for more than one year, this review is not scheduled j until August of 1997.

l ACR M1-96-0339 identified training issues in need ofimprovement that were not assigned concrete corrective

action to prevent recurrence.

i l

4.2.1.4 'INPO comments regarding self assessment On a similar note, the 1996 INPO E&A visit generated the following comments regarding self assessment and corrective actions

The long maing nature of simulator instructor performance problems was not identified by self assessment.

• Corrective actions for simulator instructor performance problems were not effective.

. Self assessment did not achieve desired results.

• Ongoing self assessments are insufficiently broad in scope to identify trauung weaknesses.

• Ongoing self assessments did not include industry experience.

• Self assessments did not include review of station problems to identify potential worker performance issues.

. Trammg self assessments did not review weaknesses identified from previous inspections and assessments.

4.2.2 Cdoclusions

4.2.2.A ne Self Assessment and Corrective Action process has not been used effectively to identify and correct weaknesses in trammg.

4.2.2.B The Self Assessment and Corrective Action process does not systematically assess past problems to ensure the adequacy and sustamability of past corrective actions.

4.2.2.C He Self Assessment and Corrective Action process in many cases focuses on a 4 single plant and does not consider if problems identified in one unit are present elsewhere.

1 4.2.2.D Some assessments in the past, including the accreditation self evaluation report, did not include line management ownership.

ru o,.a . n rum. w ,*

1996 MP-1 LOUT NRC Enemmenon vaihees IRT

4.2.3 Recommendations

4.2.3.A Review industry guidance and INPO identified "best performers" with respect to self-assessment and corrective action. Adopt best practices in the NTD self assessment and corrective action process.

4.2.3.B Conduct an ' urgency review' of previously identified problem areas and expedite close out of those relating to accreditation and / or regulatory issues.

4.2.3.C Ensure corrective actions are developed and tracked for all identified trammg issues 4.2.3.D Review effectiveness of past corrective actions. Incorporate this review into the self assessment process.

4.2.3.E Develop, communicate and reinforce clear expectations regarding the Self ' l Assessment and Corrective Action process for all levels of management, supervision and staff.

1 l

l Fmal Draft .1L February 1997 19M MP 1 LOUTN1u:: r Feihres Illt g3

Previous Inspections, Audits and Assessments Centaining issues identified in the LOUT 1RT Investigation .

Document Title Management Self Assessmentt - Systematic Experience and Selection Criterla Training Records QuaNty of NRC Exam, Oversigid Corrective Action Approach to -  ;.StafHng Training Materials Reviewand :

Training VaNdation ACR 11037 Root Cause Analysis y y y y MP-l Simulator Evaluations (Oct.1996) 1995d996 LOIT Class Company Final y y y y Examination Root Cause Evaluation Survemance A-324" Verification of LORT y y Attendance" (March 1992)

NRC Inspection 93-80 (all units) X X X X Audit A60516" Operations Activities for y y MP-3* (March 1993)

Audit A21062 *MP-1 Unit Staff Training" y y y (April 1993)

NRC Root Cause Analysis of MP-1 LORT y y y 1991 Unsatisfactory Program Determination NRC Inspection 9605 NU Assessment of y y y NRC issues ACR M2-96-0657 Failure to apply SAT at X X X MP-2 (Nov.1996)

Self Assessment of MP-1 LOIT X X X X (Jan.1996)

Self Assessment of MP-1 LORT X X X X (Jan.1996)

Temn Assessme.J of 1995 NTD ActMhes y X y y i

1996 Fi Program Coordinators y y y y y y y Assessment of Training Programs 1996 INPO ESA Visit y y X y y FeelDrst - tI Febenary,1997 1996 MP-I LEUT NRC Exemmenon Failures IRT y4

=

1 4.3 SYSTEMATIC APPROACH TO TRAINING (SAT)

4.3.1 A lack of the effective utihzation and understandmg of the SAT contributed to this event. Evident throughout the course of the program are errors in judgment and deviations from procedures with regard to the Systematic Approach to trammg.

2 4 Federal regulation and industry guidelines

• require that licensees utilize a systems approach to trauung that j incorporates the following 5 elements:

e Analysis of thejobs to be performed

, e Design of the training program

.

• Development of trammg materials i

i e Implementation of trauung e Evaluation and revision based on student performance The Nuclear Trammg Department should implement the SAT through effective mstructor trammg, policies and trammg procedures. The team identified the following occurrences where SAT concepts, policies or procedures j were not used in the design and conduct of the 1996 LOUT program I o The modification of the LOUT program was driven by available time in which to conduct the trammg NTD i staff revised and reduced trammg mycrents in order to present the program within the window established by l the plant. Under the SAT, trammg time should pnmarily be driven by the amount of time required for the candidates to obtain competency, not an arbitrary time-line.

l

NTD procedures provide detailed guidance to staff, supervision and management in the process and steps l required to satisfactorily modify trammg programs in compliance with the SAT. (Attachment V) Inteniews with the trair.ing coordinator and supervisor associated with the revision indicated that these procedures were i not used.

i

! e Paa of the reduction in course time included shortenmg the on-shift component from 13 weeks to 4 weeks l Lis action was taken even though the National Academy of Nuclear Trauung accredited trammg program  !

l required 13 weeks (see Attachment X), When questioned about this resision, the LOUT coordmator stated that j he was unaware that the accredited program had a 13 week requirement Additionally there is no evidence that a systematic approach was used to justify this change. While elements of an accredited trammg program q should be revised based on changing circumstances, careful consideration should be given in ensuring that the 4

expected outcomes of the revision meet the intent of the original program

) e ne LOUT program was origmally designed to include a four month self-study component. The intent was for candid =*~ to self-study selected systems and to receive periodic tests on the topics studied. His self-study component was included to compensate for the relatively short time available for formal trammg However, due to delays in the selection process, some candidates were not announced until May of 1996 resulting in the self-study component not being implemented as planned Instead, the self-study materials were given to candidates to study concurrently with the scheduled formal trauung This represented a significant change to the revised course design. Adhering to SAT principles would have resulted in a re-analysis of the traming program design to ensure that appropriate time and focus were given to the objectives and material affected.

• In addition to the self-study matenal mentioned above, two candidates were expected to study for the NRC

{

adnumstered General Fundamentals Exam (GFE) concurrent with formal trammg phase. This resulted in candidates missing LOUT trammg to attend GFE trammg.

2 10 CFR 55.2

• ACAD # 91015 Fmal Draft . Ii February,1997 1996 MP.I LOUT NILC Eaammatan FeGurus RT 1C a7

.- _~ . - - .- .- - . . . - _ - - - . - - _ . - . - _ - . . - - - - - . . - - - --

j l

1

• Some trauung materials used in the self-study program did not reflect actual plant changes. (LNP module, Instrumentation)

. The program re-design was based on candidates who currently held RO licenses and were pursuing an upgrade to SRO. In March,1996, after the program was revised, Trauung was notified that additional l candidates would be added These candidates were from the Unit 1 STA position and had different entry level skills and experience than the RO's. No significant modification to the program content was made to accommodate these new candidates. The following candidate comments were offered at the end of the program i

" Training incorrectly assumed that the applicants had a solid systen:, knowledge base "

"The Instants cannot be expected to know " expected operator actions" Gsat are notprocedurally governed.

"The assumption that the candidates hada LOITlevel ofknowledge at the beginning ofthe course was apoor one" (Ifelt I was playing catch upfor several reasons) "First the presumption that my level of l knowledge was at the licensed operator level. While my knowledge was good. Iwas not at a Unit l

~

1 operator level at the start ofthe class. " j 1

Evidence of weaknesses in the effective utilization and undereAng of the SAT is evident elsewhere in operator )

trauung e Interviews with traming staff and supervision indicate a weak understandmg of the SAT.  ;

• The Unit I licensed operator task list was last reviewed in 1993. NTD procedures require these task lists be reviewed every two years.

. The process in place (Taskmaster) by which plant design changes and procedure revisions are tracked and reviewed for trammg implication for Unit's 1 and 2 is not being used. A Unit 1 operator instructor stated he has no method to ensure that all applicable plant and procedure changes have been reviewed and are included in trauung e ACR M2-96-0657 was written in 11/13/96 for "Failurs to apply the SAT at Unit 2" e Operations Trauung Manager mtervention (third level batrier) was required to prevent violation of trammg procedures at Connecticut Yankee in the fall of 1996.

. The 1996 self assessment of compliance with National Academy of Nuclear Trauung Accreditation criteria by program coordinators identified that many elements of the SAT are not in compliance.

• A review of past inspections, audits and assessments indicates that numerous issues regarding the implementation of the SAT have been previously identi6ed. (Attachment I).

NOTE : During the IRT investigation, Unit 1 Operations Trauung supervision submitted ACR M1-97-0109.

This ACR documented a breakdown in the SAT for Unit I trammg activities.

4.3.2

Conclusions:

4.3.2.A The original accredited program was revised without sufficient attention placed on accreditation or regulatory commitments.

4.3.2.B NTD procedures were not used to ensure a systematic process was followed in redesigmng the program.

INiS$ECr.a-. mr 16

O e i

i

. 4.3.2.C %e analysis and needs assessment for program redesign was not dew and

therefore cannot be ve.rified.

4.3.2.D The NTD staffinvolved with design and revision were unaware of accreditation conumtments

4.3.2.E Evidence of weakness in understanding mi applying the SAT is evident elsewhere in the NTD.

{ 4.3.3 Recommendations: ,

, 4.3.3.A Evaluate existing NTD procedures and instructions (OTBI's) and ensure these procedures meet NTD expectations for the trammg process.

4.3.3.B- Revise NTD procedures and instructions (OTBI's) as required.

4.3.3.C Management and supervision should ensure compliance wi,th existing procedures.  ;

4.3.3.D Evaluate the as taught LOUT program against accreditation standards and ensure  ;

that all accreditation commitments were met. l 4.3.3.E Develop and implement corrective actions for the issues identified in the recently performed internal assessment of accreditation criteria.

i 4.3.3.F Implement the previously existing plan for reviewing the Systems Approach to Training as part of the 1997 Continuing Instructor Trammg program.

4 i

! l l

i i'

rw om . n r.% im im wi.i tour une s r.ih rat 17 i

4 -M-t

Previous inspections, Audits and Evaluations Which Identified issues With the Systems Approach to Training Document Title Analysis Design Development Implementation Evaluation ACR I1037 Root Cause Analysis X X MP-1 Simulator Evaluations (Oct.19%)

1995/1996 LOIT Class Company Final 6xamination Root Cause X Evaluation NRC Inspection 93-80 (all units) X X X X Surveillance A-324 " Verification of LORT Attendance" X (March 1992)

Audit A60516 " Operations Activities for MP-3" (March 1993) X X Audit A21062 "MPI Unit StaffTraining" (April 1993) X X NRC Root Cause Analysis of MP-1 LORT 1991 Unsatisfactory X X Program Determination NRC Inspection %-05 NU Assessment of NRC issues X X ACR M2-96-0657 Failure to apply SAT at Unit 2 (Nov.1996) '

X X Scif Assessment of MP-1 LOIT (Jan.1996) X X Self Assessment of MP-1 LORT (Jan.1996) X X Team Assessment of 1995 NTD Activities X . X l

1996 NTD Pregram Coordinators Assessment ofTraining X X X X X Programs 1996 INPO E&A Visit X X X Im"% LEU"" re mr 18 .

'.4 EXPERIENCE AND STAFFING. >

'Ibe experience level of the key individuals ===tM with the LOUT program played a significant role in this event. As a resuh of frequent personnel changes at all levels during the conduct of the program, staff experience was diluted and important historical contexts were lost. Many of the decisions made and actions taken can be directly related to this lack of experience. In addition to experience, overall staffing levels of Unit 1 Operator Traimng was also a factor in this eveet.

4.4.1.1 Organiutional Changes Organizational changes prior to and during the LOUT program Dune Dancrtpeem etChange Oingehig R ;'- --- - -

Innsehent 1093 ASOT accepts possuon m operauons. No replacement is made None due to re<rganizatiert I/96 New LOL7 program ocordmaior assigned None 2/96 One mland of. N/A

~

396 New Operabans Manager for Unrt I appooned . .

4 96 Trauur:s Director transferred to Oversight Group '.

• 696 Ops Trauung Liaison starts LOLTr No re;/w. ./ made 696 Operations Mentor appomtad (Collateral Stuft Mgr. dutv) a 496 Operations Tranung Supervisor take on sne posruert t

~

7/96 LOL7 Coordmator accepts ddierunt posamort l

896 AsOT postuon filled but Assigned to temporsrv project s96 j 1 Instructor assigned to plant procedures project 9.96 2 Contractors hired N/A 9 96 Operuuens Trauung Manager removed. no replacernent made '

1096 One Instructor terminated N/A 11/96 Director of Umt I replaced. ,

Key individuals with programmatic responsibilities to the LOUT program were removed, transferred or promoted during the term of the program.

Stemna Changes in Nuclear Training During LOUT Position- Held by: Replacedhv: Date:

V.P. Eng A. Trng . 10/96 Director Nuclear Trauung  ; 4/96 Manager Ops Trauung Not Filled 10/%

Super. Ops Trauung 4/96 Asst. Super. Ops Tranung Vacant s/96 LOUT Coorthnata 7/96 Pea 1 Duma . Il Pshnury.19P7 ,n itse IO 114UT NkC F-Pshees RT 37

. . - - . - - . _ . ~ - . - - - - - ~ . - . . . . - . . - ~ -

Stah t Chamaes in Unit i Durier IDUT Pata~en: hem by: Re=leted by: Date:

Recovery OfLeer 10/96 Umt Ops Duector . I1/96 Ops Manager 3/96 Asst. Ops Manager 4/96 Ops 12anaan '

Not Filled 6/96 4.4.1.2 Unit 1 OTB Experience Summary.

Nine instructors started in Unit I trauung over the period from Dac=har 1992 through April 1994. This staffing meresse was driven by the poor performance seen in Unit 1 Operator trauung in 19~91 and 1992. At the time these -

individuals were hired, an experienced cadre of managers, supervisors and instructors was in place. In the recent i.ast, the majority of these expenenced individuals have been promoted or transferred out of Unit 1 OTB. As indicated below, while overall trammg experience of the mstructors is adequate, there is little actual Unit 1 or commercial epergiig experience. Additionally, both supervisors had less than 1 year on the job during the LOUT program Unit 1 Operations Trammg Branch (OTB) Experience Summary (12/31/%)

1. ofSupervisors 2 Average # years onjob as Ops Trng supervisor .75

1. of Supervisors with Unit 1 operations experience 0

1. of Supervisors with commercial operations experience elsewhere 0

1. ofInstructors 9 Average # years onjob as Ops insuuctor 3.3

1. ofInstructors with Unit 1 operations experience 1

1. ofInstructors with commercial operations experience elsewhere i J of Contractors temporarily filling approved vacancies 3

1. of approved vacancies not filled 2 22 ""sf - - 20

4.4.1.3 Stafling Levels and History e As of Dec. 31,1996, Unit 1 OTB Authorized Staffing totaled 17; 2 supervisory positions and 15 instructors.

o As of Dec. 31,1996, Unit 1 OTB Actual Staffing totaled 15; 2 supervisory positions,9 utility instructors and 3 contractor instructors (one instructor is temporarily assigned to an on-site procedures project). As of 1/1/97 there were 5 vacancies posted.

The chart below displays Unit 1 OTB authorized staffing, qualified instructors and ac:ual complement through the period of the LOUT class.

3,,, Unit 1 OTB Staffing History Jan. . Dec.1996 18

1. Leaving M# Hired * *

• Authorized + Qualified Instructors ""*"" Total Staff p...a...g r~

16 e a

n... . . . .e . . . a . . 4 . . . ., . . 4 . . 4 . . . . g 12 8

6 4

0 Jan Feb. Mar Apr May Jun Jul Aug Sep Oct Nov Dec j

Unit 1 OTB staff was in flux immediately prior to and throughout the LOUT program. The net result of these changes was to weaken the depth and breadth of knowledge of the entire staff. Some of the significant facts that can be related to inexperience are listed below:

o The LOUT coordinator stated that he was unaware that the accredited program had a 13 week OJT requirement.

o The LOUT Coordinator did not utilize procedures effectively to drive program changes.

o The newly appointed Operations Trammg supervisor did not recognize that the SAT was not being followed during program design and revision.

Faul Draft Il Fetmwy.1997 1996 MP-! LOLT NILC Exammanon Feahres IRT 21

4 The Operations Manager and Operations Trauung Supervisor did not recognize that the program revision was

not in compliance with the SAT and trauung procedures.

e The Shift Manager assigned as class mentor had no previous experience in this role.

! e ne Operations Trauung Supervisor failed to respond to warmng signals given to him by staff and candidates

} regarding problems associated with the program Key individuals involved with the NRC exam review had only muumal experience with the process, e

he importance of accurate record keeping was not understood by the staff. This resulted in the inability to document completion of all required trammg by the LOUT candidates.

4.4.1.4 Changes in Management Philosophy Frequent changes in Nuclear Orani=== Management have had a disruptive affect on the Nuclear Or==ni=en as a whole and trauung in particular. Associated with these personnel changes were changes ranging fiom corporate visions to actual classroom practices. This constantly changing perspective has resulted in confusion at the lower levels of the orgaruzation about exactly what is and is not important.

The impact of these changes is two fold. First, it has appeared to confuse the staffin assessing and establishing priorities. Secondly, these changes drew staff resources and attention aw3y from the fhadamentals basics of trauung; particularly the trauung and qualification process standards set by the industry and the regulator.

Following are examples ofinitiatives that were recently suspended, e Confidence Weighted Testing . Inter Leafing

. Standardized training material development

• MentalVisualization andImagery

. Self Directed Work Teams De number and timing of personnel changes coupled with inadequate turnover and inadequate oversight led to a breakdown in program continuity and quality. Coincident with the decline in experience was the decrease in the number of available quali6ed instructors. While instruction was maintained on schedule, it was at the expense of other important training activities. Rese activities included mamtenance of trauung materials, upgrade of exam banks and timely review and revision of task lists.

Other aspects relating to experience and staffing are manifested in actual classroom and simulator tramme The -

staff and instructors involved with the conduw of the LOUT program had no actual operating plant experience. i This was remoni=d by both candidates and staff. The following comments come from LOUT candidate feedback and coordinator correspondenec:

"Having current Unit Supervisors coaching in the simulator would be ofbenept" "The LOUT class requested and strongly urged the Operations Department to i provide training support with experienced SROs" '

"The level oftraining experience has been compromised The manning issue out in training waspart oftheproblem. This willcontinue to be aproblem until they are adequately staffed" "The current resources dedicated to the LOUTprogram are not suffcient to meet the needs ofthe students" "On several occasions we would be told to perform a step one way and then the next day a different instructor would tell us we were performing this step wrong. "

IUwYNc"Ow. mr 22

Staffing and experience issues have been recognized in previous assessments both at Unit I and NTD as a whole.

Source Comment Self Assessment of Unit 1 LOIT

• Instructors with more operations experience are needmi.

(January 1996)

Self Assessment of Unit 1 LORT

• Braahing the experience base of all instructors is needed.

(January 1996)

Team Assessment of1995 Nuclear

• Instruction is skewed to the instructor's expertise, resulting in Trammg Department Activities incomplete learmng (April 1996)

Trauung focuses on passing tests not oiioperating the piant.

1996 Program Coordinators

• Many stafflack recent plant experience Assessment ofTrammg programs (November 1996) 4.4.2

Conclusions:

4.4.2.A 'Ihe frequency of change for personnel associated with Unit 1 Operations Training and the difficulty encountered in retammg qualified and experienced staffis clearly affecting the ability of NTD to deliver quality training.

4.4.2.B The succession planmng process is ineffective.

4.4.3 Recommendations

4.4.3.A Management should consider the use ofINPO Loanees or industry experts to function in the Unit 1 OTB chain of command until experience level of the incumbents can be raised to an appropriate level.

4.4.3.B Management should review current conditions and practices which result in difficulty of NTD to retain qualified and experienced staff.

4.4.3.C Management should develop and implement an aggressive succession plan for all key positions in trammg (Trammg Coordinator and above).

4.4.3.D Trammg management should take a proactive stance in ensuring that transition and tumover for individuals being promoted or transferred is completed.

%E$U7'" - r*. m 23

- . - - - - - - - - - - . = . -. . - - - . . - - -- .-.

4.5 SELECTION PROCESS and CRITERIA FOR PROGRAM ENTRY AND NRC EXAM.

4.5.1.1 Candidate Selection Process Nine andidatas were originally selected by MP-I line management to participate in trammg to gain their SRO licenses in order to strengthen the unit's work control activities. Prerequisites for entering the program are described in Section 3.0 of the LOUT Program Description (TPD), Section 6.2.4 of the LOUT Program Implementing Procedure (TPIP), ANSI 18.1-1971 and NUREG 1021. The TPD and TPIP also specify I' responsibilities for candidate nommation and selection. There was, however, no formalized process for makmg the candidate selections. The original candidates consisted of five reactor operators and four STA's (" instant" candidates). The reactor operators participated in Assessment Centers to evaluate their leadership and supenisory skills. The STA's were not evaluated. Shift Managers provided input to the Unit Operations Manager regarding the work history, performance and qualifications of reactor operators. The Trauung Department was not involved  ;

in screening the candidates.

The IRT found indications that some of the mndid*s selected for the program may have been marginally motivated to obtain an NRC license:

. One candidate withdrew from the 1996 LOUT program during the first-week.

. One candidate refused remediation after failing the company examination. This candidate has since  ;

termmated employment with the company.

. Feedback which the IRT received from interviews with management, supervision and staff was not I consistent regarding the quality of candidates selected.

On Jan.1,1997, shortly after the NRC license exammation, the MP-1 Operations Manager issued a new departmental instruction detailing the screening process for the selection of RO trairing program candidates. The manager also intends to issue a similar instruction for the selection of SRO candidates.

4.5.1.2 Selection for NRC Exam The IRT found several weaknesses in assessing the readiness of candidates to take the NRC exam:

. Verbal and written concerns from the LOUT Program Coordinator regarding candidate readiness and the recommendation for an academic review board was not addressed by trammg department and unit management.

. The Unit Operations Manager and Assistant Manager assessed the candidates' performance on their final simulator exam. The Program Coordinator recommended that an independent contractor be used to assess the readiness of the candidates to take the NRC exam.

. Section 5.4 of NTM 7.079 TPIP indicates that the Manager, Operator Trammg is responsible for approving license candidates for NRC exanunation. This position was vacant during the last few months of the program

. Records could not substantiate that candidates had completed all phases of the program trammg requirements.

INYd$$$c*ir.m at M

~ . - - -_- . . - . . . . . - - .. . . _ . - .- - - . _ - _~ -

l l

4.5.2 Conclusions l j

4.5.2.A lack of a proceduralized selection process allowed some margmally motivated l

, andidatae to enter the 1996 LOUT program i 1

4.5.2.B Candidates' readiness to take the NRC exammation could have been more thoroughly and objectively essessed.

4.5.2.C The same criteria was not applied to select STA candidates as upgrade candidates 3 assume the same SRO role in the plant.

4.5.3 Recommendations

l 4.5.3.A Operations Management should formahze the SRO candidata selection process j similar to its recent instruction for RO candidate selection. The process should include involvement and input from NTD.

4.5.3.B Benchmark industry "best performers" and develop a formal process to assess the

~

candidates' re.adiness to take the simulator exam.

! 4.5.3.C Senior Management should reinforce its expectations of Trammg Department and

, Unit personnel in fulfilling their responsibilities for thoroughly assessing and remadiating the performance oflicense candidates. ,

l

+

l

)

1 I

4 1

a l

2 4

4 Tmal Drd .11 Feteuery.1997 1996 MP.3 LOLIT N1LC 5- Faiheus IRT

i. .

i j  !

t 4.6 MAINTENANCE OF PROGRAM AND STUDENT RECORDS

]

l 4.6.1 During the investigation of this event the LOUT program personnel records were found to be inadequate.

i This represents a challenge to 10 CFR 50.120, Trauung and Quali6 cation of Nuclear Power Plant Personnel and j 10 CFR 55, Operators' Licenses i e Documentation could not be easily produced to verify that student prerequisites and prior trauung ,

requirements were met.

3  !

i e Documentation of student attendan" and completion of LOUT program requirements could not be verified 1

during the course of this investigation.

}

e Previous trauung (LOIT and STA) documentation of student attendance and completion of program requirements could not be verified during the course of this investigation.

e inconsistencies were noted on the trackmg of student **andaan and OJT time trackmg l l l e The existing computer program - T==le==er, was not used to track student qualifications.

1

{

e Excel spreadsheets are used to track ="and=a~. The lesson indentfiers on the spread sheets are not clearly i j tied to trauung program description identifiers. '

! . No "As Taught" schedule existed and had to be created for this investigation.

• j e here are no Trauung Program Control Comnuttee Meeting Minutes describing or approving any changes or ,

j deviations from the accredited program j e There is no requirement for an internal or external audit to be conducted at the completion of the program.

j e OTB1-3, Operator Trauung Anand=am and Performance and OTBI-5, Maintenance of Trauung Records and  !

Files have not been utihzed e ACR M1 97-0117 was initiated on 1/17/97 to denment and further investigate student and program records  !

l j for the 1996 LOUT Class.

)

4.6.2

Conclusions:

j 4.6.2.A Personnel and program records cannot be verified as being complete and accurate.

his presents a challenge to 10 CFR 50.120,10 CFR 55 and QA requirements 4.6.2.B Currently two computer programs exist to track personnel and program records -

neither is being used at this time.

j 4.6.2.C There was inadequate oversight of personnel and program records by trauung and j unit management as well as the oversight organization.

i 4.6.2.D Operator Trammg Branch staff, supervision and management are not following

established procedures to mamta n records.

i i

4.6.3 Recommendations

) 4.6.3.A Ensure that the investigation associated with ACR M1-97-0117 verifies that all -

4 licensed operator trauung records are in compliance with 10 CFR requirements.

j 4.6.3.B 1mplement the approved records management system.

j 4.6.3.C Implement external program auditing at a frequency appropriate to ensure record i retention requirements are being met.

Fan! Dna .1I Fehmary.1997 y i 29W6 MP-I U3UT!#LC 5- Fuhaus IRT -

)

i

. 3 .

i i

! 4.7 TRAINING MATERIALS.

5 i

i 4.7.1_ The IRT identi6ed the followmg issues relating to trauung matenals:

l

• Many system texts are out ofdate.

4 4

e PDCR (plant design changes) and procedure changes are not communicated to Trauung .

I e NTM 1.04, Commitment Tracicng is not utilized. No means of verifying that trauung material is current i i other than an instructor re-researching material prior to *~hing lesson or distributing text.  !

j e Lesson plan orgamzation does not follow text orgamzation. This interferes with effective learmag

. Exams and lesson materials are not updated concurrently. Students have been tested on material not

addressed in the lesson or in class. >

3

- = Exam bank questions are not in the industry standard four part multiple choice format.

I e

The cognitive level of many questions is not representative of current industry nonn. '

!

• Picture boards of control panels are out of date, lending to negative training.

]

• No material is provided for shift managers to effectively implement the on-shift time of License i candidates.

f-i 4.7.2 Conclusions 1

4.7.2.A The condition of trammg material does not support a program that heavily depends

, on self-study of systems and adraiai*ative topics.

} 4.7.2.B Communications between the line orgention and trauung have broken down in

• regard to plant design changes and procedure resisions.

j 4.7.2.C - Lack of procedural emnli==ce for comnutment trackmg on MP1 has lead to degradation of trammg material quality.

4.7.3 Recommendations l 4.7.3.A Review and revise all trammg material to include PDCR and procedure changes.

4.7.3.B Evaluate the issue of trauung material quality for applicability to all trammg 4

Progens l

4.7.3.C Management should ensure fae requirements of NTM 1.04, Comnutment Trackmg j are being met.

) 4.7.3.D Ensure that instructional materials are adequately upgraded and maintained before

. being used in trammg j 4.7.3.E Develop and implement a process that ensures that applicable PDCRs and procedures changes are communicated to NTD in a timely manner.

i I

1 Fan 1Dri .11 Fotnery.1997 1996 MP.814K,7NILC Exemunsaan Feamus IltT sy'y l

e

. i i .

f, 4.8 NRC EXAM, REVIEW AND VALIDATION.

4.8.1. De IRT identified several issues in the review, validation and adnunistration of the NRC exam.

(

l .

De exam bank for Unit I contains numerous three distracter questions - Confidence Weighted Testmg i (CWT) format rather than the four part muhiple choice used by the NRC. i i e The exam questions supplied to the NRC did not include appropriate rJ.,.-ces nor do they include  !

I knowledge / ability (K/A) values.

i 1 .

Exam question information in the "T==ba=='ar" system that was reviewed was incomplete.  !

i j e Nuclear Trammg Manual (NTM) 4.01 - Evaluate / Develop Test Items, was not consistently implemented I This procedure requires the documentation of K/A's, objectives, and references e

Dere is no formal method for review of the NRC wntten exam. De process was dapaadaa' on staff  ;

experience to provide an = day-** exam review. Overall experience of the reviewers was madequate.

j Only one of four reviewers had previous experience in reviewing an NRC exam.

i e There is no formalized method of selectmg validators to ensure that a representative sample of the target audience validates the exam.

j e The final validation was conducted by two experienced operators who expressed concem over the length j' of time it took to complete the exam. Both noted the difficulty of the =M =.iive questions, but

, assumed that the candidates would be more prepared for them. Both concluded that the exam was

! lengthy, but acceptable, a

4 e Questions about the length, scope, and difficulty of the exam r===iaad, but the staff was uncertain of what i_ actions could be taken to address the situation.

. During the exam, an extra half hour was given to the candidates to complete the exam. Candidates still j expressed that they felt mshed during the exam. Only one candidate completed the exam at the four hour  !

t mark. i

[

1

• After the exam,10 exam questions (10% of exam) were challenged by the unit.

, e At the request of the IRT, an indaaa= dant assessment of the wntten exam was conducted by Nuclear j Traming and Educational Services, Inc. (see Att=^=aat III). In his evaluation the consuh=at concluded -

"It is my conclusion that while there is reason to question the wiidity of the licensing exam due to its l

{ length, the information available suggests that the exam, however dQcult, was niid."

• A review by 3 of the IRT members (all industry peers and previous SROs) concluded that the written j exam, while lengthy, was within the NRC testing guidelines.

4.8.2

Conclusions:

4.8.2.A The exam bank provided to the NRC did not meet the guidelines of NUREG/BR-0122, "Exammers Handbook for Developing Operator Licensing Written Ex==ia=tions" and could not be used by the NRC.

4.8.2.B Lack of a formalized exam review and validation process contributed to an ineffective review and validation of the NRC exam, i real Dma . Il Felumery.1997 ny o 1996 MP l1.0LITNRC E- Fakses IRT **

_ ~__ __

.. ._ _. _ . _ _ _ _ ~ . . ~ _ _ _ _ . . _ _ -__ _ . _ . _ _ . _ . _ _ _ _ . _ _ _ _ _ . . _ . _ _ _ _

=. .

I l

l 1 4.8.2

Conclusions:

(cont) 4.8.2.C Lack oflicensing exam review experience by reviewers contributed to an ineffective review and validation of the NRC exam.

4.8.2.D Line ownership of the exam review and validation process needs improvement.

l l 4.8.2.E The review and validation process took place simultaneously during the same week, L thereby reducing the effectiveness of the review and validation process.

4.8.3 Recommendations

4.8.3.A Upgrade exam bank to industry standards. Implement NTM 4.01 to ensure exam questions ,

are properly constructed and docenented.

4.8.3.B Benchmark industry "best performers" and adopt practices into a formalized process for f

~

the review and validation of NRC exams Provide expectati5ns and training to individuals scheduled for exam review and validation.

4.8.3.C Line Management should incorporate exam functions into their resource scheduling to i ensure that appropriate personnel who are representative of the target audience are  ;

available for NRC exam review and validation.

4.8.3.D Separate the review and validation of NRC exams processes so that they do not occur concurrently.

l l

l

)

l l

l 1

Fmal DraA Il Fehmry,1997 im w.imer = % v m at 29

5.0 EVENT CONSEQUENCES The consequences of the LOUT failures as listed below are the result of past, present and expected outcomes of the factors discussed in the previous sections. These consequences entail regulatory as well as direct and indirect financial impacts.-

Regulatory Impact . Increased scrutiny from the NRC in the form of trammg inspections is likely.

. Action by the National Academy of Nuclear Trauung is possible.

Direct Costs: .

IRT investigation - NU and contractor IRT member salaries, lost productivity of interviewees and contributors.

1 e NTD staff salaries for NTD review ofIRT issues in other programs

. NTD salaries for trammg material and process upgrade.

. Student and instmetor salaries required for remediation.

Indirect Costs . The event required the attention of high level managers m&#ng productivity and oversight in other areas.

. The event had significant morale impact on the staff and candidates involved.

. The event will erode the credibility of the NTD with INPO and the NRC.

. The event caused additional negative publi6ty toward Millstone in the midst of already strong public opinion.

. "Get Well" actions taken as a result of this event will affect the ability of NTD to i meet the plant's trammg needs.

. Potential impact on recovery and restart due to delays in licensing additional SROs.

]

l 1

l l

i FmalDrah . Il Fetruary,1997 n 1996 MP-8 LOUTNLC f. - Falures IRT V

]

i 6.0 ROOT CAUSE, LESSONS LEARNED AND CONCLUSIONS j.

3 6.1 Root Cause I A lack of management oversight and accountability by management, supervision, and staff coupled with  ;

i the ineffectiveness of the corrective action program to remedy previously identified, long standing issues  !

j '

t ied to the degradation of training program quality.

6.2 Lessons Learned >

The following are the principal lessons learned of broad oreanir=*ianal significance:

! Management oversight shouldfocus more on specific adivities within theirfunction.

1

[ All levels of management must be more focused on specific activities within the function. Managers should carefully review decisions and plans until they have solid assurance that the subordmate functions are managed

and supervised to their expa.tgions. ._

! Strict adherence to established procedures and policies provides a barrier to program quality t deterioration.

4 Compliance with NTD procedures and mstructions ensures adherence to the Systems Approach to Training l Trammg and line management should ensure that subordinates understand and are held accountable for

implementation of policies and procedures.

1

Ineffective Corrective Action Program.

i Many of the basic causal factors contributing to the unsatisfactory performance on this exammation were found I to be similar to those that precipitated previous failures across the three Millstone units. Rese unresolved j weaknesses in operator training had been identified in numerous self-assessments, QA audits, INPO evaluations j and NRC inspections and should have been viewed as precursors to this event. Had these issues been remedied

! in a timely manner the negative outcome of the LOUT class might have been prevented.

I 1

Cross discipline and cross plant evaluation and correction ofidentifiedproblems should beperformed.

}

Problems and issues identiSed in one area result from many casual factors. In some cases these factors exist in other areas, especially where the function is similar, or common procedures are used. Self assessment i effectiveness will be increased when identified problems are evaluated and corrected cross discipline or cross unit.

} Effects ofExperience Dilution l Personnel changes can have a disruptive effect on orgamzational contmuity. The change management process for personnel important to safety (Operator Trammg for example) should warrant similar evaluation and consideration given plant hardware changes. This does not have to be taken to the extremes of a 50.59 i evaluation, but management attention to the effects of persoial change should be commensurate with the  ;

! importance of the job. In the 1996 LOUT event ALL personnel involved with the design and implementation of l the program were changed at least once. The results of this experience dilution were evident in decisions made

and actions taken. Ensuring the contmuity oflong term programs should be a priority for management.

i

(

.?

4 l FmalDinR . Il Fatsuery.1997 9g 1986 MP.I LOUT NRC Ememaston Failmes Ulf '+

i i

i 1

2 i

F I

- . 6.3 Sununary of Conclusions 4.1.2 Management Onrsight Conclusions.

4.2.2 SelfAssessmentand Corrective Action Conclusions.

i 4.3.2 Systematic Approach to Training Conclusions.

i 4.4.2 Erperience andStafng Conclusions.

4.52 Candidate Selection Criteria Conclusions. 1 i 4.6.2 Training Records Conslusions. }

+

4.7.2 TrainingMaterials Conclusions. l 4.8.2 NRC Exam, Review, and Validation Conclusions. '

l_

i The following list summarizes conclusions made during the IRT investigation:

h 4.1.2.A Managemet oversight of the 1996 IJOUT poyam was ineffective in ideratfymg and correctag challenges and

deficiencies in the posentang of qualitytramans Management =imad ownerous opportunnies to ensure that tramms l delivery was ofthe highest quality, i

j 4.1.2.B "n - failed to consider paviously identdied oversight issues (aeroes all unit) as "warnang flags" and adjust ns actions appropnately.

9 4.2.2. A The Self Assessment and Corrative Action pocess has not been used effemively to identafy and correct weaknesses j in trammg 1 4.2.2.B The Self Assessment and Corrective Action pecess does act systematically assess past problems to ensure the -

adequacy and anstamahiluy ofpast correceive maia=

f

, 4.2.2.C The Self Assessment and Corrective Action pocess in many cases focuses on a single plant and does not connder if l problems identified in one unit are present elsewhers.

4.2.2.D Some =====anen in the past, including the accrednataan utf evaluation report, did not include line == nag ===a

• • mersluP-

! 4.3.2. A The original acaedned propam was revised without sufficies an=a== pissed on accrednation comnutments 4.3.2.B NTD procedures were not used to ensure a syp=namic pocess was followed 4.3.2.C *nie analysis and needs assessmera for poyam redesip was not *==== mad and therefore cannot be venfied.

4.3.2.D ' Die NTD maffinvolved with design and revmon were unaware of accredutat==

~

4.3.2.E Evidence ofweaknees in -  ; and applying the SAT is evident elsewhere in the NTD.

3 4.4.2.A The frequency of change for personnel assectated with Unit 1 Operations Training and the ddliaaky encountered in retamms quahfied and exponenced saffis clearly affecting the ability ofNID to deliver qualny trammg.

1 4 4.2.B 1he succession planningpocessisineffective.

4.5.2. A Lack of a pocedurahzed misction pecess resuhad in some marsmally motivated and qualified candidates entenns j- the 1996 LOUT progrant 4.5.2.B Candidates' readmess to take the NRC exammation could have been ma e thoroughly and objectively assessed 4.5.2.C 1he same enteria was not applied to select STA candidates as uppade -d d='= to ==ane the same SRO rnie in the Pl ant 4.6.2.A Personnel and propam roccrds cannot be currently verified as being complete and accurate. This presents a challenge to 10 CFR $0.120,10 CFR 55 and QA requirements.

4.6.2.B Currently two computer programs exist to track personnel and poparn records meither is being used at this time 4.6.2.C 1here was inadequate oversight ofperscanel and pogram reconis by tramang and unit =anar-as as well as the overngla orgemaation.

4.6.2.D Operator Training Bnach staff, supervmon and maaap==a are not followns an=ht-had procedures to mausam records.

4.7.2.A The condnaan of trainmg matenal does not support a pogram that heavily depends on self-study of systems and adhmsustrative topics.

4.7.2.B Comnumcations between the line orpai-m== and trasmag have broken down in regard to plant design changes and poemlure revmon.

Paul tmeA . Il Fotwuary.1997 y 19es MP.I tDJT NRC '- Fehses IltT J **

.-.. .. - . .. - .- . - . - . . . . .. - -- - ~_

r 5

I 6.3 Summary of Conclusions (cont.) ,

4.7.2.C Lack of procedural compliance for comnutment tradung on MP.I has lead to degradatum of trainmg matenal P*Iity- t 4.8.2.A The exam bank provided to the NRC did not meet the guidelines of NLF.EG!BR4122, *Exammers Haneook for Developing Opers.or licensing Written Exammations" and could nor ce used by the NRC.

1 4.8.2.B Lack of a formalized exam review and validation process contribu'ad to an ineffective review and validation of the NRC exam.

4.8.2 C Lack oflicensing exam review experience by reviewers contributed to an incasetive review and validation ofik NRC exam.

4.8.2.D Line ownership of the exam review and validation process needs 4

+

. 4.8.2.E The review and validation process took place simultaneously during the same week. thereby reducing the effectiveness of the review and validation process. i L

t e

h b

5 Tsm!DrmR . Il Fetuvery,199' 1996 MP 1 LOUT NRC 1.nammunen Fahses IRT 33

i

< e I i f

7.0

SUMMARY

OF RECOMMENDATIONS ES Recommendationsfrom the Executin Summary  ;

4.1.3 Management Ontsight Recommendations.  ;

4.2.3 SelfAssessment and Correctin Action Recommendations. l l 4.3.3 Systematic Approach to Training Recommendations. l' 4.4.3 Experience andStaping Recommendations.

4.5.3 Candidate Selection Criteria Recommendations.

l 4.6.3 Training Records Recommendations.

l 4.7.3 TrainingMaterials Recommendations. ,

4.8.3 NRC Exam, Review, and Validation Recommendations. l t

' Die following list Summanzes the IRT'S recomrrendationS for corrective action:

Draw upon INFO loaness or industry e aparts to ^ , A ihnction in the MF-1 Operator Training Branch until the i ES-1 experience level of the incumbents can be raised to an appropnate level.

ES-2 Conduct m<lepth reviews for applicatility ofissues from bis invessagataan to other gauung programs for all Millstone Units. Consider temporary "stop-work

• orders for those pre, drams determmed to be unace=r hie ,

ES-3 Evaluate competencies of the license:d RO to SRO can& dates based on e=====tiaa results. Evaluate how identified weaknesses of the candidates pertair to the romaander of the licensed stafE [

ES 4 Conduct tuviews of previously iderAified problem areas and expedite close out of those relatag to accreditation, I replations, and training quality. %is should inchade re evaluating ynontaes and resource na== *=nsats against other ,

Nuclear Training Department sucellence initiatives.

ES-5 Implemem an effective correctivr; action program which includes ar*=mahility for closure in a timely and effective manner.

i ES d Senior management should clarify and remforce eNpectations ofTrainmg and Operanons management regardag their responsibilities for overught of trainmg. Implement methods and snan=nes to ensure that oversight is effective 4.13.A Review industry guidance airl INPO idenufied "best performers" with respect to manapment oversight. Adopt best practices into oversight process.

4.13.B Reinforce with Training arvi Operanons snanagement their responsibilities regardag overnight of traanms.

4.13.C Establish high expectataans regardag oversigla and implement methods to ensure that oversight is '

frequent, and useful.

4.1.3.D Ensure that oversight feedback is commumcated and addressed conustantly across all units.

4.23.A Review 1.adustry guidance and INPO identified "best perfonners" with respect to asif assessment and correcuve action. Adopt best pra:tices in the NTD self ==a6==== and correcuve acuan process.

- 4.23.B Conduct an 'uryney review' of previously idenufied probian areas and expedne close out of those relating to accrednataan and / ce regulatory issues.

4.23.C Ensure correctrve actions are developed and tracked for all intersified treamng issues.

4.23.D Review effativeness of past corrective actions. Incorporate this review isso the self ========= process.

4.2.3.E Develop, communicate and reinforce clear suparemions regardag the Self Assessment and Corrective Action process for all levels of management, supervision and statt 433.A Evaluate ensung NTD procedures and imerucuans (OTBI's) and ensure these procedures meet NTD evremias for the trammg process.

433.B Revise NTD r. - " and instructions (OTBI's) as required.

433.C Manassenant and supervmon should ensure aa-pl.aar= with enstang procedures. j

. 1 4.33.D Evaluate the as taught IMT propam against accrednationmand=ds and ensure that all accrednation commaments were inst 433.E Develop and implement corrective actions for the issues idetsified in the recently perfonned intamal amammam of accrednataan criteria 433.F ' ' the previously existing plan for revieweg the Symems Approach to Training as part of the 1997 Comunums Instructor Training program ria.irme u un.,y."

' 19es MP-1 Lot.TTNRC r% y gues tRT 34 ,

7.0 SU.MMARY OF RECOMMENDATIONS (cont.)

1 4.43.A Managemer: should consider the use ofiNPO Loanees or industry experts to function in the Unit i OTB chain of command until experience level of the incumbents can be raised to an appropriate level.

4.4.3.B Management should review current condations and practices which result in difficuhy of hTD to retain quahfied and experienced staff.

4.4.3.C Management should develop and implement an aggressive sucosesion plan for all key positions in trauung (Train;4g Coordmator and above).

4.43.D Training management should uke a proactive stance in enounng that transition and turnover for individuals being l prornoted or transferred is compMect l 1

4.53.A Operations Management should for:Jalize the SRO candidate selection process similar to its recera instruction for RO I candidate selection. The process shou;d include involvement and input from hTD. I 4.53.B Benchmark industry "best performers" and develop a formal process to asaans the candidates' readmeas to take the simulator exam.

]

4.5.3.C Senior Management should reinforce its expectations of Training Department and Unit personnel in fulfilling their responsibili'ies for thoroughly assessing and remediating the performance oflicense candidates.

4.6 3.A Ensure that the investigation associated with ACR M1-97 0117 verifies that all licensed operator training records are in compliance v ith 10 CFR requirements.

4.63.B Implement the approved records management system.

1 4.6.3.C Implement external program auditing at a frequency appropriate to ensure record retention requirements are being met.

4,73.A Review and revise all tranung matenal to include PDCR and procedure changes, 4.7.3.B Eva!uate this issue of tramung material quality for applicability to all trammg programs 4.73.C Management should ensure the requiremrins ofhTM 1.04, Comnutment Tracking are being met '

4.73.D Ensure that instrucuanal materials are adequately upgraded and maatamed before being used in training 4.73.E Develop and implement a process that ensures that applicable PDCRs and procedures changes are conn.nrucated to hTD in a timely manner.

4.83.A Upgrade exam bank to industry standards. Implement NTM 4.01 to ensure exam questions are properly constructed and documented.

4.8.3.B Benchmark industry "best performers" and adopt practices into a forralized process for the review and validation of NRC exams. Provide expeutions and training to individuals scheduled for exam review and validatiort i

4.83.C Line Management should incorporate exaru functions into their resource scheduling to ensure that appropriate )

persarmel who are repr=anrative of the target audience are available for NRC exam review and validation.

4.83.D Separate the review and validation of NRC exams processes so that they do not occur concurrently. 1 I

l i

l I

i i

. TsulDnd .1I February.1997 199s MP 1 til,T NRC a=- Feduns IRT g l 1

l 8.0 ATTACHMENTS l

i 1 Summary of Previously Identified Issues j 1

II References III NETS Review of NRC Examination l IV Evaluation of LOUT Issues Against NRC & INPO Requirements 4

i V Analysis of NTD Procedures VI Barrier and Failure Mode Analysis VII Conclusion and Causal Factor to Recommendation Matrices 1

l VIII Previous Pass Rates MP-I LOIT & 1996 Exam Results IX IRT Charter l X Accredited Program Sequence i 1 l 1 ,

I

+

i I

i 1

Fma! DraA .11 February,1991 y 1996 MP 1 LOUTNRC !.ummanon Faares RT Ju

Attachment I

.. h = = arya n ._- ', M Training Issues Listed beloware mmm==and r=rl= afprevaously idmnfiM trauung issues. Some of the issues listed i below originatM in MP-2, MP-3, or CY - and include LORT issues - but are included due to the similarity of conditions found in 1996 MP-1 IDUT pmgram.

ACR 11037 Adverse Tread Im Weeldy Operations Simulator Evaluations for MP-1 Root Cause Analysis (October,1996) e Reduced trammg contact time contributed to crew failures.

• Standards for perfonnance and evaluation ofindividuals and crews had degraded.

. Lack of ownership by both the Shift Managers and Training Mentors contributed to degraded crew performance.

1995 /1996 LOIT Class Company Final Examination Root Cause Evaluation (September,1996)

~

e ineffective strategic planning (in training).

. Lack of Self Assessment and Corrective actions.

. Human error or misjudgment on the part of supersision.

. Insufficient oversight by Operations manager.

. Insufficient oversight by trammg manager, e Insufficient oversight by supervision.

. Inadequate verbal or written communication between trammg staff and management and plant management.

NRC Inspection 93-80 (June 1993)

= Control measures do not address how training for important of significant procedure changes and license amendments is given.

. Control measures do not address when a root cause analysis for major trammg deficiencies is to be conducted.

. Some licensed operators failed to Mtmd required LORT.

. Audits of operator trammg required by technical specifications were not done (Unit 1 &3).

. Limited distribution ofTPCC mmutes and lack of detail in mmutes make it difficult for others to monitor actisities of the TPCC

= TPCC committee size limits the conumttees ability to proside checks and balances among all aspects of the trammg program

. A TPCC member approved his own trauung exemption.

. The problems . . . .will continue to be an issue until the experience level of the new staffis adequate .

. Exam banks were not kept up to date.

. 1990 assessments of LORT failed to identify problems contributmg to LORT failures in 1991 and 1992.

w omn . n %.im 1996 MP-1 LOUT NRC U - Faihre IRT

v Attachment I Summary of Previously Identified Training Issues (continued)

Surveillance A-324 " Verification of....LORT Training Attendance (March 1992)

• Attendance records were found to be of poor quality. (NRC initiated concem)

Audit A60516 " Operations Activities for MP 3 (March 1993) e Management did not observe simulator performance.

• Topics not covered in LORT were not included in sample plan (NUREG-1021).

Exams were not time validated (NUREG-1021).

. Task lists did not differentiate those tasks that require continuing training (NUREG-1220).

Leanung objectives did not have conditions, actions and standards (NUREG-1220).

• Lesson plans did not include the characteristics described in NUREG 1220.

. Test items did not cover all the la.mung objectives.

Audit A21062 "MP1 Unit Staff Training" (April 1993)

• At least 38 individuals did not attend all sessions required.

. Training effectiveness reports were not issued in 1990 and 1991.

• Management Simulator observations were not made.

• Monthly training reports were not issued.

• Instructor observations were not completed annually.

NRC Root Cause Analysis of Unit 1 LORT 1991 Unsatisfactory Program Determination ( June 1993) e Inadequate learmng objectives with respect to high standards.

• Poor practices not corrected by TPCC.

• Inadequate evaluatica criteria forjob setting.

• Inadequate audit and evaluation and inadequate external experience review e Inadequate review of external experience.

• Inadequate review ofinternal experience.

NRC Inspection 96-05 NU Assessment of NRC Issues (December,1996) e Evaluation process lacked accountability.

• Lack of Shift Manager ownership.

• Lack of critical evaluations and crew feedback.

• Weak higher order exammation skills.

• Lesson plans and texts require enhancement.

= ' Exam bank needs enhancement.

ACR M2-96-0657 Failure to apply SAT at Unit 2 (November,1996) j e

Plant evaluation feedback is not being consistently evaluated for training impact.

• Task lists have not been routinely utihzed to keep programs current.

Identified plant and industry events have not been consistently incorporated in trammg.

i e Trammg for OJT instructors has not been conducted for some OJT instructors.

l UdnENEEtiE r.a ms

4 4

\

Attachment I Sunumary of Previously Identified Training Issues (continued)

Self Assessment of Unit 1 Licensed Operator Initial Training ( January 1996)

. Need to upgrade d- .." eve tranung identified

e Candidates need more meetmgs with and observations by OPS management.

4

. Task list needs to be reviewed and irrelevant tasks deleted.

1

. Instmetors with more apannaa= experience are needed.

. Some OJT and OJT cards were not effective i . A wider variety of simulator scenanos areneeded

, Self Assessment of Unit 1 LORT (January 1996)

) . The plant needs to become more aware oftheir crew's remediation.

j . Imprevements are needed in identifying candidates who are at risk in failing.

. Broadening the experience base of all instructors is needed. ._

e The JPM process needs to be made more interesting.

Team Assessment of1995 Nuclear Training Department Activities (April 1996)

. More emphasis needs to be placed on industry events and trends in trauung.

{

! . Communication between customers and instructors is poor.

. Confidence Weighted testing is not used as designed.

. Instructors need more plant experience.

. Trammg focuses on passing tests not on operating the plant.

. Instruction is skewed to the instmetor's expertise, resulting in incomplete learmng. l

. The value of the Self Assessment Process is unclear to many NTD staff.

1996 Program Coordinators Assessment of Training programs (November 1996)

The report states that approxunately 50% of NTD programs are not being maintained. Either thejob analysis or the task analysis is out ofdate.

. Post Assessment feedback pracecere(l.ong Term) are not driving program content changes.

. Industry events beyond SOERs are not routinely incorporated into the training programs.

. Work observations are not factored into the programs.

. Prerequisite knowledge requirements are not either identified or considered by the line prior to enrolling  ;

candidates in training programs.

. Line Management and Trammg Supenisor Management obsenations of trauung are not adequate.

. Attendance problem at training has been tolerated by both Trammg and Line Management.

. Program Maintenance has not been done in accordance with the NTM e Expecations set forth by manage-e imphod that podium contact time was more important than materialin=mtemnce.

. Lack of staff resources impacted matenal mamtenance process. Budgets were cut reducing staffmg without changmg work scope, streamhmng processes, or incorporating technology to improve effectiveness. In fact a majority ofgroups were tasked to do more with less by simply being directed to work harder and longer.

. Implementation of new practices such as CWT, focus groups, and self directed teams impacted the effectiveness of the Nuclear Training D% a. Additionally this also sent a message that this was all that mattered, especially when campared to program maintenance.

ma DrtR . Il Fetary.1991 1996 MP l LOUT WILC Emanummon Faihns IllT

_ _. _ _ __ __ ._. . . . . . _ _ _ _ ___ __ _ ~ _ _ __

3 I

s Attachment I f . .

Summary of Previously Identified Training Issues (continued) I 1996 Progrwi Coordin_stors Assessment of Training Programs (November 1996) e Many TPIPs are out of date. The staff does not view this as a customer service contract that they have with the plant, the NRC, and INPO.

• Inadequate staffing exist to effectively mentor the candidates.

Weak planning skills and changing customer needs greatly effect the efficiency and the effectiveness of the Nuclear Trauung Depamnent.

Taskmaster must be replaced. The units have too many open comnutments to insure that trammg is i provided with the most recent materials.

• Many stafflack recent plant experience i e

Supervisors do not spend the appropriate amount of time coaching and developing their instructional l staff.

. Certified Instructor Trauung does not meet the staff needs. I e- Individual development plans were not effectively imple;nented at all levels of'nIanagement. ,

e In several instances task lists and initial and continuing trammg programs are not current or are in the l process of being updated.

. Exam validation process is weak due to resource limitations. Several program coordinators believe that  !

their exams do not discrmunate between competent and non-competent personnel. l

. Development and preparation times are inadequate and greatly impact the quality of service we are providing.

. Labs and In Plant trammg are not observed by trammg or line management.

• In several programs OJT cards are out of date or are in the process of being updated. )

• OJT and OJE are not properly mo.dtored by line management orjob incumbents.

• Job Incumbents are not provided refresher trammg on management's expectations regarding OJT and OJE. l

• Except for Requalification exammations, simulator trammg is not observed by line and trammg management in accordance with expectations in the approved TPIPs. I e Many simulator instructors need facilitator trainmg and continuing refresher traming in simulator iistructional skills.

. Several aspects of simulator training appear disconnected from the plant (Realism).

. Instructors do not have time to properly validate, curriculum test, or prepare for simulator trammg

~

e Core models need to be upgraded and enhaaraA on all simulators to meet current industry evpar+=tions  ;

regarding startup trauung

• DR backlog impact quality of traimng on MP 1 simulator. 1

. Quality of critiques and use of freeze, backtrack, and replay need improvement.

. NTD customer feedback process is weak and feedback indicates that in some programs feedback is not .

promptly acted upon. '

• NTD long term feedback process is not effectivc.

• NTD self assessment processes such as monitoring trairdng effectiveness and instructor assessments are weak.

• Significant Events Evaluation Committee is not being used.

• NUTIMS needs to be implemented to better track trammg commitments.

• NTD does not benchmark programs against the industry.

.e.n a ra im I996 MP 3 LOUT NRC Emanunenen Fatma IRT

Attachment II ,

Independ(nt Review Team 1996 MP-1 LOUT NRC Examination Failures f

REFERENCE DOCUMENT INDEX IDENTIFlED

~'

DOCUMENT' BY -- DATE: DOCUMENT DESCRIPTION c ,

. NUMBER- -TEAM iENTERED. .. e ,

MEMBER -

01 Muccino I/3/97 IRT Charter 02 Muccino I/3/97 Resumes -IRT Members 03 Muccino 1/3/97 Organization Charts - Nuclear Training Group 04 Muccino 1/3/97 1996 INPO Millstone Audit Report -Training Section 05 Muccino 1/3/97 19% LOUT Schedule i j

06 Muccino I/3/97 MP-1 LOUT Program Description (Rev.0) 07 Muccino 1/3/97 NU LOUT Program Implementing Procedure (NTM-7.079 Rev. 1) 08 Muccmo 1/3/97 Examination development & Administration (NTM-5-02 Rev. 3)  !

09 Muccino 1/3/97 Evaluate / Test Items (b.yd-4.01 Rev. 0) 10 Muccino 1/3/97 ACR Ml-96-1024 LOQF Written Exam Performance (12/8/96) 1I Muccino I/3/97 ACR Ml-96-1025 LOUT Simulator & JPM Exam Performance (12/20/96) i 12 Muccino 1/3/97 ACR M3-96-1154 MP-3 Operator Exam Performance (11/20/96)  !

13 Muccino 1/3/97 News Articic on Exam Failures 'The Day'(1/1/97) 14 Muccino 1/3/97 MP-1 Weekly Simulator Failures - Root Cause Evaluation (10/28/96) 15 Muccino 1/3/97 MP-2 LOIT Company Examination Failure - Root Cause Evaluation (9/27/96)  :

16 Muccino 1/3/97 NRC Inspection report (EA 93-130) Millstone LORT Program (6/7/93) l 17 Muccino I/3/97 NRC NOV and Civil Penalty - Millstone LORT (9/20/93)

I8 Muccino 1/3/97 Reply to NOV (EA 93-130)(10/19/93) I 19 Muccino 1/3/97 Letter G. Meyer (USNRC) to B. Kenyon - Meeting on Training (12/6/96) ,

20 Muccino 1/3/97 Company LOUT Written Examinations & Scores for MP-1 Candidates 21  ! Muccino 1/3/97 Previous Exam Results for LOUT,1996, Students 22 Muccino 1/3/97 NRC Written Senior Operator License Exam (12/2/96) 23 Muccino I/3/97 Comments on NRC Exam - P. Hinnenkamp to II. Miller (12/13/96) l 24 Muccino 1/3/97 Student Feedback on,1996, LOUT Program 25 Muccino 1/3/97 Preliminary Written Exam Score Analysis '

26 Muccino 1/3/97 Brief Biographics of LOUT Candidates ,

I m e-s i rennes int

IDENTIFIED DOCUMENT BY DATE DOCUMENT DESCRIPTION -

NUMBER TEAM ENTERED MEMBER 27 Muccino 1/3/97 DPUC Audit of CL&P Nuc. Operations - RC. Brown & Assoc. (12/13/96) 28 Muccino I/3/97 1991 LORT Failure - Corr. Action Plan - E. Mroccka to T. Martin (10/7/91) 29 Muccino 1/3/97 1992 LORT Failure - Root Cause Investytion 30 Muccino I/3/97 1992 NRC LORT Evaluation 50-245/2-23 (I1/2/92) 31 Muccino 1/3/97 1992 NRC LORT Inspection Report 50-245 (12/31/92) 32 Muccino 1/3/97 NRC MP-1,2 & 3 Training Inspection Exit Report (5/12/93) 33 Gilbert 1/13/97 Memo Dan Daigler(VY) to '. / - Sim. Obs.

34 Gilbert I/13/97 PECo SRO Exam 35 Gilbert 1/13/97 PECo SRO Exam 36 Gilbert 1/13/97 Susquchanna SRO Exam 37 Gilbert 1/13/97 Susquehanna RO Exam 38 Muccino 1/13/97 Catawba Operator Training Procedure 3.0 Design and Development 39 Muccino 1/I3/97 Catawba Operator Training Procedure 6.0 Training Programs 40 Muccino 1/16/97 Candidates 1995 LOIT Attendance and Grades 4i Muccino 1/16/97 1996 LOUT Attendance Reports 42 Muccino 1/16/97 Individual Training Program Descriptions for Instants 43 Muccino I/16/97 Candidates License Applications & Waivers (Form 398) 44 Muccino 1/16/97 19% LOUT Company Final Simulator Exam Results 45 Muccino 1/16/97 9/30/96 Memo , , - Resources for LOUT 46 Muccino I/16/97 _ 's Timeline on Events impacting LOUT 47 Muccino 1/16/97 19% LOUT Diagnostic Exam and Results 48 Muccino 1/16/97 Assessment of Quality of LORT Examinations (12/19/96) 49 Muccino 1/16/97 GFS Scores for LOUT Candidates (1/14/97) I 50 Muccino 1/16/97 NTD past Initiatives Status 51 Muccino 1/16/97 1996 Continuing Instructor Training Attendance Matrix 52 Muccino 1/16/97 NETS Assessment of NU & Susquehanna Exams 53 Muccino I/16/97 Unit I instructor Longevity E i E Y i"" rm

t IDENTIFIED t DOCUMENT BY DATE DOCUMENT DESCRIPTION  ;

NUMBER TEAM ENTERED MEMBER I

54 Muccino I/16/97 ACR M i-97-0109 MP-1 Operator Training Weaknesses (1/15/97) 55 Muccino I!!6/97 Misc.1996, LOUT Cormpuisi.cc from StafT 56 Muccino 1/16/97 MP-1,2 & 3 Organization Charts 57 Muccino 1/I6/97 Exam Bank -Taskmaster Printout 58 Muccino 1/I6/97 Task List to Determine OJT. (3/8/95) 59 Muccino 1/16/97 Taskmaster Student Qualification Records f 60 Muccino 1/16/97 NTD Execlience Plan  ;

61 Muccino 1/16/97 MP-1 Ops Training Resource Schedule 62 Muccino 1/16/97 ACR Ml-96-0339 Corrective Action of QA Audit (7/30/96) 63 Muccino 1/16/97 ACR M2-96-0657 MP-2 NTD Procedure Compliance (11/I3/96)  !

64 Muccino 1/16/97 Pilot SAT Training Module (5/2/96) 65 Muccino 1/16/97 MP-1 Scif-Assessments (1/16/96 & 1/18/96) 66 Muccino 1/16/97 Operators K&A's 67 Muccmo - 1/16/97 MP-1 LOIT/ LOUT Class History 1992 ,1996,  !

68 Muccino 1/16/97 1995 NTD Assessment Team Report (4/12/96) 69 Muccino I/16/97 Accreditation Renewal Scif-Evaluation Report MP-1,2 & 3 (7/94) 70 Muccino 1/16/97 1/17/97 Briefing Dm . a." IRT to J. Thayer 71 Muccino I/16/97 Attendance Roster for Simulator Sessions 72 Muccino 1/16/97 Instructor Training Program Impicmenting Proccdure NTM 7.208 73 Muccmo 1/16/97 T. Gilbert Memo to Info. Request (1/l6/97) 74 Muccino 1/16/97 1996 LOUT Student feedback Cuisewvi. dei a 75 Muccino 1/16/97 1996 NTD Organizational Changes I 76 Muccino 1/16/97 August 1996 Simulator Schedule 77 Muccino 1/16/97 NTD Coordinator Scif Review (1/2/97) 78 Muccino 1/16/97 MP-I Department Instruction I-OPS-8.06 Licensing School Selection Process (1/1/97) 79 Muccino 1/21/97 QAS Audit Report MP-1,2,3 & CY Licensed Operator Training 95024845 (4/26/95) 80 Gilbert 1/21/97 ACR Ml-97-0117 NTD incomplete Training Records (1/17/97) 81 Dooley 1/21/97 Notes from 12/6/96 NRC Exit Meeting on SRO Exam 82 Dooley I/31/97 OTBI-3 Operator Training Attendance and Performance Records 83 Deoley I/31/97 OTHI-5 Maintenance ofTraining Records and Files 84 Dooley I/31/97 Memot- . y)- (Candidates) LOUT Program Performance 1I/13/96 N EnTi"" %m

IDENTIFIED -

DOCUMENT BY . DATE DOCUMENT DESCRIPTION NUMBER TEAM ENTERED MEMBER 85 Dooley 1/31/97 NRC Letter to 1 Thayer (LOLTF Exam Results) 86 Dooley 1/31/97 Compilation of policies regarding Management Observation of Licensed Operators 87 Dooley 1/31/97 Root Cause Evaluation: Management Observation of Licensed Operators i

i Nr"-iNYi # rea== mr

. _ . ____-.__m ______m____m_ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ . - _ _ _ . _ _ _ . ----__m_____m a

--.m_ .___ __ - _--____-- _ ___-. ____ _-_ _ _ _ _ __. __m__-. -_____ J

ATTACHMENTIII i "P

NUCLEAR EDUCATION &

TRAINING SERVICES, INC.

FAX: (718) 3981659 VOICE: (718)398 1658 l D. ATE1/22/97 l THIS IS PAGE 1 OF 6 FAX TO Ken Muccino FROM Mike Woram

• SUBJECT Licensing Exam Review t

January 22,1997 '

Mr. Ken Muccino Nonheast Utilities Service Company Millstone Station Waterford. CT

Dear Mr. Muccino:

At the request of the Independent Review Team I have reviewed the rece Reactor Operator Licensing Examination administered at Millstone i Sta ,

8,1996. For an additional basis for comparison SRO licensing exam administered during the same quarter. , ,

In reviewing these examinations, I considered two major criteria:

(1)

Were the questions well writtec., addressing toples appropriate to thejob (2) the examination in the allotted time?Could a candidate with a ,

~

4 l

With regard to item (1), the appropriateness and constmetion reservations as follows: .

nor o

i .*' * {

i lt seems unlikely that the task behind Question 004 is one perform personnel.

=

However, it is unlikely that this rninor structu candidate's ability to address the question.

In general the questions were wordier than they needed to be.

to share the widely held opinion that for a question to be me operational operations story must be told as part of the question. This proble other in factors to rnake the examination unreasonably long. This.

this examination.

Concerning item (2), the length of the exam, I found several causes for con The first. and least significant was the wordiness of the cuestions.

probably not be much cause for concern. Particularly because the censinly not outside the normal range for this indu The second concern had, in my opinion, a very large impact on th noted that twenty three questions most probably required the use of Generally, the use of attached references results in a question th licensing exam because they tend to be very the RequalExam section ofthe Exarniner's Standards (NUREG 102 actually twenty nine questions for which a reference would have be However, six of these were Admin type questions for which n This is roughly the equivalent ofputting a two hour, twenty three i examination in the middle ofthe licensing exam. This leaves only two!

candidates to address the remaining seventy seven questions.

To assess this I performed the following analysis:The .

• 't it is assumed that all questions are appropriate for the evaluation Operators and that the examination is being taken by a hypothetic possesses the requisite knowledge to answer all questions correctly.

Since this candidate can be expected to score 100%, the level o

. assessed by determining how long this candidate would take to compl m.

examination.

To this end each question is given a score of 1 or 2. A score of 1 is g knowledgeable candidate could be expected to answer the question i time than it took to carefully read the question. In nearly all cases s are at the Recalllevel of knowledge.

A score of2 is given ifthe candidate will need to spend additional knowledge from several areas to figure out the answer. These always Higher knowledge level questions.

e The actual scores have debatable meanings in themselves and are mosti comparing mmimtit;ns.

l

.- i Using this scoring system I found that the Millstonc licensing exam had a The audit exam had a score of 129 and the other (non Millstone) licensini  !

score of 142. As such, it is my opinien that based on content alone, the Millstone licensing examination was significantly more time consuming that the other tw cuminations cited. The question by question details ofthis evaluation for boI examinations is shown in Tables I and 2 below.

As a result ofthe three factors affecting the length of the Millstone licens It would be reasonable to question the validity of this examination. The reas exam may have been too discuh to discriminate between a competent and a ,

operator. It is possible that a competent SRO might fail this examination due t it is my understanding that two ofyour best qualified senior operators did ta in question prior to it being administered. It is my further understanding t were 82% and 84% and that they both finished the exam in less than four hours. Th complete a time validation as would have been desirable but does indicate th incumbent SRO to pass this exam.

On the basis of the information that is presently available, i must conclude that th examination was at the highest end of acceptable difficuhy but at this time there that supports a conclusion that the examination was invalid. I must point ou your best qualified personnel score in the low eighties does not, in my opinion, resul j defmitive time validation. Ifyou wished to pursue the validity issue further four result. mainstream incumbent (i.e. competent but not superstar) SROs would g It is my conclusion that while there is reason to question the validity of th cue to its length, the information available suggests that the exam, however dimcult, Any conclusion to the contrary would have to be supported by a much more definiti validation that currently exists

i

• O 1

Yours truly

.. \

i Michael G. Woram  !

1 1 J

' l l

1 l

l WN l

2 I

i l

1

\

l l

1 i

l l

l l

e

. s Table 1. Millstone 1 SRO Licensing Exam - Evaluation Summary Q H Pt R Q H Pt R Q H Pt R Q H Pt R 1 1 26 / 2 51 1 76 / 2

. 2 / 1 27 / 2 52 1 77 / 2 3 / 1 /* 28 / 2 53 1 78 / 2 4 1 29 / 2 / 54 / 2 79 1 5 1 /* 30 1 55 / 2 80 / 2 i 6 1 /* 31 / 2 56 / 2 81 / 2

' l 7 1 /* 32 1 57 / 2 82 / 2 8 33 1 1 / 58 / 2 83 / 2 9 / 2 34 / 2 59 / 2 /

l 84 / 2 /

10 /* 35 /

1 2 / 60 / 2 85 / 1 11 1 /* 36 / 2 61 / 2 / 86 / 2 / l 12 1 37 / 2 62 / 2 87 / 2 13 1 38 / 2 / 63 / 2 88 1 14 1 39 / 2 6-1 / 2 / 89 / 2 /

15 / 2 40 / 2 65 / 2 90 1 16 / 2 41 / 2 65 / 2 91 1 17 / 2 / 42 / 2 / 67 1 92 / 2 /

18 / 2 43 / 2 68 / 2 93 / 2 /

19 / 2 44 1 69 / 2 94 1 20 / 2 45 1 70 1 95 1 21 / 2 / 46 / 2 71 / 2 /

96

/ 2 /

22 / 2 / 47 / 2 72 / 2 97 / 2 /

23 / 2 / 48 / 2 73 1 98 / 2 /

24 / 2 49 1 / 74 / 2 99 1 /

25 / 2 50 / 2 75 un 1 / 2 E 14 37 '10 E 19 44 6 E 18 43 4 E 18 42 9 l H= Higher Level Question mammmemesme-una R= Reference Required

"= Reference Not Supplied E 69 166 29 Pt= Time /Difliculty Score

i Table 2. MiHstone 1 SRO Audit Exam - Evaluation Summary i

! Q H Pt R i

! Q H Pt R Q H Pt

• R Q H Pt R j 1 / 2 26 / 2 / 51 1 76

! 1 i* 2 / 2 27 / 2 52 1 77 1

{ 3 / 2 28 1 53 1 / 78 1

{.

4 / 2 29 1 54 1 79 1

5 / 2 30 1 55 1 /

i 80 / 2 6 / 2 31 1 56 1 81 / 2

7 1 32 i 1 57 1 82 1

{ 8 1 33 / 1 58 1 / 83 / 2 /

9 / 2 34 ._

1 59 / 2 / 84 / 1 /

10 1 35 1 60 / 2 85 1

! 11

1 36 ,

1 61 1 86 / 2 i 12 i

1 37 / 2 62 i 1 87

• 1 l 13 1 38

! 1 / 63 / 2 / 88~

i 1 /  !

14 1 39 1 64 1 89 1 15 1 40 1 65 / 2 / 90 1  !

16 1 41 1 66 1 91 i

1 /

17 / 2 42 1 67 / 2 92 / 2 l 1B 1 43 1 63 / 2 93 1 19 44 1

/ 1 69 1 94 1 /

20 1 45 1 / 70 1 95 1

, 21 1 46 1 71 / 2 96 / 2 22 1 47

) 1 72 / 2 / 97

- . _ 1 1

23 1 / 48 1 / 73 / 2 98 1 24 / 2 49 1 74 / 2 99 1

'S 1 50 i 1 75 1m i

1 / 2 I 9 34 E i .

1 5 28 4 E 10 35 7 I 8 32 5 1 H= Higher Level Question numma mamm mumm-4 R-Reference Required 1

"= Reference Not Supplied PPTime/Difliculty Score E 32 129 17

. - . - , - . . -,,-- ~ , . , - - - - -

5 ATTACHMENT IV Evaluation of 19% MP-1 LOUT Issues Against NRC / INPO Requirements -

10 CFR Pan 50: Attachment B - Oeam v Assurance Criteria for Nuclear Power Plants 1

This area iregulation applies to those structures, systems and components that prevent or mitigate the consequences of postulated accidents that could cause undue risk to the health and safety of the public. In the course of the IRT review, the following challenges to the good business practices addressed by 10 CFR Part 50:  ;

] Appendix B were identified.

. Criterion I Organization.

l

{

~

(3) The authorities and duties ofpersons andindividuals performing activities afecting the safety-related i functions ofstructures, systems and components shall be clearly established and delineated in writing.

l The expectations for the following LOUT program principals was not clear - thereby allowing performance that did not result in success: LOUT Coordinator; Operations Manager; LOUT class mentor; Shift Managers.

, Criterion II. Quality Assurance Program. ,

, (4) The quality assurance program shallprovide control over activities afecting the quality ofthe idennpes l 1 structures systems and components consistent with their importance to safety.

(10) Management ofother organizations participating in the quality assurance program shall regularly review

. the status and adequacy ofthat part ofthe quality assurance program they are executing.

Tne LOUT program is of crucial importance to plant safety. Contrary to the above, Nuclear Oversight, NTD and

[ Unit I management failed to provide effective control.

3 Criterion V. Instructions, Procedures, and drawings.

1) Activities affecting quality shall be prescribed by documented instructions, Procedures, anddrawings ofa type appropriate to the circumstances and shall be accomplished in accordance with these instructions, Procedures, anddrawings.

Contrary to the above the follnwing deficiencies were observed with regard to the design and implementation of the 4

LOUT program. KfD procedures were not followed in the design and development of the LOUT program, including:

) . Training Program Analysis and Design.

l e Training Records.

. . Curriculum Control.

Criterion VI. Document Control.

2) These measures shall assure that documents including changes are reviewedfor adequacy and are distributed to and used at the location where the prescribed activity is performed l

Contrary to the above, Unit 1 OTB failed to routinely incorporate plant design and procedure changes into trammg materials.

I j 4 a

t ses It Falwei IRT

l j

l l

ATTACHMENT IV l 1

Evaluation of 1996 MP-1 LOUT Issues Against NRC / INPO Requirements (cont.) l Criterion XVL Corrective Action.

1) Measures shall be established to assure that conditions adverse to quality, such asfailures, malfunctions, 1 defciencies, deviations, defective material and equipment, and non-conformance are promptly identifed and l

corrected 1 Contrary to the above, numerous cases were identified where the Self Assessment / Corrective Action process has not been used effectively to identify and correct weaknesses in trammg.

Criterion XVII. Quality Assurance Records.

1) Sufcient records shallbe maintained tofurnish evidence ofactivities afecting quality.

3) The records shallalso include closely relateddata such as quah) cations ofpersonnel, procedures and equipment.

Contrary to the above, Unit 1 OTB was unable to produce records supporting that candidates had completed all phases of the LOUT program. --

10 CFR 50.120 Training and Oualification of Nuclear Power Plant Personnel.

This regulation directs the licensee to establish, implement and maintain a trammg program derived from a system's approach to training. The following challenges to 10 CFR 50.120 were observed.

10 CFR 50.120 (2).The trainingprogram must beperiodically revised to re)ect industry experience as wellas changes to thefacility, procedures, regulations and quality assurance requirements.

Contrary to the above, Unit 1 OTB failed to routinely incorporate plant design and procedure changes into trammg materials. '

10 CFR 50.120 (2). Suffcient records must be maintained by the licensee to maintain program integrity and kept availablefor NRC inspection to verify the adequacy ofthe program.

l; Contrary to the above, Unit 1 OTB was unable to produce records suppordng that candidates had completed all phases of the LOUT program. It also failed to document the analysis leading to the LOUT program revision.

1

e o ATTACH ~ MENT IV Evaluation of19% MP-1 LOUT Isrues Against NRC / INPO Requirements (cont.)

4 ACAD 91-015 The Obiectives and Criteria for Accreditation of Training in the Nuclear Power Industry.

, This document establishes the standards that must be followed to achieve and maintain accreditation by the National Academy of Nuclear Trauung. During the course of the IRT investigation several cases were identified where these objectives and criteria were challenged. The purpose of this summary is to provide representative l examples of these challenges. It is not an all encompassing accreditation evaluation. Management should conduct 4

a self evaluation of these items to ensure high standards of traming are being met.

Training Program Content Criteria 1.1 Industry guidance and associated training materials are used in conjunction with a systematic developmentprocess to establish and maintain trainingprogram content.

Feedback from training program coordinators suggests that trammg is more focused on passing the NRC exam than on effective operation of the plant.

Organization and Management of Training --

Criteria 2.10 Personnet entering initial training programs possess expected entry-level knowledge and skills, and experience.

a Feedback from several candidates indicated that their systems knowledge was not at the level assumed by the program.

Development and Qualification of Staff Criteria 3.1 Training managers, instructors, andprogram developmentpersonnelpossess and maintain the educational, technical and experience qualijications requiredfor theirpositions.

The inexperience of the Unit 1 OTB staff and supervision were identified as a casual factor to the event.

Analysis, Design and Development.

Objective 4, A systematicprocess is used to determinejob performance requirements, specify trainingprogram content, prepare supporting materials and maintain the trainingprogram.

NTD procedures were not used to modify training programs in compliance with the SAT.

Conduct of Classroom and Individualized Instruction and Trainee Evaluation.

Criteria 5.1 Individualizedinstruction when usedprovides trainees with sufficient guidance andsupporting materialsfor achieving the learning objectives.

Candidates commented that the individualized remediation actions taken were ineffective.

Conduct of Simulator Training and Trainee Evaluation.

Criteria 7.2 Instructors areprepared to deliver efective and consistent (simulator) training.

Some candidates commented that simulator trainmg was inconsistent from day to day regarding instructor expectations.

Systematic Evaluation of Training Effectiveness.

Criteria 8.4 Change actions (e.g. regulatory changes, changes inJob scope [e, results ofevaluations and inspections procedure changes plant system changes) are evaluatedfor applicability to initial and continuing training.

Contrary to the above, Unit 1 OTB f' ailed to routinely incorporate plant design and procedure changes into trammg materials.

rm! Draft . II Fa'omary.1991 1996 MP.I LOUT NILC f.mammanon Faawes DtT

1 l

1 l

ATTACHMENT V  !

REVIEW OF NTM TRAINING PROCEDURES 1

Modification to the accredited licensed operator trammg program was based on the proficiency of the candidates )

i and time available before the NRC exam Changes to accredited programs must be made in a systematic manner. i A review of NTM trammg procedures was conducted to determme the adequacy of these procedures to support and guide training program revision.

Based on NTD guidance the following procedures should have been followed.

- NTM414 Validation of his procedure provides a systematic method of evaluating the 1 Equivalent Training candidates previous knowledge and skills. This analysis would l then be used to determme the candidates' experience and l

knowledge with the intent of determmmg what training is needed '

This procedure was not implemented for this program NTM1.15 Individualized This procedure provides a systematic process to develop Training Curriculum individualized trammg programs based on the analysis conducted in NTM 1.14 Validation of Equivalent Trauung This plan is then followed when pursuing qualification. While these plans were developed, they were not completed in all cases.

NTM2.01 Evaluate / Dis procedure provides detailed guidance on the steps to be DevelopJob Performance followed when changmg the design of a trammg program. This Data procedure was not implemented for this program NTM101 Evaluate / Procedure NTD 2.01 directs the instructor to this procedure when Develop Learning Objectives chaging the design of trammg programs This procedure discusses the method of evaluating learnir4 objectives and details what products have to be developed in this process. His ,

procedure was not implemented for this program. l

Conclusion:

Established NTD procedures provide sufficient guidance and oversight to modify trammg programs in a systematic manner. These procedures were not used in the 1996 LOUT program l

l 996 i Feihase titT

i 4 9 d

ATTACHMENT VI

19% LOUT Program 2

Barrier and Failure Mode Analysis  !

Barr6er Failure or P " y 1st imel 2nd Level  : 3rd Level Tms Sup. Tmz Mamt ops Mamt "

systems approach to training breaks down ID ID, CA ID,CA Cl ID.CA.Cl C1

, LOUT rc~t f. cation was dnvon by available tune in wtuch MJ, M3 MJ,M3 M3 P1 to conduct the training.

Procedures were not used to guule revuton ID CA C1 Cl On-Stuft tan = cut from 13 wks to 4 wks w/o reference to IDI MJ IDI SK accreated c - = - -

l Self-Study ~.,~, a not implemented per plan. MJ MJ IDI Cl 1 Some can& dates announced only davn before class start. SK SK MI MJ Target Audience changed ulthout program modafication. SK MJ IDI SK Inexpenance of ongmal LOUT Coordinater not IDI Cl ID recognized by managemenL 4 Ops Traming management was not cosmr. ant of the IDI magrutude of program changes incorporated Two can& dates required to attend GFE simuhaneously SK SK Cl M3 with LOUT.

Required assessments of trauung by Ops and ops Tmg CA_ CA, Cl Pl. C1 Cl marnt were not conducted .-

The Ops Dept. Mentor (Stuft Manager) conducted ordy C1 C1 Pl MI Cl one observation of training a

The Cps laasson position was open throughout the class. Cl Cl P1, Cl Inadequate exam bank sent to NRC, SK SK C1

, Sr. mgmt faaled to recogmze the effect of personnel Cl Cl i changes. '

Feedback from candidates and Coordinator was not acted MJ Cl P1,C1 on 1

Andemic Review board requested by Coordmator was not MJ Cl Cl held.

Issues identified at one umt are not always reviewed for ID Cl,C3 C1 C3 Cl l 1 apphcabihtv at other units.

The existmg computer program is not used to track student CA, ID, ID CA, Cl qualifications.

Can& dates informed ofprehmi.2a.y failures prior to SK SK IDI completing operations exam j Exam teview are vahdauon process was ineffective. SK SK IDI Inconsistencies were noted on the trackmg of student ID ID C1 C1 attendance and OJT time.

, There are no TPCC Meeting Mmutes approvmg changes ID CA,ID ID, Cl ID,Cl Cl to the thUT Program Many systern texts are out of date. IAsson plans do not CA,ID ID C1 C1 Cl follow text organization The following partsal hst of FPI Failure Codes was used to group the major failures of the 1996 thUT program These groupmss were performed to identify common causes and themes of the evenL a

i Freemenev of Occurrence -

FPI Code Total 1st Barrier 2nd Barrter 3rd Barrier Description of feBure snode Cl 36 29 7 Icaha'e y.ifo,- =- -ah ngi and trendmg ID 17 6 11 Insttention to detail SK 12 6 6 tah=" skills or knowledge CA 10 2 3 Commmed actsons not carned out j MJ 9 2 7 Misjud_smas IDI 7 1 6 Unawareness P1 5 5 ta-h-te aneraion to tra"-

M3 4 1 3 Cwa.e& Mons

]

C3 2 2 Inadequate root and c~a-a cause analwis MI 2 2 Vague or unclear expectations mal Drus . Il February,19F7

ATTACHMENT VII . ,

Matrix of Conclusions and Casual Factors to Corresponding Recommendations TABLE VHa: Team Conclusions, referenced to Recommendations IRT Camelusian Reconumeindenen -

4.1.2. A Managemeed oversight of the 1996 IAUT program was ineffective in 4.13.A. Review industry gualance and INPO identified "twst perfonners" with respect to manap,ement oversight. Adopt best identifying and correcting challenges and deficiencies in the presenting of practices into oversight procent qualitytraining. M- ~ I missed numerous opportunities to ensure that training delivery wrs ofthe highest quality. 4.1JD- Reinforce with Training and Operations managemend their . , ' Zities regardmg oversight oftrauung. ,

4.13.C- Enablish high expectaHons regardag oversight and implement niethods to ensure that oversight is doeurnerned, frequent, and useful ,

4.1.2.B Management failed to connder previuw.ly idcr::if. d .-. ,*,:ir.isa 4.13.D- Ensure that oversight feedback is communicated and addressed consneently across all uruts (across all units) as "warmng flags"and adjint its actions appropriately.

4.2.2.A 1he Self Assessment and Corrective Action process has not been used 4.23.A- Review mdustry guidance and INPO identified "best performers"with respect to self- assessment and corrective action.

effectively to identify and correct weaknessen in training. Adopt best practices in the NTD self assesunent and corrective action process.

4.23.B- Conduct an

• urgency review' of previously identified problem areas and expedite close out of those relating to accrechtseinn and / or regulatory issues.

4.23.C- Ensure corrective actions are developed and tracked for allidentireed training issues.

4.23.D Review effectiveness ofpast corrective actions. Incorporate this review into the self assessment process.

4.23.E. Develop, communicate and remforce clear expectations regardmg the Self Assessment and Corrective Action process for all i levels of . . 7, supervision and staft ES. 4- Conduct reviews ofpreviously identified problem areas and expedne ekse- out ofthose relating to accreditation, regulations, ,

and training quality. This should include re- evaluating priorities and resource -

_ against other Nuclear Training Department excellence initiatives.

ES Implement an efrective corrective act%n program which includes accountability for closure in a timely and elrective manner.

4.2.2.D The Self Aseessment and Corrective Action procesa does not 4.23.D. Review effectiveness of past corrective actions. Incorporate this review into the self assessr ent process.

systematiceMy assess past problems to ensure the adequacy and sunninability ofpast corredive och 4.23.E Develop, ccmmumcate and reinforce clear expectations regenhng the Self Assessment and Corrective Action process for all levels of management, supervisions and stafE ES Conduct reviews ofpreviously identified problem areas and expedite close- out of those relating to accreditation, regulations, and training quality. This should include re- evaluating priorities and resource comnnements against other Nuclear Training Department excellence intiatives. g ES Implement an effective corrective adian program which includes accountability for closure in a timely and effective manner.

i 5

anal Drea - tI Febnery,1997 1996 hep-t LOUT NRC E Fashnes IRT

. . . - . _ . - . . . - - - - - . - - . - - - - . . - _ _ - - - - - - - - - . .- . . - _ ~ . - - . - - .

t 1 TABIE Vila: Team Conclusions, referenced to Recommendations

~

. laT c.netamw. mece mnendaewn 4.2.2.C The Self Assessment and Corrective Actica process nimany cases focuses 4.23.D- Review effectiveness of past coevective actions. hicorporate this review into the self assessment pecess. }

on a single plant and does not consider ifproblems identified in one unit

,, g % 4.23.E- Develop, commumcate and reinforte clear ----- . _ regarding the Self Assessment and Corrective Assion process for all levels of. . , , . supervision and staff. ,

ES ConAnct reviews ofpreviously identified problem areas and expedite close- out of those relating to - . I and training slity. This should include re- evaluating priorities and resource conumaments against other Nuclear Training Department excellence initiatives.

• ES Implement an effective corrective action program which includes -x ty for closure in a timely and efferteve manner.

4.2.2.D Some ======nenes in the past, inchsdag the accrediention self evaluation 4.23.E- Develop, commumcate and remforce cl8r expectations regardag the Self Assessment and Corrective Action process for all report, did not include line management ownership. levels of .

supervinon and staff.

i 43.2.A 1he onguial accredited prograni wee revised without sufficient attention 433.C- Mancgement and supervision should ensure - - with existing procedures, l I

placed on accreditation comnutmeses.  !

433.D Evaluate the as taught UXfr prograrn against accreditation standards and ensure that all accreditation comumaments were  !

l "'t i

43.2.B NTD procedures weet not used to ensure a systematic process was 433.A- Evaluate exuting NTD procedures and instructions (OTBI's) and emure these procedures meet NTD . , for the i followed training process.

433.B- Revise NTD procedures and instructions (OTBI's) as respurvd,

, 433.C- Management and supervinen should ensure m , - with existag procedures 4 3.2.C The analyses and needs assessment for prograrn redesign was not 433.F Emplement the reviously existing plan for revieweg the Systems Approach to Training as part of the 1997 Conhnusng documented and th refore cannot be verified. Instructor Training program 43.2.D The NTD staff mvolved with desipi and revneon were unaware of 433.C. Management and supervuson should ensure - , , with existuig procedures.

accreddation com,nismuits.

}

433.E. Develop and implement corrective actions for the issues identified in the recently performed internal assessment of accreditation crieeria.

43.2.E Evidence in weakness in _. ^ .and applymg the SAT is evident 433.A- Evaluate existag NTD procedures and instructions (OTBI's) and ensure these procedures meet NTD . - for die i elsewhere in the NTD. training process.

433.B. Revise NTD procedures and innructions (OTBI's) as resynred ,

t 433.E- Develop and : ,' - corrective actions for the issues identified in the recently performed internal assessrnent of i accreditation criteria.

4.4.2.A The frequency ofchange for personnel associated with Urut I operations 4.43.A. Management should consider the use of!Np0 Imances or adustry experts to funchen in the Umt i OTB chain of command ,

Training and the difficulty enconnemed in retamuns qualified and until expenence level of the incuenbents can be raised to en appropnete level. i experienced staffis clearly affecting the ability of NTD to deliver quality '

4.43.B- Management should review current conditions and practices which result in difficulty of NTD to retain qualified and exponenced staff.

l l 4.43.C- M__,, s hould develop and implement an aggressive sisccesnan plan for all key positions in treuung. (Training Coordmater and above) 4.43.D- Training management should take a proactive stena in emunng that transition and tumover for individuals being promoted or transferred is completed.

t l

W Drea 11 February.1997 sve6 MP-1 LOUT MnC E Feeuws laT

TABIE VIIA: Team Conclusions, nferenced to Recommendations .

IRT Conclusion Rec. .... "a81on 4.4.2.B De succession planning process is ineffective. 4.43.B- Management should review current conditions and practices which result in difficulty of NTD to retain quahfied and expenenced staff.

4.43.C- Management should develop and implement an aggressive succession plan for all key positions in training (Training Coordinator and above).

4.43 D- Training management should take a proactive stance in ensuring that tramition and tumover for individuals being promoted or tramferred is etwnpleted.

4.5.2.A Lack of a procedurahzed selection process resulted in some margmally 4.53.A- Operations Management should formahze the SRO candidate selec* ion process similar to its recent instruction for RO motivated and qualified candidates enterittg the 1996 LOUF progrant candidate selection. De process should include involvement and ciput from NTD.

4.53.B. Benchmark industry *1 est performers" and develop a formal process to assess the candidates' readiness to take the simulator exam.

4.5.2.B Candidates

• readmess to take the NRC exammation could have been more 4.53.C- Semor Management should reinforce its expectations of Training Department and Unit personnel in fulfilling their thoroughly and objectively assessed. respomibilities for thoroughly assessing and remediating the performance of ficense candidates.

4.6.2.A Personnel and program records cannot be cuitently verified as being 4.63A Ensure that the investigation associated with ACR M; 0117 venfirs that all hccmed operator training records are m complete and accurate. His presents a challenge to 10 CFR 50.120,10 compliance w;$ 10 CFR .ev;... ..u.

CFR 5 5 and QA ..v.........:

4.63.B. Implement the approved records management system.

4.63.C. Implement extemal program auditing at a frequency appropriate ensure record retention requirements are being met.

4.6.2.B Currently two computer programs exist to track personnel and program 4.63.B- Implement the approved records management system.

records - neither is being used at this time.

4.6.2.C Dere was inadequate oversight of personnel and program records by 4.63.A- Ensure that the investigation associated with ACR Mi-97-0117 verifies that all hcensed cperator training records are in training and unit management as well as the oversight organization. compliance with 10 CFR av;. - .

4.63.C- Implement external program auditing at a frequency appropriate ensure record retention requirements are being met.

4 6.2.D Operator Trainmg Branch staff, supervision and management are not 433.C- Management and supervision should ensure comphance with existmg procedures following established procedures to maintain records.

4.7.2. A De conddion of training material does not support a program that heavily 4.73.A- Review and revise all training material to include PDCR and procedure changes.

depends on self-study of systems and administrative topics.

4.73.B. Evaluate this issue of training material quality for applicability to aII training programs.

4.73.C- Management should emure the requirements of NTM 1.04, Commitment Tracking are being met.

4.73.D- Ensure that instructional materials are kdequately upgraded and maintained before being used in training.

4.7.2.B Commumcations between the line organization and training have broken 4.73.B- Evaluate this issue of training material quahty for applicabihty to all training progrsms.

down in regard to plant design changes and A &

4.73.C. Management should emure the .y;.... ..; of NTM 1.04, Commitment Tracking are being met.

4.73.D- Ensure that imtructional materials are adequately upgraded and maintained before being used in training.

4.73.E. Develop and implement a process that ensures that applicable PDCRs and procedures changes are communicated to NTD in a timely manner.

sulDr=ft - It Fetery.1997 1996 MF.I t.OUT NRC Ex - Faihees tRT

._. ~ _ . ._..._ _ _. _ . - . . . _ _ _ . ._ . ~ __ ._ _ _ _ _ _ . _ . . .. _ . _ _ . _ . ._

TABLE VHa: Team Gmelusions, referenced to Recommendations IRT Conctuelan Reconnernemeten 43.2_C . Iack of procedural . __ m for - tracking on MPI has lead 433.C- Managemers and supervision should ensure corsghance wdh emnting proceshnes. '

to deyedation oftreuung nimlerial spaality.

4.8.2.A The emani had provuled to die NRC dal not nieet the guidelmies of 4.83.A Uppade erans bank to indualry standards Inglement NTM 4.01 to emure exani aquestions are properly consenscsed and I NUREG/BR4122 "Exasumers Hanecok for Ikveloping operator docuenented.

Iicensing Writsen Examnistions"and could not be used by the NRC.

4.8.2.8 tack of a formahzed exam review and vetulation process corsinbuted to an 4.8.3.B- Denchmark undustry "hst performers" and adopt practices into a fonnalized process for the review and validelson of NRC .

usefective review and vahdation of the NRC exam. exams. Provide expectations and treuung to individuals scheduled for exam review and validation. .

4.8.2.C Lack of licensing emane review expenence by revieviers cominbuted to an 4.8.3.C. line Managemeed should incorporate exam functions into their resource schedehng to ensure that appropnete personnel .i ineffective review and validation of the NRC exam. who are representative of the target medience are available for NRC exam review and validation.

4.8.2 D IJne osinerslup of the exam review and vaHdation process needa 4.83.C. Line Management should incorporate exarn functions into fleeir rescence scheduhngto ensure Gast apprepnose personnel unprovement who are representative ofIhe target eenhence are available for NRC exam review and validation.

4.8.2 E The review and wahdation process took place simultaneously during the 4.83.D Sepermee the review and validation of NRC exams pr- so that they do not occur concurrently.

same week, thereby reducing the effectiveness of the review and wahdation Proce m  ;

i I

I i

i inaltholt . Il Fatsiery,1997 19s6 R4P.IIDUT pfRC r Fehmes IltT

_ _ . _ _ - . . _ . , . _ ____.__.___.___.._____________..__________________________m_ _ _ . _ , . _

ATTACHMENT VII ,

Matrix of Conclusions and Casual Factors to Corresponding Reconnnendations Attachment VH.B CasualfactorsmatrixedtoapplicableRecommendations Causal Facter ,R-- "

s)

Systems approach to treuung tweaks down. ES- 2. Conduct m<lepth reviews for applicainlity ofissues from this investigation to other traimng proyams for all Mdistone Umts. Consider temporary"stop-work" ,

orders for those programs ieternuned to be up-~ data i

ES 4- Conduct rewews of prenously identified problem areas and expedite close- out of those relating to accreditation, regulations, and training spsality. This should  !

include re- evaluating priorities and resource conumtments against other Nuclear Training Department excellence instentives.

j ES Implement an effective corrective action proyarn which includes accountability for closure in a timely and effective manner.

ES Senior _ . ^ should clarify and reinforce expectations ofTraining and Operations . regarding their responsibilities for oversight of trainig linplement snethods and measures to ensure that oversight is effective.  ;

4.13.B- Reinforce with Training and Operations .. . ---> their responsibilities regarding overwght of treuung. l 4.13.C- Establish high expectations regardmg oversegli and implement methods to ensure that oversigid is documented, frequent, and useful 4.13.D- Ensure that oversegld feedback is - .Med and addressed consistently across all units. j 4.23.B- Conduct an ' urgency review' ofprenously identified problem areas and expedite close out of those relating to accreddation and / or regulatory issues.

4.23.C- Ensure corrective actions are developed and tracked for all identified training issues.

4.23.D. Review effectiveness ofpast cor ective actions. Incorporate this review into the self ==== ament process.

4.23.E. Develop, communicate and reinforce clear expectations regarding the Self Assessment and Corrective Action process for alllevels of.. .

supervision and staff. i 433.A- Evaluate existing NTD procedures and instructions (OTBI's) and emure these procedures meet NTD expectations for the training process. ,

433.B Revise NTD procedures and instnictione (OTBI's) as requirect 433.C- Management and supervision should ensure-- __, e with existing procedures.

I.DUT noodification was driven by available tune in which 4.13.B- Reinforce with Trairung and Operations _ _

^

their responsibilities regardag overseglut of trainmg  ;

to conduct the training.

4.13.C. Establish high expectations regardag oversegid and implement methods to ensure that oversigid is documeided, frequent.

}

433.A- Evaluate existing NTD procedures and instructions (OTBI's) and ensure these procedures meet NTD expectations for the training process.

433.B- Revise NTD procedures and instructions (OTBI's) as required.

l [

433.C- M .

and supervision should ensure compliance with existing procedures. ,

433.D- Evaluate the as taught IJOUT program against accreditation standards and ensure that all accreditation conumtments were niet.

Procedures were not used to guide revision. 433.C. Management and supervision should ensure . . -- with existing procedures.

On-Shift tune cut from 13 wks to 4 wks wfo referer#.o 433.D- Evaluate the as taught IJMTI prog-am against accreditation standards and ensure that all accreditation - were enet.

accrea d_

Self-Study cornponent not i " per plert 433.D Evahrate",e as taught IJOt.fr prograrn agment accreditation standards and ensure that all accreditation were snet. t I

inal Drett - 1I Febmuy. l997 1996 MP-t t,Otfr NRC P Feines tRT

. . _ ._ _ _ _ . _ . _ _ _ _ _ _ . _ _ _ . . _ _ _ . _ _ . _ _ _ _ _ _ _.m __ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ . _ _ . _ __.

Attachment VH.B Casualfactors matrixedto applicable Recommendation) (continued) .

Causal Factor

. Recommendation (s)

Some candidates announced only days before class start.

ES- 6. Semor management should clarify and remforce expectations of Training e ed Operations management regardmg their responsib lit.es for oversight of training Implement methods and measures to ensure that oversight is effective.

4.13.B- Reinforce with Training and Operatims management their responsibilities regarding oversight of training.

4.13.C- Establish high expectations regarding oversight and implement methods to ensure that oversight is documented, frequent, and useful.

4.13 D- Ensure that oversight feedback is communicated and addressed unsistently across all unit:

Target Audience changed without spa modification. ES Senior manay.. ..; should clanfy and reinforce expectations of Training and Operations management regarding their responsibihties for oversight of trairung.

Implement methods and measures to emure that oversight is effective.

4.13.B- Reinforce with Training and Operations management their responsibilities regarding ovmight of training.

4.13.C. Establish high expectations regarding oversight and implement methods to ensure that overs:ght is documented, frequent, and useful 4.I3.D- Ensure that oversight feedback is communicated and addressed consistently maoss all units.

Inexpenence level of the ongmal IOUTi vogram Coord 4.43.0- Traimng managernent should take a proactive stance in ensurmg that transition and turnover for individuals bemg promoted or transferred is completed.

not recognized by management.

Ops. Traimng manag..-..; was not cognizant of the 4.I3 B- Reinforce with Training and Operations managemerd their responsibilities regarding oversight of training.

magnitude of changes incorporated into program.

4.13.C- Establish high expectations regarding oversight and implement methods to ensure that oversight is documented, frequent, and useful.

Two candidates required to atter'd GFE simultaneously 4.13.11- Reinforce with Training and Operations management their responsib hties regarding oversight of training.

with LOUT.

4.IJ.C. Establish high expectations regarding oversight and implement methods to ensure that oversight is documented, frequent, and useful.

Required -- - - - of trammg I y Ops and Ops. 4.13.B- Reinforce with Trainmg and O erations management their responsibthties regardmg overs!ght of training.

Trsining management were not conducted.

4. I3.C- Establish high expectations regarding oversight and implement methods to ensure that oversight is documented, frequent, and useful ne Ops. Dept. Mentor (Stuft Manager)cc,Jeted only 4.13.B- Reinforce mth Trammg and Operations management their regensibihties regarding oversigid of training.

one observation of training.

4.13.C- Establish high expectations regarding oversight and implement methods to ensure that oversight is documented, frequent, and useful.

The Ops. Ijaison position was open tuya the time of 4.13.B- Remforce with Training and Operations management their responsibilities regarding oversight of training.

the class.

4.13.C- Establish high expectations regarding oversigid and irrmlersent methuds to ensure that oversight is documented, frequent, and useful.

Poor quahty exam bank sent to NRC. 4.83.A- Upgrade exam bank to industry standards. Implement rFM 4.0I to ensure exam e-nestions are properly constructed and documented.

Sr. manar..~; failed to -r,nire the cfTect of m..J 4.43.D. Training management should take a proactive stance in ensurmg that transitied and tumover for individuals being promoted or transferred is completed.

change.

Feedback from candidates and Coosd. was not acted on. 4.!J.B- Reinforce with Training and Operations management their responsibihties regardmg oversight of trainmg, 4.13.C- Establish high expectations regarding ovmight and impicment methods to emure that oversight is documented, frequent, and useful Requested AA..ie Review board was not held. 4.13.B- Reinforce with Training and Operations management their responsibilities regarding oversight of training.

4.13.C- Fstablish high expectations regarding oversight and implement methods to ensure that oversight is documented, frequent, and useful.

-a,,a identdied at one umt are not always .m.-,; for 4.13.B- Remforce with Traimag and Operations meingement their responsibihties regardmg oversight of training.

applicability at other units.

4.13 C- Establish high expectations regarding oversight and implement methods to ensure that oversight is documented, frequent, and usefui.

De existmg m.Wer program as not used to track student 4.63.B- Implement the approved records management system.

qualifications.

inalDran It Fetmsery.1997 1906 MP-1 t OUr NRC Em Failmes IRT

. Attachment Vll.B Casualfactorsmatrixedtoapplicable Recommendations (continued) _

'i

. cw rn or , -

e3

h nfornsed of _ , failures preer to 4.1.3.B- Reedence with Trauung aml operatiera i

c ^ their responsduhtecs regardag overught of %

4.13.C. Establish high expectations regardmg oversight and : ,  %%eds to ensure that overseght is docuerassed, frequent, and usefist.

• Emani ... and vahdation process was ineffective. 4.8.3.H- penchmark industry w perferiners" and adopt practices into a farinalized process for the review and vainiseness of NRC exanes. Provide em send trasneg to individuate scheshaled for exasn review and validation.

W wree noted on Sie tracking of studed 4.63.D. hnplesnent the approved records suenagesnent syneent h and orrtime 1here are no TPCC Meeting Mmunes approving changes 4.13.B- Reinforce with Traines and Operations .- ,,

^ their . , " 'htees regardaig overseWit of treuung.

front thIDUT Progrant 4.1.3.C- Estabhsh high expectations regardmg overught and imphsned nueshads to ensure that overudht is i- ' frapsed,anduseful-Many syseene sexes are out of dues. Lesman plan does not 4.73.D- Ensure that instructional snaserials are adeemsely upgraded and nomintamed before bemg used in trasning. ,

i U.lant Wien. 'ft I

I I  !

I n

i 4

huulDuet . It Fetwaary 1997 1 89s6 MP.IIJDUT NRC r Fadures IRT '

ATTACHMENT VIII I i

PREVIOUS PASS RATES for UNIT 1 INITIAL OPERATOR TRAINING and 1996 NRC EXAM RESULTS 1

i Millstone Unit'l LOIT/ LOUT Statistics Year # of candidates: - # of Licenses obtained Pass rate on the first .

sitting for exam RO and SRO . attempt . i 1993 6 6 100 %

1994 3 2 67%

1994 6 6 100%

1996 7 1 14 %

1996 NRC EXAM RESULTS

. Candidate Status Written Exam Operations Exam : .

Candidate 'A' RO Pass Fail Candidate 'B' RO Fail Fail Candidate 'C' RO Fail Fail Candidate 'D' Instant Fail Pass Candidate 'E' Previous SRO Fail Fail Candidate 'F' Instant Pass Pass Candidate 'G' Instant Fail Fail dT non FeCurus IRT

, IRT PROJECT APPROVAL j MP1 LOIT NRC EXAMINATION FAILURES

,' Charter (Rev. 0)

. Mission, Background, Scope S

. Deliverable e Resource Allocation --

• Schedule l

2 i Prepared by John H. Ferguson An% N hnUhlN Date \11IQR(o i Accepted by Jay K. Thayer ## 4- Date N/f/%

I / / /

b I

i

Independent Review Team Charter i

Mf1 LOUT NRC Examination Failures  !

Revision 0 i

Minnien l i

An IRT has been established to investigate the circumstances related to the failure of 6 of 7 MP1

]

LOLTT candidates on the week of December 2. The team will idFA dy assess the LOUT '

process, NRC exanunation, candidate remimeen self assessment process, and root cause(s) of those failures. 1 The mission of this IRT is to produce a report as described below to be used for corrective action including orgamzational learmng If, in the conduct of the investigation, your team discovers

. signi6 cant conditions adverse to quality that could contribute to the initiation or exacerbation of a )

{

consequential event, you are to recommend immediate interun compensatory measures to neutralk  ;

such threats while permanent corrective action is being formulated and deployed.

The team is requested to produce a report with the following content:

e Executive summary that includes the most important messages to my level and above, spedfic j actions that need to be taken at those levels, and any brief details and elaboration that you ,

believe to be vital to our understandmg of the message and action. 1 A descriptive of the event, including, for every condition and action that wits not right, proper or expected, what in your view would have been the appropriate action or condi: ion.  !

. 1 Principal lessons to be learned by the orgamzation from the event (condition) you are  !

investiganng i

e The factors that controlled the consequences of the event, including 1) the triggering events or conditions that consummad he t problem,2) the pre-existing causal factors that set the stage for the problem and made the situation vulnerable to the event, 3) the factors that exacerbated the event or made the consequences worse and 4) the mitigating factors that kept the event from having more severe consequence:. This section should discuss the underlying causal factors, including all missed opportunities to have either detected, correcta.d or avoided the factors contributing to vulnerability, consummation or exacerbation.

Generic implications, including more of the same and items downstream tainted by each important condition adverse to quality.

• Extraneous conditions adverse to quality; those things found in the course of the event or its investigation that were not right, yet did not contribute to the occurrence or severity of the matter being investigated.

• Previous simdar events and conditions; thch items from experience records and nadema*d memories that indicate similarities in consequences or causation that might reflect on the station's performance in problem analysis or corrective action effectiveness.

Quality and safety impact, including separate and distinct discussions of consequences and sigm6cance

. ._ . ._. _ . . _ _.~ _ _ _ . _ _ . . _ _ _ _ . . - _ _ _ _ _. _ - . __ _

l .. .

P s

[ e Proposed corrective actions including: '

e interim c+g =Wory measums e corrective action for problem effects l

• corrective actions for causes j e corrective action for the generic implications of both the problem and its causes,

. e corrective actions for the self-assessment deficiencies and indapandant assessment deficiencies that allowed the causal factors of their underlying causal factors to lie

, unaddressed by the or-aah=en and l

} e a follow-up plan to momtor, l

• The deployment of corrective actions i

e the effectiveness of the corrective actions in accomplishing their intent, and

• any anticipated or unanticipated adverse side effects of the corrective action.

1 4

The team is encouraged to use the Phoenix manual guidelines. If those methods are not suitable, j l use methods with which the team is familiar. and which you believe to be suitable. Advise me of j the methods you intend to use and the problems with the other methods prior to initiation of team

trammg. Please include this as an attachment to your report.

Backaround

. The potential 6 of 7 LOUT candidates failing the NRC license examination indicates either a -

significant LOUT program deficiency or an unanticipated increase in NRC licensud operator

examination standards or both. - If the unacceptable results are due to an LOUT program failure it could indicate the potential for generic weaknesses in the licensed operator trammg program i Stant I The team shall consider but not limit its review to the followmg issues

i: e Any cause(s) of this event that are common to the causes of previous LOUT failures at NU. If  !

}~

so, why the corrective actions to date did not prevent this event.

l

• Whether there was adequate recognition of potential LOUT failures prior to the NRC i exammation and what actions were taken or should have been take.n.

4 e Adequacy of utility review process to measure and assess LOUT candidates readmess to pass i

, NRC exammation and assume NRC licensed operator duties.

d

. Whether unit management's expectations and training program standards are clearly stated, l

enforced and consistent with industry and regulatory standards.

t~ e Specific aspects of training program which broke down.

• Level of difficuhy of NRC examination compared NRC Region 1 norms.

1 e lhe adequacy of Oversight actions and involvement regarding this incident and earlier indicators

or precursors.

a 4 .

i l 'Ibe =%>=q of trauung, Managernent, self assessment process to develop and implement and

- maintam LOITT perfonnance standards e Any 10CFR App. B violations.

Selection process and qualification process for LOUT candidate personnel and criteria utilized to determine paranti=1 for candidates to pass NRC exam and perform.

[

l e

Adequacy of traming staff (capabilities and resource allocation).

Deliverable i The team will conduct its review and prepare its repon in accordance with NGP 3.15, Root Cause

[ Evaluation Program Any necessary or desirable deviations from company-provided guidance for l

} doing investigations should be documented, since a possible reason for the current event could

• l l involve shonfalls of the guidance given to previous investigation teams The Phoenix manual may

be utilized for evaluation process. All supponing data will be catalogued and ==ia'=iwi in IRT j reference library, s

The team will present its findmgs and recommendations in a draft (Rev.0) report to J. Thayer, J. .

Ferguson and H. Haynes according to the following schedule. Comments / corrections to factual i content will be incorporated in a final version (Rev.1) for approval.

i, j Resource Allocation '

[ Resources Tearn Memb;e l

j Team Imder Ken Muccino (NU) i One (1) NU r ie Thomas Gilben Nucl. Training Dept.

j Four(4) extemal r it's Steven Frve (Duke Power)

Wendell Barron (Duke Power)

[

Charles Sawyer Jr. (Duke Power)

Larry Dooley - Consultant (previously Asst. Train.

1-Mgr. Pilgrim Station)

Event Evaluation Trammg & Suppon William Corcoran - Consultant -NSRC

, Team Work im+ian Team Skills /Knowi~laa Tramma Building - Conf. Rm. 5 BWR basics NRC examination Personnel evaluation Process methods Resource utilization Training effectiveness measurement and methods Training standards setting and maintenance Training oversight Team leadership

_ Phoenix or FPI process controller i

.. - . . . -. - - - - - ~. ..-- - -.- - . . - . - . - . ~ ~ - . - . . . - - - . .

4 i

i

Fmecentians / R*enannihilities of T*== Leader l

• Perfonn adnunistrative and managerial duties necessary for an efficient and effective review.

Ensure team is comprised ofqualified subject-matter experts to accomplish mission. I

] Develop charter, agenda, schedules, interview lists and other guidance.

i e Ensure that appropriate background documentation, trauung and resources are provided to l achieve a successful outcome

e Ensure evaluation and report are conducted in accordance with NGP 3.15 and related

procedures and =*thMalogies.

Ensure that evaluation includes only peninent, factual, suberantiatad and venfiable findmgs, conclusions and i=- = E:ons.

5 e Work with line management to ensure corrective actions are developed which: I

- address all aspects of the problem (s) i

j. - can be reasonablyimplemented i j - have measurable results Furnish periodic feedback throughout review period to sponsor (s), appropriate line management

and communications representatives. --

ie Schedule 4

l Insk I

Responsible Schedule l Individual Target

, Develop and approve draft chaner John Ferguson 12/19/96  ;

4 i e Allocate resources and team members Jolm Ferguson 12/19/ % l

. . Team orientation / trammg Teamleader 1/12/97 l e Process (W, Cocoran 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />) j i e Team building j e Incident specifics o

• IRT Planning Team 1/13/97

• Develop plan / assignments e Finalize contact /m~ terviewee list i-

• Data documents required 1

j e Interviews / data collection Team 1/13-1/17/97 i e Analysis and Assessments Team 1/14-1/17/97 i

e Report preparation (Rev "0") Team

1/20-1/24/97 i

l e Draft report to J. K. 'Thayer, John Teamleader 1/24/97 j Ferguson and H. Haynes for review 4

l -. Final report presentation Rev. I John Ferguson 1/29/97 (approved by IRT manager) i' , , . , -- ._ _ . . _ , - -

A'ITACHMENTX j

MILLSTONE ONE OPERATOR TRAINING AND QUALIFICATION SEQUENCE ^

IDLD TRRDEEl I A IEETM DERTTWR SENUt NON-LICENSED OPERATOR OPEIW m)R MITIRL TMINWG OLIT) Cf90GITE lIEETUR CffeUITE OPElvlTm

& A &

NLIT FUNDRMENTR_S RERCTOR OPEMTOR ,

CLRSSROOM/SELF SUDY WITIAL LICENSI TMNING (LOIT) 1 WEEK CLASSROOP - 15 WEEKS FUNDRMENTALS EXflH ..

ON -SHFT - 13 WEEKS a SlHULATOR - 15 WEEKS tmgee % SUPERVISORY SKILLS - 1 WEEK (SRO OlLY)

ON-THE-Juts THf1NMG  : IRA.LY GLNLFED 2 - S WEEKS NON-LICENSED T

NLIT SYSTEMS TRRINNG KCT) SENEM REFICTOR LICENSED OPERATOR CLRS RNT RVG. 7 WEEKS / YEAR TM, CfDOM NMM LORT s SENIOR REflCTOR OPERATOR VG. 7 WEEKS /YEFIR SYSTEMS NITIRL LICENS E TMNING (LOUT)

QUALF CRTION CLFSSR001 - 7 WttKS ON-THE-JO3 TMNMG ON-SHFT 13 WEEKS as WEEKS (RVG.) SIMULRTOR - 10 WEEKS

, PLANT WRLKTHRU SUPERVISORY SKILLS - 1 WEEK A

lRA.I.Y EMLFED 18.01 L l FULLY .gdFED SRD!

RRDWSTE MERRTOR TRAIGK OWT - L2 WEEKS UIVG.' 33  :

3 TM919G PF!:GRFM C.FSSIKi:M ON-SHFT MANFIGEfENT SEMINRRS E

e N

. . -. - . . . _ . ~ . ~ .- - . . - . _ . . . . .. .. _ _ _ - . .

• =

l Docket No. 50-213 50-245 .

50-336 l i

50-423 B16253 .

l l

l 1

l i

Attachment 3 Haddam Neck Plant Millstone Nuclear Power Station, Units 1,2, and 3' ,

Summary of Expanded Reviews '

l l

1 i

1 March 1997

l l j U.S. Nucl ar R::gulatory Commission i

B16253\ Attachment 3\Page 1 l 1

Millstone Unit 1 4

Review Summarv i Summary of LOIT/ LOUT Findinas

LOIT/ LOUT program _ requirements, such as on-shift watch standing, OJT, reactivity

manipulations, and other supporting training records, have been reviewed for training

! programs leading to new RO/SRO licenses. For Millstone Unit 1, the classes reviewed 3

were the 1993 -1994 LOIT and LOUT programs; the 1995 LOIT program; and the 1996 l LOUT program. Sixteen current Millstone Unit 1 license holders have been reviewed l

through this process. The preliminary review has determined that discrepancies exist i or records cannot be found to support completion of prerequisites for licensed operator j {

training, or completion of NNECO's requirements to be licensed as an Operator / Senior  !

l Operator. A summary of the discrepancies applicable to current Millstone Unit 1 operator licensees is provided below:

i i 1. Some candidates did not complete the required number of hours of under-j instruction watch standing.

i l- 2. One candidate counted reactivity manipulations in a manner which is inconsistent

! with NNECO or industry standards. Specifically, the deficiency involved taking

( multiple credits for evolutions which by current standards would be counted as a

} single manipulation. After the re-count of reactivity manipulations by current j standards, the candidate still had greater than the minimum required five

manipulations.

{ 3. The OJT records of several candidates are either incomplete or portions are lost.

4. Some candidates completed required watch standing and/or OJT items after  !

submittal of the NRC Forms 398.

4 I.

Actions Taken for LOIT/ LOUT Findinas Millstone Unit 1 management evaluated discrepancies in the initial training of each I

licensed operator reviewed. In one case, an inactive, staff SRO was administratively i restricted from performing licensed duties pending completion of data verification. )

Subsequently, Millstone Unit 1 determined that there is no longer a_ need for this  !

j individual and another staff SRO license holder to maintain their licenses. Therefore, these licences have been terminated pursuant to 10 CFR 55.55(a). In all other cases, it

was determined that the nature of the discrepancies was such that the individuals'  !

l qualifications to stand licensed watches was not called into question. Actions to correct  !

l the program deficiencies identified in the foregoing " Summary of LOIT/ LOUT Findings" l will be completed prior ts restarting LOIT for Millstone Unit 1.

[

s

U.S. Nuclar R gulatory Commission B16253\ Attachment 3\Page 2 Summary of Licensed Operator Reaualification Trainina (LORT) Findinas

)

A review of LORT program activities against the Systems Approach to Training (SAT) I based, accredited training program (used to meet the requirements of 10 CFR 55.59) was conducted between February 12 and February 21,1997. This review evaluated i program administration, content, attendance, evaluations (examinations), and  ;

effectiveness reviews, against the requirements of the NU LORT Program I implementing Procedure (TPIP) and the Unit-Specific Training Program Description.

The review was conducted by a supervisor and an instructor from an Operator Training group other than the one being reviewed.

Approximately thirty discrepancies were documented through an Adverse Condition Report. The deficiencies were categorized by the nature of the required corrective actions, as follows:

l

1. Several discrepancies were administrative in nature rather than departures from ,

program requirements. For example, examination approval signatures were not i dated.

2. Several discrepancies were historical in nature and either did not require corrective  !

action or are resolved by following program requirements on a going forward basis.

For example, validation of equivalent training completed outside the 14-week  ;

attendance grace period.  !

3. Several discrepancies require corrective action prior to the start of LORT Cycle 97-  !

02 and each future cycle. For example, learning objectives not approved by Curriculum Advisory Committee (CAC) as required by the Nuclear Training Manual I (NTM).

F

4. Several discrepancies require corrective actions in accordance with cn approved Corrective Action Plan. For example, current process permits CAC members to approve their own training validations without higher level review.

Based on the discrepancies identified and the required corrective action, the start of Cycle 97-02 LORT instruction has been delayed from February 24,1997 to March 3, 1997. Having completed specified short term actions, Millstone Unit 1 management considers it appropriate to resume implementation of the LORT program, while longer .

term corrective actions and further enhancements are identified and implemented.-

Because of known inaccuracies, NNECO is hereby requesting that the license applications for the seven 1996 LOUT class SRO applicants for Millstone Unit 1 be withdrawn.

U.S. Nuclear Regulatory Commission B16253%ttachment 3\Page 3 l

Millstone Unit 2 4 Review Summarv

Summarv of LOIT/ LOUT Findinas LOIT/ LOUT program requirements, such as prerequisites, on-shift watch standing, OJT, j reactivity manipulations, and other supporting training records, have been reviewed for

training programs leading to new RO/SRO licenses. For Millstone Unit 2, the classes reviewed were the 1995-1996 LOIT and the 1995-1996 LOUT programs. Fifteen current Millstone Unit 2 license holders have been reviewed through this process. The

preliminary review has determined that discrepancies either exist or records cannot be l found to support completion of licensed operator training _ program prerequisites or i elements. A summary of the discrepancies applicable.to current Millstone Unit 2  ;

j operators or SROs is provided below: )

i i 1.' Some candidates did not complete the required number of hours of under-i instruction watch standing. Documented program requirements were 13 weeks on

shift with stated expectations to the candidates of approximately.10 weeks on shift.

Emphasis was placed on OJT card completion and reactivity manipulations versus hours on shift.

2. Some candidates counted reactivity manipulations in a manner which calls into question the proper completion of the requirement to perform five control manipulations which affect reactivity. Examples of this practice included taking credit for manipulations supervised by SRO candidates, and taking multiple credit for evolutions that may only allow for the credit of a single manipulation. In some cases, the preliminary re-count of reactivity manipulations by NNECO's review team may result in the conclusion that certain trainees performed fewer than the minimum required five. No active licensed watch stander has been identified who lacks the minimum five required reactivity manipulations. NNECO continues to assess the completion of reactivity manipulations.

3. The OJT records of several candidates were either incomplete or portions were lost.

4. Some candidates for instant SRO and one RO candidate did not meet all the prerequisites specified in the NU training program. For example, one previously-licerised candidate did not have the minimum six months at Millstone Unit 2 prior to the start of LOIT. Other instant SRO candidates did not complete the specified non-licensed operator qualifications. One staff candidate for RO did not have six-months experience standing watch as a non-licensed operator, as required by the NU program. The appropriate reviews and' documentation to permit such deviations from the approved training program were not completed.

l

i . .  !

U.S. Nucinar Regulatory Commission

B16253\ Attachment 3\Page 4 Actions Taken for LOIT Findinas

[

1 Active watch standers found to have incomplete training were removed from licensed

activities. ' Subsequently, documentation packages were prepared for each operator

! who was removed from licensed duties. These documentation packages provided a l basis for reinstating operators to licensed duties. For watch standers who failed to

complete the required hours of under-instruction watch standing while enrolled in LOIT, l credit was taken for under-instruction watch standing completed after NRC examination

! (a Unit-specific requirement for watch stander qualification), and for actual watch hours as a licensed operator under the supervision of a senior licensed operator. For operator licensees who were' missing one or more OJT signatures or cards, the training was completed. Additional program prerequisite or completion discrepancies  !

subsequently were identified for several watch standers. Accordingly, these individuals '

were removed from licensed duties and will be returned only after proper disposition of k the deficiency.

I An administrative control is in place to prevent inactive licensees with training -

discrepancies from being returned to active status prior to the proper disposition of the training discrepancies. l l

Management had previously deferred the start of the next Millstone Unit 2 LOIT j program, in order to upgrade training materials and processes to industry standards.  !

Actions to correct the program deficiencies identified in the foregoing " Summary of LOIT )

Findings" will be completed prior to restarting LOIT for Millstone Unit 2.

Actions Taken for LOUT Findinas 1

Active watch standers found to have incomplete training were allowed to remain on-shift. Post-NRC examination under instruction watch standing combined with actual watch hours as a licensed senior operator resulted in on-shift experience in excess of training program requirements. OJT completion discrepancies were completed, or determined to have been completed during the training program.

Summary of Licensed Operator Reaualification Trainina (LORT) Findinas A review of LORT program activities against the SAT based, accredited training program (used to meet the requirements of 10 CFR 55.59) was conducted between February 12 and February 20,1997. This review evaluated program administration, content, attendance, evaluations (examinations), and effectiveness reviews, against the requirements of the NU LORT Prog'am r implementing Procedure (TPIP) and the Unit-Specific Training Program Description. The review was conducted by a supervisor and an instructor from an Operator Training group other than the one being reviewed.

1

, vew ---,>m - e

U.S. Nuclsar Regulatory Commission B16253\ Attachment 3\Page 5 Approximately twenty-five discrepancies were documented through an Adverse Condition Report. The deficiencies were categorized by the nature of the required corrective actions, as follows:

4

1. Several discrepancies were administrative in nature rather than departures from program ' requirements. For example, simulator training crew are limited to 5 personnel, except for intact operating shifts, which are not limited and may be large based on current plans for 1997 training.

2. Several discrepancies were historical in nature and either did not require corrective action or are resolved by following program requirements on a going forward basis.

For example, . the TPIP requires that recommendations for removal from licensed activities be formally documented. However, such recommendations had been routinely made by telephone notification of line management. ,

3. Several discrepancies require corrective action prior to the start of LORT Cycle 97-02. For example, the approved Training Program Description was specific to the two-year program for 1995-1996.

4. Several discrepancies require corrective actions in accordance with an approved  ;

Corrective Action Plan. For example, the memo reporting completion of the biennial review of the task list does not contain details as to how the review was conducted.

Based on the discrepancies identified and the required corrective action, the start of Cycle 97-02 LORT instruction was delayed from February 24 to March 3,1997. Having completed specified short term actions, Millstone Unit 2 management finds it appropriate to resume implementation of the LORT program, while longer term corrective actions and further enhancements are identified and implemented.

l i

_ . _ __ _ _ _ _. _ _ _ _ . _ _ _ . . . _ _ _ . _ _ . . _ . _ ~ _ _ _ _

U.S. Nucitar Regulatory Commission

B16253\ Attachment 3\Page 6 2

Millstone Unit 3 1 4

Review Summarv ,

i l Summary of LOIT/ LOUT Findinas I I LOIT/ LOUT program requirements, such as prerequisites, on-shift watch standing, OJT,  ;

reactivity manipulations, and other supporting training records, have been reviewed for l training programs leading to new RO/SRO licenses. For Millstone Unit 3, the classes j

, reviewed were the 1994 and 1995 LOIT program, and the 1995 LOUT program. Nine '

current Millstone Unit 3 license holders have been reviewed through this process. The l

, preliminary review has determined that discrepancies exist or records cannot be found  ;

2 to support completion of prerequisites for licensed operator training, or completion of NNECO's requirements to be licensed as an Operator / Senior Operator. A summary of the discrepancies applicable to current Millstone Unit 3 operator licensees is provided i below:

1. One candidate did not complete the required number of hours of under-instruction  !

walch standing.

2. One candidate completed required OJT items after the submittal of the NRC Form 398, but prior to the NRC license examination.

3. Two candidates for instant SRO did not have an appropriately documented i individualized training program specified in lieu of the standard NU program. These two candidates were either previously licensed or were trained for NRC licensing, iand as such, were given an abbreviated training program in 1994 - 1995. No documentation of the specific program expectations can be located.

Actions Taken for LOIT/ LOUT Findinas Millstone Unit 3 management evaluated discrepancies in the initial training of each licensed operator, reviewed. In all cases, it was determined that the natute of the discrepancies was such that the individuals' qualifications to conduct licensed activities was not called into question. For the one licensed operator who has incomplete i training (under-instruction watch standing), Millstone Unit 3 management has approved

- NTD's . recommended basis for the continued acceptability of the licensee's qualifications.

Millstone Unit 3 LOIT/ LOUT programs are in progress at this time. NTD management has reviewed'a sample of the training processes and record keeping to ensure that discrepancies or results similar to those found elsewhere are not being duplicated in the current class. Actions to correct the program deficiencies identified in the foregoing i

" Summary of LOIT/ LOUT Findings" will be completed prior to NRC examination of the current LOIT/ LOUT class in July,1997.

--. ~ _ _ _ _ _ _ _ _ _ .._ _ _ _ ._ _ .. _ _ . _ _ _ _ _ _ _

U.S. Nuclacr R:gulatory Commission B16253\ Attachment 3\Page 7 Summary of Licensed Operator Reaualification Trainina (LORT) Findinas

A' review of LORT program activities against the SAT based, accredited training j program (used to meet the requirements of 10 CFR 55.59) was conducted between 2

February 12 and February 26,1997. This review evaluated program administration, content, attendance, evaluations (examinations), and effectiveness reviews, against the i

requirements of the NU LORT Program Implementing Procedure (TPIP) and the Unit-l Specific Training Program Description. The review was conducted by a supervisor and

an instructor from an Operator Training group other than the one being reviewed.

l Approximately thirty discrepancies were documented through an Adverse Condition

! Report. The deficiencies were categorized by the nature of the required corrective actions, as follows:

1. Several discrepancies were administrative in nature rather than departures from program requirements. For example, an unlicensed Shift Technical Advisor failed the 1996 Annual Examination and has not yet completed remediation.

4 l 2. Several discrepancies were historical in nature and either did not require corrective

, action or are resolved by following program requirements on a going forward basis.

i For example, missed attendance had not been made up or resolved within the

! specified time frame.

i l 3. Several discrepancies require corrective actions in accordance with an approved

Corrective Action Plan. For example, documentation of the completion of makeup i training is inconsistent from crew to crew.

! Based on the discrepancies identified and the required corrective action, Millstone Unit

{ 3 management considers it appropriate to continue implementation of the LORT j program, while longer term corrective actions and further enhancements are identified j- and implemented.

a a

i s

)

w

- .- _ . . - . . - - - . - . . - . . - ~ . - . . - . . .~ - - ~ .

U.S. Nucisar Regulatory Commission B16253\ Attachment 3\Page 8 l

i Haddam Neck Plant Review Summarv i Summarv of LOIT/ LOUT Findinas 1

LOIT/ LOUT program requirements, such as prereq. 'isites, on-shift watch standing, OJT,

! reactivity manipulations, and other supporting training records have been reviewed for l training programs leading to new RO/SRO licenses. For the Haddam Neck Plant (HNP), ,

[ the classes reviewed were the 1995-1996 LOIT program, and the 1996 LOUT program. )

The preliminary review has determined that discrepancies either exist or records cannot i

be located to support completion of prerequisites for licensed operator training, or j completion of CYAPCO's requirements to be licensed as an Operator / Senior Operator. i A summary of the discrepancies applicable to the three current HNP operator licenses  !

from the LOIT/ LOUT program is provided below:

1. Reactivity manipulations were counted in a manner which is inconsistent with ,

industry standards. An example of this practice is taking multiple credits for 1 evolutions which by current standards would be counted as a single manipulation.

In one case, the re-count of reactivity manipulations by the review team left the  ;

trainee with fewer than the minimum required five.

2. One candidate completed one OJT card after the submittal of the NRC Form 398, but prior to the NRC license examination.

Actions Taken for LOIT/ LOUT Findinas CYAPCO management immediately took action to administratively remove from licensed duties all members of the 1995-1996 LOIT/ LOUT program. Subsequently, nine of the twelve licenses were no longer needed and their licenses have been terminated pursuant to 10CFR55.55(a). The three remaining licenses continue to be restricted from licensed duties pending completion of the data review. The results of this data review will be submitted to the NRC by April 2,1997.

Summary of Licensed Operator Reaualification (LORT) Findinas The HNP LORT program continues as the unit prepares for decommissioning and the attendant changes in license status. Training topics are being selected from the task lists consistent with systems and activities (procedures) currently in use at ti;e facility.

Based on having appropriate management oversight of the HNP LORT program, the reduced number of relevant training topics, and the decreased size of the training ,

population, CYAPCO has determined that the LORT program should continue.  ;

CYAPCO has directed that a specific review of the HNP LORT program, similar to that performed for the Millstone units, be completed by April 4,1997.

4 d

t'