ML20149M855
| ML20149M855 | |
| Person / Time | |
|---|---|
| Issue date: | 05/11/1987 |
| From: | Stello V NRC OFFICE OF THE EXECUTIVE DIRECTOR FOR OPERATIONS (EDO) |
| To: | |
| Shared Package | |
| ML20147A821 | List: |
| References | |
| FOIA-87-787, TASK-PINV, TASK-SE CCS, SECY-87-121, NUDOCS 8802290154 | |
| Download: ML20149M855 (20) | |
Text
r . :., - - . . . . ~ , . . .. .. ~ .. ..
- =. _ . % ._ - - -- -
i i
b 8 ' H i *
~
- t,* 8
.% 9 #'b * [ 4
May 11, 1987 - - ~' # - '- - ~ SECY-87-121 For:
(NCtatiCn WlN The Comissioners From: Victor Stello, Jr.
Executive Director for Operations l
Subject:
THE 2-YEAR EVALUATION ON IMPLEMENTATION OF THE COPNISSION POLICY STATEMENT ON TRAINING AND QUALIFICATION Purcose: To infonn the Ccmission of the staff's evaluation of i industry's implementation of the Policy Statement on
- l Training and Qualification of Nuclear Pcwer Plant Personnel, and to obtain Cecraission guidance on revising the current Policy Statement.
Packaround: On February 7,1985, the Comission adopted the Policy Statement on Training and Qualification (50 FR 11147, '
March 20, 1985). The Policy Statement endorsed the INPO-managed Training Accreditation Program and tne industry, through NUMARC, comitment to have 10 programs at 61 sites 1 (610 programs) ready for accreditation by December 31, 1986. I In approving the Policy Statement, the Comission deferred rulemaking on training and qualification for 2 years in recognition of industry efforts and directed the staff to independently evaluate implementation of improvement programs..
SECY-86-119 provided a status report on implementation of I the Policy Statement after 1 year. It concluded tha-significant progress was being made and recomendad that rulemaking centinue to be deferred. Areas of concern were !
noted, some of which have been addressed by publication of !
the revision to 10 CFR 55 (52 FR 9453, March ?S,1987) and !
development of INPO 86-025, GuTdelines u for Continuing l Training of Licensed Personnel," issued in October 1986. l Discussion: Each of the elements of the staff's independent evaluation is discussed in the Enclosure and is sumarized belcw:
Accreditation Team Visit Observations - As of April '. 19S7, INPO has conducted 104 team visits covering 450 programs.
They also conducted 10 team visits covering 38 programs at N 6 newer facilities. NRC staff has observed 19 visits.
Centact:
F.J. Miller, NRR 4c2-4803 f) Fo7.JP- D*?I7 7AA 8802290154 880223 PDR FOIA EOLEYB7-787 PDR . . . _ _ _ _ , . . . . _ . . . . . . . . . . . . . . _ . _ . . . . . _ . .
s I
l 2-i The staff has found Accreditation Team Visits to be thorough, constructive and professional. Utilization of peer evaluators significantly enhances the process. NRC observers have expressed concern about the potential. for ,
inconsistency in applying INPO's objective * *nd criteria -l from plant to plant. Over the past year, lid 0 has increased I its attention to ensuring consistency in the application of I objectives and criteria through meetings with team leaders, training of team reviewers and senior management review of all team visit reports. - -
Observations of the National Nuclear Accrediting Board -
Ine Board has accredited 369 programs at 60 sites. Senior NRC staff members have observed all but a few of the Board meetings. The Board's review is independent, of high quality and professional. It relies heavily on the team visit report and subsequent interactions between the INPO staff and the utility. The post team visit results are provided as supplemental reports, in some cases, accreditation has been awarded based upon commitments for further training development and future program implementation. INP0's computer-based tracking system is used to ensure that these comitments are met. The Board I reviews the status of commitments monthly. The Board has deferred accreditation in two instances.
Review of Plant-Specific Self Evaluation Reports (SER) and INPO Team Reports - INPO team reports, SERs ano other documentation are available to the NRC during team visits, Board meetings and on site. Staff review of utility SERs,
~
field notes, team reports and utility responses confirms that the reports are consistent with INPO guidance, are l thorough and that INP0 and. utilities generally resolve '
recommendations prior to the Board meeting.
Training Inspections and SALP - Training inspection procedures were revised in June 1985 to be performance-based by focusing on the ability of the plant staff to i perform their jobs after training rather than on the i training program or development process.
Fonnal procedural guidance related to training program development, implementation and evaluation has improved at utilities; accreditation has increased utility management ;
attention and comitment to training; and the quality of utility lesson plans and other related training materials have improved. However, a number of training program weaknesses continue to be identified in inspection reports on both accredited and nonaccredited programs. Shortcomings in training related to plant procedures, technical specifications, design changes. LERs, operational W
- , - - - , -, 7, .- -, , , - ,.- , , -y . , , - , , , - . , . - - - - , - , ,, .-,--,r--r--.- -,--,,--_-r-c - *=-r--
i i
l experience and industry events are prevalent. Also, specific training program content was deficient in a number of programs regardless of accreditation status. .
In November of 1985, SALP procedures were revised to l evaluate licensee training programs as a separate functional l area. Review of SALP reports indicates positive effects of accreditation as demonstrated by well-staffed and qualified training departments, better-trained pl, ant personnel and,new .
or greatly enhanced training fa'cilities. Concerr.s frequently identified in SALP reports include: (1)under-standing of procedures; (2) requalification program administration; and (3) lack of nuclear specific detail in maintenance training and knowledge of the impact of maintenance activities on operations.
Post-Accreditatien Reviews - Since SECY-86-119, the staff conducteo five post-accreditation reviews (Salem, San Onofre, Callaway, Arkansas Nuclear One and WNP-2). A total of 20 individual training programs were reviewed over the 2-year evaluation period and findings from each were provided to INPO.
The Comission's Policy Statement sets forth five elements which are essential to effective performance-based training.
In a Systems Approach to Training (SAT), each element is 1 dependent on the preceding element with feedback to all l precedir.g elements. New program development typically i starts with the first element, Systematic Analysis of the !
Jobs to be Performed. However, for existing training programs, training system development can build on existing materials when used in conjunction with a needs analysis and continued evaluation of training needs based upon job performance feedback. This was the case for all but one of the programs reviewed.
The staff observations of all the accredited programs reviewed indicate that evaluation and feedback procedures are in place to allow effective modification of existing programs to be performance-based. All programs reviewed by the staff were implementing most aspects of the five essential elements. In some instances, need for further iruprovement was icentified to INPO. An important finding from these reviews is that improvements are continuing to be mdde after accreditation has been received. Also, some utilities have expanded the use of performance-based training to programs beyond the ori.ginal 10, e.g., QA/QC, Security, Emergency Response Teams.
As ciscussed in SECY 86-119, a concern identified at the sites reviewed by the staff is that a systematic approach to the development of requalification training was not performed. This was primarily due to a perceived conflict between NRC requirements and INP0 accreditation criteria.
However, based on the recently published revisions to 10 CFR Part 55 and issuance of INPO guidance on perfonnance-based requalification training (INPO-86-025), this concern should be eliminated. . .
A sumary of the findings from the post-accreditation reviews is in letters to INPO dated April 3,1986, August 8, 1986, September 10, 1986, December 3, 1986, and April 6,1987.
NRC Examination Results - Analysis of NRC examination results (inclucing requalification examination results) does not show any statistically significant difference in pass / fail rates between accredited and nonaccredited training programs. Examination reports identify deficiencies in examination performance in the areas of procedures, technical specifications, Emergency Plan, reactor theory and system operation and-desigrr; Status: The industry connitment to have 10 trainino programs at each of 61 sites "ready for accreditation"'by December 31, 1986, (i.e. , SERs accepted by INPO) was met. Based on the number of team visits remaining and programs that have not yet achieved accreditation, NRC staff expects that all programs will be accredited by mid-CY 1988. Seventeen facilities have achieved full member status in the National Academy for Nuclear Training by having all 10 training programs accredited.
Conclusion:
Significant progress is being nade by industry in improving training and implementing the Omnission's Policy Statement.
While significant training improvements have been observed, training deficiencies and weaknesses have been identified through staff inspection and review activities in both accredited and nonaccrecited training programs.
Deficiencies of the type identified are to be expected given the magnitude of the effort involved and the difficulty in 2
i
( .
implerenting performance-based concepts by an industry that until recently based training on NRC-specific requirements.
The transition process has been complicated by questions regarding the differences between NRC and INP0 review standards and criteria and by some utility uncertainty about how to meet NRC requirements and still meet INP0's accreditation criteria. On balance, however, the industry's efforts to date to improve training have been successful.
The staff's conclusions regarding futtire industry training activities can be divided into two categories:
(1) improvement and enhancement of the existing accreditation program and (2) extension of the accreditation program to address new, important areas. The examples of improvements in the first category are: applying all elements of the Policy Statement to the training of technical staff and mcnagers during the second round of accreditation; improving the review of training perfomed by contract trainers by applying the same standards used for utility training; adding emphasis to the analysis element so that all knowledges and skilis are identified for the development of complete learning and testing objectives; improvinr3 the feedback of operating experience; e.nd developing objective methods to better assess training effectiveness.
The two issues which would require extension of the accreditation program are:
Expanding the accreditation program to include two additional functional areas - Quality Assurance / Quality Centrol training and severe accident training for
- Emergency Response Personnel. QA/QC is an area of increasing concern in the industry and had originally been proposed by the staff in the proposed rule on training (SECY-84-76). INF0, at that time, indicated it i would be an area for future consideration. Additional .
IRPO guidelines and planned severe accident training for I both shift crews and plant management was discussed with INPO on February 20, 1987, and is surnarized in Enclosure 2 of SECY-87-101, April 16,1987.
Eganding the accrecitation program to ensure that utilities assure that contractor personnel who serve in functions covered by accreditation are cualified and receive training conwensurate with the training and qualification of utility personnel. NRC inspectors have i
- - - , , - - - , n - -
l
. - - . l l
l l
. p; .
1 frecuently recuested guidance in this area and have expressed concern that contractor personnel are not as well trained or qualified as utility staff.
Decomendation: The staff reccmends that the Comission:
- 1. Continue to endorse the industry accreditation program and defer rulemaking.
- 2. Note that t>e staff will:
A. Continue to evaluate industry implementation of Training and Qualification of Nuclear Power Plant Personnel as described in the existing Policy Statement.
B. ' Work with INPO and the industry to enhance the existing accreditation program to: evaluate and quantify training effectiveness; enhance the Technical Staff and Managers' training; emphasize the importance of task analysis, learning objectives and the evaluation feedback process; and improve contracted training.
C. Meet with INPO Managemetit to discuss:
(1) Accreditation of the additional functional areas of Quality Assurance / Quality Control and severe accident training for Emergency Response Personnel, and (2) Expanding the scope of accreditation to cover utility assurance of the training and qualifications of contractor personnel.
- 3. Direct the staff to propose a revised Policy Statement on Training and Qualification of Nuclear Power Plant Personnel to incorporate the results of the 2-year trial period and the results of the discussions with INPO.
l l
. : <en \
ViltorStello,Jr.
Executive Director for Operations
Enclosure:
Independent Evaluation of Industry Training {
I
l l
1 l
7 l 1
l Commissioners' comments or consent should be provided directly I to the Office of the Secretary by c.o.b. Wednesday, May 27, 1987.
. Commission Staff Office comments, if any, should be submitted to the Commissioners NLT Wednesday, May 20, 1987, with an infor-mation copy to the Office of the Secretary. If the paper is of such a nature that it requires additional time for analytical review and cem=ent, the Commissioners and the Secretariat should be apprised of when comments may be expecte6.
DISTRIBUTION:
Commission $rs OGC (H Street)
i i
! j I
l Enclosure 1 l INDEPENDENT EVAltATION OF INDUSTRY TPAINING INTRODUCTION The NRC staff has evaluated licensees' training programs independently to ensure that results are consistent with the goal of the Policy Statement on Training and Qualifications (50 FR 11147), i.e., that the NRC's endorsement of the INP0-managed Training AccFeditation Program will result in effective performance-based training in the nuclear power industry. The staff's approach was as described in SECY-SS-288. Staff members observed INPC accreditation teams and the Accreditation Board and conducted independent staff reviews of accredited training programs (i.e., pcst-accreditation reviews). In addition, the staff has compiled information from licensed operator examination reports, inspection reports, Systematic Assessment of Licensee Performance (SALP) reports, and input from NRC regional offices.
Although LERs were reviewed, they were not used because they do not include a separate cause code for training; therefore, training as a root cause could not be determined from the data. The r6sults of the staff's evaluation are discussed below.
Pcat-Accreditation Reviews The staff has reviewed 20 training programs at 8 sites (Susquehanna, Dresden, '
Oconee, Salem, San Onofre, Callaway, Arkansas Nuclear One and WNP-2) to evaluate implementation of the 5 elements of performance-based training specified in the Policy Statement. The review findings grouped by Policy Statement element for the 20 programs are:
Systematic analysis of the jobs to be performed.
A systematic method was used to define tasks performed on the job at 18 of the 20 training programs reviewed. Nine of the programs identified knowledge, skills and abilities (KSAs) from these analyses for use as the basis for ensuring ccepleteness of learning objectives and training. Nine programs developed task lists from job analysis which were compared with the centent of existing training raterials. Of the other two programs reviewed, ,
one was accredited in 1983 based en INP0's training and l qualifications guidelines, which was allowable in lieu of for al job and task analysis, and the other was in the technical sta'f and manager category for which INPO is not requiring a position-specific analysis to establish program content.
Only two programs had centinuing training needs identified from task analyses, although ratings on importance, frequency, and difficulty of task performance have been identified for the programs at other facilities. These ratings would aid in selecting tasks for continuing training.'
l
i l
- 1. earning objectives derived from the analysis which describe desired performance after training.
Tasks were supported by learning objectives for 16 of the 20 prog ams. Learning objectives existed for most, but not all, taskt for three other programs, althougt, in some cases, the learring objectives were overly broad. No learning objectives were included in one contracted training program in the technical staff and manager category.
The majority of learning objectives reviewed described expected trainee behavior in terms of conditions, actions or standards governing successful task performance. In scee cases, learning objectives were too broad and did not provide a consistent basis for trainee evaluation.
1
- l Training design and implementation based on learning objectives. l The training organizations' goals and objectives, as well as the l responsibilities of training organization personnel, were stated '
in writing at all facilities reviewed. .
Qualification and training requirements for the training staff were found to be adequate although several were not formalized by procedure.
Careful selection of training methods and media was in evidence at all facilities. The organization and secuencing of initial training programs were generally based upon the relationship among learning object';ves.
Lesson plans at all facilities provided for consistent training l delivery, and both simulator and classroom training observed during the reviews were found to be well conducted.
l All facilities had adeovate methods for maintaining training '
records.
Evaluation of trainee mastery of the objectives during training. !
Exemptions from training at all but one facility were derived from performance-based testing or other objective evaluation methods, e.g., previous experience. J Appropriate procedures for development of trainee tests and standards of performance were available and implemented at all utilities.
l l
l l
1
. -3 I
Ia 4e of the 2? training programs reviewed, test iters ceuld be !
itnkeo to task ?ists and learning objectives. In one program, the learning objectives were too broad to determine if an effective i linkage existed. Examinations were not available for review for l one contracted training program in the technical staff and manager j category. In the remaining two programs, test items covered classroom learning objectives; however, the criteria for trainee !
evaluation during simulator training consisted of bread objectives with only infomal mechanisms for evaluaticn and feectack of l specific task perfomance problems.
Criteria for evaluation of on-the-job training (OTJ) was often too broad. ;
l All programs had adequate procedures in place to prevent comprcmise of examinations, Evaluation and revision of the training based on perfomance of trained personnel in the job setting.
All sites had procedures in place for effective program evaluation and there was evidence that the procedures were being implemented.
Evaluation of examinations and operating test results were perfomed at all sites, but aggregate analyses, such as item analysis, were not performed at the majority of facilities reviewed.
Instructor and trainee critiques were in evidence at all facilities, although they varied both in femalizatien and extent to which they were used for program evaluation.
The staff at all sites were qualified to perfom their respective duties as training or subject matter specialists.
Internal and external evaluations were performed at all sites on a regular basis.
On-the-job experiences and feedback were solicited from job incumbents and supervisors at most facilities. Most facilities have formal mechanisms for evaluation of trainee perforsance after training to provide feedback from operations to the training department.
Most programs need to improve methods to ensure that training programs remain current and timely as job performance recuirerents change (e.g., as plant equipment and procedures are chanced).
4
.4.
Operator Licensing Examination Results The impact of accreditation on licensing examination performance was assessed through analysis of pass / fail rates on the written and cperating (i.e., oral and sirulator) portions of the NRC examination. The results of the analyses for FY 1985 and for the time period extending from FY 1986 through the end of.
calendar 1986 are as follows:
FY 1985 No. No. Pass No. Pass SWR Plants Plants Candidates Written Candidates Operating Accredited 7 88 86% 96 93%
Monaccredited 21 357 77% 351 86%
FWR Plants Accredited 19 323 84% 309 88%
Nonaccredited 29 349 90% 324 95%
FY 1986 . December 1986 No. No. Pass No. ? ass BWR Plants Plants Candidates Written Candidates 0;erating Accredited 12 145 90% 156 91%
Nonaccredited 19 291 90% 277 90%
PWR Plants Accredited 24 382 84% 341 88%
Nonaccredited 24 566 87t 469 89%
No statistically significant differences were found between the examination
' pass rates of accredited and nonaccredited plants for either time period.
Several explanations can be offered as to why the effect of accreditation on training programs cannot be ascertained using operator license candidate examination performance. Firs t, throughout FY 1985 and 1986, most plants were actively engaged in completing the multiple stages leading to receipt of accreditation. Thus, many candidates included under "Nonaccredited" in the tables abcve are from plants that achieved accreditation shortly after the examination. As such, the impact of accreditation status on NPC examination scores becomes somewhat blurred during this period of transition.
Nonsignificant differences may also be a result of timing between evmpletion oftraining,subsequentNRCexamination(s)andaccreditation(e.g.,
candidates at accredited plants may have completed training prior to accreditation). Following implementation of the revisions to 10 CFR 55, the staff's data base will include whether or not the candidate completed an accredited training program.
1 I
Recualification Examinations Differences in pass rates between candidates from accredited and non- I accredited plants on NRC requalification examinations for FY 1985 and FY 1986 !
are as follcws:
. l No. Pass No. Pass No. Pass l FY 1985 Candidates Written Candidates Oral Candidates Simulator Accredited 48 71% 18 100% 18 89% l Nonaccredited 242 77% 121 97% 28 90%
FY 1986 Accredited 139 78% 128 97% 53 96%
Nonaccredited 96 72% 53 94% 52 90%
Although not statistically significant, there is a marked increase in the performance of candidates at facilities with accredited training programs between FY 1985 and FY 1986. However, this increase is probably due to factors other than application of perfomance-based training methods to
' requalification programs because requalification was regulated by 10 CFR 55, Appendix A. Future evaluations should attempt to better measure the relationship between overall requalification examination pass rates and the extent to which the facility follcws a program of perfonnance-based recualification training.
Licensed Operator Examiration Reports l
Examination reports were reviewed for all plants to extract data describing i industry training program effectiveness. These reports of ten identified !
generic weaknesses found in the written and/or operating examinations. A generic weakness is defined by the operator licensing examiners as a knowledge or perfonnance deficiency evidenced in approximately cne-third 1 of the candidates at a facility. The citing of generic weaknesses is !
intended as feedback to facility training staff. I Generic weaknesses were noted in examination reports of accredited and nonaccredited programs alike. Of the 165 examination reports reviewed c.er the 2-year period,103 identified generic weaknesses. Connon ceficiencies, i weaknesses and findings are listed belew, in roughly descending order of frequency of occurrence:
Deficiencies in kncwledge and use of procedures, especially abncrmal and evnergency procedures were cited in almost half of the examination reports. Many candidates displayed weaknesses in kncwledge of precautions, entry conditions, imediate actions, use of multiple procedures, and failure to complete and verify procecural steps. The recent transition from event to syrptom based emergency procedures may be in part responsible for this frequently cited weakness.
. l l
Xnowledge of system operation and system design, including system )
purpose, location of components, operating limits and parameters and '
their bases, the effect of malfunctions on systems, understanding system response, and system logic and interlocks was cited in approximately one-third of the reports reviewed.
1 Plant reference material, st.J as lesson plans and system descriptions, were found to be obsolete, deficient, and/or vague in approximately one-third of the reports. Instances of discontinuity between training material and procedural guidance were noted.
Further, numerous problems with both hardware and software impeded the use of simulators as a mode of candidate evaluation.
Deficiencies in interpretation and use of Technical Specifications, including knowledge of bases and definitions was cited in approximately 10% of the reports.
Candidates exhibited weaknesses in radiation protection kno,vledge, including facility exposure limits and bases for their determination, sources of radiation, theory of operation of radiation monitoring instruments, and related basic health physics principles.
Candidates exhibited weaknesses in reactor theory, thermodynamics, and electrical theory, including knowledge of reactivity effects on various plant parameters and the conduct of reactivity calculations (e.g.,ECPs,shutdownmargin).
Deficiencies in communication skills and, in the case of senior reactor operator (SRO) candidates, skills in directing control room activity and in knowledge of administrative requirements, procedures, cautions and limitations were also noted in approximately 10% of the reports.
Classification of events according to the Emergency Plan was a deficiency cited in several of the reports.
Candidates exhibited. weaknesses in their ability to perform tasks that are infrecuently encountered, and in knowledge and u6e of systems outside the control room, such as the operation of the plant from the remote shutdown parel.
Refueling, including knowledge of refueling equipment, duties, operations, requirements, and refueling system interlocks was a deficiency mentioned in several reports.
~
1 1
7 l 1
Summary l
There are no statistically significant differences in :erformance on NRC 1 examinations beineen accredited and nonaccredited facilities. Since nearly l all facilities have their operator licensing programs accredited, the staff !
may not be able to cuantify the effect of accreditatior. through examination j performance in the future. Examination reports describe a variety of generic i
weaknesses in cardidates at both accredited and nonaccredited facilities.
These weaknesses are examples of performance deficiencies, which demonstrate ;
that matarials provided by utilities to examiners and tt:e NRC examinations '
are new performarce-based, which is the cemmon goal of che industry and the NRC.
NRC INSPECTION PEPORTS i NRC inspection reports for CY 1985 and CY 1986 were reviewed to determine the influence of INPC accreditation on training programs at nuclear power plants.
Deficiencies, unresolved items, and open items were noted at utilities regardless of ac:reditatinn status.
While inspection reperts are an appropriate source for determining strengths, weaknesses, and progress of facility training programs, it is difficult to draw comparisions between accredited and nonaccredited ;rograms. The training inspection modules were revised in June of 1985 to be performance-based. However, many of the inspections were not conducted using the revised inspection procedures due to implementation lag and there was considerable variation among regions in the interpretation of these :rocedures. In addition, although there were far morr. utilities with accredited programs in 1986, the status of the accreditation effort at the time of the inspection varied considerably across utilities with respect to the specific program (s) inspected.
Soce of the findings reported which may not appear to be the result of a l performance-basec inspection because they relate to progrannatic issues were indeed found in ;erformance-based inspections. They are the result of inspectors tracing a performance deficiency identified in the inspection back to the training program. l This suntary of inspection findings is organized into five areas; Adminstrative Issues, Training Program Content, Recordkeeping, Instructor Performance and Oualifications, and Training Ev61uation. Where applicable, differences between accredited and nonaccredited facilities are discussed.
Administrative Issues. The positive influence of INP0 accreditation in the area of administrative issues is evident. However, inspectors continued to identify deficier<ies in procedures or their use in the adminstration of training.
l I
l
l The positive findings, identified in 20% of the reports reviewed were:
Increased administrative oversight of training program development, implementation, and evaluation in accredited programs.
Increase in attention to training and the improved quality of performance-based training.
The following administrative deficiencies were noted in approximatel.y 2C1 of the reports reviewed: i
{
\
1.ack of procedural guidance and management attention or comitment at ;
nonaccredited facilities.
l Failure to follow or implement procedures and guidelines that are in {
place at accredited facilities.
1 Administrative shortcomings directly related to NRC regulations in
- both accredited and nonaccredited programs. 1 i
The majority of administrative deficiencies and cipen items were related to requalification training and evaluations.
Trainino Program Content. Although a majority of inspection reports for the 2-year perico icentified improved quality and detail in lesson plans and
' other course material in accredited programs, there continue to be weakresses in the following areas in both accredited and nonaccredited programs:
4
- Over half of the reports identified inadequate coverage of specific content areas, e.g., trainns to prevent personnel from overtorquing limtorque valve operators, training needed to prevent overloading of ;
emergency diesel genera
- ors, ineffective training on containment isolation valves, and inadequate training on recognition of symptoes requiring manual feedwater pump trips during an overcooling event.
4 Shortcomings in licensed operator training related to coverage of emergency and abnormal procedures were identified in over 40% of the :
reports reviewed, i.e., not enough time spent on these procedures; insufficient depth of coverage. ,
Deficiencies in the degree of specificity in certain topical areas were identified in over ene-third of the reports. In addition, 3' deficiencies were noted in training on technical specifications and i procedures, and feedback of design changes,l.ERs, cperational {
experience and industry events to the training program. l l
J I
l I
.g.
1.ack of training coverage of topics addressed in regulatory requirements ano directives was noted in over one-third of the reports reviewed.
Inappropriate management of time for specific training topics was mentioned in more than 10% of the reports reviewed, i.e., too long an interval between related topics; too little tire to cover material.
Overreliance on self-study was cited i.r. Si cf the inspections.
Weaknesses or inadequacies in learning objectives were mentioned in a few cases, e.g., objectives that are too tread or not based on appropriate analysis.
Recordkeeoing. When documentation was mentioned in inspection reports for the latter part of the 2-year reporting period, it was generally in a positive light, often noting the implementation of new and efficient computerized 'fstems for document maintenance and retrieval. Ceficiencies were noted in the following areas:
Over 10% of the reports identified inadequacies in training record documentation and maintenance, usually involving inaccuracies, records of course attendance and remedial training, examination grades and other measures of student performance.
A few reports mentioned difficulty in retrieval of information at some utilities regardless of accreditation satus.
Instructor Performance and Oualifications. Findings in the area of instructor perfomance ano qualifications were noted in only a few inspection reports. Most of the reports in which this area was addressed identified the following:
+
- Some instructors either were not SP.0-certified or had failed to perfonn licensed duties for a period of 4 conths or greater.
Thero were instances of instructors who teach systems, integrated response, transient and simulator courses who were not enrolled in appropriate portions of requalification programs.
l In addition, a few of these reports also mentioned:
Perfonnance by instructors was adequate and effective.
Instructors were adequately qualified.
I r
10-Training Evaluation.
Inspection reports for the 2-year period identified the following findings related to training evaluation: i Nearly half of the reports reviewed indicated that a number cf satisfactory training feedback mechanisms are in place at both accredited and nonaccredited facilities, e.g., follow-up on detected '
trainee deficiencies, utility-administered examination preparation and grading, mechanisms for feedback of instructor effectiveness, use of aggregate examination analysis to identify program weakresses (and also to develop subsequent requalification programs), training needs analysis forms submitted to the training group from operations, and OTJ feedback from supervisors of trainees.
Nore than 40% of the reports reviewed addressed various deficiencies ;
in utility-administered examinations, including (1) overly easy or :
simplistic axamination cuestions; (2) failure to link examination questions to learning objectives; (3) a lack of criteria for passing examinations or other performance measures (oral, written, and simulator); (4) reductions in passing scores on requalification e.xaminations; (5) inaccuracies in examination grading; and (6) breaches of examination security, including inappropriate storage, i I and repeated administration of the same questions and examinations.
In some cases, there were no4rocedures in place to ensure that giding quality was checked on written requalification examinations which, in turn, resulted in unidentified examination failures.
- 1 In approximately one-third of the reports, deficiencies were noted in 1 the imnlementation of the above evaluation methods by some licensees
' for both accredited and nonaccredited programs, such as inadeouate connunication of LERs and reports of deficient performance to the training group for factoring into training.
Over 12% of the reports identified shortcomings in programs for remedial training, such as failure to provide such training at j , individual, greep, or progrannatic levels.
Summary. In general, the impact of INP0 accreditation continues to be reflected in an increase tin procedures for training program development, implementation and evaluation. Accreditttion also appears to have brought {
about increased management support and tonnitment to performance-based l training. However, there contiAut to be weaknesses in trainino on LERs, l operational experience, industry events, plant design changes, Technical Specifications and plant procedures and shortcomings in requalification l program implementation. Regulatory deviations and violations were found !
fairly equally e. cross utilities, regardless of accreditation status.
3
, l l
t 1-
-11 i
SYSTEFATIC ASSESSMENT OF LICENSEE PERFORMANCE (SALP)
An additional 39 SALP reports from May 1985 to December 1986 were reviewed for infomation on licensee training programs and supplements the review of 44 SALP reports discussed in SECY-85-119. It is important to note that SALP rating periods are long, i.e.,12-18 months, so that in some cases (22) the SALP review ended prior to accreditation of any of that utility's programs, while in other cases (17) accreditation had been granted to at least a few of the utility's programs during the SALP reporting period.
The review enr.ompassed information found in the following SALP report categories:
(1) Plant Operations (2) Radiological Controls (3) Faintenance (4) Licensee Activities (5) Training These areas were selected for review because performance information related to training programs and job positions covered by INPO accreditation would be ,
contained in these categories. Also, SALP rating criteria (i.e., SALP l scores) were not developed specifically for the purpose of evaluating whether i perfomance-based training programs comply with the objectives contained in the NRC Policy Statement on Training and Qualification.
Based on SALP reports, licensee training programs, both accredited and nonaccredited, show evidence of a strong commitment to training. General findings regarding the effects of accreditation on licensee training programs are:
(1) well-staffed training departments. -
(2) well-trained plant personnel, (3) new/ improved training facilities ecuipped with plant-specific simulators and mock-ups.
However, generic training problems continued to be identified in SALP reports involving both accredited and nonaccredited training programs. Problems in recent SALP reports which are similar to those identified in SECY-86-119 are: '
Understanding of procedures and inadequate training on procedures were reported in over 50 percent of the SALP reports reviewed, i
1
12 In approximately 30 percent of the SALP reports, lack of management attention at scre facilities to licensed operator requalification training, including failure to establish or implement adeouate training procedures, failure to maintain adequate training records, evidence of poor requalification training and high failure rates on NRC requalification examinations was noted.
Maintenance training was described as not nuclear specific or provided in sufficient detail in about 20 percent of the SALP reports. This lec to problems such as inadvertent operation of equipment including actuation of Engineered Safety Features (ESF) while performing maintenance or trouble-shooting.
Problems which were newly identified and occurring less frequently (10-20%)
in SALP reports from May 1985 to December 1986 include:
The need for more management attention to improve performance of plant-specific simulators and need for improvement of sir.ulator training.
Lack of recognition or understanding of Technical Specifications.
Operator unfamiliarity with equipment status during abnormal plant conditions and events.
Lack of plant experience or plant-specific training for contractor personnel.
Failure to consistently perfom thorough reviews of operational events for feedback into the training program.
Adequacy of OTJ training programs.
' In conclusien, SALP repcrts indicate that accreditation is having a positive effect on improving training. However, problems in perfomance of a variety of operating activities involving trained personnel and weaknesses in training program administration and content continue to be noted as weaknesses in SALP reports.
REGIONAL EVALUATIONS OF IN00STRY TRAINING IMPROVEMENTS l
- Each region has provided contents on the effectiveness of industry training under the NRC Policy Statement on Training and Qualification. To sumari
- e, the improvements noted in the com.ents from the regions with respect to
e J 13 accreditation of training are primarily in the area of adminstrative oversight and management commitment of both human and financial resources.
Here of the regions was able to point to direct casual relationships between accreditation and improvements in training, yet they offer the opinion that training has improved since the accreditation effort. They indicate that more time is needed to make a definitive judgment because of the short time that some accredited programs have been irplemented and because very few facilities have all their programs accredited. Some concerns which were expressed regarding accreditation are: Accreditation was felt to have some regative impact because nonlicensed training was neglected while the training staff was occupied with the accreditaticn effort, accreditation has not had a positive impact on chemistry and radiation protection training, INPO has no timely rechanism for follow-up to confinn accreditation commitments and some plants have failed to implement their accredited program, training of contractors is not addressed by INPO, and there is no analysis for training of technical staff and managers.
1.
e d
l l
l 1
)
1
)
1 j