ML20206G498
| ML20206G498 | |
| Person / Time | |
|---|---|
| Issue date: | 04/18/1986 |
| From: | Stello V NRC OFFICE OF THE EXECUTIVE DIRECTOR FOR OPERATIONS (EDO) |
| To: | |
| References | |
| TASK-PII, TASK-SE SECY-86-119, NUDOCS 8606250324 | |
| Download: ML20206G498 (61) | |
Text
-
PilBLICll% RELEASED 8
I 5
4,E
\\...../
POLICY ISSUE April 18, 1986 (InfOrmation)
SECY-86-119 For:
The Comissioners From:
Victor Stello, Jr.
Executive Director for Operations
Subject:
ANNUAL STATUS REPORT ON IMPLEMENTATION OF THE COMMISSION POLICY STATEMENT ON TRAINING AND QUALIFICATION
Purpose:
To inform the Comission of industry progress in imple-menting the Policy Statement on Training and Qualification and the results to date of staff evaluation of the INP0-managed Training Accreditation Program.
Background:
On February 7,1985, the Comission adopted the Policy Statement on Training and Qualification (50 FR 11147, March 20, 1985). The Policy Statement endorsed the INPO-managed Training Accreditation Program because it encompasses the elements of effective perfomance-based training which are:
analysis of the job, performance-based learning objectives, training design and implementation, trainee evaluation, and program evaluation.
In approving the Policy Statement, the Comission deferred for 2 years rulemaking on training and qualifications in recognition of the industry efforts provided that the industry programs produce the desired results. The Comission also directed the staff to independently evaluate applicants' and licensees' implementation of improvement programs and to evaluate the possible need for further NRC action.
Subsequent to approval of the Policy Statement, Appendix Number Four of the NRC/INPO Memorandum of Understanding was signed on July 12, 1985. That Appendix describes the activities of both the NRC staff and INP0 in implementing the Policy Statement.
62 4 860418 86-119 PDR l
Contact:
W. T. Russell, NRR 492-4803 j
l PUBLICLY RREASED
The Commissioners.
Summary:
SECY-85-288 described the staff's approach to evaluating both individual utilities' progress in improving training and the INP0-managed accreditation program. The approach includes: observation of INPO Team Visits and meetings of the National Nuclear Accrediting Board; review of plant-specific Self Evaluation Reports and INP0 team reports; and independent evaluation of training programs' effectiveness through training inspections, post-accreditation reviews, event-based reviews and analysis of performance data (SALP reports, examination reports, etc.).
Discussion:
Each of the elements of the staff review is discussed below:
Team Visit Observations - As of April 1,1986, INPO has conducted 58 Training Accreditation Team Visits at 45 sites covering 228 individual training programs. NRC staff has observed 12 of these visits.
The staff has found the process to be thorough, resource-intensive, constructive and professional. Utilization of a team that includes both INP0 staff and utility peer evalu-ators enhances the process.
Some staff observers have expressed concern about the potential for inconsistency in applying INP0's objectives and criteria from plant-to-plant.
INP0 methods to ensure consistent application of objectives and criteria include meetings with team leaders, training of team reviewers and senior management review of all Team Visit reports.
Observations of the National Nuclear Accrediting Board - As of April 1,1986, the Accrediting Board has accredited 138 individual training programs at 30 sites.
NRC has had observers at all but a few of the Board meetings.
The observers at Accrediting Board meetings have commented on the professionalism, independence and quality of the Board's review. The Board relies heavily on the review work per-formed by INP0 at the Team Visit and in subsequent inter-actions between the INP0 staff and the utility. The staff has expressed some concern (letter from Mr. Denton to Mr. Pate dated January 28,1986) related to INP0's closecut of recommendations between the time of the Team Visit and the Board meeting. Subsequently, the staff has observed an INP0 follow-up visit to close out team recommendations at one plant and has questioned INP0 Team Leaders on the results of their follow-up visits. Also, the results of follow-up visits are provided to the Board as supplemental reports. The staff has also observed granting of accredita-tion status by the Board based upon commitments for further training development and future program implementation.
l
...=.
t a
The Commissioners 3-INPO has a computer-based tracking system to ensure that these commitments are met. The Board reviews the status of commitments monthly.
Review of Plant-Specific Self Evaluation Reports (SER) and INPO Team Reports - INPO team reports, 5ERs and other documentation is available to the NRC observers of Team i
Visits and Board meetings. Utilities also make the reports available to appropriate NRC personnel for review or reading on site. Staff review of utility Self Evaluation Reports, field notes, draft team reports and utility responses confirms that the reports are consistent with INPO guidance, that INPO reports are thorough and that INPO and utilities generally resolve INPO recommendations prior to the Board meeting.
Traininc Inspections and SALP - Training inspections are intendet to evaluate training effectiveness. The previous approach to inspection of training focused on the adequacy of the licensee's training program and their implementation of that program. The inspection approach contained in the
]
procedures which were revised in June 1985 is performance-based in that it focuses on the ability of the plant staff to perform their jobs after training rather than on the training program or development process.
I In November of 1985, a revised SALP NRC Manual Chapter was distributed for implementation that specified licensee training programs be evaluated as a separate functional area.
Previously, training had been used as an evaluation criterion for each of the functional areas. One purpose of establishing training as a distinct functional area was to i
provide a better basis to analyze licensee performance in this area.
Review of training inspection reports indicates that:
accreditation has formalized procedural guidance for training program development, implementation and evalu-ation; accreditation has heightened management attention to training issues; and accreditation seems to ensure that training content is based on performance-based learning objectives, course outlines and other documenta-l tion. The inspection reports do identify, however, a variety of training program weaknesses in both accredited j
and nonaccredited programs which confirm that further improvements are needed.
Review of SALP reports indicate similar positive aspects of accreditation while identifying specific problems in some operating activities and weaknesses in training program administration, content l
and evaluation of both accredited and nonaccredited sites.
i i
.---.--_-------..,.---w---_--
---_,--.-,,,-,,-.,_-_.,--._----,..~,_,---_..,--m-.-.
_,.,,-__-,r
_.--__-.,,w,-
The Comissioners 4-A detailed discussion of the analysis of inspection reports and SALP reports is provided in Enclosure 1, Post-Accreditation Reviews - To date the staff has.
4 l
conducted three post-accreditation reviews (Susquehanna, Dresden and Oconee) covering ten training programs.
Preliminary findings from these audits were provided to INPO in January 1986.
Subsequently, NRC and INPO staff met and discussed both the preliminary findings and the staff's review procedures.
The staff's reports were completed in April 1986.
The Comission's policy statement sets forth five elements which are essential to effective performance based training.
These elements formed the basis for the post-accreditation audits.
In a systematic analysis each element is dependent on the preceding element with feedback to all preceding elements.
l For example, the third element, Training Design and Implementation, is based on the second element, Learning 4
i Objectives.
New program development typically starts with the first element, Systematic Analysis of the Jobs to be Perfonned.
For existing training programs INPO recommends that training system development build on existing materials.
This was the case for all of the programs audited.
This approach is acceptable for existing training programs provided it is utilized in conjunction with a systematic 4
analysis and the utility continues to evaluate training 4
needs based upon job performance. However, it would not be l
acceptable to enter the process at the training design stage l
(element 3) and simply develop learning objectives to match existing training materials, 1
The staff observations of the ten accredited programs l
audited indicate that procedures are in place to allow i
effective backfit of performance-based training to existing programs. While all programs are implementing most aspects of the five training elements, we observed some instances l
where further improvements are warranted. However, an important finding from our audits is that improvements are i
l continuing to be made even though accreditation has been received.
l The audit findings grouped by policy statement element l
for the 10 programs are:
1 I
-m,-_,--.-.m--
,,._,_---,-----,-...-..c.,.
_ _ _,,, -.- -. - ~., - -, -, - - - - - -
t a
The Comissioners Systematic analysis of the jobs to be performed.
A systematic method was used to define tasks per-fonned on the job for eight of the ten training programs reviewed.
Of those eight programs, three used a plant-specific job and task analysis, which included identification of knowledge, skills and abilities (KSA).
The other five programs developed task lists from job analysis which were compared with the content of existing training materials.
Of the other two programs reviewed, one was accredited in 1983 based on INP0's training and qualifications guidelines, which was allowable in lieu of formal job and task analysis and the other was in the technical staff and manager category for which INPO is not requiring a position-specific analysis to establish program content.
Learning objectives derived from the analysis which describe desired performance after training.
Learning objectives could generally be related l
to the task lists in seven of the 10 programs reviewed. Two of the programs did not con-1 sistently contain learning objectives related to documented task lists. No learning objectives were included in one contracted training program in the technical staff and manager j
- category, i
Many learning objectives reviewed did not consistently describe expected trainee behavior in terms of conditions, actions or standards l
governing successful task performance and some were too broad.
These learning objectives would not provide a consistent basis for j
trainee evaluation.
Training design and implementation based on learning objectives.
3 Classroom and simulator training observed were well planned, effective and conducted in a professional and competent manner at all sites.
Lesson plans were generally comprehensive and appropriate at all sites, although in some cases learning objectives were not included and r
.-.,.,--..-..,~,--,.--.---..r.-_.----.----m-
-.-r---e...
,.r,,--
-.-r-...--
3.---
The Consnissioners in other cases plant operating procedures were used for some classroom lesson plans.
The administrative aspects of the training program; e.g., goals and objectives and training staff responsibilities and authority, were clearly articulated by all licensees.
Evaluation of trainee mastery of the objectives during training.
Appropriate procedures for development of trainee tests and standards of performance were available and implemented at all utilities.
In eight of the ten training programs reviewed, test items could be linked to task lists and learning objectives.
In one program, the learning objectives were too broad to determine if an effective linkage existed.
Examinations were not available for review for one contracted training program in the technical staff and manager category.
All programs had adequate procedures in place to prevent compromise of examinations.
Evaluation and revision of the training based on performance of trained personnel in the job setting.
All sites had procedures in place for effective program evaluation and there was evidence that the procedures were being implemented.
Internal and external evaluations were performed at all sites on a regular basis.
The staff at all sites were cualified to perform their respective duties as training or subject matter specialists.
There was evidence in all programs of some problems in implementing mechanisms to ensure that training programs remain current and timely as job performance requirements evolve (e.g., as plant equipment and procedures are changed).
l A general finding that relates to all of the elements of the Policy Statement involves requalification (continuing) i l
I s
j The ComissionGrs 1 1
training. A systematic approach to the development of requalification training has not been perfomed at any of the sites reviewed.
Rather, the requalification programs at these utilities (and at other plants where the staff observed or reviewed INPO's process and documents) are I
based upon meeting NRC regulatory requirements promulgated shortly after TMI. Utility personnel indicated that they are reluctant to change regulation based training programs.
1 However, based on the proposed revisions to 10 CFR Part 55 and staff recommendation, INPO is currently developing specific guidance for performance-based continuing training; i.e., requalification training, i
The final staff reports from the three post-accreditation reviews are attached to H. Denton's letter to Z. Pate dated April 3,1986, which is attached as Enclosure 2.
The NRC post-accreditation reviews have been controversial partly due to an industry perception that the review pro-cedures used by the NRC staff are not consistent with INPO accreditation criteria.
The staff's comparison, described in a February 27, 1986, memorandum to the Comission, con-4 cluded that there are no differences between the staff's and INP0's concept of performance-based training. There is agreement that the intent of INP0's 12 accreditation objectives (INPO 85-002) and the staff's 5 review criteria (i.e., the 5 elements essential to performance-based training from the Comission's Policy Statement) are con-sistent. However, INPO did have a concern about the meaning and interpretation of some staff review questions.
Based upon lessons learned from the first three post-accreditation reviews and our discussions with INPO, the staff has modified its review questions and procedures to clarify the intent of some questions and to increase the efficiency of staff reviews. The revised review questions are also attached to H. Denton's April 3, 1986 letter (Enclosure 2).
The staff is discussing with INPO recomendations in three areas to improve their process. These areas are:
applica-tion of performance-based training methods to requalification training, remedial training for job incumbents and examin-ation security. As part of NRC-INPO coordination activities, INPO will provide an observer at the exit briefing for the 4
next NRC post-accreditation review.
Event Based Reviews - Event-based reviews are conducted on an as-needed basis. These reviews are precipitated by an operating event at a plant or if deficiencies noted in inspection reports or SALPs warrant further investigation.
Two such reviews were conducted in 1985; one at the l
i f
The Commissioners V. C. Summer plant (accredited) in response to the February 21, 1985, high start-up rate and power range trip, and one at the Davis-Besse plant (nonaccredited) based on their SALP rating of 3 in the area of training.
The reviews found weaknesses in the implementation of the in-plant training at V. C. Summer and problems in the development of training programs at Davis-Besse. A more detailed description of these reviews is at Enclosure 1.
NRC Examination Results - Licensed operator examination reports were reviewed to extract information on licensee performance which could be linked to training effective-ness. Analysis of NRC examination results (including requalification examination results) do not yet show any i
statistically significant difference in pass / fail rates between accredited and nonaccredited training programs.
These results and an explanation of the data limitations are contained in Enclosure 1.
Status:
The industry connitment is to have 10 training programs at each of 61 sites " ready for accreditation" by a
December 31, 1986. This requires that 610 utility Self l
Evaluations Reports be accepted by INPO. As of April 1, l
1986, 348 programs at 56 sites have had SERs accepted by INPO, 228 of these have been reviewed at 45 sites in 52 INPO Team Visits, and 138 of these programs have been accredited at 30 sites.
It is estimated that the average time from the acceptance of the SER to final accreditation is about 6 to 8 months. This time varies based on the number and scope of recommendations that result from the Team Visit. The staff has found through observations of Team Visits that, in some cases (about 50%), the Team Visit may have been premature; i.e., utility implementation of accredited training programs or program development were not complete at the time of the team visit.
Regional Administrators and inspection reports indicate that there has been insufficient time and experience with accredited training programs to evaluate the effectiveness of accreditation on personnel performance.
In fact, there has been some concern among the staff that schedule pressure and resource requirements for accreditation may have a detrimental effect on existing training.
In a few areas, the utilities' training staff workload for accreditation resulted in cancellation of training or substitution with self-study.
==
Conclusion:==
Significant progress is being made by industry in improving training and implementing the Connission's
_y__~~
.-,.m
i The Commissioners Policy Statement. As of April 1, 198'6, 138 training programs at 30 sites have been accredited and 210 additional Self Evaluation Reports have been accepted by INP0.
Less than half (262 of 610) of the Self Evaluation Reports remain to be accepted by INP0. This progress has required a substantial commitment of industry resources.
While significant training improvements have been observed, there have been training deficiencies identified in both accredited and nonaccredited training programs (see ).
Deficiencies of the type identified are to be expected given the magnitude of the effort involved and the application of performance-based concepts by an industry that traditionally based training on NRC require-ments. Also, there have been instances of communication difficulties; questions about differences between training review standards and criteria; and utility enforcement concerns about modifying NRC required training programs to meet INP0's accreditation program.
The staff and INP0 are working on these problems and both believe progress is being made.
The conflict between current NRC regulations and the objectives of accreditation must be promptly addressed.
The staff has found that utility requalification training programs in general have not been systematically analyzed, developed, and implemented due to a reluctance by licensees and INP0 to address training areas which are currently regulated.
INP0 has activities underway to develop Jeneric guidance which will result in the application of performance-based training concepts to requalification training.
The changes proposed in the revisions to 10 CFR 55 - Operators' Licenses are intended to help alleviate this conflict.
The staff believes that rulemaking related to training programs as described in the Commission's Policy Statement on Training and Qualifications should continue to be oeferred.
Significant remainin~g actions are:
1.
Completion of the remaining 262 Self Evaluation Reports (SERs) and their acceptance by INP0 by December 31, 1986, is required to meet NUMARC's commitment to the Commission; 2.
Site visits to assess the quality of utility SERs are scheduled to be completed by December 1987; and
The Comissioners 3.
Accreditation is scheduled to be complete for all 610 programs by May 1988.
3
_ Ly~
-u Victor Stello, W.
Executive Director for Operations
Enclosures:
1.
Independent Evaluation of Industry Training 2.
H. Denton letter to Z. Pate dated April 3, 1986 DISTRIBUTION:
Commissioners OGC OPE OI OCA OIA ACRS EDO SECY
i i
r t
ENCLOSURE 1 7~-'
v=
w-O m-#
'-'m
^*w w
INDEPENDENT EVALUATION OF INDUSTRY TRAINING INTRODUCTION The NRC staff has independently evaluated applicants' and licensees' training programs to ensure that the desired results are achieved and to evaluate the possible need for further NRC action. As described in SECY-85-288, the staff's evaluation of training programs includes evaluations of the INP0 accreditation process (i.e., observation of INP0 accreditation teams and the Accreditation Board) and accredited training programs (i.e., independent staff reviews of accredited training programs).
In addition, the staff has compiled data from Systematic Assessment of Licensee Performance (SALP) reports, licensed operator examination reports, inspection reports, and input from NRC regional offices. Data from licensee event reports (LERs) were reviewed but not used because LERs do not include training as a separate cause code, thus, training as a root cause could not be determined from the data.
The results of the staff's evaluation are discussed below.
DISCUSSION Event-based Evaluations Two event-based evaluations of training programs (one accredited and one not accredited) were performed in response to reportable operating events during 1985. These evaluations were perfomed at V. C. Sumer in July 1985 and et Davis-Besse in September 1985.
I The review at V. C. Summer was conducted as a follow-up to a high start-up i
rate and power range trip which involved an operating error by a reactor operator trainee being supervised during on-the-job training (0JT) by a licensed senior operator. The licensed operator, nonlicensed operator and shift technical advisor (STA) programs at V. C. Sumer were accredited at the time of the event. The staff's review focused on aspects of training j
directly related to the incident.
The staff found that the trainee was i
improperly assigned to do the start-up in that he had not been prepared for i
the control manipulation to be performed and that the SRO had not received l
instruction on the procedures for conduct of OJT and failed to adequately j
supervise the trainee at the controls. This event represents a significant breakdown in the utilities' implementation of the OJT portion of its
(
accredited training programs.
The NRC review of training at Davis-Besse was performed at the Commission's request as a follow-up to a SALP rating of category 3 for the functional area of training and the loss of auxiliary feedwater event of June 1985. The training programs at Davis-Besse are not accredited. The staff found that Davis-Besse had a substantial amount of work to do to upgrade their training i
programs to achieve perfomance-based training.
j
i
, l l
In summary, the V. C. Summer event was a significant breakdown in the implementation of the 0JT portion of its accredited training program.
The staff's review did not identify a programmatic problem with INP0's accreditation criteria or objectives for the conduct of in-plant training.
Operator Licensino Examination Results License examination data was analyzed to determine the number of license candidates who passed (or failed) the written and oral licensed operator or senior operator examinations for FY 1985 and the effect of accreditation on candidate performance.
The results of the analysis and the status of accreditation of licensed operator training programs follows:
No.
No.
Pass No.
Pass Oral BWR Plants Plants Candidates Written Candidates and Simulator Accredited 7
88 86%
96 93%
Nonaccredited 21 357 77%
351 86%
PWR Plants Accredited 19 323 84%
309 88%
Nonaccredited 29 349 90%
324 95%
Statistical analysis of the differences between pass rates at accredited and nonaccredited plants were found to be nonsignificant (p >.05) for written and operating examinations for both PWR and BWR plants.
Thus, the effect of accreditation on training programs cannot yet be ascertained using operator license candidate examination performance.
This may be a result of timing between completion of training, subsequent NRC examination and accreditation (e.g., candidates at accredited plants may have completed training prior to accreditation). Following implementation of the proposed revisions to 10 CFR 55, the staff's data base will include whether or not the candidate completed an accredited training program. Also staff efforts to improve the content and quality of NRC examinations should eliminate concerns about examination validity as a potential factor influencing examination results.
Similarly, a review of NRC requalification examination results could not be differentiated by accreditation status.
This occurs for two reasons:
(1) requalification program content is currently based upon NRC requirements and has not yet been upgraded to meet performance-based training standards, (2) the changing scope and procedures for NRC requalification examinations prior to February 1985 preclude their use for comparative analysis. The requalification program results are as follows:
7 F
I 1 i
Facilities Satisfactory Marginal Unsatisfactory FY 1984 26 19 3
4 FY 1985 17 9
4 4
Note:
NRC examines approximately 20% of candidates at a facility. The requalification program is evaluated based upon the following
~
criteria:
Satisfactory = 80% of candidates passed Marginal = 60% - 79% of candidates passed Unsatisfactory = less than 60% of candidates passed Licensed Operator Examination Reports Examination reports from the five NRC regional offices were reviewed for all plants to extract data describing industry training program effectiveness.
These reports identify generic weaknesses, if any, found from written and operating examinations.
Generic weaknesses were identified in the examination reports for 10 of 18 plants with accredited programs (55%) and in 39 of 58 plants with nonaccredited programs (67%). The generic weaknesses at both accredited and non-accredited sites identified in the staff analysis are:
Most of the generic weaknesses were related to oral / simulator examinations.
Candidates had difficulty in the use of procedures of all kinds (normal, abnormal, or emergency).
In one plant, candidates had difficulty locating procedures because the numbers had been changed.
Candidates were deficient in basic electrical theory, nuclear power fundamentals, and generic and plant-specific system knowledges.
Candidates exhibited weaknesses in ability to analyze and diagnose plant conditions.
Candidates exhibited weaknesses in tasks that are infrequently performed.
Many candidates were deficient in communications skills and, in the case of senior reactor operator (SRO) candidates, skills in directing control room activity.
NRC Inspection Reports I
NRC inspection reports were reviewed to assess the progress and impact of INPO accreditation on facility training programs. While the information contained in inspection reports lends insight into the current strengths, l
t '
weaknesses, and progress of facility training programs, a direct comparison of accredited and nonaccredited training programs was hampered by two constraints.
First, for the 30 currently-accredited sites, only five facilities had training programs that were accredited at the time of the inspection. Second, few of the inspections were conducted using the revised, performance-based inspection procedures, so most of the information below is based on the superseded inspection procedures.
For these reasons, inspection results provide limited evidence of the effectiveness of accreditation but do provide a baseline for future evaluations.
Although considerable variations in training programs were reported across facilities, with few exceptions, neither the number nor the nature of training program strengths or deficiencies was found by inspectors to be directly related to accreditation status.
Training deficiencies, unresolved items and open items were noted at accredited, nonaccredited, and soon to-be-accredited facilities.
The training inspection findings are organized into five areas:
Administrative Issues, Training Program Content, Record-Keeping, Instructor Performance and Qualifications, and Training Evaluation. Differences between accredited and nonaccredited facilities, where applicable, are discussed.
Administrative Issues. Administrative oversight of training program development, implementation, and evaluation is one area where the influence of INP0 accreditation was detected.
Inspections of nonaccredited facilities often attributed training program deficiencies to a lack of procedural guidance and management attention, whereas inspections of accredited facilities generally indicated that the problems detected involved a breach of procedures and guidelines in place. Moreover, inspections of those facilities that were soon to be accredited often included comment on the increase in attention and the quality of changes occurring in training programs during this transition to a performance-based approach.
The lack of formal training program guidance and management attention in nonaccredited facilities was noted in a variety of instances, including a lack of detailed administrative instructions for implementing training programs, quality control of training not in compliance with approved administrative and procedural controls, perceptions of training department unresponsiveness, failure of training procedures to delineate comprehensive courses of study, and inadequate or nonexistent guidelines for the evaluation of trainee proficiency.
Other administrative shortcomings more directly related to NRC regulations were found in both accredited and nonaccredited facilities.
Instances of these types of training problems include:
(1) changes (particularly reductions) in training program curricula without notification of NRC; (2) failure to remove from licensed duties those operators who failed the
f y requalification examination; (3) failure of candidates to successfully i
complete all training requirements prior to taking the NRC examination; and (4) unwarranted exemptions of licensed operators from participation in requalification training and examinations.
Instances of this last problem were noted at six facilities, including one accredited facility.
Training Program Content. The positive impact of INPO accreditation was also seen in the quality of lesson plans and course outlines. While most facilities were reported to have satisfactory course plans, objectives, and outlines, those of accredited facilities were often commended for their detail and performance-based orientation.
(In one case, however, an accredited facility was found to have excellent system lesson guides, but deficient instructor guides).
Training texts were also deemed satisfactory for the majority of facilities.
In several cases, the training content itself was found to be deficient, due to either a too-general orientation, or to the absence of plant-specific information, such as procedures and technical specifications.
The mode of instruction used for training was generally found to contain a good balance of classroom, hands-on participation (particularly with regard to the use of simulators), and on-the-job components. However, in a few instances, at both accredited and nonaccredited facilities, an overreliance on self-study was identified.
In a few cases, the time allocated for training on specific topics was found to be overly compressed and concern was expressed that the time intervals between associated training topics were too long, leading to possible deterioration in knowledge and skills necessary for subsequent courses.
The three most common problems cited with regard to training program content was the lack of training coverage of certain topics, particularly those topics addressed in regulatory requirements and directives. The most frequently cited training content deficiency was the implementation of NUREG-0737 topics, especially training in the mitigation of core damage and performance of control manipulations required by the March 28, 1980, letter from Harold Denton.
Numerous shortcomings were cited with respect to these two requirements across accredited and nonaccredited facilities alike.
The second area where content was found deficient was training on operating events and training on plant and procedure changes.
A significant number of inspections found this training to be deficient either with regard to what was being taught or the manner in which it was taught.
For example, in one (accredited) facility, LERs and other operational experience information was read verbatim to students, while at another (accredited) facility, operators were required to read information on their own.
The third area of deficiency involved shortcomings in licensed operator review of emergency and abnormal procedures and technical specifications.
Often the deficiencies in this area involved the fact that these topics were 1
g e i j
taught in combination with other topics (e.g., systems, control manipulations).
Inspectors felt procedures and technical specifications were not being covered in insufficient depth.
Record Keeping. One of the most prevalent types of generic deficiency cited across facilities involved inaccuracies and inadequacies in training record documentation and maintenance.
Instances of training record shortcomings were noted in nearly half of the facilities inspected.
Inspectors repeatedly found omissions, and sometimes inaccuracies, in trainee records.
Documentation deficiencies included records of course attendance, examination grades or other measures of student performance, and records of remedial training.
In other cases, inspectors noted that although training records appeared generally complete, the manner in which this information was stored made retrieval difficult and overly time-consuming.
Instructor Performance and Qualifications.
In the few comments regarding instructor performance, inspectors noted that classes were taught in a professional, effective manner. Virtually all shortcomings in this area pertained to instances in which either instructors were not SRO-certified, or had failed to actively perform licensed duties for a period of 4 months or greater.
In a few instances, inspectors questioned the technical qualifications of an instructor based on their personal observation of the instructor or interviews with students.
Training Evaluation.
Inspection reports noted that utilities used three methods to evaluate training:
follow-up on detected trainee deficiencies, utility-administered examination preparation and grading, and mechanisms for feedback of instructor effectiveness. Differences noted across facilities
- i with regard to these three methods of evaluation were not found to be attributable to facility accreditation status.
A number of facilities were commended for their programs of training deficiency remediation.
However, there were a greater number of instances in which facilities were cited for shortcomings in this area, including failure to provide remedial training for deficiencies found at individual, group or programatic levels.
4 Instances of deficiencies in utility-administered examinations included:
(1) overly easy or simplistic examination questions; (2) failures to link examination questions to learning objectives; (3) a lack of criteria for passing examinations or other performance measures (oral, written and simulator); (4) reductions in passing scores on requalification examinations; (5) inaccuracies in examination grading; and (6) breaches of examination security, including storage of examinations in easy access areas, and repeated administration of the same examination.
Although few inspections contained evaluations of instructor feedback, in i
those reports in which it was discussed, roughly half of the comments indicated that mechanisms of instructor feedback were satisfactory, while the i
other half cited deficiencies.
In the.latter. instances, inspections-i I
i generally reported that either no formal method existed for auditing or evaluating instructor performance, or that student feedback was not encouraged.
Summary. There are a few noteworthy conclusions from the inspection reports reviewed. First, INP0 accreditation has fonnalized procedural guidance covering training program development, implementation and evaluation.
Second, accredited and to-be-accredited facilities appear to exhibit a heightened management attention to training issues. Third, INPO accreditation would seem to ensure that training content is based upon performance-based learning objectives, course outlines, and other associated documentation, though weaknesses were identified in training on operational experience information. Also, it appears that accreditation does not assure and is not intended to assure that NRC training requirements are met.
Regulatory deviations and violations were detected fairly equally across nonaccredited, accredited, and to-be-accredited facilities.
Systematic Assessment of Licensee Performance (SALP)
~
SALP reports from January 1983 to May 1985 were reviewed to provide information on both accredited and nonaccredited licensee training programs.
SALP reports for 44 operating plants were reviewed for training information.
It is important to note that SALP rating periods are long, i.e., 12-18 months, so that in some cases, the SALP review ended prior to accreditation while in other cases accreditation activities were underway at the utility during the SALP reporting period. Thus, direct comparisons between accredited and nonaccredited programs is not possible.
The review encompassed information found in the following SALP report l
categories:
(1) Plant Operations (2) Radiological Controls (3) Maintenance (4) Licensee Activities (5) Training (when provided as a separate element)
These areas were selected for review because infonnation related to training programs and job positions covered by INPO accreditation would be contained in these categories.
It is important to note that SALP rating criteria were not developed specifically for the purpose.of evaluating whether performance-based training programs comply with the objectives stated in the NRC Policy Statement on Training and Qualification. Also, only seven SALP reports for operating plants were available for this review which contained a separate training category.
Information contained in SALP reports in addition to ratings was reviewed to identify training program changes resulting from accreditation and to identify generic training program weaknesses.
I l
l l
l
1 Based on SALP reports, licensee training programs, both accredited and nonaccredited, have been described as being generally well-defined and implemented. General findings regarding effects of accreditation on licensee training programs are:
(1) management attention to training development has increased, (2) training staff resources are increasing, and (3) training programs are becoming more formalized.
Generic training problems identified in SALP reports involved both accredited and nonaccredited training programs. These problems include:
Weaknesses in incorporation of procedure changes, license changes and design changes into training.
Compliance and adherence to procedures by all categories of plant personnel was identified as a generic problem in SALP reports.
Administrative issues such as control over training documentation and training records.
j Quality of utility-administered examinations was found to be a weakness in nonaccredited programs.
The use of self-study for training in nonaccredited programs was noted and linked to a shortage of training staff.
Weaknesses in licensed operator requalification training, including inadequate operator evaluation during requalification training, exist in both accredited and nonaccredited training programs.
In conclusion, SALP reports indicate that accreditation is having a positive effect on improving training. However, problems in performance of a variety of operating activities involving trained personnel and weaknesses in training program administration, content, and evaluation need to be corrected.
Regional Evaluations of Industry Training Improvements Each NRC Regional Administrator provided comments on the effectiveness of industry training under the NRC Policy Statement on Training and
~
Qualification. Their comments are attached.
In summary, the Regional Administrators report that industry is actively pursuing accreditation and that improvements have been noted, particularly with regard to management attention to training.
Some training deficiencies have been noted at both accredited and nonaccredited sites.
In general, the regions believe that there has been insufficient time to draw definitive conclusions on the effectiveness of accreditation since only a few sites have been accredited for a long enough period to observe difference;.in performance.
jo UNITED STATES g
[
g NUCLEAR REGULATORY COMMISSION-j REGION I 0,,
,4 631 PARK AVENUE
,o KING OF PRUSSIA. PENNSYLVANIA 19406 14 MAR 1986 MEMORANDUM FOR:
Harold'R. Denton, Director, Office of Nuclear Reactor Regulation James M. Taylor, Director, Office of Inspection and Enforcement FROM:
Thomas E. Murley, Regional Administrator
SUBJECT:
REGIONAL INPUT TO PROGRESS REPORT EFFECTIVENESS OF TRAINING UNDER THE POLICY STATEMENT ON TRAINING AND
~
QUALIFICATION This refers to your request of January 10, 1986 for Region I input to the Commission progress report on the above subject. We support the industry initiative as a positive approach to training program improvements which should result in improved personnel performance.
Discussions with licensee personnel indicate their active pursuit of the INPO goals with a consider-able expenditure of resources.
With respect to the three specific questions you posed we offer the following:
(1) We believe overall improvements in personnel performance have occurred at most Region I facilities.
The extent to which this can be attri-buted t'o INPO accreditation activities is difficult to measure. We certainly believe 'it is a principal factor.
(2) We have.' observed no training areas that have experienced a decline as a result of the INPO accreditation activities.
(3) Region I licensees are actively seeking INP0 accreditation or imple-menting accredited programs according to the INPO criteria. With respect to the latter, however, there are only a few licensees who have fully accredited programs, therefore the data base is small at this point in time.
In order to enhance appreciation of licensee initiatives and INPO review methods, Region I has observed INPO site accreditation reviews at the Susque-hanna and Limerick facilities.
Region I currently tracks licensee progress toward completion of the accreditation process in conjunction with the SALP process. We will forward applicable inspection reports which address failure to implement commitments.
Trends in personnel performance attributable to INP0 accreditation activities will be addressed in SALP reports.
1 l
1
~
Multiple Addresses 2
14 MAR 1986 We. will continue to provide copies of Operator Licensing Examination Reports to OHFS by routine distribution.
/lualla t
Thomas E. Murley Regional Administrator cc:
V. Stello, EDO
l
February 27, 1986 1
MEM0RANDUM FOR: Harold R. Denton, Director Office of Nuclear Reac, tor Regulation l
J FROM:
J. Nelson Grace j
Regional Administrator f
SUBJECT:
REGIONAL INPUT TO PROGRES$ REPORT ON EFFECTIVENESS OF TRAINING
\\
l In reply to your January 10, 1986 memorandum, we have the following general comments:
1 Licensee efforts to achieve INP0 accreditation of training programs have resulted in overall improvements in the administration and quality of plant training.
SALP V training functional area category ratings and trends are provided as an enclosure to support this conclusion. To date, eleven Region !! plants have been assessed in SALP V.
Six plants, including two rated Category 3, had improving trends, four plants had constant trends, and one plant was not trended in the training functional area.
Improvements in training quality were not reflected in performance indicators.
A qualitative review of selected LERs revealed no significant improvements in plant performance directly attributable to the INP0 accreditation process.
1 Although improvements in training quality have been apparent since the INP0 accreditation initiative, Region !! inspectors have identified deficiencies in implementation of, several specific INP0 accredited training programs.
For l
example, an inspection conducted October 1-5,1984 (Report No. 50-269/84-25) i at Oconee, revealed deficiencies in the Oconee non-licensed operator, licensed reactor operator, and licensed senior reactor operator training programs not-1 withstanding INP0 accreditation on August 17, 1983.
The deficiencies were as j
follows:
I The Oconee Trainin'g Services Group prepared and implemented three programs l
to define the Duke Power Company, Oconee Nuclear Station Operator Licensing i
Program, Licensed Operator Requalification Program and Non-Licensed Operator J
Requalification Program which were not referenced or incorporated in the Duke Power Company Corporate Nuclear Production Training Plan.
These programs did not have provisions for revision, review, approval, or con-trolled distribution.
1 i
CONTACT:
C. A. Julian FTS: 242-5541 l
hA DR V
1 4
--r re -~ ~ s, e m m.-w.m an-~
-mww.
_a-,
r
a_-
L.
Harold R. Denton 2
p The Production Training Services' training sequence chart did not provide for appropriate sequencing of learning ob;ectives. The as-taught schedule l~
for the previous reactor operator class indicated that a mixture of advanced lectures such as emergency procedWes, were provided in the early weeks of training.
Topic sequencing should instead start from basic systems and components and advance to the more complicated topics of integrated plant operations.
i No formalized, approved student text materials were available.
Students j
relied instead on self written notes which do not assure consistent stand-e ards and objectives.
Formalized instructor lesson plans were being upgraded; however, formats were inconsistent ranging from detailed outlines to simple text formats.
Lesson plan upgrades were left to individual instructors without general i
i guidance to assure lesson plan consistency.
}
An inspection conducted on March 11-28, 1985 (Report No. 348/85-15), at Farley also revealed a deficiency in the INP0 accreditation process. On December 20 j
1984, Farley received accreditation of a licensed (SRO) upgrade program. As of April 1,1985 Farley had not implemented the accredited program due to oojec-l tions from the operations group. Apparently, INP0 reviewed and accredited the program description provided by the licensee prior to actual implementation.
A Safety System Functional Inspection (Report No. 50-250/85-22) conducted at Turkey point revealed a concern over the impact of accreditation efforts on ongoing training programs.
Specifically, since August 1984, formal classroom l
training sessions, for smintenance technicians had been discontinued. Training resources were dedicated to developing training materials to support INP0 accred-itation of the maintenance training program in order to meet a February 1986 Self
-Evaluation Report submittal date. The inspection team concluded that maintenance training conducted in the interim was not adequate to maintain staff proficiency.
4 An inspection (Report No. 50-302/85-01) conducted at Crystal River revealed another potential deficiency in the INp0 accreditation process involving the failure of INP0 programs to address established regulatory requirements and previous regulatory comitments. The inspection team identified a systematic reduction in the scope of the approved licensed operator requalification program.
The reductions in program scope were not submitted to the NRC for review and approval resulting in an overall failure to implement the requalification program I
approved by MRC. Licensed personnel were not all required to fully participate in the requalification program, and licensed instructors were regularly exempted from attendance of requalification lectures and participation in annual requal-ification examinations.
Finally, multiple examples of licensee failure to adequately implement NUREG 0737 comitments indicated a general lack of manage-ment control over tracking and maintenance of commitments made to the NRC.
i l
i i
.--