ML20212N643

Responds to FOIA Request for SECY-86-119 & SECY-86-123. SECY-86-119 Forwarded & Available in PDR.SECY-86-123, Final Rulemaking for Revs to Operator Licensing-10CFR55 & Conforming Amends, Withheld (Ref FOIA Exemption 5)

ML20212N643
Person / Time
Issue date:	08/27/1986
From:	Grimsley D NRC OFFICE OF ADMINISTRATION (ADM)
To:	Graber L NUS CORP.
References
FOIA-86-463 NUDOCS 8608290067
Download: ML20212N643 (1)

Text

_ - _ _ _ _ _ _ - _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ._

pm PQK-oho p *

.g UNITED STATE 3

- o NUCLEAR REGULATORY COldMISSION U j WASHINGTON, D. C. 20555

\...../

Mr. Lyle Graber, Licensing Engineer AUG 2 7 I!Ei Licensing Information Service NUS Corporation IN RESPONSE REFER 2536 Countryside Boulevard TO F01A-86-463 Clearwater, FL 33575-2094

Dear Mr. Graber:

' This is in response to your letter dated June 17, 1986, in which you requested, pursuant to the Freedom of Information Act (F0IA), copies of SECY-86-119 and SECY-86-123.

SECY-86-119 entitled, " Annual Status Report on Implementation of the Comission Policy Statement on Training and Qualification," is being placed in the NRC Public Document Room (PDR) in file folder F01A-86-463 in your name.

SECY-86-123 entitled, " Final Rulemaking for Revisions to Operator Licensing--

10 CFR 55 and Conforming Amendments," contains the predecisional advice, opinions, and recommendations of the staff to the Comission. Because the document reflects the predecisional process between the Commission and the staff, the document is exempt from mandatory disclosure pursuant to Exemption 5 of the F0IA, 5 U.S.C. 552(b)(5), and the Commission's regulations at 10 CFR 9.5(a)(5). Release of the information would tend to inhibit the open and frank exchange of ideas essential to the deliberative process. The document contains no reasonably segregable factual portiens and is being withheld in its entirety.

Pursuant to 10 CFR 9.15 of the Commission's regulations, it has been determined that the information withheld is exempt from production or disclosure and that its production or disclosure is contrary to the public interest. The person responsible for this denial is Mr. John C. Hoyle, Assistant Secretary of the Commission.

This denial may be appealed to the Secretary of the Commission within 30 days from the receipt of thi! letter. Any such appeal must be in writing, addressed to the Secretary of the Commission, U.S. Nur. lear Regulatory Commission, Washington, DC 20555, and should clearly state on the envelope and in the letter that it is an " Appeal from an Initial F0IA Decision."

Siricerely,

" N Donnie H. Grimsley, Director Division of Rules and Records Office of Administration 8608290067 860827  !

PDR FOIA i GRABER86-463 PDR

l's

,p * * "'* %

! c, s  ;

\...../ -

POLICY ISSUE April 18, 1986 (InformatIOn) SECY-86-119 For: The Comissioners l

From: Victor Stello, Jr.

Executive Director for Operations

Subject:

ANNUAL STATUS REPORT ON IMPLEMENTATION OF THE COMMISSION POLICY STATEMENT ON TRAINING AND QUALIFICATION

Purpose:

To inform the Comission of industry progress in imple-menting the Policy Statement on Training and Qualification and the results to date of staff evaluation of the INP0-managed Training Accreditation Program.

Background:

On February 7,1985, the Comission adopted the Policy Statement on Training and Qualification (50 FR 11147, March 20, 1985). The Policy Statement endorsed the INP0-managed Training Accreditation Program because it cncompasses the elements of effective performance-based training which are: analysis of the job, performance-based learning objectives, training design and implementation, trainee evaluat'on, and program evaluation.

In approving the Policy Statement, the Commissian deferred for 2 years rulemaking on training and qualifications in recognition of the industry efforts provided that the industry programs produce the desired results. The Commission also directed the staff to indeper.dently evaluate applicants' and licensees' implementation of improvement programs and to evaluate the possible need for further NRC action.

Subsequent to approval of the Policy Statement, Appendix Number Four of the NRC/INP0 Memorandum of Understanding was signed on July 12, 1985. That Appendix describes the activities of both the NRC staff and INP0 in implementing the Policy Statement.

Contact:

W. T. Russell, NRR D"* * - ' '

492-4803 UN ' # W

> &l f f -

)

. s The Commissioners

• Sumary: SECY-85-288 described the staff's approach to evaluating both individual utilities' progress in improving training and the INP0-managed accreditation program. The approach includes: observation of INPO Team Visits and meetings of the National Nuclear Accrediting Board; review of plant-specific Self Evaluation Reports and INPO team reports; and independent evaluation of training programs' effectiveness through training inspections, post-accreditation reviews, event-based reviews and analysis of performance data (SALP reports, examination reports, etc.).

Discussion: Eacn of the elements of the staff review is discussed below:

Team Visit Observations - As of April 1, 1986, INPO has conducted 58 Training Accreditation Team Visits at 45 sites covering 228 individual training programs. NRC staff has observed 12 of these visits.

The staff has found the process to be thorough, resource-intensive, constructive and professional. Utilization of a team that includes both INP0 staff and utility peer evalu-ators enhances the process. Some staff observers have expressed concern about the potential for inconsistency in applying INP0's objectives and criteria from plant-to-plant.

l INPO methods to ensure consistent application of objectives '

and criteria include meetings with team leaders, training of team reviewers and senior management review of all Team l

Visit reports.

Observations of the National Nuclear Accrediting Board - As of April 1,1986, the Accrediting Board has accredited 138 individual training programs at 30 sites. NRC has had observers at all but a few of the Board meetings. The observers at Accrediting Board meetings have commented on the professionalism, independence and quality of the Board's review. The Board relies heavily on the review work per-formed by INP0 at the Team Visit and in subsequent inter-actions between the INP0 staff and the utility. The staff has expressed some concern (letter from Mr. Denton to Mr. Pate dated January 28,1986) related to INPO's closeout of recommendations between the time of the Team Visit and the Board meeting. Subsequently, the staff has observed an INPO follow-up visit to close out team recommendations at one plant and has questioned INPO Team Leaders on the results of their follow-up visits. Also, the results of follow-up visits are provided to the Board as supplemental reports. The staff has also observed granting of accredita-tion status by the Board based upon comitments for further training development and future program implementation.

__ ___4-_ __ c- _ _ . _ __ _ - - _ _ _ _ . _ - _ _ _ _ _ - . _ _ _ . - _

~

The Comissioners . i

< l INPO has a computer-based tracking system to ensure that '

these commitments are met. The Board reviews-the status of comitments monthly.

Review of Plant-Specific Self Evaluation Reports (SER) and

,' INPO Team Reports - INPD team reports, SERs and other documentation is available to the MC observers of Team Visits and Board meetings. Utilities also make the reports available to appropriate NRC personnel for review or reading on site. Staff review of utility Self Evaluation Reports, field notes, draft team reports and utility responses confims that the reports are consistent with INPO guidance, that INPO reports are thorough and that INPO and utilities generally resolve INP0 recomendations prior to the Board meeting. ,

Traininc Inspections and SALP - Training inspections are intendec to evaluate training effectiveness. The previous approach to inspection of training focused on the adequacy of the licensee's training program and their implementation of that program. The inspection approach contained in the.

procedures which were revised in June 1985.is perfomance-based in that it focuses on the ability of the plant staff to perform their jobs after training rather than on the training program or development process.

In November of 1985, a revised SALP NRC Manual Chapter was l

distributed for implementation that specified licensee training programs be evaluated as a separate functional area. Previously, training had been used as an evaluation criterion for each of the functional areas. One purpose of establishing training as a distinct functional area was to provide a better basis to analyze licensee performance in this area.

Review of training inspection reports indicates that:

accreditation has formalized procedural guidance for training program development, implementation and evalu-

! ation; accreditation has heightened management attention to training issues; and accreditation seems to ensure that training content is based on performance-based learning objectives, course outlines and other documenta-tion. The inspection reports do identify, however, a variety of training program weaknesses in both accredited and nonaccredited programs which confim that further improvements are needed. Review of SALP reports indicate similar positive aspects of accreditation while identifying specific problems in some operating activities and l

weaknesses in training program administration, content and evaluation of both accredited and nonaccredited sites.

l

N ,

t ,<

The Comissioners ,

A detailed discussion of the analysis of inspection reports and SALP reports is provided in Enclosure 1.

Post-Accreditation Reviews - To date the staff has conducted three post-accreditation reviews (Susquehanna, Dresden and Oconee) covering ten training programs. -

Preliminary findings from these audits were provided to INPO in January 1986. Subsequently, NRC and INPO staff .

met and discussed both the preliminary findings and the staff's review procedures. The staff's reports were completed in April 1986.

The Comission's policy statement sets forth five elements which are essential to effective performance based training.

These elements formed the basis for the post-accreditation audits.

In a systematic analysis each element is dependent on the preceding element with feedback to all preceding elements.

For example, the third element, Training Design and Implementation, is based on the second element, Learning Objectives. New program development typically starts with the first element, Systematic Analysis of the Jobs to be Performed.

For existing training programs INPO recommends that training system development build on existing materials.

This was the case for all of the progiams audited. This approach is acceptable for existing training programs provided it is utilized in conjunction with a systematic analysis and the utility continues to evaluate training

. needs based upon job performance. However, it would not be acceptable to enter the process at the training design stage (element 3) and simply develop learning objectives to match existing training materials.

The staff observations of the ten accredited programs audited indicate that procedures are in place to allow effective backfit of performance-based training to existing programs. While all programs are implementing most; aspects of the five training elements, we coserved some instances where further improvements are warranted. However, an importent finding from our audits is that improvements are continuing to be made even though accreditation has been received.

The audit findings grouped by policy statement element for the 10 programs are:

,,,,my,., , . , . _ , _ , .,

s The Commissioners ,

Systematic analysis of the jobs to be performed.

f A systematic method was used to define tasks.per-formed on the job for eight of the ten training programs reviewed. Of those eight programs, three used a plant-specific job and task analysis, which included identification of knowledge, skills and ,

abilities (KSA). The other five programs developed task lists from job analysis which were compared with the content of existing training materials. Of the other two programs reviewed, one was accredited in 1983 based on INPO's training and qualifications guidelines, which was allowable in lieu of formal job and task analysis and the other was in the technical staff and manager category for which' INPO is not requiring a position-specific analysis to establish prngran content.

Learnine objectives derived from the analysis which describe desired performance after training.

- Learning objectives could generally be related i to the task lists in seven of the 10 programs reviewed. Two of the programs did not con-sittently contain learning objectives related to documented task lists. No learning objectives were included in one contracted training

!' program in the technical staff and manager category.

- Many learning objectives reviewed did not consistently describe expected trainee behavior in terms of conditions, actions or standards governing successful task performance and some were too broad. These learning objectives would not provide a consistent basis for trainee evaluation.

Training design and implementation based on learning objectives.

- Classroom and simulator training observed were '

well planned, effective and conducted in a professional and competent manner at all sites.

- Lesson plans were generally comprehensive and appropriate at all sites, although in some cases learning objectives were not included and

The Comissioners .

in other cases plant operating procedures were used for some classroom lesson plans.

- The administrative aspects of the training program; e.g., goals and objectives and training staff responsibilities and authority, were clearly articulated by all licensees.

Evaluation of trainee mastery of the objectives during training.

- Appropriate procedures for development of trainee tests and standards of performance were available and implemented at all utilities.

- In eight of the ten training programs reviewed, test items could be linked to task lists and learning objectives. In one program, the learning objectives were too broad to determine if an effective linkage existed. Examinations were not available for review for one contracted training program in the technical staff and manager category.

- All programs had adequate procedures in place to prevent compromise of examinations.

• Evaluation and revision of the training based on performance of trained personnel in the job setting.

- All sites had procedures in place for effective program evaluation and there was evidence that the procedures were being implemented.

- Internal and external evaluations were performed at all sites on a regular basis.

- The staff at all sites were qualified to perform their respective duties as training or subject matter specialists.

- There was evidence ir, all programs of some problems in implementing mechanisms to enstre that training programs remain current and timely as job perfonnance requirements evolvt (e.g., as plant equipment and procedures are changed).

A general finding that relates to all of the elements of the Policy Statement involves requalification (continuing)

e f The Comissioners - .

training. A systematic approach to the development of requalification training has not been perfomed at any of the sites reviewed. Rather, the requalification programs at these utilities (and at other plants where the staff observed or reviewed INP0's process and documents) are based upon meeting NRC regulatory requiretrents promulgated shortly after TMI. Utility personnel indicated that they are reluctant to change regulation based training programs.

However, based on the proposed revisions to 10 CFR Part 55 and staff recommendation, INPO is currently developing specific guidance for perfomance-based continuing training; 1.e., requalification training.

The final staff reports from the three post-accreditation i reviews are attached to H. Denton's letter to Z. Pate dated April 3,1986, which is attached as Enclosure 2.

The NRC post-accreditation reviews have been controversial partly due to an industry perception that the review pro-cedures used by the NRC staff are not consistent with INPO accreditation criteria. The staff's comparison, described

in a February 27, 1986, memorandum to the Comission, con-

,- cluded that there are no differences between the staff's and INP0's concept of performance-based training. There is agreement that the intent of INPO's 12 accreditation objectives (INPO 85-002) and the staff's 5 review criteria (i.e., the 5 elements essential to perfomance-based training from the Commission's Policy Statement) are con-sistent. However, INPO did have a concern about the meaning and interpretation of some staff review questions. Based upon lessons learned from the first three post-accreditation reviews and our discussions with INPO, the staff has l

' modified its review questions and procedures to clarify the intent of some questions and to increase the efficiency of staff reviews. The revised review questions are also attached to H. Denton's April 3,1986 letter (Enclosure 2).

The staff is discussing with INPO recommendations in three areas to improve their process. These areas are: applica-tion of performance-based training methods to requalification training, remedial training for job incumbents and examin-ation security. As part of NRC-INPO coordination activities, INP0 will provide an observer at the exit briefing for the next NRC post-accreditation review.

Event Based Reviews - Event-based reviews are conducted on an as-needed basis. These reviews are precipitated by an operating event at a plant or if deficiencies noted in inspection reports or SALPs warrant further investigation.

Two such reviews were conducted in 1985; one at the

The Commissioners ,

i V. C. Summer plant (accredited) in response to the February 21, 1985, high start-up rate and power range trip, and one at the Davis-Besse plant (nonaccredited) based on their SALP. rating of 3 in the area of training.

The reviews found weaknesses in the implementation of the in-plant training at V. C. Summer and problems in the development of training programs at Davis-Besse. A more detailed description of these reviews is at Enclosure 1.

NRC Examination Results - Licensed operator examination reports were reviewed to extract information on licensee performance which could be linked to training effective-ness. Analysis of NRC examination results (including requalification examination results) do not yet show any statistically significant difference in pass / fail rates between accredited and nonaccredited training programs.

These results and an explanation of the data limitations are contained in Enclosure 1.

I Status: The industry commitment is to have 10 training programs

! at each of 61 sites " ready for eccreditation" by December 31, 1986. This requires that 610 utility Self

Evaluations Reports be accepted by INPO. As of April 1, 1986, 348 programs at 55 sites have had SERs accepted by INPC, 228 of these have been reviewed at 45 sites in 52

INPO Team Visits, and 138 of these programs have been i accredited at 30 sites. It is estimated that the average l

time from the acceptance of the SER to final accreditation t

is about 6 to 8 months. This time varies based on the number and scope of recommendations that result from the Team Visit. The staff has found through observations of Team Visits that, in some cases (about 50%), the Team Visit may have been premature; i.e., utility implementation of accredited training programs or program development were not complete at the time of the team visit.

Regional Administrators and inspection reports indicate that there has been insufficient time and experience with accredited training programs to evaluate the effectiveness of accreditation on personnel performance. In fact, there has been some concern among the staff that schedule pressure and resource requirements for accreditation may have a

! detrimental effect on existing training. In a few areas, l the utilities' training staff workload for accreditation resulted in cancellation of training or substitution with self-study.

Conclusion:

Significant progress is being made by industry in improving training and implementing the Commission's

The Commissioners .

Policy Statement. As of April 1, 1986, 138 training programs at 30 sites have been accredited and -210 .

additional Self Evaluation Reports have been accepted by '

INPO. Less than half (262 of 610) of the Self Evaluation Reports remain to be accepted by INP0. This progress has required a substantial commitment of industry resources.

While significant training improvements have been observed, there have been training deficiencies identified in both accredited and nonaccredited training programs (see Enclosure 1). Deficiencies of the type identified are to be expected given the magnitude of the effort involved and the application of performance-based concepts by an industry that traditionally based training on NRC require-ments. Also, there have been instances of communication difficulties; questions about differences between training review standards and criteria; and utility enforcement concerns ebout modifying NRC required training programs to meet INP0's accreditation program. The staff and INP0 are working on these problems and both believe progress is being made.

The conflict between current NRC regulations and the objectives of accreditation must be promptly addressed.

The staff has found that utility requalification training programs in general have not been systematically analyzed, developed, and implemented due to a reluctance by licensees and INP0 to address training areas which are currently regulated. INP0 has activities underway to develop generic guidance which will result in the application of performance-based training concepts to requalification training. The changes proposed in the revisions to 10

~

CFR 55 - Operators' Licenses are intended to help alleviate this conflict.

The staff believes that rulemaking related to training programs as described in the Comission's Policy Statement on Training and Qualifi;ations should continue to be deferred.

l l

l Significant remaining actions are:

l

1. Completion of the remaining 262 Self Evaluation Reports (SERs) and their acceptance by INP0 by i

December 31, 1986, is required to meet NUMARC's commitment to the Comission;

2. Site visits to assess the quality of utility SERs are scheduled to be completed by December 1987; and l

The Commissioners ,

3. Accreditation is scheduled to be complete for all

, 610 programs by May 1988. ,

~

c-.__

Victor Stello, J .

Executive Director for Operations

Enclosures:

1. Independent Evaluation of Industry Training

2. H. Denton letter to Z. Pate dated April 3, 1986 l

l l

l

lO E Enclosure 1 ,

'- INDEPENDENT EVALUATION OF INDUSTRY TRAINING _

INTRODUCTION The NRC staff has independently evalu6ted applicants' and licensees' training programs to ensure that the desired results are achieved and to evaluate the possible need for further NRC action. As described in SECY-85-288, the staff's evaluation of training programs includes evaluations of the INPO accreditation process (i.e., observation of INPO accreditation teams and the Accreditation Board) and accredited training programs (i.e., independent -

staff reviews of accredited training programs). In addition, the staff has compiled data from Systematic Assessment of Licensee Performance (SALP) orts, and input

~

reports, from NRC licensed regionaloperator offices.examination reports, inspection Data from licensee rep (LERs) were event reports reviewed but not used because LERs do not include training as a separate

cause code, thus, training as a root cause could not be determined from the data. The results of the staff's evaluation are discussed below.

DISCUSSION Event-based Evaluations Two event-based evaluations of trainirg programs (one accredited and one not accredited) were perfonned in responst. to reportable operating events during 1985. These evaluations were perfonned at V. C. Sumer in July 1985 and at '

! Davis-Besse in September 1985.

The review at V. C. Sumer was conducted as a follow-up to a high start-up i rate and power range trip which involved an operating error by a reactor

operator trainee being supervised during on-the-job training (0JT) by a licensed senior operator. The licensed operator, nonlicensed operator and shift technical advisor (STA) programs at V. C. Sumer were accredited at the time of the event. The staff's review focused on aspects of training directly related to the incident. The staff found that the trainee w n improperly assigned to do the start-up in that he h'ad not been prepared for

! the control manipulation to be perfonned and that the SRO had not received

! instruction on the procedures for conduct of OJT and failed to adequately supervise the trainee at the controls. This event represents a significant breakdown in the utilities' implementation of the OJT portion of its accredited training programs.

The NRC review of training at Davis-Besse was perfonned at the Commission's request as a follow-up to a SALP rating of category 3 for the functional area of training and the loss of auxiliary feedwater event of June 1985. The training programs at Davis-Besse are not accredited. The staff found that Davis-Besse had a substantial amount of work to do to upgrade their training programs to achieve perfonnance-based training.

(

~

In summary, the V. C. Summer event was a significant breakdown in the implementation of the OJT portion of its accredited training program. The staff's review did not identify a programmatic problea with INP0's accreditation criteria or objectives for the conduct of in-plant training.

Operator Licensing Examination'Results License examination data was analyzed to determine the number of license candidates who passed (or failed) the written and oral licensed operator or senior operator examinations for FY 1985 and the effect of accreditation on candidate performance. The results of the analysis and the status of accreditation of licensed operator training programs follows:

No. No. Pass No. Pass Oral BWR Plants Plants Candidates Written Candidates and Simulator Accredited 7 88 86% 96 93%

Nonaccredited 21 357 77% 351 86%

PWR Plants Accredited 19 323 84% 309 88%

Nonaccredited 29 349 90% 324 95%

Statistical analysis of the differences between pass rates at accredited and nonaccredited plants were found to be nonsignificant (p > .05) for written and operating examinations for both PWR and BWR plants. Thus, the effect of accreditation on training programs cannot yet be ascertained using operator license candidate examination performance. This may be a result of timing between completion of training, subsequent NRC examination and accreditation (e.g., candidates at accredited plants may have completed training prior to accreditation). Following implementation of the proposed revisions to 10 CFR 55, the staff's data base will include whether or not the candidate completed an accredited training program. Also staff efforts to improve the content and quality of NRC examinations should eliminate concerns about examination validity as a potential factor influencing examination results.

Similarly, a review of NRC requalification examination results could not be differentiated by accreditation status. This occurs for two reasons:

(1) requalification program content is currently based upon NRC requirements and has not yet been upgraded to meet performance-based training standards, (2) the changing scope and procedures for NRC requalification examinations prior to February 1985 preclude their use for comparative analysis. The requalification program results are as follows:

l l

Facilities Satisfactory Marginal Unsatisfactory FY 1984 26 19 3 4 FY 1985 17 9 4 4 Note: NRC examines approximately 20% of candidates at a facility. The requalification program is evaluated based upon the following criteria:

Satisfactory = 80% of candidates passed Marginal = 60% - 79% of candidates passed Unsatisfactory = less than 60% of candidates passed Licensed Operator Examination Reports Examination reports from the five NRC regional offices were reviewed for all plants to extract data describing industry training program effectiveness.

These reports identify generic weaknesses, if any, found from written and operating examinations.

Generic weaknesses were identified in the examination reports for 10 of 18 plants with accredited programs (55%) and in 39 of 58 plants with nonaccredited programs (67%). The generic weaknesses at both accredited and non-accredited sites identified in the staff analysis are:

Most of the generic weaknesses were related to oral / simulator examinations.

• Candidates had difficulty in the use of procedures of all kinds (normal, abnormal, or emergency). In one plant, candidates had difficulty locating procedures because the numbers had been changed.

!

• Candidates were deficient in basic electrical theory, nuclear power l fundamentals, and generic and plant-specific system knowledges.

• Candidates exhibited weaknesses in ability to analyze and diagnose plant i conditions.

• Candidates exhibited weaknesses in tasks that are infrequently performed.

Many candidates were deficient in communications skills and, in the case of senior reactor operator (SRO) candidates, skills in directing control room activity.

NRC Inspection Reports NRC inspection reports were reviewed to assess the progress and impact of INP0 accreditation on facility training programs. While the information contained in inspection reports lends insight into the current strengths, ,

.l 4

~

weaknesses, and progress of facility training programs, a direct comparison of accredited and nonaccredited training programs was hampered by-two constraints. First, for the 30 currently-accredited sites, only five facilities had training programs that were accredited at the time of the inspection. Second, few of the inspections were conducted using the revised, performance-based inspection procedures, so most of the information below is 4

based on the superseded inspection procedures. For these reasons, inspection results provide limited evidence of the effectiveness of accreditation but do provide a baseline for future evaluations.

Although considerable variations in training programs were reported across facilities, with few exceptions, neither the number nor the nature of training program strengths or deficiencies was found by inspectors to be directly related to accreditation status. Training deficiencies, unresolved items and open items were noted at accredited, nonaccredited, and soon

to-be-accredited facilities.

The training inspection findings are organized into five areas:

Administrative Issues, Training Program Content, Record-Keeping, Instructor Perfonnance and Qualifications, and Training Evaluation. Differences between accredited and nonaccredited facilities, where applicable, are discussed.

Administrative Issues. Administrative oversight of training program development, implementation, and evaluation is one area where the influence of INPO accreditation was detected. Inspections of nonaccredited facilities often attributed training program deficiencies to a lack of procedural guidance and management attention, whereas inspections of accredited facilities generally indicated that the problems detected involved a breach of procedures and guidelines in place. Moreover, inspections of those

facilities that were soon to be accredited often included coninent on the 3

increase in attention and the quality of changes occurring in training programs during this transition to a performance-based approach.

The lack of formal training program guidance and management attention in nonaccredited facilities was noted in a variety of instances, including a lack of detailed administrative instructions for implementing training programs, quality control of training not in compliance with approved administrative and procedural controls, perceptions of training department

! unresponsiveness, failure of training procedures to delineate comprehensive

( courses of study, and inadequate or nonexistent guidelints for the evaluation '

of trainee proficiency.

Other administrative shortcomings more directly related to NRC regulations were found in both accredited and nonaccredited facilities. Instances of these types of training' problems include: (1) changes (particularly

reductions) in training program curricula without notification of NRC;

(2) failure to remove from licensed duties those operators who failed the l

l i

s__ .-__ . . . _ . _ . . . . _ _ _ - _ . . - . _ _ . . ~ . - . - _ . - _ . _ _ . , . - _ -

1 9

requalification examination; (3) failure of candidates to successfully complete all training requirements prior to taking the NRC examination; and (4) unwarranted exemptions of licensed operators from participation in requalification training and examinations. Instances of this last problem

~

were noted at six facilities, including one accredited facility.

Training Program Content. The positive impact of INPO accreditation was also seen in the quality of lesson plans and course outlines. While most facilities were reported to have satisfactory course plans, objectives, and outlines, those of accredited facilities were often commended for their detail and performance-based orientation. (In one case, however, an accredited facility was found to have excellent system lesson guides, but .

deficientinstructorguides).  !

, - Training texts were .also deeme'd satisfactory for the majority of facilities.

In several cases, the training content itself was found to be deficient, due to either a too-general orientation, or to the absence of olant-specific information, such as procedures and technical spe ifications.

The mode of instruction used for training was generally found to contain a good balance of classroom, hands-on participation (particularly with regard to the use of simulators), and on-the-job components. However, in a few instances, at both accredited and nonaccredited facilities, an overreliance on self-study was identified.

In a few cases, the time allocated for training on specific topics was found j to be overly compressed and concern was expressed that the time intervals between associated training topics were too long, leading to possible deterioration in knowledge and skills necessary for subsequent courses.

l The three most comon problems cited with regard to training program content was the lack of training coverage of certain topics, particularly those topics addressed in regulatory requirements and directives. The most frequently cited training content deficiency was the implementation of NUREG-0737 topics, especially training in the mitigation of core damage and performance of control manipulations required by the March 28, 1980, letter from Harold Denton. Numerous shortcomings were cited with respect to these two requirements across accredited and nonaccredited facilities alike.

The second area where content was found deficient.was training on operating events and training on plant and procedure changes. A significant number of inspections found this training to be deficient either with regard to what was being taught or the manner in which it was taught. For example, in one (accredited) facility, LERs and other operational experience information w n read verbatim to students, while at another (accredited) facility, operators were required to read information on their own.

The third area of deficiency involved shortccmings in licensed operator review of emergency and abnormal procedures and technical ' specifications.

Often the deficiencies in this area involved the fact that these topics were

taught in combination with other topics (e.g., systems, control -

manipulations). Inspectors felt procedures and technical specifications were .

not being covered in insufficient depth. j Record Keeping. One of the most prevalent types of generic deficiency cited across facilities involved inaccuracies and inadequacies in training record documentation and maintenance. Instances of training record shortcomings were noted in nearly half of the facilities inspected. Inspectors repeatedly found omissions, and sometimes inaccuracies, in trainee records.

Documentation deficiencies included records of course attendance, examination grades or other measures of student performance, and records of remedial training. In other cases, inspectors noted that although training records appeared generally complete, the manner in which this information was stored made retrieval difficult and overly time-consuming.

Instructor Performance and Qualifications. In the few coments regarding instructor performance, inspectors noted that classes were taught in a professional, effective manner. Virtually all shortcomings in this area pertained to instances in which either instructors were not SRO-certified, or had failed to actively perform licensed duties for a period of 4 months or greater. In a few instances, inspectors questioned the technical qualifications of an instructor based on their personal observation of the instructor or interviews with students.

Training Evaluation. Inspection reports noted that utilities used three methods to evaluate training: follow-up en detected trainee deficiencies, utility-administered examination preparation and grading, and mechanisms for feedback of instructor effectiveness. Differences noted across facilities with regard to these three methods of evaluation were not found to be attributable to facility accreditation status.

A number of facilities were comended for their programs of training deficiency remediation. However, there were a greater number of instances in which facilities were cited for shortcomings in this area, including failure to provide remedial training for deficiencies found at individual, group or programatic levels.

Instances of deficiencies in utility-administered examinations included:

(1) overly easy of simplistic examination questions; (2) failures to link examination questions to learning objectives; (3) a lack of criteria for passing examinations or other performance measures (oral, written and simulator); (4) reductions in passing scores on requalification examinations; (5) inaccuracies in examination grading; and (6) breaches of examination security, including storage cf examinations in easy access areas, and repeated administration of the same examination.

Although few inspections contained evaluations of instructor feedback, in '

those reports in which it was discussed, roughly half of the 'coments indicated that mechanisms of instructor feedback were satisfactory, while the other half cited deficiencies. In the latter instances, inspections

=- . . -- -- . . .

generally reported that either no fomal method existed for auditing or evaluating instructor performance, or that student feedback was not encouraged. ,

Summary. There are a few noteworthy conclusions from the inspection reports reviewed. First, INPO accreditation has fomalized procedural guidance covering training program development, implementation and evaluation.

Second, accredited and to-be-accredited facilities appear to exhibit a l heightened management attention to training issues. Third. INPO accreditation would seem to ensure that training content is based upon performance-based learning objectives, course outlines, and other associated documentation, though weaknesses were identified in training on operational experience information. Also, it appears that accreditation does not assure and is not intended to assure that NRC training requirements are met.

Regulatory deviations and violations were detected fairly equally across nonaccredited, accredited, and to-be-accredited facilities.

Systematic Assessment of Licensee Performance (SALP)

SALP reports from January 1983 to May 1985 were reviewed to provide information on both accredited and nonaccredited licensee training programs.

SALP reports for 44 operating plants were reviewed for training information.

It is important to note that SALP rating periods are long, i.e.,12-18 months, so that in some cases, the SALP review ended prior to accreditation while in other cases accreditation activities were underway at l the utility during the SALP reporting period. Thus, direct comparisons between accredited and nonaccredited programs is not possible.

.The review encompassed information found in the following SALP report categories:

l

) (1) Plant Operations (2) Radiological Controls (3) Maintenance (4) Licensee Activities (5) Training (when provided as a separate element)

These areas were selected for review because information related to training ,

programs and job positions covered by INPO accreditation would be contained in these categories. It is important to note that SALP rating criteria were not developed specifically for the purpose of evaluating whether performance-based training programs comply with the objectives stated in the 1 NRC Policy Statement on Training and Qualification. Also, only seven SALP

! reports for operating plants were available for this review which contained a l separate training category. Information contained in SALP reports in I addition to ratings was reviewed to identify training program changes I, resulting from accreditation and to identify generic training progran

! weaknesses, t

Based on SALP reports, licensee training programs, both accredited and nonaccredited, have been described as being generally well-defined and

, implemented. General findings regarding effects of accreditation on licensee t training programs are: ,

1) :nanagement attention' to training development his increased,

2) training staff resources are increasing, and

3) Waining programs are becoming more' formalized.

I Generic training problems identified in SALP reports involved both accredited and nonaccredited training programs. These problems include:

Weaknesses in incorporation of procedure changes, license changes and

design changes into training. ,

Compliance and adherence to procedures by all categories of plant personnel was identified as a generic problem in SALP reports.

l'

• Administrative issues such as control over training documentation and training records.

Quality of utility-administered examinations was found to be a weakness in nonaccredited programs. ,

The use of self-study for training in nonaccredited programs was noted and linked to a shortage of training staff.

Weaknesses in licensed operator requalification training, including inadequate operator evaluation during requalification training, exist in both accredited and nonaccredited training programs.

In conclusion, SALP reports indicate that accreditation is having a positive effect on improving training. However, problems in performance of a variety of operating activities involving trained personnel and weaknesses in training program administration, content, and evaluation need to be corrected.

Regional Evaluations of Industry Training Improvements Each NRC Regional Administrator provided coments on the effectiveness of industry training under the NRC Policy Statement on Training and Qualification. Their comments are attached.

In sumary, the Regional Administrators report that industry is actively pursuing accreditation and that improvements have been noted, particularly with regard to management attention to training, Some training deficiencies have been noted at both accredited and nonaccredited sites. In general, the regions believe that there has been insufficient time to draw definitive conclusions on the effectiveness of accreditation since only a few sites have been accredited for a long enough period to observe differences in performance.

- _ - _ - - _ _ _ - - - - - _ - . - . . - _ _ - - _ _ - . __ _ - . _ - _ , =

. ba Mrog j ( LNdlTED STATES .

• 8 g NUCLEAR RESULATORY COMMISSION-a f CEflON I S f 831 PARK AVENUE

%,..... f KING OF PftUSSIA. PENNSYLVANIA 19406 14 MAR 1986 ..

MEMORANDUM FOR: Harold R. Denton, Director, Office of Nuclear Reactor Regulation -

James M. Taylor, Director, Office of Inspection and Enforcement FROM: Thomas E. Murley, Regional Administrator SUBJECJ: REGIONAL INPUT TO PROGRESS REPORT EFFECTIVENESS OF TRAINING UNDER THE POLICY STATEMENT ON TRAINING AND ,

QUALIFICATION This refers to your request of January 10, 1986 for Region I input to the Commission progress report on the above subject. We support the industry initiative as a positive approach to training program improvements which should result in improved personnel performance. Discussions with licensee personnel indicate. their active pursuit of the INPO goals with a consider-able expenditure of resources.

With respect to the three specific questions you posed we offer the following:

(1) We believe overall improvements in personnel performance have occurred at most, Region I facilities. The extent to which this can be attri-buted to INPO accreditation activities is difficult to measure. We certainly believe 'it is a principal factor.

(2) We have! observed no training areas that have experienced a decline as a result of the INPO accreditation activities. .

(3) Region I licensees are actively seeking INPO accreditation or imple-menting accredited programs according to the INPO criteria. With respect to the latter, however, there are only a few licensees who have fully accredited programs, therefore the data base is small at this point in time.

In order to enhance appreciation of licensee initiatives and INPO review '

methods, Region I has observed INPO site accreditation reviews at the Susque-hanna and Limerick facilities. Region I currently tracks licensee progress toward completion of the accreditation process in conjunction with the SALP process. We will forward applicable inspection reports which address failure to implement commitments. Trends in personnel perforn.ance attributable to INPO accreditation activities will be addressed in SALP reports.

Multiple Addresses 2 g 4 ggggg ggg We will continue to provide copies of Operator Licensing Examination Reports to DHFS by routine distribution. -

ht T Thomas E. Muriey Regional Administrator cc: e V. Stello, EDO

6WI Russell, NRR J. Partlow, IE r N. Grace, RII

) J. Keppler, RIII R. Martin, RIV J. Martin, RV l

?

I l

l

. l . .

)

l February 27, 1986 ,

MEMORANDUM FOR: Harold R. Denton, Director -

Office of Nuclear Reactor Regulation l

FROM: J. Nelson Grace ,

Regional Administrator

SUBJECT:

pIGIONAL INPUT TO PROGRESS REPORT ON EFFECTIVENESS OF TRAINING In reply to your January 10, 1986 memorandum, we have the following general i connents:

Licensee efforts to achieve INP0 accreditation of training programs have resulted 1 in overall improvements in the administration and quality of plant training.

SALP V training functional area category ratings and trends are provided as an i enclosure to support this conclusion. To date, eleven Region !! plants have been l assessed in SALP V. Six plants, including two rated Category 3 had improving l trends, four plants had constant trends, and one plant was not trended in the training functional area.

Improvements in trsining quaHty were not reflected in performance indicators.

' A oualitative review of selected LEP revealed no significant improvements in

' plant performance directly attributable to the INP0 accreditation process.

- Although improvements in trat'ning quality have been apparent since the INP0

• L accreditation initiative, Region !! inspectors have identified deficiencies in .

implementation of, several specific INP0 a. credited training programs. For example, an inspection conducted October 1-5,1984 (Report No. 50-269/84-25) at Oconee, revealed deficiencies in the Oconee non-licensed operator, licensed l --

reactor operator, and licensen senior reactor operator training programs not-withstanding INP0 accreditation on August 17, 1983. The deficiencies were as follows:

• The Oconee Trainin'g' Services Group prepared and implemented three programs to define the Duke Power Company, Oconee Nuclear Station Operator Licensing Program, Licensed Operator Requalification Program and Non-Licensed Operator Requalification Program which were not referenced or incorporated in the Duke Power Company Corporate Nuclear Production Training Plan. These programs did not have previsions for~ revision, review, approval, or con-trolled distribution.

CONTACT:

C. A. Julian FTS: 242-5541 E

~

Harold R. Denton 2 February 27, 1986 ,

• The production Training Services' training sequence chart did not provide for appropriate sequencing of learning obtectives. The as-taught schedule .

for the previous reactor operator class indicated that a mixture of advanced lectures such as emergency procedW es, were provided in the early oeeks of training. Topic sequencing should instead start from basic systems and components and advance to the more complicated topics of integrated plant operations.

• No formalized, approved student text materials .wers evailable. Students relied instead on self written notes which do not assure consistent stand- e ards and objectives.

• Fomalized instructor lesson plans were being upgraded; however, formats were inconsistent ~ ranging from detailed outlines to simple text formats.

Lesson plan upgrades were left to individual instructors with(ut general guidance to assure lesson plan consistency.

An inspection conducted on March 11-28, 1985 (Report No. 348/85-15), at Farley also revealed a deficiency in the INP0 accreditation process. On December 20, 1984. Farley received accreditation of a licensed (SR0) upgrade program. As of April-1,1985. Farley had not. implemented the accredited program due to objec-tions from the operations group. Apparently. INP0 reviewed and accredited the program description provided by the licensee prior to actual implementation.

A Safety System Functional Inspection (Report No. 50-250/85-22) conducted at

. Turkey point revealed a concern 'over the impact of accreditation efforts on l

ongoing training programs. ' Specifically, since August 1984, formal classroom ,

training sessions, for maintenance technicians had been discontinued. Training resources were dedicated to developing training materials to support INP0 accred.

itation of the maintensnce training program in order to meet a February 1986 Self -

-Evaluation Report submittal date. The inspection team concluded that maintenance training conducted in the interim was not adequate to maintain staff proficiency.

An inspection (Report No. 50-302/85-01) conducted at Crystal River revealed another potential deficiency in the INP0 accreditation process involving the failure of INP0 programs to address' established regulatory requirements and '

previous regulatory commitments. The inspection team identified a systematic reduction in the scope of the approveJ licensed operator requalification program.

The reductions in program scope were not submitted to the NRC for review and approval resulting in an overall failure to implement the requalification program approved by NRC. Licensed personnel were not all required to fully participate l

in the requalification program, and licensed instructors were regularly exempted from attendance of requalification lectures and participation in annual requal-ification examinations. Finally, multiple examples of licensee failure to adequately implement NUREG 0737 comnitments indicated a general lack of manage-ment control over tracking and maintenance of commitments made to the NRC.

0

1 . . .

+

3 " February 27,1986 Harold R. Denton ...

In summary the above examples indicate the potential for programmatic deficien- -

cies in training programs despite involvement in the INPO accreditacion process.

In light of this. Region II recommendrincreased regulatory scrutfey of the INP0 accreditation process.

Should you have questions concerning our c'ommen'ts, please contact us.

Original Signed by Roger D. Walker /for o

J. Nelson Grace Regional Administrator 4

Enclosure:

- SALP V Training Function .

Area Analyses bec w/ enc 1:

NRC Resident Inspector '

Document Control Desk

\ I b

p oe een e

4 8

e 6

I

u _

u ,

' \. g

~

ENCLOSURE -

. SALP V TRAINING FUNCTION AREA ANALYSES Facility 5 ALP Dates Category Trend Farley 8/1/83 - 12/31/84 1 Constant Grand Gulf 10/1/83 - 4/30/85 3 Improving North Anna 9/1/83 - 2/28/85 2 Improving i <.

. Surry 9/1/83 - 2/28/85 2 Improving Browns Ferry 3/1/84 - 5/31/85 2 Constant

~

,, Watts Bar 3/1/84 - 5/31/85 2 Mot Determined Sequoyah 3/1/84 - 5/31/85 2 Constant Hatch - 11/1/83 - 6/30/85 3 Improving

, Catawba . 3/1/84 - 9/30/85 2 Improving

[ t .

4: Robinson 5/1/84 10/31/85' 2 Improving Brunswick ,,o 5/1/84 - 10/31/85 2 Constant I r t

4 1

i

\

l 1

_.. .._. 1 - -

l

A *

'. e ,p ass UNITE 3 STATES NUCLEAR RESULATORY COMMisslON

. " flEGION Ill

• *! 7es nooSEvsLT noAo e oLEN ELLYN. ILUNO15 Se m FEB 111986 -

A MEMORANDUM FOR: Harold R. Denton, Director, Office of Nuclear Reactor Regulation - -

James M. Taylor. Director', Office of Inspection and Enforcement FROM: James G. Keppler, Regional Administrator, Region III ,

SUBJECT:

REGIONAL INPUT TO PROGRESS REPORT EFFECTIVENESS OF TRAINING UNDER THE POLICY STATEMENT ON TRAINING AND

, QUALIFICATION This refers to your memoranda of August 14, 1985 and January 10, 1986, on the above subject. Region III has supported NRR and IE efforts in evaluating the effectiveness of the INPO accreditation. A Region III representative attended an INPO-NRC infomation meeting at INPO Headquarters in the spring

, of 1985. We observed an INPO site team accreditation visit at Prairie Island j

in June 1985. We participated in an NRR pre-accreditation training assessmcrt at Davis-Besse in September 1985, and an NRR post-accreditation training assessment at Dresden in November 1985. We also attended an INPO-NRC infomational meeting at INPO Headquarters in November 1985 in preparation for the NRC assessment of Dresden.

At this time, haviiig reviewed all SALP reports issued since July 1985 and Operator Licensing Examinations given in 1985, we see no overall change in -

personnel performance either improving or declining as a result of INPO accreditation activities. However, this is not surprising since accreditation activities and training programs are only now being fully implemented and the consequences of these activities will take time to become manifest.

Region III, as requested, has provided the Operator Licensing Examination Reports for FY '85.

I Region III intends to track licensee completion of the accreditation process and bring to your attention those licensees who fail to implement the I comitment to actively seek INP0 accreditation or who fail to implement accredited programs according to INPO criteria.

O d btb @ F.e fyJamesG.Keppler Regional Administrator See Attached Distribution l

N

4 e M%, d UNITED STATES f- i , NUCLEAR REGULATORY COMMISSION

$ ^,

'f #

REGION ly s11 RYAN PLAZA DRIVE. SUITE 1000

\> .' ' , ,

• ARUNGToN, TEXAS 70011 MAR 0 4 SES ,,

~

MEMORANDUM FOR: H. R. Denton, Director Office of Nuclear Reactor Regulations ~

FROM: Robert D. Martin, Regional Administrator, Region IV f

SUBJECT:

EFFECTIVENESS OF TRAINING UNDER THE POLICY STATEMENT ON TRAINING AND QUALIFICATION This is in response to the January 10, 1986, memo from yourself and Jim Taylor.

• We have canvased the RIV resident inspectors to obtain their perspective regarding improvements in the effectiveness of the training activities at their assigned facility.

For those plants in which the training accreditation process has had sufficient time to impact training activities, we have seen a positive effect in the training of operations and maintenance personnel. The impact appears to be more pronounced in the operations area; primarily because this is the area receiving initial emphasis by the licensee.

We have noted two potential problems with the accreditation efforts. These problems are:

1. There was a percepti.on that the maintenance training accreditation efforts at one site were adversely effecting maintenance activities because the i better people were detailed to develop the training program in lieu of i performing or directing maintenance activities.

2. In the area of licensed operator training, there was a perceived problem between the performance based training that INPO and one licensee are pursuing and the requirements of 10 CFR Part 55. Part 55 appears to require a more detailed knowledge of plant design. The.last license examination at this plant demonstrated that to rush toward the type of operator training prograr. INFO wants--feeling that such a program will

. completely meet the 10 CFR Part 55 requirements could lead to exam i

failures. Knowledge of plant design and design basis infonnation are subjects that de not nicely integrate into a performance based approach, but are part of the NRC examination and are believed by the utility to have caused several failures in that facility's latest exams.

1

- - -,, , m-,. ,-,,e .,.,.-,.-_--...,,.,--.,------,..,_3-.-o--. - . - . . - . , - , , , - - - - , . _ . - . - . - - - - - -

i ,

H. R. Denten. NRR .

If you have any questions regarding our c 'nts, we would be pleased to discuss them with you. -

7 y.- lu/ f'/ k(L& _

ert . Martin

• Regional Administrator cc: -

V. Stello. EDO ,

J. M. Taylor IE T. E. Murley, RI J. N. Grace, RII e J. G. Keppler, RIII '

J. B. Martin, RV O

l m

4 a

i

_ __ _ . _ - --- - - + - - - = - - - - - " ' ~ - ' * - ' " ~ ~ ' ' ' ' ~ ' '

. = /pn n:vq uNtrao STATES 8 g% A i j NUCLEAR REGULATORY COMM nacos v ON f

....*g 1450 MARIA LANE. SUITE O WALNUT CREEK, CALIFORNIA 94596 .

FEB 141986 -

MEMORANDUM FOR: Harold R. Denton, Director, Office of Nuclear -

Reactor Regulation FROM: J. B. Martin, Regional Administrator

SUBJECT:

REGIONAL INPUT TO FROGRESS REPORT ON EFFECTIVENESS OF TRAINING UNDER THE POLICY STATEMENT ON TRAINING AND QUALIFICATION

• Your and J. M. Taylor's memorandtsn, subjected as above, dated January 10, 1986, requested Regional input to the progress report on the effe:tiveness of training under the Comission policy statement of March 14, 1985. The following comments are intended to address the types of glestions posed by your memorandum.

Training programs covered by the INPO accreditation process have, for the most part in Region V, progressed to the point of program development for plant operations personnel, and as such have only recently reached the point of meaningful implementation. Recognizing that the final product of these programs will not be available for one to two years, or even longer, it is early in the pro. cess for us to see measurable improvements in personnel performance attributable to the INPO accreditation process. Notwithstanding this fact, however, we have had an opportunity - principally by way of the accompaniment of INPO site evaluation teams - to observe the accreditation process. This. fin turn, has allowed some perceptions to be formed with regard to utility management's commitment and response to the concept of a systematic approach to training of nuclear plant personr.el.

Overall, based upon INPO accompaniments at two utilitias within Region V, our observations have been favorable. They have revealed evidence of strong support by top utility management for the INPO accreditation process. During I one INPO visit, for example, the Chief Executive Officer personally participated in the utility's overview presentation of their training programs to the INPO evaluation team upon their arrival.

' Further evidence of the importance with which training is viewed today by utility management is the measurable improvements seen in the size, quality and overall stature of utility nuclear training organizations along with a substantial investment in markedly improved training facilities.

Many of the changes discussed above can, in our judgement, be attributed in large part to the systematic approach to training upon which the INPO accreditation process is based.

1

4 Harold R. Denton -

2- . FEB 141965 3

In sumary, it is early yet to see a measurable impact of the product of training programs under the Comission Policy Statement on Training e and Qualification of Nuclear Power Plant Personnel. It is our view, however, basec" upon a sense of the progress made in the development of revised training programs, evidence of strong commitment by utility management, and the systematic approach to performance based training required to achieve INP0 accreditation, '

measurable improvement in tenns of personnel perfonnance will best be achieved by a continuation of the current approach to training under the Commission's Policy Statement. -

Should you have questions regarding the above coments, please contact Jess Crews of my staff.

% t$ttd

' J. B.-Martin Regional Administrator

'l 4,

I' e.

- - - - ~ - - -

-&- ----- -+-w. + =. . - - -,.-e.-,+ ,. ---,e,, . ,.. - .,-~ ,7.m.m,, g-w ---ww-,w-- , , , ,*- .. ,--.w-y-

!* Enclosure 2

[p ., UNITED STATES .

. # NUCLEAR REGULATORY COMMISSION .

g wasenwom.o.c.neses

. April 3, 1986 Mr. Zach Pate ..

Institute of Nuclear Power -

Operations

. Suite 1500 .

1100 Circle 75 Parkway Atlanta, Georgia 30339

Dear Mr. Pate:

Enclosed are the final reports from the staff's post-accreditation audits at

Susquehanna. Dresden and Oconee. Also enclosed is the comparison of revised NRC review questions with INPO accreditation objectives and criteria. The NRC review questions were revised based upon our experience with the first three post-accreditation audits. These revised questions have been discussed with K. Strahm and W. Coakley of your staff. These reports and the comparison supercede the preliminary copies sent to you January 28, 1986.

By reviewing the comparison document it is clear that the NRC review l

questions and the INP0 objectives and criteria are very similar. There are only three areas for which there are NRC questions that are not explicitly -

included in the INP0 criteria: continbing training, remediation and compromise of examinations. I am pleased that INP0 is developing guidance on continuing training and remediation for job incumbents and encourage you to i promulgate that guidance expeditiously. Also, based upon staff observation.

it appears that the National Nuclear Accrediting Board is as concerned about examination compromise as the NRC. I reconnend that INP0 consider adding appropriate accreditation criteria to address these areas in a future revision of INPO 85-002.

I am encouraged by the constructive dialogue that has taken place in recent months in this important area of training.

Sincerely.

l k Harold R. Denton, Director Office of Nuclear Reactor Regulation

Enclosures:

As stated cc: K. Strahm. INP0 J. Sniezek. NRC ,

W. Russell. NRC e Regional Administrators ',

V. Stello i - - . - - -._ - - - . - _ - - . _ - . - _ - - _ ___. _ _ _ _ _ _ _ _ _

.'6 .

4 .-

POST-ACCREDITATION AUDIT -

SUSQUEHANNA NUCLEAR PLANT DECEMBER 10. 1985 d A. Introduction i 1. Background. On October 21 through 2 . the NRC staff conducted a post-accreditation audit of training at Susquehanna. All ten of the

, Susquehanna programs eligible for INPO Accreditation have received full accreditation status. A representative sample of the accredited programs at Susquehanna was reviewed. s

2. Criteria. The criteria used by,the staff to audit the implementation of I performance-based training are taken directly from the "Comission Policy Statement on Training and Qualification of Nuclear Power Plant Personnel"(50FR11147)ofMarch 20. 1985. In its Policy Statement, the NRC states that the following five elements are essential to acceptable performance-based training programs:

1. Systematic analysis of the jobs to be performed.

2. Learning objectives that are derived from the analysis and that describe desired performance after training.

. 3. Training design and implementation based on the learning objectives.

4. Evaluation of trainee mastery of the objectives during training.

5. Evaluation and revision of the training based on the performance of trained personnel in the job setting.

3. Documentation. To support its review, the NRC review team requested that the following types of documentation be provided by the facility:

o Instructions / Procedures related to:

- Systematic methods used to analyze jobs.

- Training organization goals, objectives, and plans.

- Responsib111 ties / authority of training organization personnel.

- Methods for evaluating / selecting instructional materials, methods, and media.  ;

- Methods for organizing / sequencing of training.

L.- - .- _--___. -

- ~

O O 2

Methods for keeping training programs current.

9

- Maintenance of training records.

l

- Selection of candidates for training and the granting of -

waivers / exemptions from training, I

- Evaluation of training programs, and

- Training, qualification, and evaluation of instructors o Task lists for the job (s) being reviewed o' Documentation related to:

- Development / validation of task ifsts.

- Selection of tasks for formal training.

- Analysis of tasks.

- Analysis of on-the-job perfomance problems and industry events, and -

- Evaluation /auditsofthetrainingprogram(s) l o Roster / organization chart for the training organi:ation 1

o Training schedule for the past 6 months and the next 6 months.

I

- Scsed upon NRC staff observation at INP0 Accreditation Team visits, the above documentation is the type normally associated with participation

-in the INPO Accreditation process. Documentation beyond that required by INPO for, Accreditation is not required by NRC.

4. Scope of Audit. The programs audited and the tasks selected for review within those progiams were:

TASKS PROGRAM l

1. Mechanical Maintenance o Perfom routine surveillance on emergency diesel engine; l o Take and analyze vibration readings on pumps motors, turbines, fans, etc.;

o Perform local leak rate test; o Repair motor-operated valve operators.

5

l

. . o .  !

x Non-Licensed Operator o Line up the Reactor-Core I

2. IsolationCoolingSystem(RCIC);  ;

o Perform line-ups on the ' '

Recirculation System;

.o Rack out a 480V Bus Load Breaker; -

o Conduct Equipment Tagouts

~

o Review the operating shift's

3. STA logs, turnover sheets, etemistry data, and indicators for plant status and possible trends; o Determine if indications of core damage are present; c Evaluate Reactor Cooling System l Level Parameters;

' - o Evaluate Neutron Power Level Parameters

4. Technical Training for Tech. o Provide technical direction 6

Staff and Managers during initial rod sequence Reactor Engineer exchange; o Provide technical direction during the initial heat-up; o Analyze results from

, surveillances

5. Feedback. During the audit, the team provided feedback to the Susquehanna training staff in two group meetings. This was done to give the utility training staff an opportunity to clarify any questions arising from the audit and to identify any documentation or other evidence which could resolve team concerns,

8. Results of Review j

The following discussion of the review findings correlates with the elements of the Comission's Policy Statement on Training and Qualifications.

1.0 Systemat'c Analysis of Jobs to be Perfomed 1.1 Discussion. Task analysis methods, procedures and products were reviewed using the documentation described in Section A.3 above to determine whether:

\

• 4 o Training and qualification programs were based upon systematic analysis of the skills and knowledge required for adequate job i performance.

o Training and qualification programs were based upon a systematic I analysis of the entry skills and knowledge of trainees.

o On-the-job performance problems were systematically analyzed to detemine training needs.

o Industry events were systematically analyzed to detemine training needs.

1 o The systematic analysis considered both the initial and continuing training needs of job incumbents.

! o Procedures and other implementing mechanisms ensure that the analysis infomation remains current as job perfomance requirements evolve.

1.2 Findings. The task analysis for three of the sampled programs is complete and contains systematically derived job performance i

information. The job task analyses for those programs were found to be j exceptionally well done and are an excellent source of valid information upon which to base training programs. The fourth sampled program, 6 Reactor Engineer (Technical Staff and Managers), did not have a completed task analysis at the time of review. INPO is not requiring a position-specific analysis to establish program content for technical staff and managers.

There is a formal mechanism (i.e., the curriculum comittee) used to detemir.e training needs from industry events and on-the-job perfomance of graduate trainees 6 months after training. The method of selecting tasks for training does not seem to distinguish between tasks requiring initial training only and those requiring continuing training as well.

The existing method lacks criteria fot determining the need for continuing training based on such factors as frequency of performance and safety significance.

2.0 Development of Learning Ob.iuctives 2.1 Discussion. Learning objectives were reviewed for; the subject programs to detemine whether:

o Written learning objectives were developed based upon the systematic analysis of job performance requirements.

o Learning objectives are specific and measurable, and include conditions, actions and standards based upor. job performance requirements.

i

. ' *- . l 6

A o Procedures and other implementing mechanisms ensure that the learning objectives remain current as job perfomance requirements O evolve.

2.2 Findings. The Susquehanna training staff had prepared the documentation so that the sequence from task statement to learning objectives to l

training content was easily traced. Handwritten notes placed next to

" Specific Learning Objectives" in lesson plans indicate the associated l task / element number. Sections of units of instruction (lesson plans) were

noted on a task / clement x lesson plan identification matrix.

The learning objectives did not consistently describe expected trainee behavior in tems of conditions, actions and standards governing successful task perfomance. The learning objectives reviewed for the

' selected tasks did not reflect the KSAs, nor the standards and

' conditions derived from the job task analysis. In addition, two units

' ofinstructiondevelopedanddeliveredbycontractors(ReactorEngineer and part of Mechanical Maintenance) contained no learning objectives and l

• the task elements could only be generally correlated with lesson plans.

The Susquehanna staff also provided an example of a new fomat that l

would be used for learning objective listings. In the new femat, all relevant task statements will be referenced to the terminal objective.

with specific objectives tied to the relevant task statement for that objective. The new format does not go to the element or the knowledge.

skill, or ability (KSA) level.

l 3.0 Desien and Implementation.

' 3.1 Discussion. Using the documentation described in Section A.3 above, the design and implementation of performarce-based training was reviewed

. to determine whether:

o The training organization's goals, objectives, plans, and relation-ships with the remainder of the organization are clearly stated.

I

! o The responsibilities and authority of training personnel are clearly stated.

l

{ o Instructionalsettingsselected(classroom,on-the-jobtraining,and simulator) are appropriate given job performance requirements and l

i learning objectives.

o The organization and sequencing of both initial and continuing training program content are based upon the relationships among learning objectives.

o Lesson plans have been developed that include all the instructional materia'Is required for implementation of the program.

l

\ 6

1. i i

l o Instructional materials were based upon job performance requirements and learning objectives and selected using an objective, auditable F

< evaluation of the appropriateness of these materials.

o Procedures and other implementing mechanisms ensure that the design and implementation of initial and continuing training programs remain current as job performance, requirements evolve.

i 3.2 Findings. The training organization goals and objectives are defined

' in a variety of procedures documents and selected goals are included in the utility's management by objectives program where evaluation of achievement of those selected goals is possible. The responsibilities i of training department personnel are adequately defined. There is a p(rocedure i.e. methods and which media)describes as one step the in theneed trainingto development select instructional s

process. The utility stated that it relied on staff judgment for selection of instructional settings and has not established criteria for selection decisions. These staff judgments are reviewed as part of the

approval process for the resultiig lesson plans. laboratory exercises

' and DJT qualification cards.

Lesson plans for units of instruction reviewed generally adhered to a format prescribed by the utility's procedurr.. Although learning

' objectives were a prescribed part of lesson format, the simulator training segment for STA training contained no learning objectives.

6 The utility stated that the STA training program was presently being developed as part of the effort to change over from a combined SR0/STA training program to one that is exclusively intended for STAS. Training materia,s that existed before the changeover to a performance-based training program are based on a assumed entry level of a high school diploma. Classroom and simulator training which were observed were generally well-planned and conducted. Adequate sethods have been established for maintaining training records.

j 4.0 Trainee Eva'uation l 4.1 Discussion. The methods for and use of trainee evaluations were reviewed using the documentation described in Section A.3 above to detemine whether: .

4 o Trainees are routinely pretested or otherwise objectively evaluated to ensure that their entry skills and knowledge are consistent with the training program assumptions, and with the basis for granting exemptions from training.

! o Trainee evaluations (tests) are developed based upon job performance requirements and learning objectives (i.e., skills, knowledges, and abilities).

o Trainees are regularly and objectively evaluated throughout each i

training program.

l

s.

7 4

i o Established methods exist and are used to provide.perfomance feedback to trainees and to deal with unsatisfactory trainee perfor- r, mance.

1 4.2 Findings. A formal mechanism exists for evaluating training candidates' .,

entry level skills and knowledge. Selection examinations are i,

administered for trainee aptitude serpening prior to training. Training equivalency deteminations are made by supervisory personnel and documented to justify waiver of training requirements.

Training administrative procedures, if implemented rigorously, are adequate to preclude the compromise of tests. Trainee performance is evaluated during the training programs by objective evaluations and advising is used to prcvide remedial qualification cards.1 TraineeProcedures are in place and action may be taken assistance to students.

by the curriculum comittee to require remedial training and ratesting for trainees who perfom below minimum standards.

Procedures prescribed that learning objectives should be factored into trainee evaluations. It was evident that the Susquehanna training staff

' was able to track from the analysis infomation to most of the learning objectives and units of instruction. There were numerous learning i

objectives that were not linked,to a task / element statement. In addition, the teminology often was different between task / element statements and learning objectives, so that an explanation of the linkage was necessary. Tests for lessons on the contractor delivered training discussed ir. Section 2.2 were not reviewed by the utility staff to assess whether the level of knowledge acquired by the trainee was sufficient for task performance.

5.0 Procram Evaluation 5.1 Discussion. Training program evaluation methods were reviewed using the documentation described in Section A.3 above to determine whether:

o A documented program is in place to systematically and continually evaluate the effectiveness of the training program and revise the program as required.

o The training program conduct and content are continually monitored and evaluated.

o Feedback from on-the-job perfomance is actively solicited, evaluated, and incorporated as training program modifications.

o Operating experience and other external factors are reviewed for evidence of job performance-related lessons learned.

o The training implications of plant modifications, and procedural and

' administrative changes are integrated into both initial and

. continuing training programs.

l

2 .

'e ,

e o The program evaluates the training management, staff size, workload, G and qua ifications and perfomance.

5.2 Findings. The utility has a documented method, i.e., the curriculum

i consiittee, to evaluate the effectiveness of training programs and need for revision, and to incorporate into training the lessons learned from .

operating experience, procedures changes, and plant modifications. A systematic and thorough review of aggregate trainee test performance was performed by a contractor in 1984. Examination results are reviewed and reported quarterly. Both instructor critiques and trainee critiques are used for program evaluation within 6 months following training. The utility training evaluation plan is comprehensive, leaving only one i possible area uncovered. Procedures do not appear to require supervisor l feedback beyond six months.

Both internal and external training program evaluations are performed.

Docurented qualification requirements exist for training staff that

- address subject matter and instructional skills. Instructor perfomance is evaluated and documented annually. Training department objectives l address the need for more professional development activities for the training staff. The utility training staff consists of highly qualified I

personnel. The relationship of the instructional staff and the training

support services group is a constructive one which provides for the  !

! maximum benefit to the program of nuclear technology expertise and instructional technology expertise.

C. .Summiry of Findines The findings from the training audit at Susquehanna indicate that significant progress has been made toward achieving effective performance-based training. The staff found that four of the five elements identified in the Consission's policy Statement have been met and are being effectively implemented. The fifth element, related to learning objectives will be met when the new learning objective fomat is fully implemented. Currently, learning objectives are not complete in that they do not consistently contain conditions and standards. This conclusion is based on our finding that learning objectives could not be correlated to the job task analys's data, such that the existing learning objectives may not be complete. The staff also found that some learning objectives required improvement in order to describe desired performance after training.

I f

. i

.l6 .

l l

POST-ACCREDITATION AUDIT DRESDEN NUCLEAR STATION -

NOVEMBER 18-21, 1985 S

A. Introduction . . .

1. Background. On November 18 through 21, the NRC staff ccnducted a post-secreditation audit of training.ht Dresden. Three of the Dresden programs eligible for INP0 Accreditation have received full accreditation status.

2. Criteria. The criteria used by the staff to audit the imp 16 mentation of performance-based training are taken directly from the " Commission Policy Statement on Training and QualificationInofitsNuclearPolicyPower Plant Statement, Perscnnel" (50 FR 11147) of March 20, 1985.

the NRC states that the following five elerants are essential to

• acceptable perfomance-based tra ning programs:

1. Systematic analysis of the jobs to be performed. .

2. Learning objectives that are derived from the analysis and that describe desired performance after training, l

3. Training design and implement $1on based on the learning objectives.

4. Evaluation of trainee mastery of the objectives during training,

5. Evaluation and revision of the training based on the performance of trained personnel in the job setting.

3. Documentation. To support its review, the NRC review team requested

. that the following types of documentation be provided by the facility:

Instructions / Procedures related to:

- Systematic methods used to analyze jobs.

l

- Training organization goals, objectives, and plans.

- Responsibilities / authority of training organization personnel.

- Methods for evaluating / selecting instructional materials,

' methods, and media, i

- Methods for keeping training programs current, i

- Maintenance of training records,

- Selection of candidates for training and the granting of l waivers / exemptions from training, i

i

t  : -

2 l

~

- Evaluation of training programs, and ,

- Training, qualification, and evaluation of instructors

• Task lists for the job (s) being reviewed Documentation related to: -

- Development / validation of task lists.

- Selection of tasks for formal training.

- Analysis of tasks.

- Analysis of on-the-job performance problems and industry events, and

- Evaluation /auditsofthetrainingprogram(s)

• Roster / organization chart for the training organization Training schedule for thgpast 6 months 'and the next 6 months.

' Based upon NRC staff observation at INPO Accreditation Team visits, the above do:vmentation is the type normally associated with participation

'. in the INPO Accreditation process. Documentation beyond that required by INP0 for Accreditation is not required by NRC.

4. Scope of Audit. The programs audited and the tasks selected for review within those programs were:

TASKS PROGRAM (1) Non-Licensed Operator

• HPCI surveillances (Dresden'sEquipmentAttendant)

• Return air ejector to service and/or operation

• Remove centrifugal pump from ser-vica

• Start a centrifug61 pump:

Reactor feed pump

• Charging a CRD accumulator (2) Reactor Operator

• Equipment tagouts

• Reactor scram

• Placing feedwater regulator '

into local manual operation

• ATWS

• HPCI motor operated valve and pump operation

. s .

- 3-4 (3) Senior Reactor Operator

• Verify ECCS valve and pump .

status

• Partial or complete loss of AC power , ,

'

• Direct emergency response

• Prepare and approve temporary

. procedure changes Loss of feedwater heaters ,

! 5. Feedoack. During the audit, the team provided feedback to the Dresden training staff in daily group meetings. This was done to give the utility training staff an opportunity to clarify any team questions arising from the audit or to identify any documentation or other evidence which could resolve team concerns.

j B. Results of Review The following discussion of the review findings correlates with the elements of the Commission's Policy Statement on Training and Qualifications.

1.0 Systematic Analysis of Jobs to be Perfomed ,

1.1 Discussion. Task analysis methods, procedures and products were

! reviewed using the documentation described in Section A.3 above to

' ' detemine whether:

• Training and qualification programs were based upon systematic l

analysis of the skills and knowledge required for adequate job performance.

• Training and qualification programs were based upon a systematic analysis of the entry skills and knowledge of trainees.

• On-the-job perfomance problems were systematically analyzed to determine training needs.

• Industry events were systematically analyzed to determine training needs.

• The systematic analysis considered both the initial and continuing training needs of job incumbents.

• Procedures and other implenenting mechanisms ensure that the analysis infomation remains current as job performance requirements -

evolve.

e,w-, ..w-,.wr,-,,--,-w,. w r m w w -- .- w w,-w v%. . w--w.,e -r-----,--,-. s=w------i--

/ l

! /

t .

i 1.2 Findings. A systematic method was used by Dresden to develop task lists for each of the accredited positions. The method included a 1005 review

of procedures and comparison to the INP0 taxonosy. However, there was limited involvement of operations personnel in validating the task lists ,

and in selecting tasks for which initial and continuing training is ..,, .

l provided. During operator interviews, operators stated that training '

i did not cover al' aspects of their jobs. The identification of tasks  :

requiring initial training and those Yequiring continuing training was based on practice in the preexisting training program. The knowledges,

' skills and abilities that enable a job incumbent to perform tasks were not identified as part of the task analysis. The licensed operator requalification training program 12 based on regulatory requirements rather than an analysis of operator job requirements.

I 2.0 Development of Learnino Objectives 2.1 Discussion. Learning objectives were reviewed for the subject programs to determine whether:

• Written learning objectives were developed based upon the systematic l analysis of job performance requirements.

1

• Learning objectives are specific and measurable, and include l

conditions, actions and standards based upon job performance

requirements. .

'

• Procedures and'other implementing mechanisms ensure that the learning objectives remain current as job parformance requirements evolve.

2.2 Findings. In most cases, learning objectives could be found in lesson plans that correspond with task statements (e.g., fo'r equipment attendanttraining). Licensed operator training programs, particularly simulator and on-the-job-training segments, did not consistently contain learning objectives related to documented task lists (e.g., for equipmenttagoutandlossoffeedwaterheaterstasks). Procedures were sometimes used as the lesson plan without supporting learning objectives 4 j

(e.g.,forlicensedoperatorpre-simulatorsegments).

l In the RO and SRO programs learning objectives did not consistently contain actions, conditions under which actions should take place, and standards of performance the trainee should achieve. On-the-fob training (0JT)objectivesweretaskstatementswithagenericstandard for performance in the front of the DJT training book. Standards of performance are not stated in the learning objectives as the basis for consistent evaluation of trainee mastery of learning.

t 1

l

A ,.

The training standard document for the equipment attendant-training 9

> program appears to be a complete compilation of learning objectives for that program but it is not clear whether or how this document is used to ensure accuracy or completeness of task coverage in training materials. #'

i A matrix does identify where, in classroom and OJT segments, tasks from the equipment attendant task list are found. However, there is no training standard document for licenskd operator and senior operator training programs. There is no documentation that links objectives addressed in various segments of classroom training, where basic skills i

and knowledges are taught, to OJT and simulator learning objectives which were derived from the task list.

A mechanism exists, in the form of a training inquiry, for documenting proposed training program modifications. However, observations of simulator requalification training revealed that simulator training materials had not been modified to incorporate new symptom-based emergency operating procedures in use at Dresden. Learning objectives were not used as the basis for evaluation of trainee mastery of training in the simulator. l f

The approach to Dresden training program development is unclear as reflected in Dresden program development guidelines (PDE-5E) which discuss the use of learn < ng objectives. The training department procedures state that if previously existing training can be shown to be i

adequate, then derivation of learning objectives may be unnecessary.

~

6 The same procedure states that learning objecthes which fom the link between the analysis of requirements and the rest of the program development process, must be of high quality, and shall contain actions (performances), conditions,andstandards. Implementation of learning objectives containing actions, conditions, and standards, as derived from task analysis data, was incomplete and inconsistent. The licensee i

is developing learning objectives for new programs and for needed revisions and not updating objectives for existing training.

3.0 Desian and Implementation.

3.1 Discussion. Using the documentation described in Section A.3 above, the design and implementation of performance-based training was reviewed to determine whether: .

• The training organization's goals, objectives, plans, and relationships with the remainder of the organization are clearly stated.

i

• The responsibilities and authority of training personnel are clearly

! stated.

• Instructional settings selected (classroom, on-the-job training, and simulator) are appropriate given job perfomance requirements and l

learning objectives.

l

- --.._______-.m _ _ _ . , - - _ _ - _ ~ - - - - - - - - - - - - _ . . - -..-,,,- v, , . - . . - - , . _ ,. m c - -

i .

~ '.

1 4 .. ,

• The organization and sequencing of both initial and continuing W training program content are based upon the relationships among learning objectives.

i

• Lesson plans have been developed that include all the instructional materials required for implementation of the program.

~

• Instructional materials are based 'ponu job performance requirements and learning objectives and selected using an objective, auditable evaluation of the appropriateness of these materials.

i

• Procedures and other implementing mechanisms ensure that the design and implementation of initial and continuing training programs remain current as job performance requirements evolve.

3.2 Findings. The training organization goals and objectives are defined and revised annually and used as the basis for semi-annual evaluations of corporate progress. The responsibilities of various corporate and station training department positions is defined.

Training design and implementation at Dresden has not begun with accreditation; rather the training programs, which have been in place since 1975, have gone through a process referred to at Dresden as training program verification. Tasks from the task lists have generally been identified as to whether they are trained in the classroom, i simulator, or on-the-job training segments. Tasks for training and f sequencing of training appear to be complete for equipment attendant training while the program for licensed operator training was not complete. However, the cross-referencing of tasks to OJT and simulator training was incomplete and thus does not constitute an analysis of the

- completeness of coverage of job tasks in the training curriculum. l On-the-job training requirements for equipment attendants are frequently signed off for the trainee based only on the opportunity to perform a taskormanipulationwithoutregardforsequencingoftraining(i.e.,in somecasesitmayhavebeenbeforeclassroomtraining).

The design of continuing training for nonlicensed operators is not based

. on an analysis of aggregate performance of job incumbents or performance on periodic tests. Rather, continuing training consists of retraining personnel in all training materials on a two-year cycle. The licensed operator requalification training program is based on regulatory requirements.

l l

Lesson plans do not consistently contain objectives for lessons.

suggested instructor activities, references or suggested evaluation l

standards. On-the-job training materials consist of a simple listing of

l t

i . required tasks and corresponding sign-off documentation without .-

standards for consistent evaluation of trainee mastery. Simulator

• lesson plans generally consisted of a procedure without supporting guidance to ensure consistent and objective evaluation of trainee ..-

progress and f,eedback on performance.

Classroom and simulator training, as abserved at Dresden, were conducted l

in an appropriate manner. Simulator' training, conducted at the G. E.

simulator, was well-executed in spite of the fact that lesson plans had not been revised to reflect the use of new symptom-based emergency operating procedures.

4.0 Trainee Evaluation 4.1 Discussion. The methods for and use of trainee evaluations were reviewed using the documentation described in Section A.3 above to I determine whether:

I

• Trainees are routinely pretested or otherwise objectively evaluated to ensure that their entry skills and knowledge are consistent with the training program assumptions, and with the basis for granting exemptions from training.

• Trainee evaluations (tests) iire developed based upon job performance

' requirementsandlearningobjectives(i.e., skills,knowledges,and g abilities).

• Trainees are regularly and objectively evaluated throughout each training program.

• Established methods er.ist and are used to provide performance feedback to trainees and to deal with unsatisfactory trainee ,

performance.

4.2 Findings. Exemptions from training are, as a general rule, not granted at Dresden, although there is a procec'ure and form for documenting ,

waivers of training. Written and oral excminations are routinely used and documented for evaluation of training completion. Written examinations, as reviewed, appeared to address the training for tasks

elected for review. k'ritten examinations are maintained under the control of lead instructors and exam results are maintained with individual trainee records. Documentation of trainee performance evaluations for on-the-job and sinulator segments of training consists only of the required sign-off for tasks, with no accompanying performance standards or criteria which could be used for objective and consistent performance feedback to trainees or, in the case of requalification training, to their management. The staff observed simulator requalification training in which licensed operators repeatedly experienced difficulty understanding and controlling a simulated ATW5 event.

. = . . _ _ . _ . _ . . - ._.

s .- .

l The policy for prescribing remedial training allows a case-by-case y determination of appropriate action ~by the performance evaluation connittee. Except during requalification training, trainees are not removed from training or associated job duties when they do not meet '

individual training criteria (e.g., on-the-job training tasks) which" -

they may repeat until performance is acceptable.

5.0 Program Evaluation l

5.1 Discussion. Training program evaluation methods were reviewed using the documentation described in Section A.3 above to determine whether:

• A documented program is in place to systematically and continually l evaluate the effectiveness of the training program and revise the l

program as required.

• The training program conduct and content are continually monitored and evaluated.

• Feedback from on-the-job performance is actively snlicited.

evaluated, and incorporated as training program modifications.

l I

• Operating experience and other external factors are reviewed for l

evidence of job performance-related lessons learned.

The training implications of plant modifications, and procedural and

administrative changes are integrated into both initial and continuing training programs.

• The program evaluates the training program management, staff size.

l workload, and qualifications and perfomance.

t 5.2 Findings. Periodic comprehensive reviews of Dresden training programs

! are conducted by the Production Training Department of Commonwealth 4

Edison. The evaluations include critiques of all aspects of station l

training including recommendations for areas needing improvement.

i Dresden station is not required to formally respond to nor implement the 4 recomendations arising from these evaluations, although the Dresden training staff stated that pro'olems identified are remedied. The

Production Training Center also does training evaluations of any newly developed and implemented training programs and produces training trending reports which describe trainee attrition and completion rates for Dresden operator training progra as.

t i

Many methods are utilized to gather information on training program effectiveness, however, there is no systematic analysis of aggregate trainee performance or training critiques. Individual course cr*tiques.

l i

~ . _ - - - _ _ . - . - - _ - - - - - -

.g.

1 instructor critiques, and trainee evaluations are routinely collected following training. Analysis of this information is informal and may .*, ,

f

result in tr

ining changes at the general discretion of the training instructor and training supervisor. Similarly, supervisory evaluations . . .

of job incunbents are not documented and, therefore, analysis to identify possible training program weaknesses may be incomplete.

Rather, individual training inquiries, must be submitted by job incumbents or their supervisors for consideration by training department i personnel in making program revisions.

The audit and control of contracted training at the G. E. simulator l

j appears to be weak since evaluation and feedback on simulator training

' results is not sufficiently detailed to allow analysis.

~

Qualification and training requirements for Dresden training staff address both subject matter and instructional skills. Instructional staff performance is regularly evaluated and continuing training is available for instructors. ..

C. Sunnary of Findings The findings from the audit of three accredited training programs at Dresden indicate that training program revisions and enhancements have been and are being made. The staff found that all five elements of effective performance-based training were in place for the non-licensed 1

operator programs and that four elements have been met and are being implemented for the licensed operator programs. The element related to development of learning objectives based upon job analysis is not fully implemented. This cont:1usion is based on the finding that the results of Dresden's job analysis for licensed operator positions did not j

identify the task supporting skills and knowledge to be included in the 4

training curriculum and that some learning objectives were found to be incomplete and inconsistent.

1 i

S I - - - -

POST-ACCREDITATION AUDIT DCONEE NUCLEAR PLANT

! DECEMBER 16-18, 1985 -

I A. Introduction .

l 1. Background - On December 16 through 18, the Maintenance and Training Branch of the Division of Human Factors Technology conducted a post-accreditation audit of three training programs at Oconee Nuclear i Station. The programs which have achieved INP0 accreditation status are the Non-Licensed Operator Licensed Operator, and Senior Operator /STA programs. The review focused on a sample of tasks from each of the -

accredited programs.

l 2. Criteria - To audit the im>1emestation of performance-based training, the staff used criteria ta ten directly from the "Comission Policy

, Statement on Training and Qualification of Nuclear Power Plant

. Personnel (50 FR 11147) of March 20, 1985. In this Policy Statement, the NRC states that the following five elements are essential to acceptable performance-based training programs:

1. Systematic analysis of the jobs to be performed.

2. Learning objectives that are' derived from the analysis and that' 1 describe desired performance after training,

~3. Training design and implementation based on the learning objectives.

4. Evaluation of trainee mastery of the objectives during training,

~

1 , 5. Evaluation and revision of the training based on the performance of trained personnel in the job setting.

~

I 3. Documentation - To support its review of the licensee's accredited l

programs, the staff requested, in advance, that the following types of documentatior. be made available:

Instructions / Procedures related to:

- Systematic methods used to analyze jobs,

' - Training organization goals, objectives, and plans.

- Responsibilities / authority of training organization personnel.

- Methods for evaluating / selecting instructional materials, methods, and media,

- Methods for organizing / sequencing of training, A

w--_ --

. = '.

2-

- Methods for keeping training programs current. -

Maintenance of training records.

- Selection of candidates for training and the granting of -

waivers / exemptions from training,

- Evaluation of training programs', and

- Training, qualification, and evaluation of instructors Task lists for the job (s) being reviewed Documentation related to:

- Development / val.idation of task lists,

! - Selection of tasks for formal training.

- Analysis of tasks, ,

i

- Analysis of on-the-job perfonnance prt'lems and industry events,

, and

- Evaluation / audits of the training program (s).

Roster / organization chart for the training organization i

Training schedule for the past 6 months and the next 6 months Final accreditation team report Biennial status report on accreditation Based upon hRC staff observation at INP0 Accreditation Team visits, the above documentation is the type normally associated with participation in the INPO Accreditation process. Documentation beyond that required by INP0 for Accreditation is not required by NRC.

4. Scope of Audit - Following are the tasks selected for review from each of the programs audited:

f._.. . . _ _ _ _ _ _ . _ _ . . . _ _ _ . _ . _ - - . - _. . . . _ . . _ - - -

l'. ,

, I.

3-Program Tasks - g

1. Non-Licensed Operator

• Make out, place, and remove red and white tags ..,.

• Line up emergency feedwater

. system

• Start up, shut down, and isola' te condensate booster pumps

• Line up and start recirculating cooling water system

2. Reactor Operator

• Increase reacter power according to specifications and set controls to automatic at appropriate time

• Perform reactivity balance calculation to maintain subcritical reactivity; borate

. the reactor cooling system as necessary to maintain this i

suberitical reactivity l

• Direct utility operator to set l up core flood system for emergency shutdown service; i

' check and obtain proper boron concentration for this emergency

~

service.

3. Senior Re. actor Operator /STA

• Classify emergency events

• Direct operad ons following steam generator tube rupture

• Conduct post-trip review and recovery ,

• Direct unit shutdown from outside control room.

5. Feedback - The team met with the Oconee training staff once during the review to provide feedback. In addition to this meeting, there was a great coal of individual feedback on a more informal basis as the team members sought to clarify various issues with respect to the training programs under review. Since the team completed its review earlier than expected, the formal exit meeting was held earlier than scheduled. This obviated the need for a second meeting with the training staff.

--e----- - -. ..- - ~ ~. _ _ _ , _ _ _ _ , , , , , , , , _ , , , , , __ _ _ _

- ,i .

1. ~

B. Results of Review -

v.

The following is a discussion of the review findings as they relate to the

elements of the Comission's Policy Statement on Training and Qualifications. ,,

1.0 Systematic Analysis of Jobs to be Performed 1.1 Ditcussion - Using the documentation described in Section A.3 above, task analysis methods and products were reviewed to determine whether:

• Training and qualification programs were based upon systematic analysis of the skills and knowledge required for adequate job o

performance.

Training and qualification programs were based upon a systematic analysis of the entry skills and knowledge of trainers.

On-the-job performance prob 1' ems were systematically analyzed to detemine training needs.

• Industry events were systematically analyzed to determine training needs.

• The systematic analysis considered both the initial and continuing' training needs of job incumbents.

• Procedures and other implementing mechanisms ensure that the analysis infurmation remains current as job performance requirements evolve.

1.2 Findings - The task analysis for the three programs was accomplished through internally developed, table-top analyses performed by job incumbents, senior instructors and staff from the operations department.

These internal analyses were not carried out for the SR0/STA position.

For the SR0/STA position, a task list was developed by a senior instructor and two SR0s. During NRC interviews, SRO job incumbents indicated that the SR0/STA on-the-job (0JT) task list was not complete.

Our limited review of the OJT task list identified missing tasks for on-shift training or qualification of other operators, ramote shutdown and post-trip review procedures.

The licensee has stated that the training department is in the process of implementing the Employee Training and Qualification System (ETQS).

This system, when fully implemented, should correct the deficiencies in the SR0 task list. The staff examined the full documentation of the ETQS cnd determined that the system meets the criteria in the Commission's Policy Statement on training and qualifications. It -

should, however, differentiate between tasks for initial and continuing training with respect to the significance and frequency of task and 1

i

b

" V ,'

i learning decay. Finally, while there are procedures in piece to feed ,

back task changes to lesson plans, there are currently no procedures to v factor these changes back into the task list.

2.0 Development of Learning Objective 2.1 Discussion - Learning objectives were reviewed M r the' subject programs to detemine whether:

• Written learning objectives were developed based upon the systematic analysis of job performance requirements.

• Learning objectives are specific and measurable, and include conditions, actions and standards based upon job performance requirements.

• Procedures and other implementing mechanisms ensure that the learning objectives remain current as job performance requirements evolve.

2.2 Findings - The staff was unable to determine from the licensee's documentation the correlation between tasks, learning objectives, knowledges and skills, and lesson plans. A matrix was provided in hand-written form for the non-licensed operator program for the tasks the NRC review team had pre-selected to audit. Correlation of learning o -

L objectives to tasks for the NLO and R0 programs based on training staff knowledge and INPO Training Guidelines was satisfactory.

I Many.cf the learning objectives for all three programs reviewed did not t . J include standards or conditions. In some cases, cues and standards were L implied or were tied to a procedure. Some objectives were stated in

' tems too broad for trainees to know what their expected performance should be, e.g., " Draw the Core Flood System," " Discuss post-trip procedure." ,,

There is a training module for instructors that gives excellent examples of how learning objectives should be written. Although the module has not yet been presented to all instructors, training supervisors should have noted these deficiencies before approving materials containing poorly written objectives.

3.0 Design and Ito,1 1mentation 3.1 Discussion - Using the documentation described in Section A.3 above, the design and implementation of perfomance-based training was reviewed to determine whether:

• The training organization's goals, objectives, plans, and relation-ships with the remainder of the organization are clearly stated.

6-

• The responsibilities and authority of training personnel are clearly g i stated.

• I i

• Instructional settings selected (classroom, on-the-job training, and ff simulator) are appropriate given job performance requirements and learning objectives. ,

l

• The organization and sequencing of both initial and continuing training program content are based upon the relationships among learning objectives.

• Lesson plans have been developed that include all the instructional materials required for implementation of the program.

• Instructional materials were based upon job performance

' requirements and learning objectives and selected using an objective, auditable evaluation of the appropriateness of these meterials.

• Procedures and other implementing mechanisms ensure that the design and implementation of initial and continuing training programs remain current as job performance requirements evolve.

~

3.2 Findings - The Oconee Train 1ng Plan states the training organization's goals, objectives, plans, ard relationships with other parts of the '

, organization. The training plan provides for maintaining the goals and objectives and evaluating training personnel annually against established goals. In addition. Training Managers evaluate trainee job performance and training needs although thic is not procedurally

- required. Although the responsibilities and authority of the training

. organization are stated in the training plan, the new ETQS will define more formally the interface between corporate and training

! personnel.

At present, the instructional settings are determined by subject matter experts. The ETQS evaluates instructional settings for each task. In both the R0 and SRO programs, many of the learning objectives were too feneral n one oftothe detemine if sequencing three R0 tasks selected fororreview, test questions the sequence were appropriate.

of the learning objectives wa's not consistent with the lesson plan; however, the sequencing of the lesson plan did seem appropriate. In general, istructional aids were listed in lesson plans; however, these lesson plans would have been better had these aids been cued in at the appropriate part of the lesson. Some validation of entry-level testing has been carried out against training and job performa~nce. The new ETQS will formalize this process.

In the SRO program, there is no lesson plan for the remote shutdown task. Although there is some segmented classroca training on this task, the shift personnel interviewed indicated they have never walked through the entire procedure.

i

'. _y.

The only training scheduled during NRC's review was in the RO program.

The NRC staff observed both classroom instruction and the simulator lab.

• The class and lab presentations were effective and followed approved lesson plans and laboratory guides. ,, ,

On-the-job training is presently carried out without the operations staff being instructed in evaluation.or instructional skills.  !

' Interviews indicated that there were no formal standards in OJT; according to operators, some of the staff are " easier than others." The training department has recognized this shortcoming and has issued ance that clarified the standards and outlines methods for improving As mentioned in Section 1.2 above, there is a generic concern that continuing training needs have not been identified as part of the analysis; rather, all initial training tasks are subject to continuing training.

4.0 Trainee Evaluation 4.1 Discussion - The methods for and use of trainee evaluations were reviewed using the documentation described in Section A.3 above to detemine whether: ,

• Trainees are routinely pretested or otherwise objectively eva?uated to ensure that their entry skills and knowledge are consistent with

' the training program assumptions, and with the basis for granting exemptions from training.

• Trainee evaluations (tests) are developed based upon job performance requirements and learning objectives (i.e.. skills, knowledges, and abilities).

• Trainees are regularly and objectively evaluated throughout each training program.

• Established methods exist and are used to provide performance feedback to trainees and to deal with unsatisfactory trainee performance.

4.2 Findings - Candidates for the training program are tested using the North Carolina Test Battery. To move on to the R0 training program, candidates must have appropriate senior,ity, experience, training and

. qualifications, and pass a comprehensive examination. Examinations are used to exempt trainees from plant-specific training and/or nuclear fundamentals.

There is no formal pro'cedure to see that SRO/STA tasks are correctly performed during DJT. Learning objectives in this area are too general to support test questions. These objectives are presently being upgraded to remedy the situation. In the other two programs reviewed (R0 and NLO), test items are all linked to

~

learning objectives. Appropriate precautions are taken to ensure exam security. l There is a strong trainee evaluation and feedback system, supported by ll appropriate documentation. Instructor counseling is provided, but remedial training is r.ot administratively required. The process encourages candidate initiative through strict examination standards to continue in vraining (i.e., three weekly quiz failures results in temination from training).

5.0 Program Evaluation .

5.1 Discussion e Training program evaluation methods were reviewed using the documentation described in Section A.3 above to detemine whether:

A documented program is in place to systematically and continually evaluate the effectiveness of the training program and revise the program as required.

The trai.ning program conduct and content are continually monitored and evaluated.

Feedback from on-the-job performance is actively solicited, evaluated, and incorporated as training program modifications, c

• Operating experience and other external factors are reviewed for evidence of job performance-related lessons learned.

Ine training implications of plant modifications, and procedural and administrative changes are integrated into both initial and continuing training programs. j The program evaluates the training management, staff size, workload, and quali.fications and performance. ,

. l 5.2 Findinfis - Oconee has a well-developed and implemented program in place to eva unte and revise training programs. Shift personnel interviewed expressed concern, however, about the lenath of time it takes to get training on a new plant system into the training profiram. They specifically cited three new systems in use at the p ant on which they have not yet been trained.

An analysis of examinations and tests is conducted to determine if any of the questions appear to be " outliers," i.e., questions that are answered correctly or incorrectly by a very large percentage of the test population. This information is fed back either to improve testing or to determine if there are weaknesses in the training program.

Instructor critiques are used for evaluation and revision of programs.

Trainee critiques are also used for program revision. All feedback 1

4. * ,

.g.

mechanisms identified in the staff's review procedures were.in the

. Oconee Training Plan; our review also confirmed that the feedback a procedures are effectively implemented. i The training staff is evaluated twice a year by supervisors, students, and instructional systems development specialists. In addition, instructors are videotaped and critiqued. Instructors participate in the critique of their own video tapes. Initial and continuing training are provided for the staff. Some instructors did indicate that they had to use their own time for class preparation because of their workload.

Oconee conducts a zero-based manpower study annually to determine staffing needs for training.

C. Sumar.y of Findings The findings from the review of the NLO, R0, and SR0/STA training proprams at Oconee, using today's standards, indicate effective training imp,ementation, trainee evaluation and program evaluation. The job analyses of the NLO and R0 programs was systematic and produced a list of the tasks that can be linked to learning objectives in lesson plans.

However, the SR0/STA task list is incomplete. In many cases, the learning objectives for all these programs did not include conditions or standards and many learning objectives for the SR0/STA program are too broad and vague. The SR0/STA training program would not meet two of five elements contained in the Commission's Policy Statement on training and qualifications. However, a more significant finding ic the fact i that the training department has made substantial improvements to their program since being accredited in August 1983. The staff did not

discover any weakness or deficiency that had not already been identified by the utility. In fact, utility corrective actions were already underway and are scheduled to be completed in 1986. It is clear that

- Oconee's training department has been successful in training and developing skilled operators. The operators' performance on NRC requalification exams (24 of 25 passed), their excellent operating record (nooperator-inducedtripsinover2 years),andthe high quality training observed by the staff are positive indications of effective training.

..,_m

4.

COMPARISON OF NRC REVIEW QUESTIONS AND INPO ACCREDITATION OBJECTIVES AND CRITERIA (INPO85-002)

• NRC REVIEW 00ESTIONS* INP0 OBJECTIVE / COMENTS CRITERIA 1.1 Was a systematic method 4.1 used for identifying the tasks that make up the job (c)beingevaluated?

1.2 Was a systematic method 4.2 used for selecting tasks for which training will '

be provided?

.1.3 Are tasks requiring initial 5. INP0's Objective 5 training only and those addresses requiring continuing training both initial and continuing differentiated? training. INP0 is developing i differentiated? guidance for continuing training program development.

1.4 Is analysis of tasks chosen 6.2 for training adequate for

development of learning l

objectives?

. 1.5 Are approved procedures 1.5 implemented so that 4.4 analysis information is kept current as job perforinance ,

requirements change?

2.1 Are there learning objectives 6.2 for each of the tasks selected for this evaluation?

2.2 Are learning objectives 6.2 derived from or related to the knowledges, skills, and abilities needed for successful job performance?

• revised based upon NRC experience during first three pcst-accreditation reviews and February 25, 1986, meet,ing between NRC and INPO.

i 1

~

I' ' - --- .-

1 .

NRC REVIEW OVESTIONS* _INPO OBJECTIVE / COMMENTS CRITERIA

~

2.3 Does each learning objective 6.3 .

state the dob perfomance .

behaviors (actions) expected of trainees upon completion -

of training? ,

2.4 Does each learning objective 6.3 .'

state the job performance-basedcondition(s)underwhich the trainee actions will take place?

2.5 Does each of the learning 6.3 objectives state a specific job performance-based standard for successful performance of the learning objective?

2.6 Are there written procedures 1.5 that require modification of learning objectives when related job performance -

requirements change?

c 3.1 Is there a written plan that 1.2 clearly and specifically states the training organization's goals, objectives, and plans, and ,

relationships with other parts of the facility's organization?

3.2 Are the responsibilities 1.3 and authority of training -

organization personnel clearly stated in writing?

j I

3.3 Are there documented 2.

qualification and training requirements for the training j staff that address both subject matter and instructional skills and knowledge appropriate for specific assignments?

3.4 Is there evidence that the 8.

appropriateness of 9.

instructional settings 10.

has been evaluated? 11.

4.

• e NRC REVIEW 00ESTIONS* INPO OBJECTIVE / COMMENTS CRITERIA 3.5 Is the organization and 5.1 ,

V sequencing of the initial 6.5 training program (s) based upon the relationships *

, l among leatning objectives?

3.6 Is the organization and 5.1 -

sequencing of continuing 6.5 training based upon the relationships among learning objectives?

3.7 Are lesson plans available 7.2 that provide for consistent training delivery?

3.8 Is there evidence that the 4.3 appropriateness of existing instructional materials has been evaluated based upon identified trainee needs and learning objectives?

~

3.9 Is training being conducted 8.

in an adequate manner as 9.

c determined through application 10. -

of training observation 11.

checklists?

- 3.10 Are adequate methods 1.7 est.ablished for maintaining training rect,rds?

4.1 Are exemptions from training 1.6 based upon performance-based l

l testing or other objective evaluation methods only?

4.2 Is trainee evaluation 6.2 appropriate to job perfonnance 8.6 requirements and training 9.4 l objectives? 9.5 10.5 11.6 4.3 Is trainee perfonnance 8.6 evaluated regularly during 9.4 the training program, and 10.5 prompt, objective feedback 11.6 provided on a regular basis?

m . --* .,s.

. <. k NRC REVIEW 00ESTIONS* INPO OBJECTIVE / COMMENTS CRITERIA 4.4 Are trainess who perform none INPO requires that trainees .

below minimum standards who perform below minimum provided remedial training, standards do not graduate.

ratested and removed from '

Remedial training and

l the training program if retesting are encouraged but not required.

minimum standards are not ,

met? ,

4.5 Are job incumbents who none INP0 is developing written perfom below minimum guidance with respect to removal standards during from associated job duties and

requalification or remedial training. The continuing training Accreditation Board frequently removed from associated questions utilities on their job dutirs and provided policies in these areas.

remedial training?

4.6 Are appropriate precautions none- Precautions to preclude taken to preclude compromise compromise of tests are not of test contents? covered in INP0 criteria but are usually addressed by the Accreditation Board.

5.1 Is there evidence that a 12.1 method is in place to systematically evaluate the

, effectiveness of training programs and to revise the programs as required?

- 5.2 Are examination and operating 12.3 l test results evaluated such that tests are improved and feedback is provided to improve training?

l 5.3 Are instructor critiques of 12.

training used for program evaluation?

5.4 Are trainee critiques of 12.3 training used for program

evaluation?

5.5 Are on-the-job experiences 12.4 solicit'defrom job incumbents and used for program evaluation?

5.6 Is feedback from supervisors 12.4 about job perfomance problems solicited for program evaluation?

8

.4^o NRC REVIEW OVESTIONS* INPO OBJECTIVE / COMMENTS  !

CRITERIA

~

5.7 Are both internal and 12.1 external training program audit / evaluation findings used for program evaluation? ,,

5.8 Is the performance of each 2.7 .

member of the training staff -

objectively evaluated on a regular basis?

9 4

b n

~

0

- --- . ,.,n -, , ., -------.c- -,n- - - . . . - --