ML20149C246
| ML20149C246 | |
| Person / Time | |
|---|---|
| Site: | Arkansas Nuclear, Callaway, 05000000 |
| Issue date: | 12/03/1986 |
| From: | Russell W Office of Nuclear Reactor Regulation |
| To: | Strahm K INSTITUTE OF NUCLEAR POWER OPERATIONS |
| Shared Package | |
| ML082310270 | List: |
| References | |
| FOIA-87-787, RTR-NUREG-1220 NUDOCS 8612100113 | |
| Download: ML20149C246 (33) | |
Text
'
~
4;.
DISTRIBUTION:
s Central Files De: ember 3,1986 MLRoe P0'Connor i"i"8 Mr. Ken Strahm Institute of Nuclear Power Operations 1100 circle 75 Parkway Suite 1500 Atlanta, Georgia 30339
Dear Mr. Strahm:
Enclosed are the final reports from the staff's post-accreditation audit at Callaway and Arkansas Nuclear One (Ah0) conducted in October and November 1986, respectively. The audit teams used the Training Review Criteria and Procedures published as NUREG-1220 in July 1986.
The Callaway training program was developed frcn a task analysis that identified skills and knowledges from which le&rning objectives were derived.
Though continuing training needs were not identified from this task analysis information, the criticality, frequency, and difficulty of task performance were identified, and can be used in the future to detemine tasks appropriate for continuing training.
All the members of the training staff at Callaway were able to answer questions about training beyond those addressed to their own specific area.
Both simulator and on-the-job training at Callaway include written tests and standard simulator and qualification card evaluations.
In addition, s
'!. perfomance-based training is being developed for positions in the utility beyond those currently addressed by accreditation.
The training program at ANO was also developed from a task analysis that identified knowledges and skills from which learning objectives were derived.
Continuing training needs were not identified from the analysis.
This utility is unique in that both its task analysis and program evaluation are carried out by an external group in the corpnrate personnel division.
The ANO training programs contain all five elements for perfomance-based training. There are several areas, however, that are not covered by specific procedures. This is an issue that needs to be addressed by INP0 and the NRC.
The NRC audit teams have seen programs that are proceduralized to varying degrees. Some judgnent needs to be made as to the degree of proceduralization needed to ensure consistent training over time.
It is also interesting to note that both of these programs included FeI n ->/7-767 qualifications cards for instructors that require demonstration of performance-based skills and knowledge.
U'/y It is clear from the training facilities, materials, and the increase in staff that both Callaway and ANO have made a strong management commitment to
- r fcr m
- W.-dtv
'y,
~ w.....
...g.................
y
,,4 3
.... g..
.=.
OFFICI AL R ECOR D Cbkh
""""" "~"""U$. AO5M$5 ce ro:w sie no,so acu ord
~
~
(
2 Both of these progiams were developed from complete task analysis information that included knowledge, skills and~ abilities.
This has generally resulted in comprehensive learning objectives. The majority of the other programs we have reviewed did not include complete JTAs and our ability to judge the comprehensiveness of learning objectives was hindered. We believe that these findings support our position that both job and task analyses are necessary for full implementation of performance-based training.
If you have any questions about the staff's training review activities, please contact me at 301/492-2803.
Sincerely.
Original signed by:
Bruce A. Boger for/ William T. Russell, Director Division of Human Factors Technology Office of Nuclear Reactor Regulation
Enclosures:
As stated cc:
J. Sniezek D'n'/DSM2/LTR TO STRAHM j
s, p.....a./ad,W..........y.,y...........................
->..m e a
nd,Ie.....
...w.
...s..R.n..m....
a.a
...un tm) 12/3 /86 1 /j /86 12/}/86 k2/j/86
- v.5 am in soo 24 4ac romu sia no,soiwnew ond OFFICIAL RECORD COPY
POST-ACCREDITATION AUDIT OF TRAINING PROGRAMS AT CALLAWAY A.
Introduction 1.0 Background. On October 14 through 16, 1986, the NRC staff conducted a post-accreditation audit of training at Callaway. All 10 of the eligible Callaway training programs have received INPO accreditation.
The Callaway R0 and mechanical maintenance training programs were reviewed.
2.0 Approach.
Prior to the on-site review, NRC staff selected two programs for review and informed the utility.
Five tasks for each program were selected from the utility's task listing and the utility was informed so that the materials associated with those tasks were available for review. One additional task for each program was identified on site during the review.
The remainder of the review consisted of document review, interviews and interactions with both training and operations staff and management, and classroom and simulator observations to answer the questions in the NRC's "Training Review Criteria and Procedures" (NUREG-1220, June 1986). The utility briefed the NRC review team on the general approach to training levelopment at Callaway and explained their approach to analysis.
The NRC review team consisted of three training and assessment specialists'fr6m DHFT/NRR and two subject matter experts from Region III.
3.0 Criteria. The criteria used by the staff to audit the implementation of performance-based training programs are taken directly from the "Connission Policy' Statement on Training and Qualification of Nuclear Power Plant Personnel" (50 FR 11147) of March 20, 1985.
In its Policy Statement, the NRC states that the following five elements are essential to acceptable performance-based training programs:
(1) Systematic analysis of the jobs to be performed, (2) Learning objectives that are derived from the analysis and that describe desired performance after training.
(3) Training design and implementation based on the learning objectives, (4) Evaluation of trainee mastery of the objectives during training, and (5) Evaluation and revision of the training based on the perfonnance of trained personnel in the job setting.
4.0 accumentation.
To support its review, the NRC requested that the following types of documentation be provided by the facility:
- Instructions / procedures related to:
?/
- Systematic methods used to analyze jobs,
- Training organization goals, objectives, and plans.
- Responsibilities / authority of training organization personnel.
- Methods for evaluating / selecting instructional materials, methods, and media,
- Methods for organizing / sequencing of training.
- Methods for keeping training programs current.
- Maintenance of training records.
- Selection of candidates for training and the granting of waivers / exemptions from training.
- Evaluation of' training programs, and
- Training, qualification, and evaluation of instructors.
- Task lists for the jobs being reviewed
- Documentation related to:
- Development / validation of task lists.
- Selection of tasks for formal training.
l
~
3-
- Analysis of tasks,
- Analysis of on-the-job performance problems and industry events, and
- Evaluation / audits of the training program (s).
- Roster / organization chart for the training organization
- Training schedule The above documentation is the type normally associated with
,.j participation in the INPO accreditation process.
5.0 Scope of Audit. The programs audited and the tasks selected for review within those programs were:
PROGRAM TASKS
.. i (1) Reactor Operator
- Respond to high activity in the reactor coolant system
- Perform normal blowdown of steam generators
- Respond to generator gas system malfunction
- Operate the cire water system at low loao and/or extreme air and cold 1
water temperatures
- Borate the RHR system to greater than or equal to reactor coolant system l'~~
boron concentration
- Conduct a cooldown from outside the control room (2) Mechanical Maintenance
- Use lubricants and solvents
- Perform preventive and/or corrective maintenance on pressurizer safety i
valves l
l
4
- Perform functional pre-start checkout on MS/FW isolation valve operators
- Perfonn corrective and preventive maintenance of reactor coolant pump motors
- Remove and install snubbers from/in plant systems Although five tasks were preselected for the mechanical maintenance program, it was determined on site that one of the tasks was no longer valid. Therefore, only five total tasks, including the one selected at the site, were reviewed for the mechanical maintenance program.
1 B.
Results of Review The following discussion of the review findings parallels the elements of the Comission's Policy Statement on Training and Qualification.
1.0 Systematic Analysis of Jobs to be Performed 1.1 Discussion. Analysis methods, procedures and products were reviewed using the documentation described in Section A.4 above to detennine whether:
- A systematic method was used for identifying the tasks that make upjob(s)beingevaluated.
- A systematic method was used for selecting tasks for which training is provided.
- Tasks requiring initial training only and those requiring continuing training were differentiated.
- Analysis of tasks chosen for training was adequate for development of learning objectives.
- Approved procedures are implemented so that analysis infonnation is kept current as job performance requirements change.
1.2 Findings. Tasks for the Reactor Operator (RO) position at Callaway were identified through a Job Analysis Task Force composed of Operations and Training personnel.
It was the responsibility of a Task Force to develop and administer the Job Analysis Questionnaire. Sources for questionnaire development included, but ware not limited to, the following:
- INP0 Task Data Base;
- INP0 Questionnaire for R0;
- Interviews with Job Incumbents / Supervisors;
- Procedures (Administrative, Operating, Departmental and Surveillance);
'. /
- Equipment Lists; and
- Technical Specifications.
After the development of the Job Analysis Questionnaire, a
(
Validation Board consisting of the responsible Department Head U.
designee, Superintendent.of Training and/or Applicable Training Supervisor, all members of the JA task force, and any other supervisory personnel deemed necessary produced the validated date and major task findings from which the final Job Analysis was produced. Subtasks listed under major tasks were not validated by the Board. 'Defelopment of the task list for the R0 position included a needs assessment of knowledges and skills required to l
perform each task.
l 1
l Continuing training at Callaway is treated as a part of the initial training program for which criticality, frequency, and decay of performance have been identified.
It was suggested that this information be used to determine continuing training needs in addition to sources currently used, such as Appendix A of 10 CFR Part 55, course critiques, and Training Field Reports which may be filled out by any plant employee.
Procedures have been implemented to ensure currency of task analysis information as job performance requirements change.
Deficiency Reports and the Job Analysis Change Form are two of the mechanisms that are in place at Callaway for inaintaining accurate j
analysis infonnation.
Job task analysis-based programs at Callaway include mechanical maintenance. As with the R0 program, the analysis was based on the INPO task list, Callaway technical manuals, and plant procedures.
A validation board reviewed the tasks to determine which ones actually applied to Callaway, determined the appropriate training settings and the knowledges and skills needed to accomplish the tasks. This board consisted of the training supervisor from the maintenance training group and management personnel from the plant maintenance department.
In addition, a survey was conducted to detennine frequency, difficulty, and criticality of tasks. The survey sample included at least 20% of personnel assigned to the mechanical maintenance job.
No less than five people were ever surveyed. No specific tasks were identified for continuing training for the mechanical maintenance program.
This will be l
done, however, sithin the next year.
Procedures are in place for keeping analysis information current; written examples were provided to verify implementation of these procedures.
l
(
l
-wW
- 94 w
7 2.0 Development of Learning Objectives 2.1 Discussion. Learning objectives were reviewed for the subject programs to detemine whether:
- There are learning objectives for each of the tasks selected for review.
- Learning objectives are derived from or related to the knowledge, skills and abilities needed for successful job perfomance.
- Each learning objective states the job performance behaviors expected of trainees upon completiort of training.
- Each learning objective states the job performance-based conditions under which th2 trainee actions will take place.
- Each learning objective states the specific job performance-based standard for successful performance of the objective.
' Written procedures require modification of learning objectives when related job performance requirements change.
2.2 Findings.
For the R0. program each task selected for review was supported by written learning objectives documented in lesson plans t
and on qualification cards.
The Callaway Self Evaluation Report (SER) states the knowledge and performance requirements for each R0 task.
These requirements are supported by learning objectives I
which contain conditions, actions, and standards to be met during classroom instruction. Qualification cards state performance objectives, and the method by which performance objectives may be satisfied (simulation, observation, damonstration or discussion).
t
8 Learning objectives contain a specific job performance-based standard for successful performance.
Within the classroom context, a grade of 80% is considered successful for a written examination; the standard for successful performance on oral examinations is a satisfactory rating.
For practical factors (qualifications cards),
success is determined based upon accurate completion of the procedure.
Each of the tasks selected for review in the mechanical maintenance program was supported by learning objectives.
The learning objectives were derived from knowledges identified in the analysis phase of the program.
Qualification card items were derived from skills or performance items that came from that same phase of the program. All learning objectives stated the job performance behaviors expected of trainees upon completion of training.
Conditions and standards for many of the objectives were referenced in procedures or stated as a generic condition and standard. A number of the subtasks had specific conditions and standards. All l
-l,
performance items had qualification standards.
l l
Written procedures spea,1fy that learning objectives for all programs are to be modified as job performance requirements change as identified through Training Field Reports, Course Deficiency Reports, and as a result.of the Quarterly Training Managers Meeting.
3.0 Design / Implementation 3.1 Discussion.' Using the documentation described in Section A.4 above, the design and implementation of performance-based training was reviewed to determine whether:
l l
- There is a written plan that clearly and specifically states the training organization's goals, objectives, plans and relationships with other parts of the facility's organization.
- Responsibilities and authority of training organization personnel are clearly stated in writing.
- There are documented cualification and training requirements for the training staff that address both subject matter and instructional skills and knowledge appropriate for specific assignments.
- There is evidence that the appropriateness of instructional settings has been evaluated.
~
- The organization and sequencing of the initial training programs are based upon the relationships among learning objectives.
1
- The organization and sequencing of continuing training is based upon the relationships among learning objectives.
- Lesson plans are available that provide for consistent training delivery.
- There is evidence that the appropriateness of existing instructional materials has been evaluated based upon identified trainee needs and learning objectives.
- Training is-being conducted in an adequate manner as determined through application of training observation checklists.
- There are adequate methods established for maintaining training records.
J
3.2 Findings. Nuclear Function Directive 5 and the Callaway Training Department Goa'Is and Objectives state clearly and specifically the organization's goals, objectives, plans, and relationships with other parts of the facility's organization. The Nuclear Function Program Sumary provides quarterly updates of goals and objectives.
Responsibilities and authority of training personnel are clearly described in Administrative Procedures.
There is a definitive program to ensure that qualification and training requirements for the training staff include both subject matter and instructional skills and knowledge. The Instructor Training and Qualification Program requires qualification card signatures for required courses, knowledge items, practical factors and an evaluation of practice teaching.
Qualification standards i
are also in place.
l Consideration was given to the appropriateness of instructional l
settings in the development of the task analyses. A distinction l
was made between training needs for formal (classroom) training, on the-job trair.ing (requiring completion of qualification cards),
and self-study. One of the functions of the Validation Board was an assessment of the appropriateness of the ins ~tructional settings.
Organization and sequencin'g of the initial R0 training program l
content is based upon the relationships among learning objectives.
A suggestion that arose as a result of this review was that the sequence of courses could be further refined.
The R0 program requires an audit examination of qualification cards prior to the individual beinc) certified as ready to take the NRC licensing examination.
The organization and sequencing of continued training, based upon the relationships among learning objectives was r.ot an applicable area for review at this time as continuing training needs have not been identified for Callaway's training programs.
Lessons plans it the Callaway Training Center provide for consistent delivery of instructional material by including the following:
- Learning objectives
- Presentation (Instruction)
- Sumary
- Review Questions
- References
- Student Handouts
- Audio-visual Aids (when appropriate)
The R0 training program at this site was developed without direct reference to existing instructional materials.
Following the completion of new program development, the prior program and materials were reviewed in order to incorporate any additional appropriate materials.
The conduct of training as observed was appropriate to the setting and the material.
Personnel who sign off on maintenance tasks on qualification cards are trained as on-the-job trainers.
Although there is appropriate sequencing in the mechanical maintenance program, the present group of mechanical maintenance personnel are not bound by the sequence because they are all at the journeyman level. This group participates in the portions of the training program where the greatest need exists.
However, all apprentices and new hires for 1986 will be trained according to the sequence.
In addition, all incumbent mechanical maintenan n,,ersonnel must have completed the entire program by January 1, 1989.
Lesson plans that provide for consistent training delivery were in place for all tasks selected for review in the mechanical maintenance program. These lesson plans must be reviewed by the senior training supervisor and the
instructional technologist for educational quality and to ensure that the lesson plans and their learning objectives are based on the task analysis. The lesson plans are also reviewed by the head of the maintenance department to ensure the technical quality of the course material.
Both classroom and on-the-job training were observed and found to be adequate. A review of the methods and implementation for maintaining training records at this site provided information on a relatively new and well-implemented computer system for record maintenance.
Examples of training records were examined to determine appropriate implementation.
Individual trainees can call up their own training records on computer terminals located throughout the training center and the 3
plant.
4.0 Trainee Evaluation 4.1 Discussion. The methods for and use of trainee evaluations were reviewed using the documentation described in Section A.4 above to determine whether:
- Exemptions from training are based upon performance-based testing or other objective evaluation methods.
l
- Trainee evaluation is appropriate to job performance requirements j
and training objectives.
- Trainee performance is evaluated regularly during the training program and prompt, objective feedback provided on a regular basis.
l
- Trainees who perform below minimum standards are provided remedial training, retested, and removed from the training l
program if minimum standards are not met.
1
- Job incumbents who peiform below minimum standards during requalification or cont lnuing training are removed from associated job duties and provided remedial training.
- Appropriate precautions are taken to preclude compromise of test contents.
4.2 Findings.
The review determined that waivers or exemptions from training may only be granted following administration of examinations.
The passing criterion for exemptions is 100% for Instructors, and for all other exemptions, the passing grade is noted in the course syllabus.
Trainee evaluation based upon job performance requirements is appropriate in that each task selected for review in the R0 program was evalu&ted either in formal training through testing major tasks or on qualification cards for subtasks.
Perfonmance is regularly evaluated in the R0 program through a mechanism requiring three i.'
examinations per course with the exception of simulator courses which require one written and one oral evaluation.
Procedural requirements ensure that prompt, objective feedback is provided within 3 days of the evaluation.
Trainee evaluation of training and instructors is also solicited through critiques.
Trainees who perform below minimum standards are provided remedial training, retested, and removed from the training program if minimum standards are not met. Two methods for ensuring that difficulties are used as feedback to the training program are course critiques filled out by the trainees and a review by the Academic Review Board.
Job incumbents who perform below minimum standards during requaiification or continuing training are removed from associated job duties, and provided remedial trair.ing.
Trainee critiques are also used to provide feedback to the training program.
l 4 As with the RO program, exemptions from mechanical maintenance training are based on performance-based testing. Mechanical maintenance personnel must normally go through the 36-month Callaway Apprentice program before entering the mechanical maintenance training program at the journeyman level. A procedure is in place that requires regular trainee performance evaluation and prompt, objective feedback. Trainees who perform below minimum standards are provided individualized remedial training and are retested.
Examples of implementation of this system were reviewed.
To date, it has not been necessary to drop anyone from the program since remediation and retesting has always proved effective.
Procedures are in place to ensure that tests are not used more than once and that no more than 25% of questions are repeated.
Procedures also provided for appropriate proctorir.g and security measures to preclude compromise of test contents.
It was noted that some of the test questions in the subject area of reactor coolant pumps and safety and relief valves lack the depth of the course material, 5.0 Program Evaluation 5.1 Discussion. Training program evaluation methods were reviewed using the documentation described in Section A.4 above to determine whether:
- A method is in place to systematically evaluate the effectiveness of training programs and to revise the programs as required.
' Examination and operating test results are evaluated so that tests are improved and feedback is provided to improve training.
- Instructor critiques of training are used for program evaluation.
)
__-____________.--------------------a
.
- Trainee critiques of training are used for program evaluation.
l l
- On-the-job experiences are solicited from job incumbents and used I
for program evaluation.
t
- Feedback from supervisors about job performance is solicited for program evaluation.
- Both internal and external training program audits / evaluation findings are used for program evaluation, l
l l
- The performance of each member of the training staff is objectively evaluated on a regular basis.
l 1
5.2 Findings. There is evidence that several methods are in place to i
systematically evaluate training program effectiveness and make revisions as necessary based upon the feedback. One mechanism is i
theTrainingActionTrackingSystem(TRAT). This proceduralized method ensures that all related changes in programs are i
implemented. The training department has recently developed mechanisms for test evaluation based upon trainee scores and instructors are able to refer to an examination question bank prior to the start of a course.
Critiques, to the learning objective level,oftrainingcontentandinstructors(includingvendors)are assessed as part of the evaluation of training programs. Questions asked by students during examination administration are reviewed for training program feedback.
Vendor instructors are also critiqued by training department staff.
On-the-job experiences are solicited from job incumbents for program evaluation. There are several mechanisms from which the information is drawn including formal 90-120 day follow-ups and the Training Field Report which may be filled out by any individual.
Supervisor feedback is solicited through their comments on the trainees' training evaluation form as well as through Training Field Reports. There are also forms for both trainees and supervisors to critique OJT.
C.
Summary of Findings The findings from the post-accreditation audit at Callaway indicate that comprehensive task inventories have been developed that include an assessment of the skills and knowledges which serve as the basis for learning objectives, training and evaluation.
Prior to the selection of the two programs that would be the subject of this review, the licensee provided to the NRC review team the task analyses for all 10 INPO i
accredited programs.
It should be noted that while INP0 currently does not require the development of task analysis information for the Technical Staff and Managers Program, the training program for this
(
category at Callaway is based upon a task analysis. Although continuing training has not been identified at the task level, infonnation on the criticality, difficulty, and frequency of task perfonnance is available and continuing training program development will be addressed in the future.
Selection of training methods and media at Callaway is systematically addressed as a responsibility of the Validation Board.
l Instructor qualification and training includes qualification card l
assessment of both instructional and subject matter skill and knowledge requirements. Trainee evaluation is both objective and prompt.
Evaluation mechanisms provide feedback for both trainee and training program' evaluation.
Discussions with job incumbents, supervisors, and the Training Department personnel reflected a strong management comitment to develop and maintain a dynamic training program.
This comitment is further demonstrated by the high degree to which the five essential elements of performance-based training have been implemented at Callaway.
i POST-ACCREDITATION AUDIT OF TRAINING PROGRAMS AT ARKANSAS NUCLEAR ONE (ANO)
A.
Introduction 1.0 Background.
On November 4 through 6, 1986, the NRC staff conducted a post-accreditation audit of training at Arkansas Nuclear One (ANO).
Six of the eligible ANO training programs have received INPO accreditation.
The ANO licensed reactor operator and electrical maintenance training programs were reviewed.
2.0 Approach.
Prior to the on-site review, NRC staff selected the two programs for review and informed the utility.
Five tasks for each program were selected from the utility's task listing and the utility was informed so that the materials associated with those tasks were available for review.
One additional task for each program was selected at the site during the review.
The remainder of the review consisted of document review, interviews and interactions with both training and operations staff and management, and classroom, laboratory, and simulator observations to answer the questions in the NRC's "Training Review Criteria and Procedures" (NUREG-1220, June 1986). The utility briefed the NRC review team on the general approach to training development at ANO and explained their approach to analysis. The NRC review team consisted of two training specialists and one subject matter expert (licensed operator training) from NRR and one subject matter expert (electrical maintenance) from Region IV.
3.0 Criteria.
The criteria used by the staff to audit the implementation of performance-based training programs are taken directly from the "Commission Policy Statement on Training and Qualification of Nuclear Power Plant Personnel" (50 FR 11147) of March 20, 1985.
In its Policy Statement, the NRC states that the following five elements are essential to acceptable performance-based training programs:
2-(1) Systematic analysis of the jobs to be performed, (2)
Learning objectives that are derived from the analysis and that describe desired performance after training, (3)
Training design and implementation based on the learning objectives, (4) Evaluation of trainee mastery of the objectives during training, and (5)
Evaluation and revision of the training based on the performance of trained personnel in the job setting.
4.0 Documentation. To support its review, the NRC requested that the following types of documentation be provided by the facility:
- Instructions / procedures related to:
- Systematic methods used to analyze jobs,
- Training organization goals, objectives, and plans,
- Responsibilities / authority of training organization personnel,
- Methods for evaluating / selecting instructional materials, methods, and media.
- Methods for organizing / sequencing of training,
- Methods for keeping training programs current,
- Maintenance of training records.
- Selection of candidates for training and the granting of waivers / exemptions from training,
- Evaluation of training programs, and
- Training, qualification, and evaluation of instructors.
Task lists for the jobs being reviewed
- Documentation related to:
- Development / validation of task lists,
- Selection of tasks for formal training,
- Analysis of tasks,
- Analysis of on-the-job performance problems and industry events, and
- Evaluation / audits of the training program (s).
- Roster / organization chart for the training organization
- Training schedule The above documentation is the type normally associated with participation in the INP0 accreditation process.
INP0 team visit /
accieditation reports and the Biennial Report for continuation of accreditation were also available.
5.0 Scope of audit. The programs audited and the tasks selected for review within those programs were:
PROGRAM TASKS (1) Reactor Operator
- Electrical Distribution Perfom breaker operation w/o DC control power
- Control Rod Drive
(
Recover dropped control rods
- Main Steam & Main Steam Isolation i
l l
Start up MS and MSIVs
- Emergency Feedwater Feed S/G utilizing electric driven l
EFW pump l
1
4
- Reactor Coolant Collapse steam bubble
' Fuel Handling
- Operate VCT level control in automatic
- This task was chosen at the site to replace a task previously selected, "Transfer CEA within reactor core using manual CEA change tool." The previously selected task is performed by a vendor.
(2)
Electrical Maintenance
- Analyze, troubleshoot and repair mechanical and electrical problems found in the. following valve operators Limitorque l
(
Analyze, troubleshoot and/or repair the following power equipment Main generators (GE & Westinghouse) i
- Demonstrate knowledge of the following types of evaporators l
l (1) Dry Evaporator (2) Flooded Evaporator (3)
Refrigerant-to-air evaporator (4) Refrigerant-to-liquid l
evaporator
- Perform system clean-up using 1.
Operating method 2.
Flush method
- Analyze, troubleshoot, repair fuel handling equipment
- Demonstrate knowledge of bearings in the following areas; analyze defective bearing and determine cause of failure B.
Results of Review The following discussion of the review findings parallels the elements of the Commission's Policy Statement on Training and Qualification.
l 1.0 Systematic Analysis of Jobs to be Performed 1.1 Discussion.
Task analysis methods, procedures and products were l
reviewed using the documentation described in Section A.4 above to determine whether:
- A systematic method was used for identifying the tasks that make up job (s)beingevaluated.
- A systematic method was used for selecting tasks for which training is provided.
- Tasks requiring initial training only and those requiring continuing training were differentiated, i
i l
1
- Analysis of tasks chosen for training was adequate for development of learning objectives.
- Approved procedures are implemented so that analysis information is kept current as job performance requirements change.
1.2 FindinJs.
Analysis of all job positions at ANO is performed by a separate group within the corporate personnel departnent. The initial analysis process starts with interviews of 100% of job incumbents for each position.
The interviews are structured in that a form was used to I
conduct them.
The results of these interviews are documented and reviewed by the task analysis group. The information is reviewed with job incumbents to validate the analysis.
In addition to this validation process, the respective supervisors for each position analyzed check the incumbents' comments for accuracy; all changes are checked as well.
Knowledges and skills are listed for each position but not tied to specific tasks until the task list is turned over to the training department.
Knowledges and skills for prerequisite positiors, e.g.,
Auxiliary Operator, are not included. Therefore, basic theory knowledges that would be considered part of A0 training would not be included in the R0 listing even if the RO would be required to have those knowledges. The plant manager then reviews and approves the task list. At this point, the task lists are turned over to the training department at the plant. The. task list and the list of knowledges and skills are used to write learning objectives in the development phase of the training programs for each job position. The task list is updated each year by interviewing a 20% sample of job incumbents and using the same process as for the initial analysis.
The updated, revised task lists are sent to cognizant training departments with changes noted.
Changes are also initiated at the training department level. Tasks for continuing training are not identified in the analysis stage.
Continuing training for operator programs is based on regulation, i.e.,
10 CFR 55, Appendix A, needs assessment on an annual basis, feedback from written and oral exams, supervisory feedback and on industry
-7 experience. Continuing training for electrical maintenance is based on industry experience and requests from Maintenance Supervisors on an annual basis. These requests are related to infrequently performed tasks, identified performance weaknesses, complex systems modifications, and procedure or equipment changes. The analysis of tasks is adequate for the development of learning objectives, but does not include data en frequency, difficulty, or criticality of performance.
2.0 Development of Learning Objectives 2.1 Discussion.
Learning objectives were reviewed for the subject programs to determine whether:
- There are learning objectives for each of the tasks selected for review.
- LearningAjectives are derived from or related to the knowledge, skills and abilities needed for successful job performance.
l
' Each learning objective states the job performance behaviors expected of trainees upon completion of training.
Each learning objective states the job perfortnance-based conditions under which the trainee actions will take place.
- Each learning objective states the specific job performance-based standard for successful performance of the objective.
- Written procedures' require modification of learning objectives when related job performance requirements change.
2.2 Findings.
Learning objectives were available for each of the tasks selected for the review of the licensed operator program.
Exclusive R0 skills and knowledges were identified and tied to particular tasks.
l
. Actions and conditions were clearly stated.
The standard for successful classroom performance is described as an examination score of 80%
overall, and 70% in each category with the implied standard for each learning objective being 100%. Successful completion of on-the-job training (0JT) is considered to be perfomance completed to the satisfaction of a job incumbent.
Perscns interviewed acknowledged that they had not been trained in providire er svaluating OJT.
However, a training program is being developed to address this issue. A procedure exists to require modification of isaining objectives as job perfomance requirements chaige.
Corrective action may include partial or complete reanalysis of job requirements.
There were learning objectives for the six tasks selected for review from the electrical maintenance program. These objectives were derived from the knowledges and skills needed for successful job performance and stated the job performance behaviors expected at completion of training.
For the learning objectives in the electrical maintenance program that have a related competency demonstration, the conditions are stated more
,/
specifically since these are procedure based. The testing standard for knowledge items is 80% with objectives weighted by importance.
Perfonnance items or competencies are rated on a pass / fail basis, i.e.,
if the trainee repairs a valve and it leaks when tested, the trainee fails. There are written procedures in place for modification of learning objectives in the electrical maintenance program. These procedures identify the sources of input for these changes and state that the changes are the responsibility of the lead maintenance trainer.
3.0 Design / Implementation 3.1 Discussion.
Using the documentation described in Section A.4 above, the design and implementation of performance-based training was reviewed to determine whether:
l
4 e
9 There is a written plan that clearly and specifically states the training organization's goals, objectives, plans and relationships with other parts of the facility's organization.
Responsibilities and authority of training organization personnel are clearly stated in writing.
There are documented qualification and training requirements for the training staff that address both subject matter and instructional skills and knowledge appropriate for specific assignments.
There is evidence that the appropriateness of instructional settings has been evaluated.
The organization and sequencing of the initial training programs are based upon the relationships among learning objectives.
The organization and sequencing of continuing training is based upon the relationships among learning objectives.
Lesson plans are available that provide for consistent training delivery.
1 There is evidence that the appropriateness of existing instructional materials has been evaluated based upon identified trainee needs and l
l learning objectives.
Training is being conducted in an adequate manner as determined through application of training observation checklists.
There are adequate methods established for maintaining training
- records, i
l
3.2 Findings. There is a. written plan that clearly and specifically states the training organization's goals, objectives, and plans, and its relationships with other parts of the facility's organization. The goals are frequently updated and posted on bulletin boards throughout the training department.
Responsibilities and authority of the training organization are also clearly stated in writing. The qualification and training requirements for the training staff are documented. There is a requirement for all training staff members to be trained and certified in instructional technology. This training includes a qualification card containing performance items related to instructional technology skills.
The procedure that addresses the appropriateness of instructional settings references Section 3.2 of the INP0 Training Systems Development Manual (85-006). The organization and sequencing of initial training program content is based on the task analysis. Lesson plans in the operator program also provide for consistent training delivery.
Procedures describe lesson plan content including references, materials, objectives, presentation, review and assignments.._
Appropriateness of existing instructional material was evaluated by both 5
training and operations personnel.
Classroom and simulator training, as observed, were conducted in a satisfactory manner by well-qualified instructors.
The lesson plans reviewed for the electrical maintenance program were thorough and prepared to provide consistent training delivery. Existing l
training materials were evaluated against the task analysis and, where necessary, were revised according to the procedure that outlines standards for all training documents. These standards are in accordance with a systems approach to training. A classroom lecture and related laboratory were observed for the electrical maintenance program.
Both portions of the training were conducted in an adequate manner.
Training records were maintained for all programs using a computerized system.
4.0 Trainee Evaluatioq 4.1 Discussion. The methods for and use of trainee evaluations were reviewed using the documentation described in Section A.4 above to determine whether:
- Exemptions from training are based upon performance-based testing or other objective evaluation methods.
- Trainee evaluation is appropriate to job performance requirements and training objectives.
- Trainee performance is evaluated regularly during the training program and prompt, objective feedback provided on a regular basis.
- Trainees who perform below minimum standards are provided remedial training, retested, and removed from the training program if minimum standards are not met.
- Job incumbents who perform below minimum standards during requalification or continuing training are removed from associated job duties and provided remedial training.
- Appropriate precautions are.taken to preclude compromise of test contents.
4.2 Findings. The entry level qualification for the RO position requires the trainee to be a high school graduate with a math and science background.
Upon entering the program, the student takes Basic Power Principles, General Employee Training, and Nuclear Reactor Fundamentals.
The remainder of the operator program is sequenced in four mandatory segments as follows:
(1) Auxiliary Operator, (2) Waste Control Operator, (3) Reactor Operator, and (4) Senior Reactor Operator. A person with prior experience (e.g., Navy Nuclear or previously licensed
. individuals) may be exempt from the Basic Power Principles and Nuclear Reactor Fundamentals courses with the approval of the Operations and Training Managers. A procedure is in place which indicated who is responsible for granting exemptions; however, there are no critaria established in that procedure.
The team was told that the only exemptions which are allowed in the operator program are those noted above.
Test items for all tasks are consistent with objectives and job perfomance requirements.
In practice, simulator and oral examination
~
results are imediately provided to the trainee. Written exams are generally returned within 3 days. Trainees who perfonn below minimum j
standards (a grade of less than "C" twice) prior to entering the A0 segment of training are removed from the program. A prelicense examination is also given to trainees in the initial training program with those perfonning below minimum standards being removed from the Procedures are in place to ensure that job incumbents program.
falling below minimum standards during requalification training are subject to review by the training and operations supervisors to determine the need for removing the individual from licensed duties.
The Operations Superintendent for Unit 2 reviews, and regrades if necessary, all requalification examinations.
Evidence was provided that an operator was removed from licensed duties pending successful completion of a retake examination.
l Examinations in the initial operator training program are administered no more than one time, while during the six-cycle requalification training, three different examinations are administered to the six classes.
Procedures'are in place to (1) ensure testing materials are l
f contained in a centralized file, (2) describe responsibility and authority for materials, and (3) describe monitoring of requalification testing by instructors.
L Exemptions from electrical maintenance training are based on performance-based testing or documented justification by the training supervisor. This type of exemption must also be signed and approved by the maintenance supervisor. Both the written and performance test items are consistent with learning objectives and -job performance requirements. Trainee performance is regularly evaluated.
Feedback seldom occurs any more than 3 days after evaluation. Trainees who perform below minimum standards in the electrical maintenance program retake the class and are given feedback related to their specific weaknesses as identified through evaluation. When items on evaluations are failed, trainees may not work independently or take the lead on maintenance tasks related to a failed item.. There is a procedure in place that requires tests to be kept in a locked cabinet. The procedure does not state the method for proctoring or the consequences of cheating. Maintenance personnel interviewed, however, clearly understood that the consequences of cheating are dismissal. Although by practice tests are not administered more than once, there is nothing in the procedure that states this requirement. There are ample items in l
the test bank to preclude repeated use of questions.
5.0 Program Evaluation 5.1 Discussion. Training program evaluation methods were reviewed using the documentation described in Section A.4 above to determine whether:
- A method is in place to systematically evaluate the effectiveness of training programs and to revise the programs as required.
- Examination and operating test results are evaluated so that tests are improved and feedback is provided to improve training.
- Instructor critiques of training are used for program evaluation.
l
' Trainee critiques of training are used for program evaluation.
I 1
1 o
- On-the-job experiences are solicited from job incumbents and used for program evaluation.
l
- Feedback from supervisors about job performance is solicited for program evaluation.
- Both internal and external training program audits / evaluation findings are used for program evaluation.
- The performance of each member of the training staff is objectively evaluated on a regular basis.
5.2 Findings. A number of feedback mechanisms are used to evaluate training at ANO. Training Evaluation / Action Requests (TEAR) and Training Manual Revision Requests are used to revise programs as necessary. These mechanisms are proceduralized. Test results are evaluated through item analysis to improve tests and training although this process is not proceduralized. There are procedures in place that require instructors i.'
to review training based on feedback from trainees and their own experience in presenting material. Trainee critiques of training are used to evaluate training.
Examples of these critiques were reviewed.
A poor evaluation results in a discussion between the instructor and his/her supervisor. On-the-job experiences from job incumbents and feedback from supervisors about job performance are two of the feedback mechanisms used to evaluate training programs.
This evaluation, the Training Effectiveness Evaluation (TEE), is both systematic and proceduralized. A group at Arkansas Power and Light Headquarters is dedicated to this effort.
The Quality Assurance (QA) group also evaluates the training program.
Examples of reports from both groups were reviewed.
Each member of the training staff is evaluated at least biannually by his or her supervisor.
This evaluation is covered by procedure.
Instructors are also evaluated by a contractor / professor from Arkansas Tech for platform skills.
s C.
Suramary of Findings The findings of the audit at ANO indicate that both the licensed operator and electrical maintenance training programs have been developed according to a systems approach to training. The decision was made by ANO in 1980 to use the Instructional Systems Development (ISD) model based on the Air Force program.
For this reason, very little of the ANO program had to be revised to fit this type of training system.
All five of the elements essential to performance-based training are implemented at ANO. However, there are areas throughout the program where procedures are not in place to ensure future consistent implementation. The ANO staff indicated that they felt their program was sufficiently proceduralized as it stands.
This utility's method for analysis of jobs and tasks.is unique in + hat it is conducted by a separate group attached to the corporate personcel division. The utility also has a systematic, proceduralized mechanism i
l for overall external program evaluation in its Training Effectiveness
-J.
Evaluation group, which also is based in the personnel division.
l The training facility, training materials, and increase in the size of the training staff since 1980 are indicators of strong management commitment to training.
There has been steady improvement in j
performance on NRC licensing examinations.
Personnel interviews also indicated good rapport between the plant and the training department.
l
,__