ML20204H230
| ML20204H230 | |
| Person / Time | |
|---|---|
| Issue date: | 07/31/1986 |
| From: | Office of Nuclear Reactor Regulation |
| To: | |
| References | |
| NUREG-1220, NUDOCS 8608080057 | |
| Download: ML20204H230 (123) | |
Text
.
l Training Review Criteria
~
and Procedures i
4 i
i i
i i
U.S. Nuclear Regulatory Commission Office of Nuclear Reactor Regulation pn nec i l
c y
-,1 4
S
\\
8608080057 860731 S
- f 1220 R PDR u
e...,
1
=-
i I
L NOTICE Availability of Reference Materials Cited in NRC Publications Most documents cited in NRC publications will be available from one of the following sources:
- 1. The NRC Public Document Rocm,1717 H Street, N.W.
Washington, DC 20555 i
l
- 2. The Superintendent of Documents, U.S. Government Printing Office, Post Office Box 37082, Washington, DC 20013 7082
- 3. The National Technical information Service, Springfield, VA 22161 f
Although the listing tnat follows rApresents the majority of documents cited in NRC publications, j
it is not intended to be exhaustive.
Referenced documents available for inspection and copying for : fee from the NRC Public Docu-ment Room include NHC correspondence and internal NRC memoranda; NRC Office of Inspection and Enforcement bulletins, circulars, information notices, inspection and investigation notices; Licensee Event Reports; vendor reports and correspondence; Commission papers; and applicant and licensee documents and correspondence.
The following documents in the NUREG series are available for purchase from the GPO Sales Prog am: formal NRC staff and contractor reports, NRC-sponsored conference proceedings, and l
NRC booklets and brochures. Also available are Regulatory Guides, NRC regulations in the Code of Federal Regulations, and Nuclear Regulatory Commission Issuances.
f Documents available from the National Technical Information Service include NUREG series i;
reports and technical reports prepared by other federal agencies and reports prepared by the Atomic Energy Commission, forerunner agency to the Nuclear Regulatory Commission.
Documents available from public and special technical libraries include all open literature items, such as books, journal and periodical articles, and transactions. Federal Register notices, federal and state legislation, and congressional reports can usually be obtained from these libraries.
Documents such as theses, dissertations, foreign reports and translations, and non-NRC conference proceedings are available for purchase from the organization sponsoring the publication cited.
Single copies of NRC draft reports are available free, to the extent of supply, upon written request to the Division of Technical Information and Document Control, U.S. Nuclear Regulatory Com-mission, Washington, DC 20555.
Copies of industry codes and standards used in a substantive manner in the NRC regulatory process are maintained at the NRC Library, 7920 Norfolk Avenue, Bethesda, Maryland, and are available there for reference use by the public. Codes and standards are usually copyrighted and may be purchased from the originating organization or, if they are American National Standards, from the American National Standards Institute,1430 Broadway, New York, NY 10018.
m
NUREG-1220 Training Review Criteria and Procedures Manuscript Completed: May 1986 Date Published: July 1986 Division of Human Factors Technology Office of Nuclear Reactor Regulation U.S. Nuclear Regulatory Commission Washington, D.C. 20555 p.~"
....?. ~'
Notice of Review Package Modification to Accommodate NUREG Format This package was designed for use by NRC personnel engaged in the review of performance-based training programs in nuclear power plants. The actual review package includes split pages and tabs and is contained in a multi ring binder to enable easy use.
In the actual review package, the pages on the right side contain the review questions, observations, checklists, and the scoring summary. These are the pages that are to be written on by the reviewer during each review and replaced for each new review. The pages on the left side contain the review and scoring guidance and are not replaced for each new review. Tabs are used in the original package to separate each section of the review as well as the Introduction, Appendices, and Glossary. Multi-colored pages are also used to facilitate the review.
The NRC staff decided to publish the package in NUREG format to enable large-scale production and distribution of the document. In order to do so, it was necessary to impose another set of page numbers, remove the split paging, index tabs, and multi-color pages. This explains the unusual use of blank space in the document but should enable the user to follow the logical flow of the contents without difficulty.
iii
CONTENTS
_P_ age Notice of Review Package Modification to Accommodate NUREG Format iij A ck no wl edg e m en t s..............................................................................
vii I N TR O D U CTI O N................................................................................
I REVIEW QUESTIONS 13 Section 1 -- Systematic Analysis of Jobs To Be Performed.......................................
13 S-ec tio n 2 -- Development o f Learning Objectives...............................................
29 Section 3 -- Design / Implementation 39 Section 4 -- Trainee Evaluation 55 Section 5 -- Program Evaluation 63 TR AININ G OBSERVATION CHEC KLISTS...........................................................
73 Appendix A -- Training Observation Checklist (Classroom Session)................................
73 Appendix B -- Tr aining Observation Checklist (Laboratory / Workshop Session).......................
83 Appendix C -- Training Observation Checklist (03T Session) 93 Appendix D -- Training Observation Checklist (Simulator Session).................................
103 S C O R IN G S U M M A R Y............................................................................
113 GLOSSARY.....................................................................................
115 y
ACKNOWLEDGEMENTS The review criteria and procedures in this report are a means of ensuring that the five essential i
elements of performance-based training identified in the " Commission Policy Statement on Training and Qualifications of Nuclear Power Plant Personnel" (March 20,1984,50 FR 11147) are implemented in accredited training programs. The criteria are the elements listed in the policy statement. The procedures and questions were developed by the staff of the Maintenance and Training Branch of the Division of Human Factors Technology and based on work conducted by T. J. Mazour of Analysis & Technology, Inc., North Stonington, Connecticut.
l 1
vii
e
<-1 O
-bUR Q
O cc HZ i
PURPOSE OF THESE PROCEDURES The industry, through commitment to the INPO-managed Training Accreditation Program, is implementing what is referred to as performance-based training, or the Systems Approach to Training (SAT). The NRC considers the following five elements as essential to these training programs:
1.
Systematic analysis of the jobs to be performed, 2.
Learning objectives that are derived from the analysis and that describe desired performance af ter training, 3.
Training design and implementation based on the learning objectives, 4.
Evaluation of trainee mastery of the objectives during training, 5.
Evaluation and revision of the training based on the performance of trained personnel in the job setting.
As indicated in the " Commission Policy Statement on Training and Qualifications of Nuclear Power Plant Personnel" of March 20,1985, "the NRC will continue to closely monitor the process [lNPO Accreditation] and its results, and "It remains the continuing responsibility of the NRC to independently evaluate applicants' and licensees' implementation of improve-ment programs...." These procedures provide a systematic means for implementing two of the NRC monitoring functions identified in the Commission Policy Statement:
l 1.
" Continuing evaluation of industrywide training and qualification program effectiveness," and 2.
" Monitoring plant and industry trends and events involving personnel errors."
1
IMPLEMENTING THESE PROCEDURES These procedures are organized around the five essential elements of the SAT (or performance-based training) that are listed on the preceding page.
Discussed below are the ways in which the procedures can be implemented for each of the identified purposes:
CONTINUING EVALUATION OF INDUSTRYWIDE TRAINING AND QUALIFICATION PROGRAM EFFECTIVENESS For these continuing evaluations of facility training programs, the review will proceed through these five essential elements in order. The objectives of such a review are shown on the following page.
It is anticipated that two experienced reviewers working for three days could complete such a review for one job / position.
Reviews related to training for additional jobs / positions, if conducted at the same time, would require substantially less time. This is the case because a significant fraction of the review questions apply to all facility training programs, and thus the answer for one job / position may apply to other jobs / positions.
MONITORING PLANT AND INDUSTRY TRENDS AND EVENTS INVOLVING PERSONNEL ERRORS These procedures may also be used to evaluate facility training programs where a particular human performance problem has been identified or is suspected, and a determination is to be made concerning related training programs. Each of these reviews is likely to be unique in some respects, but in many cases it may not be necessary to answer all the review questions to determine whether there is a training-related problem and to diagnose its cause. The flowchart on page 4 provides a means for determining an efficient way to use these procedures based upon the characteristics of the problem.
2
REVIEW OBJECTIVES 1.
OBJECTIVES FOR SYSTEMATIC ANALYSIS 3.
OBJECTIVES FOR DESIGN 5.
OBJECTIVES FOR PROGRAM OF JOBS TO BE PERFORMED IMPLEMENTATION EVALUATION Verify that Verify that:
Verify that:
1.1 A systematic method is used for identifying and select-3.1 The goals, objectives, responsibilities, and authority of 3.1 Methods are in place to systematically evaluate the ing tasks for training to prepare individuals to do their the training organization and staff are clearly stated.
e!!ectiveness of training programs and revise training jobs.
programs as appropriate.
3.2 Qualifications and training requirements for the train-I.2 Tasks for continuing and initial training are differen-ing staff address both appropriate subject matter and 5.2 Feedback from trainee tests, on-the-job experiences,
- tiated, instructional skills.
and supervisors is used in program evaluation.
1.3 The analysis is adequate for development of learning 3.3 Training is appropriately organized and sequenced and 5.3 Instructor and trainee critiques are used in program objectives.
instructional settings are appropriate to tasks.
evaluation.
1.4 The analysis is kept current as job performance require-3.4 Lesson plans provide for consistent training dehvery.
5.4 Both internal and external program audits are used for 3.5 Existing instructional materials have been evaluated based on training needs.
5.5 Training staff are routinely and objectively evaluated.
3.6 Training is conducted in an adequate menner and records are maintained.
t T
t T
2.
OBJECTIVES FOR DEVELOPMENT OF 4.
OBJECTIVES FOR TRAINING LEARNING OBJECTIVES EVALUATION Verify that:
Verify that' 2.1 There are learning objectives related to knowledge, 4.1 Exemptions from training are objectively determined.
skills, and abilities for each task.
4.2 Trainee performance is regularly evak.ated using appro-2.2 Learning objectives contain actions, conditions, and priate job performance measures and objectives, standards needed for job performance.
4.3 Trainees who perform below minimum standards during 2.3 There are procedures to modify learning objectives as initial and requahfication training receive remedial job performance requirements ch.nge.
training, are retested, and are removed from training or job duties if performance is not acceptable.
4.4 Precautions are in place to prevent test compromise.
3
USE OF THESE PROCEDURES FOR MONITORING EVENTS INVOLVING PERSONNEL ERRORS (START) i SELECT THE DESCRIPTION BELOW STOP. THE REVIEW
(@, @, @ OR @)
OBJECTIVES ARE MET (UNLESS OTHER AREAS THAT BEST CATEGORIZES ARE IDENTIFIED FOR THE PROBLEM EVALUATION).
YES YES DETERMeNE THOSE REVIEW 01 THE TRAINING PROGRAM (S)
QUESTIONS IN SECTIONS USE THE REVIEW WAS (WERE) JUDGED 1 THROUGH 5 THAT ARE pE THE ARE THE QUESTIONS TO IDENTIFY NO REASONS FDR THE NO UNACCEPTABLE OR WEAK RELATED TO THE PREVIOUS DESIRED ELEMENTS ADDITIONAL QUESTIONS THROUGH A PREVIOUS WEAK AREAS. ADD ESENT UNACCEPTABILITY OF THAT SHOULD DE EVALUATED RESULTS REVIEW.
ADDITIONAL QUESTIONS m mmq m wm DETERM:NED IF REQUIRED.
OF THE FINDING.
?
AN INDIVIDUAL. OR A FEW fT INEE EVALU T ON)
CAUSE OF THE NO CAUSE OF THE NO Tl AL QUESTIONS INDIVIDUALS MADE (AN) ERROR (S)
COMPLETE SECTION 5 FOR THE SPECIFIC PROBLEM DETERMINED PROBLEM DETERMINED THAT SHOULD BE THAT CONTRIBUTED TO AN LER INDIVIDUAL INVOLVED BY THESE BY THESE ADDRESSED 8ASED UPON ROGRAM EVALUATION)
OR OTHER SIGNIFICANT EVENT.
IN THE EVENT.
QUESTIONS QUESTIONS E T HE YES YES STOP (UNLESS OTHER AREAS ARE RELATED
/
/
TO THE CAUSE OF N
THE PROBLEM).
A HUMAN PERFORMANCE PROBLEM IS IDENTIFIED THAT IS COMMON TO ALL. OR MOST, OF THE INDIVIDL ALS IN A PARTICULAR JOB OR GROUP OF JOBS.
DOES THE NATURE OTHER HUMAN OF THE PROBLEM INDICATF NO PERFORMANCE AN OBVIOUS RELATIONSHIP WITH PROBLEMS ANY PARTICULAR REVIEW QUESTION OR GROUP OF QUESTIONS
?
YES DETERMINE THE eNSWERS TO THESE QUES'.lONS AND PROCEED ACCORDINGLY.
l 4
l
PREPARATIONS FOR THE REVIEW l.
It is recommended that the f acility be notified well in advance of the review, indicating the program (s) that will be reviewed, and that the facility be requested to assemble:
o Instructions / Procedures related to:
o Task lists for the job (s) being reviewed
- Systematic methods used to analyze jobs, o Documentation related to:
- Training organization goals, objectives, and
- Development / validation of task lists,
- plans,
- Selection of tasks for formal training,
- Responsibilities / authority of training organiza-tion personnel,
- Analysis of tasks,
- Methods for evaluating / selecting instructional
- Analysis of on-the-job performance problems and materials, methods, and media, industry events, and
- Methods for organizing / sequencing of training,
- Evaluations / audits of the training program (s)
- Methods for keeping training programs current, o Roster / organization chart for the training organization
- Maintenance of training records, o Training schedule for the past six months and the next six months
- Selection of candidates for training and the granting of waivers / exemptions from training, o Final accreditation team report
- Evaluation of training programs, and o Biennial status report on accreditation (if applicable)
- Training, qualification, and evaluation of instructors 2.
The f acility should be requested to provide a dedicated individual to interface with the review team. This individual should be knowledgeable of the program (s) being evaluated and know where/how to find information requested by the reviewers. Ideally, this would be the person coordinating the f acility's efforts toward INPO accreditation, it may also be helpful to interview graduate trainees (e.g., operators) and operations management regarding their roles in developing, reviewing, and recommending modifications to the training curriculum.
5
PREPARATIONS FOR THE REVIEW (C:nt'd) 3.
Prospective members of the review team should review the results of previous inspections of this facility's training programs, and LERs within the last two years.
4.
The questions in Sections 1 through 5 should be reviewed and then the five examples that follow this page should be studied as aids in filling out the checklists. The following points are emphasized:
- The scoring method used in this review package is designed to achieve consistent results among reviewers. The meaning of the numerical scores for each item is provided in the " Scoring Guidance" section of the left side pages.
Reviewers are to read carefully the statements that characterize each possible score. They should then assign an appropriate score to the item, circling the number on the right side page. Scores for each element are then totalled, averaged, and entered on the scoring summary page. It should be noted that these scores are used only to achieve internal consistency among the review team participants. (NOTE: The scores are not parrallel to the SALP in fact, they have reverse quantitative meaning in that a rating of @ in a SALP report is low while a @ process; for this review is a "best" score.)
- During the transition period to performance-based training, in situations where the facility has not yet completed the changes required to achieve a score of @, but where actions are in progress that will apparently lead to this score, it may be judged appropriate to use a score of " @ T" where the "T" indicates a transition situation. Such an example is shown on page 10.
- When the scoring guidance is closely adhered to, it is not necessary for reviewers to write lengthy narrative comments. Rather, comment can be "by exception"; that is, if only one task reviewed lacks a particular detail while others fully satisfy the criteria, a written comment should be made about the exception.
5.
It is appropriate to assign two reviewers (one subject matter expert and one training program development expert) to each training program being reviewed. For those questions that involve more than one program (e.g., those related to training program plans), the reviewers should compare their scores and resolve any difference (it may be appropriate to have different scores for these questions for different programs based upon the manner in which the plans are implemented for the different training programs).
6.
Some questions may have the same answer regardless of the training program being reviewed. An example of this can be seen in questions 3.1 and 3.2. In most facilities, the training organization does not change for each program. In general, the review team may want to have one person assigned to answer such questions for all programs rather than requiring each reviewer to separately review materials and formulate answers to each question.
t 6
l
How To Select Tasks for the Review TASKS SELECTED FOR THE REVIEW JOB }lfoJM 0)WS&sIrdFACILITY Crein Mounts),,1 Once the information has been assembled, use it to select three to u tasks from the task list for each job being evaluated. These tasks and information reltted to them will be used to aid in answering questions with respect to each of the five elements of this 1.
Per A m o /emoorors sAJ/s,h, surgev evaluat (Sections I through 5). The following are the criteria to be used in selecting
[.
7
- 1. All tasks should be among those selected for formal training.
( Tos Ic T.1 )
- 2. At least one of the tasks should be among those selected for continuing training.
- 3. At least one of the tasks should be among those selected for formal OJT.
3.
Oe/onfame/We p ersonn e /
( To
//. 2 )
- 4. At least one of the tasks should be among those selected for training using a simulation device (if such a device is used for training for this job).
4.
Per form os a ee-en +rv from
- 5. At least one of the tasks should be from the emergency / abnormal operations area (if applicable to the job).
/n trnsber (Oos t -occidenf } {1a' s A: / 2. S~)
Write the names of tasks selected for the review in the appropriate spaces on 5.
AM0/V2 d DOS /- oce/deo / s ome /e (
page 1-2R, along with any number or other designation assigned to the tasks by the facility.
Also, in the appropriate spaces on page 1-2R, indicate those tasks that meet each of the I/#
Il-five criteria above.
6.
<? N 1q 'hN\\ \\ \\ n 3k Alp. asks are among those selected for fo L='
List the num;m(s) of the task (s) above that were selected for continuing training.
3_.M,f List the numbers of the task (s) above that were selected for formal 03T.
/. 2.
List the number (s) of the task (s) above that were selected for training using a simulation device.
- 3. 5" List the number (s) of the task (s) above that are from the emergency / abnormal operations area (if applicable to the job).
V. 5~
1-2L I-2R 7
REVIEU GUIDANCE REVIE'? REOUIREMENTS 3
d D E!
1.1 Was a synematic method used for identifyire the tasks that maka up the job (s) being
< Qy evaluated?
Qgg
(
n z 1 A method or procedure is considered to be systematic if it possesses the following 1.1 Was a systematic rnethod used for ggO characteristics?
identifying the tasks that make up the job (s) being evaluated?
I@3 o Consists of a logically ordered set of steps, o is clearly documented so that it can be executed with consistent results on COMMENTS: A omuo cr y/ew o[a n
/ndufh w!de Ern k #s/ wo A different occasions by different personne!'ature) of external review / approval for f
Requires a physical record (e.g.,
sign o
historical accountability, and cond'or/ed. N arooo w_s mo/e up Results in a consistently formatted, quality product (where applicable).
of 3_/oA wuYheoAs one 74< NP o
A task is a unit of work that describes the performance of a complete meaningful Troiner. P/on/ -w<<//a tosh e ve-e function in,a job. An example of a task for the health physics (radiation protection) oc///Ed /o Phe /ddus/6-wide. fos 4 technician job is:
lIY DOC.lsmen to tion o f yhja stuttw
" Perform pre-Radiation Work Permit (RWP) surveys."
I5 "N /uc/ed f/i /Ae Fac///fy3 d he co #
- w th thD
(
,7' N @ MN b#b*
je the g ote D are C005 1.2 Was a systematic method used for 3 core 01 selecting tasks for wtuch traarung gddance-will be provided?
I 2 3 SCORING GUIDANCE Score this question as a:
@ if there is no task list, or if the task list is an industry-wide task list or a task list developed for another facility that is used for this facility without review or validation, or if there are no implementing procedures / instructions that describe the process for developing / validating the task list.
@ if a plant-specific task list has been produced based upon a systematic method M }
if both subject and training specialist personnel were involved in the performance 5 of the analysis to the minimum extent required to ensure the relevance and quality }
of the information collected.
@ if a plant-specific task list has been developed that includes all of these characteristics:
o Evidence of involvement in the process by experienced and
'e t i professionals aM job incumbents, d o Validation of the initial task li re a
nt th ou interview, survey, or a group revi o Approval of the task list by mana r erv and training personnel.
1-3L I-3R 8
REVIEW GUIDANCE REVIEO REQUIREMENTS 3
d b id 2.1 Are there learning objectives for each of the tasks selected for this evaluation?
< ;;! O E $ is' A learning objective describes precisely what is to be learned in terms of the expected t-z 2 trainee performance under specified conditions to accepted standards. These learning 2.1 Are there learnmg objectives for
@g8 objectives are of ten referred to as terminal learning objectives, and are easy to eads of the tasks selected for this 2h identify because they are similar to the task statements. An example of a terminal evaluation?
I learning objective for a mechanical technician job is:
COMMENTS: 7dru /cor,,/m ggg g "Given the disassembled parts of a service water pump, proper tools, and the 4
approved procedure, reassemble the pump in accordance with the steps of the approved procedure."
She a s socih/ed -fas 4 ore Learning objecitves should provide the instructional designer with the information 8# ##
necessary to organize objectives, select instructional settings, and other related do /ed M c /of er- /f / 97.5".
actions required to design and develop a training program. Here are two examples of good learning objectives:
"Given a handheld calculator, calculate the volume of a cy!inder, without error" and "When directed by the, shift supervisor and given the appropriate procedure, start a 2.2 Are learning objectives derived main feedwater pump in accordance with the appropriate procedure."
from or related to the knowledges, skills, and abilities needed for successful job performance?
I 2 3 COMMENTS:
AN SCORING GUIDANCE Q hy For the three to six tasks selected for this review score this question as a:
@ if there are no learning objectives for these tasks.
h
@ if there are learning objectives for those tasks:
k o Selected for initial training, o Related to emergency /abnorma tion
@ if there are learning objectives for all lected for this inspection.
2-2L 2-2R 9
REVIEW REQUIREMENTS 3
REVIED GUIDANCE dud O3.1 is there a written plan that clearly and specifically states the training organization's
<Qy y g ;2;8 goals, objectives, and plans, and relationships with other parts of the facility's p.
z organization?-
3.1 Is there a written plan that clearly 0 id O
- U U and specifically states the trauung organizationfs goals, objectives, and 7
plans, and relationships with other parts of the facility's organization?
I 3
COMMENTS: 7/je fro,nfde o
Man o o er r>rsanded a sel>edu/e tJhieb inNJ}ca/ed Yl>of such a n/on is to be deue/oped L3 /+hin the neef d rnon/As.
@"rath scored as 'e the facility has p an l s This question is uestion.
than 'Oi decausthe intent of the q Are rg.
rneet ansthorigy to ly I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if there is no written plan, or if goals, objectives, and plans are stated only in general ways that do rjot permit objective evaluation of their success.
@ if there is a written plan that addresses, as a minimum, the training organizations' goals and plans a_rjd I
some objective evaluation of the achievement of these goals and plans is possible.
@ if there is a written plan that addresses the training org '
objectives, and plans and relationships with other parts of g
n
{
and specific terms that permit an objecti ft c
there is evidence that these goals, lans revised, or that there is a procedure i so.
3-3L 3-3R 10 l
REVIEO REQUIREMENTS REVIEW GUIDANCE
.a
.a Ydd O4.1 Are exemptions from traininst based upon performance-based testire or other objec-H tive evaluation methods only?
Are exemptions from traarung based Z 0 V upon performascc-L J testing or other objective evaluation methods 2h d7 1
Y COMMENTS: ]~ob -sneci$a }cs+
\\ntot@*
i/rins were used as &e bas 4 ye the est\\**
uits i
= -So r n*wnsp inns Som various N
. zygg
'E o in SC matizing estiooS vorfs of f^< No/nhrr y \\ptu "gfagtof*ft hro,orom.
SC*
Trada,e (See 9
b **'
O at the Dfpt. Jnstrucfsn f,4) esstsoea-te to tr -
"T'. d c_,__u m 1 2 3 COMMENTS:
W ANLcuo SCORING GUIDANCE Score this question as as
@ if exemptions from training are permitted without any record of these exemptions.
@ if no exemptions from training are permitted, or if any exemptions permitted are based upon performance-based testing or other objective and documented evalua-tion methods.
@ if the requirements for a score of @ are met an_d only performance-based testing is used for granting exemptions.
- -21.
l1
REMEN RENREMENTS REVIED GUIDANCE j
a a na O5.1
[ $
E' d5 is there evidence that a method is in place to systematically evaluate the etfective-ness of tramuu programs and to revise the pronrams as required?
{
w z 1 3.1 Is there evidence that a method is in ogo This item addresses only whether an evaluation program exists. Subsequent questions P ace to systematically evaluate the l
deal with the specifics of the evaluation program.
effectiveness of training programs and to revise the programs as required?
I 2@
7de,soaram COMMENTS:
a/a s score, C* p ents,consis-l
'3
/~ cts /cturcl wi9hin Yhe /Osh e a e Note thatand *** gg guidanc g fy,, of, y,3 ge,f,f,g,yy) anc/ a sr>e t cher_k of tent.
j, g g,, pg, g,,;g;,,,, p,, y.),g the review /*srichnf3 had N been sir >p/emen f ed'.
05.2Are examination and operating test results evaluated so that tests are improved and feeM is provided to improve trammg?
! 2 3 COMMENTS:
SCORING GUIDANCE Score this question as as
@ if no evidence of an evaluation program exists.
@ if an approved evaluation program is in place.
@ if the requirements for a score of @ are met M there is evidence that the program is periodically evaluated and revised (if I
required).
15 6
5-2L 5-2R l
12
Z O
-b 00WD Cy 3W
->W Od
---w
SECTION 1 --
SECTION 1 --
SYSTEMATIC ANALYSIS OF JOBS TO BE PERFORMED SYSTEMATIC ANALYSIS OF JOBS TO BE PERFORMED Obtain the Fo!!owing:
List of Information Obtained o Instructions / procedures that describe the facility's analysis methods, Instructions / procedures that describe the facility's analysis methods, o Task lists / inventories for the job (s) of interest, Task lists / inventories for the job (s) of interest, o information related to the developmen*Jvalidation of the task lists (s)
Information related to the development / validation of the task list (s)
Information related to the selection of tasks for formal training (initial and Information related to the selection of tasks for o
continuing),
formal training (initial and continding),
o Information related to the analysis of tasks to determine corditions and standards Information related to the analysis of tasks to of performance, and enabling skills and knowledge, and determine conditions and standards of perform-ance, and enabling skills and knowledge, and o Information related to the analysis of on-the-job performance problems and Information related to the analysis of on-the-job industry events-perfeemance problems and industry events.
1-IL I-1R 13
How To Select Tasks for the Review TASKS SELECTED FOR TriE REVIED JOB FACILITY Once the information has been assembled, use it to select three to six tasks from the task list for each job being evaluated. These tasks and information rela *ed to them will be used to aid in answering questions with respect to each of the five elements of this 1.
evaluation (Sections 1 through 5). The following are the criteria to be used in selecting these tasks:
2,
- 1. All tasks should be among those selected for formal training.
- 2. At least one of the tasks should be among those selected for continuing training.
- 3. At least one of the tasks should be among those selected for forma! 03T.
3.
- 4. At least one of the tasks should be among those selected for training using a simulation device (if such a device is used for training for this job).
4*
- 5. At least one of the tasks should be from the emergency / abnormal operations area (if applicable to the job).
Write the names of tasks selected for the review in the appropriate spaces on 5.
page 1-2R, along with any number or other designation assigned to the tasks by the facility.
Also, in the appropriate spaces on page 1-2R, indicate those tasks that meet each of the five criteria above.
6.
All tasks are among those selected for formal training.
List the number (s) of the task (s) above that were selected for continuing training.
List the numbers of the task (s) above that were selected for formal OJT.
List the number (s) of the task (s) above that were selected for training using a simulation device.
List the number (s) of the task (s) above that are from the emergency / abnormal operations area (if applicable to the job).
1-2L I-2R 14
REVIEW REOUIREMENTS REVIEW GUIDANCE
-j ] j
< a e 1.1 Was a systematic niethod used for identifying the tasks that make up the job (s) being H
"}
[y$
evaluated?
"j 1.1 Was a systematic method used for o n2 o z o o A method or procedure is considered to be systematic if it possesses the following identifying the tasks that make up characteristics?
the job (s) being evaluated?
1 2 3 o Consists of a logically ordered set of steps, COMMENTS:
o is clearly documented so that it can be executed with consistent results on different occasions by different personnel, Requires a physical record (e.g.,
signature) of external review / approval for o
historical accountability, and Results in a consistently formatted, quality product (where applicable).
o A task is a unit of work that describes the performance of a complete meaningful function in a job. An example of a task for the health physics (radiation protection) technician job is:
" Perform pre-Radiation Work Permit (RWP) surveys."
SCORING GUIDANCE Score this question as a:
@ if there is no tar,k list, or if the task list is an industry-wide task list or a task list developed for another facility that is used for this facility without review or validation, or-if there are no implementing procedures / instructions that describe the process for developing / validating the task list.
@ if a plant-specific task list has been produced based upon a systematic method af if both subject and training specialist personnel were involved in the performance of the analysis to the minimum extent required to ensure the relevance and quality of the information collected.
@ if a plant-specific task list has been developed that includes all of these characteristics:
o Evidence of involvement in the process by experienced and qualified training professionals aM job incumbents, an.d o Validation of the initial task list by two or more job incumbents through interview, survey, or a group review meeting, and Approval of the task list by managers / supervisors of subject matter and training o
personnel 1-3L I-3R 15
REVIE7 GUIDANCE REVIE7 REQUIREMENTS 1.2 Cas n systematic method used for selecting tasks for which trauuna will be provided?
.d D d c j y w
a
[a To systematically select tasks for formal training, there must a basis for selections.
@ y $i The following are criteria that are generally used as the basis for selecting tasks for 1.2 Was a systematic method used for y8 traming:
selecting tasks for which training will be provided?
I 2 3
~
o Percent of job incumbents who perform the task, o Percent of time spent performing the task, COMMENTS:
o Consequences of inadequate performance, o Frequency of task performance, o Task learning difficulty (or task performance difficulty),
o Time between job entry and task performance, Entry-level skills and knowledge (e.g., job tasks performed in previous jobs),
o o On-the-job performance problems present (sources: LERs, industrial accident reports, plant events resulting from personnel errors and not requiring LERs), and o Industry events applicable to the facility (sources: INPO SEE-IN, SERs, SOERs).
Information concerning these criteria should be collected in an objective, reliable way and should be obtained from individuals who are familiar with the particular job being analyzed. Threshold values for task selection decisions should be established by qualified training specialists.
(Note to reviewer: Information reviewed in answering this review question may be useful in answering questions 5.5 and 5.6) 1 SCORING GUIDANCE Score this question as a:
@ if no systematic basis is identified for selecting tasks for initial training, or if the requirements for a score of @ are not met.
@ if the basis for selection includes consequences of inadequate performance and involvement of job incumbents in the selection process is documented.
@ if the requirements for a score of O2 are met and a generally accepted method for collecting the information (e.g., survey, interview, group consensus meeting) is implemented in such a way that a representative sample of job incumbents' opinions and performance problems are objectively and reliably identified and analyzed for inclusion in training.
1-4L I-4R 16
REVIEW GUIDANCE REVIEW REOUIREMENTS 3
d U E!
1.3 Are tasks requirina initial trainine only and those requirina continuina tralruna
<J H
differentiated?
H<
o.
w 9 z lE in general, the fo!!owing three criteria are used in selecting tasks for continuing 1.3 Are tasks requiring initial traaning O w 0 z o u training (in addition to the criteria used to select tasks for initial training):
only and those requiring continuann training differentiated?
! 2 3
- 1. Consequences of inadequate performance,
- 2. Frequency of task performance, and COMMENTS:
- 3. Decay in task performance with time (e.g., known personnel performance prob-lems).
o Consequences of Inadequate Performance. Those tasks whose performance is so critical that particular attention must be paid to ensuring adequate performance.
o Frequency of Performance. The subset of tasks selected for initial training for which proficiency is not maintained through on-the-job performance (e.g., tasks that are performed infrequently).
o Decay in Task Performance over Time. Those tasks that require periodic practice to maintain proficiency (particularly tasks requiring coordinated psychomotor skills).
SCORING GUIDANCE Score this question as a:
@ if no systematic basis is identifi for selecting tasks for continuing training, or if the requirements for a score of are ny met.
@ if consequences of inadequate performance and either frequency of performance of time decay are considered in selecting tasks for continuing training and involvement of job incumbents in the task selection process is documented.
@ if the requirements for a score of @ are met and a generally accepted method for collecting the information (e.g., survey, interview, group consensus meeting) is implemented in such a way that a representative sample of job incumbents' opinions concerning the tasks to be selected are objectively and reliably identified and interpreted.
1-5L I-5R 17
/
REVIEO REOUIREMENTS 3
REVIED GUIDANCE
.j 3.g
< a -j
[ y "$
1.4 Is analysis of tasks chosen for training adequate for development of learning i-y objectives?
0 ua O z o o The process of identifying such information as conditions, actions, standards of perfor-1.4 Is analysis of tasks chosen for trh mance, and skills and knowledge underlying successful task performance is called a ing adequate for development of task analysis, learning objectives?
I.2 3
A condition is an event that determines, in the job situation, when the job incumbent COMMENTS:
performs a particular task (i.e.,"when the tank high level alarm is activated") or that significantly influences performance of a task (i.e., "given that the reactor coolant system is at normal operating temperature and pressure"). Actions are the behaviors required to complete the task. Standards describe the criteria or standards of perfor-mance of a task that separate " acceptable" from " unacceptable" job performance (i.e.,
" tank level must be maintained between high and low level trip set points").
Shown on the blue pages at the end of Section 1 is an example of the ski!!s and knowledge and related information collected through a comprehensive task analysis (the process shown is similar to INPO's task analysis approach). Note particularly that skills and knowledge are identified by first dividing the task into elements or steps and then determining the skills and knowledge required for each element / step. This illustrates one of the basic tenets of task analysis: if a job is divided into small enough
" pieces," then specific and complete information can be determined about the skills and knowledge, and conditions, actions, and standards of job performance (e.g., a statement such as " knowledge of basic mathematics" without additional detail would be unacceptable while "use of a handheld calculator to calculate the volume of a cylinder" would be acceptable).
l l
SCORING GUIDANCE For the three to six tasks selected for this inspection, score this question as a:
l
@ if no conditions, actions, standards, skills or knowledge have been identified for these tasks, g if the results of a generic task analysis or a task analysis performed for a different facility have been used without reviewing / modifying the information to determine its applicability to this specific facility.
@ if:
A procedure for collecting information concerning conditions, actions, stan-o dards, skills and knowledge exists and o Most of the conditions, actions, ano standards are specific and measurable aj (Continued on next page) j 1-6L I-6R 18 I
l
= _ _
REVIET GUIDANCE 1.4 (Cont'd)
SCORING GUIDANCE (Cont'd) o The skills and knowledge are identified based upon tasks as they are performed for this particular job (through review of generic task analysis data for its applicability or facility-specific analysis)g o The skills and knowledge are specific enough to support the design of training, a_nd o Job incumbents or individuals with equivalent knowledge of the job provided the information or reviewed existing information for accuracy / realism.
@ if:
The requirements for a score of @ are met an_d o
o All conditions, actions, and standards are specific and measurable and o The skills and knowledge are at a level of detail similar to those shown in the example task analysis data collection form at the end of Section 1.
1-7L 19
REVIEO REQUIREMENTS 3
y REVIEW GUIDANCE
.2
>- a
$ $ ?-
I.5 A_re approved procedures impleniented so that Nnifsis information is kept current as "j
[yg s
job perfarmanca requirements change?
g 1.5 Are approved procedures imple-o ua o Job performance requirements may change over time for a variety of reasons, such as:
mented so that analysis information
- U V Backfits/ modifications of plant equipment, is W ment as A Wwmance o
c Changes in plant operating or administrative procedures, miuirements %d 1 2 3 o Lessons learned through operating experience, and COMMENTS:
o Reorganization or changes in job responsibilities.
If training is to remain performance based, then the impact of these changes must be analyzed sa the training program re(sed, as required. Procedures for accomplishing this should be in tne facility's training plan, administrative procedures er related documentation.
Identify a backfit/ plant modification or procedure change that has been implemented during the past three to six months.* Determine whether the actions specified in the modification or procedure cnange related to updating training requirements have been implemented.
(Note to reviewer: Information reviewed in answering this review question may be useful in answering Questions 3.5 and 5.6.)
- If a time frame for review is spccified in the approved procedure, use that instead of the three to six months specified above.
SCORING GUIDANCE Score this questiorras a:
@ if there are no approved procedures in place. of if the procedures were not properly implemented.
@ if procedures are in place that require job performance requirements to be updated as a result of:
o Backfits/ modifications of plant equipment, o Changes in procedures, o Operating experience, and o Reorganization or job changes ary,d,.
the procedure has been followed to the extent required to ensure that job performance requirements are generally met.
@ if approved procedures are in place to require that all the information in Questions 1.1 through 1.4 remains current as job performance requirements char ge and Il the procedure has been properly implemented in all respects.
1 - 81.
1-8R 20
~...
+e 4
TOTAL SCORE FOR SECTION 1 AVERAGE SCORE FOR SECTION 1 (to nearest tenth)
TRANSFER THE INFORMATION ABOVE TO THE SCORING
SUMMARY
(THE LAST PAGE OF THIS DOCUMENT).
e
\\
21
EXAMPLES OF SKILLS AND KNODLEDGE COLLECTED THROUGH A COMPREHENSIVE TASK ANALYSIS,
(Sheet 1 of 6)
TASK ANALYSIS DATA COLLECTION FORM
, l a i 116 Ie i 1 l,I, is I.I, I,I,I, Ima rw c-rm a==== =w=
~~'es S. -
.m sar u.u m e, w
.m --i 2" dium p g* e w m j
E"en "@JA>w A meAes %..e#Gr1 h
$f S pa, W My M & **
- O 2 h p2. r%. arse re.x 4.at =m t k *L y W=S M w
e2 R.a.saw u m
- e2 E-4 haw 4 a e chant W-wW es== =# y4 s 6 safe Ww s40 es.t.
- 32 pt Pansa toa et pms aar es.e ca ra.
w1 Definition / Explanation of Important Parts of the Task Analysis 04 Initiating Cue (s)
The stimulus or stimuli that prompt task performance. (What happens to let the job incumbent know that he should start performing the task?)
05 Terminating Cue (s)
The stimulus or stimuli that indicate task completion. (What happens to let the job incumbent know that he has completed the task?)
06 Standards The testable or measurable criteria regarding quality, sequence, accuracy, and time that must be met while performing the task.
(What are the most important dif ferences between the way someone performs this task well and someone who performs the task poorly?)
1-9 22
EXAMPLES OF SKILLS AND KNOULEDGE COLLECTED THROUGH A COMPREHENSIVE TASK ANALYS5 (Sheet 2 of 6) 1 A_
TASK ANALYSIS DATA COLLECTION FORM
==
01 ac ez cfAndd M en.&me y&.
lll2:r
@J SA P W
- l:'.L".*..!
mws 99 21,%. 01 Paa*.-
antar s s.t, sp p pamms.u tL % sos.n pk o
Mm 01 Cp4mel-wH 6L M - C1***^=*"
y,,,,
- 2 4.am.4 ea&9-=M eJ AM sey-Class
- y 02 Qt.m.: when - neG.at fu n
U.#.',M.L~ T,. '1" Y.
E E
D Q.0, E e
re
~
M
- =--
M c5 wh Sp
~~
Definition / Explanation of important Parts of the Task '~.alysis 16 Instructional Setting The recommended setting that is both optimum and most feasible to be used in training the performance of the subject task.
17 JPM Method The recommended method and setting that are optimum and most feasible to be used in testing an operator's ability to perform the subject task.
1-10 23
EXAMPLES OF SKILLS AND KNO7 LEDGE COLLECTED THROUGH A COMPREHENSIVE TASK ANALYSIS (Sheet 3 cf 6)
M k.
y m am 7
, s r.. ur- $ u
,m n.. nae,-rasrwn~
=>m.-
.J
.=1 et vu.t de r*.a e ed --'" A" wze A
- d y y
usu ca.c w agew g,
$2 ht,4r jam mas W W.5d m quer.94.a4 dy 8 ****
M*w**
y emy w wwp.s 4
- J gm r.- = =.4
- r p p., w.a mm 8%
1M
- 1 Pg m a wa % eym m v= v gg c::L or an m e-ta w i., y p.,.
33%
ps unpyspm m ma.<s y,
%~y H
M m em aa. wocw wy p, s.aa.sw.
W*
d.um m m%
ee
t % f.
a.a w a w a.a 9 m -.=
y m3 m w4 w w.
y%
03 maeap e 4 soon My u
l 10 14 w w w +a ~ y /.s p mas w
.1 ems Definition / Explanation of Important Parts of the Task Analysis 19 Element / Action The action steps used by the job incumbent in completing the subject task.
20 Conditions Any required condition necessary for the performance of a particular element.
21 Standards Any measurable criterion that must be observed or achieved during the performance of a particular element.
1-1IL 24
EXAMPLES OF SKILLS AND KNOOLEDGE COLLECTED THROUGH A COMPREHENSIVE TASK ANALYSIS (Sheet 4 of 6) xL Ramrw A w mar %ru rw.
. maa ar
, m ~ r.,
m
.o ra s.-.-
u.-
== b.e ygE==y
==.
gy--.r -w r,
s ez r y
.t w M new am
- w..p g
toadd ewe fa.,m aeSA, deat)
Lenar.a. J m f a tss e.< ha eca.and.L
.a.t.a r
noeuan o
( m W a w.,y) s.a e s me p a.sa, s eaf.e u
ez 4 (s.nl 84 P.a M
meses es
/,o p
.>.ns m.csr sm=ct--.sso.h.
w
,,aa,,,,,
0 gan r aet *Ow w G m--- 4 AA - &(@,
.-m p...n
..r.- JwnmA A Anad.4 a %p ab.n.s om zeoo n.a*pL M Amed.no m s.s.t
- z so. A.~^ c "
Womes6ae he ald 44 44 f.05d0.sM Caet.s as a a ils saa v e, aem sames wooeme,o. Msa eta w(a.y eAa as as,
Aan a.
r
- .,y - % < s en p.m. A s e e3 toevermoo fws n ma 44 e sm,yy ah.n4s M 8-
- /*8e/
nassa.a...i.a p 9.w.a naarmm88 wa-d em p maatarea tadl005#d880
% M M" O enjgggaaa.'a%mW m--
enomear ?-N / <>===t **.t.,4.e4.fu 44 a.Aas (y a ma.
e4ses' aa, alma cast ae)
Definition / Explanation of Important Parts of the Task Analysis 24 Skills and Knowledge The particular statements of required knowledge in an academic, system, and/or generic equipment component sense necessary to support the operator's performance of each of the elements. These must be related directly to a particular step number.
1-12 25 l
EXAMPLES OF SKILLS AND KNOULEDGE COLLECTED THROUGH A COMPREHENSIVE TASK ANALYSIS (Sheet 5 cf O 1 =.L
,,,, wu w
~ ~ -
e.e.e as er 9.g.m mea Taf /h fa.M ## %# flhe.=.h7 Med l
t Sg y h,,",
m e
sense aus emomsees m
% y.,.-
maveratsLaser 4 A A - A W% p y av
= -.. w lad!A014Udp f M - # vee y 64 eas' y elessa net e.ttst amt be,lan.
redd2SJen@ bA
(.4? Jr. My 5%.
ramnn3assEffeer af
'4 i
fundes.,s.h 4 h 3% Wee,(jna=&4 I
- A.msat.a.e5 ms aatmsab.op an.M Aus.d)
/
l c' ^- '- '- *G c'a
- h.* s t-e nse ye a s aa 05 Nman9mn**
m te ans m
m <
- .s ss.
as.)
rg,,,,9,e rar a,e ranwaraan y a.,
- w. p,,,
a., e raeupmara..+
e.<,ar a.,*<c wc m
86 84
%. f f 48.*
- Gad.&os,/.hsq. net s#%eg W a.,
-a.,
g. m%
ET **.T *'-'
- *~--y 46 ros moop Amar 4* f=
-aans - %r=yjunas amis.
W 4'de fman. maata *ma( %mn dja.=f wr 4'e/aAn.imment unarnramp 1,,,a,. af,6 p aatss tent
- ypm.pe samesa.g ths y gaggm EDGdSQ* Aman t aa au g
. A na d %ja fjam.ss sy.
f@ (b vee fa n-.
Jana, en.4 p7 ramistraw %
- j.=>
f v 5.=r
.=t.a raanankarLast 4 *An jm.aea em44.M64a.l*Ges.4 % r*=v W A
- mim, 9
w(,.ea r<. fw a= 4-.<% r-v **w.=
e ramrmwo a M und.=/
roate::::."t -
14A p=a= =.haa.w% pay %+ =La etwottMP r.pa t.--. ~
.,al M.mla8*dp*.dowes*
W.dm
!-13 26
EXAMPLES OF SKILLS AND KNOOLEDGE COLLECTED THROUGH A COMPREHENSIVE TASK ANALYSIS (Sheet 6 of 6) 4= L MM 9~ i.=~ w n-
. na m ynge gg mavar Mants-ft M w thi -..msra rh S= L==
id=_a==
go roavao:anw4=p3==wyp
[pp 4 4.at== W /am** a.anf** es.(
l0/8/20tMl
/ WE F*** A W Q (% amme, p.e m) 50/Wb.ent e
'-'~vdas w a.Aag
- Wu M M WMn mont osaac
@ um ss.n.,as - %
P y atemos~=.a.kws(G %.no y)Geo.asapee.6 m kunsol0itl0 6 %.tdm a)
M*Wp'Pem.'(say heat es ** Wime As. em mRep 4.-
Amo w a may1ogia Gesa.W W am l Er b emdsnam sM. Cd pagl80llt M
raalra!000Ls.r 4 W a== 4 mennaf 14 sm.as.La n m
weemAMty
- a=dem /er *s ke o.me.
1$
n%Bu%--s.t*= au $*a +Cm en M 1-14 27
SECTION 2 --
SECTION 2 --
DEVELOPMENT OF LEARNING OBJECTIVES DEVELOPMENT OF LEARNING OBJECTIVES List of Information Obtained Obtain the Following:
Learning objectives associated with the tasks selected for this review.
Learning objectives associated with the tasks o
selected for this review.
Information linking these learning objectives to job performance requirements.
Information linking these learning objectives to job o
performance requirements.
Procedures / instructions with respect to maintaining learning objectives current.
Procedures / instructions with respect to maintain-o ing learning objectives current.
l i
Documentation with respect to a recent plant modification or procedural change Documentation with respect to a recent plant
)
o that should necessitate revision of training.
modification or procedural change that should necessitate revision of training.
2-IL 2-iR 29
REVIEO GUIDANCE REVIEO REQUIREMENTS 3
d$d 2.1 Are there learnine obiectives for each of the tasks selected for this evaluation?
<gy H# g A learning objective describes precisely what is to be learned in terms of the expected y$
g 2.1 Are there learnmg objectives for o uJ O trainee performance under specified conditions to accepted standards. These learning Z U U each of the tasks selected for this objectives are often referred to as terminal learning objectives, and are easy to evaluation?
I 2 3 identify because they are similar to the task statements. An example of a terminal learning objective for a mechanical technician job is:
COMMENTS:
"Given the disassembled parts of a service water pump, proper tools, and the approved procedure, reassemble the pump in accordance with the steps of the approved procedure."
Learning ejectives should provide the instructional designer with the information necessary to organize objectives, select instructional settings, and other related actions required to design and develop a training program. Here are two examples of good learning objectives:
"Given a handheld calculator, calculate the volume of a cylinder, without error" and "When directed by the shift supervisor and given the appropriate procedure, start a main feedwater pump in accordance with the appropriate procedure."
SCORING GUIDANCE For the three to six tasks selected for this review score this question as at l
l
@ if there are no learning objectives for these tasks.
@ if there are learning objectives for those tasks:
i o Selected for initial training,
~
o Related to emergency / abnormal operations.
@ if there are learning objectives for all tasks selected for this review.
l l
2-2L 2-2R 30
REVIEW REQUIREMENTS REVIEW GUIDANCE
-j $ ;;j
< i e 2.2.
Are learnana objectives derived from or related to the imowledges.
- ills, and abilities y :g g
@ y :EdO needed ior successful inb,=ib. -- +?
r 2.2 Are learning objectives derived m
(A These learning objectives are often referred to as enabi, a learning ob,iectives.,
from or related to the knowledges, i
review of the example of the skills and knowledge collected through a comprehensive skills, and abilities needed in task analysis, provided on the pages that follow Question 1.5 (pages 1-9L through 1-14L) successful job performance?
I 2 3 willillustrate the origin of enabling learning objectives.)
I
- COMMENTS:
l i
l l
l I
I.
4 f
SCORING GUIDANCE I
For the three to six tasks selected for the review, score this question as a:
1 j
@ if learning objectives are not derived from or related to the skills, knowledge, and abilities that enable the trainee to perform these tasks.
@ if learning objectives are derived from or related to the skills, knowledge, and j
abilities that enable the trainee to perform tasks:
o Selected for initial training, Related to emergency / abnormal operations o
(the only exceptions being any skills / knowledge that are assumed to be entry-level skills / knowledge).
@ if learning objectives are derived from or related to the skills, knowledge, and abilities that enable the trainee to perform all the tasks selected for this review (the only exceptions being any skills / knowledge that are assumed to be entry-level skills / knowledge).
2-3L 2-3R 31
REVIED GUIDANCE REVIED REOUIREMENTS 3
.a
> a 2.3 Does each learnuu oblective state the job performance behaviors (actions) expected of
- d d d
$ c"$
trainees upon compaction of trainuu?
t-<
w.
Listed below are examples of these behaviors (action verbs):
2.3 Does each learrung objective state odb zo u the job performance behaviors Knowledste Skill (actions) expected of tratnees upon Analyze Name Adjust Repair Apply Prove Assemble Replace COMMENTS:
Calculate Recall Calibrate Select Check (visual)
Recite Close Set Classify Relate Communicate Start Derive Solve Inspect S top Discriminate State Open Throttle Evaluate Tell Operate Trip Identify Verify Reassemble Vent List Write Remove Monitor Examples of these behaviors (actions) are underlined in the following learning objectives:
Given the disassembled parts of a service water pump, proper tools, and the approved procedures, reassemble the pump in accordance with the steps of the approved procedure.
(Continued on next page)
SCORING GUIDANCE For the learning objectives identified in Questions 2.1 and 2.2, score this question as a:
@ if one or more of these learning objectives are not based on associated job performance behaviors.
@ if behaviors are stated for all learning objectives, and if there is an auditable link between job performan~ce requirements and the learning objective action statements for a!! learning objectives.
@ if for all learning objectives the behavior (action statement) is the same as the associated task / skill / knowledge or the difference between the two action state-ments can be reconciled, d if non-specific action verbs (such as "know" and " understand") that are difficult to evaluate are avoided.
2-4L 2-4R 32
REVIEW GUIDANCE 2.3 (Cont'd)
Given temperature indications and their associated times, calculate the heatuo rate with an accuracy of within12 degrees per hour.
In general, the behaviors stated in the learning objectives should be the same as those stated in the related tasks / skills / knowledge; however, there are reasons why there may be differences between these behaviors. For example, plant safety, potential for equipment damage, or other considerations may preclude the actual performance of the task in training. Therefore, the learning objective action statement may be
" simulate tripping the auxilliary boiler" when the task action statement is " trip the auxilliary boiler."
,l i
f i
i 1
2-5L 33
REVIE7 GUIDANCE REVIEf7 REQUIREMENTS
.a 2.4 Does each learnma chiective state the job performance-based conditiords) under which ddU the trainee actsons will take place?
Q k
e-z 2 Examples of conditions are underlined in the following learning objectives:
2.4 Does each learmng objective state
@y8 the job performance-based condi-Given the disassembled parts of a service water pump, proper tools, and the tion (s) under which trainee actions approved procedure, reassemble the pump in accordance with the steps of the will take place?
! 2 3 approved procedure.
COMMENTS:
Given temperature indications and their associated times, calculate the heatup rate with an accuracy of within y,2 degrees per hour.
SCORING GUIDANCE Score this question as a:
@ if these learning objectives do g include condition statements, g if the condition statements do not meet the requirements for a score of h
@ if conditions are stated for all the objectives in Question 2.1 (terminal learning objectives) and i
at least 2/37the objectives in Question 2.2 (enabling learning objectives), d the conditions are derived from job periormance requirements.
@ if conditions are based upon an analysis of job performance requirements, aj the conditions are as specific as the examples above.
2-61.
2-6R 34
REVIEO REOUIREMENTS
~
-j 3g REVIEW GUIDANCE
-.a H
[ $ "$
9 2.5 Does each of the learnana abiectives state a specific lob performance-based standard y
for successful performance of the learnant obiective?
Examples of standards are underlined in the following learning objectives:
s a
based standard for successful per-Given the disassembled parts of a service, water pump, proper tools, and the f
M h IMg eMM i2 3 t
approved procedure, reassemble the pump m accordance with the steps of the approved procedure.
COMMENTS:
i Given temperature indications and their associated times, calculate the heatuo rate with an accuracy of within 12 degrees per hour.
a 1
1 l
SCORING GUIDANCE Score this question as a:
@ if these learning objectives do not include specific standards g o
the standards do not meet the requirements for a score of h
@ if specific standards are identified for all objectives in Question 2.1 (terminal learning objectives),Lnd at least 2/3 cf the objectives in Question 2.2 (enabling learning objectives), and the standards are derived from job performance requirements.
@ if specific standards are identified for a!! learning objectives, and these standards are based upon an analysis of associated job performance require-ments, and the standards are as specific as the examples above.
2-7L 2-7R 35
-- =
REVIEO REQUIREMENTS 3
REVIEO GUIDANCE
-j 3d
< a -
$U 2.6 Are there written procedures that require modafication of learning objectives when related iob performance requirements chanate?
fy$
2.6 Are there written procedures that
@j$
Obtain a recent plant modification package, prxedural change, or other document require modification of learning revision that should necessitate the revision of training, and the procedure for objectives when related job perfor-modification of training programs when job performance requirements change. Deter-mance requirements change?
I 2 3 mine whether the procedure has been followed and documented in this case.
I COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the procedure has not been fo!! owed or if no procedure exists.
@ if the procedure has been followed to the extent required to ensure that job performance requirements are generally current.
@ if the procedure has been properly implemented in a!! aspects.
2-8L 2-8R 36
Jll.
1l l
EE VH OT B(
2 A
Y )T N
NR N 7
2 O
OAE 3
N T
TMM I
I O
C AMU IT E
MUC C
S RSO E
R)
O D S
Oh FGS t
R Fn N NIH O
Et e I
I F
Rt E RT E
Os HOF e
R Cr TCO O
Sa S
e R E C
En EEA G
S Go F H L
At S
P A
R(
N T T T
E A
S O
V ROA T
A TTL
SECTION 3 --
SECTION 3 --
DESIGN / IMPLEMENTATION DESIGN / IMPLEMENTATION Obtain the Followmg List of Information Obtained o
- he list c ' three to six tasks selected for this review The list of three to six tasks selected for this (page 1-2R of this procedure),
review (page 1-2R),
Instructions / procedures that describe the training organizatforts goals, objectives, Instructions / procedures that describe the training o
plans, and relationships with other parts of the facility's organization, organization's goals, objectives, plans, and rela-tionships with other parts of the facility's organiz-1
- ation, o Evidence that these goals and objectives and the evaluation of the training Evidence that these goals and objectives and the organization against these goals and objectives have been updated / revised, evaluation of the training organization against these goals and objectives have been updated /
- revised, Instructions / procedures that define the responsibility and authority of training o
organization personnel, Instructions / procedures that define the responsibil-ity and authority of training organization person-
- nel, Roster /or6anization chart for the training organization, o
Rotter / organization chart for the training organi-
- zation, Procedures / instructions for the training, qualifications, evaluation, and certifica-o Procedures / instructions for the training, qualifica-tion of trainir.g personnel, h tions, evaluation, and certification of training Personnel, Instructions / procedures that describe the method (s) used for evaluating / selecting o
instructional methods and media, Instructions / procedures that describe the method (s) used for evaluating / selecting instructional methods and media, (Continued on next page)
(Continued on next page) 3-IL 3-1R 39 i
SECTION 3 --
SECTION 3..
DESIGN / IMPLEMENTATION (Cont'd DESIGN /IMPLEAELMTATION (Cont'd List of Information Obtamed (Cont'd Obtain the Following (Cont'dk Documentation with respect to the evaluation /
Documentation with respect to evaluation / selection of the instructional methods selection of the instructional methods and media a
and media for the three to six tasks selected for this inspection, for the three to six tasks selected for this review.
Instructions / procedures with respect to the organization / sequencing of learning Instructions / procedures with respect to the organi-zation/ sequencing of learning objectives ior initial o
objectives f or initial and continuing training, and continuing training, Evidence that the organization / sequencing of the learning objectives associated Evidence that the organization / sequencing of the with the three to six tasks selected for this review has been considered, learning objectives associated with the three to six o
tasks selected for this review has been considered, Lesson plans that include the learning objectives associated with the three to six Lesson plans that include the learning objectives associated with the three to six tasks selected for o
tasks selected for this review, this review, Procedure / instruction for evaluating the appropriateness of existing instructional Procedure / instruction for evaluating the appropri-ateness of existing instructional materials based o
materials based upon job performance requirements and learning objectives, upon job performance requirements and learning objectives, Examples of existing instructional rr.aterials that were evaluated using the pro-Examples of existing instructional materials that were evaluated using the procedure / instruction o
cedure/ instruction above Of existing instructional materials were selected to train the three to six tasks selected for this review, obtain this material),
above Qf existing instructional materials were selected to train the three to six tasks selected for this review, obtain this material.),
Training schedule for the time during which the review will be conducted, Training schedule for the time during which the o
review will be conducted, Procedures / instructions for maintaining training records, and Procedures / instructions for maintaining training o
records,and Procedures / instructions for the training, qualifications, evaluation, and certifica-Procedures / instructions for the training, qualifica-tions, evaluation, and certification of training o
tion of training personnel personnel 3-2R 3-2L 40
REVIEW REOUIREMENTS REVIEW GUIDANCE 3
a >-
.a YdN 3.1 Is there a written plan that clearly and specifically states the trainuu organizatiorfs
< w$ U M
M objectives and plans, and relationshaps with other parts of the facility's 9:
organization?
3.1 Is there a written plan that clearly o3o i
Z U U and specifically states the training organizatiorfs goals, objectives, and lP ans, and relationships with other parts of the facility's organization?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if there is no written plan, g if goals, objectives, and plans are stated only in general ways that do g permit objective evaluation of their success.
@ if there is a written plan that addresses, as a minimum, the training organizations' goals and plans and some objective evaluation of the achievement of these goals and plans is possible.
@ if there is a written plan that addresses the training organizatiorfs goals, objectives, and plans and relationships with other parts of the organization in clear i
and specific terms that permit an objective evaluation of their success and there is evidence that these goals, objectives, and plans have been updated, and revised, g that there is a procedure for doing so.
3-3L 3-3R 41
REVIE7 REQUIREMENTS REVIE7 GUIDANCE dud
""H ies and authority of trannum organization personnel clearly stated
<$U 3.2 Are the.s t
in writig E$d 8 z 2 3.2 Are the
' 'Hties and
@y8
. =, _
authority of training organization personnel clearly stated in writing?
1 2 3 COMMENTS:
i n
I i
SCORING GUIDANCE Score this question as a:
@ if there is no approved document defining responsibilities and authorities of training organization personnel
@ if there is a written document that defines the responsibilities and authorities of all management, supervisory and senior instructor level personnel in the training organization.
l
@ if there is an approved document that clearly and unambiguously defines the l
responsibilities and authorities of all training organization personnel on the roster / organization chart.
3-4L 3-4R 42
i l
REVIEW GUIDANCE REVIEW REQUIREMENTS 3
J
.a
.a 3.3 Are there documented eselsfication and trainina s -_ '..- r.ts for the trainana staff Y ~! N Id that address both senect matter and instructaenal skills and knowledre amorceriate for 6-specifac assumments?
hy$
3.3 Are there documented qualification O ua O
- U U Experience has shown that, in addition to a thorough understanding of the training and traanang i@;...c.ts for the content, successful execution of a training program depends on additional skills and traanang staff that address both ad>-
knowledge in the areas of:
ject matter and instructaonal skills and knowledge appropriate for o Training program development (represented by the training professional),
specific assignment?
I 2 3 4
o Instruction (represented by the qualified instructor), and COMMENTS:
o Management (represented by the management professional with a training background).
I In addition to the subject matter expertise element, training staff qualifications should address each of these three skill / knowledge areas also.
Those staff members who are responsible for analysis of job requirements and program j
design / development should be qualified by experience and training to l
o Analyze jobs and tasks, j
o Develop learning objectives, and o Select appropriate instructional methods, settings, and media.
i i
4 SCORING GUIDANCE i
j Score this question as a:
1 i
@ if no documented qualifications for the training staff exist or l
if the only documented qualifications for the training staff pertain to subject j
matter expertise.
l
@ if, in addition to subject matter expertise, documented training staff qualifications address two of the other three areas.
@ if, in addition to subject matter expertise, documented training staff qualifications address training program development, instruction and management.
1 l
3-3L 3-3R I
43 i
REVIEO GUIDANCE REVIEO REQUIREMENTS 3
$.D i$
3.4 Is there evidence that the appropriateness of instructional settinas has been
<J evaluated?
y y
>z lE Review the instructional settings for the learning objectives associated with the three 3.4 Is there evidence that the appro-
@y@
to six tasks selected for this review. (A job aid for selection of instructional settings priateness of instructaanal settings has been evaluated?
I 2 3 is provided on the next page.)
Instructional settings are the environments in which learning occurs. In order to COidMENTS:
provide the opportunity for mastery of the training, the job performance requirements must be considered in selecting the appropriate instructional setting. For example, it would not be appropriate to select a classroom as the instructional setting for a learning objective that requires a " hands-on" environment for mastery.
The instructional settings generally used by facilities are:
o Classroom instruction, o Laboratory and workshop instruction o Formal on-the-job training (OJT),
o Simulator instruction, and o Self-study.
There should be in place a systematic process by which appropriate choices of instructional settings are evaluated during the development phase of training. Such a process should also be used to determine if existing training uses appropriate settings wherever possible.
SCORING GUIDANCE Score this question as a:
@ if there is no method for evaluating appropriateness of instructional settings for the learning objectives associated with these tasks.
@ if there has been a selection of instructional settings for these learning objectives and E selections have been evaluated for consistency with the characteristics in the job aid for selecting instructional settings for all terminal learning objectives and at least 2/3 of the enabling learning objectives.
@ if the selection of instructional settings has been evaluated and is consistent with the job aid for all the learning objectives reviewed.
3-6L 3-6R
308 AID FOR SELECTING INSTRUCTIONAL SETTINGS INSTRUCTIONAL
.SOME REASONS SOME REASONS SETTING FOR USE NOT TO USE i
r Classroom o Large group of
- Hands-on environ-j trainees can be ment is required for scheduled at the same mastery
)
time o Large amount of information is to be 4
presented I
j Laboratory and Workshop o Hands-on environment o Necessary conditions Instruction required and necessary cannot be 1
conditions can be provided/ simulated in i
provided in the the laboratory j
laboratory Formal 03T o Hands-on environment o Necessary conditions required and necessary cannot be conditionTequipment provided/ simulated in operation can be pro-the plant (e.g.,
vided in the plant emergency j
indications) o Sufficient qualified i
personnel are avail-o High fidelity of able to conduct the training is required l
03T and plant status / safety does nat permit manipulation of the j
equipment i
Simulator Instruction o High fidelity to the o Necessary conditions job task is required can be provided in a j
during training for the less costly _ setting l
skills and knowledge to be mastered by the t
j trainee
)
Self-Study o All conditions can be o Close supervision I
contained in the train-required to prevent J
ing material (or made injury or damage to available in the plant) the plant equipment o Task is identified as 1
difficult to i
learn / perform i
i 45 i
E*3 REVIEO GUIDANCE a 3 a a iaYU
[ $
E' 3.5 is the organization and w L i of the initial trainina pronram(s) based upon the relationships amore learnant abiectives?
H z 2 3.5 Is the organization and 4
@yO Learning objectives may be dependent on, independent of, or complementary to one
- I*I #
N another. Where there is a dependency of one objectave on another, the dependent objective should be taught in a sequence af ter the objective upon which it is content based upon the relationships dependent. Complementary objectives should be grouped together to allow the among learnmg objectives?
I 2 3 instructor to use these complementary relationships. An example of a dependent COMMENTS:
relationship would be that recognition of abnormal operation of a pump first requires knowledge of the indicators of normal pump operation An example of a complemen-M relationship would be " assembly of a centrifugal pump" and " disassembly of a centrifugal pump."
l i
SCORING GUIDANCE Secre this question as a:
@ if there has been no formal review of the sequencing of training based upon relationships among learning objectives.
@ if indications are that at least a general review of the sequencing of training has been conducted, ag the sequencing of the learning objectives associated with the three to six tasks selected for this inspection is appropriate, based upon your knowladge of any dependent or complementary relationships that exist among these objectives.
@ if the requirements for a score of @ are met, and a detailed, structured review of the sequencing of training has been conducted and documented.
3-8L 3-8R 46
REVIEO REQUIREMENTS 3
REVIEW GUIDANCE
"$ U Gi 3.6 Is the arrWation and sequencuw of continuing trauung based upon the relationships E ld U among tearninst objectives?
5' s z 2 3.6 is the organization and sequencing gy0 of continuing trasning based upon the relationships among learniaq objectives?
I 2 3 COMMENTS:
i I
SCORING CUIDANCE Score this question as a:
@ if there has been no formal review of the sequencing of training based upon relationships among learning objectives.
@ if indications are that at least a general review of the sequencing of training has been conducted, and the sequencing oTihe learning objectives associated with the continuing training task (s) selected for this review (see page 1-2R) is appropriate, based upon your knowledge of any dependent or complementary relationships that exist among these objectives.
@ if the requirements for a score of @ are met, aj a detailed, structured review of the sequencing of training has been conducted and documented.
3-9L 3-9R 47
REVIE7 REQUIREMENTS 5
REVIE7 GUIDANCE
- i -i W 3J Are lesson plans avadable that provide for consistent trainina delivery?
u.t l
g< $
E' Select two or more lesson plans that include learning objectives associated with the a
tasks selected for this review.
z o u 3y provide for consistent traanang A lesson plan is a structured outline that ensures consistency in instructor presentation MW I 2 3
)
l and evaluation of trainee performance. Regardless of the instructional setting (classroom, laboratory,03T, simulator, or self-study), lesson plans should include:
COMMENTS:
o Learning objectives, o Appropriate instructor teaching activities, o Instructor and trainee references, i
o Appropriate evaluation methods / standards, and I
o Required materials (audiovisual aids, test equipment, etc.).
i For formal 03T, lesson plans frequently are a series of 03T guides. These guides provide guidance end structure during training in the plant. They can also be used as 1
qualification guides or check!1sts when evaluating a trainee's proficiency in performing a task.
{
For simulator training, the lesson plan may be identified as an exercise guide and should include:
l o Indications / symptoms to be recognized, o Correct responses to the exercise, j
o Standards for objectively evaluating trainee performance, and i
o Feedback to be provided to the trainee.
i SCORING GUIDANCE Score this question as a:
@ if lesson plans are not developed, @or if the requirements for a score of are ny met.
l
@ if all the lesson plans reviewed contain the information described above.
@ if all the lesson plans reviewed contain the information described above, aj the level of detail of the information a!!ows specific reference to instructional activities of the trainee and instructor for the objectives listed.
l 3-10L 3-10R 48
3 REVIEW GUIDANCE a >.
- i -i N g uj 3.1 is there evidence that the appraoriateness of existire instructional materials has been g<
evaluated based upon identified trainee needs and learnum objectives?
w n.
> z 2 3.8 Is there evidence that the appro-o w o Z U U There are many cases where existing instructional materials may be used (either "as priate of existing insMonal is" or with modification) for implementing a performance-based tra ning program.
matalais has W evaluated M Among these cases are:
upon identified trainee needs and learring objectives?
I 2 3 o A training program has been in place for some time before a systematte analysis of job performance requirements is performed (i.e., a licensed operator COMMENTS:
training program). Because the licensee expects that much of the existing training material for this program will be suitable for a performance-based training program, the licensee chooses to revise these materials (as required) rather than develop new materials.
o A training program is being developed for a job that is "similar" to another job where performance-based training materials (based upon the other job) are available.
1 The use of these existing materials (or their modification for use) in a performance-based training program is entirely appropriate if the decision to use these materials is based upon job performance requirements (e.g., the learning objectives) and trainee entry-level skills and knowledge. When such an evaluation is n5Lt conducted, subjective decisions will be made, with a bias toward accepting materials that do not meet a!! job needs.
(Continued on next page) l.
SCORING GUIDANCE Score this question as a:
@ if there is no evidence indicating that suc., a review of existing training materials was conducted.
@ if tnere is evidence indicating that existing training materials were evaluated objectively at least once based upon the three criteria on the next page. (!! any of the learning objectives associated with the tasks selected for this review are trained through the use of existing materials, these materials should meet the three criteria on the next page.)
@ if there is evidence indicating that existing training materials are regularly and i
objectively evaluated based upon tne three criteria on the next page.
3-IIL 3-11R 49
,d*
REVIEO GUIDANCE 3.8 (Cont'd) 11 existing training materials are used, there should be evidente available that ir.dicates a review of these materials was conducted that included the following criteria:
I o Is *he material content and reading level consistent with expected entry-level
.s.
skills and knowledge?
o Does the material adequately ad. vest.he learning ot,jectives?
- ~^
o is the material consistent with other components of the training program?
]
P
.i 1
4 i
I I
i i
i j
3-12L 50
REVIEW GUIDANCE REVIEW REQUIREMENTS dU 3.9 is traininst beina conductal in an h=te manner as determined through application of the traininst observation chedclists?
[YU a
-J h y: $
Given the training schedule, select a particular training evolution for observation.
3.9 Is waining W comiuctai in an ow o Make the selection based on these criteria:
Z U V W ate manner as determined o Has a particular instructional setting (classroom, laboratory,03T, simulator) through appiscation of the trasnsng been identified as regt. iring improvement (through previous parts of this observation checklists?
1 2 3 checklist or previous inspections)?
COMMENTS:
o Are any of the training evolutions addressing a content area that has been identified as requiring improvemer.t?
o is the duration of the evclution consistent with the time available for cbservation?
Training Observation Checklists for each instruction.a! setting have been provided as separate appendices to these inspection procedures as fo!!ows:
Appendix Training Observation Checklist For A
Classroom Setting B
Laboratory Setting C
03T Setting D
Simulator Setting Complete the appropriate observation checklist provided in the appropriate Appendix (based upon instructional settingl.
SCORING GUIDANCE Using the results of the training observation, score this enestion as at
@ if a score of @ is received in any major area (as defined on the observation checklist) or
~
if the average score for all areas is less than 2.0.
@ if a score of @ or above is received in all major areas and the average score is 2.0 or greater.
@ if a score of @ or greater is received in all areas.and the instructor's overall i
performance is particularly exemplary.
l 3-13L 3-13R 51
=
3 REVIE"? GUIDANCE REVIED REQUIREAAENTS a >
.a
.a w
.a j "j 3.10 Are adequate methods established for nuintaining training records?
a.
w The following are the materials that should be included in a training records system:
3.10 Are adequate methods established Z 0 0 Program Records for maintaanang training records?
I 2 3 o Training Plans, COMMENTS:
o Training group procedures, o Training materials, o Test / examination banks, o Course schedules and trainee participatior, o Results of trainee examinations and course evaluations, o Analysis results (e.g., job and task analysis), and o Data gathered to assess program effectiveness.
Trainee Records o Records of the training and qualifications of facility employees and o Records of the training of contractors and visitors.
The program re:ords have been evaluated through individual questions in this and other sections. There fore, this question focuses on trainee records.
To evaluate trainee records, select the name of an individual who has teen qualified to perform the job in question. Ask for the training records of this individual 1
i SCORING CtJIDANCE s
Score this question as a:
@ if there are no records of this individual's training and qualifications or s
if the requirements for a score of @ are g met.
s N
@ if the trairling records for this individual indicate completion of all training and qualifications required for the job.
@ if the requirements for a score of @l training records are used to plan individual are met, d there are indications that individua i
training.
1 I
l l
3-14L 3 14R 52
mb.A a
5 L
< w _ _ _.
h-w h
TOTAL SCORE FOR SECTION 3 AVERAGE SCORE FOR SECTION J (to nearest tenth)
TRANSFER THE INFORMATION ABOVE TO THE SCORING
SUMMARY
(THE LAST PAGE OF THIS DOCUMENT).
i 53
SECTION 4 --
SECTION 4 --
TRAINEE EVALUATION TRAINEE EVALUATION Obtain the Fo!!owing:e List of Information Obtaaned o The list of three to six tasks selected for this inspection The list of three to six tasks selected for this (page 1-2R of this procedure),
review (page 1-2R),
o Procedures / instructions concerning the selection of candidates for training and the Procedures / instructions concerning the selection of granting of exemptions from training, candidates for training and the granting of exemp-tions from training, o Records concerning trainee selection and exemptions granted from training, Records concerning trainee s' election and exemp-tions granted from training, o Examples of tests and test results for the most recent course for either initial or Examples of tests and test results for the most continuing training, recent course for either initial or continuing
- training, o Records of training for facility personnel, and Records of training for facility personnel, and Procedures / instructions concerrting precautions to preclude compromise of test Procedures / instructions concerning precautions to o
content.
preclude compromise of test content.
Information collected to answer questions in this section may also be usefulin answering some questions in Section 5.
4-1L 4-!R 55
REVIEN GUIDANCE REVIED REOUIREMENTS 3
.a 4.1 Are exemptions from training based upon performance-based testinst or other obiec-M "j $
tive evaluation methodL*J1 Q>z 2
4.1 Are exemptions from traarung based
@y0 upon performss.chd testing or oder objective evaluation methods only?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if exemptions from training are permitted without any record of these exemptions.
@ if no exemptions from training are permitted, or if any exemptions permitted are based upon performance-based testing or other objective and documented evalua-tion methods.
@ if the requirements for a score of @ are met and only performance-based testing is used for granting exemptions.
4-21.
4-2R 56
REVIED REQUIREMENTS 3
REVIEf7 GUIDANCE
$$U
< uJ 4.2 is trainee evaluation appropriate to iob performance requirements and trainira p dd objectives?
- z 2 UO 4.2
!s We whtim w@te to z
For the three to six tasks selected for this review and their associated learning M #wrnance ts W objectises, identify the test items / methods used to evaluate trainee performance.
training objectives?
! 2 3 For task level learning objectives (i.e., terminal learning objectives or TLO's), the COMMENTS:
actual behavior called for in the task statement and TLO should also be required for the performance test (i.e., the test should assess the task as it is stated).
SCORING GUIDANCE Score this question as a:
@ if there is no consistent link between the job performance requirements / learning objectives and the test items.
@ if the test items for all tasks evaluate the task /TLO as stated, ad test items for at least 2/3 of the tasks are consistent with the action statement, condition, and standards of the objectives.
@ if the test items for all tasks are consistent with the objective and job performance requirements.
4-3L 4 3R 57
REVIEO REOUIREMENTS 3
REVIEC GUIDANCE
.a
.a
- $ d N
[y$
4.3 Is trainee performance evaluated reaularly durina the traininst pronram, and prompt.
> gg objective feeAmrk provided on a reaular basis?
4.3 Is trairw.e performance evaluated O ua O V
Learning is greatly enhanced when trainees are provided feedback with respect to regularly during the training pro-their performance. The more immediate and continuous the feedback the better. This gram, W prompt, objective fe feedback can h provided by self-checks that are scored by the trainee, as in back provided on a regular basis?
I 2 3 self-study workbooks, or by instructor-administered and scored tests. If instructor-administered tests are used, there should be provisions for promptly providing the COMMENTS:
results to trainees (i.e., ideally within a day, but no more than a week).
SCORING GUIDANCE Score this question as a:
@ if there is no evidence of regular, prompt, and objective evaluation / feedback of trainee performance, or if the requirements for a score of @ are g met.
@ if, within a week af ter an evaluation, trainees are provided feedback concerning performance that not only indicates whether the overall performance was satisfactory, but also provides at least general feedback concerning specific performances as related to job performance requirements.
@ if a!! elements of @ are present and prompt feedback is given to trainees related to specific skills /knowledges that do g meet job performance requirements (i.e.,
learning objective standards).
4-4L 4-4R 58
REVIEW GUIDANCE REVIET REQUIREMENTS 3
.a
.a 4.4 Are tcainees who perform below minimum standards provided remedial trainirut.
Y -j $
y "j reteste 1, and removed from the training program if minimum standards are not met?
9<
o.
w Ask for specific examples of the remediation process and procedures for accomplishing 4.4 Are trainees who perform below obb
- V U
required additional training, testing, and removal from the training program when minirr.um standards provided minimum standards are not met. The minimum standards used as the basis for remedial training, retested, and requiring remediation should be based upon job performance requirements.
removed from the training program if minimum standards are not met?
! 2 3 COMMENTS:
SCORING GUIDANCE Score this question as as
@ if minimum standards are g established and implemented as part of the program.
@ if a remedial training and retesting program are routinely implemented in cases where trainee performance is below minimum standards, arjd trainees are removed from the training program if minimum standards are not met af ter remedial training and testing.
@ if, in addition to the requirements for a score of @ special retraining techniques are used to correct the difficulties individuals experienced in the original training, and Illnformation concerning these difficulties is used as feedback into the training program.
4-5L 4 5R 59
REVIED GUIDANCE REVIED REQUIREMENTS 3
.a
.a 405 Are job incumbents who perform below minimum standards during requalification or ddM cont 2nuing trainiru removed from anariated job duties ind provided remedial
>j$
- yg trainirw?
g 4.5 Are job incumbents who perform o ai o z o o Ask for the test records for the two most recently completed requalification or below minimum standards during continuing training programs.
requalification or continumg trairw ing removed from =< =clated job duties and provided remedial Review these test records and remedial training documents.
training?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if one or more individuals in the program above have completed the training program without meeting minimum standards and have been neither removed from associated job duties nor provided remediation.
@ if, for the programs reviewed, all individuals met the minimum standards or were removed from the training program and provided remedial training.
@ if the requirements for a score of @ are met aj the root causes of failures are identified and used to improve the training processes (as appropriate).
l l
4..ol 4-6R 60
l l
REVIE7 REQUIREMENTS 3
REVIEW GUIDANCE d U ~j
< ;;! U 4.6 Are acoropriate prev-mutions taken to preclude compromise of test contents?
$d H z 2 Among the measures that should be implemented are:
4.6 Are appropriate precautions taken
@j@
to preclude compromise of test con-1.
A sufficient number of questions in a test item bank to permit variations in tents?
! 2 3 examinations from one administration of an examination to the next, COMMENTS:
2.
A single administration of a particular examination, rather than multiple administrations of the same examination, and 3.
Regular proctoring of tests.
SCORING GUIDANCE Score this question as a:
@ if the requirements for a score of @ are not met.
@ if measures 1, and 3 above are implemented.
@ if all of the measures above are implemented in an apparently effective manner.
4-71.
4-7R 61 i
TOTAL SCORE FOR SECTION 4 AVERAGE SCORE FOR SECTION 4 (to nearest tenth)
TRANSFER THE INFORMATION ABOVE TO THE SCORING
SUMMARY
(THE l
LAST PAGE OF THIS DOCUMENT).
l l
l 62
\\
SECTION 5 --
SECTION 5 -
PROGRAM EVALUATION PROGRAM EVALUATION Obtain the Following:
List of Inform _ation Obtained o Procedures / instructions for evaluation of training programs, Procedures / instructions for evaluation of training
- programs, o The most recent:
The most recent:
- Evaluation of trainee test performance, h Evaluation of aggregate trainee performance,
- Instructor critiques, Instructor critiques,
- Trainee critiques, Trainee critiques,
- Graduate trainee on-the-job experience critiques, and Graduate trainee on-the-job experience critiques, and
- Supervisor feedback, Supervisor feedback, o Results of internal and external audits for the last two years, and Results of internal and external audits for the last two years, and Results of evaluations of personnel performance problems for the last two years.
Results of evaluations of personnel performance o
problems for the last two years.
l 5-IL 5-IR 63
REVIEO REQUIREMENTS y
REVIED GUIDANCE dud 5.1 is there evidence that a method is in place to systematically evaluate the effective-
- i! U ness of traininst programs and to revise the programs as required?
< g ;l;8 w z E 5.1 is there evidence that a method is in
@y8 1his item addresses only whether an evaluation program exists. Subsequent questions lP ace to systematically evaluate the deal with the specifics of the evaluation program.
effectiveness of training programs and to revise the programs as required?
1 2 3 COMMENTS:
l l
SCORING GUIDANCE Score this question as a:
@ if no evidence of an evaluation program exists.
@ if an approved evaluation program is in place.
@ if the requirements for a score of @ are met arLd there is evidence that the program is periodically evaluated and revised (if required).
f 5-2L 5-2R 64 1
REVIEW GUIDANCE REVIEW REQUIREMENTS
"$ $ E!
.4 e-5.2 Are examination and operating test results evaluated so that tests are improved and f$
feedback is provided to improve training?
<> z 2 Just as individual trainee test performance is indicative of the strengths and 5.2 Are examination and operating test
@j@
weaknesses of individual trainees, review of the aggregate test performance of results evaluated so that tests are trainees provides insights into the strengths and weaknesses of the training program.
Improved and feceack is provided While the identification of areas where trainees had difficulty in meeting program to improve training? ~
1 2 3 standards is straightforward, the determination of the cause of these difficulties may not be. There is a tendency.o focus on a solution of "more training"; however, the COMMENTS:
analysis should also consider:
1.
The adequacy of the procedures or other job performance aids provided, 2.
The adequacy of selection criteria, 3.
The sequencing of training (is prerequisite training being provided?), and 4.
The qualifications and performance of the instructor.
Review the results of the evaluation of test results for the most recently completed course and the procedure / instruction dealing with this evaluation to score this question.
SCORING GUIDANCE Score this question as a:
@ if a review of aggregate trainee test scores is neither required nor conducted or if procedures / instructions call for such a review, but the review has not been conducted within the specified time frame (or within six months af ter the program is completed if no time frame is specified).
@ if a review of aggregate trainee test scores has been conducted aM the review evaluates the causes of any program weaknesses identified.
@ if the requirements for a score of @ are met and the evaluation considers at least three (3) of four (4) items listed above, ar)d such evaluations are regularly and routinely conducted.
5-3L 5-3R 65
REVIEC GUIDANCE REVIED REOUIREMENTS 3
$ D i$
5.3 Are instructor critiques of training used for program evaluation?
<gy
- g
.4 Instructors are in a unique position to identify problems with the technical accuracy,
[y$
completeness, sequencing, and trainee difficulties associated with the training 3.3 Are instructor critiques of the o ua o z o u materials. These problems should be identified as they occur through a diary, log, or training used for program evalua-similar means. Recommended changes should be reviewed in a timely manner and tion?
! 2 3 implemented as appropriate.
COMMENTS:
l t
SCORING GUIDANCE Score this question as a:
@ if there is no requirement for instructor critiques of training to be used for program evaluation.
@ if there are both a requirement for instructor critiques of training a_nd indications n
that these critiques are used for program evaluation.
@ if the requirements for a score of @ are met and I
there are indications that the critiques are an ongoing part of course conduct with a I
formal method for timely implementation of appropriate changes.
5-4L 5-4R 66
REVIEW GUIDANCE REVIEW REQUIREMENTS a >
ld d N j $
5.4 Are trainee critiques of traininst used for prostram evaluation?
s<
su 9 Information provided by trainees upon completion of major segments of training should 53$
focus on course effectiveness and ways to improve training. To be most effective, the 5.4 Are trainee critiques of training Z 0 U critiques should:
used for program evaluation?
I 2 3 o Be in written form and COMMENTS:
o Be structured to focus trainee comments on the desired information.
t SCORING GUIDANCE Score this question as a:
@ if there is no requirement for trainee critiques of training.
@ if there is a requirement for trainee critiques of training and indications that these critiques are being regularly solicited and used for program evaluation.
@ if the requirements for a score of @ are met d the critiques are structured so as to focus trainee comments on desired informa-1 tion.
5-5L 5-5R 67
REMO RENNEW REVIEO GUIDANCE
$$d
[ $
E' 5.5 Are on-the-iob miences solicited from lob incumbents and used for program evaluation?
hh n
ex sola Soliciting information from program graduates three to six months af ter,they g
complete training can aid in identifying strengths and weaknesses of initial training 9,
g programs. Information from more experienced job incumbents should also be collected to identify needs for continuing (refresher) training. The following types of informa-COMMENTS-*
tion should be collected:
1.
Unexpected difficulties in performing tasks on the job, 2.
Tasks that are particularly difficult or easy to perform, 3.
Differences between the way tasks are performed on the job and the way they are taught, 4.
Additional training needed to do the job, and l
5.
Kinds of errors committed on the job.
SCORING GUIDANCE Score this question as a:
@ if there is no formal program for soliciting information from job incumbents for program evaluation or if procedures /instru7tions call for co!!ecting and using this information, but the solicitation has not been conducted.
@ if information with respect to at least three (3) of the five (3) areas above has been solicited from job incumbents and, used for program evaluation.
@ if the requirements for a score of @ have been met and the types of information listed above are collected and used for evaluation of both initial and continuing training.
5-6L 5-6R 68 l
l l
REVIEW GL'IDANCE REVIEW REQUIREMENTS 3
-s
.2 5.6 Is feceack from supervisors about job performance problems solicited for program
( -j $
yj nMatim?
s<
ua 9:
Information should be solicited periodically (annually is suggested) to determine how
- V U
we!! the initial training program is preparing individuals to perform their jobs and what 5.6 Is feceack from supervisors about continuing training is needed for current job incumbents. The following types of job performance problems solici^.ed information should be collected:
for program evaluation?
I 2 3 1.
Tasks for which new job incumbents are inadequately prepared, COMMENTS:
2.
Kinds of errors being committed by job incumbents, 3.
Additional training received by new job incumbents once they are on the job, 4.
Suggestions for improvements in initial and continuing training programs, and 5.
Expected changes in job assignments, procedures, or equipments.
SCORING GUIDANCE Score this question as a:
h if there is no formal program for soliciting feedback from supervisors for program evaluation or if procedures / instructions call for collecting and using this information, but the solicitation has g been conducted.
@ if at least three (3) of the five (5) types of information above are collected a_nd used for program evaluation.
@ if the requirements for a score of @ are met and the types of information listed above are collected and used for evaluation of both initial and continuing training.
3-7L 5-7R 69
REVIED REOUIREMENTS REVIEW GUIDANCE
.a > U j
5.7 Are both internal and external
- trasnana proaram audit / evaluation findinas used for Y -j $
y uj 9
orogram evaluation?
o.
w 5.7 Are both internal and external
- In reviewing training program audit / evaluation findings for the last two years, there should be a resolution for each finding that either adequately rebuts the finding or training program audit / evaluation Z U U makes changes in the program in response to the finding. Review these rebuttals and findings used for program evaluation?
I 2 3 changes to identify whether they are appropriate responses to the findings.
COMMENTS:
l l
l External to the training department; not necessarily external to the utility itself.
SCORING GUIDANCE Score this question as a:
@ if audit / evaluation findings are not used for program evaluation.
@ if (in your opinion) audit / evaluation findings are, in general, appropriately used for program evaluation.
@ if all audit / evaluation findings are appropriately used for program evaluation.
External to the training department; not necessarily external to the utility itself.
5-8L 5 8R 70 1
1
REVIED GUIDANCE REVIED REOUIREMENTS 3
dud 5.8 Is the performance of each member of the trainirm staff objectively evaluated on a
<a H
renular hasis?
o.
w 5.8 Is the performance of each member oEb
- O U of the training staff objectively evaluated on a regular basis?
I 2 3 COMMENTS:
)
i SCORING GUIDANCE Score this question as a:
@ if no provision is made for evaluating training staff member performance against the job requirements on a periodic basis.
@ if procedures exist for the periodic (at least once every 12 months) evaluation of training staff member performance against job requirements and there is evidence that the procedures are, in general, being followed.
@ if procedures exist g there is evidence that they are being followed in a consistent, accountable manner.
5 5 9R 71 1
I l
i TOTAL SCORE FOR SECTION 5 AVERAGE SCORE FOR SECTION 5 (to nearest tenth) -
TRANSFER THE ?NFORMATION ABOVE TO THE SCORING
SUMMARY
(THE LAST PAGE OF THIS DOCUMENT).
t i
72 l
l
1 TRAINING OBSERVATION i
CHECKLISTS i
4 l
1
APPENDIX A
SUMMARY
SCORING SHEET FOR TRAINING OBSERVATION CHECKLIST "A" TRAINING OBSERVATION CHECKLIST (CLASSROOM SESSION)
(CLASSROOM SESSION)
If a classroom session is selected for observation, this checklist is to be used in support of DATE-Question 3.9, which states, INSTRUCTOR-
"Is training being conducted in an adequate manner?"
LESSON TITLE-PLANNED LENGTH-SCORING
SUMMARY
SCORE CHECKLIST INSTRUCTIONS ASSIGNED Score each question of this checklist after observing the conduct of the entire session.
Enter the score for each question in the appropriate space under SCORING
SUMMARY
on A.1 Timing and Pace 1
2 3
page A-1R.
A.2 Introduction and Lesson Context 1
2 3
A.3 Structure and Content 1
2 3
A.4 Presentation and Poise 1
2 3
A.5 Control and Flow I
2 3
A.6 Demonstrated Technical Competence 1
2 3
A.7 Completeness and Accuracy 1
2 3
A.8 Lesson Summary and Lab Prep 1 2
3 TOTAL SCORE AVERAGE SCORE FOR THIS OBSERVATION (to the nearest tenth)
A-!L A-IR 73
REVIEO REOUIREMENTS TIMING AND PACE d$d A.1 Did the instructor present the lesson in a well-paced, coherent fashion within the
[ ld U allotted time?
<$d n z 1 A.! Did the instructor present the lesson
@y@
in a well-paced, coherent fashion within the allotted time?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the lesson fell well outside planned duration; the instructor exhibited no time sense during presentation with negative effect on teaching and trainee acceptance of material.
@ if the lesson fell outside the expected duration by 15-20%; some allowance for questions was provided.
@ if the lesson was presented within the a!!otted time with ample, but not excessive, time for questions; the instructor was aware of time throughout lesson; if lesson was in excess of one hour, a planned, controlled break was provided.
A-2L A-2R 74
REVIEW REQUIREMENTS 3
INTRODUCTION AND LESSON CONTEXT d$d A.2 Did the instructor inform the class of the lesson objectives. context and structure?
<py Qe ::3 u2 :::
A.2 Did the instructor inform the class E$$
of the lesson objectives, context, z o u and structure?
I 2 3 COMMENTS:
l SCORING GUIDANCE Score this question as a:
@ if the instructor failed to provide an adequate level of lesson introduction or miscommunicated the overall intent, st-ucture, or context of the lesson with a negative ef fect on the ensuing instruction.
@ if the instructor outlined the lesson in general or generic terms, provided some direction for the ensuing instruction; the course context of the lesson was adequately explained as it related to preceding or following lessons.
@ if the instructor provided the trainee with a meaningful ar.d complete instructional context for the lesson with reference to the overall course objectives; the lesson was outlined prior to the presentation and the structure was explained in detail.
A-3L A-3R 75 l
REVIED REQUIREMENTS STRUCTURE AND CONTENT
.a
>a A3 Did the instructor adiere to the content and structure of the instructor side?
dd#
$' 3 H<
o.
w6h A.3 Did the instructor a&ere to the o
content and structure of the Z U U instructor guide?
I 2 3 COMMENTS:
i l
SCORING GUIDANCE Score this question as a:
@ if the instructor failed to utilize the instructor guide and, as a result, major errors of omission occurred within the presentation; data required for en:uing training was not covered as required.
@ if the instructor utilized the outline of instruction with minor departures into other subject areas; the subject areas of the lesson suffered from only minor deviations in the level and content of the presentation by the instructor.
@ if the instructor kept to the outline of instruction, departing only to satisfy questions or to maintain class interest / attention; all subject areas were covered at a level commensurate with the data presented in the instructor guide and with the objectives of the lesson.
A-4L A-4R 76
PRESENTATION AND POISE REVIEW REOUIREMENTS
$ D i$
A.4 Did the instructor present the lesson in an interesting and attention-metting fashion?
<gy N$d w z :E A.4 Did the instructor present the lesson
@y@
in an interesting and attention-getting fashion?
I 2 3 COMMENTS:
SCORING GUIDANCE Secre this questian as a:
@ if the instructor was confused and nervous with a large negative impact on effectiveness or the instructor was unable to relate to the knowledge level of the trainees.
@ if the instructor, in general, conveyed a feeling of certainty and presented the lesson in an effective fashion; the instructor monitored and reacted to the attention level of the class with only minor effects on the overall presentation.
@ if the instructor was poised and professional and kept the interest of the class throughout the session; the instructor gained and maintained the confidence snd trust of the class.
A-5L A-5R 77
CONTROL AND FLOW REVIEO REOUIREMENTS d3d A.5 Did the instructor maintain control of the class and the instructional flow?
<yy N,$$
>z lE A.5 Did the inst 4uctor maintain control O uJ O of the class and the instructional flow?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
I i
@ if control was never gained by the instructor and the session deteriorated, or tangential discussions were pursued resulting in substantial detriment of the subject matter.
i t
@ if the lesson was af fected in only a minor fashion by loss of controf g the end goals of the lesson were attained.
l
@ if the instruccor was in control of the class throughout its conduct and did not allow tangential discussions to disrupt the pursuit of the subject at hand; the respect of the class for the instructor allowed the successful completion of the lesson without undue distraction.
A-6L A-6R 78
\\
.y DEMONSTRATED TECHNICAL COMPETENCE REVIEW REQUIREMENTS
"$ $ id A.6 Did the instrear exhibit a level of tecfmical knowledae adeemte _to satisfy the
<gg requirements of the subject matter?
Q i-z e y@
A.6 Did the instructor exhibit a level of O
technical knowledge adequate to satisfy the requirements of the sub-ject matter?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the instructor did not seem to possess a technical level of knowledge which would allow successful completion of the lesson.
@ if the instructor's grasp of the subject matter was adequate to address the level of detail in the instructor guide (lesson plan); the instructor was able to interpret his detailed knowledge to an understandable level; questions were answered in a completion fashion or with only minor / recoverable errors.
@ if the instructor exhibited a complete and thorough grasp of the subject matter and was confident enough in this knowledge to be able to relate it in an understandable fashion; questions within the scope of the course were answered correctly and completely; questions outside the scope of the lesson or course were referred to offline discussion.
A-7L A-7R 79
COMPLETENESS AND ACCURACY REVIED REQUIREMENTS 3
.a
.a A.7 Was the lesson, as presented, technically complete and correct and was the -
p -j $
y "j information understandable?
k<
n.
u,6$
A.7 Was the lesson, as presented, tech ~
oZ 0 V nically complete and cor.ect and was the information understand-able?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if technical errors by the instructor were of such a severe and widespread nature that trainee confidence and instructor credibility could not be maintained; the instructor f ailed to bridge the gap between the instructor guide (lesson plan) and the reference materials; lack of preparation was obvious.
@ if there were a few technical errors in the presentatior, and they were of a : minor nature and had little impact on the accomplishment of the objectives; the instructor presented an understandable picture of the function or system.
@ if there were no technical errors in the presentation an,d the instructor presented a complete picture of the function / system including relating the lesson to the everall performance of the function / system.
A-8L A-8R 80
= _ _ _ _ _ = _ _
=n
LESSON
SUMMARY
AND LAB PREP REVIEW REOUIREMENTS 3
dud A.S Did the instructor adequately summarize the class and leave the trainees with a clear
<a s
picture of retriainiru requirements for satisf actory objective completion?
y 9 z 2 A.S Did the instructor adequately
@yO summarize the rima and leave the trainees with a clear picture of remaining requirements for satis-factory completion of the objectives?
! 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the instructor did not summarize the lesson or provide adequate guidance.
@ if the instructor provided a brief summary of the lesson that, in general, communicated the requirements for successful lasson completion.
@ if the instructor presented a complete and thorough summary of the subject matter and provided the trainees with a review of homework, lab and study regtirements, and schedule for completion.
A-9L A-9R 81 rs Il
APPENDIX B
SUMMARY
SCORING SHEET FOR TRAINING OBSERVATION CHECKLIST 4" TRAINING OBSERVATION CHECKLIST (LABORATORY / WORKSHOP SESSION)
(LABORATORY / WORKSHOP SESSION)
If a Iaboratory/ workshop session is selected for observation, this checklist is to be used in DATE-support of Question 3.9, which states, INSTRUCTOR-
"Is training being conducted in an adequate manner?"
LESSON TITLE-PLANNED LENGTH-SCORING
SUMMARY
CHECKLIST INSTRUCTIONS SCORE ASSIGNED Score each question of this checklist af ter observing the conduct of the entire session.
QUESTION AREA (Circle one)
Enter the score for each question in the appropriate space under SCORING
SUMMARY
on Page B-IR.
B.!
Timing, Pace, and Orientation 1 2
3 B.2 Introduction and Laboratory /
Workshop i
2 3
B.3 Structure and Content 1
2 3
B.4 Facility Support 1
2 3
B.5 Control and Flow 1
2 3
B.6 Demonstrated Technical i
Competence 1
2 3
B.7 Completeness and Accuracy 1
2 3
B.8 Scheduling and Coordination 1
2 3
B.9 Lesson Summary and Lab Prep 1 2
3 TOTAL SCORE AVERAGE SCORE FOR THIS OBSERVATION (to the nearest tenth) 1 B-IL B-IR 83
TIMING, PACE. AND ORIENTATION REVIED REQUIREMENTS 3
.a
.4 B.1 Did the instructor present the laboratory / workshop in a well-paced, coherent fashion NdN yid within the allotted time with sufficient attention to security, equipment, and u; A orier.tation?
bb B.1 Did the instructor present the oZ U V laboratory / workshop in a well-paced, coherent fashion within the allotted time with sufficient atten-tion to security, equipment, and orientation?
! 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the laboratory / workshop fell substantially outside planned duration (greater than 20%); instructor exhibited no time sense during presentation and had a negative ef fect on teaching and trainee acceptance of material.
@ if the laboratory / workshop fell outside of expected duration by 15-20%; some allowance for questions was provided.
@ if the laboratory / workshop was presented within the allotted time with ample, but not excessive, time for questions; if the laboratory class was in excess of one hour, a planned, controlled break was provided B-2L B.2R 84
INTRODUCTION AND LESSON CONTEXT REVIED REQUIREMENTS
~! $ E!
< ! y B.2 Did the instructor inform the class of the laboratory / workshop oblectives. context. and
$ gli
- 8 structure?
t-z 2 B.2 Did the instructor inform the class
@y@
of the laboratory / workshop objec-tives, context, and structure?
I 2,3 COMMENTS:
)
i l
4
)
i I
t SCORING GUIDANCE Score this question as a:
@ if the instructor failed to provide an adequate level of laboratory / workshop introduction or miscommunicated the overall intent, structure, or context of the laboratory / workshop with a negative effect on the ensuing instruction.
@ if the instructor outlined the laboratory / workshop in general or generic terms and, provided sufficient direction for the ensuing instruction; t% course context of the laboratory / workshop was adepately explained as related to preceding or following sessions.
@ if the instructor provided the trainee with a meaningful and complete instruct onal i
context for the laboratory / workshop with reference to the overall course objec-tives; the laboratory / workshop was out!M prior to the presentation and the structure was explained in detail.
B-3L B-3R 85
'l STRUCTURE AND CONTENT REVIETT REQUIREMENTS 3
d U d~
B.3 Did the instructor aeere to the content and structure of the instructor guide?
< g g-N$d e-z 2 B.3 Did the instructor a&ere to the O uJ O
- U U content and structure of the instructor guide?
I 2 3 COMMENTS:
l i
i l
SCORING GUIDANCE Score this question as a:
@ if the instructor failed to utilize the instructor guide and, as a result, major errors of omission occurred within the presentation; data required for ensuing training was not covered as required.
@ if the instructor utilized the outline of instruction with only minor departures into other subject areas; the subject areas of the lesson suffered from minor deviations in the level and content of the presentation by the instructor.
@ if the instructor kept to the outline of instruction, departing only to satisfy questions or to maintain class interest / attention; all subject areas were covered at a level commensurate with the data presented in the instructor guide and with the objectives of the lesson.
B-4L B-4R 86
FACILITY SUPPORT REVIEW REQUIREMENTS dud B.4 Did the instnsetor use the laboratory / workshop facility in a satisfactory manner to
<u H
support the achievement of the knowledge and skill requirements as specified in C
- Id e
lesson objectives?
hy$
B.4 Did the instructor use the O W O laboratory / workshop facility in a satisfactory manner to support the achievement of the knowledge and skill requirements as specified in the lesson objectives?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the instructor was confused and nervous and had a negative impact on effectiveness or if the instructor was unable to coordinate trainees with equipment and failed to support the achievement of skill objectives in the workshop.
@ if the instructor, in general, conveyed a feeling of certainty and presented the workshop in an effective fashion; the instructor provided sufficient coordination of trainees, equipment, space and time.
@ (if the instructor was poised and professional in his coordination of the facilities e.g., temperature, lighting, space); adequate room and time with the equipment were provided to satisfy skill requirements.
B-5L B-5R l
87
CONTROL AND FLOW REVIEO REQUIREMENTS dud B.5 Did the instructor maintain control of the class and the instructional flow?
< j y
$d 9 z E B.5 Did the instructor maintain control
@j8 of the class and the instructional flow?
I 2 3 COMMENTS:
i l
SCORING GUIDANCE Score this question as a:
@ if the instructor never gained control and the session deteriorated or time on equipment was inadequate to achieve expected skill level.
@ if the lesson was affected in only a minor f ashion by the loss of control and, the end goals of the workshop were attained.
@ if the instructor was always in control of the workshop; effective coordination of trainees and equipment was evident; the class' respect for the instructor allowed the successful comp!stion of the workshop without undue distraction.
B-6L B-6R i
88
DEMONSTRATED TECHNICAL COMPETENCE REVIEW REOUIREMENTS 3
.a
> a B.6 Did the instructor exhibit a level of technical knowledae adequate to satisfy the
- ! dN y
"i subiect matter requirements?
bb B.6 Did the instructor exhibit a level of oz o u technical knowledge adequate to satisfy the m& ject matter i
requirements?
! 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the instructor did not seem to have a technical skill ievel which would allow successful completion of the laboratory / workshop.
@ if the instructor's grasp of the subject matter was adequate to address the level of detail in the instructor guide; the instructor was able to demonstrate his detailed knowledge to an understandable level; questions were answered in a complete f ashion or with only minor / recoverable errors.
@ if the instructor exhibited a complete and thorough grasp of the subject matter and was confident enough in this skill be able to demonstrate it in an understandable f ashion; questions within the scope of the course were answered correctly and completely; ques *. ions outside the scope of the lesson or course were referred to of!!ine dismssion; suggestions and recommendations were offered in an acceptable manner.
r B-7L B-7R
COMPLETENESS AND ACCURACY REVIED REOUIREMENTS 3
a > a B.7 Was the laboratory /workshon, as presented, technically complete and were the skill
- i! dW expectations understandable?
{ "9:
j H<
ua B.7 Was the laboratory / workshop, as o d o-Z 0 V presented, technically complete and were the skill expectations under-standable?
! 2 3 COMMENTS:
i l
SCORING GUIDANCE Score this question as a:
@ if technical errors by the instructor were of such a severe and widespread nature that trainee confidence and instructor credibility could not be maintained; the instructor failed to bridge the gap between the instructor guide (lesson plan) and the reference materials; lack of skill and preparation were obvious.
@ if there were few technical errors in the demonstration and they were of a minor nature and had little impact on the accomplishment of the objectives; the instructor presented a complete picture of the function or system.
i
@ if there were no technical errors in the demonstration ag the instructor presented a complete picture of the laboratory equipment, including relating it to equipment in the plant.
I B-8L B-8R 90
I SCHEDULING AND COORDINATION REVIEW REOUIREMENTS
.$ $ E$
B.8 Did the instructor provide adequate laboratory time to allow for proficiency of Skill
<jy performance?
ygg 9 z :E B.8 Did the instructor provide adequate
@y@
laboratory time to allow for pro-i ficiency of skill performance?
I 2 3 COMMENTS:
il SCORING GUIDANCE Score this question as a:
@ if the instructor failed to coordinate adequate scheduling of trainees in the laboratory to a!!ow for proficiency of skill performance.
@ laboratory such that sufficient practice time was available to ensure mastery of if the instructor, in general, provided scheduling and coordination of trainees in the I
skills.
@ laboratory to permit adequate practice; supervision, guidance, and reinforcement if the instructor provided acceptable scheduling and coordination of trainees in the were provided; consumable /nonconsumable items were sufficient to support the skill objectives; flow from classroom to laboratory was effective for ensuring optimum learning by trainees.
B-9L B-9R 91 l
LESSON
SUMMARY
AND LAO PREP REVIED REQUIREMENTS dud B.9 Did the instructor asleauntely summarize the laboratory / work 4 co and leave the
$ :;i O tramees with a clear peture of remaining requirements for satisfactory completion of g Ei' the objectives?
>- z 2 B.9 Did the instructor adequately
@d8 summarize the laboratory / workshop and leave the tramees with a clear picture of remaining requirements for satisfactory completion of the objectives?
I 2 3 COMMENTS:
I SCORING GUIDANCE Score this question as a:
@ if the instructor did not summ7-N the lesson or provide adequate guidance and feedback.
@ if the instructor provided a brief summary of the lesson that, in general, communicated the requirements for successful laboratory completion.
@ if the instructor presented a corr.prehensive summary of the subject matter and provided the trainees with a complete briefing of laboratory and objective requirements and a schedule for completion; related safety precautions were reinforced in the summary and emphasized prior to the lab.
B-10L B-10R 92
APPENDIX C
SUMMARY
SCORING SIEET FOR
^
TRAINING OBSERVATION CHECKLIST (OJT SESSION)
(OJT SESSION)
If an 03T session is selected for observation, this checklist is to be used in support of DATE-Question 3.9, which states, INSTRUCTOR-
"Is training being conducted in an adequate manner?
LESSON TITLE-PLANNED LENGTH-SCORING
SUMMARY
CHECKLIST INSTRUCTIONS SCORE ASSIGNED Score each question of this checklist af ter observing the conduct of the entire session.
QUESTION AREA (Circle one)
Enter the score for each question in the appropriate space under SCORING
SUMMARY
on pap C-1R.
C.I Timing, Pace, and Orientation. I 2
3 C.2 Introduction and 03T 1
2 3
C.3 Structure and Content i
2 3
C.4 Facility Support i
2 3
C.3 Control and Flow 1
2 3
C.6 Demonstrated Technical Competence 1
2 3
C.7 Completeness and Accuracy 1
2 3
C.8 Scheduling and Coordination 1
2 3
C.9 OJT Summary 1
2 3
TOTAL SCORE i
AVERAGE SCORE FOR THIS OBSERVATION (to the nearest tenth) l i
l 1
i 4
C-IL C-IR 93
TIARNG. PACE, AND ORIENTATION REVIEN REOUIREMENTS dud C.! Did the instructor conduct the OJT in a well-paced, coherent fashion within the
< ld $
allotted time within sufficient attention to security, equipment, and orientation?
E$d 9 z lE C.1 Did the instructor conduct the OJT
@y8 in a well-paced, coherent fashion within the allotted time with sufficient attention to security, equipment, and orientation?
1 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if OJT fell well outside planned duration; instructor exhibited no time sense during presentation with negative effect on teaching and trainee acceptance of material or Evironmental conditions did not support conduct of OJT or safety precautions were not followed.
@ if OJT fell outside of expected duration by 15-20%; some a!!owance was provided for questions; environmental conditions were adequate for training; safety precau-tions were suf ficiently addressed; orientation to f acilities was adequate.
@ if OJT was presented within the allotted time with ample, but not excessive, time for questions; no apparent evidence of rushing or dragging; learning process was aided by environmental conditions; safety precautions adequately addressed; orien-tation to facilities was adequate.
C-2L C-2R e
y
INTRODUCTION AND CONTEXT REVIEW REQUIREMENTS 3
.a
.a C.2 Did the instructor inform the traanee(s) of the 03T objectives, context and structure?
YdU
- Id
.a C3 Did the instructor amere to the content and structure of the instructor mulde?
- ! -j ii8 c.
wdb C.3 Did the instructor a&ere to the o
content and structure of the Z U U instructor guide?
I 2 3 COMMENTS:
1 1
i SCORING GUIDANCE Score this question as a:
@ if the instructor failed to utilize the instructor guide (lesson plan) and, as a result, major errors of omission occurred within the presentation; data required for ensuing training was not covered as required.
@ if the instructor utilized the outline of instruction with only minor departures into other subject areas; the subject areas of the lesson suffered from only minor
]
deviations in the level and content of the presentation by the instructor.
@ if the instructor kept to the outline of trainee instruction, departing only to satisfy questions or to maintain trainee interest /attent'on; all areas were covered at a level commensurate with the data yesented in the instructor guide and with the intent of the lesson.
C-4L C-4R
^
96
FACILITY SUPPORT REVIE"'7 REQUIREMENTS dud C.) Did the instructor use the training facility in a satisfactory manner to support the
<gy H
achievement of the knowledge and skill requirements as specified in the lesson hya g
objectives?
C.4 Did the instructor use the training
@y8 facility in a satisfactory manner to support the achievement of the knowledge and skill requirements as specified in the lesson objectives?
I 2 3 COMMENTS:
I 9
i SCORING GUIDANCE Score this question as a:
@ if the instructor was unable to coordinate trainees with the equipment and failed to support the achievement of ski!! objectives of the 03T or the learning objective could not be achieved because the equipment did not support the objective.
@ if the instructor provided sufficient coordination of trainee (s), equipment, and time and adequately monitored the attention level of the trainee (s).
@ if the instructor used the equipment / facilities in a way that a!! owed complete achievement of the desired learning objectives in an efficient and effective manner.
C-5L C-5R 97
REVIEO REQUIREMENTS CONTROL AND FLOO
-j b d
.c
.a e
C.5 Did the instructor maintain control of the class and the instructional flow?
y y
9 z 2 C.5 Did the instructor maintain control O uJ O of the class and the instructional flow?
I 2 3 COMMENTS:
l SCORING GUIDANCE Score this question as a:
@ 11 the instructor never gained control and the session deteriorated; time on equipment was inadequate to achieve expected skri! level
@ if loss of control affected the lesson in only a minor fashion; related safety precautions and warnings were fo!! owed; and the end goals of the training session were attained.
@ if the instructor was in control of the training session throughout its conduct; effective coordination of trainee (s) and equipment was evident; related safety precautions and warnings were presented in such a way that a safety awareness /
attitude was instilled in the trainee (s).
C-6L C-6R 98 l
DEMONSTRATED TECHNICAL COMPETENCE REVIED REQUIREMENTS 3
a > a C.6 Did the instructor exhibit a level of techmcal imowledne adeaunte to satisfy the (y$
j uj requirements of the subiect matter?
u, a,
db l
C.6 Did ;he instructor estubit a level of Oz o u tedwucal knowledge W =te to satisfy the requirements of the sihject matter?
I 2 3 COMMENTS:
i 1
4 1
l SCORING GUIDANCE Score this question as a:
@ if the instructor did not seem to possess a technical skill level that would allow l
successful completion of the OJT.
)
i
@ if the instructor's grasp of the subject matter was adequate to address the level of detail in the instructor guide (lesson plan); the instructor was able to relate his detailed knowledge to an understandable level for the trainee; questions were answered in a complete fashion or with minor / recoverable errors.
i
@ if the instructor exhibited a comp ete and thorough grasp of the subject matter and was.ble to demonstrate it in an understandable fashion to the trainee; questions i
within the scope of the training were answered correctly and completely; questions outside the scope of the training were referred to of fline discussion.
C-7L C-7R i
99
COMPLETENESS AND ACCURACY REVIEO REQUIRE &CNTS 3
a >
.a C.7 Was the traininz **%
as presented. technically complete and were the skill pdW
$ c"$
expectations understandable?
W<
un e-.
z :E C.7 Was the training session, as o ua o Z U V presented, tedmically complete and were the skill expectations under-standable?
I 2 3 COMMENTS:
l t
i i
SCORING CUIDANCE Score this question as a:
@ if the a..tructor's technical errors were of such a severe and widespread nature that trainee confidence and instructor credibility could not be maintained; the instructor failed to bridge the gap between the instructor guide (lesson plan) and the 03T to be conducted; lack of skill and preparation were obvious.
@ if there were few technical errors in the demonstration and they were of a minor nature and had little impact on the accomplishment of the objectives; the instructor presented a complete picture of the OJT to be conducted and related the lesson to the overall performance of the system / function.
@ if there were no technical errors in the instructor's demonstrations to the trainees and safety precautions / limitations were emphasized and the 03T was related to the overall performance of the system / function.
C-81.
C-8R 100
-~
REVIEO REQUIREMENTS SCHEDULING AND COORDINATION dbd
<au C.S Did the instructor provide adequate trainina time to allow for proficiency of skill E$d performance?
H z 2 C.S Did the instructor provide adequate
@y@
training time to allow for pro-ficiency of skill performance?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the instructor failed to coordinate adequate scheduling of trainees to allow for proficiency of skill performance. Additional practice time was required.
@ if the instructor provided scheduling and coordination of trainee (s) in the OJT session, but f ailed to provide sufficient practice time to ensure skill mastery; supervision, guidance, and reinforcement were generally available.
@ if the instructor provided acceptable scheduling and coordination of trainees during the OJT session to permit adequate practice; aj sufficient supervision, guidance, and r6forcement were provided.
j 1
C-9L C-9R 101
I 3
.,s d j S $ U
'edywbU 2
- s. i s< oZ 1
S T
yef - -
hosc l
N et ie t
ta e aj EM u er sbo R
E q vu 0
at r 1
e R
dec o e ipfh C
l IJ a
t M
d C
nrtfs E
raa no R
o e e l
tTcmn 7
cu3 o
t e
E rOairi t
I t
V s
ue n e l
ihhq p E
e irm t
R t
ew o
ez gc hi n
rsi t aeny S
eir m n a o ?s T
dmimtc ve N a
iureai E
Dst rf t M
M 9
O C
C e
e da o
r c
h n
t a
n t
ad r
n o
e a
d d
ra i
l e.
e r
c i
t n t
s p
u t
a g
ao t
h s
ci a n d
?
e e
2 i
e e
is m
t ns m
iz 0
v t
w i a
u uc t e s
1 t
u ms cr a
s c q
e i h
e e e
mi d
ju p
e j d
oy bq n o a
ct ue m
b sr e
i e
a e e
t e
d f
e r
d a a v
t h i
hs hi n
t v
t t t a
e f o
a c
h o
r n
e f
y t
p og oj r
n s n b
a e o a
i r
si yo m
v t
o ed r
e e l u aT m
n l
l lp o
ec m3 u
mO s
s h n d
m E
s t
e i
n uf a o C
e t
f n so h
c N
oo eg n t
T y
A T
i n
3 r
3 yt iiv i
Y 0 o D
e f
0 alp ni e
r s
d t
I e
R e c U
e m m er c
A h
a G
h hb r
L f
mo e
o 0
M t
s t
G uc r e f
i M
e t
e s
pt n
1 z a N
z g
e i S
r i
f n ml e
C U
i s
I a r R
r ei op r
o a
in cm e
T m
O ri 3
C m
ba o
r f
0 m s r
ac e
t S
m t
dadw u
a s n u
l e
s d u eh s
,n t
ef ly m t
s t
e e o
ds nin io t
r n
ie ewot v
s i
c estiu a
o u u d
c r ee a.
r q q a
id pu pel cn e
s npeo e
s r
d a
r rr rimrips a
g o
oo oa t r o s
n n
t f o
c.
c ct cye t
r i
i s
o n s
u er eg t
uk u
t i
t t
c a s
r c r
r e
t a t n t h of n s e stf ai u m r
u sb inde sn nim t
e q
ind i
eluda s
r e
e n
is f
e e er ed er id f
t t o
h h
h iu h ve a i
t t
t td q
ohl e e e r ceh h r e
f f
n e
f u
r ia ir ips rt t
t o
di ic c
Q
)
Dp S
9 C
rI1I l
!I.ll l
l l
APPENDIX D
SUMMARY
SCORING SHEET FOR TRAINING OBSERVATION CHECKLIST "D" TRAINING OBSERVATION CHECKLIST (SIMULATOR SESSION)
(SIMULATOR SESSION)
!! a simulator session is selected for observation, this checklist is to be used in support of DATE.
Question 3.9, which states, INSTRUCTOR-
"Is training being conducted in an adequate manner?
LESSON TITLE-PLANNED LENGTH-SCORING
SUMMARY
CHECKLIST INSTRUCTIONS SCORE ASSIGNED Score each question of this checklist af ter observing the conduct of the entire session.
QUESTION AREA (Circle one)
Enter the score for each question in the appropriate space under SCORING
SUMMARY
on page D-IR.
D.1 Timing, Pace, and Orientation 1 2
3 D.2 Introduction and Simulator 1
2 3
D.3 Structure and Content i
2 3
D.4 Facility Support i
2 3
D.5 Control and Flow I
2 3
D.6 Demonstrated Technical Competence 1
2 3
D.7 Completeness and Accuracy 1
2 3
D.8 Scheduling and Coordination 1
2 3
D.9 Simulator Summary 1
2 3
TOTAL SCORE AVERAGE SCORE FOR THIS OBSERVATION (to the nearest tenth) d D-IL D-IR 103
REVIED REQUIREMENTS 3
TIMING PACE AND ORIENTATION
.a
.a i dd D.1 Did the instructor present the simulator training in a well-paced, coherent fashion y3 wittun the allotted time with sufficient attention to security, equipment, and ua S
' D.! Did the instructor present the simo.
$6$
orientation?
z o o lator training in a well-paced, coherent fashion within the allotted time with sufficient attention to security, equipment, and orienta-tion?
I 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if simulator training fel! vell outside planned duration (greater than 20%), or the instructor exhibited no time sense during presentation with negative effect on teaching and trainee acceptance of materials or no facility orientation was provided.
@ if simulator training te!! outside of expected duration by 15-20% with an a!!owance for questions and orientation to f acilities was adequate.
@ if simulator training was presented within the alloted time with ample, but not excessive, time for questions and orientation to facilities was adequate.
D-2R D-2L 104
INTOODUCTION AND CONTEXT REVIEO REQUIREMENTS 3
.a
>..a D.2 Dh! ti.e ia Mor inform the class of the simulator objectives, context, and structure?
d ~! N yW gj g<
m e:
D.2 Did the instructor inform the class o 6o Z U U of the simulator objectives, context, and struensre?
I 2 3 l
COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the instructor failed to provide an adequate level of simulator introduction or miscommunicated the overall intent, structure, or context of the training with a negative effect on the ensuing instruction.
@ if the instructor outlined the simulator training in general or generic terms and provided sufficient direction for the ensuing instruction; the course context of the training was adequately explained and the training was related to preceding or following sessions.
@ if the instructor provided the trainee (s) with a meaningful and complete instruc-tional context for the simulator with reference to the overall course objectives; the train!.ig was outlined prior to the presentation and the structure was explained in detail D-3L D-3R I05
REVIEW REOUIREMENTS 3
STRUCTURE AND CONTENT
~$ U E$
<jg Did the instructor adere to the content and structure of the exercise guide / lesson plan?
E$d D.3 e z 2 O W O D.3 Did the instructor adhere to the U
content and structure of the exercise guide / lesson plan?
! 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the instructor failed to utilize the exercise guide (lesson plan) and, as a result, major errors of omission occurred within the presentation; data required for ensuing training was not covered as required.
@ if the instructor utilized the outline of the exercise guide; any departures into other subject areas were based upon a need to support exercisc guide objectives; the exercise suffered from only minor deviations in the level and content of the presentation by the instructor.
@ if the instructor kept to the outline of the exercise guide, departing only to demonstrate related concepts / responses or to maintain class interest / attention; all subject areas were covered at a level commensurate with the intent and data presented in the exercise guide.
D-4R D-4L 106 imud
l
- 7 FACILITY SUPPORT REVIEO REOUIREMENTS
~ yi
.a
.a
"$ "$ d D.4 Did the instructor use the simulator training facility in a satisfactory manner to
[ j 'j s
=&t the achievement of the knowledge and skill requirements as wdfied in the gg lesson objectives?
D.4 Did the instruc. tor use the simulator o w o Z " U training facility in a satisfactory manner to sipport the acnievement of the knowledge and ski!! require-ments as specified in the lesson objectives?
! 2 3 COMMENTS:
SCORING GUIDANCE Score this question as a:
@ if the instructor was unable to coordinate trainees with the simulator features and capabilities and failed to support the achievement of skill objectives in the exercise.
i
@ if the instructor presented the simulator training in a manner which supported the etfectiveness of the lesson and the instructor provided sufIIcTent coordination of trainees and equipment.
h (if the instructor was poised and professional in his coordination of the facilities e.g., temperature, lighting, space) af adequate time with the equipment was provided to satisfy skill requirements.
D-5L D-5R 107 l
CONTROL AND FLO7 REVIE"7 REQUIREMENTS d3d D.5 Did the instructor maintain control of the etmu and the Instructional flow?
<py E$d 9 z 2 D.5 Did the instructor maintain control O id O zo u of the clama and the instre flow?
! 2 3 COMMENTS:
I SCORING GUIDANCE Score this question as a
@ if control was never gained by the instructor and the session deteriorated g time en equipment was inadequate to achieve expected skil! level.
@ if the lesson was af fected in only a minor f ashion by loss of control and, the end goals of the training sessions were attained.
i
@ if the instructor was in control of the simulator training session throughout its conduct; effective coordination of trainees and equipme.: was evident; related safety precautions and warnings were presented and a safety awareness / attitude was instilled in the trainees.
s D4L D-6R 108 i
DEMONSTRATED TECHNICAL COMPETENCE REVIED REOUIREMENTS 3
.a
> a D6 Did the instructor exhibit a level of technical knowledre adequate to satisfy the Y ~$ N
<y$
requirements of the subject matter?
u.a 9
D.6 Did the instructor exhibit a level of o do Z G V technical knowledge adequate to satisfy the requirements of the sub}ect matter?
I 2 3 COMMENTS:
l l
SCORING GUIDANCE Score this question as a:
@ if the instructor did not seem to possess a technical skill level which would allow successful completion of the simulator training.
@ if the instructor's grasp of the subject matter was adequate to address the level of detail in the exercise guide; the instructor was able to demonstrate his detailed knowledge to a level understandable by the trainee (s).
@ if the instructor exhibited a complete and thorough grasp of the subject matter and was confident enough in this skill to be able to demonstrate it in an understandable fashion on the simulator equipment; questions within the scope of the simulator training were answered correctly and completely; questions outside the scope of the simulator training were raferred to offline discussion.
D-7L D-7R 109
REVIED REQUIREMENTS 3
COMPLETENESS AND ACCURACY a >- a j $
Was the simulator traLia session. as presented. technically complete and were the
( $ "aj D.7 w<
skill expectations understandable?
e B5E DJ Was the simulator training sessaan.
z a v as presented, technically complete and were the skill expectations understandable?
1 2 3 COMMENTS:
l l
SCORING GUIDANCE Score this question as a:
@ if technical errors by the instructor were of such a severe and widespread nature that trainee confidence and instructor credibility could not be maintained; fack of skill and preparation were obvious; technical errors in the demonstration resulted in confusion and uncertainty in the trainee (s); errors of an obvious nature were made and resulted in the erosion of instructor credibility.
@ if there were few technical errors in the demonstration and they were of a minor nature and had no impact on the accomplishment of the objectives; the instructor presented a complete picture of the function or system and related the exercise.
@ if there were no technical errors in the instructor's implementation of the simulator exercise and the instructor preYnted an integrated picture of the function (s)/ systems (s),
including their effect on the overall performance of the system (s)/ plant.
D-8R D-8L 110
SCHEDULING AND COORDINATION REVIED REQUIREMENT _S 3
. >- a D.S Did the instructor provide =d===a=te simulator trasnirm time to allow for proficiency of skill 4dd performance?
Q y
- z 2
D.S Did the instructor provide adequate O w o
- V U simulator trainirs time to allow for proficiency of skill performance?
! 2 3 COMMENTS:
l SCORING GUIDANCE Score this question as a:
@ if the instructor failed to coordinate adequate scheduling of trainee (s) on the simulator equipment or to a!!ow for proficiency of skill performance.
@ if the instructor provided scheduling and coordination of trainee (s) in simulator training session so as to provide, in general, sufficient practice time to ensure mastery of skill and instructor supervTsion, guidance, and reinforcement were generally available.
@ if the instructor provided scheduling and coordination of trainee (s)in the simulator training session to permit adequate practice by all trainees g supervision, uidance, and reinforcement were always provided when required by the trainee (s.
D-9L D-9R 111
RE*JIED REQUIREMENTS 3
SIMULATOR
SUMMARY
.$ $ d
-8 D.9 Did the instructor adequately summarize the simulator training and leave the trainees Q
with a clear picture of remaining requirements for satisfactory objective completion?
s z a D.9 Did the instructor adequately O w 0 Z U U summarize the simulator training and leave the trainees with a clear picture of remaining requirements for satisfactory objective comple-tion?
I 2 3 COMMENTS:
1 l
l SCORING GUIDANCE Score this question as a:
@ if the instructor did not summarize the exercise or provide adequate guidance and feedback to trainees.
@ if the instructor provided a brief summary of the lesson that generally communi-cated the requirements for successful simulator training completion, including safety discussions.
@ if the instructor presented a comprehensive summary of the subject matter and provided the trainees with a complete briefing of simulator training and objective requirements and schedule for completior ; related safety precautions were I
reinforced in the summary and emphasized prior to the training session.
D-10L D-10R 112
u CEl<E Eo m
CZ O
O l
i 1
l l
SCORING
SUMMARY
AVERAGE CHARACTERISTICS SCORE SECTION SCORED AS@ *
(to nearest tenth) 1 (Systematic Analysis of Jobs to be Performed) 2 (Development of Learning Objectives) 3 (Design /
Implementation) i I
i (Trainee Evaluation) i 5
(Program Evaluation)
These items may form the basis for recommenda-tions for needed training program improvements.
I13
QC<
<r)
V)
O a
O
i l
GLOSSARY CONDITIONS - On-the-job conditions that significantly influence performance of a task (i.e.,"given that the reactor coolant system is at normal operating temperature and pressure").
CUE - Event that determines, in the job situation, when the job incumbent performs a particular task (i.e., "when the tank high level alarm is activated").
ENABLING LEARNING OBJECTIVE - Learning objective related to the skills and knowledge that enable the trainee to perform a given task.
INSTRUCTIONAL SETTING - Environment in which learning occurs.
LEARNING OBJECTIVE - Precisely what is to be learned in terms of the expected trainee performanece under specified conditions to. accepted standards; terminal learning objective.
l LES5ON PLAN - Structured outline that ensures consistency in instructor presentation and evaluation of trainee performance.
STANDARDS - Criteria or standards of performance of a task that separate " acceptable" from " unacceptable" job performance (i.e., " tank level must be maintained between high and low level trip set points").
SYSTEMATIC - Consisting of a logically ordered set of steps; clearly documented so that it can be executed with consistent results on different occasions by different personnel; requiring a physical record (e.g., signature) of external review /
approval for historical accountability; and resulting in a consistently formatted, quality product where applicable.
TASK - Unit of work that describes the performance of a complete meaningful function in a job.
TASK ANALYSIS - Process of identifying the skills and knowledge that are necessary for a job incumbent to adequately perform assigned tasks.
TERMINAL LEARNING OBJECTIVE - Learning objective directly related to the per-formance of a particular job task.
115
esRC POmas 335 U S. NUCLEAR KEGULATORY COamaiss#0N e REPORT NUM8tR fAsstwd DF IdOC. edd Fei No,8f eaFJ (2 Set
%"/E2 '
BIBLIOGRAPHIC DATA SHEET NUREG-1220 SEE INSTRUCTsONS ON T E REVER$E
- 2. TITLE *N0 5UBTsTLt 3 LE AVE BLANK Training Revie Criteria and Procedures
[
4 DATE REPOR T COMPLETED
/May l
MONr.
, EAR 1986
. AuT Oms, f
6 DATE REPORT ISSUED j
MO~T.
vEAR July 1986 7 PERFORMtNG ORGANi2 ATION NAME AND MAIL ADDRESS flac8eedele Caett 8 PROJECTIT A5K' WORK UNtiNUMSER Division of Human Facto Technology Office of Nuclear Reacto Regulation U.S. Nuclear Regulatory Co.
ission Washington, DC 20555 10 SPONSORING ORGANIZATION NAME AND MAILING ADDRESS i s dele Codes tia T YPE OP REPORT ee echnical Same as 7. above,
. PERIOD COVERED stace a,ve aumf v
12 $UPPLEMENT ARY NOTES 13 A85TR ACT (200 words or sessJ This report provides a set of training rev.
. riteria and procedures which constitute a systematic means of implementing two mo tor functions identified in the " Commission Policy Statement on Training and Oualific tions Nuclear Power Plant Personnel" (March 20, 1984, 50 FR 11147). These furlitions a s
1.
Continuing evaluation of indus y-wide trai 'ng and qualification program effectiveness, and 2.
Monitoring plant and industry rends and event involving personnel errors.
The procedures are organized around t.
five essential ele. nts of performance-based training articulated in the Policy St ~ ement.
The package was designed for use by C personnel engaged in t review of performance-based training programs in nuclear p It has been pu ished in a modified version in NUREG format to enable la$er plants.
g.ge-scale production and dist 'bution for information purposes.
W t
14 DOCUMENT AN ALYSis - e KEvwORD5/DESCRiPTORS p
AV AIL AsitiT v performance-based training performance evaluation i
systems approach to training g
Unlimited training
'e sEcumTv etAssiP CATION traini
. DENT P ERs,nfEN ENDED nRMsreview criteria F~ -
0 Unclassified aru,,->
Unclassified 17 NUMBER OF P AGES 18 PRICE
UNITED STATES snciavouAm CLA$$ AAM NUCLEAR REGULATORY COMMISSION Postace er ms Pac WASHINGTON, D.C. 20566 wfN"Sc.
P(AMIT feo G47 OFFICIAL SUS (NESS PENALTY FOR PR:VATE USE. 8300 120555078877 1 1ANITO US NRC TIOC ADM-DIV 0F& PUB MGT BR-PDR NUREG POLICY W-501 OC 20555 WASHINGTON l
-