ML20132C376
| ML20132C376 | |
| Person / Time | |
|---|---|
| Site: | Farley  |
| Issue date: | 08/08/1984 |
| From: | Buzy J, Koontz J Office of Nuclear Reactor Regulation |
| To: | Booher H Office of Nuclear Reactor Regulation |
| References | |
| NUDOCS 8408240215 | |
| Download: ML20132C376 (23) | |
Text
-. - .
b ja d E ICIC'N O C3 ,
CertifiUl 2 - u. m se nso
- a. r,voni7 J. Buzy 00 - MS 50 -364 MEMCRA'iDUil FOR: Harold Booner, Chief License Qualifications Branch Division of Human Factors Safety THRU: J. J. Persensky Personnel Qualifications Section Licensee Qualifications Branch Division of Human Factors Safety FROM: J. L. Koontz '
Personnel Qualifications Section l Licensee Qualifications Branch Division of Human Factors Safety J. J. Buzy Personnel Qualifications Section Licensee Qualifications Branch Division of Human Factors Safety SU3 JECT: OBSERVATION OF INP0 ACCREDITATI0fl TEAll VISIT AT FARLEY NUCLEAR POWER STATION On June 18, 19 and 20, Jennifer Koontz and Jose'ph Buzy observed members of the INP0 Accreditation Team at Farley Nuclear Station near Dothan, Alabama.
The purpose of the visit was to evaluate the adequacy of the IllPO Accreditation process as a means of meeting the intent of requirements for implementation of a systems approach to training (SAT) in the comercial nuclear power industry.
i I. Accreditation Process The Accreditation Process as implemented by the Accreditation of Training in the Nuclear Pcwer Industry IllP0 82-011, !1ay 1982 generally occurs in several phases. During a preliminary visit to the utility, INPO provides information on how to prepare the Self-Evaluation Report (SER). Once a draft SER is prepared, ItiP0 makes an accreditation assistance visit to discuss the SER and suggest revisions as
- appropria te. Upon receipt of the final SER, INPO conducts a review and ,
decides on the schedule for the accreditation team to visit the utility. l The accreditation team visit, which lasts for one week onsite, results l in a report which identifies training program strengths andfweaknesses I related to It:PO's accreditation criteria. Further resolution of identified problems may be necessary prior to a decision by the IllP0 Accreditation Board.
~
t i
- "C' Y,......... ...... ....... ............. ........... . .. ... ....' .. .. . ..... - ....... . .. .. . - .-
l
.wrnau m e2do _ - - - - -" _
- "5 " " "-' *" l OFFICIAL RECORD COPY- - - - . _ _ _ _ _ _ . _ _ _ _ . _ _ _ _ , . . . _ _ , _ _ , _ _ _ _ _ _ _ _ _ l
l Recw.mndations reported to the utility es a result of the accreditation tes. visit require a written response by the utility, including a ccanitnent to take action to resolve the identified problens, a plan for resolution, and initial implementation of the plan. '
The IllPO Accreditation Team at Farley consisted of peer evaluators from INP0 member utilities and personnel from INP0's Accreditation The team was and I Training Assistance Departments (see Enclosure 1).
organized in two subgroups to review the training prog specific topical area. -
Farley has proposed four training programs for accreditation at this time. These are non-licensed operator, licensed operator, licensed Responsibility operator requalification, and shift technical advisor.for review review team members.
21, 1984, and made a submittal Farley began preparing the SER on JanuaryThe SER was reviewed by the INPO st to INP0 on May 7, 1984. At this time, we have not made assigncents were completed by June 1984.
inquiry as to the labor involved in the preparation of the SER, review .
by the INP0 staff, developing and assembling assignments, Regarding the size the evaluation process and preparation of the INPO team findings.of the accre involved solely in the evaluation process and probably 6-8 man weeks in developing the team report which will be presented to the accreditation board.
The NRC team concludes that the accreditation process requires extensive use of trained and experienced personnel to prepare and carry out the review process. We hope to obtain additional information when we visit the INP0 Offices and review the Farley report.
Evaluation of, f training programs The stated purpose of the accreditation team visit is to gather facts related to verification of information contained in the utility's Self Evaluation Report (SER) and to compare training programs to INPO's accreditation criteria (from ~he Accreditation ofTo Training facilitate in the Nuclear Power Industry, INP0 32-011, May 1982).
evaluation, each team member uses questions cross-referenced to INP0 criteria related to a specific topical area and a list of indicators for each question. For exonple, for evaluatir,g training prograft development ar.d impleaentaticn: 1
"' C ' > ..................
I l
p Learning Objectives (III.c.2.b)
Question - Are behavorial learning objectives (which specify of job performance? Are the objectives written with conditions-actions-i.
standards?
Indicators - JTA task list training objectives-matching tasks !
conditions-acticns-standards Interviews are used to gather information and to obtain answers to the Evaluators look for supporting -
questions for each topical area. documentation and examples that meet an
- Documents and reference materials (e.g., lesson plans, job analysis data, training center procedures, course descriptions) are reviewed against Review of the problems identified in the SER by the IUP0 INP0 criteria.
Team and method (s) to solve those problems is one part of the review s
process. The team evaluates the efforts identified to solve the problems, resources available, accomplishments to date and makes preliminary evaluations of whether the facility can meet comitments.
Preliminary problem identification is made using the SER. Further problems or concerns are identified individually by team Group and members team meetingsand .
documented with examples and checklists daily.
are held each day to share information, summarize concerns, identify follow-up items and discuss problems.
Each day principal members of the team meet with the utility's staff and infom them of the team's progress and any problems identified durino the previous day. The utility may provide additional information to the individual team members based on these comments.
- During our visit we were able to observe one meetingWe with didthenotfacility attead staff and two meetings of the combined team members.
the final meeting of the team or the exit meeting with the team and the Farley staff.
The breakdown of individual topical areas forSharing reviewoffacilitates concerns in coverage of a wide range of subject matter.
group and team meetings ensures further coverage so that problem areas are identified and followed up as necessary. All problems or concerns, however, must be directly related to INP0's criteria used for evaluation. Problems that are identified which do not relate the INP0's Identiffad concerns are a i.e.,ated criteria are not formally pursued.so that final problem docume the level of the accreditation criteria).
A pilot review of job analysis data was conducted by one INPO team member at Farley. The review consisted of a cross-check for
. . . ~ . - - -
..................... - . . - ~ . . . - -
L-wr-manemitacu ono OFFICIAL RECORD COPY
completeness of a sample of liWO jobThe anelysis d6ta against the INP0 knowledge taxonomy wes also documented job analysis at Farley.
cross-checked against utility lesson plans to see if knowledges, skills and abilities appeared in instructional materials. The purpose of the review was to evaluate the potential usability of the IllP0 job and task analysis data for training evaluation and accreditation purposes.
II. Observations at Farley Interviews Interviews of utility personnel constitute the major data gathering
- The depth of interview method employed by the accreditation team.While some accreditation team members coverage var.ied among evaluators.
interviewed a range of utility personnel, including supervisors, program cevelopment, staff, instructors, shift personnel and trainees, not all team members interviewed trainees and shift personnel. The interviews of plant and operations personnel regarding training benefits and problems is an important source of information which should not be overlooked by any evaluator.
Interviews of training staff members by program and process team members .
was often performed on an individual basis thereby requiring the training staff mecbers to be available for extended periods of time and often requiring IflP0 team members to wait or be on call untti the interviews were ccepleted. We believe that teams of program and process members could have performed the interviews in a shorter period of time and probably have obtained more information for each of the evaluation groups. Consistency among evaluators in application of INP0 criteria may be enhanced by training of process and program evaluators and reduce redundancy in interviews of key utility training staff.
Interviews of operators and senior operators were conducted'in the control room using IHP0 evaluation workbook items during normal operation of the units. The evaluators were required to identify by position and interview specific members of the crews. Although the operators had back-up personnel available, interviews in the control room for any period of time may distract personnel from performance of duties and probably reduces the free dialogue that could occur in another setting. Interviews with operators on only one shif t results in a relatively small sample of the operating crews. Interviews with a larger number of personnel (two shifts) in each position and a different setting could produce more valid information and reduce the possible f
distraction of the crews.
ItiPO criteria on evaluation workbook items may not be consistently used by team members, some more skilled interviewers do not follow a fixed question and answer format. Subtle differences among interviewers % ;;
d t: " :cre mi2 Pit; " dept 5 rd :cepc of topf: ;;;; r;.gc .
l ...".............. .....................
. . - " - . . . - - . . " . " . . ~ . -
om> .................. .................... ..................... . . . . . . . . . . . . . . . . . . . . .
Law.ru nn no.sowcu o24o OFFICIAL RECORD COPY -* "
- 2"2-*[
1 m u -
I intervieuers consistently follow written interview proteccls with !
relatively little probing or follo.vup. While written criteria facilitate reliability (i.e., consistency) among evaluators, successful !
information gathering cay depend on flexibility and interviewing i l expertise.
Specific Coments
-Our view of the licensed operator data sheet which was furnished to the i
accreditation team indicates that operator's licenses were originally issued in !! arch 1978. A total of 73 licensed operators and senior operators are on the staff and some 26% of the licenses are effective ,
for only one of the two units. Although the accreditation process '
! reviews programs for licensed personnel, we did not observe a system l which evaluates _the differences in training between units or method (s)
> to obtain licenses on both units. - e
- Review of four training tasks by one peer evaluator took over two hours t
- to identify lesson plans and develop findings. Only one task was i readily identified and evaluated. While the other three tasks were i l related to operator responsibility, they were located in subsections of l
procedures and required extensive searching by the licensee's
- instructors and the peer reviewer. We could not determine if the .
! students were. required to perform or walk-thru each task. Interviews j with operators did not detemine if these tasks were performed as part l
of the on-the-job _ training requirements for initial licensing or during
- the requalification of operators. The labor intensive verification of
' specific training content is dependent on the judgment and expertise of
! the reviewers rather than on specific evaluation criteria. ;
-We did not observe if INPO evaluators were required to review lesson l
- plans before attending a trairing session in the classroom or for i simulator exercises. During the !.Qd training assessment audits conducted in 1983 and recent T!11-1 training at S&W, we found that prior review of lesson plans and student handouts provided the evaluators _a 4
preview of material to be covered in the class and enabled the evaluator to concentrate on the instructor's presentation, participation by i students, and the overall effectiveness of the training session. ,
Although most of the classroom training is. contained in lesson plans, -
simulator training lesson plans at Farley are in the developcent stages. ;
1 '
l The on-the-job training (0JT) checklist at Farley is used by operator, ,
upgrade, and Shift Supervisor candidates. There is no distinction between classification of operators as to performance or reduireer.ents by the-evaluators. We did not notice if 0JT was included as a deficiency in the Farley SER; however, it was noted by the peer reviewers. <
Interviews with licensed personnel who are required to evaluate the ,
i trainees reveal that operators and senior operators have distinct levels ;
i
- -*"'**> .................. ..................... ..................... ............ ~ ...... ..................... ..................... ............~...i
- **"'> ................... ..................... .......'.... ~ . ~ .................. - - ....... - - . - - ......... ~ ... - -
-..~.-~..i I
- > .................. .................... ..................... ..................... ..................... ....-~.--- ~~~~
f
.=cm2. muo,.em.c ouc , _
OFFICIAL RECORD COPY _ ,_ _
- *] ]" ] [
of rce,uiremnts. Unfortuwtely, these requirencut.s are rot an .LJ in the utility's training evaluation process (SEP). P.eview of the final INPO accreditation report to Farley will be necessary to deternine whether these probleas are docunented as a result of the accreditatior, review. ,,
Several inquiries were made by peer menbers and the NRC staff which addressed feedback of operating experience for training nodifications.
Although trainees and operating staff were aware of recent significant events, we did not deternine if the accreditaticn process ensures that significant events are contained in lesson plans for classrocn training or sinulator exercises. Stailarly, we are not sure that the accreditation process ensures feedback fren operator performance for. .s__
training program nodifications.
We noted that Farley Unit 2 operated for some 380 consecutive days at power without a shutdown. During this period Unit I had several unscheduled shutdowns and.a. refueling outage. It appears that the operating staff and other licensed personnel had limited opportunity to perform reactivity manipulations or periodic practice at the console.
We did not observe if the accredita, tion process ensures periodic training using actual plant operations or simulators for individuals or teams on a regular schedule. .
The Farley staff requested INP0 to provide more specific criteria for waiver of training requirements for experienced non-licensed and licensed personnel. Although INP0 did not nake any connitments that we are aware of, we should pursue this issue to ensure that criteria regarding waivers are compatible with the intent of current regulations and Section 5.2.1.7 of ANS/ ANSI 3.1, 1981. INP0 criteria do not appear to reference applicable codes and standards which currently cover such topics'as testing in lieu of training. "_ ,,g General Connents Daily sunnary neetings of the process and content subgroups provide for sharing of concerns and problems identified by the review team nenbers.
These group debriefings help to set goals for subsequent review activities and identify problem areas for review by other tean menbers as appropriate. This pronotes a nore thorough review and focuses concerns on INP0 criteria. The lead evaluators play a major role in aggregating and sunnarizing the subgroups' concerns. A full review team neeting at the end of each day helps to focus the review team activities for the subsequent day. Note that the NRC observers were not able to attend the final two days of the evaluatien in order to eb$erve the wrap-up of the week's activities.
CATEf .................. ..................... ..................... ....".............. . ...................
.- ro.u p. "e aom ace OFFICI AL RECORD COPY * " " ""
- a .. . .;
The participation of peer reviewers fren other utilities is seen nn means of ensuring some consistency of the ace editation process. Inere appears to be a great deal of positive trans' - of experience
- peer by peer participants reviewers to their home utilities as well as themselves. Two of-the peer reviewers had gone t...rvogh the accreditation process at their home utility.
The utility training staff views the INP0 accreditation process as an opportunity to obtain important and helpful feedback regarding their training development progress. The utility training staff also perceives the benefit of having training issues and problems brought to management attention by outside evaluators. Although the preparation '
and planning for the INP0 accreditation visit is labor-intensive, the utility perceives the accreditation process as an important tool for training program development. The utility. sees INP0 accreditation as one means of being well prepared for any training regulation that may be promulgated by the NRC.
One peer reviewer from a utility which has received accreditation status perceives that the credibility of the accreditation process still needs to be established and maintained. For a utility going through the self evaluation prior to accreditation, INP0 criteria may not provide enough guidance since they are open'to interpretation. The problem of .
subjectivity is perceived as being comon to accreditation of higher education; decisions must be made based on a set of criteria which are subject to interpretation.
An additional concern developed during the process of assembling this report. We did not observe if the accreditation process will review JTA for new tasks which are not identified in current training programs for nonlicensed and licensed personnel. The new tasks, once identified, should be included in retraining or requalification programs with a set priority and schedule.
The variability among utility training programs and training developcent activities is perceived to be. a potential problem for censistent application of INP0's criteria. For example, the criteria are not intended to be applied to a comparison of-training development activities (e.g.,taskanalysis)betweenutilities. The President of INPO states that industry variability is.a challenge to accreditation.
There is a need for flexibility and approeches which allow for plant differences without lowering the standards by which acceptability is determined. t A conclusion of the NRC observers after these days of observation was that the INPO accreditation visit at Farley was prenature. The training ~
development activities utilizing SAT at Farley are still in their infancy. Monitoring of the _ utility's progress will be necessary in ,
- r kr for "Z u Mc ;a inf,acd ;33c3 3...c it of tai.ii.y p.w., a,.ma l
..................... ..................... .................l
- ^2 4 .................. ..................... .....................
- u.s.ono ne -4eo. l
.: rmu a s no.se uncu o24o OFFICIAL RECORD COPY
prior to o decision regardirg accreditattun. Inu neWA f or resc h irn "open items" prior to granting accreditation should be nonitored by the HRC.
A major problem identified by the staff evaluation of the INP0 accreditation process is the lack of decision method for determining a utility's compliance with INP0 criteria. There is no procedure to describe how the concerns and problems identified from the application of INP0's criteria are combined in order to determine eligibility for accreditation status. INP0 does not use a weighting scheme or method of determining the relative importance of the various criteria or identified problems. The criteria used by INP0 appear to be heavily s weighted toward assessment of learning objectives as the basis for training. A stated criterion for making a recommendation to the utility in the accreditation report is whether an identified problem affects training effectiveness. The accreditation criteria cust be sufficiently rooted in conditions and standards to allow a reliable decision about accrediting utility training.
f III. Recommendations As a result of the preliminary evaluation of the INP0 accreditation process as implemented at Farley, several actions are recommended. .
- The HRC should develop an objective method by which to assess, document and evaluate both the process and the results of INPO
- accreditation at a sample of nuclear plants. 1
- The evaluation should include a review of the preparatory activities of the utility seeking accreditation (i.e. the Self Evaluation Report), the methods applied and the results obtained by the IUP0 accreditation team visit, the report of the INP0 accreditation team, and the accreditation board review process and decision.
- Since INP0 accreditation is as yet an unvalidated process, it is important to monitor the process as it natures, including a comparison among nuclear utilities which were among the first to receive accreditation and those which may be undergoing accredi-tation in the near term. In particular, the summary evaluation reports are alleged to have changed significantly since the initial accreditation reviews by IUP0 and should be reviewed by HRC as part of a ccntinuing NRC reliability check. The availability to NRC of wri' ten documentation of the IUP0 accreditationtprocess may constrain direct evaluation.
- The NRC should continue to participate as cbservers during IUP0 accreditation activities in order to evaluate the process and results c^7 '" *' '< ---^ -
2: een: c' ecser ng imp'ementat4c- ^' '
i
~ . ~ .. - ........ - ~ ....... . ~ ....... - -
"C' Y ..................... .............. ~ .... .....................
""'>f..................
..... .......... ~ ..
...... ..................... . ~ .... - ~ ~
ear) .................... ..................... .................... ..................... - . . . . . - ~ ~ ~
~ ~ ~ ~ . -
.cc ccaw re no somacu o24o OFFICIAL RECORD COPY
_7 i
- The fasibilitj of pirticipation b; th.. r: inspect m or LQ3/ Region Inspcction Tean at a utility u.;dcrsai.ig accreditat.ica should be investigated. -
- Subsequent fir.C evaluation activitics should focus en both validity and reliability (i.e., consistency) of the process and the resultant recom.:nJ4tior.s among utilities.
. Original signed bys s J. L. Koontz s Personnel Qualifications Se tion Licensee Qualificaticns Branch Division of Human Factors Safety
. Original signed by:
J. J. Buzy
- Personnel Qualifications Section Licensee Qualifications Branch Division of Human Factors Safety I
Enclosure:
As stated cc: H. Thompson i
I f
q l
JLK/ MEMO TO B00HER ea J ome) .DHFS:LQd DHFS:LQB DHl QB
'""^"*>
JK66KtY:vp ""J
"*J ' 'f 'dHsky".
j.............. . " . . * . " ". . ." ". . ." ". .. ." ".. .". ..""". .. .""".. .. ."""......."
~> et.8..t.ea..... .. . .si. . . t . . .i a. a. ..e4
. . D.e.9. .... ..................... ..................... . . . . . . . . . . . . . . . . . . . . ......-..........
.cc r:::w m no,somacu o24o OFFICIAL RECORD COPY _
- [2 "o ""-*oof