IR 05000354/1998302
| ML20217H984 | |
| Person / Time | |
|---|---|
| Site: | Hope Creek  |
| Issue date: | 10/13/1999 |
| From: | NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML20217H982 | List: |
| References | |
| 50-354-98-302-0, NUDOCS 9910220243 | |
| Download: ML20217H984 (28) | |
Text
fr
l U. S. NUCLEAR REGULATORY COMMISSION
REGION I
Docket No:
50-354 Report No:
50-354/98-302 - Supplement License No:
'
Licensee:
Public Service Electric and Gas Company Facility:
Hope Creek Nuclear Generating Station l
l
'
Location:
Hancocks Bridge, New Jersey Dates:
January 25-29,1999 (NRC onsite review); and Management Meeting March 22,1999 April 29-30,1999 (in-office review)
June 17-18,1999 (in-office review)
Telephone call on appeal results, July 20,1999 Inspectors:
Richard Conte, Review Leader William Maier, Operations Engineer (part time)
John F. Munro, Senior Examiner, NRR
.
,
Rene J. Vogt-Lowell, Operations Engineer, NRR
^
Approved by:
Wayne D. Lanning, Director Division of Reactor Safety
'
9910220243 991013 PDR ADOCK 05000354 V
.
h
s
.
EXECUTIVE SUMMARY Hope Creek Nuclear Generating Station Examination Report Nos. 50-354/98-302 - Supplement As a result of the administration of the February 1998 initial examination at Hope Creek, the NRC staff identified two facility problems dealing with: (1) the quality of initial examination submittals in meeting the examination standards; and, (2) written test results which were too close to the pass / fail demarcation of 80 percent. The two problems manifested themselves again during the administration of the December 1998 examinations. This supplemental review addressed the adequacy of the initial license training program at the facility with respect to the two problems identified. This report also gives the results of individual operator appeals for the December 1998 examinations.
The first problem on the quality of initial exam submittals recurred due, in part, to ineffective corrective actions conducted in response to the identification of the underlying causes from the February 1998 exam. For example, this occurred for the corrective action addressing the need among licensee staff to gain more knowledge and experience in the NRC examination standards. Also, the licensee's problem with the quality of initial exam submittals occurred in the December 1998 exam due to two contributing causes or factors: (1) the effectiveness of training and operations department reviews for the December 1998 exam was limited due to a combination of schedule pressure and the workload of the responsible licensee supervisors and managers; and, (2) the licensee's validation process for exams was deficient.
The second problem on written test results similarly recurred due to ineffective corrective actions for a deficient trainee evaluation process. However, unlike the exam quality problem, this problem was more complex and required more effort than realized in order to prevent it from recurring in the December 1998 examinations.
.
Both problems recurred even though there was a reasonably thorough identification of causes.
The implementation of the licensee's condition reporting process was deficient since ineffective corrective actions resulted. Also, from the results of the licensee's 1999 review, the underlying causes of the ineffective corrective actions were not fully explained.
As noted at the management meeting of March 22,1999, both problems will be further reviewed by facility license and NRC staff. (Inspector Follow item (IFI) 98-302-01)
As a result of four individual operator applicant appeals, two of the four received licenses.
These results add to the overall results as follows: five of seven reactor operator and six of nine senior reactor operator applicants passed the written test of the examination (overall eleven of sixteen applicants received licenses). All applicants passed the operating test of the examination.
il
f, i
Report Details Overview - Background and Purpose Examination Report 50-354/98-03 and two supplements documented the NRC staff conclusion on the problematic nature of the licensee's submitted initial examination for February 1998 in meeting the examination standards. The facility had performed an " apparent cause" analysis
- (not as thorough as a root cause analysis) for the problem. Likewise, the examination report documented NRC staff conclusions on the grouping of operator written test results too close to the pass / fail demarcation of 80 percent among those who took the initial examination for February 1998. The facility did a root cause analysis for the problem. Related to both problems,' a supplement to the report documented results of a Management Meeting conducted in Region I on June 29,1998. Another supplement to this report documented the results of appeals for the February 1998 exam. As a result, all eleven applicants passed the written test based on new information provided by the applicants and all applicants except one SRO passed the operating test.: Notwithstanding these results, the NRC staff characterized the issue on the written test results on the examination as a potential program problem, along with the issue on
. the quality of the licensee's exam submittal. These problems manifested again during the -
December.1998 exam resulting in licensee and NRC staff self-assessments.
- The purpose of this inspection was to review: (1) licensee corrective actions from the February 1998 exam; and, (2) review the applicant completion and content of the initial training program.
A management meeting was conducted on Mach 22,1999 to discuss the results of the respective self-assessments (Attachmente 1 and 2). The NRC review group also developed a sequence of events for this review (Attachment 3).
As a result of four individual operator appeals of grading results, the NRC issued two additional licenses.' (Two other licenses were issued since the applicants had written test scores initially between 80-81% and their written scores got better as a result of appeals.) The preliminary and final pass / fail rates for both exams are summarized below.
Dec. 98 Exam Results:
RO - 7 total SRO - 9 total Pass.
Fail Pass Fail Operating (OP) Test
0
0 Preliminary Written Test 4*
5
Preliminary Overall Pass -
4*
5
Rate.
Appeals NA
NA
]
i Final Written Test
2
3 Final Overall Pass Rate
2
3 i
- Two had scores between 80-81% and licenses were held pending appeals.
l T
.
2 Feb. 98 Exam Results:
Pass Fail Pass Fail Preliminary OP Test
0
2 Preliminary Written Test
1
3+
Preliminary Overall Pass
1
3 l
Rate i
Appeals NA
NA
Final OP Test
0
1 Final Written Test
0
0 Final Overall Pass Rate
0
1
+0ne SRO was a retake on the written exam only.
The findings below address the results of NRC staff review of the licensee's initial training program. NRC staff conducted both onsite and in-office reviews along with a management meeting.
1. ODerations 05.1 Problem Resolution on Quality of initial Examination Submittals a.
Scope of Review (41500)
The NRC group reviewed the adequacy and effectiveness of the licensee's corrective actions for the first problem identified during the February 1998 examination. The problem was that the licensee's examination submittal to the NRC staff did not completely meet the examination standards and therefore could not be administered. That problem resulted in a two week delay. The group reviewed Condition Report No. 9801290786 and i
discussed the related actions with cognizant facility representatives.
b.
Observations and Findinas The condition report on the problem of quality of the exam submittalindicated the following apparent causes for the problem:
-
Over-reliance on contractor personnel to develop the exam.
Insufficient Management oversight to ensure a quality product was provided to the
-
NRC.
-
Relative inexperience of the training staff in the NRC exam process due to the loss of experienced personnel.
-
Training staff unsure of the level of detail. required in the simulator scenario guides and job performance measure (JPMs).
The condition report also identified the following causal factors:
,
-
3
-
Lack of effective communications of the licensee's examination team witn the NRC examination team.
-
PSE&G personnel on the exarn team lacked knowledge of regulatory requirements and process due to inexperience in preparation of NUREG 1021 (Interim Revision 8) exams.
The cor.dition report reflected the below listed licensee corrective actions for the problem.
The NRC group's assessment is also provided:
-
Determine causal factors for inadequate submittal - The group noted that this action was canceled. The apparent causes were developed in a third revision to the conditinn report evaluation (CREV) section. The facility needed to correct previous CF.EVs because they were not properly completed. However, the final evaluation was reasonably thorough in identifying a number of relevant causes for the problem from which corrective actions could be developed.
-
Meet with NRC staff examiners to discuss exam issues - The group noted that the corrective action was completed based on the management meeting of June 1998 with NRC. However, the group noted that the meeting did not focus on any specific points of disagreement or provide clarity on specific exam issues. As such, this corrective action was ineffective to address the apparent cause.
-
Provide Training on the Exam Development Process - One instructor in the i
requalification program (who eventually became the coordinator for the initial exam in Dec. 98) attended a MANTG (Mid-Atlantic Nuclear Training Group)
seminar in summer 1998 at the Susquehanna nuclear training center. However, the group noted that the seminar was designed for individuals with a high working level knowledge and experience on the examination standards. The seminar was apparently used by the licensee for " initial or basic training" in this area. As such, this corrective action was ineffective for addressing the apparent t ause.
-
Develop a staffing plan for Operations Training - This was completed, but it was unc9ar how effective it was in resolving this problem. The licensee's effectiveness review was still pending.
-
Revise Program Procedure to extend review for NRC exam development - This was canceled since the licensee representatives eventually realized that this measure would not be practical.
Attend a MANTG seminar on writing NRC initial exams - This action was
-
completed but, as noted above, this seminar was not designed as " initial" training for personnel learning the NRC exam process.
-
PSE&G to develop August 1998 Audit Exam prior to NRC exam - Thi;.was canceled for resource reasons and reassigned to a contractor. The group noted that the cancellation of this activity represented a missed opportunity to build exam development (and review skills) for licensee personne.
,
Although not specifically listed in the condition report, some action was taken by the licensee to improve site specific knowledge of contractor-instructors. To train applicants
}
for the December 1998 exam, the licensee provided some training for those contractor-instructors used and some of the contractors were previously familiar with the Hope Creek design.
The group noted that licensee personnel, who wrote or reviewed this condition report, did not ensure the compete correlation of sp ecific apparent causes with corrective actions.
For example, the apparent cause.: " unsure on level of detail for simulator scenario guides and JPMs" was never specifical'y discussed in the Management Meeting of June 29, 1998, nor was the problem cornmunicated to NRC Region I staff for more detailed discussions. The incomplete correlation of cause to corrective action appeared to be a factor on why the corrective actions for the problem were ineffective. From this review, the group concluded that the licensee was not aggressive in questioning the NRC staff regarding the required quality level of the examination standards.
c.
Conc!usions The problem on the quality of initial exam submittals recurred due, in part, to ineffective corrective actions conducted in response to the identification of the underlying causes from the Febraary 1998 exam. For example, this occurred for the corrective action
addressing the need among licensee staff to gain more knowledge and experience in the NRC examination standards.
05.2 Additional issues Related to Examination Quality a.
Scope of Review (41500)
The NRC group assessed the effectiveness of the licensee's process for review of the examination material prior to submitting the final product to the NRC staff. The review included the validation process as it related to the December 1998 exam. In addition to information gathered through interviews, the group reviewed the licensee process in facility procedure NC.TQ.TC.ZZ-0027(Z), Revision 1, " Preparation for and Administration of the Initial Licensed Operator Training Program." Also, the review included the sequence of events leading to the licensee's submittal of the December 1999 exam to the NRC Region I staff.
b.
Observations and Fir'dinas The group learned that, as a part of the licensee's project plan, the licensee had planned to conduct a psychometric review of the December 1998 exam submitted by its contractor. The action was planned as a result of the problems encountered during the February 1998 examination. Due to the lack of in-house resources and problems finding such expertise for this type of review, the licensee assigned the job to the same contractor who prepared the December 1998 exam. The group noted that the deferral of this activity to the contractor represented a missed opportunity to build exam review skills and find substantial problems before identification by the NRC Region I staf.
The licensee's validation procedure /procese does not provided details on "how to" produce and review exams for NRC staff use. Further, based on interviews, the group noted that the validation process was inconsistently applied - some licensed personnel took the test as an applicant would, while others researched each question as a part of the review. Further, operations personnel used in the process had little to no training on the examination standards.
The operations training supervisor, who h&d acquired experience from the interaction with NRC staff during the February 1998 examination, and the assistant operations manager, delayed their review of the proposed examinations. Their r.ontinuing involvement with the
{
training for the applicants precluded, by security agree; ient, earlier review of the proposed licensing examination. They were involved in getting weak applicants ready for the licensing exam. As a result, their review of the exam occurred in late October 1998.
The examination was scheduled for subn ittal to Region I on November 7,1998. Upon initial review, the facility representatives noted a number of specific problems with the contractor-developed examination. The effectiveness of the licensee's review for the contractor-developed examination was apparently impacted by the delayed timing of the licensee's supervisory review.
The coordinator for the licensing exam had some training and experience on test development process. He also attended the MANTG workshop noted above. However, his review efforts were effected by an ongoing significant workload: contractor coordinator for the audit and NRC exams; coordinator for a quickly scheduled outside assistance review group; coordinator for licensed operator requalification exams and l
related NRC inspection. The effectiveness of the coordinator's review, along with resolution of problems on the contractor-developed examination, appeared to be similarly impacted as a result of his workload.
l After the contractor resolved the licensee's comments, licensee representatives did not seek out the same licensee reviewers for further review to ensure the comments were adequately resolved. Although there didn't appear to be sufficient time to incorporate
,
l validator feedback in light of the schedule discussed above, the licensee's review process
!
does not address this type of feedback.
c.
Conclusions The licensee's problem with the quality of initial (sam submittals occurred also in the December 1998 exam due to two contributing ca mes or factors: (1) the effectiveness of l
training and operations department reviews for the December 1998 exam was limited due to a combination of schedule pressure and the workload of the responsible licensee supervisors and managers; and, (2) the licensee's validation process for exams was deficien..
.
05.3 Problem Resciution on Written Test Results a.
Scope of Review (41500)
The NRO group reviewed the effectiveness of the licensee's resolution for the second problem identified during the February 1998 examination. The problem was that a grouping of operator written test results were too close to the pass / fait demarcation of 80 percent. The group reviewed Condition Report No. 980529096 and discussed the related actions with cognizant facility representatives. Also reviewed were:
NRC written results in comparison to the performance results of the applicants
-
throughout the initial training program in 1998 leading up to the exam in December
'i998.
-
A sampling of initial program lesson plans and tests for quality. The sample selected was based on an independent review of the licensee's test item analysis for the December 1998 exam, b.
Observations and Findinas Unlike the above noted Level 11 causal analysis for the quality of the exam submittal, the causes from the evaluation for the problem of poor test results were more closely coupled with the corrective actions. The condition report indicated the following causes and corrective actions (CA) for the problem:
-
The candidate selection process was cursory and SROs candidates for the February 1998 were from Salem and barely met residency requirements (6 months onsite experience eligibility criteria) at Hope Creek.
CA:
Pre-class qualification cards were established (for SROs only) but implemented after the December 1998 class was selected.
The target selection process would focus on grading of skills and work experience.
'
The group, as well as licensee representatives, noted that those corrective actions would take longer than a year for the corrective actions to effectively correct the problem. Also these actions did not apply to RO applicants who are selected based on seniority.
-
Instructor performance and Management oversight of contractors needed improvement.
l CA:
SROs from the plant were put on rotation to assist December 1998 class.
The operations superintendent was to provide oversight.
Site specific training was given for contractor E
,
,
Monthly review of contractor performance would be conducted.
Training would verify instructor qualifications.
The group noted that the licensee intends to be eventually free of the use of contractors in the initial program once sufficient in-plant licensed personnel are maintained. Again the group, as well as licensee representatives, noted that it
. would take longer than a year for those corrective actions to effectively correct the problem.
The evaluation process was not effective in identifying marginal performers.
-
CA:
A help committee was formed and was being transformed into a Training Review Board for the December 1998 applicants.
A mentoring program was established.
Candidate feedback is to be provided after each training module (there are four modules in initial licensing training program)
Based on the results of both the February and December 1998 exam, the group noted that the evaluation process continued to be deficient and the corrective actions were ineffective, overall.
The group reviewed selected Leaming Objectives and Lesson Plans. These covered systems and administrative controls for the performance of ROs and SROs. The group j
found no significant problems with the system lesson plans. However, based on interviews, the lesson plan for administrative controls was substantially revised and issued about a week before the classroom session. Further, the group noted that this particular lesson plan did not clearly distinguish expectations for ROs vs SROs and it had a substantial amount of information in it. To understand the differences in the administrative area, one needs to go into the individual administrative controls and read the distinction in RO vs SRO duties and responsibilities. Applicants interviewed reported that there was not much time allotted to the administrative area and much of it was self study. Accordingly, the expectations and standards of performance were not clearly defined in the lesson plan on administrative procedures.
The group found some problems with the facility's testing practices during the initial training program. The ROs interviewed reported being surprised by the closed book (no
'
limited references provided) on the NRC written exam. They reported there was a number of test items for which they had to answer questions from memory, especially in the administrative controls area. The group independently noted that early program tests i
relied on lower cognitive level questions and there was frequent use of open reference questions, that is, questions with particular references supplied. Tests given later in the i
I program such as for Module 111, Operating Experience, (the third of four major parts of the initial training program) appeared to be better discriminating tests. However, the Audit and NRC exams were different from the program test / exams since the former tests were closed book (no limited references provided).
l
.,
.
For the December 1998 class, the RO written test scores were nearer to 80-85 on written exams more later than earlier in the program which was indicative of a problem. Four of j
seven had an average of 80 to 85 for module lil, which also factors in the audit exam i
results. An analysis of RO standing in NRC licensing test results showed that their overall standing was fairly consistent (within one position number of each other) relative to Module lil program and audit exam standings ~. The group determined that certain ROs should have been viewed as weaker candidates as evidenced by the program written test results indicating a potential problem in passing the NRC licensing test.
The licensee's audit results similarly identified weaker candidates. However, the audit written test was only 25% of overall score in Module Ill. All personnel allowed to take the NRC December 1998 exam met program requirements of having an average score of 80% for Module 111. However, two of seven RO and one of nine SRO applicants failed the final audit test but those applicants were allowed to take the NRC exam anyway. From the above information, it would be reasonable to conclude some of the RO and SRO applicants would not pass the NRC written test. That likelihood increased with the change in forum for testing (as proposed by a contractor and accepted by the licensee) to the closed book (no limited references provided).
'
The group leamed that a management decision was made to send all applicants to the NRC December 1998 exam despite external and intemal recommendations to not do so for several applicants (RO and SRO). Both operations and training department views were factored into the decision. This decision was apparently based on placing a greater emphasis on the operational readiness of the applicants.
The group noted that, overall, the trainee evaluation process continued be deficient in not effectively identifying and remediating marginal performers.
'c.
Conclusions i
The problem on written test results recurred due to ineffective corrective actions for a deficient trainee evaluation process. However, unlike the exam quality pr'>binm, this j
problem was more complex and ree.uted moro effort than realized in edar tc prevent it i
from recurring in the December 1958 examination.
08.1 Licensee Root Cause Analyses a.
Scope of Review (40500 and 41500)
Subsequent to the management meeting of March 22,1999, NRC staff reviewed for adequacy the licensee's current root cause analyses for the problems of exam quality and written test result,
.
b.
Observations and Findinos in 1999, the analyses for both problems were Level I root cause analyses. The results identified a number of program deficiencies and inadequacies (as summarized by the facility in their meeting handout - Attachment 1 to this report). These causes explain, in part, why the problems occurred in both the February 1998 and December 1998 exams.
However, the licensee analyses fell short of fully explaining why similar problems were repeated from one exam to the other. Specifically, for the exam quality problem, the licensee identified ineffective corrective actions as a root cause. But, the analysis explained only why the corrective action for enhancing their staff's knowledge and experience with the NRC standards were ineffective - due to inadequate training. Other performance issues were not adhering to the requirements of the standards, over-reliance on the use of contract-instructors, and untimely and ineffective licensee supervisory or project management reviews of the exam. Insights were given on why those performance issues occurred but the reason why they recurred was not fully explained.
,
For the exam quality problem, the group noted that the ineffective corrective actions could have resulted from several factors: improper correlation between underlying causes and corrective actions, untimely and/or inadequate verification of completion of certain actions l
before the December 1998 exam preparation started. For the problem on written test
!
results, the group also noted that the quality of the exam material, especially for the written exam, was a factor in giving results too close to the pass / fait demarcation of 80 percent. This observation was based on the two sets of individual operator appeals processed by NRC staff in 1998 and 1999. Appeal results indicated that a number of questions needed deletion or had two correct answers for technical reasons. These results reflect the deficient quality of questions on the exam.
c.
Conclusions Both problems on exam quality and written test results recurred even though there was a reasonably thorough identification of causes. The implementation of the licensee's condition reporting process was deficient since ineffective corrective actions resulted.
Also, from the results of the licensee's 1999 review, the underlying causes of the ineffective corrective actions were not fully explained.
As noted at the management meeting of March 22,1999, these problems will be further reviewed by facility licensee and NRC staff. (Inspector Follow Item (IFI) 98-302-01)
i i
,
.
V. Manaaement Meetinas X1 Meeting Summary On March 22,1999 the respective organizations met to discuss the results of respective self-assessments.
A final exit discussion occurred on July 20,1999 by telephone between Mr. Jackson, PSE&G, and Mr. Conte, NRC, Region 1. As noted in the overview section of this report, the results of individual operator appeals were also announced.
Attachments:
1. Facility Handout for Meeting of March 22,1999
' 2. NRC Staff Handout for Meeting of MARCH 22,1999 3. Sequence of Events
,
i
,
I
L
"
,-
,
.
PARTIAL LIST OF PERSONS CONTACTED FACILITY M. Bezilla, Hope Creek Plant Manger D. Jackson, Operations Training Manager O. Havens, Operations Training Supervisor (Hope Creek)
J. McMahon, Vice President S. Thompson, Training Director NB.Q R. Conte, Chief, Human Performance and Emergency Preparedness Branch J. Wiggins Deputy Regional Administrator J. Munro, NRR R. Vogt-Lowell, NRR W. Lanning, Director, Division of Reactor Safety INSPECTION PROCEDURES 41500 Training
'
f
'
ITEMS OPENED, CLOSED, DISCUSSED
Qgened TYPE DESCRIPTION
)
.
98-302-01 IFl Licencee and NRC staff follow-up on licensee problems with the quality of initial exam materials and poor test results among certain applicants, i
i l
l l
.
.
Attachment i FACILITY MEETING HANDOUT
'
FOR MEETING OF MARCH 22,1999
,
l
,-
/
!
!
.i l
l i
l l
~
f
.
Hope Creek NRC Exam December Exam Submittal
- E We Are Responsible !
- Chronology
.
. August - Vendor Meeting
=> September / October - Vendor Submits Sample Plan and
.
Exam To PSEG November 5 - PSEG Submits Written Exam To The NRC e
November 16 - 1st NRC Meeting To Discuss Exam e
Problems -
.
- Chronology-December 3 - 2nd Meeting With NRC To Discuss Exam
Repairs December 8 - Exam Starts (1 Day Late)
e December 22 - Written Exam Given (Last Day of Exam)
e Week Of January 25 - NRC Conducts Exam Root Cause e
- Original Scope Written Exam - 59 of 125 Questions Rated Unsatisfactory
.
JPM Follow Up Questions-29 of 60 Questions Rated e
'
Unsatisfactory Administrative Questions - 9 of 20 Rated Unsatisfactory
.
Simulator Scenarios and JPMs Were Satisfactory
.
Final Review Approximately.30 of 125 Questions Were Not Satisfactory e
e. Approximately.7 of 60 JPM Follow Up Que.stions Were Not Satisfactory
. Administrative Section Was Satisfactory
. Simulator Scenarios and JPMs Were Satisfactory
.
.-
.
..
- Written Exam
. Mostly " Low Level of Knowledge" and, " Low Discriminatory Validity" Flaws
. Other Flaws Did Exist On The Written Exam
. To A Lesser Degree These Were Also Seen in The Other Parts Of The Exam
- Root Causes The Root Causes:
. PSEG's Exam Review Was Deficient
. Exam Supplied To PSEG By The Vendor Did Not Fully Meet
NUREG 1021 Regts.
. Psychometric Reviews By PSEG and Vendor Did Not Detect The Problems
- Contributing Factors
. Correct Resources
. Resource Preparation
. Management Oversight
. Structured Reviews
. Corrective Actions
!
- Corrective Actions
- Strengthen Exam Process
. Conduct Resource Loading
. Provide Exam Writing Training
. Train On NUREG 1021 Final Rev. 8
.
-* Conclusion
. We Fell Short Of The Mark
. We Can and Will Do Better in The Future
. We Will Continue To Work With You and The Industry To
,
improve The Exam Process
'
.
E
,
.
. Hope Creek December 1998 Initial License Exam Student Performance
!
Backgroun'd
. Hope Creek NRC Initial License Exam Began On December 8,1998
. There Were 9 SRO Candidates and 7 RO Candidates
. All Candidates Passed The Operating Exam l
9 of 16 Candidates Passed The Written Exam
'
l
Root Causes Exams During The Course
.
Candidate Remediation-
i i
- Contributing Causes
'
The Exam Review Process Changed The Difficulty Of The
.
Written Exam
,
'
Candidate Screening
.
.
l
- Corrective Actions Candidate Challenge
.
Remediation Process
.
Mentor Program
.
- Conclusion We Fell Short Of The Mark
.
We Will Do Better in The Future
.
We Will Continue To Work With You and The Industry To
.
improve The' Exam Process
1
.
.
Attachment 2 NRC STAFF MEETING HANDOUT FOR MEETING OF MARCH 22,1999 i
l l
l l
l
..
l.
-
-
-
_
t G
m e
t s
n n
a N
x C
m e
I E
R m
ss k
N e
s T
ee s
s E
r
-
s e
C e
A s
E e
s s
p o
f sA MH l
o p
e r
s u
k S
f l
-
G r
a P
a n
e o
S dn d
m
&
e n
e n
n g
a E
A R
o o
i s
t n
S
k g
a o
r n
i t
P
a n
t in e
a
-
m s
e s
t
,
p e
n
e n
C R
o
r e
O P
s h
i R
c t
e r
g c
r a
n u
G G
P N
M d
&
&
-
i
-
n o
C
-
-
e r
E E
R p
t S
S O
n I
P P
N
~.,, lU e
e e
e
-
.
.
-
-
-
TNEMA M
o X8 rn S
u S
E 9 R
M 9 R
n 9 EK1 N
h9
/
"%,
o1 S
'
r
-
J S Ee 3*
,
I
-
"
A Eb N
e2
t t
/
O nh
?.
RmG I
o c
-
r. '
r Ca ECe E
dM c
R ra I
h EEe c
SP D
R i
_
C O
-
_
_
-
R H
_
_
N
_
_
-
-
-
,
-
-
-
TNE M
s d
S ra S
e d
d c
n E
a e
n t
s r
S e
a s
t r
n v
m S
o e
e Ac r
n m
s o
i a
f m
m
-
-
e r
EA r
e a
o
-
x C
-
P e
-
T f
-
E C
e o
h R
s
_
t n
s S
f o
e N
y o
i t
t c
C f
a o
i n
l c
r o
a o
i P
R u
n i
w Q
a u
g t
N e
c m
n m
i i
i l
d v
a p
m a
e x
p o
r R
E A
C G
sl eU s
e lIll1l
l
.
.
e TNK M
S no E
i
,
ta S
z i
S na A
g
.
rO
.
'
F p
e u
%
L o
,
r E
G S
CRN.
N
-,
1L
.
-
s n
d o
r i
a t
l d
a e
a n
u r
l l
r a
a b
T t
e s
v m
v e
N e
o y
t r
c K
e d
a f
i s
o l
M D
r u
t a
q n
e d
e S
h n
t t
e m
S a
mu t
t E
a s
r h
y mt S
t s
s t
i a
n t
n l
i S
l o
a x
u A
u e
e se i
l s
Ri q
d b
F c
t e
a e
e r
t L
d e
e p
t e
E C
m s
c i
t n
c S
R a
s o
t i
N n
mn n
C d
a a
d d
c a
s i
R e
i
-
md a
l s
p N
wp a
m r
iH a
e d
a x
h n
f A
e T
a o
)U.
,.
e e
.,
.
- .
y s
b d
s ra n
g d
o i
n T
n t
i a
a r
t c
u N
d s
i Ed f
n s
o u
s n
m me M
i e
t o
m l
i S
b n
it o
o u
S a
c r
t
)d c
r E
i t
p n
o
'
t l
S n
p e
G p
o p
p i
(
a c
&
o S
c i
A s
f E
d s
t i
e u
m d
S e
l F
s P
s e
f f
s L
R o
o n
i.
o m
s s
E e
e p
d l
l S
p p
u e
C m
m wt c a
a o
e l
l R
x C
x l
f e
e o
e N
R f
r e
e f
mN mC f
R a
o y
o t
S b
S N
S
)
-
e e
e
s tee hs r
T d
e n
w N
a sn r
K a
r e
d l
Mo r
a r
a n
E r
i S
g g
g i
S n
l r
a o
i E
da n
tn r
i S
G g
y a
i e
c S
-
r k
il
)
o s
r p
A d
e e
p y
u w
a
'
tn b
F o
r s
s n
l e
s n
o c
L i
ia d
a (
s t
a s
f s
E o
g e
r s
t lu d
g e
n
.
S l
i c
s
-
o k
e t
o r
r o
n r
t a
R C
t e
p n
n d
o m
R o
n g
c e
it e
n y
l N
n p
d i
t p
i i
e e
l t
a a
l t
d r
u u
tAi G
Q M
n
%,d Y,f
e e
.
[
- %_
Attachment 3 SEQUENCE OF EVENTS 9/27/97-10/2/97'
Facility developed initial examination administered to 5 SRO applicants.
10/15/97
. NRC Exam Report No. 50-354/97-08 (OL) issued j
4/5 SROs nassed - 1 failed the written examination No generic performance weaknesses noted as a result of exam administration No documented problems noted on quality of licensee's submittal Details reflect four separate licensee contacts causing communications problems including the loss of some NRC comn. Me - the report also H
notes that the contacts eventually worked well with examiners.
2/2/98 NRC Letter issued to licensee on February 1998 exam delaying the exam from February 9 to 23 (for two weeks) and outlining quality problems with written exam, simulator scenarios, administrative section, JPMs and JPM Questions.
2/23-27/98 February 1998 exam administered to 5 ROs and 5 SROs. A written retake from the September 1997 exam was also administered.
4/28/98 NRC Exam Report No. 50-354/97-08 (OL) issued documenting program problem and the quality of the initial exam proposed as being inadequate No passifail decisions given since applications were under further review
)
by NRC.
5/28/98 Exam results announced to facility with copies of applicant exam reports sent to the facility training department.
6/16/98 A supplement NRC Exam Report No. 50-354/97-08 (OL) was issued documenting exam results and summarizing again program and quality issues:
4/5 ROs passed and one written test failure
.
i 3/5 SROs passed - 2 failed the operating test and 1 of the 2 failed the written also 1 SRO retake failed agai g<=.<
Issues to be addressed in management meeting of June 29,1998 were
~ identified in cover letter of the supplement.
6/29/98 Management Meeting covered the following topics as outline in above supplement:
root cause for quality problems on the submitted examination for February 1998
reason for low test scores of certain applicants and implication to LOIT program
LOIT program / procedure inadequacy on the conduct of significant
<
control manipulations (SCMs)
related to SCMs the adequacy of corrective actions from Salem events and information notice 97-67
LOIT program / procedure adequacy on how applicant's spend time on shift.
l
attention detail on the completion of NRC form 398 8/10-12/98 NRC developed exam administered to " fast track" (they had previous licenses at other BWRs) SROs 9/4/98 NRC Examination Report No. 50-354/98-301 (OL) issued - 4 of 4 applicants passed and performed well l
10/16/98 Second Supplement to NRC Exam Report No. 50-354/97-08 (OL)
documenting results of appeal, final results for the February 1998 examination, and results of management meeting of June 29,1998.
5/5 ROs passed and one written test failure won appeal @h new information provided by the applicant to Region 1.
4/5 SROs passed - 1 failed the operating test and the 1 written failure also won his operating and written test appeals with new information provided to Region I and the program branch (NRR).
Notwithstanding the above results, the Executive Summary documents exam quality problems and low test scores among certain applicants.
i
<
3..y
.
-11/5/98 For December 1998 exam, licensee delivers written test, simulator scenarios. Licensee previously obtained NRC staff concurrence to delay submitting walk-through portion of operating test (admin, JPMs and JPM questions) till 11/9/98 due to quality problems 11/9/98 Rest of exam submitted as previously agreed upon to Region I 11/16/98 Management meeting to discuss comments on written test and JPM questions.
'11/18-19/98 Telephone comments on administrative section. Preliminary review was
-
that in excess of 10% of the test items needed replacement or significant I
modification. Preliminary reviow of scenarios and JPMs indicated they
.
would be ready for validation the following week.
WK 11/23 Validation week - no major problems noted on scenarios and JPMs 12/3/98 Management Meeting on additional comments on written exam, administrative section and JPM questions.
Respective organizations agree to perform self-assessments on problems i
noted with exam quality in certain areas.
12/7/98 NRC approves administration of December 1998 exam.
12/8-21/98 December 1998 operating test administered
.12/22/98 December 1998 written test administered 12/23/98 Licensee reports by telephone raw scores on the written examination with only 5/16 passing overall.
1/7 ROs passed 4/9 SROs passed.
i Licensee agreed to review problem in self-assessment to which they previously committed.
1/21/99 After review of facility post exam comment, NRC staff prepared licenses along with failure notices to applicants. NRC staff discussed pass / fail results with licensee with written exam scores.
.1/22/99 NRC grad...g was questioned by licensee and grading errors were confirmed by the NRC staff one of which affected a pass / fail decision.
NRC regraded the exam along with doing independent checks.
NRC staff agrees to self assess the grading error problem.
Seven Licenses issued - Two additional on hold due to score between 80-81% and pending appeals.