IR 05000530/1994038
| ML17311A593 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 01/20/1995 |
| From: | Wong H NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML17311A591 | List: |
| References | |
| 50-530-94-38, NUDOCS 9501270072 | |
| Download: ML17311A593 (28) | |
Text
APPENDIX B U.S.
NUCLEAR REGULATORY. COMMISSION
REGION IV
Inspection Report:
50-528/94-38 50-529/94-38 50-530/94-38 Licenses:
NPF-41 NPF-51 NPF-74 Licensee:
Arizona Public Service Company P.O.
Box 53999 Phoenix, Arizona Facility Name:
Palo Verde Nuclear Generating Station, Units 1, 2,
and
Inspection At:
Maricopa County, Tonopah, Arizona Inspection Conducted:
December 12-21.,
1994 Inspectors:
J.
Kramer, Resident Inspector A. MacDougall, Resident Inspector
'I Approved:
ong, C
>e
,
eactor rogects rane ate Ins ection Summar Areas Ins ected Unit 3
Special, announced, resident inspection of an incomplete makeup flow path valve alignment during a, Unit 3 reduced reactor coolant system
{RCS) inventory, operation on November 28, 1994.
Results:
The primary cause was the failure by the operating crew to implement valve alignments required by a procedure in that the control room supervisor assumed that, based on observed discussions
'between the shift supervisor and reactor operators, the valve alignments had been performed.
As a result, the control room supervisor initialed the procedure step which indicated the valve alignments were complete.
Another contributing cause of the event was that the shift supervisor did not completely read the actions necessary to designate the makeup flow paths and complete the valve alignments.
The shift supervisor believed that the only requirement was to 950i270072 950120
"PDR ADOCK 05000530
0
~O
designate the makeup flow paths and not to perform the valve alignments.
Communications among the operating crew members were poor.
The
=-
information conveyed at the preevolution briefing was interpreted differently by the crew members.
Specifically, the shift supervisor, shift technical advisor, and'wo reactor operators knew the valve alignments were not complete, while the control-room supervisor and two other reactor operators thought the valve alignments were complete.
~
Sections of the RCS draining procedure were poorly written.
The procedure included a step that required two separate actions for completion, and some of the procedure appendices required operator interpretation to perform as intended.
The shift supervisor did not believe that management needed to be immediately notified of the incomplete valve alignment when it was identified by an NRC resident inspector.
As a result, the shift supervisor continued with the draining evolution until questioned further by the NRC resident inspector.
The licensee concluded that the shift supervisor should have considered the safety implications of not following procedures during the evolution and immediately notified management.
The licensee recently revised operating standards and expected that two reactor operators continuously monitor the control room boards.
The reactor operators Felt that if they did not constantly monitor their control boards, they might be reprimanded.
Licensee management had not provided specific examples of when it would. be appropriate to turn away from the control room boards.
As a result, the reactor operators were reluctant to independently review and follow the procedure being used.
Nuclear Assurance.personnel missed opportunities to identify the problem.
The Nuclear Assurance evaluator was not present for the preevolution briefing and did not arri.ve in the control room until after the draining evolution. was in progress.
The evaluator indicated a reluctance to enter the operator work area of the control room due to the revised operations standards to limit control room activity.
As a result, the evaluator did not engage in direct observation of the draining activities (such as verification of valve lineups).
The licensee's pre'iiminary corrective actions, including procedure changes, management reemphasis of notification requirements for safety significant events, and counseling of the individuals involved in the event were appropriat b
~O iO
The licensee's event investigation into the root cause of the event was thorough and objective.
The licensee's review of the event captured the concerns and weaknesses identified by the inspectors.
Licensee management involvement in the review of the event was evident,
~
The licensee's probabilistic risk assessment (PRA) group performed an evaluation to determine the relative safety significance of not aligning the makeup flow paths as required.
The evaluation supported the licensee's and'he inspector's initial assessment that this event had high safety significance.
Areas Ins ected Units
and
No inspection of Units
and 2 was performed.
Results Units
and
Not applicable.
Summar of Ins ection Findin s:
~
One violation for a failure to follow procedures in accordance with Technical Specifications 6.8. 1 was identified (Section 3. 1. I).
Attachments:
~
Attachment 1 - Persons Contacted and Exit Meeting
~
Attachment 2 - Acronyms
0
~~
DETAILS
EVENT SUNHARY On November 28, 1994, at 9: 15 a.m.,
Palo Verde Unit 3 was in Mode 5, and plant operators were preparing to enter a reduced inventory condition (RCS level less than 111 feet).
RCS level was to be lowered to midloop using Procedure 430P-32216,
"RCS Drain Operations," for installation of steam generator nozzle dams to support midcycle steam generator tube eddy current inspections.
The draining procedure requi,red two independent RCS makeup flow paths to be available and al.igned prior. to entering reduced inventory:conditions.
Two f'low paths had been selected by the shift supervisor pri'or to entering reduced inventory conditions:
(1) gravity flow from the refueling water tank, and (2)
pumped flow from a high pressure safety injection pump.
However, the appropriate procedure appendices to align the makeup flow paths were not completed prior to entering reduced inventory conditions.
Operators drained'he RCS to 103 feet 8 inches and then stopped draining to allow levels to equalize.
During the period when draining activities were temporarily suspended, an NRC resident inspector.
entered the control room to monitor the evolution.
The inspector identified, by control board valve indication, that the specified makeup flow paths were not properly ali'gned.
The inspector questioned a reactor operator who was assigned to monitor shutd'own cooling about the makeup flow paths.
The reactor operator was aware of one of the two required makeup flow paths but was not aware of the existing valve alignment.
After some discussion between the inspector and shift supervisor regarding the valve alignment requirements in. the appendices of the procedure, the shift supervisor, acknowledged the error in not aligning the makeup flow paths.
The shift supervisor directed that the appendices for aligning the makeup flow paths be completed.
After the makeup flow path alignments were complete, the shift supervisor intended to continue with the draining evolution.
The inspector was concerned that the shi,ft supervisor was proceeding with the evol'ution without informing licensee management of the error.
The inspector contacted licensee management who then placed a hold on the draining evolution and promptly initiated an
.investigation of the incident.
NRC regional management held a telephone discussion with the licensee to discuss immediate corrective actions.
The licensee conducted a detailed review of the procedure to ensure that all of the appropriate procedure prerequisites were completed.
The oncoming operating crew was briefed on the event including the safety basis for ensuring that makeup flow paths were avai-lable.
The oncoming operating crew was also briefed on the importance of stopping any evolution if any. questions or problems arose.
After the immediate corrective actions were completed and after crew turnover, the licensee recommenced draining the RCS to midloo PURPOSE OF INSPECTION The purpose of this special inspection was to review the circumstances of the incomplete makeup flow path valve al,ignment and to assess the licensee's response to the event.
The inspectors, interviewed personnel from the Unit 3 operating crew,. the shift technical advisor (STA), the outage manager, and the Nuclear Assurance evaluator involved in the event.
All personnel were cooperative and candid during their interviews.
The inspectors reviewed Procedure 430P-32Z16,
"RCS Drain Operations,"
and other supporting information to determine whether i@adequacies in the procedures contributed'o the 'event.
The inspectors also performed an independent evaluation of the licensee's incident investigation.
EVENT FOLLOWUP (92901)
3.1 Shift Su ervisor Performance 3. 1. 1 Procedure Use and Crew Briefing The inspector found that the shift supervisor had not completely read all actions associated with a procedure step prior to briefing the operating crew on the draining operation.
Additionally, the inspector concluded that the conduct of the preevolution briefing was poor because all of the operators did not participate in discussions of the makeup flow paths.
These two items were major contributors to the operators not aligning the required makeup flow paths.
Step 7.2.15 of Procedure 430P-3ZZ16,
"RCS Drain Operations,"
directed the shift supervisor to designate two makeup flow paths and to complete the appropriate procedure appendices to align the flow paths.
The shift supervisor only read the first part of the procedure step to designate the makeup flow paths and did not thoroughly review the procedure appendices which required alignment of the paths.
As a result, the shift supervisor believed that the alignments did not need to be completed.
During the preevolution briefing for entering reduced RCS inventory, the shift supervisor addressed makeup flow path requirements contained in the procedure.
The shift supervisor generally discussed the flow paths with the entire control room staff and engaged in a detailed discussion of the necessary alignments, if needed, with two of the reactor operators and the STA.
The control room supervisor and two other reactor operators, who were assigned to monitor RCS level, were not involved in the detailed discussion.
Therefore, the control room supervisor and two reactor operator's were not aware of the specific manipulations necessary to initiate makeup flow or the requirement to align the makeup flow paths prior to entering reduced inventory conditions.
The inspector found that the training department performed midloop training during licensed operator requalification training held, from September to October 1994.
The shift supervisor indicated that he had attended the training.
The inspector determined that the training specifical"ly covered the
0
requirement to align RCS makeup paths by completing the appropriate appendices prior to entering a reduced inventory condition.
The inspector concluded that a training deficiency for midloop operations did not exist.
The inspector reviewed the Unit 3 midloop briefing package provided by the training department and used by the shift supervisor during the preevolution briefing.
The inspector concluded that the briefing package was adequate.
The licensee's investigation concluded that the shift supervisor did not conduct an adequate preevolution briefing prior to entering reduced inventory and did not 'read the procedure thoroughly.
The licensee also concluded that the shift supervisor did not ensure that all the control room crew members adequately understood the information discussed during the briefing.
As corrective action.
the licensee planned to address prejob briefings as part of licensed operator continued training.
Specifically, during industry events training, the licensee committed to include the necessity of formal prejob briefings and the potential consequences of improperly conducted briefings.
In addition, the licensee indicated that all Unit 2 operators will receive this training before the upcoming Unit 2 refueling outage in February 1995.
Licensee management also reinforced their expectations regarding verbal communications and work practices to the shift supervisor.
The inspector concluded that the licensee's actions to address shift supervisor procedure use and crew briefing were adequate.
The inspector will monitor the performance of preevolution briefings during future routine inspections.
3. 1.2 Hanagement Notification The inspector concluded that the shift supervisor did not meet management's expectations for prompt notification of site management.
The inspector also observed that shift supervisors have not demonstrated sensitivity to notifying upper management in previous events (Unit I excessive draining of the RCS and Unit
RCS dilution event).
After the inspector informed the shift supervisor of the problem and the valve alignments were then completed, the shift supervisor did not believe there was an immediate safety concern and he considered the plant in a safe condition.
The shift supervisor made a courtesy call to the operations department leader and left a message when he was unable to directly contact his manager at home.
The shift supervisor did not attempt to notify the site shift manager of the procedure problem because he believed the site shift manager was only responsible for the operating units. (Units I and 2).
As a result, the shift supervisor did not directly discuss the problem with operations management and chose to continue with the draining evolution.
The NRC resident inspector determined that licensee management should be informed of the event and proceeded to inform onsite manager A week earlier, on November 21, as part of the licensee's reengineering program, the position of site shift manager was established.
The si,te shift manager position was establ'ished to ensure that senior management was promptly informed of problems.
The position is staffed 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> a day, and the site shift manager reports to the Director. of Operations.
The shift supervisor had attended briefings that explained the function of the site shift manage'r.
Also, the site shift manager was in the control room the morning of the event and emphasized that he was available if there were any problems during the midloop evolution.
The licensee determined that the shift supervisor did not consult written guidance pertaining to management notification because he believed his courtesy call to the operations department leader was an appropriate notification.
Written guidance included the licensee's sensitive issues manual and event reporting manual'.,
These manuals had a.list of specific items which require immediate management notification, but did not specifically cover this type of event.
The sensitive issues manual had. general guidance to notify management when a problem is identified and ensure the responsible parties that have ownership of the issue are addressing it.
The licensee concluded that the shift supervisor did not adequately understand existing policy guidance and management expectations concerning notification of management.
The licensee also concluded that the shift supervisor should have considered the safety impl,ications of not following procedures during a
midloop operation and immediately notified management.
The Director of Operations issued a letter to all. operators describing the philosophy of when senior management needed to be promptly notified of issues.
The letter indicated that the plant is normally operated within a set of limits defined by the licensing basis, which primarily consists of the plant Technical Specifications, and the operating basis, which primarily consists of the operating procedures.
The Director of Operations'xpectation was that whenever the plant was operated outside of these limits, even if immediate action is taken to restore the condition, then the si.te shift manager, at a
minimum, needed to be notified.
The licensee informed the inspector that the shift supervisor offered to
.present a briefing concerning the event to all shift supervisors and control room supervisors.
At the exit meeting the licensee agreed to include the briefing in the overall corrective actions.
The inspector concluded that the licensee's corrective actions were good and that the new position of site shift manager should ensure that senior management was promptly informed of safety significant issues.
3.2 Control Room Su ervisor Performance During the preevolution briefing, the control room supervisor, who would directly control the draining operation, overheard the discussion among the shift supervisor and reactor operators concerning the val,ve alignments.
The control room supervisor did not question the discussion and assumed the
0
procedure appendices for the valve alignments had been completed.'he control room supervisor also assumed that the alignments had been completed based on the shift supervisor's initi'als,in the procedure at the step which directed the alignments..
The control room supervisor did not direct nor concur with the performance of the valve alignments; however, he initialed'he procedure step for completion of the alignments.
The inspector concluded that one of the primary causes of the event was that the control room supervisor assumed the valve alignments had been performed.
The inspector concl.uded that the'ontrol room supervisor missed several opportunities to verify that the makeup flow path alignments were not c'omplete.
The valve alignment appendices were part of the procedure and easily available for reference by the control room supervisor, but were not consulted'lso, the control room supervisor did not observe the control board valve positions which clearly indicated that the makeup flow paths, were not properly aligned.
The inspector concluded that the control room supervisor did not adequately direct the evolution.
The licensee determined that the control room supervisor did not adequately maintain command and control of the evolution.
In addition, the licensee determined that the control room supervisor did not adequately attend the preevolution briefing because he was distracted by several phone calls during the briefing.
The licensee reinforced management's expectations for verbal communications and work practices to the control room supervisor.
The licensee will address prejob briefings and communications as part of the licensed operator's continued training (Section 3.I.I).
The licensee's investigation also highlighted the loss of command and.control as a significant management concern.
At the exit meeting, licensee management emphasized their concern over the lapse of command and control and that the site shift manager would be continuously assessing their expectations in this area.
3.3 Reactor 0 erator Performance On October 19, 1994, licensee management implemented new standards and principles to improve the control and formality of operations, and the inspector has subsequently observed a general improvement 'in control room operations.
However, while interviewing personnel after this event, the inspector observed that some of the reactor operators were not completely comfortable with the new standards and principles of operations.
The inspector found that the reactor operators designated to monitor the control boards felt that they would be criticized if they turned their backs to the board. to review the ongoing procedure.
The inspector was concerned that the reactor operators were not effectively backing up the overall control room team.
The inspector concluded that although the reactor.
operators'pprehension to review the procedure while monitoring the control boards did not directly contribute to the valve alignment error, it represented a missed opportunity for identifying the erro ~I!
~O
The licensee also determined that the reactor operators had'iscomfort with the new operation standards and principles, and the inspector discussed this issue with the Unit 3 operations department leader..
The operations department leader issued a night order to all three un'its emphasizing management's expectation. that the reactor operators assigned to. monitor the control boards were expected to be knowledgeable of the procedures in progress while monitoring the co'ntrol boards.
In addition, the executive vi.ce president directed line management to attend simulator training and reinforce management's expectations concerning operator performance.
The inspector noted that the licensee intended to review the effectiveness-of the new operations standards in February 1995, as part of a routine followup.
The inspector concluded that the licensee's actions to address reactor operator performance were adequate.
3.4 STA Performance
On 'the day of the event, the STA was also fil'ling the position of the midloop operations coordinator (HLOC).
The HLOC was responsible for reviewing scheduled work that might affect the availability of shutdown cooling or the integrity of the RCS, as well as the normal STA duties of accident analysis and providing advice to the shift supervisor during the draining evolution.
The STA attended the preevolution briefing conducted by the shift supervisor in preparation for entering reduced RCS inventory.
The shift supervisor included the STA in the discussion concerning the makeup flow paths and the valve alignments necessary to complete the flow paths.
As a result, the STA was aware of the available makeup paths and how to inject water into the RCS in the event of a loss of inventory.
The licensee determined that before and aFter the preevolution briefing the STA spent most of his time in a room adjacent to the control room reviewing work orders, one of his duties as HLOC.
As a result, he was not always aware of when changes were being made to the RCS while in a reduced inventory condition.
The STA did not attend a second control room.briefing prior to draining the RCS from 103 feet 8 inches to midloop because.
the control room supervisor did not inform him of the briefing.
The licensee determined that the STA was challenged by the number and scope of his responsibilities and did not place a high enough priority on maintaining an awareness
.of reduced inventory operations.
In this particular event, the inspector had some assurance that the STA was. aware of the alignment of the makeup flow paths and could have helped the control room staff if a loss of inventory occurred.
The licensee's corrective actions included reinforcing that the STA's priorities, even i.f filling the.position of the HLOC, were safety first and administration second.
Licensee management also indicated that during midloop operations additional STAs would be assigned if one person was unable to keep up with the administrative workload.
Operations management also emphasized-
'0
-10-that shift supervision should include the STA in any evolution that changes the overall plant condition or in discussions of safety significant questions.
The inspector concluded that these corrective actions were appropriate.
The inspector concluded that the event highlighted some previously identified weaknesses in the implementation of the STA program such as lack of
'nvolvement in briefings and in significant evolutions.
Although these weaknesses did not directly contribute to the event, the inspector was concerned that previous licensee actions to correct these weaknesses were not fully effective.
The licensee shared the inspector's concern and agreed to continually assess STA effectiveness and their involvement in significant evolutions.
3.5 Nuclear Assurance Observations A Nuclear Assurance operations (NAO) evaluator had been tasked with observing-the conduct of reduced inventory operations.
Based on an interview of the evaluator, the inspector identified that the evaluator missed two opportunities to have questioned the conduct of the evolution:
(1) the evaluator did not attend the preevolution briefing where he could have questioned its adequacy.
and, (2) the evaluator was reluctant to move into the operator's work area in the control room to directly observe the evolution.
The licensee agreed with the inspector's findings and concluded that the NAO evaluators exhibited inadequate oversight because they were not present for the preevolution brieFing.
The licensee also determined that the NAO evaluators were overly influenced by the new principles and standards for operations in that they avoided spending time in the operator's work area in order not to interrupt operator activities.
Nuclear Assurance management reemphasized to the NAO evaluators that they are expected to monitor activities in the operator work area in a manner not to interfere with control room operations'he operations department leader informed the control room staff that, the NAO evaluators will be spending more time in the operator work area to observe operator performance.
The inspector subsequently questioned an NAO evaluator about control room observations and found that the NAO evaluators feel they now have greater flexibility in control room movements.
The inspector noted an increase of NAO evaluator presence in the control room, but will continue to monitor the effectiveness of the NAO evaluators in future inspections.
The inspector concluded that the licensee actions to address NAO evaluator performance were adequate.
PROCEDURE WEAKNESSES (42700)
The investigation team identified that weaknesses in Procedure 430P-3ZZ16,
"RCS Drain Operations."
contributed to the shift supervisor's decision to not perform the appendices aligning the makeup flow paths.
The licensee concluded
I'
't
~O
that Step 7.2. 15 was poorly. written because the single step directed performance of two separate actions.
One of the actions was for the shi.ft supervisor. to select the makeup flow paths, and the other action was to complete the appropriate appendices to establish the valve alignments.
The licensee also found that the first section of the, appendices was required for aligning the makeup path and completion of the entire appendix would have actually injected water into the RCS.
Operations standards changed the RCS draining procedure to separate the step requiring two actions and to clearly indicate which portions of the appendices are required to be completed.
The licensee also initiated a Level 1 action plan to review all general operating procedures and identify areas that required simplification.
This action plan would include a detailed review of
.the RCS draining, procedure.
The licensee intended to complete the development of the action plan by tlarch 1995.
The inspector noted that the licensee's corrective actions appeared to be appropriate.
However, the inspector was concerned about the adequacy of the licensee's overall procedure review process.
This concern was based on several observations.
First, the operations procedure writer's guide stated that there should not be more than one action.for each step.
However, previous revisions to the RCS draining procedure had not identified that a
critical step did not follow this. guidance.
Additionally, the inspector recently identified several weaknesses in the RCS draining procedure during a
Unit 2 midcycle outage in September 1994.
Operations standards corrected the immediate problems.
but did not,identify that the procedure still needed simplification.
At the exit meeting, the licensee stated that the overall procedure review process would be evaluated during the review of the general operating procedures.
PRA (92903)
The licensee's PRA group performed an evaluation to determine the relative safety significance of not aligning the makeup flow path as required by the RCS draining procedure.
The evaluation supported the licensee's and the inspector's initial assessment that this event had high safety significance.
The inspector reviewed the evaluation and noted that the core damage frequency for midloop operations was the product of the probabi:lity of the initiating event and the human reliability probability.
The dominate initiating event was a loss of RCS inventory.
At the time of the event (three days after plant shutdown).
the time to boil was 18 minutes and the time to core uncovery and subsequent core damage was 100 minutes.
The licensee determined that the confusion with the valve lineups increased the probability oF. operators not being able to properly align the makeup sources within 100 minutes from 1 in 1000 to between 1 in 100 and 1 in 20.
The PRA group concluded that this event exposed the unit to an evolution of high safety significance that was approximately 5 to 70 times the risk of a midloop operation performed in accordance with the procedur !I
-12-The inspector concluded that the PRA review was adequate and confirmed the high safety significance that was placed on this event by both the licensee and the inspector.
EVALUATION OF LICENSEE'S INVESTIGATION The licensee formed an incident investigation team led by the Unit 3 operations department leader.
The inspector reviewed the investigation report and discussed the findings in detail with the investigation team members.
The team identified that the crew weaknesses were primarily caused by poor implementation of management expectations.
The inspector found that the team thoroughly developed the basis for the weaknesses and that the corrective actions were properly focused and detailed.
The team also identified several programmatic issues and developed good corrective actions for these issues.
Overall, the inspector concluded that the licensee performed a very good investigation of the even ~O
~O
ATTACHMENT 1
Persons Contacted 1.1 Arizona Publ'ic Service Com an R. Buzard.
Primary Plant'vent Investigator, Nuclear Assurance R. Fullmer.
Department Leader, 'Nuclear Assurance W. Hartl.ey. Staff. Nuclear Executive Admin W. Ide, Director, Operations A. Krainik. Department Leader, Nuclear Regulatory Affairs D. tlarks, Section Leader, Nuclear Assurance G. Overbeck, Assistant to the Vice President Nucl.ear Production K. Roberson, Senior Engineer..
Nuclear Regulatory Affairs W. Stewart.'xecutive Vice.President, Nuclear O'. Taylor. Department Leader, Unit 3 Operations P. Wiley, Department Leader, Operations 1.2 NRC Personnel K. Johnston, Senior Resident Inspector O'. Garcia, Resident Inspector J.
Kramer, Resident Inspector A. (1acDougall.
Resident Inspector 1.3 Others J.
Draper, Site Representative, Southern California Edison F.
Gowers, Site Representative, El Paso Electric Personnel listed, above attended the exit meeting held on December 21, 1994.
EXIT NEET ING An exit meeting was conducted on December 21, 1994.
During this meeting, the inspectors summarized the scope and findings of the report.
The licensee acknowledged the inspection findings documented in this report.
The licensee did not.identify as proprietary any information provided to, or reviewed by, the inspector I II
ATTACHMENT '2 ACRONYNS NLOC
'NAO PRA RCS STA midloop operations coordinator Nuclear Assurance operations probabilistic risk assessment reactor coolant system shift technical advisor
0, I
~O
~li