ML11243A056
| ML11243A056 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 02/03/2011 |
| From: | Southern California Edison Co |
| To: | Office of Information Services |
| References | |
| FOIA/PA-2011-0157 | |
| Download: ML11243A056 (17) | |
Text
Unit 2 System Status: ATCO NOCO ORAS User Status: CRTD MRC STA ESCP ACRC SON(.;S Notification: NN 200706227 ~IIIIIIIIIIUIIi
==
Description:==
Diesel Lube Oil Dipsticks Created on: 12/14/2009 Reported By:
Responsible: 1(b)(6) 7 Priority: 4 Medium Required Start: 02/03/2010 12:53 End: 06/02/2010 12:53 Order No: 800425078 Code:
Task Exists? [Y]
Func.Loc.: S2.EDGS.S22420MG003 EMERGENCY DIESEL GENERATOR 2MG003 Equipment:
Assembly:
Quality Class: II Location: DG Room: 107 Elevation: 030 Column: 0000 Planner Group: Maint Prod Support WorkCenter: MP Maintenance Planning Plant: 1000 SONGS - Services Reliability Classification: CRITICAL-A ARC Review Status: C Completed Feedback Req'd? [I M Rule: Sig Level: 4 Low Level Issue Breakdown (] Malfunction Start: 12/15/2009 07:07 Breakdown Duration: H End:
==
Description:==
Dipsticks for the V2 Train B diesel engines are mislabeled, this S 12 / 1 4/ 2 0 0 ..............
resulted in the dipsticks being installed in the wrong engines. This is based on the levels from LOW to FULL RUN being reversed from what is required.
Print Date: 05/18/2010 Page 1 of 17
Task Summary SONGS Notification: NN 200706227 Func.Loc.: S2.EDGS.S22420MG003 EMERGENCY DIESEL GENERATOR 2MG003 Location: DG Room: 107 Elevation:030 Column: 0000 Sort No.: 0001 Code Group:N-TS-IOD Immediate Operability Determination Short Text: no DNC..NMO800425078 restored Task Code: N045 IOD-Equipment Restored to Operable WorkCenter: EMSYA Auxiliary Systems Responsible: i(b)(6)
Sort No.: 0002 Code Group:N-EOC Extent of Condition Assessment Short Text:
Task Code: E030 EOC Complete WorkCenter: EM MA Auxiliary Process Responsible: (b)(6)
Sort No.: 0003 Code Group:N-POD Prompt Operability Determination Short Text: deleted per Engr request Task Code: P050 POD Cancelled WorkCenter: 0_OM Manager - Plant Operations Responsible:()()
Sort No.: 0004 Code Group:N-RPT Reportability Assessment Short Text: Perform Reportability assessment Task Code: RP65 RPT Complete Not Reportable WorkCenter: R C Regulatory Compliance Responsible: (b)(6)
Sort No.3 0005 Code Group:N-SPT General Support Record Short Text: Provide method to make EDG fully qualifi Task Code: ST04 SPT Complete WorkCenter: OOM Manager - Plant Operations Responsible: i(b)(6)
Sort No.: 0006 Code Group:N-PRO Procedure Change Request Short Text: -E- S023-I-8.5 Update Procedure Task Code: PR30 PRO Change Complete WorkCenter: MP Maintenance Planning Responsible: (b)(6)
Print Date: 05/18/2010 Page 2 of 17
I Trending Worksheet SON(;S Notification: NN 200706227 Func.Loc.; S2.EDGS.S22420MG003 EMERGENCY DIESEL GENERATOR 2MG003 Part: Damage:
Cause: Activity:
Part: Damage:
Cause: Activity:
Part: Damage:
Cause: Activity:
Part: Damage:
Cause: Activity:
Print Date: 05/18/2010 Page 3 of 17
Task Details SONGS Notification: NN 200706227 Func.Loc.: S2.EDGS.S22420MG003 EMERGENCY DIESEL GENERATOR 2MG003 Location: DG Room: 107 Elevation:030 Column: 0000
'rtn NoA.: 00!
Sort No.: 0001 Code Group: N-TS-IOD Immediate Operability Determination Short Text: no DNC..NMO800425078 restored Task Code: N045 IOD-Equipment Restored to Operable WorkCenter: EMSYA Auxiliary Systems Responsible: '(b)(6)
Status: TSCO Planned Start: 01/25/2010 Planned End: 01/25/2010 Complete: 12/29/2009 11:22 Task Long Text:
NOTES:
- 1) Parts 1 through 4 will be completed by the STA.
- 2) Part 5 may be completed by Operations (STA) or Engineering (Responsible Engineer) when the SSC has been restored to a fully qualified status, IOD (Immediate Operability Determination)
This is an Immediate Operability Determination (IOD).
- 1. Deficiency Identified and the Affected Functional Location:
- 2. Identify the Specified Safety Function(s); include mission time (if applicable):
- 3.
Conclusion:
Determine OPERABLE/INOPERABLE Operable
_ x Inoperable Basis (provide discussion):
EDG is inoperable in this condition because the level of the engine oil in relation to the Tech Spec minimum can not be reliably obtained.
- 4. Extent of Condition (Required for Inoperable)
Page 4 of 17 Print Date;Date; 05/18/2010 05/18/2010 Page 4 of 17
I Task Details Continued SoNGS Notification: NN 200706227
==
Description:==
Diesel Lube Oil Dipsticks EOC Created (YES or NO)? yes Describe "other train/other unit" findings (if performed):
- 5. IOD Closure Information added to 2-edmr-2009-0184 NMO complete. Oil dip sticks installed per configuration. No DNC.
component restored for operability.
Bharat Mehta (PQS T3ENI3 expires 12/16/2010 Reviewed by i(b)(6) (PQS T3ENl3 expires 9/25/2010) i ... . .. ............. ....... . . . .....
Print Date: 05/18/2010 Page 5 of 17
j Task Details SONGS Notification: NN 200706227 Func.Loc.: S2.EDGS.S22420MG003 EMERGENCY DIESEL GENERATOR 2MG003 Location: DG Room: 107 Elevation:030 Column: 0000 Sort No.: 0002 Code Group: N-EOC Extent of Condition Assessment Short Text:
Task Code: E030 EOC Complete WorkCenter: EMMA Auxiliary Process Responsible: (b)(6)
Status: TSCO Planned Start: 12/30/2009 08:55 Planned End: 01/26/2010 Complete: 12/30/2009 08:55 Task Long Text:
EOC (Extent of Condition Assessment)
Perform an Extent of Condition (EOC) evaluation for an identified deficiency to determine if the deficiency currently exists elsewhere. Do not use this method to determine cause; use a DCE, ACE or RCE as appropriate to determine the cause.
If the EOC is being used to not delay completion of an Immediate Operability Determination or a Prompt Operability Determination, consider only "other train / other unit" applications.
For all other EOC uses, determine and evaluate scope as necessary.
Refer to S0123-XV-52 and S0123-XV-50 CAP-3.
- 1. Affected Equipment (See deficiency description or, if the deficiency description is incomplete or incorrect, describe)
The SONGS diesel generator engines have specific engine lube oil level requirements to satisfy tech specs. All site emergency diesel engines dipsticks were included in this EOC
- 2. Deficiency Identification (See description or, if the description is incomplete or incorrect, describe)
The dipsticks on the B Train Unit 2 diesel engines were swapped and mislabeled. This condition results in oil levels being maintained at levels that were not as expected.
- 3. Discussion of findings, including the basis for the conclusion:
The remaining six engine dipsticks were inspected. It was confirmed that Print Date: 05/18/2010 Page 6 of 17
jTask Details Continued SONGS Notification: NN 200706227
Description:
Diesel Lube oil Dipsticks the P/N for the 16 cylinder engine dipsticks were 8333323 and the P/N for the 20 cylinder engine dipsticks were 8367720 on all engines with
-the exception of the Unit 2 B Train machine which had the P/N's reversed. The dipsticks for the train B Unit 2 diesel are in the wrong engines and are mislabeled with respect to required level. All other engines have the correct dipsticks.
The markings on the six correct dipsticks were measured and found to be consistent with marks scribed via S023-403-12-302 FCN F12252M, F12254M, and F12255M.
Performed by: Maintenance Engineer Peer Checked by: (b)(6) Systems Engineer 12/29/2009 Print Date: 05/18/2010 Page 7 of 17 Print Date: 05/18/2010 Page 7 of 17
Task Deta!Is SONGS Notification: NN 200706227 Func.Loc.: S2.EDGS.S22420MG003 EMERGENCY DIESEL GENERATOR 2MG003 Location: DG Room: 107 Elevation;030 Column: 0000 lank natUai m Sort No.: 0003 Code Group: N-POD Prompt Operability Determination Short Text: deleted per Engr request Task Code: P050 POD Cancelled WorkCenter: O OM Manager - Plant Operations Responsible: (b)(6)
Status: TSC Planned Start: 12/16/2009 03:44 Planned End: 12/16/2009 03:44 Complete: 12/16/2009 03:44 Task Long Text:
PROMPT OPERABILITY DETERMINATION TEMPLATE (Refer to S0123-XV-52)
PART 1: DEGRADED/NONCONFORMING/UNANALYZED CONDITION A. Describe the as-found condition and the equipment affected, assuring that the problem and scope have been clearly identified.
B. If it is confirmed at this stage that no degraded, nonconforming or unanalyzed condition exists, record as such and provide justification.
PART 2: SPECIFIED SAFETY FUNCTION(S) OF THE AFFECTED SSC PART 3: BASIS FOR DETERMINING IMPACT ON SPECIFIED SAFETY FUNCTION(S)
A. Technical Basis B..Status (As Found)
Specified Safety Function(s) Satisfied Specified Safety Function(s) NOT Satisfied PART 4: CONTINUED DEGRADATION PART 5: COMPENSATORY MEASURES N/A Included (describe)
Print Date: 05/1 8/2010 Page 8 of 17
Task Details Continued SONGS Notification: NN 200706227
==
Description:==
Diesel Lube Oil Dipsticks PART 6: EXTENT OF CONDITION (Required for Inoperable)
EOC Created (YES or NO)?
Describe "other train/other unit" findings (if performed):
Page 9 of 17 Date:
Print Date: 05/18/2010 05/18/2010 Page 9 of 17
I 'Task Details SONGS Notification: NN 200706227 Func.Loc.: S2.EDGS.S22420MG003 EMERGENCY DIESEL GENERATOR 2MG003 Location: DG Room: 107 Elevation:030 Column: 0000 Sort No.: 0004 Code Group: N-RPT Reportability Assessment Short Text: Perform Reportability assessment Task Code: RP65 RPT Complete Not Reportable WorkCenter: RC Regulatory Compliance Responsible: I(b)(6)
Status: TSCO. _ _
Planned Start: 01/26/2010 Planned End: 01/26/2010 Complete: 02/01/2010 17:20 Task Long Text:
REPORTABILITY ASSESSMENT TEMPLATE NOTE: Timely assessment for reportability consideration is important.
RPT assignment due date as a function of priority is as follows based on system defaults: Pri 2/3A (7 days), Pri 3B (30 days), Pri 3C (60 days)
Reference Procedure S0123-XV-52, Attachments 14 and 15 - Engineering Elements.
Questions 1 through 4 and Engineering Peer Review, to be performed by Engineering.
- 1. When did the SSC fail or first become degraded (break, code not met, out of SR range, etc.)? An SSC can be considered "failed when found" only if there is no compelling evidence of earlier failure.
Maintenance order 996060377 was the initial order to inscribe the dipsticks with additional level information.
No other orders were found which inscribed the dipsticks after the initial site inscription for the new level information. Therefore it can be concluded that the error; i.e. swapped dipsticks, occurred in 1996.
- 2. What was the apparent cause (use engineering judgment to determine, if necessary - but also describe the basis for your judgment) of the failure or degraded condition? If appropriate, generate a cause evaluation assignment.
The apparent cause of the improper dipsticks in the unit 2 B train diesel is the lack of a referenced part number for the dipsticks in the Print Date: 05/18/2010 Page 10 of 17 Print Date: 05/18/2010 Page 10 of 17
I Task Details Continued SI.)N(;S Notification: NN 200706227
==
Description:==
Diesel Lube Oil Dipsticks design or work documents used to implement the inscribing of the dipsticks. The 16 cylinder engine dipstick part number should have been listed as 8333323. The part number for the 20 cylinder engine dipstick should have been listed as 8367720. These numbers correspond with those of the other three generator sets.
- 3. Would the SSC have been able to fulfill all its intended safety function(s) as defined in the UFSAR (reference specific sections) since the failure (Operable)? Consider all plant operating Modes and the status of other equipment. If yes, why?
A review of the as found Physical conditions reveal the following:
- 1) Incorrect part number dipsticks installed in both U2 Train B engines.
The dipsticks were swapped and mislabeled.
- 2) Due to the swapped condition oil levels marked on the dipsticks were not as required. The 16 Cylinder engine oil levels were marked higher than required by approx. one inch. The 20 cylinder engine oil levels were marked lower than required by approx. one inch.
The higher levels on the 16 cylinder increased oil inventory in the engine. This engine has been extensively operated with the higher oil levels with no detrimental operating effects as noted by the operating parameters and from the results of the routine lube oil samples performed.
The lower levels in the 20 cylinder reduced the margin between the normally maintained operating minimum oil level which is at the Full run mark and the tech spec required level by approx. one inch.
The oil addition logs for the effected diesel going back to 1996 were reviewed. For the 20 cylinder engine, 40 gallons of oil was added on 6/18/06. This is the most oil added from the time of inscription to the date of this discovery. Normal maintenance practice fills the crankcase to the Full Standby Level. Using the Calc J-KJA-012 sheet 55 for the gallons per inch quantity of oil of 22.9 represents a 1.746" addition of oil for the 40 gallons added on 6/18/06. Subtracting this 1.746' from the indicated Full Standby level of 13.1875" results in a oil level prior to the greatest add at 11.44". This is 1.6 " above the required 7 Day Minimum Volume Requirement of 10.8" A review of the oil additions to the 16 cylinder engine during the period from the initial inscription in 1996 and the date of discovery reveals the max add of 35 gallons occurred on 12/19/06. This would have resulted from a pre add level of 12.25 inches. This level is .75 inches Print Date: 05/18/2010 Page 11 of 17 Print Date: 05/18/2010 Page 11 of 17
j 'Task Details Continued SONGS Notification: NN 200706227
Description:
Diesel Lube Oil Dipsticks above the 7 Day Minimum Volume requirement of 11.5"
- 4. Did the failure or failure mode affect or potentially affect another SSC or the other unit?
The only SSC's considered for this review was the site emergency diesel engines.
Engineering Peer Review (comments):
Based on the above, there is no evidence that the oil level in the 16 or 20 cylinder engine was ever reduced to a level below the 7 Day Minimum Volume Requirement.
tEngineering
)b ..... Peer Review performed by:
(6....................
...... i i . . . . . . . . .....
Peer Checked by: (b()L1-26-10.
- 5. (BY Encode qualified personnel - see Encode 270QC7: Assessing Events for Reportability) Reportability assessment summary (include references as appropriate):
A written report to the NRC is required by IOCFR50.73 for an "operation or condition which was prohibited by the plant's Technical Specifications." Technical Specification 3.8.3 requires "The stored diesel fuel oil, lube oil, and starting air subsystem shall be within limits for each required diesel generator."
Engineering evaluation of actual lube oil levels based on as-found data and volume of oil added, determine that the Technical Specification limits were not exceeded. Therefore no report is required.
(b)(6) 1/31/10 I concur with this assessment-(b)(6) - 2/1/2010 Print Date: 05/18/2010 Page 12 of 17
j "Task Details SONGS Notification: NN 200706227 Func.Loc.: S2.EDGS.S22420MG003 EMERGENCY DIESEL GENERATOR 2MG003 Location: DG Room: 107 Elevation:030 Column: 0000 Sort No.: 0005 Code Group; N-SPT General Support Record Short Text: Provide method to make EDG fully qualifi Task Code: ST04 SPT Complete WorkCenter: 0_OM Manager - Plant Operations Responsible: .(b)(6)
Status: TSCC Planned Start: 12/16/2009 02:35 Planned End: 12/16/2009 02:35 Complete: 12/16/2009 02:35 Task Long Text:
SPT (General Support Record)
Describe the General support request:
This N-SPT is created to document the process how the Unit 2 Train B EDG will become fully functional again with the reported problem of inaccurately scribed lube oil dipstick marking in both engines.
Resolution:
The Unit 2 Train B EDG lube oil dipstick marking on either engine was found to be inaccurate on 12/14/09.
The IOD made the Unit 2 Train B EDG inoperable. A POD has been given to Engineering to provide follow-up assessment to determine if the Unit 2 Train B EDG is fully qualified with the reported condition.
The following measurements were collected on Unit 2 Train B EDG lube oil dipsticks on night of 12/14/09:
Engine #1,,,,,.
,,Measured,,S023-3-3.23,,
Full Stby - Full Run,,1.5000,,1.500,,
Full Run - TSmin,,0.1250,,0.125,,
Tsmin - INOP,,1.1875,,I.200,,
INOP - LOW,,l1.3750,,10.300,,
Low to Tsmin,,12.5625,,1l.500,,
Page 13 of 17 Date: 0~/lB/2010 Print Date: 05/18/2010 Page 13 of 17
Task Details Continued SONGS Notification: NN 200706227
==
Description:==
Diesel Lube Oil Dipsticks Low to Full Run,,12.6875,,ll.6250,,
Low to Full Stby,,14.1875,,13.1250,,
Length from bottom cap to low,,27 7/16 ....
Engine #2 ......
,,Measured,,S023-3-3.23,,
Full Stby - Full Run,,l 9/16,,1.500,,
Full Run - TSmin,,l 13/16,,1.825,,
Tsmin - INOP,,l 2/16,,1.100,,
INOP - LOW,,8 11/16,,9.700,,
,ti,,
Low to Tsmin,,9 13/16,,10.800,,
Low to Full Run,,ll 10/16,,12.625,,
Low to Full Stby,,13 3/16,,14.125,,
Length from bottom of cap to low,,27 1/2,,,,
From these measurements, it is shown that the Dipstick in Engine #1 is 1.075" higher that what is required. This finding suggests that the oil inventory in Engine #1 about 1.075" more than what is required or what is believed to be in the oil loop.
However, the dipstick in Engine #2 is 1.0125" less than what is required, suggesting at TSmin, Engine #2 will be short of required oil inventory by 1.0125". However, since the oil level is maintained above the Full Run Mark, a case can be shown the engine #2 oil inventory, while lower, would not impact the functionality of the engine, thus the EDG. The argument can be made in a POD.
This argument to make the EDG fully qualified with two incorrect dipsticks will not be pursued because the Unit 2 Train B EDG is not operable currently due to bus outage, and we should have correct dipsticks in the EDG before it returns to service.
Instead the dipsticks shall either be replaced with new or be re-scribed. The maintenance activity will take place under NMO 800425078. The dipsticks shall be marked as detailed by S023-403-12-302, DCN 15.
Print Date: 05/18/2010 Page 14 of 17
I Task Details Continued SONGS Notification: NN 200706227
==
Description:==
Diesel Lube Oil Dipsticks The N-POD will be cancelled.
References:
- 1) S023-3-3.23, Attachment 11 fill 2) S023-403-12-46 sht. 1, 2, & 3
- 5) S023-403-12-302 DCNs 14, 15, 16, & 17 Prepared by i~b)(6) 12/15/09 Peer Check by i(b)(6) 12-15-09 i -
Print Date: 05/18/2010 Page 15 of 17 PrintDate 05/8/200 Pae 15of 1
I Task Details SON(;S Notification: NN 200706227 Func.Loc.: S2.EDGS.S22420MG003 EMERGENCY DIESEL GENERATOR 2MG003 Location: DG Room: 107 Elevation:030 Column: 0000 Sort No.: 0006 Code Group: N-PRO Procedure Change Request Short Text: -E- S023-I-8.5 Update Procedure Task Code: PR30 PRO Change Complete WorkCenter: MP Maintenance Planning Responsible: 1(b)(6)
Status: TSCO Planned Start: 05/07/2010 12:47 Planned End: 06/02/2010 Complete: 05/07/2010 12:47 Task Long Text:
PRO (Procedure Change Request)
Please update the procedure S023-1-8.25, add calculation info for marking dipsticks, also add ID to dip stick to ensure the dip sticks do not get mixed up. In The correct reference procedure is S023-I-8.5, Diesel Generator Lube Oil Inspection and Addition. (b)(6)
Action was completed as requested/defined: Yes: _XNo:
The intent of this task is to ensure that the requested changes are completed as described and that a review is performed of existing documents to check for other impacts. Searches may be accomplished by using the appropriate search system, Procedure references listed in the document being modified or by author knowledge of impact of changes.
If not done as requested, then describe why and actions taken:
Identify the documents changed: S023-I-8.5 Document Number: Revision/TCN: REVISION 7 Date Issued: 5/6/2010 Identify other affected procedures not covered by this Notification and list the tracking Notification Task Numbers:
Other information required to support/clarify the action taken such as Print Date: 05/18/2010 Page 16 of 17
Task Details Continued SONGS Notification: NN 200706227
Description:
Diesel Lube Oil Dipsticks type of search used to determine extent of review:
Added Caution at Steps 6.2, 6.3, and 6.4 to ensure the correct dipstick is returned to the correct engine. Dipsticks are specific to the 16- and 20-cylinder engines.
Added calculation J-KJA-012 to Developmental Resources Added part numbers for the dipsticks in Attachment 4 r-6i ..... s/6/2010 05/07/2010 ..
~L. ..........
Performed peer review of S023-I-8.5. All requested changes have been incorporated.
Page 17 of 17 Print Date:
Print 05/18/2010 Date: 05/18/2010 Page 17 of 17