ML20211M823
| ML20211M823 | |
| Person / Time | |
|---|---|
| Issue date: | 08/27/1998 |
| From: | Schneider K NRC OFFICE OF STATE PROGRAMS (OSP) |
| To: | Lohaus P NRC OFFICE OF STATE PROGRAMS (OSP) |
| Shared Package | |
| ML20211M800 | List: |
| References | |
| NUDOCS 9909090226 | |
| Download: ML20211M823 (7) | |
Text
F 2 mag a t UNITED STATES l
y j NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20666-0001 DATED: AUGUST 27,1998 SIGNED BY: KATHLEEN N. SCHNEIDER i
MEMORANDUM TO: Paul H. Lohaus, Program Manager State Programs FROM: Kathleen N. Schneider - ,
Senior Project Manager
SUBJECT:
ASSESSMENT OF QUALITY AND EFFECTIVENESS OF THE l IMPEP REPORTS AND PROCESS j l As chair of the review group, I have attached for your consideration the report on the Assessment of Quality and Effectiveness of the IMPEP Reports and Process as assigned per
. the January 24,1998 memorandum to James L. Bhaha, Executive Secretary, Executive i' Council. This report is in response to the December 1,1997 Program Review of State Programs, where the Program Review Committee recommended that the Program Manager propose an approach to assess the overall quality and effectiveness of Integrated Materials Performance Evaluation Program (IMPEP) reports.
Attachment:
As stated l
l l
l 1
9909090226 990830 l PDR STPRG ESGGEN i PDR i
t
m
' ASSESSMENT OF QUALITY AND EFFECTIVENESS OF THE IMPEP REPORTS AND PROCESS INTRODUCTION in the December 1,1997 Program Review of State Programs, the Program Review Committee recommended, and the Executive Council approved as an action item, that the Program Manager propose an approach to assess the overall quality and effactiveness of the Integrated Materials Performance Evaluation Program (IMPEP) reports and process. The project manager for IMPEP coordination, OSP, as chair and selected program staff from OSP, NMSS and the Regions, informaily established a review' group' to complete this action. The Chair prepared an assessment of the IMPEP process for the NRC and Agreement State program reviews completed from November 1996 through April 1998. " Lessons leamed" from the IMPEP reviews8, recommendations made by attendees at the IMPEP training cession conducted December 2-3,1997, and IMPEP issues identified during the Organization of Agreement States (OAS) All Agreement States Meeting held October.16,1997 were considered in the assessment. This assessment was reviewed by the members of the group and their comments have been reflected into this final report.
ASSESSMENT i
From its inception, IMPEP has been an iterative process. As the program progressed from the pilot, through interim implementation to final implementation in November 1997, NRC staff has
. factored in experience, comments and suggestions to enhance the IMPEP reports and process ;
for both NRC Regions and Agreement States. The review group believes that the reviews have been effective and will continue ^to be so with revisions that have occurred, as well as those that have been identified and are planned for future incorporation into the process. The following are issues that have been identified, any actions taken to address these issues, and the present status of these actions.
- 1. ISSUE: During the' reviews conducted in 1996 and early 1997, team leaders identified the need for and requested team leadership training in order to be better equipped for the responsibility of acting as IMPEP team leaders.
ACTION: An additional one day session on team leadership was included in the December 2-3,1997 IMPEP training for all team leaders. Although the additional day was primarily for team leaders, several team members attended. The session was conducted by an NRC contractor and customized for IMPEP reviews. Based on evaluations by course
' Members of the review group included Kathleen Schneider, Chair, OSP; Lance Rakovan, OSP; George Deegan, NMSS; James Lynch, Rill; and Richard Woodruff, Rll.
21MPEP reviews were completed for 11 Agreement States (MS, CO, IL, TX, NM, NH, NV, MA,' AZ, AK, and AL), and all four Regions during this period. One follow-up review was also conducted for NE.
participants, this training strengthened leadership qualities of IMPEP team leaders.
I STATUS: Initial action completed. Periodic team leader training should be l conducted as the cadre of qualified team leaders changes.
- 2. ISSUE: During the reviews conducted in 1996, early 1997, and in the January 31, 1997 briefing of the Commission, the need for more timely issuance of
)
1 reports that document IMPEP reviews of NRC Regional and Agreement )
State materials programs was identified. Under the then existing )
approach, delays had resulted in reports not meeting the original timeliness goal of 90 days.
ACTION: Guidance addressing the timely issuance of IMPEP reports was issued to staff and management by NMSS on February 10,1997 and OSP on ,
March 19,1997. The memoranda addressed priority of participation of the review team, timing of draft reports, availability of team members, use of laptop computers, obtainirg meeting space, and increased management oversight to improve timeliness. Since that time, with the exception of the 1997 Texas and New Mexico IMPEP reviews, IMPEP reports have achieved the current goal of issuance of all reports within 104 days from the last day of the onsite review.
STATUS: Management and staff continue to issue all reports within 104 days. A revision to the Management Review Board (MRB) procedures is planned for September 23,1998. This revision should document the existing MRB process and provide additional guidance on the timeliness of issuing final IMPEP reports.
- 3. ISSUE: NRC staff has prepared reports in 1996 and 1997 on good practices that were identified during IMPEP reviews as a way to identify and share innovative and :,iiective practices used to implement materials licensing and inspection programs throughout the nation. Region Il commented that it would be useful to identify recommendations from IMPEP reviews that may assist in identifying weakness in other materials programs.
ACTION: In connection with the Good Practice Paper for 1995-1997, NRC staff prepared a report on recommendations identified in multiple IMPEP reviews during 1995-1997. This information was provided in SP-97-081, dated November 21,1997, for use in identifying areas that could need strengthening in AEA materials programs.
STATUS: Staff should continue dissemination of a report of good practices and similar recommendations from multiple IMPEP reviews on a periodic basis. ;
i j
m
- 4. ISSUE: A procedure for Management Review Board (MRB) actions was drafted in 1994 for the pilot program. NRC Staff and the Agreement States have identified the need to update this procedure (after the 1997 Texas MRB and the 1997 OAS meeting) to reflect actual MRB practices that have been developed in implementing IMPEP, such as heightened oversight j without probation and closed executive meetings of the MRB. j
(
In the April 27,1998 SRM, the Commission directed the staff to consider '
establishing a standard period of heightened oversight after which an .
Agreement State program _still found " adequate, but needs improvement," )
would be referred to the Commission for appropriate action.
ACTION: OSP has scheduled the revision of the MRB procedures to respond to the SRM. This revision will also document existing MRB processes including additional guidance on timeliness of issuing final IMPEP reports through establishing management actions to assure timely review of draft reports and a cutoff date for incorporating new informa? ion into the final report. <
STATUS: In response to the April 27,1998 SRM, the revision to the MRB procedure {
is underway to meet a September 23,1998 due date.
l
- 5. ISSUE: 1997 Draft internal OSP Procedures for conducting IMPEP reviews and I for reviewing individual common performance indicators need to be !
finalized to address the November 25,1997 revisions to Management J Directive 5.6 and reflect IMPEP experience to date. At the 1997 OAS Meeting, several clarifications were offered by the Agreement States to enhance the IMPEP internal procedures. Procedures for the non-common performance indicators have not been drafted and need to be l developed.
In the June 29,1998 SRM, the Commission directed the staff to ensure that a State's training program is reviewed under IMPEP, regardless of the degree to which it relies on NRC-provided courses. IMPEP reviews j should identify, whenever possible, available quality training courses that j could be attended by NRC personnel and thus reduce NRC training costs. j Such information should be shared with other Agreement States for their consideration in program planning.
ACTION: OSP has begun the finalization of the 1997 draft Internal Procedures for IMPEP and the common performance indicators. The " Procedure for Reviewing Common Performance Indicator #5: Response to incidents and Allegations," was sent to NMSS and Regions for review and comment. OSP is revising the document based on these comments and preparing to transmit the document to the Agreement States for comment.
The Procedure for Reviewing Common Performance Indicator # 3:
" Technical Staffing and Training"is scheduled for revision in August 1998, l with transmittal to the Agreement States, NMSS, and Regions for )
3-i
comments by September 1998. This revision will address the June 29, 1998 SRM. Staffis preparing a schedule for completion of the remaining procedures.-
STATUS: ' OSP and NMSS should develop target dates for completion of all i procedures.
- 6. ISSUE: During the review of certain Agreement State programs in 1997 and 1998 (TX, MA, & NY) and during the 1997 OAS Meeting, both States and IMPEP team m6mbers noted that reviews of Sealed Source and Device (SS&D) programs are not performance-based as the common performance indicators.
ACTION: OSP and NMSS management should reexamine SS&D program '
evaluations to determine whether the approach needs to be more performance-based versus prescriptive as it is presently conducted. The OAS IMPEP-like evaluation of NRC's SS&D program will also provide useful information to evaluate this issue.
STATUS: At the conclusion of the OAS evaluation of NRC's SS&D program and NRC's assessment, Agreement States should be informed as to NRC's position on this indicator in writing, if changes to the approach are deemed necessary, a schedule should be established.
- 7. ISSUE: The MRB identified the need to reduce the size of the IMPEP reports in April 1998.
ACTION: OSP has revised the final IMPEP report by elimination of the State's or Region's questionnaire response from the report. For Agreement States, this information is already available in the draft and proposed final reports and is in the PDR. Also, the casework reported in appendices has been additionally streamlined to reduce the volume of the report. NMSS will effect similar changes in upcoming regional reports.
STATUS: Action completed.
- 8. ISSUE: During the 1996 and 1997 IMPEP reviews, Agreement States identified the need for NRC to maintain additional contact with an Agreement State if the interval between IMPEP reviews was extended to four years.
ACTION: OSP developed a procedure and implemented annual meetings with Agreement States for those States not receiving an IMPEP review during FY 1998. OSP is presently revising the approach to incorporate the experience from the initial implementation including alterations to meeting frequencies and will be issuing a revision to the procedure.
STATUS: Initial action completed.
4-E
- 9. ISSUE: Increase the use of video conferences during MRB meetings.
ACTION: In 1998, NRC staff from Region I,11, and lll participated in the MRB meetings for Region I, Region ll, and Alabama. Staff from the New York programs will be attending the New York MRB meeting through video conferencing on September 1,1998.
STATUS: NRC staff should continue to utilize video conferencing for both State and NRC participation in MRB meetings.
- 10. ISSUE: The transmittal letter for the draft IMPEP report requests the State or Region to focus its review of the report on the factual correctness of the information in the report. Many States, and all of the Regions, have chosen to address recommendations made by the IMPEP team at this time. With this initial response MRB discussions can be more productive and efficient.
ACTION: NRC staff has acknowledged and incorporated the States' and Regions' responses into the MRB discussions. In 1997, two of the ten State reviews did not require additional response to the final report. In 1998, three of the four completed reviews did not require a response to the final report based on the State's response to the draft report. Similarly, three of the last four Regional reviews did not require any additional responses beyond those provided to the draft reports.
STATUS: NRC staff should modify the standard letter transmitting the draft report to encourage States and Regions to respond to the recommendations, when possible, to improve IMPEP discussions and to further streamline the IMPEP process.
- 11. ISSUE: There is no specific guidance for follow-up IMPEP reviews. In the 1997 Nebraska follow-up review, the review team reviewed the status of the recommendations and made no change to the findings. In the 1998 New Mexico follow-up review, the review team performed an essentially complete IMPEP review and is recommending changes to the individual performance indicators and the overall finding for the State.
ACTION: None.
STATUS: A procedure for follow-up reviews should be developed with those to be scheduled for completion in item 5.
- 12. ISSUE: Elimination of " suggestions"in the IMPEP reports, since many are not performance based.
1
]
1 ACTION: In the 1996 IMPEP reviews, the distinction between " recommendations" '
and " suggestions" was not clear to the States, the Regions, nor to many
(
of the reviewers. Standard language was developed for 1997 IMPEP 1 reports to clarify the difference. " Suggestions" made by the team are l comments that the review team believes could enhance the State's or I NRC Regional Program. No responses are necessary. On the other hand, " recommendations" require written responses from the State or Region. . This distinction between " suggestions" and " recommendations" was discussed in the 1997 IMPEP training. Most of the " suggestions" that are documented in the IMPEP reports are not performance based.
STATUS: NRC staff will send a letter to the Regions and Agreement States inviting comments. The proposal will be discussed at the annual OAS meeting.
Then, based on the feedback from the States and Regions, we will revise procedures as necessary.
- 13. ISSUE: Massachusetts became an Agreement State in April 1997. Under the previous OSP Internal Procedure, a 3 month orientation visit and a full review were to be scheduled for 6 months after the effective date of the Agreement. NRC staff committed to this schedule for the Massachusetts I Agreement Program with its first IMPEP review scheduled 6 months after effective date of the Agreement.
ACTION: In implementation, the Massachusetts IMPEP review was conducted in January 1998, instead of September 1997, since the State had not completed enough casework to allow a complete evaluation of their performance in depth. The review team recommended that the orientation visit occur within 9 months of signing of the Agreement and the first IMPEP at approximately 18 months.
STATUS: The revision of the procedure for reviews for new Agreement States should be included with those to be scheduled for completion in item 5.
6-
-.