ML20205N610

Consults with Commission on Using Allegers Responses to Survey Forms to Measure Effectiveness of Allegation Program

ML20205N610
Person / Time
Issue date:	03/08/1999
From:	Travers W NRC OFFICE OF THE EXECUTIVE DIRECTOR FOR OPERATIONS (EDO)
To:
References
SECY-99-071, SECY-99-071-01, SECY-99-071-R, SECY-99-71, SECY-99-71-1, SECY-99-71-R, NUDOCS 9904190040
Download: ML20205N610 (9)

Text

N s

. RELsASEDeTO:1, PDW U i< ~ < ' ,iI o

f//y/F/ [#J  : '. '

I t l>....................e, dMe initiah ,

%,.....}

POLICY ISSUE (Notation Vote)

March 8.1999 SECY-99-071 FOR: The Commissioners i FROM: William D. Travers Executive Director for Operations

SUBJECT:

ASSESSING THE EFFECTIVENESS OF THE ALLEGATION PROGRAM FROM REVIEW OF SURVEY FORMS SENT TO ALLEGERS l

PURPOSE:

To consult with the Commission on using allegers' responses to survey forms to measure the effectiveness of the allegation program.

BACKGROUND:

l On July 6,1993, the Executive Director for Operations (EDO) established a review team to assess the NRC's program for protecting allegers frorn retaliation. The review team was established to determine whether the NRC had taken sufficient steps to create an atmosphere within the regulated community in which individuals who have safety concerns feel free to engage in protected activities without fear of retaliation. Within the framework of the review, the team solicited comments on the allegation program from the industry, the p blic, and individuals who had previously submitted concerns to the NRC. The latter group submitted a number of l

comments, such as expressing fear that their identity would be released, lack of communication I from the NRC on the status of the review of their concerns, objection to NRC's policy of referring concerns to the licensees to investigate, and dissatisfaction regarding the untimely resolution of their concerns.

I 4

Contact:

Carl Mohrwinkel, NRR

~

301-415-1293 i S'\}C

~T dh-I

~

9904190040 990308 l PDR SECY

O The Commissioners 2 in its report, NUREG-1499, " Reassessment of the NRC's Program for Protecting Allegers Against Retaliation," issued in January 1994, the review team made 47 recommendations for improving how the NRC handles allegations in general and allegations of discrimination in particular. The staff has completed action on 45 of the 47 recommendations. One of the recommendations still open concerns developing a standard form to be sent to allegers along with close-out correspondence to solicit feedback on the NRC's handling of the allegers' concerns.

In response to this recommendation, in SECY-94-089, " Response to the Report of the Review Team for Reassessment of the NRC's Program for Protecting Allegers Against Retaliation,"

dated March 29,1994, the staff committed to have the Agency Allegation Advisors (Advisors) develop and review various options to solicit feedback on the NRC's handling of allegations and note the resource impact. In addition, the staff stated that the options would address the format for soliciting feedback, the method of collection, the method of validation, who should receive the feedback, and how the information would be compiled and used.

In a memorandum to the Commission dated November 16,1994, " Status Report on the Review Team Recommendations for Protecting Allegers Against Retaliation," the staff reported that a form had been prepared to solicit feedback on the NRC's handling of allegations and would be used in a one-time trial program. After conclusion of the trial program, a decision would be made on whether to incorporate the feedback survey into the routine process for handling allegations.

Under the trial program, in December 1995, the staff sent feedback forms to 145 allegers randomly selected from allegations submitted to the regions, the Office of Nuclear Reactor Regulation (NRR), and the Office of Nuclear Material Safety and Safeguards (NMSS). The  !

staff received 45 responses, a 31 percent response rate. Of the 45 responses,26 (58%) made favorable comments on the allegation program and 19 (42%) commented unfavorably. The criticism reported most frequently was that the NRC was untimely in responding to allegations.

Other criticisms included failure to protect allegers' identities, incompetence of NRC staff, and disagreement with the agency's technical decisions.

At the time the survey was sent, the average time to close allegations was 248 days for those involving only technical concerns and 472 days for those involving wrongdoing. Before distributing the survey, the staff had made a number of improvements to the allegation process, which already had addressed the issues raised in the survey responses. Timeliness was stressed in refresher training and was emphasized during allegation review board meetings that assigned action to resolve allegations and during the 1995 annual audit. Additionally, before September 1994, the staff did not consistently acknowledge receipt of allegations in written correspondence to allegers, in response to the review team's recommendations, the allegation program had been modified to require that (1) ellegations be acknowledged in writing and -

reiterate the staff's understanding of the concerns submitted, (2) status letters be sent if the .

allegation was open more than 180 days, and (3) closure letters be sent which reiterate the concerns submitted and describe the staff's findings and rationale for closing the allegation.

During the initial contact, allegers were to be informed of the limits on NRC's ability to protect their identity and asked if they objected to the NRC's referring their concems to the licensee for

The Commissioners 3 investigation and resolution. Implementation of these improvements was verified during annual audits of the allegation program.

Because these changes had taken place too recently to have an effect on the handling of allegations covered by the 1995 survey results, the Advisors decided to survey allegers again, after the improvements had been in place at least a year in order to assess the effectiveness of the changes. The follow-up survey was scheduled to be conducted in the second quarter of 1997.

In March 1997, the General Accounting Office (GAO) issued a comprehensive report entitled,

" Nuclear Employee Safety Concerns, Allegation System Offers Better Protection, But important issues Remain," which assessed the NRC's allegation program. One of the GAO recommendations was: "To improve NRC's knowledge of the work environment at nuclear power plants, we recommend that the Chairman, NRC, ensJre the implementation of recommendations to provide information on the extent to y hich the environment in nuclear plants is favorable for employees to report health or safety hazards without fear of discrimination. This would include recommendations on... routinely providing feedback forms in allegation case close-out correspondence."

In response to the GAO report, the NRC reaffirmed its commitment to send feedback forms to another random sample of allegers. The staff also committed to consult with the Commission concerning whether to routinely enclose the feedback form in closure correspondence after the responses to this survey were analyzed and the resource implications were evaluated.

The second survey of 150 randomly selected allegers was conducted in December 1997. The staff received 44 responses, a response rate of 29 percent. Of the 44 responses,27 (61%)

commented positively about the program and 17 (39%) contained negative comments. Among the negative comments, untimely action by the staff was mentioned by four respondents, but was no longer the most frequent complaint. At the time of this survey, the average time to close allegations had been reduced to 145 days for those involving only technical concems and to 354 days for those involving wrongdoing. The most frequent complaint was that allegers either disagreed with or took issue with the agency's technical findings, the quality of the inspection or investigation, or the ultimate resolution of their allegations. No respondents expressed concern about an inappropriate disclosure of their identity.

DISCUSSION:

The staff considered three options in evaluating whether to continue surveying allegers as one method of assessing the effectiveness of the allegation program: (1) send a survey or feedback form with each closure letter, (2) continue periodic surveys of a sample of aHegers, and (3) end the survey process.

Currently, each acknowledgment letter to allegers describes the staff's understanding of each concem and requests that the alleger contact us if the description is not accurate. This is done to ensure that the staff fully understands the concems before expending resources to resolve them. Each closure letter restates each concern and describes its resolution. The closure letter also tells the alleger to contact the allegation coordinator should there be any questions concerning the resolution of the allegation. A poll of the regional allegation coordinctors

a The Commissioners 4 j determined that allegers rarely contact the agency upon receipt of closure letters. The regional I coordinators estimated that in only 2 percent of the cases do allegers contact the agency upon receipt of the closure letter. Using Fiscal Year 1998 statistics as a basis, this correlates to 25 contacts by allegers after receiving the closure letter. This is consistent with data from the allegation database on contacts by allegers after receiving a closure letter.

As one would expect, individuals who contact the program office or regional allegation coordinators (coordinators) after receiving a closure letter, are usually dissatisfied with some portion of the agency's action or conclusion. However, the percentage of allegers who express dissatisfaction after receiving a closure letter is significantly lower than the percentage of respondents who submitted negative comments in response to the survey (39%). This may be a reflection of the ease of completing the survey form and a genuine interest in providing feedback to the agency in response to the request.

Whatever the reason, the portion of responses to the survey that were critical of the allegation program was significantly higher than the response to closure letters. The 1995 and 1997 surveys generated approximately the same rate of response (31% in 1995 and 29% in 1997). If the agency were to include a survey form with each closure letter and receive a similar response rate, it would receive approximately 300 responses a year using Fiscal Year 1998 data as a basis. If the percentage of responses with negative comments continued at the same rate, an average of 40 percent from the two surveys, the number of responses with negative comments would be approximately 120. Depending on the approach chosen, reviewing how the agency handled those 120 cases received in response to the survey could result in spending more resources than are currently being expended resolving issues raised by allegers following closure of the allegations. However, the survey responses may present a more accurate assessment of the effectiveness of the allegation program.

As noted above, the staff had addressed issues that were raised by allegers in the 1994 study, but the changes were not made soon enough to affect the 1995 survey results. It would appear that the changes implemented in the allegation program in 1995 and 1996 and the training provided for NRC staff members involved in the allegation process have improved the program, as evidenced by the changes in the responses between the two surveys. The predominant criticisms from the 1995 survey (lack of timeliness, failure to protect alleger identity, and incompetent staff) were made much less frequently or were not made at all in the 1997 survey responses.' One of the reasons for continuing a survey would be to track the type of feedback we receive to see if our training and auditing programs have eliminated or reduced problems identified in earlier surveys, and to identify emerging trends that will have to be addressed in the future.

Although the difference in responses from the 1995 survey to the 1997 survey may indicate an improved program, both sets of responses contained comments expressing allegers' disagreement with the agency's technical findings. Although this was the most frequently mentioned criticism in the 1997 responses, this criticism was not unexpected since

' The inadvertent release of allegers' identities continues to be of concern. The staff recognizes that the protection of the identity of allegers .4 essentialif the allegation program is to be effective. This topic is stressed during annual refresher training for all employees involved in processing allegations.

l

J' The Commissioners 5.

approximately 65 percent of the allegations received were considered unsubstantiated.' If the Commission decides to continue using some type of survey instrument, it is expected that the

.' percentage of respondents who disagree with the' staff's resolution of their concems will remain >

about the same.

4 OPTIONS Option 1 Adopt the recommendations of both the NRC review team, expressed in NUREG-1499, ,

and the GAO, i.e., have each office or region routinely enclose feedback evaluation forms pre-addressed to the Advisors in all closure correspondence with allegers. The form will indicate that the feedback is being requested as part of a one year trial program. All responses would be reviewed by the Advisors and copies would be sent to the region or program office for, information, and for action if appropriate. As part of their review, the Adviso.rs would identify ~.

responses that appear to present new information or raise new concems and these responses

- will be reported to the region or office for further consideration to determine if additional action .

would be necessary. If additional action were to be taken by the office or region, a second closure letter documenting the re-review would be sent to the alleger, and in those cases in which substantive follow-up activity was performed, another evaluation form would be .

enclosed. If the alleger sent back the subsequent evaluation form and continued to express .

~

dissatisfaction, the Advisors would review the allegation again to determine if it had been ;

appropriately resolved. In addition, the Advisors would track and trend the survey responses to monitor the effectiveness of the allegation program. If problematic trends were identified, corrective actions would be developed and implemented.

PRGa Option 1 gives allegers the opportunity to easily and quickly provide their feedback to the agency.

Allegations that may not have been handled as well as they should have been may be identified and reviewed so that improvements to the process could be made to ensure that future allegations would be handled in a more timely manner. Additional or corrective action, if necessary, could be taken to resolve the alleger's concem.

The feedback forms would provide metrics for the staff to monitor the effectiveness of this aspect of the allegation program in a timely manner and identify and resolve problem areas as necessary, The feedback forms may provide insight on the allegation program's impact on public .

confidence in the program ant' the NRC.

4

'j'

,3

.:.w.'. l. A

l The Commissioners 6 CONS On the basis of a projection of current data, the agency will receive approximately 120 negative -

responses per year. This is a significant increase over the approximately 25 responses received as a result of the closure letters,2 and this increase could necessitate the Agency expending more resources resolving these responses.

Although it is difficult to determine how much time would be spent reviewing the allegations associated with the 120 projected feedback form survey responses, a reasonable estimate of review time by the responsible coordinator would be 1-2 hours per case. This could amount to 240 hours0.00278 days <br />0.0667 hours <br />3.968254e-4 weeks <br />9.132e-5 months <br /> of review by the coordinators. If some follow-up activity was necessary for 25 percent of these responses, then 30 allegations per year would require further agency review.

The coordinators will then schedule an allegation review board to discuss this new information and assign appropriate actions, if necessary. This action could be as simple as a case review j by the inspector or resident who initially reviewed or inspected the allegation, which could take J 1-2 Mars, or a new inspection of the allegation, which could take considerably more time. The average time to complete the follow-up of an allegation is approximately 50 hours5.787037e-4 days <br />0.0139 hours <br />8.267196e-5 weeks <br />1.9025e-5 months <br />. The effort to review 30 allegations would take 1500 hours0.0174 days <br />0.417 hours <br />0.00248 weeks <br />5.7075e-4 months <br />, slightly more than 1 FTE. The staff's concem with this anticipated effort is that other workload requirements of the coordinators and Advisors, such as reviewing Freedom of Information Act (FO!A) packages, giving policy guidance to staff 5

and management, corresponding with allegers, and resolving initial allegations might be J affected because of resource limitations. It should be noted, however, that the agency I received 23 percent fewer allegations in Fiscal Year 1998 than in Fiscal Year 1997. If this l reduced rate of receipt continues, it should be possible for the staff to perform the suivey )

I review follow-up within budgeted resources.

Ootion 2 Continue the periodic survey of 150 allegers selected randomly as was done in the two previous .)

surveys. This would allow the agency to capture a " snapshot" of the program's performance, j and compare the snapshots from year to year. The Advisors would track and trend the data,

]

and problem areas could be identified and solutions for these problems would be '

recommended. This approach has provided valuable insights in the previous two surveys; timeliness and quality of correspondence were significant issues in the first survey, but, one could reasonably conclude that, because these issues were addressed by the Advisors and coordinators, negative comments on these have greatly decreased.

2 A review of the 17 responses to the 1997 survey that were critical of the program indicates that all of them expressed some degree of dissatisfaction with the handling of their concems. (Due to the voluntary nature of the survey, we were only able to identify specific f

allegations if the allegers identified themselves or their allegation numbers. The staff was able '

to match responses to specific allegations in 16 instances.) Of these 16 allegations, 4 (25%)

needed and received further review. The issues ranged from explaining a late response to the alleger to a review of the case based upon the presentation of new information. The other 12 either simply restated their original complaint or made a general negative comment regarding NRC competence.

y i i

o i The Commissioners 7 PROS Would allow the agency to track and trend the effectiveness of the allegation program without an increase in resources. 1 Has previously been effective in identifying problem areas that led to corrections by the staff .

and improvements in the program.  !

CONS Because the survey would be performed periodically, and not at the time of each allegation closure, with a smaller alleger sample, this option would not provide as timely or as complete information to the agency.

Would not give all allegers the additional opportunity to share their observations about the allegation program with the agency. As currently structured, only very few allegers choose to respond to the closure letters, but when the randomly selected survey group received the survey form, about 30 percent responded. By performing only a random survey, instead of a survey of every alleger, we might be losing valuable information.

A random periodic survey would not supply sufficient feedback to provide information on the outcomes of the allegation program.

Option 3 Discontinue the survey instrument on the basis that the previous two surveys have provided us as much usefulinformation as we are likely to receive. Each of the previous two surveys has elicited approximately the same rate of return and approximately the same rate of negative and positive comments.

PROS in a time of diminishing agency resources, eliminating a survey that has already been performed twice, with relatively similar results, may be prudent.

The current system does invite allegers to contact the agency if, after they receive their closure letters, they believe their concerns were not adequately addressed.

It is possible that the agency has already gamered the most valuable information that is available by having already performed two surveys. Future surveys may only confirm the previous results.

C The Commissioners 8 CONS The staff would not have sufficient information to provide data on the allegation program.

A potential source of feedback would be eliminated. The agency's ability to identify potential problem areas would be reduced.

The agency could be perceived as not listening to one group of its stakeholders.

INTERNAL AND EXTERNAL STAKEHOLDERS VIEWS The draft of this Commission paper was reviewed by the regions and program offices, which provided their views on the three proposed options. In addition, the use of feedback forms was discussed with the Union of Concerned Scientists (UCS). NRR and NMSS as well as the <

regions favored Option 1.

The UCS was one of the respondents to the second survey. When the staff contacted UCS to discuss its response, UCS expressed the view that the NRC should include a feedback form with each closure letter. UCS felt that allegers would be nure apt to return a feedback form than respond to a closure letter.

RESOURCES As discussed under Option 1 " Cons", above, the total estimated number of hours to implement this option is estimated to be 1740. This amount of effort would provide 240 hours0.00278 days <br />0.0667 hours <br />3.968254e-4 weeks <br />9.132e-5 months <br /> for the Coordinators to review the survey responses, and 1500 hours0.0174 days <br />0.417 hours <br />0.00248 weeks <br />5.7075e-4 months <br /> for the regions and offices to conduct an in-depth review of 30 allegations. This would represent approximately 1.4 FTE, although this effort would be shared by the four regions, NRR and NMSS. Considering the number of allegations received in Fiscal Year 1998, and projected retumed surveys, it is estimated that each office and region will expend the following hours: NMSS-50.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />, ,

NRR-198.6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, RI--405.4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />, Rll-330.6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, Rill-360.2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />, RIV-396.7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />.

It should be noted, however, that the allegations received agency-wide in Fiscal Year 1998 were 23 percent fewer than in Fiscal Year 1997. If the agency continues to receive allegations at this reduced level, it should be possible for the staff to perform the survey review follow-up within budgeted resources. ,

RECOMMENDATIONS The staff has considered the pros and cons of the above options and has determined that Option 1 may be the most effective means of measuring the effectiveness of the allegation program. However, given the uncertainty of the amount of agency resources that may be entailed in implementing this option, the staff recommends that the Commission endorse Option 1 for a one-year trial period. At the end of one year, the staff will assess the benefits of-receiving the feedback and the impact on resources, and make a recommendation to the  !

Commission as to whether this method of surveying allegers should be permanently adopted.

I i

_.-._____m_________

J

y lq '

The Commissioners ~ 9 COORDINATION The Office of the General Counsel has no objection to this paper, and the Chief Financial Officer has reviewed this paper for resource implications and has no objections.

%h w M

William D. Travers Executive Director for Operations Comissioners' completed vote sheets / comments should be provided directly .

to the Office of the Secretary by COB Wednesday, March 24, 1999.

Commission Staff Office comments, if any, should be submitted to the Commissioners NLT Wednesday, March 17, 1999, with an information copy to the Office of the-Secretary. If the paper is of such a nature that it requires additional review and comment, the Commissioners and the Secretariat should be apprised of when comments may be expected.

DISTRIBUTION:

Commissioners OGC OCAA OIG OPA OCA CIO CFO EDO REGIONS SECY L