ML070780294
| ML070780294 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 03/14/2007 |
| From: | Synergy Consulting Services Corp |
| To: | Office of Nuclear Reactor Regulation |
| Markley, M T, NRR/DORL/LP4, 301-415-5723 | |
| References | |
| FOIA/PA-2007-0157 | |
| Download: ML070780294 (14) | |
Text
ATTACHMENT BACKGROUND INFORMATION ON SYNERGY Summary of Our Cultural Assessment Experience SYNERGY has extensive experience in conducting Nuclear Safety Cultural Assessments for the commercial nuclear industry. In this regard, we have performed more than 100 Nuclear Safety Cultural Assessments, including 44 nuclear power plant Sites, 60 nuclear power plants and 2 gaseous diffusion plants. We have developed a solid reputation throughout the industry for providing such services in a consistently high-quality manner. The US Nuclear Regulatory Commission is also very familiar with our approach and methodology. By virtue of the extent of our activities, we have developed a large industry database that is particularly usefil for industry benchmarking and that provides a meaningful frame of reference for assessment results.
How we got into the Cultural Assessment Business We entered the consulting business after obtaining operational experience on Navy nuclear submarines and after obtaining regulatory experience through service with the US NRC - in key technical and managerial positions. We were key players at the time that the Office of NRR created the Division of Operating Reactors, which focused on technical/regulatory issues that were emerging at operational plants. Over the years, we have obtained ahid maintained a keen sense of what is important to the NRC and to other industry nuclear oversight organizations.
During the period 1983-1995, our consulting practice primarily included the following types of engagements:
- 1. High-level support to Utility/Site management in situations where the Utility/Site had encountered significant regulatory-related difficulties in obtaining their operating licenses. Most of these difficulties had their root in management issues, but typically manifested themselves in programmatic design, construction and/or quality assurance/control issues. Most of these situations involved breakdowns in Utility-Regulator communications and in regulatory trust & confidence in Utility/Site management. Most of these situations involved problems with what we now recognize/label as "organizational culture". Some of these situations also involved problems identified by "whistleblowers" and/or by other allegations submitted to the NRC. Our typical role - on behalf of the utility - was to assess the situation and then to assist in the development of strategies, plans and actions to recover from the situation. In most cases, we were directly involved in the implementation of the strategies and plans -
most often in an on-Site mentoring/coaching/observing/reinforcement role. In a few cases, we served in a role of independent oversight of the implementation. In all of these engagements, we spent almost all of our time on-Site.
- 2. High-level support to Utility/Site management in situations where the Utility/Site had encountered significant regulatory-related operational performance difficulties. These were the days of the NRC's "troubled plant" list when Sites were targeted for intense regulatory scrutiny due to regulatory-related operational performance. These situations involved not only safety-related performance, but also breakdowns in Utility-Regulator communications and in regulatory trust & confidence in Utility/Site management. We I
ATTACHMENT provided assistance both to those Sites that were on the "troubled plant" list and to Sites that wanted to take proactive action to avoid getting on that list. Our typical role - on behalf of the utility - was to perform an in-depth assessment of all aspects of the organization and its performance, including the underlying organizational culture, and to assist in the development of strategies, plans and actions to recover from/avoid the situation. In many cases we were directly involved in the implementation of the strategies and plans, most often in an on-Site mentoring/coaching/observing/reinforcement role. In these engagements, we also spent almost all of our time on-Site.
- 3. High-level support to Utility/Site management in situations where the Utility/Site desired to continue to improve its operational performance. These engagements were similar in nature to those noted in 2 above.
By 1990, we had perform-ned a significant amount of work in these three areas. Based on that work, we developed an internal "White Paper" that identified and characterized the key attributes that differentiated high performing Sites from lower performing Sites. Although we did not formally publish this Paper, it was relatively well-known throughout the industry at the time and helped to cement our role in providing the services noted above.
In April 1992, we formed SYNERGY Consulting Services Corporation.
In 1993, we were approached by some of our clients to assist them in establishing and evaluating the effectiveness of Employee Concerns Programs and to develop an approach to systematically measure and characterize the Nuclear Safety Culture at their Sites. By 1998, due to client demand, conducting Nuclear Safety Culture Assessments had become our primary business area.
The Evolution of our Cultural Assessment Methodology Our first activity was to formalize the cultural attributes that were important to the Nuclear Safety Culture. In doing this, we drew from our experience regarding regulatory expectations, the key attributes included in our White Paper, other attributes derived from our organizational assessment experience and other attributes that were being forwarded by organizations such as INPO and the IAEA. We also recognized early on that there were certain attributes of the General Culture & Work Environment that should be included in a Nuclear Safety Culture Assessment - due to crossover effects. (It later became very clear that ratings of GCWE attributes could often serve as a leading indicator of ongoing or future challenges to the NSC.)
In the first few years, our assessments were conducted through a combination of employee interviews and program document reviews. While this approach proved to be reliable for drawing conclusions/making recommendations at the Site-wide level, we concluded that there were two disadvantages to this approach: (1) it provided limited ability to conclusively identify outlier organizations at the Site (absent a very large interview base) and (2) it was very labor-intensive.
By 1996, we had developed an approach involving an employee survey (questionnaire) that was administered to all employees and long-term contractors. This approach addressed the two disadvantages noted above. The survey also provided opportunities for write-in comments, which enabled us both to capture insights into the issues driving the numerical ratings and to 2
ATTACHMENT identify any other issues based on recurring comments. At this same time, we developed formal Models/Sub-Models of key cultural metrics - and mapped the survey questions/attributes into these models. This, of course, was necessary to improve consistency (industry benchmarking) in the interpretation of the results and to facilitate the identification/isolation of lower-rated sub-metrics, which in turn increases the specificity of problem identification. This approach greatly increased our ability to identify and target "outlier organizations" within a Site's organization based on particularly low metric ratings. This allows our clients to identify and act upon "local" cultural problems. At this stage of our development, we always augmented the survey with a set of employee interviews (random, representative sample - usually on the order of 40-60 interviews).
When we began to see the quantity and quality of information coming from the write-in comments, it confirmed to us and to our clients that this approach (survey and write-ins) was providing significantly more insight into the organization's culture than the previously used approach (interviews and document reviews).
By 1998, we had developed computer software to process survey data more efficiently and accurately. (As time has progressed, this significantly added to our industry benchmarking capabilities as well.)
In 1998, based upon client requests, we added a Leadership, Management and Supervisory Behaviors & Practices Model to our Cultural Assessment repertoire. We developed this model based upon our own experience, input provided by experts within our client base and literature on the subject. (NOTE: Some of our clients request that we fully use this model in their Cultural Assessments. Most all of our clients ask us to use at least selected portions of this Model, particularly those portions that are indirectly related to the SCWE. )
By 2000, we recognized that the write-in comment approach was providing such high-quality information that - unless unusual circumstances were found to exist - the conduct of personnel interviews should be considered to be optional or at least should be contingent on an analysis of the survey results and the write-in comments. As of 2006, we infrequently include personnel interviews as part of an assessment. We, of course, provide our clients a recommendation in this regard after we have analyzed the survey results/write-in comments. Interviews are usually conducted if one of the following two circumstances exists: (1) there is a very high level of regulatory interest in a particular cultural area or areas; (2) the write-in comments do not provide sufficient insight into the reasons why certain organizations were identified as "outliers."
Overall, we have concluded that the rated survey questions and the write-in opportunities provide more than sufficient information.
As time has passed, we have continued to "tune" our Models and the attributes that are "tested" through the survey. We have added attributes to reflect both our experience as well as industry experience. We have a large bank of survey questions to draw from to test these attributes. Some of our clients like very simple questions; some like more complex questions designed to get into cultural nuances. Some like the surveys to be as short as possible (as long as the essential attributes are tested); some like them to be longer so that many nuances are explored. Some like us to customize the survey questions. Some like us to address a variety of "special topics of client interest" in the survey - since this is an opportunity to obtain employee input. The bottom line is that we have the ability to be flexible to meet our client's objectives/desires.
3
ATTACHMENT We have also offered our clients the opportunity to obtain specific data on industry-wide areas of interest.
- We specifically modeled the attributes set forth in INPO SOER 02-04. (Note: INPO SOER 02-04 has more recently been supplanted by the INPO Principles for a Strong Nuclear Safety Culture.)
" Over the past two years we have been offering our clients the opportunity to exercise a Model of the INPO Principles for a Strong NSC that we have developed. (For the most part, this Model reflects a re-categorizing/re-assignment of attributes that were already included in our NSC Models. A few additional attributes were addressed through additional survey questions.)
- In mid-2006, we specifically modeled the Nuclear Safety Culture components and attributes set forth in the Appendix to NRC RIS 2006-13. (For the most part, this Model reflects a re-categorizing/re-assignment of attributes that were already included in our NSC Models. A few additional attributes were addressed through additional survey questions.) This model was first used in the Independent Assessment of the Nuclear Safety Culture at Davis-Besse Nuclear Power Station in the Fall of 2006.
4
Meeting with NRC Staff March 14, 2007 Independent Assessment of the PVNGS Nuclear Safety Culture SYNERGY Consulting Services Corporation Meeting Topics
- Independent Assessment Team
- Assessment Scope
- Assessment Methodology
- Assessment Coverage of RIS 2006-13 Nuclear Safety Components & Attributes
- Summary of 95003 Expectations & Issues
- Other Topics of Interest 2
1
Independent Assessment Team
- Mr. Tim Snyder, SYNERGY - Project Lead
- Mr. John Guibert, SYNERGY
° Mr. Howard Levin, SYNERGY 3
SYNERGY Background
" History and Evolution of SYNERGY Cultural Assessments (Handout)
- SYNERGY's Experience in Conducting Cultural Assessments
- 48 Nuclear Power Plant Sites
- 76 Nuclear Power Plants
- 2 Gaseous Diffusion Plants
- > 100 Cultural Assessments
- Industry Benchmarking 4
2
Assessment Scope
" SYNERGY Cultural Models
- Nuclear Safety Culture (NS VB&P; SCWE; ECP)
- General Culture & Work Environment
- Selected Leadership, Management &
Supervisory Behaviors & Practices
- NRC RIS 2006-13 Model
- INPO Principles Model
" Special Topics Assessment Methodology SOURCES OF INPUT
" Comprehensive Cultural Survey
- Rated Questions
- Write-In Comments
- Documentation Reviews
- Individual Personnel Interviews
" Selected Behavioral Observations 3
Assessment Methodology CULTURAL SURVEY
- Target Population
-All Employees and Long-Term Contractors Working at PVNGS
" Voluntary Participation
" Survey Administered by APS
- SYNERGY Guidelines
- 95003 Survey Administration Expectations
- Independent Data Processor Assessment Methodology CULTURAL SURVEY
- Survey Participation Expectations (Site)
- 95003 Expectation is 60%-70%
- Industry Average for SYNERGY Surveys is 78%
- PVNGS 2005 Survey Participation was 86.6%
4
Assessment Methodology CULTURAL SURVEY Rated Survey Questions
- Anticipate -180-200 question sub-parts Two Write-In Comment Opportunities
- Typically - 33% of survey participants provide write-in comments
- Most of these provide multiple comments Write-In Comments Are Confidential
- Redacted 9
Assessment Methodology DOCUMENTATION REVIEWS
" Policies and Programs
- Topical Areas:
-Address RIS 2006-13 (Survey Gaps)
- Performance Improvement Plan, including response to NRC inspection results
- Self-Assessments
" Other Survey Results (pre-INPO) 10 5
Assessment Methodology PERSONNEL INTERVIEWS
° Individual Personnel Interviews
- Specific Purpose for Each Interview
- Identified Outlier Organizations
- Low Participating Organizations
- RIS 2006-13 Attributes (Survey Gaps)
- PVNGS Oversight Organizations
- APS Executive Management 11 Assessment Methodology PERSONNEL INTERVIEWS
° Topics & Questions Will Be "Fit for Purpose"
- Topical areas correlated to the specific purpose for which the interview is being conducted.
These areas will vary from interview to interview.
- Interview questions will be tailored to fit the purpose of each interview.
° Interviewees Selected by Assessment Team
" Approximately 100 Interviews Anticipated 12 6
Assessment Methodology SOURCES OF INPUT Selected Behavioral Observations Potential Targets of Opportunity (TBD)
- Off-Site Nuclear Review Board Meeting
- Management Review Meeting
- Performance Improvement Program Review Meeting
- Corrective Action Review Board Meeting
- Morning Meeting
- Other 13 Assessment Methodology ISSUE IDENTIFICATION Numerical Survey Ratings and Trends
- Threshold Criteria/Rating Conventions Reflecting Industry Norms
- Criteria Vary Dependent on Nature of Each Cultural Metric/Sub-Metric Areas of Strength, Areas For Improvement, Areas in Need of Attention, Other Opportunities for Improvement Qualitative Judgment of Assessment Team
- Based on collective evaluation of all sources of input 14 7
Assessment Methodology TRENDING Direct Trending
- Comparison with January 2005 CCA Inherent Trending
- To Be Determined 15 Assessment Methodology INDUSTRY BENCHMARKING
- Provides Context For Results
- Site to Site Comparison Basis
° Recent Data
- Expect approximately 25 Sites for comparison 16 8
Reporting of Results
- SYNERGY will provide an Independent Assessment Report, which will document the key findings and conclusions of the Independent Assessment. The Report will include an Executive Summary.
- There will be several Attachments to the Report.
- Both the Assessment Report and its Attachments will be developed in a form that is suitable for docketing (i.e., non-inclusive of SYNERGY Proprietary Information).
17 Assessment Coverage of RIS 2006-13 Nuclear Safety Components &Attributes Mapping of Survey Questions to RIS 2006-13 Components & Attributes Survey Coverage "Gaps"
- Addressed through a combination of documentation reviews and personnel interviews.
18 9
SUMMARY
OF ISSUES
- Part 02.07, Part 03.07, Enclosures A,B and E
- No Significant Issues or Items Requiring Clarification
- Enclosure C:
- Focused/Specific Purpose vs. General Data Gathering
- Enclosure D:
- Limited/Selected/Specific Purpose
" Enclosure F:
- Discussion Needed; Agreement Needed 19 Other Topics of Interest
- Detailed Discussion of 95003 Expectations (As Needed)
- Other (To be determined) 20 10