ML102500657

NRC000043-Inspection Procedure Attachment 95003.02 Guidance for Conducting an Independent NRC Safety Culture Assessment.

ML102500657
Person / Time
Site:	Prairie Island
Issue date:	01/15/2009
From:	NRC/OGC
To:	Atomic Safety and Licensing Board Panel
SECY RAS
Shared Package
ML102500654	List: ML102500655 ML102500657 ML102500658 ML102500660 ML102500662 ML102500663 ML102500664 ML102500666 ML102500667 ML102500668 ML102500670 ML102500671 ML102500672 ML102500673 ML102500675 ML102500676 RIS 2006-13, 2006/07/31-Intervenor-Exhibit 10-Information On The Changes Made To The Reactor Oversight Process To More Fully Address Safety Culture, Regulatory Issue Summary 2006-13, U.S. NRC RIS 2006-13, NRC000046-NRC Regulatory Issue Summary 2006-13, Information on the Changes Made to the Reactor Oversight Process to More Fully Address Safety Culture.
References
50-282-LR, 50-306-LR, ASLBP 08-871-01-LR-BD01, RAS 18574
Download: ML102500657 (65)

Text

NRC000043 ATTACHMENT 95003.02 Guidance for Conducting an Independent NRC Safety Culture Assessment Introduction The purpose of this assessment is for the NRC to assess the licensee=s safety culture.

Safety culture is defined as Athat assembly of characteristics and attitudes in organizations and individuals which establishes that, as an overriding priority, nuclear plant safety issues receive the attention warranted by their significance.@ Therefore. an organization=s characteristics (i.e., safety culture components that comprise the visible aspects of a safety culture) can be assessed by evaluating the extent to which its policies, programs, and processes ensure that nuclear safety issues receive the attention warranted by their significance. For example, the effectiveness of the licensee=s corrective action program at identifying, prioritizing, and resolving issues with nuclear safety impacts provides important insights into the licensee=s safety culture. An organization=s members= shared attitudes and behaviors with respect to nuclear safety also provide important insights into a licensee=s safety culture and can be assessed through behavioral observations, interviews, and focus groups.

The guidance in this attachment is intended to enable inspectors to identify those consistencies in attitudes and behavior that are indicative of safety culture.

Relevant attitudes with respect to nuclear safety include:

• specific attitudes (i.e., an individual=s tendencies to react favorably or unfavorably) towards different characteristics of the organization, which may include general attitudes about the organization as a whole, the effectiveness of the job performance evaluation system in encouraging taking responsibility for nuclear safety, the effectiveness of the work control system in scheduling work activities safely and efficiently, or the procedures and work packages the individual is given to assist in performing tasks;

• perceptions, which are how an individual interprets information about the organization to form beliefs; and

• values, which are an individual=s judgments about what is important, meaningful, and worthwhile at work both to the individual and to the organization.

Consistencies in attitudes are typically identified by interviewing an organization=s members to elicit their specific attitudes, perceptions, and values, and their views of the organization=s values and attitudes as they relate to nuclear safety. Consistencies in attitudes may also be identified by asking individuals to provide examples of situations and experiences that are consistent with the attitudes, perceptions, and values they describe. If a large proportion of an organization=s members share the same specific attitudes, perceptions, and values, these become a social Afact@ within the safety culture of the organization and can influence individuals= subsequent decisions and actions.

Behavioral consistencies with respect to nuclear safety include:

• observable behaviors (e.g., how often, as indicated by the frequency or proportion of occurrences, supervisors mention safety considerations during pre-job briefs for jobs that may impact nuclear safety, how often personnel use procedures in the manner required),

• written communications (e.g., how often, as indicated by the frequency or proportion of occurrences, do annual performance reviews mention individuals=

decisions and actions related to nuclear safety in the past year; how often do the Issue Date: 01/15/09 95003.02

NRC000043 forms from a management feedback program note unsafe acts or favorably record desirable safety behaviors; how often do email communications from managers and supervisors emphasize production or safety goals), and

• verbal communications (e.g., how often, as indicated by the frequency or proportion of occurrences, do supervisors and managers mention deadlines and schedules during conversations or in meetings compared to the number of times they mention nuclear safety considerations).

Consistencies in behavior with respect to nuclear safety are typically identified by observing an organization=s members going about their daily work as well as reviewing written records of decisions made and work that was previously performed. Behavioral consistencies can also be identified by asking an organization=s members questions that focus on their perceptions of the organization=s norms. Norms are an organization=s underlying, usually unwritten, rules for behavior that establish Ahow we do things around here,@ and they may or may not coincide with the organization=s stated policies and procedures.

Therefore, the safety culture assessment will focus on the use of tools and instruments (i.e., information-collection methods) that will help the inspection team identify these consistencies in organizational attitudes and behavior through the use of interviews, structured behavioral observations, document reviews and analysis, and case studies.

1. Preparation for an Independent NRC Safety Culture Assessment

a. From the list of inspection requirements in Enclosure 95003.02-A, select the requirements that relate to the performance deficiencies that prompted this inspection, and, to the extent possible, adapt the selected requirements to focus on those performance deficiencies. Ensure that the selected requirements include at least one requirement associated with each safety culture component.

A specific performance deficiency may or may not be the result of a weak safety culture. As input to the safety culture assessment, inspectors/SCAs should independently determine whether weaknesses in one or more safety culture components played a causal or contributing role in each performance deficiency.

b. Identify the SCA lead and SCA subteam who will conduct the safety culture assessment, and ensure that at least two different SCAs (or other inspection team members as appropriate) independently collect information from each functional/organizational group. The safety culture components detailed in section 06.07 of IMC 0305 describe organizational characteristics and consistencies in attitudes and behavior with respect to nuclear safety that are indicative of safety culture.

(1) When developing the tools and instruments and selecting the methods for this assessment, include opportunities to look for evidence of all of these components.

(2) When planning focus group interviews, assign two different SCAs to lead the interviews with different groups of participants from the same functional area. If there is an insufficient number of licensee staff within a functional area to form two separate focus groups (e.g., chemistry),

consider combining focus group participants from more than one functional area or use individual interviews instead of focus groups.

(3) Assign two different SCAs to conduct structured behavioral observations of the selected work activities within a functional area.

Issue Date: 01/15/09 95003.02

NRC000043 (4) Establish a plan for communication between and coordination among the SCAs and with the other inspection team members.

c. Obtain access to a sample of the following documents, including information the team needs for its targeted review (ensure they are not duplicates of that which has already been requested by team leader):

(1) The root cause investigations of the performance deficiencies that prompted this inspection.

(2) Reports of any self or third-party organizational/safety culture types of assessments from the past five years. These reports may include:

-safety culture assessments completed either by the licensee or a third-party.

-safety conscious work environment assessments.

-leadership assessments (redacted, as necessary). Information gathered about an individual manager that provides evidence of leadership ineffectiveness or staff dislike of a particular leadership style should not be considered when developing safety culture insights. However, if there is evidence of generally ineffective leadership or style which demonstrates organizational attitudes and behaviors that are inconsistent with those described in the safety culture components, this should be noted.

-employee morale/job satisfaction assessments. Information gathered that provides evidence of low employee morale or low job satisfaction should not be considered when developing safety culture insights, except if the low morale or job satisfaction significantly contributes to weaknesses in the safety culture components.

-Nuclear Quality Assurance/Oversight assessments related to the components of safety culture.

-any focused or broad-scope organizational performance assessments.

(3) Review corporate and site safety policy statements as they relate to safety culture.

(4) A sample of redacted job performance reviews from each functional group in the organization (e.g., operations, maintenance, security, engineering) and any (redacted) agreements or documents related to the bases for management compensation and incentives. Review the sample to determine if there is evidence that the actual reviews implemented the licensees guidance for conducting performance reviews, especially with regard to the balance between safety and production.

(5) Meeting minutes from the past year for site senior management team meetings, nuclear oversight review group meetings, and corrective action review group meetings; meetings to develop and amend site financial plans and budgets, including operating, maintenance, and capital improvement plans; meetings that focus on decision-making with nuclear safety implications; and other meetings held to plan and discuss mitigating any potential or actual chilling effects from disciplinary actions. Review for Issue Date: 01/15/09 95003.02

NRC000043 evidence that safety has priority, e.g., for safety over costs in making improvements or maintaining systems.

(6) Documents describing any reward or incentive programs focused on promoting nuclear safety behaviors and documentation pertaining to the implementation of the program(s) (redacted, if necessary).

(7) Lesson plans used to train site personnel on safety culture and/or safety conscious work environment, and records that show when the training was presented and attendance. Review training lesson plans and records to determine what was presented, when it was presented, and who attended.

(8) Summaries of documents from the corrective action program that relate to the components of safety culture and were identified or resolved within the previous year. Review and code them to the applicable safety culture components.

(9) Complete and current organizational charts, including the names and site contact information for the individuals listed.

(10) Written communications (e.g. memoranda, e-mails) between management and staff related to any significant organizational changes within the past year. Significant organizational changes could include changes in organizational structures and functions, leadership, policies, programs, procedures, and resources.

(11) Documents that correspond to the selected inspection requirements in Enclosure 95003.02-A.

d. From NRC sources, obtain and review:

(1) Allegations related to the site=s safety culture received within the previous year, at a minimum; and (2) NRC inspection reports that document, in accordance with IMC0612, the performance deficiencies which prompted this inspection.

e. Review the results of the licensee=s third-party safety culture assessment to determine if particular functional/organizational groups were identified by the SCAs as having problems in any of the safety culture components or if the results indicated that there were weaknesses in any of the safety culture components across the site.

f. From the review of the documentation, determine if any functional groups, management levels, or safety culture components should receive more emphasis in the assessment based on the licensee=s assessment having identified safety culture or performance issues related to them.

g. Determine the assessment methods that are best suited to the perceived situation at the site, ensuring that each safety culture component will be assessed with at least two different methods, and develop sampling plans for each method. Assessment methods shall include individual and/or focus group interviews (see guidance in Enclosure 95003.02-B and Enclosure 95003.02-C);

structured behavioral observations (see guidance in Enclosure 95003-D.02);

and event follow-up studies (see guidance in Enclosure 95003.02-E).

Issue Date: 01/15/09 95003.02

NRC000043

h. Prepare the selected data-collection tools, which may include interview and focus group guides and behavioral observation checklists.

i. Work with the licensee to identify the appropriate means to disseminate a communication plan to site personnel that addresses the purpose of the assessment; states the team will meet with groups, observe meetings and work activities, and talk with individuals; states anyone who wants to talk with the NRC should contact the team (provide appropriate instructions); describes, to the extent possible, information obtained during the assessment will not be attributed to individual participants who are interviewed or observed by NRC inspectors; and requests sensitivity to information shared between personnel during participation in NRC activities, such as focus groups.

2. Conduct the NRC=s Independent NRC Safety Culture Assessment

a. Complete this assessment by applying the methods and sampling plans, using the data-collection tools developed during the preparation phase.

b. As the assessment progresses, the lead SCA should adjust the assessment plan as required to:

(1) ensure that the information collected will be adequate to reach valid and reliable insights about the safety culture components at each management level and in each functional group that falls within the scope of the assessment; (2) resolve inconsistencies identified in NRC assessment results; and (3) address emergent issues identified during other inspection or assessment activities.

Adjustments to the assessment plan may include increasing the number of interviewees or focus group participants, conducting additional individual interviews, adding documents to be reviewed, increasing the number and focus of behavioral observations, re-directing resources to complete a case study, and/or increasing the number of team members collecting related information.

c. Ensure that each safety culture component is assessed using at least two data-collection methods, by at least two SCAs (or other inspection team members as appropriate) independently, and that the data-collection methods are applied consistently. In this context, valid insights are based on consistent results from applying multiple information-gathering methods, and reliable insights are based on consistent results from multiple team members who are independently collecting related information.

d. Identify weaknesses in any safety culture component and the functional groups and management levels in which any weaknesses appear. As examples, identify any performance deficiencies for which safety culture weaknesses were a root or contributing cause.

e. As results from the various data-collection tools are being compiled, and particularly after all results have been obtained, aggregate those results to determine whether any consistency regarding a particular safety culture component exists among results obtained from various data-collection tools and inspectors. From this determination, develop insights about the various safety Issue Date: 01/15/09 95003.02

NRC000043 culture components and how they are reflected in the attitudes and behaviors within the various licensee functional groups.

The insights of most concern include the following:

$ Corporate and/or senior site management demonstrates attitudes and behavior with respect to nuclear safety that are substantively inconsistent with the expectations in any of the safety culture components.

$ A single critical functional group, including operations, engineering, maintenance, or security, demonstrates weaknesses across multiple safety culture components.

$ The majority of functional groups demonstrate some weaknesses in multiple safety culture components (i.e., weaknesses are widespread throughout the organization).

If any of these statements are supported by consistent results obtained through application of multiple data-collection tools in conjunction with inspector/SCA insight, then:

$ Use this information to inform the assessment of the contributors to degraded performance in the affected SPA(s). For instance, the results of this assessment can be reflected in statements such as Aweaknesses in a safety culture component (or group of components) contributed to deficiencies in (some SPA Key Attribute),@ or Aweaknesses exhibited by the operations organization related to (some specific list of safety culture components) contributed to the degraded Reactor Safety SPA (or Key SPA Attribute).@

$ Document each such statement as a key result of this assessment. Follow the requirements in section 02.12. and associated guidance in section 03.12.

to perform the documentation in coordination with the rest of the inspection team. 5003.02 Enclosures 5003.02-A: Sample Inspection Requirements for Safety Culture Components.

[ADAMS ML082670440] 5003.02-B: Sample Questions for Safety Culture Components.

[ADAMS ML082670848] 5003.02-C: Guidance for Focus Groups and Individual Interviews.

[ADAMS ML082630691] 5003.02-D: Guidance for Structured Behavioral Observations.

[ADAMS ML082670874] 5003.02-E: Guidance for Event Follow-up Studies.

[ADAMS ML082630714] 5003.02-F: Guidance for Evaluating Safety Culture Surveys.

[ADAMS ML082630765]

Issue Date: 01/15/09 95003.02

NRC000043 Enclosure 95003.02-A Sample Inspection Requirements for Safety Culture Components This enclosure contains sample inspection requirements from which safety culture assessors may select and adapt inspection requirements related to performance deficiencies, as required by section 95003.02.08.a. This enclosure also identifies documents which correspond to the requirements, to assist safety culture assessors in compiling a list of documents to request from the licensee. Hence, this enclosure is a resource that safety culture assessors may use to develop the assessment section of the inspection plan.

In this enclosure, sample inspection requirements and corresponding documents are listed in the two-column table below, under each of the safety culture components:

column one includes the inspection requirements associated with each component, while the second column describes the corresponding documents which should be requested from the licensee.

ACCOUNTABILITY - Management defines the line of authority and responsibility for nuclear safety. Specifically (as applicable):

$ Accountability is maintained for important safety decisions in that the system of rewards and sanctions is aligned with nuclear safety policies and reinforces behaviors and outcomes that reflect safety as an overriding priority.

$ Management reinforces safety standards and displays behaviors that reflect safety as an overriding priority.

$ The workforce demonstrates a proper safety focus and reinforces safety principles among their peers.

Requirement Corresponding Documents Review the policies and procedure which Policies and procedures which define the define the line of authority and lines of authority and responsibility for responsibility for nuclear safety to verify nuclear safety. A sample of redacted that those lines and responsibilities are performance reviews.

clearly identified.

If the licensee has established a system Policies and procedures for employee of rewards and/or sanctions related to rewards and/or sanctions related to plant plant performance, then verify that those performance.

rewards and/or sanctions do not conflict with nuclear safety policies and do reinforce behaviors and outcomes which reflect safety as an overriding priority.

Determine that objective measures have Evidence of objective measures taken by been taken by management to reinforce management to reinforce safety safety standards. standards In management meetings, observe Schedule of management meetings.

whether the behavior of management reinforces safety standards and displays behaviors that reflect safety as an overriding priority. See Enclosure D for guidance on structured behavioral observations.

Issue Date: 01/15/09 EA - 1 95003.02

NRC000043 Observe whether personnel reinforce safety principles among themselves. See Enclosure D for guidance on structured behavioral observations.

CONTINUOUS LEARNING ENVIRONMENT - The licensee ensures that a learning environment exists. Specifically (as applicable):

$ The licensee provides adequate training and knowledge transfer to all personnel on site to ensure technical competency.

$ Personnel continuously strive to improve their knowledge, skills, and safety performance through activities such as benchmarking, being receptive to feedback, and setting performance goals. The licensee effectively communicates information learned from internal and external sources about industry and plant issues.

Requirement Corresponding documents For each major work group (including but Lesson plans and training records for not necessarily limited to Operations, continuing training.

Engineering, Maintenance, Radiological Protection, Security), review the continuing-training program for the group:

! Review the lesson plans to verify that they include features to effectively facilitate knowledge transfer to ensure technical competency.

! Review records which identify the employees who received the training and compare those records with employee rosters to verify that employee participation was consistent with management expectations. Also verify that management expectations facilitate and enable effective knowledge transfer.

Identify the benchmarking and reverse- Records of benchmarking activities for benchmarking activities conducted during the last three years.

the previous two years.

! Review the records of benchmarking activities to verify that they included features which could improve licensee knowledge, skills, and safety performance.

! Review the actions taken by the licensee as a result of those activities, to verify that the licensee effectively integrated lessons learned from those activities into their programs and processes.

Issue Date: 01/15/09 EA - 2 95003.02

NRC000043 Review the procedures which establish Examples of communications to the and describe the licensee=s industry organization of information learned from operating experience program and review internal and external sources about selected records developed using that industry and plant issues.

program, to verify that the licensee effectively communicates information learned from internal and external sources about industry and plant issues CORRECTIVE ACTION PROGRAM - The licensee ensures that issues potentially impacting nuclear safety are promptly identified, fully evaluated, and that actions are taken to address safety issues in a timely manner, commensurate with their significance. Specifically (as applicable):

$ The licensee implements a corrective action program with a low threshold for identifying issues. The licensee identifies such issues completely, accurately, and in a timely manner commensurate with their safety significance.

$ The licensee periodically trends and assesses information from the corrective action program and other assessments in the aggregate to identify programmatic and common-cause problems. The licensee communicates the results of the trending to applicable personnel.

$ The licensee thoroughly evaluates problems such that the resolutions address the causes and extent of conditions, as necessary. This includes properly classifying, prioritizing, and evaluating for operability and reportability conditions adverse to quality. This also includes, for significant problems, conducting effectiveness reviews of corrective actions to ensure that the problems are resolved.

$ The licensee takes appropriate corrective actions to address safety issues and adverse trends in a timely manner, commensurate with their safety significance and complexity.

$ If an alternative process (i.e., a process for raising concerns that is an alternate to the licensee=s corrective action program or line management) for raising safety concerns exists, then it results in appropriate and timely resolutions of identified problems.

Requirement Corresponding documents Review the CAP program procedure to Procedures for corrective action program.

verify that it clearly states an expectation to identify issues at a low threshold.

Review a sample of recently-identified A list of corrective action documents to issues in the CAP to verify that issues select a sample from or a sample of had been identified at the threshold recent CA documents.

stated in the procedure.

Issue Date: 01/15/09 EA - 3 95003.02

NRC000043 Review a sample of recently-completed (as above) A list of corrective action evaluations to verify for each evaluation documents to select a sample from or a that sample of recent CA documents.

! the difference between the event date/time and the reported date/time is commensurate with the safety significance of the identified issue;

! the event/condition description in the completed evaluation is consistent with the event/condition description in the CAP record.

Verify that a program or process exists to Audit or trending plan for the CAP.

periodically trend and assess information from the CAP and other assessments in the aggregate to identify programmatic and common cause problems. Review a sample of results produced by that program or process to verify that it does identify such problems. Review a representative sample of those problems to verify that they were appropriately addressed.

Determine to whom the licensee A sample of trending results in the distributes the trending results to verify area(s) of inspection interest.

that the results apply to those personnel.

Determine how the recipients responded to or otherwise used the results.

For a sample of issues identified in the A list of corrective action documents to CAP, verify that the evaluations were select a sample from or a sample of thorough and that the resolutions of those recent CA documents.

issues appropriately addressed the causes. For each issue, verify that the licensee properly classified and prioritized the issue commensurate with its potential safety significance, and that the licensee properly addressed operability and reportability considerations.

For a sample of significant conditions A sample of significant conditions adverse to quality identified in the CAP, adverse to quality identified in the CAP.

verify that the licensee completed effectiveness reviews, and that those reviews verified that the associated conditions were resolved.

For a sample of safety issues identified in A list of corrective action documents to the CAP as selected by the team leader, select a sample from or a sample of verify that the licensee implemented recent CA documents.

corrective actions in a timely manner, commensurate with their safety significance and complexity.

Issue Date: 01/15/09 EA - 4 95003.02

NRC000043 For a sample of adverse trends identified A sample of trending results in the in the CAP as selected by the team area(s) of inspection interest.

leader, verify that the licensee implemented corrective actions in a timely manner, commensurate with their safety significance and complexity.

Review any self /independent Copies of self /independent assessments assessments of CAP conducted in the of CAP conducted in the past 24 months.

past 24 months, to verify that those assessments were thorough and objective.

For the issues identified in Evaluations of and corrective actions for self/independent assessments of CAP issues identified in self /independent conducted in the past 24 months, review assessments of CAP conducted in the evaluations of the issues and corrective past 24 months actions taken to address those issues, to verify that the evaluations were thorough and that the resolutions of those issues appropriately addressed the causes.

For a sample of CAP items that were CAP items that were cancelled during the cancelled, verify that no risk-significant past 24 months issues were cancelled.

For a sample of CAP items that were CAP items that were downgraded in downgraded in priority, verify that no risk- priority during the past 24 months significant items were downgraded.

Observe initial screening, management Schedule of these meetings screening, and closure meetings Determine whether any safety issues Safety issues that were identified in the were identified in the alternative process alternative process in the past two years.

within the previous two years. If so, determine how those issues were addressed, to verify that the resolutions of identified safety issues were appropriate and timely.

Issue Date: 01/15/09 EA - 5 95003.02

NRC000043 DECISION-MAKING - Licensee decisions demonstrate that nuclear safety is an overriding priority:

! The licensee makes safety-significant or risk-significant decisions using a systematic process, especially when faced with uncertain or unexpected plant conditions, to ensure safety is maintained. This includes formally defining the authority and roles for decisions affecting nuclear safety, communicating these roles to applicable personnel, implementing these roles and authorities as designed, and obtaining interdisciplinary input and reviews on safety-significant or risk-significant decisions.

! The licensee uses conservative assumptions in decision-making and adopts a requirement to demonstrate that the proposed action is safe in order to proceed rather than a requirement to demonstrate that it is unsafe in order to disapprove the action. The licensee conducts effectiveness reviews of safety-significant decisions to verify the validity of the underlying assumptions, identify possible unintended consequences, and determine how to improve future decisions.

! The licensee communicates decisions and the basis for decisions to personnel who have a need to know the information in order to perform work safely, in a timely manner.

Requirement Corresponding documents Review procedures for making decisions, Records that describe recently-made immediate and longer-term; note decisions.

definitions of authority and roles; verify that the procedures call for conservative Procedures for management decision assumptions (regarding equipment making.

degradation, human performance, unfamiliar plant conditions and tasks, etc.) and consider risk impacts; verify that procedures require effectiveness reviews and communication of decisions and bases to affected personnel.

Identify methods used to communicate Procedures for communication and these roles to site personnel. communication plans for management decisions Review procedures for obtaining Procedures for obtaining interdisciplinary interdisciplinary reviews on decisions. reviews on decisions.

Determine if training on decision-making Training materials and records.

is provided and review training materials and records.

Review records that describe recent Records or minutes of planning meetings decisions. If records don=t exist or are including modification and capital incomplete, interview involved personnel. improvement approval meetings.

Observe decision making activities in work planning meetings, plan-of-the-day meetings, and other forums.

Issue Date: 01/15/09 EA - 6 95003.02

NRC000043 ENVIRONMENT FOR RAISING CONCERNS - An environment exists in which employees feel free to raise concerns both to their management and/or the NRC without fear of retaliation, and employees are encouraged to raise such concerns.

Specifically (as applicable):

! Behaviors and interactions encourage the free flow of information related to raising nuclear safety issues, differing professional opinions, and identifying issues in the corrective action program and through self-assessments. Such behaviors include supervisors responding to employee safety concerns in an open, honest, and nondefensive manner and providing complete, accurate, and forthright information to oversight, audit, and regulatory organizations. Past behaviors, actions, or interactions that may reasonably discourage the raising of such issues are actively mitigated. As a result, personnel freely and openly communicate in a clear manner conditions or behaviors, such as fitness for duty issues, that may impact safety, and personnel raise nuclear safety issues without fear of retaliation.

! If alternative processes (i.e., a process for raising concerns or resolving differing professional opinions that are alternates to the licensee=s corrective action program or line management) for raising safety concerns or resolving differing professional opinions exist, then they are communicated, accessible, have an option to raise issues in confidence, and are independent in the sense that the program does not report to line management (i.e., those who would in the normal course of activities be responsible for addressing the issue raised).

Requirement Corresponding documents Verify that measures have been taken by Procedures and policies and training on the licensee to encourage employees to raising concerns. Samples of plant raise concerns both to their management communications that inform and reinforce and/or the NRC without fear of retaliation. the procedures and policies.

Observe licensee employee behaviors during meetings, etc. to determine whether behaviors promote the raising of safety concerns.

Interview personnel involved in recent decisions to determine whether dissenting views were heard. If so, verify that consideration of those views did not discourage employees from raising dissenting views.

Review the NRC records of allegations NRC files.

for evidence of discrimination to determine whether discrimination, chilling effect, or ineffective corrective action program issues have been raised and substantiated.

Issue Date: 01/15/09 EA - 7 95003.02

NRC000043 Review the procedures and policies Procedures and policies which establish which establish and describe the and describe the alternative process for alternative process for raising safety raising safety concerns.

concerns or resolving differing professional opinions to verify that those processes are accessible, have an option to raise issues in confidence, and are independent from management who would in the normal course of activities be responsible for addressing the issue.

Verify that actions and supporting Samples of communications that inform behaviors have been taken by the and reinforce the procedures and policies licensee to inform employees about the for raising concerns.

alternative process.

Review selected issues recorded in the Access to files for the alternative process alternative process for raising issues to for raising safety concerns.

verify that those issues were evaluated and resolved as appropriate.

OPERATING EXPERIENCE - The licensee uses operating experience information, including vendor recommendations and internally generated lessons learned, to support plant safety. Specifically (as applicable):

! The licensee systematically collects, evaluates, and communicates to affected internal stakeholders in a timely manner relevant internal and external operating experience.

! The licensee implements and institutionalizes operating experience through changes to station processes, procedures, equipment, and training programs.

Requirement Corresponding documents Verify that the licensee collected, A list of generic communications received evaluated, and communicated to affected and processed by the licensee within the staff in a timely manner the generic previous two years.

communications issued by the NRC within the previous two years that applied Procedures or policies for handling to power reactor licensees. operating experience from the NRC.

By reviewing appropriate licensee A list of OE items received from INPO, records, verify on a sampling basis that vendors and other sources and the licensee collected, evaluated, and processed by the licensee within the communicated to affected staff in a timely previous two years.

manner communications received from INPO, vendors and other sources. Procedures or policies for handling operating experience from industry.

Refer to sections 3.3 and 3.4 of the OEDO procedure 220, Coordination with the Institute of Nuclear Power Operations for guidance prior to reviewing any INPO documents.

Issue Date: 01/15/09 EA - 8 95003.02

NRC000043 For a representative sample of OE items For a sample of generic communications communicated to affected staff, verify and OE items, actions taken as a result.

that the licensee identified and implemented appropriate corresponding Refer to sections 3.3 and 3.4 of the changes to station processes, OEDO procedure 220, Coordination with procedures, equipment, and/or training the Institute of Nuclear Power programs. Operations for guidance prior to reviewing any INPO documents.

Verify that the licensee incorporates the A schedule of pre-job briefs that will be use of OE into pre-job briefs, held while the team is on site.

management meetings, and work packages. A schedule of management meetings that will be held while the team is on site.

A list of work packages completed during the recent past. (Review a sample.)

Review the CAP for issues related to the Evaluations of and corrective actions for use/effectiveness of OE. OE issues.

ORGANIZATIONAL CHANGE MANAGEMENT - Management uses a systematic process for planning, coordinating, and evaluating the safety impacts of decisions related to major changes in organizational structures and functions, leadership, policies, programs, procedures, and resources. Management effectively communicates such changes to affected personnel.

Requirement Corresponding documents Review the method(s) or process(es) Procedures and policies used for used by the licensee for planning, planning, coordinating, and evaluating coordinating, and evaluating the safety the safety impacts of decisions related to impacts of decisions related to major major changes in organizational changes in organizational structures and structures and functions, leadership, functions, leadership, policies, programs, policies, programs, procedures, and procedures, and resources. Review resources. Records which describe the records which describe the safety safety impacts of decisions evaluated impacts of decisions evaluated using using those methods/processes during those methods/processes during the the previous two years.

previous two years, to verify that the licensee effectively used the subject methods/processes.

Review the methods used by the licensee The methods used to communicate to communicate such changes to affected changes to affected personnel and a personnel, to verify that the changes sample of communications.

were effectively communicated to those personnel.

Identify methods used to communicate Procedures for communication and these roles to site personnel. communication plans for management decisions Review procedures for obtaining Procedures for obtaining interdisciplinary interdisciplinary reviews on decisions. reviews on decisions Issue Date: 01/15/09 EA - 9 95003.02

NRC000043 Determine how the licensee identifies Procedures for implementing changes.

and attempts to mitigate any unintended effects of planned changes, including A list of recent planned changes.

those associated with voluntary reductions, retirements and layoffs. For selected recent changes, interview involved personnel to determine whether any unintended effects were identified.

Determine the steps taken to get the Procedures for implementing changes.

organization culturally ready for change, to minimize fear, and increase tolerance of uncertainty.

PREVENTING, DETECTING, AND MITIGATING PERCEPTIONS OF RETALIATION

- A policy for prohibiting harassment and retaliation for raising nuclear safety concerns exists and is consistently enforced in that:

! All personnel are effectively trained that harassment and retaliation for raising safety concerns is a violation of law and policy and will not be tolerated.

! Claims of discrimination are investigated consistent with the content of the regulations regarding employee protection and any necessary corrective actions are taken in a timely manner, including actions to mitigate any potential chilling effect on others due to the personnel action under investigation.

! The potential chilling effects of disciplinary actions and other potentially adverse personnel actions (e.g., reductions, outsourcing, and reorganizations) are considered and compensatory actions are taken when appropriate.

Requirement Corresponding documents Verify that personnel have received Training plans on prohibitions of training regarding supervisor-to- harassment and intimidation.

employee and peer-to-peer behaviors that could constitute harassment, Records that indicate who received the intimidation, retaliation, and subject training.

discrimination for raising safety concerns and that such behaviors are a violation of law and policy and will not be tolerated.

Review any investigations performed by Records of investigations performed by the licensee of claims of discrimination to the licensee of claims of discrimination.

verify that those investigations were conducted consistent with the content of the regulations regarding employee protection and any necessary corrective actions are taken in a timely manner, including actions to mitigate any potential chilling effect on others due to the personnel action under investigation.

Review the NRC records of allegations NRC records of allegations for evidence of discrimination to determine whether discrimination issues have been raised and substantiated.

Issue Date: 01/15/09 EA - 10 95003.02

NRC000043 Verify that the procedures and/or policies Procedures and/or policies for disciplining for disciplining employees and employees implementing forced reductions contain sufficient provisions to preclude taking Procedures and/or policies for adverse employee actions as retaliation implementing forced reductions for protected activity.

Review the disciplinary actions taken Disciplinary actions taken against against employees within the previous employees within the previous two years.

two years, and verify that compensatory actions were taken as appropriate by the licensee to address potential chilling effects of those actions.

RESOURCES - The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to assure nuclear safety. Specifically, those necessary for:

! maintaining long-term plant safety by maintenance of design margins, minimization of longstanding equipment issues, minimizing preventative maintenance deferrals, and ensuring maintenance and engineering backlogs that are low enough to support safety

! training of personnel and sufficient qualified personnel to maintain work hours within working hour guidelines

! complete, accurate, and up-to-date design documentation, procedures, and work packages, and correct labeling of components

! adequate and available facilities and equipment, including physical improvements, simulator fidelity and emergency facilities, and equipment Requirement Corresponding documents Determine the history of reductions-in- Reports of internal or third-party staffing force or other draw-downs of the studies workforce at the site. Establish the reasons for these reductions and quantify Reports of re-engineering efforts the numbers of employees for the completed within the past two years associated organizational areas to determine the impact of the reductions on Peer group comparisons the available personnel resources.

Determine the bases for staffing level Resource assessments in the previous determinations. two business plans Strategic staffing plans Verify that long-standing equipment Justification for identified long-standing issues and deferring preventive equipment issues.

maintenance are minimized to the extent practical, and that justification for long-standing equipment issues is risk-informed.

Issue Date: 01/15/09 EA - 11 95003.02

NRC000043 Verify that the licensee knows what Procedures for design control and design design margins exist. Determine how modifications, and design basis design margins are considered in design documents.

control, and are updated as required to be current with how the plant is configured and operated.

Review the engineering backlogs Tracking records for engineering work (including engineering work requests, assignments and requests.

design mods, temp mods, drawing updates, equipment database updates, mod proposals, CAP evaluations) to determine trends. Verify that any conditions adverse to quality that are addressed by backlogged items have also been entered into the corrective action program.

Verify that decisions to place items in the Justifications for listing items in the engineering backlog are risk-informed. engineering backlog.

Review the trend in the non-outage work- Tracking system status and trends in order and workBpackage backlogs. work orders and work packages.

Review the licensee=s procedures for Procedures for control of overtime and authorizing overtime, including meeting overtime guidelines exceptions to overtime guidelines.

Review the trends in the overtime for selected work groups. Check that overtime limits are not routinely exceeded.

For selected work groups, review the Procedures for qualifying working level programs and procedures for qualifying and first line supervisors in all work personnel. For a sample of personnel in groups.

those work groups, verify that personnel qualifications are current and in accordance with those procedures.

Review the trend in update backlogs of Tracking records for trends in document procedures, calculations and drawings. updates. (Engineering is listed above)

Determine how the licensee records Record of work package deficiencies and reflect the quality of work packages; trend information.

consider rework designations in the maintenance database and cause codes in the CAP that could indicate work package quality. Determine whether the trend in work package quality is being tracked.

Issue Date: 01/15/09 EA - 12 95003.02

NRC000043 Interview selected operators and Procedures for simulator fidelity and simulator support personnel to determine identification and resolution of simulator the extent to which the simulator matches issues, and results produced using those the plant. If the simulator does not procedures.

reasonably match the plant, determine the reasons why not. Lists of discrepancies between the plant and the simulator.

Justification for the simulator backlog.

Review the trend in the simulator work The trend in simulator work orders.

order backlog.

Identify the procedure(s) used to maintain Procedures for meeting requirements for emergency facilities and equipment, and emergency preparedness.

evaluate results developed through that procedure.

Review the trend in the emergency Trend information on work orders for facility maintenance and/or upgrade emergency planning.

backlog.

SAFETY POLICIES - Safety policies and related training establish and reinforce that nuclear safety is an overriding priority in that:

! These policies require and reinforce that individuals have the right and responsibility to raise nuclear safety issues through available means, including avenues outside their organizational chain of command, and to external agencies, and obtain feedback on the resolution of such issues.

! Personnel are effectively trained on these policies.

! Organizational decisions and actions at all levels of the organization are consistent with the policies. Production, cost, and schedule goals are developed, communicated, and implemented in a manner that reinforces the importance of nuclear safety.

! Senior managers and corporate personnel periodically communicate and reinforce nuclear safety such that personnel understand that safety is of the highest priority.

Requirement Corresponding documents Review the policies and training plans The policies and training plans which which establish and reinforce that nuclear establish and reinforce that nuclear safety is an overriding priority, to verify safety is an overriding priority. Policies that those policies and plans require and and plans that reinforce that individuals reinforce that individuals have the right have the right and responsibility to raise and responsibility to raise nuclear safety nuclear safety issues through available issues through available means, means.

including avenues outside their organizational chain of command and to external agencies, and participate in the resolution of such issues.

Issue Date: 01/15/09 EA - 13 95003.02

NRC000043 Review records which identify the Training records on policies and plans personnel (including, as appropriate, that require and reinforce that individuals contractors) who have received training have the right and responsibility to raise on those policies within the last two nuclear safety issues through available years. Compare those records with means.

employee rosters.

Review the methods used by the licensee The documented methods used by the to communicate production, cost, and licensee to communicate production, schedule goals to employees, to verify cost, and schedule goals to employees that those methods reinforces the primary importance of nuclear safety.

Review the methods used by senior The documented methods used by senior managers and corporate personnel to managers and corporate personnel to periodically communicate and reinforce periodically communicate and reinforce nuclear safety such that personnel nuclear safety as the highest priority.

understand that safety is of the highest priority.

SELF-AND INDEPENDENT ASSESSMENTS - The licensee conducts self- and independent assessments of their activities and practices, as appropriate, to assess performance and identify areas for improvement. Specifically (as applicable):

! The licensee conducts self-assessments at an appropriate frequency; such assessments are of sufficient depth, are comprehensive, are appropriately objective, and are self-critical. The licensee periodically assesses the effectiveness of oversight groups and programs, such as the corrective action program, and policies.

! The licensee tracks and trends safety indicators that provide an accurate representation of performance.

! The licensee coordinates and communicates results from assessments to affected personnel and takes corrective actions to address issues commensurate with their significance.

Requirement Corresponding documents Verify that the periodic self- and The station self-assessment program and independent assessments conducted by schedule.

the licensee have been conducted at an appropriate frequency.

For a representative sample of those A representative sample of self-assessments, verify that the assessments.

assessments were of sufficient depth, are comprehensive, are appropriately objective, and are self-critical.

Verify that the licensee periodically An assessment plan and schedule of assesses the effectiveness of oversight effectiveness assessments of oversight groups and the CAP. For a sample of groups and the CAP. A sample of reports that document such assessments and the corrective actions assessments, verify that the taken.

assessments are of sufficient depth, are comprehensive, are appropriately objective, and are self-critical.

Issue Date: 01/15/09 EA - 14 95003.02

NRC000043 Review the safety indicators tracked by Performance indicator data for activities the licensee to verify that those indicators important to nuclear safety.

provide an accurate representation of performance.

For a sample of reports that document A representative sample of self assessments, verify that the issues assessment results and performance identified in those reports were indicator data for activities important to subsequently classified, prioritized, nuclear safety.

evaluated and addressed as appropriate.

WORK CONTROL - The licensee plans and coordinates work activities, consistent with nuclear safety. Specifically (as applicable):

! The licensee appropriately plans work activities by incorporating:

< risk insights

< job site conditions, including environmental conditions that may impact human performance; plant structures, systems, and components; human-system interface; or radiological safety

< the need for planned contingencies, compensatory actions, and abort criteria

! The licensee appropriately coordinates work activities by incorporating actions to address:

< the impact of changes to the work scope or activity on the plant and human performance

< the impact of the work on different job activities and the need for work groups to maintain interfaces with offsite organizations and communicate, coordinate, and cooperate with each other during activities in which interdepartmental coordination is necessary to assure plant and human performance

< the need to keep personnel apprised of work status, the operational impact of work activities, and plant conditions that may affect work activities

< the licensee plans work activities to support long-term equipment reliability by limiting temporary modifications, operator work-arounds, safety systems unavailability, and reliance on manual actions.

Maintenance scheduling is more preventive than reactive.

Requirement Corresponding documents Review the procedures used to manage Work control procedures for risk.

risk or control work to verify that the procedure requires risk considerations to be incorporated into work scheduling.

Review the procedures used to prepare Work package preparation procedures for work to verify that they require

! consideration of risk insights

! addressing job site conditions

! the impact of changes on the plant and human performance;

! the impact of the work on different job activities; and

! the need for planned contingencies, compensatory actions, and abort criteria.

Issue Date: 01/15/09 EA - 15 95003.02

NRC000043 Review the procedure(s) used to Procedures for scheduling work.

schedule and control work to verify that it Procedures for control of temporary includes features which appropriately modifications, operator work-arounds, limit temporary modifications, operator safety systems unavailability or work-arounds, safety systems degradation, and reliance on manual unavailability, and reliance on manual actions.

actions Review the procedure under which the Procedures for pre-job briefs and shift licensee conducts pre-job and shift turnover and briefings.

briefings to verify that it requires communication of the operational impact of work activities and plant conditions that may affect work activities.

Observe selected pre-job and shift Schedule of shift turnover meetings in all briefings to verify that those departments communications occur.

Observe selected meetings used to plan Schedules of work planning and and coordinate work activities to verify (1) coordination meetings.

that work groups communicate, coordinate, and cooperate with each other; (2) the free flow of information, including dissenting opinions; and (3) a strong presence by the operations organization and focus on operations=

priorities.

Determine whether licensee personnel Procedures and policies for making major have access to a process to appeal major decisions.

decisions.

Review the documents that include Procedures for interfacing with non-protocols between on-site and selected nuclear support groups providing off-site work groups (like work groups engineering and maintenance of nuclear who perform switchyard maintenance plant related structures systems or and coolant channel dredging) to verify components.

that the protocols provide adequate communication, coordination, and cooperation.

Interview the on-site personnel who Organization charts and contacts for staff administer selected interfaces with off- administering non-nuclear support site organizations to verify that the groups.

associated protocols are being followed.

Issue Date: 01/15/09 EA - 16 95003.02

NRC000043 WORK PRACTICES - Personnel work practices support human performance.

Specifically (as applicable):

! The licensee communicates human error prevention techniques, such as holding pre-job briefings, self- and peer checking, and proper documentation of activities.

These techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely. Personnel are fit for duty. In addition, personnel do not proceed in the face of uncertainty or unexpected circumstances.

! The licensee defines and effectively communicates expectations regarding procedural compliance, and personnel follow procedures.

! The licensee ensures supervisory and management oversight of work activities, including contractors, such that nuclear safety is supported.

Requirement Corresponding documents Review job preparation procedures and Procedures and training plans for working observe selected jobs to verify (1) that level work practices in all departments.

human error prevention techniques are used such as pre-job review of tasks, Schedule of pre-job briefs scheduled pre-job briefings, contingency planning, during the inspection.

peer verifications, etc., as appropriate to the work being performed; (2) the presence of peer-to-peer coaching and reinforcement; (3) that workers understand the risk impact of planned work, and discuss that impact in pre-job briefs.

Verify that employees receive training on Training records on FFD for plant staff.

fitness-for-duty policies and practices, Policies and procedures on fitness-for-and review those policies and practices, duty.

including behavior observation.

Review related records developed during Records of administrative actions taken the preceding 12 months to verify related to fitness for duty. Employee implementation of those policies and identification may be omitted.

practices.

Review policies or procedures which Policies, procedures and training records address proceeding in the face of addressing resolution of issues impacting uncertainty or unexpected circumstances completion of work.

to verify that related guidance is adequate. Verify that appropriate site personnel receive training on this topic, and that this topic is reinforced in pre-job briefs.

Review policies or procedures which Policies or procedures on procedural address procedural compliance to verify compliance.

that related guidance is adequate. Verify that appropriate site personnel receive Training plans and records on procedural training on this topic. Verify procedures compliance.

are followed during observation of work.

Issue Date: 01/15/09 EA - 17 95003.02

NRC000043 Review policies or procedures which Policies or procedures for supervisory address supervisory and management and management oversight of work oversight of work activities to verify that activities related guidance is adequate.

Interview selected supervisors and managers to determine whether they are able to spend sufficient time in the field.

If not, determine why not.

Review audits or performance metrics for Procedures or policies and audits or supervisory functions (e.g., access performance metrics for supervisory records which indicate time in the plant functions.

for managers, supervisory reports of observations of worker performance, etc.)

to verify that those managers and supervisors spend time in the plant.

Issue Date: 01/15/09 EA - 18 95003.02

NRC000043 Enclosure 95003.02-B Sample Questions for Safety Culture Components The following questions are examples of the types of questions which may be asked during focus group or individual interviews. Safety Culture components and questions selected for inspection should be based upon specific site circumstances, i.e., not all safety culture components and/or all questions will necessarily be applicable. The questions are not all-inclusive and the following list is not intended to be used in its entirety. The interviewer shouldn't simply read the questions to the focus group or interviewees as written; instead, the interviewer should understand the audience and reword/tailor the questions/topics for the audience.

The questions related to each safety culture component are organized here in alphabetical order by the title of the component, but the list of questions to be used in any particular interview or focus group can be organized differently. However, when developing the list of questions to be asked in interviews or focus groups, it is easier for respondents to answer groups of related questions than questions that skip from one topic to another. One way to introduce a group of related questions is with a statement such as, ANow let=s talk about decision making@ (or operating experience, work practices, etc.).

The first step in developing a good question is to identify exactly what kind of information you want the respondents to provide. This is important because it is easy to receive one type of information when you really want another. Virtually all questions are either open-ended or close-ended.

Open-ended questions are generally more demanding to answer, typically produce many different responses, and often produce information that can not be compared across respondents. These questions may be used when the safety culture assessor doesn't want to specify response choices or wants to (1) to give respondents a chance to state opinions, (2) let respondents vent frustrations, and (3) hear what has been overlooked, or further explore issues. Open-ended questions are helpful when they follow a close-ended question and ask respondents to explain details about their particular answer. Also, open-ended questions are desirable when respondents are being asked about behaviors.

Close-ended questions are those that provide specific choices and the respondent selects from the choices. These questions are generally less demanding to answer and the responses are easier to analyze and aggregate.

In summary, well worded questions asked in the appropriate format will make it easier for the respondent to answer. Using the opened and closed ended questions appropriately will provide useful information for the safety culture assessor. 5003-C provides additional guidance for conducting individual interviews and focus groups.

Accountability

a. Who is responsible for nuclear safety at this site? Explain.

b. Please describe your line of authority. Who revises your line of authority if necessary?

Issue Date: 01/15/09 EB - 1 95003.02

NRC000043

c. Do you have nuclear safety responsibilities in your job? If so, please describe what they are. Is nuclear safety incorporated into your job performance review?

If so, how?

d. In the past year, have you experienced any challenges or barriers to meeting your nuclear safety responsibilities? If so, please describe them. Have you had any particular successes in meeting your nuclear safety responsibilities? If so, please describe them.

e. Does the plant have any kind of program to reward staff for improving safety, such as spot awards for excellent work practices or awards for the best suggestions to improve safety? If so, please describe the program(s). Do you think they make difference in what people do around here? If not, what would be more effective?

f. Can you think of an instance from the past year in which your supervisor or a manager spoke directly to you about a nuclear safety issue? If so, please describe the circumstances. What was the nature of the discussion? Was a decision made to change anything about how you perform your work? If so, please describe the decision.

Continuous Learning Environment

a. On average, what is the proportion of staff time in your department that is devoted to training? What do you think about this proportion of time (e.g., too little, too much, right amount)? How does this compare with other departments at the site? How does this compare with other plants that you are aware of?

b. Is it ever necessary to cancel or reschedule training due to operational requirements? For what types of reasons does this occur? About how often does it occur? In the past several years, have you had planned training cancelled due to operational requirements? Were there any actions taken to reschedule the cancelled training? Who took these actions? Please describe what happened.

c. How does your management treat errors? Does your management consider errors as negative to the business, as learning opportunities, or both? Can you describe any examples of errors that were used as learning examples and/or errors that were considered to be negative for the business? If an error was used as a learning example, how was it discussed? Would you be more or less likely to report an error you or a co-worker made as a result? If an error was considered to be negative for the business, what happened? Would you be more or less likely to report an error you or a co-worker made as a result?

d. In the past several years, have you been involved in any benchmarking activities? If so, please describe them. Were any of your benchmarking ideas implemented? Please describe the ideas and how they were implemented or why they weren=t implemented. Does it seem that the changes have made a difference? If so, how?

e. Have other departments been involved in benchmarking? Were any changes made as a result of what they learned? Does it seem that the changes have made a difference? If so, how? (Limit this question to managers, supervisors or others who would know about benchmarking activities in other sections.)

Issue Date: 01/15/09 EB - 2 95003.02

NRC000043

f. Have you been involved in a major organizational change (e.g. reorganization, layoff, voluntary staff reduction/buy-outs, retirements), where knowledge was transferred to persons who might be new to a position? If so, was this done in a timely way (i.e., before the knowledgeable person leaves)? How well did this work?

Corrective Action Program

a. Are issues entered into the Corrective Action Program (CAP) prioritized correctly? Please explain. Is there any difference in the handling of an immediate nuclear safety issue vs. a more long-term nuclear safety issue at this site? Can you describe the types of nuclear safety issues that are immediately addressed if they arise? What types of nuclear safety issues are associated with longer response times? Can you think of any long-term nuclear safety issues which exist that in the past year either have not been appropriately evaluated or whose corrective actions have been inappropriately extended?

Please describe the situation. Do you know the reasons for delay in resolving the issue?

b. Is there a threshold for items identified for entry into the CAP? Is this threshold at the appropriate level? Are you aware of any abnormal or unusual or questionable conditions that may not be entered into the CAP but could potentially be indications of or precursors to nonconforming or degraded conditions?

c. Does your management encourage condition report (CR) initiation? How?

Where? Can concerns be filed anonymously through the CR process? What about the Employee Concerns Program process?

d. Describe how CAP trends information in the aggregate to identify common cause problems.

e. Please describe an example from the past year in which your supervisor or manager stopped work or delayed completing a task because of a nuclear safety issue or concern. What was the nature of the issue? How was it identified? How was it resolved? Were you personally satisfied with the resolution? If not, what would you have liked to see happen?

f. Do you believe that the site=s corrective action program is successful in addressing issues that are submitted? Can you provide an example related to your answer? Is the effectiveness of the implemented corrective actions evaluated? How? How frequently?

g. Are issues raised from CAP tracked to completion? Are initiators informed of the result? Can you describe any examples when this was true? If not, please describe what happened to the issue and why it wasnt completed or the initiators werent=t informed, if you know?

h. Have you ever submitted an issue to the corrective action program? Was the issue adequately addressed? If not, did you pursue the issue? Please explain.

i. In your experience, are issues in the CAP addressed in a timely manner? Were the issues addressed in a manner consistent with their importance?

j. Are you aware of any specific instances in which another employee (or contractor) submitted an issue to the corrective action program and considered Issue Date: 01/15/09 EB - 3 95003.02

NRC000043 the response s/he got to be incomplete or unacceptable? Please describe the situation.

k. Do you see any changes in the amount of time necessary to resolve corrective action issues over the past year? If so, do you have any ideas about the reasons for the changes?

l. Can you think of an instance from the past year in which you approached a supervisor or manager about a nuclear safety issue? If so, please describe the circumstances. What was the nature of the discussion? Was a decision made to change anything about how work is performed as a result of the discussion?

If so, please describe. If not, what was the reason(s) the decision was made not to make any changes?

m. Can you think of an instance from the past year in which you approached a co-worker or a co-worker approached you about a nuclear safety issue? If so, please describe the circumstances. What was the nature of the discussion?

Was a decision made to change anything about how work is performed as a result of the discussion? If so, please describe. If not, what was the reason(s) the decision was made not to make any change(s)?

n. When there is a problem to be solved that will affect several work groups or other departments, how is that handled? Who is usually involved with the resolution (e.g. employees or managers)? If the solutions are identified by managers, do employees have input? If so, how much? What would you do if a solution is identified that ins unworkable for you or your department?

o. Based on your experiences, does your manager fully understand technical and nuclear safety issues that you or members of your work group raise? How does your manager handle if s/he doesn=t understand your issue? Is your input solicited during the problem resolution if you enter an issue into the CAP?

p. How is the CAP assessed? What have been the results? Are such reviews effective? Please explain. How does CAP address causes and extent of conditions? Are such reviews complete? Please explain.

q. Describe any unexplainable change in the number or nature of issues raised by employees to the CAP.

r. How and at what point are employees who raised issues included in the corrective action process? Is this necessary?

s. What corrective action tracking systems exist that are not part of the official CAP? If so, why do they exist? Who uses them? What types of issues are entered into them? Do any of these tracking systems include items that are potentially important to nuclear safety or that should be in the official CAP?

Please give examples. Do these tracking systems get periodically audited to verify that any items that should be in the official CAP are put into that program?

t. Can anyone enter an issue into the CAP? When someone enters an issue into the CAP, does the entry have to be approved by a supervisor? Does anyone higher up also have to approve the entry before it goes through the remainder of the CAP process? Are you aware of any issues that someone wanted to enter into the CAP, but a supervisor or higher-level manager disapproved the entry?

Please describe the situation.

Issue Date: 01/15/09 EB - 4 95003.02

NRC000043 Decision-Making

a. When a situation arises that requires a choice between nuclear safety and production, how is the decision handled and who decides? Can you give any examples of situations in which there may be a trade-off or even a conflict between nuclear safety and production in your job (e.g., there was pressure to meet a schedule goal, but you or someone you know identified a problem which would delay the work)? Have you ever run into a situation like this? If so, what did you do? How did your supervisor react? How did your management react?

How did it work out? Would you do the same thing next time? If not, what would you do differently?

b. In your own words, what does the term, Aconservative decision-making,@ mean?

Based on your definition, can you give any examples where you have demonstrated Aconservative decision-making?@ If so, please describe. What about your supervisor? Your management? Please describe.

c. What do you take into consideration when making a decision on whether a situation is safe (in terms of nuclear safety) to continue operation? What about your supervisor? Your management?

d. Do you receive information on the basis of management decisions related to safety-significant or risk-significant decisions? If so, how would you characterize the level of information you receive (e.g., not enough, right amount, too much detail). Please explain.

e. What is the site process for making decisions related to safety-significant or risk-significant decisions such as whether a component is inoperable or whether a proposed design change assures safety? Does the process use a systematic approach? Is this process consistently followed? If not, please explain.

f. Can you recall an instance in which decisions were not made at the appropriate organizational level, and, as a result , a nuclear safety issue arose? Please describe.

g. Do management decisions regarding operational issues (such as changes to the scope of work or response to operational events) reflect the appropriate focus on safety? If so, please give examples.

h. Do management decisions related to deferred outage items, corrective vs.

elective maintenance, and treatment of emergent outage items reflect the appropriate focus on safety? If so, please give examples.

Environment for Raising Concerns (For working-level personnel)

a. Are you willing to raise a safety concern? Are there any conditions under which you would be hesitant to raise a safety concern? If yes, does that condition exist here at (Insert Plant Name)? Please elaborate.

b. Are you aware of situations in the past year, where any employee or contractor may have been hesitant to raise concerns, internally or externally? If yes, please explain. (If the NRC safety culture assessor is aware of a specific Issue Date: 01/15/09 EB - 5 95003.02

NRC000043 incident that may have caused such hesitation, then ask about it. Focus on whether or not the interviewee or others may be less likely to report concerns since that incident).

c. Where would you go to raise a safety issue? [The NRC safety culture assessor should be aware of the following avenues for raising concerns, but not prompt the interviewee: supervisor, corrective action program (CAP), alternative program (Employee Concerns Program (ECP)/Ombudsman), NRC or other avenue.] Why would you pick this avenue? Have you or others had any experiences, or know of any situations, that have influenced your decision to pick this avenue? If so, please describe.

d. Are there other avenues available to you for raising safety issues? Ask each of the questions listed in the following table for each avenue available:

Question Supervisor Corrective ECP/ NRC Other Action Ombudsman Program Have you ever submitted a safety issue to (insert method) If no, why not?

If yes, was the issue adequately addressed?

Why or why not?

If not adequately addressed, did you further pursue the issue? If not, why not?

Given the nuclear safety importance of the issue, did you receive timely feedback?

Describe any instances in which another employee who submitted an issue to (insert method) and you considered the response unacceptable?

e. Explain how to use the ECP/Ombudsman program. How did you learn about the ECP/Ombudsman program? Is the ECP/Ombudsman office accessible?

Too visible? Do you (routinely) see the ECP/Ombudsman Manager/Investigators around the plant? If yes, give examples. Is your management supportive of the ECP/Ombudsman program? If yes, how is such support demonstrated? If no, please describe what has led you to believe that they are not supportive.

f. Has your confidentiality been breached when a safety concern was raised to the ECP/Ombudsman? Describe. Do you know if anyone else=s confidentiality has been breached when a safety concern was raised to the ECP/Ombudsman? If yes, describe.

g. Would you say that your management is supportive of the SCWE policy? If yes, how is such support demonstrated? If no, please describe what has led you to believe they are not supportive.

Issue Date: 01/15/09 EB - 6 95003.02

NRC000043

h. Have events or circumstances occurred in the past year that have reduced: Your willingness to identify or raise safety issues? Your confidence in the corrective action program? Your willingness to challenge actions or decisions you believe are wrong? Your comfort level in voicing your viewpoints and opinions?

i. Have you received training concerning SCWE? If yes, describe what it covered.

When did you last receive SCWE training? Is there periodic SCWE refresher training? If yes, how often?

j. Did your supervisor receive any SCWE training? If yes, did you notice a difference in the way he or she handled concerns after the training?

k. If there has been an assessment of SCWE, were the results effectively shared with you? Were any actions taken with the results? If yes, please describe.

l. If actions have been taken by management to maintain or improve the SCWE, have they been effective? Why or why not?

Environment for Raising Concerns (For supervisors and managers)

a. Are you willing to raise a safety concern? Are there any conditions under which you would be hesitant to raise a safety concern? If yes, does that condition exist here at (Insert Plant Name)? Please elaborate.

b. Are you aware of situations in the past year, where any employee or contractor may be hesitant to raise concerns, internally or externally? If yes, please explain. (If NRC safety culture assessor is aware of specific incident that may have caused such hesitation, then ask about it. Focus on whether or not the interviewee or others may be less likely to report concerns since that incident).

c. Where would you go to raise a safety issue? [The NRC safety culture assessor should be aware of the following avenues for raising concerns, but not prompt the interviewee: supervisor, corrective action program (CAP), alternative program (Employee Concerns Program (ECP)/Ombudsman), NRC or other avenue.] Why would you pick this avenue? Have you or others had any experiences, or know of any situations, that have influenced your decision to pick this avenue? If so, please describe.

d. What are your management=s expectations of you regarding handling employee (safety) concerns (e.g. responsiveness, timeliness, availability, confidentiality) ?

How has your management documented these expectations? Please explain.

How are these expectations measured/assessed (e.g., performance appraisal)?

e. What are management=s expectations regarding employee behavior when raising safety concerns (e.g. peer-to-peer retaliation)? How are these expectations communicated?

f. How do you actively encourage your employees to bring concerns to you? Give examples (e.g. reward/incentive programs; communications).

g. Describe what an employee could do if he/she were not satisfied with how their concern was handled. For instance, is there an appeal process they may use?

If yes, have you ever been through such a process? Describe your experience.

How do you advertise this process to employees?

Issue Date: 01/15/09 EB - 7 95003.02

NRC000043

h. Have you been requested to protect the confidentiality of an employee raising safety concerns? If yes, how did you protect that? Are you aware of any instances where an employees= confidentiality was breached? If yes, describe.

i. As a manager, explain how you use the ECP/Ombudsman. Do you believe that the ECP/Ombudsman program is sufficiently independent from management?

How do you show support for the ECP/Ombudsman? Give an example. (Is there survey data that shows what percentage of employees believe their manager supports the ECP?)

j. How are contractors covered by your SCWE Policy? Are all contractors covered? How do you ensure that contractors working under your direction foster a SCWE? Ask if guidance exists for evaluating contractor SCWE programs.

k. Have you received training concerning SCWE? If yes, describe what it covered. If you have received SCWE training, when did you last receive it?

Describe any changes in the way you handled safety concerns brought to you after you took the SCWE training. Is there periodic SCWE refresher training? If yes, how often? What SCWE training do contractors receive? If none, why not?

l. Describe the tools that management uses to monitor overall SCWE performance. How are the results of these tools (e.g., SCWE surveys, etc.)

made available to the workforce?

m. Have actions been taken to maintain or improve the SCWE? Have they been effective? Why or why not?

n. Do you know what is expected of you in handling employee concerns? Do you feel able to effectively handle employee concerns? Are you held accountable for your handling of employee concerns? How?

Operating Experience

a. Are you able to find out what's going on relative to operating experience in the rest of the industry? from the NRC? Please describe how.

b. Does the site have a program for the collection of operating experience (OE) information? What about for evaluation? How are the programs communicated to the affected staff? How effective are the programs?

c. Can you think of an instance in the past year where external operating experience (OE) resulted in changes in how things are done here? If so, what was the event/problem/OE? Do you think the changes solved any problems or improved things?

d. Do you read relevant internal OE information? What about external OE information? How often? Is the information provided to you in a timely manner? Is it useful? Is it accessible? Can you quickly find what you need (without having to wade through everything)? (If a supervisor) Can you quickly find relevant information related to specific jobs to include in pre-job briefs? Do you have sufficient time to effectively use OE (i.e., is it an organizational priority?)? If no to the any of the above, please explain.

Issue Date: 01/15/09 EB - 8 95003.02

NRC000043

e. Can you think of an instance in the past year where OE information resulted in change to procedure(s)? If so, how often has this happened? Please provide an example and describe what happened.

f. Have there been any changes to training as a result of OE information in the past year? If so, Please provide an example and describe what happened.

g. Can you think of an instance in the past year where OE information did not receive attention? Was it something you thought was relevant? What was the topic? What would you have liked to see changed because of it? Please explain.

Organizational Change Management

a. Is there a change process for major organizational changes? What is that process? Please discuss. Is the process systematic? Please describe. In your experience, has the process worked well? Please provide an example.

b. Can you think of an instance in the past year in which management made a major organizational or resource change, such as a re-organization, a change in more than one senior-level manager at about the same time, layoffs, a hiring freeze or a hiring effort? If so, what was the change? How has the change worked out?

(Follow on questions 3 and 4, as appropriate)

c. How and when did you learn of the change? Were the reasons for the change communicated to you? In your experience, did the reasons for the change make sense?

d. Was the change actually carried out in the way you were told that it would be implemented? If not, were you informed of the reason(s) for the change of plans?

e. Is there a process for evaluating the impact of organizational changes on nuclear safety? If so, Explain that process. In your experience, has the process worked well? Please provide an example.

f. Have organizational changes (fill in with the specific change if applicable) impacted employees= willingness to raise concerns? Please discuss.

g. Are key management positions filled expeditiously or are Aacting@ managers common practice?

Preventing, Detecting, and Mitigating Perceptions of Retaliation

a. What are the organization=s policies regarding preventing and detecting retaliation and/or chilling effect?

b. Are you aware of any actions taken by your management to prevent and detect retaliation and/or a chilling effect? If so, were their actions effective in addressing the situation? Do you believe that management=s handling of the issues is consistent?

Issue Date: 01/15/09 EB - 9 95003.02

NRC000043

c. Are you aware of any instances in which another individual experienced a negative reaction for raising a safety issue? If yes, please describe the incident, including any information conveyed by management concerning the incident.

d. Are you aware of any specific instances in which another employee (or contractor) submitted an issue to the corrective action program or ECP and was retaliated against for pursuing the issue? Please describe the situation.

e. Are you aware of your company=s policy with regard to protecting employees against retaliations/discrimination for raising nuclear safety concerns? Does management tolerate retaliation of any kind for raising concerns? Please explain.

f. How do you/your management prevent retaliation or the perception of retaliation associated with disciplinary actions or changes to the plant organization to ensure actions do not chill others (e.g., communicate reasons for discipline)?

(This question is for managers)

g. How do you ensure that you don=t discourage the reporting of issues when there is pressure to meet production goals? (This question is for managers)

h. What actions can you take if an allegation of employment discrimination involving a contractor is made?

Resources

a. What methods does your site use to maintain plant safety long term? (e.g.,

maintenance of design margins, minimization of long-standing equipment issues, minimizing preventive maintenance deferrals, ensuring maintenance and engineering backlogs are kept low enough to support safety) Can you provide any examples or how it has (or has not) had an effect on resources?

b. Are there any nuclear safety initiatives or programs that your work group or team are currently involved in? If so, please describe the initiative/program. What is its purpose and goal(s)? What are the actions you are taking to accomplish the goal(s)? Do you have a sense of whether the program or initiative is being effective? Please describe. Does the program include ways to measure its effectiveness?

c. In the past year, have you lacked the type of tools, equipment, and other resources you need to perform your job? Can you think of an instance in the past year when you needed additional tools/equipment/resources to perform your work and were able/unable to obtain them? Please describe the request.

If it was turned down, what reasons were you given for the decision? What impact did the lack of the have on your work?

d. During the past year are you aware of situations in which the lack of staff, equipment, or facilities impacted the safe completion of a job? Are the people available qualified to perform the work? If no, why do you think that is? In the past year, can you think of any instances in which a lack of qualified staff, equipment, or facilities delayed the completion of work tasks? If so, please describe the circumstances. Are there any barriers to adding the qualified staff you need to get work done around here? Please describe.

e. Have you worked overtime under a deviation from the work hours policy in the past year? If so, which of the work hour limits was/were exceeded? What were Issue Date: 01/15/09 EB - 10 95003.02

NRC000043 the circumstances that required the extra work hours? How often does this occur?

f. Have you ever turned down overtime because you believed you were too tired to be safe? What response did your supervisor (or whoever you reported being too tired to) give you?

g. Have any instances occurred in which backlogs impacted the site=s ability to respond to nuclear safety issues? Please describe.

h. Are there adequate resources to ensure that procedures, work packages, and design documentation are complete, accurate, and up-to-date? If no to any of the these attributes, please describe.

i. Do you use the simulator for any of your work or training? How would you describe the difference in the performance of the simulator with the plant?

j. Are you aware of a emergency drill where the adequacy of the emergency facility impacted performance of personnel? If so, please describe.

k. In the past year are there any long standing equipment issues at the site that were not addressed, such as deferred maintenance and/or PM=s, deferred outage work or deferred emergent work, unaddressed operator burdens and control room deficiencies, or long-standing temporary mods? If so, please explain.

Safety Policies

a. Is there a vision/mission statement/policy that addresses safety? Is it clearly visible and understood? Where is that statement/policy located? What goals does that statement/policy specify?

b. Is there a separate policy on safety culture? If so, what do you think is the most important aspect of this policy? What is your reason? Is there a separate policy on safety conscious work environment? If so, what is the most important aspect of this policy? What is your reason? Specifically, how does management communicate these policies to the staff? How effective is this communication?

c. Other than policies, what tells you about the priority of nuclear safety at this site?

What is the message you receive? Have any additional nuclear safety goals have been formulated for your work? What are they? Have you used them to guide your work? If so, how?

d. Do decisions and actions at your level incorporate the safety goals we just discussed? How?

e. How do your managers balance production and safety? Can you give an example of a good balance between production and safety? An unacceptable example? If your work group has production goals, are these goals communicated in a manner that reinforces safety?

f. Have you received training concerning safety policies? Describe what it covered. What did you think of this training (e.g. useful/not useful, effective/not effective)? When did you last receive such training? Have any of your other training courses referred to the priority of nuclear safety? Which courses and which policies?

Issue Date: 01/15/09 EB - 11 95003.02

NRC000043

g. How are the safety policies reinforced? (e.g., by management?, goals?,

incentives? accountability?)

h. In your experience, how well do the managers at this site communicate in a clear way that nuclear safety is a high priority? Please provide an example of an effective communication. Did it change anything about how you think about your work or how you do it? If so, what changed?

Self- and Independent Assessments

a. Is self-assessment and improvement important at this site? Explain.

b. Have you ever been involved in a self-assessment or independent assessment?

If so, what was the subject? When was it done? Did you have the resources you needed to complete the task (e.g., time, qualified personnel to address the technical issues involved)? If not, please explain. Were the schedule and due dates for completing the assessment consistent with the complexity of the issue? Was the assessment of sufficient depth to address the complexity of the issue? Were the results and recommendations from your effort used? How?

Did the outcomes from the self-assessment lead to positive changes in how things are done? Did you experience any challenges while conducting the assessment? Please explain. Would you volunteer to become involved in one again? Please, describe your reasons.

if not, do you know someone who has been involved in a self-assessment who we could talk to?

c. Do you know what typically happens to results from self-assessments or independent assessments? If so, please explain. If the assessment makes recommendations, do you know what happens to those recommendations?

Please explain. Is there any tracking done on the recommendations? Please explain. Has there been a recent self-or independent assessment of your department? Were the results communicated to you? If so, What were the results/findings? What changes resulted from the findings of the assessment?

Were there any results/findings that were not addressed? Please explain.

d. Are you aware of an assessment of the effectiveness of the corrective action program? If so, did any of the findings impact your work? How?

e. In your job, do you track or trend performance indicators? What are those indicators? How are they used? Based on your experience, how would you describe the usefulness of the performance indicators? Are there any indicators you believe should be used but are not currently?

f. Are the results from self-or independent assessments typically communicated to you or affected personnel? How? If the results typically aren't communicated or you do not know, please describe the various assessments you know have been done in the past year. What do you think about not receiving this information?

Is not receiving this type of information a problem? Whose responsibility do you think it is to deal with communicating that information (management, yours, or both)? If you are aware of assessment results, could this communication be improved? How?

g. Have there been instances in which the results of some self- or independent assessment resulted in changes in how you do your job? What were the Issue Date: 01/15/09 EB - 12 95003.02

NRC000043 changes? Have the changes solved the problem(s) and/or improved things?

Was there something else that could have been done that would have worked better or been more effective?

h. What=s the reaction to independent assessments done by your QA department?

By external groups? By INPO? NRC inspections? Is this input valued by the organization? by management? by your peers? Is it used to improve?

Work Control

a. Has any change occurred in the nuclear-safety orientation of the site during the last several years? If so, has this had any effect on your work? How? Please describe.

b. What messages have you received from various people in this organization regarding the priority of safety? Please describe any different messages.

c. Does the site plan work such that job site conditions are adequate, including environmental, which may impact human performance; plant structures, systems, and components; human-system interface; and radiological safety?

(Divide this question in pieces as appropriate for the respondent) Please describe.

d. Has your supervisor provided you with Aabort criteria@ for situations or conditions in which you should stop work? What are they? Are you aware of any guidance with respect to Acompensatory actions@?

e. How does the site schedule emergent work? Can you give an example where is was not scheduled appropriately?

f. Do you get sufficient and timely information about what=s going on at the site and in your department from your supervisor? In the past year, can you think of an instance in which timely communication or lack of timely communication made a difference in how you perform your job? Please describe.

g. In the past year, have you found that the work you were assigned to do couldn=t be performed because it was scheduled in a way that it interfered or conflicted with other work being performed? Did the work have to be stopped or delayed?

Over the past year, about how often would you estimate that this has happened to you?

h. In the past year, have you had to make a decision with nuclear safety implications and did not have any procedure to follow? Please explain. Have you ever had to deviate from procedure in order to assure the nuclear safety of the site? Please explain.

i. Are you aware of an example of a repeat maintenance issue that occurred in which the licensee had previously resolved the issue with temporary modifications? Please explain.

j. Is preventive maintenance scheduled in a manner that promotes long term equipment reliability? Please provide an example.

k. Is your work impacted by Aoperator work-arounds@? If so, please describe how?

l. On a scale from 1 to 5, how reactive (rather then preventive) do you rate the plant site maintenance scheduling? When equipment failures occur, are the Issue Date: 01/15/09 EB - 13 95003.02

NRC000043 maintenance activities sufficient to address all aspects of the system which could have been impacted or are subject to the same failure types? Please explain.

m. Is there an appropriate balance between outage and on-line maintenance at this site? Are outages typically long enough to do all the necessary work? Is work sometimes moved from the outage to on-line maintenance to keep outages as short as possible?

Work Practices

a. In terms of safety, what is your personal approach to your own work? Who do you look to for guidance on nuclear safety issues?

b. What is the first thing that happens here when an event has occurred that seems to have been caused by human error? Can you give an example?

Based on your experience, what are the most common reasons behind human errors?

c. Are self and peer checking procedures used at the site? If so, how? Please describe your personal experiences. Can you think of any situations where they should be used but are bypassed? If so, please explain. Have you received any training on human error reduction techniques? How is the use of human error reduction techniques reinforced? Are you held accountable for using human error reduction techniques in your work?

d. Do you participate in pre-job briefs? If so, are pre-job briefs routinely held? For what types of work? Are they effective?

e. Over the past year, have you encountered a situation where you or someone else were getting ready to perform a job or were in the middle of performing it, and unexpected or uncertain conditions arose that were different from what you or the other person were led to expect? If so, about how often? Please provide examples and describe what you or the other individual did.

f. Do you personally have stop work authority? Have you used it or considered using it at any time within the past year? If so, please describe the situation. If you did stop work, what kind of reactions did you receive from co-workers?

Your immediate supervisor/manager? Higher-level management, if they became involved? Were you satisfied with your decision? Would you do it again?

g. What approach does this site take towards preparing for new and infrequently performed tasks? Is just-in-time training conducted for infrequently performed tasks? If so, how often? Have you ever participated in one of these training sessions? If so, please describe your experience. Do supervisors and/or managers typically provide oversight when new or infrequently performed tasks are being conducted?

h. Describe your fitness for duty program. Based on your experience, is it effective? Please explain. Based on your experience, is management supportive of the program?

i. If another department makes a change in how they perform their work that affects your work, are you informed of the change? How? Have there been Issue Date: 01/15/09 EB - 14 95003.02

NRC000043 instances where changes were made that directly impacted your job that you were not made aware of? If so, please describe.

j. Is it communicated to personnel which procedures require verbatim compliance?

Are such procedures followed? If not, please explain.

k. Do you get feedback on your nuclear safety performance from your supervisors?

Peers? Can you describe a recent (past year) situation in which you received feedback from your supervisor/peers? What was said or done to give you the feedback? Did you change anything as a result? Are you able to provide feedback on safety performance to your peers or supervisors? Please explain.

l. Do managers observe your work? If so, how often? What do they do when they are observing? Are you aware of any changes to how work is performed at the site that have resulted from management observations? If so, please describe.

m. Does the site provide oversight of contractors? If so, how? In your experience, has the oversight been adequate? Can you think of any examples where lack of oversight of contractors led to a safety issue? Please describe.

Issue Date: 01/15/09 EB - 15 95003.02

NRC000043 Enclosure 95003.02-C Guidance for Focus Groups and Individual Interviews This enclosure provides information about the strengths and limitations of different interview types and methods. It also provides guidance for conducting individual and group interviews.

This enclosure is arranged in the following five parts:

Part Contents 1 Individual Interviews 2 Group Interviews 3 Structured Interviews 4 Unstructured Interviews 5 Sampling Guidance Note that both individual and group interviews may be either structured or unstructured.

The first four parts are subdivided into Overview, Strengths, and Limitations subsections. Parts 1 and 2 also include a Guidance subsection. Part 5 presents considerations in selecting samples of participants for interviewing.

1. Individual Interviews

a. Overview Individual interviews consist of collecting information in a face-to-face, one-on-one setting where an interviewer poses a series of questions/topics to the interviewee and records the information provided. The degree of structure in the questions/topics can vary. (See parts 3 and 4.)

b. Strengths (1) Permits detailed exploration of an individual=s values, perceptions, attitudes, and views about the organization=s norms.

(2) Preferred method when information is more sensitive or personal and anonymity can be assured.

(3) Useful when additional probing on answers is required.

(4) Interviewer has good control over the session in terms of topics discussed and detail provided.

c. Limitations (1) Time-consuming and resource-intensive.

(2) Quality of interview data strongly influenced by interviewer skills.

Issue Date: 01/15/09 EC - 1 95003.02

NRC000043 (3) Interviews with approximately 10% of the population are needed to adequately understand existing issues.

(4) Information from a single interviewee cannot be considered unless validated through other sources of information.

d. Guidance Complete individual interviews as follows:

(1) Note that the main objectives of the interview is to gain opinions on:

(a) the licensee=s actual awareness of nuclear safety issues; (b) the safety-related attitudes of management and staff; (c) the degree of compliance with policies and procedures; (d) the possible reasons for observed inconsistencies or contradictions between actual and expected behavior, thus gathering data on social norms, beliefs, and values relevant to safety culture.

(2) Prepare an interview plan in advance of the interview, and follow that plan to conduct the interview.

(3) Conduct the interview in 3 stages, as follows:

(a) Establish a relationship of trust and cooperation with the interviewee (b) Gather interview data (c) Tell the interviewee that the results of all the interviews are captured and considered along with other results, and that observations, findings, and conclusions are documented in the Inspection Report.

(d) Discuss and evaluate issues raised by the interviewee (4) After the interview, prepare a written summary of the interview results, with emphasis on the issues raised by the interviewee and insights gained by the safety culture assessor.

2. Group Interviews

a. Overview Group interviews (i.e., focus groups) consist of collecting information in a face-to-face, group setting, where an interviewer poses a set of questions/topics to the participants and records the information provided. The degree of structure in the questions/topics can vary.

b. Strengths (1) Group interaction can prompt/sustain discussions without a high level of interviewer input.

Issue Date: 01/15/09 EC - 2 95003.02

NRC000043 (2) Efficient - requires fewer team resources than individual interviews to obtain adequate sample size.

c. Limitations (1) Should not be used (1) for sensitive or personal topics, (2) when there is evidence of conflict within or between work groups, or (3) when the participants have concerns about anonymity and confidentiality of responses.

(2) Interviewer does not have a high degree of control over the session; time may be lost on irrelevant issues. Participants will react to others=

statements in addition to the interviewer=s questions.

(3) The interview session may be dominated by the views of a minority of the participants.

(4) Participants may feel pressure towards consensus.

(5) Qualitative data may be difficult to analyze.

d. Guidance (1) The typical amount of time to conduct the focus group interview will range from 90 - 120 minutes. Open each focus-group interview by introducing the safety culture assessors who are in the focus group. Ask the participants to introduce themselves (first names should be sufficient) and describe how long they have worked in their current organization, as well as, on site in total. This information (time in current organization and time on site) will provide context to the data collected during the focus group interview. Verify that the attendees are as selected by the safety culture assessors. Question the inclusion of substitutes. If supervisory or management personnel are among the working-level attendees, ask them to leave and invite them to meet privately with the safety culture assessors at another time or check if they are in the supervisory focus groups. Then tell participants:

(a) that the purpose of the focus group is to determine whether and how underlying issues contributed to the performance deficiencies which prompted this inspection; (b) how interview participants were selected; (c) that the focus group will consist of discussions prompted by questions; (d) indicate that, to the extent possible, information obtained during the inspection will not be attributed to individual participants who are interviewed by the NRC. Remind the participants that the discussion is occurring in a group so others will hear what is said.

Also, state that if an individual provides details about a specific event in which he or she was uniquely involved, and if the NRC includes those details in their writeup of this inspection, then those details may identify the participant who provided the details; Issue Date: 01/15/09 EC - 3 95003.02

NRC000043 (e) all records held by the NRC of people participating will be destroyed after the focus group; that the NRC expects that cooperating with the NRC inspection, including participating in the focus group should not be used in any way to threaten, punish, or retaliate against an individual. The safety culture assessor should explain that if the individual wishes to discuss issues in a separate private interview, the safety culture assessors or residents if the safety culture assessors are no longer onsite, will make themselves available.

Such concerns may be handled as allegations (e.g., claims of retaliation or wrongdoing) or included as input into the ongoing inspection, as appropriate.

(f) that the results of all the interviews are captured and considered along with other results, and that observations, findings, and conclusions are documented in the Inspection Report.

(2) From the set of questions developed or selected for this interview (see parts 3 and 4), ask one of the questions and invite participants to respond.

Ask followup questions as necessary to clarify responses and encourage discussion from other participants. Continue followup questions until the group=s response is complete and understood, and the team has obtained from the participants as many insights into safety culture components as are reasonably associated with that question. After that, ask another question.

(3) Continue question-and-answer conversations as described above, for as long as participants willingly respond, or until the planned questions are all asked, subject to the following:

(a) Encourage all participants to speak up. Ask the same question or a variation of it to at least one other person.

(b) If any participant(s) does not answer any question directed to the group, direct a question or two to that participant, and encourage him or her to say what he or she thinks.

(c) If any participant(s) appears reluctant to answer a question or if his or her answers suggest an underlying issue or concern, make note of the question and the answers (if provided), for later follow-up. Do not persist if someone shows continued reluctance to speak.

(4) Close the focus group by thanking the participants for their participation.

Tell them that if:

(a) they have anything else to say that they did not feel comfortable saying during the session, (b) they later want to clarify or revise something they said during the focus group, (c) they later remember something they wish they had said during the focus group, or Issue Date: 01/15/09 EC - 4 95003.02

NRC000043 (d) they want to talk about something that wasn=t discussed during the focus group, then they should contact the safety culture assessors, and tell them how to do that.

(5) After closing the focus group, review the focus group notes, and add relevant safety culture assessor observations and comments.

(6) If any participant=s response identified a concern or issue related to a safety culture component, or if any discussion suggested or otherwise indicated a concern or issue related to a safety culture component, then address that concern or issue as described in 02.09. NOTE:

Information that reflects negatively on licensee performance or safety culture, if provided in the context of the purpose of the inspection, is not an allegation. However, information that describes an inadequacy in licensee performance, which is specific and outside the scope of the inspection, may be an allegation if the NRC has not already assessed the validity of the issue. Any issues related to wrongdoing, whether provided by licensee management, licensee employees or contractors, or NRC staff, are allegations. For more specific guidance, refer to Management Directive 8.8, AManagement of Allegations@, or contact the appropriate regional allegation coordinator.

3. Structured Interviews

a. Overview Structured interviews consist of using a pre-defined set of questions that are consistently asked of each interviewee or of subsets of interviewees. Can be done in a face-to-face or group setting.

b. Strengths (1) Ensures similar topic areas are explored across multiple interviewees.

(2) Reduces differences in the process followed across interviewees and interviewers that could bias the results.

(3) Semi-structured interviews still allow follow-up questions and more in-depth probing of a topic.

(4) Ensures all key topics are addressed during the allocated interview time.

(5) Provides a basis for comparison between respondents.

c. Limitations (1) If conducted as a fully structured interview, does not allow for additional follow-up on issues that arise.

(2) The question sets used require careful consideration.

4. Unstructured Interviews

a. Overview Issue Date: 01/15/09 EC - 5 95003.02

NRC000043 Unstructured interviews consist of an interviewer asking interviewees a series of questions that are developed as the interview is conducted. Can be done in a face-to-face or group setting.

b. Strengths (1) Gives interviewer complete freedom to fully explore topics of interest that arise.

(2) The interviewee largely guides the interview in terms of discussion topics.

This may lead to the identification of previously unknown issues.

c. Limitations (1) Provides no basis for comparison between respondents.

(2) The interview process is more likely to be influenced by the style and biases of interviewer.

(3) Information collected may not be highly relevant to the assessment.

(4) No framework is available to guide the interviewer.

5. Sampling Guidance

a. Develop a sampling plan that is informed by the findings of the licensee=s independent safety culture assessment and review of background material.

b. Include all functional groups at the site and possibly some corporate functional groups in the sampling plan.

c. Select licensee and contractor participants using a stratified random distribution (as described in (1) and (2) below) of personnel from organizational rosters, focused on specific groups as appropriate. Nominally:

(1) For the work group(s) most closely associated with the performance deficiencies, plan to interview approximately 20% of the working- and supervisory-level personnel.

(2) For the major functional groups that were not closely associated with the performance deficiencies, plan to interview approximately 10% of the working- and supervisory-level personnel.

(3) Plan to interview all heads of functional groups within the licensee=s management structure, and all managers organizationally above those individuals.

(4) Work with the licensee to select permissible substitutes for selected participants, based on unavailability due to shift work, vacation, sickness, or press of duty.

Issue Date: 01/15/09 EC - 6 95003.02

NRC000043 Enclosure 95003.02-D Guidance for Structured Behavioral Observations This enclosure provides guidance for developing and using structured behavioral observation checklists to identify patterns of behaving related to the components of safety culture. This enclosure does not provide behavioral observation checklists for the operating experience and self-assessment safety culture components because they are process-type components and don=t lend themselves to behavioral observation.

1. Overview Behavioral Observation Checklists involve the use of a structured format to record observational data. Key observable attributes of behaviors associated with safety culture are listed in checklist fashion, which ensures structured collection of data associated with observations. The structure also allows quantification of observational information. Behavioral Observation Checklists may also be used to guide and focus observations without quantifying the information collected.

2. Strengths

a. Data collected reflect real activities (versus respondent opinions or perceptions).

b. Multiple observations of similar activities (e.g., turnovers) allow quantification of information across multiple occurrences of the activity.

c. Observer is non-intrusive and does not interrupt activity.

d. Checklist format ensures similar information will be collected across multiple observers.

e. When quantitative data are not obtained or cannot be reported due to limited observations, qualitative data can be useful.

3. Limitations

a. Observer=s presence may affect the manner in which the activity is conducted.

b. In some cases, multiple observations of a similar activity are not possible.

c. Unless multiple observations of a similar activity are conducted, quantitative data cannot be reported.

d. Those observed may avoid discussing any sensitive topics in the presence of the observer.

4. Applications To be completed when observing:

a. licensee decision-making processes, including goal-setting, oversight, and work planning sessions;

b. the actual performance of work activities, including activities for which formal procedures and standards of behavior exist; Issue Date: 01/15/09 ED - 1 95003.02

NRC000043

c. communications, including interactions between managers and staff, between peers, as well as interdepartmental, intradepartmental and external communications; and

d. training.

5. Guidance:

a. Identify the categories of activities that will be observed. Select activities to observe based on their relevance to specific safety culture components to be assessed with this data-collection method, as defined in the assessment plan.

b. Through discussion with knowledgeable licensee personnel, identify the frequency with which the selected activities typically occur and determine the number of observations to be scheduled for each category of activity. If structured behavioral observation will be the primary method of collecting data about a specific safety culture component (e.g., decision-making), plan to observe a minimum of 25 activities of interest over the course of a one-week inspection. A minimum of 15 observations may be sufficient if behavioral observation will be used as a supplement to other information-gathering methods.

c. To develop consistency in using the checklists among different observers,

1. discuss the checklist items in advance and determine how they will be used;

2. jointly observe several of the same activities;

3. compare the results obtained by the different observers when observing the same activity;

4. discuss and resolve any differences in how the checklist items were interpreted;

5. revise the checklist items, as necessary.

d. For each category of activity to be observed, select a subset of the checklist items below or develop additional items, based on the nature of the activity and the safety culture components to be assessed. Do not plan to collect data about all of the safety culture components from any one observation, because the behaviors associated with some components do not occur with sufficient frequency to be provide an adequate sample of observations (e.g., budget planning meetings involving corporate and site management that might provide insights related to the Resources component).

e. Include no more than 15 items on a single checklist. Longer checklists are difficult to use and searching for items on the checklist can distract the observer.

f. Use the same checklist items when observing activities that fall into the same category of activities, so that the frequencies of the behaviors of interest can be determined.

g. For activities performed frequently during the inspection (e.g., shift turnover, pre-job briefs, and surveillance and maintenance activities), plan to observe up to 25 of the activities during the inspection.

Issue Date: 01/15/09 ED - 2 95003.02

NRC000043

h. For infrequently performed activities (e.g. weekly management/staff meetings, all-hands meetings, personnel action meetings) plan to observe a sample of convenience (i.e., perform the observation if one occurs during the inspection and if safety culture assessors are available at the time.)

i. Maintain the checklists used for each observation, even if no data were collected, in order to document the sample size.

j. For each checklist created, the safety culture assessor should note:

1. the date and time of the observation;

2. the activity observed (e.g., pre-job briefing, shift turnover, plan-of-the-day meeting, department meetings, a maintenance job, corrective action review meeting);

3. the levels of management and staff involved (e.g., senior management, functional area management, middle management, first-line supervisors, staff or contractors);

4. the functional area(s) involved (e.g., operations, maintenance, radiation protection, engineering);

5. the number of individuals involved, and

6. other characteristics of the activity that can be used to compare and contrast data collected from different activities.

k. Provide space on the checklist for the safety culture assessor to add notes that record more details about the interactions observed. For example, one of the checklist items below asks, "Was risk or nuclear safety discussed?" If the answer is yes, the safety culture assessor should add a description of the context in which risk or safety was discussed, the extent of the discussion, and an assessment of it. However, the additional information should be recorded only after the observation is completed, in order to ensure that the safety culture assessor is not distracted from observing.

l. Following the observation, the safety culture assessor should also document any qualitative assessment of the interaction or work activity observed, related to the safety culture components. This information will be necessary to ensure that the observation data are appropriately interpreted.

m. When all observations have been completed, summarize the following:

1. The number of observations made of each category of activity;

2. The extent to which behaviors were observed that are consistent with the safety culture components;

3. The extent to which behaviors were observed that are inconsistent with the safety culture components; and

4. Any qualitative information necessary to interpret properly the quantitative data.

Issue Date: 01/15/09 ED - 3 95003.02

NRC000043 This information can then be used to assess how the components of safety culture are integrated into day-to-day activities. This information is useful in assessing the overall safety culture as well as the safety culture of individual functional groups.

Example checklist items:

Accountability (Observed during ongoing work activities.)

Are the personnel who are performing the activities given specific success criteria that define organizational expectations before beginning the work?

Yes No N/A ____

If yes, nuclear safety was was not among the expectations.

Is performance feedback timely, so that corrections in performance can be achieved?

Yes No N/A If yes, did any feedback concern nuclear safety? Yes No N/A Is performance feedback available from verbal communication or performance evaluation reports generated at a later date  ?

If yes, did any feedback concern nuclear safety? Yes No N/A Did any supervisor offer performance feedback related to nuclear safety?

Yes No N/A Did any manager offer performance feedback related to nuclear safety?

Yes No N/A Did any peers offer performance feedback related to nuclear safety?

Yes No N/A If it was necessary to deviate from the originally planned activities, did the personnel performing the activities have the authority to approve the deviation?

Yes No N/A If yes, did the deviation have nuclear safety implications?

Yes No N/A If the work is being performed by a crew, is there an obvious structure to the group (i.e.,

there is a clearly identified group leader and specified roles and responsibilities for each of the other group members)? Yes No N/A Were the personnel selected to perform the activities familiar with the task requirements or was there obvious uncertainty regarding the tasks to be performed  ?

(To be observed during meetings.)

Were the specific individuals responsible for implementing the initiative, project, or program under discussion present? Yes No N/A Was the individual given an opportunity to present discuss or defend his or her position?

Issue Date: 01/15/09 ED - 4 95003.02

NRC000043 Yes No N/A If the responsible individual was present, did s/he receive any feedback related to nuclear safety? Yes No N/A If yes, was the feedback provided by (check all that apply):

Peers Supervisor Manager If the responsible individual was present, did s/he receive any feedback related to deadlines, costs, quality or other performance criteria? Yes No N/A If yes, was the feedback provided by (check all that apply):

Peers Supervisor Manager Corrective action program (CAP)

(Typically observed during issue screening, management screening, or closure meetings)

Which of the following individuals participated in the meeting?

Corporate management ____

Senior management ____

Functional area management ____

Middle management ____

Licensee staff ____

Contractor ____

Other (describe)? ____

Were screening criteria used? Yes ____ No ____ N/A ____

Were the screening criteria conservatively applied for every issue?

Yes ____ No ____ N/A ____

Did anyone challenge how any of the criteria were being applied?

Yes ____ No ____ N/A ____

Did anyone challenge the prioritization of any issues?

Yes ____ No ____ N/A ____

Were any issues upgraded or downgraded in priority? Yes ____ No ____ N/A ____

Upgraded ____ Downgraded ____

If so, did anyone challenge the change?

Yes ____ No ____ N/A ____

Were issues thoroughly discussed? Yes ____ No ____ N/A ____

Was safety, as applicable, considered for every issue? Yes ____ No ____ N/A ____

Issue Date: 01/15/09 ED - 5 95003.02

NRC000043 Were there any issues where it was decided not enough information was available to make the prioritization? Yes ____ No ____ N/A ____

If so, were any individuals directly involved with the issue consulted or plans made to consult the individuals involved? Yes ____ No ____ N/A ____

Did the reviewers have an understanding of the evaluation (i.e., they reviewed the evaluation prior to the meeting)? Yes ____ No ____ N/A ____

Did the reviewers place safety as the highest priority? Yes ____ No ____ N/A ____

Was there any discussion about the scope of the evaluation (i.e., what areas the evaluation covered)? Yes ____ No ____ N/A ____

What about depth (i.e., how thorough/in-depth the issue was investigated)?

Yes ____ No ____ N/A ____

Did any reviewer raise any concerns about problems not being adequately investigated in the evaluation? Yes ____ No ____ N/A ____

Was there any discussion on if the corrective action(s) presented could resolve all the problems identified in the evaluation? Yes ____ No ____ N/A ____

Did any reviewer interact with the evaluator(s) of the issue?

Yes ____ No ____ N/A ____

If so, did the reviewer(s) behave at any point in a way that could potentially discourage the evaluator from performing a thorough/in-depth investigation in the future? Yes ____ No ____ N/A ____

Were there any evaluations not accepted by the reviewers?

Yes ____ No ____ N/A ____

Is yes, was resolution on what to do about the evaluation reached?

Yes ____ No ____ N/A ____

If yes, was it through consensus-seeking ____ or top-down direction from management ____ ?

If no, was it decided to push the decision up the management hierarchy ____ or not ____?

If it was determined that the evaluation should have any rework done:

Was guidance provided on how to improve the evaluation?

Yes ____ No ____ N/A ____

Were any additional resources (e.g., training, additional evaluators, management assistance) offered to the evaluator(s)?

Yes ____ No ____ N/A ____

Were there any concerns raised about the new deadline?

Yes ____ No ____ N/A ____

If yes, who raised the concern? Reviewer(s) ____ Evaluator(s) ____

Continuous Learning Environment Issue Date: 01/15/09 ED - 6 95003.02

NRC000043 (When observing training.)

Is the training a result of an event or incident that occurred at the facility due to a human performance problem? Yes No N/A ____

Do trainees appear hesitant to ask questions or seek clarification?

Yes No N/A ____

Do trainees appear to be engaged? Yes No N/A ____

Do trainees have an opportunity to offer feedback about the training?

Yes No N/A ____

Are trainees evaluated at the completion of training?

Yes No N/A____

Are trainees provided with feedback while the training is ongoing?

Yes No N/A ____

Are lessons learned from internal or external operating experience incorporated into the training?

Yes No N/A ____

Is nuclear safety addressed during the training? Yes No N/A ____

Decision-making (and Organizational Change Management, as applicable)

(May be observed in scheduled or informal meetings or during ongoing work activities.)

Did the decision involve technical , policy , or personnel issues?

Were any uncertainties discussed? Yes No N/A ____

Were alternatives generated or not  ?

Was Arisk@ or nuclear safety discussed? Yes No N/A ____

Were conservative assumptions used? Yes____No____ N/A ____

Were any alternatives rejected because of risk or nuclear safety considerations?

Yes No N/A____

Was resolution reached or not  ?

If resolution was reached, was it through consensus-seeking or top-down direction from management  ?

If resolution was not reached, was it decided to push the decision up the management hierarchy or not  ?

If resolution was not reached, was it decided to seek more information or not  ?

Issue Date: 01/15/09 ED - 7 95003.02

NRC000043 If nuclear safety was involved, was the decision based on sufficient evidence that it was safe to proceed? Yes No N/A ____

If nuclear safety was involved, was the decision based on sufficient evidence that it was unsafe to proceed? Yes No N/A ____

If the decision concerned policies, rules, and goals, did the manager consult with his/her immediate subordinates? Yes No N/A ____

If the decision concerned staffing, did the manager consult with his/her immediate subordinates? Yes No N/A ____

If the decision concerned a technical issue, did the manager consult with any technical staff? Yes No N/A ____

If the decision concerned how to solve a work-related problem, did the individual consult his/her superior? Yes No N/A ____

Was a plan made for communicating the results of the decision?

Yes No N/A____

If yes, was communicating with the affected individuals discussed?

Yes No N/A ____

If yes, was communicating with a higher management level discussed?

Yes No N/A____

Were any previous, similar decisions discussed? Yes No N/A____

If yes, was the effectiveness of the previous decision discussed?

Yes No N/A ____

Environment for Raising Concerns (Observed during any interactions among site personnel.)

Did a subordinate(s) ask any questions of a superior during the interaction?

Yes No N/A____

If yes, did the superior answer the question(s)? Yes No N/A ____

Did a subordinate(s) raise any concerns to a superior during the interaction?

Yes No N/A ____

If yes, did the concerns involve (check all that apply):

nuclear safety radiological or industrial safety resources (e.g., staff, expertise) scheduling or deadlines other If yes, did the superior address the concerns? Yes No N/A ____

If yes, did the superior resolve the concerns? Yes No N/A____

If yes, was the supervisor=s response open and non-defensive?

Issue Date: 01/15/09 ED - 8 95003.02

NRC000043 Yes No N/A____

Did a subordinate offer any suggestions to a superior during the interaction?

Yes No N/A ____

If yes, did the superior discuss the suggestion(s)? Yes No N/A ____

Was the interaction obviously strained , obviously pleasant , or was there no apparent affect  ?

Was the interaction related to a safety issue , regulatory requirement(s) ,

production issue(s) , personal conflict , other  ?

Did the interaction include discussion of ways to improve the facility performance?

Yes No N/A ____

Did the interaction include discussion of ways to improve personnel performance?

Yes No N/A____

Did any staff member self-report an error? Yes No N/A____

If yes, did peers react favorably? Yes No N/A ____

If yes, did supervisor(s) react favorably? Yes No N/A____

Preventing and Detecting Retaliation (Observed during management or oversight meetings.)

Was there a rigorous investigation of the potential issue? Yes No N/A ____

Did the disposition seem appropriate? Yes No _N/A ____

Was the potential for the action to discourage the reporting of concerns discussed?

Yes No _N/A____

If yes, mitigation actions were were not assigned.

Resources (Observed during ongoing work activities. See also procedures-related items in Work Practices.)

Did personnel have problems reading the work package (legibility)?

Yes No N/A ____

Did personnel have problems interpreting the information in the work package?

Yes No N/A____

Was any information missing from the work package? Yes No N/A ____

Were an adequate number of staff available to perform the work?

Yes No N/A ____

Were the procedures adequate to perform the work? Yes No N/A Issue Date: 01/15/09 ED - 9 95003.02

NRC000043 Did personnel have the equipment necessary to perform the work safely?

Yes No N/A ____

Safety Policies (Typically observed in scheduled meetings.)

Was nuclear safety discussed as a goal? Yes No N/A____

Were goals other than nuclear safety discussed? Yes No N/A____

Goals were were not prioritized?

Nuclear safety was was not assigned the highest priority.

Were any target levels attached to the goals? Yes No N/A____

If goals were being set on a departmental level, were overall organizational goals factored in? Yes No N/A ____

If yes, nuclear safety was was not one of the goals.

If goals were being set on an organizational level, were corporate goals factored in?

Yes No N/A If yes, nuclear safety was was not one of the goals.

Was there overall agreement among the individuals setting the goals on what the goals and priorities should be? Yes No N/A ____

Was there any indication that the goals of different departments were in conflict?

Yes No N/A____

If nuclear safety goals were discussed, the following individuals brought them up:

Corporate management Senior management Functional area management Middle management Licensee staff Contractor Other (describe)

If production goals were discussed, was the potential impact on nuclear safety mentioned?

Yes No N/A ____

Work Control (Observed during ongoing work activities or a work planning session.)

Issue Date: 01/15/09 ED - 10 95003.02

NRC000043 When planning a work activity, were the following issues discussed (check all that apply)?

risk insights defense in depth job site conditions that could impact human performance task sequencing to optimize system availability potential impacts on nuclear safety of performing the activity at the same time as other activities are performed contingencies compensatory actions conditions under which the work would need to stop for nuclear safety reasons the impact on nuclear safety of any temporary modifications to be installed the impact on human performance of any operator work-arounds to be created any relevant internal or external operating experience A pre-job briefing was was not conducted. If it was conducted, were the following issues discussed (check all that apply)?

risk insights and/or nuclear safety considerations defense in depth job site conditions that could impact human performance and means to mitigate their potential effects contingencies for mitigating the effects of mistakes and/or possible worst-case scenarios procedure usage requirements other work activities that have the potential to interact with this one conditions under which work would be stopped for safety reasons communications requirements applicable lessons learned from internal or external operating experience When performing a work activity simultaneously with other work activities that had the potential to interact, communications were were not maintained between the individuals/groups performing the different activities.

When performing the work activity, unexpected conditions did did not arise.

Work Practices (Observed during ongoing work activities.)

Are there obvious time pressures for work completion? Yes No N/A ____

If obvious time pressures exist:

Do they appear reasonable given the activities to be performed?

Yes No N/A____

Is there evidence that those pressures compromised the quality of the work performed in any way? Yes No N/A____

Is there evidence that those pressures compromised the safety of the work performed in any way? Yes No N/A____

Issue Date: 01/15/09 ED - 11 95003.02

NRC000043 Were time constraints for the work activities clearly communicated to all individuals involved in the activity? Yes No N/A ____

The reason for the time constraints is related to (check all that apply):

nuclear safety concerns limited personnel resources other scheduled work activities pressure to get the facility back on-line other/unknown Human error prevention techniques were were not used.

Human error prevention techniques were were not discussed during the pre-job brief.

Were procedures used in performing the activity? Yes No N/A____

If procedures were used, were they conveniently located and easily accessible?

Yes No N/A____

Verbatim compliance with the procedures was was not required.

If verbatim compliance was required, was it achieved? Yes No N/A ____

If verbatim compliance was not achieved, (Note - these items relate to Resources.)

was it because the activities described by the procedure could not be performed as written, given the conditions (e.g., time constraints, personnel resources, unexpected conditions)? Yes No N/A ____

was it because the procedures not well understood or understandable?

Yes No N/A ____

The formal process for deviating from a procedure was was not followed.

Were any problems encountered during performance of the work activities?

Yes No N/A____

If yes, did the problems have any nuclear safety implications?

Yes No N/A ____

Work was was not stopped until the problem was resolved.

If a management decision or additional expertise was required to solve the problem, were the necessary individuals made available within a reasonable time period?

Yes No N/A____

Did any personnel point out conditions that could adversely impact nuclear safety?

Yes No N/A ____

Did any personnel point out behaviors that could adversely impact nuclear safety?

Yes No N/A____

Issue Date: 01/15/09 ED - 12 95003.02

NRC000043 Were any work-arounds used? Yes No N/A ____

If yes, was the work-around long-standing or created for the current work activity  ?

Was it proceduralized? Yes No N/A ____

If the work activity was considered critical, was management present?

Yes No N/A ____

If yes, did management offer direction or feedback  ?

Was the direction or feedback related to nuclear safety?

Yes No N/A ____

Issue Date: 01/15/09 ED - 13 95003.02

NRC000043 Enclosure 95003.02-E Guidance for Safety Culture Event Follow-up Studies This enclosure provides guidance for selecting and performing event follow-up studies to identify consistencies in attitudes and behaviors related to the components of safety culture.

1. Overview An event follow-up study is an in-depth investigation and analysis of an organizational event (e.g., a high-visibility disciplinary action, a significant management change, a human performance problem that resulted in an operational event) or organizational condition (e.g., weaknesses in the safety culture components). Event follow-up studies provide an opportunity to trace the progression of a single event, or the development of an organizational condition, using multiple methods, to observe how organizational behaviors impact the facility=s ability to cope with that event or condition.

2. Strengths

a. Allows for a thorough examination of a particular situation.

b. Results are documented in a narrative format providing valuable examples to support the overall findings of the assessment.

c. Most effective when the activity to be tracked is identified early in the assessment and at the beginning stages of the activity, although retrospective analyses are possible.

3. Limitations

a. Results cannot be generalized beyond the single situation studied.

b. Requires sufficient time devoted by one to two safety culture assessors, detracting from time available for other assessment activities.

c. Detailed information on the organization=s assessment of the activity or event may not yet be available.

d. If the event follow-up study requires retrospection, biases may be introduced by the effects of intervening events on individuals= memories.

4. Example Applications

a. Understanding the history of a particular functional group or specific work unit that may be demonstrating weaknesses in one or more safety culture components to identify the causes of the weaknesses and the effectiveness or ineffectiveness of the licensee=s corrective actions.

b. As part of evaluating the licensee=s organizational change management effectiveness by studying the implementation and impacts of a specific organizational change.

c. As part of evaluating the licensee=s decision-making processes by identifying the patterns of thinking and behaving that led to a specific decision.

Issue Date: 01/15/09 EE - 1 95003.02

NRC000043

d. As part of evaluating the licensee=s effectiveness in preventing, detecting, and mitigating the perception of retaliation by studying organizational events that did or did not create a chilling effect.

5. Guidance

a. Identify the organizational event or condition to be studied.

(1) Significant events and conditions that will provide useful information about the safety culture components can be identified from the team=s other assessment and inspection activities, including individual and group interviews, the review of issues entered into the corrective action program, as well as the review of allegations, previous inspection reports, and licensee self-assessments.

(2) Operational events are also typically organizationally meaningful and understanding the management, organizational, and human performance causes and contributors to the events, as well as the event=s organizational consequences, often provides useful information about the safety culture components.

b. Use a combination of interviews, document reviews, and observations, if possible, to obtain a complete understanding of why and how the event or condition occurred and its relationship to the safety culture components.

Investigating and analyzing a single event or condition often provides information related to multiple safety culture components.

c. Ensure that the information obtained that is related to the safety culture components is shared within the team.

Issue Date: 01/15/09 Encl E - 2 95003.02

NRC000043 Enclosure 95003.02-F Guidance for Evaluating Safety Culture Surveys This enclosure provides safety culture assessors with guidance for evaluating a safety culture survey that was administered by a licensee. (The NRC=s safety culture assessment will not include the use of surveys.)

Method: Quantitative surveys are structured, written questionnaires, administered to respondents. Questions are close-ended (require a single answer with no explanation) and require respondents to select the best answer from the several options provided.

Answers given can be transformed into numerical information for statistical analysis.

Strengths:

! Can be administered to a very large sample or entire population.

! Can provide precise and quantitative data.

! Usually quick and easy to complete, depending on questionnaire length.

! Data can be rapidly analyzed.

! Respondents remain anonymous while information on general demographic characteristics can be collected.

! When completed by a representative sample can provide precise and reliable information on total population and subpopulations.

! Some reliable and valid surveys already exist in the survey and public opinion research industries.

Limitations:

! Not effective for exploring complicated/ambiguous issues.

! Managers can be strongly influenced by statistics.

! Results can be misleading, especially if the design, application, or interpretation of the questionnaire is less than satisfactory.

! Requires large sample sizes to draw valid conclusions, make valid comparisons, and assure statistical validity across the population and subpopulations selected.

Guidance

1. Review the questions used, as follows, to determine whether:

(Note - the criteria listed for this step can also be used to evaluate questions used by the licensee for interviews or focus groups.)

a. Question wording is simple. Questions avoid technical or specialized words, unless the participants are highly familiar with them.

Issue Date: 01/15/09 EF - 1 95003.02

NRC000043

b. Sentences are short.

c. No ambiguous words or equivocal sentence structures.

d. Times and places and frequencies are specified, even if they are usually assumed.

e. Questions do not include double negatives.

f. Questions address only one topic at a time; questions are not embedded within questions.

g. Questions are unbiased and not leading (i.e., wording does not lead the respondent to answer one way rather than another or place the respondent in a double-bind where no answer accurately reflects his or her situation).

h. Each question is necessary and provides additional, useful information.

i. Related questions are grouped.

j. Questions are sequenced so that one question or line of questioning does not influence responses to subsequent questions.

k. Questions flow from the general to the more specific.

l. Questions flow from the least sensitive to more sensitive topics.

m. Initial questions address screening and rapport-building topics before specific questions.

n. Unique or unusual questions are prefixed with an explanation to avoid confusion. For example, terms used in the questions, such as Ayour supervisor@

or Amanagement@ should be defined, as well as any terms that may be unfamiliar to the participants, such as ASCWE.@

2. Through interviews and document reviews, evaluate whether the survey was developed in accordance with standard practices. Determine whether:

a. The survey questions were pilot-tested with respondents who were representative of the intended participants.

b. Problematic survey questions were revised, on the basis of pilot test results.

c. The revisions were again pilot-tested with representative respondents.

d. The survey developer assessed test-retest or split-half reliability of the survey instrument.

e. The survey has been previously used at the licensee=s facility, or in other organizations, and evaluate any evidence provided by the licensee that indicates whether the previous results were valid and accurately identified strengths and weaknesses that could be verified from other sources of information.

Issue Date: 01/15/09 EF - 2 95003.02

NRC000043

3. Evaluate the procedures used to administer the survey to determine whether they were systematic and were unlikely to have biased the responses. Determine whether:

a. The methods used to select the sample of participants assured representativeness.

b. Questionnaires were administered in a consistent location under a consistent set of conditions. If the survey was administered at different locations or an online survey was used, determine whether the instructions to participants and other means were used to minimize the potential spurious effects of such differences on the data.

c. Participants were monitored while taking the survey by the survey administrators and survey administrators were available to answer questions.

d. Participants had sufficient time to complete the survey.

e. All individuals in the sample had an equal opportunity to participate (e.g.,

accommodations were made to permit backshift personnel to participate).

f. Licensee supervisors or management personnel were present only to introduce the survey team or not at all.

g. The introduction to the survey clearly describes the purpose(s) of the survey, whether responses will be maintained anonymous, who will have access to the raw data, and how the information will be used.

h. Introductory information and instructions encourage the respondents to answer truthfully, indicate that there are no right or wrong answers, and avoid statements that may bias the responses.

i. The same introductory information and instructions were provided to all survey respondents.

j. Anonymity and confidentiality were discussed.

4. Evaluate the statistical methods used to analyze the results. Determine whether:

a. Sufficient responses were received to ensure statistical validity.

b. The statistical techniques applied were appropriate for the types of data collected (i.e., nominal, ordinal).

c. Any differences in responses between functional groups or levels of management were appropriately tested to determine whether the differences were likely due to chance or appear to be statistically reliable.

d. The probability level established for comparisons between responses to individual questions, question sets, and among different subgroups was sufficiently low to reduce the likelihood of Afalse positives,@ in which differences appear to be statistically reliable but are, in fact, due to chance.

Issue Date: 01/15/09 EF - 3 95003.02

NRC000043

e. Any analyses were performed to verify that scales or sets of grouped questions are internally consistent and so appear to be measuring related constructs, and that the results confirm the item groupings.

f. The conclusions drawn from the survey are supported by the results of the analyses.

5. Determine whether the quantitative survey results were supplemented with any of the following to enhance the interpretation of the results:

a. Interviews or focus groups were conducted to gain additional information, as needed, to interpret ambiguous results or gain greater insights related to any issues identified in the survey.

b. The survey provided opportunities for respondents to write-in comments, clarifications, explanations, and additional, more detailed information.

c. Additional information related to any global organizational conditions that could affect the results, such as recent reductions in force, acquisitions or mergers, incentive buy-outs leading to large-scale retirements, or other factors, was used to evaluate differences between subpopulations or responses to the same survey administered at different times.

6. Evaluate participants= responses to the survey:

a. Determine whether any issues related to the survey were entered into the CAP, raised to the Employee Concerns Program/Ombudsman or other alternate means of raising concerns, or to the NRC in allegations.

b. Elicit individuals= perceptions of the survey, the manner in which the survey was administered, the integrity of the results, the manner in which results were communicated, and the manner in which the results were used.

Issue Date: 01/15/09 EF - 4 95003.02

NRC000043 Revision History for IP 95003.02 Commitment Issue Date Description of Change Training Training Comment Resolution Tracking Needed Completion Accession Number Number Date N/A 01/15/09 Completed 4 year historical No N/A ML083430521 CN 09-002 CN search This is the initial issuance of this attachment. The bulk of the inspection guidance in this attachment pre-existed in the body of IP95003 and in Enclosures 95003-A through F. Draft IP95003 section 03.08 describes the NRCs planned approach to perform a graded evaluation of a licensees third party safety culture assessment using the guidance contained in this attachment. This includes performing an evaluation of the third-party assessment methodology and tools, as is done in the current IP95003. This addresses the disposition of ROP feedback form 95003-1261.

Issue Date: 01/15/09 Att1-1 95003.02