ML20134M437

Self Assessment Guide Program Guidelines

ML20134M437
Person / Time
Site:	Comanche Peak
Issue date:	11/13/1996
From:	TEXAS UTILITIES ELECTRIC CO. (TU ELECTRIC)
To:
Shared Package
ML20134M415	List: ML20134M413 ML20134M437
References
PROC-961113, NUDOCS 9611250102
Download: ML20134M437 (33)

Text

{{#Wiki_filter:-- i i

SUCLEAR i

ENGIS~EERING l DEPART ME:Ni' i i i I j m. a m i m l l 1 RIELECTRIC i SELF ASSESS MENT GUIDE PROGRAM GUIDELINES 78A2228a82Zeaask G PDR

-= _. . = _ -. I a Engineering SelfAuenneent Guide TABLE OF CONTENTS 1.0 Purpose 2.0 Definitions p 3.0 Responsibilities 4.0 Instructions 4.1 Station Engineering Management Overview Program. 4.2 Self Assessment Review Committee 4.3 Limited Scope Assessments 4.4 Full Scope Assessments 4 4.5 Selection of the Assessment Team Leader 1 J 4.6 Team Leader Preparation Activities 4.7 Assistant Team Leader Preparation 4.8 Team Member Preparation Activities 4.9 Pre-Assessment Team Meeting 4.10 Conduct of the Assessment 4.11 Post Assessment Activities 4.12 Assessment Report 5.0 References 6.0 Attachments Page 2 Program Guideltnes

_. - _.- _ _._ _. _ ~ y Engheerring SeVAnameent GuMe 1j 1 4 1.0 PURPOSE J This guideline establishes the policy and methodology by which Station Engineering performs self assessment. It should be used as a guideline as the process may be adjusted to meet specific needs. 4 l 2.0 DEFINITIONS 2.1 Self Assessment i The critical evaluation of an activity, process, or program performed by the individual or organization accountable for the work. 4 2.2 Limited Scope Assessment Assessment that is generally limited to a single work process or assessment module. 1 i l 2.3 Full Scope Assessment i An assessment of the general engineering capabilities, equivalent to an area of emphasis inspection by the NRC under IP 37550. I 2.4 Adverse Condition A deficiency, failure, malfunction, deviation, abnormal occurrence, defective material or equipment, or non-conformance in an item or activity which has affected or reasonably could affect: Nuclear Safety or Quality Compliance with other regulations not included in Safety or Quality Personnel Safety Plant Reliability Commercial considerations Page 3 Program Guidelines

_m _._,_m._ . _ _ _ -_. _ -.=. -.. < ~ i Engineerhng SetfAssessneent Guide 't i i 2.5 Finding e A problem, concern, or recommendation which was developed from j observations performed for a self-assessment and requires action to address. Findings may consist of adverse conditions, areas not meeting expectations, or i areas for improvement. 4 2.6 Client The responsible work process owner, typically the manager for whom the self l assessment is being performed. l 1 3.0 RESPONSIBILITIES 4 3.1 The Station Engineering Manager is responsible for the overall implementation of the self assessment process and will chair the Self Assessment Review Committee. l 3.2 The Assessment Team Leader is responsible for the following; I Communicating with the client to determine the specific attributes of the requested assessment (e.g., scope, objectives, schedule). 7 j Assembling an assessment team that meets the client's specific needs, that = possesses the necessary technical capabilities and is of sufficient size to l complete the assessment objectives in the allotted time. Preparing the assessment notebook (performance objectives and criteria, checklists, guides, interview sheets, schedules and agendas, essential site documentation, etc.). Coordinating logistical and administrative activities. Conducting the pre-assessment team meeting. Directing the entrance and exit meetings, the daily team meeting and the daily client briefings during the assessment. Page 4 Program Guidelines

- ~.. 2 i I i i l Engineering SeVAssessment Guide i l l Adjusting priorities and delegating activities to team members during the ) = assessment. . Preparing a:.d issuing the assessment report. Performing the duties of the Assistant Team Leader if one is not assigned. 3.3 The Assistant Assessment Team Leader (if assigned)is responsible for the following; Managing the assessment information generated by the team, such as, = entering data into the assessment data base, preparing subsequent data 4 base reports for use by the team and recording pertinent information from the daily team meetings. Assisting the Team Leader in other areas, as requested, during the preparation for or conduct of the assessment. - 3.4 Assessment Team Members are responsible for the following; Completing personal preparation required to complete the assessment of assigned performance objectives. i Conducting the assessment of assigned performance objectives and l writing Conclusions and Recommendations. Presenting the Conclusions and Recommendations and other pertinent j. information on assigned performance objectives at the Exit Meeting. 4.0 INSTRUCTIONS 4.1 Station Engineering Management Overview Program 4.1.1 The primary purpose of the Station Engineering Management Overview i - Program, (SEMOP), is to provide routine self assessment of the departments diverse processes and programs. Page5 Program Guldehnes d

Engineering SelfAumnerat Guide 4.1.2 Station Engineering Management should be assigned to SEMOP assessments on a schedule to ensure a complete review of all assessment areas on an annual basis. Additional SEMOP's may be assigned by the - I Responsible Discipline Managers at their discretion. i 4.1.3 Some SEMOP's require specific plant conditions to perform (e.g. Outage). If a SEMOP is scheduled during a period that plant conditions do not support the assessment any other SEMOP may be substitute in its place. 4.1.4 Prior to conducting an assessment, reference material should be reviewed for program requirements and management expectations. Previous SEMOP assessments should be reviewed for areas of weakness. 4.1.5 The SEMOP form should be filled out following the assessment with enough detail to indicate the extent of the review. All recommendations should be recorded on the SEMOP form. 4.1.6 The SEMOP should be reviewed and all recommendations dispositioned by the Responsible Discipline Manager. 4.1.7 Completed SEMOP's should be retumed to the Station Engineering Secretary for retention to file. 4.2 Self Assessment Review Committee 4.2.1 The Self Assessment Review Committee should consist of the following; Station Engineering Manager-Chair Station Engineering Line Managers Licensing Engineering Overview Engineering Training 4.2.2 The Self Assessment Committee should meet quarterly to overview the assessment process, monitor activities and indicators for trends, and status corrective actions. Page 6 Program Guidelines

_m m... _..____..__.m ._.m_ 4 z<ina<ern SelfAssessneent GuMe a 3 i 2 Quarterly review scope should include the following; i Station Engineering Performance Indicators ONE Forms Engineering Overview Activities NRC Monthly Exit Summary SEMOP Review . Open corrective actions from previous assessment activities j 4.3 Limited Scope Assessments 4.3.1 Limited scope assessments are generally one week in duration. Assessment scope needs to be carefully controlled by the Team Leader and the client to ensure that it can be completed in one week. 3 5 4.3.2 Limited scope assessments are normally planned and scheduled to allow i formation of the team up to several months prior to the assessment to j provide time for thorough preparation. A minimum lead time of at least six weeks is generally required if team participants from other utilities are desired. 4.3.3 Limited scope assessments may be requested by work pro.ess owners. When the request is accepted, the selected Team Leader will communicate with the client to determine the specific attributes of the requested assessment (e.g., scope, objectives, schedule). 4.3.4 Limited scope assessments may be condition based intitiated by reaching performance threshold values of work process performace indicators or negeative trends as determined by the Self Assessment Review Committee. The committee will determine the specific attributes of the assessment (e.g., scope, objectives, schedule, team composition). 4.4 Full Scope Assenments 4.4.1 Full scope assessments will evaluate engineering activities, particularly the i effectiveness of the engineering organization to perform routine and reactive activities, including the identification and resolution of technical Page 7 ) Program Guidelines . ~ -.. _ -

4 4 EMy sayAutumed GeMe i issues and problesns. 4.4.2 Full scope assessments generally have an assessment period of two weeks. i Assessment scope needs to be carefully controlled by the Team Leader to ensure that assessment activities can be completed in two weeks. [ 4.4.3 Full scope assessments are normally planned and scheduled to allow team i determination up to five months prior to the assessment to provide time for thorough preparation. 4.4.4 Full scope assessments that are to be submitted to the NRC for review for a limited scope inspection as an alternative to the Engineering Team Inspection will normally be submitted for internal review four months prior to the assessment and submitted to the NRC three months prior to the assessment. 4.4.5 Full scope assessments should include a vertical slice of a safety significant system including a review of the following Design Criteria Documents. Calculations. Design Changes. Safety Evaluations. Mangement Controls. System Operability Determinations. Surveillance Testing.- System performance. 4.4.6 System selection criteria should be based on the following; Probabilistic Risk Assessment significance. Maintenance Rule Catagory Systems that have undergone significant modifications. Systems that have required recent engineering attention. Number of assessment hours already spent on the system. Page 8 Program Guid. tines

I Engineering \\ SeVAssessnarnt Guinie l l r 4.4.7 Full Scope Assessments should include a horizontal review of general i engineering capabilities and selected engineering programs to provide a broad functional review. 4.5 Selection of the Assessment Team Leader The selection of the Team Leader will be based on availability, knowledge, and experience in the area to be assessed. 4.6 Team Leader / Assessment Coordinator Preoaration Activities 4.6.1 Ensure the desires of the client are fully understood and documented. A - face-to-face meeting with the client is recommended. At this time, the. Team Leader and the client should establish the assessment Performance Objectives and Criteria. 4.6.2 Establish the assessment team. The team must include personnel who have the expertise required to complete the assessment objectives. 4.6.3 Issue an assessment notification memorandum to the client and to the team j members. The memorandum should include: scheduled assessment dates. assessment agenda team memberlisting summary description of the assessment scope, objectives and criteria. I At this stage in the preparation process some of the above information may still be preliminary and should be so indicated in the memorandum. - The memorandum should also inform the client that if any of the items identified during the assessment are reportable under the quality program that formal documentation of those items is the responsibility of the client. Page 9 Program Guidehnes

i EMg se n m amem a m e i i 4.6.4 Ensure that logistical and administrative activities are properly planned i and the information communicated to team members. These activities include but are not limited to: l' ) assessment team travel and lodging = { local transportation, directions, maps meeting times and places 2 meals site access / badging 2 site specific training (PAT, etc.) i whole body counts entrance / exit scheduling = team work areas a team work hours, etc. 4.6.5 Acquire specific documentation from the client, if required, for assessment preparation. Documentation may include but is not limited to: i Procedures Policies INPO reports NRC inspection reports SALP reports QA audit reports LER's Corrective action reports Design change packages Management reports, etc. 4.6.6 Assign team members responsibility for the assessment of each of the performance objectives. 4.6.7 Prepare an assessment notebook. The Team Leader usually prepares the notebook and provides copies to the team with sufficient lead time to l allow team members ample time for personal preparation. Team members and/or the Assistant Team Leader may also be requested to participate in the preparation of this material. The notebook typically includes: Page 10 Program Guidelines

._..~ 1 4 i } Engineering senuaan,au caue t i a travelinformation/ maps l assessment objectives team list j. team assignments - 1 i' assessment schedule / agenda organization charts (client's) e performance objectives and criteria, checklists, assessment guides H I conclusion and recommendation sheets. data input forms l. copy of this instruction l selected reference documents r 4.7 Assistant Team Leader Prenaration Activities L 4.7.1 Review the assessment notification memorandum and other. documentation provided by the. Team Leader to ensure that the assessment i scope, objectives and the performance objectives and criteria are fully I understood. I 4.7.2 Prepare the assessment data base for use during the assessment.. i j-4.7.3 Assist the Team Leader in other areas, as requested. 1 4.8 Team Member Prenaration Activities i 4.8.1 Review the assessment notification memorandum and other documentation provided by the Team Leader to ensure that the assessment } scope, objectives and the performance objectives and criteria are fully understood. l 4.8.2 Participate in the preparation of assessment checklists, interview sheets and assessment guides as requested by the Team Leader. i 4.8.3 - Complete personal preparation required for assigned performance i objectives. t Page 11 Program Guidehnes 1 3

? Enginee.4eg sepuament t,aue i l 4.9 Pre-Assessment Team Meetino ) The Team Leader is responsible for conducting a pre-assessment team meeting. This meeting is normally held the first day of the assessment prior to the Entrance ' Meeting but, in some cases, may be held the day prior to the start of the j assessment. The purpose of the meeting is to discuss the conduct of the assessment, to provide training / orientation to the team members, if required, and j to resolve any last minute issues, problems or questions. 4.9.1 Preliminary information provided in the assessment notification memorandum or at other times during the preparation process should be updated with current information. 4.9.2 The assessment process and content should be discussed to ensure all team -{ members are comfortable with all aspects of the pending assessment. 4.9.2.a Process topics would include: i expectations and deliverables overview of a typical assessment j = assessment data base data input sheets areas for improvement / strengths / observations i report format a Aily team meetings 4.9.2.b Content topics would include: confirm performance objective assignments discuss overall strategy ensure each team member has a strategy to assess his assigned area discuss planned field observations discuss any special activities (i.e., observations or other activities that a i would require coordination of several team members) discuss interview questions, if applicable e Page 12 Program Guidelines

Enghetering SelfAuenneent GuMe 4.9.3 At the discretion of the Team Leader, the pre-assessment meeting may. j also be used to review observation and interview techniques with the team members. 4.10 Conduct of Assessments 4.10.1 Entrance Meeting The primary purpose of the entrance meeting is to introduce the assessment team to the client's managers, supervisors and/or key employees and, if appropriate, for each team member to identify the counterpart for his assigned performance objectives. The following additional topics should also be discussed: assessment scope. assessment agenda / schedule. up-to-date plant status, if applicable scheduled activities / evolutions that the assessment team may wish to observe last minute administrative, documentation or technical support needs. desired client briefing times schedule time for exit meeting The Team Leader should also reiterate that it is the client's responsibility to formally document any team observations that are potential violations of 10CFR50 Appendix B or that otherwise meet the criteria for reporting under the site's quality program. 4.10.2 Assessment Activities To help ensure a smooth transition into the assessment, each team member should carefully plan what his first activity will be after the entrance meeting. A decisive start builds confidence in both the assessment team members and in the organization being assessed. The actual assessment consists of a combination of document review, ) interviews and field observations. The relative proportion of each is dependent Page H Program Guidelines

.__.,_m, I i E&ering SelfAssmneent Guide 1 upon the nature and objectives of the assessment and the desires of the client. 1 In general, however, it is the goal of the assessment process to perform . performance based assessments. Therefore, as much time and efTort as possible should be spent observing the actual work process being assessed. 4.10.2.a Documentation Review j . Documentation review is an essential activity in the performance of an assessment. Documentation may include but is not limited to: Policies Procedures

• ' Directives Organization charts i

Design change packages Drawings Codes and standards Vendor manuals Logs a Turnover sheets . QA audit reports NRC reports /non compliance LER's Internal problem reports and corrective action documents l Management reports INPO reports, etc. 1 Essential documentation should be included in the assessment package and used by team members in preparing for the assessment. Each team member j should become as familiar as possible with the material that applies to his assigned performance objectives. Document review may produce the 4 following results: I Some of the assessment criteria may be satisfied by this document [ review alone, indicating no further action is required. i i ]. Page 14 j Program Gwdelines e 4

m. _m.._ I Engineering j SelfAssessment Guide d 4 The reviewer can learn how an activity or process should be performed. This knowledge can be efTectively used in personnel interviews and field observations. i Review of NRC, INPO and QA reports, LER's and internal problem - and corrective action reports may indicate areas that need additional attention during the assessment.- Document review can provide information that will be helpful in determining what personnel interviews will be necessary and what activity observations should be made. ( 4.10.2.b Interviews Personnel interviews can be an effective and efficient method to complete some aspects of an assessment. In addition, the client may have requested specific information that would necessitate a series of interviews. In general, interviews can be used to: Gather information. The interviewer can learn about an = organization' s goals and expectations, objectives, policies, procedures, work practices, responsibilities, etc., without having to read large amounts of written material. j Determine if there is a clear understanding of the goals and expectations, objectives, policies, procedures, work practices and responsibilities across and up and down the organization. Gauge attitudes, morale and client supplier relationships. Clarify or validate information obtained from document review or field observation. 4.10.2.c Field Observation ) ' I Field observation is a very powerful method to assess the performance of an organization and is almost the only way to achieve a performance Page H Program Guidelmes

Engintering seyAmmnew caue based assessment. As much field observation as possible should be performed. Observations should be well planned and should be performed as early in the assessment as possible. Observations may generate numerous questions that need follow-up either by documentation review, personnel interviews or additional observations. Sufficient time must be available to effectively complete this follow-up work. 4.10.3 Documentation of Assessment Activities Team members provide the Team Leader a daily input of assessment activity results categorized as strengths, areas for improvement or other observations. The input is submitted using a Data Input Form or similar. The Assistant . Team Leader inputs this material into the Assessment Data Base. Each day, subsequent to the initial data entry, a report is provided to each team member sorted in a manner determined by the Team Leader to best support the needs of the assessment team. This data base report is generally referred to as the " field notes". Analysis of the field notes is one of the methods used to' attempt to develop an overall picture of the assessed organization and determine if there are trends / indications that warrant some redirection or concentration of the assessment. The field notes are a compilation of the factual information discovered in the performance of the assessment and are used as the basis for the preparation of the final report but are not included with the final report. Distribution of the field notes is limited to the copy given to the client at the conclusion of the assessment and the file copy. 4.10.4 Team Conferences The Team Leader should conduct a team meetings regularly during the assessment. These meetings are an opportunity for each individual team member to provide: Page 16 Program Guidelines

1 t 4 t Engineering i SelfAssesenaent GuMe i 1 . An update on his assessment progress Information on his upcoming activities l Outlook for completing assessment objectives Input ofinterest in other team members areas l These meetings should also be used to develop the assessment " big picture". In l order to accomplish this, the data must be organized in a logical fashion and one i with which the team can work. The field notes can be used as that method or other methods such as flip charts or yellow stickies can be used. This is the place to identify trends and redirect assessment activities, if necessary. 4.10.5 Client Briefings The Team Leader should brief the client regularly. The purpose of the meeting j is to inform the client ofissues identified to date and to let the client know the status of the assessment and the outlook for completing the assessment objectives. The last client briefing before the exit meeting should include all items that will be discussed at the exit. Ifissues arise after this meeting, the i client must be informed before the exit. i j 4.10.6 Exit Meeting f a' The exit meeting should be scheduled at the entrance meeting or at one of the client briefings. For the exit, the significant assessment issues (Executive Summary) should be developed and documented. This preliminary Executive Summary will be presented at the exit and a copy will generally be left with the client. Each team member will also present the Conclusions and l Recommendations for each assigned performance objectiv_e. The Team Leader i directs the exit meeting; however, team members will make the actual [ presentations. There should be no surprises at the exit meeting. l 4 4.11 Post Assessment Activities The following activities should be performed at the conclusion of each assessment j-and are the responsibility of the Team Leader. Page 17 l Program Guidelines I- [. i

.=- -.- - ... ~.. -. l e l Eargineering SelfAssessment Guide 1 4 4.11.1 Complete the assessment report (see Section 4.12) 1 4.11.2 Send a copy of the Assessment Notebook and a copy of the field notes to I the Station Engineering secretary for filing. 1 4.11.3 Send a " letter of appreciation" to We supervisor / manager of each assessment team member. 4.12 Assessment Report The assessment report is the vehicle used to formally present the assessment i results to the client. The report also provides a tangible record of the Station Engineer' g self-assessment program. The Team Leader is responsible for the m preparation of the report. The final report should be issued to the client within 1 l approximately two weeks of the end of the assessment. 4.12.1 Report Content and Format t The assessment report contains three sources of results information: Executive Summary j Performance Objective Conclusions and Recommendations j Field Notes (separately provided only to the client) = i There is a relationship between the three sections. The Field Notes contain the detailed factual information discovered in the performance of the assessment. This information is the basis for the Conclusions and Recommendations written for each Performance Objective. t The Executive Summary is used to highlight the important issues identified in the assessment, both Strengths and Areas for Improvement. The Executive Summary is also used to provide analysis ofinformation that appears in other sections of the report. Sufficient information must always be provided in each section of the report such that the client can follow the logic of how the assessment team conclusions were determined. Page 18 Program Guidelines l T 4 d +... -.

..m.m. .._.__._=__ ___-_._..___._ __.__.. _ _.. 1 i Engineering SelfAssessneent Guide 4 4.12.2 Report Approval Process 4 The assessment report preparation, review and approval process is as follows: i l Team Leader prepares draft report. Draft submitted to the client for comments / concurrence. i i Client comments incorporated. Final report signed by the team members and sent to the client. The distribution of the report is limited to the client, individuals in the j client's chain of command. 4.12.3 Client Performance Feedback Survey When the final report is signed out a Performance Feedback Survey is also sent to the client. J I 6 i 4 i l } Page 19 Program Guidehnes 1 'l

l Engineering SelfAssessneent Guide

5.0 REFERENCES

NRC IP 37550 e l NRC IP 40500 NRC IP 40501 i NRC IP 93801 INPO 90-15 ] 6.0 ATTACHMENTS 6.1 Interview Techniques 6.2 Observation Techniques 6.3 Data Collection Sheet 6.4 SEMOP Index NOT PROVIDED WITH THIS ENCLOSURE 6.5 SEMOP's NOTPROVIDED WITH THIS ENCLOSURE l Page 20 Program Guidehnes

seyAnenmem GuMe Attachement i i INTERVIEW TECHNIQUES j l 1.0 GENERAL i . Personnel interviews are an important and effective tool in the self assessment j process. Interviews may result from a request by the client that specific individuals or groups be interviewed for a specified purpose or may just be one of the tools being i used to achieve the assessment objectives. Personnel interviews are primarily used to gather data and generally will yield some combination of the following three outcomes:

a. The interviewee's perspective, involvement and knowledge of the subject matter is determined.

b. Additional docwnents are identified to the interviewer which contain the information being sought.

c. Additional personnel are identified who can provide the information being sought or can provide additional information on the subject.

The quality and quantity ofinformation collected in an interview is directly related to i the team members ability to ask appropriate questions, gain an understanding of the interviewee's perspective, and determine the appropriate action based on the information gained. Throughout this process the interviewer will need to use a variety of communications skills which will be discussed in this attachment. 2.0 PREPARATION 2.1 As with all other phases of an assessment, preparation is the key element. The assessment team member, the interviewer, should research the subject matter of the interview to the extent necessary to be able to conduct an effective and professional interview. 2.2 The following additional items should also be considered when preparing to conduct personnelinterviews. a. Each interview should have a purpose which supports the overall objective of the assessment.

o seanummes caue AnnchemeMi b. The desired outcomes /results of an interview should be determined before the interview. A series of key questions should be developed that supports the purpose of c. the interview and will achieve the desired outcome /results. d. The questions should be written down and used during the interview to keep it on track. Interviews should be scheduled at a time mutually convenient for both e. parties. Scheduling becomes more important as you move up the organizational chain ofcommand. l f. The interview setting is very important. The best location is where the interviewee is most comfortable and where the least amount of distraction is present. 1 g. As part of the interview, the interviewee may request information such as: (1) "Why do you want to talk with me?" (2) "How long will this interview take?" (3) "What will you do with what I tell you?. " l (4) "Will my name be used?" j (5) "What is an assessment?" The interviewer should be prepared to answer these and other similar questions. h. Sufficient time should be allowed between scheduled interviews to reconstruct and complete interview notes. J

savaueu m a c ue AnschemeM1 -3.0 CONDUCTTHEINTERVIEW 3.1 General The flow ofinformation in an interview usually starts with the interviewer and generally follows the following sequence. Open the interview, state the area ofinquiry, and ask an open question. a. b. Obtain response c. Ask probing questions d. Use active listening skills e. Record information f. Repeat process 3.2 The Opening - The interview opening is very important in that it sets the tone for the rest of the interview and determines the effectiveness of the interviewer. The opening should put the interviewee at ease and establish the interviewer's credibility. The interviewer should demonstrate interest in the interviewee and his job, and i establish the pattern of the interviewee speaking with the interviewer actively listening. 3.3 Questioning The type of questions asked during an interview influences the climate of the interview situation and the amount and type ofinformation received. The four I basic types of questions are open, closed, probing, and leading or loaded. The i interviewer should plan a few key questions, usually the open type, after deciding on the objectives or desired outcomes of the interview. a. Open Questions Open questions ask for general information and allow the interviewee to i l

. ~ _.. _ _. _ - _ -. _ _ _ _ _.. _. __._-m-._ SeVAmument GuMe Anachement1 structure the response. Example: "Tell me about yourself." The advantages of open questions are. (1) Allows the interviewee to present not only the information requested but. also attitudes and feelings about the subject. L(2) Open questions provide information that is used to formulate other types of questions, and they do not create a defensive attitude because the interviewee is controlling the responses. (3) Allows the interviewee to volunteer information. The disadvantages of open questions are: (1) Increased time to conduct the interview (2) Interviewer is not in control. 1 (3) Note taking is more difficult. (4) Interviewee may feel the interviewer does not know what he is looking for. As a general rule, use open questions for starting each area ofinquiry. These questions usually begin with "What", "How",~ or "Please explain". b. Closed Questions H Closed questions are designed to limit the response available to the interviewee. Usually a closed question can be answered with a word or phrase. Example: " At what time did the trip occur." The advantages of closed questions are: ) (1) Saves time. The interviewer can ask many questions in a short time frame. (2) Note taking is easier, 6 l

4 SeVAuemarm GuMe Aneckenarnt i i 4 s (3) Interviewer retains control. (4) Appropriate when limited or specific info'rmation is required. The disadvantages of closed questions are: (1) Limits the amount ofinformation given. I (2) If not'used properly, the rapid fire questioning sequence can make the ) interviewee feel like he is being interrogated.- ] The disadvantages to closed questions must be considered carefully. The cooperative climate of the interview can be destroyed quickly if the interviewer overuses this type of question. As a general rule, use closed questions only when seeking specific items ofinformation. Do not ask more than three closed questions in succession. c. Probing Questions Probing questions are used to clarity information or gain additional information, usually based on a response to an open question. Probing questions are always based on the information given by the interviewee. They are useful because they focus the response on the information you need to know. They can be used to clarity apparent inconsistencies or discrepancies. Example: "Tell me more about... "What do you mean by... d. Leading / Loaded Questions ' Leading and loaded questions have hidden agendas and usually ask the respondent to agree with a position already held by the questioner. Asking loaded or leading questions is seldom, if ever, useful and should be avoided in - an interview situation. 3.4 Active Listening

't i i i seapueumen caue Auschement i a i i l Aner asking a question, the interviewer evaluates the information and attitude of the interviewee and reacts in a manner to foster a constructive climate. An i interviewer's reaction, verbal and nonverbal, should indicate to the interviewee that active listening is taking place. Failure to do so may reduce the amount of information given and negatively influence the climate of the interview situation. Active listening is providing the interviewee with feedback. Active listening skills are paraphrasing, reassuring sounds, summarizing, and nonverbal responses. 3-a. Paraphrasing l Paraphrasing is using your words to convey your understanding of what the interviewee stated. To use this technique, the interviewer must listen and comprehend the message. It is a technique that fosters a positive climate because it shows the interviewee that you are paying attention and feel the information being presented is important. Paraphrasing also requires the interviewer to wait until the interviewee has finished presenting the information. Not interrupting 4 the interviewee also fosters a positive climate. Paraphrasing should be used when you feel the interviewee has made a statement that clearly needs to be j understood by both parties. Notice that paraphrasing can also lead to another line of questioning. If the interviewee answers "right" or "that is correct" the paraphrase serves as a closed question. If the interviewer's perception was not correct, the paraphrase functions as an open or probing question. b. Reassuring Sounds Periodically indicate to the interviewee that you are actively listening by making reassuring sounds. These sounds can be words or phrases such as "OK", "I see", "That's interesting". Very onen the reassuring sounds are combined with body language such as nodding the head. c. Summarizing This technique should be used by the interviewer at the end of the conversation for each area ofInquiry and at the end of the interview. The difference between summarizing and paraphrasing is that summarizing covers all the key points covered related to an area ofinquiry or to the entire interview. This technique is especially useful when a great deal ofinformation is covered during an interview . It serves as a comprehension check for both parties. m v.

l I l SelfAuenneent GuMe Anachenmenti d. Nonverbals Nonverbal messages are given to the interviewee and received from the . interviewee. Fifty to eighty percent of the meaning conveyed during a face to face conversation comes from the nonverbal messages. When you send - nonverbal signals, you are confirming for the interviewee that you are or are not listening and comprehending the information. These confirming signals may be head nods, smiles, raised eyebrows, leaning forward in the chair, maintaining good eye contact, etc.. Remember, if the interviewee has to choose between what you say verbally and what you say nonverbally, he is more likely to accept the nonverbal signals over the verbal message. 3.5 Summarize and Close As previously stated, at the conclusion of the interview the interviewer should summarize all the key points covered in the interview. The ending summary serves as the last check of communication between the interviewer and interviewee. The interviewer should make a closing statement thanking the interviewee and, if not previously discussed, let the interviewee know how the acquired information will be used. 4.0 POSTINTERVIEW ACTIVITIES The most important post interview activity is to ensure that all the pertinent information provided by the interviewee is contained in your interview notes. This should be done as soon as possible after the interview, while the interview is still fresh in your mind. It is also recommended that an interview summary be written, organized around the desired outcomes or objectives of the interview. Other post interview activities might include: performing any required follow-up; briefing the team leader; and, briefing other team members. i

j = e 1 SetfAssessneent GuMe j Anechenaent 2 OBSERVATION TECHNIQUES 1.0 GENERAL Field observation is the process of observing plant personnel engaged in their daily activities. When observing an activity, the assessment team member will be looking 1 at an individual or a group ofindividuals that is likely representative of the organization's ability to train its personnel and implement its policies and = procedures. When chosen carefully, the results of field observations will provide a clear indicator of the overall performance of the organization. 2.0 SELECTION ] 2.1 In selecting activities and planning for field observations, there are several questions that can be considered to help in deciding the most effective course of action. Some basic questions with discussion are as follows:

a. Is the activity within the scope of the assessment and does it support the objectives of the assessment?

b. Is the activity significant with respect to nuclear safety?. If so, the results of an observation of this activity may carry considerable impact. If areas for improvement in this type of activity are noted, they may be significant in themselves.

c. Is the activity of sufficient complexity that a written procedure has been developed? For many of the activities that are observed, a procedure has been developed by the organization to ensure that specific steps are accomplished in a required manner such that the end product meets a minimum quality standard.

d. Does the activity involve several departments or disciplines? Although single discipline observations can be productive, those that require the cooperative effort of several elements of the plant organization often provide a more significant input to the assessment. In addition, the customer will frequently request that specific activities or functions be observed during the assessment.

serpueamen caue Attachenernt 2 l i 2.2 The following examples illustrate the type of activities that can be used for field observations: 1 a. Corrective maintenance b. Preventive maintenance c. Chemistry activities - sampling / analyses i 4 4 d. Surveillance / testing i e. Turbine building / nuclear plant operator rounds f. Industrial safety inspector on tour } g. New fuel receipt and inspection l h. Shift turnover i. Control room evolutions j. System / component clearance activities k. HP control point activities 4 1. Swipe surveys m. Hot machine shop work i n. Training - GET/requalification o. Safety review committee meetings p. Maintenance planning meetings q. Outage planning meetings i 4

savAmumed GuMe Anuhemen2 a i 3.0 PREARATION i As with all other phases of an assessment, preparation is the key element. The two most important parts of the preparation phase are the determination of what and when. The what to observe can be determined by communicating with the team leader or the customer to ascertain what activities will be going on during the period of the assessment. This will enable the team member to plan for specific activities and to conduct the necessary research and study. When to observe is, as early in the assessment as possible. By conducting results oriented observations as early as possible, the assessment team gains considerable insight into the strengths and weaknesses within the organization. This then enables the team to properly direct their activities during the remainder of the assessment period. The team member should read and become as familiar as possible with the appropriate procedures, directives, codes, regulations and similar documents prior to observing the activity. i t In addition to evaluating the performance of the activity..against the written documentation (procedures, policies and directives), the adequacy of the documentation should also be evaluated against the agreed upon assessment i acceptance criteria. i t 4.0 OBSERVATION Most observations should be planned in advance and arrangements made as to when and where the assessment team member (s) should meet with the individual (s) who j will be the subject of the observation. If possible, an observation should begin at the beginning of the shift or, for particular work assignments, at the beginning of the work. For example, if the subject of the observation is an item of preventive maintenance, the observation should commence when the worker initiates action to obtain the necessary paper work and tools to perform his task. Establishing good rapport and trust with the individuals under observation is important. They should understand that the purpose of the observation is not to criticize them personally, but to look at station training procedures, policies, and practices and their implementation. It should be made clear that they should not allow the observation to interfere wit h the way they normally conduct the activity. Questions are a necessary part of an observation, but should be asked at times when ~ they do not adversely affect the performance of the individual being questioned. Take good factual notes but do it in a manner that does not cause tension among the personnel you are observing. It may be useful to explain why you are taking notes. i

e. SeVAssmnaent Guide Anacheneent2 The following techniques may be of assistance when conducting field observations, Try to observe everything that is going on. Be creatively inquisitive and a. persistent. b. Take detailed factual notes. Sometimes apparently irrelevant material becomes quite meaningful when analyzing and summarizing an evolution for write-up. c. Include procedure numbers and other reference information in the notes for follow-up. - d. Include questions and items to follow-up in the notes. Information could be lost if memory is trusted for recall later. e. Include preparatory activities in the observation if possible. f. If communications are being conducted on a headset, ask for a second headset to improve your knowledge of what is happening. g. Don't assume - ask questions. Even if engineer A told you the answer, ask engineer B. (However, don't entrap people.) h. Constantly ask yourself: "Why is he doing that? Is it the correct thing to do?" Note the details. i. Don't just observe the activity, observe the individuals and the surroundings. Look under, over, and around. Think beyond the evolution. (1) Why doesn't the snubber have oil in it? (2) Why is the wrench he is using painted red? (3) Where did that instrument come from? (4) Why does he keep changing settings? (5) How many management personnel have I seen in the last two hours? i ]

_.. _ ~ _ _ ~ - _ _ -. _ _ _.... _....- m o ev # 2 i SeyAnasneent GeMe Attackeneent2 I i j. For longer evolutions, have another team member watch some of the evolution. Compare notes. i k. Think ahead to determine if there are other items you might want to look for other than those connected directly with the observation. 1. For complex evolutions to which sevent evaluators are assigned, coordination will be required between the observers. Joint review of the procedures and discussion of what to look for is helpful. 5.0 FOLLOW-UP/ DOCUMENTATION Subsequent to observing the work activity, the team member organizes his notes and commences analysis of his observations. This process generally results in the need for follow-up action in order to resolve unanswered questions and to determine root causes, if possible, ofidentified areas for improvement. This follow up may require additional document review, personnel interviews or additional observations. Observations should be documented such that the identified issues are clearly and concisely presented to the customer. The documentation should be written so that.it aids in focusing on the symptomatic results rather than on the individual. The ] significance of written items should be obvious, or they should be put in perspective by stating why they are significant (e.g., not in accordance with approved procedures or not in accordance with agreed upon assessment acceptance criteria, etc.). Conclusions should be fully supported by the facts. j i

_ -. = - i Station Engineering Self Assessment Program Data Collection Data Collected By: ENTRY CODES Objective (OBJ) Activity (ACT) Observed O/ Reviewed R/ Interviewed I Strength - S Area for Improvement - 1 Other - O 4 ' OBJ# -:

'ACT

S/1/O Details -

OIR/l J h 1.3 I}}