ML20236H851

Proposes Options for Assessing Performance & Competency of Licensee Mgt & Delineate Associated Policy Issues for Commission Consideration & Feedback

ML20236H851
Person / Time
Issue date:	03/26/1998
From:	Callan L NRC OFFICE OF THE EXECUTIVE DIRECTOR FOR OPERATIONS (EDO)
To:
References
SECY-98-059, SECY-98-059-01, SECY-98-059-R, SECY-98-59, SECY-98-59-1, SECY-98-59-R, NUDOCS 9807070386
Download: ML20236H851 (10)

Text

-_____

i

' #p "%% *

. RELEASED 10 THE PDR .

l' w< 1,9  : 7/y/pr pia / :

i  ;

• k Sc/p eato ininais

%..a... ....

POLICY ISSUE (Notation Vote)

March 26.1998 SECY-98-059' EQB: The Commissioners FROM: L. Joseph Callan Executive Director for Operations

SUBJECT:

PROPOSED OPTIONS FOR ASSESSING THE PERFORMANCE AND COMPETENCY OF LICENSEE MANAGEMENT PURPOSE:

The purpose of this paper is to propose options for assessing the performance and competency of licensea management and delineate associated policy issues for Commission consideration and feedback. The staff seeks guidance on the Commission's preferred options before expending significant additional resources necessary to fully develop all available options.

BACKGROUND:

In its May 30,1997, report entitled, "Nuctoar Regulation: Preventing Problem Plants Requi,es More Effective NRC Action," the General Accounting Office (GAO) stated that NRC does not have an effective process for ensuring that licensees maintain competent management in their j nuclear plants. Although NRC's regv!ations do not explicitly require the evaluation of plant gj/

management before a license to operate o nuclear plant can be issued, NRC must determine if the prospective licensee is technically and financially qualified to engage in the activities

//

authorized by the operating license. The GAO recommended that the assessment of p

management's competency and performance be a mandatory component of NRC's inspection '>f/7, process. The NRC staff responded to GAO's recommendation by stating that : has been, and '7 remains to be, the NRC staff's practice to conduct performance-baseu W a,vos in all areas of facility operation and design and on the basis of the inspection resula, f draw conclusions about the effectiveness of the licensee's management.

CONTACT: Peter J. Prescott, AEOD NOTE: TO BE MADE PUBLICLY AVAILABLE (301) 415-7591 AT COMMISSION MEETING ON 4/2/98 l

1 9807070386 980326 t '

l i

$_ROS pg '

, qu cw?.

_- _ _ _ - ~ _ _ _ _ _ _ _ _ - - _ - _ - _ _ - - - _ - _ _ - _ _ - _ - - - _ _ _ _ _ _ _ _

-4' L

- In its December 30,1990, report regarding improvements to the Senior Management Meeting (SMM) process, the Arthur Andersen Consulting Firm recommended that the NRC staff assess leading indicators, such as management and operational effectiveness, on an ongoing basis.

' As a result, the Commission issued a Staff Requirements Memorandum (SRM) dated March 14,1997, asking the staff to specifically consider the extent to which objective performance infonnation could be used in the SMM decision process, with new performance indicators being phased in as appropriate. ~ ln SECY-97-072, " Staff Action Plan to improve the Senior Management Meeting Process," dated April 2,1997, the staff responded to the Commission

. that a development process had been initiated to assess leading indicators, such as management and operational effectiveness, on an ongoing basis.

1

The NRC currently conducts limited scope evaluations of management performance in response to specific operational events or adverse human or program performance trends.

~

When evaluations of management have been conducted in response to specific events, they j have typically been conducted as elements of Special inspection Teams, Augmented Inspection Teams, and Incident investigation Teams; if there is evidence of declining i performance, Diagnostic Evaluations are conducted. The NRC also evaluates compliance of l licensee programs against licensee Quality Assurance programs, which are developed to satisfy 10 CFR Part 50, Appendix B, Quality Assurance Criteria for Nuclear Power Plants and

- Fuel Reprocessing Plants.

As part of the NRC's effort to improve the SMM process, the staff is investigating'the 4 development of management performance assessment tools for improving the current plant performance evaluation methodology. The Office of Nuclear Regulatory Research (RES) and AEOD have initiated development of a~ process to assess management performance on an _j ongoing basis. A key milestone in this effort included a week-long workshop in August 1997 i involving NRC staff and experts from the research community. The general consensus of the n workshop was that Management and Organizational (M&O) factors do influence human and hardware performance. The experts in attendance not only identified important M&O factors but also identified those with the highest potential impact on risk.

In December 1997, AEOD conducted an Organizational Effectiveness Workshop with, representatives from RES and NRR to develop characteristics, measures, and indicators to

assess management performance based on insights from existing NRC inspection programs,

. ongoing NRC research, and industry evaluation techniques. As a result of this workshop and a detailed study.of inspection information contained in the NRC Plant issues Matrix (PIM) data

• base for eight nuclear facilities, the staff deterrnined that it was possible to infer conclusions about management performance from current inspection activities. However, it was concluded

. that a management assessment based on current inspection program data would not likely

- result in a leading indicator of plant performance and would not result in a comprehensive and

' direct assessment of licensee management performance.

i DISCUSSION.

The current NRC inspection program assesses compliance with existing regulations and develops performance insights by observing the conduct of operations, material condition cf the

L

plant, performance of licensee personnel, quality of engineering work, and the licensee's

. performance in problem identification and resolution. The NRC inspection program also

( ' involves evaluation of operational events to identify root causes such as human error, design l deficiencies, and weak administrative controls. The NRC then assesses overall plant-L performance and infers licensee management performance based on a comprehensive review l: . of inspection findings, license amendments, event reports, enforcement history, and l performance indicators.

, . In response to the GAO and Arthur Andersen recommendations mentioned above, the staff proposes five options for assessing performance and competency of licensee management.'

These options are summarized as follows:

L -1. Continue to conduct performance-based inspections in all areas of facility operation and L

design; however, do not attempt to infer or articulate conclusions regarding the perfonnance or competency of licensee management.

2. Infer licensee management performance from the results of the current performance-

! based inspection program of overall plant performance. Enhance guidance to improve the quality and consistency of the management performance assessment.

3. Assess the performance of licensee management through targeted operational

. performance inspections using specific inspection procedures, trained staff, and y contractors.

L l

4.- Assess the performance of licensee management by evaluating and documenting management performance attributes as part of the routine inspection program. J Implement the necessary regulations. . Revise the inspection, staff training, and j qualification programs accordingly. j

5. Assess the competency of licensee management by evaluating management competency attributes using specific inspection procedures, trained staff, and

. contractors implement the necessary regulations. Revise the inspection, staff training, and qualification programs accordingly.

' To help determine whether the NRC should modify its current practice with respect to assessing licensee management performance, the potential benefits derived from insights into M&O L factors were considered in concert with those findings typically available from existing I

l

^'

• Management Performance Assessment. assessing characteristics (e.g. culture, organizationallearning, communications) of an organization to determine if the implementation of the organization's procedures and

- practices ensures equipment and human reliability, regulatory compliance, and safe plant performance.

Manaaement Comoetency Assessment assessing the characteristics, attributes, knowledge and skills of managers to determine if they have the necessary ability to develop and implement procedures and

. practices to ensure equipment and human reliability, regulatory compliance, and safe plant performance.

l I

3 inspection programs. In all options, the staff would continue to assess the role of management in event follow-up. In options two through five, the staff would develop improved guidance for defining and assessing management performance or competency. Currently, an ongoing RES program to develop the technical basis needed to define the relationship between M&O factors and equipment and human performance would support implementation of options two, throe, or four. The options, along with the implementation and policy issues associated with each option, are discussed below.

Option 1: Continue to conduct performance-based inspections in all areas of facility operation and design; however, do not attempt to infer or articulate conclusions regarding the performance or competency oflicensee management.

Option The NRC would continue its practice of conducting performance-based inspections in all areas of facility operation and design as required by the current inspection program. The NRC would continue inspecting for compliance with existing regulations by observing the conduct of operations, the material condition of the plant, the performance of licensee personnel, the quality of engineering work, and the licensee's performance in problem identification and resolution. In addition, the NRC would continue to examine operational events to identify root causes such as human error, design deficiencies, and administrative

. controls. The process for assessing plant performance would continue to be based on inspection findings, enforcement actions, operational events, and performance indicators.

However, contrary to current practice, the NRC would no longer infer or articulate conclusions regarding management. This option would not require additionalinspection effort beyond that required for implementing the current inspection program.

Implementation: Minimal guisance development and inspection training would be required to implement this option.

Poliev Isag: This option would entail a Commission c'> cision directing the staff ta discontinue its practi:e ofinferring management performance from the results of performance-based inspections.

Option 2: Infer licensee management performance from the results of the current performance-based inspection program of overall plant performance.

Enhance guidance to improve the quality and consistency of the management performance assessment.

Option The NRC would continue its practice of conducting performance-based inspections by observing the conduct of oparations, the material condition of the plant, the performance of licensee persor.nel, the quality of engineering work, and the licer.see's performance in problem identification and resolution. The NRC would cor'tinue to examine operational events to identify root causes such as human error, design deficiencies, and administrative controls. The ,

process for assessing plant performance would continue to oe based on inspection findings, enforcement actions, operational events, and performance indicators using newly developed tools such as the Plant Performance Template and the Trending Methodology. i

y

( Management performance would be inferred from examination of inspection results and -

operational events. In the past, this has t,sen accomplished through informal qualitative

. Judgment. Guidance currently being developed by RES would allow for a more systematic method of inferring management performance, it is estimated that this option would require the j same inspection effort and resources as those required for implementing the current inspection *

. program.'

Implementation' The NRC staff is currently exploring improvements to the process for assessing plant performance. This effort is known as the integrated review of assessment process (IRAP). The new process would involve annual meetings at the regional level to assess overall plant performance. Following the assessment, the licensee would be given the opportunity to respond to NRC's assescment findings. Subsequently, the assessment findings would be made public.

Cu rently the management performance implications of individual inspection findings are not j

' documented in the PlM. In the future, in order to be compatible with the scrutability requirements of the IRAP, the staff will be required to identify a mechanism for periodically disclosing the managment assessment for public. scrutiny.

This option is currently in place. Improvements to the process could be accomplished in FY -

-1999. ,

Poliev lssue: Although this option is not a change in current policy, the Commission may choose to explicitly direct the staff to infer management performance from the data currently documented in the inspection program.

. Option 3: Assess the performance of licensee management through targeted operational performance inspections using specific inspection procedures,

trained staff, and contractors.

Ootion NRC inspectors would continue to evaluate, respond to, and document failures to comply with existing regulations, much as they do under option 2 today. However, based on guidance currently under development by RES, inspection procedures would be developed to perform targeted operational performance inspections for specific indications of management performance Inspectors would receive training on the types of issues identified in the inspection procedures and their relationship to management performance.

'The assessment of management performance would be based primarily on the results of those

targeted inspection efforts. Insights from operational events would also be factored into the -

assessment, it is estimated that this option would require reallocation of inspection resources to management performance issues.

lmolementation';ln this option, management performance findings would be documented in the

. PIM, and would become public at the time the PIM is updated. .The overall assessment of management performance would be based on the nurnber and severity of such findings. This

,. L option would require significant training for NRC staff.

L l

?: ,

s- _ _ _ __ _ __ _ __ _ _ _

i l

This option could be implemented by FY 2000.

l I

Poliev lasue:~ This option would expand what is currently being done in the assessment process by collecting information related specifically to management performance. This option would require a change in current policy and would require the Commission to direct the staff to 2

' document information related to the assessment of management performance as part of NRC's

. formalinspection program.

Option 4:- Assess the performance of licensee management by evaluating and documenting management performance attributes as part of the routine inspection program, implement the necessary regulations. Revise the Inspection, staff training, and qualification programs accordingly.

Ootion The NRC would directly assess the performance of licensee management by defining

the attributes of ma,lagement that are related to safety performance, developing assessment ,

- tools and applying them for assessing management performance at the licensees' facilities. l The possible types of attributes that would be evaluated include: safety culture, communications, degree to which the organization is a learning organization. The NRC would develop acceptable levels of the various attributes of management performance based on available research. The necessary regulations and inspection procedures would also be '

developed. Inspectors would receive training on the types of issues identified in the revised i regulations and inspection modules and their relationship to management performance.

In'this option, the assessment would be based on the results of specific management f performance inspections. The results would be documented in the PIM. Insights from

< operational events would be factored into the assessment. The NRC would take enforcement

' action based on failure to meet management performance standards. This option would require reallocation of inspection resources to management performance issues.

1molementation: If this option is chosen, significant additional study will be required to assess 3

' implementation options and their associated impacts. The possible implementation options '

would be (1) the NRC would periodically perform management performance inspections, (2) the NRC would perform management performance inspections on a for-cause basis, (3) the NRC t, would require the licensees to perform periodic assessments, or (4) the NRC would require l

= licensees to conduct assessments on a for-cause basis. Regulations and Agency guidelines would provide consistency in evaluating management performance, provide a regulatory basis

.for developing questionnaires and inspection protocols, and ensure that the management lg issues assessed have been identified as important to risk.~ This option would require significant 3 training for NRC staff.

~ This option could be implemented by' FY 2001.-

j Pohcv Issue: This option would require the Commission to issue new regulations and would  !

constitute a significant departure from current regulatory approaches under the NRC's statutory authority.-- ,

A; Y

Option 5: Assess the competency of licensee management by evaluating management competency attributes using specific assessment tools, guidelines, train _ed staff, and contractors. Implement the necessary -

regulations. ' Revise the inspection, staff training, and qualification programs accordingly.

Option The NRC would directly assess the competency of licensee management. The NRC would define the knowledge, skills, and attributes associated with competent safety management, and develop appropriate evaluation methods. The NRC would define levels of acceptable management competency based on available research on the topic. The NRC would implement the necessary regulations to enforce compliance with those standards . ,

Inspection procedures to assess competency would be developed and implemented into the I NRC inspection program. )

in considering this option several questions, such as those noted below, need to be explored:

- Which management positions would be included in the assessment ? j

.* Would individual managers be assessed or would the assessment focus on  !

management teams ?

r.

Would there be formal qualifications or licensing requirements for licensee managers ?

The NRC would take enforcement action based on failure to meet the management competency standards. - The assessment of licensee performance would include a direct assessment of management competency. This option would require reallocation of inspection l resources to management competency issues.

]

ilmolementation: If this option is chosen, significant additional st' My will be required to assess

- implementation options and their associated impacts. The possioie implementation options would be (1) the NRC would periodically perform management competency inspections, (2) the T NRC would perform management competency inspections on a for-cause basis, (3) the NRC would require the licensees to perform periodic competency assessments, (4) the NRC would require licensees to conduct competency assessments on a for-cause basis, (f) the NRC could require competency assessments by a third party, or (6) the NRC could license individual managers.

. -This option would likely require the development of a special corps of inspectors and the use of specialized contractors to implement the methods.

This option could be implemented in FY 2001.

Poliev lasuef This option would require the Commission to issue new regulations and would

!~

constitute a significant departure from current regulatory approaches under the NRC's statutory

authority.

l L. _ __ _ _ _ _ _ _ . _ _ _ _ . . _ _ . _ _ _ _ _ _ _ _ . _ _ _ _ _ . . _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ ______ __

l i a

!1 i

I RESCURCES AND IMPACTS-l The resources and impacts associated with each of the options are listed in Table 1 (Attached).

i These include the need for policy or rule change; the costs associated with development of the l rule, necessary guidance, modifications to the inspection program and additional inspector training; impact on staff qualifications; reallocation ofinspection resources (reduction in plant performance inspection to compensate for new inspections of management performance or

, competency); probability of successfulimplementation; use as a leading indicator; and l

scrutability of findings. Each of these factors was quantitatively (where FTE or contract costs could be estimated) or qualitatively assessed on a relative basis for each option.

Inspection of Table i shows that options that depend on rulemaking have the greatest impact l on staff qualific;tions and require the most significant reallocation ofinspection resources. On .

the other hand, those options will have better scrutability of findings because of documentation I requirements and clarity of NRC expectations. The probability of successfulimplementation for these options is not as high as the other options because of the uncertainties associated with the rulemaking process and development of a defensible technical basis and regulatory analysis.

Based on Commission guidance, the earlier response to GAO will be revised as necessary.

REQUESTED COMMISSION ACTION l

The staff requests that the Commission select one of the options listed in this paper and provide guidance for proceeding further. Subsequent to the Commission's action, the staff will develop another Commission paper to address in more detail a plan of action for implementation of the l option identified by the Commission.

l COORDINATION'

, The Office of the General Counsel (OGC) has reviewed this paper and has no legal objections, l although OGC notes that there are significant legal issues that would need to be addressed if l option four or five were pursued. The Office of the Chief Financial Officer has reviewed this l paper for resource implications and has no objections.

l

s h.

RESOURCES-Currently, the FY 98 and 99 budgets support development of guidance for options one, two, or l three Options four and five are not budgeted.

L. Jo eph Callan 1 Exe tive Director i for Operations

Attachment:

Table 1," Resources and Impacts Associated with Assessing Performance and j Competency of Licensee Management" l I

Commissioners' completed vote sheets / comments should be provided directly to the Office of the Secretary by COB Friday, April 10, 1998.

Commission Staff Office comments, if any, should be submitted to the Commissioners NLT April 3, 1998, with an information copy to the Office of the Secretary. If the paper is.of such a nature that it requires additional review and comment, the Commissioners and the Secretariat should be apprised of when comments may be-expected.

DISTRIBUTION:

Commissioners OGC-OCAA

'OIG OPA OCA CIO CFO EDO REGIONS SECY l

l

)

5 K K 4 n E0 E0 E (

w T n le T0 T0 h T m h w o h

- N oit u F5 F 0, ig F iu ig o n ig E p R 31 31 H 0 d H L k n H M

E O $ $ 1 M

e U G

A N

A

- M E

E

- S N

- E )

C 4 4 K

K E (

m I

L no le E 0 E0 h T m h iu h h F u T 0 T0 ig F u ig d ig ig it F 0 F 0, O p R 2 1 21 H o i

d H e H H Y O $ $ t e M C M N

E T

E P

M O

C e D c N t n n

e a

)

A 3 1 m E m m m E n y ( K C oit u c

/A E 00 iu T F

w iu h ig iu iu mrm so o N d o d d d N p P T 7 e 0 L e H e e sf r A F $ M M M M ee M O 1

1 sp s

R ag O t n 1

F n el in leR i bE mc e e aP i g d T

G ae N nt a I

a S ic md S

E onit S 2 ) t r n oop S 1 n e ( K e e e e h t A oit n A E 00 n n n n ig w w io ep H o /

N T 5 o o o o o o t cnu T p N F $ N N N N H L L eos I

O 1 pla r

- W s o D intnct E

T ecmt ea r

- A I

ns as n oc C

O me r sf S

S o so f a e r

A 1 n y la e l

a e e ees p c~u S ic A n n m n n h A A 9 T io is ig nt lo

/

N r o i n o o /

N

/

N t

9- nan C t p P i

N i N N H

- A P O M M Ylama F pr ic

. ofi M

I dnoeg mfr n

- D ar f pis N n t A o lu 8 s n e

_ S s E c t

h:

it m

a r

t r

o c

e f

f it c

e f

s s

e f 3 o(

) s g

9- e air Yr clp uq F uf oore w g a p in

_ C a p r o p t s c ne R o d e t r s s S n c o c d in e s yl

- Um y e g n Pn i n I u it n in l O iI c t

a n e. nio C f o)2 Sn aa F d ri t i w

e la Rs f io icm f l S

i a ot n ef i lo ic c s, iot a ng Nn n( ot J ob s

- E es Pn o o s

h C n tci c on o os a n

d r Inf o tye n. n an da :.

nit it e ityt i m R c fytat io eio ii ri s ef i it n oa a c e r l t

r A le pd l g e i b e u

o fota s u iue so di a d r t ic cf cr ou iame b

inP ag a

ht p

s d lu eg t

s R GD InM AT al pa i

R o as 2l p da n t tun lt n cd o e ee o u ee n el a ca r

one D' .

mQ RR el os R

PI m NR C A B C SM I LP r r laye

. . . . . ueeh 1 2 3 4. 5 6 7. CRBT

))))

1

((((

234

> $zEmZ

.