ML20134L303

Trip Rept of 961105 Visit to Columbus,Oh to Attend & Conduct Workshop at Tenth Annual Instructor Workshop Sponsored by Midwest Nuclear Training Assoc

ML20134L303
Person / Time
Issue date:	11/12/1996
From:	Usova G NRC (Affiliation Not Assigned)
To:	Richards S NRC (Affiliation Not Assigned)
References
NUDOCS 9611200240
Download: ML20134L303 (37)

Text

a m>a 44 n a' 4 m =4 -+ea-su - Asa- i n .&-

-- A L m- W a 1 u- -\As W. Aan -,

.} g -p ucu j p -4 UNITED STATES

o 'j j NUCLEAR REGULATORY COMMISSION

• o

• WASHINGTON, D.C. 20665-0001

' o %.,*****[

November 12, 1996 MEMORANDUM T0: . Stuart A. Richards, Chief

- Operator Licensing Branch p FROM: George M. Usova Operator Licensing Bran 1b'

)'

SUBJECT:

TRIP REPORT: PRESENTATION AT THE loth ANNUAL INSTRUCTOR'S WORKSHOP, MIDWEST NUCLEAR TRAINING ASSOCIATION I attended and conducted a workshop at the Tenth Annual Instructor Training Workshop held November 5 at Columbus, Ohio. The workshop was sponsored by the Midwest Nuclear Training Association (MNTA).

In May 1996, I was contacted and invited to make a presentation by the MNTA workshop coordinator, Mr. Brian Hajek. I agreed to make two 90 minute presentations to enable more workshop participants to attend the presentation.

Approximately 35 participants attended both workshop presentations.

As I explained to all participants at the outset, the NRC was making this presentation to promote a common understanding between the NRC and industry of measurement concepts and terminology used in examination development as well as to communicate expectations of levels of knowledge and difficulty in examinations. Since the burden of examination development responsibility was shifting to industry but yet the exam'was considered to be an NRC administered examination, it was prudent for the NRC to provide to industry the type of guidance contained in the workshop.

The title of the workshop was increasing Test Iten Validity in Examination

-Development (See Attachment 1). This workshop session addressed test item and examination development guidelines established by the operator licensing branch in developing and reviewing operator license examinations. The session included a background discussion on test validity, defined and assessed higher cognitive level questions, described elements of psychometrics, and provided participant practice in critiquing test items. (See Attachment 2 for workshop slides).

The sessions proceeded very well; post workshop comments made to me were highly positive, several of which stated that there should have been more time allowed for the workshop. Each participant also completed an evaluation form .

4)

Df/h-/j' / '

NMMf5 7//

O^ 9611200240 961112

/

~

PDR ORQ NRRA 9% _ ff D g HLE CENTR COPY OW

4 3

i*,

d .

1,

, which was submitted to the workshop staff who would analyze all feedback

forms. I will receive the actual evaluation forms at a later date and will forward them to you for review.

In sum, the workshop sessions I presented were well-received and successful.

3 Attachments: As Stated I

DISTRIBUTION:

icentral: Files : a PUBLIC HOLB RF j Ta enceive a sepy of tido deevnesat. Inesses in the ben: "C" = Copy without ettechment/ enclosure "E" = Copy with attachment / enclosure 'N" = No copy l OFFICE HOLB/DRCH , lC l l l NAME GUsova:gm/rchh '

! DATE 11/12/96 0FFICIAL RECORD COPY e

i i

i 1

2' l

i i

J-i j

i usw IIRC FEE CENTER COPY

9 s l e

which was submitted to the workshop staff who would analyze all feedback .

forms. I will receive the actual evaluation forms at a later date and will l forward them to you for review.

In sum, the workshop sessions I presented were well-received and successful.

l Attachments: As Stated ,

DISTRIBUTION: l Central Files PUBLIC HOLB RF Ta resolve a copy of this deeumont, inescate in the ben: "C" = Copy wkhout attachment / enclosure *E" = Copy with attachment / enclosure *N" = No copy 0FFICE HOLB/DRCH i lC l l NAME GUsova:gm/rc b u DATE 11/12/96  !

0FFICIAL RECORD COPY l

1 l

l

- _. - . . - . - - . _ . - . - . . _ - ~ . - . -. . .. .-

l

~ '

i

. l 1

which was submitted to the workshop staff who would analyze all feedback I forms. I will receive the actual evaluation forms at a later date and will l forward them to you for review. i In sum, the workshop sessions I presented were well-received and successful.

1 l

Attachments: As Stated l l

l l

)

. . , . MNTATcnth Annual Instruct:rs' Werkshap d* November 3-6,1996

. .a Workshop Program

. Sundav. November 3 6:30 - 8:00 pm Registration Welcoming Reception (Hors d' oeuvres, cash bar)

Monday.- November 4 7:00 - 8:00 am Registration, Continental Breakfast Governor's Fover (Coffee, tea, juice, rolls, fresh fruit) 8:00 - 9:30 am Opening Comments / Keynote Presentation Governor's A MNTA Chair: Ron Seizert Workshop Director: Brian Hajek Keynote Presentation: Dr Bill Futrell(Dr. Wonderful) 9:30 - 9:45 am Break 9:45 - 11:00 am Breakout Session 1 lHP-2 Adactation of Advanced CRM Trainino to the Nuclear Industrv Governor's E H. Thomas Heinzer Aerospace Research Group This session describes advanced Crew Resource Management training in tae nuclear environment. The goal of this training is to provide operators and technicians with active tools for breaHng the chain of latent and active errors that inevitably precede a mishap. In particular, CRM intervention Training improves cognitive skills as described in NUREG/CR-6126, " Cognitive Skills Requirements in the Control Room." Experience with implementation at power plants, including pre-test and post-test comparisons will be presented along with a description of classroom and simulator activities.

MSE-1~ Trainino for Instructors and Evaluators of OJT Governor's D Mary Craver, IES Utilities, DAEC Have you had difficulty in answering the following questions for your on-the-job (OJT) trainers and evaluators?

How can I promote consistency for OJT Task Performance Es aluations (TPEs)? How can I demonstrate the importance of the OJT/TPE process? How can I emphasize the importance of the self-checking process? How can i promote the importance of procedure understanding? How can i promote classroom discussion on how individual performance techniques can affect the evaluation process?

The presentation will discuss past experiences and lessons leamed with the OJT/TPE process. The session will emphasize the use of a hands-on activity to demonstrate a way students can be involved in the leaming process and thus generate discussion on various training and evaluation techniques.

I

- _ _ _ _ from fatiguing to fascinating.

,f, ART-8d Really Usina Creative Instruction Executive C J. R. Steely, Duke Power Company 7,q

-M

'. This session deals with incorporating creativity and accelerated leaming into your presentations. )

You will see techniques presently being used to train Duke Power's nuclear operators. Increasing student involvement by adding creative exercises to the presentation is the objective. Leam by doing' l

ART-9d Whole Body Team Training Legislative Barry Wallace / Sandra Kelly / Larry Davis, Comed Mission impossible: Line Manager asks you to help improve his group's communication, problem solving, decision making, leadership, trust, and team building skills. Gulp! Not more boring classroom training! Try a new approach - Outdoor Experiential Team Building, or as we call it: Whole Body Team Training.

Whole Body Team Training (WBTT) is based upon proven experiential and accelerated leaming techniques.

All elements of the leamer are involved, including intellectual, physical, emotional, and spiritual. Using the out-of-doors as the classroom, team initiatives challenge and powerful metaphors transfer leaming to the workplace. Individuals and teams grow quickly.

Come see, participate in, and experience WBTT - it might just be the answer for your next Mission impossible.

l ART-4 Incorooratina Learner involvement in Training Classes Senate B Larry Epstein, Wisconsin Electric {

I Tired of giving the same old lecture year after year to the same people just to meet training requirements?

Tired of lecturing and having no one listen? Do you keep asking yourself *How do i get the trainees involved in the leaming process?" or "What is the advantage of being a facilitator instead of a teacher?" If you answered "YES" to any of the above questions, then this breakout session is for you. A sure-fire method for developing lessons and courses with leamer involvement will be the subject of this presentation. At least three specific leamer involved activities will be demonstrated and discussed. Although the activities will be emergency response based, we will demonstrate how to apply the methods to all technical training.

TSD-3 Inceasina Test item Validity in Exam Develooment Judicial  ;

George Usova, US NRC - OLB HQ 1

This workshop session will address test item and examination development guidelines established by the NRC l in developing and reviewing operator license examinations. The session will include a discussion and background on test validity, define and assess higher cognitive level questions, describe elements of psychometrics, and provide practice in critiquing acceptable and unacceptable items.

TSD Video Critiauina of Performance Based Training House  !

Thomas Chartrand and Bruce Jahn, Consumers Power This breakout session presents training methods which can be used to allow trainees to critique their own performance during scenario exercises. The session will contain information on the following topics:

• Description of hardware necessary to support scenarics.

e How scenario topics relate to worker performance problems.

e instsuctor techniques during facilitation of scenarios.

e instructor techniques during facilitation of cntiques.

e Group exercise to help participants identify possible uses of video critique techniques at their own -

work site.

If you want to better utilize your limited training resources while maximizing line management ownership of training, this workshop is for you!

Q: .'-

INCREASING TEST ITEM VALIDITY IN EXAMINATION DEVELOPMENT ,

MNTA INSTRUCTOR WORKSHOP NOVEMBER 5,1996 Dr. George M. Usova USNRC Training and Assessment Specialist 301-415-1064 i

t SESSION OBJECTIVE:

)

To review validity concepts affecting the NRC written examination for the purpose of: ,

instructing facility personnel toward the construction of more valid and consistent NRC license examinations l t l  !

l . .

i i

.s. ,

gfj . '.

COVERAGE e 3 Levels of Validity 4 i

i

• 3 Levels of Knowledge i

i

• Discrimination, Sampling, and Cut Scores I

e Psychometrics i

I

+

l i l l l

VALIDITY

(

l -

l A valid test is one which tests what it intends to test.

In training examinations, this means it tests the specific skills and knowledge defined and taught in the

objectives.

In // censing examinations, this means it tests the specific skills and knowledge that should have been defined and .

4 taught in the objectives.

. ..l

. g.

s

-l e

3 LEVELS OF VAllDITY j i

i r

• Content I

(

i i

1

+ Operational i i

e Discnminant  !

t t

i l

i i

i

)

CONTENT VALIDITY i

1 l

l Essentially addresses K/A coverage and i

sampling plan coverage i i

i 1

[

4

)

I F e b

{

. .,. , j

vaw-

)

(f: .'-

OPERATIONAL VALIDITY Addresses two questions:

i

1. Is the question, as stated, important to be known as a '

par: of the operator's job?

2. Does the question, as expressed, require the candidate to perform a job related mental or physical operation?

i DISCRIMINANT VALIDITY Addresses making a distinction of measurement along a continuum of candidate performance.

• The cut score is the exam performance level that

the test yields for making a pass-fail decision.

• Since the cut score is 80 percent, the exam must be written at a level of difficulty that intends to discriminate at the 80 percent level.

4

• The test item, its stem and distractor, interplay are such that, by intent and design, at least 80 percent of the candidates taking the test should answer the item correctly.

~

.. .i VALIDITY

SUMMARY

1. The exam (item) must be content valid, which ,

encompasses job safety significance and sampling.

2. The exam (item) should be operationally oriented: a conceivable mental or psychomotor performance of the job. As such the item should be written at the comprehension or analysis level vice simple fundamental knowledge; this means items that measure problem solving, prediction, and analysis--central to job performance.

3. The exam (item) must discriminate at a moderate level of difficulty as set by the cut score. This means that the stem and distractor interplay are such that at

least 80 percent of the candidates taking the test should answer the item correctly.

DETERMINANTS OF DISCRIMINATION i

t i

4

• Level of examination knowledge  !

• Level of examination difficulty a Passing score j i

e item bank use l

t i

l

.. .- .y I . .:

1

. I 3 LEVELS OF KNOWLEDGE i

Bloom's Taxonomy

• Analysis, Application, Synthesis

• Comprehension

• Fundamental (simple memory)

LEVEL OF KNOWLEDGE

• Bloom's Taxonomy is the reference benchmark NRC uses to classify the levels of knowledge of test items.

• Bloom's Taxonomy is a classification scheme that permits the classification of items by the level (depth) of mental thought and performance required to answer the items.

• Bloom's Taxonomy conceptually can be applied to i written, scenarios, or JPM items.

.. v;.:;. ';

l@: .. ;

i

.]

LEVELS DEFINED j l

LEVEL 1 Fundamental Knowledge testing is defined as a simple mental process that tests the recall or recognition of discrete information bits with concrete referents; exan:ples inckde knowledge of terminology, definitions, set points, or other specific facts.

LEVEL 2 Comprehension testing involves the mental process of i

understanding the material through relating it to its own l parts or to some other material; examples can include rephrasing information in different words, describing or recognizing relationships, showing similarities and differences among parts or wholes, recognizing how systems interact, including consequences or implications.

._.,-} ,

1

@.]

1 LEVEL 3 l

Analysis, synthesis, and application testing is a more i

active and product-oriented testing which involves the multi-part mental process of assembling, sorting, or integrating the parts (information bits and their relationships) so that the whole, and the sum of its parts i

can be used to: predict an event or outcome, solve a l

problem, or create something new, i.e., mentally using the knowledge and its meaning to solve problems.

NATURE OF EXAMINATIONS AND TESTS 1

• Tests are samples of performance -

• infer overall performance based upon sample

• Sample must be broad-based to make confident inference

• Sample must not be fully predictable otherwise inferences cannot be made on untested areas l

l

• ltems must intend to discriminate otherwise

! test has little or no value q #

• *

• a

ff:.'-j

.{

INTEGRITY AND VALIDITY

• Examination integrity is essential to making a '

l valid licensing decision.

• Integrity of an examination is central to its validity.

i

!

• A valid examination / test item must be

• Content valid.

• Job-related and operational.

• Discriminate between safe and unsafe operator performance at 80% level.

l l

PSYCHOMETRICS Items may have one or more of the following -

psychometric errors:

4

1. Low level of knowledge (fundamental knowledge)

2. Low operational validity (not clearly job operational)

3. Low discriminatory valioity (too easy or too hard)

4. Implausible distractors

5. Confusing language or ambiguous questions

6. Confusing or inappropriate negatives in the question

7. Collection of true/ false statements

8. Backward logic 9

I y 4 #

t - - - - - - -- -- .

( Q i : .'

LOW LEVEL OF KNOWLEDGE Which one of the following is powered from 4160 VAC bus 1 A7

a. RHR pump A*

b. RHR pump B

c. RHR pump C

d. RHR pump D i

LOW LEVEL OF KNOWLEDGE Select the full core display indication of a drifting control rod. -

a. Red light *

b. White light '

c. Blue light

d. Amber light

• • 1 9

LOW LEVEL OF KNOWLEDGE .

Concerning use of water as a fire extinguishing agent, SELECT the correct statement from the following:

a. Primary agent for extinguishing Class A fires and also effective on Class B and C fires *

b. Primary agent for extinguishing Class B fires band also effective on Class A and C fires

c. Primary agent for extinguishing Class A and B fires but not effective on Class C fires

d. Primary agent for extinguishing Class B and C fires but not effective on Class A fires

r i

f This level of knowledge is low because it doesn't test the candidate's ability to recognize what class a fire is then '

to select the correct extinguisher then use it. All the i candidate needs to know is that water is for class- A.

e e g *

. y,,'*.

-~

Q .'-

LOW LEVEL OF KNOWLEDGE The following plant conditions exist:

o RCP 2A tripped after running for 50 minutes.

o The RCP was restarted, but tripped within 15 seconds.

Which ONE of the following is the minimum required interval before the next attempt to start RCP 2A7 a.15 minutes

b. 30 minutes *

c. 45 minutes l d. 60 minutes

The above question might be considered a fundamental knowledge level that is too hard unless the operators are expected to know the correct time requirement in order to preclude damage to equipment. -

a i

G 9 6

• *p e 4

~

@

;i LOW OPERATIONAL VALIDITY Under which one of the following conditions should the Shift Supervisor inform the shop steward?

a. Initiation of a directed overtime request

b. Discipline action on a supervisory personnel

c. Medical injury of a contractor personnel

d. Personnel error by a bargaining unit member

• While this may be related to a SS's job, it has nothing to do with nuclear safety and should not be included in an NRC examination.

3 l

l i

w- l l

ary l

s f o e ota v n e %i 0m l

r e 0i r t

a 1 c i

s w yd r b

- Y o nw

- T t I

D c a

wlo of I

e no L

A r y k s g l V n n ei

- i o bd

- Y w? 3 dan R ops y l u s O l l d om nn oe l

T oa _

A f u hm ep s -

N

• 22 nt i

I ht R yddly ol l M

f H l

nnnn i t a _

_ I R oR oaao at .

ee ma C 11.16 r s nh S l l l l or I

D ot eeee f o y. _

het vvvv eeee nt ai t ca i

. W i ht i LLLL s rd iel i O i hpa _

Wn L i abcd Tov

't 1

G: ',

LOW DISCRIMINATORY VALIDITY The plant is recovering from a scram due to a spurious Group I isolation. The cause of the isolation has been repaired and preparations are being made to reopen the MSIVs. Reactor pressure is currently 825 psig and the main steam lines are being pressurized.

WHICH ONE (1) of the following represents the LOWEST main steam line pressure that will allow the MSIVs to be opened per procedure?

a. 625 psig l

b. 675 psig*

c. 725 psig

d. 775 psig

t;!:iil l {!li I!! 'l:i : ! : . l \ lllllllj

_ o .

• h ,

t e .

• feh

_ i t

l dl l

a e z a n

eio r ri

_ ot

_ i n mr a e

smp ee u o

_ ae cvi n ea bhw o eol

_ tt a e

_ ndb

_ i e

i mtd r e u cl

_ cpo

_ i de s xw

_ h

't ec

_ nbu st s e 'n s o

dl da n ud o on iwa t _

st e er u nu ady qc.i ei t sl ipol cdi hpr a Tapv

@: -l .

IMPLAUSIBLE DISTRACTORS Which of the following will cause the RHR. pumps to start during a design basis LOCA?

a. Low drywell pressure

b. High reactor water level

c. High drywell pressure *

d. MSIVs in the NOT OPEN position Distractors a, b, and d are implausible distractors considering minimal knowledge of the plant response to a LOCA.

r

l l lMPLAUSIBLE DISTRACTORS Which ONE of the following conditions will NOT result in a shutdown of the SBGT System? ~

a. Manual shutdown

b. High temperature 225 deg F charcoal bed

c. High temperature 180 deg F heater inlet

• l d. Overloads in local control panel l

! Distractor a is very implausible. Distractor d is l subjective.

• ~,.'.

]

• s .

CONFUSING LANGUAGE OR AMBIGUOUS STEM Which one of the following parameters will start HPCI, RCIC and SBGTS?

a. Low reactor water level

b. High primary containment pressure *

c. High reactor building exhaust radiation

d. Low reactor building differential pressure This question could result in 4 correct answers since the question could be read to be individually or collectively.

CONFUSING OR INAPPROPRIATE NEGAhlVES IN THE QUESTION Regarding temporary plant alterations (TPA), technical reviews are NOT required

a. for a TPA NOT installed using an approved procedure.

b. for TPAs installed on BOP systems but are required for safety related systems.

c. for a TPA that has NOT been directed by the shift supervisor to be an emergency TPA.

d. for all TPAs directed by the shift supervisor.

l -

~.

4 This question contains multiple problems: (1) While negative questions can be used, they should be used for good reason; in the above example, there appears to be no good basis for asking this question negatively.' (2)

Two of the distractors (a and c) also contain a negative, creating a double negative and readability confusions, a violation of good item writing practice.

This question would be more appropriately written as "Under which of the following conditions are technical reviews required?" This phrasing would eliminate the negative in the stem.,

i i

COLLECTIONS OF TRUE/ FALSE STATEMENTS Which of the following are true? l

a. High drywell pressure will auto start the emergency diesel generators.*

b. Low reactor water level will trip the main turbine.

c. High reactor pressure will initiate RCIC.

d. High reactor power with the mode switch in startup will NOT close the MSIVs.

I

: .: .l

~

COLLECTIONS OF TRUE/ FALSE STATEMENTS Which one of the following describes pump cavitation?

a. Vapor bubbles are formed when the enthalpy ~

difference between pump discharge and a pump suction exceeds the latent heat of vaporization.

b. Vapor bubbles are formed in the eye of the pump and collapse as they enter higher pressure regions of the pump.*

c. Vapor bubbles are produced vehen the !cce!! zed l pressure exceeds the vapor pressure at the existing temperature.

D. Vapor bubbles are discharged from the pump where they impinge on downstream piping and cause a water haminer.

Both examples represent an error of a collection of true/ false statements, which typically only test simple rote memory; the candidate needs only to reed a definition or condition. The question elicits no comprehension or problem-solving; hence, the question lacks operational validity. This type of question allows a candidate to answer the question without the stem of the question.

~

- w. .

- ep :

BACKWARD LOGIC i

Backward logic is a question that asks the candidate for Information normally received, and provides the candidate

with information he/she normally has to supply. In an operational setting, operators are faced with conditions and required to know what procedure (s) to use. Instead l

the question asks them to do just the opposite.

1 i

L---_-___-----__________-______-__-______________-_-____-_____

i

. i BACKWARD LOGIC Which of the following parameters will simultaneously start HPCI, RCIC and SBGTS?

a. High RPV water level

b. High drywell pressure *

c. Low RPV water level -

d. Low drywell pressure It is better to select a parameter and then request the expected system response because that is more operationally relevant.

,, PSYCH 0 METRIC GUIDELINES FROM EXAMINER STANDARDS l

Multiole-Choice Items i

l a. Does the question have one focused topic, making it something other than a collection of true-false items?

b. Is as much information as possible included in the stem?

c. Is the question or problem defined in the stem?

d. Are tricky or irrelevant questions avoided?

f l

e. Are the answer options homogeneous and highly plausible?

i i f. Are "none of the above" and "all of the above" avoided?-

l

g. Are there an appropriate number of options for each question?

l j h. . Is each item stat of what not to do,'d positively, unless the intent is to test knowledge 2

1. Is the question free of " specific determiners" (e.g., logical or grammatical inconsistencies, incorrect answers which are consistently

different, verbal associations between the stem and the answer options)?

4 j ._ Are common misconceptions used as distractors?

] k. Are the answer options of the items ordered sequentially?

f 1. Is the question free of trivial distractors?

1 i

e o*.

Nd

- r- - w - .- , _ + - - -

i l .

!h NRC CHECKLIST FOR OPEN-REFERENCE TEST ITEMS Test Item Level l 1. Does each test item have a documented link to important licensee j tasks, X/As, and/or facility learning objectives?

! 2. Is each question operationally oriented (i.e., is there a j correlation between job demands and test demands)?  ;

i

3. Is the question at least at the comprehension-level of knowledge?

1 l

4. Is the context of the questions realistic and free of window dressing and backwards logic?

5. Does the item require an appropriate use of references (i.e., use of l analysis skills or synthesis of information either to discern what procedures were applicable or to consult the procedures to obtain l the answer)? j

6. Is the question a " direct look-up" question, or does one question on the examination compromise another? A " direct look-up question" is  ;

defined as a question that immediately directs an licensee to a l particular reference where the answer is readily available.

7. Does the question possess a high K/A importance factor (3 or i greater) for the job position? j

8. Does the question discriminate a competent licensee from one who is-not? l

g. Is the question appropriate for the written examination :nd the selected written examination fomat (e.g., short answer; multiple choice)? i

10. Do questions in Section A take advantage of the simulator control room setting?

11. Does any question have the potential of being a " double-jeopardy" question?

12. Is the question clear, precise, and easy to read and understand?

13. Is there only one correct answer to the question?-

14. Does the question pose situations and problems other than those presented during. training?

_ 15. Does the question have a reasonable estimated response time?

,