ML20137N657

Trip Rept of 970325 Visit to Lombard,Il to Conduct 4 H Workshop Entitled, Principles & Practices of Measurement & Psychometric in NRC Written Exam. List of Attendees Encl

ML20137N657
Person / Time
Issue date:	04/03/1997
From:	Usova G NRC (Affiliation Not Assigned)
To:	Richards S NRC (Affiliation Not Assigned)
References
NUDOCS 9704090015
Download: ML20137N657 (59)

Text

, -

.,r .- . - - . - - -. - . - ._-.- . - .

1 i;  ; fY

. . , y 4 UNITED STATES I. j

.J' NUCLEAR REGULATORY COMMISSION k '?  !. WASHINGTON. D.C. 2006H001

\..... / .f ,

April 3, 1997 l

1 i

. MEMORANDUM'TO: Stuart A. Richards, Chief -

0 r: tor Licensing Branch FROM: George M. Usova M" OperatorLicensingBranchj)[

SUBJECT:

TRIP REPORT: WORKSHOP PRESENTATION AT THE REGION III TRAINING. SUPERVISORS MEETING -)

L j On March 25, 1997, I conducted a four-hour workshop entitled, Principles and I

^ Practices of Measurement and Psychonetrics in the NRC Written Examination,.

to Region III nuclear power plant training supervisors at a scheduled meeting hosted by COMED in Lombard, Illinois. Approximately 55 participants attended the workshop (See Attachment 1). Also in attendance from the NRC was Mr. Jay Lennartz, a license examiner from Region III. Mr. Lennartz conducted an afternoon question and answer session with the participants.

Background

In December 1996, I was contacted Mr.-Paul DiGiovanna of COMED and invited to

present a workshop on examination methods to operations training supervisors at their regularly scheduled training supervisors meeting. I had agreed to conduct the workshop and designed the agenda as shown in Attachment 2.

. Presentation As I explained to all workshop participants at the outset of the presentation, ,

the NRC was making this presentation to promote a common understanding between 1 the NRC and industry on measurement concepts and terminology used in examination davelopment and to communicate expectations of levels of knowledge

< . and difficulty in examinations. Because the burden of examination development responsibility was shifting to industry (although the exam is considered to be , ,

an NRC administered examination), it was prudent for the NRC to provide to

, )

industry the type of guidance contained in the workshop.

l 1 This wo'rkshop session addressed test item and examination development J quidelines established by the operator licensing branch in developing and

, tr. viewing operator license examinatimss. The session also included a background discussion on test validityi 4? fined and assessed higher cognitive f)-

level questions, described elements of psychometrics, and provided participant f 1

. practice in critiquing test items. (See Attachment 3 Workshop Slides).

~

Q\v N30 HLE GENTER COPY 0DW cf77o s

00M KKWS4l 9704090015 970403 PDR ORG NRRA PDR ,

i e

l S. Richards  !

! Several noteworthy questions were raised by the attendees during the presentation and my response follows.

1. Since the NRC wants an exam that intends to discriminate, what L

percentage of failures does the NRC expect on facility-developed J

examinations.

My response: The NRC does not have any preconceived expectation on percentage

, of failures. In fact, given the nature of a criterion-referenced examination, such'as that of the NRC license examination in which individuals are tested on

the content areas trained upon, the expectation is that scores will generally be high and that m6st people will pass the examination. Historically, this

.has been the case. However, occasionally there will be individuals who differ from the norm and will fail the examination; these failures may be attributable to individual differences that exist among people --- differences that manifest themselves in one or more of the following: lower levels of cognitive ability, motivation, aptitude, or personal attention to study and preparation.

On the other hand, the more reliable indicator of an examination's ability to -

discriminate lies in its average score. In this regard, average scores that ,

fall within the 86-90 range - an expected score range given the fixed cut l score and nature of the criterion referenced examination - is a reasonable '

benchmark for effective discrimination. However, examinations that consistently, and over time, yield average scores exceeding 95, for example,

.should be reexamined far effective discrimination.

j 2. What percentage of memory level items are acceptable on an NRC license examination?

My response: The NRC has no fixed percentage of memory level items that should appear on an examination. In general, the more operationally valid an examination, the more likely the examination will contain more items that measure cognitive ability at the higher levels of comprehension, application, and analysis. Realizing that there will likely be some mix of items at all cognitive levels, memory level items should be held to a minimum since they possess less operational and discriminatory validity.

-Attachments: As Stated 4

S. Richards April 3,1997 Several nateworthy questions were raised by the attendees during the

presentation and my response follows. j Since the NRC wants an exam that intends to discriminate, what

~

1. ,

percentage of failures does the NRC expect on facility-developed i examinations.

My response: The NRC does not have any preconceived expectation on percentage of failures. In fact, given the nature of a criterion-referenced examination, such as that of the NRC license examination in which individuals are tested on the content areas trefned upon, the expectation is that scores will generally

. be high and that most people will pass the examination. Historically, this has been the case. However, occasionally there will be ir.dividuals who differ from the norm and will fail the examination; these failures may be

attributable to individual differences that exist among people -- differences that manifest themselves in one or more of the following: lower levels of cognitive ability, motivation, aptitude, or personal attention to study and preparation.

! -On the other hand, the more reliable indicator of an examination's ability to discriminate lies in its average score. In this regard, average scores that j

fall within the 86-90 range - an expected score range given the 80 percent 4

fixed cut score and nature of the criterion referenced examination - is a j i - reasonable benchmark for effective discrimination. However, examinations that l

' consistently, and over time, yield average scores exceeding 95, for example, 1 l should be reexamined for effective discrimination. l i

! 2. What percentage of memory level items are acceptable on an NRC license 2

examination?'

My response: The NRC has no fixed percentage of memory level items that should appear on an examination. In general, the more operationally valid an i examination, the more likely the examination will contain more items that i measure cognitive ability at the higher levels of comprehension, application, and analysis. Realizing that there will likely be some mix of items at all cognitive levels, memory level items should be held to a minimum since they 1 possess less opert;ional and discriminatory validity.

l The workshop began at 8:15 and ended approximately at 12:00 P.M. A number of participants made positive and favorable comments about the content presentation during and after the workshop. My overall assessment was that

, the workshop content and presentation was well received and highly successful.

. Attachments: As stated ,

DISTRIBUTION:

Central Files

, PUBLIC

HOLB RF n e . .e m. - me m m. w

c . cm m .a nu.n.w r - c , e m.a nu.new. v no copy 0FFICE NAME HOLB/DRCH GUsova:ge/rc

[ L I 1 l

~

DATE 04/T/97 "

0FFICIAL RECORD COPY

-u,- , - . ., . - - . - .- , , - - . , a---. r, .,---e- - - - - - - - - - - .

Region HI Operations Training  !

March 25,1997 l

Comed's Leadership Development Center  :

Nasme i Utility Phone # Fax # E-anail Address l gc., , et 3 d. ; i.< h . ; t, - c ,- 9 i-t - A s - 2 rc. o q i t - 3 3 y - M 2 7v- g f;,i .;g cc ,gc, ecy d.Jf . _ . .

L ,.. , c 5d >9yi

g. N Au % $w s ib.n . 4-tr - Svv- //W 6 /2 - 7 3 c, -c. zvo jame s. ta A4 nsps.c% ,

k&  %.u r.\d f h3

• s n ~w

. n W,_ .o G ,, , ._ cf.a - h v su-) 4 ./ ,% ; . 4 ; 3 5

'*'^'~~"'*"

. h V&& uasa3. c f wm _

py W L F (_-o '/N -Jir. (< 3 2 ) y'/I- 19~- 2 2 7 / I AM3. eFk, '.e w ava t -

br i t t lM w2c"Ne-Lah

.J a b l &M

• Y g hl Wh*)$rY GQF Q[ hD e) *h O- "' '

b * =IV I4 *~

\ -

.3 ATTACHMENT 1 ."

.r 5:

Region IH Operations Training March 25,1997

[

Comed's Leadership Development Center 3; Nanne Utility Phone # Fa1# E-mail Address ah f/T)15l'3Y// es23I? S/S Y CS' % P] Skob (2 cerf.e,ee.a a, W 7Lno  % sd <>r>9e-m>ox wo (origa-anca>isi  ?

.9tr., inf:,p Cm iA (q.J w h a v-s n ! x c i.z e c w v- a v/ r s a c by . LJ g cca. <nc... ,

?

p:c S u,~ G ... c, u ~ s.c~ ~ n= - -

5 IL/ 0;6km Ce> >- Ecl as-vse-arn we Gi S- YSr -30s g j 4 9 c c ,,,,. ,. , ,_?

E

'  ?

2 Region IH Operations Training i March 25,1997 -

ConiEd's Leadership Development Center 3 Nanne l Utusty Phase # Fax # E-enna Address

')flil) &&<t:. 1 LSb(; {kbEtN ft9- Zy9 ZV37 f*)ONj$$'Nf'OP'N'#

M< % hwfagg, (Doi g; . ,,,,, (24 M ' !n v /Anam'rwm:wws '

b h aw cEL be fu (zd ze-sse M Vf3 (zS)2so-son 41 % - w C0t o A.u<.e - !

O e

,, f0L. EOtM des 5E Y/4 73/ 3 YM IY4-EV3/ U

a;

.g:

Region III Operations Training I March 25,1997  ;-

Comed's Leadership Development Center A Name UtiEty Phone # Fax # E-annit Addetes k ned- bfhf D2bh- l*IC Yl l't X

5 u

G E

Region III Operations Training [l March 25,1997 "

CorpEd's Leadership Developanent Center 5 Nasme l Utility Phone # Fax # E-utail Address ~

65 ~749'- 2L 6 4.

Q jgiu daw, W S'l 74'f ?/00 Al'Thtse G^w~ds GtG 5+7-WM D  ?

h la0m( Cw*= u s sv7- F/n (.I w -sy ?-syc,  !

brY0brbh 0hSv'her' JM 7ff ?27x M 7cf Ato<a 5 Ga Q[h/ 0a.rmers s i c 7 c, 4 g e n 6tc 7c t proo

(

E

Region III Operations Training 3 March 25,1997  :

Comed's Leadership Development Center i Name Utility Phone # Fat # Femail Address  !

%Hecu. %N bresne e sn~ano sxv. 3rA Johvi Wo6d CowG<t(Dio4Q <;q uw u . 7,3 g;

[dlal4k)W [1 cL Ad ADO h + 9/6/ o p;st 5 2.3. neen~m ( w a* se s oeicm <ca j D ia, O c a caa FM - m w m, , sau j,.

gA I* 1 A D_ b #* A $#l J. AS \k T # "f _ - IY{ #

• IC'# #

)g (L fa,,, y [Obh (%9~~ Nf' I/ b* - % Sk A 109 d 3 sco - x s t.rr

1. 5/' /Ns or #% OM 309 674 (124i ;b,9 -694 .2AtiI GcTQgg s s ,j , ( f g ,

E

l =

Region III Operations Training 5 L Marck 25,1997 ,

CosmEd's Leadership Developnient Center i Name Utility Phone # Fax # E-mail Address a .;

KE fi [.6FUL To.~s fe~cv .) y, mi v,,,.t OlestMy& b b 6diso 5 % - 4109 $G-45bD AJD&bYs.M G"1 5 Q

n .r dias Kaar dex 2n-nem am 4nwsr- ein E 2/7 935 888/,3?It 217-935 - DIT ,~

lilirios"s .?ou'Et

7) M b. s lf M  :

A

__  ?

E E

FROMs -40M ED PTC FAX NO.s 819 098 482 P4.e3.g7 10:36A 8.10 I k

+ ,

t 0 . $

g l 1 3'

{

. .t d'

= h s

I b::

~t i e  ;

,3 a t. u u5 i E

, u o ,

~

be N

a g s

~

s s

. ~ b

.. . . s g E M f N

.. e "

a -

W h 2 E $ @ C i& m +~ nr +

$ M- b a- >

og ,  %

g D D u

N \p N b

,n Q, M s ~

s g g 7 - ,

b#2

• k b I N d i o!c  ! wRw :n -

a e ~ r

. E g.

Y n b h w s

A B

e i

C si s Q G  % W  % T  %

@ d $ 5 $ $ M  % O

,h l*

~

U QQ u g ti n s

. s g r b 3 Q D t) M M N k vs C N a s

.* g . a u m iu W L x 1 M 3 4  %  % 4  %

% b I %

i a u g 0 4

? *$

d o k I1 < . o3 U p 4-

.) ji @$y.s;s e

o nc m so.: .is as u.. .

Announcement of next Region IH Quarterly i L Training Supervisors Meeting Hosted By i Afarting Information: __ _

Where: Comed I.cadership Developinent Cer.tcr (LDC)in Oak Brook lilinois Whcis: 0730. l(n,. . esday, March 25 IW7 0730 14m. Wednesday. March 26, IW7 Ag:enda: 1)ay 1 0730.0x30 Couce and...Inansloctions OX)n-12m Principle and Pracuces of Meu>uremen:and Dr. George Uwiva. NRC Psychometric3 in the NRC Written Emnination. Training and Anei,stoent Specialist This workshop willaddre>s test item and exam descloprnent guideltac8 established by the NRC in developmg and reviewing operor licen>e cums The sc>sion w di include a discus ion and badground on fdi vahdity, define and 4weu hisher cognitae icvel y..ruons, Jesnbe elements of psychemetno. .mJ pronic practice m critiquing acceptable and ullaccCpluhlC items 12(MblNWl Linn h.

1300-1500 Do.cusuon of Rev H ivues/ Draf t Rev 8 L Lennarv Reg ll1 NRC Additional Reg ill examiners will d:3cus,recent Acting Ch,cf Operator pilot c wn experience- Licensmg Branch 1500-INW)

Questoms and Opert Diwus3 ion (ILT and Requal)

Ag ndu,l)uy 2 0730 0830 Collec tuid.

Submit ideaVnceds for day 2 acenda. liem> that w di be docuwed are recent INPO viuts, resource etthange program, etc. A detauled iger.da will follow.

Suppcxted auend.uice would mclude the operadons tr.iinmg supervisor and > our inidal exam deseloper.

Due to the expcued high level of attendemcc, I will need a reply by Feb. 2N. lW7 Statmg the number of auendecs per udlhyNie. Please umtxt me at 815 45H0411 x 2218, or ahwpdecanail.cevo.com. idie emer the better, especially if you have mput to the agenda)

Look lofward too wemy you all soon Paul DiGurv.um.s ComF.d ATTACHMENT 2

PRINCIPLES AND PRACTICES OF MEASUREMENT AND PSYCHOMETRICS IN "

THE NRC 'NRITTEN EXAMINATION t

TRAINING SUPERVISOR'S WORKSHOP MARCH 25,1997 i

George M. Usova Training and Assessment Specialist

USNRC

301-415-1064 ATTACHMENT 3

_.-_.-___-__-..__m__.__-...-.m..- -.-t_ __..___m..____..___-_______-__.____..-_m. _ . . _ _ _ _ _ . * '-+ 1 w - *i&'ei

WORKSHOP OUTCOMES e Common terminology e Common understanding i

e Shared expectations e Less rework e Better examination product

SESSION OBJECTIVE:

I To review validity concepts affecting the NRC .

written examination for the purpose of: .

instructing facility personnel towarci the construction of more valid and consistent NRC license .

examinations  ;

i 4 4

COVERAGE i l l

l l e 3 Levels of Validity .

f r  :

I i

e 3 Levels of Knowledge  :

1 j e Discrimination, Sampling, and Cut Scores I

e Psychometrics i

I i

l

VALIDITY l

l l A valid test is one which tests what it intends to test.

l in training examinations, this means it tests the specific skills and knowledge defined and taught in the objectives.

In // censing examinations, this means it tests the specific skills and knowledge that should have been defined and taught in the objectives.

.l

-l

3 LEVELS OF VALIDITY t

• Content

• Operational 1

• D.iscriminant 3

J

__ _ _ _ _ _ _ _ _ _ _ __ _ _ _ _ _____ _ _ __._______ ___ ___ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - m _ ____ _ _ _ _ _ ____ .. _ _ . __i

. i ' ! '  ?! r. I , l 1 l: I i

d Y n a

T I

D e

_ I L

g a

A r

_ e V v _

T o _

c N

E A _

T /

K e N g _

O s a e

~

r e

C s s v

- e o r c d

d n

- a l a

y p l

l g i

a n t i n l p

e s m s a E s -

~.

OPERATIONAL VALIDITY Addresses two questions:

1. Is the question, as stated, important to be known as a part of the operator's job? i l

2. Does the question, as expressed, require the candidate to perform a job related mental or physical operation?

1 i

DISCRIMINANT VALIDITY I

l Addresses making a distinction of measurement along a l continuum of candidate performance.

i

• The cut score is the exam performance level that the test yields for making a pass-fail decision.

• Since the cut score is 80 percent, the exam must be written at a level of difficulty that Intends to discriminate at the 80 percent level.

• The test item, its stem and distractor, interplay are l such that, by intent and design, at least 80 percent of l the candidates taking the test should answer the item correctly.

i I , * ,e

s '. .

VALIDITY

SUMMARY

~

1. The exam (item) must be content valid, which ,

encompasses job safety significance and sampling.

i

2. The exam (item) should be operationally oriented: a i

conceivable mental or psychomotor performance of the job. As such the item should be written at the comprehension or analysis level vice simple fundamental ,

knowledge; this means items that measure problem solving, pr.ediction, and analysis--central to job performance.

3. The exam (item) must. discriminate at a moderate level of difficulty as set by the cut score. This means that the stem and distractor interplay are such that at least 80 percent of the candidates taking the test should answer the item correctly.

~

l

i 3 LEVELS OF KNOWLEDGE ,

Bloom's Taxonomy i

i i

• Analysis, Application, Synthesis

• Comprehension i

• Fundamental (simple memory)  :

l

LEVEL OF KNOWLEDGE

• Bloom's Taxonomy is the reference benchmark NRC uses to classify the levels of knowledge of test items.

• Bloom's Taxonomy is a classification scheme that permits the classification of items by the level (depth) of mental thought and performance required to -

answer the items.

I

• Bloom's Taxonomy conceptually can be applied to written, scenarios, or JPM items.

i 1

~

LEVELS DEFINED .

3 LEVEL 1 q Fundamental Knowledge testing is defined as a simple mental process that tests the recall or recognition of discrete information bits with concrete referents; examples include knowledge of terminology, definitions, set points, or other specific facts.

r

~

r

l LEVEL 2 l ,

Comprehension testing involves the mental process of understanding the material through relating it to its own parts or to some other material; examples can include i

! rephrasing information in different words, describing or recognizing relationships, showing similarities and i

differences among parts or wholes, recognizing how systems interact, including consequences or implications.

i

l l

LEVEL 3 .

Analysis, synthesis, and application testing is a more active and product-oriented testing which involves the multi-part mental process of assembling, sorting, or integrating the parts (information bits and their j relationships) so that the whole, and the sum of its parts can be used to: predict an event or outcome, solve a problem, or create something new, i.e., mentally using the knowledge and its meaning to solve problems.

. ..l

DETERMINANTS OF DISCRIMINATION i

s i

• Level of examination knowledge

• Level of examination difficulty

+

l

• Passing score l

l

• Item bank use

' :r !I pI .  ;.tlll  ! :l  ! ; '

t n e e iss e

l p

d i

f wa r re e s

n

- m o h ea i w

S -

a c td r T s oet e S n e h E o k a l es t o

T p bet

- D u m tan t e

N d o cu a e t i

dn n

- A s d eo i

- S e a e r pe i m

c b s r

- N a yad c

_ O n a

e c b- l ius e um l

_ I n dl a T m a d a f

- A r o o e ov beb m t N f r r r o

- I e o b t o t dnn

. M p f r e on e r A f e b nn t o n

X o p t s

t a sc ie E s l tt l

e l

a u u s sit F

O l

p r

e mec mec ul E m v o l e n l e n msa

- R a r pe pe r

sh U s e mre me mt e s T e f af n af e

- A r

a I n Si Si n lt t N s t

s * * *

• e T

PSYCHOMETRICS Items may have one or more of the following l psychometric errors:

L

1. Low level of knowledge (fundamental knowledge)

2. Low operational validity (not clearly job operational)

3. Low discriminatory validity (too easy or too hard)

4. Implausible distractors

5. Confusing language or ambiguous questions

6. Confusing or inappropriate negatives in the question

7. Collection of true/ false statements

8. Backward logic t

i

F l

LOW LEVEL OF KNOWLEDGE L

l Which one of the following is powered from 4160 VAC l

bus 1 A?

l

a. RHR pump A* .

b. RHR pump B

c. RHR pump C

d. RHR pump D ,

l t

i E

L

LOW LEVEL OF KNOWLEDGE .

l l Select the full core display indication of a drifting control l rod. .

l

a. Red light *

b. White light .

c. Blue light

d. Amber light v

i i

i

1

~

LOW LEVEL OF KNOWLEDGE Concerning use of water as a fire extinguishing agent, .

SELECT the correct statement from the following.

a. Primary agent for extinguishing Class A fires and also effective.on Class B and C fires *

b. Primarv agent for extinguishing Class B fires band also effective on Class A and C fires

c. Primary agent for extinguishing Class A and B fires I but not effective on Class C firee .

d. Primary agent for extinguishing Class B and C fires but not effective on Class A fires

This level of knowledge is low because it doesn't test the candidate's ability to recognize what class a fire is then to select the correct extinguisher then use it. All the candidate needs to know is that water is for class A.

e l

l t

i LOW LEVEL OF KNOWLEDGE The following plant conditions exist:

o RCP 2A tripped after running for 50 minutes.

o The RCP was restarted, but tripped within 15 seconds. .

4 l Which ONE of the following is the minimum required interval before the next attempt to start RCP 2A?

~

a.15 minutes  ;

b. 30 minutes *

c. 45 minutes

d. 60 minutes

. , p

The above question might be considered a fundamental knowledge level that is too hard unless the operators are expected to know the correct time requirement in order to preclude damage to equipment. ~

~

l LOW OPERATIONAL VALIDITY l Under which one of the following conditions should the l Shift Supervisor inform the shop steward?

j

a. Initiation of a directed overtime request .

b. Discipline action on a supervisory personnel

c. Medical injury of a contractor personnel

d. Personnel error by a bargaining unit member

• l While this may be related to a SS's job, it has nothing to do with nuclear safety and should not be included in an NRC examination.

l l

l ..

! . +;l > l' i  ;: '  ;;: ll<  ! ; i !!

_ ~ _.

l l

- i w-ary l

l l

s f o e ota

- v n -

e %i l

r e

0m 0r i

t 1 c a

- s w yi d _

.- b Y r nw T t o -

I D c wlo -

a of I

L e r

no -

- y k A g l n

s _

V ei i

n o bd .

n Y w? 3 da

- R ops y l u s O d l

oe l

om nn l

T oa hm A f u s

- ep 22 i N

• nt

- I ht R yddly ol l M f H l nnnn i t a I

R oR oaao at ma C ee 11.16 r s S nh l l l l or

- I D

ot eeee f o y.

het vvvv eeee nt iai t

_ W i ca i LLLL s rd iel i ht O i

. . . . hpa L Win abcd Tov ll l  ;

9 LOW DISCRIMINATORY VALIDITY The plant is recovering from a scram due to a spurious Group I isolation. The cause of the isolation has been repaired and preparations are being made to reopen the ,

MSIVs. Reactor pressure is currently 825 psig and the main steam lines are being pressurized.

WHICH ONE (1) of the following represents the LOWEST main steam line pressure that will allow the MSIVs to be opened per procedure?

a. 625 psig

b. 675 psig* .

c. 725 psig

d. 775 psig e ,

,.  : ! l!  ! ' .  : .:[

' , , I i I !!!  ; { I : !n e

h t e eh f t i

l dl l

a e na' z

eio r r i ot imrn a e ee smp u o ae cvi n

ea bhw o eol t t a e ie ndb mtdu icl r e cpo s

ixw de h t e c

_ 'nst s bu e 'n s o

_ dl da ud n

o on wa i

. t st er

_ e u ady nu .

qc ei i

t i

slpol cd i hpr a Tapv

' i' i

IMPLAUSIBLE DISTRACTORS i Which of the following will cause the RHR pumps to start during a design basis LOCA?

a. Low drywell pressure

b. High reactor water level

c. High drywell pressure *

d. MSIVs in the NOT OPEN po.sition Distractors a, b, and d are implausible distractors considering minitaal knowledge of the. plant response to a '

LOCA.

I CONFUSING OR INAPPROPRIATE NEGATIVES IN THE QUESTION

Regarding temporary plant alterations (TPA), technical

reviews are NOT required

a. for a TPA NOT installed using an approved procedure.

l'

b. for TPAs installed on BOP systems but are required for safety related systems.

c. for a TPA that has NOT been directed by the shift supervisor to be an emergency TPA.

d. for all TPAs directed by the shift supervisor.

l

. t l

This question contains multiple problems: (1) While i negative questions can be used, they should be used for good reason; in the above example, there appears to be no good basis for asking this question negatively. (2)

Two of the distractors (a and c) also contain a negative, creating a double negative and readability confusions, a  ;

violation of good item writing practice.

i This question would be more appropriately written as "Under which of the following conditions are technical reviews required?" This phrasing would eliminate the l

negative in the stem. 1

. .i

COLLECTIONS OF TRUE/ FALSE STATEMENTS Which of the following are true?

a. High dryweII pressure will auto start the emergency

diesel generators.* <

b. Low reactor water level will trip the main turbine.

c. High reactor pressure will initiate RCIC. - o

! d. High reactor power with the mode switch in startup will NOT close the MSIVs. -

l l

I

. COLLECTIONS OF TRUE/ FALSE STATEMENTS Which one of the following describes pump cavitation?

a. Vapor bubbles are formed when the enthalpy

difference between pump discharge and a pump suction exceeds the latent heat of vaporization.

b. Vapor bubbles are formed in the eye of the pump and collapse as they enter higher pressure regions of the pump.* -

c. Vapor bubbles are produced when the localized pressure exceeds the vapor pressure at the existing temperature.

D. Vapor bubbles are discharged from the pump where they impinge on downstream piping and cause a water hammer.

~~i

Both examples represent an error of a collection of -

true/ false statements, which typically only test simple i

rote memory; the candidate needs only to recall a definition or condition. The question elicits no ~

comprehens' ion or problem-solving; hence, the question.

! lacks operational validity. This type of question allows a t

candidate to answer the question without the stem of the question.

I

BACKWARD LOGIC Backward logic is a question that asks the candidate for information normally received, and provides the candidate with information he/she normally has to supply. In an operational setting, operators are faced with conditions and required to know what procedure (s) to use.. Instead the question asks them to do just the opposite.

l

_. .\

BACKWARD LOGIC Which of the following parameters will simultaneo~usly '

i start HPCI, RCIC and SBGTS?

! a. High RPV water level

b. High drywell pressure

• i

c. Low RPV water level -

d. Low drywell pressure It is better to select a parameter and then request the expected. system response because that is more i operationally relevant.

l

3 1

BACKWARD LOGIC - an everyday example l

"If it takes 12.5 cubic feet of concrete to build a square i loading pad 6 inches thick, what is the length of one side of the pad?"

i This question gives the test takers information they should be asked to calculate, while it requires them to provide information they would be supplied in an actual work situation.

b l

..l

)

PSYCH 0 METRIC GUIDELINES FROM EXAMINER STAISARDS .

Multinle-choice Items

a. Does the question have one focused topic, making 'it something other than a collection of true-false items?

b. Is as much information as possible included.in the stem?

c. Is the question or problem dafdr. d in the stem?

d. Are tricky or irrelevant questions avoided? '

e. Are the answer options homogeneous and highly plausible?

f.. Are "none of the above" and "all of the above" avoided?

g. Are there an appropriate number of options for each question?

h. Is each item stated positively, unless the intent is to test knowledge i of what not to do? l

1. Is the question free of " specific determiners" (e.g., logical or grammatical inconsistencies, incorrect answers which are consistently different, verbal associations between the stem and the answer options)?

I j . Are common misconceptions used as distractors? I

k. Are the answer options of the items ordered sequentially?

1. Is tiie question free of trivial distractors?

1 4

9 4

i

. NRC CHECKLIST FOR OPEN-REFERENCE TEST ITEMS Test Item level

1. Does each test item have a documented link to important licensee tasks, K/As, and/or facility learning objectives?

2. Is each question operationally oriented (i.e., is there a correlation between job demands and test demands)?

3. Is the question at least at the comprehension-level of knowledge?

4. Is the context of the questions realistic and free of window dressing and backwards logic?

5. Does the item require an appropriate use of references (i.e., use of analysis skills or synthesis of information either to discern what procedures were applicable or to consult the procedures to obtain the answer)?

6. Is the question a " direct look-up" question, or does one question on the examination compromise another? A " direct look-up question" is defined as a question that immediately directs an licensee to a particular reference where the answer is readily available.

7. Does the question possess a high K/A importance factor (3 or greater) for the job position?

8. Does the question discriminate a competent licensee from one who is not?

9. Is the question appropriate for the written examination and the selected written examination format (e.g., short ooswer; multiple choice)?

10. Do questions in Section A take advantage of the simulator control room setting?

11. Does any question have the potential of being a " double-jeopardy" question?

12. Is the question clear, precise, and easy to read and understand?

13. Is there only one correct answer to the question?

14. Does the question pose situations and problems other than those presented during training?

15. Does the question have a reasonable estimated response time?

s-

+

l l l i

1

.i i I

l J

COMPREHENSION ,

1 I

l i

APPLICATION S

OR MEMORY?

i ,

4 d

4 Dr. George.M. Usova Training and Assessment Specialist USNRC 4

e- --

COMPREHENSION / APPLICATION OR MEMORY LEVEL?

d Q: When is a comorehension/aoolication level auestion NOT likely a comorehension/anolication level auestion?

There are several categories of misapprehension or misconception of what distinguishes a fundamantal simple memory level question from that of a higher cognitive level question.

Although the facility never argued that Example below was more than memory level, its revision was argued as an application level question.

EXAMPLE:

As required by 10 CFR 26, " Fitness for Duty Programs," what is the MININUM time an operator must abstain from the consumption of alcohol prior to any scheduled shift?

a. 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />

b. 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />

c. 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />

d. 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> Answer: c K/A 294001K1.05 - memory Evaluation. The above question is clearly a fundamental knowledge level question since it elicits from memory a simple detail / fact.

REVISION .

, You were called this morning and requested to assist with a unit startup this evening. To enter the plant at 11 PM that evening, what is the LATEST time that you could consume alcohol?

a. 2 PM

b. 4 PM

c. 6 PM

d. 8 PM Answer: c Evaluation. Although the revised question required the secondary mental process of subtraction to arrive at the correct answer, the ancillary process of subtraction is only an intervening step t. hat : lightly complicates the process of arriving at the correct answer. subtraction is required to answer the question but is unrelated to the knowledge required of a competent operator.

I COMPREHENSION /APPLICATI0M OR MEMORY LEVEL 7 EXAMPLE:

While performing surveillance procedure W1 on the Unit-1, division-2125 VDC battery, you obtain the following data:

- ICV: 2.14

- Temperature: 86 degrees F

- Level: -1 (as recorded)

- Specific Gravity (uncorrected):~1.201

- Battery Charger Current: 5 amps on float S.G. (corrected) = 5.G. (uncorrected) + fT - 77 decrees3 F)(.001)

+ (Level Mark)(.003)

Based on the above information, the Specific Gravity is which meet the Technical Specification Category A limit.

Note: This question requires the use of TS 3.8.2.3.

. a. 1.198, does NOT

b. 1.195, does NOT )

c. 1.207, does

d. 1.201, does ,

1  :

Answer: d K/A 295004G008 (Application)

Evaluation. This question might appear to test the applicant's ability to understand and apply battery parameters to the determination of TS operability. However, as one can plainly see,.the question only tests the applicants to substitute certain parameters into a given equation and perform an arithmetic calculation. Reference to the TS noted in the question is not required based on the three different values of S.G. (corrected) supplied as I 1

distractors. Therefore, the question is of no discriminatory value since any individual possessing adequate arithmetic knowledge will arrive at the correct ,

answer. l l

l

i. .

i*

COMPREHENSION / APPLICATION OR MEMORY LEVEL 7

The facility argued that the question below was at the comprehension' level.

Since the question was not asking the expected, memorized set point of 110 F, the change to 112 F elevated the thought process beyond the memory level.

EXAMPLE r i

In ~ response to a stuck open SRV, which of the following conditions would

, reauire scramming the reactor?

a. SRV tailpipe temperature reaches 350 degrees F.

4 b. Two cyclings of the SRV's handswitch were unsuccessful, i c. One minute has elapsed since the SRV first opened.

d. Suppression pool temperature is 112 degrees F.

Answer: d i

4 Evaluation. Although TS limits are 110 degrees F, raising the number by 2 3

degrees to 112 does not meaningfully elevate the level of knowledge to a higher, more operationally valid level where relationships are interrelated

, and decision-making is required.

i 4

a 4

i

e COMPREHENSI9N/ APPLICATION OR MEMORY LEVEL 7 EXAMPLE The example below may or may not be at the comprehension level. It can represent a more subtle example that might is projected as comprehension but upon closer inspection is simply the extraction of verbatim text from a

" Discussion" segment of training / reference material.

In response to a major leak in the Instrument Air system, the operators have scrammed the reactor with the reactor mode switch taken to SHUTDOWN.

As the Instrument Air System pressure continues to decrease, what will be the effect on the MSIVs?

Inboard Outboard MSIVs MSIVs

a. Remain open Will close

b. Will close Remain open

c. Will close Will close

d. Remain open Remain open Answer: a REF: Abnormal operating procedure, Loss of Instrument Air and/or Service Air, Page 3, Section E, Discussion.

i Evaluation. Because the material was drawn from a discussion session does not automatically elevate its level of knowledge. If the response asked for in the test item requires the candidate to independently interrelate a causal relationship that' is not directly addressed in the material, + hen the question is truely at the comprehension level. On the other hand, if the causal relationship is directly presented as such in either textual material or

., classroom training, and is then again directly represented as such in a test item, there is arguably no individual independent comprehension mental process occuring. The question has been relegated to a verbatim recognition / recall level.

E