ML17291A980

Discusses 950907 Meeting W/Util to Discuss Implementation of Pilot Exam Development Program,Per GL 95-06

ML17291A980
Person / Time
Site:	Columbia
Issue date:	08/23/1995
From:	Gwynn T NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV)
To:	Parrish J WASHINGTON PUBLIC POWER SUPPLY SYSTEM
References
GL-95-06, GL-95-6, NUDOCS 9508280218
Download: ML17291A980 (39)

Text

pzroe-r IY z REGUDATOiit80QLKFSTiDiT55M'BUTIOhVEM (RIDE)

CCESSION NBR:9508280218 DOC.DATE: 95/08/23 NOTARIZED:

NO

'DOCKET FACIL:50-397 WPPSS Nuclear Project, Unit 2, Washington Public Powe -'5000397 AUTH.NAME AUTHOR AFFILIATION GWYNN,T.P.

Region 4 (Post 820201)

RECIP.NAME RECIPIENT AFFILIATION PARRISH,J.V.

Washington Public Power Supply System

SUBJECT:

Discusses 950907 meeting w/util to discuss implementation of pilot exam development program,per GL 95-06.

DISTRIBUTION CODE:

IE42D COPIES RECEIVED:LTR ENCL SIZE:

TITLE: Operator Licensing Examination Reports F

NOTES:

RECIPIENT ID CODE/NAME PD4-2 PD COPIES LTTR ENCL 1

1 RECIPIENT ID CODE/NAME CLIFFORD,J COPIES LTTR ENCL 1

1 INTERNAL: ACRS NRR/DRCH/HHFB RGN4 FILE 01 2

2 P

E CENTER 1.

1 RR~DRCH/HOLB 1

1

~

1 1

1 1

EXTERNAL: NOAC 1

1 NRC PDR 1

1 VOTE TO ALL"RIDS" RECIPIENT'TSi PLEASE HELP US TO REDUCE CTASTE! CONTACTTHE DOCL';"iIEi1 CON'TROL DESK, ROO!iI P1.37 (EXT. 5N-eau

) TO ELI!illiiATEYOUR KAME PRO.'Nil DIS I'RIDU'I'IOYLIS1'S F'R DOCL'!iIEYTSYOL'Oi'1 iLED!

TOTAL NUMBER OF COPIES REQUIRED:

LTTR 10 ENCL 10

UNITED STATES CLEAR REGULATORY COMMISSI REGION IV 611 RYAN PLAZA DRIVE, SUITE 400 AR LINGTON, TEXAS 76011 8064 AUG 23 I.":"-.-"

Wa'shington Public Power Supply System ATTN:

J.

V. Parrish, Vice President Nuclear Operations 3000 George Washington Way P.O.

Box 968, MD 1023

Richland, Washington 99352

SUBJECT:

MEETING ON PILOT EXAMINATION DEVELOPMENT PROGRAM The purpose of'his letter, is to inform you that we will be hosting a meeting on Thursday, September 7.

1995, from 1:00 p.m. until 4:00 p.m., in the Region IV offices.

We specifically invite representatives from your staff involved in training and licensing of operators, as well as other interested parties.

This working level meeting will discuss implementation of the pilot examination development program.

We have included, as Attachment 1.

a copy of the preliminary staff guidance and modifications to NUREG-1021, "Examiner Standards."

Also included, as Attachment 2, is a copy of Gt.neric Letter 95-06.

We would be happy to accept any written questions in advapce by mail or facsimile, directed as described below.

The meeting will be held in the Region IV Training Conference Room on the 4th floor of our building.

Because of the size of this facility, we anticipate no attendance limits for interested personnel on your staff; however, we ask that you notify Ms. Laura Hurley, Operator Licensing Assistant, at (817)860-8253 by September 1

~ with an attendance list so that appropriate room arrangements can be made.

We have discussed this meeting with members of your training staff.

Due to the subject matter and nature, this meeting is open to attendance by members of the general public.

If you or your staff have any suggested topics or questions, please contact John Pellet. Chief. Operations Branch, at (817)860-8159 voice or (817)860-8212 facsimile.

Sine 1

Docket:

50-397 License:

NPF-21 o

a Gwynn, Dir c r

Division of Rea a ety 95082802i8 950823 PDR ADQCK 05000397 V

PDR

Washington Public Power Supply System Attachments:

1.

ES-201 Pilot Guidance 2.

NRC Generic Letter 95-06 cc w/attachments:

Washington Public Power Supply System ATTN:

J.

H. Swai les, WNP-2 Plant Manager P.O.

Box 968, MD 927M

Richland, Washington 99352-0968 Washington Public Power Supply System ATTN:

Chief Counsel 3000 George Washington Way P.O.

Box 968, HD 396

Richland, Washington 99352-0968 Energy Facility Site Evaluation Council ATTN:

Frederick S. Adair, Chai rman P.O.

Box 43172 Olympian Washington 98504-3172 Washington Public Power Supply System ATTN:

D. A. Swank.

WNP-2 Licensing Manager P.O.

Box 968 (Mail Drop PE20)

Richland, Washington 99352-0968 Washington Public Power Supply System ATTN:

P.

R. Bemis, Director Regulatory and Industry Affairs P.O.

Box 968 (Hail Drop PE20)

Richland.

Washington 99352-0968 Benton County Board of Commissioners ATTN:

Chairman P.O.

Box 190 Prosser

~ Washington 99350-0190 Winston 8 Strawn ATTN:

H.

H. Phi lips, Esq.

1400 L Street, N.W.

Washington, D.C.

20005-3502 Washington Public Power Supply System ATTN:

Jack Baker Nuclear Training Manager 3000 George Washington Way P. 0.

Box 968, HD 1028 Richland'A 99352-0968

Washington Public Power Supply System bcc to DMB (IE42) bcc distrib.

by RIV:

L. J. Callan DRSS-FIPB Branch Chief (DRP/E, WCFO)

Senior Project Inspector (DRP/ED WCFO)

Leah Tremper (OC/LFDCB, MS:

TWFN 9E10)

S, Richards (NRR/KOLB)

Resident Inspector MIS System RIV'ile Branch Chief (DRP/TSS)

M.

Hammond (PAO, WCFO)

DOCUMENT NAME:

0:'~OSNPILOT.LTR To receive copy of document, 'indicate in box: "C" ~ Copy without enclosures "E" ~ Copy with enclosures "N" ~ No copy RE:OB:DRS JTa ia 08

" /95 C:OB:DRS C

JLPelle 08 95 D:DRS TPGwynn

. 08

/95 AL

Washington Public Power Supply System bcc to DMB ( IE42) bcc distrib.

by RIV:

L. J. Callan DRSS-F IPB Branch Chief (DRP/E.

WCFO)

Senior Project Inspector (DRP/E, WCFO)

Leah Tremper (OC/LFDCB, MS:

TWFN 9E10)

S. Richards (NRR/HOLB)

Resident Inspector MIS System RIV File Branch Chief (DRP/TSS)

M.

Hammond (PAO, WCFO)

DOCUMENT NAME:

0:NOSKPILOT.LTR To receive copy of document, indicate In box: "C" a Copy without enclosures "E"

Copy with enclosures "N" No copy RE:OB:DRS C:OB:DRS C

D:DRS JTa ia 08

/95 JLPelle 08 95 TPGwynn 08

/95 A

ATTACHHENT 1 ES-201 PILOT GUIDANCE

The facility license snould oesignate a point of c tno NRC cnief examiner ana assian aaditi (subjec to the securitv guidelines on Interim Atta n c ment 5) to ensure

. administered.

and graaea e applicable Examiner Standaras.

The point of contact or another authorized aci lit re indepenoently review and aoorove the oro os th rooosed wr'itt n exami ations a

r~t~~g tests before they ar subject o the sec rity guidelines on Int the authority to speak for the faci lit "IIITE~IM Sample Corpora-e fJoti ication Let e."

the following examination arrang=-r<<ants:

~

the guidelines for ensuring examination

'nation security (refer to ante. im

~

the need to have the examination outlines and a co

=

~

e

.raining involvement) delivered G

ay

~or=

~he scneaure" examination

<<st 0

~

tie Jic lines or'ev icolrc. acministe. in an i ations

(=~-'01 c.i ery),

deverooing and administering the operatin o

i a

e and the uideiin e operating t sts (ES-301 ana ES-302.

~

the need to have the finar examination(s) and the reference materials 'identified in ES-201 Attachment

2. aerivered to examination date (only trose

. eferences that are actu least days before the schedule'o preoare for the examination snail be re uested is encouraged:

hard cooie~ snculd normall y e

ted to suppo t

~

the reauirements (refer to '.0 CFR 55.31) ana ui e

At<acnment

1) for submit-in 1

ng -he icense applications.

The attached Form ES-ZOl-l.

"ItJTERIM Examina i c~

xamination Preparation Checklist-rac t e activities leading up to the examination.

The attached form.

"INTERIM Examinatition Outline Quality Assurance p

p exa nation outlines.

s a

e used to review the ro ose aopropriate fe dback to the fa 1 t 1

potential for significant proolems with the final examinati outlines are significantly deficient r guidance.

i nt refer to Item 8 for additional ROI 95-25 ATTACHMENT 1

0

Tne cnief examiner will work with the designated facility contact.

as necessary.

to ensure tnat the final examinat ons are developed in accordance with the applicable Examiner Standards and the previously approved.

faci 1 ity-developed examination outlines.

The chief examiner will review the proposed written examinations and operating tests for quality in accordance with the applicable INTERIM Quality Assurance Checklists (refer to ES-30l and ES-401 pilot guidance).

It is esoecially important that the exams and tests be reviewed promDtly because of the extra time that may be required if extensive changes are necessary.

The QA reviews should be completed within two weeks after the examinations and tests are receivea.

The chief examiner will note any changes that need to be made and forward the examinations and tests to the resoonsible supervisor (or a designated actor otner than the chief examiner) for review and comment before reviewing the consolidated comments with the facility licensee.

The supervisory review is not intended to be another technical review.

but rather a cneck to ensure that all the applicable administrative requirements have Deen implemented.

Tnere are no minimum or maximum limits on the number or scooe of changes the NRC may direct the facility licensee to make orovic d they are necessary to make the xamina-ion conform with estabi'si

=-= acc ptanc cr;-.eria.

he ch

ef examiner mav ciscuss the NRC's conce. ns and changes wi:n -.he fac;iity representati'~e(s)

(normally Dersonnei invoivea with tne aevelcpment oi tne Drcposed examinations) via telepnone.

in the regional office. or at the facility aepending on the extent of the changes and as approved by the resoonsible regional suoervisor.

The facility review snail be conauctea in accordance with the guidelines and instructions in Attachment 4 (less Item 7) about 2 weeks before the week in which the examinations are scnecuied to be given, If the reviews are conducted over the phone.

the written exam and oDerating test changes may be communicated separately so that the facility may begin making the requi rea revisions.

If the facility reviewers have significant disagreements with the changes directed by the chief examiner.

the chief examiner wi 11 inform the resoonsibie regional suoervisor so that the disagreements can be resolved before the examinations are aaministered.

If the faci lity-preoared examination outlines or final examinations are so deficient that tney cannot be corrected before the scheduled examination date or cause the 'chief examiner to question the adequacy of; the facility licensee's training program.

the chief examiner shall discuss the problems with the responsible supervisor to determine the appropriate course of action.

Regional management should consult HQLB as appropriate and make a decision whether to proceed with the faci lity-developed examinations or develop the examinations in-house.

If the region does not nave tne resources to ensure that acceptable ROI 95-2""

ATTACHMENT I

examinations are preDared 5y the scneduied aaministration date.

regiona!

management snail negotiat with the faci!iiy licensee io rescneduie the examinations as necessary.

Normally. the facility licensee will make the examination and test changes.

and the cnief examiner shall verify that the changes were entered as di rectea.

The final examinations

'and tests and a copy of the origina! submitta) snail be routea to the resoonsible supervisor for final aporoval regardless who enters the exam>nation changes.

As a general rule. the written examinations should be scheduled before the operating tests.

however. other sequences are permissible if agreed to by the facility licenseo.

Normally, the written exam>nations shou!d be administered no more than one week before the operating tests.

Una r extenuating c'.rcumstanc s

and with prior approval from HOLH.

written 'examinations may oe g)ven up to 30 days before the ooerating tests.

as soon as the license applications are accepteo.

any apD!icable

'aiver requests are

. eso<vea.

ana the examinations are approvea.

nlhen reviewing the;;:ritten examination administration guidelines with t e faci!ity lice..see (per

.tern i3 on interim,"orm.ES-2 Oi-'). the region snould c"nf1r i the asap!leaf'l tus on -.he examination assignment snee so tnat :ne faci lit, '~ii'. inow wno srouic be acmn ste.

o - wr~-.= n exam.

ATTACHMEHT 1

ES-201

~lh

<Adaress)

(City

State, 7i coQe Gear ~Name; I

N T E R I M

Sample Corporate Notification Letter HRC Letternead Attachment 1

date tldh I

~dt.. Il.

M.IN.~tl d

M.IN.~tel..

f th '>>

I f

I II

~t'--

I t d

I th tt I.

fd Your staff has agreed io dharti ipate in a voluntary pilot examination program in which your staff will orepare the written examinations and operating tests ana submi.i them to the NRC regional office for evaluation and approval.

Your sta will prepare the proposed examinations in accorcance with the guidelines in Revision 7.

Supplemen-. '. of NUREl"-1021.

"Operator Licensing Examiner S:araaras."

R vision " of.VURECiBR-0122. "Examiners'anobook for Develooing

"'Derazor Licensing rewritten Examinations.

and tne at-.acnmenz io this letter.

he *tVRC regional of>ice w il disc: ss wiTh your stafi any examination cnances at might be necessary prior to -;heir aaministration.

To meet the above sch du:e. it will be necessary for your staff to furnish the orooosed examination outiines by ~(date The proposed written examinations.

operating tes=s.

and the supporting rererence materiais will be due by (date).

Any delay in receivina the required rererence and exam>nation materiais or rhe suomittal of inaaequate or incomplete materia'ls may result in the examinations Deing rescheduled.

In order to conduct

.he re ueszed written examinations and operating tests. it will be necessarv for your staff to provide adequate space and accommodations in accordance with ES-402 and to make the simulation facility available on the dates noted above.

ES-~02.

Attacnment l. and =S-30Z. Aitachmeni

1. contain a number of NRC policies and guidelines that wi 11 be in effect while the written examinations and operating tests are being administered.

In accoraance with ES-302.

your slaff should retain the original simulator performance data (e.g.,

system pressures.

temperatures, and levels) generated during the dynamic operating tests until the examination results are final.

Your staff should submit, preliminary reactor operator and senior reactor operator license applications and waiver requests at least 30 days before the first examination da-.

so ':hat the NRC wi 11 be able to review the applications and the medical certificai'ons and evaluate any requested waivers.

If the applications are not received ai,least 30 days before the examination date.

a RO! 95-25 ATTACHMENT 1

ES->Ol Attachment i

postponement may be necessary.

Signea applications certifying that all training has been completed should be suomitted at least 14 days oefore the first examination date.

This reouest is covered by Of ice oa Management and Budget (OHB) Clearance Number 3150-Ol01.

which expires "pri 1 30.

199?.

The estimated average burden is 7.7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> per response.

including gaihering.

copying and mailing the required material.

Send comments regarding this burden estimate or any other asoect of this collection of information. including suggestions for reducing this ouraen.

to the information and Recoras Management Branch.

Hai 1 Stop T-o F33. Office of information Resources Management.

U.S.

Nuclear Regulatory Commission.

Mashington.

D.C. 20555:

ana to tne Paperwork Reduction Projec~

(3156-0101)

. Office oi Infolmation anQ Regul ai:ory Affairs. NEOB-10202, Office of Management and Budget.

Mashington.

D.C.

20503.

ihank you ror your coopers.ion in this natter

~Mama!

has policies and guidelines ref rene d in this letter.

if you regarcing tne NRC's examination ". ocedvre~

-no guidelines.

(Name rf <ere;Qnal "cnt"c

- (teie~~one

~i~ oerl or (Na;.,z been advised of the have an'y auestions please contact s:~on< 1 g ) sa r'rare i pr!

t: nor

~ q Zr ~ '-. '-= rC~nr nC r ~-~"r

~

~

C>>

Sincereiy.

Attacnient:

Pilot "xamination Guidelines (Appropriate regional representative)

Docke No.: 50-(Number)

Distrioution:

Public NRC Document Control System Regional Distribution ROi 95-25 ATTACHMENT 1

P. LOT E XAMINATjON GU ( DE> rNES.

Facility licensees will prepare the written examinations and operating tests (dynamic simulator and walkthrough) in accordance with the instructions in Revision

7. Supplement.l.

of NUREG-1021 ("Operator Licensing Examiner Stanaards")

and Revision 5 of NUREGiBR-0122 ("Examiners'andbook for Developing Operator Lice~sing Mritten Examinations" ).

sub 'ect t th f 11 additional criteria:

jec o

e o

owing (1)

The facility license wi1 1 prepare an integrated examination outline (written and operating test) and submit it to the NRC regional office for review.

comment.

and approval at least 60 days before the scheduled examination date.

The NRC chief examiner will work with the facility licensee to resolve any problems and to avoid unnecessary revision of the final examination products.

I 2

<2) racility employees:;no are involved with developing the examinations ana t sts will sign a stanaara security agreement (Form ES-201-2) before they gain specific xnowieage of the exam~nations and test F

1't p oye s who Diayed a suostantial role in training the license applicants will gen rally not be involved in developing the license examinations or tests.

if the faci lity licensee consiciers sucn emoioyees necessary

-or ceveioping the examinations or tests. it will aefine tne proc ss :: will taKe to ensul e that the inTegrit f'

amirat;ons is no=

=".-.,Drcmlsea ana discuss

-. ne process th th NRC-(4)

Ine written examina:.ons ana operating tests will satisf th

=

11 specific criteria:

a s y e ro lowing A maximum of 50 percent of the written examination questions may be taken directly -rom the facility question bank:

up to an add't' pe c nt or

~he questions may be taken from the facility bank but must be significantly modified:

and a minimum of 10 percent of the questions will be newly developed.

As discussed in NUREGlBR-0122.

new questions snould emonasize the applicants'nderstanding at the comprenension or application levels of knowledge because they have the greatest ope. ationai and discriminatory validity.

re o more than 25 Dercent of the questions on the examinat b

peated from examinations.

quizzes.

or tests administered to the ion may e

license applicants during their license training class.

or from the past two NRC license examinations at the f'acility.

No questions may be drawn directly from the applicants'udit examination or similar testing vehicle given at the end of the license training class.

Each walkthrough test wi"ll include at least two job performance measures (JPMs) tnat are either'ew or significantly altered and ROI 95-25 ATIACHMENT x

eacn simulator.scenario sez will incluGe az least one new or significantly alzereG scenario.

Otner scenarios used may be drawn directly from the facility scenar;o bank:

however.

they will be altered to the degreo neeGeG to prevent zne applicants from immediately recognizing the scenarios based on initial condition other cues.

lons ol

~

A significant modification. for purposes of the written questions.

-means a change to zhe conditions. in the stem and at least one diszraczor significantly changed.

Similarly. JPMs and simulator scenarios will have az least one substantive event or conaition change that alters the course of action in the JPM or scenario.

The facility licensee -will suDmit the final written examinations and operating tests to the HRC regional office at least 30 days before the scheouled examination date.

in izs suomittal.

the faci lity lice..see will include a history (e.g..

bank.

revised, new, and daze last used) of eacn zest item used on the written and operating zests.

Th>

'RC The

<,RC regional office will assign a chief examiner to coordinate the review. revision (as deze.minea necessary by tne NRC).

and valioazion of the written examinations anc cperazing tests with:he facility licensee,

'dG 1cnal NRC staff exam<inc. s wi 1 1 be assignee as necessarv

(""

1" cr l'.wo. de nQ'ng on re numDer oT aDDlican:s) to as 1

=xam<lner wit<

aG <iniszel 'nafiQ gl aG1n lie oDe. azing

<eszs 1ll accoroanc

iih exist ng "rcceaures.

lne aci.izy licensee wi 11 aGmiriszer anG grade zhe written examirazions:

.",RC examiners will review ard approve zne licensee's graGing.

RO! 9"-25 A&iACHMENT 1

~

~

I I ~

~

~

~

~

I-

~

~

I

~

~

~

~

~

~

~

I

~ I ~

~

~

~ II

~

~

~

~

~ IO

~

t I

~

I

~

I

~

~

I

~ ~ I

~

~

~

I

~

~ ~ ~

t

~

~

~

~

I III 55 mm

~ I

~

~

I I

I ~

I

~

~

~

~

~

~ II

~

I

~

I

~

~

~

~

~

~

~ I

~

~

~ ~ I

~

~

~

~ It

~

t II

~

I

~

~

\\ I ~

~

~

~

~

t ~ ~

~

~

~

~

~ ~

~

~

I I

~

I

~

I t I ~

t I

~

~

~ ~

~

~

I

FS-201 I

N T E R I H ExamirraLi<)r) Sec<<r il.y Agreemer)L Form FS-201-2 I'r(.-exa)))i>><)Linr)

I ackr)nwledge that I have ac(I>>ired specialized knowle<lge al)o()L the NRC licensing examinat.ions scheduled for Lhe week(s) of as nf the date of my si<lr)aL(rr'e arrd agree

that, I will not. k>>owir)gly div<<lge any infnrmat.inn aborrt Lhese examir)aLior)s to ar)y per's<)r)s wlro l)ave r)ot: been authorized by the NRC chief exarr)incr'.

I understar)d LhaL I

am not Lo participate ir) ar)y ir)stnrcLinr) i>>volvir)g tl)ose appl)car)ts scheduled to be admi))istere(l these licer)sing examir)atior)s fr'<)))) this date unLi I completinn of examination

<)(Ill)i))ist.r at in>>.

I f(rrther (rt)(l(.'r'SLar)d Ll)aL viol at in>> of Ll)e <:nr)(li Lior)s of I I)is agreement may resulL in

<',<)>>collat.ior) of the exarrrir)aLI<)r)s <lr)<llor ar) er)forcem<.r)t. act.in>> <)gainst me or'the facility licer)see.

2.

I'ns t.- ex()rr>> r)<}L)n)1 I did r)oL. to I.he best of my k>>owle(lge. divulge ar)y ir)fnr'rr)rt.inr) cnr)cernir)q the NltC licensing examinations a(I)))ir)istered dr)r'ing tf)e week(s)

()f I() ar)y rrr)<)<<tl)<)r'ized I)er'sor)s.

I di(l i)ot I)articipat<.

it>

I>>sL)'(rctir)<g Lhose applicar)Ls who were a(hnir)istet'<'.<I Lhese li<:(.'>>sir)<g examir)aLior)s from the date that I entered ir)to this security agreement ur)ti1 Lhe coinpletior) of examinaLion administration.

PRINTEO NAME JOB TITI.E 1 RESI'ONSI 0ILIly S IGNAIIIRl'. (I )

DAIE SIGNA'IIIRE (2) l.

2..

3.

5 6.

7.

0.

9.

10.

11.

12.

]3 15.

. 1)AlE Ia)l

<)<) 2!)

AI I'ACIIHENl'

~

~

~,:

~

~

~

~

~

~

~ 0

~

~ ~

~

~

~',

~

\\

~

~ ~

~ ~

~;

~

~ ~

~

~

0 0

~

~

~

M5

~

~

~

~

~

~

~

~

~

~

~ ~

~

~

~

~

~

~

'I

~

~

~

~

~

~

~

~

~ ~

~

~

~

~

~

~,

~ t,

~

~

~

~

~ ~

~ ~

~

~

~

t

~

~

~

~

~

~

~

~ ~

~

~ ~

~

~

~

~

~

~

~

~ ~

~,

~

~

~ ~

~

~ \\

~

~

~

~

~

~

~

~

~

~

~

~,

~ I 5

~

~

~

~

~

~

~

~

~ ',

~

~

~

~

~

~

~ ~

~ t',

~

~

~

~

~

~

~

~

~

~

~

~

~

~',

~

~

~

~

~

~

+

~

~

~

~

~

~

~

~

HS

~M r%8 L

SrR rER

$5L e

~

~ ~

~

~

'I

~

ES-201 I

N T E. R I M

Examination Securitv Guidelines Attachment 5

In accordance with 10 CFR 55.49. facility licensees (as well as applicants and Iicense s) shall not engage in any activity that compromises the integrity of any examination required Dy Part :5.

NRC examiners and faci liiy licensees

'rIust be attentive to examination security measures to ensure compliance with the regulation.

The following guidelines covering personnel restrictions.

pnysicai security.

and limitations on the use of examination banks shall be reviewed with the facility license at the time ihe examination arrangements are confirmed.

Personnel Restrictions Facility employees wno Diayea a substantial role in training the license applicants will gene. ally not-be involved in developing the license examinations or tests.

If ~ne facility licensee considers sucn emoIoyees necessarv to deveIoD the examinations or tests. it shall define the orocess it will tave to ensure that exam integrity is not compromised and disc's th process with the NRC chief examiner.

The region snail keep HOLY informea of any comDensatory measures tnat mignt be nec ssary.

c="

a i Iea iicensee SuDmlt a

1 ic="nse= s!la I:

. nliiIlze tri nuiilDel oi personineI wIlo naYe kncwiIe ce

-. :I:e

.".=,

.ice,.s ng examinat on..'e iaciIity oeiIieve==

-..".=-: ;ore :nan 5 peoDIe are necessarY lt should Justifica l

. to ne regional office for approval.

All personnel who re eive detailed knowledge of any portion of the NRC license examination.

including the examination outline.

must acknowledge tneir responsibilities by signing a form. such as the attached "INTERIM Examination Security Agreement." at the time they obtain detailed knowledge and again airier the examinations are compIete.

The facility licensee will provide

=- copy of the form (listing the expected signatories) to the criei examiner at the time the examination arrangements are conf~rmed.

The facility licenseo shall inform the chief examiner if add',tionaI oersonnei need to be added.

The original'orms must be submitteQ to the Ieglonai office after the examinations are comDIete.

"-'sicai Security The NRC expects that :he facility licensee will exercise the same physical security pr cautions with the initial examinations as it does with its requalification exam~nations.

If the facility licensee has a

procedure in place. it is exo cted to be implemenred.

All examination-soecific materials (i.e.. the prooosed examination outlines and iinai examinations) snail-be controlled and protected as sensitive informatior and snail not be transmitted via non-secure eIectronic means.

RC'5-25 ATTACHMENT i

ES-201 Aiiachmeni 5 The proposed examination outlines., wriiien examinations.

ana operating tests inat are mailea io tne regional office snail be @laced in a double envelope.

The inner envelooe snail be conspicuously mar<ed "FOR OFFiCIAL USE ONLY" and "TO BE OPENED BY ADDRESSFE ONLY."

Furthermore.

the cover leiier forwaraing -.ne examination materials should reauesz thar. the materials'be wiinheid from public disclosure unzi 1 afie.

the examinations are complete.

The outlines and proposed examinations may be transmitted via the NRC's AUTOS neiwork.

3.

The aci lity licensee and :he cnief examiner wi 1 1 review the "So'curiiy Consideraiions for Simulator Operating Fxaminations" io ensure that the instructor station features.

programmers'ools.

and ex'al interconnections do noi comoromise examination integrity.

The primary objective is to ensure that the exam material cannot be reaa or recoraea at.oiner unsecured consoles and that examination mat rial is eiiher pnysic-liy securea or elec-.ronica lly protected wnen not ;n use by indlvlouals on the securi.v agreement.

'n NRC'examiner may insoec-.

-.ne faciliiy licensee's aciminisiration of tne writien examinations.

The NRC will also review ine results of the examination zo determine if tnere is any indication of compromise.

The faci liiy licensee and the NRC should determine if examination securiiy problems were noted in the pasi and ensure that corrective actions have Deen taker. io pre lude recurrence.

Examination Bank Limitations The faci liiy licensee ana chief examiner shall ensure that written examinations ana ooerating ie is conform with the guidelines in ES-301 and ES-401 (as modifiea h rein) regarding the use of bank items directly from the bank. moaified it ms.

and new items.

The guidance here.n suoerseaes conflict,ing cuiaance in NUREG-1021: for example, the 10~

limit on facility banE use (ES-301.

Paragraph D. l.b.

ana ES-401.

Paragraph C.l.c) would noi apply during the pilot examinaiions.

2.

if the facility license has an open bank. it will not place any new or modified items zo De used on the examination (wriiien questions.

job perrormance measures.

or simulator scenarios) in their examination bank unzi'1 after the las-examination has been administered.

RQI c5 12 ATTACHMENT 1

SECURITY CONSIDERATIONS FOR SIHULATOR OPERATING EXAHINATIQNS Simulators present a uniaue

=-=-

ue==. of integrity concerns for operating ions.

e.xaminers snculd ensure that operating examihation security is in re areas.

the ns".ructor station, the programmers'ools.

ana ct ons.

This note lists representative features that o e su e that the oianned exam nation is not y

e i e ina a!.ring pre-examination activities.

eatures can be checked througn the tableau or Most of the instructor station f ~

at the instructor's conso!e.

The 0rogrammers' e ext mal int

, connections are noi generally apparent to the er.

The simu!ator -swaff snould be consulted to e

s atus o

in~se latter items.

iNSTRUCTOR STATION FEATURES 0

TREND RECORDING

- Most simulators have ors ave the ability to monitor and aJ'aoh 11o 1o' i bla e

eoe tea by the

-o y

vari e:o be assianed io the trending feature.

..ke tren data can be spooled to a file for later printing.

0 SNAPSHOTS

- All simu!ators have 0

at ve snaosnoi capability.

Initial conditions 0

recor e

d'or future recall.

a are snapsnots hat al" autcmaticallv BACKTRACK - Back:rac<,iie a'l e-ae ermlnea in 4~

e-i t

. va!

. usua!ly up to" 2 nour oi oo ion 0 ck l acK iles r

usua on.

"EPLAY/PLAYB'CK - The

,"= lav/

c

" o

=p v oiayback reature steps throuah a series of vs tne i/0 status

( lignts, meters, etc.) for each ia y.

ten.

tne replay feature uses the backtrack files.

separate reoIay file storage may be proviaed.

o SCRIPTS/COMPUTER ASS that 1

c

-'SIKD EXERCISES

- Many simulators have a feature funct I

orbs Dr pro rammed implementation of malfunctions and remote-conditions.

Scripts may be used o!

Sta to faci 1 iiate scenario administration.

Scripts a

ica y

e storea for future use. 'tored scripts can also be ected Yor review ana editing from the instructor station.

c INITIAL CONDITIONS

SUMMARY

n

- Snapshots are usually labeled on the h

enu wiin date/time recoraed.

oer tinent ana instructor comments.

Even if the comment field cnangoa to inaicate that a snapshot is available for re-use.

a a

scenario initialization) may still be reoresentative of test

'un i e snapsnot is actually overwritten or updated.

a unction summary menus disolay the status of MALFUNCTION

SUMMARY

- Malfun e

ma unctions.

both active and inactive.

The malfunction y

a oenaent and therefore d~p~cts the maif~~ct~~~s

~ w re active or stagea wnen an IC, such as a scenario validation, Roi g~-""

ATTACHMENT "

0 MONITORED PARAMETERS - instructors are aiforaed the capabilitv to define inaividuai or grouos of parameters for disoiay or orintout.

The monitorea Darameter grouo assignments can be recalled for review and editing. if used to facilitate scenario validation or exam~nation aaministration.

the monitored parameters can provide insight into the focus of the examination.

0 VIDEO 8 AUDIO RECORDING Many simulators are equipped with video ana audio recording capability in the control room.

'Video and audio controls are typically located at tne instructor station.

PPOGRAHHERS'OOLS o

SOFY~ARE TERMINALS - Simulator engineers have access to real-time monitoring ana control oT simulator dna moael conditions :hrougn software suooort terminals.

These -erminais may be located in the co,"purer iaci >ity or at tno enginee~'s oesk.

0

'ND:"PENDENT EXECUTiVES - The conoitions for scenarios can sometimes be replicated off-line using inaepencient executive programs.

These programs snould nor. be in communication with the IlO.

Independent executives and their associated initialization files may provide an inaication of planned exercises if they have been used to resolve problems during scenario validation.

0 GRAPHICAL USER INTERFACES

- Instructor station graphical user interfaces often display simulated Diam conditions and performance in real-time.

At remote locations.

sucn as a programmer's

desk, the GUI could display the full scenario.

EXTERNAL INTERCONNECTIONS 0

ESF FEEDS

- Many simulators have data links to the ESF and the operations management offices for emergency planning drills.

These links can display simulated plant condition to observers outside the simulated conr.roi room during scenario validation or examinations.

REMOTE PLANT PROCESS COMPUTER 8

iNSTRUCTOR STATION SCREENS

- Repeater screens in the training area can display scenarios in real time'o observers outside the simulated control room.

o MODEMS 8

REMOTE SIMULATOR SUPPORT SYSTEMS

- Many simulators are equipped with modems from the instructor station or simulation comouters for outside monitoring and control of simulaior status and activities by par.ies off site.

0 0

'REND RECORDING - Groups of parameters can be defined and assigned to trend recoraers.

The recorders may be. bui is not necessarily.

locatea at the instructor station.

The recoraing may also be in file format for presentation on instructor station screens.

Recording-sessions are typically activated or de-activatea at the instructor station.

o STUDENT PERFORMANCE MONITORING - Special groups of parameters and simulated plant operating conditions can often be assigned to a tracking ana recording function tnat plots an individual student's oerformance during training exercises.

Recording sessions are typica11y activated or de-activated at the instructor station.

Roi 95-25 ATTACHMENT 1

ES-301 PILOT GUIDANCE 1.

The facility license will preoare=the prooosed ooerating iesi outlines (i.e..

Forms ES-301-i.

2.

and 3) in accoraance with Section D and submit them to the NRC regional office with the writien exam outlines for review'and approval 60 aays before the scheduled examination date.

Sufficient ooerating test outlines shall be developed io ensure that all the applicants can oe tesied with the personnel available on the schedule agreed upon by the NRC regional office and the facility licensee.

Day-to-day overlap or duolication between walk-through tests ana dynamic simulator scenarios shall be limited or precautions shall be iaken io ensure that test integrity is noi, compromised.

Each walk-through test shall include at least iwo (one for uparade SROs) job performance measures (JPMs) that, are either new or sianificantly altered and each simulator scenario set shall include at Yeast one new or significantly alterea scenario.

Other scenarios used mav be drawn directly from tne facility's scenario bank.

however they will be altered to the degree needea to prevent the apo1icants from recognizina the scenarios early basea on initial conditions or other cues.

A significant modification. for purooses of the JPMs and simulator scenarios means at lcasi one substaniive event or condiiion cnange that alters the course of action in the JPM or scenario.

The facility lic ~se should evaluat the dominant accident seauences (DAS) for tne faciii-.'o cetermine if they are suii-ble fot t st'ing on the simulator or aur;ng the uaik-inrough.

DAS -re ihose seauonces here th freauency of core carnage is greatest as ae rmined by the faci iity licensee s probaoilis ic risK assessment (PRA) or inaiviaual p~ant examinaiion

( IPE).

Each scenario set should expose the applicants to situations in which their performance could cause plant degradation or threaten the health and safety of the ouolic.

The scenarios should ensure that all the rating factors within each competency can be evaluated and should incluae events thar. would require unsatisfactory rating factor evaluations if an applicant performs poorly.

The administrative tooics should be evaluated in a oerformance-based (i.e.. using JPMs) moae whenever possible.

rather than by asking prescripted auestions.

The region will review. revise as necessary.

and approve the facility-prepared operaiing tesi outline(s) in conjunction with the written examination using the for'm provided (refer to ES-201 pilot guidance).

The facility licensee ui 11 prepare the operating tests in accordance with the previously aoproved examination outline(s) and the instructions in Section D.

An authorized facility representative should review and approve the tests before submitting them to the NRC regional office for review and aporoval 30 days before the scheduled examination date.

If it is necessary to deviate from the previously approved operating test outlines.

the facility should explain each deviation.

The explanation should include a reason why the original proposal could not, be implemented and a just-'fication why the proposed replacement is considerea an acceptaole suostitute.

ROI 95-25 ATTACHMENT 1

The facility licensee shail;densify wnich test items are new. which ones are taken directly from the facility licensee's item banks, the origin of items that were modified from existing bank items.

and the date on which eacn item was last used on a licensing or practice examination or qu>z.

The region will revie~,

work with the facility licensee to revise as necessary.

and approve the final operating tests in accordance with the "INTERIM Operating Test QA Sheet."

and the instructions in the ES-201 pilot guidance.

ROI 9:"-2~

16 ATTACHMENT 1

~

~

~ ~

~

~

0

~

~

~ ~

~

~

~

~

~

~

~

~ ~,

~ ~

~

~

~

~

~

~

~

~

~ t ~

~

~

~

~

~

~

4

~

~ ~

~

~

~

~

~

j

~

~

~

~

~ ~

~

~

~ ~

~

~

~ ~

~

~

~

~ ~ ~

~

~

~

~

0

~

~

\\

~

~

~

~

~ ~ ~

~

~ ~

P

~

~

~

~ '.

~

~ ~

~

~

~

~

~

~

~

~

~

~

~

~

~

~ ~

~

~

mar

~IS

~RS

~

~

~

~

~ ~

~ ~

~

\\

4 ~

~

~

~

~

~

~

~

+

~

~

~

~

~,

~

~

~

~

~

~

~ai

~

~

~ I

~

ES-3Q2

>ISO<<".UIOANCE The chief examiner should confirm with the facility licensee that the simulator instructor's station.

programmers'ools.

and external interconnections do not compromise operating test security while conaucting the examinations.

The primary objective is to ensure that the exam material cannot be read or recorded at other unsecured consoles and that examination material is either physically secured or agreement.

electronically protected when not in use by individuals on "he t

. e securi y ROI 95-25 18 ATTACHMENT 1

The facility license will prepare the proposed written examination outlines in accordance witn the ES ana the Examiners'andbook and suomit them to the WRC regional of ice with the operating test outlines for review and approval 60 days before the scneduled examinaiion date.

Facility licensees are encouraged to use the appropriate K/A record form(s) in the Examiners'anabook (or a facsimile) to faci litate the quality assurance reviews.

2.

The region will review. work with the facility licensee to revise as necessary.

and approve the facility-prepared examination outl,ine in conjunction with the operating. test outline(s) using the form provided (refer to ES-201 pi lot guidance).

Tiie facility licensee will prepare the examination in.accordance with the previously approved outline ano tho instructions in the ES.

An authorized facility representative snould review and approv'e the exam before suomi:ting '; to the NRC regional office for review and approvai 30 days before the scheduled 'examination date.

If it is necessary to deviate from the Previously approvea exam outline.

the facility snould exolain acn deviat'.on.

Tne xoiar:=--ion should inc'ude a reason whv the original ore."osai could not be imp.emented ana a justification wny the prGDoseo reDlacemen is conslQere~

"n acceDtabie sUDstltute.

4 maximum o-50~ o- -.h. written examination questions may be taken directly -. cm their bank:

up to an additional

~0 of the questions may be taken from the Dank but must be significantly modified; and a minimum of 10'of -.he auestions will be ne>>iy developed.

As discussed in the Examiners'anabook.

the new quest cns snould emphasize the applicants'nderstanding at the comprehension or application levels of know)cage because they have -he greatest operational and discriminatory validity.

No more than 25K of the qu stions on the examination may be repeated from examinations.

Guizzos.

or tests administered to the license applicants during their license training class.

or from the past two NRC license examinations at the facility.

No questions may be drawn directly from the aopiicants'udit xamination or similar testing vehicle given at th~ end of the license training class.

The examinations shall be 100K mult'Die choice A significant modification. for pur"oses of the written questions.

means

-=.a. chang to the conditions in the stem and at least one distractor significantly chang a.

ROI o5 25 10 ATTACHMENT I

Tne facility licensee shall identify which zest items are new, which ones are taken direczly from zhe facility licensee's item banks (or another source such as the NRC EQB or an old NRC exam).

the origin of items that were modified from existing bank items.

and the date on which each item was last used on a licensing or practice examination.

The region will revie~. work with the facility licensee to revise as-necessary.

and approve the prooosed written exams in accordance with the "IWTERIH Written Examinazion Quality Assurance Checklist."

The facility licensee is primarily responsible for ensuring the technical accuracy of the license examinations and compliance with the question duplication and distribution guidelines (i.e..

Items

1. 4. 5.

and 6 on the QA Checklist).

However. the chief examiner is expected to use his or her best judgment and take reasonable measures to verify these items when reviewing the examination.

If the ouestions are not tecrnically accurate. it should be self-revealing during the grading Drocess if the facility licensee accepts double answers for items or recommends delezions.

RQI c5 2 20 ATTACHMENT 1

~

~

I

~

~

~

~

~

I

'I

~ ~ )

'I 'I

~

I-

~

~

~

~

~

~ ~

~ ~

~

~

~

~

~

~

~

~

immi&

~Rm 85 RRQ

~

~

~

~

~

e

~

~

0

~

4

~ ~

I

~.

'I

~ o

- ~

I;

~ ~

~

~

4

~ ~

~

~

~

~

I -

~

o

~

~ t ~

~

o

~

~

~ 4I ~ ~

~

~

a a

~

I ~ ~ 0

+

e

~ I

~

o

~

. ~

~

~ -

~

~ 0 ~

~

~

~

II

~ -

~

I

~

~

~

J ~

E

~

~

~

~

0

~

0

~

~ ~

ES-~02 PILOT CUIDANCE.

Tne faci lity licensee shall orovide the necessary copies of the approved examinations.

answer sheets.

and handouts (e.g.,

equation sheets.

selected tecnnical soecifications.

ana steam tables) for each applica'nt, as approved by the NRC chief examiner.

The facility licensee may use machine-gradable answer sheets if desired.

but it is NOT required.

The facility licensee will administer the examinations to the apol'icants identified on the.examination assignment sheet (Attachment 3 of ES-201) as arranged with the NRC chief examiner and in accordance with the specific instructions in 5ect;:on D.

As a general rule. the written examinations should be scheduled before tne operating tests.

however.

other sequences are permissible if agreed to by the facility 1;cense Normally. the written exam~nations snouia be aaministered no more than one week before the operating tests.

Under extenuating circ mstanc s and with prior aoproval from.HOLH.

wr tten examinations may be given up to 30 days before the operating t s;s.

as soon as the '!icense applications are accepted.

any apoiicabie waiver requests are resolvea.

and the xaminations are approved.

<-. :ne -,ac; lity lic nsee wii'e conducting the written examinations

nile -.ne NRC examine."s are c.;-site.

the chief examiner should insDec-t,e examination fac'1':;es t ensure tneir aaequacy and perioaicall mcnitor the exam to

e. sure the proctor is appropriately addressing the ape'iicants'uestions.

If th-Is is not feasible.

the region should consider having an examiner check the facilities upon arrival at the site for the operating tests cr via another method as determined appropriate by the resocnsible regional supervisor.

At least one individual uho is ami liar with the intent of the questions (i.e.,

a facility empiovee wno took part in the examination devetooment) snail be available io c arify examination questions for the applicants during the examination.

Tnis individual must be extremely careful not to lead the aoplicants or give away answers when clarify ng questions.

If he or she nas any doubt aoout now to respond tc an apD;icant's question. it is best zo withnold aaditiona1 guidance ana instruct the applicant to do his or her best with the inTormation tnat is provided.

All questions asked and statements of clarification must be documented verbatim for later review by the NRC chief examiner.

The proctor should cons-.ruct a "hart illustrating the seating arrangement of the aopiicants curing the examination.

The proctor should also record the stariin= time of the examination and the time at which each applicant co,",Dieters the exam.

Th proctor shall coile : the xaminations, exam cover,and answer sheeTs for graaing in accordance with =5-403: the examinations may be ROI 95-2" 22 ATTACHMENT 1

distributed to the applicants after the last examination is collected.

Nhen four hours have -elapsed.

the proctor shall instruct the remaining applicants to stop work. sign their examination cover sheets.

and turn in their examinations.

The proctor shall deliver the completed examination packages.

the marked-up master examinations.

and the list of applicant auestions to the appropriate facility representative for grading per ES-403.

ROI 95-25 23 ATTACHMENT 1

ES-~03 PILOT "UIOANCE 1.

The facility licensee will grade the written license examinations in accordance with the instructions in the ES.

In order not to bias the operating test evaluations.

the facility licensee should not provide the examination results to the NRC examiners until after the operating tests are complete.

The facility should not disseminate the examination grades to the license applicants until after the NRC reviews and approves the grading.

2.

The facility will evaluate all the questions posed by the applicants and the pen-and-ink changes made on the master examination while it was administered to determine if any of the questions should be deleted from the examination or any of the answers chanced.

The faci litv will document the reason for every change or de1etion.

The facility shall not change the examination unless there is a valid reference to support the change.

It is exoected that the facility licensee will analyze the examination for problems with the test items and possible training deficiencies that might indicate a need for remedial training or program changes.

Machine grading and computerized item analysis are acceptable.

but AOT required.

The chief examiner should re uest the facility licensee to prepare a

matrix su7marizing

-.he aopiic=-nrs'erformance on each exam'ination uestion (i.e..

answers Der cistractor) and forward it to the regional office with tne examinat on package.

'n authorized facility representative should review the grading and submit the answer and cover sneets.

applicant questions and clarifications. grading matrix. master exam changes and justifications to the NRC regional office within 5 working days after the exit meeting.

The chief examiner shall review the examination grading using the attached

" INTERiM Examination Grading Quality Assurance Checkoff Sheet."

The chief examiner should applv his or her judgment when reviewing the examination results and aajust the level of the review based on the applicants'nd the facility licensee's performance (e.g..

the number of items changed or deleted.

the average grade.

the number of marginal or fai ling grades.

etc.).

The chief examiner should also be alert for any indication that the examination was comoromised.

If 5X or more of the examination answers are changed or questions are deleted during the grading process.

the regional office should request the facility licensee to explain why so many post-examination changes were necessary and what actions will be taken to improve future license examinations.

The region should also consult the Operator Licensing Branch staff for further guidance.

If 10 or more of the questions are deleted.

the region should take action per the ES to verify the validity of the examination.

ROI 95-25 24 ATTACHMENT I

0

~

\\

~

I ~

~

~ ~ ~

~

~

~

~

~

~

~

~

~

~

~

~

~ I

~

I

~ ~

~

~

~ ~ ~

~

~

~

~

0

ES-501 P~LOT GUIDELINES The examination report should document any significant deficiencies in the original examinations submitted by the facility licensee and other problems (e.g., security concerns) encountered during the examination development, administration.

and grading process.

If necessary, the region should request the facility licensee to describe the actions it wil') take to improve future performance.

2.

The examiners involved with the pilot examinations shall document any significant questions or problems that they or the facility licensees may have regarding the new examination process:

answers and solutions should also be documented, if available.

As soon as possible after the examinations are complete.

the examiners involved shall respona to the attached questionnaire and return it to HOLB along with the questions from item 2.

HOLB will evaluate the feedback.

disseminate the information as necessary.

and incorporate lessons (earned in future examinations.

In addi-.ion to the ;:- ms identified in ES-501.

the region will place a

copy of -.ne propose" outlines and the original examinations submitted by the facility licensee.

including the NRC's comments and resolutions.

in the mas-. e.

examinaticn file.

The region wi 11 ensure that a copy of the facility licensee's submittals are placed in the PDR after the examinations are administered.

ROI 95-25 26 ATTACHMENT 1

PILOT OUESTiONNAlRE Please complete this questionnaire electronically and submit it to HOLB via e-mail (SAR. AJH.

and SXG) as soon as possible after the examinations are complete.

Facility Name:

Date of Examinations:

How did the overall quality of the examinations compare with recently administered NRC-developed examinations?

Are the following QA checklists adequate?

Are there any items that should be ad'ded or deleted?

Examination outline.

Operating test.

Should it replace the competency and event checkiists in Rev.. 8?

r.'riiten xam.

.r;I i7. en exam gl aG)ng.

aha~ leve. of facility management should QA the materials before they're sent :o the NRC?

2.

Mow much iime did it I.ake to perform the following activities?

Outline review:

Operating test review:

';,'ritten exam review:

Other (not chief) examine," preparation (how many):

Administer the operating tests (including travel time):

Review the written exam grading:

Document and grade the operating tests:

Prepare the report:

Fill out this questionnaire:

ROI 95-25 ATTACHHENT 2

3.

Mas a preparation trip conducted for this examination?

Explain why.

'How many examiners participazed in the preparation trip?

4.

'Nhat items on the routine-reference list can be deleted if the facility develops the examinations?

Mere reforence mazerials that were submitted in electronic format adequate for reviewing and preparing for the examinations?

Mere there any significant problems?

5.

Are the interim security guidelines adequate?

Should anything be added or deleted for Rev.

8?

How many personnel were signed up on the securizy agreement?

How did the facility licenseo deal with the restrictions on personnel who can participate in exam development?

Whaz problems, if any. did they create for the facility licensee?

Qo thev need to be more specifically defined?

r r~as -;here any value added Dv having the expecze" signazories idenzified earlv jn Zhe pl OceSS?

're zne due dates

-,or the exam~nation reference materials.

outlines.

and final examinations reasonaDie:

do they need to be adjusted?

~hat criteria should be used zo delay or reject the exams?

8.

Qescribe any signif cant problems encountered in meeting the following exam inzegrity crizer>a?

The 50~/40~/10~ wrizten exam question'distribution.

,The 25 duplication limit from previous exams and quizzes.

Tne prohibition on duplicating any written exam items from the audit exam.

One "new" scenario for each applicant.

Two "new" JPNs on each operating test (one for upgrades).

Interday overlap/duplication for JPHs/scenarios.

Qoes there need to be a specific limit?

Is the definition of "modified" material sufficient?

Do we need to aefine "new?"

ROI 95-25 ATTACHMENT 2

9.

How did the overall level of difficulty of the examination compare with recently admini stered NRC-devel ooea examinati ons?

Is there a need for more guidance on level of difficulty?

10.

Should we encourage and further evaluate the practice of using the same walk-through for all the applicants (split over 2-3 days as necessary)?

This would cut down on the resources necessary to develop and review the tests and hopefully result in better tests since resources could be more focused on quality than quantity.

13 How are facj lity licensees incorporating PRA/IPE/DAS insights into their exams?

Is guidance to consider the insights when developing the exams sufficient or is more definitive guiaance necessary?

Did restricting the timing of the written examination administration to one week prior to ine, operating tests create any problems?

Should any additional provisions be made for the NRC to inspect the written exam administration process?

I'.

Did allowing the facility to answer questions during the written exam create anv apparent problems with exam integrity?

Mhat additional restrictions. if any.

snould be placed on proctors?

Should we adopt the GFES no ou stion policy?

16.

'B.

19.

Did the requirement to grade the written exams within 5 days create any problems for the licensee'?

Is there an alternate proposal?

Is 5 written exam item changes a reasonable threshold for asking the licensee to explain the examination deficiencies?

Mere facility licensee comments received on preparing and submitting a

matrix of applicant responses to every test question?

Do you think the overall examination process was less effective'ust as effective. or more effective tnan the old method?

Briefly explain.

Do you have any other commen-.s?

ROI 95-25 ATIACHHENT 2

ATTACHMENT 2 NRC GENERIC LETTER 95-06