Text

12-2003 - Draft Exam Comments
	ML050480177
Person / Time
Site:	San Onofre
Issue date:	12/30/2003
From:	Gody A; Operations Branch IV
To:	Ray H; Southern California Edison Co
References
	50-361/03-301, 50-362/03-301 50-361/03-301, 50-362/03-301
	Download: ML050480177 (15)
	v • d • e

ES-401 Written Examination Form ES-401-9 (R8, S1)

Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= S R O U/E /S Explan ation Focus Dis t. Link units ward K/A O nly R1 F 2 T U KA m ismatch; stem wording (lit twice procedure name needed)

R2 H 4 S R3 H 4 T U KA mismatch R4 F 3 S R5 H 4 T U KA mismatch R6 H 3 E KA C E/A11 EK2.2 ???

R7 F 2 S R8 F 2 S R9 F 2 S R10 F 3 T U KA mismatch Instructions

[Ref er to Sec tion D of E S-4 01 an d A ppen dix B f or additional inf ormation regard ing eac h of th e following c onc epts .]

1. En ter th e level of kn owled ge (L O K) of eac h q ues tion as either ( F)un dam ental or ( H )igh er c ogn itive level.

2. Enter the level of difficulty (LO D) of each ques tion using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

@ Th e stem lacks su fficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless inform ation).

@ Th e stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ More th an one d istrac tor is not c redible.

@ On e or more distractors is (are) partially correct (e.g., if the applicant can m ake unstated assu mptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ Th e question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ Th e question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from mem ory).

@ Th e question contains data with an unrealistic level of accu racy or incons istent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Ch eck q ues tions th at are sam pled for c onform anc e with the ap proved K /A an d thos e that are des igna ted S R O -only (K/A and license level mismatches are unacceptable).

6. Based on the review ers jud gm ent, is the qu estion as written (U )nacceptable ( req uiring rep air or rep lac em ent), in need of ( E)ditorial enh anc em ent, or (S)atis fac tory?

7. At a m inimu m, explain an y U ratings (e.g., how the A ppen dix B p syc hom etric attribu tes are n ot being m et).

ES-401 2 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly R11 F 3 S R12 F 2 S R13 H 3 T E 69 E K1 .2 ?? ?; C cred ible ?; clarif y respon ses R14 H 4 T E Choice C correct???

R15 H 4 T E Trend information on parameters needed?

R16 F 3 S R17 H 3 S R18 H 4 T T T U Pow er inc rease on drop C EA ?; C hoice A correc t?; B , D c redible?

R19 H 4 T U KA mismatch R20 H 4 S R21 H 4 E 09 EA 3.26?; delete boldface R22 H 4 E E0 4 E A 2 .01? ; delete boldfac e; Ch oice A cred ible?

R23 H 3 S R24 H 4 E Edit answer C to reflect long term effect R25 H 4 S R26 H 4 T U S am e as qu es tion R 2 R27 F 3 T T E Ed it (rem ove at; r eplac e plan t with reac tor); S pec ify wh ich level R28 F 2 S R29 F 3 E Edit (remove redundant slight meter change indication)

R30 F 2 T U KA mismatch R31 H 3 T T T U KA mis matc h; N o ques tion asked in stem ; ans wer/s tem d isc onnec t; Xe?

R32 H 4 E Edit (spell out iaw)

R33 F 2 S

ES-401 3 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly R34 H 3 T E Edit (is/actions); procedure name?; Choice B correct?; KA AK3.02?

R35 H 2 E Edit ( ans wer need s id entified in key)

R36 F 2 E Edit (KA designator)

R37 H 3 E Ed it (remove c omm ent plant s pec ific d ata)

R38 F 2 T E KA 01 A A 3 .06 d oes not exis t?

R39 H 3 E Ed it (remove c omm ent tem ps and p ress ures ; add w hich of the to Q R40 F 2 E Ed it stem (add of the ... R CS leak rate)

R41 H 2 T E incom plete stem inform ation; No an sw er identified R42 F 2 S R43 H 3 S R44 F 2 E Ed it (remove bold fac e)

R45 H 3 E Edit (remove boldface; include system 015 in KA designator)

R46 F 2 U Same question as R38 R47 F 2 S R48 F 2 T U Stem cues answer of Question R47; KA mismatch R49 H 2 E g pm or gallon s in s tem ?

R50 F 3 S R51 F 3 S R52 H 3 S R53 H 3 E No an sw er identified R54 F 2 S R55 F 2 E Same KA as question R54; Q not related to liquid rad waste (generic)

R56 F 2 S

ES-401 4 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly R57 F 2 S R58 F 3 S R59 H 3 S R60 H 3 S R61 H 4 S R62 H 4 S R63 H 3 T U KA mismatch R64 F 2 T U KA mismatch R65 H 3 T U KA mismatch R66 H 3 S R67 H 2 S R68 H 2 T U KA mismatch R69 F 2 S R70 H 3 S R71 F 2 T U KA mismatch R72 F 3 E Ed it (remove bolf ace)

R73 H 3 E I nc lu de p roc ed ur e n am e R74 H 3 E Edit KA 063 K2.01???

R75 F 2 S R76 F 2 E KA m ismatch?

R77 F 2 S R78 F 2 S R79 F 3 S

ES-401 5 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly R80 H 2 S R81 H 3 S R82 F 2 T E SRO only question?

R83 H 2 S R84 H 2 S R85 H 3 S R86 F 2 S R87 H 3 S R88 F 2 S R89 H 4 T E SRO only question?

R90 H 4 E Ed it (remove bold fac e)

R91 F 2 S R92 F 2 S R93 F 2 S R94 F 2 T E SRO only question?

R95 F 2 E Ed it (remove bold fac e)

R96 F 2 S R97 H 3 S R98 F 2 S R99 F 3 S R100 F 2 S S1 F 3 T U KA mismatch S2 F 3 S

ES-401 6 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. O ther 6. 7.

Q# LOK LOD

( F/H ) (1-5) Stem Cu es T /F Cred. Partial Job- Minut ia #/ Back- Q= SRO U/E /S Explan ation Focus Dis t. Link units ward K/A O nly S3 H 3 T U R O s ys tem level; E d it ( plac e q ues tion at en d of s tem )

S4 F 2 S S5 F 3 T U R O sys tem only S6 H 3 S S7 F 2 S S8 F 2 S S9 H 3 S S10 F 3 S S11 F 3 S S12 F 2 S S13 F 3 T U R O sys tem only S14 H 3 T U R O sys tem only S15 F 2 T U KA mismatch S16 F 3 T U R O sys tem only S17 F 2 T U R O sys tem only S18 H 3 T U R O sys tem only S19 H 3 S S20 H 3 T U R O sys tem only S21 F 2 S S22 F 2 T U R O sys tem only S23 H 3 S S24 F 2 T U R O sys tem only S25 F 3 S

S:\DRS\OB \EXAMS\SO NG S\SON GS -INIT-EXAM 12-2003\SONG S-12-2003-NRC D RAFT EXAM C OM MEN TS\RO Admin JPM review.wpd APPENDIX E - R EG IO N IV O PE RA T ING T ES T JO B P ER FO RM AN CE ME AS UR E Q UA LIT Y R EV IEW MA T RIX

1. 2. 3. A ttributes 4. Job Content 5. 6.

JPM# D yn LOD Errors U/E /S Explan ation (D/S) (1-5) (See below for instructions)

IC Cues Cr itical Scope Over- Job- Minut ia Focus Steps (N/B) lap Link RO A1a S 3 S RO A1b S 3 E Step 9 of th e JP M lis ted the ans wer as 8.8 D K/K . T he key has the ans wer as 8.2 D K/K us ing F ig 4.3.

W e get 8.2 DK/K u sing Fig 4.3. W hich is correct? In addition, there is no tolerance for this answer -

W hy?

RO A2 S 3 E Prints provided to applicant have different valve designations than those documented in the JPM. Step 2 ref ers to th e casing vent valve - W hy?

RO A3 S 3 X E IC d oes not p rovide des ignate dr ain valve to be opened. S tep 1 listed as c ritical for ap plican t to deter mine ap plic able s urvey m ap f rom han dou ts p rovided . O nly s urvey m ap p rovided is th e app licab le one.

RO A4 S 2 S

- Page 1 -

S:\DRS\OB \EXAMS\SO NG S\SON GS -INIT-EXAM 12-2003\SONG S-12-2003-NRC D RAFT EXAM C OM MEN TS\RO Admin JPM review Resolution.wpd APPENDIX E - REG ION IV OP ER AT ING TE ST JO B PE RFO RM AN CE MEA SU RE QU ALITY R EVIEW MAT RIX (COM M ENT RE SOLUTION)

1. 2. 3. A ttributes 4. Job Content 5. 6.

JPM# D yn LOD Errors U/E /S Explan ation (D/S) (1-5) (See below for instructions)

IC Cues Cr itical Scope Over- Job- Minut ia Focus Steps (N/B) lap Link RO A1a S 3 S RO A1b S 3 S Will determine correct answer and revise JPM . Will also add a tolerance for this value.

RO A2 S 3 S Will revise JPM to include valve numbers as shown on print. Valve designations in JPM are the design ations th at will be reco rded in the tag o ut.

RO A3 S 3 S W ill check on th e drain v alve (the re m ay b e only one). If ther are mo re than one, w ill designate what valve to go to. Will revise the JPM to state to find the correct page of the provided survey map.

RO A4 S 2 S Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.

2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.

3. Check the appropriate box when an attribute weakness is identified:

The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.

The JPM does not contain sufficient cues that are objective (not leading).

All critical steps (elements) have not been properly identified.

Scope of the task is either too narrow (N) or too broad (B).

Excessive overlap with other part of operating test or written examination.

4. Check the appropriate box when a job content error is identified:

Topics not linked to job content (e.g., disguised task, not required in real job).

Task is trivial and without safety significance.

5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 1 -

S:\DRS\OB \EXAMS\SO NG S\SON GS -INIT-EXAM 12-2003\SONG S-12-2003-NRC D RAFT EXAM C OM MEN TS\SR O AD MIN JPM review.wpd APPENDIX E - R EG IO N IV O PE RA T ING T ES T JO B P ER FO RM AN CE ME AS UR E Q UA LIT Y R EV IEW MA T RIX

1. 2. 3. A ttributes 4. Job Content 5. 6.

JPM# D yn LOD Errors U/E /S Explan ation (D/S) (1-5) (See below for instructions)

IC Cues Cr itical Scope Over- Job- Minut ia Focus Steps (N/B) lap Link SRO A1a S 2 S SRO A1b S 2 U J P M tas k is to r eview S hu td ow n M ar gin C alc u lat ion an d d ir ec t ap pr op riat e ac tion s , J P M pr ovid ed . S A M E as RO J PM has applicant calculate SDM and nothing else. W hile there are some errors in the calc ulations, th e resu lt is the s ame - n eed to ch ange. L ow dis crim ination.

SRO A2 S 2 E Couldnt find in procedure requirement for discharge valve to be shut before suction valve - does such a requirem ent exist?

SRO A3 S 3 X E IC d oes not p rovide des ignate dr ain valve to be opened. S tep 1 listed as c ritical for ap plican t to deter mine ap plic able s urvey m ap f rom han dou ts p rovided . O nly s urvey m ap p rovided is th e app licab le one.

SRO A4 S 2 S

- Page 1 -

S:\DRS\OB \EXAMS\SO NG S\SON GS -INIT-EXAM 12-2003\SONG S-12-2003-NRC D RAFT EXAM C OM MEN TS\SR O AD MIN JPM review Resolution.wpd APPENDIX E - REG ION IV OP ER AT ING TE ST JO B PE RFO RM AN CE MEA SU RE QU ALITY R EVIEW MAT RIX (COM M ENT RE SOLUTION)

1. 2. 3. A ttributes 4. Job Content 5. 6.

JPM# D yn LOD Errors U/E /S Explan ation (D/S) (1-5) (See below for instructions)

IC Cues Cr itical Scope Over- Job- Minut ia Focus Steps (N/B) lap Link SRO A1a S 2 S SRO A1b S 2 S W ill revise the JPM such that the re sult is also inco rrect.

SRO A2 S 2 S W i ll c h ec k if a p r oc ed u re re q ui re m e nt fo r th e di sc h ar g e v a lv e to b e s h ut be fo r e t he su c ti on v a lv e is clo sed exis ts. If it d oe s ex ist w ill insu re th at th e JP M is co rre ct.

SRO A3 S 3 S W ill check on th e drain v alve (the re m ay b e only one). If ther are mo re than one, w ill designate what valve to go to. Will revise the JPM to state to find the correct page of the provided survey map .

SRO A4 S 2 S Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.

2. Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.

3. Check the appropriate box when an attribute weakness is identified:

The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.

The JPM does not contain sufficient cues that are objective (not leading).

All critical steps (elements) have not been properly identified.

Scope of the task is either too narrow (N) or too broad (B).

Excessive overlap with other part of operating test or written examination.

4. Check the appropriate box when a job content error is identified:

Topics not linked to job content (e.g., disguised task, not required in real job).

Task is trivial and without safety significance.

5. Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

6. Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).

- Page 1 -

APPENDIX E - R EG IO N IV O PE RA T ING T ES T JO B P ER FO RM AN CE ME AS UR E Q UA LIT Y R EV IEW MA T RIX

1. 2. 3. A ttributes 4. Job Content 5. 6.

JPM# D yn LOD Errors U/E /S Explan ation (D/S) (1-5) (See below for instructions)

IC Cues Cr itical Scope Over- Job- Minut ia Focus Steps (N/B) lap Link S1 D 3 X E Proc edure s tates ins ertion limit of 1 15" for P LC EA s, J PM cu e only gives 11 2.5" T ech . Sp ec. lim it. How do we know it is an alternate path?

S2 D 3 X E Need prereq uisite inf ormation (e.g., m ode). C losing HV 935 2 is n ot mark ed as c ritical. W hat is ef fec t on system if left open?

S3 D 3 X X E JP M trans itions to Proc edu re S O 23 2.1 8 an d s tarts at step 6 .3.7 . T he J PM does not p rovide th is inform ation f or the examin er. N eed p roc edu re s teps and nu mber. A lso n eed ter mination point for t his JP M (procedure step).

S4 D 2 X E JP M has reac tor s hu tdow n, th erefo re wh y do th ey trip th e reac tor af ter th e sh aft s heer? If th e reac tor trip is required by the ARP , what about the 5 second limit imposed by the ARP?

S5 D 3 S Do we need a note for examiner at beginning of JPM that keys # 8 and 23 required to complete task?

S7 D 3 S Do w e need a note f or examiner at b eginnin g of J PM that key 52 r equired to com plete task? Need proc edure s teps in JP M b ecau se J PM starts at step 6 .3.9 p age 7 of proc edure.

S6 (C1) S 3 E Attac hm ent 1 of Proc edure S O 23- 6-2 does n ot list a 2A 06 b us to 2XR 1 trans form er con nec tion as s tated in th e J P M .

P1 S 3 E Procedure gives L-85 (L-86) vice L-45 (L-46) given in JPM. P rocedure has step to open Neutral Breaker, t his s tep is n ot in J P M.

P2 S 3 S W hat is the purpose of the cue Atmospheric Dum p Valve HV-8419 has been selected.?

P3 S 2 E Need to add that s teps com pleted up to 3.4 to I.C . JP M r eferenc es d ifferen t proc edure f or D iesel shutdow, procedure refers to SO 23-2-13, JP M refers to SO 23-3-3.23. Steps f or diesel shutdown are not c ons istent w ith the s teps listed in P roced ure S O 3 .23.

APPENDIX E - REG ION IV OP ER AT ING TE ST JO B PE RFO RM AN CE MEA SU RE QU ALITY R EVIEW MAT RIX (COM M ENT RE SOLUTION)

1. 2. 3. A ttributes 4. Job Content 5. 6.

JPM# D yn LOD Errors U/E /S Explan ation (D/S) (1-5) (See below for instructions)

IC Cues Cr itical Scope Over- Job- Minut ia Focus Steps (N/B) lap Link S1 D 3 S W ill revise the JPM to state the correc t insertion lim it.

S2 D 3 S Le av ing HV -93 52 o pe n w ill hav e n o e ffec t be cau se it is in se ries with an oth er v alv e th at is c los ed .

S3 D 3 S Procedure steps will be added to the JPM. The candidate will not have Procedure SO23-3-2.18

( st ep s wi ll b e do n e b y m e m or y .

S4 D 2 S This JPM is not a RCP Shaft Sheer event, only a thrust bearing overheating event. Tripping of the reactor is a requirement of the procedure for an overheated thrust bearing. Will check on the 5 se co nd limit . If req uire d b y t he AR P, th ey will re vis e th e JP M acc ord ing ly.

S5 D 3 S They will provide the keys to the examiner who will than give them to the candidate when requested.

S7 D 3 S W ill pro vid e K ey 53 to the exa m ine r to giv e to the can did ate wh en req ue ste d.

S6 (C1) S 3 S The designation of 2XR1 in the JPM (should be 2XR2) is a typographical error that will be co rre cte d.

P1 S 3 S The L-85 (L-86) designation is for the M G sets while the L-45 (L-46) designation is the panel designation, therefore the JPM is correct. The candidates are not expected to open the Neutral bre ake rs (it is no t ne ces sar y) .

P2 S 3 S T he cu e w ill be d elete d.

P3 S 2 S Th e JPM will be rev ised to m atch the proc edur e. Th at will make the JPM steps co nsistent w ith the pro ced ure .

GENERIC COMMENTS FOR ALL JPMS:

ADD PROCEDURE STEPS TO THE JPMS

DESIGNATE ON THE JPM COVER SHEETS IF THE JPM IS AN ALTERNATE JPM.

APPENDIX E - REG ION IV OP ER AT ING TE ST JO B PE RFO RM AN CE MEA SU RE QU ALITY R EVIEW MAT RIX (COM M ENT RE SOLUTION)

1. 2. 3. A ttributes 4. Job Content 5. 6.

JPM# D yn LOD Errors U/E /S Explan ation (D/S) (1-5) (See below for instructions)

IC Cues Cr itical Scope Over- Job- Minut ia Focus Steps (N/B) lap Link S1 D 3 S W ill revise the JPM to state the correc t insertion lim it.

S2 D 3 S Le av ing HV -93 52 o pe n w ill hav e n o e ffec t be cau se it is in se ries with an oth er v alv e th at is c los ed .

S3 D 3 S Procedure steps will be added to the JPM. The candidate will not have Procedure SO23-3-2.18

( st ep s wi ll b e do n e b y m e m or y .

S4 D 2 S This JPM is not a RCP Shaft Sheer event, only a thrust bearing overheating event. Tripping of the reactor is a requirement of the procedure for an overheated thrust bearing. Will check on the 5 se co nd limit . If req uire d b y t he AR P, th ey will re vis e th e JP M acc ord ing ly.

S5 D 3 S They will provide the keys to the examiner who will than give them to the candidate when requested.

S7 D 3 S W ill pro vid e K ey 53 to the exa m ine r to giv e to the can did ate wh en req ue ste d.

S6 (C1) S 3 S The designation of 2XR1 in the JPM (should be 2XR2) is a typographical error that will be co rre cte d.

P1 S 3 S The L-85 (L-86) designation is for the M G sets while the L-45 (L-46) designation is the panel designation, therefore the JPM is correct. The candidates are not expected to open the Neutral bre ake rs (it is no t ne ces sar y) .

P2 S 3 S T he cu e w ill be d elete d.

P3 S 2 S Th e JPM will be rev ised to m atch the proc edur e. Th at will make the JPM steps co nsistent w ith the pro ced ure .

GENERIC COMMENTS FOR ALL JPMS:

ADD PROCEDURE STEPS TO THE JPMS

DESIGNATE ON THE JPM COVER SHEETS IF THE JPM IS AN ALTERNATE JPM.

S:\DRS\OB\EXAMS\SONG S\SONG S-INIT-EXAM 12-2003\SONG S-12-2003-NRC DRAFT EXAM COMMEN TS\Scenario review.wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen 1 2 3 4 5 6 7 8 9 10 Explanation (See below for instructions)

Set ES TS Crit IC Pred TL L/C Eff U/E/S 1 Yes Yes Yes Yes Yes No Yes Yes E Needs a time line.

Yes Yes Yes Yes Yes No Yes Yes E Needs a time line.

2 3 No Yes Yes Yes Yes No Yes Yes E Needs a time line.

Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.

1. ES: ES-301 checklists 4, 5, & 6 satisfied.

2. TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.

3. Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.

4. IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.

5. Pred: Scenario sequence and other factors avoid predictability issues.

6. TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.

7. L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.

8. Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.

9. Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

10. Provide a brief description of problem in the explanation column

- Page 1 -

S:\DRS\OB\EXAMS\SONG S\SONG S-INIT-EXAM 12-2003\SONG S-12-2003-NRC DRAFT EXAM COMMEN TS\Scenario review Resolution.wpd APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX (COMMENT RESOLUTION)

Scen 1 2 3 4 5 6 7 8 9 10 Explanation (See below for instructions)

Set ES TS Crit IC Pred TL L/C Eff U/E/S 1 Yes Yes Yes Yes Yes No Yes Yes E Will provide a time line.

Yes Yes Yes Yes Yes No Yes Yes E Will provide a time line.

2 3 No Yes Yes Yes Yes No Yes Yes E Will provide a time line.