ML071920480

NRC Comments on Draft License-Developed Exam (Written & Operating Tests) (Folder 2)

ML071920480
Person / Time
Site:	Seabrook
Issue date:	06/29/2007
From:	Caruso J NRC Region 1
To:	Wright K Florida Power & Light Energy Seabrook
Sykes, Marvin D.
Shared Package
ML062050109	List: IR 05000443/2007301 ML070811054 ML071920480 ML071920500 ML071930191 ML071930235 ML071930238 ML071930241 ML071930246 ML071930252 ML071930259 ML072080317 ML072390373 ML072920097 ML072960127 SBK-L-07066, Draft - Outlines and Facility Ltr. Dtd 4/11/07 (Folder 2)
References
Download: ML071920480 (19)

Text

ES-401 Written Examination Review Worksheet Seabrook 2007 Form ES-401-9 Notes:

1. LOK listed is licensees evaluation. NRC exception noted in explanation. If licensees H correct, then may exceed 60%.

2. Resolution of comments is included in italics.

3. BOLD question numbers had a detailed technical review performed.

-3. Psychometric- Flaws 4. Job Content Flaws 5. Other 6. 7.

Stem

-0cus Cred.

Dist. I Job- Minutia Link Q= SRO UIWS WA Only I N S This is a F level.

Explanation Changed to memory level.

I J E This is a GFES question- okay in limited numbers. Question replaced with more discriminatins operationally oriented question

-I X D Implausible during a Cooldown and Depressurization Re vised distractor

( IN IS I Revised question stem to make higher order

( IN Is I i X N U Looks like DLO with graph provided. It is pretty boderline unless you read the graph wrong it is unlikely that any applicant will miss it. Why are distractors credible and/or how a less than competent applicant could get the answer incorrect short of an inability to use a graph. New Q written using same KIA

~ ~

X I N E ICImplausible with only one PT failure. Set points for PORVs lift 2385 both valves have

- same set point. Revised C&D

i..1.;r;.I

3. Psychomet c Flaws

- 5. Other

4. Job Content FI IWS 6. 7.

IP-

!I ; Cred. Partial Job- Minutia Back- Q= SRO U/US Explanation Focus Dist. Link -

units ward WA Only I

Y N E D distractor doesnt appear to be plausible given head limitations since you are using the boric acid pump or SI pump (740#)into pressurized plant. Also A is only borderline since it is not approved for use at Seabrook

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

1# LOK LOD (NH) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward KIA Only Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 -

4 range are acceptable).

Check the appropriate box if a psychometric flaw is identified:

0 The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed or too much needless information).

0 The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc.).

0 The answer choices are a collection of unrelated true/false statements.

0 The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

0 One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

Check the appropriate box if a job content error is identified:

0 The question is not linked to the job requirements (Le., the question has a valid WA but, as written, is noL operational in content).

0 The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it is not required to be known from memory).

0 The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

0 The question requires reverse logic or application compared to the job requirements.

Check questions that are sampled for conformance with the approved WA and those that are designated SRO-only (WA and license level mismatches are unacceptable).

Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement),

in need of (E)ditorial enhancement, or (S)atisfactory?

At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9 7.

Explanation Explain why distractors are plausible e.g., C appears implausible. Where is cold safety injection? Revised Cydistractor to make more plausible.

A - clearify justification revised A to make i a vital power supply which is more plausible.

Wording issue does charging system flow refer to battery charger in A? Please specify.

Re vised A to specify charging system flow.

B not plausible with containment level unchanged & D Implausible if containment pressure is normal replaced with new Q and new randomly WA.

~ ~~

Overlap primarily redundant or similar to sim JPM

1. 9. D not plausible question very easy and leading with the loss of both RHR pumps replaced with a new Q and new randomly selected WA.

3. Psychometric Flaws 4. Job Content Flaws

- 7.

Stem >red. Sack- Explanation

-0cus ward X { N U Why would anyone consider C&D to be plausible?? Replaced with a bank Q usin!

- same U A X ( N U Dont see how C&D would be considered plausible with heaters de-energized. Replaced with a bank Q using same WA.

X ( N E Procedure does not direct reducing to 350 Mwe.

Change to 360 Mwe. Note: This is a technical error detected during technical review. Why is A plausible - trip the turbine cases Rx trip.?

Provided basis for plausiblity for A.

/ N E B makes condition worse and C Implausible during RED path ICC. Provided better more detailed plausibility explanations

~

{ N S This is a F level. Re-designated as memory level question and revised plausibility justification statements for distractors.

\ ] N E Stem cues correct answer. Why are the other answers considered plausible? Revised Cy distractor and revised plausibility justification statements for distractors.

t N E significantly is too subjective for stem condition:

Deleted word siunificantlvin stem.

7.

Explanation 3 D appears to be also correct under stem conditions; shortly is pretty subjective Also how and why would A&B be plausible. Revised D distractor and revised plausibi/ityjustification statements for distractors.

ES-401 2 Form ES-401-9

1. / Back- Q= SRO U/E/S Explanation inits ward WA Only Y N E Correct answer is the only one with the term thermal Revised answer to remove the word thermal.

N N U WA mismatch. This is a motor current (load) question; not a cooling or ventilation question.

Replaced with new Q and new randomly selected WA.

ES-401 2 Form ES-401-9 Ainutia #/ Back- Q= SRO u/E/ Explanation S

Dist. Link - inits ward KIA Only Y N S Y N U B&C doesnt seem if you know anything about ECCS power supplies and intiating signals. Very simple question LOD=l-2. Replaced with a bank Q testing same WA.

~ ~~ ~ ~

Y N E May provide a cue to Q# 18 testing same or similar concepts. C distractor not plausible.

Q 18 replaced and plausibility for C provided I II IY N -

E Memory level question Re-designated memory Level.

I IY IN Is I Y N E D implausible; not enough oil to cause PCCW head tank issues, regardless of pressures.

Changed D distractor.

I Iy IN Is I Y N S Y N S A is really none of the above; not recommended revisded A

.I I 2. I I

1. 3. Psychometric Flaws 4.Job Content Flaws 7.

ies TIF Cred. vlinutia 3ack- Explanation

-Dist. - -

{ N S X \ I N U 1) W A mismatch. This is a question regarding CRDM ventilation; not related to reading containment parameters. 2) Knowing that 2 CRDM fans are normally running eliminates 2 distractors - seems not appropriately discriminating - testing very basic system knowledge Replaced with bank Q testing same WA ES-401 2 Form ES-401-9 Q#

1.

LOK LOD 2.

- 3. Psychometric Flaws I 4. Job Content Flaws 5. Other 6. 7.

(F/H) (1-5) Stem Explanation Dist.

A and B seem implausible. Where was steam flow directed if dumps are closed at the onset?

Replaced with bank Q testing same WA.

~ ~~~~

I6 F 2 WA mis-match How does this match ability to monitor program levels. Recommend revising question to make more operationally oriented.

Replaced with a new Q testing same WA.

17 F 4 I8 H 3

3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q= SRO U/E/S Explanation WA Only Y N E This is a GFES question; not very operationally oriented. License decided to replace with bank Q testing same WA.

IN Is I Y N S Y N S Note: Licensee stated T=O, 1, 2 are there just ct indicate sequencing - actual time minutes vs seconds not important.

IN /s Okay in limited number power simple supply lquestion IN Is I IN Is I

i

3. Psychometric Flaws 4. Job Content Flaws 5. Other 7.

Q# LOK Explanation

~~~~~

B is longer than all other distractors. Add procedurally directed before manual actions in stem. Explain WA match loss or malfunction of ICs, S/G, CRDS controllers and positioners??

Comments incorporated.

3 H 6implausible if steam tables are available.

Revised distractors B&C to improve plausibility.

4 H really just a set point question turbine runback vs setback load reduction the same 55% also 220# vs 250# is this even required knowledge from memory. LOD borderline. Revised Q to improve discriminatory value.

i5 H Correct answer longer than distractors. This seems to be a pattern throughout the exam.

Revised distractors to balance lengths to avoid cuing i6 F 17 H Xtractor 9 not plausible in Mode 1 Suspend Core U s . Revised distractor

~~~~~

1% F I 1) This is a GFES question; not discriminating at

!he RO level 2) distractors are not plausible if iou know anything about pump design. 3)

This is a F level Replaced with new Q testing same MA.

9 F 2orrect answer stands out by capitalizing the Nord NOT Comment accepted

I I 1

~ - ~ -

~~~ -

4. Job Content Flaws 5. Other 6. 7.

1. I Back- Q= SRO UIWS Explanation inits ward WA Only Y N S Y J E 1) A Implausible. This is the normal condition for Mode 5. 2) this is a GFES type question and discriminatory value and operational significanct low. This is a F level Replaced with a bank Q testing same WA.

Y N S A GET question.

Is this required knowledge from memory with a LO to know and apply this formula? Replaced with a bank Q testing same KIA. Licensee statec that this Q is operationally oriented and therefore will not accept from applicants regarding job applicability.

\J E 1 ) C implausible. Licensees do not get permission from Resident Inspector for anything

2) This is a F level 3) Is this required RO knowledge from memory with a valid LO?

Replaced with a new Q testing same WA.

\1s Y \1 E State in the stem all other plant systems behave as expected since you are not telling them the Rx trips as expected. Comment incorporated Y U K/A mismatch WA is for an ATWS this is not ATWS the Rx is s/d and we are not entering FR -S-1. Replaced with a new Q testing same KIA.

1.

LOK F/H Focus I

3. Psychometric Flaws X

artial vlinutia

- (5. O l r y

4. Job Content Flaws 3ack- Q= SRO ward KIA On1 iE 6.

U/E&

7 Explanation Given a Large Break LOCA A Natural Circ does not seem plausible? Replaced A distractor.

X / Y E B and D not plausible at 65% power (automatic RX Trip). Are you allowed to operate single loop - no and what would be the power restriction with one pump? Why not change stem condition to 50% vice 65 % power maybe more discriminating? Enhanced plausibilfy statements and changed power level in stem Iy Is I Iy Is I Y S Do you expect an SRO to recall from memory a step in an appendix. Yes important concert to test.

ly Is I IN Is I ~

X N U Not SRO-only; no selection of procedures, simply a knowledge of what is in the selected procedure. Distractors not plausible. Why couldnt EDGs be running?? Replaced with a new Q testing same WA.

Explain basis for why the distractors are plausible. Provided statements why plausible

3. Psychometric Flaws I 4. Job Content Flaws [ 5. Other 6. 7.

Stem h e s UIWS Explanation

-0cus J Not SRO-Only. This is basically a system knowledge question. Also doesnt match W A which requires selection of procedure to correct, control, or mitigate. Maybe okay as an RO system question. Replaced with a new Q testing same WA.

S I I

Y J Not SRO-Only. This is basically a system knowledge question.

A and D not plausible. Would not expect a 1permissive to actuate anything. Replaced 1 with a new Q testing different randomly selected

~ WA.

Y S I Y E Another example of the longest distractor being the correct answer. Modified correct answer Y J I Extremely easy. B not plausible. Question ldoesnt discriminate at the appropriate level. TS depth of testing is surface not testing operator 1 understanding. Replaced with a new Q using same WA Y J Not SRO-Only. This is basically a system knowledge question Replaced with a new Q I

and a new randomly selected WA Y S I Y 1D not plausible. Changed distractor

3. Psychometric Flaws 4. Job Content Flaws 5. Other I 6. 7.

Explanation Y U Looks like DLO to me. Took me less than 30 seconds to come up with the right answer using the provided reference material. Maybe they can show me differently on site. Recommend revising to make it an integrated plant question.

Q replaced with a bank Q using the same WA.

16 H v u Distractors A&D are not plausible to review adequacy of post maintenance retest requirements after completion of re-test makes no sense. This is not an SRO level question that discriminates at an appropriate level. Q reploaed with a bank Q.

17 F Y S Made a fundamental le vel, 18 F 3 Y E Adoesnt seem plausible to allow HP supervisor alone to approve. Change A to I & C Supt and The Health Physics Department Manager. Replaced with RO #73 which was wriffen at the SRO level. Memory level.

19 H f S 00 F f S

SUMMARY

OF COMMENTS FOR SEABROOK OP TEST Where are updated JPM outlines for Admin and Sim/ln-plant JPMs??

RO Admin JPMs #1 provide completed Form B template - key not provided - key provided

1. 2 provide completed Form C SDM Determination - key provided

1. 3 poor JPM - appears to be very simplistic marginal task can we beef it up a bit to make more discriminating leaning towards overly simplistic doesnt demonstrate understanding or competence. Also prepare appropriate Form as a Key. Redesigned to perform in simulator key -

provided

1. 4 poor JPM - dont see where the discriminatory value comes in -

applicant doesnt demonstrate knowledge and understanding just ability to transcribe data . Also prepare appropriate Form as a Key. JPM replaced SRO Admin JPMs #1: Unsat ... non-discriminatingfor SRO Replaced JPM

1. 2: OK, but delete paragraph C-J of Directions to Students (gives clue).

Initiating Cue these are SRO applicants why do we need to tell them using ODI-30 to leading they should know this. Also delete the first Cue from the body of the JPM and from the tear off sheet and section 10 direction to students - unnecessary for same reason SRO applicants.

Actually this would be a good opportunity to make this integrated TS call and include e.g. a CCP. Modified to have applicant determine TS LCO and action outage time

1. 3: OK.

1. 4: Ok

1. 5: Revised to administer after each scenario to first classify before making the notifications.

Sim JPMs Big Picture: 3 Unsats JPMs; 2 marginal-to-Unsat JPMs ... as a set, the sim JPMs have poor discriminating validity.

General Notes: 1) For SRO applicants do not provide the procedure numbers for Simulator JPMs they are expected to be able to chose the right procedure. 2) General Comment for all JPMs - Clean-up tear off sheets for each JPM delete introductory statements Ensure task is done correctly, and You may asked follow-up questions - that goes without saying and could add to applicant stress. 3) General Comment for all JPMs -Directions to the students delete all the instructions after B on this one and the other JPMs in general. We already have a

detailed briefing sheet for the exam APP E. We dont need this stuff and it is confusing to cover performance standards we are not going to read these to your applicants.

1. 1: consider replacing JPM ... hold your nose ... barely 2 Critical steps -

see that an ASDV is open, attempt to manually close it from control board (it wont) then call an EO to isolate the ASDV locally ... minimal board operations. Very weak JPM - spell out the acronym VAS in evaluator cue. Replaced JPM

1. 2: OK except must delete paragraph C in Directions to Students ... the paragraph will give away to the applicants that this JPM is faulted. .

Spell out all acronyms(ex. MPCS step #7) in the the JPMs at least once which is standard practice. Revised

1. 3: OK except the JPM tells applicant that the examiner will read the steps of the procedure ... we dont do that must maintain independence.

Applicants should do their own self-check STAR unless physically not possible this SOP for initial license exams. This is a standard JPM and normally don without any outside assistance. Only one RO applicant no reader will be allowed for uniform exam consistency not allowed

1. 4: Unsat ... 2 Critical steps, which require the applicants to merely observe - but not take action for - an automatic action which did not occur

... NO BOARD OPERATIONS. Created newalt path JPM

1. 5: OK

1. 6: Unsat ... at a stretch, 1 Critical step - applicant must merely VERIFY (because plant conditions indeed support) that the running SI pump should NOT be stopped ... NO BOARD OPERATIONS. Created new alt path JPM

1. 7: Unsat JPM ... like #4, is this a newly proposed replacement JPM?

Number doesnt agree with origninally submitted outline?? Is this being considered alternate path because it really isnt? applicants must verify automatic isolation actions should have occurred ... detect that 1 valve did NOT auto close, so then close it manually ... ONE BOARD MANIPULATION. Very little to no discriminatory value the procedure to mainly isolate systems as required. Deleted JPM

1. 8:OK

1. 9:OK Plant JPMs JPM 01 - Tighten initiating cue to be specific ...simulate transferring power panel PP-1A (b, C, D) pick one - any one. Are they all in the same area or pick the one that is most accessable. Designated panel 7A

JPM Step 3 marked WA.

JPM -03This JPM is designated alternate path - For this JPM dont agree with the direct cuing to carry out emergency boration when pump fails.

Why cant we modify the initiating cue to establish rapid boration and then later just give the cue CS-P-3A will not start. If asked Boration is required what do you recommend and then concur to their recommendation to initiate emergency boration. We expect the applicants to take the alternate paths on their own without direction that is the whole point of the alt path JPM. It is suppose to be a procedurally driven alternate path that is an expected path given the plant conditions. This JPM should be able to be modified to do that. Deleted a/t path Scenarios General Comment : Scenario outlines D-1 dont seem to match the scenarios need revision A// D-7 forms revised