ML071920480

From kanterella
Revision as of 07:33, 23 October 2018 by StriderTol (talk | contribs) (Created page by program invented by StriderTol)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
Seabrook Station - NRC Comments on Draft License-Developed Exam (Written & Operating Tests) (Folder 2)
ML071920480
Person / Time
Site: Seabrook NextEra Energy icon.png
Issue date: 06/29/2007
From: Caruso J G
NRC Region 1
To: Wright K
Florida Power & Light Energy Seabrook
Sykes, Marvin D.
Shared Package
ML062050109 List:
References
Download: ML071920480 (19)


Text

5. Other Q= SRO WA Only INS 6. 7. UIWS Explanation This is a "F level. Changed to memory level. E This is a GFES question- okay in limited numbers. Question replaced with more discrimina tins opera tionally oriented question U Looks like DLO with graph provided.

It is pretty boderline unless you read the graph wrong it is unlikely that any applicant will miss it. Why are distractors credible and/or how a less than competent applicant could get the answer incorrect short of an inability to use a graph. New Q written using same KIA ~ I ~ N E 'IC" Implausible with only one PT failure. Set points for PORVs lift 2385 both valves have same set point.

Revised "C&D" ES-401 Written Examination Review Worksheet Seabrook 2007 Form ES-401-9 Notes: 1. LOK listed is licensee's evaluation. NRC exception noted in explanation.

If licensee's "H" correct, then may exceed 60%. 2. Resolution of comments is included in italics. 3. BOLD question numbers had a detailed technical review performed.

3. Psychometric Flaws
4. Job Content Flaws - Stem -0cus - - - Cred. Dist. - - Job- Minutia Link I I J -I X "D" Implausible during a "Cooldown and De p ress u r izat io n" Re vised distractor ( IN IS I Revised question stem to make higher order ( IN Is I i X N X -
3. Psychomet
I!; i..1.;r;.

Focus 5. Other Q= SRO WA Only Y N c Flaws Cred. Partial I Dist. I 6. 7. U/US Explanation E "D" distractor doesn't appear to be plausible given head limitations since you are using the boric acid pump or SI pump (740#) into pressurized plant.

Also "A is only borderline since it is not approved for use at Seabrook I 4. Job Content FI Job- Minutia Link units P- IWS - Back- ward - -

1. 2. 3. Psychometric Flaws 4. Job Content Flaws
5. Other 6. 1# LOK LOD (NH) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Focus Dist. Link units ward KIA Only Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable). Check the appropriate box if a psychometric flaw is identified:

0 The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc.).

The distractors are not credible; single implausible distractors should be repaired, more than one is One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions or too much needless information).

The answer choices are a collection of unrelated true/false statements.

u naccepta ble. that are not contradicted by stem). 0 0 0 0 Check the appropriate box if a job content error is identified:

0 The question is not linked to the job requirements (Le., the question has a valid WA but, as written, is noL The question requires the recall of knowledge that is too specific for the closed reference test mode (Le., it The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter The question requires reverse logic or application compared to the job requirements. operational in content). is not required to be known from memory). in percent with question in gallons).

0 0 0 Check questions that are sampled for conformance with the approved WA and those that are designated SRO- only (WA and license level mismatches are unacceptable).

Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met). 7. Explanation ES-401 2 Form ES-401-9 7. Explanation Explain why distractors are plausible e.g., "C" appears implausible. Where is "cold safety injection"?

Revised "Cy' distractor to make more plausible.

"A - clearify justification revised "A" to make i a vital power supply which is more plausible.

Wording issue does "charging system flow" refer to battery charger in "A? Please specify. Re vised "A " to specify charging system flow. "B" not plausible with containment level unchanged

& D" Implausible if "containment pressure is normal" replaced with new Q and new randomly WA. ~ ~~ Overlap primarily redundant or similar to sim JPM #9. "D" not plausible question very easy and leading with the loss of both RHR pumps replaced with a new Q and new randomly selected WA.

3. Psychometric Flaws Stem -0cus - - - >red. - X - X X 4. Job Content Flaws - Sack- ward - - - {NU (NU (NE /NE {NS \]NE tNE 7. Explanation Why would anyone consider "C&D" to be plausible?? Replaced with a bank Q usin! same UA Don't see how "C&D" would be considered plausible with heaters de-energized.

Replaced with a bank Q using same WA. Procedure does not direct reducing to 350 Mwe. Change to 360 Mwe. Note: This is a technical error detected during technical review. Why is "A plausible - trip the turbine cases Rx trip.? Provided basis for plausiblity for "A". "B" makes condition worse and "C" Implausible during RED path ICC. Provided better more detailed plausibility explanations

~ This is a "F level. Re-designated as memory level question and revised plausibility justification statements for distractors.

Stem cues correct answer. Why are the other answers considered plausible?

Revised "Cy' distractor and revised plausibility justification statements for distractors. "significantly" is too subjective for stem condition:

Deleted word "siunificantlv" in stem.

3 7. Explanation "D" appears to be also correct under stem conditions; "shortly" is pretty subjective Also how and why would "A&B" be plausible.

Revised "D distractor and revised plausibi/ity justification statements for distractors.

ES-401 2 Form ES-401-9 #/ Back- Q= SRO Explanation U/E/S inits ward WA Only Y N E Correct answer is the only one with the term "thermal" Revised answer to remove the word "thermal".

NNU WA mismatch. This is a motor current (load) question; not a cooling or ventilation question.

Replaced with new Q and new randomly selected WA.

ES-401 2 Form ES-401-9 Dist. Link Ainutia - - u/E/ Explanation S #/ Back- Q= SRO inits ward KIA Only YNS Y N U "B&C" doesn't seem if you know anything about ECCS power supplies and intiating signals.

Very simple question LOD=l-2. Replaced with a bank Q testing same WA. ~ ~~ ~ ~ Y N E May provide a cue to Q# 18 testing same or similar concepts. "C" distractor not plausible.

Q 18 replaced and plausibility for "C" provided Y N E Memory level question - Re-designated memory I I I I Level. I IY IN Is I Y N E "D" implausible; not enough oil to cause PCCW head tank issues, regardless of pressures.

Changed "D " distractor.

I Iy IN Is I YNS YNS "A is really "none of the above"; not recommended revisded "A "

1. I 2. I 3. Psychometric Flaws {NS \INU ies TIF I - Cred. Dist. - - X 4. Job Content Flaws - vlinutia - - - 3ack- - .I 7. Explanation I 1) WA mismatch.

This is a question regarding CRDM ventilation; not related to "reading containment parameters".

2) Knowing that 2 CRDM fans are normally running eliminates 2 distractors - seems not appropriately discriminating - testing very basic system knowledge Replaced with bank Q testing same WA ES-401 2 Form ES-401-9 1. 2. 3. Psychometric Flaws I 4. Job Content Flaws 5. Other 6. Q# LOK LOD - (F/H) (1-5) Stem I6 F 2 17 F 4 I8 H 3 Dist. 7. Explanation "A and "B seem implausible.

Where was steam flow directed if dumps are closed at the onset? Replaced with bank Q testing same WA. ~ ~~~~ WA mis-match How does this match ability to monitor program levels. Recommend revising question to make more operationally oriented. Replaced with a new Q testing same WA.

3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. Q= SRO U/E/S Explanation WA Only Y N E This is a GFES question; not very operationally oriented.

License decided to replace with bank Q testing same WA. IN Is I YNS Y N S Note: Licensee stated T=O, 1 , 2 are there just tc indicate sequencing - actual time minutes vs seconds not important. Okay in limited number power simple supply IN /s lquestion IN Is I IN Is I

3 ;4 i5 i6 17 1% 9F H H H F H F Q# LOK i 3. Psychometric Flaws
4. Job Content Flaws 5. Other 7. Explanation

~~~~~ "B" is longer than all other distractors. Add "procedurally directed" before manual actions in stem. Explain WA match "loss or malfunction of ICs, S/G, CRDS controllers and positioners??

Comments incorporated.

"6" implausible if steam tables are available.

Revised distractors "B&C" to improve plausibility.

really just a set point question turbine runback vs setback load reduction the same 55% also 220# vs 250# is this even required knowledge from memory. LOD borderline.

Revised Q to improve discriminatory value. Correct answer longer than distractors.

This seems to be a pattern throughout the exam. Revised distractors to balance lengths to avoid cuing Xtractor "9 not plausible in Mode 1 Suspend Core Us. Revised distractor I ~~~~~ 1) This is a GFES question; not discriminating at !he RO level 2) distractors are not plausible if iou know anything about pump design.

3) This is a "F level Replaced with new Q testing same MA. 2orrect answer stands out by capitalizing the Nord "NOT" Comment accepted

~~~ ~-~-- 4. Job Content Flaws I 5. Other I 6. 1 7. #I Back- Q= SRO UIWS inits ward WA Only YNS Explanation Y \J E \1s JE 1) "C" implausible. Licensees do not get permission from Resident Inspector for anything 2) This is a "F" level 3) Is this required RO knowledge from memory with a valid LO? Replaced with a new Q testing same WA. 1) "A Implausible.

This is the normal condition for Mode 5. 2) this is a GFES type question and discriminatory value and operational significanct low. This is a "F" level Replaced with a bank Q testing same WA. \1 E Y State in the stem all other plant systems behave as expected since you are not telling them the Rx trips as expected. Comment incorporated NS Y U A GET question.

Is this required knowledge from memory with a LO to know and apply this formula?

Replaced with a bank Q testing same KIA. Licensee statec that this Q is operationally oriented and therefore will not accept from applicants regarding job applicability.

K/A mismatch WA is for an ATWS this is not ATWS the Rx is s/d and we are not entering FR -S-1 . Replaced with a new Q testing same KIA. Y Y S Do you expect an SRO to recall from memory a step in an appendix.

Yes important concert to test. ~ 'Not SRO-only; no selection of procedures, simply a knowledge of what is in the selected procedure. Distractors not plausible.

Why couldn't EDG's be running?? Replaced with a new Q testing same WA. 1. LOK F/H 4. Job Content Flaws (5. Olry iE 6. Q= SRO U/E& KIA On1 7 3. Psychometric Flaws Focus I - 'artial - 3ack- ward - - - vlinutia - - Explanation Given a Large Break LOCA "A Natural Circ does not seem plausible?

Replaced "A" distractor.

X X X "B and "D not plausible at 65% power (automatic RX Trip). Are you allowed to operate single loop - no and what would be the power restriction with one pump?

Why not change stem condition to 50% vice 65 % power maybe more discriminating?

Enhanced plausibilfy statements and changed power level in stem /YE Iy Is I Iy Is I ' ly Is I IN Is I 'NU Explain basis for why the distractors are plausible.

Provided statements why plausible

3. Psychometric Flaws I 4. Job Content Flaws [ 5. Other J Stem -0cus - - Not SRO-Only. This is basically a system knowledge question.

Also doesn't match WA which requires selection of procedure to correct, control, or mitigate. Maybe okay as an RO system question.

Replaced with a new Q testing same WA. hes I 'Not SRO-Only.

This is basically a system 'knowledge question.

"A and "D" not plausible.

Would not expect a 1"permissive" to actuate anything.

Replaced 1 with a new Q testing different randomly selected

~ WA. Y Y Y Y Y Y Y 6. UIWS E 7. Another example of the longest distractor being the correct answer. Modified correct answer Explanation I Extremely easy. "B" not plausible.

Question ldoesn't discriminate at the appropriate level.

TS depth of testing is surface not testing operator 1 understanding. Replaced with a new Q using 'same WA J SI Not SRO-Only.

This is basically a system knowledge question Replaced with a new Q and a new randomly selected WA J SI J I S I 1"D not plausible.

Changed distractor

3. Psychometric Flaws 4. Job Content Flaws 5. Other I 6. 7. Explanation Looks like DLO to me. Took me less than 30 seconds to come up with the right answer using the provided reference material. Maybe they can "show me" differently on site. Recommend revising to make it an integrated plant question.

Q replaced with a bank Q using the same WA. YU vu YS YE fS fS 16 H 17 F 18 F 19 H 00 F Distractors A&D are not plausible to review adequacy of post maintenance retest requirements after completion of re-test makes no sense. This is not an SRO level question that discriminates at an appropriate level.

Q reploaed with a bank Q. Made a fundamental le vel, 3 "A doesn't seem plausible to allow HP supervisor alone to approve. Change "A to "I & C Supt and The Health Physics Department Manager".

Replaced with RO #73 which was wriffen at the SRO level. Memory level.

SUMMARY

OF COMMENTS FOR SEABROOK OP TEST Where are updated JPM outlines for Admin and Sim/ln-plant JPMs?? RO Admin JPMs #1 provide completed Form B template - key not provided - key provided #2 provide completed Form C SDM Determination - key provided

  1. 3 poor JPM - appears to be very simplistic marginal task can we beef it up a bit to make more discriminating leaning towards overly simplistic doesn't demonstrate understanding or competence. Also prepare appropriate Form as a Key. Redesigned to perform in simulator - key provided #4 poor JPM - don't see where the discriminatory value comes in - applicant doesn't demonstrate knowledge and understanding just ability to transcribe data . Also prepare appropriate Form as a Key. JPM replaced SRO Admin JPMs #1: Unsat ... non-discriminating for SRO Replaced JPM #2: OK, but delete paragraph C-J of "Directions to Students" (gives clue). Initiating Cue these are SRO applicants why do we need to tell them "using ODI-30" to leading they should know this. Also delete the first Cue from the body of the JPM and from the tear off sheet and section 10 direction to students - unnecessary for same reason SRO applicants.

Actually this would be a good opportunity to make this integrated TS call and include e.g. a CCP. LCO and action outage time Modified to have applicant determine TS #3: OK. #4: Ok #5: Revised to administer after each scenario to first classify before making the notifications. Sim JPMs Big Picture: 3 Unsats JPMs; 2 marginal-to-Unsat JPMs ... as a set, the sim JPMs have poor discriminating validity. General Notes:

1) For SRO applicants do not provide the procedure numbers for Simulator JPMs they are expected to be able to chose the right procedure.
2) General Comment for all JPMs - Clean-up tear off sheets for each JPM delete introductory statements "Ensure task is done correctly, and You may asked follow-up questions - that goes without saying and could add to applicant stress. 3)

General Comment for all JPMs -Directions to the students delete all the instructions after B on this one and the other JPMs in general. We already have a

detailed briefing sheet for the exam APP E. We don't need this stuff and it is confusing to cover performance standards we are not going to read these to your applicants.

  1. 1: consider replacing JPM

... hold your nose ... barely 2 Critical steps - see that an ASDV is open, attempt to manually close it from control board (it won't) then call an EO to isolate the ASDV locally ... minimal board operations.

Very weak JPM - spell out the acronym "VAS in evaluator cue. Replaced JPM #2: OK except must delete paragraph C in "Directions to Students"

... the paragraph will give away to the applicants that this JPM is faulted. . Spell out all acronyms(ex.

"MPCS step #7) in the the JPMs at least once which is standard practice.

Revised #3: OK except the JPM tells applicant that the examiner will read the steps of the procedure

... we don't do that must maintain independence. Applicants should do their own self-check STAR unless physically not possible this SOP for initial license exams. This is a standard JPM and normally don without any outside assistance. Only one RO applicant no reader will be allowed for uniform exam consistency not allowed #4: Unsat ... 2 Critical steps, which require the applicants to merely observe - but not take action for - an automatic action which did not occur ... NO BOARD OPERATIONS.

Created newalt path JPM #5: OK #6: Unsat ... at a stretch, 1 Critical step - applicant must merely VERIFY (because plant conditions indeed support) that the running SI pump should NOT be stopped

... NO BOARD OPERATIONS.

Created new alt path JPM #7: Unsat JPM ... like #4, is this a newly proposed replacement JPM? Number doesn't agree with origninally submitted outline??

Is this being considered alternate path because it really isn't? applicants must verify automatic isolation actions should have occurred

... detect that 1 valve did NOT auto close, so then close it manually ... ONE BOARD MANIPULATION.

Very little to no discriminatory value the procedure to mainly isolate systems as required. Deleted JPM

  1. 8: OK #9:OK Plant JPMs JPM 01 - Tighten initiating cue to be specific "...simulate transferring power panel PP-1A (b, C, D) pick one - any one. Are they all in the same area or pick the one that is most accessable. Designated panel 7A JPM Step 3 marked WA. JPM -03 This JPM is designated alternate path - For this JPM don't agree with the direct cuing to carry out emergency boration when pump fails. Why can't we modify the initiating cue to establish rapid boration and then later just give the cue CS-P-3A will not start. If asked Boration is required what do you recommend and then concur to their recommendation to initiate emergency boration.

We expect the applicants to take the alternate paths on their own without direction that is the whole point of the alt path JPM. It is suppose to be a procedurally driven alternate path that is an expected path given the plant conditions.

This JPM should be able to be modified to do that. Deleted a/t path Scenarios General Comment : Scenario outlines D-1 don't seem to match the scenarios need revision A// D-7 forms revised