ML18065B212
| ML18065B212 | |
| Person / Time | |
|---|---|
| Site: | Palisades  |
| Issue date: | 04/21/1998 |
| From: | Grove J NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | Thomas J. Palmisano CONSUMERS ENERGY CO. (FORMERLY CONSUMERS POWER CO.) |
| References | |
| NUDOCS 9804240307 | |
| Download: ML18065B212 (12) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION REGION Ill Mr. Thomas J. Palmisano 801 WARRENVILLE ROAD LISLE, ILLINOIS 60532-4351 April 21, 1998 Site Vice President and General Manager Palisades Nuclear Generating Plant 27780 Blue Star Memorial Highway Covert, Ml 49043-9530
SUBJECT:
REGION Ill EXAMINATION WRITING WORKSHOP
Dear Mr. Palmisano:
Region II I staff will conduct an Examination Writing Workshop in the Chicago area on Wednesday, May 27 and Thursday, May 28, 1998. The workshop is designed to inform your management and operations and training staffs of examination writing techniques for the licensing of reactor operators and senior reactor operators in accordance with the guidance contained in NUREG 1021, "Operator Licensing Examination Standards for Power Reactors."
I would like to extend an invitation to you and to members of your operations and training staffs, to attend this workshop. Our goal for the workshop is to provide a learning experience for your staff, to provide opportunities to develop and review examination materials, and to provide an opportunity to interact with regional and headquarters operator licensing staffs. The subject audience for this workshop is those members of your staff who will prepare and review your initial examination submittals.
The location for this conference will be the Doubletree Guest Suites, 2111 Butterfield Road, Downers Grove, IL 60515. This hotel is near the intersection of Interstates 88 and 355, which is about 3 miles from the Region Ill offices. A block of roorris have been reserved on May 26 and 27 for attendants at the Doubletree at the rate of $125 plus tax per night.. Please confirm your reservations by May 12, 1998, by calling the hotel directly at (630) 971-2000. In addition, please notify Ms. Mary Ann Bies of my staff at (630) 829-9711 with the names and positions of people who plan to attend the workshop.
Some background information is enclosed which supports our discussions of the fundamentals of question development. We request that all attendees at the workshop be familiar with this material. This will allow us to quickly move into the question writing workshop by having all attendees at a similar starting point. However, we will allow for some time to provide an overview and answer questions on the material.
9804240307 980421 PDR ADOCK 05000255 V
-~-------** --- - -- - --------.-----
T. Palmisano 2
A draft agenda is enclosed. We look forward to meeting with you and your staff in May. If you have any questions, any comments on the agenda, or need additional information, please contact Mr. Melvyn Leach, Chief, Operator Licensing Branch, at (630) 829-9705.
Docket No:
50-255
Enclosures:
As stated Sincerely, original signed by M. N. Leach/for John A. Grobe, Director Division of Reactor Safety cc w/encls:
E. Chatfield, Training Manager Distribution:
Docket File w/encl PUBLIC IE-01 w/encl SRIPalisades w/encl Riii PRR w/encl DOCDESK (E-Mail)
Greens DOCUMENT NAME: G:PAL-SHOP.OL b *c* eo To receive a copy of this document, indicate in the ox
=
PY w/o attacil/en cl *e* eo
=
OFFICE Rlll:DRS 1¥ Rlll:DRS I A/
NAME Leach/le ~
t..
Grobe t?';?'h"<~
DATE 4/~/98 4/~/98 OFFICIAL RECORD COPY.
PY w atta ci1/
cl "N" N
en
= o copy I
I
.. -'..i
8:00 to 8:15 a.m.
8:15 to 10:00 a.m.
10:00 to 10:15 a.m.
10:15to 11:45a.m.
11 :45 to 12:45 p.m.
12:45 to 2:45 p.m.
REGION Ill EXAMINATION WORKSHOP DRAFT AGENDA WEDNESDAY MAY 27 Introductory remarks Discussion of measurement concepts including: validity, levels of knowledge, level of difficulty, & sampling.
Break Review examples of acceptable and unacceptable test items. Review examples of before and after versions of test items.
- Lunch Examination question development workshop. Attendees will prepare questions based on lesson plan *material provided.
2:45 to 3:00 p.m.
Break 3:00 to 4:00 p.m.
4:00 to 5:00 p.m.
Examination question review workshop.
Attendees will review examination
- questions developed by others.
Group critique of questions. -
Melvyn Leach Dr. George Usova Riii Examiners Riii Examiners Riii Examiners RI 11 Examiners
8:00 to 9:30 a.m.
9:30 to 9:45 a.m.
10:00 to 11 :45 a.m.
11 :45 to 12:45 p.m.
12:45to1:45 p.m.
REGION Ill EXAMINATION WORKSHOP DRAFT AGENDA THURSDAY MAY 28 Group presentations of questions which presented learning opportunities.
Break Operating test discussion - dynamic scenarios and job performance measures:
Lunch Operating test discussion -
administrative section.
1 :45 to 2:00 p.m.
Break 2:00 to 2:30 p.m.
Operations and Operator Licensing.
2:30 to 3:00 p.m.
Closing remarks and questions.
-~*---*~--~--S--~-----.*--,*-*--**--.0'""*-*"**-*w **-.....-.,.--...-...-* *--****- ***--<
- RI II Examiners Riii Examiners A. Bill Beach Melvyn Leach
THE NRC WRITIEN EXAMINATION:
PRINCIPLES AND PRACTICES Dr. George M. Usova USN RC Training and Assessment Specialist 301-415-1064 REGION 3 WORKSHOP MAY 27, 1998 SESSION OBJECTIVE:
To review validity eoncepts affecting the NRC written examination for the purpose of:
instructing facility personnel toward the construction of more valid and consistent NRC license examinations COVERAGE 3 Levels of Validity 3 Levels of Knowledge Discrimination, Sampling, and Cut Scores Psychometrics VALIDITY A valid test is one which tests what it intends to test.
_ID training examinations, this means it tests the specific skills and knowledge defined and taught in the objectives.
In licensing examinations, this means it tests the specific skills and knowledge that should have been defined and taught in the objectives.
3 LEVELS OF VALIDITY Content Operational Discriminant CONTENT VALIDITY Essentially addresses KIA coverage and sampling plan coverage OPERATIONAL VALIDITY Addresses two questions:
- 1.
Is the question, as stated, important to be known as a part of the operator's job?
- 2.
Does the question, as expressed, require the candidate to perform a job related mental or physical operation?
DISCRIMINANT VALIDITY Addresses making a distinction of measurement along a continuum of candidate performance.
The cut score is the exam performance level that the test yields for making a pass-fail decision.
Since the cut score is 80 percent, the exam must be written at a level of difficulty that intends to discriminate at the 80 percent level.
The test item.its stem and distractor, interplay are such that, by intent and design, at least 80 percent of the candidates taking the test should answer the item correctly.
VALIDITY
SUMMARY
- 1.
The exam (item) must be content valid, which encompasses job safety significance and sampling.
- 2.
The exam (item) should be operationally oriented:.a conceivable mental or psychomotor performance of the job. As such the item should be written at the comprehension or analysis level vice simple fundamental knowledge;
- this means items that measure problem solving, prediction, and analysis-central to job performance.
- 3.
The *exam (item) must discriminate at a moderate level of difficulty as set by the cut score. This means that the stem and distractor interplay are such that at least 80 percent of the candidates taking the test should answer the item correctly.
3 LEVELS OF KNOWLEDGE Bloom's Taxonomy Analysis, Application, Synthesis Comprehension Fundamental (simple memory)
LEVEL OF KNOWLEDGE Bloom's Taxonomy is the reference benchmark NRC uses to classify the levels of knowledge of test items.
Bloom's Taxonomy is a classification scheme that permits the classification of items by the level (depth) of mental thought and performanee required to answer the items.
Bloom's Taxonomy conceptually can be applied to written, scenarios, or JPM items.
LEVELS DEFJNED LEVEL 1 Fundamental Knowledge testing is defined as a simple mental process that tests the recall or recognition of discrete information bits with concrete referents; examples include knowledge of terminology, definitions, set points, or other specific facts.
LEVEL 2 Comprehension testing involves the mental process of understanding the material through relating it to its own parts or to some other material; examples can include rephrasing information in different words, describing or recognizing relationships, showing similarities and differences among parts or wholes, recognizing how systems interact, including consequences or implications.
LEVEL 3 Analysis, synthesis, and application testing is a moreactive and product-oriented testing which involves the multi-part mental process of assembling, sorting, or integrating the parts (information bits and their relationships) so that the whole, and the sum of its parts can be used to: predict an event or outcome, solve a problem, or create something new, i.e., mentally using the knowledge and its meaning to solve problems.
DETERMINANTS OF DISCRIMINATION Level of examination knowledge.
Level of examination difficulty Passing score Item bank use NATURE OF EXAMINATIONS AND TESTS Tests are samples of performance Infer overall performance based upon sample Sample must be broad-base.d to make confident inference Sample must not be fully predictable otherwise inferences cannot be made on untested areas Items must intend to discriminate otherwise test has little or no value PSYCHOMETRICS Items may have one or more of the following psychometric errors:
- 1. Low level of knowledge (fundamental knowledge)
- 2. Low operational validity (not clearly job operational)
- 3. Low discriminatory validity (too easy or too hard)
- 4. Implausible distractors
- 5. Confusing language or ambiguous questions
- 6. Confusing or inappropriate negatives in the question
- 7. Collection of true/false statements
- 8. Backward logic LOW LEVEL OF KNOWLEDGE Which one of the following is powered from 4160 VAC bus 1 A?
- a. RHR pump A*
- b. RHR pump B
- c. RH~ pump C d.- RHR pump D
LOW LEVEL OF KNOWLEDGE Select the full core display indication of a drifting control rod.
- a. Red light*
- b. White light
- c. Blue light d, Amber light LOW LEVEL OF KNOWLEDGE Concerning use of water as a fire extinguishing agent, SELECT the correct statement from the following:
- a. Primary agent for extinguishing Class A fires and also effective on Class B and C fires *
- b. Primary agent for extinguishing Class B fires band also effective on Class A and C fires
- c. Primary agent for extinguishing Class A and B fires but not effective on Class C fires
- d. Primary agent for extinguishing Class B and C fires but not effective on,Class A fires LOW LEVEL OF KNOWLEDGE The following plant conditions exist:
RCP 2A tripped after running for 50 minutes.
The RCP was restarted, but tripped within 15 seconds.
Which ONE of the following is the minimum required interval before the next attempt to start RCP 2A?
- a. 15 minutes
- b. 30 minutes*
- c. 45 minutes
- d. 60 minutes LOW OPERATIONAL VALIDITY Under which one of the following conditions should the Shift Supervisor inform the shop steward?
- a. Initiation of a directed overtime request
- b. Discipline action on a supervisory personnel
- c. Medical injury of a contractor personnel
- d. Personnel error by a bargaining unit member*
While this may be related to a SS's job, it has nothing to do with nuclear safety and should not be included in an NRC examination.
- ~.
LOW DISCRIMINATORY VALIDITY Which one of the following reactor water levels will initiate the RHR pumps?
- a. Level 1 only*
- b. Level 1 and 2 only
- c. Level 1 and 2 and 3 only
- d. Level 6 only This information should be known by 100% of all operators at all times and is of low discriminatory validity.
LOW DISCRIMINATORY VALIDITY The plant is recovering from a scram due to a spurious Group I isolation. The cause of the isolation has been repaired and preparations are being made to reopen the MSIVs. Reactor pressure is currently 825 psig and the main steam lines are being pressurized.
WHICH ONE (1) of the following represents the LOWEST main steam line pressure that will allow the MSIVs to be opened per procedure?
- a. 625 psig
- b. 675 psig*
- c. 725 psig
- d. 775 psig IMPLAUSIBLE DISTRACTORS Which of the following will cause the RHR pumps to start during a design basis LOCA?
- a. Low drywell pressure
- b. High reactor water level
- c. High drywell pressure*
- d. MSIVs in the NOT OPEN position Distractors a, b, and d are implausible distractors considering minimal knowledge of the plant response to a* LOCA.
CONFUSING OR INAPPROPRIATE NEGATIVES IN THE QUESTION Reg?rding temporary plant alterations (TPA), technical reviews are NOT required
- a. for a TPA NOT installed using an approved procedure.
- b. for TPAs installed on BOP systems but are required for safety related systems.
- c. for a TPA that has NOT been directed by the shift supervisor to be an emergency TPA.
- d. for all TPAs directed by the shift supervisor.
This question contains multiple problems: (1) While negative questions can be used, they should be used for good reason; in the above example, there appears to be no good basis for asking this question negatively. (2) Two of the distractors (a and c) also contain a negative, creating a double negative and readability confusions, a violation of good item writing practice.
This question would be more appropriately written as "Under which of the following conditions are technical reviews required?"
- This phrasing would eliminate the negative in the stem.
COLLECTIONS OF TRUE/FALSE STATEMENTS Which of the following are true?
- a. High drywell pressure will auto start the emergency diesel generators.*
- b. Low reactor water level will trip the main turbine.
- c. High reactor pressure will initiate RCIC.
- d. High reactor power with the mode switch in startup will NOT close the MSIVs.
COLLECTIONS OF TRUE/FALSE STATEMENTS Which one of the following describes pump cavitation?
- a.
Vapor bubbles are formed when the enthalpy difference between pump discharge and a pump suction exceeds the latent heat of vaporization.
- b.
Vapor oubbles are formed in the eye of the pump and collapse as they enter higher pressure regions of the pump.*
- c.
Vapor bubbles are produced when the localized pressure exceeds the vapor pressure at the existing temperature.
- d.
Vapor bubbles are discharged from the pump where they impinge on downstream piping and cause a water hammer.
Both examples represent an error of a collection of true/false statements, which' typically only test simple rote memory; the candidate needs only to recall a definition or condition. The question elicits no comprehension or problem-solving; hence, the question lacks operational validity. This type of question allows a candidate to answer the question without the stem of the question.
BACKWARD LOGIC Which of the following parameters will simultaneously start HPCI, RCIC and SBGTS?
- a. High RPV water level
- b. High drywell pressure*
- c. Low RPV water level
- d. Low drywell pressure It is better to select a parameter and then request the expected system response because that
'is more operationally relevant.
BACKWARD LOGIC Backward logic is a question that asks the candidate for information normally received, and provides the candidate with information he/she normally has to supply. In an operational setting, operators are faced with conditions and required to know what procedure(s) to use.
Instead the question asks them to do just the opposite.
BACKWARD LOGIC-an everyday example "If it takes 12.5 cubic feet of concrete to build a square loading pad 6 inches thick, what is the length of one side of the pad?"
This question gives the test takers information they should be asked to calculate, while it requires them to provide information they would be supplied in an actual work situation.