ML20247M677
| ML20247M677 | |
| Person / Time | |
|---|---|
| Site: | Grand Gulf  |
| Issue date: | 05/15/1998 |
| From: | Pellet J NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | Hagan J ENTERGY OPERATIONS, INC. |
| References | |
| NUDOCS 9805260331 | |
| Download: ML20247M677 (16) | |
Text
7 g* "%s,I UNITED ST ATES I
[.,
,. t NUCLEAR REGULATORY COMMISSION R EGION IV S P* '^
,f 611 RY AN PLAZA DRIVE. SUITE 400
./
AR LINGTON. T E XAS 76011 8064 9 *****
May 15,1998 Joseph J. Hagan, Vice President Operations - Grand Gulf Nuclear Station Entergy Operations, Inc.
P.O. Box 756 Port Gibson, Mississippi 39150
SUBJECT:
REGION IV EXAMINATION WRITING WORKSHOP
Dear Mr. Hagan:
Region IV and Office of Nuclear Reactor Regulation staffs will conduct an Examination Writing Workshop in the Region IV Training Conference Room, starting at noon on Wednesday, July 22 through noon on Friday, July 24,1998. The workshop is designed to follow up on our May 1997 workshop. It will convey examination test item review techniques for operator licensing examinations, provide skill-building exercises in the presented techniques, and discuss potential changes in the final version of NUREG-1021, " Operator Licensing Examination Standards for Power Reactors," Revision 8.
You and members of your operations and training staffs are invited to attend this workshop. Our goal for the workshop is to provide a learning experience for your staff, to provide opportunities j
to review examination materials, and to provide an opportunity to interact with regional and i
headquarters operator licensing staffs. The subject audience for this workshop is those members of your staff who will prepare and review your operator licensing examination submittals.
The location for this conference will be the Region IV Training Conference Room at the Region IV office, located at 611 Ryan Plaza Drive, Suite 400, Arlington, Texas,76011. Please notify Ms. Laura Hurley of my staff at (817)860-8253 with the names and positions of your staff who plan to attend the workshop.
Some background information is enclosed, which supports our discussions of the fundamentals of examination question development. We request that all attendees at the workshop familiarize themselves with this material prior to the beginning of the workshop. However, we will allow for some time to provide an overview and answer questions on the material.
I 9805260331 980515 PDR ADOCK 05000416 V
{
a
Engergy Operations, Inc. A draft agenda is enclosed. We look forward to meeting with your staffin July. If you have any questions, any comments on the agenda, or need additionalinformation, please contact me at (817)860-8159.
Sincerely, hM
~
John L. Pellet, Chief Operations Branch s
Division of Reactor Safety
Enclosures:
As stated Docket No.: 50-416 License No.: NPF-29 cc w/ enclosures:
Don Janacek, Director Entergy Operations, Inc.
P.O. Box 756 Port Gibson, Mississippi 39150 Executive Vice President and Chief Operating Officer Entergy Operations, Inc.
P.O. Box 31995 Jackson, Mississippi 39286-1995 Wise, Carter, Child & Caraway P.O. Box 651 Jackson, Mississippi 39205 Winston & Strawn 1400 L Street, N.W. - 12th Floor Washington, D.C. 20005-3502 Sam Mabry, Director Division of Solid Waste Management Mississippi Department of Natural Resources P.O. Box 10385 Jackson, Mississippi 39209
4 Engergy Operations, Inc. President Claibome County Board of Supervisors Port Gibson, Mississippi 39150 General Manager Grand Gulf Nuclear Station Entergy Operations, Inc.
P.O. Box 756 Port Gibson, Mississippi 39150 The Honorable Richard leyoub Attorney General Department of Justice State of Louisiana P.O. Box 94005 Baton Rouge, Louisiana 70804-9005 Office of the Govemor State of Mississippi Jackson, Mississippi 39201 Mike Moore, Attomey General Frank Spencer, Asst. Attomey General State of Mississippi P.O. Box 22947 Jackson, Mississippi 39225 Dr. F. E. Thompson, Jr.
State Health Officer State Board of Health P.O. Box 1700 Jackson, Mississippi 39205 Robert W. Goff, Director State Liaison Officer Division of Radiation Health Mississippi Department of Health P.O. Box 1700 Jackson, Mississippi 39215-1700 i
Vice President Operations Support Entergy Operations, Inc.
P.O. Box 31995 l
Jackson, Mississippi 39286-1995 I
l i
k_____.___________-.___________________._-.-_.._
_.._______________..m
_____.m.____.______m_-
Engergy Operations, Inc. Director, Nuclear Safety and Regulatory Affairs Entergy Operations, !nc.
P.O. Box 756 Port Gibson, Mississippi 39150 i
Vice President, Operations Grand Gulf Nuclear Station Entergy Operations, Inc.
P.O. Box 756 Port Gibson, Mississippi 39150
Engergy Operations, Inc. bec to DCD (IE42) bcc distrib. by RIV:
Regional Administrator Senior Resident inspector (River Bend) l DRP Director DRS-PSB Branch Chief (DRP/F)
MIS System l
Project Engineer (DRP/F)
RIV File Branch Chief (DRP/TSS)
Resident inspector
)
I l
DOCUMENT NAME: R:\\_GG\\GG515lt.imb l
To re:elve copy of document, indicate in box: "C" = Copy wit:. Jut enclosures "E" = Copy with enclosures "N" = No copy RIV:C:OB 6
JLPellet/imbM 054/98 F' '
I OFFICIAL RECORD COPY l
i I
Engergy Operations, Inc. bec to DCD (IE42) bec distrib. by RIV:
Regional Administrator Senior Resident inspector (River Bend)
DRP Director DRS-PSB Branch Chief (DRP/F)
MIS System Project Engineer (DRP/F)
RIV File Branch Chief (DRP/TSS)
Resident inspector l
'I l
l 4
i iti0008 DOCUMENT NAME: R:\\_GG\\GG515lt.imb To receive copy of document, indicate in box: "C" = Copy without enclosures "E" = Copy with enclosures "N" = No copy RIV:C:OB 6
JLPellet/imbM 05 6/98 F"
OFFICIAL RECORD COPY l
ATTACHMENT 1 REGION IV EXAMINATION WORKSHOP DRAFT AGENDA Wednesday 12:00 - 12:15 pm Introductory remarks Ellis Merschoff Arthur Howell John Pellet Wednesday 12:15 - 2:00 pm Discussion of measurement concepts Dr. George Usova including: validity, levels of l
knowledge, level of difficulty, and sampling.
{
Wednesday 2:00 - 2:15 pm Break Wednesday 2:15 - 3:00 pm What is a " direct lookup" question Dr. George Usova and how to identify one during a review.
Wednesday 3:00 - 3:15 pm Break Wednesday 3:15 - 4:15 pm Discussion of proposed Revision 8 John Pellet security and reviewer restriction changes.
Wednesday 4:15 - 4:45 pm Question and answers from the day.
John Pellet Wednesday 4:45 - 5:00 pm Day 1 closing remarks John Pellet Thursday 8:00 - 8:15 am Day 2 Opening Remarks John Pellet Thursday 8:15 - 9:00 am Discussion of " scrutable" examination John Pellet sampling methodology.
Thursday 9:00 - 9:30 am The difference between a " correct" John Pellet answer and a legally defensible, objectively demonstrable answer, Thursday 9:30 - 10:00 am One review methodology for the John Pellet operating test tasks and questions.
Thursday 10:00 - 10:15 am Break Thursday 10:15 - 11:30 am Group review of potential test items.
Thursday 11:30 - 12:30 pm Lunch Thursday 12:30 - 1:30 pm Workshop discussion of tasks and John Pellet questions.
f Thursday 1:30 - 2:00 pm One review methodology for written John Pellet test questions.
Thursday 2:00 - 2:15 pm Break Thursday 2:15 - 4:00 pm Group review of proposed test items.
Thursday 4:00 ~- 4:45 pm Workshop discussion of written test John Pellet items.
Thursday 4:45 - 5:00 pm Day 2 Closing Remarks John Pellet Friday 8:00 - 8:30 am Opening Remarks and questions and John Pellet answers from Day 2.
Friday 8:30 - 10:00 am Proposed changes in finalissue of John Pellet Revision 8 to NUREG-1021.
Applications and eligibility Test development Test administration Post exam process Friday 10:00 - 10:15 am Break Friday 10:15 - 10:30 am Examination scheduling John Pellet Friday 10:30 - 11:15 am Questions and Answers John Pellet Friday 11:15 - 11:30 am Closing Remarks Ellis Merschoff Arthur Howell John Pellet i
l.
ATTACHMENT 2 THE NRC WRITTEN EXAMINATION:
PRINCIPLES AND PRACTICES Dr. George M. Usova USNRC Training and Assessment Specialist 301-415-1064 SESSION OBJECTIVE:
To review validity concepts affecting the NRC written examination for the purpose of:
instructing facility personnel toward the construction of more valid and consistent NRC license examinations COVERAGE e 3 Levels of Validity e 3 Levels of Knowledge o Discrimination, Sampling, and Cut Scores e Psychometrics VALIDITY A valid test is one which tests what it intends to test.
In training examinations, this means it tests the specific skills and knowledge defined and taught in the objectives.
In licensing examinations, this means it tests the specific skills and knowledge that should have been defined and taught in the objectives.
3 LEVELS OF VALIDITY Content l
Operational Discriminant
- I
__._____._m._____
____s_______..._
CONTENT VALIDITY Essentially addresses K/A coverage and sampling plan coverage OPERATIONAL VALIDITY Addresses two questions:
1.
Is the question, as stated, important to be known as a part of the operator's job?
2.
Does the question, as expressed, require the candidate to perform a job related mental or physical operation?
DISCRIMINANT VALIDITY Addresses making a distinction of measurement along a continuum of candidate performance.
The cut score is the exam performance level that the test yields for making a pass-fail decision.
Since the cut score is 80 percent, the exam must be written at a level of difficulty that intends to discriminate at the 80 percent level.
The test item,its stem and distractor, interplay are such that, by intent and design, at least 80 percent of the candidates taking the test should answer the item correctly.
VAllDITY
SUMMARY
1.
The exam (item) must be content valid, which encompasses job safety significance and sampling.
2.
The exam (item) should be operationally oriented: a conceivable mental or psychomotor performance of the job. As such the item should be written at the comprehension or analysis level vice simple fundamental knowledge; this means items that measure problem solving, prediction, and analysis-central to job performance.
3.
The exam (item) must discriminate at a moderate level of difficulty as set by the cut score. This means that the stem and distractor interplay are such that at least 80 percent of the candidates taking the test should answer the item correctly. - _ _ _ _ _ _ _ _ _ _ _
l 3 LEVELS OF KNOWLEDGE Bloom's Taxonomy Analysis, Application, Synthesis Comprehension Fundamental (simple memory)
LEVEL OF KNOWLEDGE B!oom's Taxonomy is the reference benchmark NRC uses to classify the levels of knowledge of test items.
Bloom's Taxonomy is a classification scheme that permits the classification of itemt by the level (depth) of mental thought and performance required to answer the items.
Bloom's Taxonomy conceptually can be applied to written, scenarios, or JPM items.
l LEVELS DEFINED LEVEL 1 Fundamental Knowledge testing is defined as a simple mental process that tests the recall or recognition of discrete information bits with concrete referents; examples include knowledge of terminology, definitions, set points, or other specific facts.
LEVEL 2 Comprehension testing involves the mental process of understanding the material through relating it to its own parts or to some other material; examples can include rephrasing information in different words, describing or recognizing relationships, showing similarities and differences among parts or wholes, recognizing how systems interact, including consequences or implications.
LEVEL 3 Analysis, synthesis, and application testing is a moreactive and product-oriented testing which involves the multi-part mental process of assembling, sorting, or integrating the parts (information bits and their relationships) so that the whole, and the sum of its parts can be used to: predict an event or outcome, solve a problem, or create something new, i.e., mentally using the knowledge and its meaning to solve problems. - _ _ _ _ _ _ _ _ _
d P
l DETERMINANTS OF DISCRIMINATION Level of examination knowledge Level of examination difficulty Passing score
=
Item bank use NATURE OF EXAMINATIONS AND TESTS Tests are samples of performance Infer overall performance based upon sample Sample must be broad-based to make confident inference Sample must not be fully predictable otherwise inferences cannot be made on untested a
areas Items must intend to discriminate otherwise test has little or no value PSYCHOMETRICS Items may have one or more of the following psyt hometric errors:
- 1. Low level of knowledge (fundamental knowledge)
J
- 2. Low operational validity (not clearly job operational)
- 3. Low discriminatory validity (too easy or too hard)
]
- 4. Implausible distractors j
- 5. Confusing language or ambiguous questions
- 6. Confusing or inappropriate negatives in the question
- 7. Collection of true/ false statements
- 8. Backward logic LOW LEVEL OF KNOWLEDGE Which one of the following is powered from 4160 VAC bus 1 A?
- a. RHR pump A*
- b. RHR pump B
- c. RHR pump C
- d. RHR pump D t
LOW LEVEL OF KNOWLEDGE l
Select the full core display indication of a drifting control rod,
- a. Red light *
- b. White light l
- c. Blue light i
- d. Amberlight LOW LEVEL OF KNOWLEDGE Concerning use of water as a fire extinguishing agent, SELECT the correct statement from the j
following:
- a. Primary agent for extinguishing Class A fires and also effective on Class B and C fires *
- b. Primary agent for extinguishing Class B fires band also effective on Class A and C fires r
- c. Primary agent for extinguishing Class A and B fires but not effective on Class C fires d.
Primary agent for extinguishing Class B and C fires but not effective on Class A fires LOW LEVEL OF KNOWLEDGE The following plant conditions exist:
o RCP 2A tripped after running for 50 minutes, o The RCP was restarted, but tripped within 15 seconds.
Which ONE of the following is the minimum required interval before the next attempt to start
]
RCP 2A7 a.15 minutes
- b. 30 minutes *
- c. 45 minutes
- d. 60 minutes LOW OPERATIONAL VALIDITY i
Under which one of the following conditions should the Shift Supervisor inform the shop steward?
- a. Initiation of a directed overtime request
- b. Discipline action on a supervisory personnel
- c. Medicalinjury of a contractor personnel
- d. Personnel error by a bargaining unit member
- While this may be related to a SS's job, it has nothing to do with nuclear safety and should not be included in an NRC examination.
LOW DISCRIMINATORY VALIDITY Which one of the following reactor water levels willinitiate the RHR pumps?
- a. Level 1 only*
- b. Level 1 and 2 only
- c. Level 1 and 2 and 3 only
- d. Level 6 only l
This information should be known by 100% of all operators at all times and is of low l
disenminatory validity.
LOW DISCRIMINATORY VALIDITY The plant is recovering from a scram due to a spurious Group I isolation. The cause of the isolation has been repaired and preparations are being made to reopen the MSIVs. Reactor pressure is currently 825 psig and the main steam lines are being pressurized.
WHICH ONE (1) of the following represents the LOWEST main steam line pressure that will allow the MSIVs to be opened per procedure?
- a. 625 psig
- b. 675 psig*
- c. 725 psig
- d. 775 psig IMPLAUSIBLE DISTRACTORS Which of the following will cause the RHR pumps to start during a design basis LOCA?
- a. Low drywell pressure
- b. High reactor water level
- c. High drywell pressure *
- d. MSIVs in the NOT OPEN position Distractors a, b, and d are implausible distractors considering minimal knowledge of the plant response to a LOCA.
l l I
CONFUSING OR INAPPROPRIATE NEGATIVES IN THE QUESTION Regarding temporary plant alterations (TPA), technical reviews are NOT required
- a. for a TPA NOT installed using an approved procedure.
- b. for TPAs installed on BOP systems but are required for safety related systems.
- c. for a TPA that has NOT been directed by the shift supervisor to be an emergency TPA.
- d. for all TPAs directed by the shift supervisor.
This question contains multiple problems: (1) While negative questions can be used, they should be used for good reason; in the above example, there appears to be no good basis for asking this question negatively. (2) Two of the distractors (a and c) also contain a negative, creating a double negative and readability confusions, a violation of good item writing practice.
l This question would be more appropriately written as "Under which of the following conditions I
are technical reviews required?" This phrasing would eliminate the negative in the stem.
I COLLECTIONS OF TRUE/ FALSE STATEMENTS Which of the following are true?
- a. High drywell pressure will auto start the emergency diesel generators.*
- b. Low reactor water level will trip the main turbina.
- c. High reactor pressure willinitiate RCIC.
- d. High reactor power with the mode switch in startup will NOT close the MSIVs.
COLLECTIONS OF TRUE/ FALSE STATEMENTS Which one of the following describes pump cavitation?
- a. Vapor bubbles are formed when the enthalpy difference between pump discharge and a pump suction exceeds the latent heat of vaporization.
- b. Vapor bubbles are formed in the eye of the pump and collapse as they enter higher pressure regions of the pump.*
- c. Vapor bubbles are produced when the localized pressure exceeds the vapor pressure at the existing temperature.
- d. Vapor bubbles are discharged from the pump where they impinge on downstream piping and cause a water hammer.
Both examples represent an error of a collection of true/ false statements, which typically only test simple rote memory; the candidate needs only to recall a definition or condition. The question elicits no comprehension or problem-solving; hence, the question lacks operational validity. This type of question allows a candidate to answer the question without the stem of the question.
7-
BACKWARD LOGIC Which of the following parameters will simultaneously start HPCI, RCIC and SBGTS?
- a. High RPV water level
- b. High drywell pressure *
- c. Low RPV water level
- d. Low drywell pressure it is better to select a pararreter and then request the expected system response because that is more operationally releve'1.
BACKWARD LOGIC Backward logic is a question that asks the candidate for information normally received, and provides the candidate with information he/she normally has to supply. In an operational s3tting, operators are faced with conditions and required to know what procedure (s) to use. Instead the question asks them to do just the opposite.
BACKWARD LOGIC - an everyday example l
"If it takes 12.5 cubic feet of concrete to build a square loading pad 6 inches thick, what is the length of one side of the pad?"
This question gives the test takers informatbn they should be asked to calculate, while it requires them to provide information they would be supplied in an actual work situation.
l i
l I _ _ _ _ - _ _ - _ - _ _ _ - - _ _ _ _ _ _ _. _ _
__-