ML17191A610
| ML17191A610 | |
| Person / Time | |
|---|---|
| Site: | Dresden, Byron, Braidwood, Quad Cities, LaSalle  |
| Issue date: | 04/21/1998 |
| From: | Grobe J NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | Kingsley O COMMONWEALTH EDISON CO. |
| References | |
| NUDOCS 9804240367 | |
| Download: ML17191A610 (17) | |
Text
UNITED STATES NUCLEAR AEGULATORY COMMISSION REGION Ill Mr. Oliver D. Kingsley President, Nuclear Generation Group Commonwealth Edison Company ATTN: Regulatory Services Executive Towers West Ill 1400 Opus Place, Suite 500 Downers Grove, IL 60515 801 WARRENVILLE ROAD LISLE, ILUNOIS 60532-4351 April 21. 1998
SUBJECT:
REGION Ill EXAMINATION WRITING WORKSHOP
Dear Mr. Kingsley:
Region Ill staff will conduct an Examination Writing Workshop in the Chicago area on Wednesday, May 27 and Thursday, May 28, 1998. The workshop is designed to inform your management and operations and training staffs of examination writing techniques for the
. licensing of reactor operators and senior reactor operators in accordance with the guidance contained in NUREG 1021, *operator Licensing Examination Standards for Power Reactors.*
I would like to extend an invitation to you and to members of your operations and training staffs, to attend this workshop. Our goal for the workshop is to provide a learning experience for your staff, to provide opportunities to develop and review examination materials, and to provide an opportunity to interact with regional and headquarters operator licensing staffs. The subject audience for this workshop is those members of your staff who will prepare and review your initial examination submittals.
The location for this conference will be the Doubletree Guest Suites, 2111 Butterfield Road,.
DoWners Grove, IL 60515. This hotel is near the intersection of Interstates 88 and 355, which is about 3 miles from the Region Ill offices. A block of rooms have been reserved on May 26 and 27 for attendants at the Doubletree at the rate of $125 plus tax per night. Please confirm your reservations by May 12, 1998, by calling the hotel directly at (630) 971-2o00. In addition, please notify Ms. Mary Ann Bies of my staff at (630) 829-9711 with the names and positions of people who plan to attend the workshop.
Some background information is enclosed which supports our discussions of the fundamentals of question development. We request that ail attendees at the workshop be familiar with this material. This will allow us to quickly move into the question writing workshop by having all attendees at a similar starting point. However, we will allow for some time to provide an overview and answer questions on the material.
)'
- 0. Kingsley 2
A draft agenda is enclosed. We look forward to meeting with you and your staff in May. If you have any questions, any comments on the agenda, or need additional Information, please contact Mr. Melvyn Leach, Chief, Operator Licensing Branch, at (630) 829-9705.
Docket No: 50-456; 50-457 50-454; 50-455 50-237; 50-249 50-373; 50-37 4 50-254; 50-265
Enclosures:
As stated Sincerely, original signed.by M. N. Leach/for John A. Grobe, Director Division of Reactor Safety cc w/encls: T. Tulon, Site Vice President Pistdbytjon:
K. Graesser, Site Vice President M. Heffley, Site Vice President F. Daclmo, Site Vice President E. Kraft, Jr., Site Vice President J. Walker, Training Manager T. Schmidt, Training Manager J. Almon, Training Manager D. Sanchez, Training Manager F. Tsakeres, Training Manager Docket File w/encl PUBLIC IE-01 w/encl SRI Braidwood; Byron; Dresden; LaSalle; Quad Cities w/encl Riii PRR w/encl DOCDESK (E~Mail)
Greens DOCUMENT NAME: G:CECO-SHOP.Ol To receive a co i>Y of this document, Indicate In the box "C" = Copy w/o attac:hlend "E" = Coov w/attac:hlend "N" = No mov OFFICE Rlll:DRS I 4"" Rlll:DRS I.-\\I" I
I NAME Leach/le
~N..(..
Grobe h'""" 4L!.._
DATE 4~/98 4/JJ/98 OFFICIAL RECORD COPY
REGION Ill EXAMINATION WORKSHOP DRAFT AGENDA WEDNESDAY MAY 27 8:00 to 8: 15 a.m.
Introductory remarks Melvyn Leach 8:15 to 10:00 a.m.
Discussion of measurement concepts Dr. George Usova including: validity, levels of knowledge; level of difficulty, & sampling.
10:00 to 10:15 a.m.
Break 1O:15 to 11 :45 a.m.
Review examples of acceptable and Riii Examiners unacceptable test items. Review examples of before and after versions of test items.
11 :45 to 12:45 p.m.
Lunch 12:45 to 2:45 p.m.
Examination question development Riii Examiners workshop. Attendees will prepare questions based on lesson plan material provided.
2:45 to 3:00 p.m.
Break 3:00 to 4:00 p'.m.
Examination question review workshop.
Riii Examiners Attendees will review examination questions developed by others.
4:00 to 5:00 p.m.
Group critique of questions.
Riii Examiners
Jl REGION Ill EXAMINATION WORKSHOP DRAFT AGENDA THURSDAY MAY 28 8:00 to 9:30 a.m.
Group presentations of questions which presented learning opportunities.
9:30 to 9:45 a.m.
Break 10:00 to 11 :45 a.m.
Operating test discussion - dynamic Riii Examiners scenarios and job performance measures.
11 :45 to 12:45 p.m.
Lunch 12:45to1:45 p.m.
Operating test discussion -
Riii Examiners administrative section.
1 :45 to 2:00 p.m.
Break 2:00 to 2:30 p.m.
Operations and Operator Licensing.
A. Bill Beach 2:30 to 3:00 p.m.
Closing remarks and questions.
Melvyn Leach
THE NRC WRITTEN EXAMINATION:
PRINCIPLES AND PRACTICES Dr. George M. Usova USN RC
- Training and Assessment Specialist 301-415-1064 REGION 3 WORKSHOP MAY27, 1998 SESSION OBJECTIVE:
To review validity ooncepts affecting the NRC written examination for the purpose of:
instructing facility personnel toward the construction of more valid and consistent NRC license examinations
- COVERAGE.
3 Levels of Validity 3 Levels of Knowledge Discrimination, Sampling, and Cut Scores Psychometrics VALIDITY A valid test is one which tests what it intends to test.
In training examinations, this means it tests the specific skills and knowledge defined and taught in the objectives.
In licensing examinations, this means it tests the specific skills and knowledge that should have been defined and taught in the objectives.
3 LEVELS OF KNOWLEDGE Bloom's Taxonomy Analysis, Application, Synthesis Comprehension Fundamental (simple memory)
LEVEL OF KNOWLEDGE Bloom's Taxonomy is the reference benchmark NRC uses to classify the levels of knowledge of test items.
Bloom's Taxonomy is a classification scheme that permits the classification of items by the level (depth) of mental thought and performance required to answer the items.
Bloom's Taxonomy conceptually can be applied to written, scenarios, or JPM items.
LEVELS DEFINED LEVEL 1 Fundamental Knowledge testing is defined as a simple mental process that tests the recall or recognition of discrete information bits with concrete referents; examples include knowledge of terminology, definitions, set points, or other specific facts.
LEVEL 2 Comprehension testing involves the mental process of understanding the material through relating it to Its own parts or to some other material; examples can include rephrasing information in different words, describing or recognizing relationships, showing similarities and differences among parts or wholes, recognizing how systems interact, including consequences or implications.
LEVEL 3 Analysis, synthesis, and application testing is a moreactive and product-oriented testing which involves the multi-part mental process of assembling, sorting, or integrating the parts (information bits and their relationships) so that the whole, and the sum of Its parts can be used to: predict an event or outcome, solve a problem, or create something new, i.e., mentally using the knowledge and its meaning to solve problems.
3 LEVELS OF VALIDITY Content Operational Disaiminant CONTENT VALIDITY Essentially addresses KJA coverage and sampling plan coverage OPERATIONAL VALIDITY Addresses two questions:
- 1.
Is the question, as stated, important to be known as a part of the operator's job?
- 2.
Does the question, C;iS expressed, require the candidate to perform a job related mental or physical operation?
DISCRIMINANT VALIDITY Addresses making a distinction of measurement along a continuum of candidate performance.
The art score is the exam performance level that the test yields for making a pass-fail decision.
Since the Cut score is 80 percent, the exam must be written at a level of difficulty that intends to disaiminate at the 80 percent level.
The test item.Its stem and distractor, interplay are sucti that, by intent and design, at least 80 percent of the candidates taking the test should answer the item correctly.
VALIDITY
SUMMARY
- 1.
The exam (item) must be content valid, which encompasses job safety significance and sampling.
- 2.
The exam (Item) should be operationally oriented: a conceivable mental or psychomotor performance of the job. As such the Item should be written at the comprehension or analysis level vice simple fundamental knowledge; this means items that measure problem solving, prediction, and analysis-central to job performance.
- 3.
The exam (item) must discriminate at a moderate level of difficulty as set by the cut score. This means that the stem and dlstractor interplay are such that at least 80 percent of the candidates taking the test should answer the item correctly.
DETERMINANTS OF DISCRIMINATION Level of examination knowledge level of examination difficulty Passing score Item bank use NATURE OF EXAMINATIONS AND TESTS Tests are samples of performance Infer overall performance based upon sample Sample must be broad-based to make confident inference Sample must not be fully predictable otherwise inferences cannot be made on untested areas Items must intend to discriminate otherwise test has little or no value PSYCHOMETRICS
. Items may have one or more of the following psychometric errors:
- 1. Low level of knowledge (fundamental knowledge)
- 2. Low operational validity (not clearly job operational)
- 3. Low discriminatory validity (too easy or too hard)
- 4. Implausible distractors
- 5. Confusing language or ambiguous questions
- 6. Confusing or inappropriate negatives in the question
- 7. Collection of true/false statements
- 8. Backward logic LOW LEVEL OF KNOWLEDGE Which one of the following is powered from 4160 VAC bus 1A?
- a. RHR pump A*
- b. RHRpump B
- c. RHRpumpC
- d. RHRpump D
LOWLEVEL OF KNOWLEDGE Select the full core display indication of a drifting control rod.
- a. Red light*
- b. White light
- c. Blue light
- d. Amber light LOW LEVEL OF KNOWLEDGE Concerning use of water as a fire extinguishing agent, SELECT the correct statement from the following: *
- a. Primary agent for extinguishing Class A fires and also effective on Class Band C fires *
- b. Primary agent for extinguishing Class B fires band also effective on Class A and C fires
- c. Primary agent for extinguishing Class A and B fires but not effective on Class C.fires
- d. Primary agent for extinguishing Class Band C fires but not effective on Class A fires LOW LEVEL OF KNOWLEDGE The following plant conditions exist RCP 2A tripped after running for 50 minutes.
The RCP was resta~. but tripped within 15 seconds.
Which ONE of the following is the minimum required Interval before the next attempt to start RCP2A?
- a. 15 minutes
- b. 30 minutes*
- c. 45 minutes
- d. 60 minutes LOW OPERATIONAL VALIDITY Under which one of the following conditions should the Shift Supervisor inform the shop steward?
- a. Initiation of a directed overtime request
- b. Discipline action on a supervisory personnel
- c. Medical Injury of a contractor personnel
- d. Personnel error by a bargaining unit member*
While this may be related to a SS's job, it has nothing to do with nuclear safety and should not be Included In an NRC examination.
LOW DISCRIMINATORY VALIDITY Which one of the following reactor water levels will initiate the RHR *pumps?
- a. Level 1 only*.
- b. Level 1 and 2 only
- c. Level 1 and 2 and 3 only
- d. Level 6 only This information should be known by 100% of all operators at all times and is of low discriminatory validity..
LOW DISCRIMINATORY VALIDITY
- The plant is recovering from a scram due to a spurious Group I isolation. The cause of the isolation has been repaired and preparations are being made to reopen the MSIVs. Reactor pressure is currently 825 psig and the main steam lines are being pressurized.
WHICH ONE (1) of the followingrepresents the LOWEST main steam line pressure that will allow the MSIVs to be opened per procedure?
- a. 625 psig
- b. 675 psig*
- c. 725 psig
- d. 775 psig IMPLAUSIBLE DISTRACTORS Which of the following will cause the RHR pumps to start during a design basis LOCA?
- a. Low drywall pressure
- b. High reactor water level
- c. High drywall pressure*
- d.
- MSIVs in the NOT OPEN position Distractors a, b, and d are implausible distractors considering minimal knowledge of the plant response to a LOCA.
CONFUSING OR INAPPROPRIATE NEGATIVES IN THE QUESTION Regarding temporary plant alterations (TPA), technical reviews are NOT required
- a. for a TPA NOT installed using an approved procedure.
- b. for TPAs installed on BOP systems but are required for safety related systems.
- c. for a TPA that has NOT been directed by the shift supervisor to be an emergency TPA.
- d. for all TPAs directed by the shift supervisor.
This question contains multiple problems: (1) While negative questions can be used, they should be used for good reason; in the above example, there appears to be no good basis for asking this question negatively. (2) Two of the distractors (a and c) also contain a negative, creating a double negative and readability confusions, a violation of good Item writing practice.
This question would be more appropriately written as "Under which of the following conditions are technical reviews required?" This phrasing would eliminate the negative in the stem.
COLLECTIONS OF TRUE/FALSE STATEMENTS Which of the following are true?
- a. High drywell pressure will auto start the emergency diesel generators.*
- b. Low reactor water level will trip the main turbine.
- c. High reactor pressure will initiate RCIC.
- d. High reactor power with the mode switch in startup will NOT dose the MSIVs.
COLLECTIONS OF TRUE/FALSE STATEMENTS Which one of the follo~ng desaibes pump cavitation?
- a.
b.
- c.
- d.
Vapor bubbles are fonned when the enthalpy difference between pump discharge and a pump suction exceeds the latent heat of vaporization.
Vapor bubbles are fonned in the eye of the pump and collapse as they enter higher pressure regions of the pump.*
Vapor bubbles are produced when the localized pressure exceeds the vapor pressure at the existing temperature.
Vapor bubbles are discharged from the pump where they impinge on downstream piping and cause a water hammer.
Both examples represent an error of a collection of true/false statements, which typically only test simple rote memory; the candidate needs only to recall a definition or condition. The question elicits no comprehension or problem-soMng; hence, the question lacks operational validity. This type of question allows a candidate to answer the question without the stem of the question.
BACKWARD LOGIC Which of the following parameters will simultaneously start HPCI, RCIC and SBGTS?
- a. High. RPV water level
- b. High drywell pressure*
- c. Low RPV water level
- d. Low drywall pressure It is better to select a parameter and then request" the expected system response because that is more operationally relevant.
BACKWARD LOGIC
- Backward logic is a question that asks the candidate for infonnation nonnally received, and provides the candidate with information he/she normally has to supply. In an operational setting, operators are faced with conditions and required to know what procedure(s) to use.
Instead the question asks them to do just the opposite.
BACKWARD LOGIC - an everyday example "If it takes 12.5 cubic feet of concrete to build a square loading pad 6 inches thick, what is the length of one side of the pad?"
This question gives the test takers information they should be asked to calculate, while it requires them to provide infonnation they would be supplied in an actual work situation.
8:00 to 8:15 a.m.
8:15 to 10:00 a.m.
REGION Ill EXAMINATION WORKSHOP AGENDA WEDNESDAY MAY 27 Introductory remarks Discussion of measurement concepts including: validity, levels of knowledge, level of difficulty, & sampling.
10:00 to 10:15 a.m.
Break 10:15 to 11:45 a.m.
11 :45 to 12:45 p.m.
12:45 to 2:45 p.m.
. Review examples of acceptable and unacceptable test items. Review examples of before and after versions of test items.
Lunch Examination question development workshop. Attendees will prepare questions based on lesson plan material
- provided.
2:45 to 3:00 p.m.
Break 3:00 to 4:00 p.m.
4:00 to 5:00 p.m.
Examination question review workshop.
Attendees will review examination questions developed by others.
Group critique of questions.
Melvyn Leach Dr. George Usova Dave Muller&
Mike Bielby Riii Examiners Riii Examiners Riii Examiners
REGION Ill EXAMINATION WORKSHOP AGENDA THURSDAY MAY 28 8:00 to 9:30 a.m.
Group presentations of questions which presented teaming opportunities.
9:30 to 9:45 a.m.
Break 9:45 to 10:45 a.m.
Operating test discussion - dynamic Max Bailey scenarios 10:45 to 11:45 a.m.
Operating test discussion - administrative T.R. Jone~ &
section.
Jim Blis 11:45 to 12:45 p.m.
Lunch 12:45 to 2:00 p.m.
Operating test discussion - Job Dell McNeil Perfonnance Measures.
2:00 to 3:00 p.m.
Closing remarks and questions.
JohnA Grobe
Seminar Evaluation NAME (Optional), ________________ ---------------------
SEMINAR TITLE __________ ~-----~-~-~~--~~~-~~~-~
SEMINARDAT8LOCATION __ ~-~-----~~~---~~----~-------
Please take a few minutes to fill out this seminar evaluation form. Your comments will be used to improve future offer-ings of this seminar.
- 1.
APPRAISAL OF SEMINAR CONTENT:
- a.
How much INTEREST did you have in the subject matter of this seminar?
- b.
How much VALUE did you receive from this seminar?
- c.
Would you recommend this course to others who are interested in the s.ubject area covered by this seminar?
HIGH 10 9
8 10 9
8 10 9
8 7
6 5
7 6
5 7
6 5
4 4
4 3
3 3
LOW 2
2 2
WHY? --------------------------------------
- d.
In many cases it is difficult to find the right mix between theory and practical application: In terms of your needs, how do you evaluate this seminar?
Much Too Theoretical Too Theoretical Properly Balanced Too Practical Much Too Practical
- e.
Based on the stated objectives, did this seminar meet your expectations? -------------
- 2.
SPEAKER EVALUATION:
Please evaluate the Speaker's effectiveness in the following areas:*
- a.
Style and Delivery
- b.
Responsiveness to Participants
- c.
Knowledge of Subject EXCELLENT 10 9
8 10 9
8 10 9
8 7
6 7
6 7. 6 5
5 5
4 4
4 3
3 3
POOR 2
2 2
Other comments on Speaker's effectiveness---------------.,-------------
- d.
Please comment on Speaker's 'use of Audio-Visuals (transparencies, slides, flip charts, etc.), group participation and role playing, seminar materials and handouts
- 3.
SEMINAR EVALUATION:
- a.
What were the MOST effective segments? Why? ----------------------
- b.
What were the LEAST effective segments? Why?-----'-------------------
Practice Activity Please review the following test items.
Identify any psychometric errors you see for each item.
RO QUESTION #050 In FR-H.5, Response to Steam Generator Low Level, AFW flowrate is procedurally restricted to 50,000 lbm/hr when recovering a steam generator level if the level has fallen below 24% wide range indication?
Which ONE of the following indicates why?
A. Minimize thermal stress conditions on steam generator components.
C. Ensure RCS inventory demand does not exceed normal charging pump capacity.
D. Ensure pressurizer level transient does not result in pressure transient that would actuate SI.
ANSWER:
A.
SROX94 KIA X000054K102 OBJECTIVE #0030260S
REFERENCES:
T61.0030.6, LP-#26 SRO