IR 05000245/1992032
| ML20127A038 | |
| Person / Time | |
|---|---|
| Site: | Millstone  |
| Issue date: | 12/29/1992 |
| From: | Conte R, Florek D NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML20127A025 | List: |
| References | |
| 50-245-92-32OL, NUDOCS 9301110156 | |
| Download: ML20127A038 (108) | |
Text
,n.,.
~.n....
... - - - - _ -...
- -.
- -.. - - --
- - -. -. - - -
- - _ _
!'
i J
-
.
p
.i U. S. NUCLEAlt itEGULATORY COMMISSION
- '
'
ItEGION 1
'
REPORT NUMilER:
50 245/92-32 (OL)
l DOCKET NUMilER:
50-245 LICENSE NUMilElt:
,
i
,
LICENSEE:
Northeast Nuclear Energy Company P.O Box 270
,
llartford, CT 06141-0270
.
t'
FACILITY:
Millstone Unit 1 Nuclear Power Station INSPECTION DATES:
December 14 - 17, 1992 ACCOMPANIED llY:
J.11. Williams, Sr. Operations Engineer J. S. Stewart. Operations Engineer M. Parrish, EG&G A
n
.
/ 7, 3}
'
LEAD INSPECTOR /
g
~
- ~
EXAMINER:
/
'
Donald Florek, r. 6perations Engineer.
Dat'e
/
BWR Section, perations Ilranch
,
Division of Reactor Safety
'
rt u
/
d9
APPROVED llY:
_
Date
<
Richard J. Conte, Chief llWR Section, Operations 13 ranch Division of Reactor Safety INSPECTION SUMMARY: Examination / Inspection conducted December 14 - 17. 1992 (Inspection Renort 50-245/92-32MOL)
AREAS INSPECTED: Routine announced examination / inspection to administer retake L
requalification examinations to two operators and three senior operators. The inspection reviewed facility licensee actions on previous inspection Endings related to emergency i
{
operating procedures, independent audits of operator training programs and status of selective corrective actions from December 3,1992, meeting to correct performance problems identified during the requaliGeation examination period.
o 9301110156 921231 PDR-ADOCK 0500024S.
l V
PDR E
.
.
..
..
..
.
.
_
.
2 ItFSUl!!'S: Two operators and two senior oix rators passed the examination. One senior operator failed in use of the emergency plan as part of the simulator evaluation. Unresolved item (Ult! 245/89-19-01) regarding emergency operating procedures was closed. A new unresolved item (Ult! 245/92-32-01) was opened for issuance of nowchart based emergency operating procedures. The performance of the operator training audits required by ANSI N18.7 and Technical Speci0 cation 6.5.3.7(b) is unresolved (Ult! 245/92-32-02). The facility lictnsee i: proceeding on with the specine commitments made at the December 3,1992, mec ing. The effectiveness of the licensee actions can only be assessed based on future obst. vations after more data is obtained, e
I
.
_....
_ _ __
_
__
_ _ _
_- _ _.
- _ _ _ _
_ _ _ _. _
.,.
l
-
~
- DETAILS 1.0 INTRODUCTION Examination report 50-245/92-23 (OL) documented the unsatisfactory hiillstone Unit I requalification program evaluation. Following the unsatisfactory program evaluation, a Confirmatory Action letter (CAL) 1-92-014 was issued that restricted those licensed
operators who failed either the NRC or facility requalification examination until either the facility requalification program was determined to be at least provisionally satisfactory or until the licensed operators have been evaluated as successful by the NRC. During the week
,
of December 14, 1992, the NRC staff administered requalification retake examinations to 4 licensed operators (2 ROs and 2 SROs) who had failed NRC administered requalification examinations and one SRO who failed a facility requalification examination. One RO and two SROs were administered a simulator examination, one RO was administered a JPhl examination and one SRO was administered JPhis to evaluate his ability to classify an event.
The JPhi methodology was used in lieu of classification in a dynamic simulator environment.
The examiners used the process and criteria described in NUREG 1021, " Operator Licensing Examiner Standards," Revision 6. A listing of the items used in the examination is contained
-
in Attachment 1.
2.0 SUMMARY OF EXAMINATION 2.1 Individual Examinntion Results
,
NRC GRADING RO SRO TOTAL Pass / Fail
_ Pass / Fail _
Pass / Fail.
Written N/A N/A N/A
'
Simulator 1/0 2/1-3/1 W alk.
1/0
.N/A 1/0 through -
Overall 2/0 2/1 4/1
...--.. -.
.,
. -. ~.
..
-. _-.
-
.
-..
.
..
.. -.-
.
.
i 2.2 Strengths and Weaknesses Based on Individual Operator Performance No generic weakness was notui. The examiners noted that the teamwork of the crew examined in the scenario has signincantly improved from prior examinations, j
2.3 Simulator Remedial Training Program The facility licensee had initially planned on the NRC reexamining two reactor operators on the simulator who had failed the prior NRC administered requalification. Millstone Unit 1 operates with three reactor operators in the RO, BOP and RPO position so an operator who had previously passed an NRC examination would also be used. The two scenarios developed for the examination resulted in critical tasks for the RO and BOP position but no critical tasks for the RPO position. The NRC planned to rotate the two reactor operators into the RO and BOP position and use the third operator in the RPO position that i:ad no critical tasks to perform.
After the facility licensee had observed the validation of the two simulator scenarios, they requested that one of the reactor operators not rotate into the BOP position. The basis for the facility request was that the RO had not practiced in the BOP position during the remediation training program since no weakness was identi0ed for the individual while in the BOP position. The facility licensee indicated that while they had con 0dence that the individual could perform in the BOP position they did not want to subject the individual and crew to an examination situation that they were not prepared for. The other reactor operator being examined operated in all panel positions during the remediation program. The facility licensee requested an additional day of training and evaluation for the crew in the proposed
.
configuration. The NRC examiner indicated that additional training was inappropriate. L On the morning of the examination, the facility licensee elected to substitute another licensed operator from the training staff who worked with the crew during the remediation period to complete the crew. The facility licensee requested that two short 5-10 minute warmup scenarios with the additional substitute operator. The NRC examiner was present at the warmup scenarios and determined that no compromise of the examination occurred.
2.4 Emergency Plan Classincation JPM Results The SRO who was administered JPMs to evaluate his ability to classify an event did not pass the retake examination. The JPM methodology was used in lieu of classification in a dynamic simulator environment. Two JPM scenarios were used. One was selected from the bank and the other was modined from the bank.
Following the examination results, the examiner reviewed the activities of the remediation peri (xi. The SRO had asked for and was provided a copy of the JPM scenarios in the bank
_
(12 scenarios) for emergency classification. Since the facility licensee uses the bank for
'
quarterly take-home-then-send-back-results testing of SROs, which was ongoing, the facility _
Llicensee did not provide the answers to the SRO. The examiner reviewed a cope of the
-
.,
.-
-
.
-
--
-
-. - -
-
. - -.
- -_
_ _ _ _ - _ _ - _ _ _ _ - _ _ _ _ _ -
_
l
-
.
~
SRO's personal bank with the answers that were filled in. Two of the classifications were in error; one of these was selected for the examination. The SRO determined the emergency classincation in the examination to be the same as in his personal copy. At the end of the examination week, the facility licensee had not completed their review to determine if the SRO or training staff filled in the answers for the SRO's personal bank.
The examiner was concerned about the release of the emergency plan classi0 cation bank when only 12 scenarios were available in the bank. The size of the emergency plan classincation bank is not suf6ciently large to release to preclude examination preconditioning of the operators. At the exit meeting, the facility licensee representatives acknowledged that additional controls were needed for the JPMs that contain scenarios for emergency classification and agreed to review this area.
3.0 LICENSEE ACTIONS ON PREVIOUS INSPECTION FINDINGS (Closed) Unresolved item 245/S9-19-01 Adequacy of three technical deviations from the BWR Owners Group Emergency Procedure Guidelines regarding use of the isolation condenser. initiation of primary containment flooding and emereeney depressuri7ation to nrotect primary containment and secondary containment.
Inspection Report 50-245/92-11 (OL) indicated that the additional facility licensee iaformation provided in a letter dated February 10, 1992, required further evaluation. NRC Letter dated July 17, 1992, resolved the outstanding technical issue regarding the three technical deviations and indicated that the related procedure changes will be assessed in ftdure inspections of licensed activities. The NRC inspector reviewed current draft versions of the EOP flowcharts and determined that the procedure changes required were contained in the draft flowcharts. The licensee has indicated in a letter dated December 2,1992, that the implemente. tion of flowchart based EOP will not be completed until the end of July 1993.
Pending issuance of the flowchart EOPs and review by NRC staff, this area is unresolved.
(URI 245/92-32-01)
4.0 INDEPENDENT AUDITS OF OPERATOR TRAINING PROGRAMS The inspector discussed quality assurance and other licensee-performed independent audits of operator training programs with the licensee. The inspector also reviewed how the technical
_ specifications audit requirements of paragraph 6.5.3.7 were being met. The licensee's QA plan commits to ANSI N18.7-1976 which requires audits of programs for training, retraining, qualification and performance of operating staff. Technical Speci0 cation 6.5.3.7(b) requires audits be performed of training of the unit staff at least once per 12 months. In the period of 1986 to 1992, there were only two narrowly focused audits associated with operator training identined. Both audits were performed in 1992. One audit of class attendance was requested by the Training Department and the other dealt with lesson plan modiReation.
. _ _ _ _ _ _ - - _ _ _ _ - _ _ -
-
,
'
This inspection is incomplete because of the operator requalification retake examination etforts. Therefore, more information and discussion with the licensee is needed to resolve the issue of auditing training programs of the unit staff. The performance of the operator training audits required by ANSI N18.7 and Technical Specification 6.5.3.7(b)is unresolved (URI 245/92-32-02).
5.0 MANAGEMENT MEETING ON DECEMilER 3,1992, AND INSPECTION FOLLOW-UP On December 3,1992, a management meeting was held at the Region 1 offices to allow the facility licensee to present the facility analysis of why the 1991 remediation effort failed to prevent the 1992 weaknesses and the corrective actions that the facility licensee will take.
Attachment 3 contains a listing of the individuals in attendance at the meeting. The facility licensee handouts provided at the meeting are contained in Attachment 4. The facility formally submitted the short and long term corrective actions and the status of previously committed long term corrective actions in a letter dater December 7,1992.
During the site visit from December 13 - 17, 1992, the inspector also reviewed and discussed with facility licensee personnel some of the corrective actions identified at the December 3,1992, meeting. The inspector reviewed the facility licensee activities in criterion referenced testing, advanced training classroom, quantifying simulator performance, use of Vermont Yankee simulator, use of split page format for instructional materials, spacing effect, interleaving, instructor crew accountability a'id visualization. The effectiveness of most of the items cannot be assessed until they have been used more and additional information is obtained.
Criterion referenced testing was used to identify generic weakness by testing 25% of the licensed operators using an extensive series of tests. When 30% or more of the operators
.
examined answered incorrectly, a generic weakness.was identified. Electrical, instrumentation logic, recirculation system, isolation condenser and control rod drive were the major generic system weaknesses identified. Purportedly, criterion referenced testing effectively eliminates guessing and encourages operators, in a nonthreatening manner, to admit or demonstrate a knowledge weakness. The facility licensee is also planning to apply the methodology to pre-test areas for future lessons to adjust the training for the specific knowledge weaknesses of operators to be trained. The examiner concluded that criterion referenced testing appeared to be a good tool to provide feedback to the trainers for the areas that operators need remediation taining.
The facility licensee has set up two classrooms using a computer based individual stud' ent feedback system on the material to be covered. The lesson plans have preestablished questions in the lesson to ask the students, as they are being trained, to get prompt feedback on whether the concept is understood by all students. Individual, as well as group, data is obtained. The instructor can easily identify if the students as a group or individually are ~
having difficulty understanding the lesson being taught. Because entire class feedback is
_
.
.
required, student attentiveness is also increased. The examiner concluded that this type of classroom setup also appears to have a high potential to improve student learning.
The facility licensee also has setup a computer based system to try to measure student improvement in the simulator. When the students are being evaluated, instructors quantitatively evaluate communications, analytical skills, technical performance and teamwork at predetermined intervals. Superimposed on the quantitative evaluations are verbal comments of the evaluators as feedback to the individuals being evaluated. This system has recently been installed.
Facility licensee evaluation of the use of the Vermont Yankee simulator as part of teamwork enhancement training has been positive. The training reinforced working together to
-
understand plant response because of the unfamiliarity of the VY control panel and procedures.
The facility licensee is investing resources in using split page training material, spacing and interleaving of training presentations and visualization methodology to enhance student learning and retention. These methods are founded in various research studies to improve student learning and retention. The effects of these methods are pending after more data is obtained on operator performance.
The facility licensee is establishing an instructor / operator accountability by assigning a specific instructor for each crew. The instructor will be responsible for assuring that each member of the crew is performing up to expectations.
The facility licensee is proceeding with the specific commitments made at the December 3,1992, meeting. The effectiveness of the licensee actions can only be assessed based on future obseivations after 'nore data is obtained.
6.0 EXIT MEETING Region I conducted an exit meeting on December 17, 1992. The NRC presented the results of the examinations and discussed inspection related findings as discussed in this report.
Those persons in attendance are noted in Attachment 2.
{
___-___
_ - _ - ___-___ -___-____
. _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _. _ _ _ _ _ _. _ _ _ _
_
.
.
NI'l'ACilN1ENT I ItEQUALIF! CATION EXAhllNATION TEST ITEhlS Jolt PEltFOlth1ANCE AIEASUltE'S USED
Classify an Event Using limergency Plan Implementing Procedures
hianual Start from Diesel lloom
Transfer lleactor Protection System llus A to lleactor Protection hiotor Generator A
Initiate Drywell Cooling
1solate Control lhxl Drive 30-39 113 Standby Liquid Control Pump Operability Test (Faulted)
--
'Cransfer to hiechanical Pressure llegulator at Power
Core Spray Operability Test
Recire Pump Startup
Iteturn isolation Conder -
to Standby
ltestore power to resc <e startup transformer after loss of normal power Sih1ULATOlt SCENA'JOS 1.
Failure of I'.e reactor protection system to automatically initiate a scram with an increasing scram discharge volume water level. The 11 battery fails following the manua' scram resulting in loss of about 50% of the annunciators. An un-isolatable leak in reactor water cleanup then occurs.
2.
Control axi drive pump A is out of service. The diesel generator fails during a surveillance. A loss of power occurs and the control nxis do not insert into the core.
_
The 11 standby liquid control (SLC) pump trips resulting in inability to either manually
_
drive rods or initiate SLC When operators cross connect the gas turbine to the diesel bus, operators will be able to manually drive control rods and inject baron with SLC,
-
- - - - - - -
-
_ _
-
. - -. -
. _ - -
_ _.
.. -. - -.
_
- - _
- _.
=
. -... - - -.
.
i
.
ATTACIIMFNT 2 PERSONS CONTACTED NORTilEAST UTILITIES
- 11. Ilaynes, Director Unit 1
,
- R. Heidecker, Supervisor Operator Training Unit 1 F. Libby, Quality Services Department
- P. Przekop, Operations Manager
- M. Ross, Operations Assistant
- II. Ruth, Manager Operator Training A. Saunders, Quality Services Department NUCLEAR REGULATORY COMMISSION
- D. Florek, Sr. Opciations Engineer
- K. Kolaczyk, Resident inspector
- R. Laufer, Examiner NRR/OLB
- 11. Williams, Sr. Operations Engineer
'
- Denotes those present at the exit meeting on December 17, 1992
I l
l i
!
l
!
1
_ _ _. - -
.... _... _, _ _ _,.. _... _ _.,, -
.
..
.
.
. -
. -,
..
.
l
.
.
ATTACilh1ENT 3
,
PERSONS IN ATTENDANCE AT DECE31BER 3,1992, h1EETING NORTIIEAST UTILITIES M. Black, Director Nuclear Training M. Brown, Project Manager Operator Training Unit 1 H. Hayr Director Unit 1 R. HeidecKer, Supervisor Operator Training Unit 1 P. Przekop, Operations Manager W. Romberg, Vice President Nuclear Operations Services M. Ross, Operations Assistunt M. Wilson, Supervisor Nuclear Licensing NUCLEAR REGULATORY COh1511SSION J. Anderson, Project Manager, NRR L. Bettenhausen, Chief Operations Branch P. Bissett, Sr. Operations Engineer L. Briggs, Sr. Operations Engineer R. Conte, Chief BWR Section.
L. Doerficin, Chief Reactor Project Section 4A D. Florek, Sr. Operations Engineer W. Hodges, Director Division of Reactor Safety T. Martin, Regional Administrator J. Munro, Sr. Operations Engineer, NRR R. Pelton, Training Specialist, NRR C. Sisco, Operations Engineer J. Stewart. Operations Engineer H. Williams, Sr. Operations Engineer
,
=, - -, - - - -
~.,,y ww
-
.
.
.
,
f
,
A'ITACilMENT 4 FACILITY IIANDOUTS AT Tile DECEAlllER 3,1992, MEETING
.
l'
.,i._,,... _, _ _ _. - _,,,,,,. -,... - =. -
.
.
- -, -... - =
. - -.
-.
.--
- - - - -
--
, -,...
.
.
millstone
"
t Unit 1 Nuclear Power E'
-
Station s
Northeast Nuc ear Energy Company Millstone Unit No.1 Licensed Operator Requalification Program Presentation
~
.
i December 3rd 1992
'
<
'
'
_
_
_
-w
_ _ _ _
I 8v3 b
G)
to h
O
'
._
j z
2o E
CO
,
O c
O CC Q)
"
O E
O c
w
.;s
>
O Cd
,
'
cc u
>
Q.
Q)
'
>
Q g
.
o O
>
EI 4mm 1"1 3?Es
_ _--_--_-
.
.
.
'
'
'
millstone Unit 1 Nuclear Power Briefing Cutline Ma.colm Black i
Director Nuclear Training i
!
,
. _ _ _ _.
1 24:e
-
'
NRC BRIEFING
/
-
-
KING OF PRUSSIA
5
.
m///StODO DECEMBER 3RD,1992 Unit i Nuc6*v Prwet g
PROGRAM OUTLINE no#
_
INTRODUG7!ON
'NAYNE ROMBERG
,
BRIEFING OUTL!NE MALCOLM BLACK PROJECT TEAM INTRODUCTION MALCOLM BLACK HARRY HAYNES CURRENT STATUS OF MILLSTONE UNITS 2, 3, AND HADDAM NECK PLANT TRAINING PROGRAMS MALCOLM BLACK EXPLANATION OF WHY THE 1991 REMEDIATION EFFORT FAILED TO PREVENT THE 1992 WEAKNESSES BRAD RUTH TRAINING PROGRAM FAILURE ANALYSES ROBERT HEIDECKER TRAINING STRATEGIES AND PROGRESS REPORT MICHAEL BROWN PARTNERSHIP EFFORT PETE PRZEKOP
STRATEGIC RECOVERY PLAN -
SHORT AND LONG TERM ROBERT HEIDECKER CLOSING WAYNE ROMBERG
l
.
_-
~
'
l millstone w
j Unit 1 Nuclear Power
!
Station
_
g
i The
,
L Project Team
'
.
.
l Introduction
t
!
!
-
-
. -. -.
.
.
-.
.
---
..
.
m w
Malcolm Black - Director Nuclear Training
!
Harry Haynes - Director Vlil stone Unit 1
-\\
.
_ _ _ _ _ _ _ _
.,,,n,--.,_-.--,-,,.w.
,, -..,,...
.
_
.
.
.
millstone m
Unit 1 Nuclear Power Mike Brown Project Position:
Project Manager Recent Assignment: Manager, Technical Training - 6 yrs (managed the initial accreditation of NUs' Technical Training Programs)
Experience:
o l&C Manager, Millstone Unit 3 - 9 years (Design thru commerical operations)
o Head up l&C startup of Millstone 2 CPF - 2 years O Millstone 2 I&C - 3 years (start up)
O Millstone 2 Operations Dept. - 1 year 0 Navy Nuclear Program - 6 years (Reactor Operator)
'
,
, _ _ _ _=
sw
.
- -- -
-
.
.
gg i'
'S millstone uns Unit 1 Nuclear Power
Robert Heidecker Project Position:
Supervisor, Operator Training
.
Recent Assignment:
Supervisor, Operator Training, Haddam Neck past 5 years Experience:
o 4 yrs consultant at Haddam Neck Training o. 6 yrs training / supervisory experience at Westinghouse o
12 yrs Navy Nuclear experience Educational Experience / Licenses:
BS Degree - Nuclear Engineer, Penn State SRO License - Beaver Valley Unit 1 SRO Certified (NRC) - Haddam Neck.
Professional Associations: Representative, BWR/PWR owners group -
simulator complexity / critical tasks
,
- _____
-
-
.
.
>
millstone
,
Unit 1 Nuclear Power Station
-
Burns Mixon Project Position:
Human Performance Specialist
<
Recent Assignment:
Consultant to Washington Public Power and Commonwealth Edison's (NU employee as of 12/15/92) recovery effort Experience:
o Commonwealth Edison & Washington Public Power Human Performance consultant focusing on leadership, teamwork, decision making, and communciation skills - 1989 - present.
o Human Resource consultant to industry other than nuclear-related - 14 years.
o Faculty U.S. Air Force Academy - 7 years (lecturing on leadership
-
topics).
O BA and MA - Florida State University - 1965 and 1971
.
. - _ - -
.
.
.
.
,
millstone Unit 1 clear Power
.
Peter Przekop Project Position:
Operations Manager Recent Assignment:
l&C Manager, Millstone Unit 1 - 7 years Experience:
o SRO licensed MP-1 - 18 years o Unit 1 Duty Officer o New Haven College 1970 BS Physics o Florida State University Physics o Staff Assistant-Millstone Unit 1 Superintendent - 3 years o Staff Assistant-Millstone Station Superintendent - 2 years o Engineering Supervisor, Millstone Unit 1 - 2 years o Engineer, Millstone Unit 1 - 3 years o Maintenance Engineer, Millstone Unit 1 - 3 years l
l
10
.
-
'
.
.
millstone Unit 1 Nuclear Power Mike Ross Project Position:
Operations Department Liaison
,
Recent Assignment:
Operations Assistant past 4 years
.
Experience:
o SRO Licensed o
Millstone Unit 1 Duty Officer o
University of Connecticut BME o
Operations Shift Engineer-2 years o
Unit 1 Engineering (Qualified Reactor Engineer) 5 years
.
-
--
,
e
)
[
'
millstone uniu Nge;r P=er
,
,
p,
_
CURRENT STATUS
,
OF
.
MILLSTONE UNITS 2,3 AND HADDAM NECK REQUALIFICATION. TRAINING PROGRAMS
,
l
%
Emses-
-
.
.
.
.
mdistone MU umog;erPer MILLSTONE UNITS 2,3 AND HADDAM NE i
REQUALIFICATION PROGRAM ASSESSMENTS o
NRC REQUALIFICATION EXAMINATIONS-49 OF 50 PASSED (1991 TO PRESENT)
o NRC REQUALIFICATION TRAINING INSPECTION-MEET 10CRF55 REQUIREMENTS o
NU ASSESSMENTS USING NUREG - 1220 CRITERIA-QUALITY' ASSESSED AS SATISFA.CTORY i
NU BIENNIAL ACCREDIATION SELF ASSESSMENT
-PROGRAMS MEET ACCREDITATION OBJECTIVES
_
..
..
.
,,
,
. _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _
-. _
. _ -
il millstone
,
Unit 1 Nuclear Power Station
_
CURRENT STATUS
.
o MILLSTONE UNITS 2,3 AND HADDAM NECK REQUALIFICATION PROGRAMS ARE SATISFACTORY o SOME ACTIONS ARE NECESSARY
~
.
.
.
.
A
'
$#Mocog NU ACTIONS an, Station
.
IN PROGRESS
'
o PERFORMANCE ENHANCEMENT PROGRAM
- INCREASE AND RETAIN QUALIFIED STAFF
- STAFF AUGMENTED WITH CONTRACT INSTRUCTORS
- TRAINING MATERIAL IMPROVEMENTS
- STAFFING CONTINGENCY FOR UNPLANNED ATTRITION SITUATIONS o TRAINING MANAGEMENT SYSTEM UPGRADE o OPERATOR TRAINING BRANCH ADMINISTRATIVE PROCEDUREIMPROVEMENTS
'
<
PLANNED o EVALUATE AND IMPLEMENT A MORE RIGOROUS
.SELF ASSESSMENT PROCESS
-'
-
-
.
.'
.
.
I millstone
,
,
uniu g;ie r Puer
_
j WHY THE 1991 REQUALIFICATION 3ROGRAM AND CORRECTIVE ACTIONS FAILED TO PREVENT 992 WEAKNESSES
,
- - - - - - - _.
_ _ _ _ _ - - - _ _ _ - _ -. _ _ _ _ _ _ - _ _ _ _ _ _ _ - _ _ _ - _ _ _
_
_
.
.
.
,
ga%g
[
millstone
"" "": ' " ~"
Ypd 1992 REQUALIFICATION OPERATOR PERFORMANCE SUMMARY 1991 ASSESSMENTS 1992 ACTIONS RESULTS STRENGTHS ADVANCED SCENARIOS INCORPORATED WRITTEN EX AMIN ATION WRITTEN EX AMIN ATION PERFORMANCE OPERATIONS AND STAFF PERFORMANCE DECLINED CREWS TRAINED TO NEW JPM PERFORMANCE COMMAND AND CONTROL JPM PERFORMANCE DECLINED STANDARDS EOP IMPLEMENTATION AND WEAKNESSES MONITORING OF KEY PARAMETERS STRESSED
~
SRO COMMAND, CONTROL, CREW COMMAND, CONTROL AND COMMUNICATIONS LICENSE HOLDERS FAILING AND COMMUNICATIONS 1991 EXAM REMEDIATED AND IMPROVED
CREW TEAMWORK RE-EXAMINED EOP IMPLEMENTATION ADDITIONAL TEAMWORK SPECIFIC EOP WEAKNESSES TRA!NING CORRECTED CREW MONITORING OF KEY FREQUENCY OF CREW OVERALL CREW ATTENTION TO
-
PARAMETERS EVALUATIONS INCREASED
'*llTICAL PARAMETERS AND SAFETY SYSTFM ACTUATIONS SIMULATOR CHANGES IMPROVED IMPLEMENTED
-
.
-_
. - - _
'
~
.
.
II
- "%g
-
millstone Unit 1 Nuclear Power Station 1992 REQUALIFICATION TRAINING PROCESS SUMMARY 1991 ASSESSMENTS 1992 ACTIONS RESULTS WEAKNESSES SIMULATOR EV ALU ATION EVALUATORS OBJECTIVE TOOLS REVISED TO REFLECT EVALUATOR OBJECTIVITY NEW STANDARDS JPM MEASUREMENT STANDARDS ACCEPTABLE EXAMINATION MATERIALS EVALUATION TEAM OBSERVED AND FEEDBACK PROVIDED JPM QUESTIONS REQUIRED REVISION FOR EXAMINATION ADVANCED SCENARIOS INCLUDED IN EXAMINATION REVISED OPEN REFERENCE BANK QUESTIONS NOT WRITTEN TO HIGHER COGNITIVE LEVELS JPM MEASUREMENT STANDARDS REVISED JPM PROCESS PROBLEMS WERE IDENTIFIED JPM QUESTIONS NOT ADDRESSED OPEN REFERENCE QUESTIONS REVISED
,
.
.
.
millstone m
Unit 1 Nuclear Power
" " " "
__
CAUSES FOR 1992 WEAKNESSES o
NU SELF ASSESSMENT FOCUSED ON OBSERVED WEAKNESSES o
LOWER COGNITIVE LEVEL OF EXAMINATION BANK ITEMS FAILED TO EXPOSE EXISTING OPERATOR KNOWLEDGE DEFICIENCIES o
1992.REQUALIFICATION TRAINING CENTERED ON THE SIMULATOR
'
EMPHASIZING " CREW SKILLS" AND "EOP IMPLEMENTATION SKILLS" o
JPM TRAINING WAS NOT FORMALLY SCHEDULED INTO THE REQUALIFICATION TRAINING PROGRAM o
EXTENDED PERIODS WITH REDUCED NUMBERS OF TRAINING STAFF AND LOW EXPERIENCE LEVELS CONTRIBUTED TO PROGRAM DEGRADATION
-
- - -
-
-
-
.
.
esee
-
"
ANNUAL EXAMINATION RESULTS Simulator Examinations o 86% pass rate o One crew failure o Three operators failed due to E-plan classification (2 CO's & 1 SCO)
.
o Overall crew performance improved
.
.
.
millstone
,
Unit 1 Nuclear Power Station EXAMINATION RESULTS JPM Examinations o 97.6% pass rate o 19 of 430 JPMs were failed o 4 JPMs were missed twice o One individual failed (1 RO)
-
I
-
,
,
millstone
,
Unit 1 Nuclear Power
_
Station EXAM l\\ ATION RESULTS l
Written Examinations
o 84% pass rate o 7 individuals failed (4 ROs,1 SS, 2 Staff SRO)
,
o Marginal performance compared to other
,
units / industry o 65% of scores were greater than 85%
(average score = 86.8)
i
.,_
__-
_
.
millstone E
g uniu nucie r puer SELF ASSESSMENT PROCESS o ROOT CAUSE ANALYSIS o EXAM ASSESSMENTS
.
o CONFIDENCE WEIGHTED TEST
.
.
,,_ s millstone
\\
m
""'" 'fll~ " ~
Se f Assessment Analysis
-
Simulator Weaknesses o Simulator training focused primarily at SRO level o Limited focus on EOPs versus tasks and system knowledge o'E-plan classifications not reinforced at crew level
o Exercise guides did not contain instructional activities / strategies
-
o Exercise guides did not contain same standards used during
-
evaluations o, Simulator time was focused on the EVALUATION mode versus TRAINING mode I
,
- - - _
,
-
e
,
..
millstone m
__
Unit 1 Nuclear Power Station Se " Assessment Analysis _
JP VI Wea<nesses-o JPMs not formally scheduled into training program o Individuals failed to maintain their own proficiency o Time critical JPMs were not trained on o Several JPM failures attributed to lack of self checking o Operators improperly processed initiating cues or initial conditions
_
,,
- - -_____________-_ -.
_-__ _
_ -_____
.
,
millstone m
Unit 1 Nuclear Power Self Assessmen~: Analysis
_
" ' " " "
Written Weakne-sses o Exam bank did not adequately assess higher cognitive knowledge levels o 40% of exam items tested objectives that were outside the two year sample plan
.
o Operators had difficulty processing information due to systems / procedure knowledge deficiencies
.
-
..
'
~
ewe
'w Programmatic Weaknesses o Selec~:ec systems knowlecge o Selectec EOP/ONPs o E-3:an c assification "or non SS/SCO wa:chstanc er l
l l
!
-
'
'
.
.
ll p uns
,
millstone m
Unit 1 Nuclear Power
.
Station Se f Assessment Analysis Process Wea<nesses o JTA and associated training materials being updated o Program not. designed with proper emphasis in all environments
...
.
,,
- 30
@
l
="W!"!
%zb Project Goal
.
'h To e eva:e
- le NiIstone Unit -
!.
0 aerator Training 3rograms-i rom their current leve of margina acceatability to that
-
o' an incustry eacer
.
-.
..
'
'
v9?!M2*
&
'
'
Station Strategies to Strengthen:
'
l l
0 The Training Process l
o The Individual Knowledge Base o Performance Skills l;
o Teamwork o JPM Im alementation
~
o Program Evaluation
.
.~....g..
.
.
.
.
millstone
\\
m Unit 1 Nuclear Power
_
Station g
The Training Process o Reconstructing the Job Analysis for all MP-1 Operator positions o Reconfiguring tle objectives at
.
the a33roariate cognitive evels o U3 gracing all MP-1 Lesson Plans that have programmatic c eficiencies l
,
.
)
I'
'
millstone
,
Unit 1 I clear Power
_
The Training Process
.
o U3 grading the exam bank to the appropriate cognitive level and congruent with the revisec objectives, using the NRC
reviewed mocel 0 Reformatting all student text in a scholar text ormat to i
enlance learning
~'
o Uogracing stucent anc teaching text graahics to enhance the isual learning element v
.
- _ _ _ _
-
-
-
- -
--
..
.
r i
__
A
'
millstone Q
""'" :::::"~
The Individual Knowledge Base )
o Caaitalize on acvancec learning research concucted at NU-Advanced Training Classroom-Accelerated Learning-Examinations as a Preservative of Knowledge l-Harvard (Medical School) study groups " Pathways to the Future"
!
-Interleafing l-Spacing Effect
!
-Introduction of flow charts, job aids, and mental cueing to enhance student l
retention l-Upgrade of the visual component of the student text and classroon)
'
presentation materials
l
..
.
'
'\\
mees v.
Station gg The Individual Knowledge Base o Dr. Robert Blork wiI consult mont11y on
'
'
the ef"ectiveness of the learning strategies employec l
0 Ex3anced use o" interactive video ins *:ruc: ion o introduction o" designer software to teach circuit ogics
-
l
.
.
.
.
millstone neu Unit 1 Nuclear Power
___
Station
,
Performance Skills o Enhance Simulator Exercises
!
-Greater use of freeze and video critquing techniques-Improved facilitation by the
-
Operations Management-Implementation of Rev. 7 (a crew success concept)
-New EOP flow charts
..
.
..
.
.
=-_
-
-
,
e e
'
'
e w
Performance Skills o implement the use of mental imagery o Develop a model to measure individual and crew performance and improvements over time o improve OJT guices o New barrier failure table training aid o Expanded use of Technical Training labs
-
and instructors o The use of non plant-specific simulators to evaluate crew analytical skills
,
'
a
..
.
millstone
,
Unit 1 Nuclear Power
_
Station Teamwork I
o Acci:iona
- eams<i s vraininc, ao:lincivicua anc Crew o DCreased em31 asis on Crew evalua: ions on :le simu ator o Adcition o" a -uman 3er"ormance S3eciais: :o :le s"a=
,
l
,
..
_ _ _...
.
.
,
,
,
..
I i
egrez w
< M m: eTierra':im
o Upg rac e o J 3 V's o <oin~: uni: eva uations o incivicuals Jerorming < 3V's o marovec me: lod o ima emen":ing < 3V's ' rom a
~
3rocess 3ers3ec:ive}
l L
.
w T
-
'
.
.
/per millstone
\\
Proaram Evaluation m
.
'
u-- r-
-
Station
-
g o introduction of confidence weig 7ted testing o The use of a curriculum cevelopment coordinator to monitor t7e quality of curriculum improvements o Preparation of an INPO self evaluation report o Programs will undergo reaccreditation-first quarter.1994 o NRC initial and requal examinations-1993 o Evaluation team of various Region 1 utiities o Requested-INPO Assist Visit - 1993
-
39.
.
-
.
- -----
--
..
_.
?
'
$#8! 08 (
+
"""
Improvements to Date o Upgrade of the \\lLO 1: ext o Acvance training classroom o New mocel to set proper cognitive level o New barrier failure :able training aic
'
o Technica: Trainers anc! l abs in1:o the opera:or programs o Confidence weighted testing o Revisec JTA and objectives (in arog ress)
-
-
-
^
.
.
millstone
'
man Unit 1 Nuclear Power Station
--
Improvements to Date o improved visuals (in progress)
L o Curriculum development coordinator o Accelerated learning techniques o Programmatic review of JPMs and OJT guides o Diagnostic evaluations performed on the individuals who
~
failed this past year
.
o Actively pursuing job offers to fili vacant positions t
t
,
B
i
.
- - - - - - -
- -
- - - - - - - -
- - -
_.
.
-
.
ll p~m i
s millstone nu
)
Unit 1 Nuclear Power
Partnership o Training initiatives and improvements are only part of the equation
!
.
~
o Success requires Operations Department input, proactive
support, and commitment
.i
,
,
--
-- _ _
.
_
.
.
,
'
!
\\
millstone i
"" "l.'2" """
%fp/
Expectations o Training programs must reflect Operation philosophy
-
o Standards of operations will be consistent in the Unit and y
Training environments
.
.
i I
i r
-
..
A
~
millstone seu Unit 1 Nuclear Power commitment o Unit Operations will be an integral part of the project initiatives
!
'
,
'
- - - - - _
,,
.
'
f {8f 98 Strategic Recovery Plan
\\
'
Short Term o Pro lect team out in alace o Remecia: ion plans
-
o Train on 4 common JPMs o Enhance system knowiecge on generic weaknesses (irst cycle 93?
o Use o~ acvanced c assroom training
.
(first cycle,1993)
,
i
-
,
,
-
-
millstone m
Unit 1 Nuclear Power
-
Strategic Recovery Plan
="
Short Term o Reinforced in plant knowledge with JPM training (first cycle 93)
o Use of confidence weighted testing results to design future cycles o Train all operators on SOER 92-1 - Self checking (first cycle 93)
l.
- _ _
.
-
.
~
mittstone
,
""'" ::;;--
Strategic Recovery Plan
-
Long Term o Project team to remain in place through the recovery effort o Exam materials upgraded in accordance with NUREG 1021 requirements
.
o Training schedule will incorporate systems training based on revised Job Analysis information o JPMs will be structured in program over two year
~
cycle based on revised Job Analysis information
'
-
.
millstone m
Unit 1 Nuclear Power Station
_
Strateg,c Recovery Plan i
Long Term o Simulator exercise guides will include additional instructional strategies and standards o Periodic independent review of LORT program
.
o Assess all operators in third quarter of 1993 to determine effectiveness of pilot effort
.
,
o EOP flow charts trained on prior to July 1,1993
'
.
.
-
- -
_
,
+
.
n Exams and Use as Preservatives of Course-Acquired Knowledge
,
_
E'
THOM AS K. LANDAU! R L4TitLEEN 1. AtNSUE
"
bed Latwaunes ABSTRACT Resencon cf materiallearned in couege-style exhnkal courses eas meemetod by a repested naal examinat6on sivun one yest ister.
Matched groups cf students also repestr4 the Anal entminonon sin weeks or sin months artes the end of the courses. The at month group showed almost so has over the year. The sts week group suftered about the some conseeerable toes as did a control group with no netemn6ag esarrunstion. Ovestionnaire measures indkated that studenu who e,asd over 10 percent or the esamined materislin their I
everty montMy work also forgot eery Utoe. Arpervacy, cowtsestugu4 red knowledes een be hart fresh over long pertods by small j
smounts or property syeced rev6ew activities.
THERE IS WIDESPREAD concern that knowledge and two4emester schedule. Classes meet about two hours per I
skill acquired through college style courses are too often week for 16 weeks during ordinary work hours, and stu.
lost before they can effecuvely be put to use by the student.
dents are given homework assignments intended to re.
T1usi a report of an investigation of the use and retendon quire about four hours per week. Instructors are Ph.D..
of knowledge from some advanced technical counes over kvel rewarch scientists and engineers.The courses have a period of one yest, with particular attention to the pos 1 final exams, and while there is no formal grading, succeuful ble loss prewnting funcuan of retaking examinations after completion of a course is recorded in the employee's per.
a course is ont.
maner,t record. De students art au technical employees of The initial motiw for the investigadon was to see whe.
Beu 1.aboratories.There is a wide range of educadonal a.id ther a simple, easy.to use technique could be developed employment history and ages among the students. Alinost that would help itudents to retain more of their new knowl.
allitudents haw at least bachelors degrees, and many hold edge over long periods. A substantial body ofliterature in masters degrees or doctcestes.
educational and leaming psychology demonstrates that Prior to the spring semester of 1971. instructors were re.
tests, recitation, and actin practice in general, are efrective cruited for participation in the study. Rn instructors vol.
In reducir.3 the rate of decay of knowledge and skill (e.g.,
untured to do the considerable extra administrauon and I,2..: 4,6,7,8). However, the efrects of tests on toten.
examinadon scoring required. Students in their counes tion over periods of pracucal significance-months and were asked to participate on a voluntary basis.Participa.
,
yeen rather than days or weeks-do not wem to han bun tion required that students take. in addition to the ordinary m
studjed.1)kewise, tesu or other review activities at delays end.of course examination, another examination a year of months han not been studied. College. type courses are later and in most cases. still another exam sometime be.
often aimed at applications at least a year after their com.
tween.The students were informed as to the nature of the pletion.The purpose of this study was to attack directly experirnent. Howewr, they did not know whether they such very long term retention and methods ofimproving would be a6ked to take the interim exam, or whether they it under practical constraints.The proposed technique for woukt be in the control group that faced only an exam a providing active practice, and at the same time measuring year later. Dey did not know when interim ex sms would
-
the retention of course engendered knowledge our long be stven. Approximately one-third of the students in these periods, wu as fouows. Students took the usual end of.
five courus volunteered to parucipate, and retumed in a coune examinadon.Then, at a later time, they were siwn timely fashion au of the examinations which they were an opportuntry to take the identical examination slain, as asked to take.The main data from the experiment upon a " refresher."
wtdch results are to be reported were obtained from $$
In order to measure the long term effectiwness of the students, technique, U students were asked to taka a terminal exam.
De design of this study allows the following questions instion-afsm idendcal-a year after the course was over, to be explored. First, how do the knowledge and skills ac.
In addidon, a questionnaire was administered to estimate quired in such courses last owr the succeeding ye ar? See:
how much och student used material from his course.
ond, how much use is made of the information gained The courses studied were part of the Beu 1.aboratories from ouch a course over t}w next yeart nird is the me.
In Hours Continuing Educauon program. This is a large.
terial better retained if it is used, as is commonly assumea, scale program offering postgraduate counes in a variety and if so, to what extent? Fourth,1f students take addi.
of fields ofidence, mathematics and engineering.The tional exams after the end of the coune,does this aid re.
courws are taught in standard college style fonnats in a tention? If so, doesit matter when the exama are takent
.
._
~-.6.yne
_
_
... _ _. _ _. _ -. - - - - - - - - - - - - - - - " - - ~ " ' - - "- - - - - -"-"~
~ - - - - ~ -
--.
.
_-
. _ _. - -
-
.
--.
.
. _
,
s
.
300 Tile JOURNAL OF EDUCATIONAL RESEARCll TotJe I.-Numtmer or studente assesned (sotal) and who retwned comp 6ese (good) data.
.
-
6 weeks Group 6 MontasGroup Comuol Geoup TM N
Good Good Good Good Course Studeata Total Dsta Total Dsta Total Data Data Random Pronne:
1
1 I
I I
Phrucal Destan
3
3
5
3 Derul $ystem Desisn
3
3
3
i PL t Prteramming Lariguage
to
10
$
$
Asiewtly Laeruage
27
$
il
il
TOTAL 3 145
14 a6
46
$5 j
l Ttus paper will present results relevant to all of these keep track of the students, but identities were not avail.
problerm, along with data analyses by which some poten-able to the instructors who later scored them.
iial biases arising from the less than-perfect participation Approximately two weeks after the examinations, are dealt with. Finally,it discusses what conclusions rnay foUow up letters were sent to those who had not yet re.
i reasonably be based on the results.
tumed them. The number of examinations returned by
!
memben of the various experimental groups and classes is Design shovm in Table 1. The proportion of exams returned was Titjes of the five courus involved, along with the num, about the same for the control and six month groups, and ben of students who completed vanous parts of the study, somewhat lower for the six week group.The total number are shown in Table 1. A total of 145 students took exam.
of complete seta of data was too smsU to suow separate instions at the end of the various counes.These examina, analyses by courses;in order to obtain interpretable data, tions were scored for each class by its instructor, and results from all of the courses were combined.Thusit is members of each class were ranked in terms of their difficult to characterire predsely the kind of course to scores. The students in eoch class were then divided into which the results apply. However, since weu over half of matched triplets (the three best students, the next three, the students were in programming courses, and the rat etc.)on the basia of these ranks. From each triplet, one
"re in technical or mathematical courses, there la a cer.
student was picked randomly to be in each of thrn expert.
tain common character to the material, mental groups. Where the number in the class was not even.
All of the Interim snd Terminal examinations were col.
ly divisible by three, the extra students were not assigned lected before any of them was scored. After they were all to experimental groups. The three experimental groups.
returned,in August 1972, the examinations were returned were: a control group that received no examination until to the originalinstructors for scoring. Instructors did not one year later; a six week group that receiwd an examina, know which student had provided which exam paper, nor tion six weeks after the end of the course and another from which of the examination periods any exam had after a year; and a six-rnonth group that received an examin.
come. '!he instructors wre asked to review their scoring
ation six months after the end of the couras and again after methods from the previous par and to fouow them as a year.
closely U poseble.They were asked to score aD of the Table 2 summarizes the daign.The end.of. course exam exstrinations together and to try to maintain consistent is teferred Io as the Initial examination, the examinations criteria.
,
occurring at six weeks or six months will be referred to as Scores from the initial examination alone were used to Interim examinations, aad the examination occurring after establish a common scale.The Initial scores for each class a year will be called the Temunal exandnation. Note that wre first transformed to : scores.This aDowed combining au students received an initial examination in June 1971.
of xores actoss clasers. Raw scores for the Interim and All studenu also receiwd a Terminal exandnation in July Terminal exarninations were converted to the common scale 1972. Those in the six week and six.rnonth groupe received by subtracting the class mean for the initial examination, Interim examinationa in July 1971 or January 1972, and then dividing by the standard deviation of the initial respectively.
exam scores. For purposes of estabitshing the scale, all-Methods and Procedures The examinations were sent by mail to each student at Taw 2.-Deelen of Itetention study the appropriate time. The examinations, which were Xerox talaal Cm M h""
Terminal copics of the original extrns takan by the students, wre l'inal han ham accompanied by a cover letter requating cooperation and Osee 71) Unh 71) tJammary 72) Ush 72)
instructing the student to take the exam!nstion in the same control n.anner as had been required at the end of the c' urse (all Group X
X-o courses used take home exams).The students were assured 6-week that their results would be kept anonymous, and they Gmp X
X X
wre asked not to put their naines on the examination g%'ath X
X X
sheets. A code number on each examination was used to
_
_
_
_
_..
. __ _
_.
_ _. _ _ _ _.. _
. _ _
_ _ _ _ _ _ _ _..~.
._
..m
-
e
,
lANDAUER-AINSt.IE 101 fsgure 1.- Ayerste etam acotes over a one year period foteowi.g technscal courses. All scores are hased on standar4 (5D) usu ts dertved itone oriesal endof<ourse final eassns (INIT), and an adjusted io equal saarnng lents. The overga are foe 6ndependent groups of studenes who took a sinn esam nrst at the end of the course. sad next after the tone inservalladhisted. Dets an from flee sea Lab-orsionies Coatinuing Educetion tourses Srven in spring 1971.
Gt
8
$
-
O h
-
W
.I
'
i e
i INIT 6 WKS 6 MOS I YR EXAM
.
scores for the Irdtlal examinations were used, including knowledge owr a year in the absence ofintervention. As those of students who failed to take one or more of the can be seen, the mean scores first go up and then decline -
succeeding exams. This was done in order to provide as owr the year: 041,0.72,-D3,
.73. At six weeks the stable md meaninsful a scale as possible.
scores were significantly higher (t,, a 3.03, p <. 01) than One quntion was whether the three experimental they wre on the Initial exam, probably because the stu.
grops were similar at the outset.The Initial exam scores dents found answers to qundons they had not known, for students with complete data in the six wek, six month Howowr, by six months the swraos student's scores were and control group wre.37 D3 and
.06, respectinly, at the same lowl or slightly less than they were on the These means do not differ from each other slgnificantly Irdtlal exam. By a year the average student had lort a sig.
(F,s = 0.91). Howwr, for further analyals, an additional nificant.73 of a standard deviation unit of what he had adjustment was made in the acores no as to giw all groups originaEy learned (ts, = 2.35, p < 43). Some additional equal starting lewls. From each of the interim and Tor.
intuitin meaning can be gained by considering what such a minal : ecores was subtracted the initial s ocore for the loas meant for students in a typecal couros. For exarnple,in same student, orw of the courses there was an exam grading scale of 200 Sowrsl weeks aftse the Terminal examination, a ques.
painu, for which the initial mean was 117 and the atandard tionnaire was sent out in order to obtain information about deviation 30.8.The range of scores was from 44 to 178.
the amount of use that had been made of the material ow:
Thus, an awrage student would have slipped from an initial the preceding year.The questionnaire seked the students raw score of 117 to 94 by a year later. Stated differently, to estimete for each month between the end of the course the loss owr a year took the average student from about-and the Terminal examination the percentage of the meter.
the 50th percentile on the Initial exam to a score corres.
1:1 covered on the end.of course exandnation that had been ponding approximately to the 23rd percentile on that used at least once during that month. A calendar was pro.
exam.
vided in which they were to wdte their estimated percent.
Conaldering the effect of the Interim examination, Fig.
ages. Eighty seven questionnaires were returned, ure 2 shows the losses our the whole year for students in the control roup, the six week interim enam group, and G
Results the six month intedm exarn group. Losses for the six week Figure I shon the mean adjusted exam scores for the and control groups wre almost identical; the loss for the six. week exam, the six month exam, and ths Termir.al six month group was very little and significantly less than exam. Note thet the Terminal eram score rnean is for the that of the control group (tn = 1.86, p <.05, one. tailed).
control group ordy. Thus, this figure reflecu change in That the effect of a test is greeter if the test is delayed
-
.-
-
.
.
103 Tif E JOURNAL OF EDUCA710NAL RESEARCH ligere 2.-Det bne in eJpated enam scores o,w e one year pered for independent groups having an,aservening esam (contr<d) or havins aa interiesing easm afsee es ther as wwks or as montho (Veragallanes indicate standsid error or means)
-
$
-
I
-.
GMOS-GROUP l*
6 WK1
.
GROUP
CONTROL GROUP
'I END OF COURSE 1 EAR EXAM i
rather than given immediately has precedent in experi.
The three sets of students differing on estimated usage menis on paired 4ssociated learning in which a non monotonic did not differ systematically or significantly in their Initial efrect of delay of testing has been observed (2,3). Whether test scores, having means of
.05. 43, and.09, respectiwly, the some explanadon will serve both for one. minute delays for the none,less4han.10 percent and more4han.10 per.
of tests on palted associates as studiod by Landauer (2,3),
cent groups (Fage = 031). Thus, differences in initial -
,.
and for six month deleys of course tests is an open ques-tion. If the explanation is that tests as studied here do most good when the S has forgotten some,but not all of what Figure 3.-8sbisc$se esehneses of proport6am of enseerial sessed on end ru M exams used La e 12 seem months he once knew, then the two phenomena may well be related.
Figure 3 shows the estimated awrage amount of the material covered on exams which Se used in various months o
after the end of the courses. The low range of these esti.
mates seems somewhat surprising. Almost half of the stu.
-
dents asserted that they have newt used any of the meta'*
tal cowred on the examinations.We doubt that this is be.
utAN WONTHLv USA 0c -
or exam uartmat, cause the examinations did not cour the important ele, toyvt pl.
pwtato ments of the course, since both the Instructors and the nn.
vestigators were convinced that the examinations used in
the courses were quite comprehensbe and reflected a good g
job of choosing representative and important material.
g 80
~
The studenu who had both takaa Terminal examinations and completed questionnaires were divided into three rough.
E ly equal-stud groups on the basis of the amount of use
'g-they estimated they had made of the course material,
averaged over the whole year.The thrw groups were:(1)
21 who reported that they had never used any of the ma.
tertal;(2) 15 who estimated they used an awrage oflese i o'
i than 10 percent-for whom the median use per month was 1.7 percent-and (3) 13 who estimated they used 10 pet.
-
cent or more in the average month-for whom the median
^ %
tae per month was 204 percent.The adjusted Terminal examinadon scores for these three groups are shown in Figure 4. Retention was better among thoes who reported
'
'
'
'
'
'
'
'
'
using the material rnost. The differences shown in Figure
'
3
'
'
'
'
'O
"
4 are sta tistically sign 10 cant (Fsea = 4.22, p <.05).
WoNTH Artta teer.cousut l
_
-
_.
..
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
'
'
.
regure 4.-Change en easm eccess over one rent period as e runcreon or estidents' esurnstes or she amoval or ensmaned material wructi thew used en an everage enanth.
Et-
o zO dm sg hQ O
-
U 5$
new. e
&
59<
d P
5
h N
10 %
ESTIMATED PROPORTION OF EXAM MATERIALS USED IN AVERAGE MONTH talent do not explain the differences in adjusted Terminal no reason to belaeve that the onee'who dropped out of the scores as a function of the amount of use. His suggests control group would have done better than those who that it was the use that kept the material fresh,rather dropped out of the other poupe.Therefore, there is no than that good students used the matenal more.
direct evidence on which to believe that those who remained Potential Bias Problems to take the Terminal examination differed rystematically
.etmen the groups, As noted above, the three experimental groups were Nonetheless, the investigators felt that it was important similar with respect to their scores on the initial exam. In to astimate whether the results might han been different particular, the control group and the sixconth group hoc all the students who took the six wek or siteonth the groups whose tetendon differed significantly owr a exan a subsequently also taken Terminal exams. This was year-were almost idendcal on Initial scores. It therefore act enplished by deriving a prediction equation for Ter.
appears that these groupe were similar with regard to minst scores on the basis of Initial and Interim scores for original talent. Nonetheless,it is possible that the taking thoes.tudents who took all three exams. For example, for e
of exams at six weeks or six months aerwd to selectinly the sixeonth group, a linser regression equation, based on dissuade students who forgot more from taking the a least equares fit, showd that Terminal examination examinations a year later, thus introducks a bias. There scorse wre predicted as w =.03 +.52x +.30y,where were,in fact,17 students in the sis week and nine in tM w is the unadjwted Terminal examination a4 core, x the original sixeonth group who took the laterim examine.
unadjusted initial s4 core, and y the unadjusted Inte rim tion but failed to return Terminal examination papers, s* core, (One student took tb Terminal but not the Interim
. nose prediction equations having been determined for exam.) We calculated the mean (adjusted) performance of the students who took all three exams, they were then ap.
these groups on the Intsrim examination and thy were, plied to the nontakers," who had taken initial and in-indeed, considerably lows than those for the average stu.
terim, but not Terminal exams. Predicted adjusted scores dents in these groups: 43 and.77 for the six week and for them-what they probably would have gotten on the sixconth groups, respectively. Honver,it should be Terminal examinations had they taken them, minus their noted that this self4clection is not necessarily indicatlw actual scons on the Initial examinations-were -1.07 for of a bias in the comparison of rnean results for the experi.
the six week group and.36 for the six month group.ne mental groups with the control group. The proportion of padicted six week group score was below the actust con-students in the sixconth group who took the Terminal trol group score whlis the predicted sixconth group score exarr.instion was almost Identical to that in the cor, trol ws: abon tM control group. For the control group, there group;Lt., there was the same ontall dropout rata among is, of course, no corresponding estimate for dropouts, the control group as among the sixeonth group.Dere is Howowr,on the assumption that control group members
__________-_____________--_-______-________-_-____-_a
.
_. _ _ _
~
.
- 104 Tile JOURNAL OF EDUCATIONAL RESEARCH
.
-
who fa21ed to take the Termmaleaam would have done if months apart. While the present study gives evidence of anything, somewhat wone than those who actuaUy took ef fectiveness only for esams and reported actual use, other it, as predicted for experimental groups, the "true" Ter.
active recau esercises such as papen or problern sets should j
rninal score for contsol students wou'd be even lower than also work. llowever, eaaminations, ideally with shoitly observed. Thus, had au students completed the Terminal following (orrections, utve the function in an easily aJ.
exams.it appears very unhkely that the overau direction m,nistered fashion.
and signincance of the differencesin average perfc,rrnance among the three groups would have been changed.
,
"
REFE RENCl3 The Orst thing that is worthy of note is the very smau amount of use that students claimed to make of the ma.
3. Ausubet. D. P.,"Early versus Delayed Review in Meanmgrul teriallearned in these counes. The average student seerns 14a nung " hnhokey da ihe schets. 3: IM 198.1966.
"*
"'
^'
to have taken a course at least a year before he actuaUy
^
^
ch e t
9 n'eeded it.One rnight guess that technical counes given in 3.1.and.ver.T. K.; Ekiridee, L. "Effect of Tests without i eed-a technjcal work environment would suffer comparadvely tieck and Presentatforv. Test intervalin Faired Associate learnmg,"
little from this problem. The fact that there 1 a relation
/o"'ael of DPerimen tet hyrhoJosy. 73: 290 298.1967.
between the arnount of use and the amount ofloss over a 4. Pmri n. H. A.: Etus. M.: Toohill N.; Kloess. P. "Some Measurernents of the Effects of Revwws Journelo/Educermet year would seem to suggest that it is a potentiaUy serious hyc,vWory. 26 65 72,1935.
matter for a student to take a course more than a year be.
5. Roderick, M.; Andmon, R. C.,"Prcgrammed Instrucuon fore he will use its contents, to Psychology wrsus Tent-book Style Summary of the same Lee.
On the posidve side, an Interim test at six months-but son." Journal o/ Ede rbast hychology. 59:381 387, 1968.
not one at six weeks-seems to help students tetsin infor.
g,j,fg,Y,y,[,,Ij
,7,f,'y,$$[37,y,',',",y,',*,,
mation over a period of a year.Thus,one possible partial like Ennts, A
,,ve,n Educerknet Arverch /ourut. 3:241 249, solution to the problem of getting knowledge through to 1966.
Its temporal desdnation la ta provide occasional retests, or 7 80a's. A. W.; stioud, J. B.,"Revtew, with sp.cial Reference perhaps other " refresher" setivities, for those who con *
f65 urar /
r ehh ty. 31:
,
11nue to anticipate future use.The results of this study 8. Spitser. H. F., " Studies in Re tention." Jourcat o/ Educe.
suggest that such activities should be spaced about six thaet hyrhoJory. 30:641 657,1939.
.
CO6(PARISGH OF Tite STANFORD 4tNET SCA12 WTTil THE FEADODY 11CTUR.E VOCASUI.ARY 1TJT s
.a a,
a s, Ja pa.. r nmn r.
Wercosaw State Catege THE WELL KNOWN 'fT.3TS of sneessment, the Revised Staa-palaned and eneetal devia tes. Of 11 retaatrittty studnes. nine were car-fntd-Binet Scale of Intalagence and the Wechster latelligence Scale rted out with a devtsse population. The manual states that "coefn.
Ice Children, an the commonly used diagnostic toons treen individs-cients of equkvalence and temporal statslity appear to be astsfactory any. Another teet less ihme senaam these and with a absi-for both eversp(PPVF manud, p. 32). It is qvwtlonable as children and for those who have one of a number mura of propratka neessaary le the y Pletan Vocabulary of dinebdtties.
Test (PFVF).
the dieshtilty la and how sneny average children formed pan of the lhe test is cornposed of 150 aanbered pistes, each costalains standardtaatsoa population.
four pectures;the lesene asust identify one of the four fross a quee-No rennerch date on preschool ch11dren and on those with above tion and/or statement by the tesast. Basal and ceiling ages are eas!!y aversee abGity were available. Of the valMity studies. 36 percent obtamed. Frora the forms (A-8), percentiles and tanniligence quo.
treated mental ntardates,18 percent examined everage chihiren uen to are obtained for $4 rompias from C.A. of 24 to I 8 3.
and 44 percent treated various other forms of deviation. Buros cites ReliatGty Andings showed coefDetenta of.97 to.61 based on the spunous effects of the val &dity costucients because of the ranse data frces both forms admkustemd to & ;M Sa-deaf, cerebra!
in age end vastance in dettetion (1:330h Table I.-Wesa, Sissna and Standard Error for Stamford44eet scale of lasellionace and the Pinbody Pkture Vocabulary Test Stamfor+56 met Scah of latelligence Peabody Pnetare Vocabulary Test Age N
BD RE
'
32 134.40 6.53 1.16 117.00 14.50 2.58 i
30 130.50 9.35 1.73 114.50 10.10 1.87
32 129.30 13.23 2.36 117.75 LOO 1.42
-
-
-
.
.
-_
-
-.
--.
.
-
_..
.
., _.
The Spacing Effect A Case Study in the Failure to Apply the Results ofPsjthological Research Frank N. Dempster University ofNevada. Las Vegas ABSTMCT: The spacing efect would appear to have recent sampling of practitioner. oriented tentbooks suit-considerable potentialfor improving classroom learning, able for use in teacher education programs,1 found either yet there is no evidence ofits widespread application. I little or no mention of the practical benefits of the spacing consider nine possible impediments to the implementation effect, and in sorne cases the spacing effcct was confused of research)ndings in the classroom in an efort to de-with other phenomena (e.g., Good & Brophy,1986:
termine which. (/any, apply to the spacing efect. I con.
Mayer,1987; Slavin,1986; Woolfolk,1987). One well.
clude lhat the apparent absence ofs)'stematic application known educator, in fact, advised against spaced practice may be due, in part, to the ahistorical character ofresearch at least in the early stages oflearning (Hunter,1983),
on Ihe spacing efect and certain gaps in our understanding Why is it that rescarch bodings that appeat to have ofboth the spacmg efect and classroom praalce. Howewr, signi6 cant implications, such as the spacing effect, often because none of these concerns seems especially discour-are not utilized by teachers and curriculum makers 7 in aging, and in view of what we do know about the spacing general, the problem is that there is no well developed efect, classroom application is recommended.
implementation model, nor is there a standard method.
ology for analyzing the conditions that foster the transfer of knowledge from the laboratory to the classroom (see ne spacing effect-which refers to the fmding that for a Hosford,1984, for a discusslor}). Obviously, inues re.
giwn amount of study time, spaced presentations yield garding the utilization of 6ndings from basic research are substantially better learning than do massed present&*
Complicated, and there are many potential impediments tions-is one of the most remarkable phenomena to to the implementation of research fmdings in the class-emerge from laboratory research on learning. It is re-room, in this article, I explore nine potential impedi.
markable in several respects. First, the spacing effect is ments, all of which seem reasonable at hnt glance, in an one of the most dependable and replicable phenomena effort to determine which, if any, apply to the spacing in experimental psychology. Second, it is remarkably ro-effect.
bust. In many cases, two spaced presentations are about Impediments to Application twice as effective as two massed presentations (e.g.,
Hintzman,1974; Melton,1970), and the difference p The PAeneewmen Has Not seen Known long EnougA tween them increases as the frequency of repetition in-creases (Underwood,1970). Morcowr, demonstrations Although the time lag between discovery and application of achiewment following massed presentations often are vasies greatly, some considerable period of time often in-only slightly higher than that following a single presen-terwnes between the publication of research 6adings and tation (e.g., Melton,1970). Third, the spacing effect is their application. In the case of the spacing effect, however, truly ubiquitous in scope. It has been obserwd in virtually a considerable period of time already has passed since its
,
enry standard experimental learning paradigm, with all initial documentation. The spacing effect was known as sorts of traditional research material (Dempster,1987a; cariy as 1885 when Ebbinghaus published the results of i
Hintzman,1974; Melton,1970).
his seminal experimental work on memory. With himself With all of these characteristics in its favor, the spac-as the subject, Ebbinghaus found that for a single 12 -
ing effect would seem to haw considerable potential for syllable series,68 immediately successive repetitions had improving classroorn learning. Howewe, there is little ev-the effect of making possible an erroriess recital after seven idence that this potential hu been reshzed. Neither additional repetitions on the following day. However, the American classrooms nor American textbooks appear to same effect wts achiewd by only 38 distributed repetitions implement spaced reviews in any systematic way, and by spread over three days. On the basis of this and other comparison, Soviet mathematics textbooks ptcnide a related 6ndings, Ebbinghaus concluded that "with any much more distributed method of presentation than do considerable number of repetitions a suitable distribution their American counterparts (Stigler, Fuson, Ham, &
of them owr a space of time is decidedly more advaa-l Kim,1986). Nor is there much evidence that the next tageous than the massing of them at a single time" (Eb-
)
generation of educaton is being better informed. In a binghaus, t 885/1913, p. 89). Jost, also working w,th non-
!
,
627 August 1988 * American Psychologist cygag - No c= a-o= =wuanwe n
--
,,, _ _ _ - - -
' - - ~ ~
- _ _ -
-
,
-
sense syllables, reported similar 6adings end in 1897 for.
mutated what was to tecome known as Jost's Law:"If related esplicitly to lane. scale educationalissues
'
two associations are ofequal strength but ofdifferent a&e, uriables. Whereas the relationships between the ra
.
a new repetition hu a greater value for the oldet one" Sne-grained analyses ofleaming and memory co (McGeoch,1943, p.14c).
by researchen and the practice of education u to 1928, Ruch published a redew ofdozens ofstud-not self eddent to the teacher s.ed must be fcrward in the case of the spaci ies of the spacios effect. Although interpretation of the results of these studits (e g.,
Dearbom,
19t0; Petkins, 1914; Pyle,1913; Starch,1912)is complicated by other,spacing effect has immediate and obvious implicatio potentially confounded variables, the results tend,in gen.for how time in the classroom may be distributeu eral, to confirm the earlier work by Ebbinghaus and bymally. In the wake of recent critiques and studies of Jost. Thus, published reports of the spacing effect haveschooling, such as A Nation at Risk (National Comm been in existence since the latter part of the 19th century sion on Excellence in Education,1983), Ume to Lea (Denharn & Lieberman,1980), and Perspecthes on and the early part of the 20th century, The Phenomenon Has Not Recetred structional Dme(Fisher & Berliner 1985), the use of ti Recent Documentation in the classroom has become a majo,r educationa The Phenomenon Has Not Been Demonstrat In the absence of recent documentation, research 6ndings Satisfactorily in Schoo6LIAs Actlettles may seem stale or anachronistic, but, as most, if not all, students of the learning literature know, the spacing effect Such demonstrations are arguably the mos+ imp has been well-documented in recent times. Many studiesbridge between basic research and educational p of this phenornenon were published during the 1960s andin the case of the spacing effect, however, this brid the 1970s, as teviews by Hintzman (1974), Melton (1970),
to have been crossed. Several demonstrations of and Glenbers (1979) attest.
ing effect reviewed by Ruch Although much of the research included in these
" intended for schoolroom app (1928) were, as he lication" (p. 20). One of reviews was reminiscent of the work of Ebbinghaus in the most iuteresting of these effects, from an educ using easily analyzable simple verbal ur.its perspective, is that of Pyle (1913), who had a group this research are considerable from any pen, she fruits of third-graders drilled in addition, either twice a day pective For example, the ubiquitous, highly replicable character of days (once in the morning and once in the aftemoon th( spacing effect fostered the notion that its caistence once a day for 10 days. Their improvement in recall of must be telling us something important about memory addition facts, which was decidedly in favor of the la (e g., Hintzman,1974). Also,it clearly demonstrated thatinstructional method, provided perhaps the earliest the Total Time Law, which states that the amount learned pertmental confirmation of William James' (1901) adsice is a direct function of study time regardless of how that to teachers and students that it is better to repeat an as-time is distributed, was in deep trouble or at least in needsociation on many different days than again and ag of a major overhaul (Melton,1970; Underwood,1970).
on just a few days (p.129).
More recently, the spacing of repetiuons has been Most of the early demonstrations of the spacing eff the subject of studies reported in a variety ofjournals,
" intended for schoolroorn application" focused on te including some with an applied perspective (Bahrick &processing tasks, and in three recent studies text pro-Phelps,1987; Cuddy & Jacoby,1982; Dellarosa &
cessing again has been the focus. In a study by Kraft a Bourne,1985; Dempster,1987b; Elmes. Dye, & Herdelin, Jenkins (1081), subpects attempted to free recall the 1983; Glenberg & Lehmann,1980; Glover & Corkill, and one ides unit from each of a series of twice. present 1987; Toppino & Digeorge,1984; Toppino & Gracen. passages, with each repetiti 1935). Thus, documented evidence of the spacing effect eight latervening passages.on separated by tags of up to As in much standard serbal has appeared in the literature continually for the past 100 learning research, recall was a linear and much impr function oflag. One practicallimitation of this study years.
course, is that students rarely are asked to recall so little The Phenomenon Cannot Be Linked to issues of (win a series of passages.
Currvat Concern to Educators in the second of these recent spacing effect studies I agree with Glaser (1982) that research knowledge is Dempster (1986) contrasted lags--defmed as the inter
,
.
most likely to inform educational practice ifit can be between two opportunities to read a,assage of text ma terial-of 30 seconds, 5 minutes, 'a minutes, and 48 Prepmuon or tha aruele m supported, in isn. try a srant frcen u, hours in one experiment, and 5 minutes and 30 minute coticapes John A. Bates and Dons Cmy for umt usatance, u MUBarnck Res in a second experiment. In the first experiment, sub in the 48 hour5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> condition recalled significantly more idea
,
for their helpful rescuons to an earlief mon of this trucie.u Arthur Glenbers. Dousin units than did subjects in either of the two thortest la Conditions. In the second experiment, recall was sigmf Cormpordence concermns eus artic5e show be ddms@o Frank cantly higher m'
N. Dempnet. Department of Educauonal Pmhology, Unrversty of N,.the 30. minute condition than in the 5-oda 14: Vesas. NV 19154 min'tte condition, in both experiments, the recall advan.
tage associated with the best performing group was about 628 August 1988 * American Psycholopst
=
. - ~ _.
_ -
_ _. - - - -
_ - - -
-
- - - -
_ -
1, the same (betwere 25% and 3M), even though different make cestain that students iridependently retrien tarset infor.
text passages were used in the two experiments. Thus, mation at mternis that are as ions as I month, over a rmod the effect was both robust ard replicable. FinaJly, Glover of 5'"f*3 Ye*'5 I"St**d of the more t# cal intervah of I to :
and Corkill (1987) obsentd the spacing effect (0 lag wrsus days e peri ds of from to to 15 mis. m M a 30-minute lag)in subjects' memory for paragraphs they There Are Serious Discontinultles I: he Literature on read as well as for brieflectures (12$ words) they heard.
tat Spacing Efter in addition to text recall. spacing effects have been demonstrated in programmed instruction, where the ob-Another possible reason for the fadue to apply the results jective has been the learning of science and mathematical of research on the spacing effect is that there are serious concepts in one study, the sneanings of a series of pro.
discontinuities in the literature on the spacing effect, such grammed scienti6e ter ns were learned ruuch more efrec.
that most recent studies seem uninformed by the research tively when repetitior.s were spaced than when they were of earlier ones. Although the spacing effect has a lengthy massed (Reynold* & Glaser,1964). In another study, ar.
published history, there are discontinuities of this sort, ithmetical u!es presented by a computer assisted in.
For example, much of the important work from an ap-struction system and empressed as verbal statemenu were plied perspective done in the early 1900s is not cited by learned better when reviews occurred one and seven days studies published later. More generally, most recent studies after original learning than when they occurred one and tend to she the impression that, with the exception of two days after original learning (Gay,1973, Experi.
the work of Ebbinghaus and Jost, all we know about the ment 2).
spacing efrect dates only as far back as the 1960s. I cannot Finaliy, spacing effects have been found in vocabu.
help but wonder how widespread a problem this is in the lary learning. In a study by Dempster (1987b),38 un.
learning literature, and what sorts orconsequences it has common English words and their defmitions were pre.
for the evolution of the science oflearning and its appli.
sented three times, either with each repetition of any giwn cation. Why is it that we occasionally-and perhaps fre-word separated by every other word (i.e., each repetition quently-give up on, or simply lose interest in, a phe.
of a word was separated by 37 other words or 4 minutes, nomenon before we have def nitise answers to basic ques.
19 seconds) or with each repetition of a word massed in tions and, then much later, return to the phenomenon as succession. In addition, the words were presented either though we had just recently discowred it?
with or without sentence contexts. The results were quite Upon reacction, this ahistorical character of research clear. In three experiments in which spaced versus massed on the spacing effect would seem to have at least two presentations were evaluated in this manner, spaced pre-unfortunate consequences, either of which could impede sentations yielded substantially higher lewis of vocabulary application. From a scientific perspective, mch rescatch learning than did massed presentations. In some cases, is less likely to be as cumulative-where cumulatise refers in fact, the number of word meanings recalled was over to empirical laws and theoretical structures building on 50% greater under spaced conditions than under massed one another so that later developments extend and unify conditions, carher work (Hedges,1987)-as would be mort histoncal in Dempster's (1987b) study the retention interval esearch. From a practical perspectiva, it will seem as if awraged less than an hour-short. from a practical per.
the spacing effect hu not weathered continuous scrutiny spective. Thus, the question minit be asked, "Would owr a lengthy period of time, spacing effects emerge in the setention of vocabulary There are, of course, many possible reasons for dis.
words tested at much longer intervals?" Fortunately, there continuities in the literature of a phenomenon. In the are data relevant to this question. Bahrick and Phelps case of the spacing effect, however, three appear to stand (1987) tested 35 individuals who had learned and re-out. First, for the most part, studies conducted from an learned 50 English-Spanish word pairs for recall and rec-applied perspective and those cc.uducted from a basic re.
ognition after an interval of eight yean. One nriable of search perspective constitute two distinct streams of re.
interest was the interval between successive relearning search. For example, although widely cited reviews by sessions--either 30 days, I day, or 0. The data show that Crowder (1976), Glenberg (1979), Hintzman (1974), and the intersession interval had a wry larse effect on recall, Melton (1970) repoet dozens of studies of the spacing with the recall probability associated with the 30-day in-effect using simple list learning materials, none of the terval about 2.5 times the probability associated with the many studies using more compter. material with clearer zero interval. In turn, the 1. day interval was associated classroom analogues are reported.This is the case despite with much better retention than the zero interval. For the fact that some still siable accounts of the spacing effect both comparisons, the effect on recognition, exclusive of (see for etar.sple Cuddy & Jacoby,1982, and Hintzman, recall, was much less pronounced. Moreover, their data 1974) were anticipated and supported in earlier studies clearly indicate that even five or more presentations are done from a more applied perspective (see, for example, unlikely to facilitate long term retention if the interval Ausubel,1966).
between successive presentations is one day or less. With Secor d. different terminology has been used to refer respect to the educational implications of their study, the to similar, though distinguishable, phenomena-a situ.
authors concluded that long. term retention would almost ation that seems to have created some confusion. For certainly be enhanced if foreign language courses example, a number of writers hase distinguished between August 1988 * American Psychologist 629 i
_ _ _ _ _.
_ _ _
.- _
_
_
_ -. _..
._ _ _ _
_ -.. _ _ _ _ _ _ _ _. _. _
.-
the " spacing efrect" and toth the " Melton" or " lag efrect" Also, Reder arid Anderson (191:2) failed Io cite Edwar and the " maned versus-distributed practice"(treet. Lag (1917), and neither Bloom and Shuell(1981) nor Smi erTects have been taid to occur when performance im-and Rethkopf(1984) seemed aware of the work of A
,
prows as a function of the number ofintentning items (1950), who found orJy minor differences in retenta betwren sucressist presentations, whereas massed wrsus between groups treated hke theirs.
distnbuted practice effects onen have been restricted to comparisons betwren spacings of zero (massed practice)
Too Afany Studies lising School Like Aeririties Hart and all spacings greater tha zero (distributed practice).
I#II'# '# #A#" 'A' 8Pd'I"I EI'#'
The use of this terminology, howtwr,is somewhat uneven.
Although the spacing effect is one of the most dependal Some researchers hase used these terms interchangeably phenomena in the learning of standard Serbal learni or have included other variables (e g., length of period of lists, there have been more than a half dozen document study, retention interval)in their characterizations of the failures to observe the spacing effect in tasks with cla spacing cfrect (e g., Ruch,1928). Also, rouch of the massed room analogues. The results of these studies make it qu versus distributed practice research, which has focused clear that the spacing effect is subject to certain not fu on perceptual motor skills tasks and lists of nonsense syl-understood boundary conditions. Specifically, five so lables, has yielded weak effects of spacing. According to of boundary conditions are suggested. First,it has be !
Underwood (1961), who reviewed 10 years of distributed found that under certain circumstances spaced pre c,
practice research, "Even under the most favorable con-tations are no better than (Austin,1921) and sometirr ;
ditions for facilitation by distributad practice, one could even worse than (Gordon,1925) massed presentations not recommend its use in an applied setting" (p. 230).
tests ofimmediate recall. For example, Austin found it
,
Apparently following his lead, some older educational massed readings (e.g., five times in one day) of text rr psychology texts advised that there were no clear practical terial proved as effective as spaced readings (e.g., da implications to be drawn from distributed practice re-for five days) in tests ofimmediate recall, whereas t scatch (e g., DeCecco,1968; Mathis, Cotton, & Sechtest, spaced readings were much more effective in delayed te*
1970).
particularly if they came two to four weeks aAct learnii Third, the spacing effect is just one of a family of Second, it has been found that massed practice ollen similar, though less thoroughly investigated, phenomena more efficient for certain simple, isolated skills, such that are occasionally confused in the literature. One such writing the products of number pairs as rapidly as possit phenomenon is the so-called " test spacing" effect, which (Thorndike,1916), Third, evidence from traditioi refers to the fact that spaced tests, particularly tests with learning research suggests that the spacing effect may :
intertest intervals of an expanding nature, result in greater apply to preschool age children, although it does eme'
retention than do massed testings (Landauer & Bjork, in a robust manner by age seven (Toppino & DiGeor 1978; Rea & Modigliani,1985; Spitzer,1939). Another 1984). Fourth, two studies have shown that the spact related phenomenon has been observed when once pre-effect can be eliminated if paraphrased rather than s sented written exercises or materials in a short course in batim versions of the repeated materials are used (D statistics either are spread our the course of sewral ses-larosa & Bourne,1985; Glowr & Corkill,1987).
sions or are presented in a single session-a situation that Finally, the results of a number of studies seem
-
is analogous to " cramming" for a test. In this case, stu-suggest that beyond a certain lag intena], further increa
,
dents learn more when the material is distributed ostr in lag are not always associated with further increases several sessions (Bloom & Shuell,1981; Smith & Roth-learning. For example, English, Wellborn, and Kill kcyf,1984). Finally, Reder and Anderson (1982) found (1934) found that four readings of a text at three h<
that, with total study time equated, repeated, well spaced inten21s were associated with better learning than fi presentations of a test were more effective than was a consecutive unspaced readings; however, r:adings at thi single, longer presentation. Similarly, Edwards (1917)
hour intensis were no better thrn readings at either o compared groups who studied strious school materials, or three day intervals. Similarly, Lyon (1914), Peters including history and geography, six and one half minutes Ellis, Toohill, and Kloess (1935), and Sones and Strc continuously or with the same amount of time divided (1940) reported essentially no differences in tetention into a study period of four minutes follcmtd some days tween groups with rereading reviews spaced I and i later by a rniew of two and one half minutes. Lag and and 9, and 1 and 17 days aner original learning. Th test inten21s, difficulty of the material, and ace of the findings wtre later corroborated by Ausubel(1966) r subjects were so variable that it is difheult to interpret by Gay (1973).
the results; however, without exception they favor the re-Of these boundary conditions, the last two seem peated, spaced study groups.
most serious and most puzzling in view of the fact t To complicate matters further, research on allied the spacing efrect has been found in paraphrased mate phenomena also has an ahistorical character. For example, (Rothkopf & Coke,1966) and in siew of long lag ett Rea and Modigliani(1985) failed to cite Spitzer (1939),
obtained with traditional serbal learning matenal(G even though Spitzer's work showed that if the intensi berg & Lehmann,1980) and in vocabulary lean between original learning and the first test in a senes is (Bahriek & Phelps,1987). With respect to the latte too lengthy, test spacing effects are likely to be vitiated.
may be that under certain lengthy lag conditions, the u
,
630 August 1988. American Pychok
,
_,
-
_
.
.
m__.
-A--
"*
__
_ _
_
,
--
-
,
.
'
benefas of spaced repetitions do not obtain because the results ofinitist proceuing efforts hase been forgetten of clautoom practice, the application potential o
'
(see Lyon,1914, and Sones & Stroud,1940, for earlier spacing effect cannot be estimated with any preci discuuions of this hypothesis).
According!r, it may be auumed wrongly that efforts In any case, the spacing effect, especially in text pro-imp'ement the spacing effect would result in little b
'
ceuing, cannot be taken for granted. Although there havt been relatively few documented failures to obtain the The Phenomenon is Nor Subitiently Understood spacing effect, they certainly could have raised enoughPhenomena that are not well understood,are l doubt about the dependability of the effect to discourage invite skepticism among practitioners--particularly th application.
who are famthat with any one of several mstances which the application of a poorly understoo The Phenomenon 1/as Nor Been Demonstrated Sarlsfactorily la rat Classroom had extremely unfavorable consequences (d fmd e g., thalido-mide). For example, educaton, who often give the With very few exceptions (Dempster 1986: Glowr &
impteuion of having a low regard for fact memor Corkill,1987; Pyle,1913), even resea,rch using educa. might feel that the spacing effect would interfere wit tionally relevant materials has been conducted in the lab-peration of 'more laudable, higher mental procene oratory. Moreover, the rather simple learning situations because it is exactly such memorization to w created in clauroom studies of the spacing effect fail to app approach the complexities facing curriculum developers etical picture surrounding the spac-e and teachen. Curriculum developers and teachen han ing effect is confused and uncertain, despite num to concern themselves with the design ofinstruction con.
attempts at clarification (e.g., Dellarosa & Bourne,1985 seyed in classes that often meet every day for a school Hintzman,1974). A recent case in point is a publish term, under conditions in which much if not most of the failure to replicate findingsimplicating a component le content is organized in ways that imply systematic trove-els interpretation of the spacing effect in standard v ment through learmng hierarchies, curricular sequences,learning tasks (Toppino & Gracen,1985). The co and so forth. Thus, it may be argued that it is not at all nent levels hypothesis is one of a gr clear what specific irnplications demonstrations of the that Miri*uute the spacing effect to m.oup o.f hypothe spacing effect in simple, isolated classroom situations have
.
dence f ene dm creatmg indepen.
about how much material to include in a course, how to I***"y;e[u on 'g events with increasing for curriculum designers and teachers faced with decisions
,e naturalistic material, theoretical work sequence it, and how to opumally phase in new material has been slow to develop, and little has been done in and phase out old material.
way of theoretically derived hypothesis testing. How Arguably, the relatise lack of applied research in ed.
it is worth noting that Ausubel(1966), in a rarely cit
.
ucational settings is, from an educational perspectiw, thestudy, offered the following explanation for the adent most serious shortcoming of resea'ch on the spacing er.
of s aced review in meaningful learning:
feet. There is no substitute for applied research, and the absence of at least several convincing demonstrations ofin the first place, after a longer retention interval, when the spacing effect in ongoing classroom situations undermaterial is forgotten, the learner is more highly mouvated proSt from the @portunity for review. He is less likely to naturahstic conditions may well have been an impedimentthis pp rtunity as unnecessa y and superfluous and is he to application'
more disposed to take good advantage ofit in terms of effo attention, and concentration, (p.197)
Too Lirrte Is Known Abour Actual Classroom Procric'
to Insrify Widespread Applicarlon ofrA'
Apparently, Ausubel(1966) was on the right track Specang Efter when he stressed the role of attention and e To the best ofmy knowledge, nothing has been published spacing effect. Recent evidence indicates that some concerning the proportion of time in the classroom of attentional account of the spacing efrect is at least a teachers usually devote to review (i.e., any re-presentation viable as any other account and well worth pursuin or practice activity pertaining to a particular educational (Dellaross & Bourne,1985; Dempster,1986; Elmes,
& Herdelin,1983; Magliero, 1983). For example, objectini, and to what extent reviews are massed as op-posed to spaced. Moreover, the general nature of clan.
Dempster (1986) had subjects respond to a questionnai room review activities apparently has not been charac-administered following a recall test of a twice read pas tetized. For example, to what extent are reviews verbatim sage, with the two readings spaced either 30 minutes ap
!
or paraphrased? Because several studies have failed to or 5 minutes apart. The questionnaire consisted of 10 i
t l
show spacing effects in text processing when paraphrased items, each followed by a 10-point rating scale, which rather than verbatim versions of the repeated materialswas designed to elicit self reports of various cognitise and
l were used (Dellarosa & Bourne,1985; Glover & Corkill,affective states and processes during reading and te
'
1987), the answer to this question is of practical impor.
included were questions concerning lesels of attenuon.
interest, antiety, rehearsal, and changes ofinterpretauon tance.
in short, due to significant gaps in our understanding from one reading to the next. The results were quite clear t
Signi6 cant group differences emerged on only two of the
,
August 1988 * American Psychologist 6.11
____ __
_ _ _ _.. _ _ _.. _ _ _ _ _ _ _ _ _ _
. _ _ _
_ _
_. _ _.
_ _
items, speci6cally one asking the subjects to indicate how tation of research fmdings in the classroom. Of the nine,
" interested" they were during the second reading-an fne appear to apply to the spacing etTect. These include affective state-and one asking them to indicate how the ahistorical character of research on the spacing effett, much "attendon" they paid dunng the recond reading-some failures to obtain the e' rect with School like activ.
a cognitive process. In both cases, the aserage ratings of ities, a paucity ofimpressive classroorn demon!trations students in the spaced 30-minute condition (those who of the phenomenoa, limited knowledge of classroom
'
also did best on the recall test) were higher than those in practice, and an incomplete under:tanding of the psy-the massed 5-minute condition. Mortoser, a correlational chological bases of the spacing effett. By contrast, the fact analysis, applied to the scores of both groups combined, that the phenomenon (a) has been known for a long penod revealed a signi6 cant correlation betwten recall and only of time, (b) has received recent documentation. (c) can one of the questionnaire items-that is, the attention paid be linked to current educational issues, and (d) has been during the second reading. Those who reported having shown to extrapolate to school like activities sutraests that pajd more attention tended to have learned more from the first four dimensions of analysis considered do not the text.
apply.
However, why should spaced presentations be more The following question now arises: To what extent
,
interesting (see also Elmes et al.,1983) and receive more sho'dd the list of plausible impediments to the application attention than massed presentations? Here again, Ausubel of the spacing effect in the classroom discourage appli.
(1966) seemed to have anticipated more recent develop.
cation? AAer addressing each of these concerns 1 conclude ments when he suggested that it has to do with the relative that we do know enough about the effect of spacing to accessibility of previous encodings (in his words, the ac-make a very strong argument for application without any
'
tivity of "trying and failing to remember material" p.
additional knowledge about the spacing effect or clan-197). If a student receives massed presentations, the in-room practice.
fortnation teamed during earlier presentations should be Consider first the ahistorical and somewhat confus-relatively easy to retneve from memory during subsequent ing character of research on the spacing effect. Given the
,
presentations. Thus, subsequent presentations should be long history of rescarch on the spacing effect and the con-relatively redundant or familiar and thus relatively boring.
siderable recent documentation, the discontinuities in the By contrast, if a student receives two well. spaced presen-literature on the spacing effect do not seem ternbly im.
tations, the information leamed during initial presenta-portant. In fact, the spacing effect has weathered contin-tions should be relatively inaccessible during subsequent uous scrutiny over a lengthy period of time. Of course, presentations, which should heighten interest level and the fact that the spacing effect is just one of a family of the amount or quality of attention subsequent presenta-similar phenomena would seem only to strengthen and tions receive. According to this account, then, massed broaden the appeal of" spacing" in the classroom.
presentations are relatively ineffective because they may A seco id plausible reason for the absence of wide-not actually result in much repetitive processing (see also spread application of the spacing effect is that there hast Cuddy & Jacoby,1982; Dellarosa & Bourne,1985; been some failures to obtain the effect in school like tasks.
Greeno,1970; Jacoby,1978; Underwood,1970).
The spacing effect, in fact, does appear to be subject to One implication of this account is that anything that certain, not fully understood boundary conditions. How-increases the likelihood that a repetition will receive full ever, it would be unrealistic to expect the spacing effect processing, such as esents that make it difficult to retrieve to apply in every situation, and the relatively few failures the results of prior encodings, should improve learning.
to obtain the spacing effect seem trivial in light of its Thus, this account helps to explain failures to obtain the roany demonstrations. For this reason and in the absence spacing effect with paraphrased repetitions, that is, rep-of any serious contraindications to the application of the etitions having a changed surface structure (Dellarosa &
spacing effect, the fact that the spaciog effect does not Boume,1985; Glover & Corbil,1987), and under lengthy always work hardly seems to justify resistance to its ap-lag conditions (Ausubel,1966; English et al.,1934: Gay, plication.
1973; Ly on,1914; Peterson et al.,1935, Sones & Stroud, Arguably, the most serious of the plausible imped-1940).
iments to the application of the spacing effect is the pau-In sum, although recent studies have yielded some city ofimpressive classroom demonstrations of the phe-promising clues to the mechanisms underlying the spacing nomenon. Clearly, programmatic research on the efrects effect, our theoreticalignorance may hast been and may of spacing in education settings is long overdue, as the continue to be an impediment to application or might results of such efforts would likely aid in its application.
contnbute to inappropriate applications.
The most useful studies of this sort would be those in-volving curriculum design and classroom teaching that help shed light on the implications of the spacing etfect Summary and Conclusions for specific applied issues, such as homework, frequency The spacing effect would seem to have considerable po-of testing and feedback, teaming hierarchies, master)
tential for improving classroom learning, yet there is no learning, and questioning strategies.
evidence of widespread application. In this article, I have Although additional classtoom research is desirabh
,
considered nine possible impediments to the implemen-and even necessary in order to make the most erTectm 632 August 1988. Amencan Psychologas
- - - _ - - _ _ _ _.-
-.
____.
- -..
_.
._.
-
-.-
.
use of spacing. I do not think we have to withhold all increase the les ning of only certain kinds ofinformation judgment on the applicability of the spacing efect at this (the qua!itative hypothests). So far, these hypotheses have point. For extrnple, we know that children study vocab.
been esamined only in relation to the more general issue ulary in a variety of subjects and that the spacing effect of how repeutions improve learning, with mixed results facilitates vocabulary learning. I.ikewise, children are c x.
(Annis & Annis,1987: Bromage & Mayer,1986: Mayer, posed to numerous scientific terms and arithmetical rules 1983). They have not been examined in the contett of while in school-c.ach of which is a domain in which comparisons of massed and spaced repetitions, although spacing has been found to be effective. When coupled such comparisons might be fruitful from beth a theoret-with what we already know about the effects of spacing ical and a practical perspective.
in text processing, there is ample reason to teliese that Recently, I was told that the spacing effect has been the spacing effect will improve classroom learning in a
" studied to death," and on another occasion that "we wide variety of subject areas.
know all that we need to know about the spacing effect."
Although we do not know much about current Clearly, the spacing effect is one of the most studied phe.
classroom applications of the spacing effect, there are nomena in the 100 year history oflearning research, and many reasons to believe that the spacing effect is under.
we do know a lot about it---enough to recommend ap.
utilized in the classroom in terms ofits potential for im-plication. However, it would be a mistake to do what proving learning. In addition to the reasons indicated these comments imply-which is simply to stop inves.
earlier, there is the fact that the spacit+ effect is somewhat tigating the phenomenon. Although it may take some counterintuitive. The spacing effect typically refers to a cleser research to avoid diminishing returns, continued phenomenon that occurs under conditions in which the experimental study of the spacing effect can yield valuable retention interval between the last presentation and the information regarding its parameters and cause(s). Then test is held constant. Thus, one might reason that because too, applied research and widespread application should the retention interval between the first presentation and produce the data base needed to evaluate the particular the test is shorter under massed conditions, this condition conditions under which the spacing efrect works best.
should result in superior performance. Even experienced educators, when judging the instructional effectistness of REFERENCU text passages, tend to rate prose in which the repetition of a giveu unit ofinformation is massed as better than Annis, L F., & Annis, D. B. (1987. Apnlt Dots procrict meat perfect?
those in which it is spaced (Rothkopf,1963).
The cas verpension on student Icemmt Presented at the annual in short, the spacing effect is neither intuitively ob.
meeting [the Arnencan Educau nal Research Assocahon. Wu
,
vious, nor well known among educators. Accordingly,it Ash. P. (1950). The relauw esecuwness of massed unus spaced ftim is reasonable to assume that those who become teachers, presentauous./oumale/Educatwnaltsghology. 4/.19 30.
administrators, curriculum developers, or wTiters of Auson. S. D. M. tl921L A study in logical memoryJmerfran /oumal reading series are ignorant of the spacing effect,just u
many psychologists are not clear about the totality of p,gy,,y j,,3,3,3,g,, j, i,delswd,,rewew in meanansfullear DP nus 3.i, educational situations that call for its application Thus, Bahrick. H. P., & Phelps, E. (1987L Retention of Spanish vocabulary our ignorance of actual classroom practice should not be owr a nors./oumalo/EspenmearalrixWory*amm Jtemory.
interpreted to mean that widespread implementation of g Corngi 1.
Jg the spacing effect hu little or no potential for improving
,
pmeue ncarmns aM muon diecow.languapocabulary.
classroom learning.
joumat of tsaauwnal pe,rarra 74, ga s.2sg.
The Snal plausible impediment to the application Bromaer, B. K., & Mene. R. E. (19861 Quantitative and qualmuve of the spacing effect is that it is not well understood theo.
esats of repeution on leamins frorn technical test. /oumal of Edu.
retically. However, in view of the absence of evidence 40**'#3g%7(2 27:
g, q{Itwmin cM mtmory Httisdale, linking the spacing effect to some undesirable psycholog*
NL Ertbaum.
ital process or outcome, this impediment is extremely cuddy, L J., & Jacoby. L L (19s2K When forgetung helps rnemory:
hard to justify For example, there is no evidence that the An snabus of repeuuan esecta. Joumat o/ nreeltrammt av iMal spacing effect impairs the ability to conceptuahze or to 8'ho w JJ.451-467.
think critically. Even ifit is found that the psychological (I[($' DOL En enmentsinlearnins./ouma basis of the spacing effect applies only or largely to mem-o,ceeg j, p,(1968L tar psgholofF dlfomMI 08d Miffb(ho9' Ed*
orization (Asks, memory is of central impGrtance to any watwnalps>cMory Enslewood Clifs NJ: Prenuce. Hall.
complex intellectual activity.
Dellatosa. D., & Soume. L E. (19sSL Surface form and'the trenns Nevertheless, a fuller understanding of the spacing
[ [ g j,g % [ ;,l [,~j37,,g, g,,,,, g,,,
effect could eventually and n its application ano might f,,,,, rk learatory m th classroom. Manusenpi submmed for pub-help to asoid some inappropriate applications. Hence, i 1. cation.
believe the search for the underlying cause(s) of the spac.
Demscer. F. N (1987st Time aM the producuan of classroom learmes.
ing effect should proceed As part of this effort, some Damg implasums froru basic tuar& Edsmonal PmWost
attempt should be made to determine if spaced presen.
pl[
I F. N. (19:7bt Efects or vsnable encodies and spaced pre.
tations tend simply to increase the amount ofinformation wntiums ao vocabulary learturt /oumal o/ Edeanomal PWfv learned (the quantitative hypothesis), or if they tend to r9.162-170.
August 1988 * American Psycholopst 633
-
,
_ --
.. - - ---
-.
..~- _-
.__ -.--_ - _
a
.
Denham,C. A Ueterman. A.(1980t Time tolearn Washingtoo DC:
NatiooalInsuture of Education.
Mswr, R. E.(1987L Educaliceia/ptgmicry A cernant approac
,
tan: Usue, Bronti.
Ethrghana H.(1913t Memory (H. A. Rurr & C E. Bumenius Tranal New Wrt: Teactwrs Course. (Ong:ral it pubhshed 1885; paperteck McGeoch.J. A.(1943). TArpig%gyyAmman teamsg New bk:
.
Lonsmans Grien.
ed., New Wrkt Dowt,1964)
Edesrdt A. S. (19171 The datthsoon of tirne in learnAng small amountsMehen. A. W.(1970t The sirtsdon mth rapco to the spann of matenal in Studies in hghdogy 11tchener commemorattw sd.
couans ard memary, Joumalof thbalLoresag and iHs/ Br4rvror, 9. $9f>-606.
i ume (pp. 209-213t Worcester. MA: Wila:m.
Elmes, D. G., D>e, C J., & Herdehn, N. J. (1983L what is the role of Nauenti Commimon on Excellence in Educenondill
hs&c The imperanwfor afrational trform. Washington, DC: U.
aKect in the spaang efect? Memory J Cognition. II. 144-131.
Gowrnment Printir.8 Ofre.
Enshsh. H. 8., Wruborn, L L., & Kathan, C D (1914). Stud.es in sutstance memorustion. /oumal of Genrral hg4 dory //. 233-Perkins, N. L (1914L The value af dtstributed repetiums in ro 260.
Br ruh /gurnalothwh&ry. 7.253-261.
Fisher. C. W., & Berheer. D. C (1985) /irnecersws on instructional Peterwn H. A., Eths, M., Toohill. N., & Kloets. P. (1935L Some ssme New Wk: Longman.
surements of the effects of renews /pumalo/Educarronal P:nhdor>
16.65-72.
Gey. L R. (1973). Temporal posadon of reviews ud its eKcct on the retenuon of mathemaucal rules. /ournalo/Educetsonal hwhdory Pfe, W. H. (1913). Economical learning /oumal of Educasio 64. 171-182.
chdorx /. 148-158.
Olaser, R. (1982). Instruc6onal ps)chology: Past, present, and future.
Res. C P., & Modashani, Y.(1981k The efract of enpanded v Amertton hyhdottst. /7. 292 303.
Iluman heming. 4, Ii-l8.prsetace on the retenuon of mut Glenberg. A. Mdl979). Component. lewis thson c(the efects af spedes of repetinons on recall arm! remsnition. Memory a Corntnon. 7,93* Rader. L M., & Anderson, J. R. (1982). EKeets of spacing and
'
,
I I 2.
tahment on memory for the main points of a test. Memo Glenters. A. M., & Lehmann T. 5. (1980) $pecing repetidons ver i
'
naion.10.97-102.
Reynokia. J. H., & Glasce, R. (1964). EKects of repeuuon a G
&C 9 7) Inb o(raraphrased repeuuons
[ 7 h n b ory
"
""
on the spacing efect. Journal of Educational hxhdory 79,198-J 97 304
"'"
'
Good T. L, & Brwhy. J. E. (1986L Educationalpixhdory (3rd ed L Rothkw(, E. Z. (l%3t some obeetwauons on predicting instruc New Wrt: Longman.
' by amp 6e inspecumdoumal(Arnammed /anruason, Gordan, K. (1923) Class results with specas and urispece<f memaruire.
Rothkgf. E Z & Coke, E. V.(1966L Vanadons in phrauna a
/oumalo/Expersmentalhghdorx 3. 337-341.
eution interval and the recall of sentence materials. J Greeno. J. G. (1970K Conservation ofinformadan-processing capacity Learnmg aM f >r6al 8thavior. 3. 86-91, in paired amoctate memont.ars /ournaloflerbalhornmr and Hrr6a/
9. $81-386, Ruch. T. C. (19281 Factors induenons the reladve cronomy o Behanoe:
Hedges, L V (1987). How hard is hard science, how son is soft science?
and distributed prac6ce in learnirt hghdogscal Arverw.1119-4L American hnkdotust. 42,441-45S.
-
Hanuman, D. L (1974L Theorrucalimpbcadons of the spectng efect.
51snn, R. E. (l986L Educationalpsykdory: Theory mio procssce In R. L Solso (Ed K TAcortes m cornasw psgAalery tar toola slewood CLRs, NJ. Prenuce Hall.
Srnrosiurn (pp. 77-99). Potoms. MD Erlbaum.
Smith,5. M., & Rothkepf E. Z. (1984). Conteatual ennchment a Hosford. P. L (Ed.). (19841 Usses what aw know about traching. Al-datribuuan o(proctics in tlw classroom. Cognaron and /nstructw etandria, VA: Associauon for Supervision aM Curnculum Deveb 1.341-)$8.
Sooes, A. M., & Stroud, J. B. (1940). Review with special tefettnce ement.
Hunter, M. (19831/ncreannsieur traching rfterswntu Belmont. CA:
temporal poution. /oumal o/ Educational hghdory 11. 665 4 76.
The L4anung lastatute.
Spitzet. H. E (1939L 5tudies in tetenuon. /ournal o/Educarsona Jamby L L (1978L On interpreung the efects of repeution: Solving a chdorx /0. 641-656.
problem versus remembenes a miunon Journalof Hrrballsarvung Starch, D. (1912). Penods of mort in learnir4 /oumalof Educationa aM frrbal 8thavior. 17,649-667.
hwMory J, 209-213'
James. W. (l901) TM.s to teachers m psykdory:And to students onSuskt. J. W., Fuson, K. C Ham, M., & Kim, M, Sdl986L An analys K ll, R.
&J b ( 811 lag edoct with suraPy presented tan raathemancs textbooka Cognusen aWasuumon,1 Ob penases. k*letan of the h>chonomsc,5ocorry 17,132-134.
I *Mauet. T. K., & Bjork, R. A. (1978k Opdmurn rehearsal pattems Thomdike E. L (1916L Notes on practice. imptovabihty and the cu
'
and %me Icaming. In M. M. Grurwberg. P. E. Morns, & R. N. Sykes o(work. Amnican /oumalc/hxWorx /1, W6L homcal aspras of memory (pp. 625-612L New Wth:
T@pino, T. C, & Digeorge, W. (198 Il The sesong erect in free re Acs-
'ress, Lyon,15. O (1914L The relauon o(length c(matenal to time taken for emerges with devtlement. Memory a Cognuson. //. 118-122.
I
!
lesetusa and the amusum daartbutaan c(ume /oumal(Educational Toppino, T. C. & Gracen, T. E (1983). The las cKect and differenu
{
hxhdcry J. 1-9,85 91,155-163.
,
orsaassanon thory. /ournal #Espeamensal hxhology Lcornias.
Memorx and Cornuton. 11,183-191.
(
MasberD. A. (198)k Pupilddadans folloming smits cdidenucal and related Underwood. B. L (19611 Ten years of maued practice on distribute
[
to be-remembered words. Memory 4 Cornuion. II. 609 4 15.
pree ucs, hghdorscal Arvirm 64,229-247, Mathts. B. C, Cotton, L W., & Sechtest, L (1970k hghdoric 1/oun.
Underwood, B. J. (1970). A breakdown of the totabtime law in free dations ofeduconon New Wrk: Academic Press.
Wrycr. R. E. (1983) Can you repeat that? Qualitauve estets of tepeution recati learnirs /oumal of l'erbal traming and irrbal 8thavior 9
$73-580.
ard advanced organasers on learning from ecience peces. Joumal of Educanceal h>chdory 71 40-49.
Woolfolk. A. E. (1987L Educationalpsghdogy (3rd ed t Eegje*oo CL5s. NJ: Prentice Hall.
634 August 1988 * American Psychologsst
_ _.
. _ _ _ _
.
_.
-
.
-
'
.
,
.
.
-_
-
-
i
__
I V S A _ Z A~~ O \\
~~9A \\,\\ G 3 70G RAV i
l jf
%h t
.
\\
s fl
-
d-y (p-
!
/
i
'
)
-f
~
l
.f \\ \\
.
/
\\1
'\\
,
A i
'
l
,
.,
i
',
-
'
!
i i
.
!
!
i i
!
,
.
!
,
I
!
-
\\
t
!
U.S. Olympic Training Center
.
i
- 1 l
... _ _ - _. _ _ _.. _ _ _ _. _ _...
. _ _ - _ - _ _..... _. _ _. _. _ _ _ _ _. _ _ _..
_ _ _ _
.
-
.
.
.
.
.
IMAGERY TRAINING FOR OLYMPIC ATHLETES
!
INTRODUCTIOff define the concept of imagery as it applies to spo tThe s to explain basic imagery technique and describe methods for r,
to improving imagery skill and specific ways to apply th imagery skills to your individual training.
ese preparation and competition." Imagery" is a very relevant to the athletes >
raining, Many olympic athletes and performance enhancement. coaches are familar with and use imagery techn provide practical methods for improving your athleticThe following inf performance.
Author Richard M. Fenker e
r Compiled By Shirley Kay Durtschi Research Assistant-Sport Psychology.
U.S. Olympic Training Center
,
-
=e-n n.
- - _ _ - -
. _ _ -
-. -.. -..
. -. _. - -.
.
-. -._
-
--
~
.
.
.
t.'h a t Is Imagery?
i Imagery is cne language of the body / mind.
often use words or relf-talk as the basis Just as you you also frequently use images.
for " thinking" common means for communication with theWords or verbal langua are our most external world (and for thinking about this world)
are most often used for internal communication.
Images
.
be visual (mental pictures or visualizations)
Imagery can also be auditory (involving sounds),
but it can olfactory (involving smells or tastes),
kinesthetic (involving your sense of body position ortactile (involving or motion).
involve more than one sensory modality.An extremely vivid or clea For example, were a swimmer, you might imagine a mental picture of if you of your teammates cheering,yourself swimming in a competition and also
'
against your skin feel the coolness of the water your body as you e,xecute a quick,have an acute kinesthetic awareness of
,
the odor of chlorine from the pool. smooth turn and finally, smell
_I_magery Is The Lanouage Of Experience imagery we can create, Images are our mental versions of experience.
Using experience; or, in vivid detail, a replay of a past our mind.
In sports,we can create a new event or experience in we are free to capture the image of another athlete's moves or form and build on this pattern to develop our own style.
Imagery guides much of an athlete's experience because it i a more efficient, describe how to execute a golf-swing, complete language than self-talk.
s Try to You could write a book!
in detail, using words.
information in a few seconds!An appropriate image communicates t imagery.
How Does Imagery Work In-Sports?
performing your spcrt might have an impact on-your act swimming performance.
First
_
performing an action, you-are, when you imagine yourself'
transmitting electrical-These neuromuscular impulses will, impulses to the muscles invo the same form or pattern--as the impulsen_for vivid imagery, have
.
are actually performing (but be somewhat weaker).-generated when you neural pathways involved when these movements arementa Thus, performed.
Second, or." representing" instructions.for movement. images may be an efficient w given above, a simpler yet more complete _ description-than is possiblefo
_
-
with words.
.
.
.
.
u
.
_.... _._. _ _
_
---
. _.
._
-_
-_ _.. _ _ _ - _. _. _ _ _ _ _. _
.
l
.
r What If My Imagery Is Very Poor?
First, remember that poor visual imagery doesn't necessarily mean your other imagery senses are diminished.
Some people rely more heavily en their auditory images or kinesthetic images than on visual imagery.
Imagery is a skill and like all skills can be improved with practice.
Think of it as exercising a very special mental muscle.
If your images are weak or unclear or lack quality in any modality, relax.
Probably what you need most is practice, using some of the exercises given in.this brochure.
Your coach or sports psychologist can also help by providing training materials for enhancing imagery
,
skills.
Learning to talk took some time, so will learning to develop vivid imagery.
But, you can do it with practice.
What's The Most Effective Way To Use Im.agery?
No one knows for certain, but current research suggests the following guidelines:
1. Use a combination of imagery practice with actual practice for the best results.
2. The most effective types of imagery-probably depend on skill level.
Beginning-and intermediate athletes may want to focus on basic skills while more advanced athletes may work on physiological controls, problem solving or competition scenarios.
3. Cassette tape programs can_be useful for improving imagery skills.
Commercially produced e
tapes or other tapes made for a group.are probably less effective than tapes designed for individual athletes.
4. Choose a quiet. place where youtare not likely to be disturbed for practicing imagery.
Using one of-the trance techniques discussed below will-also enhance the effectiveness of your imagery.
-
., _ _.
-
.
- _ _
.
. _ _ __
_
.
_._
_
._ _ _ _ _ _ _
_
__ _.
_,.
.
.
'
.
.
Imagery With Relaxation, Meditation Or Hypnosis?
dono in conjunction with a " trance" Imagery practice will generally be m technique such as self-hypnosis, relaxation, meditation or hypnosis.
There are three reasons for the increased effectiveness.
First.
focusing attention or improving concentration." trance" techniques are m
'
any of these methods inaures that your perception of theAnd using images will be enhanced.
Second, using one of these techniques is likely to improve the quality or vividness of
.
your imagery.
The " letting go" that takes place during.
relaxation or self-hypnosis is conducive to the production of imagery.
Finally, quality that keeps your everyday, trance states have a non-evaluative rational mind from censoring the ideas or images that you create.
plastic and you can leap high over the other players on For example, basketball court, images as irrational; but,your logical mind might reject those is possible and can be believed.in trance, anything you imagine imagination!
There are no limits to you USING IMAGERY SKILLS IN SPORTS training and competition for athletes.This section describes the ma Mental Rehearsal competition in the mind. Imagery 1s used to practice some' aspect of training
'
The images might relate to:
specific skills the. athlete needs to master, well-rehearsed skills core automatic or performing in making-competitions.
Skill Learning Imagery is used to teach basic skills.
given by the coach might be used as "models" for-theskil
'
athletes.
Competition Programs future competition.Imageryjis used to rehearse a detailed program for'a meet-will be-held,-the events in.which he or she will beA gymnast m competing, How'will the gymnast get ready?and the~ details of the Mpla
-
event.
mental details will be monitored?
What physical and-L
- be started,
' How will the competition experienced and then finished?
ahead in points or-for handling pain or distractions? strateg or i
!
. _.
l e
-
'
-r N
Y
_-
-
-"
"
.
.
.
Problem Solving It iu very difficult imagine that you ccn solve it.to solve a problem unless you Imagery can be used to rehearse crucial problem situations and to find solutions.
Unexpected Contingencies Athletes are often confronted with new and different situatior.s that require adjustments.
Imagining these situations in advance and making the required mental adjustments increases the chance that the appropriate response will be made when it counts.
Controlling Emotional Or Peak States
-
Each athlete has an image of himself~or herself performing at a peak level.
The thoughts, feelings associated with this image are part of theemotions and body athlete's " peak state".
By knowing which physical or mental factors are associated with " peak" and then creating images that describe (accurately or symbolically) these factors, an athlete can gain some degree of control over the performance states.
For example, one championship sprinter uses the images " relaxed, focused as a beam of light and gliding smooth and strong down the track" to describe her ideal performance state.
Controlling Anxiety, Tension Or Stress Imagery can be used to prepare for situations where an athlete is likely to feel tension or stress.
During rehearsal, a tension-producing situation is imagined and then " defused" by creating the appropriate corrective image.
For example, excessive tension associated with the start of a race.might be linked with the " image" of a simple breathing exercise (the exercise would be practiced as part
.
of the imagery experience)
relaxation or control.
that produces sensations of Imagery For Centering, Focusing Attention, Blocking Distractions, Managing Pain Or Energizing There are many'other types of imagery programs that can
-
be use to achieve _these and other specific goals.
When an athlete experiences difficulty with attention, distractions, pain or other problems it may be valuable to design a program specifically for that problem.
.The appropriate imagery in each case depends to a large degree on the individual.
Usually, however, such a program would involve:
creating imagery of the. problem situa'cion (eg.,
experiencing pain): creating an appropriate " image" to handle the problem (eg., imagining that with each step some of the pain is pushed away from tne body and left behind on the track);
finally, and rehearsing the use of this Jmage in practice or competition,
i
. - -
_ - - _ - _ _ - _ _ _ _ - _ _ _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ - _ _ _ _
_ _ _ _ - _ _ _
.. _.. -
.
-_
..
.. -
.
--
.-
'
s e
F
.6
.
Im_agery For Goals Athletes can " program" their goals by using imagery to
"see", " hear" or " feel" themselves achieving the goal.
imagery can be very realistic (a diver imagines her routine The off the high p?.atform--getting set, executing the dive perfectly, the judges award a high scora) feeling the coolness of the water and watching or very unrealistic and exaggarated (a discus thrower imagines himself as superman lifting the discus without effort and hurling it out of sight).
.
GOAL VERSUS PROCESS IMAGERY Should athletes use " goal oriented imagery" or " process" imagery in a performance enhancement program? Both types of
-
imagery can be very valuable.
Goal-oriented _ images of winning races, receiving medals or~ breaking records can be before competitions.very effective images when used before or during workouts or Such images which remind an athlete of his or her goals can be potent incentives for keeping energetic, att?ntive and motivated during practice.
rehearsing one-a goals before a competition can also have a And, motivational energizing effect.
But during the competition, be distracting ra+%e;any concern over goals or outcomes is likely to than helpful.
Thus,-for most athletes it is better to e a;
- oal-oriented imagery during-competition.
Images af winning or setting records can shift quickly into concerns over losing or slow times:and:in either case such images are likely to be distracting and to interfere with performance.
Process imagery is-focused or. the _ performance itself.
What sights, best performances?-feelings and sounds are appropriate for_your your fastest, What images are necessary for you to ski distracting thoughts? making the quickest turns and eliminating all Each athlete has his or her own keeping the mind focused and "into" the' competition. appropriate important during competition because it keeps the athlete's
-
mind on the key elements of-performance instead of winning,.
losing, how well opponents are doing, the crowd and other-i extraneous factors.
In general, rely on goal-oriented imagery as a motivator in practice or before a meet and use process imag you are performing.
-
ery while
.
-
v-
w.
.
'
,
.
.-
SHOULD IMAGERY BE REALISTIC OR EXAGGERATED?
Researchers have not definitive manner.
answered this question in a
"Should Goals be Realistic or Unrealistic"?Another version of this qu ask, character of the athlete.in both cases depends to a great deal on the psych The answer likely to be a problem for athletes who become frustratedUnrealistic g
"
discouraged when these goals are not achieved.
or athletes, setting extremely high goais, For other even when unattainable, their best.
acts as an incentive or motivator to perform individually.Each athlete or coach needs to make this judgment lead to frustration,If unrealistic goals.or goal-oriented images images are effective as motivatore,then set more realistic goals.
If such then use them.
describe "how" Exaggerated images are often used by athletes to they want to perform, not what goal ~they are seeking.
like a punching bag, Examples might be images of a boxer pumme opponent over the bar and soar to an indefinite height, Hor-of aof a high jumper who ca tor.is player who has eyes on her racquet and can "s
- quet's perspective where to send the ball. Such ee" from
-
communicate in a symbolic manner,
.
a clear message c >;
what is expected.
Many athletes have used urirealistic images that work for you. goals and exaggerated imagery successfully
,
Select GUIDE FOR INCLUDING IMAGERY IN TRAINING PROGRAMS 1.
Imagery is like any other skill: it must be practiced regularly to be effective.
2.
Imagery training should be included as part of the athlete's regular practice routine.
If it is added as an " extra" to-be done outside practice, sends a clear message about this to physical practice.
its priority relative 3.
Imagery can be used effectively before, after physical practice.
during and (a) "Before" sessions can be used_to prepare athletes for what will happen in practice or to create a particular mental state.
(b) "During" sessions can reinforce how imagery might be used during an actual competition or help athletes learn new skills or strategies.
(c) "After" during the day or give athletes the chance tosessions can help e recover after a strenuous practice.
.
.
,
.
I
-
4.
General imagery skills such as rehearsal or'
concentration can often-be taught in'a group setting.
Specific skills such as problem solving or developing competition scenarios are best done individually.
5.
_The evaluation of athletes' imagery skills is a good starting point for an imagery program.
The feedback will help build interest in the program and motivate individuals to participate fully.
6.
Regard imagery as simply another useful skill in an athlete's repertoire of physical and mental.
_
skills.
For some people it.will be-of enormous benefit, while for others it will.have little effect.
Used properly, imagery training is a powerful tool for integrating the body and mind in sports.
f
!
b _.
_ _ _ _ _ _ _ _ _ - _ _._ _ _ _ _. _ _ _ _
. _. _.
. _ _ _
1~
,
,
.
s
.
-ADDITIONAL INFORMATION SOURCES Suinn, Richard, Seven Steps To Peak Performance.
Syor, John and Christopher Connolly, Sporting Body,_
Sporting Mind, Cambridge University Press, New York, 1984.
Williams, Jean (Ed.), Applied Sports Psychology, Mayfield Publishing, Palo Alto, 1986.
_
p==
.
.
I
.
-
,..
cQh
3 e
,ron g
,
,
=
-
.-
,i
b
\\
l %'
&
s
%
t OW yy,qgALIZATION TApg This Handout includes:
I.
Purpose of Visualization Tapes.
II.
Preparing to make a Tape.
III.
How to structure your own Tape.
- _ _ - _ _ - - _ _ _ _ - - -
.
.
I. Purpose of Visualization Tapes'
before competitions. Virtually all successful athletes use mental preparati For some athletes, on simple as thinking about general strategies for the upcomingthe prepa competition.
Many elite athletes, however, and organized approach to mental preparation, and visualizatitake a more is often a central part of that on fantasizing or wishful thinking, pre-competition training.
Unlike and organized use of your mind to improve performancevisualization is a purpo physical training. requires time and practice, and it is not a substitute for It
.
your mind let your body down in a tough competition,If you h ac visualization could be very useful for you.
then you to prepare for competition situations, Visualization helps successful, and maintain confidence in yourself.see yourself being successful performance, you can anticipate obstacles,By rehearsing a the opportunity to use recognize performance situation. your strengths, and stay calmer during a II. Preparations For Making A Visualization Tape What You Need:
SCRIPT-Your tape will work best about what you want to include and how you want to presentif you have t Like a carefully planned competition strategy, your tape it.
be detailed and organized.
should that you make the experience as real as possibleWhen visualizing, it is importan
~
involve all of your senses, and that you just
,
It is much easier to make a thorough tape if you write downlike you would
" script" ahead of time.
Before making your script, a
think abouc a few aspects of the upcoming competition:
you need to 1) Your Strengths-One of the most essential parts of a visualization tape is a focus upon your strengths as an athlete In preparation, write down a list of your strengths as a competitive athlete
.
What parts of your competition performance have help the past?
ou in about?
Are you stronger than most opponents?
Are you usually quicker or faster than opponents?Have yo
Are you steady?Are you mentally tough and good at enduring pain?
Are you aggressive?
Have you developed specific skills?
i I
_
_ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _. - _ _ _ _ _ _ _ _ _ _ _ _ _. _ _ _. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.
.
respect.
important If you do use someone else's voice, it fairly detailed and personal.is especially to make the script What to sav en the tace?
and someti=es to a specific competition, butThe tape should b sport following four sections as general guidelines:
we suggest the 1)
relaxation portiThe relaxation phase your tape should begin with a bri f and nervousness.on which allows you to shut out outside thoughts e
taking long slow deep breaths from your diaphragmOne goo breath a bit, then slowly exhaling.
, holding the and you should notice feeling more Do this a number of times relaxed almost If you use another relaxation technique, write thatimmediately.
script here.
into the 2) Pre-Event-Going back to images you collected, your competitors, you feel calm and confident.a very posit try to simulate As you notice know you will have a good performance.about your prepara
, you the two or three things you need to focus on in this comYou remind yours and begin to think about your strengths and why ycu will do well petition, 3)
that you feel calm and powerful and in control of yo mind.
You focus on the two or three things that are ke o y and success, and you see yourself successfully doing them. ys to your get too nervous or start ((If you stop for a second and go back to some relaxation exerc just Don't get mad at yourself, because it is natural to have to work through the rough spots.
Remember,
.
if you get nervous or get during actual competition. negative taages during visualization, it is lik have an advantage over others who must deal with this duri competition))
When you can see yourself performing successfully, follow your performance and. notice how good it feels to be do well.
Try to use all your senses and feel the different sensations as you begin to recognize that you are succeeding.
You stay calm and relaxed and keep up your strategy until the very end.
You are not cocky, but you are not defensive:
that maintaining your strategy will enable you to succeed you know
.
4) Post-Event-a moment to feel whatAfter seeing yourself perform successfully, take success is like.
have a good clear image of whatEnjoy the success for a while.Tak and notice everything.
After you it feels like to succeed, go back l
.
.
_
..EbA
3Ja 2_y_
cl::
2.
J a_>
-
-__L.._i.-4_
.a_s
.mv._.__
a
,
e w,.m_J h
_.#.<_
w a.6 m
a
.'
t to a few deep breathing / relaxation exercises determine how you feel.from the visualization, gradually becoming more
,
You should be calm and relaxed Try to
.
are very aroused and excited, you should do a few more m.If you relaxation.
It is sometimes helpful to replay your
. inutes of visualization, because you may nocice aspects of your perfo that you don't notice in the heat there is some*hing you don't like about your perfoof actual competition.
rmance If visualization, write a corrective statement i rmance in the in your next visualization you can modify the behavior.nto the script and l
l l
l l
l
!
l
!
,
._
- -,
- _ _ _ _ _ _ - - - _ _ - - - - - - - - - - - - - - - - - - - - - - --
.
.3 j
Qe
-
-
uc
'
-
J:
~m
.
,,.
v.
,
a..
.
'
c,gp:'j
u
...
.
,,
,
,/./
.
_
_
?.m
.,
gm
-
'F
>
h bo
-
-
2 A1
'
y
...
. - - -
.
...
.
. -
_ -.- - __ -
_. -
... _ _ -... - - - -
-
_ _.
-. -.. ~
-.
- _.
.
.
.-
SUGGESTIONS FOR MAKING.A PRECOMPETITI Many athletes like to get in competition by listening _to a the right frame of mind ' for can be played a
week _before they compete.
~ Tapes tape before competition, and now, competition, the night before competition.with the invention of Walkmans, listened to AT their favorite they can be Some athletes use tapes that have
'
psyching-up or others listen to tapes that contain relaxing music on them, while both music AND instructions. things to concentrate on before
.
on.
Some tapes contain You can make a tape, or tapes, for yourself quite easily.
First, decide if you want a music tape or a sugg both.
If you want a music tape, decide what estions tape, BEST before competition.
or music makes you FEEL Remember, you may be tape on the team bus or perhaps at the track or the listening to this pool, wherever your sp, ort event takes place.
swimming Next get a copy of the music you want it is on a record, CD, or cassette.
to listen to, whether in a boxing event might make a tape with the An athlete-about
" Top Gun", and " Star Wars".
to compete Another athlete who is tense before a shThis tape will getthemes fr him psyched up.
make a tape with music from "Chariotsooting competition might Canon", and some of Fire",
"Pachabel's
,
calm the athlete down. favorite classical music.
!
This tape will help
-
A suggestions tape will help you get mind for your best in the right state.of performance.
It things you need to be thinking about will help you cue in on the make your own tape, more effective you can personalize it.before competition.
be If-you-attuned to your special and unique circumstanceto a ma than listening not that.is s.
First, athletes preferdecide whose voice you want to be listening _ to themselves, listening to. Some
someone else to but be others much prefer speaking.
If you select someone else, it should be someone them who.. g1ves you confidence when you listen _ to
,
--.a coach, Next, remember _ a close companion, or your sport psychologist.
to try your tapes out before you go
' to competition ~
Try them out at competition situations, practices, and under-simulated so that you can be sure they have the right feel, and that need to modify the tape, you will have plenty oftheir timing and p
,
right.
If you time that way.
l
"
,
l
._, _., _
_
,_
.
.
.
.
~....
-
-. - -
_
-
...
.
SAMPLE TAPE
.
1.
Relaxing music.
2.
Relaxation instructions.
this is just a relaxation. These should be fairly brief unless-tape.
An -example of a-complete relaxation scr1pt is provided in another handout. Concentrate en deep breathing and relaxing the muscles.
3.
Self-suggestions.
Some examples include:
- You have every reason to believe in yourself, as a competitor and as a person.
- You are well prepared.
You are physically ready.
You'are mentally ready.
- Remember your goals.
You have the ability and the determination to reach them.
- You are fully prepared to dig deep within yourself to meet these challenges.
- Focus on what'YOU have to do and on what YOU are going to do.
Nothing and nobody else matters.
- Focus on WHAT.IS WITHIN YOUR CONTROL.
4.
Final suggestions.
Go over the key points that you need to:
focus on.
These cues may be thoughts that have helped you to attain your previous performance bests.
For example:
- Be powerful at the start.
- Nothing will stop you today.
- Concentrate on good extension.
- Just focus on your body and let your performance come naturally.
f p
_.,,
y
,
,,
- -.. -
-
-
--
.
. _.
.
- -
. - -
'
,
-
i
\\
,
ADDITIONAL SUGGESTIONS
.
These self-suggestions have been found helpful by other athletes.
Some of them may work-for you:
- The challenge before me will require an all o am fully prepared for the challenge - I will push'my li i ut effort. I
-
- Because of the physical preparation I m ts.
mental plan I have done and the have developed Ih
- I can draw upon all my resour,ces. ave an advantage.
- I will be as I choose to be.
- I choose to be my best.
- I choose to be the best.
- As I prepare to start, concentrated and absorbed by the task.I will become completely and in control.
me or distract me.I will follow my plan. I will feel energized Nothing will disturb
- During training for this event, for a superior performance.
I prepared my body and mind
I planned and prepared my precompetition warm up t an ideal feeling state for my performance.
-
o create
- I developed a refined competition plan to take me t goal.
o my
- I tactically. simulated my performance physically, So I am ready.
mentally, and
- I have every reason to feel confident.-
better than anyone here.
I feel good.
I am prepared.
I am the best.
I am fully completely determined to achieve my goal.I am confident in m expecting me to do well, People are and I am going to show them what can do.
I These are some tape for you:
suggestions if someone else is reading the
- Stand tall!
Let
- You are already starting to feel the way you wa tthe champ and you will perform the way you want to perform.
.
n to feel,
- You are and will remain mentally calm absolutely determined throughout.
alert, and
,
- You have a well-practiced plan that will unf ld imagined, hoped, and planned.
o as you have
transformed into the explosive power of your stIf you
, it will be
- You are capable of adapting and refocusing in the f art.
any obstacle.
- ace of
- You can control what you do.
be focused and in control throughout.* In the competition and you will
,
- You can control.your own destiny.
.
,
!
i
,,
..
._- -
._
_ _
. -
. _.
_- _. _ -
.
---
.. _
_
__.
.
.--
. -
.
- 1 LOST PRE-EVENT-TAPES t-What happens if you prepare a pre-event tape and then can't
. find it.when you need it.at the competition site?
Treat it as
.
you would a piece of your equipment, and you will not likely lose it.
In case of loss or breakdown, perhaps-it would be worthwhile to carry a-copy of~your tape or a written copy of your' pre-event
.self-suggestions as'a backup.
If you end up not having a copy of -
your tape to listen to or a copy-of your self-suggestion-to read to yourself, sit down quietly in your own space and think about:
how you want to feel.
Try.
to recall your major self-suggestions.
That should not. pose a problem, because as a result of your previous experience with the tape,-
your self-suggestions should be well-learned by this' time.
.
=s-v v
-<
~
,
,
.
8 e.
.
RE-EXAMINATION OF THE ROLE OF ASSESSMENT IN PROMOTING STUDENT LEARNING By James E. Bruno There are three major types of policy analysis studies that are associated with issues that involve student testing and assessment, sThese are che "why" of assessment, the "how" of assessment and the "what" of assessment.
Why societies assess their young is embedded in the collective culture of mankind or as Carl Jung would say the
" collective unconscious *of man.
The rite of passage of youth, the Bar Hitzvah, the confirmation, the SAT, etc., always seems to entail some sort of " ordeal" or rite or performance based test for the youth of a culture.
Even the ancient Greeks were keenly aware of the need for assessment.
When the Sphinx asked Oedipus "what walks with four legs then two legs then three legs" and Oedipus answered man, he was " judged" by the Sphinx as being " king" material.
History's first Criterion Referenced l
Test was thus recorded.
Over the millennia of man, only the l
format of testing -- oral, performance, written-essay, and multiple choice has changed.
The purposes of testing and assessment seem to remain the same regardless of culture --
sorting of individuals.
l
.
-
-.
- - -
- - -
.
.
- - - -
-.
. -
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _
.
..
.
A, why We Test In a modern twentieth century, industrial state, the "why" of testing in the classroom has two important purposes-These are:
(1)
Summative Evaluation -- judgment, selection or
]}
sorting of students (2)
Formative Evaluation -- assessment of a student to
_
promote lassroom learning.
Sometimes tests are used in the classrom for both summative and formative evaluation purposes.
NaturaJly it is the latter purpose of testing and assessment that is perhaps most important for promoting student learning.
Assessments for formative evaluation have to be concerned about the natural tendency of students to forget material (previously taught) and the propensity of students to learn new material (to be taught).
,
.
Both forgetting old information and learning new information are strongly related to the expressed confidence that students have in their current information base -- i.e. their levels of reliable (accurate and confident) information.
See Figure 1 for
,
an analysis of how learning new information (i.e. percents) is based on the confidence in previously learned information (i.e.
decimals). Also note how previously learned information is forgotten over time in direct relationship to expressed confidence in that information.
.
.
.-
.
-
..,
..
. --
. ~. -. -..
-. _
.
.
'
.
...
.
Thus "why" we assess students, at least from the
.,
-classroom teacher perspective, is basically twofold.
.1 (1)
Judgement, Selection and Sorting (grades and Summative Evaluation placement)
--
(2)
Enhancing Learning of new information and minimizing the forgetting of old information -- Formative Evaluation From the student (and parent) perspective the "why" of testing and assessment includes:
(1)
Insuring that the student-is "on line"-in terms of meeting the educational objectives of'the family and culture (2)
Articulating the school's instructional program to-students (and parents) so.that students can help themselves learn.--
From the school administrative perspective the "why" of testing includes:
-
(1)- Resource allocation (micro level)-to promote the effectiveness-of the instructional program (2)
Teacher accountability and assignment (3). Curriculum alignment and planning.
-.
.
,
-
.
.-
.
. --
.
.
.
-
-'
,
From the state perspective, the "why" of testing includes:
(1)
Educational accountability (2)
Resource allocation (macro level) to promote the state's instructional objectives.
B.
How We Test
.
The " bow" we assess students is based on two major perspectives to assessment.
Will the evaluation be objectively scored (multiple choice, low cost, machine scoring) or subjectively scored (performance based, portfolio analysis, high cost teacher evaluation) ?
There are important reasons why objective testing is presently favored over subjective testing.
Some of these reasons include economic (cost and viability), Legal (bias and due process), and Educational (domain of skills, information for instructional decision making) considerations.
In general objective testing is preferred over subjective assessment because it can assess a wider domain of skills, eliminates evaluator bias from the assessment process and most important is economically viable for the educational agency to perform on a mass scale.
Presently there are three types of objectively scored tests that are available to the classroom teacher.
These are the Norm Referenced Test (NRT), the Criterion Referenced Test (CRT), the Information Referenced Test (IRT).
The following table depicts some of the main characteristics of each type of test.
. -
-
-
- -
_
.
r
,
Type of Type Mechanics Objective Scoring Norm Referenced Test Student Score is Referenced Right or Wrong to a Normal Distribution (one dimensional and Cu rve non-reproducible)
Criterion Reference Student Score is Referenced Right or Wrong Test to a Selection Standard (one dimensional and Score non-reproducible)
Infarmation Student Score is Referenced MCW-APM Referenced Test to Information Standard of (two dimensional and attainment reproducible)
The Norm Referenced Test such as the Stanford Achievement Test (SAT), and the Comprehensive Test of Basic Skills (CTBS),
is typically used for summative evaluation.
They are notoriously sensitive to student guessing, test preparation and culture bias.
These limitations make them nearly useless sources of information for the teacher for promoting classroom learning.
The fact that the scoring system used on a Norm referenced test is not reproducible, since guessing is implicitly encouraged, further limits its practical use in classroom formative evaluation.
Finally the tautology of assuming that the normal distribution model of attainment ~is r
perfect, then designing test items to fit that model, underscores the pseudoscientific nature of the Norm Referenced Test for use in Selection or Summative evaluation.
In short, the Norm Referenced Test is generally considered by teachers to be useless for measuring anything but the narrowest, instructionally irrelevant set of skills.
The Criterion Referenced Test (CRT) such as mathematics or language placement test are far more "in line" with teacher needs for instructional decision making and used in promoting
.
.
_._
_
_ _.. -.
__ _.._._ _.
.
'
.
..
-.
classroom learning.
Unfortunately, due to the one dimensional-system of evaluating (Right or Wrong)-or scoring these types of j
tests, they are also extremely sensitive'to guessing.
In i
addition there is a strong inverse relationship between_the selection standard or criterion score used on the CRT and the i
number of false positives generated by the assessment process.-
)
The Information Referenced Test is a hybrid NRT and.CRT.
Like a CRT, the main emphasis is for classroom formative
,
evaluation and like an NRT the score is referenced to some fixed standard or model of attainment The Information Referenced model of attainment evaluates students on a standard of being informed, part informed, uninformed and misinformed.
The i
scoring system used in an IRT is also a reproducible system.
With a reproducible test scoring system the maximum score for the student is obtained if and only if the student does not-guess, but-answers each test item on the assessment with his or her actual information.
The IRT procedure is fully optically
!
acannable (see Figure 2) and generates an individual i
education plan (IEP) for formative evaluation of each student'
assessed (see-Figure 3).
A class information needs profile (CINP)
(see Figure 4) -and a school information needs-
-Profile (SINP) (see Figure 5) are also produced with the-IRT procedure.
..
,
.
..
-.
.-
- - -.
.
-____
_ _ _ _ _ _ _ _ - _ _ - _ _ _ _ _ _ _ _ - - _ _ _ - - _ _ _ _ - - _ - - _ _ - _ _
- _ _ - - _ _
__
'd
.
e
.
C.
What Are the Limitations of Present Testing Practices Policy studies that deal with the "what" of testing and assessment practices, tend to underscore the severe limitations of presently used CRT and NRT formats.
The search for alternative testing and assessment formats such as the Information Referenced Testing (IRT) format and the Answer Until Correct format, are research efforts to address some of these concerns.
Here are just a few of some of the major concerns, as found in the research literature, with presently used CRT's and NRT's for classroom use.
Teachers are not provided with information needed for
instructional decieton making.
In many inner city schools, nearly 1/3 of the students
-
test at or below the random chance or expectation
_
level.
The correction for guessing formula assumes that all
wrong answers on a test were the result of random guesses.
This formula thus over corrects and yields a biased estimates downward of actual student attainment.
When corrections for guessing are not used the estimated
of student attainment are a biased estimate upwards of actual student attainment.
Students actually know less.
.
--.
.
.- -.-
.. ~.
.,
-.
-..
-.
- -.--.- -..
.
,..
$
....
,
.
. 8
!
The one dimensional Right or Wrong (RW) test' scoring _
system is in reality only a recognition (and non recognition) system and promotes a student test score maximiring strategy that necessitates guessing.
It is 14 non-reproducible test scoring system.
There is no partial credit given with presently used
'1 multiple choice testing formats.
This limitation leads to large disparities between the student test-score maximiring and teacher conforming strategies.
This disparity in strategies is inverse to student attainment levels.
Because of the large amounts of misinformation, lack of
i information and partial information, presently used
NRT's and CRT's scored with one dimensional R-W test
-
scoring systems-are insensitive or-yield erroneous
)
results when used.with low and medium attaining students.
,
,
'
Misinformation knowing something wrongly 1/2 + 1/3 = 2/5
cannot be assessed with present practices..Thus-the important pedagogical strategy. of reeducation for -
. misinformation cannot-be-addressed.
The instructional leadership TRIAD. -- student (and
parent), teacher and-school administrator are not part
. of the evaluation-loop of instructional support.
,
...~-t
,
w a
e
.
.
.
,
.
Current CRT and NRT testing practices assume that the
signal of instruction is perfect and that all imperfections in instruction are assumed to be with the student.
In essence, instructional validity is not considered nor is it assessed.
The Normal Curve Assumption of ability, which is the
foundation of the NRT format, is predicated on two erroneous propositions.
These are that all student responses are certain (flipping a coin -- heads or tails
-- right or wrong) and that the narrow domain covered by the test is an unambiguous measure of general student ability.
Basically the normal curve model is assumed to be perfect and all imperfections to produce the normal curve model reside with the test item rather than the student or the model itself.
There is by design a ceiling and collar effect with
grade level NRT.
This limitation can and does lead to enormous distortions at each end of the distribution for assessing student attainment.
There is no cumulative feedback curricula information
provided to the teacher or school principal regarding common areas of misinformation and lack of information.
.
.
.
.
.
D.
Conclusion In conclusion there will always.be problems with assessing student attainment in the classroom.
Biochemistry has not advanced to the state where students will produce an enzyme when he or she has learned algebra.
If this were ever the case teachers, by means of a simple blood test, could then clearly and unambiguously determine if actual learning had taken place.
Until these advances in biochemistry come to fruition, all educational assessments will have to be " passive" or one step removed assessments.
What seems to be needed in education is a total re-examination of the basic philosophy of why we test.
The philosophy presented here is that all testing should be used to promote learning.
The way medical doctors use. testing to promote the health of a patient, teachers should use testing to promote the learning of the child.
The way medical testing is used to monitor health and suggest strategies for improving health, educational testing should monitor learning and suggest appropriate instructional interventions to enhance learning.
The design of the Information Referenced Testing concept is directed specifically at addressing the formative
!
evaluation needs of an individual student and thus promoting learning.
As other multiple choice objective formats and subjective testing formats become available, they should all be evaluated against the economic, legal as well as the educational standards for formative as well as summative evaluation.
Unless l
an alternative to present multiple choice, CRT and NRT testing
i
.
..
,
,~
'
practices occur, school reform vill have no chance of succeeding.
A "fix a on exactly what has to be reformed to improve learning will be missing and the important relationship between testing and learning left unaddressed.
Information l
Referenced Testing is a significant step in filling the l
important formative evaluation niche in schooling and might provide the type of " sensitive" instrumentation that is needed to service the demands of the school reform movement.
I i
l
f Levelof cmranee a
in Present
- '
Infonnation
'
(decimals)
Cmadencein
_
/nfc-_
1.0. Level of I
emed\\
Levej or 1/i 1.0 Ccondencein Leammg (Remembering)
New Prevmusly Izformanon lemmed P et Informed (percents)
Old !nforma: ion (decimals)
.75
.7%
.7%
,
E
.ht
.50
.5g
,
inf o ed f
.
.25 a
.24
,
e si o formeN 0.0 I
.0
"
3-
1
1
3
Time toleam New Tune toForptOki Information Informanon (Percents)
(Decimals)
(units - weeks - months - years)
Linkage Between Forgetting (old information)
Learning (new information) and Confidence in presentInformation Figure 1 Learning New Information and Forgetting Previously Learned Information as a Function of Expressed Confidence in Information
-
- -
.
_. __
_
..
_ _
_ _ -.
-
.
_. - - _ __
.
+,;,
.,.91 g. g o.:9:g.
o
. man,,..u.a nea. ~u.u ~ e,a
.
g
-
.-
apj 3.$-@ @'
@..QLg
!Q l AM sURE 1 1 AM NOT SU7.E I DO.rT KNON
- g.
O : 9. O.
O O:O O
sO' O O OOO O
=
'
iO O O O :O. O
- O.-
..
.,.
-
,,.
--
Ma.a..w.,~~.
,*..
-
=
-
-
co" n n.O O O
.O O 'O O
n. "on..
"
-=
ioco...n
,,
-
- 1. 0. O. O O00
1 AM SURE
+30 100
,,
n.O O O OOO O
OOO
=
=
i. O O O.
O O _O O
l ^= 5=
--
see, ann.,
seen.e -
+10 100
,,,
is.O O O OOO O
O O
O
'
=
=
li.O O O OOO O
' aa' =
=
G
-0-
.amm
,,,
2 @@@@@@@@QG@OS I
! =
A l
! -
3 @@@@@@@@QG@OS I
! =
I t -
4 @@ @ @ @ @ @ @(D G C O 3
! asm I
!" "
F G
l @ @ @ @ @ @ @ @ CD CD @ Q S I
!==
I
! am s @ @ @ @ @ @ @ @ CD CD O C S I
! mas I
! =
E H
! mm 1 @ @ @ @ @ @ @ @ CD CD E C S I
t sum M
I 8 @@@@@@ @@CD CD OC @
!==
,
.
t -
o I
I t _== -
8 @ @ @ @ @ @ @ @ CD CD C C S I
'-
t aus I
te @ @ @ @ @ @ @ @ CD CD C C S I
( -
(==
I 11 @ @ @ @ @ @ @ @ CD G @ C S aus e
,
i
,
s l
! -
L K
J
,
12 @ @ @ @(D @ @ @ CD CD @ C S I
! man
! =
,
I sus : 13 @ @ @ @ @ @ @ @ CD CD C C S I
i t am 14 @ @ @ @ @ @ @ @ CD CD @ C S I
I==
g-m.
-l-I t==
15 @ @ @ @ @ @ @ @ CD @ @ C S iueowtAwt oinernows t eue ILon MAnKING ANSWCAS I-
-l ens 18 @ @ @ @ @ @ @ @ CD G S C O EXAMPLES
,,,
oney m at g-t==' 17 @ @ @ @ @ @ @ @ CD CD @ 0 9 wnomo
! sum 1@@@@@_.
,o way,,, g, i,.-
- I. som - II @ @ @ @ @ @ @ @ CD G @ C S
_.
n-
I - uma wnowo I
't
= = - 18 @ @ @ @ @ @ @ @@ CD @ C S 2 @ @ @@@
- Ma8* '**v tand a****
I visum ww ene.
-l
,
- ,,,
- "*"
-l
'
t enn 38 @ @ @@ @ @ @ @CD CD @ C S wnowo-I
- t aus 3@@@@@
.g,,,,,,,,,,,,,, _
,
21 @@ @ @ @ @ @ @(D G @ C S v
n m asung.
I'
-
t==
! ase neowT I
( = 22 @@@@@@@@CDG@CS 4@@@S@
' Ma** a* ***r "*** *
'l
- 8'*""
I i em
'
Figure.2 Optical-Scan' Sheet to Support Two Dimensional Information Referenced Testing (IRT)
'
in the Classroom
.
I r
--.y-c
.m r.
-
,,
, + _ -
,,
-.-c
-,,
..
.
-
=s
-
,
_ _
-
-_ _ _ _ _-- _ _ _ _
-
- ,
.
-e
,
.
,
.
L
... IET.predamed N shade plas GIP) to asypert
hisenssedemand seseplag af medens hi tornamesEvelmass h HMahrmasse af Emmannass emessys wtses yes sus awe W e aswear km was avang I Eeve tassment uplain wery the answer was osmag that yes shongk was asamt and wtry anseer antwa was emmest Tes Ann CHF FIATE) Desertpass haeusdessi O m amness
M m-hau 1,3. ad 3 Asses Omas: Mah. p. M -
M Prebisa Selvtag (WP; "M"-
2 Sep Censre: Mah. p. 32
M Addhan and Sekunndes af hogers
-
Osmeral Man p. e
M hopes NaherIJssa Ommal Mah, p. 393
~
.
.
.
.
.
.
.
.
.
Esissises thWwasd Gash N11supasas Cuessys ths you said yes didn't lase -esis your housaur amplais esas seassys
,
a pen.
Tam hun GNP STATI)
har e nosal Qes EEAuseas
V Lies Malapas
,
Osman! Man p. 190
_*
11 -
V hables saistas ("dF)- Addiden seg u
.-
G una s! M ak, p. 306
V Adenes and satirestaa Whud Numbas Osmars! Mak, p.190 IJ V
Pues Posen hoe se % p. 38
.
.
.
.
.
.
.
.
-.
ir.--h pm,esay latened hses as m
'
Commys whsse you wason't aan af the sesse=4sve per lamresar metse esse mesmys vte yen.
Tea hun GNF STATE) Dunerlydse harusdonal Dou Ranrunas
P Prebisus Selvens (WF. " 1"
_. I ag
-
Oman! Mah, p. 41 3i P
penant af aMumher Oussret p. 300 II
g Dummelsgud 3 Phases
.
,
P Calculats anartsa!Espressieme hos a p.1,pA4
.
.
.
.
.
.
.
.
.
.,.
Figure 3 Student Individual Education Plan (IEP)
Generated with the 'IRT Procedure 7'"________u
_
_ _ _ _ _ _ _
_ _ _ _ _ _
___
_ _ _ _ _ - - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _
.,.
.
e
..
,
o:.
.~ ',. 3ALTgrusend das hearmadas profte a suppest h peoptag:
,-
aumreemanal et)isens ser as hade Mdalensed Ennalnames hans far Your Qam
. Yew andas how *teeg htennases b esa annaspt N. Develop Jersweseal massaah a ganeessus ham & 7-a peia= as hashesess wta haak hermudas k hans sesa.
Tes han Pensms of Qass cosampt Dessnydences Raimums:
M 4.N Nummen! L,
' - '._..
hase a Alselma.7A4
.1 4.0
'.Munkw&hs Gamme! Man, p. 395 to RJ7 laest Onnass Mahpla Ouesni Mah, p.130
g,gg e M.r " -"--' hanhas
_
OsamW haam, p. 413
.-
.
.
.
.
.
.
.
.
-
Ushtermed Test !asus por Year Gass Russh.:. pense,lly leak hissmaaso h esa essespa seas. Saak tassessnes t
.aedad. pumpen bask W asmaak a sansk mass bask assasps.
Test kna Pereunt of Qass Ceasapt DesertyneMll pees RaAmunes Ceaselammu
0.21 Calaulsdens Guesen) Mst,p.300
-
4.l0 Addisee -Cusanals 1.3 & 3 Plasms Quesm! Man, p. 94
0.23 Pro 61am Salves (WP)==
unwi - %
r
.
.
.
.
.
.
.
.
.
'
Persty Infocused Comesyt Aress ser Yaur them Yew summans how insensplass er sammhls tsfunnsees 4: esas unnampt asses. A tessuth srnsw is aseded b Aans sea.
Tomtass Peneet of cas Coussyt Dessrlydan.cois Rafasenes-3.
Anheessen (1 & 1 Dusama! Passa)
Ovesm1 Wah. p. W 9'
8.74 W hip h Pueym0
~
Osamm1 Maa, p. 304
4.7f Coevwwee-Purums a a Deshmal Omanal Man, p.3M
-
,
.
.
.
.
.-
.
.
.
Figure 4 Class Information Needs Profile (CINP)
Generated with the IRT-Procedure
.
h
.
'tr
,'
.
-
a
- '
- - _ _ _ _ _ _ _ _. _ _ _
_ _ - - - _ _ - - - - - _ - - - _ _ - _ - - - - - - _ _ _ - _ _. _ _. _ - _ _ _ - - - - - _ -. _ -. _. - - -
. - _
--
-
--
- - - - _
_ -
- - - - -
.
e
_ _.
_ _
. _/. _.:.
e
..... _....
-
'g'!.Pi."1: 11:1.!.**P!'!.M. 7,1:8.:J: 'a't:!!!"
.
..
.
a uu u-
.
___n.f.i.r..J..FLituti: Ult.!!st L11U."UP:no"It""'
".
,
. n, g n u..,h n. n n. n n u
-._ p..
. g......_ g. g n, f
,.
rin!.F::1:-'rf!p gg.g,..,g p
......
.
_....
.
...
_. _ _
.
! * '-5"
'
" ' ' "
"
-;rane..ag.-riira""-
---
-
n
---
-....j g g f gg.m g u..
u en.n
...
... n.i
.
sta ge 1, m.
ta l.ne63 p.estWL a.
.M 9898 8.4?
.......
.
,t..,l nf.a.n. c.,i.i n n n
.I m
u.. n
..a
..L
.e.
'""k - -
..
a
.
...
.
i.. n b...... it; pan.g,.P.ld"ini.
_
u___sn.e-
-a6
._g
-- -
a n....
..n l..ag..... gnpgie.emmu..
i_
n._.in.n._.
.
..n...
.a r.fh,g; m.f.., unintin g;;
- ,.
.
.
.
in.n
..n p
Jgjjg.g-plentne. ause _ _gety'=n =.tnen f!"r.
.g
.___su.n
.,3
.
_ _ _.
8!j'
j u
en.n
..n i tt :til!:: 1 "1<t " ". * ". "i n
'
"
Sua"-
- * "
-
-
.
- gg,p;;g.,
qing.
.
e n..
..n
.
>.___i a.. &
e.'
..;g.gtgg;; g=f.. na n........__..
y;96t.ge,gpyg,= numi
.
n.n
....
an. im.u...,.. n i n e n n
.n
-
u n.n..
..n
.
-
..n,_._.; gen.b..;,.;;.ge;..
u e.n gggp
,p,nv.m.
,
n.n g,.ga =
...___n. n
...
...
_.
gmurn.i...usu nn.
..n e
n
,i.n
..g.
!! int'!ii.ni.l.
u s..i.
.
n.n
..u u..
mi
..
"[M.u.l,T!'ent!.mi.!! 17.3Jtill Pl.*%ut i
'
-
unu
,
M8tt testauttlemai. Mit.ta6s aunt attats in tatt 443 t er ac e e,g g g.t.,.csl.eteeres.es.e.,ait on. cst.es n
ava.
es. stet to.ttet e. a e.6ert aLu 6.as s as. o.it e.se
'
.
a e
e
.
e a
Figure 5
"
School Information Needs Profile (SINP)
Generated with the IRT Procedure
.
.... ~.
- .
.
.
..
C___.__________.__________._________.______________.________.
_ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _
- _ _