NL-21-0051, Quality Assurance Topical Report, Reduction in Commitment (Pandemic Related)
| ML21127A184 | |
| Person / Time | |
|---|---|
| Site: | Hatch, Vogtle, Farley, 07200042, 07200036, 07201039, 07100333, 07100521, 07100726  |
| Issue date: | 05/07/2021 |
| From: | Gayheart C Southern Nuclear Operating Co |
| To: | Document Control Desk, Office of Nuclear Material Safety and Safeguards, Office of Nuclear Reactor Regulation |
| References | |
| NL-21-0051 | |
| Download: ML21127A184 (124) | |
Text
&KHU\O$*D\KHDUW &RORQQDGH3DUNZD\
5HJXODWRU\$IIDLUV'LUHFWRU %LUPLQJKDP$/
FDJD\KHD#VRXWKHUQFRFRP
0D\ &)5 D
1/
'RFNHW1RV
$771 'RFXPHQW&RQWURO'HVN
861XFOHDU5HJXODWRU\&RPPLVVLRQ
- DVKLQJWRQ'&
(GZLQ,+DWFK1XFOHDU3ODQW8QLWVDQG
-RVHSK0)DUOH\1XFOHDU3ODQW8QLWVDQG
9RJWOH(OHFWULF*HQHUDWLQJ3ODQW8QLWVDQG
4XDOLW\$VVXUDQFH7RSLFDO5HSRUW5HGXFWLRQLQ&RPPLWPHQW 3DQGHPLF5HODWHG
/DGLHVDQG*HQWOHPHQ
,QDFFRUGDQFHZLWKWKHUHTXLUHPHQWVRI&)5 D 6RXWKHUQ1XFOHDU2SHUDWLQJ
&RPSDQ\ 61& KHUHE\UHTXHVWVDSSURYDORIDUHYLVLRQWRWKH61&4XDOLW\$VVXUDQFH7RSLFDO
5HSRUW 4$75 7KHSURSRVHGFKDQJHLVGHHPHGWRFRQVWLWXWHDUHGXFWLRQRIFRPPLWPHQWLQWKH
SUHYLRXVO\DFFHSWHGTXDOLW\DVVXUDQFHSURJUDP6SHFLILFDOO\61&¶V4$75ZLOOEHH[SDQGHGWR
SURYLGHDOORZDQFHVIRUSHUIRUPLQJIXOO\UHPRWHYHQGRUDXGLWVGXULQJSHULRGVRIH[LJHQW
FRQGLWLRQV HJSDQGHPLFV
7KHIROORZLQJHQFORVXUHVDUHLQFOXGHGSXUVXDQWWR&)5 D DQGLQVXSSRUWIRUWKH
SURSRVHG4$75FKDQJH
(QFORVXUH³'HVFULSWLRQDQG-XVWLILFDWLRQIRU&KDQJH'H[SODLQV61&¶VSURSRVHG
FKDQJHWRWKH4$75SURYLGHVUHDVRQVIRUWKHFKDQJHDQGSURYLGHVWKHEDVLVIRU
FRQFOXGLQJWKHUHYLVHG4$75ZLOOFRQWLQXHWRVDWLVI\WKHFULWHULDRI
&)5$SSHQGL[%&)56XESDUW+DQG&)56XESDUW*$WWDFKPHQWV
DQGWR(QFORVXUHSURYLGHPDUNXSDQGFOHDQW\SHGDIIHFWHGSDJHVUHVSHFWLYHO\RI
WKH61&4$75
(QFORVXUHRemote Assessment Techniques: Planning and Conducting Audits and Surveys Using Remote Techniques During Exigent Conditions.(35,3DOR$OWR&$
LVDFRS\RIWKH(OHFWULF3RZHU5HVHDUFK,QVWLWXWH (35, WHFKQLFDO
UHSRUWWKDWSURYLGHVVXSSRUWLQJLQGXVWU\MXVWLILFDWLRQIRUWKHSURSRVHG4$75FKDQJH
1RWHWKDWWKLVHQFORVXUHLVDFRPSUHKHQVLYHGRFXPHQWSURYLGLQJJXLGDQFHDERYHDQG
EH\RQGWKHVSHFLILFFKDQJHVEHLQJUHTXHVWHGIRUUHYLHZDQGDSSURYDO7KHHQWLUHW\RI
WKHGRFXPHQWLVSURYLGHGIRUHDVHRIUHYLHZDQGWRHQVXUHVXIILFLHQWFRQWH[W
861XFOHDU5HJXODWRU\&RPPLVVLRQ
1/
3DJH
2Q-DQXDU\WKH86'HSDUWPHQWRI+HDOWKDQG+XPDQ6HUYLFHVGHFODUHGDSXEOLF
KHDOWKHPHUJHQF\ 3+( IRUWKH8QLWHG6WDWHVWRDLGWKHQDWLRQ¶VKHDOWKFDUHFRPPXQLW\LQ
UHVSRQGLQJWRWKH&RURQDYLUXV'LVHDVH &29,' 2Q0DUFKWKH&29,'
RXWEUHDNZDVFKDUDFWHUL]HGDVDSDQGHPLFE\WKH:RUOG+HDOWK2UJDQL]DWLRQDQGRQ0DUFK
3UHVLGHQW'RQDOG7UXPSGHFODUHGWKH&29,' SDQGHPLFDQDWLRQDOHPHUJHQF\3+(
GHFODUDWLRQVDUHYDOLGIRU GD\SHULRGVRUXQWLOWKH6HFUHWDU\RIWKH'HSDUWPHQWRI+HDOWKDQG
+XPDQ6HUYLFHVGHFODUHVWKDWWKH3+(QRORQJHUH[LVWV7KH&29,' 3+(GHFODUDWLRQKDV
EHHQUHQHZHGDQGPDLQWDLQHGIRUHYHU\ GD\SHULRGVLQFHWKHRULJLQDOGHFODUDWLRQDQGWKH
ODWHVWUHQHZDOEHFDPHHIIHFWLYH$SULO'XHWRWKHH[WHQGHGGXUDWLRQRIWKH&29,'
3XEOLF+HDOWK(PHUJHQF\DQGWKHUHVXOWLQJPLWLJDWLQJDFWLRQV UHVWULFWLRQVRQVXSSOLHUIDFLOLW\
DFFHVVWUDYHOEDQVHWF SUHFOXGLQJRUFRPSOLFDWLQJFRPSOHWLRQRIYHQGRUDXGLWV61&
UHVSHFWIXOO\UHTXHVWVDSSURYDORIWKLV4$75FKDQJHZLWKLQDGD\SHULRG
2QFHDSSURYHGWKLVFKDQJHZLOOLQFRUSRUDWHJXLGDQFHIURP(35, LQWR61&¶V
4$75DQGLWZLOOSURPRWHFRQVLVWHQF\DPRQJOLFHQVHHVDQGVXSSOLHUVWKDWGHVLUHWRXWLOL]H(35,
LQWKHLUTXDOLW\DVVXUDQFHSURJUDPV
7KLVOHWWHUFRQWDLQVQR15&FRPPLWPHQWV,I\RXKDYHDQ\TXHVWLRQVSOHDVHFRQWDFW
-DPLH&ROHPDQDW
,GHFODUHXQGHUSHQDOW\RISHUMXU\WKDWWKHIRUHJRLQJLVWUXHDQGFRUUHFW([HFXWHGRQWKH
WKGD\RI0D\
5HVSHFWIXOO\VXEPLWWHG
&KHU\O$*D\KHDUW
5HJXODWRU\$IIDLUV'LUHFWRU
&$*ZDJ
(QFORVXUH+/-'HVFULSWLRQDQG%DVLVIRU&KDQJH
$WWDFKPHQW+/-0DUNXSRI4$753DJHV
$WWDFKPHQW+/-&OHDQ7\SHG4$753DJHV
(QFORVXUH+/-Remote Assessment Techniques: Planning and Conducting Audits and Surveys Using Remote Techniques During Exigent Conditions.(35,3DOR$OWR&$
FF 5HJLRQDO$GPLQLVWUDWRU5HJLRQOO
1553URMHFW0DQDJHU+/-)DUOH\+DWFK9RJWOH8QLWVDQG
6HQLRU5HVLGHQW,QVSHFWRU+/-)DUOH\+DWFK9RJWOH8QLWVDQG
'LUHFWRU'LYLVLRQRI6SHQW)XHO0DQDJHPHQW2IILFHRI1XFOHDU0DWHULDO6DIHW\DQG
6DIHJXDUGV
'LUHFWRU$ODEDPD2IILFHRI5DGLDWLRQ&RQWURO
'LUHFWRU(QYLURQPHQWDO3URWHFWLRQ'LYLVLRQ+/-6WDWHRI*HRUJLD
61&'RFXPHQW&RQWURO57\SH &*$
(GZLQ,+DWFK1XFOHDU3ODQW8QLWVDQG
-RVHSK0)DUOH\1XFOHDU3ODQW8QLWVDQG
9RJWOH(OHFWULF*HQHUDWLQJ3ODQW8QLWVDQG
4XDOLW\$VVXUDQFH7RSLFDO5HSRUW5HGXFWLRQLQ&RPPLWPHQW 3DQGHPLF5HODWHG
(QFORVXUH
'HVFULSWLRQDQG%DVLVIRU&KDQJH
(QFORVXUHWR1/
'HVFULSWLRQDQG%DVLVIRU&KDQJH
'HVFULSWLRQRIWKH&KDQJH
7KHSURSRVHGFKDQJHWRWKH6RXWKHUQ1XFOHDU2SHUDWLQJ&RPSDQ\ 61& 4XDOLW\$VVXUDQFH
7RSLFDO5HSRUW 4$75 ZLOODGGWKHRSWLRQWRXWLOL]HUHPRWHDVVHVVPHQWWHFKQLTXHVWRSHUIRUP
YHQGRUDXGLWV7KLVDELOLW\WRSHUIRUPIXOO\UHPRWHDQGSURYLVLRQDOUHPRWHDVVHVVPHQWVZLOOEH
DYDLODEOHRQO\GXULQJSHULRGVRIH[LJHQWFRQGLWLRQV HJSDQGHPLFV 7KHFKDQJHZLOOEHSODFHG
LQ4$756HFWLRQ³14$ &RPPLWPHQW'DQG3DUW,,,³5HJXODWRU\&RPPLWPHQWV'
$WWDFKPHQWSURYLGHVWKHPDUNHGXSSDJHVRI61&¶V4$75VKRZLQJWKHSURSRVHGFKDQJH
DQG$WWDFKPHQWSURYLGHVWKHFOHDQW\SHGYHUVLRQRIWKHVDPHSDJHV
$QHYDOXDWLRQDJDLQVWWKHFULWHULDLQ&)5 D RIWKHSURSRVHGFKDQJHFRQFOXGHGWKDW
WKHDELOLW\WRSHUIRUPIXOO\UHPRWHYHQGRUDXGLWVHYHQGXULQJSHULRGVKDYLQJH[LJHQWFRQGLWLRQV
FRQVWLWXWHVDUHGXFWLRQLQFRPPLWPHQWSHU&)5 D 7KHUHIRUH15&DSSURYDOLV
UHTXLUHGSULRUWRLPSOHPHQWDWLRQ1RWHWKDWDOWKRXJKWKHGRFXPHQWRemote Assessment Techniques: Planning and Conducting Audits and Surveys Using Remote Techniques During Exigent Conditions.(35,3DOR$OWR&$ DOVRLQFOXGHVJXLGDQFHIRU
FRQGXFWLQJDQDXGLWXWLOL]LQJK\EULGWHFKQLTXHVWKDWWHFKQLTXHLVQRWUHTXHVWHGIRU15&UHYLHZ
DQGDSSURYDOVLQFHWKDWWHFKQLTXHZDVHYDOXDWHGDJDLQVWWKHFULWHULDRI&)5 D DQG
IRXQGWRQRWFRQVWLWXWHDUHGXFWLRQLQFRPPLWPHQW$OWKRXJKWKHK\EULGDXGLWWHFKQLTXHJXLGDQFH
LVFRQWDLQHGLQDGRFXPHQWODEHOHGIRUXVHLQH[LJHQWFRQGLWLRQVWKHUHDUHQRUHJXODWRU\
UHVWULFWLRQVRUFRQGLWLRQVIRUZKHQWKHK\EULGDXGLWWHFKQLTXHPD\EHXWLOL]HG
7KLVFKDQJHZLOOLQFRUSRUDWHJXLGDQFHIURP(35, LQWR61&¶V4$75(35,ZLOO
PDLQWDLQYHUVLRQFRQWURORIWKHLU7HFKQLFDO5HSRUWE\XVLQJXQLTXHGRFXPHQWQXPEHUV LH
IRUPDMRUUHYLVLRQV PLQRUHGLWRULDOFKDQJHVZLOOEHKDQGOHGWKURXJKHUUDWD
DWWDFKPHQWV
5HDVRQIRUWKH3URSRVHG&KDQJH
7KHZRUOGZLGHSDQGHPLFFDXVHGE\&RURQDYLUXV'LVHDVH &29,' KDVLPSDFWHGWKH
DELOLW\IRUOLFHQVHHVWRSHUIRUPQHFHVVDU\DQGUHJXODWRULO\UHTXLUHGDXGLWVDWYHQGRUORFDWLRQV
3XEOLF+HDOWK(PHUJHQF\ 3+( PLWLJDWLQJDFWLRQV FRQWDFWWUDFLQJTXDUDQWLQLQJVRFLDO
GLVWDQFLQJWUDYHOUHVWULFWLRQVHWF HQDFWHGE\OLFHQVHHVYHQGRUVDQGE\JRYHUQPHQWVRQ
WUDYHOWRDGGUHVVWKHSDQGHPLFMHRSDUGL]HWKHDELOLW\RI61&WRPHHWLWV4XDOLW\$VVXUDQFH
3URJUDPUHTXLUHPHQWV
'XHWRWKHH[WHQGHGGHOD\LQWUDYHODQGDELOLW\WRDFFHVVYHQGRUIDFLOLWLHVDXGLWVRIFHUWDLQ
YHQGRUVLQSDUWLFXODUWKRVHZLWKLQWHUQDWLRQDOORFDWLRQVZHUHQRWDEOHWREHFRPSOHWHGZLWKLQWKH
JUDFHSHULRG7KHDELOLW\WRXWLOL]HUHPRWHDVVHVVPHQWWHFKQLTXHVZRXOGSURYLGHDQRSWLRQ
GXULQJH[LJHQWFRQGLWLRQVWRXVHRQHRIWKHIROORZLQJWHFKQLTXHVHYDOXDWHGLQ(35,
EHIRUHDYHQGRUH[SLUHV
x )XOO\5HPRWH$VVHVVPHQW$IXOO\UHPRWHDVVHVVPHQWLVDQDVVHVVPHQWFRQGXFWHGZLWK
DOOWHDPPHPEHUVSDUWLFLSDWLQJUHPRWHO\)XOO\UHPRWHDVVHVVPHQWVHYDOXDWHDOO
DSSOLFDEOHWHFKQLFDODQGTXDOLW\UHTXLUHPHQWVRILQWHUHVW
(
(QFORVXUHWR1/
'HVFULSWLRQDQG%DVLVIRU&KDQJH
x 3URYLVLRQDO5HPRWH$VVHVVPHQW+/-$SURYLVLRQDOUHPRWHDVVHVVPHQWLVDQDVVHVVPHQW
FRQGXFWHGZLWKDOOWHDPPHPEHUVSDUWLFLSDWLQJUHPRWHO\WKDWGRHVQRWDVVHVVDOO
WHFKQLFDODQGTXDOLW\UHTXLUHPHQWVRILQWHUHVW7KHREMHFWLYHRIDSURYLVLRQDOUHPRWH
DVVHVVPHQWLVWRDSSURYHDVXSSOLHUIRUXVHZLWKDSSURSULDWHUHVWULFWLRQVGXULQJH[LJHQW
FRQGLWLRQVXQWLODQDVVHVVPHQWWKDWHYDOXDWHVDOOWHFKQLFDODQGTXDOLW\UHTXLUHPHQWVRI
LQWHUHVWFDQEHSHUIRUPHG
%DVLVIRU&RQFOXGLQJWKDWWKH5HYLVHG3URJUDP&RQWLQXHVWR6DWLVI\WKH&ULWHULDRI
&)5$SSHQGL[%DQGWKH3UHYLRXVO\$FFHSWHG4XDOLW\$VVXUDQFH3URJUDP
&RPPLWPHQWV
7KHSURSRVHGFKDQJHWKRXJKFRQVLGHUHGDUHGXFWLRQLQFRPPLWPHQWVDVLWUHODWHVWREHLQJ
SUHVHQWDWWKHIDFLOLW\WRSHUIRUPDQDVVHVVPHQWZLOOSURYLGHDGGLWLRQDOIOH[LELOLW\IRU61&WR
SHUIRUPYHQGRUDVVHVVPHQWVXVLQJRQHRIWKHIROORZLQJWHFKQLTXHVLGHQWLILHGLQ
(35, GXULQJH[LJHQWFRQGLWLRQV
x 3URYLVLRQDOUHPRWHDVVHVVPHQWVZKHUHDOODVVHVVRUVDUHUHPRWHFDQEHXVHGLQFDVHV
ZKHUHRQVLWHLQSHUVRQSUHVHQFHLVQHFHVVDU\EXWLVQRWSRVVLEOHGXHWRH[LJHQW
FRQGLWLRQV7KHUHVXOWVRIDSURYLVLRQDOUHPRWHDVVHVVPHQWDUHHYDOXDWHGWRGHWHUPLQHLI
DSSURYDORIWKHDVVHVVHGRUJDQL]DWLRQFDQEHSURYLVLRQDOO\UHQHZHGZLWKDSSURSULDWH
UHVWULFWLRQVDSSOLHGWRHDFKRUGHUGXULQJWKHSURYLVLRQDODSSURYDOSHULRG
x )XOO\UHPRWHDVVHVVPHQWVZKHUHDOODVVHVVRUVDUHUHPRWHFDQEHXVHGLQOLPLWHGFDVHV
ZKHQRQVLWHSUHVHQFHLVQRWQHFHVVDU\VXFKDVZKHQWKHVXSSOLHUGRHVQRWKDYHD
SURGXFWLRQIDFLOLW\
7KHIROORZLQJUHJXODWRU\GRFXPHQWVSURYLGHWKHEDVLVIRUWKH61&4$75SHUWDLQLQJWRVXSSOLHU
DXGLWV
x $60(14$ 4XDOLW\$VVXUDQFH5HTXLUHPHQWVIRU1XFOHDU)DFLOLW\$SSOLFDWLRQV
%DVLF5HTXLUHPHQW&RQWURORI3XUFKDVHG,WHPVDQG6HUYLFHV6XSSOHPHQW6
6XSSOHPHQWDU\5HTXLUHPHQWVIRU&RQWURORI3XUFKDVHG,WHPVDQG6HUYLFHV6HFWLRQ
6XSSOLHU6HOHFWLRQ6XE6HFWLRQ6RXUFH(YDOXDWLRQDQG6HOHFWLRQLQFOXGHVODQJXDJH
WKDWVWDWHVLQSDUW
o %DFNJURXQGDQG2EMHFWLYH
³0HDVXUHVIRUHYDOXDWLRQDQGVHOHFWLRQRISURFXUHPHQWVRXUFHVDQGWKHUHVXOWV
WKHUHIURPVKDOOEHGRFXPHQWHGDQGVKDOOLQFOXGHRQHRUPRUHRI D WKURXJK F
EHORZ
F 6XSSOLHU¶VWHFKQLFDODQGTXDOLW\FDSDELOLW\DVGHWHUPLQHGE\DGLUHFWHYDOXDWLRQ
RIKLVIDFLOLWLHVDQGSHUVRQQHODQGWKHLPSOHPHQWDWLRQRIKLVTXDOLW\DVVXUDQFH
SURJUDP'
(
(QFORVXUHWR1/
'HVFULSWLRQDQG%DVLVIRU&KDQJH
x $16,1
o 6RXUFH,QVSHFWLRQDQG$XGLW3URYLVLRQVIRUDFFHVVWRWKHVXSSOLHU V
IDFLOLWLHVDQGUHFRUGVIRUVRXUFHLQVSHFWLRQDQGDXGLWZKHQWKHQHHGIRU
VXFKLQVSHFWLRQRUDXGLWKDVEHHQGHWHUPLQHG
o &RQWURORI3XUFKDVHG0DWHULDO(TXLSPHQWDQG6HUYLFHV0HDVXUHV
VKDOOEHSURYLGHGWRDVVXUHWKDWSXUFKDVHGLWHPVDQGVHUYLFHVZKHWKHU
SXUFKDVHGGLUHFWO\RUWKURXJKFRQWUDFWRUVFRQIRUPWRWKHSURFXUHPHQW
GRFXPHQWV7KHVHPHDVXUHVVKDOOLQFOXGHSURYLVLRQVDVDSSURSULDWHIRUVRXUFH
HYDOXDWLRQDQGVHOHFWLRQREMHFWLYHHYLGHQFHRITXDOLW\IXUQLVKHUE\WKH
FRQWUDFWRULQVSHFWLRQDQGDXGLWDWWKHVRXUFHDQGH[DPLQDWLRQRILWHPVXSRQ
GHOLYHU\
x 5HJXODWRU\*XLGH5HYLVLRQ$XJXVW +/-4XDOLW\$VVXUDQFH3URJUDP
5HTXLUHPHQWV
o 7KHDSSOLFDQWRUOLFHQVHHVKRXOGHLWKHUDXGLWLWVVXSSOLHU VTXDOLW\
DVVXUDQFHSURJUDPRQDWULHQQLDOEDVLVRUDUUDQJHIRUVXFKDXGLW
(35, SURYLGHVJXLGDQFHDQGDIUDPHZRUNWRVXSSRUWWKHSODQQLQJDQGFRQGXFWRI
DXGLWVXWLOL]LQJUHPRWHWHFKQLTXHV$WWKHWLPHWKHDERYHUHJXODWRU\UHTXLUHPHQWVDQGVWDQGDUGV
ZHUHGHYHORSHGDQGSODFHGLQHIIHFWWHFKQRORJ\RSWLRQVDYDLODEOHWRGD\ZHUHQRWHYHQ
FRQWHPSODWHG7KH(35,JXLGDQFHVXSSRUWVWKHDELOLW\WRDFFHVVDVXSSOLHU¶VIDFLOLW\IRUHIIHFWLYH
DVVHVVPHQWVWREHFRQGXFWHG7UDGLWLRQDODVVHVVPHQWWHFKQLTXHVUHTXLUHGDQRQVLWHSUHVHQFH
SULPDULO\GXHWRQRRWKHURSWLRQVH[LVWLQJWRVXSSRUWREVHUYDWLRQRIWHFKQLFDODQGTXDOLW\
SURFHVVHVRILQWHUHVWRUUHYLHZRIGRFXPHQWDWLRQ7KHRQO\GLIIHUHQFHLQWKH(35,JXLGDQFH
IURPWKHWUDGLWLRQDODVVHVVPHQWSURFHVVLVWKDWDFFHVVWRDVXSSOLHU¶VIDFLOLW\DQGGRFXPHQWDWLRQ
LVREWDLQHGWKURXJKWHFKQRORJLFDOPHDQVWRWKHH[WHQWDVXSSOLHULVDEOHWRVXSSRUW:KHUHWKHUH
DUHFKDOOHQJHVWRVXSSRUWLQJDFFHVVIXOO\UHPRWHO\IRUDOOWKHWHFKQLFDODQGTXDOLW\UHTXLUHPHQWV
RILQWHUHVWWKHSURYLVLRQDOUHPRWHDVVHVVPHQWSURFHVVZLOODGGUHVVWKRVHDVSHFWV7KHUHIRUH
WKLVJXLGDQFHZLOOFRQWLQXHWRVXSSRUWWKHDELOLW\WRDVVHVVDQGHYDOXDWHDVXSSOLHU¶V
LPSOHPHQWDWLRQRIWKHLUTXDOLW\DVVXUDQFHSURJUDP
61&¶V4$75DOVRVXSSRUWVWKHTXDOLW\DVVXUDQFHSURJUDPIRUUDGLRDFWLYHZDVWHVKLSSLQJGU\
FDVNVWRUDJHDQG,QGHSHQGHQW6SHQW)XHO6WRUDJH)DFLOLW\VXSSOLHUV7KLVMXVWLILFDWLRQDOVR
VXSSRUWVWKHFRQFOXVLRQWKDWWKHUHTXLUHPHQWVRI6XESDUW+RI&)5DQG6XESDUW*RI
&)5ZLOOFRQWLQXHWREHPHWDV61&PDQDJHVWKHVHVXSSOLHUVWKHVDPHDV&)5
$SSHQGL[%VXSSOLHUV
(
(QFORVXUHWR1/
'HVFULSWLRQDQG%DVLVIRU&KDQJH
&RQFOXVLRQ
61&EHOLHYHVUHPRWHO\SHUIRUPHGYHQGRUDXGLWVFDQEHSHUIRUPHGHIIHFWLYHO\DQGPD\EH
UHTXLUHGWRPDLQWDLQVXSSOLHVRITXDOLW\UHODWHGHTXLSPHQWWRQXFOHDUSODQWVGXHWRWUDYHO
UHVWULFWLRQLPSRVHGE\SDQGHPLFVVXFKDV&29,' RUIRURWKHUSHUVRQQHOVDIHW\LVVXHV
%DVHGRQ61&¶VHYDOXDWLRQRIWKHSURSRVHGFKDQJHDJDLQVWWKHFULWHULDRI
&)5 D WKHSURSRVHGFKDQJHUHSUHVHQWVDUHGXFWLRQLQFRPPLWPHQWLQWKH4$75DVLW
UHODWHVWRWKHSHUIRUPDQFHRIIXOO\UHPRWHDVVHVVPHQWVDQGSURYLVLRQDOUHPRWHDVVHVVPHQWV
WKHUHIRUH15&DSSURYDOLVUHTXLUHGSULRUWR61&¶VLPSOHPHQWDWLRQ7KHSURSRVHGFKDQJHLV
FRPSOLDQWZLWKDQGFRQWLQXHVWRVDWLVI\WKHFULWHULDRI&)5$SSHQGL[%DQGFRQWLQXHVWR
VDWLVI\WKHUHTXLUHPHQWVRI&)56XESDUW+DQG&)56XESDUW*DVLQFRUSRUDWHG
LQWKH61&4$75
5HIHUHQFHV
Remote Assessment Techniques: Planning and Conducting Audits and Surveys Using Remote Techniques During Exigent Conditions.(35,3DOR$OWR&$
$PHULFDQ1DWLRQDO6WDQGDUGV,QVWLWXWH $16, $PHULFDQ6RFLHW\RI0HFKDQLFDO
(QJLQHHUV $60( 14$ ³4XDOLW\$VVXUDQFH3URJUDP5HTXLUHPHQWVIRU1XFOHDU
3RZHU3ODQWV'
$PHULFDQ1XFOHDU6RFLHW\ $16 $PHULFDQ1DWLRQDO6WDQGDUGV,QVWLWXWH
$16, ³$GPLQLVWUDWLYH&RQWUROVDQG4XDOLW\$VVXUDQFHIRUWKH2SHUDWLRQDO
3KDVHRI1XFOHDU3RZHU3ODQWV'
5HJXODWRU\*XLGH5HYLVLRQ³4XDOLW\$VVXUDQFH3URJUDP5HTXLUHPHQWV 'HVLJQ
DQG&RQVWUXFWLRQ '861XFOHDU5HJXODWRU\&RPPLVVLRQ:DVKLQJWRQ'&
(
(GZLQ,+DWFK1XFOHDU3ODQW8QLWVDQG
-RVHSK0)DUOH\1XFOHDU3ODQW8QLWVDQG
9RJWOH(OHFWULF*HQHUDWLQJ3ODQW8QLWVDQG
4XDOLW\$VVXUDQFH7RSLFDO5HSRUW5HGXFWLRQLQ&RPPLWPHQW 3DQGHPLF5HODWHG
$WWDFKPHQW
0DUNXSRI4$753DJHV
6287+(5118&/($523(5$7,1*&203$1<,1&
48$/,7<$6685$1&(723,&$/5(3257
,WLVYDOLGDWHGDWUHFHLSWLQVSHFWLRQWKDWWKHODERUDWRU\ VGRFXPHQWDWLRQFHUWLILHV
WKDW
D 7KHFRQWUDFWHGFDOLEUDWLRQRUWHVWVHUYLFHKDVEHHQSHUIRUPHGLQDFFRUGDQFH
ZLWKWKHLU,62,(& SURJUDPDQGKDVEHHQSHUIRUPHGZLWKLQWKHLU
VFRSHRIDFFUHGLWDWLRQDQG
E 7KHSXUFKDVHRUGHU VUHTXLUHPHQWVDUHPHW
- )RU6HFWLRQ61&FRQVLGHUVGRFXPHQWVWKDWPD\EHVWRUHGLQDSSURYHG
HOHFWURQLFPHGLDXQGHU61&FRQWURODQGQRWSK\VLFDOO\ORFDWHGRQWKHSODQWVLWHEXW
ZKLFKDUHDFFHVVLEOHIURPWKHUHVSHFWLYHQXFOHDUIDFLOLW\VLWHDVPHHWLQJWKH14$
UHTXLUHPHQWIRUGRFXPHQWVWREHDYDLODEOHDWWKHVLWH
- $VFODULILFDWLRQIRU6HFWLRQ61&DOORZVIRUWKHHVWDEOLVKPHQWRIDSSURSULDWH
FRQWUROVDVVRFLDWHGZLWKFRQGLWLRQDOUHOHDVHVRIQRQFRQIRUPLQJLWHPVIRULQVWDOODWLRQ
FRQVLVWHQWZLWKWKHSURYLVLRQVRI6HFWLRQRIWKLV4$75$FFRUGLQJO\LWHPVPD\EH
FRQGLWLRQDOO\UHOHDVHGDVGHVFULEHGLQ6HFWLRQSULRUWRUHFHLSWRIGRFXPHQWDU\
HYLGHQFHWKDWLWHPVFRQIRUPWRSURFXUHPHQWGRFXPHQWV
- ,QOLHXRIWKHUHTXLUHPHQWVRI6HFWLRQ&RPPHUFLDO*UDGH,WHPVFRQWUROVIRU
FRPPHUFLDOJUDGHLWHPVDQGVHUYLFHVDUHHVWDEOLVKHGLQ61&GRFXPHQWVXVLQJWKH
JXLGDQFHRI(35,13GDWHG-XQH DVGLVFXVVHGLQ*HQHULF/HWWHU
DQG*HQHULF/HWWHU
x 3DQGHPLF ([LJHQW&RQGLWLRQV
- :LWKLQ5HTXLUHPHQW6WKHWHUPV³GLUHFWHYDOXDWLRQ'DQG³DXGLW'PD\EHVDWLVILHG
XVLQJUHPRWHDVVHVVPHQWWHFKQLTXHVLQSDQGHPLFRUH[LJHQWFRQGLWLRQV
- ,QWKHFDVHRISDQGHPLFRUH[LJHQWFRQGLWLRQV61&KDVDGRSWHG(35,'RFXPHQW+/-
+/-5HPRWH$VVHVVPHQW7HFKQLTXHV 3ODQQLQJDQG&RQGXFWLQJ$XGLWV
DQG6XUYH\V8VLQJ5HPRWH7HFKQLTXHV'XULQJ([LJHQW&RQGLWLRQV
,, 9HUVLRQ
6287+(5118&/($523(5$7,1*&203$1<,1&
48$/,7<$6685$1&(723,&$/5(3257
3$57,,,5(*8/$725<&200,70(176
15&5HJXODWRU\*XLGHVDQG4XDOLW\$VVXUDQFH6WDQGDUGV
7KLVVHFWLRQLGHQWLILHVWKH15&5HJXODWRU\*XLGHVDQGWKHRWKHUTXDOLW\DVVXUDQFH
VWDQGDUGVZKLFKKDYHEHHQVHOHFWHGWRVXSSOHPHQWDQGVXSSRUWWKH61&4$3URJUDP
6RXWKHUQ1XFOHDUFRPPLWVWRFRPSOLDQFHZLWKWKHVHVWDQGDUGVWRWKHH[WHQWGHVFULEHG
KHUHLQ&RPPLWPHQWWRDSDUWLFXODU5HJXODWRU\*XLGHRURWKHU4$VWDQGDUGGRHVQRW
FRQVWLWXWHDFRPPLWPHQWWRWKH5HJXODWRU\*XLGHVRU4$VWDQGDUGVWKDWPD\EHUHIHUHQFHG
WKHUHLQ
x 5HJXODWRU\*XLGH+/-4XDOLILFDWLRQDQG7UDLQLQJRI3HUVRQQHOIRU1XFOHDU3RZHU3ODQWV
- 6RXWKHUQ1XFOHDUPHHWVWKHUHTXLUHPHQWRIWKHDSSOLFDEOH7HFKQLFDO6SHFLILFDWLRQV
IRUHDFKSODQWIRUWKHVHOHFWLRQDQGWUDLQLQJRIQXFOHDUSRZHUSODQWSHUVRQQHO
- 7KH61&QXFOHDUSODQWTXDOLILFDWLRQDQGWUDLQLQJSURJUDPLVGHVFULEHGLQ6HFWLRQV
DQGRIWKH)6$5V
x 5HJXODWRU\*XLGH5HYLVLRQ$XJXVW +/-4XDOLW\$VVXUDQFH3URJUDP
5HTXLUHPHQWV 'HVLJQDQG&RQVWUXFWLRQ
- 6RXWKHUQ1XFOHDUPHHWVWKHUHTXLUHPHQWVRIWKLVUHJXODWRU\JXLGHIRU&RQVWUXFWLRQ
$FWLYLWLHVFRQGXFWHGE\6RXWKHUQ1XFOHDUH[FHSWWKDW$60(14$ HGLWLRQ DV
PRGLILHGE\WKHH[FHSWLRQVWR14$ DVVKRZQLQWKLV4$75 ZLOOEHXVHGLQ
SODFHRI$16,$60(14$ DQGWKH$16,$60(14$D $GGHQGD
- LWKLQ5HJXODWRU\*XLGHWKHWHUPV³GLUHFWHYDOXDWLRQ'DQG³DXGLW'PD\EH
VDWLVILHGXVLQJUHPRWHDVVHVVPHQWWHFKQLTXHVLQSDQGHPLFRUH[LJHQWFRQGLWLRQV
x 5HJXODWRU\*XLGH5HYLVLRQ)HEUXDU\ +/-4XDOLW\$VVXUDQFH3URJUDP
5HTXLUHPHQWV 2SHUDWLRQ
- 6RXWKHUQ1XFOHDUFRQVLGHUVWKDWWKHFROOHFWLYHTXDOLW\DVVXUDQFHUHTXLUHPHQWVRIWKLV
4$75DQGWKH4$UHTXLUHPHQWVRI$60(14$ DUHHTXLYDOHQWWR$16,
1 $16DQG5HJXODWRU\*XLGH5HYLVLRQ&RQVHTXHQWO\
6RXWKHUQ1XFOHDUGRHVQRWFRPPLWWR$16,1 RUWR5HJXODWRU\*XLGH
H[FHSWWKDW$SSHQGL[$RI5HJXODWRU\*XLGHVKDOOEHXVHGDVJXLGDQFHIRU
HVWDEOLVKLQJWKHSURFHGXUHVUHTXLUHGIRUSODQWRSHUDWLRQDOSKDVHDFWLYLWLHV
,,, 9HUVLRQ
(GZLQ,+DWFK1XFOHDU3ODQW8QLWVDQG
-RVHSK0)DUOH\1XFOHDU3ODQW8QLWVDQG
9RJWOH(OHFWULF*HQHUDWLQJ3ODQW8QLWVDQG
4XDOLW\$VVXUDQFH7RSLFDO5HSRUW5HGXFWLRQLQ&RPPLWPHQW 3DQGHPLF5HODWHG
$WWDFKPHQW
&OHDQ7\SHG4$753DJHV
6287+(5118&/($523(5$7,1*&203$1<,1&
48$/,7<$6685$1&(723,&$/5(3257
,WLVYDOLGDWHGDWUHFHLSWLQVSHFWLRQWKDWWKHODERUDWRU\ VGRFXPHQWDWLRQFHUWLILHV
WKDW
D 7KHFRQWUDFWHGFDOLEUDWLRQRUWHVWVHUYLFHKDVEHHQSHUIRUPHGLQDFFRUGDQFH
ZLWKWKHLU,62,(& SURJUDPDQGKDVEHHQSHUIRUPHGZLWKLQWKHLU
VFRSHRIDFFUHGLWDWLRQDQG
E 7KHSXUFKDVHRUGHU VUHTXLUHPHQWVDUHPHW
- )RU6HFWLRQ61&FRQVLGHUVGRFXPHQWVWKDWPD\EHVWRUHGLQDSSURYHG
HOHFWURQLFPHGLDXQGHU61&FRQWURODQGQRWSK\VLFDOO\ORFDWHGRQWKHSODQWVLWHEXW
ZKLFKDUHDFFHVVLEOHIURPWKHUHVSHFWLYHQXFOHDUIDFLOLW\VLWHDVPHHWLQJWKH14$
UHTXLUHPHQWIRUGRFXPHQWVWREHDYDLODEOHDWWKHVLWH
- $VFODULILFDWLRQIRU6HFWLRQ61&DOORZVIRUWKHHVWDEOLVKPHQWRIDSSURSULDWH
FRQWUROVDVVRFLDWHGZLWKFRQGLWLRQDOUHOHDVHVRIQRQFRQIRUPLQJLWHPVIRULQVWDOODWLRQ
FRQVLVWHQWZLWKWKHSURYLVLRQVRI6HFWLRQRIWKLV4$75$FFRUGLQJO\LWHPVPD\EH
FRQGLWLRQDOO\UHOHDVHGDVGHVFULEHGLQ6HFWLRQSULRUWRUHFHLSWRIGRFXPHQWDU\
HYLGHQFHWKDWLWHPVFRQIRUPWRSURFXUHPHQWGRFXPHQWV
- ,QOLHXRIWKHUHTXLUHPHQWVRI6HFWLRQ&RPPHUFLDO*UDGH,WHPVFRQWUROVIRU
FRPPHUFLDOJUDGHLWHPVDQGVHUYLFHVDUHHVWDEOLVKHGLQ61&GRFXPHQWVXVLQJWKH
JXLGDQFHRI(35,13GDWHG-XQH DVGLVFXVVHGLQ*HQHULF/HWWHU
DQG*HQHULF/HWWHU
x 3DQGHPLF ([LJHQW&RQGLWLRQV
- :LWKLQ5HTXLUHPHQW6WKHWHUPV³GLUHFWHYDOXDWLRQ'DQG³DXGLW'PD\EHVDWLVILHG
XVLQJUHPRWHDVVHVVPHQWWHFKQLTXHVLQSDQGHPLFRUH[LJHQWFRQGLWLRQV
- ,QWKHFDVHRISDQGHPLFRUH[LJHQWFRQGLWLRQV61&KDVDGRSWHG(35,'RFXPHQW+/-
+/-5HPRWH$VVHVVPHQW7HFKQLTXHV 3ODQQLQJDQG&RQGXFWLQJ$XGLWV
DQG6XUYH\V8VLQJ5HPRWH7HFKQLTXHV'XULQJ([LJHQW&RQGLWLRQV
,, 9HUVLRQ
6287+(5118&/($523(5$7,1*&203$1<,1&
48$/,7<$6685$1&(723,&$/5(3257
3$57,,,5(*8/$725<&200,70(176
15&5HJXODWRU\*XLGHVDQG4XDOLW\$VVXUDQFH6WDQGDUGV
7KLVVHFWLRQLGHQWLILHVWKH15&5HJXODWRU\*XLGHVDQGWKHRWKHUTXDOLW\DVVXUDQFH
VWDQGDUGVZKLFKKDYHEHHQVHOHFWHGWRVXSSOHPHQWDQGVXSSRUWWKH61&4$3URJUDP
6RXWKHUQ1XFOHDUFRPPLWVWRFRPSOLDQFHZLWKWKHVHVWDQGDUGVWRWKHH[WHQWGHVFULEHG
KHUHLQ&RPPLWPHQWWRDSDUWLFXODU5HJXODWRU\*XLGHRURWKHU4$VWDQGDUGGRHVQRW
FRQVWLWXWHDFRPPLWPHQWWRWKH5HJXODWRU\*XLGHVRU4$VWDQGDUGVWKDWPD\EHUHIHUHQFHG
WKHUHLQ
x 5HJXODWRU\*XLGH+/-4XDOLILFDWLRQDQG7UDLQLQJRI3HUVRQQHOIRU1XFOHDU3RZHU3ODQWV
- 6RXWKHUQ1XFOHDUPHHWVWKHUHTXLUHPHQWRIWKHDSSOLFDEOH7HFKQLFDO6SHFLILFDWLRQV
IRUHDFKSODQWIRUWKHVHOHFWLRQDQGWUDLQLQJRIQXFOHDUSRZHUSODQWSHUVRQQHO
- 7KH61&QXFOHDUSODQWTXDOLILFDWLRQDQGWUDLQLQJSURJUDPLVGHVFULEHGLQ6HFWLRQV
DQGRIWKH)6$5V
x 5HJXODWRU\*XLGH5HYLVLRQ$XJXVW +/-4XDOLW\$VVXUDQFH3URJUDP
5HTXLUHPHQWV 'HVLJQDQG&RQVWUXFWLRQ
- 6RXWKHUQ1XFOHDUPHHWVWKHUHTXLUHPHQWVRIWKLVUHJXODWRU\JXLGHIRU&RQVWUXFWLRQ
$FWLYLWLHVFRQGXFWHGE\6RXWKHUQ1XFOHDUH[FHSWWKDW$60(14$ HGLWLRQ DV
PRGLILHGE\WKHH[FHSWLRQVWR14$ DVVKRZQLQWKLV4$75 ZLOOEHXVHGLQ
SODFHRI$16,$60(14$ DQGWKH$16,$60(14$D $GGHQGD
- LWKLQ5HJXODWRU\*XLGHWKHWHUPV³GLUHFWHYDOXDWLRQ'DQG³DXGLW'PD\EH
VDWLVILHGXVLQJUHPRWHDVVHVVPHQWWHFKQLTXHVLQSDQGHPLFRUH[LJHQWFRQGLWLRQV
x 5HJXODWRU\*XLGH5HYLVLRQ)HEUXDU\ +/-4XDOLW\$VVXUDQFH3URJUDP
5HTXLUHPHQWV 2SHUDWLRQ
- 6RXWKHUQ1XFOHDUFRQVLGHUVWKDWWKHFROOHFWLYHTXDOLW\DVVXUDQFHUHTXLUHPHQWVRIWKLV
4$75DQGWKH4$UHTXLUHPHQWVRI$60(14$ DUHHTXLYDOHQWWR$16,
1 $16DQG5HJXODWRU\*XLGH5HYLVLRQ&RQVHTXHQWO\
6RXWKHUQ1XFOHDUGRHVQRWFRPPLWWR$16,1 RUWR5HJXODWRU\*XLGH
H[FHSWWKDW$SSHQGL[$RI5HJXODWRU\*XLGHVKDOOEHXVHGDVJXLGDQFHIRU
HVWDEOLVKLQJWKHSURFHGXUHVUHTXLUHGIRUSODQWRSHUDWLRQDOSKDVHDFWLYLWLHV
,,, 9HUVLRQ
(GZLQ,+DWFK1XFOHDU3ODQW8QLWVDQG
-RVHSK0)DUOH\1XFOHDU3ODQW8QLWVDQG
9RJWOH(OHFWULF*HQHUDWLQJ3ODQW8QLWVDQG
4XDOLW\$VVXUDQFH7RSLFDO5HSRUW5HGXFWLRQLQ&RPPLWPHQW 3DQGHPLF5HODWHG
(QFORVXUH
Remote Assessment Techniques: Planning and Conducting Audits and Surveys Using Remote Techniques During Exigent Conditions.(35,3DOR$OWR&$
Remote Assessment Techniques Planning and Conducting Audits and Surveys Using 3002020796 Final Report, April 2021 EPRI Project Manager M. Tannenbaum All or a portion of the requirements of the EPRI Nuclear Quality Assurance Program apply to this product.
ELECTRIC POWER RESEARCH INSTITUTE 3420 Hillview Avenue, Palo Alto, California 94304- -
DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES (A
FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR CIRCUMSTANCE; OR PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT.
THE ELECTRIC POWER RESEARCH INSTITUTE (EPRI) PREPARED THIS REPORT.
THE TECHNICAL CONTENTS OF THIS PRODUCT WERE NOT PREPARED IN ACCORDANCE WITH NOT REMENTS OF 10 CFR PART 21.
NOTE For further information about EPRI, call the EPRI Customer Assistance Center at 800.313.3774 or e-Electric Power Research Institute, EPRI, and TOGETHER Copyright © 2021 Electric Power Research Institute, Inc. All rights reserved.
ACKNOWLEDGMENTS The Electric Power Research Institute (EPRI) prepared this report.
Principal Investigator M. Tannenbaum This report describes research sponsored by EPRI.
EPRI thanks the following individuals who participated in the technical advisory committee and contributed to the development of this report. Their valuable insights and experience were essential to the successful completion of this project.
Earl Mayhorn Ameren UE Paul Lang ASME Accreditation/Conformity Assessment Tim Franchuk Curtiss-Wright Nuclear Division Mark Coren Duke Energy Jon Thomas Duke Energy Jessica Lemieux EPRI Allison Read EPRI Brian Hendel Energy Northwest Guy Robinson Entergy Stanley Griffin GE Hitachi Nuclear Energy Mark Soler Holtec International Alka Malur Hydro Aire Spencer Daw Idaho National Laboratories John Simmons Luminant Hussain Al Jaberi Nawah Energy Company Gerald Ice Oak Ridge Y-12 Uranium Processing Facility Tracy Bolt Paragon Energy Solutions Les Taggart Paragon Energy Solutions Sham Beri Portland General Electric This publication is a corporate document that should be cited in the literature in the following manner:
Remote Assessment Techniques: Planning and Conducting Audits and Surveys Using Remote Techniques During Exigent Conditions. EPRI, Palo Alto, CA: 2021. 3002020796.
iii
Rachel Czuba Sonic Systems International Amy Aughtman Southern Nuclear Zack Betsill Southern Nuclear Greg Przyjemski Talen Energy Jon Talbott Tennessee Valley Authority Norm Moreau Theseus Professional Services Jim Garrison United Controls, International Paul Prescott U.S. Nuclear Regulatory Commission Kerri Kavanagh U.S. Nuclear Regulatory Commission Ed Renaud WestinghouseGlobal Components Manufacturing Dave Soward Xcel Energy Lynn Valdez Arizona Public Service iv
ABSTRACT In the first quarter of 2020, when many parts of the world began to experience an unprecedented pandemic due to the rapid spread of COVID-19, the global supply chain was impacted in multiple ways.
One impact of the pandemic was an inability to perform quality assurance activities such as audits, commercial-grade surveys, and source verifications at suppliers facilities. In April 2020, the Electric Power Research Institute published Remote Source Verification During a Pandemic or Similar State of Emergency: Screening Criteria and Process Guidance, 3002019436-A (EPRI 3002019436-A). The scope of the report was limited to performance of remote source verification in extreme circumstances where on-site verification is not possible due to conditions that threaten the health and safety of individuals performing the verification. As the pandemic endured, inability to perform traditional audits and commercial-grade surveys threatened the ability to maintain qualified suppliers beyond the expiry date of their pre-pandemic assessments.
The report considers the experience of nuclear licensees, suppliers, and regulators in Canada, Finland, United Arab Emirates, the United States, and other jurisdictions that have used remote techniques to conduct supplier assessments. It describes remote assessment techniques for the purposes of supplier assessments. It provides a process for determining the extent to which remote techniques can be used and includes considerations for planning and executing assessments involving remote techniques. Remote techniques can be used to complete supplier assessments during exigent conditions with appropriate planning, communication, technology, and other tools described in this report.
Keywords Assessment Audit Commercial-grade survey COVID-19 Pandemic Quality assurance v
EXECUTIVE
SUMMARY
Deliverable Number: 3002020796 Product Type: Technical Report Product
Title:
Remote Assessment Techniques: Planning and Conducting Audits and Surveys Using Remote Techniques During Exigent Conditions PRIMARY AUDIENCE: Nuclear quality assurance, quality control, and procurement engineering professionals SECONDARY AUDIENCE: Nuclear supply chain professionals KEY RESEARCH QUESTION
-site audits, commercial-grade surveys, and other assessments of suppliers might not be possible due to conditions that threaten the health and safety of individuals performing the assessment. Can technology-based remote assessment techniques enable remote assessment? If so, what considerations apply to planning and conducting assessments that involve remote assessment techniques?
RESEARCH OVERVIEW A technical advisory group composed of procurement quality professionals met virtually to discuss the topics of remote assessment and remote assessment techniques with regulatory staff, identify potential challenges, and develop a plan for enabling use of remote assessment techniques during . The team developed a performance-based approach to remote assessment. The approach focuses on technical and quality requirements related to the ability of items purchased from the supplier to perform their design was also developed. This process can result in one of three optionshybrid assessment, provisional remote assessment, or fully remote assessment. Hybrid assessments are conducted when at least one member of the assessment team can be on-site and in-person at the suppliers facility. Provisional remote assessments none of the assessment team members can be on-site, in-person, but on-site presence is required to adequately assess performance-based technical and quality requirements. Fully remote assessments are
-site, in-person and on-site presence is not required to adequately assess performance-based technical and quality requirements.
KEY FINDINGS Planning is essential to the success of remote assessment techniques.
With proper planning, remote assessment techniques can be used to observe and assess a broad range of technical and quality requirements.
can be used for each assessment based on the technical and quality requirements of interest.
Supplier readiness and willingness to accommodate remote assessment techniques are prerequisites.
vii
EXECUTIVE
SUMMARY
WHY THIS MATTERS The process and techniques described in this report can serve as a basis for successful application of remote assessment techniques. This report can serve as the basis for developing effective processes for remote assessment in order to maintain approval of suppliers that provide services, equipment, and replacement HOW TO APPLY RESULTS
, and pre- d to develop effective procedures for application of remote assessment techniques. In addition, this report can be used to successfully plan and conduct audits and commercial-grade surveys that involve remote assessment techniques and technologies.
LEARNING AND ENGAGEMENT OPPORTUNITIES Procurement and quality professionals involved in nuclear and other forms of generation can benefit from the information in this report.
Members of the American Society of Mechanical Engineers (ASME) NQA-1, Nuclear Procurement Issues Corporation, and the Nuclear Industry Assessment Committee might be interested in this report.
EPRI CONTACT: Marc Tannenbaum, Senior Technical PROGRAM: Plant Engineering, P41 IMPLEMENTATION CATEGORY: Category 1, Regulatory Together...Shaping the Future of Electricity Electric Power Research Institute 3420 Hillview Avenue, Palo Alto, California 94304- -0813 USA 800.313.3774 * *
- www.epri.com
© 2021 Electric Power Research Institute (EPRI), Inc. All rights reserved. Electric Power Research Institute, EPRI, and
CONTENTS ABSTRACT ..................................................................................................................................V EXECUTIVE
SUMMARY
............................................................................................................VII 1 BACKGROUND AND OBJECTIVE ....................................................................................... 1-1 1.1 .................................................................................................................. 1-1 1.2 Objective ...................................................................................................................... 1-1 1.3 Applicability of Guidance ............................................................................................. 1-2 1.4 Intent and Precautions ................................................................................................. 1-2
........................................... 1-3 Entities That Perform Supplier Assessments ............................................................... 1-3 1.7 Regulatory and Standard Perspective ......................................................................... 1-4 1.7.1 Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants ............................................................................... 1-4 1.7.2 ANSI N18.7-Operational Phase of Nuclear Power Plants ................................................................... 1-4 1.7.3 -
Nuclear Facilities ............................................................................................................. 1-1.7.4 ASME NQA-1-1994 Applications ..................................................................................................................... 1-ASME NQA-1a-2009 ........................................................................................... 1-ASME NQA rements for Nuclear Facility Applications ..................................................................................................................... 1-1.7.7 ASME NQA-1-2019 Applications ..................................................................................................................... 1-7 1.7.8 DOE- -1230-2019, Commercial Grade Dedication Application
......................................................................................................................... 1-7 1.8 Acceptance of Remote Assessment Activities ............................................................. 1-7 1.9 Typical Assessment Activities ...................................................................................... 1-8 1.10 Structure of This Report .......................................................................................... 1-8 ix
2 LEXICON AND ACRONYMS .................................................................................................2-1 2.1 ........................................................................................................................ 2-1 2.2 Acronyms and Abbreviations ....................................................................................... 2-3 ACTIVITIES ASSOCIATED WITH PERFORMING ASSESSMENTS .................................... 3-1 3.1 Review of Documentation ............................................................................................ 3-1 3.2 Access to Production Facilities .................................................................................... 3-2 3.3 Access to Supplier Personnel ...................................................................................... 3-3 3.4 Access to Processes for Observation .......................................................................... 3-Assessment Team Collaboration ................................................................................. 3-Activities Related to Management of the Assessment Team for an On-Site Assessment ...................................................................................................................... 3-Assessment and Assessment Resource Planning ...................................................... 3-7 4 REMOTE ASSESSMENT TECHNIQUES .............................................................................. 4-1 4.1 Review of Documentation ............................................................................................ 4-2 4.2 Access to Production Facilities .................................................................................... 4-3 4.3 Access to Supplier Personnel ...................................................................................... 4-4.4 Access to Processes for Observation .......................................................................... 4-7 Assessment Team Collaboration ................................................................................. 4-9 Assessment and Assessment Resource Planning .................................................... 4-10 5 REVIEW OF THE PERFORMANCE-BASED SUPPLIER ASSESSMENT PROCESS ......... 5-1 Performance- Assessment Process .................................................... -1 Planning Phase of the Performance- ................ -3 Conduct of Assessment Phase of the Performance-Process ................................................................................................................................. -3 Documentation Phase of the Performance-Process ................................................................................................................................. -3 Follow-Up Phase of the Performance- .............. -3 Maintaining a Performance- .................................................................... -4 6 PROCESS FOR EVALUATING USE OF REMOTE ASSESSMENT TECHNIQUES ............ 6-1 Assessment Objectives................................................................................................ -4
................................................................................... -4 Identify Technical and Quality Requirements of Interest ............................................. -4 x
Identify Supplier Activities Related to Technical and Quality Requirements of Interest for Assessment......................................................................................................... -
Determine If On-Site Presence Is Practical ................................................................. -
Plan and Conduct Traditional Assessment .................................................................. -
Identify the Technical and Quality Requirements of Interest............................... -8 Does Assessment of the Activity Normally Require On-Site, In-Person Presence? ........................................................................................................................ -8 Identify the Assessment Techniques Typically Used When Present In-Person/On-Site ................................................................................................................. -8 Can Remote Assessment Techniques Provide Results Similar to On-Site, In-Person Techniques? .................................................................................................... -8 the Activity ........................................................................................................................ -8 Identify That On-Site Presence Is Necessary for Adequate Performance-
.................................................................................... -9 Do Any Activities Require On-Site Presence? ............................................................. -9 Plan and Conduct Fully Remote Assessment.............................................................. -9 Eligibility Requirements for Fully Remote Assessment ....................................... -9 Assess the On-Site Activities .............................................................................................. -10 Is Hybrid Assessment Possible? ........................................................................... -10 Plan and Conduct Hybrid Assessment .................................................................. -10 Plan and Conduct a Provisional Remote Assessment .......................................... -10 Eligibility Requirements for Provisional Remote Assessment........................ -11 Is Provisional Approval of the Supplier Appropriate? ............................................ -11 Supplier List ........................................................................................................................ -12 Complete Assessment of the Supplier, Including Review of Items Provided During Provisional Approval, as Soon as Practical ............................................................. -12 Pursue Alternative Options for Procurement and Acceptance .............................. -12 7 PLANNING FOR AN ASSESSMENT THAT USES REMOTE TECHNIQUES ...................... 7-1 7.1 Planning Phase ............................................................................................................ 7-1 7.1.1 Establish Assessments Objective ...................................................................... 7-1 7.1.2 Identify Items/Services Provided by the Supplier ................................................ 7-2 7.1.3 ............. 7-2 xi
7.1.4 Identify Technical and Quality Requirements of Interest or Critical Characteristics .................................................................................................................. 7-2 Identify Suppliers Processes Related to Technical and Quality Requirements of Interest .................................................................................................. 7-2 Identify Acceptance Criteria and Methods for Supplier Processes Related to Technical and Quality Requirements of Interest ........................................................... 7-3 Develop a List of Documents Required to Support the Assessment ............. 7-3 7.1.7 Complete Assessment Team .............................................................................. 7-3 7.1.7.1 Determine Suppliers Ability to Support Remote Techniques........................ 7-4 7.1.7.2 Determine the Type of Remote Assessment to Perform ............................... 7-4 7.1.7.3 Select the Assessment Team and Develop the Assessment Plan................ 7-4 7.1.7.4 Assessment Team Preparation ..................................................................... 7-7.2 Conduct of Assessment Phase .................................................................................... 7-7.2.1 Establish Protocol for Remote Communication ................................................... 7-7.2.2 ...................................................................... 7-7.2.2 Pre- ................................................................................................. 7-7.3 Documentation and Follow-Up..................................................................................... 7-7.3.1 (Provisional Remote Assessments) .................................................................................. 7-7.3.2 ................................. 7-8 REMOTE ASSESSMENT TOOLS/APPLICATIONS .............................................................. 8-1 8.1 ..................................................................................... 8-1 8.2 ..................................................................................... 8-1 8.3 Sampling of Remote Assessment Tools/Applications.................................................. 8-4 8.4 Document Sharing ....................................................................................................... 8-4 9 REFERENCES ....................................................................................................................... 9-1 A CURRENT APPLICATION AND EXPERIENCE .................................................................. A-1 A.1 ............................................................. A-1 A.1.1 Limited Scope Audit of a Third-Party Qualifier by a LicenseeSupplier Perspective ...................................................................................................................... A-1 Type and Description of Assessment ......................................................................... A-1 Remote Techniques Used .......................................................................................... A-1 Tools Used .................................................................................................................. A-1 xii
Lessons Learned ........................................................................................................ A-2 Limitations ................................................................................................................... A-2 A.1.2 Limited Scope AuditAssessment Team Perspective ...................................... A-2 Type and Description of Assessment ......................................................................... A-2 Remote Techniques Used .......................................................................................... A-2 Lessons Learned ........................................................................................................ A-2 Limitations ................................................................................................................... A-3 A.1.3 Full Triennial AuditAssessment Team Perspective ........................................ A-3 Type and Description of Assessment ......................................................................... A-3 Remote Techniques Used .......................................................................................... A-3 Lessons Learned ........................................................................................................ A-3 Limitations ................................................................................................................... A-4 A.1.4 Full Triennial AuditAssessment Team Perspective ........................................ A-4 Type and Description of Assessment ......................................................................... A-4 Remote Techniques Used .......................................................................................... A-Lessons Learned ........................................................................................................ A-Limitations ................................................................................................................... A-Assessment Team Perspective .......................................... A-Type and Description of Assessment ......................................................................... A-Remote Techniques Used .......................................................................................... A-Lessons Learned ........................................................................................................ A-Limitations ................................................................................................................... A-A.2 ........................................................... A-7 A.2.1 CANDU Procurement Audit Committee ............................................................. A-7 A.2.2 CANDU Industry Assessment Committee .......................................................... A-7 A.2.3 Emirates Nuclear Energy Corporation/Nawah Energy Company ...................... A-9 Hybrid Audit Processes .............................................................................................. A-9 Lessons Learned ...................................................................................................... A-10 A.2.4 Standard Discussion on Remote Assessment and Audit ................................. A-10 Nuclear Industry Evaluation Program .............................................................. A-10 Lessons Learned ...................................................................................................... A-10 Limitations ................................................................................................................. A-11 B REMOTE ASSESSMENT PLANNING WORKSHEET ......................................................... B-1
........... -1 xiii
C PERFORMANCE-BASED ASSESSMENT AND CRITICAL CHARACTERISTIC WORKSHEETS ........................................................................................................................ C-1 C.1 Instructions for Completing the Performance-
............................................................................................................................ C-1 C.2 .............................. C-3 D PRE-ASSESSMENT CHECKLISTS FOR REMOTE ASSESSMENTS ................................ D-1 D.1 Instructions for Completing the Pre- ...................................... D-1 E FREQUENTLY ASKED QUESTIONS .................................................................................. E-1 xiv
LIST OF FIGURES Figure 1-1 Report structure ........................................................................................................ 1-8
-1 Performance-based assessment process based on EPRI NP- ........................ -2
-1 Process for evaluating use of remote assessment techniques ................................ -3
- ................................................................. -7 Figure 8- ing
............................................................................................. 8-2 Figure 8- -mounted tablet camera remote meeting participants ............................................................................................... 8-3 Figure 8-3 The RealWear head-mounted tablet supports real-time collaboration, including transmission of documents via the heads-up display that provides a view equivalent to a 7-in. (178-mm) tablet at arms length ..................................................................................................8-3 Figure A-1 Partial Quality Assurance Standard Clause (Part A) section of CANIAC Audit
............................................................................................... A-8 Figure A-
............................................................................................................................ A-9 xv
LIST OF TABLES Table 3-1 Assessment activities associated with access to documents .................................... 3-2 Table 3-2 Assessment activities associated with access to production facilities when in-person and on-site ............................................................................................................. 3-3 Table 3-3 Assessment activities associated with direct access to personnel ............................ 3-4 Table 3-4 Assessment activities associated with direct access to processes ........................... 3-Table 3-assessment team ............................................................................................................... 3-Table 3- ........................ 3-7 Table 4-1 Remote assessment techniques for review of documentation .................................. 4-2 Table 4-2 Remote assessment techniques for successful access to production facilities ......... 4-4 Table 4-3 Remote assessment techniques for successful observation and interviews of personnel ........................................................................................................................... 4-Table 4-4 Remote assessment techniques for successful observation of processes and individuals performing them ............................................................................................... 4-7 Table 4- ement of the assessment team ............. 4-9 Table 4- .............. 4-11
- ........................................................ -2
- ................................................................. -3 Table C-1 Performance-based supplier assessment wo .............................................. C-2 Table C- ............................................................................. C-4 Table D-1 Pre- ........................................ D-1 Table D-2 Pre-assessment screeni ............................... D-3 xvii
1 BACKGROUND AND OBJECTIVE
1.1 Background
In the first quarter of 2020, the world began to experience an unprecedented pandemic due to the rapid spread of COVID-19. The global supply chain was impacted in a number of ways.
Manufacturing facilities worldwide temporarily closed or experienced reduced capacity as stay-at-home orders restricted employees from traveling to work. Although certain nuclear suppliers and manufacturers were categorized as essential by regulatory agencies and were able to remain open, some experienced occasional shutdowns to disinfect facilities after employees tested positive for the disease or came into contact with someone who tested positive.
Due to robust supply chains and on-site inventory, nuclear plant supply chain organizations ability to provide the spare and replacement items needed to operate and maintain plants experienced relatively low impact, even as the spring outage season approached. One impact, however, was an inability to perform quality assurance activities such as audits, commercial-grade surveys, and source verifications at suppliers facilities. In April 2020, the Electric Power Research Institute (EPRI) published Remote Source Verification During a Pandemic or Similar State of Emergency: Screening Criteria and Process Guidance, 3002019436-A [1]. The U.S.
Nuclear Regulatory Commission (NRC) found the report acceptable in a safety evaluation addressing a request by Energy Northwest to change Columbia Generating Stations operational quality assurance program description, Columbia Generating Station - Correction to Reduction in Commitment to the Operational Quality Assurance Program Description [2], and in a topical report safety evaluation (ADAMS accession nos. ML20205L619 and ML20205L618) [3]. The scope of the EPRI report [1] was limited to performance of remote source verification in extreme circumstances where it is not possible to perform on-site source verification due to conditions that threaten the health and safety of individuals performing the verification.
As the pandemic endured, inability to perform traditional audits and commercial-grade surveys continued to threaten the ability to maintain qualified suppliers beyond the expiry date of their pre-pandemic assessments. The purpose of this report is to present a methodology for remote assessment techniques pursuant to supplier assessments, such as audits and commercial-grade surveys.
1.2 Objective The objective of this report is to provide guidance for the application of remote assessment techniques and associated information and communication technology (ICT), such as content management platforms, audio, video, telecommunication, and other real-time tools, to facilitate successful completion of audits, commercial-grade surveys, and other supplier assessments.
1-1
Background and Objective Because remote assessment techniques can limit the ability to observe and assess, the techniques included in this report are primarily intended to achieve results similar to traditional on-site and in-person performance-based assessments.
Successful use of remote assessment techniques involves:
Maintaining a performance-based approach Clearly defined scope and objectives Identification of activities that should be witnessed as part of the assessment Identification of those activities that can be witnessed/assessed only in-person When possible, identification of means to use remote assessment techniques for witnessing/assessing If on-site assessor presence is limited, careful planning to ensure that on-site assessors maintain focus required to complete witnessing and assessment of all applicable on-site activities Coordination of virtual meetings and interviews to ensure that all aspects of the assessment are completed during the assessment period Technology required to remotely witness/assess important aspects of the activity in real time Contingency plans to address the potential for technology failures during the activity Means for documenting any assessment activities accomplished using remote assessment techniques in a format that provides the verifying entity with objective evidence of the witnessing/verification 1.3 Applicability of Guidance The guidance in this report is intended to be used in conducting external supplier assessments.
However, the concepts and techniques could be applied to internal assessments as permitted by applicable regulatory and quality assurance program requirements.
1.4 Intent and Precautions This report was developed to provide an alternative to conducting on-site, in-person assessments of suppliers (audits, commercial-grade surveys, and so on) during exigent conditions, such as a pandemic.
The intent of the report is to provide a technically sound process for evaluating the use of remote assessment techniques (included in Section 6 of the report) that is focused on performance-based technical and quality requirements and activities of interest related to the products and services provided by the supplier. This process should be used on a case-by-case basis to determine the extent to which remote assessment techniques can be applied and identify appropriate remote assessment techniques. The possible determinations include the following:
Conduct of a hybrid assessment (some assessment team members are on-site, in-person while others are remote) that applies appropriate remote assessment techniques Conduct of a provisional remote assessment that addresses the technical and quality activities of interest that do not require on-site presence (completing the assessment to the extent 1-2
Background and Objective possible when on-site presence is not possible due to exigent conditions) in an effort to maintain approval of the supplier, provided appropriate restrictions are applied to each procurement during the provisional approval period In certain cases, conduct of a complete, fully remote assessment that ensures and adequately documents objective evidence that all the quality and technical activities of interest are satisfactory implemented The report does not attempt to assert regulatory boundaries. The extent to which remote assessment techniques can be used might be limited by applicable regulations. Although regulations in some jurisdictions may permit hybrid assessments while not permitting fully remote assessments, others might permit both. The guidance included in this report for application of remote assessment techniques can be applied for hybrid, provisionally remote, and fully remote assesments. Therefore, the guidance should be applied on a case-by-case basis and as permitted by regulatory and quality assurance program requirements.
1.5 Difference Between Assessment and Source Verification Whereas source verifications are typically focused on a limited scope of quality activities or manufacturing processes, assessments, such as audits and commercial-grade surveys, are typically focused on a broader scope and involve observation of numerous processes and activities. Additionally, assessments may serve different purposes, such as the following:
Initial qualification of a supplier Limited-scope or follow-up assessments Triennial or other regularly scheduled assessments necessary to maintain supplier qualification 1.6 Entities That Perform Supplier Assessments Supplier assessments may be performed by licensees, suppliers, or third parties. Audits and commercial-grade surveys are typically led by a certified lead auditor, with the assistance of qualified auditors. Performance-based assessment teams are often supplemented with technical specialists, who can be engineers, technicians, or other subject matter experts familiar with the suppliers products and processes.
Examples of entities that perform assessments include:
Conformity assessment teams (the International Organization for Standardization [ISO],
American Society of Mechanical Engineers [ASME], and so on)
Licensees Maintenance and operations, engineering, procurement and construction contractors (working on U.S. Department of Energy [DOE] projects, for example)
Nuclear Industry Assessment Corporation (NIAC) teams Nuclear Procurement Issue Corporation (NUPIC) teams Suppliers that assess subtier suppliers 1-3
Background and Objective 1.7 Regulatory and Standard Perspective Depending on the language and interpretation of applicable regulations and standards, remote assessment techniques might require prior authorization by regulatory agencies. Examples of regulations and standards to which U.S. licensees may be committed are included in Sections 1.7.1-1.7.8.
1.7.1 Appendix B to Part 50Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants [4]
As an example, criterion VII of 10 CFR 50, Appendix B [4], includes language suggesting that source evaluation and selection involves activities at the source or suppliers facility:
VII. Control of Purchased Material, Equipment, and Services Measures shall be established to assure that purchased material, equipment, and services, whether purchased directly or through contractors and subcontractors, conform to the procurement documents. These measures shall include provisions, as appropriate, for source evaluation and selection, objective evidence of quality furnished by the contractor or subcontractor, inspection at the contractor or subcontractor source, and examination of products upon delivery. Documentary evidence that material and equipment conform to the procurement requirements shall be available at the nuclear powerplant or fuel reprocessing plant site prior to installation or use of such material and equipment. This documentary evidence shall be retained at the nuclear powerplant or fuel reprocessing plant site and shall be sufficient to identify the specific requirements, such as codes, standards, or specifications, met by the purchased material and equipment. The effectiveness of the control of quality by contractors and subcontractors shall be assessed by the applicant or designee at intervals consistent with the importance, complexity, and quantity of the product or services. [Emphasis added.]
1.7.2 ANSI N18.7-1976, Administrative Controls and Quality Assurance for the Operational Phase of Nuclear Power Plants [5]
American National Standards Institute (ANSI) N18.7-1976 [5], 5.2.13.2, Control of Purchased Material, Equipment and Services, clearly indicates that audits take place at the suppliers facility:
Control of Purchased Material, Equipment and Services. Measures shall be provided to assure that purchased items and services, whether purchased directly or through contractors, conform to the procurement documents. These measures shall include provisions, as appropriate, for source evaluation and selection, objective evidence of quality furnisher by the contractor, inspection and audit at the source and examination of items upon delivery. [Emphasis added.]
In addition, ANSI N18.7-1976 [5], Section 5.2.13.1, addresses the need for including provisions in procurement documents to secure the rights for individuals conducting inspections and audits to access the suppliers facilities.
(3) Source Inspection and Audit. Provisions for access to the suppliers facilities and records for source inspection and audit when the need for such inspection or audit has been determined. [Emphasis added.]
1-4
Background and Objective 1.7.3 ANSI/ASME N45.2-1977, Quality Assurance Program Requirements for Nuclear Facilities [6]
ANSI N45.2-1977 [6], Section 8, Control of Purchased Material, Equipment, and Services, includes similar language that clearly indicates that audits take place at the suppliers facility:
Measures shall be established and documented to assure that purchased items and services, whether purchased directly or through contractors, conform to the procurement documents. These measures shall include provisions, as appropriate, for source evaluation and selection, objective evidence of quality furnished by the contractor, inspection and audit at the source, and examination of items upon delivery. [Emphasis added.]
ANSI N45.2.10-1973 [7], endorsed by NRC Regulatory Guide 1.74 [8], defines audit as:
An activity to determine through investigation, the adequacy of, and adherence to, established standards or other applicable contractual and licensing requirements, and the effectiveness of implementation.
ANSI N45.2.13-1976 [9], Section 4, Selection of Procurement Sources, Subsection 4.2, Selection Measures, states in part:
Methods to be utilized in evaluation of Supplier sources, and the results therefrom, shall be documented and shall include any or all of the following... The Suppliers technical and quality capability as determined by a direct evaluation of his facilities and personnel, and the implementation of his quality assurance program. [Emphasis added.]
NRC Inspection Procedure 43004 [10], on commercial-grade surveys pursuant to accepting an item for use as a basic component using commercial-grade dedication, defines source verification as follows:
The entity should have a documented and effectively implemented program and/or procedures to control the critical characteristics of the item(s) being procured.
- 1. The survey should be conducted by an individual(s) that is also trained in auditing and knowledgeable in the operation of the item(s) and the associated critical characteristics to be verified. The verification is accomplished by reviewing the vendors program/procedures controlling these characteristics and observing the actual implementation of these controls in the manufacture of items identical or similar to the items being purchased. [Emphasis added.]
1.7.4 ASME NQA-1-1994 [11], Quality Assurance Requirements for Nuclear Facility Applications ANSI/ASME NQA-1-1994 [11], Basic Requirement 7, Control of Purchased Items and Services, Supplement 7S-1, Supplementary Requirements for Control of Purchased Items and Services, Section 3, Supplier Selection, Subsection 3.1, Source Evaluation and Selection, includes language that states in part:
Measures for evaluation and selection of procurement sources, and the results therefrom, shall be documented and shall include one or more of (a) through (c) below...
(c) Suppliers technical and quality capability as determined by a direct evaluation of his facilities and personnel and the implementation of his quality assurance program.
[Emphasis added.]
1-5
Background and Objective In this context, direct evaluation is interpreted as implying evaluation at the facility being assessed.
1.7.5 ASME NQA-1a-2009 [12]
Many suppliers are committed to ANSI/ASME NQA-1a-2009 [12], endorsed by NRC Regulatory Guide 1.28 [13]. In this standard, Part 1, Requirement 7, Control of Purchased Items and Services, Section 200, Supplier Evaluation and Selection, states in part:
Prior to award of a contract, the Purchaser shall evaluate the Suppliers capability to provide items or services in accordance with the requirements of the procurement documents. Supplier evaluation and selection and the results therefrom shall be documented and shall include one or more of the following:
(a) Suppliers history of providing an identical or similar product that performs satisfactorily in actual use. The Suppliers history shall reflect current capability.
(b) Suppliers current quality records supported by documented qualitative and quantitative information that can be objectively evaluated.
(c) Suppliers technical and quality capability as determined by a direct evaluation of the facilities, personnel, and the implementation of the Suppliers quality assurance program. [Emphasis added.]
In this context, direct evaluation is interpreted as implying evaluation at the facility being assessed.
1.7.6 ASME NQA-1-2015, Quality Assurance Requirements for Nuclear Facility Applications ASME NQA-1-2015 [14], endorsed by NRC Regulatory Guide 1.28 [13], includes the following definitions:
assessment: an all-inclusive term that may include review, evaluation, inspection, test, check, surveillance, or audit to determine and document whether items, processes, systems, or services meet specified requirements and perform effectively.
audit: a planned and documented activity performed to determine by investigation, examination, or evaluation of objective evidence the adequacy of and compliance with established procedures, instructions, drawings, and other applicable documents, and the effectiveness of implementation. An audit should not be confused with surveillance or inspection activities performed for the sole purpose of process control or product acceptance.
audit, external: an audit of those portions of another organizations quality assurance program not under the direct control or within the organizational structure of the auditing organization.
NQA-1 requirement 4, Section 204, Right of Access:
The procurement documents shall provide for access to the Suppliers and subtier Suppliers facilities and records for surveillance, inspection, or audit by the Purchaser, its designated representative, and others authorized by the Purchaser. [Emphasis added.]
1-6
Background and Objective 1.7.7 ASME NQA-1-2019 [15], Quality Assurance Requirements for Nuclear Facility Applications ASME NQA-1-2019 [15], Part II, Subpart 2.14, Quality Assurance Requirements for Commercial Grade Items and Services, Section 603(a), states:
A commercial grade survey is performed in accordance with a checklist or plan at the suppliers facility and includes or addresses the following... [Emphasis added.]
Although the authors of these documents probably did not anticipate the availability of (and therefore did not consider the efficacy of) real-time communication technologies for remote assessments, the language in regulations and standards to which licensees in the United States are committed implies that assessments such as audits and commercial-grade surveys are conducted in-person at the source location.
1.7.8 DOE-HDBK-1230-2019, Commercial Grade Dedication Application Handbook The DOE handbook provides information to assist DOE and National Nuclear Security Administration sites in commercial-grade dedication of items and services. Section 6.5.3, Method 2, Commercial Grade Survey of the Supplier, states:
Method 2 activities are performed at the suppliers facilities by the dedicating entity or its agent before releasing the commercial grade item or commercial grade service from the supplier or test laboratory facility. [Emphasis added.]
1.8 Acceptance of Remote Assessment Activities The traditional approach to supplier evaluation is to perform assessments such as audits and surveys on-site and in-person. This approach is based on and therefore consistent with the regulatory documents and industry standards cited in Section 1.7. Although many remote communication technologies exist today that would have been considered science fiction at the time the documents cited in Section 1.7 were written, the traditional approach has proven effective.
One benefit of the COVID-19 pandemic is that it has necessitated innovative approaches to conducting business as well as everyday living. Some of the compensatory measures developed to facilitate daily business tasks and everyday life may be adopted post-pandemic. Precedents exist for adoption of remote technologies and techniques, such as electronic signatures, digital record storage, and virtual medical appointments. Although these precedents provide a foundation for establishing legitimacy and legal acceptance of activities enabled by use of remote techniques, the technical advisory committee that developed this report recognizes that nuclear quality assurance is of utmost importance because it is relied on to ensure nuclear safety.
Those involved are deeply committed to nuclear safety. For this reason, sufficient operating experience, a successful performance record, incorporation of lessons learned, and perhaps regulatory approval are prerequisite to expanding use of remote assessment techniques (beyond hybrid assessments) during normal conditions.
1-7
Background and Objective 1.9 Typical Assessment Activities For the purpose of this report, assessments typically include two activitiesaudits and commercial-grade surveys. Audits verify the adequacy of quality program implementation for suppliers that provide items or services in accordance with quality assurance programs.
Commercial-grade surveys verify that identified critical characteristics are controlled by suppliers who provide commercial-grade items that will be dedicated by entities relying on the assessment to accept the identified critical characteristics.
Section 202 of ASME NQA-1, Part I, Requirement 18, Audits [14, 15], includes the following language about external audits:
External audits (e.g., Supplier audits) shall be performed on a triennial basis and supplemented by annual evaluations of the Suppliers performance to determine if the regular schedule audit frequency shall be maintained or decreased or if other corrective action is required. A continuous or ongoing evaluation of the Suppliers performance may be conducted in lieu of the annual evaluations, provided that the results are reviewed in order to determine if corrective action is required.
1.10 Structure of This Report Figure 1-1 shows the contents and organization of this report.
1 *Background - the need for remote assessment techniques and review of regulatory and standard language 2 *Lexicon and Acronyms 3 *Activites Associated with Performing Assessments 4 *Remote Assessment Techniques 5 *Review of the Performance-Based Supplier Assessment Process 6 *Process for Evaluating Use of Remote Assessment Techniques 7 *Planning for An Assessment That Uses Remote Assessment Techniques 8 *Remote Assessment Tools/Applications 9 *References A *Current Application and Experience B *Remote Assessment Planning Worksheet C *Performance-Based Assessment and Critical Characteristic Worksheets D *Pre-Assessment Checklists for Remote Assessments
- Frequently Asked Questions E
Figure 1-1 Report structure 1-8
2 LEXICON AND ACRONYMS 2.1 Lexicon assessment For the purpose of this report, assessments typically include audits to verify the adequacy of suppliers that provide safety-related items and commercial-grade surveys to verify that identified critical characteristics are controlled by suppliers that provide commercial-grade items that will be dedicated by entities relying on the assessment to accept the identified critical characteristics.
A review, evaluation, inspection, test, check, surveillance, or audit to determine and document whether items, processes, systems, or services meet specified requirements and perform effectively. (ASME NQA-1, Part II, Subpart 2.22, Section 101 [14, 15].)
An all-inclusive term that may include review, evaluation, inspection, test, check, surveillance, or audit to determine and document whether items, processes, systems, or services meet specified requirements and perform effectively. (ASME NQA-1, Part I, Introduction, Section 400
[14, 15].)
audit A planned and documented activity performed to determine by investigation, examination, or evaluation of objective evidence the adequacy of and compliance with established procedures, instructions, drawings, and other applicable documents and the effectiveness of implementation. An audit should not be confused with surveillance or inspection activities performed for the sole purpose of process control or product acceptance. (ASME NQA-1, Part I, Introduction, Section 4 [11] and Section 400 [12, 14, 15].)
audit, external An audit of those portions of another organizations quality assurance program not under the direct control or within the organizational structure of the auditing organization. (ASME NQA-1, Part I, Introduction, Section 4 [11] and Section 400 [12, 14, 15].)
checks The tests, measurements, verifications, or controls placed on an activity by means of investigations, comparisons, or examinations to determine satisfactory condition, accuracy, safety, or performance. (ASME NQA-1, Part II, Subpart 2.8, Section 1.1 [11] and Section 101 [12, 14, 15].)
2-1
Lexicon and Acronyms commercial-grade Activities conducted by the dedicating entity or its agent to verify that survey a supplier of commercial-grade items controls, through quality activities, some or all of the critical characteristics of the designated commercial-grade items to be purchased. The verification can be used as a method to accept those characteristics. The commercial-grade survey should include verification of the supplementary documentation and the effective implementation of the commercial-grade quality program. (NRC IP-43004 [10].)
computer program A combination of computer instructions and data definitions that enables computer hardware to perform computational or control functions. (ASME NQA-1, Part I, Section 400 [12, 14, 15].)
critical characteristics The important design, materials, and performance characteristics of a commercial grade-item thatonce verifiedwill provide reasonable assurance that the item will perform its intended safety function.
(10CFR21 [17].)
examination An element of inspection consisting of investigation of materials, components, supplies, and services to determine conformance to those specified requirements that can be determined by such investigation.
Examination is usually nondestructive and includes simple physical manipulation, gauging, and measurement. (ANSI N45.2.10 [7].)
exigent conditions For the purposes of this report, extreme circumstances where it is not possible to perform on-site verification or assessment. For example:
Conditions that threaten the health and safety of individuals performing the verification or assessment Local, national, or global declaration of an emergency Local, national, or global restrictions on travel foreign material Any material that is not part of the system or component as designed.
Examples include dirt, debris, broken or missing parts, oil, slag, tools, rags, chemicals, machine tailings, lapping compounds, grinding particles, paint chips, leak-sealing compounds, and any other items that could adversely affect the intended operation, components, or chemistry of the system. (EPRI 1019805 [18].)
foreign material The plant processes and practices for preventing the introduction of exclusion foreign material into a system, equipment, or component. (EPRI 1019805 [18].)
2-2
Lexicon and Acronyms fully remote A fully remote assessment is an assessment conducted with all team assessment members participating remotely. Fully remote assessments evaluate all applicable technical and quality requirements of interest.
hybrid assessment A hybrid assessment is an assessment conducted with one or more team members participating in-person at the facility being assessed while other team members are participating remotely. Hybrid assessments evaluate all applicable technical and quality requirements of interest.
information and Information technology and audiovisual equipment and applications communication used during an assessment to enable audio and video communication, technology (ICT) document storage and sharing, and management of the assessment.
inspection Examination or measurement to verify whether an item or activity conforms to specified requirements. (ASME NQA-1, Part I, Introduction, Section 4 [11] and Section 400 [12, 14, 15].)
inspector A person who performs inspection activities to verify conformance to specific requirements. (ASME NQA-1, Part I, Introduction, Section 4 [11] and Section 400 [12, 14, 15].)
nondestructive The development and application of technical methods to examine examination materials and/or components in ways that do not impair future usefulness and serviceability in order to detect, locate, measure, interpret, and evaluate flaws. (2019 Edition, ASME Section V, Nondestructive Examination [19].)
performance-based An assessment using a methodology that evaluates processes or assessment activities on the basis of their performance and allows subsequent conclusions about the products of the process or activity and the quality assurance program of the supplier audited. (EPRI NP-6630 [20].)
provisional remote An assessment conducted with all team members participating assessment remotely that does not assess all technical and quality requirements of interest. The objective of a provisional remote assessment is to approve a supplier for use with appropriate restrictions during exigent conditions until an assessment that evaluates all technical and quality requirements of interest can be performed.
remote assessment Techniques, including the use of ICT, that are used for assessment techniques activities when assessors are not physically present at the facility where program implementation is being assessed.
2-3
Lexicon and Acronyms remote source A method used by the purchaser or its agent to witness activities, in verification real time, using technology (such as video communication) in lieu of in-person activities to verify that specified requirements are met. (EPRI 3002019436 [1].)
safety culture The U.S. Nuclear Regulatory Commission (NRC) defines nuclear safety culture as the core values and behaviors resulting from a collective commitment by leaders and individuals to emphasize safety over competing goals to ensure protection of people and the environment. (76 FR 34773; June 14, 2011 [21].)
software Computer programs and associated documentation and data pertaining to the operation of a computer system. (ASME NQA-1, Part I, Introduction, Section 400 [12, 14, 15].)
source surveillance A review, observation, or inspection for the purpose of verifying that an action has been accomplished as specified at the location of material procurement or manufacture. (ANSI N45.2.10 [7].)
source verification Activities witnessed at the suppliers facilities by the purchaser or its agent before releasing the commercial-grade item from the vendor or test laboratory facility to confirm by direct observation that the selected critical characteristics are verified by the vendor. (NRC IP-43004 [10].)
special process A process, the results of which are highly dependent on the control of the process or the skill of the operators, or both, and in which the specified quality cannot be readily determined by inspection or test of the product. (ASME NQA-1, Part I, Introduction, Section 4 [11] and Section 400 [12, 14, 15].)
surveillance The act of monitoring or observing to verify whether an item or activity conforms to specified requirements. (ASME NQA-1, Part I, Introduction, Section 4 [11] and Section 400 [12, 14, 15].)
testing An element of verification for the determination of the capability of an item to meet specified requirements by subjecting the item to a set of physical, chemical, environmental, or operating conditions. (ANSI N45.2.10 [7], ASME NQA-1, Part I, Introduction, Section 4 [11] and Section 400 [12, 14, 15].)
verification The act of reviewing, inspecting, testing, checking, auditing, or otherwise determining and documenting whether items, processes, services, or documents conform to specified requirements. (ASME NQA-1, Part I, Introduction, Section 4 [11] and Section 400 [14, 15].)
2-4
Lexicon and Acronyms 2.2 Acronyms and Abbreviations ADAMS Agencywide Documents Access and Management System (U.S. Nuclear Regulatory Commission)
ANSI American National Standards Institute ASME American Society of Mechanical Engineers A/V audiovisual CANDU Reactor design name derived from Canada deuterium uranium CANPAC CANDU Procurement Audit Committee CC critical characteristic CFI counterfeit and fraudulent item CFR Code of Federal Regulations COVID-19 coronavirus disease 2019 DOE U.S. Department of Energy EPRI Electric Power Research Institute FME foreign material exclusion FTP file transfer protocol ICT information and communication technology M&TE measuring and test equipment NDE nondestructive examination NIAC Nuclear Industry Assessment Corporation NQA nuclear quality assurance NRC U.S. Nuclear Regulatory Commission NUPIC Nuclear Procurement Issues Corporation (formerly Nuclear Procurement Issues Committee)
PBSA performance-based supplier audit Wi-Fi wireless fidelity 2-5
3 ACTIVITIES ASSOCIATED WITH PERFORMING ASSESSMENTS The traditional approach to assessing nuclear suppliers quality programs involves being on-site, in-person at their facilities. There are inherent advantages to this approach. For example, there is direct, in-person observation of activities affecting quality as well as interactive communication and immediate access to records, production facilities, and equipment involved in activities affecting quality.
Observing activities affecting quality is vital to performance-based assessments that focus on determining the adequacy of the suppliers controls over the product characteristics necessary for the product to perform its intended safety functions. This section describes typical assessment activities and the benefits associated with being in-person and on-site. This information provides a foundation for identifying and understanding assessment techniques that can be used to achieve similar results when performing assessments that involve one or more assessors working remotely. Section 4 of this report identifies remote assessment techniques and considerations associated with activities involved in an assessment. Section 5 reviews the performance-based approach to assessment of suppliers.
3.1 Review of Documentation Documentation provides objective evidence of processes and activities affecting quality. Many types of documents and quality records are reviewed during an assessment, including but not limited to drawings, procedures, manuals, viewing screens for software programs, calculations, customer and supplier procurement documents, notes, assessments, training records, required posted notices, and applicable supporting documentation.
Review of documentation involves direct access to controlled documents and quality records.
The following activities are typically involved when reviewing documentation:
Timely access to requested documents and records, including hardcopy and other document formats that cannot be easily shared or reviewed electronically Sharing of proprietary documents and records in a secure environment Direct observation of controlled documents in use at workstations Observation of indications of ad hoc deviation from documented processes and indications that documents are not frequently used Assessing the suppliers generation and maintenance of appropriate records Table 3-1 provides examples of assessment activities associated with review of documents.
3-1
Activities Associated with Performing Assessments Table 3-1 Assessment activities associated with access to documents Activity Examples/Practical Experience Timely access to requested documents/ Review subprocesses and associated activities records and access to hardcopy and other easily document types that cannot be easily Review availability and accessibility of reviewed electronically (such as work documents/records travelers, inspection reports, verification and validation binders) Ability to assess control, availability, and accessibility of documents Sharing of proprietary documents in a secure Review of customer purchase orders, design environment documents, production drawings, failure data, commercial-grade dedication plans Direct observation of controlled documents at Review of adequacy of document usage work locations practices versions Ability to observe use of uncontrolled or outdated Observation of indications of ad hoc deviation D that contain controlled from documented processes and indications documents that documents are not frequently used Difficult retrieving controlled documents Assessing the generation and maintenance of Storage of quality records in appropriate locations records 3.2 Access to Production Facilities The production facilities for a product are central to a suppliers quality program implementation. Assessing program implementation involves observation of production and associated activities affecting quality while they are in progress. These observations enable the assessment team to evaluate the quality program and its effectiveness.
Observation and assessment of the following factors is typically enabled by direct access to production facilities:
Foreign material exclusion controls Flow of processes involved in production, including manufacturing and inspection activities Condition of manufacturing equipment and tools used to conduct activities affecting quality Processes or machinery that involve software or computer programs New or existing supplier capabilities that are not included in the current assessment scope Safety culture Table 3-2 provides examples of assessment activities associated with access to production facilities.
3-2
Activities Associated with Performing Assessments Table 3-2 Assessment activities associated with access to production facilities when in-person and on-site Activity Examples/Practical Experience Foreign material exclusion controls Product flow throughout manufacturing Observe/inspect completed products process and inspection activities completed products Evaluate segregation of nonconforming items and accessibility of associated documentation Compare as-built condition to design drawing/ documents Review measuring and test equipment (M&TE) controls Observe activities in the periphery of process flow Condition of manufacturing equipment Observe control of M&TE, including identification and and tools used to conduct activities affecting quality Evaluate general cleanliness and state of repair of equipment that may affect product quality Processes or machinery that involve Observe computer terminals and digital controls software controls integrated with machinery Supplier capabilities that are not Observe machinery, processes, and products not included in the current assessment included in current approved scope of supply scope (new or existing)
Nuclear safety culture Evaluate ability to raise safety and quality concerns without fear of reprisal Assess supplier personnel awareness that products perform important functions in nuclear power facilities Observe other indications of safety culture, such as industrial safety practices, including availability and use of personal protective equipment and safety precautions 3.3 Access to Supplier Personnel Supplier personnel who are directly or indirectly involved in activities related to the successful function and reliability of their products are also central to implementation of a suppliers quality program. Interaction between supplier personnel and the assessors enables a mutual understanding of applicable processes and requirements. These interactions may also address supplier concerns with a customers quality and technical requirements.
Direct interaction with supplier personnel helps build rapport between supplier personnel and assessors. Assessment of the following factors is typical during direct interaction with supplier personnel on-site:
Quality functions independence and sufficient authority Knowledge base of supplier personnel in quality outside the quality functions Rapport between suppliers quality departments and other supplier departments 3-3
Activities Associated with Performing Assessments Table 3-3 summarizes assessment activities associated with access to and interaction with supplier personnel.
Table 3-3 Assessment activities associated with direct access to personnel Activity Examples/Practical Experience Quality functions independence and Gain an understanding of the chain of command and other sufficient authority -to personnel and practices when resolving questions or uncertainty Knowledge of regulatory requirements, such as reporting of defects and noncompliance Witness interactions between suppliers quality departments and other departments Knowledge base of supplier Ability to directly interview personnel being observed about personnel, including those not in the the activity, applicable controls, and procedures instead of quality assurance organization relying on the designated supplier representatives to respond Evaluate awareness of Counterfeit and fraudulent items (CFIs) and measures to prevent their use Presence of safety and quality cultures, questioning attitude, and freedom to raise concerns Assess supplier personnel reactions to questions, including Amount of time required to respond Rapport between suppliers quality , and departments and other supplier activities necessary to answer questions departments Gain insights into safety culture and cooperation Rapport-building between supplier Develop a level of trust and enhanced communication and personnel and assessors understanding of requirements Allows ad hoc questions and emerging concerns to be addressed with minimal impact on schedule
, and activities necessary to answer questions
-up questions, and so on Q
availability Supplier personnel gain an understanding of customer requirements and the purpose behind them Ability of the suppliers personnel to learn from participating in assessment activities and interviews 3-4
Activities Associated with Performing Assessments 3.4 Access to Processes for Observation Performance-based assessment relies primarily on observing in-process activities affecting quality. Therefore, observation of these processes is vital when evaluating implementation of a suppliers quality program.
Assessment of the following factors is typically enabled through on-site access when observing supplier processes that affect the quality of the final product:
Indications of effective traceability and materials controls Special process implementation, such as nondestructive examination (NDE), welding, and heat treatment Processes affecting product fit, form, and ability to perform safety-related functions Process flow of items, materials, and products In-process and final inspection and testing Supplier controls for the prevention and detection of CFI Table 3-4 provides examples of assessment activities related to observation of processes.
Table 3-4 Assessment activities associated with direct access to processes Activity Examples/Practical Experience Indications of effective traceability Control of limited-life items (shelf life) and material controls Controls to mitigate use of suspected CFI Control of nonconforming items Identification of stored and in-process materials and items Special processes implementation Witness activities such as NDE, welding, and heat treatment to assess compliance with applicable procedures and controls Processes affecting product fit, Technical specialist observation and evaluation of important form, and ability to perform safety- processes related functions Process flow of items and Compare observed sequence to documented instructions materials Review of hold-point processing Determine if any processes underway are undocumented accordance with controlled documents Control In-process and final inspection and Review of commercial-grade dedication acceptance methods testing Supplier controls for the Personnel awareness of the threat caused by CFI, preventive prevention and detection of CFI measures during specification and procurement Measures to detect CFI during receiving, fabrication, inspection Process for controlling suspected CFI 3-5
Activities Associated with Performing Assessments 3.5 Assessment Team Collaboration Effective collaboration between assessment team members is necessary to ensure that the assessment is completed. To this end, the assessment team leader is responsible for coordinating activities of the team and addressing any emergent issues. Communication between the team leader and team members is essential because team members typically rely on each other for assistance as priorities change or emergent concerns are addressed. In addition to improving coordination of the assessment, collaboration promotes enhanced understanding of the suppliers processes and knowledge transfer and retention between team members.
3.5.1 Activities Related to Management of the Assessment Team for an On-Site Assessment The following types of coordination activities are typical during an on-site assessment:
Identification of potential concerns and emergent issues Collaboration on areas of the assessment checklist that overlap Development and transfer of knowledge and experience Table 3-5 provides examples of assessment activities related to flexibility and management of the assessment team.
Table 3-5 Assessment activities associated with flexibility and management of assessment team Activity Examples/Practical Experience Identification of potential concerns, Potential significant deficient condition found with discussion of emergent issues and team impact to product already shipped.
redirection Impromptu or unplanned meetings/discussions with all assessment team members.
Ensuring schedule adherence during the assessment.
Inter-relationship of assessment Collaboration between members with different areas checklistcollaboration on assessing of concern (such as a member evaluating commercial-areas of the assessment checklist that grade dedication discussing issues with another overlap member evaluating procurement and supplier evaluations).
Assessment activity is difficult to complete in timeframe, and other team members can help complete the assessment.
Knowledge and experience attainment and to mentor and train transfer the assessment.
Obtain hands-with the types of facilities, processes, and equipment used by suppliers.
3-6
Activities Associated with Performing Assessments 3.6 Assessment and Assessment Resource Planning Careful and deliberate planning is one of the most essential activities involved in performing an assessment.
The following activities are typically involved in assessment planning:
Identification of quality program requirements and quality and technical requirements of interest Identification of assessors and technical specialists Assigning assessment areas of interest/concern (for example, checklist sections) to team members Identification of any technological, logistical, or other assessment challenges Table 3-6 provides examples of assessment activities related to assessment and assessment resource planning.
Table 3-6 Activities related to assessment and assessment resource planning Activity Examples/Practical Experience Quality program requirements and Identifying the technical attributes associated with the technical requirements scope of supply Identifying the quality program requirements that will be the focus of the assessment Planning activities, interviews, and observations to be conducted while at the suppliers facilities Assigning assessment areas of Identifying which assessors are responsible for each interest/concern to team members Identifying which technical attributes associated with the products form, fit, and function the technical specialist will evaluate Identification of any challenges for the Establish a baseline schedule for assessment assessment and the team (such as activities logistical and technological)
Determining if there are periods during which supplier personnel may be unavailable meeting 3-7
4 REMOTE ASSESSMENT TECHNIQUES The purpose of this section is to identify and describe assessment techniques that can be used to successfully complete assessment activities remotely (remote assessment techniques) when it is not possible for all assessment team members to be present in-person and on-site. It identifies the methods that may be used to best mitigate any gaps that arise when using a remote assessment instead of a traditional on-site, in-person assessment.
Remote assessment techniques typically involve employing ICT to enable evaluation of supplier processes, controls, and corresponding documentary evidence of implementation in a way that provides results similar to those obtained from in-person, on-site assessments.
As covered in Section 3, assessments typically involve a number of activities, including but not limited to the following:
Review of documentation A walk-through of the facility being assessed (sometimes in sequence from order entry to shipping) and observation of work activities Observation of work activities, processes, and the individuals performing them Interviews of supplier personnel Assessment team meetings Traditional on-site assessments afford flexibility for completing these activities in terms of schedule, timely access to personnel and records, ability to reassign assessors, and ability of assessors to observe multiple objects at once with a turn of the head or other movement as necessary to achieve the desired line of sight during observation. The remote assessment techniques described in this section involve an increased amount of advance planning and consideration to ensure similar capabilities during an assessment using remote assessment techniques.
Although the techniques in this section are described in terms of assessing technical and quality requirements of interest, they may also be applied to other assessment activities because the scope of assessment typically includes the entire quality program.
4-1
Remote Assessment Techniques 4.1 Review of Documentation Most documentation review activities can be performed remotely provided that the supplier is willing to permit remote access to the documents. Remote review of documentation requires advance preparation and consideration, such as the following activities:
Arranging the necessary permissions, controls, and (if required) nondisclosure agreement with the supplier Arranging controlled avenues of sharing documents, such as cloud or local content management and file-sharing services that can be configured to provide secure access to specified individuals at the desired level (for example: upload, download, preview, edit)
Advance identification of the documents or types of documents that are anticipated to be requested and notifying the supplier so that arrangements can be made to provide requested documents in advance of and on request during the assessment Table 4-1 provides examples of remote assessment techniques that can be applied to enable successful review of documentation.
Table 4-1 Remote assessment techniques for review of documentation Technique Examples/Practical Experience Establish a secure means file-sharing platform.
sharing. The platform Cloud or local content management and file-sharing services or platforms may drive the duration can be configured to provide secure access to specified individuals at the and efficiency of the , upload, download, preview, edit).
document review include , Microsoft SharePoint, process. t One Drive, file transfer protocol (FTP) sites, Google Drive, and Use the same primary and contingency ICT (software, applications, and/or hardware) to hold pre-assessment briefings to establish the necessary functionality, ICT , and establish functionality for all involved.
Evaluate document control and processes to generate and maintain quality records through interviews and, if necessary, video access to Arranging the necessary Obtain agreement and cooperation from supplier, including arranging the permissions, controls, necessary permissions to review documents through the content and (if required) management platform and (if required) a nondisclosure agreement with nondisclosure agreement the supplier.
with the supplier. -sensitive or proprietary information is typically not available for off-site review. Measures such as nondisclosure agreements and controlled-use file-share sites are required to enable off-site review.
Schedule a preliminary meeting with the supplier to verify that the selected technologies (primary and contingency plan) function as intended.
4-2
Remote Assessment Techniques Table 4-1 (continued)
Remote assessment techniques for review of documentation Technique Examples/Practical Experience Advance identification of Notify the supplier of documents needed, and request that they be the documents or types provided in advance. Some documents typically requested during an of documents that are assessment, such as travelers, procedures, or inspection reports, anticipated to be should be requested in advance to enable remote auditing.
requested.
themselves with the structure of documents in advance (if possible) and identify any additional reference documents required to complete the review.
Arrangements should be made in advance with the supplier to determine how documents not provided in advance can be provided during the assessment.
Advance access to the suppliers quality assurance program and implementing procedures can inform the assessor as to what additional documents to request for review, whereas remote assessments require advanced review to achieve similar results.
Ad hoc mobile phones with adequate resolution to be legible, particularly when and sent to recipients for ease of viewing.
4.2 Access to Production Facilities Remote assessors cannot physically walk through the facility, but it is possible to use live video and audio communication to permit remote assessors to virtually accompany an on-site assessor or supplier representative on a walk-through of the facility and observation of any in-process work activities. Compact (wearable) cameras that are capable of capturing high-definition video can be used by on-site personnel to provide visual observation capabilities to remote assessors through live steaming. Two-way audio communication by mobile phone or live-streaming mobile phone applications can be used to enable assessors to direct local personnel in situations where a closer look, longer look, change of direction, or change of focus is needed during the walk-through. In addition, wearable cameras can record video of the walk-through that could be stored in a file-sharing location approved by the supplier and used for reference during the assessment. Conducting a remote walk-through requires advance preparation, such as the following:
Advance discussion of required safety and security protocols, such as health screening, testing, social distancing, and personal protective equipment (for example, during hybrid assessments)
Agreement and approval to use cameras and other ICT from supplier management Acquisition, setup, and testing of ICT, such as cameras, live-streaming, and audio communication applications Training personnel in use of ICT equipment 4-3
Remote Assessment Techniques Trial run of primary and contingency ICT equipment and internet access before the assessment starts Development and communication of contingency plans to address loss or deterioration of audio or video signals Table 4-2 provides examples of remote assessment techniques that can be applied to enable successful access to production facilities.
Table 4-2 Remote assessment techniques for successful access to production facilities Technique Examples/Practical Experience Advance discussion of required safety Identify applicable safety and security protocols prior to and security protocols, such as health start of the assessment.
screening, testing, social distancing, and For hybrid assessments, requirements for testing and personal protective equipment evidence of testing/vaccination (CLEAR 1 Health Pass, vaccination cards, and so on) need to be communicated far enough in advance to enable compliance.
Assessor should identify in advance Request the entity being assessed to provide a site map or associated considerations and areas of floor plan before the assessment. -through interest during the observation, such as plan, complete with questions for supplier staff and specific cleanliness, safety culture, and industrial areas of focus for the camera, can be prepared in advance.
safety practices Advance identification of the desired sequence for the
-through or tour and specific production areas or activities related to the technical and quality requirements of interest.
Acquisition, setup, and testing of Obtain agreement and approval to use cameras and other appropriate cameras, live-streaming, and ICT from supplier management. This should be addressed audio communication applications prior to issuance of the audit/survey plan. Successful using technical support staff from the audit team leaders s
A wearable camera ° camera is effective in enabling remote observation of production facilities and processes.
Training personnel in use of the equipment training protocol and/or guidance may be required with the assistance of technical support staff.
Development and communication of This should include the availability of multiple sources of contingency plans to address loss or internet connections, such as hot spots and smartphones.
deterioration of audio or video signals Ensure that sufficient time has been allotted for activities in the event that ICT does not function as desired.
1 CLEAR is a registered trademark of CLEAR.
4-4
Remote Assessment Techniques Table 4-2 (continued)
Remote assessment techniques for successful access to production facilities Technique Examples/Practical Experience Trial run of primary and contingency Once the audit plan has been issued, trial run(s) should be equipment and internet access before conducted prior to the audit/survey entrance meeting to the assessment starts ensure that all ICT equipment functions as desired.
processes and procedures should operate the video and desirable by the supplier to operate a camera if the primary assessor and maneuver a camera in a fashion that supports the assessors ability to view the facility. Safety individual holding the camera to prevent tripping, should be arranged. Hands-free or worn cameras may not require spotters, and so on.
Address who will operate equipment and how to do it safely in pre-to apply and discuss these needs and their application.
This may involve personnel other than those normally involved in an assessment.
or interference from equipment nearby that could challenge the clarity of necessary communications.
Determine how two-way communication will be enabled between remote assessors and supplier or other assessor individuals on-site.
Two-way communication through a videoconferencing integrated laptop cameras may not have sufficient field of vison or resolution for witnessing production activities, smartphones or similar cameras are better suited steady. Employment of a smartphone camera in combination with an application identified in Section 8
( ) can provide high-resolution images with two-way video/audio.
Determine the supplier hosts language proficiency to hear and understand questions or requests for additional camera views from the remote assessor.
Consider if facility structures (metal roof ) or layout impacts the type of audio communication needed.
Determine if on-site personnel must wear headphones to mitigate high sound in machining or fabrication areas.
4-5
Remote Assessment Techniques 4.3 Access to Supplier Personnel In-person communication allows all parties to participate both verbally and nonverbally. When remote techniques are warranted, supplier interviews can be accomplished using a variety of applications that enable Web-based meetings and communication. These tools typically allow for sharing of documents as well as two-way audio and video communication. Use of ICT, such as Web-based meeting and communication tools for remote assessment interviews, requires advance preparation. Examples are as follows:
Advance identification of interview schedules, locations, and interviewee expectations Acquisition, setup, and testing of remote and local computers (at necessary locations),
including necessary ICT, such as cameras and microphones Development and communication of contingency plans to address loss or deterioration of audio or video signals Table 4-3 provides examples of remote assessment techniques that can be applied to enable successful observation and interviews of supplier personnel.
Table 4-3 Remote assessment techniques for successful observation and interviews of personnel Technique Examples/Practical Experience Advance identification of interview schedules, Determine the setting the interviewee will be in locations, and during the interview.
Get agreement from interviewee to maintain video and audio communication during interviews (for s in the local environments during interviews.
adding attendees to virtual meetings and sharing documents, images, and other information.
as needed.
Acquisition, setup, and testing of remote and Ensure that remote and local computers being used local computers, including cameras and have adequate resolution and are positioned to microphones as necessary ensure adequate lighting and visual field necessary t
transmission.
may be required to achieve adequate clarity.
Ensure that internet bandwidth can accommodate the video/audio platform, including document sharing and multiple locations.
Development and communication of Conduct trial run of primary and contingency contingency plans to address loss or equipment before the assessment starts.
deterioration of audio or video signals 4-6
Remote Assessment Techniques 4.4 Access to Processes for Observation Similar to remote assessment techniques that can be applied to a facility walk-through, it is possible to use live two-way video and audio communication to enable observation by remote assessors. Remote observation of work activities, processes, and individuals performing them requires advance preparation, such as:
Advance identification of the processes and work activities anticipated to be observed and notifying the supplier so that appropriate ICT equipment and internet access can be in place before the assessment Acquisition, setup, and testing of ICT, such as appropriate cameras, live-streaming, and audio communication applications Training personnel in use of ICT equipment Trial run of ICT equipment before the assessment starts Development and communication of contingency plans to address loss or deterioration of audio or video signals Table 4-4 provides examples of remote assessment techniques that can be applied to enable successful observation of processes and the individuals performing them.
Table 4-4 Remote assessment techniques for successful observation of processes and individuals performing them Technique Examples/Practical Experience Advance identification of the processes nclude but are not observed and notifying the supplier so that appropriate equipment and internet access can be in place in advance of the assessment Product handling Calibration activities Welding, control of welding material Calibrati document identification, personnel badges remote observation equipment is critical to the success of the remote assessment.
4-7
Remote Assessment Techniques Table 4-4 (continued)
Remote assessment techniques for successful observation of processes and individuals performing them Technique Examples/Practical Experience Acquisition, setup, and testing of Communicate with supplier to determine the type and appropriate cameras, live-streaming, and amount of ICT equipment required as well as personnel audio communication applications to operate the equipment.
Mobile phones (iPhone, Samsung, and so on)
Cameras (GoPro, RealWear, and so on)
Laptops Tablets (such as an iPad)
Microphones Recording options Evaluate the ability of the primary assessor to observe, or individual performing the activity for limitations in controlling audio and video. Are different angles or fields of view required? In addition, their ability to enact contingency plans may be limited.
Consider arranging for another individual to manage the ICT equipment to avoid distracting the individual performing the activity and allow a more efficient process for the assessor to obtain necessary information.
Training personnel in use of the equipment Ensure that supplier personnel and assessment team equipment and software.
rial run of the equipment performed in the most challenging (such as the noisiest or dimmest) areas ns that can be corrected in advance.
Development and communication of contingency plans to address loss or a landline in the event that Wi-Fi communications are deterioration of audio or video signals inadequate. Wi-Fi or cellular connections often have poor reception because of interference from building structures.
messaging, or alternative devices or points of contact.
contingencies other than to pause assessment of the the supplier to resolve or reschedule to a later time during the assessment.
4-8
Remote Assessment Techniques 4.5 Assessment Team Collaboration Assessment team meetings can be accomplished using ICT, such as Web-based meetings and the communication tools covered in Section 4.4. Lower-resolution video and audio quality may suffice for team meetings. Use of Web-based meeting and communication tools for remote assessment team meetings requires advance preparation, such as:
Advance identification of team meeting schedules and locations so that appropriate equipment can be in place in advance of the assessment. This may involve mobile ICT equipment, such as laptops at the suppliers facility and internet hot spots provided by assessment team members or the suppliers internet connection.
Acquisition, setup, and testing of ICT, including remote and local computers and cameras and microphones as necessary.
Development and communication of contingency plans to address loss or deterioration of audio or video signals.
Trial run of primary and contingency equipment before the assessment starts.
Measures to capture successful use of ICT and communicate the methods and equipment used to improve future use of remote techniques.
Table 4-5 provides examples of remote assessment techniques that can be applied to enable successful management of the assessment team.
Table 4-5 Remote assessment techniques for management of the assessment team Technique Examples/Practical Experience Advance identification of team meeting schedules Plans are made to acquire internet connectivity and locations so that appropriate equipment can and to establish methods of remote file transfer in be in place in advance of the assessment. This advance of the assessment. are may involve use of mobile equipment, such as and OneHub.
laptops at the suppliers facility and internet hot spots provided by assessment team members or the suppliers internet connection.
Acquisition, setup, and testing of remote and local Online tools mentioned in Section 8, such as computers, including cameras and microphones Avatour and Auditboard, allow for virtual as necessary. inspections with both video and audio capabilities.
Development and communication of contingency Contingency plans should be made to mitigate plans to address loss or deterioration of audio or potential problems with remote technology.
video signals.
Trial run of primary and contingency equipment Testing and validation of the remote technology before the assessment starts; consider if lower- will ensure success during the assessment.
resolution video and audio quality may suffice.
Assessments conducted remotely may be recorded for future reference. transfer as well as for training purposes to improve future performance. Permission should be obtained in advance from individuals being recorded.
4-9
Remote Assessment Techniques Table 4-5 (continued)
Remote assessment techniques for management of the assessment team Technique Examples/Practical Experience Establish appropriate meeting and communication Web-based meeting tools (Microsoft Teams, tools, including means of sharing assessment and so on).
team documents.
During pre-assessment planning, develop During pre-assessment planning, primary and communication plan, including methods and secondary communication methods should be schedules for team communications. establishe when scheduling meetings with both the remote team as well as supplier staff.
For international engagements, differences in languages and arrangements tor translation should be considered. Translators may need to be trained in ICT.
Provide opportunity during planning phase to Use the communication tools that will be used orient team members with the planned during the assessment when conducting planning communication technology. meetings with the assessment team and the supplier.
Establish the method for timely communication of The lead assessor is advised when established issues identified during assessment to lead priority thresholds are met in accordance with the assessor. communication plan.
4.6 Assessment and Assessment Resource Planning Careful and deliberate advance planning is essential to successful application of remote assessment techniques. In addition to traditional planning activities, provision for use of primary and contingency ICT, detailed coordination with the supplier, and coordination with the assessment team members are required. Planning an assessment that involves remote assessment techniques involves:
Determining the extent to which remote assessment techniques can be used. (Section 6 describes a process for making this determination.)
Planning the activities, interviews, and observations to be conducted and determining which will be conducted remotely and which will be entirely in-person and on-site. Careful, detailed scheduling with the supplier in advance is needed to ensure that the appropriate supplier staff, assessors, and technical specialists (if necessary) are available for each interview.
Identification of ICT that will be used during the assessment, as follows:
- Identifying a platform for sharing documented content and establishing the correct levels of access for supplier and assessment team personnel
- Identifying audio and video equipment to be used for each assessment activity and the individuals who will be responsible for operating it 4-10
Remote Assessment Techniques
- Developing contingency plans, including secondary forms of content sharing, audio, and video communication
- Conducting a test or trial run of primary and contingency ICT, including transitioning from primary to contingency technologies Table 4-6 provides examples of remote assessment techniques that can be applied to enable successful management of the assessment team.
Table 4-6 Remote assessment techniques for planning assessment and resources Technique Examples/Practical Experience assessment techniques can be used determination.
Planning activities, interviews, and observations ence at to be conducted and determining which will be the facility being assessed and/or have conducted remotely and which will be entirely in-person and on-site Careful, detailed scheduling with the supplier in advance is needed to ensure that the appropriate supplier staff, assessors, and technical specialists (if necessary) are available for each interview.
Identification of technologies that will be used Identifying a platform for sharing documented during the assessment content and establishing the correct levels of access for supplier and assessment team personnel.
Identifying audio and video equipment to be used for each assessment activity as well as the individuals who will be responsible for operating the equipment.
Developing contingency plans, including secondary forms of content sharing, audio, and video communication.
Conducting a test or trial run of primary and contingency technologies, including transitioning from primary to contingency technologies.
4-11
5 REVIEW OF THE PERFORMANCE-BASED SUPPLIER ASSESSMENT PROCESS 5.1 Performance-Based Supplier Assessment Process This report uses the term performance-based supplier assessment instead of audit because the performance-based methodology presented herein is intended to be applied to any type of assessment, including audits and commercial-grade surveys. As covered in the EPRI report Guidelines for Performance-Based Supplier Audits (NCIG-16) (EPRI NP-6630) [20], the concept of a performance-based approach is to focus the assessment on the suppliers in-process activities and, to a lesser extent, on a programmatic review of compliance with the quality assurance program. To this end, the scope and activities selected for evaluation should allow the assessment team to reach conclusions about the ability of the items provided by the supplier to perform their intended safety-related functions.
A performance-based assessment is enabled by advance preparation and planning that involves:
Understanding the scope of items provided by the supplier Identification of the safety function(s) performed by items provided by the supplier Identification of the attributes necessary for the items the supplier provides to perform their safety functions Identification of the suppliers activities that impart the necessary attributes Identification of a means for evaluating those supplier activities The performance-based approach to auditing was developed and adopted pursuant to the Nuclear Management and Review Council (NUMARC) Comprehensive Procurement Initiative [21] in the early 1990s to improve the programmatic approach to supplier audits.
The programmatic approach tended to assess compliance with regulatory requirements through a detailed review of documentation. The performance-based approach includes review of documentation but is primarily based on engineering input to emphasize evaluation of supplier activities that have significant effects on the important properties or attributes of items being procured that ensure the ability to perform their design functions.
Figure 5-1 illustrates the performance-based assessment process. This process consists of four phases:
Planning Conduct of assessment Documentation Follow-up 5-1
Review of the Performance-Based Supplier Assessment Process 5.2 Planning Phase of the Performance-Based Supplier Assessment Process The planning phase is critical to the performance-based process. It includes:
Establishing the assessments objective Identifying the items provided by the supplier Reviewing operating experience related to the items and supplier Identifying technical and quality requirements of interests based on engineering input Identifying supplier processes related to technical and quality requirements of interest Identifying acceptance criteria and methods for the supplier processes Selecting the assessment team and developing the assessment plan 5.3 Conduct of Assessment Phase of the Performance-Based Supplier Assessment Process The conduct of assessment phase is based on a logical approach to accomplishing the objectives of the assessment by coordinating activities with the supplier and understanding the flow of the suppliers processes, how the processes are controlled, and how procurement document requirements are met. This logical approach includes the following:
An entrance meeting to explain the assessment objectives and support required to accomplish them A walk-through of the facility by the entire assessment team, typically starting with receipt of an order and proceeding as an order would through order entry and review, engineering, manufacturing processes, inspections, certification, preparation for shipment, and so forth More detailed investigation of processes and activities of interest assigned to each assessment team member Identification and investigation of any areas of concern Determining if any significant problems exist and making notifications if appropriate An exit meeting to discuss assessment results with the supplier and coordinate any immediate follow-up activities required 5.4 Documentation Phase of the Performance-Based Supplier Assessment Process The documentation phase consists of sharing preliminary results of the assessment with interested individuals, gathering their feedback, and preparing the assessment report.
5.5 Follow-Up Phase of the Performance-Based Supplier Assessment Process Resolution, follow-up, and verification of any long-term corrective actions associated with the assessment are performed in this phase.
5-3
Review of the Performance-Based Supplier Assessment Process 5.6 Maintaining a Performance-Based Focus When applying remote assessment techniques, it is important to maintain focus on the technical and quality requirement of interest. In particular, this includes observation of the suppliers in-process activities that are important in ensuring that their products are capable of performing their intended safety-related functions. Clear identification of the technical and quality requirements of interest is essential. These technical and quality requirements, typically identified by engineering, are related to the capability of the suppliers products to perform their safety functions and provide a foundation for evaluating the extent to which remote assessment techniques can be used.
Section 6 describes a process that can be used to evaluate the technical and quality requirements of interest to determine the extent to which remote assessment techniques can be effectively applied. This evaluation includes considering the types of activities (such as the activities described in Section 3) associated with assessing each requirement of interest and determining if remote assessment techniques (such as techniques described in Section 4) can be used to provide similar results. This process can be used to determine if a fully remote, hybrid, or provisional remote assessment is appropriate based on the technical and quality requirements of interest.
5-4
6 PROCESS FOR EVALUATING USE OF REMOTE ASSESSMENT TECHNIQUES As explained in Section 5 of this report, a performance-based assessment focuses on supplier activities that are related to the technical and quality requirements of interest. This section includes a process that focuses on these performance-based activities to evaluate the extent to which remote assessment techniques can be successfully applied to observe each activity. When a traditional in-person, on-site assessment is not practical or possible, this process can be used during initial planning of an assessment to identify an appropriate approach to assessing the suppliers qualification. The process does not attempt to find basis in or advocate specific regulatory requirements, but instead focuses on evaluating the efficacy of available remote assessment techniques in accomplishing adequate observation of the technical and quality activities of interest. The process can be used as part of the assessment planning process during exigent conditions to determine which of three approacheshybrid assessment, provisional remote assessment, or fully remote assessmentis appropriate.
Each approach is described in more detail in the following, and Table 6-1 compares the three types of assessments:
Hybrid assessment (one or more assessment team members are on-site and in-person).
If it is possible for any members of the assessment team to be on-site at the suppliers facility, a hybrid assessment can be planned and conducted. A hybrid assessment would involve one or more (depending on the technical and quality activities of interest to be observed on site) assessment team members acting as the eyes and ears on-site, while the balance of the assessment team works remotely and is in communication with the on-site assessors. Hybrid assessments are covered in Section 6, Step 6.12.
Provisional remote assessment (no assessment team members are on-site, in-person). If it is not possible for any members of the assessment team to be on-site due to exigent conditions and other eligibility requirements described in Section 6.13.1 are met, a provisional remote assessment can be planned and conducted. The provisional remote assessment plan would be the same as that for a hybrid assessment because it would identify all the technical and quality activities to be assessed. However, the assessment would be completed to the extent possible without on-site, in-person presence. The only activities not assessed would be those that could not be adequately observed using remote techniques like those described in Section 4.
If review of supplier history (covered in Section 6.14) and the results of the provisional remote assessment provide adequate confidence that the supplier is effectively implementing the aspects of their quality assurance program that could be assessed, the provisional remote assessment can be used as the basis for maintaining supplier approval with restrictions. This 6-1
Process for Evaluating Use of Remote Assessment Techniques means that restrictions related to the performance-based technical and quality requirements of interest that could not be assessed remotely are imposed on each procurement. Examples of restrictions are as follows:
- Additional purchase order requirements
- Enhanced tests or inspections at receipt or prior to installation
- Document reviews prior to shipping
- Remote source verification Provisional remote assessments are covered in Sections 6.13-6.16.
Fully remote assessment. If it is not possible for any members of the assessment team to be on-site due to exigent conditions, a fully remote assessment can be planned and conducted if the eligibility requirements described in Section 6.9.1 are met.
A fully remote assessment is one performed without any on-site/in-person presence. In certain cases, a completely remote assessment may be possible. Fully remote assessments are covered in Section 6.9.
Table 6-1 Basic comparison of remote assessment types Hybrid Assessment Provisional Remote Fully Remote Assessment (Sections 6.11 and 6.12) Assessment (Section 6.9)
(Sections 6.13-6.16)
One or more assessor is on-site All assessors are remote. All assessors are remote.
and in-person.
No eligibility requirements. Eligibility requirements in Eligibility requirements in apply. apply.
Preferable that at least one Preferrable that at least one Preferable that at least one assessor is familiar with the assessor is familiar with the assessor is familiar with the facility. facility and most recent facility.
assessment.
Establishing or renewing Renewing suppliers approval. Renewing suppliers approval.
suppliers approval.
Results in typical period of Results in reduced period of Results in typical period of approval. approval. approval, but could be reduced if appropriate.
The numbering in the following dialog corresponds with the step numbers in the process for evaluating remote assessment techniques illustrated in Figure 6-1.
6-2
Process for Evaluating Use of Remote Assessment Techniques 6.1 Assessment Objectives The assessment objectives must be defined. Assessment objectives for audits of basic component suppliers include verifying the adequacy and effectiveness of the suppliers quality assurance program as it relates to the items supplied by the supplier and nuclear quality assurance requirements.
With regard to commercial-grade surveys, the objectives of the survey include verification that the supplier effectively controls the identified critical characteristics and implements a quality program that supports control of the critical characteristics.
In addition, in both audits and surveys, verification of the effectiveness of corrective actions for any findings identified during the previous assessment is required.
6.2 Identify Items to Be Considered Assemble a list of the items typically procured from the supplier. This list would include items and services supplied by the supplier. If the supplier provides support services for the items supplied, these services must be identified. For instance, if the supplier provides engineering services associated with the items provided, this should be identified. The same is true for repair services.
6.3 Identify Technical and Quality Requirements of Interest Identify the most important technical, functional, and performance attributes required for the types of items provided by the supplier to perform their design functions. Considerations in this area include the following:
Design and design verification Equipment qualification, including seismic and environmental Manufacturing/fabrication/assembly activities Special processes, including welding NDE Heat treatment Inspection/testing Calibration Document control Personnel qualifications/training Control of records With regard to commercial-grade suppliers that are approved through commercial-grade surveys, the technical and quality requirements of interest equate to the critical characteristics of the item being procured along with the supporting quality controls needed to ensure that the critical characteristics are effectively controlled.
6-4
Process for Evaluating Use of Remote Assessment Techniques 6.4 Identify Supplier Activities Related to Technical and Quality Requirements of Interest for Assessment Identify the most important supplier processes and activities that are related to the ability of the item being produced to perform their design functions. This step may require communication with the supplier in advance of the assessment to determine which specific processes require on-site, in-person presence. When performing a traditional assessment, for identification of categories of controls, for example, inspection or testing may be sufficient. When considering remote assessment, it may be necessary to identify the specific types of testing and inspection, such as tensile strength, hydrostatic/pneumatic test, performance test, or insulation resistance test, to determine if and how observation of the controls could be achieved using remote techniques.
Processes and activities may include the following categories as they relate to the technical and quality requirements of interest:
Design requirements and control Procurement and acceptance, including commercial-grade dedication Materials control, handling, storage, and shipping Fabrication and assembly processes Special processes (welding, heat treatment, and so forth)
Repairs Tests and inspections Environmental qualification testing Detection of suspect counterfeit and fraudulent items (SCFIs) as described in Plant Support Engineering: Counterfeit and Fraudulent ItemsMitigating the Increasing Risk, Revision 1 of 1019163 [23]
Foreign material exclusion (FME) controls Calibration Documentation, document controls, and control of quality records 6.5 Determine If On-Site Presence Is Practical This determination is based on existence of exigent conditions, applicable regulatory and quality assurance program requirements, and so on.
6.6 Plan and Conduct Traditional Assessment When it is possible or practical for assessment to take place at the suppliers facilities, plan and conduct a traditional performance-based supplier assessment.
6-5
6.7.1 Identify the Technical and Quality Requirements of Interest Document the performance-based technical and quality requirements of interest using the remote assessment planning worksheet.(see Figure 6-3, Appendix B).
6.7.2 Does Assessment of the Activity Normally Require On-Site, In-Person Presence?
Steps 6.7.2-6.7.6 are intended to be performed for each identified technical and quality activity of interest. A prudent approach may be to use a table such as the one in Figure 6-3 and Appendix B to complete these steps for each quality and technical requirement of interest.
Determine if on-site presence is normally required to assess the requirement of interest. For example, if the activity normally involves only review of documentation that takes place in a conference room, on-site presence would not be considered necessary. Conversely, if the activity normally involves witnessing a fabrication method or observing processes on the factory floor, on-site presence would be considered necessary. If the activity normally does require insight from on-site activities, proceed to Step 6.7.3. If not, proceed to Step 6.7.5.
6.7.3 Identify the Assessment Techniques Typically Used When Present In-Person/On-Site Using the information included in Section 3 of this report, consider the activities typically involved in assessing each quality and technical requirement of interest. This may require the assistance of an assessment team member familiar with the suppliers facilityfor example, a team member who has visited the suppliers facility within the past 36 months. Review of past assessment reports may also be helpful in identifying assessment techniques used during normal conditions.
6.7.4 Can Remote Assessment Techniques Provide Results Similar to On-Site, In-Person Techniques?
Determine if remote assessment techniques, such as those discussed in Section 4 of this report, can be used to provide successful observation and assessment of each technical and quality requirement or activity of interest in the worksheet. If they can, indicate YES in the Can remote assessment techniques provide insight/benefit? column of the worksheet illustrated in Figure 6-3 and proceed to Step 6.7.5. If they cannot, indicate NO in the Can remote assessment techniques provide insight/benefit? column of the worksheet and proceed to Step 6.7.6.
6.7.5 Identify the Remote Assessment Techniques That Will Be Used to Assess the Activity If remote assessment techniques can be used to accomplish the activity associated with the technical and quality requirement of interest, identify the remote assessment technique that will be used. For example, Provide supplier personnel with a live camera (smartphone and gimbal) and audio communication equipment that enable the remote assessment team to do a virtual walk-through of the facility and to direct the supplier personnel to take them to the desired locations, and so forth.
6-8
Process for Evaluating Use of Remote Assessment Techniques 6.7.6 Identify That On-Site Presence Is Necessary for Adequate Performance-Based Assessment of the Activity If remote assessment techniques cannot provide the necessary ability to observe and assess, indicate that on-site presence is necessary for adequate performance-based assessment of the technical or quality activity of interest in the On-Site Necessary column of the worksheet.
6.8 Do Any Activities Require On-Site Presence?
If column 6 of the completed remote assessment planning worksheet depicted in Figure 6-3 indicates that in-person and on-site presence is required to assess one or more technical and quality requirements of interest, proceed to Step 6.10, Complete Assessment Team Roster Based on Individuals Necessary for On-Site Activities. If column 6 of the completed remote assessment planning worksheet depicted in Figure 6-3 indicates that in-person and on-site presence is not necessary to assess any of the technical and quality requirements of interest, proceed to Step 6.9, Plan and Conduct Fully Remote Assessment.
6.9 Plan and Conduct Fully Remote Assessment Plan and conduct a fully remote assessment using the guidance provided in Section 7, Planning Process for Assessments Involving Remote Assessment Techniques.
6.9.1 Eligibility Requirements for Fully Remote Assessment The following conditions must be met in order to conduct a fully remote assessment:
In-person, on-site presence of assessors is not possible due to exigent conditions.
The supplier has been previously qualified by a traditional assessment (a fully remote assessment cannot be used to qualify a new supplier).
A review of the procurement history since the last triennial or other regularly scheduled audit/survey, including receipt inspection results, does not identify potential issues that could not be addressed without on-site, in-person presence during the assessment. [22]
Evaluation (through completion of a remote assessment planning worksheet, such as the one depicted in Figure 6-3 and Appendix B) leads to the determination (in Step 6.8) that on-site presence is not required to observe any of the technical and quality requirements of interest because remote techniques alone can be used to successfully complete activities associated with assessing the technical and quality requirements of interest.
If practical, additional confidence in a fully remote assessment can be obtained if at least one member of the fully remote assessment team (preferably the assessment leader and the technical specialist) is familiar with the suppliers facility and participated in the most recent assessment of the supplier. As an alternative, the assessment leader could consult with the assessment leader(s) and technical specialist(s) from previous assessments.
A shorter period of approval may be considered if indicated by the results of the fully remote assessment and other factors, such as a review of operating experience and the suppliers historical performance.
If the eligibility requirements for a fully remote assessment cannot be met, proceed to Step 6.10.
6-9
Process for Evaluating Use of Remote Assessment Techniques 6.10 Complete Assessment Team Roster Based on Individuals Necessary to Assess the On-Site Activities Develop an assessment team roster, identifying the individuals who can best serve as the teams eyes and ears on-site and are most familiar with the supplier, facility, and activities associated with assessing the technical and quality requirements of interest for which in-person, on-site presence is required.
6.11 Is Hybrid Assessment Possible?
Determine if it is possible to perform a hybrid assessment based on factors such as:
Exigent conditions Governmental and corporate travel restrictions Ability of assessment team members to travel Willingness of the supplier to permit entry to non-employees Willingness of the supplier to permit use of remote assessment technologies The pre-assessment checklists in Appendix D include several considerations for hybrid assessments.
6.12 Plan and Conduct Hybrid Assessment Plan and conduct a hybrid assessment using the guidance provided in Section 7 of this report.
6.13 Plan and Conduct a Provisional Remote Assessment If column 6 of the completed remote assessment planning worksheet depicted in Figure 6-3 indicates that in-person and on-site presence is required to assess one or more technical and quality requirements of interest and the eligibility requirements described in Section 6.13.1 are met, a provisional remote assessment can be planned and conducted. The assessment plan would be the same as that for a hybrid assessment because it would identify all the technical and quality activities to be assessed. However, the assessment would be completed to the extent possible without in-person, on-site presence. The only activities not assessed would be those that could not be adequately observed using remote techniques.
If practical, additional confidence in a provisional remote assessment can be obtained if at least one member of the provisional remote assessment team (preferably the assessment leader and the technical specialist) is familiar with the suppliers facility and participated in the most recent assessment of the supplier. As an alternative, the assessment leader could consult with the assessment leader(s) and technical specialist(s) from previous assessments.
A shorter period of approval may be considered if indicated by the results of the provisional remote assessment and other factors, such as a review of operating experience and the suppliers historical performance.
The results of the provisional remote assessment are evaluated by the assessment team along with the historical performance of the supplier to determine if provisional approval of the supplier is appropriate (Step 6.14).
6-10
Process for Evaluating Use of Remote Assessment Techniques The premise behind completion of a provisional remote assessment is to maintain provisional approval of an existing nuclear supplier (during exigent conditions) provided that appropriate restrictions are imposed on each order.
6.13.1 Eligibility Requirements for Provisional Remote Assessment The following conditions must be met in order to conduct a provisional remote assessment:
In-person, on-site presence of assessors is not possible due to exigent conditions.
The supplier has been previously qualified by a traditional assessment (a provisional remote assessment cannot be used to qualify a new supplier).
A commitment can be made to complete the assessment of the supplier as soon as practical when the exigent conditions no longer exist (Step 6.16).
6.14 Is Provisional Approval of the Supplier Appropriate?
After the provisional remote assessment is completed, assessment team members determine if provisional approval of the supplier is appropriate based on the scope and results of the assessment activities successfully completed and other pertinent factors, including a review of operating experience, procurement history, and supplier performance since the last traditional assessment.
If the assessment team determines that the results are satisfactory to provide adequate confidence that the supplier is effectively implementing the aspects of their quality assurance program that could be assessed, the provisional remote assessment can be used as the basis for maintaining supplier approval with restrictions. A shorter period of supplier approval may be considered if indicated by the results of the provisional remote assessment and other factors, such as a review of operating experience and the suppliers historical performance.
In addition, approval based on a provisional remote assessment involves a commitment (by action tracking or a similar process) to complete assessment of the supplier when the exigent conditions no longer exist (Step 6.16).
Appropriate restrictions are applied to each order placed during provisional approval. The restrictions are determined based on the results of the provisional remote assessment and should be related to the technical and quality requirements of interest that could not be sufficiently observed using remote assessment techniques. These restrictions impose additional activities affecting quality, such as:
Remote source verification Submittal of additional documentation, such as drawings, work travelers, and test procedures Review of documentation Testing and inspection at receipt Planned restrictions should be discussed with the supplier to ensure that they can be successfully applied to each procurement. If provisional approval is appropriate, proceed to Step 6.15. If not, proceed to Step 6.17.
6-11
Process for Evaluating Use of Remote Assessment Techniques 6.15 Prepare Basis for Restricted/Provisional Approval and Update Approved Supplier List Prepare a documented basis for provisional approval. This would include a summary of considerations and development of restrictions based on the quality and technical requirements of interest for which on-site presence is required (identified in column 6 of the remote assessment planning worksheet shown in Figure 6-3). Restrictions might include measures such as:
Remote source verification Submittal of additional documentation, such as drawings, work travelers, and test procedures Review of documentation Testing and inspection at receipt 6.16 Complete Assessment of the Supplier, Including Review of Items Provided During Provisional Approval, as Soon as Practical Complete the assessment activities that could not be completed using remote technologies as soon as possible. A commitment should be made in the applicable tracking system to ensure that this assessment occurs. This assessment should include a review of items purchased during the provisional approval period. A full on-site, in-person assessment is preferable, but at a minimum, an on-site, in-person assessment of the technical and quality activities of interest not assessed during the provisional remote assessment should be completed.
6.17 Pursue Alternative Options for Procurement and Acceptance Pursue other procurement options for the required items. These include but are not limited to:
Inter-utility transfer Commercial-grade dedication Use of alternative suppliers Use of third-party dedication suppliers Use of a combination of acceptance methods, such as source evaluation and selection, objective evidence of the quality furnished by the supplier, remote source verification at the supplier source, and examination of products on delivery Engineering change evaluation to permit use of an alternative replacement item 6-12
7 PLANNING FOR AN ASSESSMENT THAT USES REMOTE TECHNIQUES The process for planning an assessment that uses remote techniques is the same as that process for planning a traditional in-person and on-site assessment. However, additional considerations are required for successful planning and conduct of an assessment that involves remote techniques. Important considerations beyond those for a traditional assessment include:
Determining the suppliers willingness and capability to accommodate remote assessment techniques Detailed identification, planning, and scheduling of assessment activities that involve remote assessment techniques Identification of technologies to be used to assess activities, including equipment, equipment operators, communication capabilities, and contingency plans Advance preparation and training of supplier and assessment staff in use of the equipment Dry runs to test both the primary and contingency technologies in advance, including how to handle failure of the primary technology and smoothly transition to the contingency Three important elements of planning an assessment that will use remote techniques are:
Using the remote assessment planning worksheet included in Appendix B (or similar method) to determine the type of remote assessment to perform Using a performance-based supplier assessment or critical characteristics worksheet, such as those included in Appendix C (or similar method), to identify the technical and quality requirements of interest Using pre-remote-assessment checklists included in Appendix D (or similar method) to ensure that considerations and preparations necessary for a successful remote assessment are completed in advance of the assessment The planning process described in the rest of this section follows the traditional performance-based supplier assessment approach outlined in Section 5 while incorporating the three elements previously listed and additional considerations that are necessary for the successful application of remote assessment technologies.
7.1 Planning Phase 7.1.1 Establish Assessments Objective The assessment objective should be identified and documented.
7-1
Planning for an Assessment that Uses Remote Techniques When applicable, the objective should address the intent to perform portions of the assessment using remote techniques.
7.1.2 Identify Items/Services Provided by the Supplier A list of items included in the suppliers scope of supply should be identified and documented.
7.1.3 Review Operating Experience Related to Supplier and Items Provided A review of operating experience, such as the historical performance of the supplier, is conducted. Some of the information that can be considered includes but is not limited to the following:
Supplier performance history Receiving inspection history Input from licenses who use the supplier Industry issues or corrective action reports captured in available databases NRC inspection reports and follow-up available at nrc.gov Reports of defects and noncompliance available at nrc.gov or equivalent reports in applicable jurisdictions 7.1.4 Identify Technical and Quality Requirements of Interest or Critical Characteristics Technical and quality requirements or critical characteristics of interest are typically identified by Engineering. It is usually not necessary to include every technical and quality requirement applicable to an item in the scope of the assessment. Engineering identifies those that are most important to the entity performing the assessment.
More information on identification of technical and quality requirements of interest can be found in EPRI NP-6630, Guidelines for Performance-Based Supplier Audits (NCIG-16) [20].
More information on identification of critical characteristics for use in commercial-grade surveys can be found in EPRI 3002002982, Plant Engineering: Guideline for the Acceptance of Commercial-Grade Items in Nuclear Safety-Related Applications: Revision 1 to EPRI NP-5652 and TR-102260 [25].
7.1.5 Identify Suppliers Processes Related to Technical and Quality Requirements of Interest Identification of the suppliers processes related to technical and quality requirements of interest can be facilitated through a performance-based supplier assessment or critical characteristics worksheet similar to those included in Appendix C. Individuals who have previously performed assessments at the suppliers facility or similar supplier facilities and often representatives from the facility being assessed are often able to identify processes related to the technical and quality requirements of interest.
7-2
Planning for an Assessment that Uses Remote Techniques As explained in Section 6.4, this step may require communication with the supplier in advance of the assessment to determine which specific processes require on-site, in-person presence. When performing a traditional assessment, identification of categories of controlsfor example, inspection or testingmay be sufficient. When considering remote assessment, it may be necessary to identify the specific types of testing and inspection, such as tensile strength, hydrostatic/pneumatic test, performance test, and insulation resistance test, to determine if and how observation of the controls could be achieved using remote techniques.
7.1.6 Identify Acceptance Criteria and Methods for Supplier Processes Related to Technical and Quality Requirements of Interest Identification of acceptance criteria and methods for supplier processes related to technical and quality requirements of interest can be facilitated through a performance-based supplier assessment or critical characteristics worksheet similar to those included in Appendix C.
7.1. Develop a List of Documents Required to Support the Assessment For an assessment involving remote assessment techniques, it is important to develop a list of documents required to support the assessment during the planning phase. This list can be used when discussing the suppliers willingness to support remote assessors by sharing documents through a secure content management platform. The list can also be used to identify and request certain documents to be provided for review in advance of the assessment to enable more efficient use of time available for interviews and observations during the assessment. Documents such as the following should be considered:
Suppliers quality assurance program manual, secondary/supporting procedures, and other documents that define the controls and implementation of the quality program Procedures and work instructions associated with technical and quality assurance requirements of interest Design documents and engineering evaluations Training and qualification records Internal/external assessments Nonconformance and corrective action reports Equipment design and qualification records Manufacturing and in-process quality control records Drawings Calculations Commercial-grade dedication plans and results of acceptance activities Test procedures and resultant tests 7.1.7 Complete Assessment Team Prior to selecting the assessment team, the ability of the supplier to support remote assessment activities, the type of remote assessment to perform, and the teams ability to support the remote assessment need to be determined.
7-3
Planning for an Assessment that Uses Remote Techniques 7.1.7.1 Determine Suppliers Ability to Support Remote Techniques Determining the suppliers readiness and willingness to support an assessment using remote assessment techniques is one of the most important steps involved in the planning process. The following considerations should be discussed with the supplier prior to selecting assessment team members:
The suppliers willingness to provide documents (for example, those identified in Section 7.1.6.1) to assessors for remote viewing through a secure content management application and their willingness to do so in advance of the start of the assessment to enable more effective use of time during the assessment Need for nondisclosures or similar legal vehicles to enable sharing of documents Availability of technology (internet connectivity, computers, cameras, Web meeting tools, content-sharing platform) at the suppliers facility and in required locations during the assessment Willingness of supplier to permit photography in their facility for the assessment Willingness of supplier personnel to appear on camera for interviews and during observation of work activities Medical screening or testing required for assessors access to suppliers facilities Use of personal protective equipment, social distancing, and similar protocols during the assessment A checklist of preliminary considerations is included in Appendix D.
7.1.7.2 Determine the Type of Remote Assessment to Perform Using the process included in Section 6 and the remote assessment planning worksheet included in Appendix B or a similar method, determine the appropriate type of remote assessment to perform. If a hybrid assessment is possible, base the selection of in-person, on-site assessment team members on their expertise related to the technical and quality requirements of interest that require on-site presence for observation as indicated in the assessment planning worksheet.
7.1.7.3 Select the Assessment Team and Develop the Assessment Plan Ideally, the assessment team leader and technical specialist will have visited the facility being assessed in the past and be familiar with the products and processes of the entity being assessed.
As alternatives, the assessment team leader could consult with the assessment team leader(s) and technical specialist(s) from previous assessments or arrange for the supplier to provide a virtual tour of the facility before considering remote techniques. Assessment team members should be selected based on their ability to assess the identified technical and quality requirements of interest.
Selecting assessment team members and their assignments for an assessment that will include remote techniques involves considerations such as the following:
Appropriate knowledge and experience, including experience or familiarity with the supplier being assessed Ability of team members to travel to suppliers facility to support hybrid assessments 7-4
Planning for an Assessment that Uses Remote Techniques Travel restrictions Availability of internet service and required hardware/software Ability of each team member to access video meeting/conferencing tools, taking into account cybersecurity Differences in time zones (ability of all necessary parties to attend scheduled team meetings, interviews, and observations)
Limitations to access imposed by the supplier and local jurisdictions:
- Number of assessors permitted
- Health screenings
- Vaccination records
- Quarantine requirements 7.1.7.4 Assessment Team Preparation Similar to a traditional assessment, review of previous assessment reports, including the performance-based supplier assessment or CC Worksheet and Technical Specialists summary, will help familiarize team members with the suppliers activities prior to the start of the assessment.
Holding pre-assessment planning meetings using the Web conferencing tools and content management sharing platforms that will be used during the assessment will help reduce technical issues at the start of the assessment. Holding the pre-assessment meetings at times consistent with the schedule to be used during the assessment will reduce the likelihood of confusion caused by differing time zones at the start of the assessment.
Arranging for translators and training them in ICT may facilitate international assessments.
7.2 Conduct of Assessment Phase Conduct of the assessment should be very similar to that of a traditional assessment, other than reliance on communication technology for observation, interviewing, and team discussion. The remote nature of assessment activities should not impact the assessment scope, types of activities observed, or types of objective evidence reviewed.
The following techniques can be used to facilitate assessment activities that involve remote techniques.
7.2.1 Establish Protocol for Remote Communication Establish and communicate how virtual meetings will be controlled to optimize communication and minimize distractions, such as two people talking at the same time. This typically involves application of tools available in the meeting application, such as requesting to speak in chat and raising virtual hands. The individual leading each session uses these techniques to ensure successful communication.
7-5
Planning for an Assessment that Uses Remote Techniques 7.2.2 Daily Team and Supplier Briefings Establish a communication plan for the remote assessment. This might include scheduled daily team and supplier briefings as well as notification when urgent meetings are required to address emerging concerns.
7.2.2 Pre-Job Briefings Hold pre-job briefings at the start of observations, interviews, and meetings involving remote techniques. These briefings should address:
Safety hazards and precautions, designated spotters, and so forth Primary and contingency equipment or technology to be used, including identification of the individuals responsible for operating the equipment Confirmation of internet signal, communications equipment, and document availability A review of contingency plans in the event that communication is lost or primary equipment fails If appropriate, plans for a trial run of technology 7.3 Documentation and Follow-Up Documentation and follow-up should be very similar to that of a traditional assessment.
However, the following actions will be necessary for some assessments involving remote assessment techniques.
7.3.1 Develop Restrictions That Should Be Applied to Each Procurement (Provisional Remote Assessments)
Develop language that should be included in future procurements to impose restrictions on the supplier that can be used to gain additional assurance that the technical and quality requirements of interest are successfully applied to each purchase.
7.3.2 Update Approved Supplier List/Action Tracking Programs Approved supplier lists should be updated to include the results of the assessment.
In the case of a provisional remote assessment, this includes documenting restrictions that should be imposed on each procurement during the provisional approval period, such as:
Enhanced receipt inspection Completion of remote source verification Documentation reviews In the case of provisional remote assessments, appropriate action-tracking tools should be used to ensure that a traditional assessment is performed as soon as practical.
7-6
8 REMOTE ASSESSMENT TOOLS/APPLICATIONS Appropriate remote assessment ICT should be selected to accomplish the objective of the assessment. In addition to audio or verbal communication tools and applications, many video-based products are available. The products reviewed during preparation of this report are either free or subscription-based, or they are charged per use based on increments of time.
Although the products identified in this section are representative of those available, many more exist. Mention in this report does not indicate endorsement or vetting of the products.
8.1 Verbal/Audio Communications When selecting primary and contingency audio communication tools, consider options that might not fail at the same time as video communication tools. For example, if there is a chance that internet bandwidth might cause poor quality or cessation of internet-based audio, a mobile phone, landline, or satellite phone might be a good choice for contingency or even concurrent communication during the activity. Audio communication tools include:
Satellite phones Mobile phones Internet-based applications, such as Skype, FaceTime, and WeChat Landline phones 8.2 Video/Visual Communications A wide variety of video communication tools and applications exist that can be up and running very quickly. When selecting a product, verify compatibility with the equipment that will be used by the assessor and the organization being assessed. Although standard office-variety tools might suffice for some activities, specialized remote assessment applications offer features that may enable a greater range of activities. Some of these features include:
Ability to accommodate multiple users at the same time.
Ability to display, share, and review documents on screen.
Ability to switch the source of video feed to any person connected.
Ability to capture photos or snapshots and mark them up while the verification is being performed.
Timestamping and geotagging of photos and videos, as illustrated by the location and date information at the bottom of the image in Figure 8-1.
Ability to capture, share, and download video of the assessment as is possible with cameras that can be worn (see Figure 8-2).
Smartphone cameras as the means of video communication.
8-1
Remote Assessment Tools/Applications 8.3 Sampling of Remote Assessment Tools/Applications Some of the products available when this report was developed are:
Avatour Auditboard Blitzz Facebook Portal FedRamp Google Meet GoToAssist Seeit Microsoft Teams Skype SnapInspect Zoom WebEx 8.4 Document Sharing A variety of applications specializing in content management can be used to securely manage sharing of documents. Some of these tools include:
Box.com Dropbox FTP sites Google Drive Microsoft OneDrive Microsoft SharePoint Microsoft Teams Realwear My Files, My Media, and My Camera WebEx Teams Wiki site Although some of these applications permit users to be assigned various levels of access, others include features to restrict how uploaded files can be used. For example, Microsoft SharePoint includes a feature that enables attachments to be viewed but not edited or saved.
8-4
9 REFERENCES
- 1. Remote Source Verification During a Pandemic or Similar State of Emergency: Screening Criteria and Process Guidance, 3002019436-A. EPRI, Palo Alto, CA: 2020. 3002019436-A.
- 2. Safety Evaluation Report, Safety evaluation by the Office of Nuclear Reactor Regulation, Request for change to the Columbia Operational quality assurance program description.
Docket No. 50-397, ADAMS accession no. ML20203K876. U.S. Nuclear Regulatory Commission, Washington, D.C.: July 2020.
- 3. Topical Report Safety Evaluation EPRI 3002019436, Remote Source Verification During a Pandemic or Similar State of Emergency: Screening Criteria and Process Guidance, ADAMS accession no. ML20244A017. U.S. Nuclear Regulatory Commission, Washington, D.C.: September 2020.
- 4. U.S. Code of Federal Regulations, Title 10, Chapter 1, Part 21, Reporting of Defects and Noncompliance. Office of the Federal Register, National Archives and Records Administration, U.S. Government Printing Office, Washington, D.C.
- 5. Administrative Controls and Quality Assurance for the Operational Phase of Nuclear Power Plants. ANSI N18.7/ANS 3.2. American National Standards Institute, Washington, D.C.:
1976.
- 6. Quality Assurance Program Requirements for Nuclear Facilities. ANSI/ASME N45.2-1977.
American Society of Mechanical Engineers, New York, NY: 1977.
- 7. Quality Assurance Requirements Terms and Definitions. ANSI N45.2.10. American National Standards Institute, Washington, D.C.: 1973.
- 8. Quality Assurance Terms and Definitions. Regulatory Guide 1.74. U.S. Nuclear Regulatory Commission, Washington, D.C.: February 1974.
- 9. Quality Assurance Requirements for Control of Procurement of Items and Services for Nuclear Power Plants. ANSI N45.2.13. American National Standards Institute, Washington, D.C.: 1976.
- 10. Inspection of Commercial-Grade Dedication Programs. NRC Inspection Manual, Inspection Procedure 43004. U.S. Nuclear Regulatory Commission, Government Printing Office, Washington, D.C.: January 2017.
- 11. Quality Assurance Requirements for Nuclear Facility Applications. ASME NQA-1-1994.
American Society of Mechanical Engineers, New York, NY: 1994.
- 12. Quality Assurance Requirements for Nuclear Facility Applications (QA). ASME NQA-1a-2009 (addenda). American Society of Mechanical Engineers, New York, NY: 2009.
9-1
References
- 13. Quality Assurance Program Criteria (Design and Construction). Regulatory Guide 1.28, Revision 5. U.S. Nuclear Regulatory Commission, Government Printing Office, Washington, D.C.: October 2017.
- 14. Quality Assurance Requirements for Nuclear Facility Applications. ASME NQA-1-2015.
American Society of Mechanical Engineers, New York, NY: 2015.
- 15. Quality Assurance Requirements for Nuclear Facility Applications. ASME NQA-1-2019.
American Society of Mechanical Engineers, New York, NY: 2019.
- 16. Commercial Grade Dedication Application Handbook. DOE-HDBK-1230-2019, DOE Handbook. U.S. Department of Energy, Washington, D.C.: July 2019.
- 17. U.S. Code of Federal Regulations, Title 10, Chapter 1, Part 21, Reporting of Defects and Noncompliance. Office of the Federal Register, National Archives and Records Administration, U.S. Government Printing Office, Washington, D.C.
- 18. Generation Maintenance Applications Center: Foreign Material Exclusion Guidelines.
EPRI, Palo Alto, CA: 2010. 1019805.
- 19. 2019 ASME Boiler and Pressure Vessel Code,Section V, Nondestructive Examination.
American Society of Mechanical Engineers, New York, NY: 2019.
- 20. Guidelines for Performance-Based Supplier Audits (NCIG-16). EPRI, Palo Alto, CA: 1990.
NP-6630.
- 21. Nuclear Procurement Program Improvements. NUMARC 90-13. Nuclear Utility Management and Resources Council, Washington, D.C.: 1990.
- 22. Safety Evaluation Report, Safety Evaluation by the Office of Nuclear Reactor Regulation, Change to the Operating Quality Assurance Manual, Revision 34b Union Electric Company, Callaway Plant, Unit No. 1. Docket nos. 50-483 and 72-1045, ADAMS accession no.
ML20216A681. U.S. Nuclear Regulatory Commission, Washington, D.C.: August 2020.
- 23. Plant Support Engineering: Counterfeit and Fraudulent Items. EPRI, Palo Alto, CA: 2010.
1021493.
- 24. CANDU Industry Assessment Committee (CANIAC) Audit Program Guideline: Remote Supplier Audits. CANDU Owners Group, Incorporated, Toronto, Canada: August 2020.
- 25. Plant Engineering: Guideline for the Acceptance of Commercial-Grade Items in Nuclear Safety-Related Applications: Revision 1 to EPRI NP-5652 and TR-102260. EPRI, Palo Alto, CA: 2014. 3002002982.
- 26. Quality Assurance Program Requirements for the Supply of Items and Services for Nuclear Power Plants, Category 1. CSA N299.1. CSA Group, Toronto, Ontario, Canada: 2016.
- 27. Guidelines for Auditing Management Systems. ISO 19011:2018. International Standards Association, Geneva, Switzerland: 2018.
- 28. Mandatory Document for the Use of Information and Communication Technology (ICT) for Auditing/Assessment Purposes. IAF MD 4:2018, Issue 2. International Accreditation Forum, Incorporated, Quebec, Canada: July 2019.
9-2
A CURRENT APPLICATION AND EXPERIENCE A.1 Remote Assessment Experience Summaries A.1.1 Limited Scope Audit of a Third-Party Qualifier by a LicenseeSupplier Perspective Type and Description of Assessment The assessment was a fully remote, limited-scope audit performed by a utility at a supplier that is a third-party qualifier. The scope of the audit was limited to current civil/structural safety-related orders (for example, concrete, soil, and grout). The audit took place over a two-week period. The licensee audit team included three members. A team of five supplier employees supported the audit. This summary includes perspectives of the entity being assessed.
Remote Techniques Used The entity being audited created a private shared folder on their network that was accessible to all members of the audit team and shared the folder with the team about a week prior to the entrance meeting. The folder contained a list of all quality control procedures and instructions as well as the suppliers quality assurance manual. The licensee reviewed this information and requested additional procedures and several examples of concrete/grout/soil orders, personnel training records, corrective action reports, M&TE records, and several other documents to be added to the shared folder.
An open phone line was set up for the audit entrance, daily briefs, and exit meetings.
Daily briefings were held each day of the two-week audit. The audit team leader emailed a list of questions that would be discussed during the daily briefs to the suppliers quality manager ahead of time so that the supplier could review the questions and be ready with answers during the daily brief.
The audit team used email to request additional documents throughout the audit. Questions relating to reviewed documentation and procedures were answered by email, over the phone, or during the daily briefs.
Tools Used DropBox was used for secure online content management to share documents, procedures, and records.
An open phone line was used for audit entrance, daily briefs, and exit meetings.
A-1
Current Application and Experience Lessons Learned Additional time was required to complete assessment planning.
The audit took place over a two-week period, longer than an in-person audit would have taken.
Time was needed for the back-and-forth of daily questions. The lead auditor would email questions (for audit record purposes) to the quality assurance manager. These were then reviewed and answered during the daily briefings.
Limitations Auditors were not able to physically see the suppliers facility or talk to lab technicians.
A.1.2 Limited Scope AuditAssessment Team Perspective Type and Description of Assessment The assessment was a limited-scope audit of a supplier by a NUPIC assessment team; originally, the assessment had been planned as an in-person, on-site audit. The audit was conducted as a hybrid assessment, with the audit team leader present on-site and in-person. The audit team leader performed verifications to support team members working remotely. This summary includes perspectives of the assessment team.
Remote Techniques Used The supplier was very adept at coordinating virtual team meetings using experience gained from operating their day-to-day business since the outset of the pandemic.
The supplier was very diligent in maintaining proper safety protocols, such as requiring masks and social distance spacing.
The supplier provided excellent support responding to documentation requests, answering questions, and making appropriate employees available for interviews/discussions electronically.
Team meetings held throughout the day, text messages, emails, and regular phone calls made communication seamless.
Lessons Learned A team effort was required to complete most sections of the checklist. Remote auditors could answer programmatic questions, but the on-site auditor reviewed most of the objective evidence, such as purchase orders and dedication plans.
Rigid implementation of proper safety protocols and concern for safety at the suppliers facility were reassuring to supplier and assessment team personnel.
The supplier was very receptive to the hybrid approach and did everything possible to make it successful.
Although coordination was required, the available technology enabled the assessment to proceed with no disruption in discussions between the supplier and assessment team.
A-2
Current Application and Experience The absence of conference room chatter that goes on during many audits made it easier to stay focused with minimal distractions. However, this also reduced team discussion synergies, which is noted as a limitation. You must manage the positive against the negative.
Due to the limited scope (four checklist sections, three auditors, three days), the work scope was not a challenge and allowed enough time for a satisfactory audit.
Limitations An audit that has auditors working remotely might not be as intrusive and thorough as one with all members at the facility.
Open dialog between team members was limited, which impacted the synergy resulting from open discussion and exchange of information with less experienced auditors.
The audit team leader was limited in which audit checklist sections could be assigned to remote auditors because the supplier was not willing to share proprietary documents electronically with remote auditors.
The audit team leader spent a large amount of time following up on previous findings and deficiencies.
A.1.3 Full Triennial AuditAssessment Team Perspective Type and Description of Assessment The assessment was a full (triennial) audit of an engineering services company by a NUPIC assessment team. The audit was originally planned as an in-person, on-site audit. Due to pandemic conditions, the supplier advised that they could permit only two auditors to be present on-site. The audit was conducted as a hybrid assessment, with the audit team leader and technical specialist present on-site and in-person. The audit team leader performed verifications to support team members working remotely. This summary includes perspectives of assessment team members.
Remote Techniques Used Web-based meetings twice a day using the suppliers conferencing tool.
The supplier assigned two designated supplier representatives as the sole points of contact for requests by remote auditors during conduct of the audit.
Lessons Learned The expectation that supplier representatives communicate openly with auditors needs to be made clear by the audit team leader and supplier management at the start of the assessment.
Remote auditors should be able to freely contact supplier representatives during the audit to request documents, ask questions, or locate needed information.
It is difficult to assess some of the suppliers responses without the benefit of observing body language and facial expressions.
A-3
Current Application and Experience When auditing engineering activities, it is imperative that engineering documents can be shared in a way that makes them legible to auditors and permits auditors to review the documents at their own pace.
Establish a way to organize documents being shared remotely early in the auditperhaps in clearly labeled subfoldersand request the supplier to provide a cross-referenced list of file names to contents when file-naming conventions do not clearly identify the contents to external personnel.
The audit team should plan, communicate, and cooperate enough to avoid two different remote auditors requesting the same documents or information from the supplier.
Limitations The suppliers management limited remote auditors to routing all requests through one of two supplier contacts, which resulted in frustration because one of the two designated contacts was frequently unavailable and neither contact had the experience or knowledge to provide timely, adequate responses.
It can be difficult to view documents on a laptop screen when multiple applications are being usedfor example, a document opened by a supplier in a document management system window being shared with auditors in a Microsoft Teams window. In some cases, documents were not legible.
The initial setup for access to another organizations Web conferencing tools can be difficult, depending on the company firewall; it can also require multiple logons each day.
Review of certain documents that could not be added to the document-sharing application or emailed to an auditor required the remote auditor to have a video meeting with the supplier to share the document, which was not conducive to a thorough review.
It was difficult and time-consuming for the auditors to locate information and documents needed because the documents shared with auditors were not clearly organized in the shared location.
Supplier personnel did not seem to be interested in supporting the remote audit activities.
Several instances were noted where the supplier received duplicate requests (the same request for information from two different remote auditors).
Audit team collaboration was limited, and time zone differences required some remote auditors to work outside of their normal business hours.
A.1.4 Full Triennial AuditAssessment Team Perspective Type and Description of Assessment The assessment was a full (triennial) audit of a supplier by a NUPIC audit team. The audit was conducted as a hybrid assessment, with some of the auditors present on-site and in-person. This summary includes perspectives of assessment team members.
A-4
Current Application and Experience Remote Techniques Used The supplier shared documents on their SharePoint site.
The supplier made the documentation available to the team before the assessment, which enabled some auditors to prepare in advance of the audit.
Microsoft Teams was used for meetings and interviews.
Lessons Learned Maintaining an open line (Microsoft Teams) all day that auditors could use to ask questions was effective.
ICT was successful in bringing the on-site and remote auditors together.
All of the originally assigned utility auditors were able to participate in the hybrid audit.
Remote and on-site members of the audit team were able to work well together to support activities that needed to be verified at the suppliers facility.
It might be useful to establish a white board document in a location readily available to all audit team members.
The audit team should test primary and contingency ICT equipment and applications in advance of the audit.
Limitations Communication had to be coordinated with an audit team member at a different supplier location.
Issues could not be tracked as usual on a white board.
It was more difficult for audit team members to interact with the technical specialist.
It was difficult to achieve the team dialogue that typically occurs during the course of the audit.
The camera feature in Microsoft Teams was not working for everyone on the audit team.
A.1.5 Full Triennial Audit: Assessment Team Perspective Type and Description of Assessment The assessment was a full (triennial) audit of a supplier of a wide range of mechanical devices.
Due to pandemic conditions, the supplier advised that they were limiting the auditors to be present on-site. The audit was conducted as a hybrid assessment, with the audit team leader, technical specialist, and two other auditors on-site and four auditors working remotely.
Remote Techniques Used The suppler established two secured FTP sites. One was loaded with quality program manual and procedures, and the other one was loaded with the documents requested by the audit team leader during audit planning and was used during the audit to provide documents requested by audit team members. All auditors had access to the password-protected FTP sites.
A-5
Current Application and Experience The supplier established points of contacts for all audit team members based on the assigned checklist section and set up a virtual Microsoft Teams meeting schedule for the audit week.
The suppliers conferencing equipment enabled excellent audio and visual quality.
Lessons Learned Start preparation as soon as possible (two or three weeks prior to the audit).
Virtual meetings and interviews required more time than planned, so scheduled durations may need to be longer than normal for virtual meetings.
The supplier was willing to share all needed documents and agreed to leave the FTP site online until the audit report is issued.
Designate a contact, such as the supplier quality assurance manager, to handle requests for documents.
If possible, review documents and request follow-up documents from the suppliers quality assurance manager in advance. Once the audit starts, the quality assurance manager will be extremely busy. (This is if the manager can share documents prior to the audit. Some documents are proprietary.)
Limitations Communication with remote auditors was not optimal.
The on-site audit team leader had to handle a lot of tasks that could not be performed by the remote auditors.
When using Microsoft Teams for interviews, only one person at a time could share the screen, so it was difficult to share notes.
The response time to requests for documents was slow due to the large number of documents requested.
Some electronic record packages provided did not provide complete information, and response to follow-up requests was slower than anticipated.
Reviewing documents online proved to be challenging.
The audit team leaders ability to provide oversight was limited by the number of activities being performed for remote auditors.
The large number of invites for Web meetings caused some confusion; as a result, some auditors and supplier personnel missed scheduled meetings.
Although setting up meetings for planning purposes was effective, there were no provisions for accommodating an auditor who was ready to start sooner than the scheduled time.
Supplier personnel seemed uncomfortable sharing images of documents in use at workstations in the field, particularly with auditors from organizations in a different country than the supplier.
Interaction with other auditors was missed.
Some remote auditors were challenged by requests to do non-audit-related work and ICT problems that would not be a factor for an on-site auditor.
A-6
Current Application and Experience A.2 General Experience with Remote Assessment A.2.1 CANDU Procurement Audit Committee The CANDU Procurement Audit Committee (CANPAC) has established a guideline that applies when audits cannot be performed at the suppliers facility due to exigent conditions. The report discusses a phased approach in conducting hybrid audits. Hybrid audits at CANPAC consist of the initial remote audit (herein, the Phase 1 audit), followed by the performance-based on-site audit (herein known as Phase II audit). This hybrid approach is invoked only when the utilities are soon approaching the allowable nine-month supplier expiry date and require some form of audit/assessment of these suppliers in order to keep them active in their ASL with appropriate restrictions. Approximately three months before the nine-month allowable expiry date, a Phase I audit would be planned and conducted. The Phase I audits are 100% remote.
The Phase I review of documentation and objective evidence may include any or a combination of the following audit techniques:
Review of emailed documentation provided by supplier Remote/virtual interviews Review of objective evidence by videoconference Any elements of the standard that require further on-site verification during the Phase II audit are clearly outlined in the Phase I audit summary. Upon completion of the Phase I audit, the audit report and audit checklists are submitted to the utilities for their acceptance. The completion of the Phase I audit allows the utilities to extend their supplier expiry dates with any applicable restrictions necessary until the Phase II can be undertaken.
When travel restrictions are lifted and on-site audits can be resumed safely, the Phase II audits are scheduled, and, typically, the same audit team will be assigned to finalize the audit and verify any areas outlined in the Phase I report that require further verification.
Regardless of when the actual remote Phase I audit or Phase II audit takes place (months after the original expiry), their new expiry date will be three years from the original expiry date.
A.2.2 CANDU Industry Assessment Committee The CANDU Owners Groups (COGs) Industry Assessment Committee (CANIAC) joint audit program is establishing a guideline that applies when audits cannot be performed at the suppliers facility due to exigent conditions. The draft guideline addresses on-site audits, remote audits, and combination audits.
The guideline includes a checklist for remote assessments that captures the method of assessment for individual quality requirements included in Canadian Standards Association Quality Assurance Program Requirements for the Supply of Items and Services for Nuclear Power Plants
[26]. Part A of the checklist lists the options to consider for each quality standard clause, including document review, interviews, and direct observations. Figure A-1 shows a partial section of the checklists Part A.
A-7
Current Application and Experience Lessons Learned It is very important to maintain regular remote interfaces during all major audit phases; this includes audit planning, scope development, pre-audit briefing, physical check/verification of sample activities of records, and audit report review.
The hybrid audit emphasis is on obtaining documentation in advance and performing document review prior to conducting the on-site verification.
All interfaces between the audit team and auditee, including communication, decisions, and action agreed to during the audit, should be documented in writing.
The audit team leader should ensure that the team maintains a good balance of remote review and physical verification to provide assurance that the quality assurance program controls are adequately established and effectively implemented.
A.2.4 Standard Discussion on Remote Assessment and Audit Annex A of ISO 19011:2018, Guidelines for Auditing Management Systems [27], includes discussion on virtual audits since the 2011 revision. In addition, IAF MD 4:2018, IAF Mandatory Document for the Use of Information and Communication Technology (ICT) for Auditing/Assessment Purposes [28], includes discussion on the importance of data protection measures and other considerations when using ICT to conduct remote assessments and audits.
A.2.5 Nuclear Industry Evaluation Program This section summarizes experience and lessons learned by the nuclear industry evaluation program organization during peer assessments of licensee programs that can be applied to any type of remote assessment.
Lessons Learned The assessment team manager must determine the remote meeting platform to be used and communicate this choice early. This allows assessment team members and personnel from the entity being assessed to become familiar with the platform before the assessment.
Start preparations early, and complete as much advance work and review as possible prior to conduct of the assessment.
Although no major changes were necessary to the typical preparation schedule, additional work was necessary to meet schedule milestones.
Although internet-based teleconferencing tools may be familiar to some assessment team members and supplier personnel, others may have never used such tools and will need time and assistance in becoming familiar with the functionality.
Different organizations have elected to use different Web-based teleconferencing tools.
Major platforms used so far include WebEx, Microsoft Teams, and Skype. These tools generally allow for participation by guests. However, guests may not be able to access certain features, such as file sharing. It is important to test software and features that will be used in advance.
A-10
Current Application and Experience Establish a communications plan and protocol early, such as one or more daily status updates and providing an open questions/issue list to the primary contact at the organization being assessed after the last daily status update.
The WebEx virtual meeting tool (other software may be similar) allows the host to establish a private virtual room that can be left open during working hours as long as the host is present. This type of feature permits the host to provide access information to all team members so that any team member could join the room and speak to the host at any time to ask questions or address concerns.
It may be beneficial for assessment teams to use collaborative tools (such as Microsoft Office 360 or Box) that allow all team members to concurrently access and modify a single document in real time. This could be used to maintain a common white board or issues list and to promote timely update of assessment checklists.
Applications such as WebEx can be loaded on smartphones and used to record and transmit video of production activities while maintaining audio communication between the assessor and individual performing activities. When using mobile phones, consider availability of a backup battery.
Use calendar invites to schedule status update meetings so that clear expectations and deadlines are set and known by all team members and the organization being assessed.
Encourage all assessment team members and supplier staff supporting the assessment to commit as much time as possible to the assessment each day. The risk of distractions impacting assessment schedule are greater when assessors are located remotely.
Consider scheduling additional time for exit briefings due to the nature of telecommunications and remote presentation.
Effective three-way communication is essential during a remote assessment. Confirm that the entity being assessed understands what is needed at the end of each conversation. An example of three-way communication follows:
- 1. The assessment team leader communicates during a daily briefing what actions are required of the supplier to move forward with the assessment.
- 2. The lead supplier representative acknowledges the communication by repeating the essence of the request or required action. If clarification is appropriate, the assessment team leader provides it and asks the supplier representative to repeat the essence of the request as clarified.
- 3. The assessment team leader acknowledges the supplier representatives reply to confirm that the request is properly understood. If the supplier then has questions, the process can be repeated until the request is mutually understood.
Limitations Loss of ad-hoc communication between assessment team members in the conference room, hallway, and so forth.
Confusion resulting from differing time zones when scheduling and conducting meetings, briefings, and interviews.
A-11
Current Application and Experience Interviews are less insightful due to reduced ability to observe body language.
Increased opportunity for assessment team members working remotely to be distracted.
A-12
B REMOTE ASSESSMENT PLANNING WORKSHEET The remote assessment planning worksheet can be used in conjunction with Section 6, Steps 6.7.1-6.7.6, to determine which of the following three assessment options is appropriate based on the technical and quality requirements of interest:
Hybrid assessment (one or more assessment team members are on-site, in-person)
Provisional remote assessment (no assessment team members are on-site, in-person)
Fully remote assessment B.1 Instructions for Completing the Remote Assessment Planning Worksheet The numbers in the following steps correspond with the numbers indicated in parenthesis below the headings on the remote assessment planning worksheet shown in Table B-1:
- 1. Enter a number for each technical or quality requirement of interest.
- 2. Enter a brief description of the technical or quality requirement interest (for audits) or critical characteristic (for commercial-grade surveys).
- 3. Enter the activity or activities used to assess the technical or quality requirement of interest or critical characteristic.
- 4. If remote assessment techniques can be used to obtain similar results, enter YES. If remote assessment techniques cannot provide similar results, enter NO.
- 5. If a YES is entered in the previous step, enter the type of remote assessment technique that will be used. If NO is entered, leave blank.
- 6. If on-site presence is necessary, enter YES; if not, enter NO.
- 7. Add rows (and complete Steps 1-6) as necessary to accommodate all technical and quality requirements of interest.
- 8. Select the type of assessment to pursue:
- Mark the Hybrid Assessment box if remote assessment techniques alone cannot provide the necessary insight for performance-based assessment of the technical and quality requirements of interest and at least one team member can be present in-person, on-site.
- Mark the Provisional Remote Assessment box if remote assessment techniques alone cannot provide the necessary insight for performance-based assessment of technical and quality requirements of interest or critical characteristics and it is not possible for any team members to be on-site, in-person.
- Mark the Fully Remote Assessment box if remote assessment techniques alone can provide all necessary insight for performance-based assessment of technical and quality requirements of interest or critical characteristics (no on-site necessary items indicated in column 6).
B-1
Remote Assessment Planning Worksheet
- 9. Document a basis for the type of assessment selected.
B-2
Table B-1 Remote assessment planning worksheet No. Activity Used to Assess Can Remote Remote Assessment Technique to Apply On-Site Technical/Quality Assessment Necessary?
Requirement of Interest or (more than one can be Techniques Provide Critical Characteristic entered) (YES/NO)
Insight? (YES/NO)
(1) (2) (3) (5) (6)
(4)
Type of Assessment to Pursue (8)
Hybrid Assessment Remote assessment techniques alone cannot provide the necessary insight for performance-based assessment of technical and quality requirements of interest or critical characteristics.
At least one team member can be present in-person, on-site.
Provisional Remote Assessment Remote assessment techniques alone cannot provide the necessary insight for performance-based assessment of technical and quality requirements of interest or critical characteristics.
It is not possible for any team members to be in-person, on-site.
Fully Remote Assessment Remote assessment techniques alone can provide all necessary insight for performance-based assessment of technical and quality requirements of interest or critical characteristics (no on-site necessary items indicated in B-3
C PERFORMANCE-BASED ASSESSMENT AND CRITICAL CHARACTERISTIC WORKSHEETS Performance-based supplier assessment and critical characteristic worksheets, such as the ones included in this appendix, are useful in identifying the technical and quality requirements of interest and the related activities affecting quality.
C.1 Instructions for Completing the Performance-Based Assessment Planning Worksheet The numbers in the following steps correspond with the numbers indicated in parenthesis below the headings on the performance-based supplier assessment worksheet shown in Table C-1:
- 1. Enter the items and/or services included in the scope of supply to be assessed.
- 2. Enter a brief description of the technical or quality requirement interest. These are typically established through engineering input and are the most important requirements.
- 3. Enter the suppliers processes, activities, or methods of control used to ensure that technical and quality requirements of interest are met. This step may require communication with the supplier in advance of the assessment to determine which specific processes require on-site, in-person presence. When performing a traditional assessment, identification of categories of controls (for example, inspection or testing) may be sufficient. When considering remote assessment, it may be necessary to identify the specific types of testing and inspection, such as tensile strength, hydrostatic/pneumatic test, performance test, and insulation resistance test, to determine if and how observation of the controls could be achieved using remote techniques.
- 4. Enter a brief description of the applicable acceptance method or criteria.
- 5. Identify the technique(s) that will be used to assess the suppliers process, activity, or method of controlfor example, observation of in-process work, observation of inspection and testing, demonstration of procedure implementation, facility walkdown, documentation review, interview, or another technique including remote assessment techniques.
- 6. During or after the assessment, indicate the results of the assessment (for example, satisfactory or unsatisfactory).
C-1
Table C-1 Performance-based supplier assessment worksheet Items in Scope of Supply:
(1)
Technical Characteristics and Supplier Processes, Activities, Acceptance Method/Criteria Verification Technique Results Other Items of Interest (Essential Methods of Control to Be to Fit, Form, Function) Evaluated (2) (3) (4) (5) (6)
C-2
C.2 Instructions for Completing the Critical Characteristic Worksheet The numbers in the following steps correspond with the numbers indicated in parenthesis below the headings on the critical characteristics worksheet shown in Table C-2:
- 1. Enter the items and/or services included in the scope of supply to be assessed.
- 2. Enter the critical characteristics to be verified. These are typically established in a commercial-grade dedication technical evaluation prior to the commercial-grade survey and are specific to the item(s) that will be dedicated.
- 3. Enter the suppliers processes, activities, or methods of control for the critical characteristic.
- 4. Enter a brief description of the applicable acceptance criteria. Acceptance criteria are typically established in a commercial-grade dedication technical evaluation prior to the commercial-grade survey and are specific to the item(s) that will be dedicated.
- 5. Identify the technique(s) that will be used to assess the suppliers process, activity, or method of controlfor example, observation of in-process work, observation of inspection and testing, demonstration of procedure implementation, facility walkdown, documentation review, interview, or another technique including remote assessment techniques.
- 6. During or after the assessment, indicate the results of the assessment (for example, satisfactory or unsatisfactory). Documented controls associated with each activitys critical characteristic should also be recorded.
C-3
Table C-2 Critical characteristics worksheet Items in Scope of Commercial-Grade Survey:
(1)
Critical Characteristics to Be Supplier Processes, Activities, Acceptance Criteria Verification Technique Results Verified Methods of Control to Be Evaluated (2) (3) (4) (5) (6)
C-4
D PRE-ASSESSMENT CHECKLISTS FOR REMOTE ASSESSMENTS This appendix includes two checklists intended to be used prior to the assessment to ensure that the necessary preparations are made with the assessment team and the entity being assessed.
D.1 Instructions for Completing the Pre-Assessment Checklists Pre-assessment planning worksheets (see Table D-1) include considerations for the assessment team as well as considerations for the entity being assessed.
If the response to any of the checklist items cannot be confirmed, additional coordination and planning may be required to ensure successful completion of the assessment.
An assessment using remote techniques may not be possible if the entity being assessed is not willing to agree to the checklist items included in the pre-assessment screening checklist for entity being assessed (see Table D-2).
Table D-1 Pre-assessment planning checklist for assessment team No. Criteria Access Requirements
- 1. Access requirements imposed by the entity being assessed, such as daily health screening measures or evidence of vaccination or testing, have been identified and agreed to by team members.
Assessment Team Composition
- 2. If practical, at least one member of the team is familiar with has previously been to the facility).
Ideally, the assessment team leader and technical specialist will have visited the facility being assessed in the past and be familiar with the products and processes of the entity being assessed. As an alternative, the assessment team leader could consult with the assessment team leader(s) and technical specialist(s) from previous assessments.
- 3. It is preferable that one or more members of the assessment team have previous 4.
assessed or the types of products and services provided have been identified.
Assessment team members have reviewed the most recent previous assessment, summary.
D-1
Pre-Assessment Checklists for Remote Assessments Table D-1 (continued)
Pre-assessment planning checklist for assessment team No. Criteria Information and Communications Technology (ICT)
ICT has been arranged, acquired, set up, and tested. ICT includes the remote and local computers, cameras, and microphones as necessary for real-time, two-way communication between remote and on-site assessors and remote and on-site representatives from the entity being assessed.
- 7. Contingency plans to address loss or deterioration of audio or video signals or failure of primary ICT have been developed and communicated to assessment team members and the entity being assessed.
- 8. Determine how two-way communication will be enabled between team members and representatives of the entity being assessed during observation of production by processes and use of ICT. Headphones or an equivalent might be needed.
- 9. Trial run of primary and contingency equipment is completed before the assessment starts.
Assessment Team Preparation
- 10. A method for timely notification to the assessment team leader of issues identified during assessment or has been established and communicated to team members.
- 11. Differences in time remote assessors have confirmed that calendar invites are at the correct local time.
- 12. For international engagements, differences in languages and arrangements tor translation have been made (as necessary). Translators are trained in use of ICT.
13.
in front of the camera.
- 14. Assessment team members have successfully participated in dry runs to confirm successful operation and use of primary and contingency ICT to be used by the assessment team.
A successful trial run of primary and contingency ICT has been completed prior to start of the assessment. If necessary (due to bandwidth, signal strength, and so on),
lower-resolution video and audio have been considered.
Identification of Remote Assessment Activities Advance identification of interview schedules and locations so that appropriate equipment and internet access can be in place in advance of the assessment.
- 17. Advance identification by the assessment team of activities that need to be observed and the specific attributes of the activities that should be visible. Communicate the observation needs to the entity being assessed to allow them to determine if they can adequately support the needs.
D-2
Pre-Assessment Checklists for Remote Assessments Table D-2 Pre-assessment screening checklist for entity being assessed No. Criteria Access RequirementsFacility
- 1. Access requirements, such as daily health screening measures and evidence of vaccination or testing, have been discussed with the entity being assessed.
- 2. The entity being assessed is willing to allow observation of any activities on shop floor as selected by the assessor.
- 3. The entity being assessed is willing to allow remote interviews and interactions with personnel, including shop personnel as necessary.
- 4. The entity being assessed understands the types of assessment activities that will be remote and is willing to provide the necessary time and support to allow for a thorough and efficient assessment.
When different regions or languages are involved, the camera operators language proficiency in understanding requests and instructions should be considered.
- 7. For international engagements, differences in languages and arrangements tor translation have been made (as necessary). Translators are trained in use of ICT.
- 8. Entity being assessed has video and audio capabilities to allow observation on the assessment team.
Access RequirementsDocumentation
- 9. The entity being assessed is willing to share necessary documents with remote assessors through a content-sharing application.
- 10. Requirements (as applicable) for nondisclosure agreements have been discussed start of assessment.
- 11. The entity being assessed agrees to promptly provide the assessment team with documentation requested during the assessment to provide objective evidence of quality assurance program implementation.
- 12. The assessment team will have access to the quality assurance manual, second-level procedures, and other necessary documents to enable timely review and reference as required during the assessment.
Entity Being AssessedPersonnel
- 13. Personnel from the entity being assessed are willing to appear on camera for interviews and observations.
D-3
Pre-Assessment Checklists for Remote Assessments Table D-2 (continued)
Pre-assessment screening checklist for entity being assessed No. Criteria Information and Communications Technology (ICT)
- 14. The entity being assessed is able and willing to support remote, real-time observation of the performance of activities that are required to be observed as to provide objective evidence of the quality assurance program implementation.
The entity being assessed is able and willing to provide or permit use of the ICT needed to provide sufficient audio and visual clarity and mobility.
The entity being assessed is aware that activities to be observed may include welding, inspection, machining, materials processing, and testing. In addition, that observation may include real-
- 17. The entity being assessed has ICT sufficient to enable video and audio observation on the shop floor or is willing to use ICT provided by the assessment team.
- 18. The entity being assessed will support remote, real-time observation of the performance of activities that are required to be observed, and it will follow the assessors directions related to what needs to be in the field of vision, camera movement, and so forth.
- 19. Determine if interference from equipment nearby that could challenge clarity of necessary audio communications and establish measures to mitigate or eliminate Entity Being Assessed Preparations 20.
confirmed that calendar invites are at the correct local time.
- 21. Scheduled times for daily briefings with appropriate supplier management have been agreed to and scheduled.
- 22. The ICT to be used for daily briefings, interviews, and observations of production activities has been identified, agreed to, and tested.
- 23. The supplier had confirmed that production activities to be assessed will be ongoing during the assessment period.
D-4
E FREQUENTLY ASKED QUESTIONS The following questions were encountered by individuals contributing to this report during its development. Reference to the location of answers in this report is included for each question as applicable.
- 1. Does this guidance apply only to assessments of suppliers, or can it also be used for internal assessments?
See Section 1.3.
- 2. What is the difference between an assessment and a source verification?
See Section 1.5.
- 3. What is an example of a regulatory or quality assurance standard perspective on remote assessments?
See Section 1.6.
- 4. Are remote auditing techniques directly addressed by NQA-1?
Not in or before NQA-1-2019 [15].
- 5. How can access to view propriety information outside of a suppliers facility be arranged to support remote assessment?
See Section 4.1.
- 6. How does a traditional performance-based assessment differ from an assessment involving remote techniques?
See Section 5 and Section 6.
- 7. What is the difference between hybrid, fully remote, and provisional remote assessments?
See Section 6 and Table 6-1.
- 8. What types of information and communication technology are typically employed to enable remote assessment activities?
See Section 8 of this report.
E-1
Frequently Asked Questions
- 9. When is use of remote assessment techniques possible?
See Section 1.4, Section 6.9.1, Section 6.13.1, and Section 7.1.7.1.
- 10. Can participation in remote or hybrid assessments be used as a basis for certifying an assessment team member as an assessment team leader or lead auditor?
The documents reviewed for this report do not address remote or hybrid assessments as a basis for certification.
E-2