Text

NRC-2019-000317 - Resp 1 - Interim, Agency Records Subject to the Request Are Enclosed
	ML21210A145
Person / Time
Issue date:	07/26/2021
From:	; NRC/OCIO
To:	;
Shared Package
ML21210A143	List: ML21210A144; ML21210A145;
References
	FOIA, NRC-2019-000317
	Download: ML21210A145 (273)
	v • d • e

From: w....fm To: Cubellis Louis Subject : Re: Recommendations RE: Sandia Draft Outline for VA Tool NUREG Date: Thursday, February 21, 2019 12:05:20 PM Thanks. I will put together what you provided and my comments. I have not had a ch ance to look at it yet.

From: Cu bellis, Louis Sent: Thursday, February 21, 2019 7:41 AM To: Lee, Pete

Subject:

Recommendat ions RE: Sandia Draft Outline for VA Tool NUREG Good morning, Pete, I had a couple of hours this morning to review and comment on Sandia's draft outline for the NUREG. The outline includes widely accepted VA methodologies and considerations, and the sections appeared to me to correspond to the task order requirements. For example, requirement "a" is Section 1.3, requirement "b" is Section 2, requirement "c" is Section 3, etc. It seems like Sandia has begun pulling together information for all of the tasks, except for "j"; there's just a placeholder consisting of highlighted text from the task order in Section 10.

Per your request, I recommend you consider including the following points before you provide feedback to NRO and Sandia. I put the task order requirements in brackets at the end of each comment.

Section 1.3: Add as references [task a]:

o June 2018 NNSA Supplemental Directive 470.4-2 o January 2018 DoE validation and verificat ion report tor AVERT o December 2011 DoD validation and verification report tor AVERT

Sections 7 and 1O: Add information on using visual depictions of response force fields of tire (aka heat maps) to help validate the tools' pathway analysis results.

For example, I was able to detect a programming limitation in one VA tool by reviewing heat maps generated by the tool and then comparing the pathway analysis results to the vulnerabilities I identified in the maps. When none of the 1,000 computer-generated paths corresponded to a route the heat map indicated had nearly a 90% adversary success rate, I was able to ask why. I consulted the lead programmer for the company that produced the tool. He was able to take what I discovered, do some research, and then tell me that except tor avoid detection, the tool did not consider facility access levels for adversary path strategies. This can cause the tool to make adversaries enter detection zones sooner or remain there longer than expected, which can prevent the adversaries from exploiting the most vulnerable pathways (MVPs) or give guard forces significantly longer periods to interrupt them en route to the MVPs. In the 1,000 runs we ran for the demo, the

adversaries immediately entered detection zones outside the protected area and then tried to walk as many as 200 meters along the outer edge of the detection zone before breaching the protected area fence. This resulted in the guard force having the maximum interdiction time possible and none of the adversaries made it to the start of the MVPs. [g]

Sections 5, 8, and 10: In addition to pathway analyses, it seems like it would be helpful for a VA analyst to know that workarounds may also be necessary to ameliorate unrealistic response force or adversary capabilities provided by a VA tool (e.g., all-knowing response force, perfect response force communication at all times, lack of adversary coordination, etc.). For example, some tools give response force members 360-degree hearing and eyesight to infinite distances. One way to counteract that would be to put relevant response force members into 'closets', and then use a 'respond to' or similar command to cause the members to exit the closets after someone or something else detects the adversary. [d, e, f, h, i, j]

Section 8: The NUREG should include some guidance for the VA analyst to determine how a VA tool's combat simulations are calculated. I'm aware of one tool that uses a ratio of the size of a firing port to a human body to calculate the number of rounds that would pass through the port in an engagement, and that ratio remains constant regardless of the type of weapon engaging the port (i.e., precision-engagement weapon vs standard assault weapon). For example, consider an adversary who fires 20 shots from a sniper rifle with a 50% hit probability and 25%

kill probability at the given range at a response force member who is inside a firing port that is one-tenth the size of a standing human. AVERT would say two of those shots (i.e., .1Ox 20) would get through the port. Each of those two rounds would have a 50% chance of hitting, and a 25% chance of killing, the response force member. Hit and kill probabilities would not be applied to the other 18 rounds (i.e.,

no chance of hitting or killing the guard). Such a process seems skewed to benefit the response force and may overestimate the probability of neutralization, and by extension, system effectiveness. [h]

Respectfully, Lou From: Lee, Pete Sent: Thursday, February 07, 2019 7:07 AM To: Cubellis, Louis <Louis.Cubellis@nrc.gov>

Subject:

Fw: [WARNING: MESSAGE ENCRYPTED](OUO Attachment) Password protected draft outline When you get a chance over the next two weeks, could you take a look at the attached and provide some feedback on (1) whether there is additional topica l area that should be included; (2) for what has been identified, topics that should be included or expanded; and (3) any suggestions such that it would be a useful generic guidance for applying tool for analyzing systems to informed risk decision making.

I have not had a chance to look through it yet, but above is the feedback that \ like to provide.

For right now, this proj ect is you and I to keep it on track. We wil l so licit other for input for the deliverable fu rther dow n the road. M y reason is not to go off track and the waste of time and resources t o address inputs based on a false sense or assu med working knowledge and understanding of what is a physical protection system, a system approach to analysis or assessment, and basic modeling approaches, principles, and their limitatio ns.

Pete From: Snell, Mark K <mksnell @sand ja gov>

Sent: Monday, February 4, 20 19 3:39 PM To: Lee, Pete; Tarta l, George Cc: Whalen, Ryan T; Dennis, Matthew L; Parks, Mancel Jordan

Subject:

[Ext ern al_Se nder] [WARN ING: MESSAGE ENCRYPTED](OUO Attachme nt) Password protected draft outli ne Official Use Only Exemption 7 - Law Enforcement Attachment contains OUO Peter and George, Here is a password-protected versio n of the draft out line of the NU REG tec hnical report concern ing guidance on VA ModSim for your considerat ion . If you have any questions or comment s please let me kn ow .

The password w il l follow shortly.

Thanks, Mark

Note to requester: The attachment is immediately following this email record.

From: w....fm To: Cubellis Louis

Subject:

RE: Recommendations RE: Sandia Draft Outline for VA Tool NUREG Date: Wednesday, February 27, 2019 10:19:39 AM Attachments: NSIR - Review Comments LTD#4 Deliverable Submitted on 2-2-2019.docl\

imaae001.ona Lou Take a look at the attached. I believe I captured what you provided below in the table for feedback SNL.

I will not ask responding to them , as we will see in the next deliverable how they address these suggestion.

Go ahead and mark it up.

Thanks.

Senior Security Program Manager U.S. Nuclear Regulatory Commission 11545 Rockville Pike, Rockvil le, MD 20852-2738 Phone: 301-287-3690 pete.lee@nrc.gov

(

Website: www ore gov From: Cubel lis, Louis Se nt: Thursday, February 21, 2019 7:42 AM To: Lee, Pete <Pete.Lee@ nrc.gov>

Subject:

Recommendat ions RE: Sandia Draft Outline for VA Tool NU REG Good morning, Pete, I had a couple of hours this morning to review and comment on Sandia's draft outline for the NUREG. The outline includes widely accepted VA methodologies and considerations, and the sections appeared to me to correspond to the task order requirements. For example, requirement "a" is Section 1.3, requirement "b" is Section 2, requirement "c" is Section 3, etc. It seems like Sandia has begun pulling together information for all of the tasks, except for "j"; there's just a placeholder consisting of highlighted text from the task order in Section 10.

Per your request, I recommend you consider including the following points before you provide feedback to NRO and Sandia. I put the task order requirements in brackets at the end of each comment.

Section 1.3: Add as references [task a]:

o June 2018 NNSA Supplemental Directive 470.4-2 o January 2018 DoE validation and verificat ion report for AVERT o December 2011 DoD validation and verification report for AVERT

Sections 7 and 1O: Add information on using visual depictions of response force fields of fire (aka heat maps) to help validate the tools' pathway analysis results.

For example, I was able to detect a programming limitation in one VA tool by reviewing heat maps generated by the tool and then comparing the pathway analysis results to the vulnerabilities I identified in the maps. When none of the 1,000 computer-generated paths corresponded to a route the heat map indicated had nearly a 90% adversary success rate, I was able to ask why. I consulted the lead programmer for the company that produced the tool. He was able to take what I discovered, do some research, and then tell me that except for avoid detection, the tool did not consider facility access levels for adversary path strategies. This can cause the tool to make adversaries enter detection zones sooner or remain there longer than expected, which can prevent the adversaries from exploiting the most vulnerable pathways (MVPs) or give guard forces significantly longer periods to interrupt them en route to the MVPs. In the 1,000 runs we ran for the demo, the adversaries immediately entered detection zones outside the protected area and then tried to walk as many as 200 meters along the outer edge of the detection zone before breaching the protected area fence. This resulted in the guard force having the maximum interdiction time possible and none of the adversaries made it to the start of the MVPs. [g]

Sections 5, 8, and 1 O: In addition to pathway analyses, it seems like it would be helpful for a VA analyst to know that workarounds may also be necessary to ameliorate unrealistic response force or adversary capabilities provided by a VA tool (e.g., all-knowing response force, perfect response force communication at all times, lack of adversary coordination , etc.). For exam~le, some tools give response force members 360-degree hearing and eyesight to infinite distances. One way to counteract that would be to put relevant response force members into 'closets', and then use a 'respond to' or similar command to cause the members to exit the closets after someone or something else detects the adversary. [d, e, f, h, i, j]

Section 8: The NU REG should include some guidance for the VA analyst to determine how a VA tool's combat simulations are calculated. I'm aware of one tool that uses a ratio of the size of a firing port to a human body to calculate the number of rounds that would pass through the port in an engagement, and that ratio remains constant regardless of the type of weapon engaging the port (i.e., precision-engagement weapon vs standard assault weapon). For example, consider an adversary who fires 20 shots from a sn iper rifle with a 50% hit probability and 25%

kill probability at the given range at a response force member who is inside a firing port that is one-tenth the size of a standing human. AVERT would say two of those shots (i.e., .10 x 20) would get through the port. Each of those two rounds would have a 50% chance of hitting, and a 25% chance of killing, the response force member. Hit and kill probabilities would not be applied to the other 18 rounds (i.e.,

no chance of hitting or killing the guard). Such a process seems skewed to benefit the response force and may overestimate the probability of neutralization, and by extension, system effectiveness. [h]

Respectfully, Lou From: Lee, Pete Sent: Thursday, February 07, 2019 7:07 AM To: Cubellis, Louis <Louis Cubellis@nrc gov>

Subject:

Fw: [WARNING: MESSAGE ENCRYPTED](OUO Attachment) Password protected draft outline When you get a chance over the next two weeks, could you take a look at the attached and provide some feedback on (1) whether there is additional topica l area that should be i ncluded; (2) for what has been identified, topics that should be included or expanded; and (3) any suggestions such that it would be a useful generic guidance for applying tool for analyzing systems to informed risk decision making.

I have not had a chance to look through it yet, but above is the feedback that I like to provide.

For right now, this project is you and I to keep it on track. W e will solicit other for input for the deliverable further down the road. My reason is not to go off track and the waste of time and resources to address inputs based on a false sense or assumed working knowledge and understanding of what is a p hysical protection system, a system approach to analysis or assessment, and basic modeling approaches, principles, and t heir limitations.

Pete From: Snell, Mark K <mksnell @sandia gov>

Se nt: Monday, February 4, 2019 3:39 PM To: Lee, Pete; Tarta l, George Cc: Whalen, Ryan T; Dennis, Matthew L; Parks, Mancel Jordan

Subject:

[Externa l_Sender] [WARN ING: MESSAGE ENCRYPTED](OUO Attachment) Password protected draft outline Official Use Only Exemption 7 - Law Enforcement Attachment contains OUO Peter and George, Here is a password-protected version of the draft out line of the NUREG technical report concerning guidance on VA ModSim for your consideration. If you have any questions or comments please let me know.

The password wi ll fo llow shortly.

Thanks, Ma rk

NSIR/DPCP/RSB Comments Guidance on Applications of Vulnerability Assessment Models For Assessing the Effectiveness of a Physical Protection System NRCHQ2514D005-NRCHQ2517T004 Tasks 3a and 3b Deliverables Date: February, 2019 No. Source Section Comment 1 NSIR/DPCP Acronyms and Use appropriate generic terms to describe the organizations or persons that provides specific

/RSB Units of Measure functions or information that may be important in the guidance. To the extent possible avoid (Plee) (Pages 11-12 ) references and acronyms (e.g., Office of Radiological Security, Radiation Safety Officer, etc.) that are specific to an organization or a person that are unique or specific to the DOE or NRC.

Basis: Guidance is intended to be generic and should, to the extent possible, avoid to the use of specific terms or acronyms unique to the DOE, NRC, and/or other federal agencies.

2 NSIR/DPCP Section 1.3 Include references that would assist users' basic understanding of quantification or the numerical

/RSB (Pages 14-15) methods (e.g., Monte Carlo or others) and their application in VA tools for modeling and simulations.

(Plee) Guidance should provide understanding of strength, weaknesses, and imitations of numerical methods and underlying techniques that may be applied in the VA tool to determine and solve for probabilities and/or system effectiveness.

Basis: User or modelers should have a basic reference on the Monte Carlo methods or other stochastic methods for the numerical technique applied in the VA tools and appropriate understanding of their application and limitations.

3 NSIR/DPCP Section 1.3 Review of existing guidance should include the DOE NNSA Supplemental Directive 470.4-2,

/RSB (Pages 14-15) "Enterprise Safeguards and Security Planning and Analysis Program," and "Implementation Instruction (LCubellis} 470.4-2," dated June 23, 2018 for developing guidance with respect to the use of VA model and simulations. Information and insights that are relevant to the criteria of "helpful in addressing the use of VA model and simulations for physical protection purpose," should be consider for guidance.

4 NSIR/DPCP Section 1.3 (a) Review DOE's "Validation and Verification Report for AVERT," January 2018 and Defense Threat

/RSB (Page 14-15) Reduction Agency, "Automated Vulnerability Evaluation for the Risk of Terrorism (AVERT) Software (LCubellis) Version 5.1.2, Accreditation Report," December 2011 with respect to providing guidance the "dos and don'ts" on VA tool for modeling and simulations for quantifying PPS effectiveness and their use, including the strengths and weaknesses that may be inherent to the how VA tool are built to perform modeling and simulation.

(b) Provide guidance on appropriate verification and validation (V&V) process to ensure that model and simulations are correct and reliable (i.e., credibility of results). Guidance should identify V&V challenges and how they may be appropriately addressed, including how VA tools may be accredited Special Note: On the basis that guidance being develop is generic and not guidance on the use of a specific VA tool, the Information that is helpful should be captured generically and specific references to the reports of a specific V A tool should not be captured either Section 1.3 and as References.

Basis: Insights from these evaluations of an existing VA tool for modeling simulation should be captured generically as guidance for users' dos and don'ts and strengths and weaknesses that may be encounter of VA tools.

5 NSIR/DPCP Section 2.ii (Page Ensure that the guidance in this section address the accuracies, interpretations, and limitations of data

/RSB 15) collected from performance tests, limited drills and exercises, including simulated force-on-force, with (Plee) respect to application reasonable insights that considers human factors and behaviors (i.e .,

uncertainties) under extreme stress situations such as combat, which solicits physiological and psychological responses that are not simulated.

Basis: Specific guidance on the insights of realistic and limitations of data collected from performance testing (e.g., limited scope drills and exercises and force-on-force exercises) will ensure the realism of modeling and simulation of individuals (responders and adversary) performance under fire (i.e. ,

combat). How VA tool capture reasonable simulations and interpretation of results characterizing the uncertainties of human performance under fire is of significance to the results on reliability and availability of interdiction/neutralization functions.

6 NSIR/DPCP Section 4.iv Ensure guidance address the subject of communications and how model and simulation should

/RSB (Page 17) accurately capture planned and performance of the security communication capabilities of the PPS ..

(Plee)

Basis: Communications is a key and critical element in the assessment and the command and control of security response in the sequences of events to achieve interdiction/neutralization functions. The accuracy and realism of modeling and simulation need to address the capabilities of security specific and plant systems relied on for communications for reasonable realistic outcomes.

7 NSIR/DPCP Section 4.iv(a) Ensure guidance include in-depth guidance on how to appropriately determine pathways and

/RSB (Page 17) "pathway analysis" and how models and simulation should determine pathways based on the PPS (Plee) design and the DBT characteristics. Include guidance on how modeling and simulation should sufficiently address all pathways, including those pathways with large delay time - greater tasks completion time or greater field of fire other PPS features. Provide guidance on modeling and simulation and user should determine and assess the likelihood of adversary's selection of pathway or pathways (speed , cover/concealment, avoid detection, etc.).

Basis: In-depth guidance on the subject of pathways will permit users to understand how the modeling and simulations should determine and selections of pathways and how the pathways contribute or impact determining the probabilities of neutralization and its contribution to the resulting overall PPS effectiveness.

8 NSIR/DPCP Section 4.iv(e) Ensure guidance address specifics for determining appropriate and collecting data for performance

/RSB Plee) (Page 17) testing that are reasonably realistic response capabilities and standards. Specifically include guidance on considerations of human factors and behaviors for person or persons under fire.

Basis: Performance tests has limitations that must be accounted in modeling and simulation. Specific guidance on insights of realistic and limitations of data collected from performance testing will ensure the realism of performance under fire and reasonable interpretation of results to capture uncertainties of human performance under fire.

9 NSIR/DPCP Section 5.ii (Page Ensure the guidance include how to establish the minimum and acceptable value(s) for system

/RSB 18) effectiveness (PE)- Include discussion of what values means, specific to upper bound or lower bounds (Plee) that would not be acceptable, one that would be unrealistic of modeling and simulation results and other unacceptable performance, respectively.

Basis: Guidance to users should understand what range of value are for PE that would be reasonably acceptable for the PPS effectiveness and values that may not acceptable or suspect of modeling and simulations. For example, why is PE= 1 is a mathematically improbable based on the effectiveness of each elements and sequences of events for a PPS and would not be reasonable value or why a PE=

0.5 may not a reasonable standard value for effective of a PPS for meeting the criteria of reasonable assurance.

10 NSIR/DPCP Section 5.ii (Page Ensure guidance include specifics on the distinctions between and the use and application of

/RSB 18) "performance testing" and "Force on Force exercises," in conjunction with modeling and simulation for (Plee) determining PE.

Basis: The applications and limitations of performance tests and Force-on-Force exercises should be understood by the users and how they are used in and valid portions or parts of the modeling and simulation results.

11 NSIR/DPCP Section 5.ii Ensure guidance provide discussions on the applications and limitations on "use of multiple M&s tools

/RSB (Page 18) that analyze and utilize mass amounts of human performance testing that should provide a better PE (Plee) as it relates to response capabilities." Include guidance on how users should address differing results from using multiple tools.

Basis: Guidance on the using multiple modeling and simulation tools, including "probability of neutralization calculations" (Section 4.iv(a)), should be provided to users on how to assess tools available, compare, and interpret varying PE for the effectiveness.

12 NSIR/DPCP Section 5.iv (Page Ensure guidance address the advantages and advantages for using various VA tools (qualitative,

/RSB 19) quantitative, or a combinations) available to assess physical security functions and system (Plee) effectiveness.

Basis: Guidance should provide users the pros and cons of particular type of VA tools or assessment methods, TTX, human-in-the loop combat computer simulation, and batch code simulations, indicated in this section.

13 NSIR/DPCP Section 5.iv(a) Ensure this section include guidance the minimum considerations for conducting quality TTX. In

/RSB (Page 19) addition, include in this or subsequent section (i.e., Section 10) of the guidance, the specifics (Plee) considerations of subject matter experts (e.g. , knowledge, skills, and abilities) that would be necessary and sufficient for using and implementing TTX as VA tool for assessing the effectiveness of a PPS.

Basis: Guidance should allow user to understand baseline or minimum considerations or reasonable criteria on how a quality TTX may be conducted to assess vulnerabilities and capabilities of elements of a PPS. Guidance on the subject matter expertise needed and necessary is critical for applying TTX as a method of qualitatively assess a PPS and ensure that quality implemented TTX results would be consistent or comparable to results from quantified PPS effectiveness.

14 Sections 5, 8, & 10 Ensure guidance for user, VA analyst, describe how to identify and how to address potential NSIR/DPCP (Pages 18-20, unrealistic default inputs a and/or conditional values ((e.g. , all-knowing response force, perfect

/RSB Pages 22-23, and response force communication at all times, lack of adversary coordination, etc.). Provide guidance of (LCubellis) Page 24) what may appropriate to recalibrate, workaround, or other measures that may be necessary to ameliorate unrealistic response force or adversary capabilities programmed or default conditions of a VA tool.

Basis: Some VA tools give response force members 360-degree hearing and eyesight to infinite distances. Guidance for such programmed logic or default values maybe to isolate or delay relevant response force members (e.g., put them into 'closets', and then use a 'respond to' or similar command to cause the members to exit the closets) until after someone or IDS detects the adversary.

15 NSIR/DPCP Section 7 Ensure that guidance in this section, on "pathway analysis," include how modeling and simulation tool

/RSB (Page 22) capture pathway analysis. Guidance should address "should" and "should not" for modeling and (Plee) simulation of pathways and how key elements of the PPS are quantified and/or qualified in a modeling and simulation tool.

Basis: In addition to describing the details on the concept of "pathway analysis" that is need to assess a PPS, the guidance should include how it is captured or should be captured in modeling and simulation tools.

16 NSIR/DPCP Sections 7 and 10 Ensure guidance in Sections 7 and 10 address the use of visual depictions of response force fields of

/RSB (Pages 21-22 and fire (i.e. , heat maps) in validating of VA tools' pathway analysis results, to verify, evaluate, and identify (LCubellis) Page 24) potential programing limitations of pathway analysis results to vulnerabilities shown on heat maps.

Basis: A user is able to detect a programming limitation in a VA tool by reviewing heat maps generated by the tool and then comparing the pathway analysis results to the visually identified vulnerabilities (e.g., lack of coverage or gaps, minimum over lapping field of fire, blocked line of sights, etc.) on the maps. For example, when none of a VA tool's 1,000 computer-generated paths corresponded to a route the heat map indicated had nearly a 90% adversary success rate, a user would ask and investigate the results of reasonable and realistic representation of pathways.

Programing logic, with exception of condition for avoiding detection, did not consider facility access levels for adversary path strategies, which caused adversaries enter detection zones sooner or remain there longer than expected. The resulting programing logic can prevent the adversaries from exploiting the most vulnerable pathways (MVPs) or give guard forces significantly longer periods to interrupt them in the route to the MVPs. It was determined that in the 1,000 runs, the adversaries immediately entered detection zones outside the protected area and then tried to walk as many as 200 meters along the outer edge of the detection zone before breaching the protected area fence.

This resulted in the guard force having the maximum interdiction time possible and none of the adversaries made it to the start of the MVPs.

17 Section 8 Ensure guidance include how users, VA analyst, would determine that a VA tool's combat simulations NSIR/DPCP (Page 22-23) is reasonable and realistic (i.e. , appropriate determined or calculated). Include in the guidance how

/RSB they should not be determined or calculated, and how to identify unreasonable and unrealistic (LCubellis) program or default conditions.

Basis: Some VA tool uses a ratio of the size of a firing port to a human body to calculate the number of rounds that would pass through the port in an engagement, and that ratio remains constant regardless of the type of weapon engaging the port (i.e., precision-engagement weapon vs standard assault weapon). For example, consider an adversary who fires 20 shots from a sniper rifle with a 50% hit probability and 25% kill probability at the given range at a response force member who is inside a firing port that is one-tenth the size of a standing human. A VA tool may be programmed or uses a default condition such that two of those shots (i.e. , .10 x 20) would get through the port. Each of those two rounds would have a 50% chance of hitting , and a 25% chance of killing, the response force member. The probabilities for hit and kill would not be applied to the other 18 rounds (i.e., no chance of hitting or killing the guard). Such a process seems skewed to benefit the response force and may overestimate the PN, and by extension overestimate the PPS effectiveness.

18 NSIR/DPCP Section 8.v Ensure descriptions and discussions of statistical methods is sufficiently in depth to provide the users

/RSB (Page 22) understanding to adequately and reasonably apply and interpret results of system effectiveness from (Plee) a modeling and simulation tool. The guidance should include the appropriate characterization of statistical significance of limited drills and exercises, performance tests, which may be relied on for performance insights and validation of results.

Basis: The statistical methods, including sufficient sampling, applied to appropriate interpretations of modeling and simulation results for the PN to ensure reasonable characterization of the uncertainties of security response and the resulting overall PPS effectiveness.

19 NSIR/DPCP Section 10 Provide guidance addressing minimum specific knowledge, skills, and abilities (KSA) needed and

/RSB (Page 24) necessary for individuals from developer to end users, and in between, for quality and adequate (Plee) application of various VA tools. The minimum training and qualification for individuals should address the application of qualitative assessment (i.e., TTX) and quantitative assessment (modeling and simulations). Guidance should also include the KSA for individuals performing quality assurance, validation and verification, and auditing.

Basis: Appropriate basic KSA of various individuals provide assurance of the quality from developer to end user of VA tools for their intended purpose and assurance of reasonably realistic assessment of PPS effectiveness and identifying vulnerabilities or weaknesses.

20 NSIR/DPCP References Ensure references identified in Section 1.3 are captured in the list of references.

/RSB (Page 25)

(Plee)

Note to requester: This version of the email record was provided to the NRC FOIA staff, with only the header on final email on the next page.

From: w...fm To: Cubellis Louis

Subject:

Re: Comments - Draft Outline and Content for Guidance on Application of VA - NRCHQ2514D005-NRCHQ2517T004 Date: Monday, March 04, 2019 7:21 :09 AM Attachments: imaae001.ona Great. I will let them know.

I just talk to Rocky, fill him in on discussions I had with managem ent last Friday.

Pete From: Cubel lis, Louis Sent: M onday, March 4, 2019 7:14 AM To: Lee, Pete

Subject:

Re: Comments - Draf t Outline and Content for Guidance on Application of VA -

N RCHQ2514D005-N RCHQ2517T004 Good morning, Pete. I reviewed the information you provided to SN L and NRO, and it accurate ly captures my comments.

Louis J. Cubellis, Jr U.S. Nuclear Regulatory Commission Office of Nuclear Security and Incident Response Phone : (301) 287-3670 E-mail: Louis Cubellis@nrc gov From: Lee, Pete Sent: Friday, March 1, 2019 9:06 AM To: Parks, Ma ncel Jordan; Dennis, Matthew L Cc: Whalen, Ryan T; Tarta l, George; Snell, Mark K; Vechioli Feliciano, Lucieann; Cubellis, Louis

Subject:

Comments - Draft Outline and Content for Guidance on Application of VA -

N RCHQ2514D00S-N RCHQ25 l 7T004 Jordan/Matt:

Attached is the subject. It's being peer reviewed Lou Cubellis of my branch that has extensive knowledge on the subject and physical security and combat, and also attended the recent DOE NTC training on VA.

I requested him to verify that I captured his input adequately and peer review my comments/suggestion.

Any additions or changes will be forwarded to you once that is complete.

There is no need to provide responses to the comments, as they are reminder in nature for what I believe are planned for populating the content outlined.

We will review the next deliverable on how they are addressed. Please give me a call if you need to touch based on the comments/suggestions.

Based on the outline and content, I believe you are on the way 10 meeting our expectation for a guidance that will be useful to a wide audience, accordance with the milestones in LTD#4.

Thank you ..

Senior Security Program Manager U.S. Nuclear Regulatory Commission 11545 Rockville Pike, Rockville, MD 20852-2738 Phone: 301-287-3690 pete.lee@nrc.gov

(

R

,.. r,. 1" *w, ,__.,

Website: www arc e;ov From: Snell, Mark K l@sandia.gov

Sent: Monday, February 04, 2019 3:37 PM To: Lee, Pete <Pete.Lee@nrc.gov>; Tartal, George <George.Tartal@nrc.gov>

Cc: Whalen, Ryan T <rtwhale@sandia.gov>; Dennis, Matthew L <mldenni@sandia.gov>; Parks, Mancel Jordan <mjparks@sandia.gov>

Subject:

[External_Sender] FW: Draft Outline

Note to requester: The February 201 9 Comments document, immediately following this email, was included as part of this record, as provided to the FOIA staff.

From: w...fm To : Parks Mancel Jordan: Pennis Matthew L Cc: Whalen Ryan T: Tartal George: Snell Mark K: Vechioli Feliciano Lucjeann: Cubellis Louis Subject : Re: Comments - Draft Out line and Content for Guidance on Application of VA - NRCHQ2514D005-NRCHQ2517T004 Date: Monday, March 11, 2019 2:09:39 PM Attachments: iroaoeoo1 ono Just to close the loop. There is no additio ns or changes.

M att. Good working with you on past tasks and this one . Thank you .

Pete From: Lee, Pete Sent : Friday, March 1, 2019 9:06 AM To: Parks, Ma ncel Jordan; Dennis, Matthew L Cc: Whalen, Ryan T; Tarta l, George; Snell, Mark K; Vech ioli Felicia no, Lucieann; Cubellis, Louis

Subject:

Comments - Draft Outline and Content for Guidance on Application of VA -

NRCHQ2514D005-NRCHQ2517T004 Jordan/Matt:

Attached is the subject. It's being peer reviewed Lou Cubellis of my branch that has extensive knowledge on the subject and physical security and combat, and also attended the recent DOE NTC training on VA.

I requested him to verify that I captured his input adequately and peer review my comments/suggestion.

Any additions or changes will be forwarded to you once that is complete.

There is no need to provide responses to the comments, as they are reminder in nature for what I believe are planned for populating the content outlined.

We will review the next deliverable on how they are addressed. Please give me a call if you need to touch based on the comments/suggestions.

Based on the outline and content, I believe you are on the way to meeting our expectation for a guidance that will be useful to a wide audience, accordance with the milestones in LTD#4.

Thank you ..

Senior Security Program Manager U.S. Nuclear Regulatory Commission 11545 Rockville Pike, Rockville, MD 20852-2738 Phone: 301-287-3690 pete.lee@nrc.gov

Website: www. nrc.sov From: Snell, Mark K [mailto :mksnell@sandia .gov]

Sent: Monday, February 04, 2019 3:37 PM To: Lee, Pete <Pete. Lee@nrc.gov>; Tarta l, George <George.Tarta l@nrc.gov>

Cc: Whalen , Ryan T <rtwha le@sand ia.gov>; Denn is, M atthew L <m ldenni@sandia.gov>; Parks, Mancel Jordan <mjparks@sandia .gov>

Subject:

[Ext ernal_Se nder] FW: Draft Outline

NSIR/DPCP/ RSB Comments Guidance on Applications of Vulnerability Assessment Models For Assessing the Effectiveness of a Physical Protection System NRCHQ2514O005-NRCHQ2517T004 Tasks 3a and 3b Deliverables Date: February, 2019 No. Source Section Comment 1 NSIR/DPCP Acronyms and Use appropriate generic terms to describe the organizations or persons that provides specific

/RSB Units of Measure functions or information that may be important in the guidance. To the extent possible avoid (Plee) (Pages 11-12 ) references and acronyms (e.g., Office of Radiological Security, Radiation Safety Officer, etc.) that are specific to an organization or a person that are unique or specific to the DOE or NRC.

Basis: Guidance is intended to be generic and should, to the extent possible, avoid to the use of specific terms or acronyms unique to the DOE, NRC, and/or other federal agencies.

2 NSIR/DPCP Section 1.3 Include references that would assist users' basic understanding of quantification or the numerical

/RSB (Pages 14-15) methods (e.g., Monte Carlo or others) and their application in VA tools for modeling and simulations.

(Plee) Guidance should provide understanding of strength, weaknesses, and imitations of numerical methods and underlying techniques that may be applied in the VA tool to determine and solve for probabilities and/or system effectiveness.

Basis: User or modelers should have a basic reference on the Monte Carlo methods or other stochastic methods for the numerical technique applied in the VA tools and appropriate understanding of their application and limitations.

3 NSIR/DPCP Section 1.3 Review of existing guidance should include the DOE NNSA Supplemental Directive 470.4-2,

/RSB (Pages 14-15) "Enterprise Safeguards and Security Planning and Analysis Program," and "Implementation Instruction (LCubellis) 470.4-2," dated June 23, 2018 for developing guidance with respect to the use of VA model and simulations. Information and insights that are relevant to the criteria of "helpful in addressing the use of VA model and simulations for physical protection purpose," should be consider for guidance.

4 NSIR/DPCP Section 1.3 (a) Review DOE's "Validation and Verification Report for AVERT," January 2018 and Defense Threat

/RSB (Page 14-15) Reduction Agency, "Automated Vulnerability Evaluation for the Risk of Terrorism (AVERT) Software (LCubellis) Version 5.1.2, Accreditation Report," December 2011 with respect to providing guidance the "dos and don'ts" on VA tool for modeling and simulations for quantifying PPS effectiveness and their use, including the strengths and weaknesses that may be inherent to the how VA tool are built to perform modeling and simulation.

(b) Provide guidance on appropriate verification and validation (V&V) process to ensure that model and simulations are correct and reliable (i.e., credibility of results). Guidance should identify V&V challenges and how they may be appropriately addressed, including how VA tools may be accredited Special Note: On the basis that guidance being develop is generic and not guidance on the use of a specific VA tool, the Information that is helpful should be captured generically and specific references to the reports of a specific VA tool should not be captured either Section 1.3 and as References.

Basis: Insights from these evaluations of an existing VA tool for modeling simulation should be captured generically as guidance for users' dos and don'ts and strengths and weaknesses that may be encounter of VA tools.

5 NSIR/DPCP Section 2.ii (Page Ensure that the guidance in this section address the accuracies, interpretations, and limitations of data

/RSB 15) collected from performance tests, limited drills and exercises, including simulated force-on-force, with (Plee) respect to application reasonable insights that considers human factors and behaviors (i.e.,

uncertainties) under extreme stress situations such as combat, which solicits physiological and psychological responses that are not simulated.

Basis: Specific guidance on the insights of realistic and limitations of data collected from performance testing (e.g., limited scope drills and exercises and force-on-force exercises) will ensure the realism of modeling and simulation of individuals (responders and adversary) performance under fire (i.e.,

combat). How VA tool capture reasonable simulations and interpretation of results characterizing the uncertainties of human performance under fire is of significance to the results on reliability and availability of interdiction/neutralization functions.

6 NSIR/DPCP Section 4.iv Ensure guidance address the subject of communications and how model and simulation should

/RSB (Page 17) accurately capture planned and performance of the security communication capabilities of the PPS ..

(Plee)

Basis: Communications is a key and critical element in the assessment and the command and control of security response in the sequences of events to achieve interdiction/neutralization functions. The accuracy and realism of modeling and simulation need to address the capabilities of security specific and plant systems relied on for communications for reasonable realistic outcomes.

7 NSIR/DPCP Section 4.iv(a) Ensure guidance include in-depth guidance on how to appropriately determine pathways and

/RSB (Page 17) "pathway analysis" and how models and simulation should determine pathways based on the PPS (Plee) design and the DBT characteristics. Include guidance on how modeling and simulation should sufficiently address all pathways, including those pathways with large delay time - greater tasks completion time or greater field of fire other PPS features. Provide guidance on modeling and simulation and user should determine and assess the likelihood of adversary's selection of pathway or pathways (speed, cover/concealment, avoid detection, etc.).

Basis: In-depth guidance on the subject of pathways will permit users to understand how the modeling and simulations should determine and selections of pathways and how the pathways contribute or impact determining the probabilities of neutralization and its contribution to the resulting overall PPS effectiveness.

8 NSIR/DPCP Section 4 .iv( e) Ensure guidance address specifics for determining appropriate and collecting data for performance

/RSB Plee) (Page 17) testing that are reasonably realistic response capabilities and standards. Specifically include guidance on considerations of human factors and behaviors for person or persons under fire.

Basis: Performance tests has limitations that must be accounted in modeling and simulation. Specific guidance on insights of realistic and limitations of data collected from performance testing will ensure the realism of performance under fire and reasonable interpretation of results to capture uncertainties of human performance under fire.

9 NSIR/DPCP Section 5.ii (Page Ensure the guidance include how to establish the minimum and acceptable value(s) for system

/RSB 18) effectiveness (PE). Include discussion of what values means, specific to upper bound or lower bounds (PLee) that would not be acceptable, one that would be unrealistic of modeling and simulation results and other unacceptable performance, respectively.

Basis: Guidance to users should understand what range of value are for PE that would be reasonably acceptable for the PPS effectiveness and values that may not acceptable or suspect of modeling and simulations. For example, why is PE= 1 is a mathematicallly improbable based on the effectiveness of each elements and sequences of events for a PPS and would not be reasonable value or why a PE=