Sampled NetFlow: Difference between revisions

From wiki.comcert.com
Jump to navigation Jump to search
No edit summary
No edit summary
 
(2 intermediate revisions by the same user not shown)
Line 14: Line 14:
=== Rate Calculation ===
=== Rate Calculation ===


Compared to SFlow, NetFlow lacks the ability to report metrics such as bit rate (bps) and&nbsp;packet rate (pps) exactly as they are.&nbsp; A single&nbsp;NetFlow&nbsp;record&nbsp;reports on&nbsp;<u>flow duration</u> and quantitative metrics such as the <u>number of bytes and packets</u> transferred during that time interval.&nbsp; Next, NetFlow reporter <u>calculates</u>&nbsp;to its best abilities <u>bit rate</u> and <u>packet rate</u> based on that information.&nbsp; Flow sampling&nbsp;at a&nbsp;rate that is not adapted to the port speed of the interface, will compromize the exactitude of that calculation, even when all other precautionary measures such as "active time-out" and "inactive time-out are applied correctly.
Compared to SFlow, NetFlow lacks the ability to report metrics such as bit rate (bps) and&nbsp;packet rate (pps) exactly as they are.&nbsp; A single&nbsp;NetFlow&nbsp;record&nbsp;reports on&nbsp;<u>flow duration</u> and quantitative metrics such as the <u>number of bytes and packets</u> transferred during that time interval.&nbsp; Next, NetFlow reporter <u>calculates</u>&nbsp;to its best abilities <u>bit rate</u> and <u>packet rate</u> based on that information.&nbsp; Flow sampling&nbsp;at a&nbsp;rate that is not adapted to the speed of the circuit, will compromize the exactitude of that calculation, even when all other precautionary measures such as "active time-out" and "inactive time-out are applied correctly.


&nbsp;
&nbsp;
Line 26: Line 26:
=== Impact on Network Analysis ===
=== Impact on Network Analysis ===


Tools for flow-based network analysis are used to monitor and baseline network behavior and increasingly by security teams to detect persistent threats.&nbsp; Sampling rates that are not adapted to the port speed of the interface may lead to inaccurate reporting of quantitative metrics such as bit rate, packet rate and utilization.&nbsp; It may&nbsp;also lead to some short-lived flows no longer being reported at all.&nbsp; When security professionals need to go back in time, the flows they want to investigate may no longer be there.
Tools for flow-based network analysis are used to monitor and baseline network behavior and increasingly by security teams to detect persistent threats.&nbsp; Sampling rates that are not adapted to the circuit speed&nbsp;may lead to inaccurate reporting of quantitative metrics such as bit rate, packet rate and utilization.&nbsp; It may&nbsp;also lead to some short-lived flows no longer being reported at all.&nbsp; When security professionals need to go back in time, the flows they want to investigate may no longer be there.


&nbsp;
&nbsp;
Line 36: Line 36:
#Validate the data reported.&nbsp; Compare&nbsp;metrics such as bit rate and packet rate against the same metrics obtained by other tools such as SNMP plotters.&nbsp; <u>You must find a&nbsp;balance between sampling rate and report&nbsp;exactitude.</u>  
#Validate the data reported.&nbsp; Compare&nbsp;metrics such as bit rate and packet rate against the same metrics obtained by other tools such as SNMP plotters.&nbsp; <u>You must find a&nbsp;balance between sampling rate and report&nbsp;exactitude.</u>  
#Please realize that the "raw" flow records database kept&nbsp;by some type of collectors&nbsp;will&nbsp;probably be incomplete; i.e. the tool may no longer be suitable for forensic flow analysis by the security team.  
#Please realize that the "raw" flow records database kept&nbsp;by some type of collectors&nbsp;will&nbsp;probably be incomplete; i.e. the tool may no longer be suitable for forensic flow analysis by the security team.  
#If device/network efficienty is a concern, and SFlow is an option, use SFlow instead of Sampled NetFlow.&nbsp; SFlow is reporting interface counters and the exactitude of metrics derived from those counters is independent from the sampling rate.&nbsp;  
#If device/network efficienty is a concern, and SFlow is an option, use SFlow instead of Sampled NetFlow.&nbsp; SFlow exporting devices send interface counters to the collector and the exactitude of metrics derived from those counters is therefore independent from the sampling rate.&nbsp;  


&nbsp;
&nbsp;


&nbsp;
&nbsp;

Latest revision as of 17:09, 4 December 2020

Introduction

NetFlow was designed to process all IP packets on an interface and to report on metrics such as source and destination of traffic, class of service, and the causes of congestion. 

Increasingly higher traffic rates mean higher numbers of NetFlow records and increasing load on exporting device resources.  The increasing number of records may require too much bandwidth when send over the network to the collector.  Therefore, it may become necessary finding a balance between reporting needs and device/network efficiency.  Vendors may recommend or even force the use of Sampled NetFlow on some of their devices.

NetFlow was designed by Cisco in the late 90's and Cisco still owns the term NetFlow.  This artice applies to all varieties of NetFlow that support sampling.  

 

Solution

Rate Calculation

Compared to SFlow, NetFlow lacks the ability to report metrics such as bit rate (bps) and packet rate (pps) exactly as they are.  A single NetFlow record reports on flow duration and quantitative metrics such as the number of bytes and packets transferred during that time interval.  Next, NetFlow reporter calculates to its best abilities bit rate and packet rate based on that information.  Flow sampling at a rate that is not adapted to the speed of the circuit, will compromize the exactitude of that calculation, even when all other precautionary measures such as "active time-out" and "inactive time-out are applied correctly.

 

Sampled NetFlow

While a sampling rate of 1 out-of 100 may reduce the exports of NetFlow records by as much as 50%, you must keep in mind that the collector will only receive a small percentage of the traffic and it will use the reported sampling rate to "inflate" the metrics reported for #bytes and #packets.

 

Impact on Network Analysis

Tools for flow-based network analysis are used to monitor and baseline network behavior and increasingly by security teams to detect persistent threats.  Sampling rates that are not adapted to the circuit speed may lead to inaccurate reporting of quantitative metrics such as bit rate, packet rate and utilization.  It may also lead to some short-lived flows no longer being reported at all.  When security professionals need to go back in time, the flows they want to investigate may no longer be there.

 

Using Sampled NetFlow

  1. The use of Sampled NetFlow instead of "regular" NetFlow will always come at a cost.  Use Sampled NetFlow only if you must.  This could be the case when monitoring a high-speed interface (n x Gbps), limited network capacity or when the device's vendor forces you to use a sampler.
  2. Start at sampling rates 1 out-of tens or 1 out-of hundreds.  A rates of 1 out-of-thousand or smaller is seldom acceptable. 
  3. Validate the data reported.  Compare metrics such as bit rate and packet rate against the same metrics obtained by other tools such as SNMP plotters.  You must find a balance between sampling rate and report exactitude.
  4. Please realize that the "raw" flow records database kept by some type of collectors will probably be incomplete; i.e. the tool may no longer be suitable for forensic flow analysis by the security team.
  5. If device/network efficienty is a concern, and SFlow is an option, use SFlow instead of Sampled NetFlow.  SFlow exporting devices send interface counters to the collector and the exactitude of metrics derived from those counters is therefore independent from the sampling rate.