Configuring flow export: Difference between revisions

From wiki.comcert.com
Jump to navigation Jump to search
No edit summary
No edit summary
 
(57 intermediate revisions by the same user not shown)
Line 2: Line 2:
== Introduction ==
== Introduction ==


 
Correct configuration of Flow Exporting Devices (FED) is key to Flow Based Network Analysis. Mistakes made during configuration will result in missing and duplicated data.  They are difficult to identify. Vendors of network equipment are not making it easy as many of them are using some sort of proprietary method to enable flow export on their devices.
 
[[File:WIP.jpg|center|150px|WIP.jpg]]
 
 


Correct configuration of Flow Exporting Devices (FED) is key to Flow Based Network Analysis. Errors made during configuration will result in missing or duplicate data and is difficult to spot. Vendors of network  equipment are not making it easy and most of them are using a proprietary method in order to enable network flow record exports to the collector (in our case TVF or TVA).
Furthermore, Flow Collector (FC) and FED must allign on configuration parameters in order to interpret the records correctly.


Furthermore, some collectors require a special setting on FED in order to undersand its flow records correctly.
This article creates a vendor independant framework to configure flow exports.  We hope it will help you to confgure your FED.


This article is trying to decribe the idea behind network flow reporting and we hope it will help you to determine the correct procedure on how to confgure your FEDs.
<span style="background-color:#FFFF00;">We ask you kindly to read the entire article and not to jump to the confuration examples towards&nbsp;the end.</span>


For a complete overview of&nbsp;IP Flow Information eXchange (IPFIX), we recommend the reading of [https://tools.ietf.org/html/rfc3917 RFC 3917].
For a complete overview of&nbsp;IP Flow Information eXchange (IPFIX), we recommend reading&nbsp;[https://tools.ietf.org/html/rfc3917 RFC 3917].


If you are aware of vendor specific issues with the configuration of flow exports, it is greatly appreciated if you [mailto:wiki@comcert.com let us know].
If you come aware of vendor specific issues thay may help others, please&nbsp;[mailto:wiki@comcert.com let us know].


&nbsp;
&nbsp;
Line 24: Line 20:
=== Definitions ===
=== Definitions ===


==== FED ====
First, let us make sure we speak the same language.
 
==== Flow Enabled Device or Flow Exporter ====


A&nbsp;''Flow Enabled Device'' is any L3 and in some cases L2 device sending flow records to a collector.
''Flow Enabled Device''&nbsp;of ''Flow Exporter'' is a device configured to send flow records to one or more collectors.&nbsp; This can be a L2 or L3 device.


==== Flow ====
==== Flow or IP Flow ====


A&nbsp;''flow''&nbsp;is defined as a stream of packets between a given source and a given destination. For example, in client-server computing a TCP sessions consists of two flows: one flow from client to server and the second flow from server to client.
''Flow''&nbsp;is formed by all&nbsp;packets between two TCP/IP sockets.&nbsp; Therefore, what we call a TCP sessions consists of two flows: one flow from client to server and the second flow from server to client.


==== Flow record ====
==== Flow record ====


A ''flow record'' contains information about a specific flow that was&nbsp;metered at an observation point.&nbsp; A flow record contains measured&nbsp;properties of the flow (e.g., the total number of bytes of all packets of the flow, flow duration and the physical or virtual input interface i.e. where the packets entered the FED) and usually characteristic properties of the&nbsp;flow (e.g., source/destination IP address, protocol, port, ToS/DiffServ marking, ...).
''Flow record'' contains information about a specific flow that was&nbsp;metered at an observation point.&nbsp; A flow record contains measured&nbsp;properties aggregated for the flow.&nbsp; Key record fields are source and destination IP addess, protocol, source and destination port, number of bytes, number of packets, duration and input and output&nbsp;interfaces.&nbsp; There could be many more.


==== Sensor ====
==== Sensor ====


A ''sensor'' is a meter located at an observation point. It will "observe" packets going by the observation point and&nbsp;"build" the flow record decribing&nbsp;the flow.&nbsp; A sensor will meter&nbsp;packets in one direction only (ingress/in or egress/out) but some vendors use sensors that are bidirectional. Historically, a sensor was ingress/in only, so if left unspecified by a vendor, you may assume the&nbsp;sensor being unidirectional and ingress/in only.
''Sensor'' is a metering point in the device. Sensors are associated with interfaces.&nbsp; It will quantify and qualify packets in or&nbsp;out of that interface, leading to the creation of a flow record decribing&nbsp;the flow.&nbsp; It is very important to realize a sensor will measure packets in one direction only (ingress or egress).&nbsp;


==== Collector ====
==== Collector ====


A flow ''collector'' is most likely a server that collects the flow records and stores them in in a file or database. In TruView, TruView Flow is the collector.
Flow ''collector'' is a server application&nbsp;that receives&nbsp;the flow records and stores them in a database. In TruView, TruView Flow is the collector.&nbsp; In nGeniusONE, the Flow Collector is the collector.&nbsp; The sizeof the hardware will depend on the number of flows received per second and the aging time of the "raw" flow records.


Reporter
==== Reporter ====


The ''reporter'' is an application running on the collector that data mines the records to produce statistical results. In some cases, collector and reporter are the same server. In TruView,TruView Central is the reporter.
Flow ''reporter'' is a server application&nbsp;mining the data records,&nbsp;producing&nbsp;tables and charts and possibly alerts.&nbsp; In TruView, TruView Central is the reporter.&nbsp;&nbsp;In nGeniusONE, nGenius Server&nbsp;is the reporter.&nbsp; When the number of flows received per second is low&nbsp;(<50k/s),&nbsp;collector and reporter can be installed on the same hardware.


&nbsp;
&nbsp;
Line 52: Line 50:
=== Global configuration ===
=== Global configuration ===


The way to enable flow monitoring&nbsp;and flow exports will vary from vendor to vendor.&nbsp; Basically you will have to configure&nbsp;these parameters:
The way to enable flow monitoring&nbsp;and flow exports will vary from vendor to vendor.&nbsp; In general there are two major portions.&nbsp; In Global Setting&nbsp;you set parameters on the flow and the export mechanism itself.&nbsp; Interface Settings let&nbsp;you decide where to deploy sensors so that all flows through the device are accounted for.


==== Destination ====
==== Destination ====


IP address of the collector
*IP address of the collector  


==== Port ====
==== Port ====


UDP port where the collector is listening
*UDP port where the collector is listening on


==== Active flow timeout ====
==== Active flow timeout ====


Interval to send flow record updates in cse of&nbsp;a long duration flow. This setting should match the smallest granularity of the database on the collector.&nbsp; In case of TruView, this is one&nbsp;minute.
*Interval at which flow record updates are send in case of&nbsp;long flows. A flow is considered long when it exceeds the smallest database granularit. Consequently, this setting should match the smallest granularity of the database on the collector.&nbsp; In most cases, this is one minute or&nbsp;60 seconds.&nbsp;


==== Inactive flow timeout ====
==== Inactive flow timeout ====


Interval of inactivity (no packets) that marks a flow as inactive. The recommended setting is 15 seconds.
*Interval of inactivity (no packets) that marks a flow as inactive. The recommended setting is 15 seconds.  


==== Flow record format ====
==== Flow record format ====


With the introduction of CISCO's Flexible NetFlow&nbsp;and the standardization of&nbsp;IPFIX, the PDU of a flow record is no longer uniformely defined.&nbsp;The fields contained in the PDU should reflect the reporter's capabilities. In case of TruView the flow record must contain at least the following information:
With the introduction of CISCO&nbsp;Flexible NetFlow and the standardization of&nbsp;IPFIX, the PDU of a flow record is no longer uniquely defined.&nbsp;The fields contained in the PDU should match the reporter's capabilities of reporting on the value represented in that field.&nbsp; These fields should be present at all times:


*source address  
*source address  
*destination addresstos
*destination address
*type of service
*protocol  
*protocol  
*source port  
*source port  
*destination port  
*destination port  
*input interface  
*input interface  
*output interface
*tcp flags  
*tcp flags  
*number of bytes (long format)
*number of bytes&nbsp;
*number of packets (long format)
*number of packets&nbsp;
*timestamp of the first packet of the flow (absolute time)
*timestamp of the first packet of the flow&nbsp;
*timestamp of the last&nbsp;packet of the flow (absolute time)
*timestamp of the last&nbsp;packet of the flow  
 
Additional fields may be required in case of [[Sampled_NetFlow|sampling]] or to help indicating&nbsp;the in and out interfaces on a L2 device.


&nbsp;
&nbsp;
Line 92: Line 94:
&nbsp;
&nbsp;


{{#invoke:Message box|ambox |type=info|text=If this is your first experience with configuring flow exports on network devices, we recommend you make a drawing of the device, its interfaces (physical and virtual) and every flow that goes through the device.}}
{{#invoke:Message box|ambox |type=info|text=If this is your first experience with configuring flow export on network devices, we recommend you make a drawing of the device, its interfaces (physical and virtual) and every flow that goes through the device.}}
 
&nbsp;
 
Basically, this is&nbsp;the science of deploying sensors so that <span style="background-color:#FFFF00">EVERY&nbsp;flow that goes through the device is metered once and&nbsp;</span><span style="background-color:#FFFF00">NO&nbsp;flow is metered more than once</span>.&nbsp;&nbsp;
 
==== Example 1: ingress sensor on all interfaces ====
 
Flows from user to network are&nbsp;metered by the ingress sensor on interface 0/0 while flows from network to user are&nbsp;metered by the ingress sensor on interface 0/1.&nbsp; If the device has more than two active interfaces, deploy ingress sensors on all these interface.&nbsp;
 
&nbsp;
 
[[File:2ArmedRouterNIUI.png|border|center|2ArmedRouterNIUI.png]]
 
&nbsp;
 
==== Example 2: egress sensor on all&nbsp;interfaces ====
 
Flows from user to network are&nbsp;metered by the egress sensor on interface 0/1 while flows from network to user are&nbsp;metered by the egress sensor on interface 0/0.&nbsp;If the device has more than two active interfaces, deploy egress sensors on all these interface.&nbsp;
 
&nbsp;
 
[[File:2ArmedRouterNEUE.png|border|center|2ArmedRouterNEUE.png]]
 
&nbsp;
 
==== Example 3 and 4: ingress&nbsp;and egress&nbsp;sensors on some interfaces ====
 
In example 3, flows from user to network are&nbsp;metered by the ingress/in sensor on interface 0/0.&nbsp; Flows from network to user are&nbsp;metered by the egress sensor on that same interface 0/0. This configuration is prefrerable when interface 0/1 is complex like a composed of a multitude of virtual interfaces.&nbsp; In some vendors it may also be required when the&nbsp;network side is a&nbsp;tunnel interface.&nbsp;
 
It is the method preferred by&nbsp;network service providers as they only "care" about the wide area interface.&nbsp; In that case, sensors will be deployed on the network side of the router.&nbsp;&nbsp;
 
As a general rule of thumb, this method should not be used on more than one interfaces when these are carying the same flow.&nbsp;&nbsp;
 
&nbsp;
 
[[File:2ArmedRouterUIE.png|border|center|2ArmedRouterUIE.png]]
 
[[File:2ArmedRouterNIE.png|border|center|2ArmedRouterNIE.png]]
 
&nbsp;
 
=== Cisco NetFlow Configuration Example ===


&nbsp;
&nbsp;


Basically, this is&nbsp;the science of deploying sensors so that EVERY<span style="background-color:#FFFF00;">&nbsp;flow that goes through the FED is metered</span>.&nbsp; It is equally important to make sure <span style="background-color:#FFFF00;">NO&nbsp;flow is metered more than once</span>.&nbsp;&nbsp;
[[File:CiscoFlowExample.png|border|center|CiscoFlowExample.png]]


&nbsp;
&nbsp;


=== Cisco configuration example ===
This is how Flexible NetFlow configuration (without [[Sampled_NetFlow|sampling]]) would look on a CISCO 4000 Series ISR router with a recent IOS. The&nbsp;router has two physical interfaces (20 and 10Mbps) facing the network.&nbsp; At no point in time there is traffic between these two interfaces.&nbsp;
 
To avoid manual set-up of the interfaces in the collector, the collector must&nbsp;be able to discover the device's configuration using SNMP (read-only).&nbsp; Also, you need to set the correct speed (unspecified line rate or commited interface rate)&nbsp;for&nbsp;the circuits&nbsp;behind the physical interfaces to allow correct calculation of interface utilization.


This is how a Flexible NetFlow configuration would look like on a CISCO 3850 router. This router has two physical interfaces facing the network (WAN).&nbsp; Router configuration that is not part of NetFlow have&nbsp;been left out for clarity. Please note that in most cases the collector needs to be able to query the router by SNMP in order to "discover" the FED's configuration.
On devices handling a lot of traffic, you&nbsp;may be required to increase the size of the cache.&nbsp; This will consume additional memory on the device.
<pre>ip access-list standard SNMP-NS
permit 10.200.255.123
snmp-server community N3tsc0ut RO SNMP-NS


==== Global settings ====
<pre>ip access-list standard SNMP-RO
permit <IPCollector>
snmp-server community notsecure RO
!
!
!
flow record record-ns-1-in
flow record recordCFLOW
  match ipv4 tos
  match ipv4 tos
  match ipv4 protocol
  match ipv4 protocol
  match ipv4 source address
  match ipv4 source address
match ipv4 source mask
  match ipv4 destination address
  match ipv4 destination address
match ipv4 destination mask
  match transport source-port
  match transport source-port
  match transport destination-port
  match transport destination-port
  match interface input
  match interface input
collect interface output
collect routing source as
collect routing destination as
  collect transport tcp flags
  collect transport tcp flags
  collect counter bytes long
  collect counter bytes long
  collect counter packets long
  collect counter packets long
  collect timestamp absolute first
  collect timestamp sys-uptime first
  collect timestamp absolute last
  collect timestamp sys-uptime last
!
!
flow record record-ns-1-out
match ipv4 tos
match ipv4 protocol
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
match interface output
collect transport tcp flags
collect counter bytes long
collect counter packets long
collect timestamp absolute first
collect timestamp absolute last
!
!
!
!
flow exporter netscout-1
flow exporter exporterCFLOW
  destination 10.200.255.123
  destination <IPCollecor>
  source Vlan1301
  source <SourceInterfaceForExports>
  transport udp 2055
  transport udp 2055
  template data timeout 60
  template data timeout 300
option sampler-table timeout 60
!
 
!
flow monitor monitor-ns1-in
exporter netscout-1
cache timeout active 60
&nbsp;cache timeout inactive 14
record record-ns-1-in
!
!
!
!
flow monitor monitor-ns1-out
flow monitor monitorCFLOW
  exporter netscout-1
  exporter exporternfv9
cache entries 60000
  cache timeout active 60
  cache timeout active 60
&nbsp;cache timeout inactive 14
cache timeout inactive 15
  record record-ns-1-out
  record recordnfv9</pre>
!
 
!
==== Interface specific settings ====
interface GigabitEthernet1/1/1
<pre>interface GigabitEthernet1/1/1
&nbsp;bandwidth 20000
bandwidth 20000
  ip flow monitor monitor-ns1-in input
  ip flow monitor monitorCFLOW input  
  ip flow monitor monitor-ns1-out output
  ip flow monitor monitorCFLOW output  
!
!  
!
!
interface GigabitEthernet2/1/1
interface GigabitEthernet2/1/1
&nbsp;bandwidth 20000
bandwidth 10000
  ip flow monitor monitor-ns1-in input
  ip flow monitor monitorCFLOW input
ip flow monitor monitor-ns1-out output
ip flow monitor monitorCFLOW output</pre>
</pre>
 
&nbsp;
 
=== Fortinet NetFlow Configuration Example ===
 
Let's assume that the router in the previous example is now a Fortigate NG Firewall.&nbsp;&nbsp;
 
Fortinet defaults to NetFlow version 9.&nbsp; It&nbsp;can be configured for the firewall as a whole or per Virtual Domain.&nbsp; In this example&nbsp;we configure the firewall for flow exports.
 
==== Global settings ====
<pre>config system netflow
      set collector-ip <IPCollector>
      set collector-port 2055
      set source-ip <SourceInterfaceForExports>
      set active-flow-timeout 1
      set inactive-flow-timeout 15</pre>
 
==== Interface specific settings ====
<pre>config system interface
      edit wan1
&nbsp;            set inbandwidth 20000
&nbsp;            set outbandwidth 20000
            set netflow-sampler both
      end
config system interface
      edit wan2
&nbsp;            set inbandwidth 10000
&nbsp;            set outbandwidth 10000
            set netflow-sampler both
      end</pre>
 
&nbsp;

Latest revision as of 16:34, 24 November 2020

Introduction

Correct configuration of Flow Exporting Devices (FED) is key to Flow Based Network Analysis. Mistakes made during configuration will result in missing and duplicated data.  They are difficult to identify. Vendors of network equipment are not making it easy as many of them are using some sort of proprietary method to enable flow export on their devices.

Furthermore, Flow Collector (FC) and FED must allign on configuration parameters in order to interpret the records correctly.

This article creates a vendor independant framework to configure flow exports.  We hope it will help you to confgure your FED.

We ask you kindly to read the entire article and not to jump to the confuration examples towards the end.

For a complete overview of IP Flow Information eXchange (IPFIX), we recommend reading RFC 3917.

If you come aware of vendor specific issues thay may help others, please let us know.

 

Solution

Definitions

First, let us make sure we speak the same language.

Flow Enabled Device or Flow Exporter

Flow Enabled Device of Flow Exporter is a device configured to send flow records to one or more collectors.  This can be a L2 or L3 device.

Flow or IP Flow

Flow is formed by all packets between two TCP/IP sockets.  Therefore, what we call a TCP sessions consists of two flows: one flow from client to server and the second flow from server to client.

Flow record

Flow record contains information about a specific flow that was metered at an observation point.  A flow record contains measured properties aggregated for the flow.  Key record fields are source and destination IP addess, protocol, source and destination port, number of bytes, number of packets, duration and input and output interfaces.  There could be many more.

Sensor

Sensor is a metering point in the device. Sensors are associated with interfaces.  It will quantify and qualify packets in or out of that interface, leading to the creation of a flow record decribing the flow.  It is very important to realize a sensor will measure packets in one direction only (ingress or egress). 

Collector

Flow collector is a server application that receives the flow records and stores them in a database. In TruView, TruView Flow is the collector.  In nGeniusONE, the Flow Collector is the collector.  The sizeof the hardware will depend on the number of flows received per second and the aging time of the "raw" flow records.

Reporter

Flow reporter is a server application mining the data records, producing tables and charts and possibly alerts.  In TruView, TruView Central is the reporter.  In nGeniusONE, nGenius Server is the reporter.  When the number of flows received per second is low (<50k/s), collector and reporter can be installed on the same hardware.

 

Global configuration

The way to enable flow monitoring and flow exports will vary from vendor to vendor.  In general there are two major portions.  In Global Setting you set parameters on the flow and the export mechanism itself.  Interface Settings let you decide where to deploy sensors so that all flows through the device are accounted for.

Destination

  • IP address of the collector

Port

  • UDP port where the collector is listening on

Active flow timeout

  • Interval at which flow record updates are send in case of long flows. A flow is considered long when it exceeds the smallest database granularit. Consequently, this setting should match the smallest granularity of the database on the collector.  In most cases, this is one minute or 60 seconds. 

Inactive flow timeout

  • Interval of inactivity (no packets) that marks a flow as inactive. The recommended setting is 15 seconds.

Flow record format

With the introduction of CISCO Flexible NetFlow and the standardization of IPFIX, the PDU of a flow record is no longer uniquely defined. The fields contained in the PDU should match the reporter's capabilities of reporting on the value represented in that field.  These fields should be present at all times:

  • source address
  • destination address
  • type of service
  • protocol
  • source port
  • destination port
  • input interface
  • output interface
  • tcp flags
  • number of bytes 
  • number of packets 
  • timestamp of the first packet of the flow 
  • timestamp of the last packet of the flow

Additional fields may be required in case of sampling or to help indicating the in and out interfaces on a L2 device.

 

Interface configuration

 

 

Basically, this is the science of deploying sensors so that EVERY flow that goes through the device is metered once and NO flow is metered more than once.  

Example 1: ingress sensor on all interfaces

Flows from user to network are metered by the ingress sensor on interface 0/0 while flows from network to user are metered by the ingress sensor on interface 0/1.  If the device has more than two active interfaces, deploy ingress sensors on all these interface. 

 

2ArmedRouterNIUI.png
2ArmedRouterNIUI.png

 

Example 2: egress sensor on all interfaces

Flows from user to network are metered by the egress sensor on interface 0/1 while flows from network to user are metered by the egress sensor on interface 0/0. If the device has more than two active interfaces, deploy egress sensors on all these interface. 

 

2ArmedRouterNEUE.png
2ArmedRouterNEUE.png

 

Example 3 and 4: ingress and egress sensors on some interfaces

In example 3, flows from user to network are metered by the ingress/in sensor on interface 0/0.  Flows from network to user are metered by the egress sensor on that same interface 0/0. This configuration is prefrerable when interface 0/1 is complex like a composed of a multitude of virtual interfaces.  In some vendors it may also be required when the network side is a tunnel interface. 

It is the method preferred by network service providers as they only "care" about the wide area interface.  In that case, sensors will be deployed on the network side of the router.  

As a general rule of thumb, this method should not be used on more than one interfaces when these are carying the same flow.  

 

2ArmedRouterUIE.png
2ArmedRouterUIE.png
2ArmedRouterNIE.png
2ArmedRouterNIE.png

 

Cisco NetFlow Configuration Example

 

CiscoFlowExample.png
CiscoFlowExample.png

 

This is how Flexible NetFlow configuration (without sampling) would look on a CISCO 4000 Series ISR router with a recent IOS. The router has two physical interfaces (20 and 10Mbps) facing the network.  At no point in time there is traffic between these two interfaces. 

To avoid manual set-up of the interfaces in the collector, the collector must be able to discover the device's configuration using SNMP (read-only).  Also, you need to set the correct speed (unspecified line rate or commited interface rate) for the circuits behind the physical interfaces to allow correct calculation of interface utilization.

On devices handling a lot of traffic, you may be required to increase the size of the cache.  This will consume additional memory on the device.

Global settings

ip access-list standard SNMP-RO
 permit <IPCollector>
snmp-server community notsecure RO 
!
!
flow record recordCFLOW
 match ipv4 tos
 match ipv4 protocol
 match ipv4 source address
 match ipv4 source mask
 match ipv4 destination address
 match ipv4 destination mask
 match transport source-port
 match transport destination-port
 match interface input
 collect interface output
 collect routing source as
 collect routing destination as
 collect transport tcp flags
 collect counter bytes long
 collect counter packets long
 collect timestamp sys-uptime first
 collect timestamp sys-uptime last
!
!
flow exporter exporterCFLOW
 destination <IPCollecor>
 source <SourceInterfaceForExports>
 transport udp 2055
 template data timeout 300
!
!
flow monitor monitorCFLOW
 exporter exporternfv9
 cache entries 60000
 cache timeout active 60
 cache timeout inactive 15
 record recordnfv9

Interface specific settings

interface GigabitEthernet1/1/1
 bandwidth 20000
 ip flow monitor monitorCFLOW input 
 ip flow monitor monitorCFLOW output 
! 
!
interface GigabitEthernet2/1/1
 bandwidth 10000
 ip flow monitor monitorCFLOW input
 ip flow monitor monitorCFLOW output

 

Fortinet NetFlow Configuration Example

Let's assume that the router in the previous example is now a Fortigate NG Firewall.  

Fortinet defaults to NetFlow version 9.  It can be configured for the firewall as a whole or per Virtual Domain.  In this example we configure the firewall for flow exports.

Global settings

config system netflow
      set collector-ip <IPCollector>
      set collector-port 2055
      set source-ip <SourceInterfaceForExports>
      set active-flow-timeout 1
      set inactive-flow-timeout 15

Interface specific settings

config system interface
       edit wan1
             set inbandwidth 20000
             set outbandwidth 20000
             set netflow-sampler both
       end
config system interface 
       edit wan2 
             set inbandwidth 10000
             set outbandwidth 10000
             set netflow-sampler both
       end