JKFlow.pm, the XML-configurable Flowscan module

What is JKFlow.pm?

JKFlow is an easy XML configurable Flowscan module for analyzing flowfiles exported with NetFlow from Cisco routers. In JKFlow you can isolate network parts using "directions", using source- and destination subnets/sites or routergroups. Using these directions you can isolate parts of the captured netflow to measure several parameters like total, protocols, services, etc... The design makes JKFlow perfect for branch site WAN-traffic monitoring. This module can serve as a basis of your network monitoring/billing infrastructure.

Example schema 1:

This scheme shows the principe behind JKFlow. We have 3 offices connected with routers over the Internet. Each site is defined using a subnet and contains 2 further subnets: subnet desktops and subnet servers. We want to monitor server related traffic between the servers and to monitor applications between the desktops. JKFlow defines these entities: Sites are defined as a part of the network using (no-)subnets. Directions are defined as sections using source and destination subnets or sites, for specifying flows to monitor. Inside each direction we define services, protocols, applications, tos, total traffic monitoring. Appications are groups of services, like web=http+https and mail=pop+imap+smtp. Also you can use of definesets in directions, to template and reuse services, subnets, applications, tos and total. In every direction we can define different traffic monitoring. Inside every direction we collect RRDTool-data. We can also define scoreboarding based on IP-address or port inside each direction, which are reported in html-files. All the collected RRDTool-data is accessable using a CGI-script JKGrapher, which produces graphs.

Example schema 2:

This scheme shows the new abilities of JKFlow3. You can define sections of routers or router interfaces called routergroups and monitor traffic passing over directions associated with these routergroups. You can even monitor traffic of a specific direction of source and destination subnets over these routergroups. The best of all it that even in this scenario maximum benefit of Net::Patricia subnet lookup speed was maintained.

Is it fast?

Yes!, JKFlow is evolved through many versions and much effort is put make this module both flexible and fast:

It uses C coded Net::Patricia subnet matching module for fast subnet lookups.
It uses Hash-lookups, which are effective. The access-time of a hash-entry is invariant to the amount of entries. Used during counting of multiple protocols/services, this will result in no speed penalty.
It uses a single source/destination inbound+outbound Net::Patricia match per flow. Returning all matching subnets, included and excluded, all matching directions are found via a hash entry. Using more directions won't slow parsing flows much.
It creates for every direction it's own autogenerated evaluation subroutine Every direction can have it's own set of services/protocols/ftp/total/tos to monitor. Apart from it's great flexibility, the several features won't slow parsing, because a unused feature will not be included in the autogenerated code. No other report module use this technique!
Even the wanted subroutine is autogenerated! When you only want to measure overall traffic, only the overall counting function is generated in the wanted subroutine for evaluating flows.
Direction-matching is adapted to best If you create directions based only on source/destination subnets/sites, the code for evaluation of route exporters is not used in the wanted subroutine. When creating directions based on route exporters only, it won't include any Net::Patricia matching code.

Demo:

This Flash animation (4 Meg download) shows a demo of the JKGrapher CGI-interface, showing traffic over 2 interfaces on a router (located on sourceforge)
Flash Demo JKFlow

Needed skills:

You need to know how to install and run FlowScan, located on the Caida website. All configuration is done with a XML-file. If you understand some XML and know the configuration rules, you can configure any monitoring configuration you want. Some little Perl debugging/programming knowledge would be a advantage.

Status:

-JKFlow version 3.5.2 is the latest version and includes aggregated scoreboarding.
-Older JKflow 1, 2 are available but not recommended anymore, because of missing features, confusing configuration, lack of speed etc...
-the new SMP version are still a bit buggy, most synchonisation problems.

Prerequisites:

JKFlow.pm needs Net::Patricia for fast subnet lookup and XML::Simple for
configfile parsing

Legal:

JKFlow was developed started from CUFlow and is GPL'ed.

Documentation:

Introduction Brochure JKFlow
The Manual of JKFlow (08/09/2005) (recommended reading!):
-NetFlow and flow monitoring issues.
-Design of FlowScan flow reporing architecture.
-Installation of FlowScan in Linux/Solaris.
-Comparisons with different FlowScan reporting modules.
-Argumentation / internals JKFlow design.
-Configuration rules JKFlow 3.5.1.
The internal JKFlow::mylist datastructure layout.
Some example configurations for JKFlow v3.4.
This is a powerpoint presentation that I gave on Groep T Leuven.

Screenshots:

FlowScan running JKFlow.pm
FlowScan running JKFlow.pm (2)
Daily Windows syncronisation
Daily FTP syncronisation
Hourly Router statistics query
6-hourly Exchange replication
Traffic profile of a subnet
Attack of the Slapper & Code Red worm
Interface monitoring of an inbound FTP-session
Interface monitoring of the same FTP-session outbound
JKGrapher in JKFlow 3.1 produces thin style graphs
Flowscan + JKFlow 3.2.2 on a Multiprocessor Machine
Menu JKGrapher.pl 3.5
The scoreboard directories, + scoreboardfile 3.5
Part of datastructure showing scoreboarding

Download:

JKFlow3 branch:

Automatic Install Script:

-Downloads and installs FlowScan 1.006, flow-tools-0.67, RRDTool 1.2.11 & JKFlow 3.5.1 (development, 20/11/2005)
-Download the script, chmod +x flowscan_install.sh, execute ./flowscan_install.sh
-Tested in Centos 4.2 (compiling flow-tools fails in Fedora 4 due to beta gcc compiler)
-Warning: This script disables SELinux
Beta flowscan_install.sh

Version 3.5.2 (stable, 9/02/2006):

Fixes:
-Bugfix in combination of routergroups and Autonomous Systems in directions.
-Bugfix in DSCP for VoIP traffic.
-Removed hardcoded Scorekeep and Numkeep attributes.
-Updated directory creation code to only create directories when needed.
Change:
-Changed ordering in/out direction in scoreboards.
Stable jkflow-v3.5.2.tgz

Version 3.5.1 (development, 11/11/2005):

Added features:
-DSCP values
-Absolute values in JKGrapher
-Defining custom tuples in scoreboarding
-Added flow-tools 0.67 patch for gcc 3.4
-Added AS-support
Fixes:
-Division by Error bug
-Numkeep incorrect implemented max number of tuples to keep
-Correction in example JKFlow.xml
Beta jkflow-v3.5.1.tgz

Version 3.4 (stable, 7/01/2005):

Added features:
-Updating link to latest individual scoreboard
-Aggregated scoreboarding, added <report/> element
-Aggregated scoreboarding possible without individual flowfile scoreboarding
-Added 'other' direction for services not reported by other directions.
-Added sampletime element.
BugFix:
-Warn bug in 3.3.2 fixed.
jkflow-v3.4.tgz
multiprocessor jkflow-v3.4.smp.tgz

Version 3.3.2 (stable, 25/07/2004):

Added features:
-<otherservices/> and <otherprotocols/> elements.
-Now only 1 single src/dst Net::Patricia matching is done for both inbound+outbound direction.
-<scoreboardother/> will report hosts/ports not included in <services> and <ftp/>.
You can see this as a (very) basic NIDS. This feature will be improved in the future.
Changes:
-scoreboarding is now done in daily, not hourly directories.
Bugfixes:
-Broken generateCountPackets generated code 3.3.1 (wrong handling missing <ftp/> element).
-Broken monitor attribute in 3.3.1.
-Handling missing protocols/services names in /etc/protocols, /etc/services.
-Broken <all> resolved.
-Other (now otherservices) not excluding ftp.
jkflow-v3.3.2.tgz
multiprocessor jkflow-v3.3.2.smp.tgz

Version 3.2.2 (19/06/2004):

Bugfix:
-Bug which prevented using routergroup with localsubnets used in directions with subnets/sites + routergroup attributes.
-Recommended upgrade from version 3.0, 3.1, 3.2
jkflow-v3.2.2.tgz
multiprocessor jkflow-v3.2.2.smp.tgz

JKFlow2 branch (old):

Version 03/11/2003:

Added features:
-Scoreboard host & port reporting
-The xml element <scoreboard/> has 2 new attributes called "hosts" and "ports",
-specifying with the value "1" enables the host and port reporting functionality
-The host & port reports will be written in separate directories.
jkflow2-v03112003.tgz

JKFlow1 branch (old):

Version 02/06/2003:

Added features:
-Improvement in localsubnets, less Net::Patricia->match_integer calls. -Removed unused Multicast-Tos monitoring. Localsubnets are mandatory for all and routers. Migrated countmulticasts into countpackets. The performance should be beter. Avoiding including totals of directions inside selected router/subnet/direction totals.
jkflow-v02062003.tgz

If you have JKFlow running on your network, I really would like to know the size of your network, your operating system,CPU,speed and the size/structure of your configuration file, the number of flows and the time it takes to process it. Flowfiles are appreciated too. Please send it to jurgen.kobierczynski (at) pandora.be . Thanks!

CVS:

Browse the cvs-tree

FAQ:

faq.html
Post questions / Browse the mailinglist
SourceForge.net Logo
This project is hosted on sourceforge, Visit my homepage