What is JKFlow.pm?JKFlow is an easy XML configurable Flowscan module for analyzing flowfiles exported with NetFlow from Cisco routers. In JKFlow you can isolate network parts using "directions", using source- and destination subnets/sites or routergroups. Using these directions you can isolate parts of the captured netflow to measure several parameters like total, protocols, services, etc... The design makes JKFlow perfect for branch site WAN-traffic monitoring. This module can serve as a basis of your network monitoring/billing infrastructure.
Example schema 1:
This scheme shows the principe behind JKFlow. We have 3 offices connected with routers over the Internet. Each site is defined using a subnet and contains 2 further subnets: subnet desktops and subnet servers. We want to monitor server related traffic between the servers and to monitor applications between the desktops. JKFlow defines these entities: Sites are defined as a part of the network using (no-)subnets. Directions are defined as sections using source and destination subnets or sites, for specifying flows to monitor. Inside each direction we define services, protocols, applications, tos, total traffic monitoring. Appications are groups of services, like web=http+https and mail=pop+imap+smtp. Also you can use of definesets in directions, to template and reuse services, subnets, applications, tos and total. In every direction we can define different traffic monitoring. Inside every direction we collect RRDTool-data. We can also define scoreboarding based on IP-address or port inside each direction, which are reported in html-files. All the collected RRDTool-data is accessable using a CGI-script JKGrapher, which produces graphs.
Example schema 2:
This scheme shows the new abilities of JKFlow3. You can define sections of routers or router interfaces called routergroups and monitor traffic passing over directions associated with these routergroups. You can even monitor traffic of a specific direction of source and destination subnets over these routergroups. The best of all it that even in this scenario maximum benefit of Net::Patricia subnet lookup speed was maintained.
Is it fast?
Yes!, JKFlow is evolved through many versions and much effort is put make this module both
flexible and fast:
This Flash animation (4 Meg download) shows a demo of the JKGrapher
CGI-interface, showing traffic over 2 interfaces on a router
(located on sourceforge)
Needed skills:You need to know how to install and run FlowScan, located on the Caida website. All configuration is done with a XML-file. If you understand some XML and know the configuration rules, you can configure any monitoring configuration you want. Some little Perl debugging/programming knowledge would be a advantage.
Status:-JKFlow version 3.5.2 is the latest version and includes aggregated scoreboarding.
-Older JKflow 1, 2 are available but not recommended anymore, because of missing features, confusing configuration, lack of speed etc...
-the new SMP version are still a bit buggy, most synchonisation problems.
Prerequisites:JKFlow.pm needs Net::Patricia for fast subnet lookup and XML::Simple for
Legal:JKFlow was developed started from CUFlow and is GPL'ed.
Documentation:Introduction Brochure JKFlow
The Manual of JKFlow (08/09/2005) (recommended reading!):
-NetFlow and flow monitoring issues.
-Design of FlowScan flow reporing architecture.
-Installation of FlowScan in Linux/Solaris.
-Comparisons with different FlowScan reporting modules.
-Argumentation / internals JKFlow design.
-Configuration rules JKFlow 3.5.1.
The internal JKFlow::mylist datastructure layout.
Some example configurations for JKFlow v3.4.
This is a powerpoint presentation that I gave on Groep T Leuven.
Screenshots:FlowScan running JKFlow.pm
FlowScan running JKFlow.pm (2)
Daily Windows syncronisation
Daily FTP syncronisation
Hourly Router statistics query
6-hourly Exchange replication
Traffic profile of a subnet
Attack of the Slapper & Code Red worm
Interface monitoring of an inbound FTP-session
Interface monitoring of the same FTP-session outbound
JKGrapher in JKFlow 3.1 produces thin style graphs
Flowscan + JKFlow 3.2.2 on a Multiprocessor Machine
Menu JKGrapher.pl 3.5
The scoreboard directories, + scoreboardfile 3.5
Part of datastructure showing scoreboarding
Automatic Install Script:-Downloads and installs FlowScan 1.006, flow-tools-0.67, RRDTool 1.2.11 & JKFlow 3.5.1 (development, 20/11/2005)
-Download the script, chmod +x flowscan_install.sh, execute ./flowscan_install.sh
-Tested in Centos 4.2 (compiling flow-tools fails in Fedora 4 due to beta gcc compiler)
-Warning: This script disables SELinux
Version 3.5.2 (stable, 9/02/2006):Fixes:
-Bugfix in combination of routergroups and Autonomous Systems in directions.
-Bugfix in DSCP for VoIP traffic.
-Removed hardcoded Scorekeep and Numkeep attributes.
-Updated directory creation code to only create directories when needed.
-Changed ordering in/out direction in scoreboards.
Version 3.5.1 (development, 11/11/2005):Added features:
-Absolute values in JKGrapher
-Defining custom tuples in scoreboarding
-Added flow-tools 0.67 patch for gcc 3.4
-Division by Error bug
-Numkeep incorrect implemented max number of tuples to keep
-Correction in example JKFlow.xml
Version 3.4 (stable, 7/01/2005):Added features:
-Updating link to latest individual scoreboard
-Aggregated scoreboarding, added <report/> element
-Aggregated scoreboarding possible without individual flowfile scoreboarding
-Added 'other' direction for services not reported by other directions.
-Added sampletime element.
-Warn bug in 3.3.2 fixed.
Version 3.3.2 (stable, 25/07/2004):Added features:
-<otherservices/> and <otherprotocols/> elements.
-Now only 1 single src/dst Net::Patricia matching is done for both inbound+outbound direction.
-<scoreboardother/> will report hosts/ports not included in <services> and <ftp/>.
You can see this as a (very) basic NIDS. This feature will be improved in the future.
-scoreboarding is now done in daily, not hourly directories.
-Broken generateCountPackets generated code 3.3.1 (wrong handling missing <ftp/> element).
-Broken monitor attribute in 3.3.1.
-Handling missing protocols/services names in /etc/protocols, /etc/services.
-Broken <all> resolved.
-Other (now otherservices) not excluding ftp.
Version 3.2.2 (19/06/2004):Bugfix:
-Bug which prevented using routergroup with localsubnets used in directions with subnets/sites + routergroup attributes.
-Recommended upgrade from version 3.0, 3.1, 3.2
JKFlow2 branch (old):
Version 03/11/2003:Added features:
-Scoreboard host & port reporting
-The xml element <scoreboard/> has 2 new attributes called "hosts" and "ports",
-specifying with the value "1" enables the host and port reporting functionality
-The host & port reports will be written in separate directories.
JKFlow1 branch (old):
Version 02/06/2003:Added features:
-Improvement in localsubnets, less Net::Patricia->match_integer calls. -Removed unused Multicast-Tos monitoring. Localsubnets are mandatory for all and routers. Migrated countmulticasts into countpackets. The performance should be beter. Avoiding including totals of directions inside selected router/subnet/direction totals.
If you have JKFlow running on your network, I really would like to know
the size of your network, your operating system,CPU,speed and the size/structure
of your configuration file, the number of flows and the time it takes
to process it. Flowfiles are appreciated too. Please send it to jurgen.kobierczynski (at) pandora.be
CVS:Browse the cvs-tree
Post questions / Browse the mailinglist
This project is hosted on sourceforge, Visit my homepage