Publication Details
Low Overhead Distributed IP Flow Records Collection and Analysis
Žádník Martin, Ing., Ph.D. (DCSY)
NetFlow, IPFIX, IP flow collector, distributed system, parallel computing,
Hadoop, big data
Collection and analysis of IP flow records belong to a class of data-intensive
tasks, the class for which big data analytics systems should be effective.
Several Hadoop-based solutions for network traffic processing exist but are
generally suitable only for truly big data, otherwise the disadvantages of Hadoop
dominate. In this work, we present a distributed platform for IP flow records
collection and analysis together with a reference implementation. It focuses on
smaller clusters, has low overhead, allows interactive work, and exploits the
prospects of distributed systems like high throughput and scalability.
Experiments show low query latency and linear scalability with respect to the
growth of both amount of work and computer cluster. Extensions for data mining
and machine learning are easy to include and are already work in progress.
Moreover, the whole software stack is open-source.
@inproceedings{BUT161793,
author="Jan {Wrona} and Martin {Žádník}",
title="Low Overhead Distributed IP Flow Records Collection and Analysis",
booktitle="2019 IFIP/IEEE International Symposium on Integrated Network Management",
year="2019",
pages="557--562",
address="Washington DC",
isbn="978-3-903176-15-7",
url="https://ieeexplore.ieee.org/document/8717873"
}