PFQ I/O

pfq-logo

What is PFQ?

PFQ is a functional framework designed for the Linux operating system built for efficient packets capture/transmission (10G, 40G and beyond), in-kernel functional processing, kernel-bypass and packets steering across groups of sockets/end-points.

It is highly optimized for multi-core architecture, as well as for network devices equipped with multiple hardware queues. Compliant with any NIC, it provides a script that generates accelerated network device drivers starting from the source code.

PFQ enables the development of high-performance network applications, and it is shipped with a custom version of libpcap that accelerate and parallelize legacy applications. Besides, a pure functional language designed for early stages in-kernel packet processing is included: pfq-lang.

Pfq-Lang is inspired by Haskell and is intended to define applications that run on top of network device drivers. Through pfq-lang it is possible to build efficient bridges, port mirrors, simple firewalls, network balancers and so forth.

The framework includes the source code of the PFQ kernel module, user-space libraries for C, C++11-14, Haskell language, an accelerated pcap library, an implementation of pfq-lang as eDSL for C++/Haskell, an experimental pfq-lang compiler, and a set of diagnostic tools.

Features

Data-path with a full lock-less architecture.
Preallocated pools of socket buffers.
Compliant with a plethora of network devices drivers.
Rx and Tx line-rate on 10-Gbit links (14,8 Mpps), on-top-of Intel ixgbe vanilla drivers.
Kernel threads for asynchronous packets transmission.
Transmission with active timestamping.
Groups of sockets for multi-threaded applications.
Per-group packet steering through randomized hashing or deterministic classification.
Per-group Berkeley and VLAN filters.
User-space libraries for C, C++11-14 and Haskell language.
Functional engine for in-kernel packet processing with pfq-lang.
pfq-lang eDLS for both C++11-14 and Haskell language.
pfq-lang experimental compiler.
Accelerated pcap library for legacy applications (line-speed tested with captop).
Memory-mapped I/O between user and kernel on top of HugePages.
pfq-omatic script to build accelerated vanilla network drivers.
pfqd daemon used to configure and parallelize (pcap) legacy applications.

Performance

PFQ performance highly depends on the hardware used.

Running on top of a Xeon processor equipped and the Intel 82599 10G controller, PFQ can process, steer to user-space and transmit the line speed (~14,8 Million packets per second) using 2 or 3 kernel threads.

The following link shows the configuration file and the performance measured running PFQ on top of Intel 82599 10/20G NICs.

Releases

The current stable release is the master branch (v6.1).

Source code

The package provides the source code of the PFQ kernel module, user-space libraries for the C, C++11 and Haskell language, utilities, scripts a set of diagnostic tools.

Install

Linux kernel headers (3.x/4.x), gcc/g++ 4.7 (or higher), GHC 7.8 (or 7.10), alex, happy, CMake and make tool are required for the compilation.

The guideline covers the principal problems of compilation and installation.

Wiki

New to PFQ programming? The wiki pages can get you started quickly.

Publications

"PFQ: a Novel Engine for Multi-Gigabit Packet Capturing With Multi-Core Commodity Hardware" Best-Paper-Award at PAM2012

"A Purely Functional Approach to Packet Processing" ANCS 2014 Conference - Marina del Rey (LA)

"Network Traffic Processing with PFQ" JSAC-SI-MT/IEEE journal Special Issue on Measuring and Troubleshooting the Internet (April 2016)

"Enabling Packet Fan--Out in the libpcap Library for Parallel Traffic Processing" Network Traffic Measurement and Analysis Conference (TMA 2017)

"A Pipeline Functional Language for Stateful Packet Processing" IEEE International Workshop on NEtwork Accelerated FunctIOns (NEAF-IO '17)

"The Acceleration of OfSoftSwitch" IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN '17)

Invited Talks

"Functional Network Programming" at Tyrrhenian International Workshop on Digital Communication - (Sep. 2016)

"Software Accelerations for Network Applications" at NetV IRISA / Technicolor Workshop on Network Virtualization (Feb. 2017)

Licenses

The PFQ kernel module is distributed with GPL license. User-space libraries and tools, instead, are distributed with a dual license scheme, so they are available either as GPL or with a Commercial License. For more information contact the author.

Author

Nicola Bonelli nicola@pfq.io