Metadata export

Cubro SG allows to export IPFIX data to third-party systems. It supports any ipfix collector with template customization or ipfixreceiver2 utility which is already included to the main package. The utility allows to export received data into text format for next processing and inserting data into the database, as well as further copy and re-exporting data via ipfix protocol to multiply destinations.

Cubro SG supports 4 type of IPFIX channels:

• fullflow
• clickstream
• metadata
• extended metadata
• dns

Fullflow — ipfix flow contains information about connections pass through DPI, full session statistics and enriched DPI information (dpi protocol, subscriber information: login if exists).

Clickstream — ipfix flow contains information about subscriber’s visits to web pages (HTTP, HTTPS, QUIC).

Metadata — ipfix flow contains fields specified for protocols SIP, XMPP, MAIL (POP, IMAP, SMTP), FTP.

Extended (Raw) metadata — ipfix flow contains raw truncated IP packets for some protocols like STUN sequences and voip control protocol sessions. DPI sends raw data to LI-subsystem to postprocess data if needed.

DNS — ipfix flow contains all domain name service requests.

IPFIX export template for IPv6.
IPv6 template excluded fields:

• SOURCE_IPV4_ADDRES
• DESTINATION_IPV4_ADDRES
• POST_NAT_SOURCE_IPV4_ADDRESS
• POST_NAT_SOURCE_TRANSPORT_PORT

The following fields are present:

IPFIX export template for IPv6.
The format of IPFIX templates for IPV6 differs in the format of the IP_SOURCE and IP_DESTINATION fields.

Note:
LOCKED = 1 — blocked by HTTPS, 2 — HTTP redirect, 3 — blocked by HTTP (bitmask)
HOST_TYPE = 1 — HTTP, 2 — CNAME, 3 — SNI, 4 — QUIC
METHOD = 1 — GET, 2 — POST, 3 — PUT, 4 — DELETE
If the http_parse_reply=1 setting is enabled, information from responses to requests will be additionally transmitted. You can bind them to the responses by session identifier SESSION_ID.

If the ssl_parse_reply=1 setting is enabled, information from the responses to requests will be sent in addition. You can link bind to the responses by session ID SESSION_ID, taking into account the order of the responses.

Note:
IP_SRC — IP_SOURCE.
IP_DST — IP_DESTINATION.
GATEWAYS — comma separated list of gateways (IP or hostname).

Note: field “MODE” contains the type of FTP connection: (0 — active, 1 — passive).

Note: IM_PROTOCOL field contains the type of protocol used: 0 — ICQ, 7 — XMPP, 106 — ZELLO

Note: EVENT field indicates the type of event: 1 — send, 2 — receive,
ATTACHMENT — sign of attachment: mail_protocol = 0 — smtp, 1 — pop3, 2 — imap

Note:
FLW_DIR — packet direction by interface: 0: sub — inet, 1: inet — subs.
DIR_DATA — packet direction by session: for TCP 0: client → server, 1: server → client, for UDP — from whom the first packet is recorded, the first packet is considered to be a client.
VDPI_PROTO — DPI protocol
META_PROTO — internal protocol identifier (3 — SIP, 4 — FTP, 5 — SMTP, 6 — POP3, 7 — IMAP, 8 — XMPP, 9 — ICQ, 10 — RSS, 11 — NNTP, 12 — H323, 13 — ZELLO, ETC).
RAW_DATA — Truncated IP packets.

Customer can write metadata of HTTP, SSL/TLS, SIP, DNS to the txt file local or remote system. Customer have select fields of protocol to storing in a file and set save path on the server.

ajb_save_url activate recording of HTTP metadata
ajb_url_path directory to place files with these records (/var/dump/dpi by default)
ajb_url_ftimeout recording frequency by default 30 seconds
ajb_save_url_format list of metadata to record, should be separated by colon.

ajb_save_url=-1
ajb_save_url_format=ts:prg:login:ipsrc:ipdst:host:path:ref:uagent:cookie:tphost:blockd:method
ajb_url_path=/var/dump/dpi
ajb_url_ftimeout=30

Possible options for ajb_save_url_format parameter

ts	time stamp 
prg	id of the active services at the moment of request 
login	subscriber's login 
ipsrc	subscriber's IP address 
ipdst	host IP address (that of the request's addressee) 
host	the host name (Host field) 
path	the path to the requested resource (URI) 
ref	where from (Referer field) 
uagent	browser's type (User-Agent field) 
cookie	Cookie 
ssid	session ID (for binding with Netflow/IPFIX volume data) 
tphost	data type of Host (HTTP=1/CNAME=2/SNI=3/QUIC=4) 
blockd	bit mask, sign of blocking/forwarding (0x3 - for HTTP, 0x1 - for others) 
method	method 1 - GET, 2 - POST, 3 - PUT, 4 - DELETE

ajb_save_ssl activate recording of SSL metadata
ajb_ssl_path directory to place files with records (/var/dump/dpi by default)
ajb_ssl_ftimeout recording frequency by default 30 seconds
ajb_save_ssl_format list of metadata to record, should be separated by colon.

ajb_save_ssl is a mask:
0 - not saved
1 - sni (SSL)
2 - cname
3 - sni (QUIC)
-1 - to record everything

ajb_save_ssl=-1
ajb_save_ssl_format=ts:prg:login:ipsrc:ipdst:host:tphost:blockd:method
ajb_ssl_path=/var/dump/dpi
ajb_ssl_ftimeout=30

Possible options for ajb_save_ssl_format parameter

ts	time stamp 
prg	id of the active services at the moment of request 
login	subscriber's login ipsrc subscriber's IP address 
ipdst	host IP address (that of the request's addressee) host the host name (Host/CNAME/SNI/QUIC field) 
path	the path to the requested resource (URI) 
ref	where from (Referer field) 
uagent	browser's type (User-Agent field) 
cookie	Cookie 
ssid	session ID (for binding with Netflow/IPFIX volume data) 
tphost	data type of Host (HTTP=1/CNAME=2/SNI=3/QUIC=4) 
blockd	bit mask, sign of blocking/forwarding (0x3 - for HTTP, 0x1 - for others) 
method	method 1 - GET, 2 - POST, 3 - PUT, 4 - DELETE

ajb_save_sip activate recording of SIP metadata
ajb_sip_ftimeout recording frequency by default 30 seconds
ajb_sip_path directory to place files with records (/var/dump/dpi by default)
ajb_save_sip_format list of metadata to record, should be separated by colon.

ajb_save_sip=1 
ajb_sip_ftimeout=15 
ajb_sip_path=/home/sip ajb_save_sip_format=ts:ssid:ipsrc:ipdst:login:msg:scode:from:to:callid:uagent

Possible options for ajb_save_sip_format parameter

ts	time stamp 
ssid	session identifier (it's used to link to Netflow/IPFIX data to get bytes volume) 
ipsrc	subscriber’s IP ipdst server IP
login	subscriber’s LOGIN (from RADIUS) 
msg	message type
scode	status-code 
from	phone/identifier of calling party 
to	phone/identifier of called party 
callid	call identifier 
uagent	type of handset (User-Agent)

ajb_save_dns — flag for writing to a text file
ajb_dns_ftimeout — timeout (minutes) for switching to the next file
ajb_dns_bufsize — file write buffer
ajb_dns_fsize — file size limit
ajb_dns_path — path where to write

ajb_save_dns_format: format for writing to a text file
"ts" — time
"ipsrc" — ip source
"ipdst" — ip destination
"ssid" — session id
"login" — understandable
"host" — the name of which the information was requested
"rrtype" — RR types
"rrclass" — RR class
"ttl" — TTL
"rdlen" — rdata size
"rdata" — the resource itself
"psrc" — port source
"pdst" — port destination
"transport" — how the DNS query was received.

Default: ts:ssid:login:ipsrc:ipdst:psrc:pdst:transport:host:rrtype:rrclass:ttl:rdlen:rdata