py-idstools 
py-idstools is a collection of Python libraries for working with IDS
systems (typically Snort and Suricata).
Included Programs
- rulecat - Basic Suricata rule management tool suitable as a
replacement for for Oinkmaster and Pulled Pork.
- eve2pcap - Convert packets and payloads in eve logs to pcap.
- u2json - Convert unified2 files or spool directories to JSON.
- gensidmsgmap - Easily create a sid-msg.map file from rule files,
directories or a rule tarball.
- dumpdynamicrules - Helper for dumping Snort SO dynamic rule stubs.
- u2eve - Convert unified2 files to EVE compatible JSON.
Library Features
- Snort/Suricata unified2 log file parsing.
- Continuous unified2 directory spool reading with bookmarking.
- Snort/Suricata rule parser.
- Parser and lookup maps for classification.config.
- Parser and lookup maps for gen-msg.map and sid-msg.map.
Requirements
- Python 2.7 or newer.
- Currently only tested on Linux.
Installation
Latest Release (Recommended)
pip install idstools
or on Fedora and CentOS (with EPEL):
yum install python-idstools
Latest from Git
pip install https://github.com/jasonish/py-idstools/archive/master.zip
Manually
The idstools programs do not have to be installed to be used, they can
be executable directly from the archive directory:
./bin/idstools-rulecat
Or to install manually:
python setup.py install
Examples
Reading a Unified2 Spool Directory
The following code snippet will "tail" a unified log directory
returning each record as a dict-like object:
from idstools import unified2
reader = unified2.SpoolRecordReader("/var/log/snort",
"unified2.log", follow=True)
for record in reader:
if isinstance(record, unified2.Event):
print("Event:")
elif isinstance(record, unified2.Packet):
print("Packet:")
elif isinstance(record, unified2.ExtraData):
print("Extra-Data:")
print(record)
See the idstools unified2
documentation for more information on read and parsing unified2 files.
Parse Suricata/Snort Rules
The following code snippet will parse all the rules in a rule file:
from idstools import rule
for rule in rule.parse_file(sys.argv[1]):
print("[%d:%d:%d] %s" % (
rule.gid, rule.sid, rule.rev, rule.msg))
In addition to parsing files,
file objects
and strings
containing individual rules can be parsed.
Update Suricata Rules
The following command will update your Suricata rules with the latest
Emerging Threats Open ruleset for the version of Snort you have
installed:
idstools-rulecat -o /etc/suricata/rules
See the idstools-rulecat documentation for
more examples and options.
Documentation
Further documentation is located at http://idstools.readthedocs.org.
Changelog
0.6.4 - 2020-08-02
- eve2pcap: fix displaying of errors from libpcap
- eve2pcap: python3 fixes
- eve2pcap: print number of packets converted on exit
- rules: fix parsing of rules where the address or port list has a space
- Commit log
0.6.3 - 2017-11-20
- eve2pcap: fix segfault when calling libpcap functions.
- rulecat: for Emerging Threat rule URLs, use the Suricata version as found
- rulecat: default to Suricata 4.0 if it can't be found.
- rule parser: fix case where rule option does not end in ; and is
last option (https://github.com/jasonish/py-idstools/issues/58)
- Commit log
0.6.2 - 2017-08-09
- rulecat: ignore *deleted.rules by default. Provide --no-ignore
option to disable default ignores without having to add a new
ignore.
- rulecat: suppress progress bar if quiet
- rulecat: fix output filenaming for downloads that are a single rule
file
- rulecat: more python3/unicode fixes
- rule parser: if metadata is specified more than once, append to the
existing metadata list instead of replacing it
(https://github.com/jasonish/py-idstools/issues/57)
- Commit log
0.6.1 - 2017-05-25
0.6.0 - 2017-03-29
- idstools-u2eve - output packet records
- idstools-rulecat: allow --local to be specified multiple times
- idstools-rulecat: --ignore option to ignore filenames
- More python 3 fixups.
- unified2 - deprecate event readers, use record readers instead
(https://github.com/jasonish/py-idstools/issues/14)
- u2json: --packet-hex and --printable to print raw buffers as printable
chars and hex in addition to base64.
- u2eve: --packet-printable to include a "packet_printable" field
- u2eve: include Snort extra-data with printable data.
- Commit log
0.5.6
0.5.5
- unified2: fix reading of ipv6 events
- idstools-u2json: option to sort the keys
- u2spewfoo: IPv6 printing fixes
- idstools-rulecat: use ET "enhanced" rules by default
- idstools-rulecat: suricata inspired colour logging
- idstools-rulecat: handle URLs ending with query parameters
- Commit log
0.5.4
- idstools: handle rules with no msg in rule parser
- idstools-rulecat: support a drop.conf for setting rules to drop
- idstools-eve2pcap: allow link type to be set on command line
- unified2: handle large appid buffer in newer versions of Snort.
- Commit log
0.5.3
- idstools-rulecat: better documentation
- idstools-rulecat: use ET Pro https URL
- Commit log
0.5.2
- idstools-u2json: fix --delete
- idstools-u2json: add --verbose flag for debug logging
- idstools-rulecat: allow multiple urls
- Commit log
0.5.1
- New tool: eve2pcap. Converts packets and payloads found in Suricata
EVE logs to pcap files.
- Rule parser: handle multi-line rules.
- Commit log
0.5.0
- New tool: idstools-dumpdynamicrules. A wrapper around Snort to dump
dynamic rule stubs and optionally repack the tarball with the new
stubs.
- New tool: idstools-u2eve. Basically a copy of the current u2json,
but will aim to keep a compatible eve output style. idstools-u2json
will probably become more of a basic example program.
- A basic packet decoding module.
- New tool: rulecat. A basic Suricata rule management tool.
- Commit log
0.4.4
- Fix reading of growing file on OS X.
- Fix error in parsing decoder rules introduced in 0.4.3.
- Commit log
0.4.3
- Make the rule direction an accessible field of the rule object.
- Commit log
0.4.2
- Fix issue loading signature map files (GitHub issue #2).
- Commit log
0.4.1
- Fix IPv6 address unpacking.
- In u2json, if the protocol number can't be converted to a string,
encode the number as a string for a consistent JSON data type.
- Commit log
0.4.0
- New tool, u2json to convert unified2 files to JSON.
- Commit log
0.3.1
- Support the new appid unified2 event types introduced in Snort
2.9.7.0.alpha.
- Commit log
