Repository files navigation

                       _                                    (_) ___  _   _                      | |/ _ \| | | |                      | | (_) | |_| |                     _/ |\___/ \__, |                    |__/       |___/        A package for capturing and analyzing network     flow data and intraflow data, for network research,          forensics, and security monitoring.

Addendum: For our most recent network fingerprinting tools and data, please seeMercury.

Joy is a BSD-licensed libpcap-based software package for extractingdata features from live network traffic or packet capture (pcap)files, using a flow-oriented model similar to that of IPFIX orNetflow, and then representing these data features in JSON. Italso contains analysis tools that can be applied to these datafiles. Joy can be used to explore data at scale, especiallysecurity and threat-relevant data.

JSON is used in order to make the output easily consumable by dataanalysis tools. While the JSON output files are somewhat verbose,they are reasonably small, and they respond well to compression.

Joy can be configured to obtain intraflow data, that is, data andinformation about events that occur within a network flow,including:

• the sequence of lengths and arrival times of IP packets,up to some configurable number of packets.

• the empirical probability distribution of the bytes within thedata portion of a flow, and the entropy derived from that value,

• the sequence of lengths and arrival times of TLS records,

• other non-encrypted TLS data, such as the list of offeredciphersuites, the selected ciphersuite, the length of theclientKeyExchange field, and the server certificate strings,

• DNS names, addresses, and TTLs,

• HTTP header elements and the first eight bytes of the HTTPbody, and

• the name of the process associated with the flow, for flowsoriginate or terminate on the host on which pcap is running.

Joy is intended for use in security research, forensics, and forthe monitoring of (small scale) networks to detect vulnerabilities,threats and other unauthorized or unwanted behavior. Researchers,administrators, penetration testers, and security operations teamscan put this information to good use, for the protection of thenetworks being monitored, and in the case of vulnerabilities, forthe benefit of the broader community through improved defensiveposture. As with any network monitoring tool, Joy couldpotentially be misused; do not use it on any network of which youare not the owner or the administrator.

Flow, in positive psychology, is a state in which a personperforming an activity is fully immersed in a feeling of energizedfocus, deep involvement, and joy. This second meaning inspiredthe choice of name for this software package.

Joy is alpha/beta software; we hope that you use it and benefitfrom it, but do understand that it is not suitable for productionuse.

We have recently released the largest and most informative open sourceTLS fingerprint database. Among other features, our approach builds on previous work by being fully automated and annotating TLS fingerprints with significantly more information. We have built a set of python tools to enable the application of this database, as well as the generation of new databases with the help of Joy. For more information, please see theTLS fingerprinting documentation.

Joy has helped support the research that paved the way for Cisco’s EncryptedTraffic Analytics (ETA), but it is not directly integrated into any of theCisco products or services that implement ETA. The classifiers in Joy weretrained on a small dataset several years ago, and do not represent theclassification methods or performance of ETA. The intent of this feature isto allow network researchers to quickly train and deploy their own classifierson a subset of the data features that Joy produces. For more information ontraining your own classifier, see saltUI/README or reach out tojoy-users@cisco.com.

This package was written by David McGrew, Blake Anderson, Philip Perriconeand Bill Hudson {mcgrew,blaander,phperric,bhudson}@cisco.com of Cisco SystemsAdvanced Security Research Group (ASRG) and Security and Trust Organization (STO).

• Added the ability to use AF_PACKET v3 and AF_FANOUT instead of libpcapuse --enable-af_packet on the configure command for AF_PACKET
• minor bug fixes

• Fix SIGSEGV in DNS parsing (recursion depth bug)
• Fix bug in IPv6 payload calculation
• Fix bug in IPv6 IDP length
• Bump MAX library contexts to 64 (user request)

• Add IPv6 support to Joy and libjoy
• IPFix collection and export only support IPv4
• NFv9 only supports IPv4
• Anonymization only supports IPv4 addresses
• Subnet labeling only supports IPv4 addresses

• Re-write joy.c to use libjoy library
• Updated joy.c to utilize multi-threads for flow processing
• Updated unit tests and python tests to reflect new code changes
• Removed guts of the updater process to prepare for re-write
• Fixed bug in processing multiple files on the command line
• Other minor bug fixes

• Added support for make install for Centos

• Add support for fingerprinting

We are pleased to announce the 4.0.1 release of the package, which has these features:

• Add additional API's for parent application processing of Flow Records and data features
• Fixed TCP retransmission and out of order detection
• Better identification of IDP packet
• Fixed some memory usage issues
• Fixed minor bugs
• Removed dead code

We are pleased to announce the 4.0.0 release of the package, which has these features:

• Add support for building with autotools. ./configure;make clean;make

We are pleased to announce the 3.0.0 release of the package, which has these features:

• Modified JOY infrastructure code to be thread safe.
  • Allowed support multiple work threads for packet processing.
  • Each worker thread uses own output file.
  • Removed global variables for Config.
  • Modified code infrastructure to use Config Structure.
• Modified the Makefile system to build the JOY infrastructure as a static and shared library.
• Implemented an API for utilizing the JOY Library (joy_api.[hc]).
• Implemented a Vector Packet Processing integration scheme to utilize VPP native infrastructure when building that integration.
• Created 2 API test programs, joy_api_test.c and joy_api_test2.c.
• Modified existing test programs to link against static JOY library instead of re-compiling the infrastructure code.
• Modified versioning to use Common Security Module (CSM) conventions.
• Modified build_pkg to accept package version on the command line.
• Cleaned up coverity errors and warnings.
• Various bug fixes.

We are pleased to announce the 2.0 release of the package, which has these features:

• The JSON schema has been updated to be better organized, more readable, and more searchable (by putting searchable keywords as the JSON names),
• The new sleuth tool replaces query/joyq, and brings new functionality such as —fingerprint,
• Much improved documentation, which covers the joy and sleuth tools, examples, and the JSON schema(seeusing-joy)

Joy has been successfully run and tested on Linux (Debian, Ubuntu,CentOS, and Raspbian), Mac OS X and Windows. The system has been built withgcc and GNU make, but it should work with other developmentenvironments as well.

Go to the Wiki for a guide on building:Build Instructions

SeeLicense of Joy

SeeCopying for licenses of external libraries