Repository files navigation

⚠️This project is no longer actively maintained: feel free tocontact me if you need more information.

LoggyLogs is aparser for log servers. Its goal is tohelp detecting vulnerabilities in HTTP requests. It supportsNGINX andApache logs. This tool was created during a professional engagement, and was shared foreducational purposes only.

🔍

For each HTTP request,LoggyLogs is capable of detecting:

• If thehost is valid
• If theIP address is valid
• If theHTTP status code is valid
• IfHTTPS is properly enforced, to ensure the traffic is encrypted
• If theuser agent is a recognized web browser
• If endpoints are actually related to the API, to detect potentialbrute-force attacks
• If the body size is not too big, as those could lead todenial-of-service (DoS) attacks
• If non-secure HTTP methods are used (i.e.TRACE orDEBUG), as those could lead to severalinformation disclosures

Note: All the tests mentioned above can be run at once!

🔍

• Getting started: This section provides everything that is required to install LoggyLogs.
• Usage: This section shows how to configure LoggyLogs to have the best outputs possible. It includes:
  1. First run: How to run LoggyLogs for the first time
  2. Tweak parameters: How to modify several values in the source code
• Examples: A few examples of outputs are shown in this section.
• Going further: LoggyLogs has been designed as an open source project since day 1. This section includes:
  1. Code documentation: To clarify the tool's internals, and explaining how to generate the source code documentation
  2. Improvements: Additional features that could be implemented to make LoggyLogs even more efficient
• License: The license of the project.

🔍

• Clone this repository:git clone https://github.com/sljrobin/LoggyLogs
• Go to the LoggyLogs directory:cd LoggyLogs/
• Create and activate aPython virtual environment:
  • python3 -m venv venv
  • source ./venv/bin/activate
• Install the requirements withpip3:pip install -r requirements.txt

• Ensure first the Python virtual environment is enabled by runningsource ./venv/bin/activate
• RunLoggyLogs for the first time:python loggylogs.py --help

% python loggylogs.py --helpusage: loggylogs.py [-h] [--all] [--bodysize] [--host] [--ip] [--httpmethod] [--httpprotocol] [--statuscode] [--url] [--useragent]optional arguments:  -h, --help      show this help message and exit  --all           Perform all the available tests  --bodysize      Check the body size is not too important  --host          Check the host is valid  --ip            Check the IP address is valid  --httpmethod    Check which HTTP method is being used  --httpprotocol  Check if HTTP connections (i.e. without encryption) are made to the server  --statuscode    Check the HTTP status code  --url           Check the URL is an endpoint available within the API documentation  --useragent     Check the user agent is a recognized web browser

⚠️ Before running LoggyLogs to analyze logs, several parameters need to be tweaked.

Under thedata/ directory:

• Put the logs to analyze here:
  • A file example namedsamples.log is already present
  • Feel free to replace it
  • In case the analysis of several files is required, change theLOGS_PATH constant inloggylogs.py (e.g.LOGS_PATH = './data/new-samples.log')
• If need be, add moreuser agents:
  • Theuser-agents.txt file already contains an exhaustive list ofuser agents
  • Other lists are available on the Internet, likethis one

In thelib/scanner.py file:

• Depending on which endpoints are valid or not, modify theself.__hosts andself.__urls variables
• An example is shown below:

self.__hosts= ['https://www.google.com/','https://status.google.com/']self.__urls= ['/1/indexing','/1/infrastructure','/1/inventory','/1/latency','/1/reachability','/1/status']

• In addition, it might be useful to modify theself.__limit_body_size variable to inform LoggyLogs when raising an alert when an HTTP request is considered too large
• The default size has been set to 1,500 bytes, as shown below:

self.__limit_body_size=1500

🎉Everything is now ready!

Once LoggyLogs is set, it can parse HTTP requests, and eventually detect suspicious activities. To run it, use the following command:

python loggylogs.py --<test>

Example of output with the--bodysize test:

python loggylogs.py --bodysize

Example of output with the--host test:

python loggylogs.py --host

Example of output with the--statuscode and--ip tests:

python loggylogs.py --statuscodepython loggylogs.py --ip

As a project never ends, the following might help the most curious.

• The source code ofLoggyLogs has been thoroughly documented in order to help people adding new features or simply improving the code
• Because the code is commented, generating a documentation becomes easy
• Amongstmost popular solutions, we recommend usingpydoc for the documentation generation process.
• Examples:
  • Generate the documentation for theParser() class:python -m pydoc lib/parser.py
  • Generate the documentation for the entire library:python -m pydoc lib/*
  • Follow thistutorial for more information aboutpydoc
• Snippet of the documentation for thelib/scanner.py class:

NAME    scannerCLASSES    builtins.object        Scanner    class Scanner(builtins.object)     |  Scanner(display)     |     |  Methods defined here:     |     |  __init__(self, display)     |      Initialize the Scanner object.     |     |      :param Display display: a Display object from the internal library.     |     |  check_all(self)     |      Perform all the checks previously described.     |     |  check_body_size(self)     |      Check the size (in bytes) of the HTTP request. Large requests might be used during denial of service (DoS)     |      attacks to exhaust the server. If the request appears too large, display an error.     |     |  check_host(self)     |      Check the host used to perform the request is related to the API, following a whitelist approach. If not,     |      display an error.[...]

Improvements could be added toLoggyLogs to make it even more efficient. Several thoughts are shared below:

• For the IP address, aDNS lookup could be implemented. This, used in conjunction with a blacklist of domains, could be used to detect potential DoS or DDoS attacks
• Groups of requests from the same IP address could be made. From there,inspection of the timestamps between each log could be performed – note that a method to extract the date of each log entry has successfully been created. This could help in detecting potential DoS attacks
• For each URL, asearch for specific characters could help detecting potential injection attacks. For instance, characters like<,{,;, or', might be injected to test for cross-site scripting (XSS) attacks or SQL injections
• Still, with the URL,lists of known endpoints could be banned. For example, the lists ofDirBuster or otherweb wordlists could be used as blacklists of endpoints. On the server-side, it would be required to ensure all the proper checks have been implemented so that those resources cannot be accessed without proper authorization
• Instead of using a file,regular expressions for user agents could be added, just likethis one
• Optimization of the log analysis could be made with specialized libraries. Theadvertools library seems to be an interesting approach. Indeed, the logs are being compressed and the analysis appears to be much faster

LoggyLogs is licensed under the MIT License (Source).