AIL v6.5 introduces several major improvements to strengthen dark web monitoring and analysis workflows:

• I2P Crawling Support
  The crawler now supportsI2P, extending coverage beyond Tor and traditional web sources.

• Enhanced Search with Description Indexing
  Search capabilities have been improved withdescription indexing, making it easier to discover and correlate relevant content across large datasets.

• Improved Image Analysis Workflows
  Image analysis has been optimized to provide more efficient processing, categorization, and contextual enrichment of visual material.

🚀 New Features

• Crawler
  • Added fullI2P crawler support, includingauto-discovery.
• Search Engine
  • Now supportssearching descriptions of images, screenshots, and domains.
  • Added newdescription indexes for domains, images, and screenshots.
• Image Engine
  • Introduced a function toautomatically describe all images for better indexing and search.
• System
  • Added support forkvrocks installed via deb package.
  • Flask will nowskip SSL context if certificates or keys are missing, improving deployment flexibility.

Changes & Improvements

• Correlation Engine
  • Added functions tosearch for IDs in one-depth correlations, with support for returning intermediate objects.
• I2P Crawler
  • Improved domain filtering (e.g., unreachable, unknown, invalid destinations).
  • Enhanced crawler stats reporting.
• psl_faup
  • Added support foradditional TLDs (like.b32.i2p).
  • Performance improvements for parsing.
• Settings
  • Dashboard now showsnumber of active organizations, logged users, and active users.
• Search Engine
  • Enhanced image description handling and UI display.
• Build System
  • MovedYARA installation fromupdate_thirdparty.sh toinstalling_deps.sh for consistency.

🛠 Fixes

• Crawler / I2P
  • Fixed localhost redirection issues.
  • Corrected filtering of errored and invalid I2P pages.
  • Fixed crawler stats accuracy.
• Onion Module
  • Fixed domain recrawling and general module issues.
• Search Engine
  • Fixed indexing naming issues.
  • Corrected blueprint redirects and pagination.
  • Fixed broken links in image/object descriptions.
• UI & Dashboard
  • Fixed domain type selector in search by date range.
  • Improved crawler dashboard layout and fixed HTML tag issues.
• Tests & Docs
  • Fixed API/UI tests.
  • Corrected documentation for pystemon config directory.

Contributors

• Raphaël Vinot
• Thirion Aurélien

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on themisp-lea.org website.

Release date: 2025-08-22

🚀 New Features & Changes

• Core Update

  • Introduced v6.4 update. [terrtia]

• Crawler

  • Added function to recrawl onion services by month or crawl all at once. [terrtia]
  • The crawler is now namedLacus. [Alexandre Dulaunoy]

• CVE Integration

  • Migrated CVE information UI toVulnerability Lookup. [terrtia]

• Module Extractor

  • Improved email extraction. [terrtia]

• Mail Search

  • Increased cache for better performance. [terrtia]
  • Refactored mail search with improved options:
    • Domain + user fragment
    • User + optional domain fragment
    • Exact address
    • Domain fragment only
  • Improved overall search performance, added caching and pagination for domain search. [terrtia]

• ZMQ Importer

  • Display triggered filters. [terrtia]
  • Added filter content. [terrtia]
  • New option to filter by file start/end. [terrtia]

• Abstract Object

  • Improved ID iterator performance. [terrtia]

• Language

  • Added function to pingLibreTranslate. [terrtia]

🛠 Fixes

• TheHive

  • Adapted to unexpected breaking change. [terrtia]

• Crawler Recrawl

  • Fixed last parent issue. [terrtia]

• CVE

  • Fixed typo. [terrtia]
  • Corrected CVE Vulnerability Lookup UI URL. [terrtia]

• Module Extractor

  • Limited email extraction to prevent excessive data collection. [terrtia]

• ZMQ Importer

  • Fixed content filters. [terrtia]

• Language

  • Improved exception handling for unknown ISO1 language codes. [terrtia]

• Onion Lookup

  • Fixed parsing of invalid URLs (multiple improvements). [terrtia]

✨ Contributors

• terrtia
• Alexandre Dulaunoy

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on themisp-lea.org website.

AIL v6.3 adds Passive SSH integration, enabling correlation of SSH keys across onion services, IPs, and domains. This helps identify shared infrastructure and supports onion deanonymization efforts.

Key Feature: Passive SSH Integration for Onion Correlation

AIL now integrates withPassive SSH, allowing:

• SSH key correlation across IPs, domains, and onion services.
• A newSSH key object with sidebar display and linking.
• Passive SSH search and lookup within AIL.
• New IP object to correlateIPs andSSH keys.

This enables deanonymization of onion services through infrastructure fingerprinting based on shared SSH keys.

Notable Changes

Onion Module

• Reduced redundant duplicate checks.
• Only print task UUID when a new task is created.
• Fixed exceptions for invalid URLs andNone domains.

QR Code Extraction

• Added support forcolor-inverted QR codes.

IP & Domain Handling

• New IP object with SSH key correlation.
• Print deanonymized hostnames.
• Replaced and removed FAUP withpsl_faup.
• Improved domain parsing (including missing schemes).

Image Engine

• Added domain description functionality.
• Improved progress logging and display.

Language Handling

• Avoid sending unsupported languages to LibreTranslate.
• Added support forbe (Belarusian).
• Improved language selection and translation handling in UI.

Tracker & Stats

• Added heatmap: matches by year.
• Option to avoid duplicate notifications.
• New function to get AIL-wide stats.

ZMQImporter

• Content filtering byfeeder_name and pattern.
• Improved debug messages and output.

API

• Added endpoint: get onions grouped by month.

Fixes

• Removed all uses of FAUP and migrated to pythonpsl_faup.
• Fixed:
  • Domain extraction and parsing bugs.
  • IP-to-SSH key correlation.
  • Sidebar rendering for IPs and SSH keys.
  • Retro hunt filters and metadata cleanup.
  • CE Detector retagging behavior.
  • Various UI issues (icons, sparkline removal, template bugs).
  • Updater version tagging and leftover debug output.

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on themisp-lea.org website.

We’re excited to releaseAIL Framework v6.2, a major update with new features and improved performance. This version makes analysis easier and the overall experience faster and more user-friendly.

Among the highlights are a fully revamped search engine powered byMeiliSearch, improved language detection for short text, local AI-driven image descriptions, and a retro-hunting editor tool.

What's New in AIL 6.2

• Integrated YARA Editor: AIL now includes a built-in CodeMirror editor with full YARA rule support. Analysts can write, edit, and manage YARA rules directly within AIL, complete with syntax highlighting for an improved editing experience. (Thanks to Sami Mokaddem for this contribution!)
• AI-Powered Screenshot Descriptions with Ollama: The images engine now leverages Ollama to generate descriptions for screenshots and images. This AI-powered capability helps you quickly understand the content of images without needing to view them directly, adding another layer of insight. Descriptions are also saved in the database.
• Expanded Search Horizons:
  • Chat Message Search: You can now search the content of chat messages using MeiliSearch, making it easier to find specific conversations or keywords.
  • Tor Content Search: The new search engine also supports full-text search of crawled Tor pages, helping you quickly locate relevant information across hidden services.
• Enhanced Data Ingestion & Processing:
  • Matrix Feeder: AIL now supports ingesting data from Matrix export via a new chat feeder.
  • Google Tracking Module: A new module and object have been added for tracking Google analytics ID.
• Streamlined User Management & Configuration:
  • Welcome Emails for New Users: New users can now automatically receive a welcome email upon account creation.
  • New User Configuration Engine: Users can now create and save MISP accounts and API keys directly within AIL, enabling seamless data export to MISP instances or generation of MISP JSON files.
• Richer Correlation Capabilities: Discover new connections with added correlations for chat-to-CVE, chat-to-cryptocurrencies, and domain-to-chat/message.
• New Mail Object: A new mail object has been introduced, improving search and correlations with chats, domains, and crawled data.
• Module Statistics: The settings module now includes statistics for your AIL modules.
• Language Statistics in Chat Viewer: Gain insights into language distribution within chats directly in the chat viewer.

Key Enhancements in This Release

Alongside the new features, AIL 6.2 brings a wave of improvements to existing functionalities:

• Performance Boosts:
  • Significant performance improvements for mail and Gtracker searches.
  • Language detection is now more performant.
• Improved Search and Dashboards:
  • The general search dashboard has been revamped for better usability and now includes a helpful search assistant.
• Advanced Language Processing:
  • Language detection accuracy has been increased by removing special characters before analysis and improving the old Lexilang detector.new language detector for short text
  • The language engine has been refactored, allowing retrieval of chat messages and user messages filtered by language.
• Deeper Data Analysis & Correlation:
  • The reprocess functionality now includes a TrackingId module.
  • Enhanced correlation cards for file names, mail, and Gtracker entries.
• User Experience & System Management Refinements:
  • The module queue now displays the number of FeederModuleImporters.
  • The "Create New Tracker" button has been conveniently moved to the top of the trackers page.
  • HOTP users can now easily print their next 50 tokens via a new button in user settings.
  • The number of messages per participant is now shown in the chat participants view.
• Mail & Chat Improvements:
  • Punycode encoding issues in mail have been addressed.
  • Message cards now display subchannels and protocol information for better contextual understanding.
• Image Engine Refinements:
  • Beyond the new Ollama integration, image descriptions are stored along with the model used to generate them.
• Domain Analysis:
  • A new button has been added in domain search to directly crawl unknown onion sites.

Important Fixes

As with every release, AIL 6.2 includes a multitude of bug fixes to improve stability and reliability. Some notable areas include:

• Improved translations for several languages (BG, EL, HI, JA, ZH, RU) and user chat message translation.
• The installation process has been made smoother, with fixes for dependencies and submodule initialization.
• MISP export errors and empty relation issues have been resolved.
• Fixes for mail search and mail content display.
• Several fixes related to language detection, manual language selection, and LibreTranslate ISO codes.
• Crawler dashboard and domain onion cache fixes.
• And many more under-the-hood tweaks for a better overall experience!

Documentation

• We've added documentation for the tracker functionality to help you get the most out of it.

A Big Thank You!

This release wouldn't have been possible without our dedicated community and contributors. Special thanks to Sami Mokaddem, Thirion Aurélien, Aaron Kaplan, NMD03 for their extensive work and valuable contributions to this version.

We encourage you to update to AIL 6.2 to take advantage of these new features and improvements.

🔗Download & Documentation:AIL Project GitHub

💡Feedback & Contributions: As always, we welcome community feedback and contributions to make AIL even better!

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on themisp-lea.org website.

We are excited to announce the release ofAIL Project v6.1, bringing significant improvements, new features, and important bug fixes. This update enhances the crawler's control mechanisms, improves file and chat processing capabilities, and refines various UI elements to provide a more efficient user experience.

AIL introduced a pre-filtering mechanism to limit the risk of potential crawling of unsafe content. This feature is now enabled by default, allowing administrators to decide whether to enable or disable it according to their specific needs.

🍰 Highlights

• Enhanced Crawler Control: Added an option to control whether the crawler should proceed with crawling unclassified onion domains.
• New Unsafe Onion Filter: Introduced an additional filtering option to improve onion domain classification.
• Improved File and Chat Handling: File name searches are now case-insensitive, and chat files now support correlation processing.
• Retro Hunt Enhancements: Now displays the last seen date of matched objects.
• Performance and Stability Improvements: Optimized regex and YARA timeouts, removed SIGALRM usage to prevent Flask server termination, and various UI fixes.

New Features & Enhancements

Crawler Updates

• Added an option to control crawling of unclassified onion domains.
• Updated blocklist for better filtering.

File and Chat Processing

• Added afile name dashboard for improved file search and management.
• Improved case-insensitive file name searches.
• Chat explorer now displays file tags.
• Enhancedchat text processing with correlations.

Mail and Message Processing

• Removed DNS checks for UI extraction in theMail Extractor module.
• Mail exporter now adds object URLs only if the email is an AIL user.
• Display file names and message content in search results.

Tracker and Rule Management

• Added a button tohide/show long rules in trackers.
• Collapse long rules for improved readability.

Retro Hunt Improvements

• Now displayslast seen dates for matched objects.
• Fixed issues withretro hunt item restarts.

Performance & UI Improvements

• Signal timeout for global extraction and reduced regex/YARA timeout.
• Sidebar fixes and UI enhancements in organization views.

🛠️ Bug Fixes

• Fixedretro hunt item restarts.
• Corrected atypo in MailExporter.
• Resolved ascheduler role issue in documentation.
• Fixedtemplate errors in user creation when providing an invalid password.
• Addressed asidebar display issue in organization views.
• Fixed aweird encoding issue in string item content retrieval.
• Prevented file links from appearing if files are not downloaded.
• Fixedcrawler task user_org issue in the API.
• RemovedSIGALRM usage to prevent Flask server termination.

This release significantly enhances the usability, performance, and stability of AIL Project. We encourage users to update to v6.1 for the latest improvements.

🔗Download & Documentation:AIL Project GitHub

💡Feedback & Contributions: As always, we welcome community feedback and contributions to make AIL even better!

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on themisp-lea.org website.

AIL Framework v6.0.1 Release Notes

Release Date: 2025-01-23

This release includes several new features, improvements, and bug fixes. We recommend users to upgrade to the latest version.

New Features and Improvements

• User Management:
  • User login is now case-insensitive.
  • User updates now include v6.0.1 specific changes.
• Chat Explorer:
  • Added the ability to show messages and open them in their respective chat/subchannel/thread.
  • Chat tags are now displayed.
  • Basic chat card improved, includes a chat participants button and improved button styling.
• Chat Viewer:
  • Added messages heatmap by year for user-account viewer.
  • Added heatmap messages and year selector for chat viewer.
  • Added chat messages by current year heatmap feature.
• Message Description:
  • Chat name and username are now included in the message description.
• Retro Hunt:
  • OCR retro hunt added.
• Crawling:
  • Improved queued error log.
  • Improved title extraction, resolving issues with signal.alarm and sleep.
• Cryptocurrency:
  • Added ripple address subtype and correlation.
• Taxonomies and Galaxy:
  • MISP taxonomies and galaxy have been bumped to the latest versions.
• Image Handling:
  • Implemented blurring of unsafe images with violence and pornography-illicit-or-illegal tags with a warning message.
• Domain Search:
  • Domain name sanitization added to search by name and displays domain.
• Updater:
  • Now uses tag subversion (e.g., v6.0.1).
• Tracker:
  • Filter result by object type.
• Flask:
  • Update Flask_config.py.
  • Set proxy using ProxyFix.
• Other:
  • README updated with a new dashboard image.

Bug Fixes

• AIL Updater:
  • Removed updates between tags.
  • Fixed upper tags list and is_fork issues.
• Domain Display:
  • Unblurred default image if domain is down.
• Module Extractor:
  • Invalid object meta logging added and fixed.
  • Fixed onion extraction and prevents onion extraction from crawled items.
• Retro Hunt:
  • Fixed object to resume functionality.
  • Fixed item iterator issues.
  • Force pause state before deleting a retro hunt.
  • Fixed retro hunt resume.
• Crawler:
  • Increased timeout for queued captures.
  • Fixed issues related to signal timeouts during title extraction.
  • Debugged signal timeout issues.
  • Addressed issues with crawler queued capture loop and debugged related errors.
  • Avoid crawler loop if a capture end up in an invalid state.
  • Fixed reload_crawlers_stats queues stats.
• Chat Messages:
  • Fixed subchannelnb_max issue for chat messages by year.
  • Fixed get years date range for chat forum.
• Chat Explorer:
  • Fixed protocols name list order.
• Exifs Module:
  • HandledMp4 UnidentifiedImageError.
• Investigation:
  • Addressed issue with adding objects with spaces in their IDs to an investigation.

Other Changes

• Merged several pull requests:
  • Merged branch 'master' of github.com:ail-project/ail-framework.
  • Merged pull request#253 from eltociear/patch-1 (chg: [flask] update Flask_config.py).
  • Merged pull request#250 from FafnerKeyZee/patch-1 (Update Tracker.py).
  • Merged pull request#251 from FafnerKeyZee/patch-2 (Update abstract_chat_object.py).
    *Fixed an issue where stat was only performed on subchannels for a forum.
  • Merged pull request#249 from vncloudsco/master (Update Install silent and Dockerfile update).
  • Improved tracker level validation preventing a crash when the level field is None.
  • Updated Dockerfile and install scripts.

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on themisp-lea.org website.

The AIL project team is thrilled to announce the release ofAIL v6.0, a significant milestone in our continuous effort to deliver cutting-edge tools for analyzing information leaks and malicious activities on the dark web. This release introduces a host of new features, improvements, and fixes designed to streamline workflows and provide deeper insights for analysts. Here's what you can expect in this version:

New Features and Enhancements

1. Updated Dashboard for Enhanced Usability

• Tag Monitoring: Detection are now displayed by tags, enabling quicker categorization and prioritization.
• Tracker Descriptions: A new feature that displays tracker descriptions directly on the dashboard.
• Crawler Stats & Object Tooltips: Gain a comprehensive view of crawler statistics and in-depth information about objects through intuitive tooltips.
• Enhanced Object Metrics: Real-time updates and insights into the number of objects per day, supported by a new WebSocket-enabled interface.
• EChart Feeder Graph: A sleek, interactive graph for visualizing data, along with a cleanup of outdated graph libraries.

2. Backend Improvements

• Module Management Overhaul: Removal of legacy modules and module loaders, paving the way for a leaner and more efficient system.
• PubSubLogger Upgrade: The oldredis_logger has been retired for a more modern and reliable implementation.
• Queue Enhancements: The queue system now records module start times and process IDs for better process tracking.

3. New Object Type: Barcode

• AIL v6.0 introduces support for barcodes, allowing analysts to extract and analyze barcode objects efficiently.

4. Visual and UI Upgrades

• Migration toFontAwesome v6.6.0, ensuring compatibility with the latest icon set and a more polished visual interface.
• A dedicated date view for daily analysis, improving workflow for time-sensitive investigations.

🛠️ Fixes and Optimizations

• Dashboard Fixes:
  • Resolved formatting issues with day display.
  • Corrected feeder names for better clarity.
• Retro Hunts: Enhanced functionality by removing outdated objects and fixing tag addition errors.
• Object Item Display: Fixed URL visibility for item objects.
• Barcode Message Card: Addressed issues with the display of barcode-related messages.
• Sidebar Fixes: Improved layout and display for organizational information.

Why Upgrade to AIL v6.0?

This release is a leap forward for dark web analysts and cybersecurity professionals. With a more intuitive dashboard, enhanced tracking capabilities, and support for new object types, AIL v6.0 empowers users to handle complex investigations with greater speed and accuracy.

Get Started with AIL v6.0 Today!

To upgrade, after the update, the launch script can be called doing the update automatically. Detailed instructions and documentation are available in ourofficial repository.

Let’s shape the future of open-source dark web analysis together. Stay tuned for more updates, and as always, your feedback is invaluable to us!

AIL Project Release v5.9 - 2024-10-18

We are glad to announce the release of AIL v5.9, packed with numerous updates and fixes that enhance the performance and features of the framework. This release focuses on improvements in qrcode handling,dom-hash support, title processing, and crawler operations, among other critical updates.

dom-hash is a structural fingerprint of the HTML’s Document Object Model (DOM) originally developed by CERT.PL.

The fingerprint is calculated by extracting all the tag names (ignoring the content itself as well as attributes of the HTML Page). The tag names are concatenated with a pipe value |, hashed using the SHA-256 algorithm, and truncated to the first 32 characters.

Software such asLookyLoo orMISP have implemented the algorithm, the AIL framework now support thedom-hash algorithm to cluster and group similar page structure.

What's New in v5.9

Changes

• [qrcodes] Daterange search: Sort qrcode by content. [terrtia]
• [dom-hash] Adddom-hash object to computedom-hash for domains and crawled items. [terrtia]
• [CEDetector] Tag domains. [terrtia]
• [CEDetector] Add detect message functionality. [terrtia]
• [titles] Setup Titles queues and integrate CEDetector module. [terrtia]
• [tools] Add reprocess option for Titles + CEDetector. [terrtia]
• [crawler] Disable unsafe title auto-tagging. [terrtia]
• [module] General debugging and improvements. [terrtia]
• [module] Add CEDetector module. [terrtia]
• [crawler] Tag domain by vanity. [terrtia]
• [crawler] Crawl list URLs: Filter duplicates. [terrtia]
• [crawler] Submit free text of URLs to crawl. [terrtia]
• [onion module] Filter out Onion v2 domains. [terrtia]
• [show domain] Display title content and fix item tags. [terrtia]
• [crawler] Update TOR user agent for more efficient crawling. [terrtia]
• [message image] Show extracted QR codes in messages. [terrtia]
• [domain lookup] Extract domains from URL input. [terrtia]
• [api] Rename domain lookup function. [terrtia]

Fixes

• [module] Fixed a typo in the module. [terrtia]
• [tag] Tag delete: Fixed update for first/last seen times. [terrtia]
• [show domain] Fixed the correlation button when correlation count is zero. [terrtia]
• [crawler] Filter lookup tags for better accuracy. [terrtia]
• [crawler] Filter lookup for parent + domain daterange. [terrtia]

We hope these updates enhance your experience with the AIL framework. Stay tuned for future updates as we continue to improve and expand AIL’s capabilities.

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on themisp-lea.org website.

We announce the release of AIL Framework v5.8, packed with new features, improvements, and bug fixes to facilitate the usage. This release focuses on expanding functionality and improving efficiency in key areas such asQR code extraction, domain lookups, image handling, and more.

We would like to thank LEAs and police officers for their feedback during the ENISA / Europol EC3 Workshop.

What’s New in v5.8?

Enhanced Features:

• QR Code Search by Tags: You can now search for QR codes using tags, making it easier to identify and track items based on tag data.
• Domain Lookup API: A domain lookup function has been added to the API, streamlining investigations with a direct lookup feature.
• Image Lazy Loading in Chats Explorer: To optimize performance, image lazy loading has been implemented, ensuring quicker and smoother browsing through chat logs.
• Favicon Search by Date Range: Favicons can now be searched by date range, coupled with lazy loading, for an efficient exploration process.
• Improved QR Code Extraction: The QR code extractor has been upgraded to provide better extraction from images and screenshots, with added correlation features.
• New Text Wrapping Option in Item Display: You can now replace the canvas with image blur for improved visuals and use a new button to wrap text for clearer readability.
• Username Search: We’ve introduced the ability to search by username, making it easier to locate users in your dataset.
• Tags Search Improvement: If no result is found when searching by tags, the last associated date is now shown for better context.

User Interface Enhancements:

• Hunter Sidebar Icon Alignment: Icons in the hunter sidebar are now aligned for a cleaner and more intuitive interface.
• Improved Domain Screenshot Display: The canvas display for domain screenshots has been replaced by an image blur effect for a smoother visual experience.
• Organization Selector Search in User Creation/Edit: Searching for organizations is now faster and more user-friendly with the new org selector search feature.

Key Fixes:

• QR Code Extractor Improvements: Multiple issues related to the QR code extractor have been resolved, including fixing exceptions, handling empty content, filtering invalid images, and debugging image formats like JPEGs.
• Bitcoin Bech32 Address Validator: A bug affecting the Bech32 address validator has been addressed, improving accuracy when validating Bitcoin addresses.
• Improved Mail Module: Minor typos in the mail module have been fixed for better clarity.
• Crawler Dashboard: A fix has been applied to the domains up/down links, ensuring accurate daily domain tracking.
• Investigation Object Table: Long object IDs in the investigation table have been fixed to prevent display issues.
• UI Fixes: Several visual improvements were made, including resolving overlapping table/image displays and updating crawler bar chart colors for up/down states.

Conclusion:

With the release of AIL Framework v5.8, we continue to build on our commitment to delivering a powerful and intuitive tool for your intelligence investigations. From improved search functionality to enhanced image handling and bug fixes, this release is designed to optimize your day-to-day investigations.

As always, feel free to provide feedback or report any issues to help us make AIL even better.

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on themisp-lea.org website.

AIL Framework v5.7 Released!

We are thrilled to announce the release ofAIL Framework version 5.7. This update brings a host of new features, improvements, and bug fixes designed to enhance performance, security, and user experience.

Highlights

• Two-Factor Authentication (2FA): Enhance your account security with new TOTP and HOTP authentication methods.
• Organizational Support: Introduced organization-level access controls and management for users, trackers, retro hunts, investigations, and cookie jars.
• Improved Chat Monitoring: Added support for new chat types, placeholders, and enhanced message viewing with usernames and relationships.
• Dashboard Enhancements: Real-time updates with event streams replacing interval requests for a smoother experience.
• User Management Overhaul: Refactored user creation and editing processes, including organization assignments and session management.

What's New

Security Enhancements

• Two-Factor Authentication (2FA):

  • Implemented TOTP and HOTP methods for additional account security.
  • Users and admins can manage 2FA settings directly from their profiles.

• User Session Management:

  • Administrators can now manually log out users or terminate user sessions.
  • Added metadata for users, including creation date, last edit, last login, last seen, and login status.

Organizational Features

• Organization-Level Access Control:

  • Introduced organizations to structure data and access.
  • Trackers, retro hunts, investigations, and cookie jars now support organization-specific ACLs.
  • Users can view their organization information in their profiles.

• User Roles Update:

  • Renamed the "coordinator" role to "org_admin" for clarity.
  • Refactored user roles to better align with organizational structures.

Chat and Messaging Improvements

• Chat Monitoring:

  • Added support for additional chat types and placeholders.
  • Chats now display usernames for better context.
  • Implemented chat monitoring requests in the chats explorer.

• Message Relationships:

  • Enhanced relationships by adding message mentions linking chats and user accounts.
  • Introduced chord diagrams to visualize message flow between chats and users.

User Interface and Experience

• Dashboard Updates:

  • Replaced interval-based requests with event streams for real-time updates.
  • Improved performance and reduced server load.

• Error Handling:

  • Enhanced UI to gracefully handle 403 and 404 errors.
  • Refined logs to filter out unnecessary SSL errors when clients disconnect.

• Visualization Tools:

  • Updated D3.js to the latest version.
  • Migrated heatmap to version 7 with improved tooltips.
  • Circos graphs now display the number of inbound and outbound messages in tooltips.

Other Notable Additions

• Crawler Management:

  • Added functions to delete schedules and manually clear queues.
  • Improved crawler statistics with monthly domain-type stats.

• Export and Import:

  • Filtered out non-MISP objects during MISP exports.
  • Updated MISP taxonomies and galaxies to the latest versions.

Bug Fixes

• Crawler Queue Statistics:

  • Resolved multiple issues causing inaccurate crawler queue stats.

• User Management:

  • Fixed role editing and user validation processes.
  • Corrected issues with users changing their own passwords.

• Trackers and Retro Hunts:

  • Addressed ACL issues for global trackers.
  • Fixed webhook exports and post-filter selectors.

• Cookie Jar:

  • Resolved problems when adding cookies with UUIDs.

• Object Handling:

  • Fixed errors when retrieving objects withNone values.
  • Corrected display issues in the object subtype dashboard.

• User Accounts:

  • Fixed tooltips in chord graphs.
  • Corrected last username timestamp displays.

Upgrading to v5.7

To upgrade to the latest version:

1. Pull the Latest Changes: Update your local repository to include the latest commits.
2. Restart Services: Restart the AIL Framework services to apply the new changes. The update script is started automatically.

For a detailed list of changes, visit ourGitHub repository.

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on themisp-lea.org website.

Stay Connected:

• Website:ail-project.org
• GitHub:github.com/ail-project/ail-framework
• Mastodon@ail_project
• Twitter/X/N*zi platform:@AIL_Project

Thank you for your continued support. We look forward to your feedback!