Repository files navigation

Self-hostedVirusTotal /OPSWAT MetaDefender wannabe API for scanning URLs and files by multiple antivirus solutions.

IMPORTANT: version 1.5 introduces breaking changes in containers configuration and docker-compose.yaml layout. Please seereleases page and changelog ofdocker-compose.yaml andREADME.md for the additional details.

I faced a need to scan user-uploaded files in one of my work projects in an automated mode to ensure they don't contain any malware. Using VirusTotal was not an option because of a) legal restrictions and data residency limitations b) scanning by hash-sums would not be sufficient because the majority of files are generated / modified by users.

After googling, I stumbled upon a fantasticmaliceio/malice project. Unfortunately, itlooks abandoned, and most plugins do not work for the moment. In addition to that, I had an intention to use the .NET stack to align with the internal infrastructure.

In the end, it's nothing but the set of Docker containers running the agent. That agent downloads the remote file to the temp folder, then launches the vendor command-line scanning utility with proper arguments, and parses the output with a regular expression to extract a detected malware name.

IMPORTANT: MalwareMultiScan is not intended as a publicly-facing API / UI. It has (intentionally) no authorization, authentication, rate-limiting, or logging. Therefore, it should be used only as an internal / private API or behind the restrictive API gateway.

Whole solution can be started withdocker-compose up executed in a root folder of repository.

It can be also deployed to the Docker Swarm cluster by using the commanddocker stack deploy malware-multi-scan --compose-file docker-compose.yaml.

After the start the Demo Web UI will become available underhttp://localhost:8888.

Seecomponents chapter below and thedocker-compose.yaml file.

Configuration of API and Scanners is performed by passing the environment variables. Descriptions and default values are provided below.

• MONGO_ADDRESS=mongodb://localhost:27017 - MongoDB connection string.

• MONGO_DATABASE=MalwareMultiScan - MongoDB collection name.

• REDIS_ADDRESS=localhost:6379 - Redis address for the distributed task queue.

• CONSUL_ADDRESS=http://localhost:8500 - Consul address for the service registration.

• FILE_SIZE_LIMIT=52428800 - Maximum size of a file that can be handled for the file scanning. The size of the URL content is not verified. Set to 0 to disable the validation.

• BACKEND_ID=dummy - Id of a backend.

• REDIS_ADDRESS=localhost:6379 - Redis address for the distributed task queue.

• CONSUL_ADDRESS=http://localhost:8500 - Consul address for the service registration.

• MAX_SCANNING_TIME=60 - Scan time limit. It is used not just for actual scanning but also for getting the file.

• WORKER_COUNT=4 - Number of workers for parallel scanning.

• API_URL=http://localhost:5000 - Absolute URL incl. port number for the running instance of MalwareMultiScan.Api.

• POST/api/queue/url with aurl parameter passed via the form data.. Returns201 Accepted response with aScanResult or400 Bad Request error.

• POST/api/queue/file with afile parameter passed via the form data. Returns201 Accepted response with aScanResult or400 Bad Request error.

• GET/api/results/{result-id} where{result-id} corresponds to the id value of aScanResult. Returns200 OK response with aScanResult or404 Not Found error.

Both/api/queue/url and/api/queue/file also accept an optionalcallbackUrl parameter with the http(s) URL in it. This URL will be requested by the POST method with JSON serializedScanResultMessage in a body on every update from scan backends. Query string will containid parameter that corresponds to the id of the scan result andbackend parameter with the id of backend which completed the scan.

I.e. when you definecallbackUrl=http://localhost:1234/scan-results, the POST request will be made tohttp://localhost:1234/scan-results?id=123&backend=dummy with a body

{"Status":1,"Duration":5,"Threats": ["Malware.Dummy.Result"]}

More scan backends can be added in the future. Some of the popular ones do not have command line scanning utility, Linux version, or don't start in Docker container. Feel free to raise an issue if you know any in addition to the list above.

1. On startup allScanners register themselves inConsul with a service name equal toscanner and theBackendId metadata field equal to the value ofBACKEND_ID environment variable. They also register a TTL check and listen forHangfire background job in a queue named under theBackendId metadata field.

2. Third-party client triggers/api/queue/url or/api/queue/file of theMalwareMultiScan.Api.

3. MalwareMultiScan.Api sends a query toConsul and receives the list of alive scan backends with the service namescanner.

4. MalwareMultiScan.Api schedules aHangfire background job in a queue named under theBackendId metadata field.

5. Scanners picks up a job from queue, starts the scan and sends result back to thedefault queue ofHangfire.

6. MalwareMultiScan.Api picks a job from the default` queue ofHangfire and updates the state of the scan.

7. If callback URL was specified during the step #2,MalwareMultiScan.Api triggers a HTTP POST request to the specified URL. SeeCallback URL for details.

• MongoDB of version 3.x or above. Used for storing scan results and files in GridFS. The communication is happening through theofficial C#/.NET driver.

• Redis of version 5.x or above. Used for tasks queueing. The communication is happening through theHangfire library.

• Consul of version 1.8.x or above. Used for service registration of scan backends.

• Docker anddocker-compose running under Windows (in Linux containers mode), Linux, or OSX. Docker Compose is needed only for test / local deployments.

• Optional: DockerSwarm / Kubernetes cluster for scaling up the scanning capacities.

• MalwareMultiScan.Api - Simple ASP.NET Core WebApi for queueing files & urls for the scan and returning the result. Also acts as a receiver of scan results from the scanning backend nodes. SeeDockerfile.

• MalwareMultiScan.Backends - Scan backends logic. Includes Dockerfiles and implementation classes for third-party vendor scan backends.

• MalwareMultiScan.Shared - Shared components.

• MalwareMultiScan.Scanner - .NET Core Worker service subscribes to messages corresponding to the backend id, then fires up scanning command-line utility, and parses the output. SeeDockerfile. The image of MalwareMultiScan.Scanner acts as a base image for the rest of the scan backends. Check Dockerfiles from thetable above for details.

• MalwareMultiScan.Ui - Nuxt.js TypeScript SPA for demoing the API capabilities. SeeDockerfile.

Seeissues for the list of planned features, bug-fixes, and improvements.