Repository files navigation

Instant Observability for Cloud & AI Applications

Zero Code, Full Stack, eBPF & Wasm

English |简体中文 |日本語

The DeepFlow open-source project aims to provide deep observability for complex cloud-native and AI applications. DeepFlow implementedZero Code data collection with eBPF for metrics, distributed tracing, request logs and function profiling, and is further integrated withSmartEncoding to achieveFull Stack correlation and efficient access to all observability data. With DeepFlow, cloud-native and AI applications automatically gain deep observability, removing the heavy burden of developers continually instrumenting code and providing monitoring and diagnostic capabilities covering everything from code to infrastructure for DevOps/SRE teams.

• Universal Map forAny Service: DeepFlow provides a universal map withZero Code by eBPF for production environments, including application services, AI services, and infrastructure services in any language. In addition to analyzing common protocols, Wasm plugins are supported for your private protocols.Full Stack golden signals of applications and infrastructures are calculated, pinpointing performance bottlenecks at ease.
• Distributed Tracing forAny Request:Zero Code distributed tracing powered by eBPF supports applications in any language and infrastructures including gateways, service meshes, databases, message queues, DNS and NICs, leaving no blind spots.Full Stack network performance metrics and file I/O events are automatically collected for each Span. Distributed tracing enters a new era: Zero Instrumentation.
• Continuous Profiling forAny Function: DeepFlow collects profiling data at a cost of below 1% withZero Code, plots OnCPU/OffCPU/GPU/Memory/Network function call stack flame graphs, locatesFull Stack performance bottleneck in business functions, library and framework functions, runtime functions, shared library functions, kernel function, CUDA functions, and automatically relates them to distrubuted tracing data.
• Seamless Integration with Popular Stack: DeepFlow can serve as storage backed for Prometheus, OpenTelemetry, SkyWalking and Pyroscope. It also providesSQL, PromQL and OLTP APIs to work as data source in popular observability stacks. It injects meta tags for all observability signals including cloud resource, K8s container, K8s labels, K8s annotations, CMDB business attributes, etc., eliminating data silos.
• Performance 10x ClickHouse:SmartEncoding injects standardized and pre-encoded meta tags into all observability data, reducing storage overhead by 10x compared to ClickHouse String or LowCard method. Custom tags and observability data are stored separately, making tags available for almost unlimited dimensions and cardinalities with uncompromised query experience likeBigTable.

For more information, please visitthe documentation website.

There are three editions of DeepFlow:

• DeepFlow Community: for developers
• DeepFlow Enterprise: for organizations, solving team collaboration problems
• DeepFlow Cloud: SaaS service, currently in beta

The DeepFlow Community Edition consists of the core components of the Enterprise Edition.

Please refer tothe deployment documentation.

At the same time, we have also built a completeDeepFlow Community Demo, welcome to experience it. Login account/password: deepflow/deepflow.

You can visit theDeepFlow Enterprise Demo, currently available in Chinese only.

• compile deepflow-agent

DeepFlow Community Edition consists of two components, Agent and Server. An Agent runs in each K8s node, legacy host and cloud host, and is responsible for AutoMetrics and AutoTracing data collection of all application processes on the host. Server runs in a K8s cluster and provides Agent management, tag injection, data ingest and query services.

Here is ourfuture feature plan. Issues and Pull Requests are welcome.

• Discord：Clickhere to join our discussion.
• Twitter：DeepFlow
• WeChat Group：

• ThankseBPF, a revolutionary Linux kernel technology.
• ThanksOpenTelemetry, provides vendor-neutral APIs to collect application telemetry data.

• The paperNetwork-Centric Distributed Tracing with DeepFlow: Troubleshooting Your Microservices in Zero Code has been accepted by ACM SIGCOMM 2023.
• DeepFlow enriches theCNCF CLOUD NATIVE Landscape.
• DeepFlow enriches theCNCF CNAI (Cloud-Native AI) Landscape.
• DeepFlow enriches theeBPF Project Landscape.