Repository files navigation

SQLog - Connecting the dots

SQLog is aGolang library that simplifies log management usingslog.SQLog offers a lightweight and reliable solution for logging, making it perfect for developers seeking a cost-effective, high-performance way to monitor their applications.

Below is an example of usingSQLog with the SQLite storage, with the interface exposed on port8080. When you accesshttp://localhost:8080?msg=test, any query parameter will be sent to the log.

You can view the generated logs athttp://localhost:8080/logs/.

import ("log/slog""net/http""os""strings""github.com/nidorx/sqlog"// sqlite storage"github.com/nidorx/sqlog/sqlite"// sqlite driver    _"github.com/mattn/go-sqlite3")funcmain() {storage,_:=sqlite.New(nil)logger,_:=sqlog.New(&sqlog.Config{Storage:storage,})// magicslog.SetDefault(slog.New(logger.Handler()))iftrue {// In the local environment, you can send the log to standard output.logger.Fanout(slog.NewTextHandler(os.Stdout,nil))}// SQLog ui/api handlerlogHttpHandler:=logger.HttpHandler()// http handler (... nidorx/chain, gin-gonic/gin, gorilla/mux, julienschmidt/httprouter)httpHandler:=http.HandlerFunc(func(w http.ResponseWriter,r*http.Request) {ifidx:=strings.LastIndex(r.URL.Path,"/logs");idx>=0 {logHttpHandler.ServeHTTP(w,r)}else {args:= []any{}msg:="I ❤️ SQLog"fork,v:=ranger.URL.Query() {ifk=="msg" {msg=strings.Join(v,",")}else {args=append(args,slog.Any(k,strings.Join(v,",")))}}slog.Log(r.Context(),slog.LevelInfo,msg,args...)w.Write([]byte("🫶"))}})http.ListenAndServe(":8080",httpHandler)}

You can view the current version of the SQLog demo at the links below:

The demo project captures all mouse movements and sends them to the server, which logs the received parameters.

• Log generator:https://sqlog-demo.onrender.com/

• SQLog UI :https://sqlog-demo.onrender.com/logs/

  IMPORTANT! I am using thefree version of Render, so there is no guarantee of service stability or availability.

The source code is in thedemo directory.

SQLog is an efficient library for capturing and managing logs, designed with a focus on performance and low latency. The architecture of the solution consists of four main components:Handler,Ingester,Chunk, andStorage. Each of these components plays a crucial role in the ingestion, storage, and retrieval of logs.

1. Handler

   TheHandler is an implementation ofslog.Handler, allowingSQLog to be easily integrated as the standard logger in Go. It transforms logs coming fromslog into JSON format using the default encoderslog.NewJSONHandler or a custom encoder, providing flexibility in log formatting.

2. Ingester

   TheIngester manages the ingestion of logs from theHandler in a non-blocking manner. It maintains an in-memory buffer that stores log entries in blocks calledChunks. When aChunk reaches its maximum capacity, the Ingester starts using the nextChunk, marking the current one for writing toStorage. At regular intervals, the Ingester invokes theFlush method of theStorage to persist filledChunks.

   Performance: One notable feature of the Ingester is its non-blocking implementation, which avoids using mutexes to ensure concurrency. Instead, it utilizes atomic operations from Go (sync/atomic), allowing multiple goroutines to write logs simultaneously without waiting on each other, resulting in superior performance and reduced latency.

3. Chunk

   TheChunk is a non-blocking structure that allows continuous writing of log entries. EachChunk is a node in a linked list that has a reference to the nextChunk. The Ingester maintains references to two nodes: the currentChunk, which is still accepting new entries, and the flushChunk, which contains completed entries that have not yet been persisted.

   Data Structure: This linked list approach not only simplifies the management ofChunks but also facilitates fast, non-blocking writing. By updating the internal references during the flush process, the Ingester ensures that logs are written efficiently with minimal performance impact.

4. Storage

   TheStorage is responsible for the persistence of logs.SQLog's modular architecture allows for the implementation of different types of storage, such as disk files, databases, and external systems. Currently, there is anative implementation for SQLite (see more details) and another in development for in-memory persistence.

   Responsibilities:

   • Persistence: The Storage receives and storesChunks from the Ingester.
   • Search: It also allows for the retrieval of records, which is used by theSQLog API for log visualization.

The combination of these layers makesSQLog a robust and efficient solution for log management, optimizing performance through a non-blocking architecture and the use of atomic operations. This results in fast, real-time log capture capable of handling high workloads without compromising efficiency.

To use the builtin SQLite Storage implementation, you need to register the driver of your choice.

Examples:

• github.com/mattn/go-sqlite3
• modernc.org/sqlite

If you would like to contribute to this project, here are some tasks and ideas listed below. Feel free to suggest and implement new features inSQLog.

If you decide to work on a task, please leave a comment on the Issue so that others can collaborate.

• InMemory Storage
• Alerts - Enable the creation of alerts within SQLog. The solution should leverage the syntax of the language to evaluate logs at regular intervals and trigger alerts when specific conditions are met.
• Metrics (Count, AVG, Dashboards)
• NOT, REGEX (https://www.sqlite.org/lang_expr.html)

All kinds of contributions are welcome!

🐛Found a bug?
Let me know bycreating an issue.

• This project was inspired byBlacklite
• https://github.com/IGLOU-EU/go-wildcard
• The interface is inspired byDatadog

This code is distributed under the terms and conditions of theMIT license.