Souin Table of Contents

Souin HTTP cache

Project description

Warning

Sincev1.7.0 Souin implements only one storage. If you need a specific storage you have to take it fromthe storages repository and add it either in your code, during the build otherwise.
(e.g. with otter using caddy) You have to build your caddy module with the desired storagexcaddy build --with github.com/darkweak/souin/plugins/caddy --with github.com/darkweak/storages/otter/caddy and configure otter in your Caddyfile/JSON configuration file.
See thestorages section or thedocumentation website about the storages.

Configuration

Required configuration

Souin as plugin

default_cache:# Requiredttl:10s# Default TTL

Souin out-of-the-box

default_cache:# Requiredttl:10s# Default TTLreverse_proxy_url:'http://traefik'# If it's in the same network you can use http://your-service, otherwise just use https://yourdomain.com

Optional configuration

# /anywhere/configuration.ymlapi:basepath:/souin-api# Default route basepath for every additional APIs to avoid conflicts with existing routesprometheus:# Prometheus exposed metricsbasepath:/anything-for-prometheus-metrics# Change the prometheus endpoint basepathsouin:# Souin listing keys and cache managementbasepath:/anything-for-souin# Change the souin endpoint basepathcache_keys:'.*\.css':disable_body:true# Prevent the body from being used in the cache keydisable_host:true# Prevent the host from being used in the cache keydisable_method:true# Prevent the method from being used in the cache keydisable_query:true# Prevent the query string from being used in the cache keydisable_scheme:true# request scheme the query string from being used in the cache keydisable_vary:true# Prevent the varied headers string from being used in the cache keyhash:true# Hash the cache key instead of a plaintext onehide:true# Prevent the cache key to be in the response Cache-Status headerheaders:# Add headers to the key      -Authorization# Add the header value in the key      -Content-Type# Add the header value in the keytemplate:"{http.request.method}-{http.request.host}-{http.request.path}"# Use caddy placeholders to create the key (when this option is enabled, disable_* directives are skipped)cdn:# If Souin is set after a CDN fill these informationsapi_key:XXXX# Your provider API key if mandatoryprovider:fastly# The provider placed before Souin (e.g. fastly, cloudflare, akamai, varnish)strategy:soft# The strategy to purge the CDN cache based on tags (e.g. soft, hard)dynamic:true# If true, you'll be able to add custom keys than the ones defined under the surrogate_keys keydefault_cache:allowed_http_verbs:# Allowed HTTP verbs to cache (default GET, HEAD).    -GET    -POST    -HEADallowed_additional_status_codes:# Allowed additional HTTP status code to cache.    -202    -400cache_name:Souin# Override the cache name to use in the Cache-Status headerdistributed:true# Use Olric or Etcd distributed storagekey:disable_body:true# Prevent the body from being used in the cache keydisable_host:true# Prevent the host from being used in the cache keydisable_method:true# Prevent the method from being used in the cache keydisable_query:true# Prevent the query string from being used in the cache keydisable_scheme:true# Prevent the request scheme string from being used in the cache keydisable_vary:true# Prevent the varied headers string from being used in the cache keyhash:true# Hash the cache key instead of a plaintext onehide:true# Prevent the cache key to be in the response Cache-Status headerheaders:# Add headers to the key      -Authorization# Add the header value in the key      -Content-Type# Add the header value in the keytemplate:"{http.request.method}-{http.request.host}-{http.request.path}"# Use caddy placeholders to create the key (when this option is enabled, disable_* directives are skipped)etcd:# If distributed is set to true, you'll have to define either the etcd or olric sectionconfiguration:# Configure directly the Etcd clientendpoints:# Define multiple endpoints        -etcd-1:2379# First node        -etcd-2:2379# Second node        -etcd-3:2379# Third nodemode:bypass# Override the RFC respect.olric:# If distributed is set to true, you'll have to define either the etcd or olric sectionurl:'olric:3320'# Olric serverregex:exclude:'ARegexHere'# Regex to exclude from cachestale:1000s# Stale durationtimeout:# Timeout configurationbackend:10s# Backend timeout before returning an HTTP unavailable responsecache:20ms# Cache provider (badger, etcd, nutsdb, olric, depending the configuration you set) timeout before returning a missttl:1000s# Default TTLdefault_cache_control:no-store# Set default value for Cache-Control response header if not set by upstreamlog_level:INFO# Logs verbosity [ DEBUG, INFO, WARN, ERROR, DPANIC, PANIC, FATAL ], case do not matterssl_providers:# The {providers}.json to use  -traefikurls:'https:\/\/domain.com\/first-.+':# First regex route configurationttl:1000s# Override default TTL'https:\/\/domain.com\/second-route':# Second regex route configurationttl:10s# Override default TTL'https?:\/\/mysubdomain\.domain\.com':# Third regex route configurationttl:50s# Override default TTL'default_cache_control:public, max-age=86400# Override default default Cache-Controlykeys:The_First_Test:headers:Content-Type:'.+'The_Second_Test:url:'the/second/.+'The_Third_Test:The_Fourth_Test:surrogate_keys:The_First_Test:headers:Content-Type:'.+'The_Second_Test:url:'the/second/.+'The_Third_Test:The_Fourth_Test:

APIs

Prometheus API

Souin API

Security API

Diagrams

Sequence diagram

Cache systems

"badger": {"configuration": {"ValueDir":"default","ValueLogFileSize":16777216,"MemTableSize":4194304,"ValueThreshold":524288,"BypassLockGuard":true  }}

"nuts": {"configuration": {"Dir":"default","EntryIdxMode":1,"RWMode":0,"SegmentSize":1024,"NodeNum":42,"SyncEnable":true,"StartFileLoadingMode":1  }}

"otter": {"configuration": {"size":9999999  }}

"distributed":true,"olric": {"configuration": {# Olric configuration here...  }}

"distributed":true,"etcd": {"configuration": {# Etcd configuration here...  }}

GraphQL

default_cache:allowed_http_verbs:    -GET    -POST    -HEAD

Cache invalidation

Plugins

Beego filter

import ("net/http"httpcache"github.com/darkweak/souin/plugins/beego")funcmain(){// ...web.InsertFilterChain("/*",httpcache.NewHTTPCacheFilter())// ...}

Caddy module

xcaddy build --with github.com/darkweak/souin/plugins/caddy

{    log {        level debug    }    cache {        allowed_http_verbs GET POST PATCH        allowed_additional_status_codes 202        api {            basepath /some-basepath            prometheus {                security            }            souin {                security            }        }        badger {            path the_path_to_a_file.json        }        cache_name Souin        cache_keys {            .*\.something {                disable_body                disable_host                disable_method                disable_query                disable_scheme                disable_vary                headers X-Token Authorization                hide                hash            }        }        cdn {            api_key XXXX            dynamic            email darkweak@protonmail.com            hostname domain.com            network your_network            provider fastly            strategy soft            service_id 123456_id            zone_id anywhere_zone        }        key {            disable_body            disable_host            disable_method            disable_query            disable_scheme            disable_vary            hash            hide            headers Content-Type Authorization        }        log_level debug        etcd {            configuration { # Your Etcd configuration here            }        }        olric {            url url_to_your_cluster:3320            path the_path_to_a_file.yaml            configuration { # Your Olric configuration here            }        }        regex {            exclude /test2.*        }        stale 200s        timeout {          backend 20s          cache 5ms        }        ttl 1000s        default_cache_control no-store    }}:4443respond"Hello World!"@match path/test1*@match2 path/test2*@matchdefault path/default@souin-api path/souin-api*cache@match {    ttl 5s    badger {        path/tmp/badger/first-match        configuration { # Required value            ValueDir <string> # Optional            SyncWrites <bool>            NumVersionsToKeep <int>            ReadOnly <bool>            Compression <int>            InMemory <bool>            MetricsEnabled <bool>            MemTableSize <int>            BaseTableSize <int>            BaseLevelSize <int>            LevelSizeMultiplier <int>            TableSizeMultiplier <int>            MaxLevels <int>            VLogPercentile <float>            ValueThreshold <int>            NumMemtables <int>            BlockSize <int>            BloomFalsePositive <float>            BlockCacheSize <int>            IndexCacheSize <int>            NumLevelZeroTables <int>            NumLevelZeroTablesStall <int>            ValueLogFileSize <int>            ValueLogMaxEntries <int>            NumCompactors <int>            CompactL0OnClose <bool>            LmaxCompaction <bool>            ZSTDCompressionLevel <int>            VerifyValueChecksum <bool>            EncryptionKey <string>            EncryptionKey <Duration>            BypassLockGuard <bool>            ChecksumVerificationMode <int>            DetectConflicts <bool>            NamespaceOffset <int>        }    }}cache@match2 {    ttl 50s    badger {        path/tmp/badger/second-match        configuration {            ValueDir match2            ValueLogFileSize 16777216            MemTableSize 4194304            ValueThreshold 524288            BypassLockGuard true        }    }    default_cache_control"public, max-age=86400"}cache@matchdefault {    ttl 5s    badger {        path/tmp/badger/default-match        configuration {            ValueDir default            ValueLogFileSize 16777216            MemTableSize 4194304            ValueThreshold 524288            BypassLockGuard true        }    }}route/no-method-and-domain.css {    cache {        cache_keys {            .*\.css {                disable_host                disable_method            }        }    }    respond"Hello without storing method and domain cache key"}cache@souin-api {}

Chi middleware

import ("net/http"cache"github.com/darkweak/souin/plugins/chi""github.com/go-chi/chi/v5")funcmain(){// ...router:=chi.NewRouter()httpcache:=cache.NewHTTPCache(cache.DevDefaultConfiguration)router.Use(httpcache.Handle)router.Get("/*",defaultHandler)// ...}

Dotweb middleware

import (cache"github.com/darkweak/souin/plugins/dotweb""github.com/go-dotweb/dotweb/v5")funcmain(){// ...httpcache:=cache.NewHTTPCache(cache.DevDefaultConfiguration)app.HttpServer.GET("/:p",func(ctx dotweb.Context)error {returnctx.WriteString("Hello, World 👋!")}).Use(httpcache)// ...}

Echo middleware

import ("net/http"souin_echo"github.com/darkweak/souin/plugins/echo""github.com/labstack/echo/v4")funcmain(){// ...e:=echo.New()s:=souin_echo.New(souin_echo.DefaultConfiguration)e.Use(s.Process)// ...}

Fiber middleware

import (cache"github.com/darkweak/souin/plugins/fiber""github.com/gofiber/fiber/v2")funcmain(){// ...httpcache:=cache.NewHTTPCache(cache.DevDefaultConfiguration)app.Use(httpcache.Handle)// ...}

Gin middleware

import ("net/http"souin_gin"github.com/darkweak/souin/plugins/gin""github.com/gin-gonic/gin")funcmain(){// ...r:=gin.New()s:=souin_gin.New(souin_gin.DefaultConfiguration)r.Use(s.Process())// ...}

Go-zero middleware

import ("net/http"cache"github.com/darkweak/souin/plugins/go-zero")funcmain(){// ...httpcache:=cache.NewHTTPCache(cache.DevDefaultConfiguration)server.Use(httpcache.Handle)// ...}

Goa middleware

import ("net/http"httpcache"github.com/darkweak/souin/plugins/goa"goahttp"goa.design/goa/v3/http")funcmain(){// ...g:=goahttp.NewMuxer()g.Use(httpcache.NewHTTPCache(httpcache.DevDefaultConfiguration))// ...}

Goyave middleware

import ("net/http"cache"github.com/darkweak/souin/plugins/goyave""goyave.dev/goyave/v4")funcmain() {// ...goyave.Start(func(r*goyave.Router) {r.GlobalMiddleware(cache.NewHTTPCache(cache.DevDefaultConfiguration).Handle)// ...})}

Hertz middleware

import ("context""net/http"// ...httpcache"github.com/darkweak/souin/plugins/hertz")funcmain() {// ...h.Use(httpcache.NewHTTPCache(httpcache.DevDefaultConfiguration))// ...}

Kratos filter

import (httpcache"github.com/darkweak/souin/plugins/kratos"kratos_http"github.com/go-kratos/kratos/v2/transport/http")funcmain() {kratos_http.NewServer(kratos_http.Filter(httpcache.NewHTTPCacheFilter(httpcache.DevDefaultConfiguration),),)}

server: #...data: #...# HTTP cache parthttpcache:  api:    souin: {}  default_cache:    regex:      exclude: /excluded    ttl: 5s  log_level: debug

import (httpcache"github.com/darkweak/souin/plugins/kratos"kratos_http"github.com/go-kratos/kratos/v2/transport/http")funcmain() {c:=config.New(config.WithSource(file.NewSource("examples/configuration.yml")),config.WithDecoder(func(kv*config.KeyValue,vmap[string]interface{})error {returnyaml.Unmarshal(kv.Value,v)}),)iferr:=c.Load();err!=nil {panic(err)}server:=kratos_http.NewServer(kratos_http.Filter(httpcache.NewHTTPCacheFilter(httpcache.ParseConfiguration(c)),),)// ...}

Roadrunner middleware

[velox]build_args = ['-trimpath','-ldflags','-s -X github.com/roadrunner-server/roadrunner/v2/internal/meta.version=${VERSION} -X github.com/roadrunner-server/roadrunner/v2/internal/meta.buildTime=${TIME}'][roadrunner]ref ="master"[github]    [github.token]token ="GH_TOKEN"    [github.plugins]logger = {ref ="master",owner ="roadrunner-server",repository ="logger" }cache = {ref ="master",owner ="darkweak",repository ="souin",folder ="/plugins/roadrunner" }# others ...[log]level ="debug"mode ="development"

# .rr.yamlhttp:# Other http sub keyscache:api:basepath:/httpcache_apiprometheus:basepath:/anything-for-prometheus-metricssouin:{}default_cache:allowed_http_verbs:        -GET        -POST        -HEADallowed_additional_status_codes:        -202        -400cdn:api_key:XXXXdynamic:truehostname:XXXXnetwork:XXXXprovider:fastlystrategy:softregex:exclude:'/excluded'timeout:backend:5scache:1msttl:5sstale:10slog_level:debugykeys:The_First_Test:headers:Content-Type:'.+'The_Second_Test:url:'the/second/.+'surrogate_keys:The_First_Test:headers:Content-Type:'.+'The_Second_Test:url:'the/second/.+'middleware:    -cache# Other middlewares

Skipper filter

package mainimport (souin_skipper"github.com/darkweak/souin/plugins/skipper""github.com/zalando/skipper""github.com/zalando/skipper/filters")funcmain() {skipper.Run(skipper.Options{Address:":9090",RoutesFile:"example.yaml",CustomFilters: []filters.Spec{souin_skipper.NewSouinFilter()}},)}

hello: Path("/hello")   -> httpcache(`{"api":{"basepath":"/souin-api","security":{"secret":"your_secret_key","enable":true,"users":[{"username":"user1","password":"test"}]},"souin":{"security":true,"enable":true}},"default_cache":{"regex":{"exclude":"ARegexHere"},"ttl":"10s","stale":"10s"},"log_level":"INFO"}`)  -> "https://www.example.org"

Træfik plugin

# anywhere/traefik.ymlexperimental:plugins:souin:moduleName:github.com/darkweak/souinversion:v1.7.6

# anywhere/dynamic-configurationhttp:routers:whoami:middlewares:        -http-cacheservice:whoamirule:Host(`domain.com`)middlewares:http-cache:plugin:souin:api:prometheus:{}souin:{}default_cache:regex:exclude:'/test_exclude.*'ttl:5sallowed_http_verbs:              -GET              -HEAD              -POSTallowed_additional_status_codes:              -202              -400default_cache_control:no-storelog_level:debugurls:'domain.com/testing':ttl:5s'mysubdomain.domain.com':ttl:50sdefault_cache_control:public, max-age=86400ykeys:The_First_Test:headers:Content-Type:'.+'The_Second_Test:url:'the/second/.+'The_Third_Test:The_Fourth_Test:surrogate_keys:The_First_Test:headers:Content-Type:'.+'The_Second_Test:url:'the/second/.+'The_Third_Test:The_Fourth_Test:

# anywhere/docker-compose.ymlservices:#...whoami:image:traefik/whoamilabels:# other labels...      -traefik.http.routers.whoami.middlewares=http-cache      -traefik.http.middlewares.http-cache.plugin.souin.api.souin      -traefik.http.middlewares.http-cache.plugin.souin.default_cache.ttl=10s      -traefik.http.middlewares.http-cache.plugin.souin.default_cache.allowed_http_verbs=GET,HEAD,POST      -traefik.http.middlewares.http-cache.plugin.souin.log_level=debug

Tyk plugin

{"name":"httpbin.org","api_id":"3","org_id":"3","use_keyless":true,"version_data": {"not_versioned":true,"versions": {"Default": {"name":"Default","use_extended_paths":true      }    }  },"custom_middleware": {"pre": [],"post": [      {"name":"SouinRequestHandler","path":"/opt/tyk-gateway/middleware/souin-plugin.so"      }    ],"post_key_auth": [],"auth_check": {"name":"","path":"","require_session":false    },"response": [      {"name":"SouinResponseHandler","path":"/opt/tyk-gateway/middleware/souin-plugin.so"      }    ],"driver":"goplugin","id_extractor": {"extract_from":"","extract_with":"","extractor_config": {}    }  },"proxy":{"listen_path":"/httpbin/","target_url":"http://httpbin.org/","strip_listen_path":true  },"active":true,"config_data": {"httpcache": {"api": {"souin": {"enable":true        }      },"cdn": {"api_key":"XXXX","provider":"fastly","strategy":"soft"      },"default_cache": {"ttl":"5s"      }    }  }}

Webgo middleware

import ("net/http""github.com/bnkamalesh/webgo/v6"cache"github.com/darkweak/souin/plugins/webgo")funcmain(){// ...httpcache:=cache.NewHTTPCache(cache.DevDefaultConfiguration)router.Use(httpcache.Middleware)// ...}

Credits