- Notifications
You must be signed in to change notification settings - Fork96
Graceful shutdown and Kubernetes readiness / liveness checks for any Node.js HTTP applications
License
godaddy/terminus
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Adds graceful shutdown and Kubernetes readiness / liveness checks for any HTTP applications.
Install via npm:
npm i @godaddy/terminus --saveconsthttp=require('http');const{ createTerminus}=require('@godaddy/terminus');functiononSignal(){console.log('server is starting cleanup');returnPromise.all([// your clean logic, like closing database connections]);}functiononShutdown(){console.log('cleanup finished, server is shutting down');}functionhealthCheck({ state}){// `state.isShuttingDown` (boolean) shows whether the server is shutting down or notreturnPromise.resolve(// optionally include a resolve value to be included as// info in the health check response)}constserver=http.createServer((request,response)=>{response.end(`<html> <body> <h1>Hello, World!</h1> </body> </html>`);})constoptions={// health check optionshealthChecks:{'/healthcheck':healthCheck,// a function accepting a state and returning a promise indicating service health,verbatim:true,// [optional = false] use object returned from /healthcheck verbatim in response,__unsafeExposeStackTraces:true// [optional = false] return stack traces in error response if healthchecks throw errors}, caseInsensitive,// [optional] whether given health checks routes are case insensitive (defaults to false) statusOk,// [optional = 200] status to be returned for successful healthchecks statusOkResponse,// [optional = { status: 'ok' }] status response to be returned for successful healthchecks statusError,// [optional = 503] status to be returned for unsuccessful healthchecks statusErrorResponse,// [optional = { status: 'error' }] status response to be returned for unsuccessful healthchecks// cleanup optionstimeout:1000,// [optional = 1000] number of milliseconds before forceful exiting signal,// [optional = 'SIGTERM'] what signal to listen for relative to shutdown signals,// [optional = []] array of signals to listen for relative to shutdown useExit0,// [optional = false] instead of sending the received signal again without beeing catched, the process will exit(0) sendFailuresDuringShutdown,// [optional = true] whether or not to send failure (503) during shutdown beforeShutdown,// [optional] called before the HTTP server starts its shutdown onSignal,// [optional] cleanup function, returning a promise (used to be onSigterm) onShutdown,// [optional] called right before exiting onSendFailureDuringShutdown,// [optional] called before sending each 503 during shutdowns// both logger// [optional] logger function to be called with errors. Example logger call: ('error happened during shutdown', error). See terminus.js for more details.};createTerminus(server,options);server.listen(PORT||3000);
consthttp=require('http');const{ createTerminus, HealthCheckError}=require('@godaddy/terminus');createTerminus(server,{healthChecks:{'/healthcheck':asyncfunction(){consterrors=[]returnPromise.all([// all your health checks goes here].map(p=>p.catch((error)=>{// silently collecting all the errorserrors.push(error)returnundefined}))).then(()=>{if(errors.length){thrownewHealthCheckError('healthcheck failed',errors)}})}}});
consthttp=require("http");constexpress=require("express");const{ createTerminus, HealthCheckError}=require('@godaddy/terminus');constapp=express();app.get("/",(req,res)=>{res.send("ok");});constserver=http.createServer(app);functionhealthCheck({ state}){returnPromise.resolve();}constoptions={healthChecks:{"/healthcheck":healthCheck,verbatim:true,__unsafeExposeStackTraces:true,},headers:{"Access-Control-Allow-Origin":"*","Access-Control-Allow-Methods":"OPTIONS, POST, GET",},};terminus.createTerminus(server,options);server.listen(3000);
consthttp=require('http');constexpress=require('express');const{ createTerminus}=require('@godaddy/terminus');constapp=express();app.get('/',(req,res)=>{res.send('ok');});constserver=http.createServer(app);constoptions={// opts};createTerminus(server,options);server.listen(PORT||3000);
consthttp=require('http');constKoa=require('koa');const{ createTerminus}=require('@godaddy/terminus');constapp=newKoa();constserver=http.createServer(app.callback());constoptions={// opts};createTerminus(server,options);server.listen(PORT||3000);
If you want to use (cluster)[https://nodejs.org/api/cluster.html] to use more than one CPU, you need to useterminus per worker.This is heavily inspired byhttps://medium.com/@gaurav.lahoti/graceful-shutdown-of-node-js-workers-dd58bbff9e30.
Seeexample/express.cluster.js.
When Kubernetes or a user deletes a Pod, Kubernetes will notify it and wait forgracePeriod seconds before killing it.
During that time window (30 seconds by default), the Pod is in theterminating state and will be removed from any Services by a controller.The Pod itself needs to catch theSIGTERM signal and start failing any readiness probes.
If the ingress controller you use route via the Service, it is not an issue for your case. At the time of this writing, we use the nginx ingress controller which routes traffic directly to the Pods.
During this time, it is possible that load-balancers (like the nginx ingress controller) don't remove the Pods "in time", and when the Pod dies, it kills live connections.
To make sure you don't lose any connections, we recommend delaying the shutdown with the number of milliseconds that's defined by the readiness probe in your deployment configuration.To help with this, terminus exposes an option calledbeforeShutdown that takes any Promise-returning function.
Also it makes sense to use theuseExit0 = true option to signal Kubernetes that the container exited gracefully.Otherwise APM's will send you alerts, in some cases.
functionbeforeShutdown(){// given your readiness probes run every 5 second// may be worth using a bigger number so you won't// run into any race conditionsreturnnewPromise(resolve=>{setTimeout(resolve,5000)})}createTerminus(server,{ beforeShutdown,useExit0:true})
Due to inherent platform limitations,terminus has limited support for Windows.You can expectSIGINT to work, as well asSIGBREAK and to some extentSIGHUP.HoweverSIGTERM will never work on Windows because killing a process in the task manager is unconditional, i.e., there's no way for an application to detect or prevent it.Here's some relevant documentation fromlibuv to learn more about whatSIGINT,SIGBREAK etc. signify and what's supported on Windows - http://docs.libuv.org/en/v1.x/signal.html.Also, seehttps://nodejs.org/api/process.html#process_signal_events.
About
Graceful shutdown and Kubernetes readiness / liveness checks for any Node.js HTTP applications