Basics of Graceful Shutdowns on Cloud Run

24 March 2022

Cloud Run

It is a totally managed compute platform which lets us run any stateless containers on serverless.No need for Infrastructure with cloud run.

It focuses on rapid, automatic scalings that are request aware. So you’ll scale right down to Zero and only pay what you used for.

Graceful shutdowns

It is generally performed intentionally by users, as a neighborhood of their daily routines, at the end of a piece day or when finished with home use of a computer. 

Basically it’s for some questions of safety like computer fires or security issues like malware or hacked computers, hard shutdowns could even be a security precaution.

Once the container instance goes close up on Cloud Run, a SIGTERM signal must be sent to the container and therefore the application will have around 10 seconds to exit. If the container doesn’t exit by then, a SIGKILL signal goes to be sent to unexpectedly close your application. If you select to not write a sign handler for SIGTERM, the method is terminated instantly.

One can perform various “graceful shutdown” tasks within the application code by Using this termination signal.

• Flush monitoring data:

If one can use Cloud Trace or upload metrics from the appliance, so you’ll develop a sign handler and call the function that flushes out the trace spans collected before the container quits and loses these in-memory trace spans that aren’t uploaded.

• Log termination of the container:

By logging the termination event of the container, please ask your application logs to ascertain when a selected container instance has started and exited, and obtain the entire visibility into the lifecycle of individual container instances.

• Close file descriptors or database connections:

Some containers abruptly quit connections that confuse the connected servers and cause them to stay connections open for an extended time rather than gracefully disconnecting.

The graceful termination signal is first sent to the application when it’s cutting down the container instances that are traffic free. Therefore, we do not have to handle draining in-flight requests within the signal handler. Although, we’d sometimes get this signal before the container is going to be packed up thanks to underlying infrastructure reasons and therefore the container might still have in-flight connections. thanks to which the graceful termination isn’t always guaranteed.

Trapping signals, the right way

Many programming languages provide some libraries to trap the variety of termination signals like SIGTERM and run routines before the program terminates.

Make sure if the application doesn’t receive the termination signal on Cloud Run, therefore the most probable reason for this could be because the underlying application isn’t running because the init process (PID 1) and its parent process is certainly not forwarding the signal appropriately.

The main reason why this happens is that the ENTRYPOINT statement within the underlying container image’s Dockerfile isn’t set on to the application process. 

For example, the Dockerfile statement:

ENTRYPOINT node server.js

internally is transformed to:

ENTRYPOINT [“/bin/sh”, “-c”, “node server.js”]

when the Dockerfile is executed to build a container image.

Most prominently, the GNU /bin/sh and other shells like bash don’t forward signals to child processes by default. Therefore, you want to write the entrypoint statements within the vector form, just like the following, to stop your app to be executed because the sub-process of a shell:

ENTRYPOINT [“node”, “server.js”]

Similarly, if we use an entrypoint script to begin background processes within the containers, think about using a correct init process which will forward signals to child processes, like tini, dumb-init or supervisord.

How graceful shutdowns come in action

For the instance purpose, create small node js server application for Cloud Run to trap the SIGTERM signals in Cloud Run

Add this snippet of code to index.js:

process.on('SIGTERM', function () {

   console.log('helloworld: received SIGTERM, exiting gracefully');

   process.exit(0);

});

Once completed the above code , you’ll now build and push this container image, and deploy it to Cloud Run. As a part of the new deployment a container instance is spun up to handle the request.

After some passing a little time, the container will scale right down to zero since it’s not received any. ( you’ll also edit your Cloud Run application’s settings like CPU or memory on the Google Cloud Console, If you would like to trigger a scale-to-zero event. it’ll deploy a replacement revision, and therefore the existing old revision is going to be automatically turned off.

The scale-to-zero will trigger a SIGTERM signal to be sent to the before it goes pack up , and you’ll see the graceful shutdown routine executed within the Logs tab finally.

As we will observe during this process, Cloud Run didn’t require additional settings to enable graceful shutdowns. It’s turned on automatically. 

If you are working on some cleanup work or you want to monitor the data to push it out before the serverless container instances on Cloud Run shutdown, so you can give Signals a try.