Understanding Graceful Shutdowns on Cloud Run

27 January 2021

Cloud Run

It is a completely managed compute platform which lets us run any stateless containers on serverless.

No need for Infrastructure with cloud run.

It focuses on rapid, automatic scalings that are request aware. So you can scale down to Zero and only pay what you used for.

Diagram:

Graceful shutdowns

It is  generally performed intentionally by users, as a part of their daily routines, at the end of a work day or when finished with home use of a computer.

Basically it is for some safety issues like computer fires or security issues like malware or hacked computers, hard shutdowns may be a safety precaution.

Graceful shutdowns on cloud run

Once the container instance goes shut down on Cloud Run, a SIGTERM signal must be sent to the container and the application will have 10 seconds to exit. If the container does not exit by then, a SIGKILL signal will be sent to unexpectedly close your application. If you choose not to write a signal handler for SIGTERM, the process is terminated instantly.

One can perform various “graceful shutdown” tasks in the application code by Using this termination signal.

Flush monitoring data

If one can  use Cloud Trace or upload metrics from the application, so you can develop a signal handler and call the function that flushes out the trace spans collected before the container quits and loses these in-memory trace spans that are not uploaded.

Log termination of the container

By logging the termination event of the container, please refer to your application logs to see when a specific container instance has started and exited, and get complete visibility into the lifecycle of individual container instances.

Close file descriptors or database connections

Some containers abruptly quit connections that confuses the connected servers and cause them to keep connections open for a long time than gracefully disconnecting.

The graceful termination signal is first sent to the application when it is scaling down the container instances that are traffic-free. Therefore, we don’t need to handle draining in-flight requests in the signal handler. Although, we might sometimes get this signal before the container will be shut down due to underlying infrastructure reasons and the container might still have in-flight connections. Due to which the graceful termination is not always guaranteed.

Trapping signals, the right way

Many computer programming languages provide some libraries to trap the different termination signals like SIGTERM and run routines before the program terminates.

Make sure If the application does not receive the termination signal on Cloud Run, so the most catchy reason for this might be because the underlying application is not running as the init process (PID 1) and its parent process is definitely not forwarding the signal appropriately.

The main  reason why this happens is that the ENTRYPOINT statement in the underlying container image’s Dockerfile is not set directly to the application process.

For example, the Dockerfile statement:

ENTRYPOINT node server.js

internally is transformed to:

ENTRYPOINT ["/bin/sh", "-c", "node server.js"]

when the Dockerfile is executed to build a container image.

Most prominently, the GNU /bin/sh and other shells like bash do not forward signals to child processes by default. Therefore, you must write the entrypoint statements in the vector form, like the following, to prevent your app to be executed as the sub-process of a shell:

ENTRYPOINT ["node", "server.js"]

Similarly, if we use an entrypoint script to kick off background processes in the containers, consider using a proper init process that can forward signals to child processes, such as tini, dumb-init or supervisord.

How graceful shutdowns come in action?

For the example purpose, create a small node js server application for Cloud Run to trap the SIGTERM signals in Cloud Run

Add this snippet of code to index.js:

process.on('SIGTERM', function () {

   console.log('helloworld: received SIGTERM, exiting gracefully');

   process.exit(0);

});

Once  completed the above code , you can now build and push this container image, and deploy it to Cloud Run. As the part of the new deployment a container instance is spun up to handle the request.

After some passing some time, the container will scale down to zero since it has not received till any. (, you can also edit your Cloud Run application’s settings such as CPU or memory on the Google Cloud Console, If you want to trigger a scale-to-zero event. It will deploy a new revision, and the existing old revision will be automatically  turned off

The scale-to-zero will trigger a SIGTERM signal to be sent to the before it goes shut down, and you can see the graceful shutdown routine executed in the Logs tab finally

As we can observe during this process, Cloud Run did not require additional settings to enable graceful shutdowns. It’s turned on by automatically

If you have some cleanup work or you want to monitor the data to push out before the serverless container instances on Cloud Run shutdown, so you can give Signals a try.