Graceful shutdown

Trino has a graceful shutdown API that can be used exclusively on workers in order to ensure that they terminate without affecting running queries, given a sufficient grace period.

You can invoke the API with a HTTP PUT request:

# When using no authorization:
curl -v -X PUT -d '"SHUTTING_DOWN"' -H 'Content-type: application/json' -H 'X-Trino-User: authorizeduser' \
    http://worker:8081/v1/info/state

# Or when using basic auth:
curl -v -X PUT -d '"SHUTTING_DOWN"' -H 'Content-type: application/json' -u 'authorizeduser:password' \
    https://worker/v1/info/state

A successful invocation is logged with a Shutdown requested message at INFO level in the worker server log.

Keep the following aspects in mind:

If your cluster is secure, you need to provide a basic-authorization header, or satisfy whatever other security you have enabled.
If you have TLS/HTTPS enabled, you have to ensure the worker certificate is CA signed, or trusted by the server calling the shut down endpoint. Otherwise, you can make the call --insecure, but that isn't recommended.
The default System access control does not allow graceful shutdowns. You can use the allow-all system access control, configure system information rules with the file system access control or use Apache Ranger Ranger access control. Note that the access control configuration must be present on all workers.
The user must have permissions to write "system information". In Apache Ranger this permission can be found under "System Information - Write System Information".

Shutdown behavior

Once the API is called, the worker performs the following steps:

Go into SHUTTING_DOWN state.
Sleep for shutdown.grace-period, which defaults to 2 minutes. : - After this, the coordinator is aware of the shutdown and stops sending tasks to the worker.
Block until all active tasks are complete.
Sleep for the grace period again to ensure that the coordinator sees that all tasks are complete.
Shutdown the application.

Shutdown behavior​

Shutdown behavior