Instance Mangement

This section describes how to manage the instances of a Virtual Cluster. There are at least two instances (“service” and “frontend”) which need to be operating whenever the Virtual Cluster is used (see Subnets). Depending on the scale of the system services might be spread out to more instances than just the “service” and “frontend” instance. In this case these additional instances need to be up and running as well when the system is used. Ephemeral instances in the “backend” subnet(s) are created on demand based on the scientific workload using so-called instance templates. The current status of such ephemeral instances can be displayed by issuing:

sudo -i vc-instance-manager instance list

on the “service” (or the “slurm”) instance. If no entries are returned, no dynamically created instances exist and the Virtual Cluster is “idle”. The cluster can have one or more partitions. A partition is a group of ephemeral instances with certain instance properties (e.g., instance type). The properties of all partitions can be displayed using the command:

sudo -i vc-instance-manager partition list

Dynamic instances can be created and terminated manually using the following commands:

sudo -i vc-instance-manager instance start <instance-name>
sudo -i vc-instance-manager instance stop <instance-name>

where <instance-name> is the internal DNS name of one of the instances in one of the cluster partitions. Note that a number (index) has to be appended to the internal DNS name of compute instances: To start a compute instance for the "cpu-compute" partition, the command is:

sudo -i vc-instance-manager instance start cpu-compute-<index>

However, starting dynamic instances is typically not required as the workload management system of the Virtual Cluster performs these operations automatically. The creation and termination of instances for scaling the Virtual Cluster is logged in the system log (/var/log/messages) on the “service” (or the “slurm”) instance. All events of this type can be inspected by using the following command with root permissions:

sudo grep -E "VIRTUAL-CLUSTER-GROW|VIRTUAL-CLUSTER-SHRINK"

Template Instances

All dynamically created instances are created using so called instance templates. If a setting needs to be changed for the dynamically created and terminated instances, the setting must be changed in a template instance which is then converted into an instance template replacing the previous template as settings in the dynamically created ephemeral instances are not persistent.

A template instance for a certain partition can be started using the following command:

sudo -i vc-instance-manager instance-template start <compute|session>

This starts a template instance for either the compute instances (compute) or for the session instances hosting a virtual desktop. The instance allows ssh login once it is up and running. Once all modifications have been applied, the instance template can be updated using the command:

sudo -i vc-instance-manager instance-template update <compute|session>

If the keyword (compute or session) is not specified, the instance templates for both "session" and "compute" instances will be updated.

Warning

Changing instance templates requires good knowledge of the Virtual Cluster internals. Contact the Schrödinger support team before applying changes if there are any doubts about the modifications applied to the instance templates.

It is also possible to stop template instances and apply the changes later on by using the command:

sudo -i vc-instance-manager instance-template stop <compute|session>

Note

Changes in the instance templates will only apply for instances created after the instance template has been changed. Running instances are not affected.