Instance Mangement

This section describes how to manage the instances of a Virtual Cluster. There are at least two instances (“service” and “frontend”) which need to be operating whenever the Virtual Cluster is used (see Subnets). Depending on the scale of the system services might be spread out to more instances than just the “service” and “frontend” instance. In this case these additional instances need to be up and running as well when the system is used. Ephemeral instances in the “backend” subnet(s) are created on demand based on the scientific workload using so-called instance templates. The current status of such instances can be displayed by issuing:

vc-instance-manager instance list

on the “service” (or the “slurm”) instance. If no entries are returned, no dynamically created instances exist and the Virtual Cluster is “idle”. The cluster can have one or more partitions. A partition is a group of identical ephemeral instances with certain properties. The properties of all partitions can be displayed using the command:

vc-instance-manager partition list

Dynamic instances can be created and terminated manually using the following commands:

vc-instance-manager instance start <instance-name>
vc-instance-manager instance stop <instance-name>

where “” is the internal DNS name of one of the instances in one of the cluster partitions. Note that a number has to be appended to the internal DNS name of compute instances: To start a compute instance for the “cpu-compute” partition, the command is:

vc-instance-manager instance start cpu-compute-1

However, starting dynamic instances is typically not required as the workload management system of the Virtual Cluster performs these operations automatically. The creation and termination of instances for scaling the Virtual Cluster is logged in the system log (/var/log/messages) on the “service” (or the “slurm”) instance. All events of this type can be inspected by using the following command with root permissions:

cat /var/log/messages | egrep "VIRTUAL-CLUSTER-GROW|VIRTUAL-CLUSTER-SHRINK"

Template Instances

All dynamically created instances are created using so called instance templates. If a setting needs to be changed for the dynamically created and terminated instances, the setting must be changed in a template instance which is then converted into an instance template replacing the previous template as settings in the dynamically created instances are not persistent.

A template instance for a certain partition can be started using the following command:

vc-instance-manager instance-template start <partition-name>

This starts a template instance for the specified partition, i.e., a virtual machine with the name “”. Note that the 2023-2 version of the Virtual Cluster introduced a single template instance for all compute partitions. The instance allows ssh login once it is up and running. Once all modifications have been applied, the instance template for the particular partition can be updated using the command:

vc-instance-manager instance-template update <partition-name>

If the particular partition is not specified, the Virtual Cluster automation will update the instance templates for all partitions for which template instances exist.

Warning

Changing instance templates requires good knowledge of the Virtual Cluster internals. Contact the Schrödinger support team before applying changes if there are any doubts about the modifications applied to the instance templates.

It is also possible to stop template instances and apply the changes later on by using the command:

vc-instance-manager instance-template stop <partition-name>

Note

Changes in the instance templates will only apply for instances created after the instance template has been changed. Running instances are not affected.