Run Docker Container
Intro
Shadeform primarily provides GPUs as virtual machines (VM). Once you launch a Shadeform instance, you’re given the IP address and SSH key of the VM. However, for many use cases, it’s easier to use a pre-built docker container inside of that VM. Shadeform supports a launch configuration in the Create Instance API that allows you to provide a docker container image and properties. The docker container will be automatically launched on the VM.
Shadeform’s Docker feature is not a serverless container. You still have full access to the VM that it is running on.
Basic Example
For this guide, we will be using the vLLM container for LLM model serving on an A6000 GPU.
For a more in-depth guide on using vLLM, see this guide on running vLLM on Shadeform.
To launch a VM instance with the vLLM container automatically started, we need to call the Create Instance API with the launch_configuration property.
In addition to the base Create Instance API call, we are specifying a launch_configuration object.
The launch_configuration.type property must be set to docker and an additional docker_configuration object must be specified.
In the docker_configuration object, we specify image as the image that we want to run and args as any arguments that should be passed to the container image.
This API call will spin up the base A6000 VM and, once the VM is active, pull and launch the docker container. We can get the status and IP address of the instance by calling the Instances API.
The status returned by the Instances API call is the status of the VM instance and not the status of the container. The status of the container is not surfaced through Shadeform’s API.
We can follow the progress of the docker container launch by SSHing into the instance and running:
Once the container is running, you follow the progress of the container by running:
After vLLM is fully launched, you can query vLLM using the request:
Launch Image from Private Registry
If the Docker image you wish to use is in a private Dockerhub repo or private container registry, you must first configure your container registry access here.
Find the Container Registry row in the table and click on Set Up.
Enter your container registry username and password to store your credentials.
We currently only support username and password authentication for container registries. If you require another type of authentication, please reach out to founders@shadeform.ai.
Setting Environment Variables
If your docker container needs environment variables at runtime, you can pass the environment variables to the container via the envs property.
In the example below, we are passing in the HUGGING_FACE_HUB_TOKEN environment variable which is needed by vLLM to access private HuggingFace repos.
Setting Volume Mounts
If you would like to mount a volume from your VM to a filepath on your container, you can specify the mount using the volume_mounts property.
Using volume_mounts is the equivalent of using the -v flag in the Docker cli.
Setting Port Mappings
By default, all containers launched through Shadeform use --network=host which means that the ports of the container will be directly mapped to the VM.
Therefore, if the container runs on port 8000, then accessing the VM’s port 8000 will reach the container.
If you want the container port to be mapped to a different VM port, you can configure port_mappings like in the example below.
In the example, the vLLM container natively runs on port 8000. However, by mapping the port to 80, we can call the API directly using the IP address.
Setting Shared Memory
By default we run containers with --ipc="host" for shared memory. If you would like to limit shared memory, you can configure the property like in the example below:
Security
All container arguments, environment variables, and container registry credentials are encrypted at rest.