diff --git a/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc b/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc new file mode 100644 index 000000000..6059f0c8d --- /dev/null +++ b/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc @@ -0,0 +1,101 @@ +--- +# The directory name, the aliases, and the GitHub links in this page set use the +# current repository name, rhvp-vllm-ray-inference-server. That name can change, +# pending two open questions to the Validated Patterns team: whether the "rhvp-" +# prefix is acceptable under the Red Hat trademark guidelines, and whether the +# repository moves to the validatedpatterns-sandbox organization. If either +# changes, rename the directory, the aliases, the image folder, and every link +# in both pages together. +title: vLLM Ray Inference Server +date: 2026-11-08 +validated: false +tier: sandbox +summary: This pattern deploys an OpenAI-compatible large language model (LLM) server that runs vLLM on Ray Serve with one GPU, built inside the cluster from upstream packages and managed by OpenShift GitOps. +rh_products: + - Red Hat OpenShift Container Platform + - Red Hat OpenShift GitOps +# partners: left out on purpose. The pattern deploys software from NVIDIA (the +# GPU Operator and the CUDA base image) and from the Ray and vLLM projects, but +# no partner organization takes part in the pattern. A partners entry suggests a +# partnership, and the site prints a partner trademark notice for it. Add an +# entry only when a partner organization joins the pattern. +industries: + - General +focus_areas: + - AI +aliases: /rhvp-vllm-ray-inference-server/ +links: + github: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server + install: getting-started + arch: /images/rhvp-vllm-ray-inference-server/architecture.svg + bugs: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues + feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform +--- +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +:vllm-ray-pattern: vLLM Ray Inference Server pattern + +// The pattern name can change. See the comment at the top of the front matter. + +[id="about-vllm-ray-inference-server-pattern"] += About the {vllm-ray-pattern} + +The {vllm-ray-pattern} deploys a private, OpenAI-compatible large language model (LLM) endpoint on {rh-ocp}. {rh-gitops} deploys every component from your fork of the pattern repository, and the cluster builds the server image itself from upstream packages. Eugo maintains the pattern as community content in the {sandbox-tier-first}. + +[WARNING] +==== +On {AWS}, the default installation creates one billable GPU instance: a `g6.2xlarge` instance with one NVIDIA L4 GPU. AWS bills the instance for as long as it runs. To skip the instance, set `global.gpuMachineSet: none` in the `values-global.yaml` file in your fork, and push the change before you install the pattern. On other platforms, the pattern creates no instance. Uninstalling the pattern does not delete the instance. For the steps to delete it, see link:getting-started/#uninstalling-the-vllm-ray-pattern[Uninstalling the pattern]. +==== + +[id="background-vllm-ray"] +== Background + +A team that wants a private, OpenAI-compatible LLM endpoint on its own {ocp} cluster usually assembles the parts by hand: GPU enablement, a serving runtime, a model cache, a secret for gated models, and a way to rebuild the server image when the software stack changes. This pattern describes all of these parts in one Git repository. You fork the repository, set the model, and push the change, and the cluster converges to a running endpoint. + +The pattern pins every Python package by hash and the base image by digest, so each rebuild installs the same software stack, and each change is a reviewable commit. The pattern uses upstream open source components only: vLLM, Ray Serve, and Python wheels from PyPI, on the NVIDIA CUDA image that is based on Red{nbsp}Hat Universal Base Image 9. + +[id="architecture-vllm-ray"] +== Architecture + +.Architecture of the {vllm-ray-pattern} +image::/images/rhvp-vllm-ray-inference-server/architecture.svg[Diagram of the vLLM Ray Inference Server pattern. The Validated Patterns Operator and OpenShift GitOps deploy secrets management, GPU enablement, the inference server, and the notebook from your Git fork.] + +The pattern works in the following way: + +* The {validated-patterns-op} installs {rh-gitops} and points the Argo CD instance at your fork. The `clustergroup` Helm chart then creates every application that the `prod` variant lists. +* {hashicorp-vault} and the {eso-op} deliver an optional Hugging Face token to the server as a Kubernetes secret. The default model is not gated and needs no token. +* The {rh-nfd} and the NVIDIA GPU Operator prepare the GPU node. On AWS, an imperative job creates the GPU node. +* A `BuildConfig` object builds the server image inside the cluster. The server deployment runs one pod with a Ray head, Ray Serve, and the vLLM engine on one GPU. The pod reports `Ready` only after the model loads. +* A persistent volume claim keeps the model weights across pod restarts. +* A JupyterLab notebook runs behind the OpenShift OAuth proxy. Only users who can edit the `rhvp-notebook` namespace can open it. The notebook sends requests to the server through the in-cluster service. + +[id="components-vllm-ray"] +== Components that the pattern deploys + +The pattern deploys the following Operators: + +* {rh-gitops} +* The {validated-patterns-op}, a community Operator +* The {eso-op} for Red{nbsp}Hat OpenShift +* The {rh-nfd} +* The NVIDIA GPU Operator, from the certified Operators catalog + +The pattern deploys the following applications: + +* {hashicorp-vault}, from the {solution-name-upstream} Helm chart. +* The inference server. Ray Serve LLM runs the vLLM engine and serves an OpenAI-compatible API on port 8000. The default model is `Qwen/Qwen2.5-0.5B-Instruct`. The server image turns off the usage statistics of Ray and vLLM. +* The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, and the token throughput at several concurrency levels. + +The link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. + +[id="support-vllm-ray"] +== Support + +The {vllm-ray-pattern} is community content in the {sandbox-tier-first}. Eugo maintains the pattern and supports it on a best-effort basis through link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues[GitHub issues] in the pattern repository. The maintainers do not guarantee a response time. Red{nbsp}Hat does not produce, support, or validate this pattern. For an issue with a product that the pattern deploys, contact the vendor of that product. + +[id="next-steps-vllm-ray"] +== Next steps + +* link:getting-started[Getting started] diff --git a/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc b/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc new file mode 100644 index 000000000..9d1592400 --- /dev/null +++ b/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc @@ -0,0 +1,206 @@ +--- +# The pattern name in this page can change. See the comment at the top of the +# front matter in _index.adoc in this folder. +title: Getting started +weight: 10 +aliases: /rhvp-vllm-ray-inference-server/getting-started/ +--- + +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +:vllm-ray-pattern: vLLM Ray Inference Server pattern + +[id="deploying-the-vllm-ray-pattern"] +== Deploying the {vllm-ray-pattern} + +You deploy the pattern from your own fork of the pattern repository. {rh-gitops} deploys the content of your fork on GitHub, not the files on your workstation, so commit and push every change before you install the pattern. + +.Prerequisites + +* An {ocp} 4.18 or later cluster, and cluster administrator access. To create a cluster, go to the link:https://console.redhat.com/[{hybrid-console-first}] and select *Services -> Containers -> Create cluster*. +* A storage class that provisions persistent volumes dynamically. The model cache uses one persistent volume claim. +* The {ocp-registry} in the `Managed` state, and the `Build` and `ImageRegistry` cluster capabilities enabled. The cluster builds the server image and the notebook image itself. On bare metal and on vSphere, the registry starts in the `Removed` state. To check the registry state, run the following command: ++ +[source,terminal] +---- +$ oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec.managementState}' +---- +* A GPU node: +** On AWS, the pattern creates one `g6.2xlarge` node with one NVIDIA L4 GPU by default. AWS bills this instance for as long as it runs. +** On an AWS cluster that was installed into existing subnets, the job that creates the GPU node fails. Set `global.gpuMachineSet: none` and add a GPU node yourself. +** On other platforms, add a node with an NVIDIA GPU that the NVIDIA GPU Operator supports. +* The server runs on `amd64` nodes by default. If your GPU node is an `arm64` node, read the prerequisites in the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md#prerequisites[AGENTS.md file] in the pattern repository before you install. +* On your workstation, Podman 4.3 or later, Git, and the `oc` CLI, logged in to the cluster. If you set the `KUBECONFIG` environment variable, it must point to a file in your home directory. For the other installation tool dependencies, see link:https://validatedpatterns.io/learn/quickstart/[Patterns quick start]. + +.Procedure + +. Fork the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server[rhvp-vllm-ray-inference-server] repository on GitHub. + +. Clone your fork: ++ +[source,terminal,subs="+quotes"] +---- +$ git clone https://github.com/____/rhvp-vllm-ray-inference-server.git +---- + +. Go to the root directory of the repository: ++ +[source,terminal] +---- +$ cd rhvp-vllm-ray-inference-server +---- + +. Edit the `values-global.yaml` file: +** Set `global.gpuMachineSet` to `create` to let the pattern create one AWS `g6.2xlarge` GPU node, or to `none` to use a GPU node that you provide. +** Optional: To serve a different model from Hugging Face, set `global.model.id`, `global.model.source`, and `global.model.revision`. + +. Commit and push the change: ++ +[source,terminal] +---- +$ git commit -am "Configure the pattern" +$ git push +---- + +. Optional: To serve a gated model, create the secrets file for the pattern and add your Hugging Face token: +.. Copy the secrets template: ++ +[source,terminal] +---- +$ mkdir -p ~/.config/validated-patterns +$ cp values-secret.yaml.template \ + ~/.config/validated-patterns/values-secret-rhvp-vllm-ray-inference-server.yaml +---- +.. In the copied file, set the `value` of the `hftoken` field to your Hugging Face token, which starts with `hf_`. Do not commit this file. + +. Log in to the cluster: ++ +[source,terminal,subs="+quotes"] +---- +$ oc login --token=____ --server=____ +---- + +. Install the pattern: ++ +[source,terminal] +---- +$ ./pattern.sh make install +---- ++ +The installer waits up to 60 minutes for every Argo CD application to report `Synced` and `Healthy`. A first installation can take longer, because the cluster builds a large server image, the GPU node joins the cluster, and the server downloads the model. If the wait times out, the pattern stays installed. Check the progress with the commands in the next step, and then run `./pattern.sh make argo-healthcheck`. + +. Watch the deployment: ++ +[source,terminal] +---- +$ oc get applications.argoproj.io -A +$ oc get builds -n rhvp-inference +$ oc get pods -n rhvp-inference -w +---- ++ +The server pod stays in the `Pending` state until a node advertises the `nvidia.com/gpu` resource. The pod reports `Ready` after it loads the model. + +[id="verifying-the-vllm-ray-pattern"] +== Verifying the inference server + +The server is reachable inside the cluster at `http://ray-serve-llm.rhvp-inference.svc:8000`. To send a test request from your workstation, forward the server port. + +.Procedure + +. Forward the server port to your workstation: ++ +[source,terminal] +---- +$ oc port-forward svc/ray-serve-llm 8000:8000 -n rhvp-inference +---- + +. In another terminal, list the models that the server serves: ++ +[source,terminal] +---- +$ curl -s http://localhost:8000/v1/models +---- + +. Send a chat request: ++ +[source,terminal] +---- +$ curl -s http://localhost:8000/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "qwen2.5-0.5b-instruct", + "messages": [{"role": "user", "content": "Say hello in one sentence."}], + "max_tokens": 64 + }' +---- ++ +The `model` field takes the value of `global.model.id`. Any OpenAI-compatible client works with the base URL `http://localhost:8000/v1`. + +[id="opening-the-notebook-vllm-ray"] +== Opening the notebook + +The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can edit the `rhvp-notebook` namespace, and cluster administrators, can open it. + +.Procedure + +. Get the URL of the notebook route: ++ +[source,terminal] +---- +$ oc get route -n rhvp-notebook +---- + +. Open the URL in a browser and log in with your OpenShift account. + +. Open the `stock_vs_eugo_inference.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. + +. Optional: To give another user access to the notebook, grant the `edit` role in the `rhvp-notebook` namespace: ++ +[source,terminal,subs="+quotes"] +---- +$ oc adm policy add-role-to-user edit ____ -n rhvp-notebook +---- + +[id="uninstalling-the-vllm-ray-pattern"] +== Uninstalling the pattern + +The {solution-name-upstream} framework marks uninstallation as experimental. Uninstalling the pattern leaves the {validated-patterns-op} and {rh-gitops} installed, and it does not delete the AWS GPU node. AWS bills the GPU instance until you delete its machine set. + +.Procedure + +. Uninstall the pattern: ++ +[source,terminal] +---- +$ ./pattern.sh make uninstall +---- + +. List the machine sets with their instance types: ++ +[source,terminal] +---- +$ oc get machinesets -n openshift-machine-api \ + -o custom-columns=NAME:.metadata.name,TYPE:.spec.template.spec.providerSpec.value.instanceType +---- + +. Delete the machine set of the `g6.2xlarge` instance type. Delete it only after the uninstallation finishes, or after the `global.gpuMachineSet: none` setting syncs. Otherwise, the job creates the machine set again. ++ +[source,terminal,subs="+quotes"] +---- +$ oc delete machineset ____ -n openshift-machine-api +---- + +. Check for a leftover persistent volume from the `rhvp-inference/ray-serve-llm-models` claim: ++ +[source,terminal] +---- +$ oc get pv +---- + +[id="next-steps-vllm-ray-getting-started"] +== Next steps + +* For the web console installation, private forks, updates, and troubleshooting, see the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md[AGENTS.md file] in the pattern repository. +* For the pattern overview, see link:../[{vllm-ray-pattern}]. diff --git a/static/images/rhvp-vllm-ray-inference-server/architecture.svg b/static/images/rhvp-vllm-ray-inference-server/architecture.svg new file mode 100644 index 000000000..9ddcbee4b --- /dev/null +++ b/static/images/rhvp-vllm-ray-inference-server/architecture.svg @@ -0,0 +1 @@ +OpenShift clusterNamespacerhvp-notebookNamespacerhvp-inferenceGPU enablementSecretsGithftokenSecrethuggingface-secretmodel weights at/modelsOpenAI-compatible HTTPYour fork of this repositoryYou, in a browserValidated Patterns OperatorOpenShift GitOps: Argo CD instancevp-gitopsclustergroup app-of-apps, variant prodHashiCorp VaultExternal Secrets OperatorNode Feature DiscoveryNVIDIA GPU OperatorMachineSet job, AWS only: oneg6.2xlarge with one NVIDIA L4BuildConfig ray-serve-llmImageStream ray-serve-llmDeployment ray-serve-llm: Ray head +Ray Serve + vLLM, one GPUPVC ray-serve-llm-modelsService ray-serve-llm, port 8000BuildConfig and ImageStreamrhvp-notebookNotebook pod: JupyterLab +oauth-proxyRoute with OpenShift login \ No newline at end of file