The AI platform for teams on Kubernetes
forgeai gives every team a project on a shared GPU cluster, with quotas, network isolation and member access. Training jobs, sandboxes and volumes are ordinary Kubernetes objects, so they work with the tools you already run.
- Kubeflow Trainer v2
- Kueue
- agent-sandbox
- Harbor
- dufs
apiVersion: forgeai.wutz.dev/v1alpha1
kind: Project
metadata:
name: demo
spec:
displayName: Demo project
quota:
gpus:
- class: rtx-5090
count: 8
storage: 1Ti
volumes: 10
network:
isolation: Isolated
members:
- name: alice
role: edit
deletionPolicy: RetainEverything a shared AI cluster needs, in one place
forgeai is a Kubernetes operator with a desktop console. It adds the parts that Trainer, Kueue and the sandbox controller leave out: tenancy, lifecycle, retries and access.
Multi-tenant projects
A Project is a tenant: its own namespace, GPU quota per class, storage and volume limits, network isolation, a registry robot and members with edit or view access.
Distributed training
Jobs are plain Kubeflow Trainer v2 TrainJobs, admitted by Kueue. Jobs over quota wait in the queue instead of failing.
Failure handling
Stuck jobs are detected and stopped. Failures are classified (node failure, eviction, OOM, image pull, config, exit code) and retried when the policy allows.
Agent sandboxes
Interactive workloads run as agent-sandbox Sandboxes under gVisor or Kata. Idle sandboxes suspend, and a TTL stops them on schedule.
Volumes and files
Shared ReadWriteMany volumes resize, go to a trash with an expiry, and restore. An on-demand file server browses them and scales to zero when idle.
Instance types
GPU classes and sizes are generated from the nodes of the cluster, with live capacity per node. Members pick an instance type; they never edit the catalogue.
No API server of its own
The operator reconciles Kubernetes objects. The desktop console and kubectl create those objects with each member's own credentials, so Kubernetes RBAC decides what everyone can do.
An admin creates a Project
Quota, network isolation, trash retention and the list of members go into one cluster-scoped resource.
The operator provisions the tenant
It creates the namespace, queues, quotas, network policy, registry project and member credentials.
Members work inside their project
They submit jobs, start sandboxes and manage volumes with their own kubeconfig. They cannot change quotas, policies or access.
| Object | Name | Purpose |
|---|---|---|
| Namespace | forge-<project> | Pod security baseline, network policies and labels that tie it to the project. |
| ClusterQueue + LocalQueue | forge-<project> / default | GPU quota per class. Jobs over quota wait in the queue. |
| ResourceQuota | forgeai-quota | Backstop for workloads that bypass Kueue: GPUs, CPU, memory, storage, volumes and pods. |
| NetworkPolicy | forgeai-isolation | Ingress only from the namespace and platform namespaces; optional internet egress block. |
| Registry project + robot | forge-<project> | Harbor project with a pull secret and push credentials, plus a storage quota. |
| Member ServiceAccount | forgeai-members/<project>.<member> | A token bound to the project edit or view role. Removing the member revokes it. |
Jobs that recover on their own
Training runs as Kubeflow Trainer v2 TrainJobs in the project namespace, queued by Kueue. The operator adds the lifecycle features Trainer does not provide, so a long run does not need someone watching it.
- Stop and start release the pods and keep the job, then reactivate its Kueue workload.
- Rerun submits a new job from the form of an existing one.
- Stuck detection stops jobs whose pods cannot start, such as image pull errors or crash loops.
- Retries resubmit failed jobs as
<group>-r<n>up to a limit. Infrastructure failures retry by default.
| Failure reason | Retried by default |
|---|---|
| NodeFailure | Yes |
| Evicted | Yes |
| OOMKilled | Only with retry-on: Any |
| ExitCode | Only with retry-on: Any |
| ImagePullFailed | Never |
| ConfigError | Never |
Interactive work without idle GPUs
- Sandboxes are agent-sandbox resources with gVisor or Kata isolation, a bash command, env, optional GPU placement and project volumes.
- Idle suspend stops a sandbox nothing has used for its timeout. Open clients send a heartbeat to keep it awake.
- Volumes are PVCs from the platform storage types. They can grow (never shrink) and go to a trash with an expiry before they are purged.
- File server serves each volume over HTTPS with basic auth and scales to zero when it is not in use.
apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
name: agent
namespace: forge-demo
labels:
forgeai.wutz.dev/project: demo
spec:
operatingMode: Running
podTemplate:
spec:
runtimeClassName: gvisor
containers:
- name: sandbox
image: <registry>/forge-demo/agent:latestInstall the operator and the console
forgeai needs a Kubernetes cluster with Kubeflow Trainer v2 and Kueue installed. Sandboxes need agent-sandbox, and project registries need Harbor. The steps below run from a checkout of the repository.
1. Install the CRDs and the operator
Targets the cluster in your current kubeconfig context. Replace the image with one your cluster can pull.
make install
make deploy IMG=<registry>/forgeai:<tag>2. Create a project
The repository ships a sample Project you can adapt.
kubectl apply -f config/samples/forgeai_v1alpha1_project.yaml
kubectl get fproj3. Build the desktop console
The console is a Wails app that talks to the cluster through a kubeconfig context.
make guiFull reference: the README covers the object model, andgui/ covers the console.