# Stealthium

> Runtime observability and security for GPUs and AI workloads. Stealthium is a runtime observability and security platform for GPUs and AI workloads. It turns low-level GPU telemetry — kernel traces, driver events, NVML counters and fabric metrics — into Hyperprints: high-level, actionable signals about AI workload performance and security. Stealthium integrates with CUDA, NVIDIA drivers and toolkits, and the major AI frameworks, across single nodes, multi-GPU instances and multi-cluster deployments.

Stealthium runs alongside GPU workloads and reports on them; it does not schedule, rent or host GPUs. Everything below is served from stealthium.io and needs no credentials to read.

## When to use Stealthium

- A GPU-accelerated training or inference job slowed down and you need to know which layer — application, CUDA runtime, driver, GPU, or fabric — is responsible.
- You need per-workload GPU telemetry (kernel traces, NVML counters, memory and SM utilisation, fabric throughput) correlated with the process, container or pod that produced it.
- You are investigating suspected misuse of GPU capacity — cryptomining on idle accelerators, unauthorised kernels, or a job consuming far more GPU time than it requested.
- You need to detect or triage GPU-specific attack activity at runtime: side channels between MIG instances, malicious or tampered CUDA fatbins, driver-level exploitation.
- You need an audit trail of who ran which AI workload on which GPU, for compliance evidence over a shared or multi-tenant GPU fleet.
- You are evaluating GPU security posture across a multi-node or multi-cluster fleet and need one inventory of accelerators, drivers, toolkit versions and their exposure.

## When not to use Stealthium

- General-purpose CPU/host APM or log aggregation — Stealthium is GPU- and AI-workload-specific and complements, rather than replaces, those tools.
- Provisioning, scheduling or renting GPU capacity — Stealthium observes and secures GPUs, it does not supply them.
- Static analysis of model weights or prompt-level LLM safety — Stealthium works at the GPU runtime layer, not the model-content layer.

## How an agent should call Stealthium

- [Agent instructions](https://stealthium.io/agents.md): When to reach for Stealthium and the exact requests to make.
- [OpenAPI 3.1 document](https://stealthium.io/openapi.json): Every operation on stealthium.io, with typed parameters, response schemas and scopes.
- [Site identity endpoint](https://stealthium.io/api/site): JSON identity, contact details and when-to-use guidance.
- [Developer resources](https://stealthium.io/developers): Human-readable index of every API, spec and machine-readable file.
- [Product documentation](https://docs.stealthium.io): Install the Stealthium client, create an API key and stream GPU telemetry.
- [Quickstart](https://docs.stealthium.io/intro): From nothing to GPU telemetry in Stealthium in under 10 minutes.

## Content

- [Homepage](https://stealthium.io/): What Stealthium observes and secures.
- [Blog index](https://stealthium.io/blog): GPU security research from the Stealthium team.
- [Blog index as JSON](https://stealthium.io/api/posts): The same index, machine-readable.
- [Blog RSS feed](https://stealthium.io/feed.xml): RSS 2.0 for new research.
- [Terms of service](https://stealthium.io/terms): Legal terms for using Stealthium.

## Research

- [Root on Every GPU Container Start: Catching NVIDIA Container Toolkit Compromise](https://stealthium.io/blog/catching-nvidia-container-toolkit-compromise): The NVIDIA Container Toolkit is mandatory, privileged, and almost entirely unmonitored — and most detection content written for it watches binaries that no longer execute. One docker run --gpus all produced six root executions of nvidia-cdi-hook on our test host, before the workload's first instruction. Here is what actually runs on a modern GPU node, and how Stealthium separates legitimate hook execution from abuse.
- [ShadowRay 2.0: Turning Exposed Ray Clusters Into a GPU-Mining Botnet — and How Stealthium Catches It](https://stealthium.io/blog/shadowray-ray-jobs-api-gpu-cryptomining): An internet-exposed Ray dashboard is an open door: an unauthenticated job submission runs attacker code on every node, and within minutes a crypto-miner is pinning your GPUs while your model weights walk out — all while monitoring reports a training job as running. We reproduced the ShadowRay 2.0 kill chain end to end against a real Ray cluster on an NVIDIA L40S and watched Stealthium catch every stage, from the unauthenticated request to the CUDA proof-of-work kernel.
- [Investigating Behind Bars: Model Fingerprinting By Nosy Neighbors.](https://stealthium.io/blog/behind-bars-mig-side-channel-model-fingerprinting): Behind Bars (USENIX Security '26) found a side channel that survives NVIDIA MIG's L2 partitioning: GPU-wide memory barriers cross every partition on the card. We reproduced it independently on current H100 hardware, identified which of five LLMs a neighbor was serving from 0.558 seconds of trace at 97.0% accuracy, and caught the probe live with Stealthium.
- [Across The AI Accelerator-verse: A Need for Security and Observability](https://stealthium.io/blog/stealthium-x-tenstorrent): As the demand and potential of AI scales exponentially, the infrastructure landscape is both fragmenting and flourishing. New neo-cloud providers and silicon providers are reestablishing a multi-cloud, multi-accelerator wave. This evolution of compute has not had a matching evolution of security.
- [Stealthium and Tenstorrent Partner to Deliver Runtime Observability for AI Infrastructure](https://stealthium.io/blog/stealthium-x-tenstorrent-press-release): Stealthium's runtime observability platform integrates with Tenstorrent's open AI compute platform to provide visibility into AI workloads running on Tenstorrent systems.
- [Stopping A Two-Faced Greedy Guest: How Stealthium Protects Against Januscape.](https://stealthium.io/blog/januscape-kvm-guest-to-host-escape): Januscape (CVE-2026-53359) is a shadow-MMU bug in KVM that lets a malicious guest fully escape to the host — on both Intel and AMD from the same trigger. It is invisible to every tool that watches only guests and host userspace. Here is how the attack unfolds, and how Stealthium stops its exploitation before impact.
- [Your GPU Can Be Rooted From Inside a CUDA Kernel. Here Is Exactly How.](https://stealthium.io/blog/gpubreach-lateral-movement-cuda-kernel): An unprivileged process inside a CUDA container can gain arbitrary read/write to every byte of VRAM on a shared GPU — model theft, model poisoning, private inference data exposure. No special permissions, no kernel exploits, invisible to endpoint security. This is GPUBreach: Lateral Movement, Collection, and Impact.
- [GPUBreach: A Root Shell Through GPU Abuse — and How Stealthium detects it.](https://stealthium.io/blog/gpubreach-root-cause-analysis-detection): Before the full GPUBreach paper was published, the Stealthium security team traced the vulnerable path in NVIDIA's open-source kernel modules, confirmed the overflow dimensions with an instrumented kernel build, and had detection in place. This post documents that analysis.
- [The Trillion-Dollar Blind Spot: Inside the Black Box of AI Factories](https://stealthium.io/blog/trillion-dollar-blind-spot): The US government has committed $500 billion to AI infrastructure. Enterprises are dropping eight figures on H100 clusters. And not one of them has a tool that tells them what is actually executing inside those accelerators at runtime.
- [Inside CUDA Fatbins - The Hidden World of GPU Binary Formats - Part 1](https://stealthium.io/blog/fatbins-cuda-gpu-binary-formats-part-1): When compiling a CUDA program, nvcc creates something more complex than a simple executable. This post explores the internals of CUDA's fatbin format — the multi-architecture binary container that makes 'compile once, run anywhere' possible, and the security implications hiding in plain sight.
- [How Stealthium Built a Faster GPU Monitoring Layer](https://stealthium.io/blog/stealthium-gpu-monitor-nvml): NVML is the industry standard for GPU monitoring, but at production scale its per-call overhead compounds fast. Stealthium GPU Monitor is our answer: a monitoring layer we built and own, designed for deep observability with faster startup, lower memory footprint, and richer data than NVML in most cases.
- [The GPU Security Gap Nobody Owns](https://stealthium.io/blog/gpu-security-gap-nobody-owns): Enterprises are deploying AI workloads with zero GPU-level security visibility, caught in a dangerous shared responsibility blind spot. Cloud providers secure the infrastructure—but what runs inside your GPU allocation has always been your responsibility.
- [Substrate AI and Stealthium Partner to Deliver Runtime Security for AI Infrastructure](https://stealthium.io/blog/substrate-ai-partnership): As artificial intelligence infrastructure scales across enterprises and public sector organizations, security, transparency, and compliance have become critical operating requirements. Modern AI environments are highly distributed, GPU-accelerated, and often shared across multiple tenants, yet visibility and runtime protection at the GPU layer remain limited.
- [NVIDIA Driver Vulnerabilities: Deep Dive and Runtime Detection Strategies](https://stealthium.io/blog/nvidia-gpu-vulnerabilities): The AI revolution is happening whether you're on board or not, and that means something you're doing requires GPUs, probably something business critical. And when NVIDIA's kernel modules leaked two privilege-escalation vulnerabilities this fall, most organizations learned about GPU attack surfaces the hard way—through CVE notices rather than telemetry.

## Optional

- [llms-full.txt](https://stealthium.io/llms-full.txt): Every page and post on this site as one markdown file.
- [Sitemap](https://stealthium.io/sitemap.xml): Every indexable URL.