NVIDIA introduces generative AI into surgical scenarios, launching the world's first real-time interactive surgical simulator trained on actual gallbladder removal videos, marking a milestone in surgical robot simulation research that has reached a clinical-level visual environment.

NVIDIA released Cosmos3-H-Dreams on October 20th. This system learns visual dynamics from clinical surgical videos on the CMR Surgical Versius platform and is capable of generating new frames for surgical scenarios in real-time based on instrument control instructions, rather than simply playing back existing surgical videos. The system can run interactively on a single RTX PRO with 6000 GPU.

NVIDIA clearly states that Cosmos3-H-Dreams is a research-oriented simulator that has not yet been validated for patient care and is not suitable for clinical diagnosis and treatment. Visual realism does not equate to biomechanical accuracy. The system currently only covers specific surgical procedures and instrument combinations, and there are still multiple validation tasks that need to be completed before it can be applied clinically. For investors interested in the medical AI sector, this system represents more of NVIDIA's technological layout in the fields of physical AI and surgical robot simulation, rather than a near-term commercialization.

From desktop integration to clinical applications

The predecessor of Cosmos3-H-Dreams, Cosmos-H-Dreams, focused on the desktop stitching tasks of the dVRK platform, and had a relatively simple background. However, once entering the human body cavity, the visual complexity increases significantly: tissues deform under tension, reflections change as instruments move, anatomical structures are obscured, and surgical smoke affects visibility.

The training data selected by NVIDIA this time comes from videos of clinical cholecystectomies performed using the Versius platform, focusing on two visible instruments that are in operation: the left Cadiere forceps and the right monopolar hook. The model is based on videos with a resolution of 832×480, with 28 control dimensions covering the posture of both arms, the state of the clamps, energy control, and joystick input. The motion data is aligned with the video at a frequency of 10Hz.

NVIDIA emphasizes that visual realism is the starting point of verification, not proof of biomechanical accuracy. The generated images must also respond to the operator's device control commands—commands for translation, rotation, and gripping need to produce reasonable anatomical consequences. A system that ignores control responses in favor of merely visual realism cannot serve as an effective interactive environment.

From Teacher Models to Real-Time Inference

Cosmos3-H-Dreams is built upon Cosmos3-H-Embodied-Nano, which is a 16 billion parameter model pre-trained on the Open-H-Embodiment dataset. NVIDIA employs a "teacher-student" distillation pipeline, first fine-tuning the bidirectional teacher model over 24-frame, and then 72-frame prediction spans to enable it to fully learn surgical dynamics; subsequently, through distillation, the behavior is transferred to the causal student model, allowing it to generate videos piece by piece as new actions occur.

To address the issue of error accumulation over time in autoregressive generation, NVIDIA introduced the Self-Forcing technology. This technology allows student models to use the historical frames they generate themselves as context during training, and is guided jointly by a teacher model and an acquired evaluator, thereby suppressing the phenomenon of gradual drift in device shape or anatomical structure across generations of generation.

Stream inference is driven by the FlashDreams library. To address the issue of continuously expanding attention history during long-term generation, the system has integrated a modified FadeMem module, which compresses and integrates key-value caches during the stream generation process, complementing Self-Forcing. The former focuses on how the model learns from the generation history, while the latter deals with how this history is stored during inference.

Two checkpoints: speed or quality can be chosen.

NVIDIA is simultaneously releasing two independently trained inference checkpoints this time.

The four-step checkpoint performs denoising steps four times for each video generated, prioritizing visual quality; the two-step checkpoint, on the other hand, reduces the number of steps to two through further specialized training, sacrificing some visual quality in exchange for faster inference speed. NVIDIA specifically notes that the two-step model is not obtained by directly reducing the sampling steps from the four-step checkpoint, but rather through independent training.

Both checkpoints are open to developers and come with a complete teacher fine-tuning and Self-Forcing distillation training scheme, supporting adaptation to other datasets that have synchronized video and action data. When connecting custom controllers, the action conventions and normalization settings must be consistent with those used during training.

Clear research value, but a long way to go in clinical application

NVIDIA positions Cosmos3-H-Dreams as a controllable simulation research platform, whose core value lies in providing a generative environment for surgical robot research that can respond to device commands and possesses clinical-level visual complexity. This environment supports the exploration of trajectories not seen in training data, as well as the study of device responsiveness, consistency in tissue interaction, and temporal coherence of anatomical structures.

In the system demonstration, two videos started from the same initial frame and generated different subsequent frames of surgical scenes based on the sequences of device movements recorded by VR headsets. This showcased the model's ability to respond to device control.

However, NVIDIA clearly lists the research topics that have not yet been completed: broader procedural coverage, biomechanical validation, benchmark testing involving surgeons, and the assessment of applicability to patient care. These are all areas of research, not established clinical capabilities. The current significance of this system is to provide a new simulation infrastructure for academic research and device development, rather than a medical product that can be commercialized in the near term.