The question matters enormously. A perfect depth map shown in the wrong way at the wrong time is useless — or worse, distracting during a critical manoeuvre. There are four distinct presentation paradigms, each with different tradeoffs, and the right choice depends on exactly what surgical task is being supported.
The Four Display Paradigms
1. Pseudo-colour depth overlay (the most direct option)
The depth map is converted to a colour scale (typically cool-to-warm: blue = far, red = close) and blended semi-transparently over the live endoscope video image. The surgeon sees the normal video with a colour tint that conveys depth — brighter red zones are closer to the endoscope, blue zones are further away.
Pros: Requires no new display hardware — it runs on the existing OR monitor the surgeon is already looking at. No learning curve for the display medium. Intuitive once calibrated. Consistent with how the Lumina-bone heatmap output is already rendered.
Cons: Adding colour over the video can obscure tissue detail if opacity is too high. Requires careful tuning of the colour scale and transparency. The surgeon must mentally fuse the depth colour with the video image — a modest but real cognitive demand. Colour perception can be affected by OR lighting and monitor calibration.
Best for: Spatial orientation — understanding the general topography of the bone surface in view. Shows ridges, valleys, the step between lamina and ligamentum flavum, pedicle wall curvature.
2. Contour lines / depth isocontours overlay
Rather than colouring the whole image, draw 1–2 mm depth isocontour lines over the video — like topographic map contours on the bone surface. The surgeon sees the normal video with thin coloured lines showing "this ring is 5 mm from the endoscope, this ring is 8 mm."
Pros: Much less visually disruptive than a full colour overlay — the video is largely unobstructed. Provides precise metric depth reference at specific levels. Surgeons already understand topographic maps conceptually.
Cons: Contour lines can be hard to read on a curved moving surface. Dense contours in steeply curved zones become cluttered. Requires the absolute depth calibration (not just relative shape) to be reliable.
Best for: Instrument-to-bone distance during drilling — the surgeon can see exactly which depth contour their instrument is approaching.
3. Synthetic shading / relighting (the most natural option)
Instead of overlaying depth as colour, use the recovered surface normals to re-render the bone surface with a simulated directional light from a fixed known direction — regardless of where the actual LED illumination comes from. The result is a naturally-shaded 3D-looking image of the bone that has strong, consistent shading gradients everywhere, giving the brain's visual system powerful depth cues without any added colour or annotation.
This is sometimes called "shape enhancement" — the Durr/JHU MLE paper used this approach in the colonoscopy context, showing the photometric stereo height map rendered as a 3D surface with a simulated overhead light to enhance the visual perception of mucosal topography.
Pros: Leverages the brain's evolved 3D perception from shading — no training required. Looks like a more "3D" version of the existing image rather than a separate overlay. Cognitively minimal — the surgeon doesn't need to learn a new code.
Cons: The rendered appearance changes with surface orientation in ways the surgeon may not be used to. Sharp ridges can appear exaggerated. If the underlying normal map has artefacts, the rendering makes them visually prominent.
Best for: General intraoperative spatial awareness — making the bone surface "look 3D" throughout the procedure. This is the most surgeon-intuitive presentation.
4. Side-by-side or picture-in-picture 3D surface view
Display the live endoscope video on the main monitor and add a second window showing a 3D rendered point cloud or mesh of the reconstructed bone surface — either side by side or as a small inset. The 3D view can be rotated to help the surgeon understand the anatomy from a different perspective.
Pros: The 3D model is separate from the video, so the primary surgical view is unobstructed. Allows the surgeon to virtually "look around" the anatomy. Can be registered with preoperative CT for a complete anatomical picture.
Cons: The surgeon must look away from the primary surgical field to read the 3D view — a significant attention cost during active dissection. Requires the most real-estate on the monitor stack. Registration with the preoperative CT is an additional technical step.
Best for: Pause moments — before making a key cut, the surgeon briefly checks the 3D view to confirm orientation, then returns to the live video. Not suitable for continuous use during active instrument manipulation.
The Cognitive Load Question
Recent work introducing mixed reality head-mounted displays in biportal endoscopic spine surgery found that MR technology did not increase the operating surgeon's perceived cognitive workload as measured by the SURG-TLX tool — but the study used the technology for display of the standard endoscope video, not as an additional information overlay. This matters: adding any depth information layer carries a cognitive cost, and the display mode that minimises that cost for a given clinical benefit is the right one to use.
The hierarchy, from lowest to highest cognitive load:
- Synthetic relighting — no decoding required; looks naturally 3D
- Pseudo-colour overlay — requires learning the colour code; fast to read once learned
- Contour line overlay — requires reading spatial lines against a background video
- Separate 3D view — requires attention switching between two displays
What the Literature Shows Actually Works
Autostereoscopic displays and AR overlays for MIS have been extensively studied — the key finding is that colocating depth information with the primary surgical view (rather than on a separate monitor) consistently reduces cognitive load and improves spatial accuracy, but only when the overlay is rendered at low opacity and does not obscure tissue detail.
For spine specifically, overlay visualisation in endoscopic ENT surgery — a closely analogous application — has demonstrated that superimposing navigation information on the endoscopic image allows the use of guiding lines for distance visualisation, with accuracy verified in cadaver studies and promising results in clinical paranasal sinus interventions.
The Practical Recommendation for Lumina-Bone
Given the surgical workflow and the specific tasks in spine endoscopy, the most practical approach is a two-mode display system:
Mode 1 — Always-on synthetic relighting (low transparency, background running)
The live video is continuously enhanced by re-rendering with consistent synthetic shading from the recovered surface normals. This gives the surgeon a naturally "more 3D" view throughout the procedure without any explicit depth readout or annotation. Cognitive cost is minimal. This is the baseline spatial awareness mode.
Mode 2 — On-demand pseudo-colour depth overlay with metric labels
Triggered by a foot pedal or voice command, the surgeon activates the depth overlay for 2–3 seconds to check depth at a specific moment — before making a key cut, when approaching the pedicle wall, when verifying decompression extent. The colour overlay with millimetre labels appears briefly, the surgeon reads it, and returns to the clean video. This is the metric guidance mode.
This two-mode design borrows directly from how MLE (Durr/JHU) handles the toggle between clinical white light and research imaging modes — less than 1 second to switch, surgical workflow uninterrupted.
Connection to CT Registration — The Longer-Term Picture
Once the photometric stereo pipeline produces a reliable 3D surface map of the bone in the endoscope's field of view, that surface can be registered to the preoperative CT using ICP — exactly the next milestone listed in the Lumina-bone development notes. This registration enables the most powerful display mode of all: overlaying the preoperative CT anatomy (spinal canal boundary, pedicle walls, nerve root trajectories, intended screw path) directly onto the live endoscope video in the correct spatial position.
This is the full AR navigation pipeline: photometric stereo provides the intraoperative surface → ICP registers it to CT → CT anatomy is overlaid on live video. The depth map is not just displayed to the surgeon; it becomes the registration anchor that makes all subsequent AR guidance possible. That's the strategic value of Lumina-bone beyond just "showing depth" — it provides the markerless, radiation-free registration that current spine navigation systems lack.