Camera-Only vs Lidar Models Disagree in Self-Driving Scenarios: Understanding the Risks and Realities

As autonomous vehicles steadily transition from prototypes to everyday realities, their perception systems must grapple with complex, safety-critical decisions. Among the myriad sensors involved, cameras and lidar have emerged as primary modalities — each offering unique strengths and limitations. But what happens when camera-only and lidar-based models disagree in their predictions? This disagreement isn't just noise; it acts as a powerful signal highlighting high-risk scenarios, edge cases, and fundamental challenges in self-driving perception.

In this post, we’ll unpack the phenomenon of model disagreement between camera-only and lidar inputs in self-driving contexts, leveraging tools like disagreement rate and predictive entropy. We’ll explore why these disagreements matter, touching key themes like sensor occlusion, trajectory divergence, distribution shifts, and losses optimized for different objectives. Along the way, we’ll reveal "things accuracy hides" to drive home why understanding multimodal perception divergence is essential for robust autonomous driving.

Why Focus on Camera-Only vs Lidar Models?

The current landscape of autonomous vehicle perception relies heavily on fusing data from multiple sensor types:

Cameras provide rich semantic and color information, capturing high-resolution RGB images of the environment.
Lidar offers precise 3D spatial maps via laser pulses, yielding accurate geometry even in challenging lighting.

Individually, models trained on each have distinct views of the environment. But it’s in their disagreements that we find crucial insights:

Edge Cases and Ambiguity: When sensors disagree, it often reflects scenarios that are neither fully captured by camera imagery nor lidar scans.
Risk Signaling: Disagreements often pinpoint safety-critical uncertainties, such as occluded objects or fast-moving agents.
Model Validation: Comparing camera-only vs lidar predictions illuminates distribution shifts and data gaps, guiding where retraining or additional sensors can help.

Before we dive deeper, let's clarify the core measurement tools we'll use.

Measuring Disagreement: Disagreement Rate and Predictive Entropy

AI safety in production

Disagreement Rate

Disagreement rate quantifies how often two models — here, camera-only and lidar-based — give conflicting predictions on the same input scenario. Formally, it can be expressed as:

Disagreement Rate = \frac\textNumber of samples with differing predictions\textTotal number of samples

This simple metric becomes a powerful lens for locating potentially ambiguous, occluded, or out-of-distribution cases.

Predictive Entropy

While disagreement measures conflict between models, predictive entropy captures uncertainty within a single model’s prediction distribution. For a classification model that outputs probabilities p_i over classes, entropy H is:

H = - \sum_i p_i \log p_i

Entropy highlights model confidence—or lack thereof—reinforcing risk signals when combined with disagreement rates. High entropy predictions can coincide with model disagreements, especially in complex scene elements like pedestrians partially occluded by vehicles or weather effects.

Core Themes Illuminated by Model Disagreement

1. Sensor Occlusion: Where Seeing and Sensing Diverge

Occlusion challenges perception because sensors have limited viewpoints:

Cameras rely on direct line-of-sight and can be thwarted by shadows, glare, or objects blocking view.
Lidar can provide 3D geometric readings around some occlusions but suffers from beam sparsity and returns that can be noisy or missing.

Camera-only and lidar models thus frequently disagree on objects partially hidden in the scene: a pedestrian stepping out from behind a parked van may appear prominently in the camera but only sparsely in lidar returns.

Such sensor occlusion-driven disagreements serve as high-signal flags, emphasizing critical regions where trajectory predictions might diverge or require fallback strategies. For instance, safety systems can heighten alertness or reduce speed in these scenarios.

2. Trajectory Divergence: Predictive Disagreements Foreshadow Divergent Behaviors

Self-driving vehicles don’t just perceive static objects — they predict trajectories for dynamic agents like cars, cyclists, and pedestrians. When camera-only and lidar models offer divergent forecasts for these trajectories, fallback safety considerations become vital.

Disagreements in predicted paths magnify risk because the controller depends on accurate trajectory estimations for collision avoidance. Multimodal perception through sensor fusion aims to reconcile these differences, but persistent disagreements highlight:

Uncertainty in agent intent or motion (e.g., a cyclist weaving through traffic).
Partial observability due to occlusion or sensor noise.
Model limitations from training data that may underrepresent rare maneuvers.

Monitoring disagreement patterns in trajectory space helps teams identify and mitigate scenarios prone to safety-critical mispredictions.

3. Edge Cases and Distribution Shift: When Training Distribution Meets the Real World

Disagreement strongly correlates with distribution shifts — scenarios not well represented in training sets:

Unusual weather (fog, heavy rain) changes sensor reliability unevenly.
Rare events like emergency vehicles or ad-hoc road signs generate uncertain predictions.
Geographical or cultural differences alter scene contexts unexpectedly.

Camera-only and lidar models may respond differently to these shifts due to modality-specific sensitivities, revealing data gaps and subgroup coverage weaknesses. A high disagreement rate signals the models are extrapolating rather than interpolating, warranting careful validation and possible data collection to fill coverage gaps.

4. Objective Mismatch and Loss Function Tradeoffs

Camera and lidar models often optimize distinct objectives or loss functions, reflecting their sensing modalities and downstream tasks. For example, a camera-based model might prioritize pixel-level semantic segmentation, while lidar may emphasize accurate object localization in 3D space.

This objective mismatch contributes to divergences, because each model "sees" and prioritizes the environment differently. The choice of loss functions — cross-entropy, focal loss, or regression metrics — influences calibration and uncertainty estimates, potentially misaligning final outputs.

Recognizing this tradeoff encourages the development of unified loss frameworks or multimodal architectures that better harmonize objectives, reducing uninformative disagreement and enhancing safe decision-making.

What Happens on the Worst Day in Prod?

Imagine an urban intersection with heavy traffic and poor weather conditions: a pedestrian partially occluded by a delivery truck steps onto a crosswalk. The camera-only model confidently spots the pedestrian, while the lidar-based model misses or misclassifies the pedestrian due to sensor sparsity and rain reflections. Their trajectory predictions diverge sharply, indicating conflicting beliefs about the pedestrian's movement.

If the self-driving system blindly fuses these outputs without accounting for disagreement, it may compute an optimistic trajectory and fail to brake in time, risking collision. Here, disagreement acts as a vital beacon: triggering increased caution, comprehensive sensor reprocessing, or human intervention if enabled.

This “worst day” scenario spotlights the critical role of disagreement monitoring, calibrated uncertainty, and cost-weighted thresholds — rather than naive accuracy metrics — to safeguard safety margins.

Things Accuracy Hides: Why Reporting Test-Set Scores Alone Isn’t Enough

Masked Rare Events: Test accuracy often emphasizes common scenarios, glossing over boundary cases where models diverge crucially.
Bias in Subgroup Performance: Overall accuracy hides disparities in detecting underrepresented classes or environments.
Calibrated Confidence Matters: High accuracy with poor uncertainty calibration leads to overconfidence in dangerous situations.
Ignoring Disagreement Masks Distribution Shift: When two models strongly disagree, it signals out-of-distribution data that vanilla accuracy can't capture.

Therefore, incorporating disagreement rates and predictive entropy in model validation cycles uncovers hidden risks and promotes transparency.

Table: Summary of Disagreement Drivers in Camera-Only vs Lidar Models

Factor Effect on Disagreement Why Important Sensor Occlusion High due to partial or blocked views Highlights unknown zones needing caution or fallback Trajectory Divergence High in dynamic, ambiguous motions Crucial for collision avoidance and safety Distribution Shift / Edge Cases Increases with rare or unseen scenarios Signals data gaps and model brittleness Objective / Loss Mismatch Causes systematic differences in predictions Requires harmonized training strategies

Why Disagreement is a High-Signal Risk Indicator

Beyond a mere curiosity, disagreement functions as a critical early warning system. It draws attention to scenarios where:

Models are extrapolating due to inadequate or conflicting data.
Sensor failures, occlusions, or noise impair perception quality.
Safety margins tighten due to uncertainty in agent behavior.
Operational design domains are being tested beyond training bounds.

In practice, integrating disagreement metrics into validation and live monitoring pipelines allows teams to:

Spot localization or sensing failures early.
Trigger uncertainty-aware fallback controls.
Guide data collection for rare or poorly covered cases.
Improve sensor fusion algorithms to reconcile conflicts.

Conclusions: Navigating the Complexity of Multimodal Perception

Camera-only and lidar models each offer invaluable but distinct perspectives in self-driving perception. Their disagreements reveal essential nuances often invisible through traditional accuracy metrics alone. Leveraging disagreement rate and predictive entropy provides a robust framework to understand when and why these models diverge.

By interpreting disagreement as a high-signal risk indicator — impacted by sensor occlusion, trajectory divergence, distribution shift, and objective tradeoffs — engineers can better minimize surprises on the road. This understanding directly informs safer multimodal fusion strategies, calibrated confidence measures, and targeted data improvements.

In the end, asking “what happens on the worst day in production?” guides us to treat model disagreement not as a problem to hide, but as an opportunity to diagnose, prioritize, and manage risk in the harsh realities of autonomous driving.