NURO·Software Engineer·Onsite - System Design / Architecture·Senior

Senior

Jun 2026

Summary

System design question at Nuro for a software engineer role, focused on multi-sensor fusion for autonomous vehicle perception. Pretty niche problem but made sense given what they're building.

Questions Asked (1)

You have multiple independent visual recognition systems all running on the same video frame, each outputting bounding boxes with coordinates, class labels, and confidence scores. Design the integration logic that merges all these outputs into a single deduplicated set of detections. Cover how you'd handle boxes that refer to the same object across systems, how to reconcile disagreeing class labels or confidences, and how to deal with partial overlaps. Define your inputs and outputs precisely and talk through the complexity.

System DesignAlgorithms & Data StructuresTechnical Trade-offs

Author's notes

This one took me a minute to even scope properly.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by precisely defining the inputs (list of detections per system) and outputs (merged list of deduplicated detections). Then outline a pipeline: group overlapping boxes across systems, merge each group into a single detection by reconciling labels and confidences, and handle partial overlaps via IoU thresholds. Finally, discuss algorithmic choices (e.g., union-find, clustering) and complexity trade-offs.

Pro tip: Mention that you would make the IoU threshold and confidence fusion method configurable and validate them on a small labeled set, because the optimal values depend on the specific systems and object types. This shows you think about real-world tuning and evaluation, not just theory.

1. Define inputs and outputs precisely

Input: a list of detections from each system, each detection is (system_id, bbox (x1,y1,x2,y2), class_label, confidence). Output: a list of merged detections, each with a single bbox, class_label, confidence, and optionally the set of contributing system_ids.

2. Group overlapping boxes across systems

Use spatial overlap (e.g., IoU > threshold) to cluster detections that likely refer to the same object. Consider using union-find or graph-based clustering to handle transitive overlaps. Discuss how to handle partial overlaps (e.g., one box mostly inside another) by using containment or intersection-over-min-area.

3. Merge each group into a single detection

For each group, compute a merged bounding box (e.g., weighted average by confidence, or the box with highest confidence). Reconcile class labels: if all agree, use that label; if disagree, use majority vote, highest confidence, or a weighted vote. Fuse confidences: average, max, or a probabilistic combination (e.g., noisy-OR).

4. Handle edge cases and partial overlaps

Decide on a policy for boxes that partially overlap but may be different objects (e.g., two people close together). Use a higher IoU threshold or additional cues (e.g., class consistency) to avoid merging distinct objects. Also consider boxes that are completely contained within another.

5. Analyze complexity and trade-offs

Naive pairwise comparison is O(N^2) where N is total detections. With union-find and spatial indexing (e.g., grid or R-tree), can reduce to near O(N log N). Discuss trade-offs: accuracy vs. speed, simplicity vs. robustness, and how to tune thresholds.

Key Points to Mention

Intersection over Union (IoU) and alternative overlap metrics like Intersection over Minimum Area (IoM) for partial overlaps.
Clustering algorithms: union-find (disjoint set) for transitive grouping, or graph-based connected components.
Class label reconciliation strategies: majority vote, highest confidence, weighted vote by confidence, or probabilistic fusion.
Confidence fusion methods: max, average, weighted average, or noisy-OR for combining independent probabilities.
Handling partial overlaps: using containment checks, adaptive thresholds, or class-aware merging to avoid false merges.
Complexity analysis: O(N^2) naive vs. O(N log N) with spatial indexing, and trade-offs between accuracy and efficiency.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.