← UiPath Interview Insights

UiPath·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
Apr 2026

Summary

Technical phone screen for an ML Engineer role at UiPath. One question, pretty deep on transformer internals, and the follow-ups were where things got interesting.

Questions Asked (1)

Q1

What is the time and memory complexity of self-attention for a sequence of length N with model dimension d?

Technical Trade-offsAlgorithms & Data StructuresSystem Design
Author's notes

I got the O(N² · d) time part right away but fumbled explaining why memory is O(N²) specifically.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the self-attention operation and its inputs: queries, keys, and values matrices of shape N x d. Then derive the time and memory complexity by analyzing the matrix multiplications and the attention weight matrix. Finally, discuss the implications and potential optimizations.

Pro tip: Mention that while the standard self-attention has O(N^2 d) time and O(N^2) memory, many efficient variants reduce this to O(N d^2) or O(N log N) by approximating the attention matrix, which is crucial for long sequences.

1. Define self-attention computation

Explain that self-attention computes attention scores as QK^T, applies softmax, and multiplies by V. Q, K, V are linear projections of the input, each of shape N x d.

2. Analyze time complexity

Break down the matrix multiplications: QK^T takes O(N^2 d), softmax takes O(N^2), and multiplying by V takes O(N^2 d). Total time is O(N^2 d).

3. Analyze memory complexity

The attention weight matrix (N x N) dominates memory, requiring O(N^2) storage. Additionally, storing Q, K, V takes O(N d), but for large N, O(N^2) is the bottleneck.

4. Discuss practical implications

Highlight that quadratic complexity limits sequence length, and mention common optimizations like sparse attention, low-rank approximations, or kernel methods that reduce complexity.

Key Points to Mention

  • Self-attention computes pairwise interactions between all N positions, leading to quadratic scaling.
  • Time complexity: O(N^2 d) due to matrix multiplications QK^T and (softmax) V.
  • Memory complexity: O(N^2) for storing the attention matrix, plus O(N d) for Q, K, V.
  • The softmax operation adds O(N^2) time but does not change the asymptotic complexity.
  • Efficient attention variants (e.g., Linformer, Performer, Reformer) reduce complexity to O(N d^2) or O(N log N).
  • For typical transformer models, d is often fixed (e.g., 512, 768), so complexity is often quoted as O(N^2).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.