Openai·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

May 2026

Summary

System design round at OpenAI for a software engineer role. The whole session was basically one big question about building a GPU credit management system, which sounds straightforward until you start pulling on the concurrency and expiration threads.

Questions Asked (1)

Design a GPU credit manager for a compute cluster. It needs to support granting, consuming, and refunding credits per user, querying a user's balance, and retrieving the top K users by balance. Also enforce per-user caps, optional org-level aggregate caps, optional credit expiration, no negative balances, and thread safety. Target O(log n) per operation. Walk through data structures, pseudocode for each API, and complexity analysis.

System DesignAlgorithms & Data StructuresTechnical Trade-offs

Author's notes

This one took me a second to scope properly.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then propose a core data structure that supports O(log n) operations for all APIs. Use a combination of hash maps and balanced trees (or skip lists) to manage per-user balances and global ordering, and carefully handle concurrency with fine-grained locking or lock-free techniques.

Pro tip: Emphasize the trade-offs between different data structures (e.g., skip list vs. balanced BST) and locking strategies (e.g., per-user locks vs. global lock) to demonstrate depth. Also, discuss how to handle expiration efficiently without scanning all users.

1. Clarify Requirements and Constraints

Ask questions to understand expected scale (number of users, operations per second), consistency requirements, and whether expiration is lazy or active. Confirm that O(log n) is per operation and n is number of users.

2. Design Core Data Structures

Propose a hash map for O(1) user lookup and a balanced BST or skip list for maintaining global order by balance. For org-level caps, consider a separate map from org to aggregate balance and a tree for org ordering if needed.

3. Define APIs and Pseudocode

For each API (grant, consume, refund, getBalance, topK), write pseudocode showing how to update the user's balance, adjust the ordered structure, and enforce caps and expiration. Include checks for negative balances and cap violations.

4. Address Concurrency and Thread Safety

Choose a locking strategy: per-user locks for balance updates, and a global lock for the ordered structure, or use a concurrent skip list. Discuss trade-offs between coarse and fine-grained locking.

5. Analyze Complexity and Trade-offs

Analyze time and space complexity for each operation, highlighting O(log n) for updates and topK. Discuss alternative designs (e.g., using a heap for topK) and their limitations.

Key Points to Mention

Use of a balanced BST (e.g., Red-Black Tree) or skip list to maintain sorted order by balance for O(log n) topK queries.
Hash map for O(1) user lookup, storing user metadata including balance, cap, and expiration timestamp.
Handling expiration: lazy deletion during balance checks or a priority queue for active expiration, ensuring O(log n) amortized.
Enforcing per-user and org-level caps: check before granting/consuming, and update org aggregate balance accordingly.
Thread safety: per-user locks for balance updates and a global lock for the ordered structure, or a concurrent data structure like ConcurrentSkipListMap.
Complexity analysis: all operations O(log n) due to tree operations, with O(1) hash map lookups; space O(n).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.