The scaffold was actually a relief at first because I didn't have to argue about project structure.
Start by clarifying requirements and constraints (e.g., I/O-bound, file sizes, error handling). Then outline a design using an async queue with a bounded worker pool, explaining how backpressure is applied via queue capacity and await-based scheduling. Finally, detail graceful shutdown with cancellation and result aggregation, and discuss trade-offs.
Pro tip: Demonstrate awareness of real-world issues like handling partial failures and ensuring idempotency, and mention that you'd use asyncio.Semaphore or bounded queues to control concurrency and prevent resource exhaustion.
Ask about expected scale, file types, error handling, and whether ordering matters. Confirm that the crawler is I/O-bound and should handle backpressure.
Propose an async queue (e.g., asyncio.Queue) with a maximum size to apply backpressure. Explain how tasks are enqueued and how producers await when the queue is full.
Create a fixed number of worker tasks that consume from the queue. Use a semaphore or the queue's maxsize to limit concurrent operations and avoid overwhelming the system.
Describe how to signal workers to stop (e.g., sentinel values or cancellation), wait for in-flight tasks to complete, and ensure resources are cleaned up.
Collect results in a thread-safe manner (e.g., using asyncio.gather or a results list with locks). Discuss error propagation and retry logic.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.