Start by clarifying requirements and constraints (model, tasks, metrics, scale), then present a modular architecture with clear separation of concerns: task loading, model inference via Ollama API, evaluation logic, and result aggregation. Walk through the data flow from dataset to report, emphasizing reliability (retries, caching, validation) and extensibility (plugin-based metrics, config-driven tasks).
Pro tip: Mention that you'd version both the harness and the model (e.g., via Ollama model digests) to ensure reproducibility, and implement a caching layer for model responses to avoid redundant inference during iterative development.
Ask about the model(s), task types (e.g., QA, summarization), evaluation metrics, dataset size, and expected throughput. Confirm whether the harness runs locally or in CI, and any latency/cost constraints.
Outline components: Task Loader (parses YAML/JSON configs), Ollama Client (handles API calls with retries and timeouts), Evaluator (computes metrics), Result Store (saves outputs and scores), and Reporter (generates summaries). Emphasize interfaces for extensibility.
Describe the pipeline: load tasks -> for each sample, call Ollama API -> collect responses -> compute metrics -> aggregate results -> generate report. Include error handling and logging at each stage.
Discuss retries with exponential backoff, caching responses, input validation, and deterministic seeding. For extensibility, propose plugin-based metrics and task types, and configuration-driven execution.
Highlight trade-offs like synchronous vs. asynchronous calls, local vs. remote evaluation, and simplicity vs. feature richness. Suggest future enhancements like parallel execution, distributed evaluation, or integration with experiment trackers.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.