← Point72 Asset Management Interview Insights
The filtering logic in the second method is where I spent most of my time.
Start by clarifying the data schema and business rules, then outline the class structure with a SparkSession builder and lazy evaluation. For the join and filtering, use a broadcast join on accounts and apply conditions to exclude invalid transfers. Finally, implement the aggregation methods using distinct count and groupBy with orderBy/limit, returning results as a dictionary.
Pro tip: Mention that you would cache the filtered DataFrame if it's reused across methods, and discuss the trade-off between broadcast joins and shuffle joins based on data size.
Ask about the expected schema of accounts and transactions DataFrames, and confirm the exact filtering conditions (e.g., amount > balance, missing destination).
Create a class with a method to initialize a local SparkSession using builder with appName and master('local[*]').
Join transactions with accounts on source and destination account numbers, then filter out rows where amount > source balance or destination is null.
For distinct source count, use select('source_account').distinct().count(). For top 10, groupBy source_account, count, orderBy desc, limit 10, and collect as dictionary.
Consider broadcast join for accounts, caching, and handling nulls or negative amounts. Also discuss partitioning and skew.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.