← Experian Interview Insights

Experian·Data Scientist·Technical Phone Screen·Senior

Senior
Oct 2023

Summary

Interviewed for a Data Scientist role at Experian and got hit with a pretty deep distributed computing question early on. The focus was clearly on big-data tooling and whether you actually understand what's happening under the hood, not just which API to call.

Questions Asked (1)

Q1

How do Spark RDDs, DataFrames, and Spark SQL differ from each other, and what performance benefits does lazy evaluation actually give you? When would you pick one abstraction over the others?

Technical Trade-offsSystem DesignAlgorithms & Data Structures
Author's notes

I started with RDDs and immediately felt myself going too deep into the weeds on lineage graphs before I'd even touched DataFrames.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining each abstraction (RDD, DataFrame, Spark SQL) and their core differences in API, optimization, and use cases. Then explain lazy evaluation and its performance benefits, such as reduced data shuffling and optimized execution plans. Finally, give concrete scenarios for when to choose each abstraction, tying back to data science workflows.

Pro tip: Emphasize that DataFrames and Spark SQL leverage Catalyst optimizer and Tungsten engine, which often make them faster than RDDs for structured data—but RDDs still shine for unstructured data or custom low-level transformations. Mention that lazy evaluation enables whole-stage code generation and avoids unnecessary intermediate materialization.

1. Define the abstractions

Briefly describe RDDs as low-level, distributed collections with fine-grained control; DataFrames as structured, schema-aware distributed tables; and Spark SQL as a SQL interface over DataFrames with additional optimizations.

2. Explain lazy evaluation

Describe how transformations are not executed until an action is called, allowing Spark to build a DAG and optimize the entire workflow. Highlight benefits like pipelining, avoiding unnecessary shuffles, and enabling Catalyst optimizations.

3. Compare performance and optimization

Discuss how DataFrames and Spark SQL use Catalyst and Tungsten for query optimization and code generation, often outperforming RDDs. Note that RDDs lack these optimizations but offer flexibility for complex, non-structured data.

4. Provide selection criteria

Give scenarios: use RDDs for unstructured data, custom partitioning, or when you need low-level control; use DataFrames for structured/semi-structured data and when you want optimization; use Spark SQL for SQL-centric workflows or integration with BI tools.

5. Tie to data science context

Relate to Experian's use cases: e.g., DataFrames for feature engineering on structured credit data, Spark SQL for ad-hoc analysis, and RDDs for processing raw logs or custom ML algorithms.

Key Points to Mention

  • RDD: immutable, distributed collection, low-level API, no schema, manual optimization.
  • DataFrame: distributed table with schema, Catalyst optimizer, Tungsten execution, higher-level API.
  • Spark SQL: SQL interface over DataFrames, leverages same optimizer, integrates with Hive, supports SQL queries.
  • Lazy evaluation: builds DAG, optimizes before execution, reduces data shuffling, enables pipelining.
  • Performance: DataFrames/Spark SQL often faster due to Catalyst and Tungsten; RDDs better for custom, unstructured data.
  • Selection: RDD for unstructured/custom, DataFrame for structured/optimized, Spark SQL for SQL/BI integration.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.