Databricks Data Engineer Associate Flashcards

Master key concepts with interactive flashcards

Stay focused on Databricks-DE-Associate

Foci runs Pomodoro-style focus sessions and blocks distractions while you study for your certification.

Try Foci

Card 1 of 10

0 known

ELT with Spark SQL and Python

Question

How do you handle duplicate records in Delta Lake?

Click to reveal answer

ELT with Spark SQL and Python

Answer

Multiple approaches: 1) **INSERT with NOT EXISTS**: `INSERT INTO target SELECT * FROM source WHERE NOT EXISTS (SELECT 1 FROM target WHERE ...)` 2) **MERGE with WHEN NOT MATCHED**: Only inserts new records 3) **Window functions**: `ROW_NUMBER() OVER (PARTITION BY key ORDER BY timestamp DESC) = 1` 4) **dropDuplicates()** in PySpark

Click to show question

Tip: Use ← → arrow keys to navigate, Space to flip card