Back to Security, Compliance, and Identity Fundamentals

Security, Compliance, and Identity Fundamentals Flashcards

Master key concepts with interactive flashcards

Need help staying focused? Foci helps you manage tasks and focus while you study.

Card 1 of 50

0 known

ELT with Spark SQL and Python

Question

How do you handle duplicate records in Delta Lake?

Click to reveal answer

ELT with Spark SQL and Python

Answer

Multiple approaches: 1) **INSERT with NOT EXISTS**: `INSERT INTO target SELECT * FROM source WHERE NOT EXISTS (SELECT 1 FROM target WHERE ...)` 2) **MERGE with WHEN NOT MATCHED**: Only inserts new records 3) **Window functions**: `ROW_NUMBER() OVER (PARTITION BY key ORDER BY timestamp DESC) = 1` 4) **dropDuplicates()** in PySpark

Click to show question

Tip: Use ← → arrow keys to navigate, Space to flip card

Quick Feedback

Security, Compliance, and Identity Fundamentals Flashcards

Question

Answer

Security, Compliance, and Identity Fundamentals Flashcards

Question

Answer