Databricks Databricks-Certified-Data-Engineer-Associate Dumps

To sync a Databricks Repo with the changes from a central Git repository, the data engineer needs to run the Git pull operation. This operation fetches the latest updates from the remote repository and merges them with the local repository. The data engineer can use the Pull button in the Databricks Repos UI, or use the git pull command in a terminal session. The other options are not relevant for this task, as they either push changes to the remote repository (Push), combine two branches (Merge), save changes to the local repository (Commit), or create a new local repository from a remote one (Clone). References:

Run Git operations on Databricks Repos

Git pull

Option B reads the relational source through JDBC and writes the resulting Spark DataFrame directly as a registered Unity Catalog table using its three-level name: catalog, schema, and table. Once written with saveAsTable, the data is immediately discoverable and queryable through Databricks SQL, subject to Unity Catalog permissions. Because the read and write operations occur through supported Databricks workloads, Unity Catalog can capture lineage between the source operation, notebook or job, and destination table. Writing files only to DBFS or an external path does not by itself create the required governed table. Introducing a separate staging and loading job adds unnecessary processing and operational complexity. A Pandas extraction also reduces scalability and does not satisfy the direct managed-table requirement. Therefore, option B is correct.

================

Serverless compute for workflows is designed to minimize operational overhead for Databricks Jobs while enforcing Unity Catalog governance. When a job is configured to run on serverless compute (in supported regions), Databricks automatically selects instance types , manages autoscaling , enables Photon where applicable, and handles runtime upgrades without user intervention. This directly satisfies the requirement to reduce operational toil. All-purpose clusters (option B) and traditional job clusters (option C) still require users to manage cluster sizing, instance families, and runtime versions, which contradicts the goal of automation. SQL warehouses (option A) are intended for SQL workloads and do not support running general Python notebook job tasks as workflow steps. Databricks documentation specifies that serverless workflows are compatible with Unity Catalog and are the recommended approach when users want fully managed compute for scheduled pipelines and tasks, provided the workspace is in a supported region. This makes serverless compute the correct and fully compliant solution.

Lakeflow Connect managed connectors are designed to simplify ingestion from supported enterprise applications and databases. They provide managed authentication, incremental processing, schema handling, and integration with Unity Catalog. Salesforce is supported as a managed application source, while SQL Server ingestion can accommodate database connectivity patterns, including privately connected or on-premises environments when the required networking is configured. Auto Loader is the appropriate service for incrementally discovering and processing new JSON files arriving in ADLS. It supports schema inference and evolution and can be used within Lakeflow Spark Declarative Pipelines. Option B would require substantial custom API and CDC code. Option D incorrectly treats cloud files as the same connector use case as Salesforce and SQL Server. Therefore, the combination in option C minimizes maintenance while matching each source with its appropriate ingestion mechanism.

================

Databricks recommends managed tables when you want the platform to control both the table metadata and the underlying storage lifecycle. For Unity Catalog managed tables, Databricks manages storage layout and applies platform-managed optimization features over time. Databricks also documents that when you drop a managed table , the underlying cloud data is deleted automatically after the managed retention period, which simplifies cleanup and reduces storage administration. That behavior matches the requirement exactly. External tables, by contrast, are used when you want to retain direct control over the storage path and lifecycle. Databricks explicitly states that dropping an external table removes only the metadata and does not delete the underlying files . Foreign tables are for federation scenarios against external systems, not for Bronze ingestion storage managed by Databricks. Temporary views also do not provide the desired governed lifecycle. Because the team wants Databricks-managed storage, managed retention, and simplified lifecycle handling, managed tables are the correct choice for these Bronze ingestion tables.

=========

Option A enables predictive optimization, which is designed to eliminate manual maintenance scheduling for Unity Catalog managed tables. Databricks evaluates table activity and automatically runs operations such as OPTIMIZE, VACUUM, and ANALYZE when beneficial, using serverless jobs compute. Predictive optimization can be enabled at multiple hierarchy levels, allowing tables to inherit the applicable setting. Auto Loader schema evolution handles changes in incoming file schemas but does not schedule table optimization. Liquid clustering improves data organization and query pruning, but it does not independently provide automated stale-file removal. Optimized writes and auto compaction reduce the creation of small files during write operations, but they do not replace adaptive VACUUM scheduling and the broader maintenance capabilities of predictive optimization. Therefore, option A directly satisfies the requirement for platform-managed table maintenance.

================

DataFrame.dropna() removes rows containing NULL or NaN values. Its default how= " any " behavior removes a row when any column included in the specified subset is NULL. By passing [ " customer_email " , " customer_phone " ] as the subset, the operation removes a record when either required field is missing, while ignoring NULL values in unrelated columns. Option B can produce the same filtering result, but it uses two chained where() calls and therefore does not meet the single-method-call requirement. Option C uses how= " all " and would remove a row only when all evaluated columns are NULL. Option D contains one filter() operation but requires multiple column-function calls. Consequently, option A is the direct PySpark method designed for this requirement.

================

Lakeflow Jobs provides task-level Run if conditions such as All succeeded, All done, At least one failed, and All failed. No single Run if condition directly represents “A succeeded and B failed” when Task C depends directly on both tasks. The required outcome can be modeled with two guarded dependency branches: one gateway proceeds only after Task A succeeds, and another proceeds only after Task B fails. Task C then uses All succeeded against those two successful gateway tasks. A negative dependency is not a supported dependency type. Task values pass arbitrary application values and do not automatically represent task outcomes. Notebook widgets are task parameters rather than dependency-state conditions. Therefore, the two-chain configuration described in option B correctly represents the required conjunction.

================

File notification mode is the recommended option for scalable file discovery. On the stream’s first execution, Auto Loader performs a complete directory listing to discover existing files and establish a valid read position in its checkpoint and file-events cache. It can therefore ingest the 50,000 historical files without requiring a separate directory-listing stream. Subsequent executions discover newly arriving files through cloud-storage events instead of repeatedly scanning the entire directory, improving discovery performance and reducing cloud API and compute costs. Auto Loader maintains its checkpoint state and exactly-once processing guarantees when using either detection mode. Although switching modes across restarts is supported, it is unnecessary for this requirement. Options B, C, and D incorrectly imply that notification mode cannot efficiently initialize from existing files. Therefore, option A is correct.

================

Databricks documentation recommends serverless compute as the simplest and most reliable compute option when the workload is supported. Serverless compute is designed to automatically provision resources, scale with demand, reduce infrastructure management, and apply platform optimizations without requiring users to configure clusters manually. This directly supports the requirement for automatic scaling, reduced operational overhead, and better alignment with strict SLAs. Databricks also states that serverless compute is always available and scales according to workload, making it a strong fit for organizations seeking high availability and minimal downtime. Fixed-configuration job clusters in option B still require manual sizing decisions and do not meet the “no manual cluster management” requirement. Spot instances in option C may reduce costs but can be interrupted, which makes them a poor choice when reliability is a top requirement. Interactive clusters in option D are intended more for development and exploration and still need manual management. Based on Databricks guidance, serverless compute is the correct choice.

=========

The only option that is stored in the Databricks customer’s cloud account is data. Data is stored in the customer’s cloud storage service, such as AWS S3 or Azure Data Lake Storage. The customer has full control and ownership of their data and can access it directly from their cloud account.

Option A is not correct, as the Databricks web application is hosted and managed by Databricks on their own cloud infrastructure. The customer does not need to install or maintain the web application, but only needs to access it through a web browser.

Option B is not correct, as the cluster management metadata is stored and managed by Databricks on their own cloud infrastructure. The cluster management metadata includes information such as cluster configuration, status, logs, and metrics. The customer can view and manage their clusters through the Databricks web application, but does not have direct access to the cluster management metadata.

Option C is not correct, as the repos are stored and managed by Databricks on their own cloud infrastructure. Repos are version-controlled repositories that store code and data files for Databricks projects. The customer can create and manage their repos through the Databricks web application, but does not have direct access to the repos.

Option E is not correct, as the notebooks are stored and managed by Databricks on their own cloud infrastructure. Notebooks are interactive documents that contain code, text, and visualizations for Databricks workflows. The customer can create and manage their notebooks through the Databricks web application, but does not have direct access to the notebooks.

Databricks Architecture

Databricks Data Sources

Databricks Repos

[Databricks Notebooks]

[Databricks Data Engineer Professional Exam Guide]

 Delta Live Tables expectations are optional clauses that apply data quality checks on each record passing through a query. An expectation consists of a description, a boolean statement, and an action to take when a record fails the expectation. The ON VIOLATION clause specifies the action to take, which can be one of the following: warn, drop, or fail. The drop action means that invalid records are dropped from the target dataset before data is written to the target. The failure is reported as a metric for the dataset, which can be viewed by querying the Delta Live Tables event log. The event log contains information such as the number of records that violate an expectation, the number of records dropped, and the number of records written to the target dataset. References:

Manage data quality with Delta Live Tables

Monitor Delta Live Tables pipelines

Delta Live Tables SQL language reference

According to the medallion lakehouse architecture, gold tables are the final layer of data that powers analytics, machine learning, and production applications. They are often highly refined and aggregated, containing data that has been transformed into knowledge, rather than just information. Silver tables, on the other hand, are the intermediate layer of data that represents a validated, enriched version of the raw data from the bronze layer. They provide an enterprise view of all its key business entities, concepts and transactions, but they may not have all the aggregations and calculations that are required for specific use cases. Therefore, gold tables are more likely to contain aggregations than silver tables. References:

What is the medallion lakehouse architecture?

What is a Medallion Architecture?

A materialized view is appropriate because it stores the results of a defining query and refreshes them as source data changes. Expensive joins and aggregations can therefore be computed before BI users execute their queries, improving response time and providing a consistent definition of business metrics. This aligns with the Gold layer, which normally contains highly refined, aggregated, and business-oriented datasets designed for reporting and analytics. A view over raw Bronze tables would expose unvalidated source data and repeatedly execute its underlying transformations. A streaming table is more suitable for incremental data processing than for presenting precomputed business metrics. CSV files would not provide the managed table semantics, centralized definitions, or efficient query experience required. Therefore, option A best satisfies the Gold-layer requirements.

================

Databricks documentation for Unity Catalog lineage states that lineage is captured across queries run on Databricks , is supported for all languages , and is tracked down to the column level . It also explicitly says that lineage data includes related notebooks, jobs, and dashboards , and that the lineage can be visualized in Catalog Explorer in near real time as an interactive graph. This directly matches option D. The value of this feature is that it helps engineers understand where a table’s data came from, which upstream notebooks or jobs produced it, and which downstream assets, including dashboards, depend on it. Column-level lineage adds another layer of traceability by showing how specific fields are derived through transformations, which is especially useful for impact analysis, governance, and troubleshooting. Options A, B, and C each remove documented capabilities that Unity Catalog lineage actually supports. Therefore, the Databricks-documented answer is the full interactive graph across Delta tables and related assets, with column-level tracking included.

=========

 To append a new record to an existing Delta table, you can use the INSERT INTO statement with the VALUES clause. This statement will insert one or more rows into the table with the specified values. Option A is the only code block that follows this syntax correctly. Option B is incorrect, as it uses the UNION operator, which will return a new table that is the union of two tables, not append to an existing table. Option C is incorrect, as it uses the INSERT VALUES statement, which is not a valid SQL syntax. Option D is incorrect, as it uses the UPDATE statement, which will modify existing rows in the table, not append new rows. Option E is incorrect, as it uses the UPDATE VALUES statement, which is also not a valid SQL syntax. References: Insert data into a table using SQL | Databricks on AWS, Insert data into a table using SQL - Azure Databricks, Delta Lake Quickstart - Azure Databricks

A temporary view is a relational object that is defined in the metastore and points to an existing DataFrame. It does not copy or store any physical data, but only saves the query that defines the view. The lifetime of a temporary view is tied to the SparkSession that was used to create it, so it does not persist across different sessions or applications. A temporary view is useful for accessing the same data multiple times within the same notebook or session, without incurring additional storage costs. The other options are either materialized (A, E), persistent (B, C), or not relational objects ©. References: Databricks Documentation - Temporary View, Databricks Community - How do temp views actually work?, Databricks Community - What’s the difference between a Global view and a Temp view?, Big Data Programmers - Temporary View in Databricks.

The engineer should inspect the job’s maximum concurrent-runs configuration and its queueing setting. max_concurrent_runs controls how many active runs of the same job can execute simultaneously. When that limit has already been reached, Databricks can skip the newly triggered run if queueing is disabled. When queueing is enabled, the additional run waits until capacity becomes available instead of being skipped. Spot-instance availability might delay or prevent compute startup, but it would not normally produce a job-level concurrency message. SQL warehouse permissions affect SQL task execution after a run begins, while timeout_seconds determines how long a task may run before timing out. Therefore, option C directly addresses the reported run state. This candidate was also compared against the existing Word question bank.

================

 To set up the new task to run a new notebook prior to the original task in a single-task Job, the data engineer can use the following approach: In the existing Job, create a new task that corresponds to the new notebook that needs to be run. Set up the new task with the appropriate configuration, specifying the notebook to be executed and any necessary parameters or dependencies. Once the new task is created, designate it as a dependency of the original task in the Job configuration. This ensures that the new task is executed before the original task.

In Databricks Asset Bundles , the databricks.yml file located at the repository root serves as the single authoritative configuration file for the entire project. This file defines key elements such as the bundle name, deployment targets (for example, dev, test, prod), workspace configurations, permissions, and resource definitions including jobs and pipelines. Databricks documentation specifies that there must be one primary databricks.yml file at the root , ensuring consistency and simplifying deployment processes. However, to support modularity and maintainability, this root configuration file can reference additional YAML files using the include mapping. This allows teams to organize configurations across directories while still maintaining a single source of truth. Option A is incorrect because multiple root-level configuration files are not supported as independent authorities. Option C is invalid since configuration files are not restricted to hidden directories. Option D introduces ambiguity and contradicts the requirement for a single authoritative configuration. Therefore, using one root-level configuration file with optional includes is the correct and recommended approach.

In the Medallion Architecture promoted by Databricks, the Bronze layer represents the raw ingestion layer where data is stored in its original form with minimal transformation. The primary goal is to ensure data fidelity and traceability , not to apply business logic or heavy transformations. Typical operations in the Bronze layer include schema inference (or enforcement), ingestion of raw data, and the addition of metadata columns such as ingestion timestamp, load date/time, and process identifiers. These metadata fields help track lineage, enable debugging, and support incremental processing downstream. Extensive data cleansing (A), aggregations (B), and business transformations (C) are intentionally deferred to the Silver and Gold layers , where data is refined and structured for analytics. By keeping Bronze data close to the source and append-only, organizations maintain a reliable audit trail and can reprocess data if needed. Therefore, adding operational metadata like load timestamps and process IDs is the correct characteristic of Bronze layer transformations.

=========

The correct command to grant the necessary permission on the entire database to the new team is to use the GRANT USAGE command. The GRANT USAGE command grants the principal the ability to access the securable object, such as a database, schema, or table. In this case, the securable object is the database customers, and the principal is the group team. By granting usage on the database, the team will be able to see what tables already exist in the database. Option E is the only option that uses the correct syntax and the correct privilege type for this scenario. Option A uses the wrong privilege type (VIEW) and the wrong securable object (CATALOG). Option B uses the wrong privilege type (CREATE), which would allow the team to create new tables in the database, but not necessarily see the existing ones. Option C uses the wrong securable object (CATALOG) and the wrong principal (customers). Option D uses the wrong securable object (team) and the wrong principal (customers). References: GRANT, Privilege types, Securable objects, Principals

The STREAM function is used to process data from a streaming live table or view, which is a table or view that contains data that has been added only since the last pipeline update. Streaming live tables and views are stateful, meaning that they retain the state of the previous pipeline run and only process new data based on the current query. This is useful for incremental processing of streaming or batch data sources. The customers table in the query is a streaming live table, which means that it contains the latest data from the source. The STREAM function enables the query to read the data from the customers table incrementally and create another streaming live table named loyal_customers, which contains the customer IDs of the customers with high loyalty level. References: Difference between LIVE TABLE and STREAMING LIVE TABLE, CREATE STREAMING TABLE, Load data using streaming tables in Databricks SQL.

Option B keeps environment differences entirely in the bundle configuration. Declarative Automation Bundles support custom variables that can be referenced by resource definitions, making the same job and notebook definitions reusable. Databricks targets represent deployment environments such as development, test, and production, and target-level settings take precedence over top-level settings, including variable overrides. This allows the team to deploy the same codebase while supplying different workspace paths, compute identifiers, catalog names, or other values for each environment. Editing deployed jobs in the workspace UI creates configuration drift and breaks the bundle as the authoritative source. Separate Git branches duplicate configuration and promotion logic, while notebook parameters move environment logic into application code. Therefore, defining variables and overriding them through targets is the correct configuration-only approach.

================

Option D satisfies the requirements because a Lakeflow Connect managed SaaS connector provides source-specific authentication, incremental ingestion, schema evolution, and automated retries as managed capabilities. Managed connectors use serverless infrastructure and publish governed destination tables that can be processed downstream with Lakeflow Spark Declarative Pipelines. This avoids custom code for OAuth token handling, API pagination, CDC state management, retry behavior, and schema-change recovery. Option A still requires the engineering team to implement and maintain CDC and operational logic. Options B and C place most authentication, schema evolution, change tracking, and failure recovery responsibilities in custom notebooks or jobs, conflicting with the minimal-maintenance requirement. Therefore, the fully managed connector described in option D is the only solution that delivers the requested ingestion and operational behavior as an integrated platform capability.

================

A left anti-join returns rows from the left-side relation for which no matching row exists in the right-side relation. By placing customers on the left and orders on the right and joining by customer_id, the result contains precisely the customers who have no corresponding order. A left anti-join also returns only columns from the left relation, which satisfies the requirement to exclude all order-related fields. A left outer join could produce the desired result only with an additional NULL filter and correct table order. An inner join returns customers who have placed orders, which is the opposite requirement. A full outer join can introduce order-only rows and unnecessary columns. Therefore, the left anti-join described in option C is the most exact and efficient expression for this requirement.

================

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations such as commit, pull, push, branch management, and visual comparison of diffs when committing1. However, merge is not supported in the Git dialog2. You need to use the Repos UI or your Git provider to merge branches3. Merge is a way to combine the commit history from one branch into another branch1. During a merge, a merge conflict is encountered when Git cannot automatically combine code from one branch into another. Merge conflicts require manual resolution before a merge can be completed1. References: 4: Run Git operations on Databricks Repos4, 1: CI/CD techniques with Git and Databricks Repos1, 3: Collaborate in Repos3, 2: Databricks Repos - What it is and how we can use it2.

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations such as commit, pull, push, merge, and branch management. However, to clone a remote Git repository to a Databricks repo, you must use the Databricks UI or API. You cannot clone a Git repo using the CLI through a cluster’s web terminal, as the files won’t display in the Databricks UI1. References: 1: Run Git operations on Databricks Repos | Databricks on AWS2

Option D provides a practical balance between the 15-minute latency objective and compute cost. A scheduled Lakeflow Job running every 12 minutes leaves a small margin for startup and processing while avoiding an always-running stream. Trigger.AvailableNow processes all records available when the query starts in one or more incremental micro-batches and then terminates, allowing the associated compute to be released after the backlog is consumed. Running the job every minute would introduce unnecessary job-start and storage-check overhead. A continuous trigger would keep compute active even when little or no data is arriving. Triggering a separate run for every individual file can also create excessive runs and operational overhead. Therefore, the scheduled AvailableNow strategy in option D supports bounded latency while minimizing idle compute consumption.

================

When Automatically detect column types is disabled during file upload, Databricks does not infer specialized data types from the CSV values. Instead, every uploaded column is created as STRING. This behavior prevents values that resemble numbers, dates, or Boolean values from being automatically assigned potentially incorrect types. The engineer can subsequently transform or cast individual columns into the required target types as part of a controlled Silver-layer process. Option B is incorrect because partial numeric inference does not occur when automatic detection is disabled. Option C is incorrect because Databricks can create the table without manually supplied types by using STRING for every column. Option D is also incorrect because disabling detection determines the schema behavior for the table creation operation. Therefore, option A accurately describes the result.

================

To minimize the total running time of the SQL endpoint used in the refresh schedule of a dashboard in Databricks, the most effective approach is to utilize the Auto Stop feature. This feature allows the SQL endpoint to automatically stop after a period of inactivity, ensuring that it only runs when necessary, such as during the dashboard refresh or when actively queried. This minimizes resource usage and associated costs by ensuring the SQL endpoint is not running idle outside of these operations.

Databricks describes the Bronze layer in the Medallion Architecture as the raw ingestion layer , where source data is landed with minimal validation or transformation so that the original records are preserved for auditability and reprocessing. This is especially important for sources like Kafka clickstream events and file-based product catalogs, because downstream logic such as cleansing, deduplication, conformance, and business modeling should happen later in Silver and Gold layers. Bronze data is commonly stored in an append-only pattern and may include basic ingestion metadata, but it should not be heavily transformed at this stage. Databricks explicitly recommends limiting cleanup and validation in Bronze and using Silver for validation and deduplication. That makes option C the best match to Databricks guidance. Options A, B, and D all skip or misuse the Bronze layer, which reduces traceability and makes recovery or replay harder if business logic changes later.

=========

The engineer should use Repair run . A repair reruns unsuccessful tasks as part of the original job run while preserving the successful status of tasks that do not need to be executed again. In this scenario, Tasks 1 and 2 remain successful, Task 3 is rerun after its problem is corrected, and Task 4 can execute after its dependency succeeds. Run now starts a separate job run and normally executes the workflow from its beginning. Cloning a task creates another task definition and does not resume the failed run. Reset job is not the appropriate Lakeflow Jobs recovery operation. Databricks also notes that repaired tasks should use idempotent write logic where possible because a failed task might have produced partial output before failing. Therefore, option C correctly resumes the workflow without unnecessarily repeating the successful upstream tasks.

================

When migrating to Delta Live Tables (DLT) with a data pipeline that involves different programming languages across various data layers, the migration does not require unifying the pipeline into a single language. Delta Live Tables support multi-language pipelines, allowing data engineers and data analysts to work in their preferred languages, such as Python for data engineering tasks (raw, bronze, and silver layers) and SQL for data analytics tasks (gold layer). This capability is particularly beneficial in collaborative settings and leverages the strengths of each language for different stages of data processing.

 Option E is the correct answer because it is the only mandatory requirement when creating a new Delta Live Tables pipeline. A pipeline is a data processing workflow that contains materialized views and streaming tables declared in Python or SQL source files. Delta Live Tables infers the dependencies between these tables and ensures updates occur in the correct order. To create a pipeline, you need to specify at least one notebook library to be executed, which contains the Delta Live Tables syntax. You can also specify multiple libraries of different languages within your pipeline. The other options are optional or not applicable for creating a pipeline. Option A is not required, but you can optionally provide a key-value pair configuration to customize the pipeline settings, such as the storage location, the target schema, the notifications, and the pipeline mode. Option B is not applicable, as the DBU/hour cost is determined by the cluster configuration, not the pipeline creation. Option C is not required, but you can optionally specify a storage location for the output data from the pipeline. If you leave it empty, the system uses a default location. Option D is not required, but you can optionally specify a location of a target database for the written data, either in the Hive metastore or the Unity Catalog.

 Tutorial: Run your first Delta Live Tables pipeline, What is Delta Live Tables?, Create a pipeline, Pipeline configuration.

databricks bundle validate should be run first because it checks whether the bundle configuration files are syntactically valid and reports configuration warnings. After validation succeeds, databricks bundle deploy creates or updates the bundle-defined resources and uploads the required artifacts to the selected workspace target. The final command, databricks bundle run, starts a deployed job or Lakeflow pipeline by using its resource key. bundle init creates a new project from a template and is not the validation step for an already developed bundle. bundle sync synchronizes files for development workflows but does not replace a complete deployment. The commands bundle execute, bundle check, bundle upload, and bundle start are not the appropriate command sequence. Therefore, option B is correct.

The DELETE command in Delta Lake allows you to remove data that matches a predicate from a Delta table. This command will delete all the rows where the value in the column age is greater than 25 from the existing Delta table my_table and save the updated table. The other options are either incorrect or do not achieve the desired result. Option A will only select the rows that match the predicate, but not delete them. Option B will update the rows that match the predicate, but not delete them. Option D will update the rows that do not match the predicate, but not delete them. Option E will delete the rows that do not match the predicate, which is the opposite of what we want. References: Table deletes, updates, and merges — Delta Lake Documentation

A Gold table is a table that contains highly refined and aggregated data that powers analytics, machine learning, and production applications. It represents data that has been transformed into knowledge, rather than just information. A Gold table is typically the final output of a medallion lakehouse architecture, where data flows from Bronze to Silver to Gold tables, with each layer improving the structure and quality of data. A job that queries aggregated data designed to feed into a dashboard is an example of a data workload that will utilize a Gold table as its source, as it requires data that is ready for consumption and analysis. The other options are either data workloads that will use a Bronze or Silver table as their source, or data workloads that will produce a Gold table as their output. References: Databricks Documentation - What is the medallion lakehouse architecture?, Databricks Documentation - What is a Medallion Architecture?, K21Academy - Delta Lake Architecture & Azure Databricks Workspace.

In Production mode, the pipeline runs continuously and updates the output tables whenever new data is available in the input sources. The compute resources are allocated on demand and released when the pipeline is stopped. This mode is suitable for production workloads that require high availability and reliability. References: Configure pipeline settings for Delta Live Tables, Tutorial: Run your first Delta Live Tables pipeline, Building Reliable Data Pipelines Using DataBricks’ Delta Live Tables

A table update trigger is designed for this dependency. Databricks Lakeflow Jobs can monitor one or more supported Unity Catalog tables and start a job when the monitored tables are updated. Supported changes include data updates, merges, and deletes, so the Silver pipeline can begin only after the Bronze table receives new or changed data. This event-driven approach avoids polling and does not require a continuously running cluster. A continuous trigger keeps the job running rather than reacting to a completed table update. A scheduled trigger runs according to time, regardless of whether the Bronze table changed. A file arrival trigger monitors new files in a Unity Catalog storage location, not updates to an upstream table. Therefore, option A directly implements the required table-to-pipeline orchestration.

================

A webhook alert destination is a notification destination that allows Databricks to send HTTP POST requests to a third-party endpoint when an alert is triggered. This enables the data engineer to integrate Databricks alerts with their preferred messaging or collaboration platform, such as Slack, Microsoft Teams, or PagerDuty. To set up a webhook alert destination, the data engineer needs to create and configure a webhook connector in their messaging platform, and then add the webhook URL to the Databricks notification destination. After that, the data engineer can create an alert for their Databricks SQL query, and select the webhook alert destination as the notification destination. The alert can be configured with a custom condition, such as when the number of stores with $0 in sales is greater than zero, and a custom message template, such as “Alert: {number_of_stores} stores have $0 in sales”. The alert can also be configured with a recurrence interval, such as every hour, to check the query result periodically. When the alert condition is met, the data engineer and their team will receive a notification via the messaging webhook, with the custom message and a link to the Databricks SQL query. The other options are either not suitable for sending notifications via a messaging webhook (A, B, E), or not suitable for sending recurring notifications ©. References: Databricks Documentation - Manage notification destinations, Databricks Documentation - Create alerts for Databricks SQL queries, Databricks Documentation - Configure alert conditions and messages.

 Create a table using SQL | Databricks on AWS, Create a table using SQL - Azure Databricks, Delta Lake Quickstart - Azure Databricks

Auto Loader is a tool that can incrementally and efficiently process new data files as they arrive in cloud storage without any additional setup. Auto Loader provides a Structured Streaming source called cloudFiles, which automatically detects and processes new files in a given input directory path on the cloud file storage. Auto Loader also tracks the ingestion progress and ensures exactly-once semantics when writing data into Delta Lake. Auto Loader can ingest various file formats, such as JSON, CSV, XML, PARQUET, AVRO, ORC, TEXT, and BINARYFILE. Auto Loader has support for both Python and SQL in Delta Live Tables, which are a declarative way to build production-quality data pipelines with Databricks. References: What is Auto Loader?, Get started with Databricks Auto Loader, Auto Loader in Delta Live Tables

COPY INTO is idempotent by default and tracks files that have already been loaded into the target table. A corrected file uploaded under the same name and path can therefore be treated as previously processed and skipped. The reliable pattern is to publish the correction under a new unique filename so that ingestion detects it as a new file. The pipeline can then use a business key with MERGE, UPDATE, or equivalent transformation logic to apply the corrected values to the target Delta table. Overwriting or recreating the complete table risks unnecessary data loss, increases processing cost, and disrupts downstream consumers. Reloading every historical source file is also inefficient and can introduce duplicates. Therefore, option B supports incremental ingestion while applying corrections safely and efficiently.

================

The File Arrival trigger and Auto Loader perform separate, complementary functions. The trigger monitors the configured storage location and starts the Lakeflow Job after detecting new files. After the notebook starts, Auto Loader examines its own checkpoint and file-tracking state to determine which files have not yet been committed successfully. This separation provides incremental processing and exactly-once ingestion behavior. With availableNow=True, Auto Loader processes all data available when the query starts and then stops, allowing the job compute to terminate and control costs. File Notification mode is not required solely because a File Arrival trigger is used. Disabling checkpoints would remove progress tracking and could cause duplicate processing. The trigger also does not eliminate the need to configure the Auto Loader source path. Therefore, option A is correct.

================

To read from a stream source, the data engineer needs to use the spark.readStream method instead of the spark.read method. The spark.readStream method returns a DataStreamReader object that can be used to specify the details of the input source, such as the format, the schema, the path, and the options. The spark.read method is only suitable for batch processing, not streaming processing. The other changes are not necessary or correct for reading from a stream source. References: Structured Streaming Programming Guide, Read a stream, Databricks Data Sources

The Databricks web application is the user interface that allows you to create and manage workspaces, clusters, notebooks, jobs, and other resources. It is hosted completely in the control plane of the classic Databricks architecture, which includes the backend services that Databricks manages in your Databricks account. The other options are part of the compute plane, which is where your data is processed by compute resources such as clusters. The compute plane is in your own cloud account and network. References: Databricks architecture overview, Security and Trust Center QUESTION NO: 4

Which of the following benefits of using the Databricks Lakehouse Platform is provided by Delta Lake?

A. The ability to manipulate the same data using a variety of languages

B. The ability to collaborate in real time on a single notebook

C. The ability to set up alerts for query failures

D. The ability to support batch and streaming workloads

E. The ability to distribute complex data operations

Answer: D

Delta Lake is the optimized storage layer that provides the foundation for storing data and tables in the Databricks lakehouse. Delta Lake is fully compatible with Apache Spark APIs, and was developed for tight integration with Structured Streaming, allowing you to easily use a single copy of data for both batch and streaming operations and providing incremental processing at scale1. Delta Lake supports upserts using the merge operation, which enables you to efficiently update existing data or insert new data into your Delta tables2. Delta Lake also provides time travel capabilities, which allow you to query previous versions of your data or roll back to a specific point in time3. References: 1: What is Delta Lake? | Databricks on AWS 2: Upsert into a table using merge | Databricks on AWS 3: [Query an older snapshot of a table (time travel) | Databricks on AWS]

Learn more

The small country lookup DataFrame should be broadcast to the executors. A broadcast join copies the small side of the join to each worker, allowing every executor to join its partition of the billion-row events DataFrame locally. This avoids shuffling the very large DataFrame across the network and is effective when one relation is small enough to fit comfortably in executor memory. Broadcasting events_df would be impractical because it contains billions of rows. Caching or persisting both DataFrames can help when data is reused, but neither action by itself changes the join strategy or removes the expensive shuffle. Databricks exposes the PySpark broadcast function specifically to mark a DataFrame as suitable for a broadcast join. Therefore, option D is the correct optimization.

================

JSON data is a text-based format that represents data as a collection of name-value pairs. By default, when Auto Loader infers the schema of JSON data, it treats all columns as strings. This is because JSON data can have varying data types for the same column across different files or records, and Auto Loader does not attempt to reconcile these differences. For example, a column named “age” may have integer values in some files, but string values in others. To avoid data loss or errors, Auto Loader infers the column as a string type. However, Auto Loader also provides an option to infer more precise column types based on the sample data. This option is called cloudFiles.inferColumnTypes and it can be set to true or false. When set to true, Auto Loader tries to infer the exact data types of the columns, such as integers, floats, booleans, or nested structures. When set to false, Auto Loader infers all columns as strings. The default value of this option is false. References: Configure schema inference and evolution in Auto Loader, Schema inference with auto loader (non-DLT and DLT), Using and Abusing Auto Loader’s Inferred Schema, Explicit path to data or a defined schema required for Auto loader.

Workspace-specific deployment settings should be defined under each environment in the targets section of databricks.yml. A bundle target represents a deployment environment such as development, staging, or production. Each target can contain its own workspace mapping, including the workspace host and an optional root_path. This structure allows one bundle codebase to be deployed predictably to different environments without embedding infrastructure details in notebooks or pipeline source code. A README file is documentation and is not interpreted as bundle configuration. Hardcoding workspace details in Python couples transformation logic to a particular environment. Global local environment variables are machine-specific and do not provide a complete, version-controlled bundle definition. Therefore, option C correctly places the workspace-specific settings in the declarative configuration used during validation and deployment.

================

The < = > operator performs null-safe equality in Databricks SQL. It returns TRUE when both operands contain the same non-NULL value and also when both operands are NULL. It returns FALSE when only one operand is NULL. By contrast, the standard equality operator returns UNKNOWN when either side is NULL, so it cannot match two NULL values in a WHERE condition. Option C implements inequality logic and does not provide null-safe equality. Option D can incorrectly treat a genuine empty string as equivalent to NULL, changing the meaning of the data. Therefore, option A is the only predicate that preserves SQL values while implementing the required NULL comparison behavior. This question was also checked against the existing Word question bank.

================

 The keyword that can be used to compact the small files associated with a Delta table is OPTIMIZE. The OPTIMIZE command performs file compaction on a Delta table by rewriting a set of small files into a set of larger files1. This can improve the performance of queries that scan the table by reducing the number of files that need to be read and the amount of metadata that needs to be processed1. The OPTIMIZE command can also optionally sort the data within each file by a given set of columns, which can further improve the query performance by enabling data skipping and predicate pushdown1. The OPTIMIZE command can be applied to the whole table or to a specific partition of the table1.

The other keywords are not suitable for compacting the small files associated with a Delta table. REDUCE is a keyword used in the SQL syntax for aggregating data using a user-defined function2. COMPACTION is not a valid keyword in SQL or Python. REPARTITION is a keyword used in the Python syntax for changing the number of partitions of a DataFrame or an RDD3. VACUUM is a keyword used to remove files that are no longer referenced by a Delta table and are older than a retention threshold4.

1: OPTIMIZE | Databricks on AWS

2: REDUCE | Databricks on AWS

3: repartition | Databricks on AWS

4: VACUUM | Databricks on AWS

 A serverless SQL endpoint is a compute resource that is automatically managed by Databricks and scales up or down based on the workload. A serverless SQL endpoint can be used to run queries and dashboards without requiring manual configuration or management. A serverless SQL endpoint is only active when it is needed and shuts down automatically when idle, minimizing the total running time and cost. A serverless SQL endpoint can be created and assigned to a dashboard using the Databricks SQL UI or the SQL Analytics API. References:

Create a serverless SQL endpoint

Assign a SQL endpoint to a dashboard

SQL Analytics API