Databricks (DE)

Consultant on this level understands and can clearly communicate what is Databricks Lakehouse Platform (high level), what is Delta Lake (general concepts), what is Databricks SQL and what is Databricks Machine Learning (ML) workspaces. Able to navigate between and within workspaces and perform basic operations on existing data sets. Can support existing routines with guidance and requires support in unfamiliar circumstances.

DENG

Consultant at this level should be able comfortably operate with databases, tables and views on Databricks. Using basic Databricks SQL comfortable with creating db objects, writing data into tables, cleaning data, combining and reshaping tables in SQL Analytics workspace. Familiar with basic Python and able to do basic data manipulations using PySpark in Data Engineering Workspace.

DENG

Consultant on this level is comfortable with ingesting data into Databricks platform, integrating BI Tools with Databricks, and able to create data visualizations on Databricks. Comfortable with theory and implementation of multi-hop architecture (bronze-silver-gold), and able to freely work with Delta Live Tables and DLT pipelines.
Comfortable with building production pipelines for data engineering applications and Databricks SQL queries and dashboards, including Jobs/Workflows (scheduling, task orchestration) and Dashboards (endpoints, scheduling, alerting, refreshing). For more complex solutions requires guidance from the lead.

DENG

Consultant at this level should have a deep knowledge and understanding of Databricks Lakehouse architecture and its benefits. Familiar with different types of clusters and compute configuration.
Comfortable with streaming and incremental data processing techniques including Spark structured streaming, watermarks and Auto Loader. Able to perform basic query execution plan analysis using Spark UI.
Achieved industry recognition on Associate Certification level or similar. Comfortable working independently or a part of the team.

DENG

Consultant on this level should be comfortable with optimizing data storage, understanding delta lake transactions and its properties, features and limitations. Able to perform quality enforcement on streaming data and implement slowly changing dimensions (Type 1 or Type 2). Comfortable with securing data on Databricks platform, including accessing PII, implementing RBAC etc. Comfortable with monitoring, logging and handling errors as well as programmatic platform interactions (Databricks CLI, REST API)
Comfortable working independently or leading the team.

DENG

Consultant at this level should be comfortable with cluster policies and Spark cluster configuration for high performance. Comfortable with query optimization tasks, able to identify and rectify five most common performance issues with Spark applications: "Spill", "Skew", "Shuffles", "Storage", "Serialization". With the guidance of Databricks Architect should be able to come up with Databricks Lakehouse cost-effective solution architecture diagram and implementation plan for a given environment.
Achieved industry recognition on Professional Certification level or similar. Comfortable working independently or leading large internal or external team.

DENG

Consultant on this level should be able to facilitate strategic and architectural conversations with a Customer, deeply understand Databricks product, its use cases, benefits and limitations. Able to guide internal and external teams on any Databricks related topic. Achieved wide industry recognition (i.e. Databricks Solution Architect Champion or equivalent).

DENG