Databricks Sme

**Databricks Expert will be required to set up the Databricks Platform on AWS, so he or she needs to be proficient with infrastructure as code technology such as CloudFormation and Terraform to work together with the Platform Engineering team.**

**Must have knowledge in Networking and Security Architecture for a successful implementation.**
- **Design & Build Data Pipelines**: Develop, optimize, and manage large-scale data engineering workflows using **Databricks**and **Apache Spark**, ensuring scalable and efficient data processing.
- **Data Lake & Delta Lake**: Build and manage **lake houses/data lakes**using **Databricks Delta Lake**to ensure high-performance, scalable, and reliable storage for structured and unstructured data.
- **Collaboration with Cross-Functional Teams**: Work with data scientists, analysts, and business stakeholders to understand data needs, provide insights, and deliver solutions using Databricks.
- **Performance Tuning & Optimization**: Tune and optimize Spark-based jobs and other Databricks workflows for high performance and low latency across large datasets.
- **Data Integration**: Integrate and orchestrate data from multiple sources (e.g., databases, APIs, cloud services) into the Databricks environment, ensuring data quality and consistency.
- **Cloud Infrastructure Management**: Leverage **cloud platforms**(e.g., AWS, Azure, GCP) for efficient use of **Databricks**clusters, ensuring cost optimization and scalability.
- **Machine Learning Pipeline Development**: Collaborate with data scientists to design and implement end-to-end machine learning pipelines, utilizing **MLflow**for model tracking, versioning, and deployment.
- **Model Deployment & Monitoring**: Collaborate on model deployment strategies within **Databricks**, using **MLflow**for tracking, managing experiments, and deploying models into production environments.
- **Automation & CI/CD**: Implement continuous integration and continuous delivery (CI/CD) pipelines to automate the deployment of data pipelines and machine learning models.

**Skills & Qualifications**:
**Required**:

- **Bachelor’s / Master’s degree in computer science**, Engineering or a related field.
- **7+ years**of experience in **data engineering**, with hands-on expertise in **Databricks**and **Apache Spark**.
- Proficiency in **Python**and **SQL**, with extensive experience in data processing and model development in **Databricks**.
- Strong experience with **Databricks Delta Lake**for efficient data storage and management.
- Experience building and optimizing **ETL**/ **ELT**pipelines for large-scale data processing and integration.
- Knowledge of **cloud platforms**(AWS, Azure, or GCP) and experience managing **Databricks**environments in the cloud.
- Experience in creating scalable data models, workflows, and pipelines for structured and unstructured data.
- Expertise in **performance tuning**and optimization of Spark-based jobs in **Databricks**.
- Having worked under Agile project environment

**Preferred**:

- Databricks certifications such as Apache Spark Programming, Data Engineer Associate or Professional.
- Experience with **Apache Kafka**or other streaming technologies in **Databricks**for real-time data processing.
- Knowledge of **DevOps**practices and **CI/CD**for automating data workflows and model deployments.
- Experience with **data governance**, **security**, and **privacy**best practices in cloud environments.

**Soft Skills**:

- Strong problem-solving abilities and the ability to think critically about data challenges.
- Excellent communication skills, with the ability to explain complex data engineering concepts to non-technical stakeholders.
- Adaptability and willingness to stay up to date with the latest Databricks and data engineering trends and tools.