Databricks Architect

Job Description

- Overall 10-18 yrs. of Data Engineering experience with Minimum 4+ years of hands on experience in Databricks. Ready to travel Onsite and work at client location.
- Proven hands-on experience as a Databricks Architect or similar role with a deep understanding of the Databricks platform and its capabilities.
- Analyze business requirements and translate them into technical specifications for data pipelines, data lakes, and analytical processes on the Databricks platform.
- Design and architect end-to-end data solutions, including data ingestion, storage, transformation, and presentation layers, to meet business needs and performance requirements.
- Lead the setup, configuration, and optimization of Databricks clusters, workspaces, and jobs to ensure the platform operates efficiently and meets performance benchmarks.
- Manage access controls and security configurations to ensure data privacy and compliance.
- Design and implement data integration processes, ETL workflows, and data pipelines to extract, transform, and load data from various sources into the Databricks platform.
- Optimize ETL processes to achieve high data quality and reduce latency.
- Monitor and optimize query performance and overall platform performance to ensure efficient execution of analytical queries and data processing jobs.
- Identify and resolve performance bottlenecks in the Databricks environment.
- Establish and enforce best practices, standards, and guidelines for Databricks development, ensuring data quality, consistency, and maintainability.
- Implement data governance and data lineage processes to ensure data accuracy and traceability.
- Mentor and train team members on Databricks best practices, features, and capabilities.
- Conduct knowledge-sharing sessions and workshops to foster a data-driven culture within the organization.
- Will be responsible for Databricks Practice Technical/Partnership initiatives.
- Build skills in technical areas which support the deployment and integration of Databricks-based solutions to complete customer projects.

Roles & Responsibilities

- Bachelor's or Masters degree in Computer Science, Information Technology, or related field.
- In depth hands-on implementation knowledge on Databricks. Delta Lake, Delta table - Managing Delta Tables, Databricks Cluster Configuration, Cluster policies.
- Experience handling structured and unstructured datasets
- Strong proficiency in programming languages like Python, Scala, or SQL.
- Experience with Cloud platforms like AWS, Azure, or Google Cloud, and understanding of cloud-based data storage and computing services.
- Familiarity with big data technologies like Apache Spark, Hadoop, and data lake architectures.
- Develop and maintain data pipelines, ETL workflows, and analytical processes on the Databricks platform.
- Should have good experience in Data Engineering in Databricks Batch process and Streaming
- Should have good experience in creating Workflows & Scheduling the pipelines.
- Should have good exposure on how to make packages or libraries available in DB.
- Familiarity in Databricks default runtimes
- Databricks Certified Data Engineer Associate/Professional Certification (Desirable).
- Should have experience working in Agile methodology
- Strong verbal and written communication skills.
- Strong analytical and problem-solving skills with a high attention to detail.

Role:Solution Architect

Industry Type:IT Services & Consulting

Department:Engineering - Software & QA

Employment Type:Full Time, Permanent

Role Category:Software Development

Education

UG:BCA in Any Specialization, B.Tech/B.E. in Any Specialization

PG:MCA in Any Specialization, M.Tech in Any Specialization