Lead Databricks Engineer

Company Overview
Milestone Technologies is a global IT managed services firm that partners with organizations to scale their technology, infrastructure and services to drive specific business outcomes such as digital transformation, innovation, and operational agility. Milestone is focused on building an employee-first, performance-based culture and for over 25 years, we have a demonstrated history of supporting category-defining enterprise clients that are growing ahead of the market. The company specializes in providing solutions across Application Services and Consulting, Digital Product Engineering, Digital Workplace Services, Private Cloud Services, AI/Automation, and ServiceNow. Milestone culture is built to provide a collaborative, inclusive environment that supports employees and empowers them to reach their full potential.

Our seasoned professionals deliver services based on Milestone's best practices and service delivery framework. By leveraging our vast knowledge base to execute initiatives, we deliver both short-term and long-term value to our clients and apply continuous service improvement to deliver transformational benefits to IT. With Intelligent Automation, Milestone helps businesses further accelerate their IT transformation. The result is a sharper focus on business objectives and a dramatic improvement in employee productivity. Through our key technology partnerships and our people-first approach, Milestone continues to deliver industry-leading innovation to our clients. With more than 3,000 employees serving over 200 companies worldwide, we are following our mission of revolutionizing the way IT is deployed.

Job Overview
Role Overview:
We are looking for a seasoned and visionary Lead Databricks Engineerwith over 14 years of experience to lead the design and implementation of enterprise-grade data architecture, governance frameworks, and scalable data platforms. This role requires deep expertise in modern data technologies, big data ecosystems, and cloud-native solutions. The ideal candidate will be a strategic leader who can bridge business needs with technical solutions, ensuring data integrity, quality, and accessibility across the organization.

Responsibilities

• Lead the design and implementation of Data Architecture, Data Quality frameworks, and Data Governance initiatives.
• Define data modeling standards and establish foundational best practices across data platforms.
• Champion and promote data quality principles by developing and institutionalizing enterprise-wide standards and methodologies.
• Build and oversee data governance policies, procedures, and frameworks to ensure data integrity, security, and compliance.
• Drive enterprise-wide adoption of self-service data platforms and enablement of organizational data literacy and utilization.
• Define and maintain critical data standards and metadata to facilitate the consistent understanding and use of data as a shared enterprise asset.
• Create reusable templates and establish coding standards for sourcing, collecting, and transforming both streaming and batch data.
• Design scalable data schemas, object models, and flow diagrams to effectively structure, process, and integrate data across systems.
• Provide architectural strategy, assessments, and roadmaps to guide enterprise data management practices.
• Offer hands-on expertise in architecting and managing Big Data platforms and Data Lake technologies (e.g., AWS S3, Hive), along with exposure to ML and Data Science tools.
• Implement and manage industry-standard tools and frameworks such as Databricks, Delta Lake, Apache Spark, Airflow, Hive Metastore, Redshift, Kafka, Kubernetes, Docker, and CI/CD pipelines.
• Translate complex big data and analytics requirements into high-performance, scalable data models and guide analytics engineers on implementing these models.
• Define reusable templates and processes for data modeling, data flow design, and system integration.
• Lead, mentor, and upskill data analytics teams by promoting best practices, modern technologies, and effective processes in data platform engineering.

Tech Skills

• 6+ years of hands-on experience in Data Warehouse, ETL, Data Modeling & Reporting
• 5+ years of hands-on experience in productionizing and deploying Big Data platforms and applications.
• Minimum of 3 years of hands-on DataBricks experience required.
• Hands-on experience working with: Relational/SQL, distributed columnar data stores/NoSQL databases, time-series databases, Spark streaming, Kafka, Hive, Delta Parquet, Avro, and more
• Extensive experience in understanding a variety of complex business use cases and modeling the data in the data warehouse
• Highly skilled in SQL, Python, Spark, AWS S3, Hive Data Catalog, Parquet, Redshift, Airflow, and Tableau or similar tools
• Proven experience in building a Custom Enterprise Data Warehouse or implementing tools like Data Catalogs, Spark, Tableau, Kubernetes, and Docker
• Demonstrated industry leadership in the fields of Data Warehousing, Data Science, and Big Data related technologies
• Design and Implement Machine learning models with Tensorflow ,Keras and pytorch And MLops

Compensation
Estimated Pay Range:

Exact compensation and offers of employment are dependent on circumstances of each case and will be determined based on job-related knowledge, skills, experience, licenses or certifications, and location.

*Our Commitment to Diversity & Inclusion*
At Milestone we strive to create a workplace that reflects the communities we serve and work with, where we all feel empowered to bring our full, authentic selves to work. We know creating a diverse and inclusive culture that champions equity and belonging is not only the right thing to do for our employees but is also critical to our continued success.

Milestone Technologies provides equal employment opportunity for all applicants and employees. All qualified applicants will receive consideration for employment and will not be discriminated against on the basis of race, color, religion, gender, gender identity, marital status, age, disability, veteran status, sexual orientation, national origin, or any other category protected by applicable federal and state law, or local ordinance. Milestone also makes reasonable accommodations for disabled applicants and employees.

We welcome the unique background, culture, experiences, knowledge, innovation, self-expression and perspectives you can bring to our global community. Our recruitment team is looking forward to meeting you.