Genzeon - Databricks Migration Consultant/Architect

Databricks Migration Architect / Consultant 8+

Location - Remote

Contract Directly with Genzeon| Duration: 2-3 Overview :

We are seeking a specialized Databricks Architect with deep expertise in cost optimization and migration strategies, particularly focused on transitioning away from Databricks platforms. The ideal candidate will have extensive experience in Spark cluster solutions and a proven track record of reducing Databricks operational costs while architecting successful migration paths to alternative Responsibilities : Cost Optimization :

- Conduct comprehensive cost analysis and auditing of existing Databricks deployments across multiple workspaces

- Develop and implement aggressive cost reduction strategies targeting 30-50% savings through cluster optimization

- Design and deploy automated cost monitoring solutions with real-time alerts and budget controls

- Optimize cluster configurations, auto-scaling policies, and job scheduling to minimize compute costs

- Implement spot instance strategies and preemptible VM usage for non-critical workloads

- Establish cost allocation frameworks and implement chargeback mechanisms for business unit accountability

- Create cost governance policies and developer guidelines to prevent cost overruns

- Analyze and optimize storage costs including Delta Lake table optimization and data lifecycle from Databricks Architecture :

- Lead strategic initiatives to migrate workloads away from Databricks to cost-effective alternatives

- Assess existing Databricks implementations and create detailed migration roadmaps to target platforms

- Design migration architectures for transitioning to open-source Spark on Kubernetes, EMR, or other platforms

- Develop automated migration tools and frameworks to minimize business disruption

- Create comprehensive migration strategies including data export, job conversion, and dependency mapping

- Establish parallel running environments to ensure zero-downtime migrations

- Lead post-migration validation and performance benchmarking against original Databricks solutions

- Document lessons learned and create reusable migration playbooks for future Spark Cluster Solutions :

- Design high-performance, cost-optimized Spark cluster architectures outside of Databricks ecosystem

- Implement custom Spark solutions on Kubernetes, YARN, and standalone cluster managers

- Optimize Spark job performance through advanced tuning of memory management, serialization, and parallelism

- Develop custom Spark operators and applications for specialized business use cases

- Troubleshoot complex Spark performance bottlenecks and implement optimization strategies

- Create cluster auto-scaling solutions and dynamic resource allocation frameworks

- Design fault-tolerant Spark architectures with disaster recovery and high availability

- Implement monitoring and alerting for Spark cluster health and job performance Planning & Execution :

- Collaborate with finance teams to develop multi-year cost reduction roadmaps

- Evaluate and recommend alternative platforms based on cost-benefit analysis

- Create business cases for migration projects with detailed ROI calculations

- Establish technical debt reduction strategies related to Databricks dependencies

- Partner with procurement teams on contract negotiations and vendor Qualifications : Experience :

- 8+ years of experience in big data architecture with focus on cost optimization

- 5+ years of hands-on Databricks experience with proven cost reduction achievements

- Demonstrated experience architecting and executing complete platform migrations from Databricks to alternative solutions with successful outcomes

- 6+ years of advanced Apache Spark development and cluster management experience

- Track record of achieving significant cost savings (minimum 40%+) in cloud data Optimization Expertise :

- Expert knowledge of Databricks pricing models, compute types, and cost drivers

- Experience with FinOps practices and cloud cost management tools

- Proven ability to implement automated cost controls and budget management systems

- Knowledge of alternative platforms and their cost structures (EMR, HDInsight, GCP Dataproc, & Spark Technical Skills :

- Deep expertise in migrating complex data workloads between different Spark platforms

- Advanced knowledge of Spark internals, catalyst optimizer, and performance tuning

- Experience with Kubernetes-based Spark deployments and container orchestration

- Proficiency in infrastructure-as-code for multi-cloud Spark cluster provisioning

- Strong background in data pipeline migration and ETL/ELT conversion & Platform Skills :

- Expert-level proficiency in Scala, Python, and Java for Spark development

- Advanced SQL skills and experience with multiple database technologies

- Experience with open-source alternatives to Databricks (Apache Spark, Delta Lake OSS, MLflow OSS)

- Knowledge of streaming platforms (Kafka, Kinesis, Pulsar) and real-time architectures

- Proficiency with monitoring tools (Prometheus, Grafana, ELK Qualifications :

- Databricks certifications combined with experience in competitive platforms

- Cloud cost management certifications (AWS Cost Optimization, Azure Cost Management)

- Experience with vendor negotiations and contract optimization

- Background in building business cases for platform migrations

- Knowledge of data governance during platform transitions

- Experience with Apache Iceberg, Hudi, or other open table formats as Delta Lake :

- Bachelor's degree in Computer Science, Engineering, Information Technology, or related field

- Master's degree preferred but not required with equivalent experience

(ref:hirist.tech)

---