Data Architect/Databricks Consultant

Position OverviewWe are seeking a specializedDatabricks Architect with deep expertise in cost optimization and migrationstrategies, particularly focused on transitioning away from Databricksplatforms. The ideal candidate will have extensive experience in Spark clustersolutions and a proven track record of reducing Databricks operational costswhile architecting successful migration paths to alternative platforms.Key ResponsibilitiesDatabricks Cost OptimizationConduct comprehensive cost analysis and auditing of existing Databricks deployments across multiple workspacesDevelop and implement aggressive cost reduction strategies targeting 30 -50% savings through cluster optimizationDesign and deploy automated cost monitoring solutions with real -time alerts and budget controlsOptimize cluster configurations, auto -scaling policies, and job scheduling to minimize compute costsImplement spot instance strategies and preemptible VM usage for non -critical workloadsEstablish cost allocation frameworks and implement chargeback mechanisms for business unit accountabilityCreate cost governance policies and developer guidelines to prevent cost overrunsAnalyze and optimize storage costs including Delta Lake table optimization and data lifecycle managementMigration from Databricks ArchitectureLead strategic initiatives to migrate workloads away from Databricks to cost -effective alternativesAssess existing Databricks implementations and create detailed migration roadmaps to target platformsDesign migration architectures for transitioning to open -source Spark on Kubernetes, EMR, or other platformsDevelop automated migration tools and frameworks to minimize business disruptionCreate comprehensive migration strategies including data export, job conversion, and dependency mappingEstablish parallel running environments to ensure zero -downtime migrationsLead post -migration validation and performance benchmarking against original Databricks solutionsDocument lessons learned and create reusable migration playbooks for future projectsAdvanced Spark Cluster SolutionsDesign high -performance, cost -optimized Spark cluster architectures outside of Databricks ecosystemImplement custom Spark solutions on Kubernetes, YARN, and standalone cluster managersOptimize Spark job performance through advanced tuning of memory management, serialization, and parallelismDevelop custom Spark operators and applications for specialized business use casesTroubleshoot complex Spark performance bottlenecks and implement optimization strategiesCreate cluster auto -scaling solutions and dynamic resource allocation frameworksDesign fault -tolerant Spark architectures with disaster recovery and high availabilityImplement monitoring and alerting for Spark cluster health and job performance metricsStrategic Planning & ExecutionCollaborate with finance teams to develop multi -year cost reduction roadmapsEvaluate and recommend alternative platforms based on cost -benefit analysisCreate business cases for migration projects with detailed ROI calculationsEstablish technical debt reduction strategies related to Databricks dependenciesPartner with procurement teams on contract negotiations and vendor managementRequired QualificationsSpecialized Experience8+ years of experience in big data architecture with focus on cost optimization5+ years of hands -on Databricks experience with proven cost reduction achievementsDemonstrated experience architecting and executing complete platform migrations from Databricks to alternative solutions with successful outcomes6+ years of advanced Apache Spark development and cluster management experienceTrack record of achieving significant cost savings (minimum 40%+) in cloud data platformsCost Optimization ExpertiseExpert knowledge of Databricks pricing models, compute types, and cost driversExperience with FinOps practices and cloud cost management toolsProven ability to implement automated cost controls and budget management systemsKnowledge of alternative platforms and their cost structures (EMR, HDInsight, GCP Dataproc, etc.)Migration & Spark Technical SkillsDeep expertise in migrating complex data workloads between different Spark platformsAdvanced knowledge of Spark internals, catalyst optimizer, and performance tuningExperience with Kubernetes -based Spark deployments and container orchestrationProficiency in infrastructure -as -code for multi -cloud Spark cluster provisioningStrong background in data pipeline migration and ETL/ELT conversion strategiesProgramming & Platform SkillsExpert -level proficiency in Scala, Python, and Java for Spark developmentAdvanced SQL skills and experience with multiple database technologiesExperience with open -source alternatives to Databricks (Apache Spark, Delta Lake OSS, MLflow OSS)Knowledge of streaming platforms (Kafka, Kinesis, Pulsar) and real -time architecturesProficiency with monitoring tools (Prometheus, Grafana, ELK stack)Preferred QualificationsDatabricks certifications combined with experience in competitive platformsCloud cost management certifications (AWS Cost Optimization, Azure Cost Management)Experience with vendor negotiations and contract optimizationBackground in building business cases for platform migrationsKnowledge of data governance during platform transitionsExperience with Apache Iceberg, Hudi, or other open table formats as Delta Lake alternativesEducationBachelor's degree in Computer Science, Engineering, Information Technology, or related fieldMaster's degree preferred but not required with equivalent experience