Principal Data Engineer

2 weeks ago

Bengaluru, India Autodesk Full time

# 25WD9141Position OverviewWe are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform , which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data platform capabilities. You will partner closely with Engineering Managers, Architects, Product teams, and Partner Engineering to modernize our data lakehouse architecture, deliver highly reliable data pipelines, and establish technical excellence across ingestion, processing, and governance.ResponsibilitiesTechnical LeadershipDefine and drive the technical strategy for Batch Processing within ADPDesign and scale PySpark based distributed processing frameworks integrated with Airflow for orchestrationChampion Apache Iceberg-based data lakehouse adoption (versioning, branching, WAP, indexing)Introduce performance optimization patterns for data pipelines (e.g., partition pruning, caching, resource tuning)Establish patterns for metadata-driven, intent-based data processing aligned with AI-assisted pipelinesArchitecture & DeliveryPartner with architects to design multi-tenant, secure, and compliant (SOC2/GDPR/CCPA) batch processing servicesDefine reference architectures and reusable frameworks for cross-domain processing workloadsLead technical reviews, solution designs, and architectural forums across ADPGuide the evolution of Airflow APIs and SDKs to enable scalable pipeline self-serviceMentorship & CollaborationMentor senior engineers, staff engineers, and tech leads within the Batch Processing and adjacent teamsPartner with the Batch Ingestion and Data Security teams to deliver unified ingestion + processing flowsInnovation & ImpactDrive modernization initiatives to migrate away from legacy toolsPioneer AI-augmented data engineering practices (e.g., pipeline optimization agents, anomaly detection)Ensure scalability, cost-efficiency, and reliability for thousands of production pipelines across AutodeskInfluence company-wide data engineering strategy by contributing thought leadership and whitepapersMinimum Qualifications8+ years of experience in software/data engineering, with at least 3 years in big data platform leadership rolesExpert in distributed data processing (Spark, PySpark, Ray, Flink)Deep experience with workflow orchestration (Airflow, Dagster, Prefect)Strong hands-on expertise in Lakehouse technologies (Iceberg, Delta, Hudi) and cloud platforms (AWS/Azure/GCP)Proven track record in architecting secure, multi-tenant, and compliant data platformsSkilled in SQL/NoSQL databases, Snowflake, Glue, and modern metadata/catalog toolsStrong problem-solving, communication, and cross-geo collaboration skillsExperience mentoring engineers, building strong technical culture, and influencing at scalePreferred QualificationsExposure to AI/ML-driven data engineering (optimizers, anomaly detection, auto-scaling)Experience with data governance, lineage, and observability tools (Atlan, Databand, Collibra, etc.)Familiarity with streaming + batch hybrid architectures (Kappa/Lambda)#LI-KS2

  • Principal Data Engineer

    3 days ago

    Bengaluru, India K2 Partnering Solutions Full time

    Job Title: Principal Data/App Engineer, CelonisWork Location: Bengaluru, Karnataka Job Type: Full-time/Permanent with the Client directlyJob DetailsOur Client’s Center for Process Excellence team is on a mission to improve business outcomes and employee experiences by driving step-change improvements in critical enterprise processes, and the technology and...

  • Principal Data Engineer

    3 days ago

    Bengaluru, India K2 Partnering Solutions Full time

    Job Title: Principal Data/App Engineer, CelonisWork Location: Bengaluru, Karnataka 560058Job Type: Full-time/Permanent with the Client directlyJob DetailsOur Client’s Center for Process Excellence team is on a mission to improve business outcomes and employee experiences by driving step-change improvements in critical enterprise processes, and the...

  • Principal Data Engineer

    1 day ago

    Bengaluru, India K2 Partnering Solutions Full time

    Job Title: Principal Data/App Engineer, CelonisWork Location: Bengaluru, Karnataka 560058Job Type: Full-time/Permanent with the Client directlyJob DetailsOur Client’s Center for Process Excellence team is on a mission to improve business outcomes and employee experiences by driving step-change improvements in critical enterprise processes, and the...

  • Principal Data Engineer

    1 day ago

    Bengaluru, India K2 Partnering Solutions Full time

    Job Title: Principal Data/App Engineer, CelonisWork Location: Bengaluru, Karnataka 560058Job Type: Full-time/Permanent with the Client directlyJob DetailsOur Client’s Center for Process Excellence team is on a mission to improve business outcomes and employee experiences by driving step-change improvements in critical enterprise processes, and the...

  • Principal Data Engineer

    3 days ago

    Bengaluru, India K2 Partnering Solutions Full time

    Job Title: Principal Data/App Engineer, Celonis Work Location: Bengaluru, Karnataka Job Type: Full-time/Permanent with the Client directly Job Details Our Client’s Center for Process Excellence team is on a mission to improve business outcomes and employee experiences by driving step-change improvements in critical enterprise processes, and the technology...

  • Principal Data Engineer

    23 hours ago

    Bengaluru, India K2 Partnering Solutions Full time

    Job Title: Principal Data/App Engineer, Celonis Work Location: Bengaluru, Karnataka 560058 Job Type: Full-time/Permanent with the Client directly Job Details Our Client’s Center for Process Excellence team is on a mission to improve business outcomes and employee experiences by driving step-change improvements in critical enterprise processes, and the...

  • Principal Data Engineer

    6 days ago

    Bengaluru, Karnataka, India Amadeus Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Job TitlePrincipal Data EngineerSummary of the RoleAs a Principal Data Engineer, you'll design and maintain scalable data systems that support analytics and decision-making. You'll collaborate with cross-functional teams and help shape our data capabilities while supporting others in their growth.In this role you'llDesign and maintain reliable, scalable data...

  • Principal Data Engineer

    4 days ago

    Bengaluru, Karnataka, India K2 Partnering Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Title: Principal Data/App Engineer, CelonisWork Location: Bengaluru, Karnataka 560058Job Type: Full-time/Permanent with the Client directlyJob DetailsOur Client's Center for Process Excellence team is on a mission to improve business outcomes and employee experiences by driving step-change improvements in critical enterprise processes, and the technology...

  • Principal Data Engineer

    6 days ago

    Bengaluru, Karnataka, India Amadeus Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Job TitlePrincipal Data EngineerSummary of the RoleAs a Principal Data Engineer, you'll design and maintain scalable data systems that support analytics and decision-making. You'll collaborate with cross-functional teams and help shape our data capabilities while supporting others in their growth.In this role you'llDesign and maintain reliable, scalable data...

  • Principal Data Engineer

    6 days ago

    Bengaluru, Karnataka, India Kwalee Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Join the teamAs we continue to grow, we are now looking for a Principal Data Engineer to join our Data Engineering team.As a Principal Data Engineer, you will be responsible for designing, developing, and maintaining Kwalee's core data systems — including data pipelines, data warehouses, and data lakes. You'll ensure data is accessible, reliable, and...