Principal Data Engineer

1 week ago

bangalore, India Autodesk Full time

#25WD9141Position Overview We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform, which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data platform capabilities. You will partner closely with Engineering Managers, Architects, Product teams, and Partner Engineering to modernize our data lakehouse architecture, deliver highly reliable data pipelines, and establish technical excellence across ingestion, processing, and governance.ResponsibilitiesTechnical LeadershipDefine and drive the technical strategy for Batch Processing within ADPDesign and scale PySpark based distributed processing frameworks integrated with Airflow for orchestrationChampion Apache Iceberg-based data lakehouse adoption (versioning, branching, WAP, indexing)Introduce performance optimization patterns for data pipelines (e.g., partition pruning, caching, resource tuning)Establish patterns for metadata-driven, intent-based data processing aligned with AI-assisted pipelinesArchitecture & DeliveryPartner with architects to design multi-tenant, secure, and compliant (SOC2/GDPR/CCPA) batch processing servicesDefine reference architectures and reusable frameworks for cross-domain processing workloadsLead technical reviews, solution designs, and architectural forums across ADPGuide the evolution of Airflow APIs and SDKs to enable scalable pipeline self-serviceMentorship & CollaborationMentor senior engineers, staff engineers, and tech leads within the Batch Processing and adjacent teamsPartner with the Batch Ingestion and Data Security teams to deliver unified ingestion + processing flowsInnovation & ImpactDrive modernization initiatives to migrate away from legacy toolsPioneer AI-augmented data engineering practices (e.g., pipeline optimization agents, anomaly detection)Ensure scalability, cost-efficiency, and reliability for thousands of production pipelines across AutodeskInfluence company-wide data engineering strategy by contributing thought leadership and whitepapersMinimum Qualifications8+ years of experience in software/data engineering, with at least 3 years in big data platform leadership rolesExpert in distributed data processing (Spark, PySpark, Ray, Flink)Deep experience with workflow orchestration (Airflow, Dagster, Prefect)Strong hands-on expertise in Lakehouse technologies (Iceberg, Delta, Hudi) and cloud platforms (AWS/Azure/GCP)Proven track record in architecting secure, multi-tenant, and compliant data platformsSkilled in SQL/NoSQL databases, Snowflake, Glue, and modern metadata/catalog toolsStrong problem-solving, communication, and cross-geo collaboration skillsExperience mentoring engineers, building strong technical culture, and influencing at scalePreferred QualificationsExposure to AI/ML-driven data engineering (optimizers, anomaly detection, auto-scaling)Experience with data governance, lineage, and observability tools (Atlan, Databand, Collibra, etc.)Familiarity with streaming + batch hybrid architectures (Kappa/Lambda)#LI-KS2

  • Principal Data Engineer

    1 week ago

    bangalore, India Kwalee Full time

    Join the teamAs we continue to grow, we are now looking for a Principal Data Engineer to join our Data Engineering team.As a Principal Data Engineer, you will be responsible for designing, developing, and maintaining Kwalee's core data systems — including data pipelines, data warehouses, and data lakes. You'll ensure data is accessible, reliable, and...

  • Principal Data Engineer

    17 hours ago

    bangalore, India Commonwealth Bank of Australia Full time

    Organization: At CommBank, we never lose sight of the role we play in other people's financial wellbeing. Our focus is to help people and businesses move forward to progress. To make the right financial decisions and achieve their dreams, targets, and aspirations. Regardless of where you work within our organisation, your initiative, talent, ideas, and...

  • Principal Data Engineer

    1 week ago

    bangalore, India Amadeus Full time

    Job TitlePrincipal Data EngineerSummary of the RoleAs a Principal Data Engineer, you'll design and maintain scalable data systems that support analytics and decision-making. You'll collaborate with cross-functional teams and help shape our data capabilities while supporting others in their growth.In this role you'llDesign and maintain reliable, scalable data...

  • Principal Data Engineer

    5 days ago

    Bangalore, India Autodesk Full time

    #25WD9141 Position Overview We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform , which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in...

  • Principal Data Scientist

    16 hours ago

    bangalore, India HiLabs Full time

    Role: Sr. Principal Data Scientist / Principal Data ScientistLocation: Bangalore, IndiaCompany: HiLabsWe are a leading Software as a Service (SaaS) company that specializes in the transformation of data in the US healthcare industry through cutting-edge Artificial Intelligence (AI) solutions. We are seeking job Principal Data Scientist. As a Principal Data...

  • Principal Data Engineer

    1 day ago

    bangalore, India Tata Consultancy Services Full time

    Skill : Principal Data Engineer Exp range : 12 yrs to 25 yrs Job location : Bangalore PFB Job Description : Possesses hands-on technical experience working in AWS. The individual should possess a robust set of technical and soft skills and be an excellent AWS Data Engineer with a focus on complex Automation and Engineering Framework development. Being...

  • Principal Data Engineer

    2 weeks ago

    Bangalore, India MakeMyTrip Full time

    Role: Principal Data Engineer Location: Bangalore Years of Experience: 8-10 Years About Role: We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences. About us: Nurtured from the seed of a single great idea - to empower the traveler -...

  • Principal Data Engineer

    2 weeks ago

    bangalore, India MakeMyTrip Full time

    Role: Principal Data Engineer Location: Bangalore Years of Experience: 8-10 Years About Role: We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences. About us: Nurtured from the seed of a single great idea - to empower the traveler -...

  • Principal Data Engineer

    5 days ago

    Bangalore, India Autodesk Full time

    We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform , which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data...

  • Principal Engineer

    3 days ago

    bangalore, India Antal International Full time

    Principal Software Engineer – Core Platform (Scala / Java)We're partnering with a US-based AI and computer vision company that's transforming how people experience movement in the physical world — building technology that powers seamless, real-time transactions across mobility and infrastructure.The India engineering team is growing rapidly, and we're...