Principal Data Engineer

3 days ago

bangalore, India Autodesk Full time

#25WD9141Position Overview We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform, which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data platform capabilities. You will partner closely with Engineering Managers, Architects, Product teams, and Partner Engineering to modernize our data lakehouse architecture, deliver highly reliable data pipelines, and establish technical excellence across ingestion, processing, and governance.ResponsibilitiesTechnical LeadershipDefine and drive the technical strategy for Batch Processing within ADPDesign and scale PySpark based distributed processing frameworks integrated with Airflow for orchestrationChampion Apache Iceberg-based data lakehouse adoption (versioning, branching, WAP, indexing)Introduce performance optimization patterns for data pipelines (e.g., partition pruning, caching, resource tuning)Establish patterns for metadata-driven, intent-based data processing aligned with AI-assisted pipelinesArchitecture & DeliveryPartner with architects to design multi-tenant, secure, and compliant (SOC2/GDPR/CCPA) batch processing servicesDefine reference architectures and reusable frameworks for cross-domain processing workloadsLead technical reviews, solution designs, and architectural forums across ADPGuide the evolution of Airflow APIs and SDKs to enable scalable pipeline self-serviceMentorship & CollaborationMentor senior engineers, staff engineers, and tech leads within the Batch Processing and adjacent teamsPartner with the Batch Ingestion and Data Security teams to deliver unified ingestion + processing flowsInnovation & ImpactDrive modernization initiatives to migrate away from legacy toolsPioneer AI-augmented data engineering practices (e.g., pipeline optimization agents, anomaly detection)Ensure scalability, cost-efficiency, and reliability for thousands of production pipelines across AutodeskInfluence company-wide data engineering strategy by contributing thought leadership and whitepapersMinimum Qualifications8+ years of experience in software/data engineering, with at least 3 years in big data platform leadership rolesExpert in distributed data processing (Spark, PySpark, Ray, Flink)Deep experience with workflow orchestration (Airflow, Dagster, Prefect)Strong hands-on expertise in Lakehouse technologies (Iceberg, Delta, Hudi) and cloud platforms (AWS/Azure/GCP)Proven track record in architecting secure, multi-tenant, and compliant data platformsSkilled in SQL/NoSQL databases, Snowflake, Glue, and modern metadata/catalog toolsStrong problem-solving, communication, and cross-geo collaboration skillsExperience mentoring engineers, building strong technical culture, and influencing at scalePreferred QualificationsExposure to AI/ML-driven data engineering (optimizers, anomaly detection, auto-scaling)Experience with data governance, lineage, and observability tools (Atlan, Databand, Collibra, etc.)Familiarity with streaming + batch hybrid architectures (Kappa/Lambda)#LI-KS2

  • Principal Engineer – Cloud

    5 days ago

    bangalore, India Bean HR Consulting Full time

    Job Title: Principal Engineer – Cloud & DataLocation: Bangalore (BLR) Experience: 12–16 yearsRole Overview:Lead the design and development of cloud-based applications. Architect scalable, secure, and high-performance systems, mentor the engineering team, and collaborate with product and architecture teams to deliver high-quality solutions.Key...

  • Principal Data Engineer

    2 days ago

    Bangalore, India Autodesk Full time

    #25WD9141 Position Overview We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform , which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in...

  • Principal Data Engineer

    2 days ago

    Bangalore, India Autodesk Full time

    We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform , which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data...

  • Principal Data Engineer

    2 days ago

    bangalore district, India Autodesk Full time

    #25WD9141 Position Overview We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform , which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in...

  • Principal Data Engineer

    10 hours ago

    Bangalore - Manyata Tech Park Road, India Commonwealth Bank Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    Organization: At CommBank, we never lose sight of the role we play in other people's financial wellbeing. Our focus is to help people and businesses move forward to progress. To make the right financial decisions and achieve their dreams, targets, and aspirations. Regardless of where you work within our organisation, your initiative, talent, ideas, and...

  • Principal Engineer

    2 days ago

    bangalore, India Coffeee.io Full time

    Role- principal EngineerLocation- Bangalore (Hybrid)Experience- 10-12 yearsEmployment- Full timeAbout the RoleWe are looking for a highly skilled Principal Engineer with strong technical expertise andproven experience in engineering leadership. This role is ideal for someone who enjoys beinghands-on with coding while also managing and mentoring a small team....

  • Principal Engineer

    2 days ago

    bangalore, India Coffeee.io Full time

    Role- principal Engineer Location- Bangalore (Hybrid) Experience- 10-12 years Employment- Full time About the Role We are looking for a highly skilled Principal Engineer with strong technical expertise and proven experience in engineering leadership. This role is ideal for someone who enjoys being hands-on with coding while also managing and mentoring a...

  • Principal engineer

    4 weeks ago

    Bangalore, India CoffeeBeans Full time

    About Us Coffee Beans Consulting is a tech consulting firm focused on making organizations AI-ready by structuring their data efficiently across various sources and enabling AI-driven solutions. We specialize in data architecture, pipelines, governance, MLOps, and Gen AI solutions, helping clients achieve faster go-to-market and cost efficiency . We...

  • Principal Machine Learning Engineer

    2 weeks ago

    bangalore, India Informatica Full time

    Principal Machine Learning EngineerInformatica is on a journey to leverage generative AI to simplify cloud data management. Principal Machine Learning Engineer will be responsible for driving the overall architecture and pipelines to enable Machine Learning engineers to build, train and deploy Models at scale across multiple cloud services providers in a...

  • Principal Machine Learning Engineer

    1 week ago

    bangalore, India Informatica Full time

    Principal Machine Learning Engineer Informatica is on a journey to leverage generative AI to simplify cloud data management. Principal Machine Learning Engineer will be responsible for driving the overall architecture and pipelines to enable Machine Learning engineers to build, train and deploy Models at scale across multiple cloud services providers in a...