Big Data Developer

Job profiles - Big Data Developer (Senior Manager)Experience - 7-10 yrsLocation - PuneJob DescriptionWe are seeking a highly skilled Lead Big Data Developer to architect, drive, and optimize large-scale data engineering solutions in a modern, modular, and highly automated environment. The ideal candidate will have extensive experience in building and leading end-to-end big data pipelines, with deep expertise in distributed systems, data transformation frameworks (such as DBT), and advanced data engineering best practices. Key ResponsibilitiesTechnical Leadership:Lead the design, development, and deployment of big data solutions using Java, Apache Spark, and modern orchestration tools.Architect and optimize data workflows, ensuring modularity, scalability, and maintainability across multiple domains and business units.Guide the adoption and integration of low-code transformation platforms (e.g., DBT) and ensure best practices in SQL-based transformation logic.Should be able to debug and fine tune workload , processing billions of recordsSolution Delivery & Collaboration:Oversee the development and management of APIs for data ingestion, transformation, and consumption, ensuring seamless integration with upstream and downstream systems.Collaborate with product managers, data scientists, analysts, and platform teams to translate business requirements into technical deliverables.Drive the implementation of metadata-first strategies, ensuring comprehensive cataloging, lineage, and data quality across the platform.Quality, Observability & Governance:Implement and enforce data engineering standards for quality, integrity, and security.Leverage observability suites for active and passive data validations, anomaly detection, and performance monitoring.Ensure robust governance by supporting RBAC/ABAC controls and compliance requirements.Mentorship & Team Development:Mentor and coach a team of big data engineers, fostering a culture of innovation, continuous improvement, and technical excellence.Conduct code reviews, knowledge sharing sessions, and technical upskilling initiatives.Continuous Improvement:Stay abreast of emerging trends in big data, cloud-native technologies, MLOps, and GenAI integration.Drive proof-of-concept (POC) initiatives and evaluate new tools and frameworks for adoption.Required Qualifications:Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field.8+ years of experience in big data engineering, with at least 2 years in a technical leadership or lead developer role.Advanced proficiency in Java, Apache Spark, and distributed data processing frameworks.Hands-on experience with data transformation tools such as DBT, and orchestration platforms (Airflow, Nifi, etc.).Strong expertise in API development, integration, and performance tuning.Deep understanding of data engineering principles, data mesh concepts, and multi-domain data architectures.Proficiency with Hadoop ecosystem tools (HDFS, Hive, HBase, Kafka, etc.).Experience with SQL and NoSQL databases.Strong problem-solving, analytical, and communication skills.Preferred QualificationsExperience with cloud platforms (AWS, Azure, GCP), Spark/Delta Lake, and modern data lake architectures.Knowledge of containerization (Docker, Kubernetes) and CI/CD pipelines.Familiarity with MLOps, GenAI integration, and machine learning frameworks.Experience in implementing metadata management, data lineage, and observability solutions.Exposure to federated governance, RBAC/ABAC, and security best practices.BenefitsCompetitive salary and comprehensive benefits package.Opportunity to lead and shape the next-generation data engineering platform in a highly collaborative environment.Access to cutting-edge technologies and continuous professional development.Work with a diverse, talented team on impactful, large-scale data initiatives.If you are passionate about leading technical teams, architecting scalable data solutions, and driving innovation in a fast-paced environment, we invite you to join us as our Lead Big Data Developer.