▷ (Urgent) Data Engineer- Databricks

Quay Building 8th Floor Bagmane Tech Park Bengaluru IN Line Of Business Data Estate DE Job Category Engineering Technology Experience Level Experienced HireAt Moody s we unite the brightest minds to turn todays risks into tomorrows opportunities We do this by striving to create an inclusive environment where everyone feels welcome to be who they are-with the freedom to exchange ideas think innovatively and listen to each other and customers in meaningful ways If you are excited about this opportunity but do not meet every single requirement please apply You still may be a great fit for this role or other open roles We are seeking candidates who model our values invest in every relationship lead with curiosity champion diverse perspectives turn inputs into actions and uphold trust through integrity Job Summary We are seeking a highly experienced and motivated Senior Data Engineer to join our dynamic team The ideal candidate possesses a strong software engineering background and deep expertise in designing building optimizing and maintaining scalable data pipelines and infrastructure You will leverage your extensive experience with Apache Spark Apache Kafka and various big data technologies to process and manage large datasets effectively Working within an Agile Scrum environment you will take ownership of complex tasks delivering high-quality well-tested solutions independently Responsibilities Design develop implement and maintain robust scalable and efficient batch and real-time data pipelines using Apache Spark Python PySpark and Scala and Apache Kafka Work extensively with large complex datasets residing in various storage systems e g data lakes data warehouses distributed file systems Build and manage real-time data streaming solutions to ingest process and serve data with low latency using Apache Kafka Optimize data processing jobs and data storage solutions for performance scalability and cost-effectiveness within big data ecosystems Implement comprehensive automated testing unit integration end-to-end to ensure data quality pipeline reliability and code robustness Collaborate closely with data scientists analysts software engineers and product managers to understand data requirements and deliver effective solutions Actively participate in Agile Scrum ceremonies including sprint planning daily stand-ups sprint reviews and retrospectives Take ownership of assigned tasks and projects driving them to completion independently while adhering to deadlines and quality standards Troubleshoot and resolve complex issues related to data pipelines platforms and performance Contribute to the evolution of our data architecture standards and best practices Mentor junior engineers and share knowledge within the team Document technical designs processes and implementation details Required Qualifications Bachelor s or Master s degree in Computer Science Engineering Information Technology or a related field or equivalent practical experience 10 years of professional software engineering experience with a proven track record of building complex scalable systems Significant hands-on experience typically 5 years specifically in data engineering roles Expert-level proficiency in designing and implementing data processing solutions using Apache Spark with strong skills in both Python PySpark and Scala Demonstrable experience building deploying and managing data streaming pipelines using Apache Kafka and its ecosystem e g Kafka Connect Kafka Streams Solid understanding and practical experience working with big data technologies and concepts e g Hadoop ecosystem - HDFS Hive distributed computing partitioning file formats like Parquet Avro Proven experience working effectively in an Agile Scrum development environment participating in sprints and related ceremonies Demonstrated ability to work independently manage priorities and deliver end-to-end solutions with a strong focus on automated testing and quality assurance Excellent problem-solving debugging and analytical skills Strong communication and interpersonal skills Preferred Qualifications Experience with cloud-based data platforms and services e g AWS EMR S3 Kinesis MSK Azure Databricks ADLS AWS Glue Experience with workflow orchestration tools e g Airflow Dagster Prefect Experience with containerization technologies Docker and orchestration Kubernetes Familiarity with data warehousing solutions e g Snowflake Redshift BigQuery Experience with Infrastructure as Code IaC tools like Terraform or CloudFormation Knowledge of CI CD practices and tools e g Jenkins GitLab CI GitHub Actions applied to data pipelines Experience with data modeling and database design SQL and NoSQL Moodys is an equal opportunity employer All qualified applicants will receive consideration for employment without regard to race color religion sex national origin disability protected veteran status sexual orientation gender expression gender identity or any other characteristic protected by law Candidates for Moody s Corporation may be asked to disclose securities holdings pursuant to Moodys Policy for Securities Trading and the requirements of the position Employment is contingent upon compliance with the Policy including remediation of positions in those holdings as necessary For more information on the Securities Trading Program please refer to the onPlease note STP categories are assigned by the hiring teams and are subject to change over the course of an employees tenure with Moodys