Big Data Engineer

Description :

Position : Big Data Engineer

Location : Hyderabad, India (Hybrid, 2-3 times per week on any weekdays)

Duration : 1 year

Job Summary :

Experian FSD is seeking a skilled and motivated Big Data Engineer to join our dynamic data engineering team. This hybrid role based in Hyderabad offers the opportunity to work on cutting-edge data platforms and services that power Experians global data ecosystem. The engineer will be instrumental in designing and building scalable data pipelines, optimizing big data solutions on AWS using Spark/PySpark, and developing robust RESTful API services using Scala/Java. Success in this role requires strong expertise in AWS Big Data services (EMR, EC2, ECS) and a commitment to data quality and performance.

Job Description :

Big Data Pipeline Development and Optimization :

- Design, develop, and maintain highly scalable and fault-tolerant ETL/ELT data processing pipelines leveraging Apache Spark and PySpark on the AWS platform.

- Optimize Spark jobs and clusters for performance tuning, resource utilization, and cost efficiency, specifically within AWS EMR and custom compute environments.

- Implement and manage data storage solutions on Hadoop Distributed File System (HDFS) and cloud storage services like Amazon S3 for data ingestion, processing, and consumption.

- Utilize Scala or Java to write and optimize core data processing logic, complex transformations, and highly performant backend services.

- Ensure strict data quality, governance, and validation checks are integrated into all data pipelines to maintain accuracy and reliability of the data ecosystem.

Cloud Services and API Engineering :

- Deploy, manage, and scale data applications and services using various AWS compute resources, including Amazon EC2 instances, AWS EMR clusters, and containerized environments via Amazon ECS.

- Design and develop robust, high-performance RESTful APIs and microservices using Scala/Java to facilitate real-time data access and transactional services.

- Securely expose data and service endpoints by implementing and managing resources within AWS API Gateway, including authentication, throttling, and request/response mapping.

- Implement Infrastructure-as-Code (IaC) principles where applicable to manage and provision data-related AWS resources efficiently and repeatably.

- Collaborate with security and infrastructure teams to ensure all API and data services adhere to strict compliance and security standards, including encryption and access control.

Collaboration, Testing, and Operations :

- Collaborate with cross-functional teams, including product managers and data scientists, to understand complex data requirements and translate them into efficient, well-architected technical solutions.

- Participate actively in comprehensive code reviews, ensuring adherence to coding standards, performance requirements, and best practices for large-scale distributed systems.

- Configure and maintain continuous integration and continuous deployment (CI/CD) pipelines and version control workflows using tools like Git for automated testing and deployment.

- Troubleshoot and resolve complex production issues and performance bottlenecks across the big data ecosystem and associated API services.

Required Skills & Qualifications :

- Experience : Mandatory 35 years of experience in the field of Big Data Engineering.

- Technical Proficiency : Strong hands-on experience with Spark and PySpark, and a foundational understanding of the Hadoop ecosystem.

- Programming : Mandatory proficiency in Scala or Java for backend development, API creation, and core processing logic.

- AWS Services : Proven experience with AWS EMR, Amazon EC2, Amazon ECS, and configuring services within AWS API Gateway.

- API Development : Solid technical understanding of RESTful API design principles, development, and large-scale service integration.

- DevOps Tools : Familiarity with CI/CD pipelines and expert-level use of Git for collaborative development and version control.

- Core Skills : Excellent analytical capabilities, strong problem-solving skills, and clear communication abilities for working in a hybrid, cross-functional environment.

Preferred Skills :

- Containerization & Orchestration : Direct experience with Docker for containerizing applications and knowledge of Kubernetes for container orchestration and scaling.

- Data Architecture : Exposure to modern Data Lakehouse architectures (e.g., Delta Lake, Apache Hudi) and experience with real-time data processing technologies (Kafka, Kinesis, Spark Streaming).

- Cloud Certification : Current AWS Certification (e.g., AWS Certified Data Analytics Specialty or AWS Certified Data Engineer Associate).

- Database Knowledge : Experience with both traditional relational databases (SQL) and NoSQL databases (e.g., DynamoDB, MongoDB) for diverse data storage needs.

- Security & Compliance : Practical experience implementing data security best practices, access controls (IAM), and compliance requirements within cloud environments.