Senior Data Engineer/Lead Data Engineer

Job Description :

The primary purpose of this role is to translate business requirements and functional specifications into logical program designs and to deliver modules, stable application systems, and Data or Platform solutions. This includes developing, configuring, or modifying complex integrated business and/or enterprise infrastructure or application solutions within various computing environments. This role facilitates the implementation and maintenance of complex business and enterprise Data or Platform solutions to ensure successful deployment of released applications.

As a "Senior Data Engineer "/ "Lead Data Engineer " with our client you will be part of a fast-paced team designing, developing, testing, integrating, and supporting technically innovative solutions for our Fortune 500 customers. You will leverage your wide range of experiences, developed professional concepts as well as understanding of the industry, customer, and company objectives to resolve complex issues in creative ways.

In a nutshell, we are looking for an incumbent expertise in :

- Build different types of data warehousing layers based on specific use cases.

- Lead the design, implementation, and successful delivery of large-scale, critical, or difficult data solutions involving a significant amount of work.

- Build scalable data infrastructure and understand distributed systems concepts from a data storage and compute perspective.

- Utilize expertise in SQL and have a strong understanding of ETL and data modelling.

- Ensure the accuracy and availability of data to customers and understand how technical decisions can impact their business's analytics and reporting. s to Companies in Retail, CPG, Manufacturing & Healthcare

Head Quarter: New York

R&D Lab: Poland

ODC: Bengaluru Structured approach to define & deliver customer success

500K: Model Instances Tuned & Trained

8 Business Models Productionized

10 Use Cases across Retail & CPG

Capabilities :

Clearly defined interfaces between DS & ML engineering teams

Pre-built tools & frameworks for faster build & productionization

75% Code reuse with open & custom frameworks for faster delivery throughput

Key Resposibilities :

As a Senior Data Engineer / Lead data Engineer , you will partner with Software Engineers, Data Scientists and Business Intelligence Engineers. You will gain a deep understanding of our services and the data they produce and become our resident expert in how to transform that data into a format that is useful for analytics and business intelligence. You will proactively help to identify new data for integration with the platform and propose and implement new technologies to help us better understand our data.

- Collaborate with cross-functional teams including data scientists, analysts, and software engineers to understand data requirements and design optimal data solutions on AWS.

- Design, develop, and maintain data pipelines and ETL processes to move data from various sources into our data lake and data warehouse on AWS. Implement efficient and reliable data transformation, cleansing, and enrichment processes to ensure high-quality data for downstream analytics and reporting.

- Optimize data pipelines for performance, scalability, and cost-effectiveness using AWS services such as Amazon Redshift, Amazon S3, AWS Glue, AWS Lambda, etc.

- Ensure data security and compliance by implementing appropriate data access controls, encryption mechanisms, and auditing processes. Monitor and troubleshoot data pipeline issues, performance bottlenecks, and data quality problems, and implement solutions to address them.

- Design and implement product features in collaboration with business and Technology stakeholders. Anticipate, identify, and solve issues concerning data management to improve data quality.

- Design and build reusable components, frameworks, and libraries at scale to support analytics products. Clean, prepare and optimize data at scale for ingestion and consumption.

- Clean, prepare and optimize data at scale for ingestion and consumption. Drive the implementation of new data management projects and re-structure of the current data architecture.

- Implement complex automated workflows and routines using workflow scheduling tools. Build continuous integration, test-driven development, and production deployment frameworks.

- Drive collaborative reviews of design, code, test plans and dataset implementation performed by other data engineers in support of maintaining data engineering standards. Analyse and profile data for the purpose of designing scalable solutions.

- Troubleshoot complex data issues and perform root cause analysis to proactively resolve product and operational issues. Mentor and develop other data engineers in adopting best practices.

Attributes & Competencies :

- Education: - BE / BTECH | MTech | MCA

- Minimum 4 years for Senior Data Engineer & Min 7 Yrs for Lead Data Engineer of work experience in development / Migration project.

- Experience working with distributed technology tools for developing Batch and Streaming pipelines using. - SQL, Spark, Python

- Airflow

- Scala

- Experience in Cloud Computing, e.g., AWS, GCP, Azure, etc.

- Able to quickly pick up new programming languages, technologies, and frameworks.

- Strong skills building positive relationships across Product and Engineering.

- Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

- Experience with creating/ configuring Jenkins pipeline for smooth CI/CD process for Managed Spark jobs, build Docker images, etc.

- Working knowledge of Data warehousing, Data modelling, Governance and Data Architecture

Mandatory Skill Set :

- Python / Pyspark / Spark

- Databricks / EMR

- SQL

- AWS/ Azure

- Kafka

- Data Modelling

- Performance Tuning

- GIT / Jenkins

Good to have :

- Experience working with Data platforms, including EMR, Airflow, Data bricks (Data Engineering & Delta Lake components, and Lakehouse Medallion architecture), etc.

- Experience working in Agile and Scrum development process.

- Experience in EMR/ EC2, Data bricks etc.

- Experience working with Data warehousing tools, including SQL database, Presto, and Snowflake

- Experience architecting data product in Streaming, Server less and Microservices Architecture and platform.

Interested candidate, who has same above expertises, ready for Bangalore location, who can be on board within 30 days max, those candidates can share their updated resume along with below details:- -

Total Exps:-

Min Exps as a Data Engineer:-

Rel Exps in Python / Pyspark / Spark :-

Rel Exps in Databricks / EMR :-

Rel Exps in SQL :-

Rel Exps in AWS/ Azure :-

Rel Exps in Kafka :-

Rel Exps in Data Modelling :-

Rel Exps in Performance Tuning :-

Rel Exps in GIT / Jenkins :-

Current Loc :-

Pref Loc : B'lore-Yes/No

Reason for looking for change:-

Your LinkedIn URL:-

Current CTC:-

Expected CTC :-

Notice Period :-

Any Offer in hand:-

If Yes, what is the offered CTC & DOJ:-

Looking forward to hear from you soon