Senior Data Engineer

4 weeks ago

Bangalore, Karnataka, India Scaling Theory Full time

As a Senior Data Engineer, you will partner with Software Engineers, Data Scientists, and Business Intelligence Engineers. You will gain a deep understanding of our services and the data they produce and become our resident expert in how to transform that data into a format that is useful for analytics and business intelligence. You will proactively help to identify new data for integration with the platform and propose and implement new technologies to help us better understand our data.

Responsibilities:


- Collaborate with cross-functional teams including data scientists, analysts, and software engineers to understand data requirements and design optimal data solutions on AWS.


- Design, develop, and maintain data pipelines and ETL processes to move data from various sources into our data lake and data warehouse on AWS.


- Implement efficient and reliable data transformation, cleansing, and enrichment processes to ensure high-quality data for downstream analytics and reporting.


- Optimize data pipelines for performance, scalability, and cost-effectiveness using AWS services such as Amazon Redshift, Amazon S3 AWS Glue, AWS Lambda, etc.


- Ensure data security and compliance by implementing appropriate data access controls, encryption mechanisms, and auditing processes.


- Monitor and troubleshoot data pipeline issues, performance bottlenecks, and data quality problems, and implement solutions to address them.


- Design and implement product features in collaboration with business and Technology stakeholders. Anticipate, identify, and solve issues concerning data management to improve data quality.


- Design and build reusable components, frameworks, and libraries at scale to support analytics products. Clean, prepare, and optimize data at scale for ingestion and consumption.


- Clean, prepare, and optimize data at scale for ingestion and consumption. Drive the implementation of new data management projects and re-structure the current data architecture.


- Implement complex automated workflows and routines using workflow scheduling tools.


- Build continuous integration, test-driven development, and production deployment frameworks.


- Drive collaborative reviews of design, code, test plans, and dataset implementation performed by other data engineers in support of maintaining data engineering standards.


- Analyze and profile data to design scalable solutions.


- Troubleshoot complex data issues and perform root cause analysis to proactively resolve product and operational issues.


- Mentor and develop other data engineers in adopting best practices.


- Build different types of data warehousing layers based on specific use cases.


- Lead the design, implementation, and successful delivery of large-scale, critical, or difficult data solutions involving a significant amount of work.


- Build scalable data infrastructure and understand distributed systems concepts from a data storage and computing


- Experience working with distributed technology tools for developing Batch and Streaming pipelines using: SQL, Spark, PySpark [4+ years], Airflow [3+ years],


- Able to write code that is optimized for performance. Experience in Cloud platforms, e. g., AWS/ GCP/ Azure.


- Experience with creating/ configuring Jenkins pipeline for smooth CI/CD process for Managed Spark jobs, building Docker images, etc.


- Working knowledge of Data warehousing, Data modeling, Governance, and Data Architecture.


- Experience working with Data platforms, including EMR, Airflow, Databricks (Data Engineering and Delta Lake components, and Lakehouse Medallion architecture), etc.


- Experience working in Agile and Scrum development processes.


- Experience in EMR/ EC2 Databricks etc.


- Experience working with Data warehousing tools, including SQL database, Presto, and Snowflake.


- Experience architecting data products in Streaming, Serverless, and Microservices Architecture and platforms.


- Excellent problem-solving and analytical skills.


- Able to quickly pick up new programming languages, technologies, and frameworks.


- Strong communication and collaboration skills.


- Strong skills in building positive relationships across Product and Engineering.


- Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders.


- Passionate for data solutions. Self-motivated and able to work in a fast-paced environment.


- Utilize expertise in SQL and have a strong understanding of ETL and data modeling.


- Ensure the accuracy and availability of data to customers and understand how technical decisions can impact their business's analytics and reporting.

(ref:hirist.tech)

  • Senior Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India Squareroot Consulting Pvt Ltd Full time

    We are seeking a highly skilled and experienced Senior Data Engineer to join our team.Location : Bangalore (Work from Office all 5 days)Experience : 6+ YearsRole : Senior Data EngineerThe ideal candidate will have a strong background in big data technologies and extensive hands-on experience with Cloudera, Spark, Hive, HDFS, Flink, Data Lake, and...

  • Senior Data Engineer

    2 months ago

    Bangalore, Karnataka, India Squareroot Consulting Pvt Ltd Full time

    We are seeking a highly skilled and experienced Senior Data Engineer to join our team.Location : Bangalore (Work from Office all 5 days)Experience : 6+ YearsRole : Senior Data EngineerThe ideal candidate will have a strong background in big data technologies and extensive hands-on experience with Cloudera, Spark, Hive, HDFS, Flink, Data Lake, and...

  • Senior Engineer

    2 weeks ago

    Bangalore, Karnataka, India Prudential Manpower Pvt.lTD Full time

    Immediately hiring for our client for the role of Senior Engineer Data & Analytics. Total Exp : 8+ years. Location: Anywhere in India. What you'll do:As the senior manager of the data engineering team, you will be entrusted with the delivery of scalable services that empower our customers to maximize return on brand outreach and recruitment.- Data...

  • Senior Engineer

    2 months ago

    Bangalore, Karnataka, India Prudential Manpower Pvt.lTD Full time

    Immediately hiring for our client for the role of Senior Engineer Data & Analytics. Total Exp : 8+ years. Location: Anywhere in India. What you'll do:As the senior manager of the data engineering team, you will be entrusted with the delivery of scalable services that empower our customers to maximize return on brand outreach and recruitment.- Data...

  • Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India Magna Hire Full time

    Responsibilities :Support the data engineering team in implementing analytics solutions, ensuring adherence to project timelines and quality standards.Participate in at least one full implementation lifecycle of dbt (data build tool), contributing to design, development, and deployment phases.Assist senior team members in optimizing dbt models and...

  • Data Engineer

    2 months ago

    Bangalore, Karnataka, India Magna Hire Full time

    Responsibilities :Support the data engineering team in implementing analytics solutions, ensuring adherence to project timelines and quality standards.Participate in at least one full implementation lifecycle of dbt (data build tool), contributing to design, development, and deployment phases.Assist senior team members in optimizing dbt models and...

  • Senior Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India TecHHire Global Full time

    Job Description :We are looking for a Senior Data engineer-GCP who can design and build production data pipelines from ingestion to consumption within a big data architecture using Python, Scala, PySpark, Java implementing data engineering, ingestion, and curation functions on GCP :Must be able to write quality code and build secure, highly available...

  • Senior Data Engineer

    1 month ago

    Bangalore, Karnataka, India TecHHire Global Full time

    Job Description :We are looking for a Senior Data engineer-GCP who can design and build production data pipelines from ingestion to consumption within a big data architecture using Python, Scala, PySpark, Java implementing data engineering, ingestion, and curation functions on GCP :Must be able to write quality code and build secure, highly available...

  • Senior Director

    2 weeks ago

    Bangalore, Karnataka, India Recruise India Consulting Full time

    Role : Senior Director - Data Engineering & AnalyticsResponsibilities :- Provide strategic leadership and vision for the data engineering and analytics function, aligning it with the overall business goals.- Lead and manage a diverse team of data engineers, data architects, data scientists, and cloud operations specialists.- Foster a culture of innovation,...

  • Senior Director

    1 month ago

    Bangalore, Karnataka, India Recruise India Consulting Full time

    Role : Senior Director - Data Engineering & AnalyticsResponsibilities :- Provide strategic leadership and vision for the data engineering and analytics function, aligning it with the overall business goals.- Lead and manage a diverse team of data engineers, data architects, data scientists, and cloud operations specialists.- Foster a culture of innovation,...

  • Senior Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India Infimetrics Info Solutions Full time

    Position : Senior Java (Data Engineer). Reports to : Technical Lead. Responsible for : Contributing to software development as per business requirements.Purpose of Role :- We are a data driven asset management firm with a very wide variety of datasets with varying frequency of updates that essentially drive our research models which ultimately determine the...

  • Senior Data Engineer

    2 months ago

    Bangalore, Karnataka, India Infimetrics Info Solutions Full time

    Position : Senior Java (Data Engineer). Reports to : Technical Lead. Responsible for : Contributing to software development as per business requirements.Purpose of Role :- We are a data driven asset management firm with a very wide variety of datasets with varying frequency of updates that essentially drive our research models which ultimately determine the...

  • Senior Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India SPN Globe Full time

    Must Have Skills : Spark streaming, Kafka, AWS, Python, Data Engineering, PySpark, CI/CD, Kubernetes, Docker, Dynamo DB.Job Description :1) Proven experience as a Senior Data Engineer.2) A well-rounded engineer with good appetite for learning and ramping up on cutting edge technologies.3) Good analytical and debugging skills with self-starting capabilities...

  • Senior Data Engineer

    2 months ago

    Bangalore, Karnataka, India SPN Globe Full time

    Must Have Skills : Spark streaming, Kafka, AWS, Python, Data Engineering, PySpark, CI/CD, Kubernetes, Docker, Dynamo DB.Job Description :1) Proven experience as a Senior Data Engineer.2) A well-rounded engineer with good appetite for learning and ramping up on cutting edge technologies.3) Good analytical and debugging skills with self-starting capabilities...

  • Senior Data Engineer

    2 weeks ago

    Bangalore/Hyderabad, Karnataka, India Sam manpower services & Career LLP Full time

    As a Senior Data Engineer, you will play a critical role in our organization by leading the design, development, and maintenance of our data pipeline infrastructure. You will work closely with cross-functional teams to ensure seamless data integration and accessibility for analytical and operational purposes.Key responsibilities:- Design, construct, and...

  • Senior Data Engineer

    2 months ago

    Bangalore/Hyderabad, Karnataka, India Sam manpower services & Career LLP Full time

    As a Senior Data Engineer, you will play a critical role in our organization by leading the design, development, and maintenance of our data pipeline infrastructure. You will work closely with cross-functional teams to ensure seamless data integration and accessibility for analytical and operational purposes.Key responsibilities:- Design, construct, and...

  • Senior Lead

    2 weeks ago

    Bangalore, Karnataka, India Peepal Technology and Management Consultant Full time

    Key Responsibilities :Data Engineer Senior Lead role is to coordinate, and execute all activities related to the requirements interpretation, design, and implementation of Data Analytics applications. This individual will apply proven industry and technology experience as well as communication skills, problem-solving skills, and knowledge of best practices...

  • Senior Lead

    2 weeks ago

    Bangalore, Karnataka, India Peepal Technology and Management Consultant Full time

    Key Responsibilities :Data Engineer Senior Lead role is to coordinate, and execute all activities related to the requirements interpretation, design, and implementation of Data Analytics applications. This individual will apply proven industry and technology experience as well as communication skills, problem-solving skills, and knowledge of best practices...

  • Senior Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India Web Idea Solution LLP Full time

    Role : Senior Data Engineer. Experience : 4-6 years. To be onboarded by : Immediate. Location : Bangalore. Type : Onsite/Full-Time. Responsibilities :- Design and build reusable components, frameworks and libraries at scale to support analytics products.- Design and implement product features in collaboration with business and Technology stakeholders.-...

  • Senior Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India Web Idea Solution LLP Full time

    Role : Senior Data Engineer. Experience : 4-6 years. To be onboarded by : Immediate. Location : Bangalore. Type : Onsite/Full-Time. Responsibilities :- Design and build reusable components, frameworks and libraries at scale to support analytics products.- Design and implement product features in collaboration with business and Technology stakeholders.-...