Senior Data Engineer

2 weeks ago

Bengaluru, Karnataka, India Scaling Theory Full time

As a Senior Data Engineer, you will partner with Software Engineers, Data Scientists, and Business Intelligence Engineers. You will gain a deep understanding of our services and the data they produce and become our resident expert in how to transform that data into a format that is useful for analytics and business intelligence. You will proactively help to identify new data for integration with the platform and propose and implement new technologies to help us better understand our data.

Responsibilities:

Collaborate with cross-functional teams including data scientists, analysts, and software engineers to understand data requirements and design optimal data solutions on AWS.

Design, develop, and maintain data pipelines and ETL processes to move data from various sources into our data lake and data warehouse on AWS.

Implement efficient and reliable data transformation, cleansing, and enrichment processes to ensure high-quality data for downstream analytics and reporting.

Optimize data pipelines for performance, scalability, and cost-effectiveness using AWS services such as Amazon Redshift, Amazon S3 AWS Glue, AWS Lambda, etc.

Ensure data security and compliance by implementing appropriate data access controls, encryption mechanisms, and auditing processes.

Monitor and troubleshoot data pipeline issues, performance bottlenecks, and data quality problems, and implement solutions to address them.

Design and implement product features in collaboration with business and Technology stakeholders. Anticipate, identify, and solve issues concerning data management to improve data quality.

Design and build reusable components, frameworks, and libraries at scale to support analytics products. Clean, prepare, and optimize data at scale for ingestion and consumption.

Clean, prepare, and optimize data at scale for ingestion and consumption. Drive the implementation of new data management projects and re-structure the current data architecture.

Implement complex automated workflows and routines using workflow scheduling tools.

Build continuous integration, test-driven development, and production deployment frameworks.

Drive collaborative reviews of design, code, test plans, and dataset implementation performed by other data engineers in support of maintaining data engineering standards.

Analyze and profile data to design scalable solutions.

Troubleshoot complex data issues and perform root cause analysis to proactively resolve product and operational issues.

Mentor and develop other data engineers in adopting best practices.

Build different types of data warehousing layers based on specific use cases.

Lead the design, implementation, and successful delivery of large-scale, critical, or difficult data solutions involving a significant amount of work.

Build scalable data infrastructure and understand distributed systems concepts from a data storage and computing

Experience working with distributed technology tools for developing Batch and Streaming pipelines using: SQL, Spark, PySpark (4+ years), Airflow (3+ years),

Able to write code that is optimized for performance. Experience in Cloud platforms, e. g., AWS/ GCP/ Azure.

Experience with creating/ configuring Jenkins pipeline for smooth CI/CD process for Managed Spark jobs, building Docker images, etc.

Working knowledge of Data warehousing, Data modeling, Governance, and Data Architecture.

Experience working with Data platforms, including EMR, Airflow, Databricks (Data Engineering and Delta Lake components, and Lakehouse Medallion architecture), etc.

Experience working in Agile and Scrum development processes.

Experience in EMR/ EC2 Databricks etc.

Experience working with Data warehousing tools, including SQL database, Presto, and Snowflake.

Experience architecting data products in Streaming, Serverless, and Microservices Architecture and platforms.

Excellent problem-solving and analytical skills.

Able to quickly pick up new programming languages, technologies, and frameworks.

Strong communication and collaboration skills.

Strong skills in building positive relationships across Product and Engineering.

Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders.

Passionate for data solutions. Self-motivated and able to work in a fast-paced environment.

Utilize expertise in SQL and have a strong understanding of ETL and data modeling.

Ensure the accuracy and availability of data to customers and understand how technical decisions can impact their business's analytics and reporting.

)

  • Senior Mechanical Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Abha Engineer Full time

    We are looking for a Senior Mechanical EngineerRoles are described below.Manpower Planning.Preparing of Project Cost.Schedule wise work execution.As Drawing & quality work execution.Client & Third Party Manage.Working Team Manage & Review.Reporting to Management.ROB & FOB Fabrication & Erection Work Knowledge.

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Description Lead Data Engineering Project Team, Expertise in Databricks, Expertise in Data Management & Data Governance,Expertise in Azure Cloud and its services, Experienced in Data Modelling and Data Pipelines,Experienced in Healthcare projects, Experience implementing FHIR Data model and FHIR Server Apply from local Career site Listen to the story...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Description Lead Data Engineering Project Team, Expertise in Databricks,Expertise in Data Management & Data Governance,Expertise in Azure Cloud and its services, Experienced in Data Modelling and Data Pipelines,Experienced in Healthcare projects, Experience implementing FHIR Data model and FHIR Server Apply from local Career site Listen to the story of...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Services Full time

    Lead Data Engineering Project Team, Expertise in Databricks, Expertise in Data Management & Data Governance,Expertise in Azure Cloud and its services, Experienced in Data Modelling and Data Pipelines,Experienced in Healthcare projects, Experience implementing FHIR Data model and FHIR Server

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Giant Eagle GCC Full time

    Senior Software Engineer - Data AnalyticsGrow. Learn. Thrive. Giant Eagle is where your career can soar high. At Giant Eagle, you are so much more than an employee. You are part of the Family.About the CompanySince our founding in 1931, Giant Eagle, Inc. has evolved into one of the top 40 largest private corporations in the U.S. and one of the country's...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Giant Eagle GCC Full time

    Sr Data Engineer Data AnalyticsGrow. Learn. Thrive. Giant Eagle is where your career can soar high. At Giant Eagle, you are so much more than an employee. You are part of the Family.About the CompanySince our founding in 1931, Giant Eagle, Inc. has evolved into one of the top 40 largest private corporations in the U.S. and one of the country's largest food...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job DescriptionData Engineer - Snowflake, Oracle/MySQL, NoSQL, Python8 to 13 years of experience in Data engineeringExtensive experience in different database platforms both RDBMS (Oracle, MySQL) and NoSQL (MongoDB)Hands-on experience in SnowflakeProficient in Data Pipeline OrchestrationProficient in any one of the programing languages (python or...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Huron Full time

    The OpportunityHuron helps its clients drive growth, enhance performance and sustain leadership in the markets they serve. We help healthcare organizations build innovation capabilities and accelerate key growth initiatives, enabling organizations to own the future, instead of being disrupted by it. Together, we empower clients to create sustainable growth,...

  • Senior Data Engineer/ Lead Data Engineer

    5 days ago

    Bengaluru, Karnataka, India Upstox Full time

    Apply for Upstox Senior Data Engineer/ Lead Data Engineer, Career Progress Consultants in Bengaluru/ Bangalore for Year of Experience on

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Mitchell Martin Inc. Full time

    JOB DESCRIPTIONThe client is building the Data Center of Excellence team in Bangalore, India. The client is looking to hire Senior Data Engineers.The Client Data and Commercial Technology Team is looking for Senior Data Engineers with a passion for building robust, scalable, efficient, and high-quality Data Engineering solutions to join our Engineering...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Lorven Technologies Full time

    JOB TITLE: Senior Data Engineer - Cloud Analytics SpecialistJoin our team as a Senior Data Engineer specializing in Cloud Analytics! You will have a pivotal role in designing, constructing, and managing scalable data pipelines and analytics solutions on various cloud platforms. Utilize your expertise in data engineering, cloud technologies, and analytics...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Twenty20 Systems Full time

    We are seeking a talented and experienced Senior Data Engineer to join our dynamic team at Twenty20 Systems. As a Senior Data Engineer, you will play a pivotal role in designing, building, and maintaining data pipelines that empower our organization to make data-driven decisions. You will collaborate closely with cross-functional teams to gather...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Twenty20 Systems Full time

    We are seeking a talented and experienced Senior Data Engineer to join our dynamic team at Twenty20 Systems. As a Senior Data Engineer, you will play a pivotal role in designing, building, and maintaining data pipelines that empower our organization to make data-driven decisions. You will collaborate closely with cross-functional teams to gather...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Squareroot Consulting Pvt Ltd Full time

    We are seeking a highly skilled and experienced Senior Data Engineer to join our team.Location : Bangalore (Work from Office all 5 days)Experience : 6+ YearsRole : Senior Data EngineerThe ideal candidate will have a strong background in big data technologies and extensive hands-on experience with Cloudera, Spark, Hive, HDFS, Flink, Data Lake, and...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Commonwealth Bank Full time

    Organisation: At CommBank, we are always aware of the impact we have on people's financial wellbeing. Our goal is to support individuals and businesses in moving forward towards achieving their dreams and aspirations. No matter where you are in our team, your creativity, skills, ideas, and enthusiasm all contribute to the meaningful work we do. Together, we...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Philips Full time

    Job Title: Senior Data EngineerJob Description:Overall Job SummaryThe Senior Data Engineer specializes in designing and developing cutting-edge data pipelines and Analytical Solutions. In this role, you will play a key part in the company's transition from On-Premises to Cloud. Collaborate with Data Scientists, Business Analysts, and Clinical teams; provide...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Infometry Full time

    Senior Data Engineer Company: Infometry IncLocation: RemoteRole : Senior data EngineerShift : 2PM-11PMEXP:4-7 yearsNP: ImmediateJob Description :We are seeking a skilled Data Engineer to join our dynamic team. As a Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines and infrastructure to support our data...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India TalentPulse Consultant Full time

    Job Description:We are looking for a talented Senior Data Engineer to join our team in Bangalore. As a Senior Data Engineer, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support our data-driven initiatives. This position offers a hybrid work model, allowing a combination of remote and onsite...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Pplability INC Full time

    Job Title: Senior Data Engineer-Data Modeling Location: Bengaluru, Chennai, Hyderabad.Curious about the role? What your typical day would look like? As a Senior Data Engineer, you will work to solve some of the organizational data management problems that would enable them as a data-driven organization; Seamlessly switch between roles of an Individual...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Mitchell Martin Inc. Full time

    JOB DESCRIPTION The client is building the Data Center of Excellence team in Bangalore, India. The client is looking to hire Senior Data Engineers. The Client Data and Commercial Technology Team is looking for Senior Data Engineers with a passion for building robust, scalable, efficient, and high-quality Data Engineering solutions to join our Engineering...