Data Pipeline Engineer

2 weeks ago

Anywhere in IndiaMultiple Locations Robosoft Technologies Full time
Data Pipeline Engineer

6 MONTHS CONTRACT

Roles and Responsibilities:

  • Develop, design and automate large scale, highperformance data processing systems (batch and/or streaming) to drive business growth and improve the product experience.
  • Evangelize high quality software engineering practices towards building data infrastructure and pipelines at scale.
  • Lead data engineering projects to ensure pipelines are reliable, efficient, testable, & maintainable
  • Design our data models for optimal storage and retrieval and to meet critical product and business requirements.
  • Understand and influence logging to support our data flow, architecting logging best practices where needed
  • Contribute to shared Data Engineering tooling & standards to improve the productivity and quality of output for Data Engineers across the company
  • Partner with leadership, engineers, program managers and data scientists to understand data needs.
  • Educate your partners: Use your data engineering experience to 'see what's missing', identifying and addressing gaps in their existing logging and processes.
  • Work with stakeholders to build data lineage, data governance and data cataloging.
  • Leading projects using agile methodologies.
  • Communicate effectively with people of all levels in the organization.
  • Recruit, retain and develop people skills to take bigger responsibilities and challenges.
Experience & Skills:

7+ years of relevant industry experience.

  • Experienced in custom ETL design, implementation and maintenance.
  • Working knowledge of relational databases and query authoring (SQL).
  • Experience with Java / Scala / Spark / Pyspark / Python/Databricks is preferred
  • Any prior cloud experience (AWS preferred)
  • Working experience with data at the large scale.
  • Experience designing, building and operating robust data pipelines.
  • Experience designing and deploying high performance pipelines with reliable monitoring and logging practices.
  • Effectively work across team boundaries to establish overarching data architecture and provide guidance to individual teams.
  • Optimize your ETL pipeline to minimize data transfer and storage costs.
  • Use Data Bricks SQL's auto scaling feature to adjust the number of workers based on the workload,
Tech Stack:

  • AWS Services
  • Databricks SQL Serverless
  • Databricks SQL,
  • Databricks workspaces,
  • Databricks notebooks,
  • Scheduling Databricks Job,
  • Delta Lake Architecture,
  • Lakehouse Concepts and understanding,
  • Spark Structured/Streaming
  • File Formats (CSV, Avro, Parquet)
  • CI/CD for ETL
  • Databricks SQL unified governance model (using Unity catalog) across clouds, open formats, APIs
)

  • Lead Data Engineer

    1 month ago

    Anywhere in India/Multiple Locations Pro5.ai (formerly Mangtas) Full time

    Job Description :Technical delivery lead on platform and project builds for building and maintaining optimised and highly available data pipelines that facilitate data management and data analytics solutions. The role will drive the development of data processing frameworks and data architectural approaches that scale to handle the business's growing...

  • Lead Data Engineer

    4 weeks ago

    Anywhere in India/Multiple Locations, IN Pro5.ai Full time

    Job Description :Technical delivery lead on platform and project builds for building and maintaining optimised and highly available data pipelines that facilitate data management and data analytics solutions. The role will drive the development of data processing frameworks and data architectural approaches that scale to handle the business's growing...

  • Robosoft Technologies

    3 weeks ago

    Anywhere in India/Multiple Locations Robosoft Technologies Full time

    Data Pipeline Engineer 6 MONTHS CONTRACT Roles and Responsibilities: - Develop, design and automate large scale, high-performance data processing systems (batch and/or streaming) to drive business growth and improve the product experience.- Evangelize high quality software engineering practices towards building data infrastructure and pipelines at scale....

  • Robosoft Technologies

    2 weeks ago

    Anywhere in India/Multiple Locations, IN Robosoft Technologies Full time

    Data Pipeline Engineer6 MONTHS CONTRACTRoles and Responsibilities:- Develop, design and automate large scale, high-performance data processing systems (batch and/or streaming) to drive business growth and improve the product experience.- Evangelize high quality software engineering practices towards building data infrastructure and pipelines at scale. - Lead...

  • Heroku Developer

    4 weeks ago

    Anywhere in India/Multiple Locations, IN HiVision software solutions llp Full time

    Job Description :Key Responsibilities :- Designing and implementing data pipelines to extract, transform, and load (ETL) data from various sources into Heroku Postgres.- Optimizing data pipeline performance for scalability and reliability.- Collaborating with cross-functional teams to identify data integration requirements and develop solutions that meet...

  • Heroku Developer

    4 weeks ago

    Anywhere in India/Multiple Locations HiVision software solutions llp Full time

    Job Description : Key Responsibilities :- Designing and implementing data pipelines to extract, transform, and load (ETL) data from various sources into Heroku Postgres.- Optimizing data pipeline performance for scalability and reliability.- Collaborating with cross-functional teams to identify data integration requirements and develop solutions that meet...

  • Cloud Data Engineer

    2 months ago

    Bangalore/Hyderabad/Chennai/Pune/Anywhere in India/Multiple Locations Amk technologies Full time

    Job Description : We are seeking a talented and motivated Cloud Data Engineer to join our dynamic team. As a Cloud Data Engineer, you will be responsible for designing, implementing, and maintaining data pipelines and infrastructure on Azure Cloud Platform. You will work closely with cross-functional teams to ensure efficient data flow and integration...

  • Data Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations BlueByte Technologies Full time

    Position : Data engineerLocation : Remote/ HybridMust have - AKS, Python and SQLGood to have - Erwin toolJob Description :seeking a skilled and motivated AKS (Azure Kubernetes Service) Data Engineer to join our dynamic team (can be open for GKS exp in place of AKS too). The ideal candidate will have a strong background in data engineering, experience with...

  • Data Architect

    2 weeks ago

    Anywhere In India/multiple Locations Aptus Data Labs Full time

    Role - Data Architect/Data Engineer Lead- AWSExp - 8+ Years Location - Remote Skills - Data architecture, Data modelling & design, Data Governance, MDM, Python, SQL, ETL, AWS Cloud We are seeking a highly skilled and experienced Data Architect to join our team.The ideal candidate will have 8+ years of experience in data architecture, with a focus on AWS...

  • Data Warehouse Architect

    2 months ago

    Bangalore/Anywhere in India/Multiple Locations Info Origin Inc Full time

    We are seeking a highly skilled and experienced Data Warehouse Architect to join our team! In this pivotal role, you will play a key role in building and maintaining our data warehouse infrastructure, with a specific focus on integrating data from Workday. You will leverage your expertise in data warehousing, ETL development, and Workday technologies to...

  • Data Warehouse Architect

    2 months ago

    Bangalore/Anywhere in India/Multiple Locations, IN Info Origin Inc Full time

    We are seeking a highly skilled and experienced Data Warehouse Architect to join our team! In this pivotal role, you will play a key role in building and maintaining our data warehouse infrastructure, with a specific focus on integrating data from Workday. You will leverage your expertise in data warehousing, ETL development, and Workday technologies to...

  • Data Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations Shankheshwar Enterprises Full time

    Our client is a leading player in the Digital Transformation space have 17 centers in USA and IndiaJob Description :Analyze and organize raw data, developing and maintaining data systems and pipelines, and collaborating closely with data scientists and business analysts to ensure data quality and reliability.Quickly grasp and translate business requirements...

  • Data Engineer

    4 weeks ago

    Trivandrum/Thiruvananthapuram/Cochin/Kochi/Anywhere in India/Multiple Locations ProPMO Services Private Limited Full time

    Job Description : We have a requirement for 3 Data Engineers who could join us on a short notice. 3 to 8 years of professional software development/engineering experience. Specialist experience working in Python coding as a Data Engineer. Must have good experience working with databases PostgreSQL/MySQL/MS SQL. Must have good experience working in AWS/GCP...

  • Azure Data Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations TekNinjas Full time

    Role :Data Engineer - Azure Synapse Analytics- Experience in Data engineering projects using Microsoft Azure platform (Minimum of 23 projects)- Strong expertise in data engineering tools and storage such as Azure ADLS Gen2, Blob storage- Experience implementing automated Synapse pipelines- Ability to implement Synapse pipelines for data integration ETL/ELT...

  • Azure Data Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations Vikisol Technologies Full time

    Azure Data Engineer Opportunity : Remote Exp : 5+ years Location : Remote Skills : Azure Databricks Azure ADF PySpark Python SQLResponsibilities : Collaborate on ETL data requirements. Design and optimize data pipelines (Azure Databricks, ADF). Develop PySpark scripts for efficient data processing. Ensure ETL tasks' performance and scalability (Python,...

  • Aptus Data Labs

    4 weeks ago

    Anywhere in India/Multiple Locations Aptus Data Labs Full time

    Role - Data Architect/Data Engineer Lead- AWSExp - 8+ YearsLocation - RemoteSkills - Data architecture, Data modelling & design, Data Governance, MDM, Python, SQL, ETL, AWS CloudWe are seeking a highly skilled and experienced Data Architect to join our team. The ideal candidate will have 8+ years of experience in data architecture, with a focus on AWS cloud...

  • Aptus Data Labs

    4 weeks ago

    Anywhere in India/Multiple Locations, IN Aptus Data Labs Full time

    Role - Data Architect/Data Engineer Lead- AWSExp - 8+ YearsLocation - RemoteSkills - Data architecture, Data modelling & design, Data Governance, MDM, Python, SQL, ETL, AWS CloudWe are seeking a highly skilled and experienced Data Architect to join our team. The ideal candidate will have 8+ years of experience in data architecture, with a focus on AWS cloud...

  • Data Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations Circuit Compilers Full time

    Job Description : 5 to 8 years in the build release management, software configuration, design, development, and cloud implementation. Data Engineer with expertise in Python, Scala, PySpark, AWS S3, Glue, Redshift, Lambda, DBT, Serverless, Snowflake, SQL, Datalake, and Data Pipeline. Leveraged AWS Glue for data cataloging, ETL job orchestration, and schema...

  • Azure Data Engineer

    4 weeks ago

    Anywhere in India/Multiple Locations Vikisol Technologies Full time

    Azure Data Engineer Opportunity : Remote Exp : 5+ years Location : Remote Skills : - Azure Databricks- Azure ADF- PySpark- Python- SQL Responsibilities : - Collaborate on ETL data requirements.- Design and optimize data pipelines (Azure Databricks, ADF).- Develop PySpark scripts for efficient data processing.- Ensure ETL tasks' performance and...

  • Data Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations XANDER CONSULTING AND ADVISORY PRIVATE LIMITED Full time

    Primary Roles and Responsibilities: Developing Modern Data Warehouse solutions using Databricks and AWS/ Azure Stack Ability to provide solutions that are forwardthinking in data engineering and analytics space Collaborate with DW/BI leads to understand new ETL pipeline development requirements. Triage issues to find gaps in existing pipelines and fix the...