Data Engineer

3 weeks ago

Anywhere in IndiaMultiple Locations, IN Excelra Knowledge Solutions Full time

Job Description :


At Excelra ,We are seeking a highly skilled Data Engineer with expertise in Databricks, SQL, Python, and data flow pipelines.

The ideal candidate will have experience in creating Azure Data Factory (ADF) pipelines, deploying code in Azure DevOps using Git Repos and YAML files, and exposure to Snowflake. This role involves designing, implementing, and maintaining robust data solutions to support various business needs.

Key Responsibilities :

Databricks Expertise :

- Develop and maintain data processing workflows using Databricks.

- Optimize Databricks jobs for performance and cost efficiency.

- Implement data transformations and aggregations within the Databricks environment.

- Collaborate with data scientists and analysts to integrate and process large datasets.

SQL Proficiency :

- Write complex SQL queries to retrieve and manipulate data from various databases.

- Ensure data accuracy and consistency through effective querying and data validation techniques.

- Optimize SQL queries for performance and scalability.

Python Programming :

- Develop data pipelines and ETL processes using Python.

- Create reusable code modules and libraries for data processing tasks.

- Perform data cleaning, transformation, and analysis using Python-based tools and frameworks.

Data Flow Pipelines :

- Design and implement data flow pipelines to move and transform data across different systems.

- Ensure data integrity and reliability in data pipelines.

- Monitor and troubleshoot data pipelines to resolve issues promptly.

Azure Data Factory (ADF) Pipelines :

- Create and manage ADF pipelines to orchestrate data movement and transformation activities.

- Develop custom ADF activities using Azure Functions or other services.

- Implement logging, monitoring, and alerting mechanisms for ADF pipelines.

Azure DevOps Deployment Pipelines :

- Set up and maintain deployment pipelines in Azure DevOps.

- Use Git Repos for version control of data flow related code and Databricks notebooks.

- Create and manage YAML files for continuous integration and deployment (CI/CD) processes.

- Ensure seamless deployment of Databricks and other data-related code through Azure DevOps.

Snowflake Exposure :

- Work with Snowflake data warehouse to store and manage large volumes of data.

- Write and optimize SQL queries for Snowflake.

- Integrate Snowflake with other data processing tools and platforms.

Qualifications :

- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field. A master's degree is a plus.

Experience :

- Proven experience as a Data Engineer or similar role.

- Hands-on experience with Databricks and Spark.

- Strong SQL skills with experience in query optimization.

- Proficiency in Python for data processing and automation.

- Experience in building and managing ADF pipelines.

- Familiarity with Azure DevOps, Git, and YAML for CI/CD processes.

- Exposure to Snowflake and understanding of its architecture and capabilities.

Skills :

- Strong analytical and problem-solving skills.

- Excellent communication and collaboration abilities.

- Ability to work in a fast-paced, dynamic environment.

- Attention to detail and commitment to data quality.

Additional Desirable Skills :

- Experience with other cloud platforms (AWS, GCP).

- Knowledge of data governance and security best practices.

- Familiarity with data visualization tools (e.g., Power BI, Tableau).

(ref:hirist.tech)

  • Aptus Data Labs

    1 month ago

    Anywhere in India/Multiple Locations, IN Aptus Data Labs Full time

    Role - Data Architect/Data Engineer Lead- AWSExp - 8+ YearsLocation - RemoteSkills - Data architecture, Data modelling & design, Data Governance, MDM, Python, SQL, ETL, AWS CloudWe are seeking a highly skilled and experienced Data Architect to join our team. The ideal candidate will have 8+ years of experience in data architecture, with a focus on AWS cloud...

  • Aptus Data Labs

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Aptus Data Labs Full time

    Role - Data Architect/Data Engineer Lead- AWSExp - 8+ YearsLocation - RemoteSkills - Data architecture, Data modelling & design, Data Governance, MDM, Python, SQL, ETL, AWS CloudWe are seeking a highly skilled and experienced Data Architect to join our team. The ideal candidate will have 8+ years of experience in data architecture, with a focus on AWS cloud...

  • MLOps Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN GOOGNU DATA SERVICES PRIVATE LIMITED Full time

    Job Description :- Design and Implement MLOps Infrastructure: Build and maintain robust MLOps workflows using Azure Machine Learning and other Azure services to automate machine learning lifecycle from data preparation to model deployment and monitoring.- Collaborate with Cross-Functional Teams: Work closely with data scientists, software developers, and IT...

  • MLOps Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN GOOGNU DATA SERVICES PRIVATE LIMITED Full time

    Job Description :- Design and Implement MLOps Infrastructure: Build and maintain robust MLOps workflows using Azure Machine Learning and other Azure services to automate machine learning lifecycle from data preparation to model deployment and monitoring.- Collaborate with Cross-Functional Teams: Work closely with data scientists, software developers, and IT...

  • Data Engineer

    4 weeks ago

    Anywhere in India/Multiple Locations, IN Coders Brain Technology Pvt. Ltd. Full time

    Position Name : Data Engineer (Integration)Experience Required : 10+yrsSalary : As per the Market Standard or Else whatever budget you haveNotice period : Immediate joiner Job Type : Full timeLocation : RemoteJob Description :- Understand all forms of data integration including APIs - Event-based, Streaming, REST and GraphQL API, CDC/Replication, Batch,...

  • Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Coders Brain Technology Pvt. Ltd. Full time

    Position Name : Data Engineer (Integration)Experience Required : 10+yrsSalary : As per the Market Standard or Else whatever budget you haveNotice period : Immediate joiner Job Type : Full timeLocation : RemoteJob Description :- Understand all forms of data integration including APIs - Event-based, Streaming, REST and GraphQL API, CDC/Replication, Batch,...

  • Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Coders Brain Technology Pvt. Ltd. Full time

    Position Name : Data ArchitectExperience Required : 10yrsSalary : As per the Market Standard or Else whatever budget you haveNotice period : Immediate joiner Job Type : Full timeLocation : RemoteJob Description : 10+ years of proven experience as a Data Engineer with 5+ years as Data Architect.- Bachelor's degree in computer science, Information...

  • Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Coders Brain Technology Pvt. Ltd. Full time

    Position Name : Data ArchitectExperience Required : 10yrsSalary : As per the Market Standard or Else whatever budget you haveNotice period : Immediate joiner Job Type : Full timeLocation : RemoteJob Description : 10+ years of proven experience as a Data Engineer with 5+ years as Data Architect.- Bachelor's degree in computer science, Information...

  • Senior Data Engineer

    5 days ago

    Anywhere in India/Multiple Locations, IN MindBrain Full time

    Company Description. MindBrain seamlessly integrates innovation, education, and strategic workforce solutions. As a dynamic software company, we shape the future of technology. With a focus on comprehensive IT and non-IT training, we nurture talent, empowering individuals to lead in a rapidly evolving landscape. Through strategic resource augmentation, we...

  • Senior/Staff Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Grorapid labs Full time

    Senior / Staff Engineer Data EngineeringExperience Required : 8 - 14 years working Days 5 days/week Office Location : Remote WorkingRole & Responsibilities:- Design, architect, and develop code in a testable, scalable, and performant manner.- Work closely with Product Managers, Cross Product Engineers and Data Engineers to optimize and deliver innovative...

  • Senior/Staff Engineer

    2 months ago

    Anywhere in India/Multiple Locations, IN Grorapid labs Full time

    Senior / Staff Engineer Data EngineeringExperience Required : 8 - 14 years working Days 5 days/week Office Location : Remote WorkingRole & Responsibilities:- Design, architect, and develop code in a testable, scalable, and performant manner.- Work closely with Product Managers, Cross Product Engineers and Data Engineers to optimize and deliver innovative...

  • Lead Data Engineer

    4 weeks ago

    Anywhere in India/Multiple Locations, IN Pro5.ai Full time

    Job Description :Technical delivery lead on platform and project builds for building and maintaining optimised and highly available data pipelines that facilitate data management and data analytics solutions. The role will drive the development of data processing frameworks and data architectural approaches that scale to handle the business's growing...

  • Lead Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Pro5.ai Full time

    Job Description :Technical delivery lead on platform and project builds for building and maintaining optimised and highly available data pipelines that facilitate data management and data analytics solutions. The role will drive the development of data processing frameworks and data architectural approaches that scale to handle the business's growing...

  • Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN ARAHAS TECHNOLOGIES PRIVATE LIMITED Full time

    We are currently hiring a Pyspark Data Engineer with AWS expertise with7 years of experience Job Description :- The candidate should have strong skills in Pyspark coding and have a background in AWS technologies such as Airflow- Experience working with complex SQL queries- Good with PySpark, Databricks, and Airflow- Bachelor's or Master's degree in...

  • Data Engineer

    2 months ago

    Anywhere in India/Multiple Locations, IN Paralleldots Full time

    Job Description :As a Senior Data Engineer at ParallelDots, you will be responsible for overseeing the design and development of advanced data solutions. You will work closely with data scientists, analysts, and other stakeholders to ensure that our data infrastructure meets the highest standards of scalability, reliability, and :- Lead the design,...

  • DevOps & Data Engineer

    2 months ago

    Anywhere in India/Multiple Locations, IN Dotflick Solutions Full time

    Profile : DevOps and Data EngineerJob description : This role is a Hybrid DevOps and Data Engineering.Mode : WFO/WFHRoles & responsibilities : - Setting up tools and required infrastructure- Defining and setting development, test, release, update, and support processes for DevOps operation- Have the technical skill to review, verify, and validate the...

  • DevOps & Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Dotflick Solutions Full time

    Profile : DevOps and Data EngineerJob description : This role is a Hybrid DevOps and Data Engineering.Mode : WFO/WFHRoles & responsibilities : - Setting up tools and required infrastructure- Defining and setting development, test, release, update, and support processes for DevOps operation- Have the technical skill to review, verify, and validate the...

  • Data Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN Perfecthire Full time

    Job Description : We are seeking a talented and experienced Data Engineer to join our team and play a key role in the development and implementation of our end-to-end 834 EDI solution. A robust level of experience or familiarity with 834 EDI (Electronic Data Interchange) transactions is critical for this role. As a Data Engineer, you will be responsible for...

  • Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Perfecthire Full time

    Job Description : We are seeking a talented and experienced Data Engineer to join our team and play a key role in the development and implementation of our end-to-end 834 EDI solution. A robust level of experience or familiarity with 834 EDI (Electronic Data Interchange) transactions is critical for this role. As a Data Engineer, you will be responsible for...

  • Data Platform Engineer

    5 days ago

    Anywhere in India/Multiple Locations, IN Zyoin group Full time

    We're seeking a talented and experienced Data Platform Engineer to join our team and play a key role in optimizing, maintaining, and evolving our data infrastructure. You'll be responsible for ensuring our data pipelines run smoothly, efficiently, and cost-effectively, while contributing to a stable and exceptional user Pipeline Powerhouse:...