Data Engineering

1 week ago

Remote, India VSHR UNNATI LLP Full time

SALARY : 10LPA - 16LPA


Job Overview :

Our client is seeking an experienced and detail-oriented Data Engineering and Machine Learning Tester to join our Quality Assurance team. The ideal candidate will have strong expertise in testing data pipelines, machine learning models, and large-scale data processing systems, specifically within the Databricks environment. This role is critical to ensuring the quality, accuracy, and reliability of our data engineering workflows and machine learning solutions.

Key Responsibilities :

Test Planning and Strategy :

- Develop, document, and maintain comprehensive test plans, test cases, and test scripts for data engineering and machine learning workflows.

- Collaborate with data engineers, data scientists, and product managers to understand requirements and define testing strategies tailored to the Databricks environment.

Data Pipeline Testing :

- Validate data ingestion, transformation, and processing workflows within Databricks.

- Perform end-to-end testing of ETL/ELT processes to ensure data accuracy, consistency, and performance.

- Verify data integrity across different stages of the pipeline and ensure that data meets business requirements.

Machine Learning Model Testing :

- Test and validate machine learning models developed on Databricks, ensuring they meet accuracy, performance, and reliability standards.

- Develop and execute tests for model training, validation, and deployment pipelines.

- Evaluate the performance of models on large datasets and ensure that they generalize well to unseen data


Automation and Continuous Integration :

- Implement automated testing solutions for data pipelines and machine learning models using tools compatible with Databricks.

- Integrate automated testing processes into the CI/CD pipeline to ensure continuous quality checks. Azure DevOps experience is a plus.

- Leverage Databricks notebooks and workflows for test automation where applicable.

Performance Testing :

- Conduct performance testing of data pipelines and machine learning models to identify bottlenecks and optimize processing time.

- Test the scalability of data engineering solutions to handle large volumes of data efficiently.

Collaboration and Communication :

- Work closely with cross-functional teams, including data engineers, data scientists, and DevOps, to resolve technical challenges and ensure the quality of data solutions.

- Provide clear and concise feedback on test results and collaborate with teams to address issues.

- Participate in sprint reviews and retrospectives, providing insights on test outcomes and quality metrics.

Qualifications :

Technical Skills :

- Strong experience with Databricks and related tools (Apache Spark, Delta Lake, MLflow).

- Proficiency in Python, SQL, and testing frameworks relevant to data engineering and machine learning.

- Experience with big data technologies on Azure platform.

- Knowledge of testing automation tools and CI/CD pipelines.

- Understanding of data governance, data quality, and data privacy practices.

Experience :

- 3+ years of experience in testing data engineering workflows and machine learning models, with a focus on big data environments.

- Proven experience working within Databricks or similar platforms.

- Prior experience in an Agile development environment is preferred.

Soft Skills :

- Excellent analytical and problem-solving skills.

- Strong communication skills, both verbal and written.

- Ability to work independently and as part of a collaborative team.

Benefits :

- Opportunities for professional growth and development.

- Collaborative and innovative work environment.

(ref:hirist.tech)

  • Azure Data Architect

    6 months ago

    Remote, India Data PlatformExperts Full time

    Job Description: Azure Data Engineer (Architect Level) We are seeking an experienced and highly skilled Azure Data Engineer at the Architect level to lead the design and implementation of complex data solutions on the Microsoft Azure platform. As an Azure Data Engineer Architect, you will be responsible for providing technical expertise, guiding the...

  • Azure Data Architect

    6 months ago

    Remote, India Data PlatformExperts Full time

    **Job Description: Azure Data Architect** **Key Responsibilities**: - **Design and Implementation**: - Design and implement end-to-end data solutions (data models, pipelines, data lakes, data warehouses) in Azure. - Optimize and maintain existing Azure data structures and integration processes. - Ensure architectural solutions are scalable, maintainable,...

  • Azure Data Engineer

    2 weeks ago

    Remote, India Yallo Management Consultancies L.L.C Full time

    Role Type: Contract Location: Remote Duration: 6 months+ Start: ASAP Job Role: As an Azure Data Engineer, you will play a crucial role in designing, developing, and maintaining data engineering solutions on the Azure cloud platform. You will collaborate with cross-functional teams to understand data requirements, architect scalable data pipelines, and ensure...

  • Data Engineering Expert

    13 hours ago

    Remote, India CodeKart Full time

    Job Title: Senior Data Pipeline Developer">About CodeKart:">We are a cutting-edge technology firm dedicated to delivering innovative solutions. Our team is passionate about data engineering and committed to excellence.">Job Overview:">We are seeking an experienced Senior Data Pipeline Developer to join our team. As a key member, you will design, develop, and...

  • Data Engineer

    1 week ago

    Remote, India Cloudcover Consultancy Private Limited Full time

    Job Overview :The Data Engineer will be responsible for designing and building modern data platforms to support data-driven decision making. The Data Engineer will execute technical implementation of data engineering and visualization projects.The Data Engineer will help build a data and analytics consulting practice by taking part in recruiting efforts,...

  • Data Engineering Expert

    1 month ago

    Remote, India Quantumbricks Full time

    Job Summary: We are seeking a highly skilled Senior Data Engineer with over 10 years of experience in designing, developing, and managing robust data pipelines, with specific expertise in Informatica Intelligent Cloud Services (IICS). The ideal candidate will have extensive experience in handling complex ETL/ELT processes, cloud data integration, and data...

  • CodeKart - Data Engineer - ETL/Data Warehousing

    1 week ago

    Remote, India CodeKart Full time

    Role : Data Engineer. Experience :3 yr.Location : Remote. Responsibilities : - Provide ongoing support and monitoring for AWS-based data engineering infrastructure. - Ensure the smooth operation of data pipelines and applications in a production environment. - Develop and maintain data pipelines, ETL processes, and data warehouses using AWS services. ...

  • Azure Data Engineer

    1 month ago

    Remote, India Racrosoft Technologies Full time

    Role : Azure Data Engineer Exp - 4+ years of experience Job Type : Contract Mode of Work : Remote Qualification : - Very Good Communication (Client engagement)- Able to answer Scenario Based Questions on your past work- Able to answer Deep down on certification - Able to answer Past work with deep explanation. - Good exposure to Pyspark - Focus on Azure...

  • Lead Data Engineering Architect

    6 days ago

    remote, India TWO95 International, Inc Full time

    Job Title: Lead Data Engineering ArchitectCompany Overview: TWO95 International, Inc is a leading organization in the healthcare industry, seeking a highly skilled Data Engineer to join our team.Estimated Salary Range: $180,000 - $250,000 per year, depending on experience and location.Job Description:We are looking for an experienced Data Engineer to lead...

  • Marketing Intern

    6 months ago

    Remote, India 5x Data Full time

    We are seeking a dynamic Marketing Intern to join our team at 5X, a leader in simplifying the modern data stack for businesses. This role is designed for an ambitious individual eager to dive into the world of marketing within the fast-paced tech industry, particularly focusing on data engineering and SaaS environments. As a Marketing Intern, you will play a...

  • Data Engineer

    1 week ago

    Remote, India ThinkMove Solutions Full time

    Salary : 14 - 18 LPA Overview : We are seeking a skilled and experienced Data Engineer with a specialization in DataBricks to join our dynamic team. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining our data infrastructure, with a specific focus on leveraging DataBricks for data processing and analytics. Your...

  • Senior AWS Data Engineer

    1 week ago

    Remote, India Quantumbricks Full time

    Job Summary. We are seeking a highly skilled and experienced Senior AWS Data Engineer to lead our cloud data architecture, design, and development initiatives. The ideal candidate will have deep expertise in building and managing large-scale data pipelines and solutions on AWS, as well as strong experience in data architecture, ETL, and real-time data...

  • Senior Azure Data Engineer

    1 week ago

    Remote, India Quantumbricks Full time

    Job Overview : We are seeking a highly skilled and experienced Senior Azure Data Engineer to join our team. The ideal candidate will have over 10 years of experience in designing, developing, and implementing data solutions on Azure. This role involves working closely with cross-functional teams to build scalable, efficient, and secure data platforms. You...

  • Snowflake Engineer

    1 week ago

    Remote, India InCommon Full time

    Position Summary :We are looking for seasoned Snowflake Architects and Engineers to join our team and lead the design and implementation of large-scale data solutions on the Snowflake platform. This role is suited for professionals with deep expertise in data architecture and extensive experience with Snowflake.Key Responsibilities : - Lead the architecture,...

  • Aws Architect

    6 months ago

    Remote, India True Data Software Full time

    Experience - 8+ Years Remote Job - Remote, Anywhere Vacancy - 1 **Requirements**: Bachelor's degree in either Computer Science, Computer Engineering, Software Engineering, MIS, or other IT-related discipline - Minimum 8+ years experience with following AWS services: VPC, Route53, CloudFront, WAF WebACL, CloudWatch, EKS + Fargate, EC2, Lambda, AppMesh,...

  • Senior Data Engineer

    1 week ago

    Remote, India Quantumbricks Full time

    Job Summary : We are seeking a highly skilled Senior Data Engineer with over 10 years of experience in designing, developing, and managing robust data pipelines, with specific expertise in Informatica Intelligent Cloud Services (IICS). The ideal candidate will have extensive experience in handling complex ETL/ELT processes, cloud data integration, and data...

  • Azure Data Engineer

    2 months ago

    Remote, India Talpro Full time

    Azure Data Engineer :We are looking for a skilled and experienced Azure Data Engineer with expertise in ADF (Azure Data Factory), strong RDBMS and SQL skills, and familiarity with Informatica PWC and Oracle PL/SQL. As an Azure Data Engineer, you will be responsible for understanding and converting Informatica PWC and Oracle PL/SQL code into ADF and Azure...

  • Data Engineer

    1 month ago

    Remote, India BlueByte Technologies Full time

    **Job Summary**BlueByte Technologies is seeking an experienced Data Engineer - Cloud Migration Expert to lead the migration of data from Google Cloud Platform (GCP) to Snowflake. This is a 12-month contract position with the possibility of extension.Job Description:We are looking for a highly skilled Data Engineer with expertise in GCP, Snowflake, and dbt to...

  • Freelance Visual Designers

    6 months ago

    Remote, India 5x Data Full time

    We are seeking a talented and creative Visual Designer (Freelancer) to join our team and contribute to a variety of marketing projects with a keen eye for detail and a passion for design excellence. **Experience Required**: 3-6 years **Hours**: 3-4 hours every weekday **Roles & Responsibilities**: - Utilize knowledge of visual hierarchy, grid systems,...

  • Protegrity Engineering Manager

    2 months ago

    Remote, India PROTEGRITY INDIA PRIVATE LIMITED Full time

    Job Title: Protegrity Engineering Manager - Big DataJob Summary:We are seeking a highly skilled Engineering Manager to lead our Big Data team at Protegrity. As a key member of our engineering team, you will be responsible for managing the development of our Big Data products and ensuring they meet the highest standards of quality and performance.Key...