Data Engineer with Python

2 months ago


india Unison Consulting Pte Ltd Full time

  • Develop and implement data pipelines for ingesting and collecting data from various sources into a centralized data platform.
  • Develop and maintain ETL jobs using AWS Glue services to process and transform data at scale.
  • Optimize and troubleshoot AWS Glue jobs for performance and reliability.
  • Utilize Python and PySpark to efficiently handle large volumes of data during the ingestion process.
  • Design and implement scalable data processing solutions using PySpark to transform raw data into a structured and usable format.
  • Apply data cleansing, enrichment, and validation techniques to ensure data quality and accuracy.
  • Create and maintain ETL processes using Python and PySpark to move and transform data between different systems.
  • Optimize ETL workflows for performance and efficiency.
  • Collaborate with data architects to design and implement data models that support business requirements.
  • Ensure data structures are optimized for analytics and reporting.
  • Work with distributed computing frameworks, such as Apache Spark, to process and analyze large-scale datasets.
  • Manage and optimize databases, both SQL and NoSQL, to support data storage and retrieval needs.
  • Implement indexing, partitioning, and other database optimization techniques.
  • Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver effective solutions.
  • Work closely with software engineers to integrate data solutions into larger applications.
  • Implement monitoring solutions to track data pipeline performance and proactively identify and address issues
  • Ensure compliance with data privacy regulations and company policies.
  • Stay abreast of industry trends and advancements in data engineering, Python, and PySpark.

Requirements

  • Proficiency in Python and PySpark.
  • Strong knowledge of data engineering concepts and best practices.
  • Hands-on experience with AWS Glue and other AWS services.
  • Experience with big data technologies and distributed computing.
  • Familiarity with database management systems (SQL and NoSQL).
  • Understanding of ETL processes and data modeling.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration skills.
  • Bachelor's degree in Computer Science, Information Technology, or a related field.

  • Data Engineer

    3 weeks ago


    india Unison Consulting Full time

    Bachelor qualification in a computer science or STEM (science, technology, engineering, or mathematics) related field At least 8+ years of strong data warehousing experience using RDBMS and Non-RDBMS databases At least 5 years of recent hands-on professional experience (actively coding) working as a data engineer (back-end software engineer considered) ...

  • Data Engineer

    2 months ago


    india Unison Consulting Pte Ltd Full time

    Bachelor qualification in a computer science or STEM (science, technology, engineering, or mathematics) related field. At least 8+ years of strong data warehousing experience using RDBMS and Non-RDBMS databases. At least 5 years of recent hands-on professional experience (actively coding) working as a data engineer (back-end software engineer...

  • Data Engineer

    3 weeks ago


    India Unison Consulting Full time

    Bachelor qualification in a computer science or STEM (science, technology, engineering, or mathematics) related field At least 8+ years of strong data warehousing experience using RDBMS and Non-RDBMS databases At least 5 years of recent hands-on professional experience (actively coding) working as a data engineer (back-end software engineer considered)...

  • Data Engineer

    1 week ago


    india Unison Consulting Pte Ltd Full time

    Bachelor qualification in a computer science or STEM (science, technology, engineering, or mathematics) related field. At least 8+ years of strong data warehousing experience using RDBMS and Non-RDBMS databases. At least 5 years of recent hands-on professional experience (actively coding) working as a data engineer (back-end software engineer...

  • Operations Specialist

    3 weeks ago


    india Data Engineer Academy Full time

    Job Title: Operations Analyst Location : Remote Company Overview: Data Engineering Academy is a pioneering force in data engineering education, founded by FAANG data engineers with extensive industry expertise. Our platform provides comprehensive programs tailored to empower aspiring data engineers, equipping them with the skills and confidence necessary...

  • Operations Specialist

    3 weeks ago


    india Data Engineer Academy Full time

    Job Title: Operations Analyst Location : Remote Company Overview: Data Engineering Academy is a pioneering force in data engineering education, founded by FAANG data engineers with extensive industry expertise. Our platform provides comprehensive programs tailored to empower aspiring data engineers, equipping them with the skills and confidence necessary...

  • Operations Specialist

    3 weeks ago


    India Data Engineer Academy Full time

    Job Title: Operations Analyst Location : RemoteCompany Overview:Data Engineering Academy is a pioneering force in data engineering education, founded by FAANG data engineers with extensive industry expertise. Our platform provides comprehensive programs tailored to empower aspiring data engineers, equipping them with the skills and confidence necessary to...


  • india Unison Consulting Full time

    Develop and implement data pipelines for ingesting and collecting data from various sources into a centralized data platform Develop and maintain ETL jobs using AWS Glue services to process and transform data at scale Optimize and troubleshoot AWS Glue jobs for performance and reliability Utilize Python and PySpark to efficiently handle large volumes of data...


  • India Unison Consulting Full time

    Develop and implement data pipelines for ingesting and collecting data from various sources into a centralized data platformDevelop and maintain ETL jobs using AWS Glue services to process and transform data at scaleOptimize and troubleshoot AWS Glue jobs for performance and reliabilityUtilize Python and PySpark to efficiently handle large volumes of data...

  • Data Engineer

    2 months ago


    india Muoro Full time

    Responsibilities :- Design, develop, and maintain robust data pipelines using Apache Spark, Python, and other relevant technologies.- Implement data ingestion processes from various sources, ensuring data quality and integrity.- Optimize and tune data pipelines for performance and scalability.- Collaborate with data scientists and analysts to deploy machine...

  • Data Engineer

    2 months ago


    india Paralleldots Full time

    Job Description : As a Senior Data Engineer at ParallelDots, you will be responsible for overseeing the design and development of advanced data solutions. You will work closely with data scientists, analysts, and other stakeholders to ensure that our data infrastructure meets the highest standards of scalability, reliability, and : - Lead the design,...

  • Data Engineer

    3 days ago


    india Shankheshwar Enterprises Full time

    Our client is a leading player in the Digital Transformation space have 17 centers in USA and India Job Description : - Analyze and organize raw data, developing and maintaining data systems and pipelines, and collaborating closely with data scientists and business analysts to ensure data quality and reliability. - Quickly grasp and translate business...

  • Data Engineer

    2 weeks ago


    india Edstem Technologies Full time

    Role : Data Engineer Roles : We are looking for Data Engineers for an AI based product development for one of our clients in the banking industry. As a data engineer you will play a pivotal role in collaborating with a team of skilled data engineers to develop and operationalize machine learning models that drive customer satisfaction and enhance our...

  • AWS Data Engineer

    2 months ago


    india Change Leaders Full time

    Job Description : AWS Glue + Python PysparkJob Overview :Candidate should be responsible for the configuration, administration, and optimization of the AWS Glue + Python Pyspark platform. This role involves ensuring the smooth operation of AWS Glue + Python Pyspark for data integration, transformation, and loading tasks in alignment with the organization's...

  • Data Engineer –

    2 months ago


    india HARMAN International Full time

    HARMAN’s engineers and designers are creative, purposeful and agile. As part of this team, you’ll combine your technical expertise with innovative ideas to help drive cutting-edge solutions in the car, enterprise and connected ecosystem. Every day, you will push the boundaries of creative design, and HARMAN is committed to providing you with the...

  • Data Engineer –

    1 week ago


    india HARMAN International Full time

    HARMAN’s engineers and designers are creative, purposeful and agile. As part of this team, you’ll combine your technical expertise with innovative ideas to help drive cutting-edge solutions in the car, enterprise and connected ecosystem. Every day, you will push the boundaries of creative design, and HARMAN is committed to providing you with the...

  • Data Engineer

    3 weeks ago


    india Acesoft Labs (India) Pvt Ltd Full time

    Responsibilities :- Build extremely efficient and reliable data pipelines for training examples generation and real-time online training, etc.- Design and build data transformations efficiently and reliably for different purposes (e.g. reporting, growth analysis, multi-dimensional analysis)- Design and implement reliable, scalable, robust, and extensible big...

  • Data Engineer

    1 month ago


    india Teksands.ai Full time

    The Data engineer is responsible for development & maintenance of the Data Lake within the Advice & Wealth Management technology team. You should be able to collaborate with technology cross-commits and business teams to code, test and deploy scalable solution and adhering to required quality standards. You must be a self-starter, think outside the box and...


  • india Infometry Full time

    Senior Data Engineer Company: Infometry IncLocation: RemoteRole : Senior data EngineerShift : 2PM-11PMEXP: 4-7 yearsNP: ImmediateJob Description :We are seeking a skilled Data Engineer to join our dynamic team. As a Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines and infrastructure to support our data...

  • Senior Data Engineer

    4 weeks ago


    india Rigel Networks Pvt.Ltd Full time

    Role Description:This is role for a Senior Data Engineer specializing in PySpark, Python, and SQL.As a Senior Data Engineer, you will be responsible for designing, building, and maintaining scalable data infrastructure and pipelines.You will collaborate with cross-functional teams to analyze data requirements, develop data models, and implement data...