Data Engineer

4 weeks ago

Chennai, Tamil Nadu, India O2F Info Solutions Pvt. Ltd. Full time

Job Summary :

We are seeking a highly skilled Senior Data Engineer with 4 to 8 years of experience in building robust data pipelines and working extensively with PySpark to join our data engineering team.

Key Responsibilities :

Data Pipeline Development :

- Design, build, and maintain scalable data pipelines using PySpark to process large datasets and support data-driven applications and analytics.

ETL Process Automation :

- Develop and automate ETL (Extract, Transform, Load) processes using PySpark, ensuring efficient data processing, transformation, and loading from diverse sources into data lakes, warehouses, or databases.

Distributed Computing with PySpark :

- Leverage Apache Spark and PySpark to process large-scale data in a distributed computing environment, optimizing for performance and scalability.

Cloud Data Solutions :

- Develop and deploy data pipelines and processing frameworks on cloud platforms (AWS, Azure, GCP) using native tools like AWS Glue, Azure Databricks, or Google Dataproc.

Data Integration & Transformation :

- Integrate data from various internal and external sources, ensuring data consistency, quality, and reliability throughout the pipeline.

Performance Optimization :

- Optimize PySpark jobs and pipelines for faster data processing, handling large volumes of data efficiently with minimal latency.

- Proven experience as a Data Engineer or similar role, with a strong background in database development, ETL processes, and software development.

- Proficiency in SQL and scripting languages such as Python, with experience working with relational databases.

- Proficiency in dataProc (PySpark), Pandas or other data processing libraries

- Experience with data modeling, schema design, and optimization techniques for scalability.

- Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues and optimize data processing pipelines for scale

Required Qualifications :

- 4-8 years of experience in data engineering, with a strong focus on PySpark and large-scale data processing.

Technical Skills :

- Expertise in PySpark for distributed data processing, data transformation, and job optimization.

- Strong proficiency in Python and SQL for data manipulation and pipeline creation.

- Hands-on experience with Apache Spark and its ecosystem, including Spark SQL, Spark Streaming, and PySpark MLlib.

- Solid experience working with ETL tools and frameworks, such as Apache Airflow or similar orchestration tools.

(ref:hirist.tech)

  • Data Engineer

    3 weeks ago

    Chennai, Tamil Nadu, India Quantrail Data Full time

    Location: Chennai, India (On-site)Duration: 3 monthsStipend: UnpaidEligibility: 2025 graduates onlyConversion: Opportunity for a full-time role based on performance Note: Please apply only if you are serious about joining us. This is a learning-focused internship, and we expect interns to be committed to upskilling themselves and contributing to real-world...

  • Data Engineer

    7 days ago

    Chennai, Tamil Nadu, India NTT DATA North America Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer - Azure to join our team in Chennai, Tamil Nādu (IN-TN), India (IN)."Job Duties: Key Responsibilities:•  ...

  • Data Analyst

    4 weeks ago

    Chennai, Tamil Nadu, India NTT DATA Full time

    Req ID 324657 NTT DATA strives to hire exceptional innovative and passionate individuals who want to grow with us If you want to be part of an inclusive adaptable and forward-thinking organization apply now We are currently seeking a Data Analyst to join our team in Chennai Tamil Nxc4x81du IN-TN India IN Key Responsibilities ...

  • Eucloid - Senior/Lead Data Engineer - Python

    2 weeks ago

    Chennai, Tamil Nadu, India Eucloid Data Solutions Pvt Ltd Full time

    Job Description : Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications. The ideal candidate will support development of data infrastructure for our clients by participating in activities which may include starting from up- stream and down-stream technology selection to designing and building of...

  • Data & Business Insights Analyst

    2 weeks ago

    Chennai, Tamil Nadu, India NTT DATA Full time

    Key Responsibilities Designing data models using Power BI Desktop and DAX Data Analysis Expressions Cleaning and transforming data using Power Query Editor Integrating data from multiple sources SQL Excel cloud services APIs Creating interactive dashboards and reports tailored to business needs Implementing visualizations that clearly communicate...

  • Data Engineer

    7 hours ago

    Chennai, Tamil Nadu, India Seven Consultancy Full time

    JOB DETAILS 1.Experience in Big Data technologies and working exposure in at least one cloud platform on related data services (AWS / Azure / GCP)2.Hands-on experience with the Hadoop stack – HDFS, sqoop, kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, hive, oozie, airflow and other components required in building end to end data...

  • Data Engineer

    2 days ago

    Chennai, Tamil Nadu, India xponent Full time

    xponent.ai is on a mission to empower aspiring enterprises to be their best and achieve exceptional business outcomes by unlocking the innovation and transformative power of data, analytics, and AI.We are building a Data and AI first company which will deliver innovative and impactful, sustainable Data and AI led solutions for its clients. We are open,...

  • Devops Lead Engineer

    3 weeks ago

    Chennai, Tamil Nadu, India NTT DATA Full time

    Req ID 304648NTT DATA strives to hire exceptional innovative and passionate individuals who want to grow with us If you want to be part of an inclusive adaptable and forward-thinking organization apply now We are currently seeking a DevOps Lead Engineer to join our team in Chennai Tamil Nxc4x81du IN-TN India IN Once You Are Here You Will Be...

  • Data Engineer

    3 weeks ago

    Chennai, Tamil Nadu, India NatWest Group Full time

    Join us as a Data Engineer This is an exciting opportunity to use your technical expertise to collaborate with colleagues and build effortless digital first customer experiences Youll be simplifying the bank by developing innovative data driven solutions using insight to be commercially successful and keeping our customers and the banks data safe and...

  • Systems Engineering Specialist

    2 weeks ago

    Chennai, Tamil Nadu, India NTT Data Full time

    Job DescriptionNTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Systems Engineering Specialist to join our team in Chennai, Tamil Ndu (IN-TN), India (IN).- Work closely with the...