Data Engineer

2 weeks ago

Bengaluru, Karnataka, India O2F Info Solutions Pvt. Ltd. Full time

Job Summary :

We are seeking a highly skilled Senior Data Engineer with 4 to 8 years of experience in building robust data pipelines and working extensively with PySpark to join our data engineering team.

Key Responsibilities :

Data Pipeline Development :

- Design, build, and maintain scalable data pipelines using PySpark to process large datasets and support data-driven applications and analytics.

ETL Process Automation :

- Develop and automate ETL (Extract, Transform, Load) processes using PySpark, ensuring efficient data processing, transformation, and loading from diverse sources into data lakes, warehouses, or databases.

Distributed Computing with PySpark :

- Leverage Apache Spark and PySpark to process large-scale data in a distributed computing environment, optimizing for performance and scalability.

Cloud Data Solutions :

- Develop and deploy data pipelines and processing frameworks on cloud platforms (AWS, Azure, GCP) using native tools like AWS Glue, Azure Databricks, or Google Dataproc.

Data Integration & Transformation :

- Integrate data from various internal and external sources, ensuring data consistency, quality, and reliability throughout the pipeline.

Performance Optimization :

- Optimize PySpark jobs and pipelines for faster data processing, handling large volumes of data efficiently with minimal latency.

- Proven experience as a Data Engineer or similar role, with a strong background in database development, ETL processes, and software development.

- Proficiency in SQL and scripting languages such as Python, with experience working with relational databases.

- Proficiency in dataProc (PySpark), Pandas or other data processing libraries

- Experience with data modeling, schema design, and optimization techniques for scalability.

- Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues and optimize data processing pipelines for scale

Required Qualifications :

- 4-8 years of experience in data engineering, with a strong focus on PySpark and large-scale data processing.

Technical Skills :

- Expertise in PySpark for distributed data processing, data transformation, and job optimization.

- Strong proficiency in Python and SQL for data manipulation and pipeline creation.

- Hands-on experience with Apache Spark and its ecosystem, including Spark SQL, Spark Streaming, and PySpark MLlib.

- Solid experience working with ETL tools and frameworks, such as Apache Airflow or similar orchestration tools.

(ref:hirist.tech)

  • Data Entry Operator

    2 days ago

    Bengaluru, Karnataka, India Design Engineer Full time ₹ 2,64,000 per year

    We are seeking a dedicated and detail-oriented Data Entry Operator (DEO) to support R&D projects. The role involves accurate data entry, documentation, and maintenance of research records in secure systems. The DEO will assist scientists, engineers, and administrative staff by ensuring timely and error-free handling of project information.Key...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Req ID:321800We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karntaka (IN-KA), India (IN)."Job Duties: Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical...

  • Technical Lead- Data Engineer

    2 days ago

    Bengaluru, Karnataka, India Astar Data Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Kindly find the Job Description Below.Job Title:Technical Lead- Data EngineerLocation: BangaloreYears of Experience: 8+ years of experienceSigmoidworks with a variety of clients from start-ups to fortune 500 companies. We are looking for a detailed oriented self-starter to assist our engineering and analytics teams in various roles as a Software...

  • Data Engineer

    2 days ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Job Duties:Migrate ETL workflows from SAP BODS to AWS Glue/dbt/Talend. Develop and maintain scalable ETL pipelines in AWS. Write PySpark scripts for large-scale data processing. Optimize SQL queries and transformations for AWS PostgreSQL. Work with Cloud Engineers to ensure smooth deployment and performance tuning. Integrate data pipelines with existing Unix...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    We are currently seeking a Data Engineer (HRIS) to join our team in Bangalore, Karntaka (IN-KA), India (IN).Job Duties: Key Responsibilities -Data Management - Develop data mapping specifications and transformation rules Perform data cleansing, validation, and reconciliation activities Create and execute data conversion scripts and processes Document data...

  • Data Engineer

    6 days ago

    Bengaluru, Karnataka, India NTT DATA North America Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karnātaka (IN-KA), India (IN)."Job Duties: Key...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA North America Full time

    Job DescriptionReq ID: 340517NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Bangalore, Karntaka (IN-KA), India (IN).- Hands-on experience in...

  • Data Visualization Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 1,04,000 - ₹ 13,08,780 per year

    Req ID: 340987NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Visualization Engineer - Power BI to join our team in Bangalore, Karnātaka (IN-KA), India (IN). "Job Duties: As...

  • AWS Sr Data Engineer

    2 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a AWS Sr Data Engineer to join our team in Bangalore or Remote, Karnātaka (IN-KA), India (IN). Position Location:OUS with...

  • High Salary) Data Engineer

    3 weeks ago

    Bengaluru, Karnataka, India NTT Data Full time

    Job DescriptionNTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer (HRIS) to join our team in Bangalore, Karntaka (IN-KA), India (IN).Job Duties: Key ResponsibilitiesData...