Netscribes - Data Engineer - Python/PySpark

24 hours ago

Bengaluru, Karnataka, India NETSCRIBES DATA INSIGHTS PRIVATE LIMITED Full time

We are seeking a highly motivated Data Engineer to join our data engineering team. In this role, you will design and develop scalable data pipelines and solutions leveraging Databricks, Python, PySpark, and SQL. You'll work closely with cross-functional teams to ensure clean, high-quality data is available for analytics, reporting, and machine learning use :

- Design, build, and maintain reliable and scalable ETL/ELT pipelines using PySpark, SQL, and Databricks.

- Collaborate with data analysts, scientists, and business stakeholders to gather requirements and build data solutions.

- Develop data models and support the creation of data lakes and data warehouses.

- Implement data quality checks, monitoring, and error-handling mechanisms.

- Optimize the performance of data workflows and queries for efficiency and scalability.

- Manage data integrations from diverse sources such as APIs, cloud storage, RDBMS, and flat files.

- Maintain documentation of pipelines, processes, and data flows.

Requirements :

- Proficient in Python with strong knowledge of libraries related to data manipulation (e. g., pandas, pyodbc, pyspark).

- Hands-on experience with PySpark for large-scale data processing.

- Strong command over SQL, including complex joins, window functions, and optimization.

- Experience with Databricks for building notebooks, jobs, and managing clusters.

- Familiarity with data lakehouse concepts, Delta Lake, and data versioning.

- Understanding of data pipeline orchestration tools (e. g., Airflow, Azure Data Factory).

- Good problem-solving skills and the ability to work in a fast-paced environment.

Preferred Qualifications :

- Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, or a related field.

- Experience with cloud platforms like Azure, AWS, or GCP.

- Knowledge of data governance, data cataloging, and metadata management.

- Experience with CI/CD pipelines and version control (Git).

- Exposure to ML pipelines and real-time data processing (Kafka, Spark Streaming) is a plus.

(ref:hirist.tech)

  • Senior Python and PySpark Data Engineer

    1 day ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time

    Senior Python and PySpark Data EngineerWe are seeking an experienced Senior Python and PySpark Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining complex data processing systems using Python and PySpark.Your primary focus will be on developing efficient and scalable...

  • Lead Data Engineer

    1 week ago

    Bengaluru, Karnataka, India beBeeData Full time

    Job Title: Python Pyspark DeveloperWe are seeking a highly skilled Python Pyspark Developer to join our team. As a key member of our development team, you will be responsible for designing and implementing scalable data processing solutions using Python and PySpark.Key Responsibilities:Design and develop high-performance data pipelines using Python and...

  • Data Processing Engineer

    2 weeks ago

    Bengaluru, Karnataka, India beBee Careers Full time

    Job Title: Data Processing Engineer - Python and PySpark Developer",

  • Python and PySpark Specialist

    6 days ago

    Bengaluru, Karnataka, India beBeePySpark Full time

    Job Title:Pyspark Developer RoleJob Description:We are seeking a highly skilled Python and PySpark professional to join our team. This position will be responsible for developing and maintaining complex data processing systems using Python and PySpark, ensuring high performance and scalability.Key Responsibilities:Develop and maintain robust data processing...

  • Pyspark Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Pan Asia HR Solutions Full time

    Job Description : We are looking for a skilled PySpark Developer with expertise in ETL processes, Python, and SQL to design, develop, and optimize large-scale data processing pipelines. The ideal candidate will have hands-on experience in PySpark for data transformation, aggregation, and analysis in a Big Data environment. Key Responsibilities : - Design,...

  • Netscribes - LLM Engineer - Generative AI

    1 week ago

    Bengaluru, Karnataka, India NETSCRIBES DATA INSIGHTS PRIVATE LIMITED Full time

    Responsibilities :- Design, fine-tune, and deploy Large Language Models using Vertex AI.- Develop end-to-end GenAI pipelines including data preprocessing, model training, evaluation, and inference.- Integrate LLMs into applications via APIs and custom interfaces.- Optimize and monitor model performance using Vertex AI tools and best practices.- Collaborate...

  • Python/pyspark

    2 weeks ago

    Bengaluru, Karnataka, India Tata Consultancy Services Full time

    Dear AssociateGreetings from TATA Consultancy ServicesThank you for expressing your interest in exploring a career possibility with the TCS Family.We have a job opportunity for Python/Pyspark at Tata Consultancy Services.Hiring For: Python/PysparkInterview date: 07-May-25Location: BangaloreExperience: 4-6 yearsMust Have:Develop, test, and deploy scalable...

  • Pyspark Engineer

    4 days ago

    Bengaluru, Karnataka, India Pan Asia HR Solutions Full time

    Job Description : We are looking for a skilled PySpark Developer with expertise in ETL processes, Python, and SQL to design, develop, and optimize large-scale data processing pipelines. The ideal candidate will have hands-on experience in PySpark for data transformation, aggregation, and analysis in a Big Data environment.Key Responsibilities : - Design,...

  • Pyspark Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Pan Asia HR Solutions Full time

    Job Description : We are looking for a skilled PySpark Developer with expertise in ETL processes, Python, and SQL to design, develop, and optimize large-scale data processing pipelines. The ideal candidate will have hands-on experience in PySpark for data transformation, aggregation, and analysis in a Big Data environment.Key Responsibilities : - Design,...

  • Python Pyspark Developer

    2 weeks ago

    Bengaluru, Karnataka, India Virtusa Full time

    Job DescriptionSkills Required:Python, PySpark, Azure Databricks, Shell Scripting, DB2, CI/CD (GIT, Jenkins), Java understandingExperience & Requirements- 5+ years of professional Python/PySpark development experience- Strong experience with FastAPI or similar framework (Flask, Django REST)- Deep understanding of REST API design principles- Expertise in...