SPARK Data Onboarding Engineer

19 hours ago

India Photon Group Full time ₹ 12,00,000 - ₹ 36,00,000 per year
Description

Job Title: PySpark Data Engineer

Summary:

We are seeking a skilled PySpark Data Engineer to join our team and drive the development of robust data processing and transformation solutions within our data platform. You will be responsible for designing, implementing, and maintaining PySpark-based applications to handle complex data processing tasks, ensure data quality, and integrate with diverse data sources. The ideal candidate possesses strong PySpark development skills, experience with big data technologies, and the ability to work in a fast-paced, data-driven environment.

Key Responsibilities: Data Engineering Development:

  • Design, develop, and test PySpark-based applications to process, transform, and analyze large-scale datasets from various sources, including relational databases, NoSQL databases, batch files, and real-time data streams.
  • Implement efficient data transformation and aggregation using PySpark and relevant big data frameworks.
  • Develop robust error handling and exception management mechanisms to ensure data integrity and system resilience within Spark jobs.
  • Optimize PySpark jobs for performance, including partitioning, caching, and tuning of Spark configurations.

Data Analysis and Transformation:

  • Collaborate with data analysts, data scientists, and data architects to understand data processing requirements and deliver high-quality data solutions.
  • Analyze and interpret data structures, formats, and relationships to implement effective data transformations using PySpark.
  • Work with distributed datasets in Spark, ensuring optimal performance for large-scale data processing and analytics.

Data Integration and ETL:

  • Design and implement ETL (Extract, Transform, Load) processes to ingest and integrate data from various sources, ensuring consistency, accuracy, and performance.
  • Integrate PySpark applications with data sources such as SQL databases, NoSQL databases, data lakes, and streaming platforms

Qualifications and Skills:

  • Bachelor's degree in Computer Science, Information Technology, or a related field.
  • 5+ years of hands-on experience in big data development, preferably with exposure to data-intensive applications.
  • Strong understanding of data processing principles, techniques, and best practices in a big data environment.
  • Proficiency in PySpark, Apache Spark, and related big data technologies for data processing, analysis, and integration.
  • Experience with ETL development and data pipeline orchestration tools (e.g., Apache Airflow, Luigi).
  • Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions.
  • Excellent communication and collaboration skills to work effectively with data analysts, data architects, and other team members.

  • Data Engineer

    2 days ago

    India Crayon Data Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Job DescriptionData Engineer – ChennaiBuild the foundation of data intelligence — design, develop, and optimize data pipelines that power AI-driven insights.Location: Chennai, IndiaExperience: 2–3 years in data engineering or data-driven product developmentRole OverviewAs a Data Engineer, you will be responsible for building and maintaining scalable...

  • Azure Data Engineer

    4 weeks ago

    Pune, India Fragma Data Systems Full time

    Job Description Technology Skills - Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub. - Experience in...

  • Senior Spark Data Engineer

    3 weeks ago

    Pune, India Barclays Full time

    Job Description Join us as a Senior Spark Data Engineer at Barclays, where you'll take part in the evolution of our digital landscape, driving innovation and excellence. You'll harness cutting-edge technology to revolutionize our digital offerings, ensuring unparalleled customer experiences. As a part of the team, you will deliver technology stack, using...

  • SPARK Data Reconciliation Engineer

    22 hours ago

    India Photon Group Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    DescriptionJob Title: PySpark Data Reconciliation Engineer7+ years of relevant experience in technology and development.Technical skillset required:(1) Language: Java or Python(2) Experience with BigData/Hadoop/Spark/Kafka(3) Experience with APIs and microservices architecture (4) UI Development and integration experience would be a plus.Domain...

  • Big Data Engineer

    4 weeks ago

    Mumbai, India Strategic HR Solutions Full time

    Job Description Spark Scala Developer Location : Bengaluru, Mumbai Employment Type : Full-time What Were Looking For Were hiring a Spark Scala Developer who has real-world experience working in Big Data environments, both on-prem and/or in the cloud. You should know how to write production-grade Spark applications, fine-tune performance, and work fluently...

  • Software Engineer III

    3 weeks ago

    Hyderabad, India Chase Bank Full time

    Job Description We have an exciting and rewarding opportunity for you to take your software engineering career to the next level. As a Software Engineer III at JPMorgan Chase within the Consumer and Community Banking- Data Technology, you serve as a seasoned member of an agile team to design and deliver trusted market-leading technology products in a secure,...

  • Software Engineer III

    3 weeks ago

    Hyderabad, India JPMorganChase Full time

    Job Description Job Description We have an exciting and rewarding opportunity for you to take your software engineering career to the next level. As a Software Engineer III at JPMorgan Chase within the Consumer and Community Banking- Data Technology, you serve as a seasoned member of an agile team to design and deliver trusted market-leading technology...

  • NucleusTeq - Big Data Engineer - Spark/Scala

    15 hours ago

    Anywhere in India/Multiple Locations NucleusTeq Consulting Private Limited Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role : Big Data Engineer Responsibilities : - Big Data Engineer with 5-6 years of experience - Experience in leading a team of engineers and a good attitude toward learning the domain and implementation - Strong exposure and expertise in Spark (Primary), Scala/Java (Scala Primary), Airflow Orchestration and AWS. - Finalizing the scope of the...

  • ▷ Apply Now: Azure Data Engineer

    4 days ago

    India LTIMindtree Full time

    Role Senior Data Engineer 8 years of experience Key responsibilities Build reusable utilities templates and automation pipelines Design scalable data engineering frameworks standards and best practices Provide architectural guidance cost optimization and performance tuning support for Data Engineering solutions mainly on Azure Evaluate and onboard new...

  • Big Data Engineer/DE with SCALA Spark

    4 weeks ago

    Nagpur, Maharashtra, India, Maharashtra HCLTech Full time

    We are Hiring Big Data Engineer/Data Engineer with SCALA & SparkLocation: NagpurNP: 60 days & LessExperience: 5-9 YearsSkills:Preferred Skillset: Spark, Scala, Linux Based Hadoop Ecosystem (HDFS, Impala, Hive, HBase, etc.), SQL, Jenkins, and Unix Commands.Experience in Big data technologies , real time data processing platform(Spark Streaming - Kafka)...