Senior Data Engineer

3 weeks ago

Hyderabad, Telangana, India Risk Resources LLP Full time
Job Summary

We are seeking a highly skilled Pyspark Databricks professional to join our team at Risk Resources LLP. The ideal candidate will have a strong background in developing and implementing data processing pipelines using Pyspark on the Databricks platform.

Key Responsibilities
  • Data Pipeline Development: Design and develop efficient data processing pipelines using Pyspark and Databricks.
  • Performance Optimization: Optimize and tune the performance of Spark jobs and clusters to ensure efficient data processing.
  • Data Transformations: Implement data transformations and aggregations for analysis and reporting.
  • Collaboration: Collaborate with data engineers and data scientists to integrate data from various sources into the pipelines.
  • ETL Process Development: Develop and maintain ETL processes to extract, transform, and load data into the data lake.
  • Job Monitoring: Monitor and troubleshoot data processing jobs to ensure reliability and stability.
  • Code Review: Conduct code reviews and provide technical guidance to junior team members.
  • Best Practices: Implement best practices for data pipeline development and documentation.
  • Requirements Gathering: Work with cross-functional teams to gather and analyze requirements for data processing.
  • Technology Evaluation: Participate in the evaluation and implementation of new technologies related to big data processing.
Requirements
  • Education: Bachelor's degree in Computer Science, Engineering, or a related field.
  • Experience: Proven experience in developing data processing pipelines using Pyspark and Databricks.
  • Programming Skills: Strong proficiency in Python programming language.
  • Big Data Technologies: Hands-on experience with Big Data technologies and distributed computing.
  • Spark Optimization: Expertise in optimizing Spark jobs and clusters for performance.
  • ETL Processes: Knowledge of ETL processes and data warehousing concepts.
  • Data Integration: Experience with data integration and data modeling.
  • Troubleshooting: Ability to troubleshoot and debug complex data processing issues.
  • SQL and NoSQL: Excellent understanding of SQL and NoSQL databases.
  • Communication Skills: Strong communication and collaboration skills to work effectively in a team environment.
  • Certifications: Certifications in Pyspark Databricks or related technologies are a plus.
  • Cloud Platforms: Familiarity with cloud platforms such as AWS, Azure, or Google Cloud is beneficial.

python,bigdata,etl,spark,data

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India NTT DATA Business Solutions Full time

    A Senior Data Engineer plays a pivotal role in technology-driven organizations, responsible for designing, building, and managing the infrastructure and tools that enable efficient processing and analysis of large data sets.Responsibilities:Designing and Implementing ETL Processes:Develop and maintain scalable data pipelines.Create efficient Extract,...

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, Telangana, India The Modern Data Company Full time

    Job Description :The Senior Data Engineer will provide technical expertise in needs identification, data modeling, data movement, and transformation mapping (source to target), automation and testing strategies, translating business needs into technical solutions with adherence to established data guidelines and approaches from a business unit or project...

  • Big Data Engineer

    3 weeks ago

    Hyderabad, Telangana, India data Full time

    Overview: We're seeking a skilled Big Data Engineer to join our team at data.ai. As a key member of our engineering team, you will be responsible for designing and implementing innovative data solutions that drive business growth. If you're passionate about working with large data sets, have experience with big data technologies, and are eager to...

  • Big Data Engineer, Data Modeling Expert

    4 weeks ago

    Hyderabad, Telangana, India data Full time

    Unlock the Power of Data with UsWe're seeking an experienced Big Data Engineer to join our team and contribute to the development of innovative analytics and data products. As a key member of our engineering team, you will work closely with Data Scientists, Data Engineers, Product Managers, and Software Architects to design and implement complex data product...

  • Digital Engineering Senior Software Engineer

    4 weeks ago

    Hyderabad, Telangana, India NTT DATA Full time

    Job Title: Senior Data Migration ConsultantWe are seeking an experienced Senior Data Migration Consultant to join our team at NTT DATA Services. As a Senior Data Migration Consultant, you will be responsible for performing data migration of SAP Asset Master Data objects from SAP ECC to SAP S4/HANA.Key Responsibilities:12 years of experience in SAP Data...

  • Digital Engineering Senior Engineer

    1 month ago

    Hyderabad, Telangana, India NTT DATA Full time

    Job Title: Senior Syniti ADM ConsultantAbout the RoleWe are seeking an experienced Senior Syniti ADM Consultant to join our team. As a key member of our data migration team, you will be responsible for performing data migration of SAP Asset Master Data objects from SAP ECC to SAP S4/HANA.Key Responsibilities12 years of experience in SAP Data Cleansing,...

  • Big Data Engineer, Data Modeling Expert

    4 weeks ago

    Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking an experienced Big Data Engineer to join our team at data.ai. As a key member of our engineering team, you will be responsible for designing and implementing complex data product components based on requirements with possible technical solutions.Key ResponsibilitiesDesign and implement data processing programs using Python (e.g.,...

  • Big Data Engineer

    1 month ago

    Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking an experienced Big Data Engineer to join our team in Hyderabad, India. As a key member of our engineering team, you will be responsible for designing and implementing complex data product components based on requirements with possible technical solutions.Key ResponsibilitiesDesign and implement data processing programs using...

  • Senior Digital Engineering Expert

    2 weeks ago

    Hyderabad, Telangana, India NTT DATA Full time

    Job DescriptionReq ID: 295140 NTT DATA is seeking a talented individual to fill the role of Digital Engineering Senior Engineer. We strive to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, we encourage you to apply.We are currently seeking...

  • Data Engineer

    1 month ago

    Hyderabad, Telangana, India NTT DATA Services Full time

    Job Title: Data EngineerNTT DATA Services is seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, building, and testing scalable Big Data ecosystems for our clients.Key Responsibilities:Analyze and understand data sources and APIsDesign and develop methods to connect and collect data from...

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India Experian Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our agile team at Experian. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems that meet the needs of our customers.Key Responsibilities:Design and develop data pipelines to extract, transform, and...

  • Senior Data Migration Developer

    1 month ago

    Hyderabad, Telangana, India NTT DATA Full time

    Job Title: Data Migration Senior DeveloperAbout the Role:We are seeking a highly skilled Data Migration Senior Developer to join our team at NTT DATA Services. As a key member of our team, you will be responsible for developing and implementing data migration solutions using SAP technologies.Key Responsibilities:Develop and configure data migration tools...

  • Digital Engineering Senior Consultant

    4 weeks ago

    Hyderabad, Telangana, India NTT DATA Full time

    Job Title: Senior Syniti ADM ConsultantAbout the Role:We are seeking an experienced Senior Syniti ADM Consultant to join our team. As a key member of our data migration team, you will be responsible for performing data migration of SAP Asset Master Data objects from SAP ECC to SAP S4/HANA.Key Responsibilities:12 years of experience in SAP Data Cleansing,...

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India Experian Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our agile team at Experian. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems that meet the needs of our customers.Key Responsibilities:Design and develop data pipelines using Spark and...

  • Senior Data Engineer

    1 month ago

    Hyderabad, Telangana, India ValueLabs Full time

    Senior Azure Data Engineer Role at ValueLabsWe are seeking a highly skilled Senior Azure Data Engineer to join our team at ValueLabs. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining data pipelines, data warehouses, and data lakes for our organization.Key Responsibilities:Design and develop data...

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India DBS Bank Full time

    Job SummaryDBS Bank is seeking a highly skilled Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems and pipelines.Key ResponsibilitiesDesign and develop data pipelines using Python, Hadoop, Spark, and Hive.Develop and maintain data warehouses and data...

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India DBS Bank Full time

    Job SummaryDBS Bank is seeking a highly skilled Senior Data Engineer to join our Group Technology and Operations team. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data pipelines and architectures.Key ResponsibilitiesDesign and develop data pipelines using Python, Hadoop, Spark, and HiveDevelop and...

  • Senior Data Engineer

    1 month ago

    Hyderabad, Telangana, India 451 Discovery Comm. India Full time

    About the Role:We are seeking a highly skilled Senior Data Engineer to join our team at 451 Discovery Comm. India. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems and architectures that support our business operations.Key Responsibilities:Analyze complex data sets to identify trends and...

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India Accellor Full time

    Job Title: Senior Data EngineerAbout the Role:We are seeking a highly skilled Senior Data Engineer to join our team at Accellor. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining ETL pipelines using PySpark Notebooks and Microsoft Fabric.Key Responsibilities:Design and develop ETL pipelines using PySpark Notebooks...

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India Alignity Solutions Full time

    Job Title: Senior Data EngineerJob Summary:We are seeking an experienced Senior Data Engineer to join our team at Alignity Solutions. As a Senior Data Engineer, you will be responsible for architecting, designing, and building data engineering solutions and platforms, including data warehouses and data platforms using Redshift or Snowflake.Key...