Senior Data Engineer

3 days ago


Guntur, Andhra Pradesh, India Xped pvt Ltd Full time ₹ 12,00,000 - ₹ 36,00,000 per year

JOB DESCRIPTION :

Responsibilities :

- Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights.

- Optimize PySpark jobs and data pipelines for performance, scalability, and reliability.

- Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows.

- Ensure data quality and integrity throughout all stages of data processing.

- Develop and implement strategies to mitigate financial risks associated with data transformation and aggregation.

- Troubleshoot and debug issues related to data pipelines and processing.

- Ensure compliance with regulatory requirements and industry standards in all data processing activities.

- Implement best practices for data security, compliance, and privacy within Azure environment.

- Document technical specifications, data flows, and solution architecture

- Design and build reusable components, frameworks and libraries at scale to support analytics products

- Design and implement product features in collaboration with business and Technology stakeholders

- Design and develop scalable data pipelines using Azure Databricks and PySpark. - Transform raw data into actionable insights through advanced data engineering techniques.

- Build, deploy, and maintain machine learning models using MLlib, TensorFlow, and MLflow.

- Optimize data integration workflows from Azure Blob Storage, Data Lake, and SQL/NoSQL sources.

- Execute large-scale data processing using Spark Pools, fine-tuning configurations for performance and cost-efficiency.

- Collaborate with data scientists, analysts, and business stakeholders to deliver robust data solutions.

- Maintain and enhance Databricks notebooks and Delta Lake architectures.

- Architect, build, and maintain scalable and reliable data pipelines from diverse data sources.

- Design effective data storage, retrieval mechanisms, and data models to support analytics and business needs.

- Implement data validation, transformation, and quality monitoring processes.

- Collaborate with cross-functional teams to deliver impactful, data-driven solutions.

- Proactively identify bottlenecks and optimize existing workflows and processes. - Provide guidance and mentorship to junior engineers in the team.

- Anticipate, identify and solve issues concerning data management to improve data quality

- Clean, prepare and optimize data at scale for ingestion and consumption

- Drive the implementation of new data management projects and re-structure of the current data architecture

- Implement complex automated workflows and routines using workflow scheduling tools

- Build continuous integration, test-driven development and production deployment frameworks

- Drive collaborative reviews of design, code, test plans and dataset implementation performed by other data engineers in support of maintaining data engineering standards

- Analyze and profile data for the purpose of designing scalable solutions

- Troubleshoot complex data issues and perform root cause analysis to proactively resolve product and operational issues

- Mentor and develop other data engineers in adopting best practices

Qualifications :

- 3 years experiencing developing scalable Big Data applications or solutions on distributed platforms

- Able to partner with others in solving complex problems by taking a broad perspective to identify innovative solutions

- Strong skills building positive relationships across Product and Engineering.

- Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

- Able to quickly pick up new programming languages, technologies, and frameworks

- Experience working in Agile and Scrum development process

- Experience working in a fast-paced, results-oriented environment

- Experience in Amazon Web Services (AWS) or other cloud platform tools

- Experience working with Data warehousing tools, including Dynamo DB, SQL, Amazon Redshift, and Snowflake

- Experience architecting data product in Streaming, Serverless and Microservices Architecture and platform.

- Experience working with Data platforms, including EMR, Data Bricks etc

- Experience working with distributed technology tools, including Spark, Presto, Scala, Python, Databricks, Airflow

- Working knowledge of Data warehousing, Data modelling, Governance and Data Architecture

- Working knowledge of Reporting & Analytical tools such as Tableau, Quick site etc.

- Demonstrated experience in learning new technologies and skills

- Bachelor's degree in Computer Science, Information Systems, Business, or other relevant subject area


  • AWS Data Engineer

    4 weeks ago


    Guntur, Andhra Pradesh, India Teamware Solutions Full time

    About the CompanyGreetings from Teamware Solutions a division of Quantum Leap Consulting Pvt. LtdAbout the RoleWe are hiring an AWS Data EngineerLocation: BangaloreWork Model: HybridExperience: 5-7 yearsNotice Period: Immediate- 15 daysOpen roles: 22Data Engineer: Data Engineer, DevOps, Data Modeller, QA, Business systems analyst, Data Analyst, DBAAs an AWS...

  • AWS Data Engineer

    4 weeks ago


    Guntur, Andhra Pradesh, India 9NEXUS Full time

    We are seeking a skilled AWS Data Engineer to design, build, and maintain scalable data pipelines and ETL solutions in a cloud environment. The ideal candidate will have hands-on experience with AWS EMR, Glue, , PySpark, Airflow and Python and will work closely with data analysts, data scientists, and other engineering teams to support data-driven...

  • Gcp Data Engineer

    3 weeks ago


    Guntur, Andhra Pradesh, India Tech T7 Innovations Full time

    Job Title: Google Cloud (GCP) Data EngineerLocation: RemoteJob Type: Full-TimeExperience Level: Minimum 5 years +Mandatory Skills: GCS + Google BigQuery + Airflow/Composer + PythonCompany Description:Tech T7 Innovations is a company that provides IT solutions to clients worldwide. The team consists of highly skilled and experienced professionals who are...


  • Guntur, Andhra Pradesh, India Imetris Corporation Full time

    Imetris is hiring 3 Software Engineer in Test having experience in Automation coding in C# using Selenium ( not JAVA)Job Title: Mid to senior Software Engineer in TestWorksite: Remote ( preferable Mumbai, Pune)Experience Level: Senior (5+ years)We are looking for a Offshore Software Engineer in Test with strong experience in C#, Selenium, API testing, and...


  • Guntur, Andhra Pradesh, India People Prime Worldwide Full time

    About Company: Our Client Corporation provides digital engineering and technology services to Forbes Global 2000 companies worldwide.Our Engineering First approach ensures we can execute all ideas and creatively solve pressing business challenges.With industry expertise and empowered agile teams, we prioritize execution early in the process for impactful...


  • Guntur, Andhra Pradesh, India NIT DATA Full time

    HiPlease find the below JDRole: Electrical engineer – Engineering designMandatory Experience: Oil & Gas DomainExperience and essential skills3–5 years of experienceElectronic circuit design (analog & digital)Circuit simulation (PSpice / LTSpice / Cadence)PCBA testing, troubleshooting & mixed-signal electronicsRework/repair, grounding & shielding...

  • Data Quality

    4 weeks ago


    Guntur, Andhra Pradesh, India People Prime Worldwide Full time

    About the jobAbout Client:Our Client is a global IT services company headquartered in Southborough, Massachusetts, USA. Founded in 1996, with a revenue of $1.8B, with 35,000+ associates worldwide, specializes in digital engineering, and IT services company helping clients modernize their technology infrastructure, adopt cloud and AI solutions, and accelerate...


  • Guntur, Andhra Pradesh, India DBiz Full time

    Job Description:We are seeking a highly skilled and experienced Senior Unreal Artist to join our innovative team.The ideal candidate will have extensive expertise in Unreal Engine development, with additional skills in CAD, Maya, Max, and Revit.This role requires a creative and technically proficient individual who can lead projects, collaborate effectively,...


  • Guntur, Andhra Pradesh, India ELDYNE Full time

    Position: Senior Executive, HR ManagementLocation: KolkataAbout us:Eldyne, based in Kolkata since 1987, is a pioneer in modernised digital and automated Railway Signalling Systems. Our expertise spans concept-to-commissioning including design, development, integration, testing and maintenance. With over three decades of experience and multiple projects...


  • Guntur, Andhra Pradesh, India FNZ Full time

    Role DescriptionThis role is technical in nature and is responsible for providing high quality and timely IT infrastructure support to ensure the availability, stability and performance the FNZ datacentres, platforms and corporate systems.AIX Server managementSAN and storage array management would be advantageousBackup, Replication, HA and Recovery...