Data Engineer

18 hours ago

Vijayawada, Andhra Pradesh, India VSquare Systems Pvt. Ltd. Full time

Job Description:

We are looking for a highly skilled Databricks PySpark Developer to join our data platform implementation team. In this role, you will be instrumental in designing, developing, and maintaining ETL processes to ensure efficient extraction, transformation, and loading of data from various sources into data lake and data warehouse. You will work closely with data engineers, data scientists, and business intelligence teams to build and optimize data workflows that support the project's analytics and reporting needs.

Key Responsibilities:

1. ETL Development:

  • Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.
  • Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
  • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
  • Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers.
  • Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.

2. Data Pipeline Management:

  • Hands On experience in creating advanced data pipelines using databricks workflows
  • Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
  • Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.

3. Data Analysis and Query Optimization:

  • Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.

4. Project Coordination and Continuous Improvement:

  • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
  • Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.
  • Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.

Required Qualifications:

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • Minimum of 3 years of experience as an ETL developer, with a strong focus on Databricks PySpark development.
  • Proficiency in Python programming, with extensive experience in developing and debugging Databricks PySpark applications.
  • In-depth understanding of Spark architecture and internals, with hands-on experience in Spark RDDs, DataFrames, and Spark SQL.
  • Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation, and analysis.
  • Proven experience in working with large-scale data warehousing and ETL frameworks.
  • Strong problem-solving skills and the ability to troubleshoot and resolve ETL process issues.
  • Excellent communication and collaboration skills, with the ability to work effectively in a team environment.

Preferred Qualifications:

  • Experience with cloud platforms with a preference for AWS.
  • Experience with data platform tools such as DataBricks, Snowflake, and Tableau.
  • Demonstrated ability to implement best practices for ETL processes and data management.
  • Strong understanding of data governance and data quality principles.
  • Relevant certifications in Databricks PySpark, Spark SQL, or related technologies.

  • Data Scientist/ML Engineer

    5 days ago

    Vijayawada, Andhra Pradesh, India Quant-data Full time

    We're Hiring: Machine Learning Engineer / Data Engineer (Remote | Full-Time) Build AI-powered credit decisioning systems on Microsoft AzureWe're looking for a Machine Learning Engineer / Data Engineer with 5+ years of experience to join our AI-driven credit lending platform team. In this role, you'll design and deploy scalable ML solutions that power loan...

  • Data Engineer

    2 hours ago

    Vijayawada, Andhra Pradesh, India Moutups Full time

    Job Title: Data Engineer (Remote)Job Type: Full-TimeAbout the Role: We are looking for a highly motivated and detail-oriented Data Engineer to join our remote team. In this role, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure that support real-time and batch data processing. You will collaborate...

  • Data Engineer

    18 hours ago

    Vijayawada, Andhra Pradesh, India beBeeIngestion Full time ₹ 15,00,000 - ₹ 20,00,000

    Ingestion engineers are crucial for our organization's data strategy, responsible for crafting efficient data ingestion pipelines that integrate multiple sources into Databricks.Key Responsibilities:Data Ingestion Pipelines: Develop and optimize data ingestion pipelines for integrating multiple sources into Databricks.CI/CD Pipeline Implementation: Implement...

  • Senior Data Engineer

    8 hours ago

    Vijayawada, Andhra Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    About the RoleAt our organization, we are seeking a skilled Senior Software Engineer/Technical Specialist to join our team. This role is ideal for an individual with expertise in data engineering, including experience with Azure Data Factory (ADF), Databricks, and Synapse Analytics.As a Senior Software Engineer/Technical Specialist, you will be responsible...

  • Principal Data Engineer

    1 day ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data Engineering ProfessionalOverview:Develop, optimize and maintain scalable data pipelines to support business intelligence and analytics initiatives.Description:Key ResponsibilitiesPipeline Development: Design, build and deploy robust ETL/ELT pipelines in Azure Databricks (PySpark, SQL, Delta Lake) to ingest, transform and curate governance and...

  • Cloud Data Engineering Leader

    2 days ago

    Vijayawada, Andhra Pradesh, India beBeeCloudEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Unlock your potential as a Cloud Data Engineering Leader at our innovative organization. We are seeking an experienced professional to spearhead the development of scalable data pipelines and analytics platforms in the cloud.The ideal candidate will lead a team of talented data engineers, guiding them to design and deliver high-performance data solutions...

  • Spatial Data Engineer

    1 day ago

    Vijayawada, Andhra Pradesh, India beBeeData Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Title: Spatial Data EngineerWe are seeking an experienced Spatial Data Engineer to join our team. This role is crucial in the production and management of geospatial data across various projects.The position encompasses tasks such as digitization, geocoding, and conducting surveys when necessary.Key Responsibilities:Geospatial Data Production: Engage in...

  • Senior Data Engineering Strategist

    8 hours ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Data Engineering Strategist A senior data engineering strategist is required to lead and manage a team of experienced data engineers. The ideal candidate should have a strong background in enterprise-level data engineering roles with expertise in OLAP design, data warehousing concepts, and Snowflake. Key Responsibilities: Leverage...

  • Leading Data Systems Engineer

    1 day ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 90,00,000 - ₹ 1,20,00,000

    Data Engineer RoleOur organization is seeking a seasoned Data Engineer to collaborate with our global engineering team on a multi-year project initiative.Key Responsibilities:Cultivate data engineering expertise within the project teamDevelop and maintain large-scale data systems utilizing Palantir Foundry, Workshop, PySpark, and TypescriptRequirements:5-8...

  • Senior Data Engineering Expert

    2 days ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineer Full time US$ 90,000 - US$ 1,20,000

    Job Title: Senior Data EngineerWe are looking for a skilled data engineer to join our team.The ideal candidate will have 5+ years of experience in ETL development and data engineering.In this role, you will be responsible for developing and maintaining data pipelines using FiveTran, DBT Cloud, and custom frameworks.You will work closely with the QC team to...