Data Engineer

4 weeks ago

Vijayawada, Andhra Pradesh, India VSquare Systems Pvt. Ltd. Full time

Job Description:

We are looking for a highly skilled Databricks PySpark Developer to join our data platform implementation team. In this role, you will be instrumental in designing, developing, and maintaining ETL processes to ensure efficient extraction, transformation, and loading of data from various sources into data lake and data warehouse. You will work closely with data engineers, data scientists, and business intelligence teams to build and optimize data workflows that support the project's analytics and reporting needs.

Key Responsibilities:

1. ETL Development:

  • Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.
  • Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
  • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
  • Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers.
  • Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.

2. Data Pipeline Management:

  • Hands On experience in creating advanced data pipelines using databricks workflows
  • Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
  • Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.

3. Data Analysis and Query Optimization:

  • Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.

4. Project Coordination and Continuous Improvement:

  • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
  • Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.
  • Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.

Required Qualifications:

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • Minimum of 3 years of experience as an ETL developer, with a strong focus on Databricks PySpark development.
  • Proficiency in Python programming, with extensive experience in developing and debugging Databricks PySpark applications.
  • In-depth understanding of Spark architecture and internals, with hands-on experience in Spark RDDs, DataFrames, and Spark SQL.
  • Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation, and analysis.
  • Proven experience in working with large-scale data warehousing and ETL frameworks.
  • Strong problem-solving skills and the ability to troubleshoot and resolve ETL process issues.
  • Excellent communication and collaboration skills, with the ability to work effectively in a team environment.

Preferred Qualifications:

  • Experience with cloud platforms with a preference for AWS.
  • Experience with data platform tools such as DataBricks, Snowflake, and Tableau.
  • Demonstrated ability to implement best practices for ETL processes and data management.
  • Strong understanding of data governance and data quality principles.
  • Relevant certifications in Databricks PySpark, Spark SQL, or related technologies.

  • Data Engineer

    4 weeks ago

    Vijayawada, Andhra Pradesh, India Moutups Full time

    Job Title: Data Engineer (Remote)Job Type: Full-TimeAbout the Role: We are looking for a highly motivated and detail-oriented Data Engineer to join our remote team. In this role, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure that support real-time and batch data processing. You will collaborate...

  • Digital Marketing Associate

    1 week ago

    Vijayawada, Andhra Pradesh, India Political Engineer Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Company DescriptionTechnical Avengers Pvt. Ltd. specializes in precision-driven strategies, visually-striking branding, and high-performance digital campaigns. Focused on delivering measurable ROI, we ensure clarity, speed, trust, and result-oriented execution. Every solution is tailored, data-backed, and aimed at amplifying reach, boosting engagement, and...

  • Data Engineer

    2 weeks ago

    Vijayawada, Andhra Pradesh, India People Prime Worldwide Full time

    About Company :Our client is a trusted global innovator of IT and business services. They help clients transform through consulting, industry solutions, business process services, digital & IT modernization and managed services. Our client enables them, as well as society, to move confidently into the digital future. We are committed to our clients'...

  • Technical Support Engineer(Data Storage)

    3 weeks ago

    Vijayawada, Andhra Pradesh, India Smart Moves Consultants Full time

    Job Title: Support Engineer (Data Storage )Location: Sion, MumbaiAbout Us: We are leading manufacturer of workstations, servers, and data storage solutions with over a decade of experience in delivering enterprise-grade technology to prominent clients, including government and corporate sectors.Job DescriptionPosition: Support Engineer (Data...

  • Senior Big Data Engineer

    2 weeks ago

    Vijayawada, Andhra Pradesh, India Veltris Full time

    Veltris is a Digital Product Engineering Services partner committed to driving technology-enabled transformation across enterprises, businesses, and industries. We specialize in delivering next-generation solutions for sectors including healthcare, technology, communications, manufacturing, and finance. With a focus on innovation and acceleration, Veltris...

  • Lead Aws Data Engineer

    3 weeks ago

    Vijayawada, Andhra Pradesh, India ACL Digital Full time

    Lead Data Engineer7+ yrs expLocation: Balewadi, PuneImmediate joiners only.ResponsibilitiesMulti-year work experience as a Platform Lead, Delivery Leader or similar role.Good working knowledge of Aws Services, Python, Pyspark, data modelling.Knowledge of columnar and NoSQL databases, predictive analytics, data visualization, and unstructured data.Knowledge...

  • Data Engineer

    4 weeks ago

    Vijayawada, Andhra Pradesh, India 7hillsTS Full time

    Experience : 5+ YearsLocation : BangaloreApache Trino JD:Strong hands-on experience with Trino (PrestoSQL) in production environments.Proficiency in SQL performance tuning and query optimization.Experience with distributed computing frameworks and large-scale data systems.Knowledge of Hive Metastore, Iceberg, Delta Lake, HDFS, Kafka, S3, GCS, or...

  • Data Scientist

    2 weeks ago

    Vijayawada, Andhra Pradesh, India 72 Dragons Full time

    About the company72 Dragons is a global worldwide production company as well as a set of global services in production, social media, software, data science, design, and curation. We are based and work in China, India, Qatar, and USA.Our company has 3 core areas of business -ArtFilmHealthJob DescriptionWe are looking for a Data Analyst with strong front-end...

  • Network Engineer- L3

    16 hours ago

    Vijayawada, Andhra Pradesh, India Pi DATACENTERS Pvt. Ltd. Full time US$ 90,000 - US$ 1,20,000 per year

    Position: Network Engineer- L3 (Data Center)Location: VijayawadaReports to: Director- Operations.CTC: As per the market standardNotice Period: Immediate JoinersJob SummaryThe Data Center Network Engineer L3 is responsible for the design, implementation, management, and optimization of data center network infrastructure. This role involves working with...

  • Senior Data Analyst

    3 weeks ago

    Vijayawada, Andhra Pradesh, India Nityo Infotech Full time

    Job ResponsibilitiesDevelop a strong understanding of internal stakeholders' data and analytical needs, with a focus on customer-facing leadership models.Advocate for the integration of local customer insights into strategic planning and resource allocation decisions.Partner with US Sales Force Operations to drive key business processes and collaborate on...