GCP Data Engineer

5 hours ago


Palakkad, India NASTech, IT Full time
Job Title: GCP Data Engineer


About the Role:

We are seeking an experienced GCP Data Engineer to join our dynamic data engineering team. As a Data Engineer, you will design, develop, and maintain data pipelines on Google Cloud Platform (GCP) to support large-scale data analytics and machine learning initiatives. The ideal candidate will have a strong background in working with BigQueryPython, and Databricks.

Key Responsibilities:
  • Design and Build Data Pipelines: Develop scalable and efficient data pipelines using GCP services such as BigQueryCloud DataflowCloud Storage, and Pub/Sub.
  • BigQuery Optimization: Optimize queries, build materialized views, and ensure the efficiency and performance of large datasets in BigQuery.
  • Data Modeling and ETL: Create and maintain ETL (Extract, Transform, Load) processes, transforming raw data into structured, easily accessible formats for analytics.
  • Automation and Scripting: Write Python scripts to automate data workflows, integrate data sources, and manage metadata for data governance.
  • Databricks Integration: Work with Databricks for data processing, including the use of Apache Spark for batch and stream processing.
  • Collaboration: Collaborate with Data Scientists, Analysts, and other stakeholders to understand data requirements and provide effective solutions.
  • Data Quality and Governance: Ensure data integrity, quality, and security by following best practices for data governance, monitoring, and logging.
Required Skills and Qualifications:
  • Experience with GCP Services: Hands-on experience with Google Cloud Platform tools, especially BigQueryDataflowPub/SubCloud StorageCloud Functions, and Cloud Composer.
  • Proficiency in Python: Strong Python programming skills, especially for data engineering, data transformation, automation, and working with APIs.
  • Databricks/Apache Spark: Experience working with Databricks or Apache Spark for distributed data processing and data pipeline orchestration.
  • SQL Expertise: Advanced knowledge of SQL, specifically with BigQuery for creating complex queries and optimizing large-scale datasets.
  • Data Modeling: Experience with data modeling and designing data architectures that support high-performance analytics and reporting.
  • ETL Tools: Experience with ETL tools and frameworks (e.g., Apache AirflowDataflow, or dbt).
  • Version Control: Experience with Git or similar version control systems.
  • Cloud Data Architecture: Familiarity with best practices for cloud data architecture, including data lakes, data warehouses, and cloud-native technologies.
  • Agile Methodology: Experience working in Agile development environments, collaborating in cross-functional teams.
Desired Skills:
  • Data Engineering Frameworks: Familiarity with frameworks such as Apache KafkaApache Beam, or Flinkfor real-time data processing.
  • Machine Learning Integration: Basic understanding of integrating data pipelines with machine learning models and platforms.
  • Security & Compliance: Knowledge of GCP security best practices (e.g., IAM, data encryption, auditing) and compliance standards (e.g., GDPR, HIPAA).
  • CI/CD Pipelines: Familiarity with setting up CI/CD pipelines for automated testing and deployment of data pipelines.
Education & Experience:
  • Bachelor’s or Master’s degree in Computer ScienceInformation TechnologyEngineering, or a related field.
  • 5+ years of experience as a Data Engineer, with a focus on GCP, data pipelines, and analytics.

  • GCP DevOps Engineer

    3 months ago


    palakkad, India Zemoso Technologies Full time

    What You will doWe're looking for bright DevOps engineers that are experienced individual contributors and dynamic teamleaders, to join our dedicated DevOps Practice Group. As a DevOps practitioner at Zemoso you’ll useyour expertise to provide DevSecOps services to our customers and to our inhouse Dev teams.Basic Qualifications● Bachelor's degree in...

  • Data Engineer

    4 months ago


    Palakkad, India GeoTech Full time

    About Us: We are an innovative technology company focused on delivering data-driven solutions that power our business operations and drive growth. We are looking for a skilled Data Engineer with deep expertise in Google Cloud solutions to join our team. This role is integral to building and maintaining our data infrastructure, ensuring scalability, security,...

  • Data Scientist

    4 days ago


    Palakkad, India Moative Full time

    Energy & Utilities Data ScientistAbout MoativeMoative, an Applied AI Services company, designs AI roadmaps, builds co-pilots and predictive AI solutions for companies in energy, utilities, packaging, commerce, and other primary industries. Through Moative Labs, we aspire to build micro-products and launch AI startups in vertical markets.Our Past: We have...

  • Data Engineer

    4 weeks ago


    Palakkad, India Truelancer.com Full time

    Job Title: Data Engineer - DBT (Data Build Tool)Experience: 5-8 yrsMust-Have Skills: Airflow, S3, Snowflake, SQL/No SQL, Talend, AWSEssential Skills, Education and Experience:Should have a B.E. / B.Tech. / MCA or equivalent degree along with 4-7 years of experience in Data EngineeringStrong experience in DBT concepts like Model building and configurations,...


  • Palakkad, India Mastek Full time

    Job SummaryWe are seeking a Data Governance Engineer to manage and enforce governance policies across Mashreq’s hybrid data ecosystem. This role is pivotal in ensuring data quality, compliance, and security while supporting data platform operations as the first line of support. The ideal candidate should have expertise in Informatica EDC, Axon, and Azure...


  • Palakkad, Kerala, India ANSR Full time

    We are seeking a highly experienced Senior Data Engineering Expert to join our talented team at ANSR in Bangalore. This strategic role will lead our data engineering efforts, designing and implementing complex data solutions using Python, Spark, SQL, Snowflake, dbt, machine learning, and other relevant technologies.Key ResponsibilitiesLead the design,...

  • DevOps/SRE Engineer

    3 days ago


    Palakkad, Kerala, India BayOne Solutions Full time

    Seeking an experienced DevOps/SRE Engineer to join our team at BayOne Solutions in Gurugram/Bengaluru. This role involves migrating teams to new deployment settings, updating Python scripts for deployment artifacts and pipelines, and maintaining Kubernetes operators.Key ResponsibilitiesMigrate teams from current deployment settings to new onesUpdate and...


  • palakkad, India Atrium Full time

    Unlock yourself. Take your career to the next level.At Atrium, we live and deliver at the intersection of industry strategy, intelligent platforms, and data science — empowering our customers to maximize the power of their data to solve their most complex challenges. We have a unique understanding of the role data plays in the world today and serve as...


  • Palakkad, Kerala, India CSG Full time

    Job Title: Senior Operational EngineerAbout the Role:The ideal candidate will excel in a hybrid work environment, with the ability to choose their work style. This role requires a grower who can learn new skills and adapt quickly to enhance results. The individualist will succeed by leading their own projects and producing impactful outcomes. A time-manager...

  • Back End Engineer

    2 weeks ago


    Palakkad, India Ampera Full time

    Ampera DEIAs an expert in DEI strategy and implementation leadership, Ampera brings thought leadership to diverse areas through customized, end-to-end services, including DEI Data-Driven Solutions and Diversity Scorecards, among others. Ampera provides organizations with comprehensive solutions to achieve their DEI goals, offering access to a diverse talent...

  • Data Annotator

    4 days ago


    Palakkad, India Jupiter AI Labs ✔ Full time

    Position : Data Annotation fresherWorking Hours : Full Time (Slot : 1 PM to 9 PM)Salary : Up to 15kExperience : Fresher and Non TechNote : Only boys Preferred for this role and 1pm to 9pm slotWalking Venue : SECOND FLOOR, PLOT NO. A-61, B/4, Spring Meadows Business Park, Sector 63, Noida, Gautam buddha Nagar, Uttar Pradesh, 201301 Time : 10 am to 5pmNote :...


  • palakkad, India ClearML Full time

    Information Technology Manager, AI ComputingCompany DescriptionClearML is a unified, open source platform for continuous AI/ML, trusted by forward-thinking Data Scientists, ML Engineers, DevOps, and decision makers at leading Fortune 500, enterprises, academia, and innovative start-ups worldwide. We enable customers to achieve the fastest time to production,...


  • Palakkad, India Avow Solutions Inc Full time

    Position OverviewWe are seeking a detail-oriented and analytical Data Analyst to join our team. The successful candidate will collect, process, and analyze data to provide actionable insights that drive business decisions. This role requires proficiency in data manipulation, statistical analysis, and data visualization.Key Responsibilities1.Data Collection...

  • Databricks Architect

    1 month ago


    Palakkad, India HS Solutions Inc Full time

    About the JobWe are having an Urgent opening for one of our MNC client for PAN India Locations.Position Title: Databricks ArchitectNotice Period: Immediate to 30 days maximumLocation: PAN IndiaExperience: 10-15 YrsIn this role, the Databricks Architect is responsible for providing technical direction and lead agroup of one or more developer to address a...


  • palakkad, India Infoya Full time

    Company DescriptionInfoya is a global IT consulting and services organization. We collaborate with visionary leaders to achieve outsized business impact by providing innovative solutions and exploring cutting-edge technologies. We work hand-in-hand with our clients to facilitate transformative change, guiding them through strategic shifts in technology,...


  • Palakkad, Kerala, India ZeroNoticePeriod Full time

    About UsAt ZeroNoticePeriod, we're committed to innovation and excellence in our field.Job SummaryWe're seeking an experienced Data Science Team Lead to join our team. As a key member of our organization, you will be responsible for leading our data science efforts and driving business growth through data-driven insights.Key ResponsibilitiesLead the...


  • Palakkad, Kerala, India ZeroNoticePeriod Full time

    We are seeking a highly skilled Senior Data Science Professional to join our team at ZeroNoticePeriod.As a key member of our data science team, you will be responsible for driving business growth through data-driven decision making.The ideal candidate will have a strong background in statistics, economics, or a related field, with a minimum of 5 years of...

  • Full Stack Engineer

    4 days ago


    Palakkad, India BeGig Full time

    Job Description: Full Stack DeveloperNoida (In-office role)Budget-5-8 LPACompany Description:Client is a biotech company dedicated to solving diagnostic challenges anddeveloping therapeutic solutions for rare diseases through the power of genomics andartificial intelligence. Based in Noida, India, our interdisciplinary team comprisesscientists,...


  • Palakkad, India Commtech Asia Full time

    JOB DUTIESPerform task as directed by the Project LeaderManage compilation of technical reports and submission of status report, including progress updateTechnically responsible for delivery of service in accordance with Commtech / Client expectationsReview of multi-discipline Testing and Commissioning documents for compliance with specification and...


  • Palakkad, India YourDOST Full time

    Who we are:Founded in 2014 by Richa Singh and Puneet Manuja, YourDOST is one of India’s first and leading holistic emotional wellness organisations. We help corporates, colleges build resilient communities.We are enabling leading corporates like TCS, CK Birla Group, UpGrad, Darwinbox, Tata Consumer Products and educational institutions like IIT Madras, IIT...