Data Engineering Pipeline Architect
6 days ago
Web Crawler Architecture LeadWe are seeking a skilled Web Crawler Architecture Lead to oversee our web crawling and pipeline architecture teams.Oversee web engineering and crawling teams, ensuring high-quality delivery and adherence to best practices.Design, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.Develop and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies.Define and enforce data quality, validation, and security measures across all data flows and pipelines.The ideal candidate will have:A Bachelor's or Master's degree in computer science, information technology, or a related field.7-12 years of relevant experience in web engineering, pipeline design, or large-scale web crawling systems.Strong expertise in Python, SQL, and modern data processing practices.Experience working with Airflow, Celery, or similar workflow automation tools.Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture.Hands-on experience with cloud data platforms (AWS/GCP/Azure).Experience with AI/LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar).
-
Chief Data Architect
6 days ago
vellore, India beBeeData Full timeLead Data EngineerUnlock your potential as a talented data engineer, contributing to our mission of harnessing the power of data.As a lead data engineer, you will play a pivotal role in designing and implementing scalable data pipelines and ETL processes that drive business success.You will collaborate with cross-functional teams to understand business...
-
Senior Analytics Pipeline Architect
4 days ago
vellore, India beBeeDataEngineer Full timeKey Roles and Responsibilities:We are seeking a highly skilled Senior Data Engineer to design and implement robust data pipelines that power nationwide analytics and support machine learning systems.This is an exciting opportunity to build scalable ETL pipelines using Python and SQL, orchestrated with Airflow. The ideal candidate will have a strong...
-
Associate Architect: Data
4 weeks ago
Vellore, India Quantiphi Full timeCompany ProfileQuantiphi is an award-winning AI-first digital transformation engineering company driven by the desire to solve transformational problems at the heart of the business. We are passionate about our customers and obsessed with problem-solving. Quantiphi solves our clients’ toughest and most complex business problems by combining deep industry...
-
Strategic Data Architect
6 days ago
vellore, India beBeeDataEngineer Full timeOur organization is seeking a highly skilled Senior Data Engineer to fill a key role in our data management team. In this position, you will be responsible for designing and architecting large database systems, particularly on Microsoft platforms such as SQL Server and Azure Database.The ideal candidate will have a minimum of 7 years of experience in data...
-
Senior Data Solutions Architect
4 days ago
vellore, India beBeeDataEngineering Full timeData Engineering SpecialistThis role involves leading the development, optimization, and maintenance of data pipelines and back-end services within a microservices environment.Key Responsibilities:Design and implement scalable data pipelines for ingestion, processing, and storage of data from various sources.Craft robust data solutions and APIs that power...
-
Cloud Data Solutions Architect
4 days ago
vellore, India beBeeAzure Full timeJob Title: Azure Data ArchitectThe ideal candidate should have a deep understanding of data warehousing concepts, medallion architecture, and data pipeline engineering.This role requires strong leadership to manage the successful delivery of multiple data projects and provide technical guidance to data engineering teams. The successful candidate will be...
-
Chief Data Architect
10 hours ago
vellore, India beBeeData Full timeJob OpportunityData Engineering ExcellenceWe are seeking an experienced Senior Data Engineer with in-depth knowledge of Oracle Data Integrator (ODI) and a strong understanding of Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) frameworks. This role involves designing and developing robust data pipelines across on-premises and cloud Oracle...
-
Expert Data Architect
4 days ago
vellore, India beBeeDataLead Full timeAzure Data Architect RoleWe are seeking a seasoned Data Engineer to lead our Azure data initiative. The ideal candidate will have at least 6 years of experience in a data engineering environment and expertise in CI-CD, PySpark, SQL, ADF, ADLS, ADB, and Azure Services.The successful candidate will also have experience with GIT, Jenkins, and Azure DevOps. They...
-
Freelance Data Architect
2 weeks ago
Vellore, India Whatjobs IN C2 Full timeLooking for a Freelance Data Architect to join a team of rockstar developers. The candidate should have a minimum of 12+ yrs. of experience. There are multiple openings. If you're looking for freelance/ part time opportunity (along with your day job) & a chance to work with the top 0.1% of developers in the industry, this one is for you! You will report into...
-
Big Data Architect
6 days ago
vellore, India beBeeDataEngineer Full timeJob DescriptionAn Azure Data Engineer will build, implement and maintain scalable big data architectures using Microsoft Azure Databricks, Synapse Analytics, ADLS Gen 2 and other related technologies. This role requires a strong background in data engineering, with experience in ETL, ELT, data warehousing, data governance and compliance.Required Skills and...