Senior Data Engineer

Senior Data Engineer - Data Processing & Feature Engineering 

Location: Coimbatore

Experience Level: 6+ years

About the Role

We are seeking exceptional Senior Data Engineers to build the data foundation powering Velogent AI's autonomous agents. You will design and implement large-scale data ingestion, processing, and feature engineering systems that transform unstructured enterprise data (invoices, documents, transactions, RFQs) into structured, high-quality datasets. Your work enables agentic AI systems to make accurate, compliance-aware decisions while maintaining data quality, lineage, and auditability standards required by regulated industries.

Core Responsibilities

• Design and architect end-to-end data pipelines processing large volumes of unstructured enterprise data (documents, PDFs, transaction records, email, etc.)
• Build sophisticated data ingestion frameworks supporting multiple data sources and formats with automated validation and quality checks
• Implement large-scale data processing solutions using distributed computing frameworks handling terabytes of data efficiently
• Develop advanced feature engineering pipelines extracting meaningful signals from unstructured data (document classification, entity extraction, semantic tagging)
• Design data warehousing architecture supporting both operational (near real-time) and analytical queries for agentic AI reasoning
• Build robust data quality frameworks ensuring high data accuracy critical for agent decision-making and regulatory compliance
• Implement data governance patterns including lineage tracking, metadata management, and audit trails for regulated environments
• Optimize data pipeline performance, reliability, and cost through intelligent partitioning, caching, and resource optimization
• Lead data security implementation protecting sensitive information (PII, financial data, healthcare records) with encryption and access controls
• Collaborate with AI engineers to understand data requirements and optimize data for model training and inference
• Establish best practices for data documentation, SLA management, and operational excellence

Must-Have Qualifications

• Unstructured Data Expertise: Production experience ingesting and processing large volumes of unstructured data (documents, PDFs, images, text, logs)
• Large-Scale Data Processing: Advanced expertise with distributed data processing frameworks (Apache Spark, Flink, or cloud-native alternatives like AWS Glue)
• Feature Engineering: Deep knowledge of advanced feature engineering techniques for ML systems, including automated feature extraction and transformation
• Python Proficiency: Expert-level Python for data processing, ETL pipeline development, and data science workflows
• NLP/Text Processing: Strong background in NLP and text analysis techniques for document understanding, entity extraction, and semantic processing
• Data Architecture: Experience designing data warehouses, data lakes, or lakehouse architectures supporting both batch and real-time processing
• ETL/ELT Pipeline Design: Proven expertise building production-grade ETL/ELT pipelines with error handling, retry logic, and monitoring
• Cloud Data Platforms: Advanced experience with AWS data services (S3, Athena, Glue, RDS, DynamoDB) or equivalent cloud platforms
• Data Quality & Governance: Understanding of data quality frameworks, metadata management, and data governance practices

Nice-to-Have Qualifications

• Experience with document parsing and layout analysis libraries (Pydantic, , PyPDF, etc.)
• Knowledge of information extraction pipelines and vector databases for semantic search
• Familiarity with Apache Kafka or other event streaming platforms for real-time data processing
• Experience with dbt (data build tool) or similar data transformation frameworks
• Understanding of data privacy and compliance frameworks (GDPR, HIPAA, SOC2)
• Experience optimizing costs in cloud data platforms through intelligent resource allocation
• Background in building recommendation systems or ranking systems using feature engineering
• Knowledge of graph databases and knowledge graphs for relationship extraction
• Familiarity with computer vision techniques for document analysis and processing
• Published work or open-source contributions in NLP, document processing, or data engineering

What You'll Work With

• Large-scale document processing pipelines handling millions of invoices, contracts, and business documents
• Apache Spark and distributed computing frameworks for ETL
• AWS data services (S3, Glue, Athena, RDS) for data infrastructure
• Advanced NLP and text processing libraries (spaCy, transformers, LangChain)
• Vector databases and semantic search infrastructureData quality and monitoring frameworks
• Cloud data warehouses and data lakes on AWS
• Compliance and governance frameworks for regulated industries