Senior Data Engineer

We are looking for a hands-on Senior Data Engineer to lead the end-to-end design, development, and optimization of our data architecture. You will be responsible for extracting, processing, validating, and managing vast volumes of technical data from diverse manufacturer sources using Python and manual methods when necessary.

A key focus is implementing a hybrid data architecture that efficiently handles both structured data (CSV, Excel, SQL tables) and unstructured or semi-structured data (PDFs, images, graphs, text files). This will combine the power of data lakes and data warehouses, similar to platforms like Databricks and Delta Lake, for optimized ingestion, storage, retrieval, and querying at massive scale.

You will own the entire pipeline from data scraping and extraction to database architecture design, pipeline implementation, and security enforcement — ensuring high concurrency and low latency for potentially millions of users.

Key Responsibilities

· Lead the design, implementation, execution, debugging, and optimization of end-to-end data pipelines across diverse data sources and formats.

· Develop and maintain Python scripts using BeautifulSoup, Selenium, Scrapy for web scraping and automated data extraction.

· Define data formats and requirements for ingestion, including handling manual data extraction when necessary to ensure data quality.

· Organize, clean, validate, and transform large datasets into structured and semi-structured formats suitable for database ingestion.

· Digitize critical graphical data such as performance curves using appropriate tools.

· Architect and maintain scalable hybrid data storage solutions integrating relational databases (PostgreSQL, Snowflake), data lakes, and NoSQL systems.

· Implement robust data quality controls, error handling, and monitoring to maintain data integrity.

· Optimize database I/O throughput and concurrency for high-volume, low-latency data access.

· Enforce data security best practices including authentication, authorization, and encryption (Azure RBAC, Key Vault, etc.).

· Collaborate closely with product and engineering teams; document workflows and maintain version control.

· Employ AI and automation tools to accelerate extraction and processing while maintaining manual quality validation.

Qualifications

· Proven experience designing, implementing, and maintaining full data pipelines and architectures for large-scale, complex datasets.

· Strong hands-on Python programming skills, especially in web scraping libraries (BeautifulSoup, Selenium, Scrapy).

· Strong experience in relational (PostgreSQL, Snowflake) and NoSQL databases and hybrid data storage architectures.

· Experience in pipeline debugging, bottleneck identification, and performance tuning.

· Skilled in data quality assurance, validation processes, and defining data ingestion standards, including manual extraction workflows.

· Experience in modern platforms like Databricks and Delta Lake.

· Expertise in database optimization for concurrency, speed, and reliability.

· Ability to implement and maintain security controls in data environments.

· Strong communication skills and a self-driven, independent work ethic in remote settings.

· Interest in AI/automation integration and continuous process improvement.

· Work remotely with flexible hours.

· Competitive salary ranging from ₹15–40 LPA depending on experience.

· Hands-on, end-to-end ownership of data pipelines and architecture, enabling rapid learning and career growth.

· Opportunity to innovate with AI and modern cloud data platforms in a cutting-edge environment.