Principal Data Engineer

1 month ago

mumbai, India Sony Pictures Networks India Full time

Principal Data Engineer:

Experience:

  • 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments.
  • Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of end-to-end Big Data solutions.
  • Advanced hands-on experience with:
  • ETL/ELT frameworks: (e.g., Apache Nifi, Airflow, dbt)
  • Data Warehousing: (e.g., Snowflake, Redshift, BigQuery)
  • Big Data Technologies: (e.g., Hadoop, Apache Spark, HDFS)
  • Streaming Data Platforms: (e.g., Apache Kafka, Kinesis, Flink)
  • Cloud Data Solutions: (AWS, GCP) with a focus on services like Lambda, EMR, Glue, and DataFlow.
  • Have 8+ year of experience in ClickHouse/Redshift or any other OLAP system
  • Proficient in data processing and programming with Python, Scala, or Java, leveraging these languages for building high-performance data pipelines.
  • Strong understanding of distributed computing, microservices architecture, and containerization technologies (e.g., Kubernetes, Docker).
  • Deep knowledge of data security, data governance, and compliance frameworks in large-scale, complex data ecosystems.
  • Experience in performance optimization and troubleshooting in high-throughput data environments.
  • worked in Hadoop eco-system with AWS cloud distribution and worked at least on one or more big data ingestion tools (Sqoop, Flume, NiFI, etc), distributed messaging and ingestion frameworks (Kafka, Pulsar, Pub/Sub, etc).

Core Roles and Responsibilities:

  • Architect and build large-scale, robust, and high-performance data pipelines that process both batch and streaming data.
  • Implement, automate, and maintain real-time data processing systems, ensuring high availability and low latency.
  • Collaborate with data scientists, machine learning engineers, and analysts to optimize data pipelines and integrate advanced analytics workflows.
  • Develop and maintain scalable ETL processes that handle vast amounts of structured and unstructured data from diverse sources.
  • Focus on continuous improvement, driving the evolution of architecture to ensure optimal performance, cost-efficiency, and scalability.
  • Optimize distributed systems for performance and cost-efficiency, including caching, compression techniques, and optimizing query performance.
  • Maintain, monitor, and troubleshoot complex distributed systems, implementing monitoring solutions and ensuring reliability through proactive alerting.
  • Lead technical deep dives into performance bottlenecks, debugging, and optimization of data processing workflows.
  • Manage and ensure data security, availability, and reliability by following best practices and aligning with industry standards (e.g., encryption, IAM policies).

  • Data Warehouse Engineer

    1 week ago

    Mumbai, India Data Warehouse Engineer Full time

    Experience : 2- 5 YearsPrimary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills.Good to have skills: Power BI, Databricks, Python.

  • Principal data engineer

    4 weeks ago

    Mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer: Experience: 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments. Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of...

  • Principal Data Engineer

    4 weeks ago

    mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer:Experience:- 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments.- Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of...

  • Principal Data Engineer

    4 weeks ago

    mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer: Experience: - 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments. - Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of...

  • Principal Data Engineer

    1 month ago

    Mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer:Experience:- 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments.- Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of...

  • Principal data engineer

    1 month ago

    Mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer:Experience:10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments.Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of end-to-end...

  • Principal Data Engineer

    4 weeks ago

    mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer:Experience:- 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments.- Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of...

  • Principal Data Engineer

    1 month ago

    mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer:Experience:10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments.Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of end-to-end...

  • Principal Data Engineer

    3 weeks ago

    Mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer:Experience:10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments.Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of end-to-end...

  • Principal Data Engineer

    4 weeks ago

    mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer:Experience:10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments.Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of end-to-end...

  • Principal Data Engineer

    1 month ago

    Mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer:Experience:10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments.Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of end-to-end...

  • Principal Data Engineer

    1 month ago

    Mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer: Experience: 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments. Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of...

  • Principal Data Engineer

    1 month ago

    Mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer: Experience: 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments. Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of...

  • Principal Data Engineer

    1 month ago

    mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer: Experience: 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments. Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of...

  • Principal Data Engineer

    1 month ago

    Mumbai, India Sony Pictures Networks India Full time

    Principal Data Engineer: Experience: 10-12 years of deep technical experience in data engineering, primarily focusing on high-tech or cloud-based environments. Extensive expertise in building, maintaining, and optimizing large-scale data platforms and pipelines. Must have knowledge of Big Data Architecture Patterns and experience in the delivery of...

  • Data Engineer and Principal Consultant Position

    1 week ago

    Mumbai, Maharashtra, India Genpact Full time

    About GenpactGenpact is a global professional services and solutions firm that delivers outcomes to shape the future. Our team of 125,000+ professionals across 30+ countries is driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients.Salary and BenefitsWe offer an estimated salary of $120,000 per annum based on...

  • Principal BI Engineer

    3 months ago

    Mumbai, India Cornerstone OnDemand Full time

    We're looking for a Principal BI Engineer This role is Office Based, Pune Office Principal BI Engineer We are seeking a talented  Principal BI Engineer  in Pune, India. The right candidate has strong communication skills, passion for solving business problems with data, domain knowledge in Sales & Marketing, Customer Success & Product,...

  • Principal gcp data engineer

    4 weeks ago

    Mumbai, India IKS Health Full time

    What will you be responsible for? ● The ideal candidate will oversee end to end data pipeline stack, including design, development, testing, documenting and deploying the pipeline, by following the defined/proven data engineering practices in a cloud based environment (GCP). ● As principal you will provide subject matter expertise in Pipeline...

  • Principal GCP Data Engineer

    4 weeks ago

    mumbai, India IKS Health Full time

    What will you be responsible for? ● The ideal candidate will oversee end to end data pipeline stack, including design, development, testing, documenting and deploying the pipeline, by following the defined/proven data engineering practices in a cloud based environment (GCP). ● As principal you will provide subject matter expertise in Pipeline...

  • Principal GCP Data Engineer

    4 weeks ago

    mumbai, India IKS Health Full time

    What will you be responsible for?● The ideal candidate will oversee end to end data pipeline stack, including design, development,testing, documenting and deploying the pipeline, by following the defined/proven data engineeringpractices in a cloud based environment (GCP).● As principal you will provide subject matter expertise in Pipeline Architecture,...