Brightedge - Big Data Engineer - ETL/Python

1 week ago

Hyderabad, India brightedge Full time

About UsBrightEdge is a leading SEO and content performance marketing platform that transforms online content into tangible business results.Our platform processes massive amounts of data to provide actionable insights to our clients.We're looking for a talented Big Data Engineer to join our Professional Services to help us scale and optimize our data processing capabilities.Role Overview :As a Big Data Engineer at BrightEdge, you will design, build, and maintain high-performance data pipelines that process terabytes of data.You'll work on optimizing our existing systems, identifying and resolving performance bottlenecks, and implementing solutions that improve the overall efficiency of our platform.This role is critical in ensuring our data infrastructure can handle increasing volumes of data while maintaining exceptional performance standards.Key Responsibilities :- Design and implement scalable batch processing systems using Python and big data technologies.- Optimize database performance, focusing on slow-running queries and latency improvements.- Use Python profilers and performance monitoring tools to identify bottlenecks.- Reduce P95 and P99 latency metrics across our data platform.- Build efficient ETL pipelines that can handle large-scale data processing.- Collaborate with data scientists and product teams to understand data requirements.- Monitor and troubleshoot data pipeline issues in production.- Implement data quality checks and validation mechanisms.- Document data architecture and engineering processes.- Stay current with emerging big data technologies and best practices.Qualifications :Required :- Bachelor's degree in Computer Science, Engineering, or related technical field.- 4+ years of experience in data engineering roles.- Strong Python programming skills with focus on data processing libraries.- Experience with big data technologies (Spark, Hadoop, etc.- Proven experience optimizing database performance (SQL or NoSQL).- Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.- Understanding of performance optimization techniques and profiling tool.Preferred :- Master's degree in Computer Science or related field.- Experience with SEO data or web crawling systems.- Experience with Clickhouse Database.- Knowledge of distributed systems and microservices architecture.- Familiarity with container orchestration (Kubernetes, Docker).- Experience with real-time data processing.- Contributions to open-source projects.- Experience with machine learning operations. (ref:hirist.tech)

  • Data Engineer

    4 weeks ago

    Hyderabad, India AI Start Up Full time

    Description :Now Hiring: Data Engineer (Python, PySpark, Palantir Foundry) | Pune | Contract-to-Hire | Hybrid | US ClientVopais is looking for experienced Data Engineers to join a high-impact team in Pune, supporting a major client in the utilities/energy sector. This contract-to-hire opportunity offers hands-on leadership in building robust,...

  • Data Engineer

    2 weeks ago

    Hyderabad, India Orbion Infotech Full time

    Key Responsibilities :- Design, develop, and maintain robust, scalable, and efficient data pipelines.- Work with structured and unstructured data to enable analytics and reporting.- Collaborate with data scientists, analysts, and business stakeholders to understand requirements and deliver data-driven solutions.- Optimize and automate ETL processes to ensure...

  • ETL or Big data tester

    2 weeks ago

    Hyderabad, Telangana, India Altura Innovative Technologes Full time ₹ 5,00,000 - ₹ 18,00,000 per year

    We're Hiring – ETL or Big data Testing (Hyderabad, India)Altura Innovative Technologies Pvt Ltd is looking for a QA Testing to join our client teamExperience: 6 to 12 yearsAnalyze business requirements and technical specifications to design test scenarios for ETL and big data processes.Develop, maintain, and execute test cases, test scripts, and SQL...

  • Etl or Big Data Tester

    4 days ago

    Hyderabad, Telangana, India Altura Innovative Technologes Full time

    We're Hiring - **ETL or Big data Testing** (Hyderabad, India) Altura Innovative Technologies Pvt Ltd is looking for a QA Testing to join our client team! Experience: 6 to 12 years - Analyze business requirements and technical specifications to design test scenarios for ETL and big data processes. - Develop, maintain, and execute test cases, test scripts, and...

  • Big Data Engineer

    5 days ago

    Hyderabad, India ANB IT Solutions Pvt Ltd Full time

    Job Description :- Minimum of 7+ years of progressive experience in Data Engineering, with a proven track record in pipeline creation and implementation.- Strong awareness and hands-on experience with various data tools and platforms, including Hive, Hadoop, Starburst, and other Big Data technologies.- Proficiency in programming languages such as SQL,...

  • Big Data Python Engineer

    4 weeks ago

    Hyderabad, India Virtusa Full time

    Big Data Python Engineer - CREQ Description Role GDT WS Big Data Python Engineer Exp 5+ Location Pune/Hyderabad MUST Have Skills Design and implement data processing pipelines using Python + SCALA programming and big data technologies such as Hadoop, MapReduce, Hive, and Spark. Airflow Jenkins Understand business requirements to code up data assets in...

  • Big Data Python Engineer

    4 weeks ago

    Hyderabad, India Virtusa Full time

    Big Data Python Engineer - CREQ Description Role GDT WS Big Data Python Engineer Exp 5+ Location Pune/Hyderabad MUST Have Skills Design and implement data processing pipelines using Python + SCALA programming and big data technologies such as Hadoop, MapReduce, Hive, and Spark. Airflow Jenkins Understand business requirements to code up data assets in...

  • Senior Data Engineer

    5 days ago

    Hyderabad, India Zensar Technologies Full time

    Position : Senior Data Engineer (ETL/Big Data/Streaming)Experience : 810 yearsLocation : Onsite - Hyderabad Job Overview :We are looking for an experienced Senior Data Engineer with strong expertise in ETL/ELT, big data, and real-time data streaming to join our team.The ideal candidate will have deep hands-on experience with Apache Flink, Confluent Kafka,...

  • Data Engineer

    2 weeks ago

    Hyderabad, India Phenom Full time

    Job Requirements :Responsibilities :- Translate business and functional requirements into robust, scalable solutions that work well within the overall data architecture.- Develops and maintains scalable data pipelines and builds new API integrations.- Design, develop, implement, test, document, and operate large scale, high volume and low latency...

  • Python,pyspark,apachespark

    2 days ago

    Hyderabad, Telangana, India NTT DATA Full time

    **Req ID**: 327063 We are currently seeking a python,pySpark,ApacheSpark to join our team in Hyderabad, Telangana (IN-TG), India (IN). "At NTT DATA, we know that with the right people on board, anything is possible. The quality, integrity, and commitment of our employees are key factors in our company's growth, market presence and our ability to help our...