About this Job

EcoRatings is seeking a robust and detail-oriented Data Engineer to build the critical data pipelines that ingest massive datasets from enterprise systems into our proprietary AI ecosystem.

As a core member of the Intelligence & Data track, you will bridge the gap between raw enterprise data (Oracle, SQL, FTP) and our generative AI models. This role is ideal for an engineer who thrives on solving complex data ingestion challenges and ensuring data integrity in high-stakes environments.

Responsibilities

• Data Ingestion & ETL:

  Design, develop, and maintain scalable ETL/ELT pipelines to ingest data from diverse enterprise sources, including SQL databases, Oracle ERPs, and flat files (FTP/SFTP).

• Pipeline Orchestration:

  Build and manage data workflows using tools like Apache Airflow or Prefect to ensure timely, reliable data delivery to the AI/ML team.

• Schema Design:

  Collaborate with the Lead AI Engineer to define and implement data schemas that optimize the performance of RAG (Retrieval-Augmented Generation) pipelines.

• Database Management:

  Optimize and manage both relational (PostgreSQL/MySQL) and non-relational storage solutions, specifically Vector databases like Pinecone or Weaviate.

• Data Cleaning & Validation:

  Implement automated data validation and cleaning scripts to ensure that the "intelligence" layer receives high-quality, audit-ready data.

• API Development:

  Build and maintain internal APIs and connectors that facilitate seamless communication between the data warehouse

• Security & Compliance:

  Ensure all data pipelines adhere to strict enterprise security protocols, including encryption at rest and in transit, to protect sensitive client information.

• Collaboration:

  Work closely with the Full Stack Lead and AI Engineers to synchronize data ingestion logic with application requirements and AI "thinking" processes.

Qualifications