AI Agent experience coordinator

Overview

We are seeking an AI Agent Experience Coordinator to support the development, optimization, monitoring, and quality assurance of our AI-powered solutions.

This role is ideal for a highly organized professional who combines strong project coordination skills with hands-on experience using AI tools and prompt engineering techniques. You will help manage AI agent development workflows, maintain prompt libraries, coordinate testing and quality assurance activities, and ensure that AI-generated outputs consistently meet quality and business standards.
Role:

• RXChainge project coordination
• Agent creation & monitoring
• Prompt engineering
• Quality assurance, testing & optimization

Key Responsibilities

Project coordination

• Coordinate AI agent development initiatives and related project activities.
• Manage timelines, priorities, deliverables, and stakeholder communications.
• Facilitate collaboration between client services, medical writing and dev. team.
• Track project progress, identify risks, and escalate issues when necessary.
• Support documentation, reporting, and process improvement initiatives.

Agent creation, monitoring and prompt engineering

• Execute the creation, configuration, testing, and demo-ing of AI agents.
• Maintain and organize the prompt library, ensuring consistency, version control, and documentation.
• Design and optimize prompts across multiple use cases, therapeutic areas, and user roles.
• Benchmark prompts and agent performance across models and configurations to improve output quality and reliability.
• Analyze conversation transcripts to identify prompt optimization opportunities
• Monitor agent performance and identify opportunities for improvement.

Quality assurance, testing & optimization

• Coordinate QA and user acceptance testing activities with internal stakeholders.
• Develop and execute testing plans for AI agents and prompting workflows.
• Identify and address quality issues, edge cases, and performance gaps.
• Review AI-generated content for clarity, consistency, and tone, and facilitate the content review of subject matter experts for accuracy.
• Execute on analysis and reporting steps, primarily related to transcripts, to track performance, quality, user feedback, and identify opportunities for optimization.

Required Qualifications

• 2–5+ years of experience in project coordination, project management, digital production, or a related field (flexible based on experience)
• Professional experience using AI tools and large language models (e.g., ChatGPT, Claude, Gemini, Copilot).
• Understanding of prompt engineering principles and generative AI workflows.
• Strong organizational and multitasking skills with the ability to manage multiple priorities simultaneously.
• Excellent written and verbal communication skills.
• Strong analytical and problem-solving abilities.
• Comfortable working in an evolving environment with changing priorities.
• Demonstrated attention to detail and commitment to quality.
• Bilingual (English and French)

Additional Assets

• Experience with prompt optimization or experimentation workflows
• Familiarity with evaluation methods for AI outputs (qualitative or quantitative)
• Exposure to software product development or agile environments
• Experience working with regulated or specialized content, including healthcare or life sciences.
• Experience in writing, editing, proofreading or related fields