|
Описание: |
We are an IT services company based out of San Francisco, USA. We build and acquihire expert engineering and data science teams to help our clients meet their business needs and solve their operational challenges
About the role
We are looking for an experienced Prompt Engineer with a strong background in QA and Product Management. This person will design, test and maintain prompts and prompt workflows for generative AI systems, focusing on answer quality, safety and product alignment. This role is non‑technical in the sense that heavy software development or ML engineering is not required; strong domain knowledge in testing, product requirements and evaluation is essential. You will take part in working with a lead conversion platform, powered by AI and integrations, helps local businesses convert leads faster, communicate easier, and make more sales.
Key responsibilities
* Design, document and iterate prompt templates, system instructions and example-based scaffolding for LLM-driven features.
* Run structured test campaigns: create test cases, input datasets, acceptance criteria and test plans for prompt outputs.
* Send queries to AI models, evaluate responses for correctness, relevance, completeness, tone and safety; record findings and recommend prompt adjustments.
* Perform manual and semi-automated QA of model outputs across scenarios and edge cases; report bugs, hallucinations and biased or unsafe outputs.
* Define and maintain quality metrics, benchmarks and validation suites specific to prompt performance.
* Collaborate with Product Managers, UX designers, data owners and engineers to translate product requirements into prompt specifications and acceptance criteria.
* Prioritize issues and propose product-focused remediation (prompt rephrasing, context changes, guardrails, fallback behaviors).
* Maintain prompt documentation, versioning and change logs; create reproducible prompt examples and playbooks for internal teams.
* Monitor production prompt performance, escalate regressions and support rollout/testing of prompt updates.
* Contribute to ethical and safety reviews, help mitigate prompt injection and other prompt-specific risks.
Required qualifications
* Minimum 8 years total experience in the IT industry, with solid experience in QA and/or Product Management.
* Demonstrable experience evaluating and testing outputs of generative AI or rule-based systems (manual testing, test plans, acceptance criteria).
* Strong abilities in crafting clear, unambiguous written instructions and test cases.
* English: Upper-Intermediate (B2) or higher — comfortable reading technical docs and communicating with international teams.
* Analytical mindset, attention to detail, strong documentation skills and experience with defining QA metrics.
* Familiarity with common LLM usage patterns (prompt templates, few/zero-shot, RAG) and safety concerns (hallucination, bias, prompt injection).
* Comfortable using APIs or UI tools to interact with AI systems (no deep programming required).
* Experience working cross-functionally with product, design and engineering teams.
Preferred
* Prior hands-on experience as QA lead, Product Manager, or similar role in AI/ML product context.
* Experience with test automation frameworks or low-code tools for running prompt evaluation pipelines.
* Familiarity with tooling: OpenAI, Anthropic, Hugging Face, Retrieval systems, or prompt management platforms.
* Knowledge of privacy/regulatory requirements relevant to model outputs.
Відгукнутись на вакансію |
