Case study / Syncareer
Engineering around real LLM failure modes.
Syncareer is an AI-integrated career platform built to give students access to career guidance. Its AI feature worked in the demo. Use beyond the demo exposed behaviour the product could not rely on.
SynAI makes the product context visible: degree, interests and profile skills inform the career-guidance interface.
Product and system context
The model was only one part of the product.
Syncareer combined career guidance, a CV builder and interview preparation. The AI career guidance feature used Claude with custom system instructions and context from a user’s activity.
- 01User activityActivity, progress and weaknesses
- 02Context + instructionsExplicit context and custom system instructions
- 03ClaudeModel interaction
- 04Career guidanceProduct-facing response
Interactive failure trace / full inspector
Inspect the path: input → validation → failure → fix → valid output.
Click / focus a stage to reveal its detail. Keyboard: Tab + Enter, arrows to move. Values marked ILLUSTRATIVE are demo shapes, not historical logs.
01 / INPUT
User activity + profile context
Syncareer supplies Claude with explicit context from the student profile so guidance can be tailored. Context includes degree, interests, skills, CV progress and interview activity.
02 / MODEL OUTPUT
Raw LLM response (illustrative bad output)
ILLUSTRATIVE EXAMPLE — not a historical Syncareer log. Shows the kind of format drift observed in production: malformed fields, prose bleed, missing required keys.
To become senior, you should...
- build projects
{
title: Senior Dev
level:
years: "five?"Marked illustrative — not a historical Syncareer log. Used to show format drift type.
03 / VALIDATION
Schema check
Gate that checks required fields and type expectations before the product uses the response. If validation fails, the path routes to failure diagnosis.
04 / FAILURE
Observed failure modes
Separate failures treated as distinct: inconsistent formatting, dropped context (activity/progress/weaknesses not retained), unpredictable response structure.
// failure signals
- formatting varied
- context not always retained
- structure variable
// no measured failure rate suppliedBoundary No measured before-and-after failure rate is claimed here. Evidence boundary preserved.
05 / DIAGNOSIS
Why it failed
Root cause not reduced to one “AI problem”. Each mode mapped to a specific missing constraint. Interpretation: prompt structure, context handling and expected output needed tighter control.
Needed explicit schema
Needed explicit re-injection
Needed few-shot anchor
06 / INTERVENTION
What Stephen changed
Implemented: restructured prompts, added few-shot examples to anchor expected structure, managed context explicitly, set clearer constraints around expected output. ILLUSTRATIVE pattern shown for constraint — not claiming exact historical prompt text.
// illustrative constraint pattern
{
"type": "object",
"required": ["title","level","years"],
"properties": {
"title": {"type": "string"},
"level": {"enum": ["junior","mid","senior"]},
"years": {"type": "number"}
}
}
// illustrative few-shot anchor
Example 1:
Input: CS student, 2yr exp
Output: {"title":"...","level":"mid"...}07 / OUTPUT AFTERWARD
Valid product output (illustrative)
ILLUSTRATIVE valid output showing shape after intervention. Not claimed as historical Syncareer log. Measured reliability threshold not yet supplied — evidence boundary preserved.
{
"title": "Senior Developer",
"level": "senior",
"years": 5,
"next_steps": [
"ship one AI feature",
"add validation",
"test beyond happy path"
]
}Marked illustrative — shape matches product contract. No fabricated metrics.
Boundary No measured before-and-after failure rate is claimed here. Evidence boundary preserved.
Product surfaces
The AI feature sits inside a broader workflow.
The dashboard brings career progress, CV work, interview practice and applications into one product surface.
Job discovery connects to interview practice and CV tailoring rather than existing as a separate demo.
Reliability investigation
From product decision to engineering response.
Decision record
Choosing the model strategy
Fine-tuning was considered, then rejected.
It required data the team did not have. Instead, Claude received custom system instructions and context from the user’s activity on the platform.
ConsideredFine-tuningRejected / required unavailable dataSelectedClaude + tailored contextCustom instructions and user activity contextRecordTrade-off / higher cost and more backend complexity for tailored guidance
Observed behaviour
Format drift
Output formatting was inconsistent.
The feature could respond, but its formatting did not always match what the product expected.
Product expectedCompatible formatObservedFormatting variedRelated intervention / prompt restructuring and clearer expected-output constraints
RecordObserved / inconsistent output formatting
Observed behaviour
Context loss
Relevant context was dropped.
That mattered because the guidance was intended to use a student’s activity, progress and weaknesses.
ActivityProgressWeaknessesContext not always retainedRelated intervention / explicit context management
RecordObserved / information supplied to the model was not always retained
Observed behaviour
Response variance
The response structure was unpredictable.
A useful answer was not enough. The surrounding product also needed a structure it could handle consistently.
Expected structureVariable response structureRelated intervention / few-shot examples and clearer expected-output constraints
RecordObserved / response structure varied
Diagnosis
Separate the failures
These were not one general “AI problem.”
Formatting, retained context and response structure were treated as distinct failure modes. That made the next changes more deliberate.
FormattingContextStructureRecordEvidence boundary / behaviours identified; root causes are not documented here
Implemented changes
Constrain what was breaking
The response path was made more explicit.
- Restructured prompts.
- Added few-shot examples to anchor the expected structure.
- Managed context explicitly.
- Set clearer constraints around the expected output.
RecordImplemented / no measured before-and-after reliability rate is available
Evidence & Limitations
What this does not prove yet
- Observed behaviours identified
- Engineering responses implemented
- No before-and-after outputs supplied
- No repeatable evaluation supplied
- No measured failure rate supplied
- No quantified reliability threshold established
Product traction / not a reliability measure
What came of Syncareer
- Sign-ups
- 80+
- Daily page visits
- About 50
These numbers show product use. They are not presented as evidence that the LLM became more reliable.
Contact