AI Des2Code - Test Case Design

This document defines the test contract for apps/des2code/. Tests must cover the strict storage boundary and the public artifact/backend fixtures, not only the internal ranking helpers.

Test Strategy

Layer	Scope	External systems
Unit	SQS/result schemas, aligned terms, candidate merging, model alias validation, family/structural/state helpers	None; typed fakes only
Repository	scoped DocumentDB reads, Atlas/DocumentDB vector pipelines, graph/catalog loading, S3 artifacts, webhook client	Collection/S3/HTTP doubles
Process	complete `process_record()` success/failure and artifact identity	Injected repository, matching gateway, logger
Contract	AI artifact fixtures parsed by backend and exposed by REST/MCP	Shared production fixtures
Smoke	SQS + Lambda + DocumentDB + S3 + backend webhook and latest-artifact GET	Deployed services

No unit/component test calls a live model provider. Structured model behavior is tested with bounded typed outputs. Local accuracy evaluations are separate from the deterministic CI suite.

Fixtures

Fixture	Purpose
canonical `design`	512-dimensional semantic/structure/region vectors and component occurrences
canonical `code`	source, semantic/context metadata, and both 512-dimensional vectors
canonical `code_variation`	rendered Storybook state with 512-dimensional visual vector
canonical `code_graph`	resolved scoped source/target dependency edge
ready/stale `code_index`	current context terms, variation policy, and rebuild lease
production Des2Code artifact	backend/AI shared success and failure contract
malformed legacy documents	prove strict adapters fail closed

Fixture vectors must always contain exactly 512 finite values. Tests that need invalid dimensions construct them explicitly rather than weakening canonical fixtures.

Unit Tests

Schema Validation

ID	Case	Expected
`U-SCH-001`	snake_case SQS body	accepted
`U-SCH-002`	camelCase aliases	accepted and normalized
`U-SCH-003`	missing design/project/organization	rejected
`U-SCH-004`	non-positive scope ID	rejected
`U-SCH-005`	empty or >255 `request_id`	rejected
`U-SCH-006`	unknown SQS body field	rejected because `extra=forbid`
`U-SCH-007`	complete success artifact	aliases and ISO timestamps serialize correctly
`U-SCH-008`	failure artifact	preserves optional request ID and bounded error
`U-SCH-009`	result counts and arrays	camelCase production contract is stable
`U-SCH-010`	provider key missing for selected model/embedding provider	config readiness fails
`U-SCH-011`	embedding dimension not 512	config readiness fails

Retrieval Merge And Rerank

ID	Case	Expected
`U-MRG-001`	duplicate semantic/context/region hits	merge by code ID and retain each axis maximum
`U-MRG-002`	variation hit for code outside initial vector pool	candidate is created with `variationSimilarity`
`U-MRG-003`	graph neighbor	hydrate scoped source and apply graph evidence
`U-MRG-004`	current index context	aligned terms are derived in memory only
`U-MRG-005`	deterministic rank	stable order for identical evidence/input
`U-MRG-006`	full catalog component misses top-k	occurrence planner may select it after ID validation
`U-MRG-007`	reranker returns non-retrieved ID	rejected from retrieval order
`U-MRG-008`	matcher returns unknown code/node ID	removed before hydration
`U-MRG-009`	duplicate `(nodeId, codeId)` usages	first valid occurrence retained
`U-MRG-010`	identity-only component	no state is required
`U-MRG-011`	state-required component	variation resolves only to known scoped rendered state
`U-MRG-012`	unsupported same-anchor alternatives	lower-evidence competitors suppressed
`U-MRG-013`	empty retrieval	no model call; empty successful result
`U-MRG-014`	model stage output violates typed bounds	fail the record; never accept unknown identity

Repository Tests

ID	Case	Expected
`U-REP-001`	fetch design	query includes `_id`, organization, and project; strict adapter used
`U-REP-002`	load code index	missing/stale rejected; ready returned
`U-REP-003`	Atlas vector search	inline scope filter and configured candidate/result bounds
`U-REP-004`	DocumentDB exact search	leading scoped `$match`, scope ceiling, in-process cosine ordering
`U-REP-005`	malformed/missing result vector	fail closed
`U-REP-006`	semantic/context projection	canonical metadata/vector paths and source fields
`U-REP-007`	variation visual search	canonical visual vector path and state projection
`U-REP-008`	graph neighbors	scoped source/target query and unique neighbor IDs
`U-REP-009`	full catalog	code/variation reads remain organization/project scoped
`U-REP-010`	success artifact	exact `{org}/{project}/des2code/{design}/{request}.json` key and object tag
`U-REP-011`	failure artifact	exact `.../{request}-failed.json` key
`U-REP-012`	webhook	shared payload, API key, timeout, and non-2xx delivery failure

Process Tests

ID	Case	Expected
`I-HP-001`	normal success	strict reads, ready index, retrieval/matching, success artifact, success webhook
`I-HP-002`	backend request ID	artifact filename and `requestId` use supplied value
`I-HP-003`	request ID absent	fallback to SQS message ID, then design ID
`I-HP-004`	design scope mismatch	no retrieval; failed artifact/webhook; re-raise
`I-HP-005`	design embedding invalid	failed artifact/webhook; re-raise
`I-HP-006`	code index missing/stale	failed artifact/webhook; re-raise
`I-HP-007`	model success	result counts equal final arrays and `codeIndexId` pins current index ID
`I-HP-008`	S3 success written, webhook fails	retain success artifact and return record for retry
`I-HP-009`	invalid body	do not invent unsafe scope or artifact key
`I-HP-010`	mixed SQS batch	return only failed `messageId` values
`I-HP-011`	empty candidate catalog	persist/report successful empty result
`I-HP-012`	backend artifact fixture	backend mapper accepts every required result/artifact field

Smoke Tests

ID	Scenario	Verification
`S-001`	ready project success	artifact exists under request key; backend GET returns it; webhook accepted
`S-002`	stale project index	backend trigger returns `409 CODE_INDEX_NOT_READY`; no SQS send
`S-003`	current code/variation/index rebuild	worker sees ready index and expected catalog/graph counts
`S-004`	duplicate SQS delivery	same request object is reused and remains contract-valid
`S-005`	second trigger for same design	new request ID creates a new object; backend GET returns newest by `LastModified`
`S-006`	model/provider secret missing	Lambda readiness/init fails rather than silently changing model
`S-007`	strict legacy-data rejection	incompatible DocumentDB shape fails before matching

Coverage Goals

Target	Goal
`schemas.py`	100% for public queue/result/artifact models and aliases
Matching helpers	all identity, suppression, structural, and state branches
`repo.py`	both vector providers, strict adapters, scope checks, S3, webhook
`service.py`	success plus every persistence/webhook failure ordering
Handler	full partial-batch response behavior
Cross-repo contract	AI artifact fixture and backend parser must agree

CI Commands

uv run --project apps/des2code pytest apps/des2code/__tests__
uv run pytest apps/__tests__/integration/test_production_contract_fixtures.py
uv run ruff check apps/des2code packages/des2code-core packages/models
uv run mypy apps/des2code/src packages/des2code-core/src packages/models/src

Backend contract tests must also run when the artifact or REST projection changes. Deployed smoke tests are not part of pull-request unit CI because they require AWS, DocumentDB, model-provider, and backend credentials.