Skip to content

AI Design Import โ€” Test Cases

This page is the test plan for apps/design-import/. Every row in io-definition.md โ€” fields, processing steps, and failure categories โ€” must correspond to at least one test case. If a contract clause has no test, fix that first.

Convention: Test IDs follow the format DI-<layer>-<NN>. Layers are U (unit, no I/O), C (component โ€” real S3 / Mongo testcontainer, PydanticAI + OpenAI + Webhook are mocked), and E (end-to-end โ€” real cloud). Each case explicitly states Given / When / Then. The "Then" block is the assertion contract โ€” anything not written there is not verified (callers must not depend on it either).

Test Layers and Tooling

These cases are designed as a manual / local test plan. They are not integrated into CI and are not enforced by automated gates โ€” they exist for engineers to verify that the worker behaves according to the I/O contract when changes are introduced. Run them locally before merging or use them as part of a release checklist.

Layer Scope Tooling
Unit (U) Pure functions: SQS body parsing, body unwrapping, S3 URL parsing, image format detection, document builder, Webhook payload builder pytest, no network. PydanticAI / OpenAI / httpx faked at the SDK boundary
Component (C) End-to-end process_record. Real localstack S3 + testcontainer Mongo (DocumentDB API-compatible image) + mock PydanticAI agent + mock openai.embeddings.create + mock Webhook httpx client pytest, testcontainers, localstack, respx/unittest.mock
End-to-end (E) Real dev AWS โ€” real SQS, S3, DocumentDB, OpenAI dev key, real Webhook endpoint pytest @e2e marker, smoke only

Shared Fixtures

Fixture Provides Used by
sqs_record(**overrides) A well-formed SQS record dict with overridable fields U, C
s3_bucket Localstack bucket pre-populated with a 360ร—707 PNG and a Figma JSON sample C
mock_vision_agent(visual_features=..., semantic_words=[...]) Returns a fixed DesignDescription from agent.run() U, C
mock_openai_embed(vector=[0.1]*512) Returns deterministic 512-float vectors for visual, semantic, and structural inputs U, C
mock_webhook(http_status=200) Captures the outbound Webhook payload and returns the specified status C

sqs_record(...) is the single source of truth for payload shape โ€” always synthesize from it rather than hand-writing individual dicts.

Coverage Matrix

Horizontal axis: contract clauses from io-definition.md. Vertical axis: outcomes. Each cell points to one or more test IDs.

Happy path Failure path Idempotency
Payload schema DI-U-01 DI-U-02 โ€”
Body unwrapping DI-U-03 DI-U-04 โ€”
Image format detection DI-U-05 โ€” โ€”
S3 image GET DI-C-01 DI-C-02, DI-C-03 โ€”
Vision agent DI-C-01 DI-U-06, DI-C-04 โ€”
Embedding (batch) DI-C-01 DI-U-07, DI-C-05 โ€”
Document builder DI-U-08 โ€” โ€”
DocumentDB upsert DI-C-01 DI-C-06 DI-C-07
Success Webhook DI-C-01 DI-C-08 DI-C-09
Failure Webhook โ€” DI-C-02, DI-C-03 โ€”
Re-raise to SQS on failure โ€” DI-C-02, DI-C-03 โ€”

Unit (U) Cases

No network. Pure Python.

DI-U-01 โ€” Valid SQS body is parsed

  • Given A well-formed body containing the required fields (organization_id, project_id, img_url, node_id, file_id, design_id, design_name, json_schema_url).
  • When The parser runs.
  • Then The model has organization_id, project_id, img_url, node_id, file_id, design_id, design_name, and json_schema_url.

DI-U-02 โ€” Rejected when required field is missing

  • Parametrize over each required field.
  • Given A body with that one field removed.
  • Then ValidationError. The field name appears in the error.

DI-U-03 โ€” Body unwrapping: SQS-in-SQS

  • Given body = json.dumps({"Records": [{"body": json.dumps(payload)}]}).
  • Then The parser unwraps one level and returns the inner payload model.

DI-U-04 โ€” Body unwrapping: API Gateway + base64

  • Given body = {"body": base64(json.dumps(payload)), "isBase64Encoded": true}.
  • Then Decoded, parsed, and model returned.

DI-U-05 โ€” Image format detection (magic bytes)

  • Applies only if the implementation includes a magic-byte format detector. The contract does not require one.
  • Parametrize: PNG, JPEG, GIF, WEBP samples, and one unknown byte sequence.
  • Then The detector returns "png", "jpeg", "gif", "webp", and None respectively.
  • And If the implementation puts the detected MIME type into the data URL, assert that; if it uses a fixed MIME, assert that instead. Pin whichever policy the implementation chooses.

DI-U-06 โ€” Vision agent returns empty semantic_words

  • Given Mock Vision agent returns DesignDescription(visual_features="x", semantic_words=[]).
  • Then Downstream embedding cannot run (input is empty). The worker raises, and the handler emits a failure Webhook before re-raising.

DI-U-07 โ€” Embedding response has wrong dimensions

  • Given Mock embeddings client returns a vector of 1536 floats instead of 512.
  • Then (If the document builder includes a length check) The worker rejects it, the handler emits a failure Webhook, and re-raises.
  • Note: This assertion applies only if the implementation includes a length check. The contract does not require one. If it is absent, delete this case rather than faking a pass.

DI-U-08 โ€” Document builder field copying

  • Given A parsed SQS body and a DesignDescription.
  • When The document builder runs.
  • Then The resulting dict contains:
  • _id == body.design_id
  • organization_id == body.organization_id
  • project_id, name == body.design_name
  • visual.metadata.name == body.design_name
  • visual.metadata.image_url == body.img_url
  • visual.vector_embedding has length 512
  • semantics.metadata.words == agent.semantic_words
  • semantics.vector_embedding has length 512
  • structural.vector_embedding has length 512
  • query_terms is populated
  • figma_node.file_id == body.file_id
  • figma_node.node_id == body.node_id
  • json_schema key is absent
  • components key is absent
  • Top-level id or design_id key is absent
  • Top-level img_url โ€” if the implementation follows the DocumentDB document structure in overview.en.md, assert it is absent. Decide on one policy in the implementation and pin the test to it.
  • type and based_on follow the defaults defined in the I/O contract.

Whether img_url lives at the document top level is an implementation decision โ€” update this test to match that policy. Treat the DocumentDB document structure in overview.en.md as the source of truth.

Component (C) Cases

Real localstack S3 + testcontainer Mongo. Mock PydanticAI agent, OpenAI embeddings, and Webhook httpx client at the SDK boundary.

DI-C-01 โ€” Full happy path

  • Given:
  • An image exists in S3 at img_url.
  • json_schema_url points to a valid Figma node JSON object in S3.
  • mock_vision_agent returns DesignDescription(visual_features="blue rounded card", semantic_words=["login","auth","form"]).
  • mock_openai_embed returns [0.1] * 512 for all three inputs.
  • mock_webhook(http_status=200).
  • When The handler processes the message.
  • Then:
  • The Mongo design collection contains a document with _id == design_id, organization_id == body.organization_id, visual/semantic/structural vectors of length 512, semantics.metadata.words == ["login","auth","form"], matching img_url and json_schema_url from SQS, figma_node.node_id == node_id, populated query_terms, and name == design_name.
  • The Webhook was called exactly once with {"type": "design-import", "recordId": design_id, "projectId": <pid>, "organizationId": <org_id>, "status": "success", "result": {"keywords": ["login","auth","form"], "embeddingDims": 512, "vectorsProduced": ["visual","semantic","structural"]}}.
  • No exception is re-raised.
  • The worker GETs json_schema_url and fails the record if it cannot parse the Figma node JSON.

DI-C-02 โ€” S3 image 404 (NoSuchKey)

  • Given img_url points to a non-existent key.
  • Then:
  • Failure Webhook called with {"type":"design-import","recordId":...,"projectId":...,"organizationId":...,"status":"failed"} (no result).
  • Handler re-raises; SQS sees the failure.
  • No Mongo write.
  • No success Webhook.

DI-C-03 โ€” S3 image 500

  • Given Localstack chaos returns 500 for the image key.
  • Then:
  • Failure Webhook emitted.
  • Exception re-raised (SQS retry).
  • No Mongo write.

DI-C-04 โ€” Vision agent raises

  • Given mock_vision_agent raises a PydanticAI exception (e.g. provider 5xx).
  • Then:
  • Failure Webhook emitted.
  • Exception re-raised.
  • No embeddings call. No Mongo write.

DI-C-05 โ€” Embeddings API raises

  • Given Vision agent succeeds; mock_openai_embed raises a transient APIError.
  • Then:
  • Failure Webhook emitted.
  • Exception re-raised.
  • No partial Mongo document exists โ€” assert db.design.find_one({"_id": design_id}) is None.

DI-C-06 โ€” Mongo write raises (transient)

  • Given Vision + embeddings succeed. Pause the Mongo testcontainer.
  • Then:
  • Failure Webhook emitted.
  • Exception re-raised.

DI-C-07 โ€” Redelivery atomically overwrites the document

  • Given DI-C-01 has run once, leaving a document with vector vA.
  • And The same message is redelivered with the same design_id, but the mock now returns vector vB.
  • Then:
  • The Mongo design collection contains exactly one document with _id == design_id.
  • The document's visual.vector_embedding reflects vB (last write wins).
  • A success Webhook is observed twice across the two runs (at-least-once).

DI-C-08 โ€” Webhook POST fails on the success path

  • Given Mongo write succeeds. mock_webhook(http_status=500).
  • Then:
  • A WARN log line (design_import.webhook.failed or equivalent).
  • The handler re-raises or marks the record in batchItemFailures. SQS retries because the backend did not receive the authoritative status notification.

DI-C-09 โ€” Webhook at-least-once across attempts

  • Given Attempt 1: Vision agent raises (DI-C-04).
  • And Attempt 2: everything succeeds (DI-C-01).
  • Then:
  • Two Webhooks observed: 1 failed, 1 success.
  • Mongo contains only the document from attempt 2.

DI-C-10 โ€” Persisted document has no json_schema field

  • Given DI-C-01 happy path.
  • Then db.design.find_one({"_id": design_id}) has no json_schema key. The DocumentDB document holds only embeddings and metadata. The Figma JSON lives in S3 and is referenced via json_schema_url in the SQS body.

DI-C-11 โ€” Persisted document has no components field

  • Given DI-C-01 happy path.
  • Then The document has no components key. Des2Code is computed downstream, so the design document does not hold it.

End-to-end (E) Cases

Smoke set only. Run manually against the real dev environment with real services and a real OpenAI dev key. Cost is non-zero, so keep the suite small.

DI-E-01 โ€” Round-trip on real dev SQS

  • Given guinness-backend dev enqueues a message via the actual design-create flow.
  • Then Within 90 seconds:
  • The DocumentDB design collection contains a document with two 512-float vectors.
  • The backend received a success Webhook (verify via backend logs or design row state).
  • CloudWatch shows the worker lifecycle log lines.
  • (Backend status = 1 depends on record_id resolution; treat as informational until resolved.)

DI-E-02 โ€” Retry on real OpenAI 429

  • (Manual only. Run during a period when throttling is occurring.)
  • Given The dev OpenAI key is under a tight rate limit.
  • Then The message is redelivered by SQS and eventually succeeds. The final Webhook is success.

Cross-cutting Checks

Conditions that must hold across all applicable cases. These are not standalone test rows โ€” verify them while reviewing / running the tests above.

  • No partial DocumentDB documents: Every test that fails between Vision and the Mongo write must assert db.design.find_one({"_id": design_id}) is None.
  • Embedding length is always 512: Every test that persists data must assert both vector lengths.
  • Exactly one Webhook per process_record call: Either success or failed, never both within a single call.
  • Failure Webhook is sent before re-raising: Intercept both the Webhook mock and the raise point, and assert the ordering.
  • No stack trace strings leak into the Webhook body: Webhook payload assertions are exact-match โ€” no extra fields.
  • PostgreSQL / MySQL connections are never opened: Run tests with a DB driver unreachable on localhost (or set a guard environment variable that fails loudly at import time). This surfaces any path that accidentally uses a DB driver immediately.
  • Only one document identifier: Persisted documents hold only _id. Fail if a top-level id or design_id key is present โ€” the design id flows as _id and appears nowhere else.

Intentionally Out of Scope

Listed explicitly so that future tests land in the right bucket.

  • Backend design.status state machine: Owned by guinness-backend tests. The worker only emits a Webhook.
  • Vector search / Des2Code fusion: Owned by the apps/des2code/ test suite.
  • Authentication: The worker has no HTTP entrypoint; authentication is upstream.
  • DLQ operations / SQS infrastructure: Owned by infrastructure tests. The worker only verifies re-raise behavior.
  • Code preview rendering: Owned by the code import/rendering pipeline. This worker does not render code entries.
  • Cross-tenant isolation: Purely RBAC. Happens in the backend. Worker tests assume the message is already authorized.
  • record_id resolution strategy: Currently an open contract issue. Tests should pin the current behavior so that future fixes are visible as test changes.
  • I/O Definition โ€” The contract these tests enforce.
  • Overview โ€” Processing flow diagrams, module breakdown, agent definitions, and DocumentDB document structure.