Skip to content

AI Code Import โ€” I/O Definition

This page is the official I/O contract for the code-import worker implemented in apps/code-import/ (Python package code_import under src/code_import/). Any change to fields, shapes, status values, or failure categories is a contract change and must be reflected in this page and test-cases.en.md before merging code.

Reading order: Overview โ†’ Input โ†’ Processing Contract โ†’ Output โ†’ Error Handling โ†’ Idempotency. The Field Reference at the end of the page is for lookup purposes.

Overview

flowchart LR
  api[guinness-backend] -->|INSERT code status=0| RDB[(PostgreSQL)]
  api -->|SendMessage| Q[(SQS code-import)]
  Q --> L[AI Code Import worker]
  L -->|validate optional reference| S3[(S3 preview metadata)]
  L -->|one semantic run| LLM[(DESC_MODEL)]
  L -->|one batch, two inputs| EMB[(Embeddings 512)]
  L -->|transactional replace + invalidate| Doc[(DocumentDB code + code_index)]
  L -->|POST /v1/webhooks/ai-status| api
  api -->|UPDATE code status=1 or 2| RDB
Item Value
Trigger SQS record on the code-import queue
Input SQS message + optional canonical S3 preview reference
Output DocumentDB 1 upsert (code collection) + Webhook POST 1 call
DocumentDB writes Scoped code replacement and code_index.status=stale in one transaction when evidence changed
RDB writes None โ€” the worker does not access PostgreSQL / MySQL
External calls Deterministic source parsing, one semantic agent via PydanticAI, provider-selected embeddings (1 call, 2 inputs), Webhook POST
AI framework PydanticAI (1 Agent run per changed record); embeddings through the configured OpenAI-compatible client

Input

1. SQS Message (Trigger)

Lambda receives a standard AWS SQS event. The backend (apps/app/src/services/code.ts) enqueues to the code-import queue, and each job's Records[*].body contains a JSON string.

{
  "Records": [{
    "messageId": "0192a3b4-c5d6-7e8f-9a0b-1c2d3e4f5a6b",
    "body": "{\"code_id\":\"660e8400-e29b-41d4-a716-446655440001\",\"organization_id\":1,\"project_id\":42,\"name\":\"button-primary\",\"type\":1,\"based_on\":0,\"source_code\":\"<button className='btn btn-primary'>Click me</button>\",\"css_code\":\".btn-primary { background:#007bff; }\",\"img_url\":\"s3://bucket/code/42_button.png\"}",
    "attributes": { "ApproximateReceiveCount": "1" },
    "eventSource": "aws:sqs"
  }]
}

Body fields (parsed from Records[*].body, validated by Pydantic's CodeImportMessage โ€” Overview ยง1).

Field Type Required Notes
code_id string (UUID) โœ“ RFC 4122 UUID v4 โ€” becomes _id in DocumentDB and recordId in Webhook
organization_id integer โœ“ Tenant isolation; persisted to DocDB
project_id integer โœ“ For downstream vector search scoping
name string โœ“ Stored as-is
type integer โœ“ 0 = page, 1 = code
based_on integer โœ“ 0 imported, 1 design, 2 wireframe
source_code string โœ“ Non-empty; upstream recommends โ‰ค200 KB โ€” Open Issue
css_code string | null โœ— Stored as-is if present
img_url string | null โœ— Optional canonical preview reference; stored as metadata and never downloaded by this worker
component_level string | null โœ— Optional hint: atom, molecule, organism, template, or component

Body unwrapping. To accommodate differences between Lambda and API Gateway, the handler accepts the following (in priority order):

  1. Flat โ€” the object itself is the body
  2. SQS-in-SQS โ€” inner JSON string inside the outer Records[0].body
  3. API Gateway proxy โ€” body.body (optionally isBase64Encoded)
  4. Double-wrapped (legacy) โ€” extra Records nesting found in test fixtures; unwrap until a flat CodeImportMessage is visible

Any other shape is a validation error.

2. S3: Optional Code Preview Screenshot

Property Value
Source img_url, when non-null
Validation Bucket must equal S3_BUCKET_NAME; key must equal the canonical preview key for org/project/code with png, jpg, or webp extension
Worker access No S3 GET; the URL is persisted in visual.metadata.image_url only
When absent visual.metadata.image_url is null; semantic/context import is unaffected

3. RDB Row State (Read by Other Systems)

The worker does not directly read the PostgreSQL code row. The backend creates the row before publishing to SQS (typically with status = 0). Completion is signaled via Webhook, and the state machine is owned by the backend.

Processing Contract

Each SQS record is processed synchronously in the following order:

  1. Parse and validate CodeImportMessage, unwrapping if necessary.
  2. Validate preview reference โ€” when img_url is present, require the configured bucket and exact canonical key; do not download it.
  3. Extract code context โ€” deterministically parse source_code, css_code, name, and component_level to produce family, variant, imports, exports, referenced components, class names, source tokens, and context text.
  4. Code analysis agent (text) โ€” Input: source_code + optional css_code; output: semantic_words (5โ€“10 strings).
  5. Embeddings โ€” call the configured provider once with 2 inputs: deterministic semantic identity text and context text. data[0] โ†’ semantic; data[1] โ†’ context; both are exactly 512 dimensions.
  6. Document assembly โ€” locked shape per Output 1.
  7. DocumentDB write โ€” scoped replace_one(..., upsert=True) and set the scoped singleton code_index.status to stale in one transaction when the code changed.
  8. POST success Webhook with semantic_words in result.keywords (embeddingDims always 512).

Before model/embedding work, a matching processing.hash reuses the existing valid vectors, patches only a changed preview URL, and sends success. Do not write partially until all vectors are available.

Output

Output 1: DocumentDB code Collection (Upsert)

Conforms to Spec ยง5.2 โ€” summary:

{
  "_id": "660e8400-e29b-41d4-a716-446655440001",
  "organization_id": 1,
  "project_id": 42,
  "name": "button-primary",
  "type": 1,
  "based_on": 0,
  "source_code": "<button className='btn btn-primary'>Click me</button>",
  "css_code": ".btn-primary { background:#007bff; }",
  "visual": {
    "metadata": {
      "name": "button-primary",
      "image_url": "s3://bucket/1/42/code/660e8400-e29b-41d4-a716-446655440001.png"
    }
  },
  "semantics": {
    "metadata": { "words": ["submit", "checkout", "transaction", "action", "primary"] },
    "vector_embedding": [/* 512 floats */]
  },
  "context": {
    "metadata": {
      "component_level": "atom",
      "family": "button",
      "variant": "primary",
      "imports": [{"source": "react", "specifiers": [], "default_import": "React", "namespace_import": null}],
      "exports": ["Button"],
      "referenced_components": ["button"],
      "class_names": ["btn-primary"],
      "source_tokens": ["button", "primary", "click"],
      "text": "level=atom family=button variant=primary exports=Button references=button tokens=button primary click"
    },
    "vector_embedding": [/* 512 floats */]
  },
  "processing": {
    "hash": "<64-character SHA-256 fingerprint>"
  }
}
Field Rule
visual.metadata.image_url Copy of img_url; null when no screenshot
semantics.metadata.words Identical to semantic_words from the Code Semantics agent
semantics.vector_embedding Exactly 512 dimensions โ€” embedding of deterministic semantic identity text
context.metadata Component level, family, variant, structured imports, exports, references, class names, source tokens, and canonical text
context.vector_embedding Exactly 512 dimensions โ€” embedding of context.metadata.text
processing.hash Stable SHA-256 of source/model/prompt/embedding inputs used for unchanged-import reuse

Vector indexes (created idempotently by packages/models/documentdb/code.py):

Index name Path Dimensions Similarity HNSW m efConstruction
codeSemanticVectorIndex semantics.vector_embedding 512 cosine 16 64
codeContextVectorIndex context.vector_embedding 512 cosine 16 64

Output 2: Webhook โ€” Success

POST {WEBHOOK_BASE_URL}
Content-Type: application/json
X-API-Key: {WEBHOOK_API_KEY}

{
  "type":      "code-import",
  "recordId":  "660e8400-e29b-41d4-a716-446655440001",
  "projectId": 42,
  "organizationId": 1,
  "status":    "success",
  "result": {
    "keywords":      ["submit", "checkout", "transaction", "action", "primary"],
    "embeddingDims": 512
  }
}
Field Notes
type Always "code-import"
recordId Echo of code_id (UUID string) โ€” Open Issue
projectId Echo of project_id
organizationId Echo of organization_id
result.keywords Identical to semantics.metadata.words
result.embeddingDims Always 512

Output 3: Webhook โ€” Failure

POST {WEBHOOK_BASE_URL}
Content-Type: application/json
X-API-Key: {WEBHOOK_API_KEY}

{
  "type":      "code-import",
  "recordId":  "660e8400-e29b-41d4-a716-446655440001",
  "projectId": 42,
  "organizationId": 1,
  "status":    "failed"
}

No result on failure. Exception details are not included in the Webhook (sent to CloudWatch instead).

The failure Webhook is sent before re-raising the exception.

Error Handling

Code and variation imports retry transient database transactions up to 8 attempts with exponential backoff and jitter. Each attempt uses a fresh session and preserves atomic evidence writes and index invalidation; AI calls are not repeated by this retry. Permanent errors fail immediately. Exhausted retries are delegated to SQS + DLQ. Webhook delivery failures remain retryable record failures.

Scenario Webhook (if code_id is known) Worker side
Malformed envelope / no Records โŒ ValueError, etc.
JSON parse failure โŒ JSONDecodeError
Pydantic validation Best-effort failed ValidationError
Invalid preview reference failed Re-raise
Semantic AgentRunError failed Re-raise
Embeddings API error failed Re-raise
DocumentDB write error failed Re-raise
Webhook HTTP error WARN log Re-raise so SQS retries

The handler processes every record and returns SQS batchItemFailures; one failed record does not stop later records in the same batch. ReportBatchItemFailures must be enabled on the event source mapping.

Idempotency

  1. Scoped replacement by _id โ€” reprocessing the same scoped code_id atomically replaces the previous document and invalidates the current index only when evidence changed.
  2. No RDB writes โ€” success may be delivered more than once via at-least-once Webhook. The backend tolerates duplicate posts of (type, recordId).
  3. Fingerprint short circuit โ€” identical evidence reuses stored vectors and keywords; only preview metadata may be patched.

Locked Parameters

Parameter Value / Rule Reference
Semantic word count 5โ€“10 Structured output from code analysis
Embedding dimensions 512 HNSW + Webhook embeddingDims
Embedding API calls 1 call, 2 inputs Semantic identity and deterministic context
Webhook identifier "type": "code-import" ai-status routing
HTTP client Webhook retries timeout=10s, retries=2, exponential backoff Spec ยง2
Event Level Timing
code_import.received INFO Start of process_record
code_import.parsed INFO After Pydantic validation
code_import.semantic.done INFO After semantic agent
code_import.embeddings.done INFO After provider-selected embeddings
code_import.documentdb.upserted INFO After transactional replacement/index invalidation
code_import.webhook.sent INFO After HTTP 2xx
code_import.failed ERROR Terminal failure
code_import.webhook.failed WARN Webhook-only failure

Recommended values for extra: code_id, project_id, organization_id, message_id.

Open Issue: recordId Type

Resolved: PostgreSQL code.id, the worker code_id, and webhook recordId are the same UUID v4 string. The backend webhook schema accepts this value.

Open Issue: Source Size Cap

Spec ยง11 states "source_code is recommended to be โ‰ค200 KB" โ€” decide whether to enforce the same cap via Pydantic, or revisit a common cap by agreement with the backend.

Field Reference (Lookup Table)

Field SQS Input DocumentDB Output Webhook Output
code_id โœ“ _id recordId
organization_id โœ“ โœ“ organizationId
project_id โœ“ โœ“ projectId
name / type / based_on โœ“ โœ“ โ€”
source_code / css_code โœ“ โœ“ โ€”
img_url โœ“ visual.metadata.image_url* โ€”
component_level โœ“ context.metadata.component_level โ€”
derived family / variant โ€” context.metadata.* โ€”
semantic_words (agent) โ€” semantics.metadata.words result.keywords
Source parser output โ€” context.metadata.* โ€”
semantics.vector_embedding โ€” โœ“ โ€”
context.vector_embedding โ€” โœ“ โ€”
stable fingerprint โ€” processing.hash โ€”
status โ€” โ€” success / failed

*always present; null when no preview reference is supplied.

  • Overview โ€” Processing flow, embedding rationale, agent code examples, dependencies, sequence diagrams.
  • Test Case Design โ€” Catalog validating this contract.