AI Code Import

Processes code import jobs from the code-import SQS queue. It validates an optional preview reference, extracts semantic words, derives deterministic source context, generates two 512-dimensional vectors (semantic + context), transactionally replaces the scoped code document, marks the current code_index stale when evidence changes, and notifies backend through the shared webhook.

Replaces: V1 code import predecessor in guinness-ai-develop Queue: code-import Max execution time: approximately 1–3 minutes Source: guinness-ai-v2 (new repository)

Tech Stack

Runtime: Python 3.12 on AWS Lambda
AI Framework: PydanticAI (agent orchestration, provider-agnostic)
LLM: Provider-prefixed DESC_MODEL — one structured semantic analysis call for changed source
Embeddings: EMBEDDING_PROVIDER + EMBEDDING_MODEL, exactly 512 dimensions for semantic and context vectors
Database: Amazon DocumentDB (MongoDB-compatible) — code collection, AI-exclusive
Storage: Optional canonical S3 preview reference is validated and stored as metadata; this worker does not download it
Queue: AWS SQS (code-import queue)
Status Notification: Webhook POST /v1/webhooks/ai-status → guinness-backend
No MySQL/PostgreSQL access: strict database isolation

Key Changes from V1

Item	V1	V2 (`code-import`)
AI Framework	LangGraph `StateGraph`	PydanticAI — 1 semantic agent (`run_sync`)
Screenshot	`img_url` required	`img_url` is optional metadata; rendered states are imported separately by `code-variation-import`
Source code LLM usage	Stored only, not analyzed	Semantic agent reads `source_code` + `css_code`
Embeddings	1 (vision keywords `vector_embedding`)	2 — `semantics` + `context`
Visual evidence	Image analyzed inline	`code` stores preview metadata only; `code_variation` owns rendered visual vectors
DB writes	MySQL + DocumentDB	DocumentDB only + Webhook (no RDB access)
DocDB collection	legacy flat code-like collection	`code`
CSS	Not stored	`css_code` (optional)
Batch processing	Full retry on first failure	`batchItemFailures` (per-record)
`organization_id`	Not in DocDB	Stored in all documents

Official per-field contract: io-definition.en.md.

Processing Flow

flowchart TD
    A["SQS: code-import\n(code_id, source_code, img_url?)"] --> B["Parse + Validate CodeImportMessage"]
    B --> C["Semantic Agent — text only\nsource_code + css_code → semantic_words"]
    B --> X["Deterministic source parser\nimports, exports, references, classes,\ntokens, family, variant, level"]
    B --> D["Validate optional canonical\nS3 preview reference"]
    D --> H
    C --> H
    X --> H
    H["Provider embeddings — 1 batch call\n[semantic identity, context text] → 512 dims"]
    H --> I["Replace scoped CodeDocument +\nmark code_index stale in one transaction"]
    I --> J["POST webhook success"]

    style A fill:#f9f,stroke:#333
    style C fill:#bbf,stroke:#333
    style F fill:#bbf,stroke:#333
    style G fill:#bbf,stroke:#333
    style I fill:#bfb,stroke:#333
    style J fill:#fdb,stroke:#333

Implementation: apps/code-import/src/code_import/service.py (process_record). Agents are built in handler.py from packages/agentic.

Embedding Strategy

Code import owns implementation identity, semantic intent, and deterministic source context. Rendered visual evidence is deliberately separated into code_variation, so this worker never creates a visual embedding.

Embedding	Agent	Input	What it encodes	Use in des2code
`semantics`	Code Semantics	`source_code`, `css_code`	Purpose/function keywords (5–10)	Code entries close to the function the design requires
`context`	Deterministic parser	Source, CSS, name, and matching hints	Imports, exports, JSX/HTML tags, referenced components, source tokens, family, variant, level	Code entries related by structure, hierarchy, and component composition

Both vectors are generated in one provider-selected embeddings call.

The only optional matching hint accepted from SQS is component_level; family and variant are derived from the name/source.

1. SQS Message Schema (Input)

Envelope

The backend (apps/app/src/services/code.ts) enqueues to the code-import queue. Lambda receives via the standard SQSEvent wrapper:

{
  "Records": [{
    "messageId": "0192a3b4-c5d6-7e8f-9a0b-1c2d3e4f5a6b",
    "body": "{\"code_id\":\"660e8400-e29b-41d4-a716-446655440001\",\"organization_id\":1,\"project_id\":42,\"name\":\"button-primary\",\"type\":1,\"based_on\":0,\"source_code\":\"<button className='btn btn-primary'>Click me</button>\",\"css_code\":\".btn-primary { background:#007bff; }\",\"img_url\":\"s3://bucket/code/42_button.png\"}",
    "attributes": { "ApproximateReceiveCount": "1" },
    "eventSource": "aws:sqs"
  }]
}

Body Fields

Records[0].body is a JSON string. After parsing:

Field	Type	Required	Description
`code_id`	string (UUID)	Yes	Primary key. Same value as PG `code.id` and DocumentDB `_id`.
`organization_id`	int	Yes	Owning organization. Used for tenant isolation in DocumentDB.
`project_id`	int	Yes	Owning project. Used as `$match` scope for vector search.
`name`	string	Yes	Human-readable name (e.g. `"button-primary"`).
`type`	int	Yes	`0` = page, `1` = code.
`based_on`	int	Yes	`0` = imported, `1` = design-derived, `2` = wireframe-derived.
`source_code`	string	Yes	Code source (JSX/TSX/HTML).
`css_code`	string	No	Optional CSS. Stored as-is if present.
`img_url`	string	No	Optional canonical preview S3 reference; metadata only.
`component_level`	string	No	Optional `atom`, `molecule`, `organism`, `template`, or `component` hint.

Validation Rules

code_id must conform to RFC 4122 UUID v4.
type ∈ {0, 1}, based_on ∈ {0, 1, 2} — otherwise rejected.
source_code must be non-empty (length ≥ 1, recommended ≤ 200 KB).
img_url, when present, must target S3_BUCKET_NAME and the exact canonical org/project/code preview key with png, jpg, or webp extension.

Pydantic Model

from pydantic import BaseModel, Field

class CodeImportMessage(BaseModel):
    code_id: str = Field(..., description="UUID v4; matches PG code.id and DocDB _id")
    organization_id: int
    project_id: int
    name: str
    type: int = Field(..., ge=0, le=1, description="0=page, 1=code")
    based_on: int = Field(..., ge=0, le=2, description="0=imported, 1=design, 2=wireframe")
    source_code: str = Field(..., min_length=1)
    css_code: str | None = None
    img_url: str | None = None
    component_level: str | None = None

    model_config = {"extra": "forbid"}

Note (V1 → V2 rename): V2 normalizes the legacy import identity fields to code_id / name to match the unified code collection.

2. Webhook Payload Schema (HTTP Output)

After processing completes, the worker POSTs once to WEBHOOK_BASE_URL (POST /v1/webhooks/ai-status). Authentication uses X-API-Key (timing-safe comparison). See ai-status webhook for the receiver contract.

Success

{
  "type": "code-import",
  "recordId": "660e8400-e29b-41d4-a716-446655440001",
  "projectId": 42,
  "organizationId": 1,
  "status": "success",
  "result": {
    "keywords": ["submit", "checkout", "transaction", "action", "primary"],
    "embeddingDims": 512
  }
}

Failure

{
  "type": "code-import",
  "recordId": "660e8400-e29b-41d4-a716-446655440001",
  "projectId": 42,
  "organizationId": 1,
  "status": "failed"
}

Field Reference

Field	Type	Description
`type`	string	Always `"code-import"` (discriminator).
`recordId`	string	Echo of `code_id` from SQS body. Used by the backend to update the PostgreSQL `code` row.
`projectId`	int	Echo of `project_id`.
`organizationId`	int	Echo of `organization_id`. Used by backend scope validation.
`status`	string	`"success"` or `"failed"`.
`result`	object	Present only when `status="success"`.
`result.keywords`	string[]	Semantic words extracted from source code (5–10 items).
`result.embeddingDims`	int	`512` (matches HNSW index dimensions).

Type consistency note: recordId is a string for this flow because backend code.id is a UUID string. The backend webhook contract must accept the same value as code_id.

Send Timing

Event	When sent	Body
All steps complete without exception	After successful DocumentDB upsert	Success payload
Exception raised in `process_record()`	Immediately before re-raising the exception	Failure payload

HTTP calls use httpx with timeout=10s, retries=2 (exponential backoff). Webhook failures are retryable processing failures; the worker re-raises so SQS can redeliver the message.

3. PydanticAI — LLM Role Definitions

The worker makes one structured LLM call. Deterministic source parsing is the second evidence-producing role but does not call a model.

Role	Architecture Node	Input	Output
Code Analysis LLM	`C` — source code analysis → semantic words	Source code + CSS (text)	`semantic_words` list
Source context parser	`X` — deterministic identity/graph evidence	Name + source code + CSS + optional level hint	family, variant, imports, exports, references, classes, tokens, context text

The semantic call uses provider-prefixed DESC_MODEL. The parser is local and deterministic. Both outputs contribute to the two embedded retrieval texts.

3.1 Code Analysis LLM (text only)

Corresponds to architecture node C. Reads source code (and optional CSS) and outputs 5–10 semantic keywords representing what the code does.

from pydantic_ai import Agent
from pydantic import BaseModel, Field

class CodeSemantics(BaseModel):
    semantic_words: list[str] = Field(
        min_length=5, max_length=10,
        description="5–10 keywords representing the code's purpose, function, and behavior",
    )

code_analysis_llm = Agent(
    DESC_MODEL,
    output_type=CodeSemantics,
    instructions=(
        "You are a frontend code analyst. Given a code source and "
        "optional CSS, extract 5–10 concise keywords representing its purpose, "
        "function, and behavior. Examples: 'submit', 'checkout', 'navigation', "
        "'modal', 'form', 'pagination'."
    ),
)

Input (constructed by worker): a single user message in the following format

SOURCE CODE:
<source_code>

CSS (optional):
<css_code>

Output: CodeSemantics.semantic_words → used for Webhook result.keywords and the semantic embedding.

3.2 Vision LLM (vision-capable, conditional branching)

This original section anchor is retained for documentation compatibility, but the current worker has no Vision LLM. It calls extract_code_context for every changed record, does not download or analyze img_url, and delegates rendered visual evidence to code-variation-import.

context = extract_code_context(
    name=body.name,
    kind="page" if body.type == 0 else "component",
    source_code=body.source_code,
    css_code=body.css_code,
    component_level_hint=body.component_level,
)

Evidence	Input	Architecture Node
Semantic identity	name, kind, parsed context, semantic-agent words	C
Source context	imports, exports, JSX/HTML references, classes, literals	X

Output: strict context.metadata plus canonical text for the context embedding and graph/index building.

4. Embedding Generation

PydanticAI does not handle embeddings. The worker uses the configured OpenAI or OpenRouter embedding client and makes one batched call.

def generate_embeddings(
    semantic_identity: str,
    context_text: str,
) -> tuple[list[float], list[float]]:
    response = client.embeddings.create(
        model=EMBEDDING_MODEL,
        input=[semantic_identity, context_text],
        dimensions=512,
    )
    return response.data[0].embedding, response.data[1].embedding

Embedding	Input Text	Source Agent	Index Field
`semantics`	deterministic semantic identity text including model terms	Semantic Agent + source parser	`semantics.vector_embedding`
`context`	`context_text` from source parser	Deterministic parser	`context.vector_embedding`

All vectors are 512 dimensions to match the DocumentDB HNSW indexes.

5. DocumentDB Operations

5.1 Upsert

with db.client.start_session() as session, session.start_transaction():
    collection.replace_one(
        {"_id": code_id, "organization_id": organization_id, "project_id": project_id},
        document,
        upsert=True,
        session=session,
    )
    code_index.update_one(scope, {"$set": {"status": "stale"}}, upsert=True, session=session)

The replacement and index invalidation are atomic. A matching processing.hash skips model/embedding/write work and reuses the current document.

5.2 Document Structure (`code` collection)

{
  "_id": "660e8400-e29b-41d4-a716-446655440001",
  "organization_id": 1,
  "project_id": 42,
  "name": "button-primary",
  "type": 1,
  "based_on": 0,
  "source_code": "<button className='btn btn-primary'>Click me</button>",
  "css_code": ".btn-primary { background:#007bff; border-radius:8px; }",
  "visual": {
    "metadata": {
      "name": "button-primary",
      "image_url": "s3://bucket/1/42/code/660e8400-e29b-41d4-a716-446655440001.png"
    }
  },
  "semantics": {
    "metadata": {
      "words": ["submit", "checkout", "transaction", "action", "primary"]
    },
    "vector_embedding": [0.45, 0.22, -0.10, "..."]
  },
  "context": {
    "metadata": {
      "component_level": "atom",
      "family": "button",
      "variant": "primary",
      "imports": [{"source": "react", "specifiers": [], "default_import": "React", "namespace_import": null}],
      "exports": ["Button"],
      "referenced_components": ["button"],
      "class_names": ["btn-primary"],
      "source_tokens": ["button", "primary", "click"],
      "text": "level=atom family=button variant=primary exports=Button references=button tokens=button primary click"
    },
    "vector_embedding": [0.18, 0.31, -0.22, "..."]
  },
  "processing": {
    "hash": "<64-character SHA-256 fingerprint>"
  }
}

Field	Source	Notes
`_id`	`code_id` from SQS body	UUID; matches PG `code.id`
`organization_id`	SQS body	Tenant isolation
`project_id`	SQS body	`$match` scope for vector search
`name`, `type`, `based_on`	SQS body	Stored as-is
`source_code`, `css_code`	SQS body	Stored as-is; `css_code` is optional
`visual.metadata.image_url`	SQS `img_url` or `null`	Metadata only; no vector in `code`
`semantics.metadata.words`	Code Semantics agent output	5–10 strings
`semantics.vector_embedding`	Embedding of deterministic semantic identity text	512 dimensions
`context.metadata`	Deterministic source evidence and canonical `text`	Used by context retrieval and graph/index building
`context.vector_embedding`	Embedding of `context.metadata.text`	512 dimensions
`processing.hash`	Stable input fingerprint	Unchanged-import short circuit

5.3 Vector Indexes

Created once at startup (idempotent) in packages/models/documentdb/code.py:

Index	Field	Type	Dimensions	Similarity
`codeSemanticVectorIndex`	`semantics.vector_embedding`	HNSW	512	cosine
`codeContextVectorIndex`	`context.vector_embedding`	HNSW	512	cosine

HNSW parameters: m=16, efConstruction=64.

des2code searches code by semantic/context vectors and searches rendered appearance in the separate code_variation collection.

des2code search	design-side query	code-side path
Visual/state	design screen/region image query vectors	`code_variation.visual.vector_embedding`
Semantic	`design.semantics.vector_embedding`	`code.semantics.vector_embedding`
Context / structural	`design.structure.vector_embedding`, regions, and aligned terms	`code.context.vector_embedding` and `context.metadata`

The source of semantics differs per worker:

Worker	Source of `semantics.metadata.words`
design-import	Single vision call → `semantic_words` from design image
code-import	Text-only agent → from source code + CSS

The embedding model and 512 dimensions are shared across retrieval collections. code.visual.metadata.image_url is only a representative preview pointer.

6. Error Handling

Transport clients apply bounded retries; record failure is reported through SQS partial-batch response and then follows the queue redrive policy.

def handler(event, context):
    if "Records" not in event:
        raise ValueError("Invalid SQS event")

    failures = []
    for record in event["Records"]:
        try:
            process_record(record)
        except Exception as e:
            logger.error(f"Failed: {record.get('messageId')} — {e}")
            failures.append({"itemIdentifier": record.get("messageId", "")})
    return {"batchItemFailures": failures}

Failure Matrix

Stage	Exception Class	Behavior	Webhook sent?
No `Records` in SQS envelope	`ValueError`	Re-raise → SQS retry	❌ (no record context)
Body JSON parse failure	`json.JSONDecodeError`	Re-raise → SQS retry	❌ (no `code_id`)
Pydantic validation failure	`ValidationError`	Re-raise → SQS retry	⚠️ Partial (best effort if `code_id` is parseable)
Invalid S3 preview reference	`ValueError`	Record failure	✅ failed
Semantic LLM failure	`pydantic_ai.exceptions.AgentRunError`	Record failure	✅ failed
Embeddings API failure	`openai.APIError`	Re-raise → SQS retry	✅ failed
DocumentDB write failure	`pymongo.errors.PyMongoError`	Re-raise → SQS retry	✅ failed
Webhook POST failure	`httpx.HTTPError`	Log and re-raise so SQS retries	n/a
Max SQS retries exceeded	—	Delivered to DLQ (configured in Terraform)	n/a

Retryable vs Terminal

Type	Example	SQS handling
Terminal (retry is pointless)	Invalid JSON, validation/reference error	Repeated delivery eventually reaches DLQ
Retryable (transient)	Provider 429/5xx, DocDB connection reset, webhook timeout	Bounded client retry, then SQS redelivery/DLQ

The distinction is implemented via SQS partial batch response (batchItemFailures).

7. Folder Structure

guinness-ai-v2/
  packages/
    agentic/
      __init__.py
      orchestrator.py            # build_manager_agent(), build_tool_agent() — PydanticAI
      agents/
        __init__.py
        vision.py                # Vision LLM + DesignDescription             (for design-import)
        code_semantics.py        # ★ Code Analysis LLM + CodeSemantics      (for code-import, arch node C)
    models/
      documentdb/
        __init__.py              # initialize_collections()
        design.py                # DesignCollection
        code.py                  # ★ Code collection + semantic/context indexes
        code_index.py            # Current scoped index state
    des2code-core/
      src/des2code_core/
        code_context.py          # Deterministic source parser
    utils/
      settings.py                # shared Pydantic Settings base
      documentdb.py              # shared DocumentDB connection singleton

  apps/
    code-import/                 # directory name in repository (hyphen); uv workspace path
      pyproject.toml             # Hatch project name code_import matches import path
      Dockerfile
      README.md
      __tests__/                 # not yet placed — add per test-cases.en.md
      src/
        code_import/             # Python package (underscore)
          handler.py             # Lambda entry lambda_handler (Dockerfile CMD)
          config.py              # Pydantic Settings scaffold
          schemas.py             # CodeImportMessage / shared CodeDocument / Webhook
          prompts.py             # prompt strings / versions
          repo.py                # DocumentDB transaction + webhook boundary
          service.py             # Per-record orchestration

File Responsibilities (App Level)

Path	Responsibility
`src/code_import/handler.py`	Lambda `lambda_handler(event, context)` — SQS batch processing and per-record workflow
`src/code_import/config.py`	Validated model/provider, DocumentDB, bucket, and webhook settings
`src/code_import/schemas.py`	Input/output models (`CodeImportMessage`, DocDB / Webhook payloads)
`src/code_import/prompts.py`	Prompt assets / versions
`src/code_import/repo.py`	Scoped transactional replacement, fingerprint lookup, preview patch, webhook
`src/code_import/service.py`	Parse, fingerprint, semantic/context vectorization, persistence, notification

File Responsibilities (Shared Packages)

File	Responsibility
`packages/agentic/agents/code_semantics.py`	Code Analysis LLM (arch node C) + `CodeSemantics` output model
`packages/des2code-core/code_context.py`	Deterministic source/context extraction and semantic identity text
`packages/models/documentdb/code.py`	Collection helpers and semantic/context index definitions
`packages/models/documentdb/code_index.py`	Stable scoped index ID and stale/ready state
`packages/guinness-ai-sdk`	Shared webhook contract and delivery helper

Removed from V1: config/mysql.py, all save_*_to_mysql helpers (no MySQL access in V2).

8. Environment Variables

Variable	Description	Example
Provider API key	Key required by `DESC_MODEL`; one of OpenAI, Google, Anthropic, or OpenRouter	`(secret)`
`DESC_MODEL`	Semantic analysis model with provider prefix	`openai:gpt-5.4-nano`
`EMBEDDING_PROVIDER`	Embedding client	`openai` or `openrouter`
`EMBEDDING_MODEL`	Embedding model	`text-embedding-3-large`
`EMBEDDING_DIMENSIONS`	Vector dimensions	`512`
`DOCUMENTDB_CONNECTION_STRING`	DocumentDB connection string	`mongodb://...`
`DOCUMENTDB_NAME`	DocumentDB database name	`guinness_v2`
`CODE_TABLE_NAME`	DocumentDB collection name	`code`
`DOCUMENTDB_CA_PATH`	TLS CA (Lambda)	`/var/task/global-bundle.pem`
`VECTOR_SEARCH_PROVIDER`	Index backend used during collection initialization	`documentdb` or `atlas`
`S3_BUCKET_NAME`	Bucket accepted for canonical preview references	deployment bucket
`WEBHOOK_BASE_URL`	Backend Webhook URL (private VPC)	`https://api.internal/v1/webhooks/ai-status`
`WEBHOOK_API_KEY`	API key sent as `X-API-Key` (from Secrets Manager)	`(secret)`

Not needed in the worker runtime: queue URL; Lambda receives the event through its event-source mapping.

Removed from V1: DATABASE_RDS_PROXY_ENDPOINT, DATABASE_NAME, DATABASE_USER, DATABASE_PASSWORD, DATABASE_PORT (no MySQL in V2).

9. Package Dependencies

Package	Purpose
`openai`	OpenAI SDK (embeddings)
`pydantic-ai`	PydanticAI (agent orchestration, provider-agnostic)
`pydantic`	Data models, structured output
`pydantic-settings`	Environment variable configuration
`pymongo`	DocumentDB driver
`boto3`	AWS integration used by shared runtime utilities
`structlog` / observability package	Structured logging and metrics
`httpx`	HTTP client (Webhook POST)

Removed from V1: langchain-openai, langgraph, PyMySQL.

10. End-to-End Sequence

sequenceDiagram
    participant SQS as code-import SQS
    participant L as Lambda code-import
    participant SA as Semantic Agent (PydanticAI)
    participant E as Embedding provider
    participant D as DocumentDB (code + code_index)
    participant WH as POST /v1/webhooks/ai-status

    SQS->>L: SQSEvent (1 record)
    L->>L: parse + validate (CodeImportMessage)
    L->>L: validate preview ref + fingerprint
    L->>L: deterministic source parser -> context.metadata + text
    L->>SA: source_code + css_code
    SA-->>L: semantic_words[5..10]
    L->>E: embeddings.create([semantic_identity, context_text], dim=512)
    E-->>L: [semantic_emb, context_emb]
    L->>D: transaction: replace scoped code + mark code_index stale
    D-->>L: ack
    L->>WH: POST {type:"code-import", recordId, status:"success", result:{...}}
    WH-->>L: 200

On exception:

sequenceDiagram
    participant L as Lambda
    participant WH as Webhook
    participant SQS as SQS
    L->>L: catch exception
    L->>WH: POST {type:"code-import", recordId, status:"failed"}
    WH-->>L: 200 (best effort)
    L-->>SQS: batchItemFailures includes messageId
    Note over SQS: SQS redrives according to RedrivePolicy

11. Open Questions / Future Work

Topic	Notes
Source code length limit	Tentative 200 KB. Enforce after confirming with backend.
Preview extensions	Canonical preview references currently allow PNG, JPG, and WebP; rendered-state ingestion is owned by `code-variation-import`.
Cross-organization search	Vector search is scoped by `organization_id` + `project_id`; Des2Code validates both fields before search.