AI and ML Integration

AI Co-Pilot Assistant

Technical architecture of the generative AI strategy assistant — prompt hydration with live market context, multimodal image ingestion, multi-provider abstraction (Google Gemini, OpenAI, Qwen, OpenRouter, Vertex Agent Builder), response validation, and security filtering.

⏱️ 6 min read📊 Level: Intermediate

The AIAssistant (api/ai_assistant.py, ~3,760 lines) allows users to generate strategy configurations from plain-text descriptions or chart screenshots. It acts as a bridge between the visual editor frontend and Large Language Models (Google Gemini / OpenAI / Qwen / OpenRouter / Vertex Agent Builder), transforming natural language into executable strategy JSON.


AI Ingestion Architecture

The Co-Pilot service accepts text and media assets, enriches them with real-time market data, and requests structured JSON outputs from the LLM.

Rendering diagram...

Provider Selection

Sources:

The provider is chosen by the AI_PROVIDER environment variable (or by admin settings persisted in the database) and validated against the supported set — Google's direct GenAI SDK, OpenAI, Qwen, OpenRouter, or Vertex Agent Builder (_get_active_ai_provider(), lines 748–770).


Real-Time Context Hydration

To prevent the LLM from generating obsolete or irrelevant strategy recommendations, DepthSight "hydrates" the user's prompt with real-time telemetry before sending the query.

enrich_market_context_for_ai() (lines 566–634)

The function:

  1. Parses the user prompt for ticker symbols matching \b[A-Z0-9]{2,10}USDT\b.
  2. Filters out stop-words (LONG, SHORT, GRID, DCA, AND, OR).
  3. Queries the Screener API (http://localhost:8050/api/v1/metrics/{symbol}) for up to 2 detected symbols:
Sources:

The resulting context block is injected into the LLM prompt, providing the AI with current market conditions to base its strategy recommendations on.

Context Content Injected

Data PointSourcePurpose
Last PriceScreener APIEntry/exit level calibration
NATR (Volatility)Screener APIStop-loss distance suggestion
Macro TrendScreener APIDirectional bias alignment
Oracle RegimeScreener APIRisk mode adjustment
24h VolumeScreener APILiquidity assessment

Image-to-Strategy Ingestion

Gemini and modern multimodal models can read image pixels directly. When a user uploads a screenshot of a trading setup:

1. Image Normalization (lines 1339–1356)

Sources:

The image is stripped of any data: URI prefix and paired with a normalized MIME type (defaulting to image/jpeg); the result is a (data, mime_type) tuple.

2. Provider-Specific Attachment

Google Gemini (lines 1373–1381): Converts base64 to bytes via base64.b64decode() and attaches as types.Part.from_bytes(data=..., mime_type=...).

OpenRouter (lines 1503–1511): Embeds as a {"type": "image_url"} content part whose URL is a data:{mime};base64,{data} URI.

3. Visual Parsing

The system prompt instructs the model to:

  • Extract technical indicators visible on the screen (Bollinger Bands, RSI, MACD).
  • Identify chart structures (ascending triangle, breakout levels, moving average crosses).
  • Map visual patterns to mathematical block JSON nodes:
    • Ascending triangle → local_level + level_touch_analyzer + price_action_analyzer.
    • RSI divergence → rsi_condition with specific threshold + trend filter.

4. JSON Mapping

The model generates block JSON nodes mapped to the visual builder canvas schema, which the frontend renders as draggable logic blocks.


Response Validation

LLMs can suffer from hallucinations, producing invalid JSON or referencing nonexistent logic blocks. DepthSight implements multi-layer validation:

Layer 1 — Python Code Detection (lines 523–562)

A critical security filter scans for Python code injection:

Sources:

In Advisor mode, if Python code is detected in the text response, it is blocked and a safe message is returned (api/ai_assistant.py:2455-2465).

Layer 2 — JSON Extraction (lines 1942–1948)

If the AI wraps JSON in markdown or explanatory text, the first {...} block is extracted via str.find("{") / str.rfind("}"):

Sources:

Layer 3 — Block Schema Validation

Generated strategies are checked against the user's plan restrictions (pro_only / kline_only blocks are rejected for ineligible users, api/ai_assistant.py:440-451); structurally unknown blocks are repaired by default-parameter injection rather than rejected.

Layer 4 — Parameter Default Injection (lines 244–301)

Missing parameters are merged with defaults via _ensure_default_params():

Sources:

The recursive walk also assigns a missing block id (lines 254–257) and migrates legacy block types (lines 259–269). This prevents bot engine crashes from missing configuration fields.

Layer 5 — Pydantic Validation

The final JSON is validated against schemas.StrategyV2ConfigData.model_validate() (lines 2179–2181), ensuring type correctness.


Full Query Flow

Generator Mode (get_chat_response(), line 2002)

Used for strategy generation from text/screenshots:

  1. Achievement Grant (line 2014): Unlocks "used_ai_assistant" achievement.
  2. Quota Check (lines 2029–2034): QuotaManager.check_and_consume("use_ai_assistant").
  3. Chat History Retrieval: Loads full session history; the modification path scans it for the previous advisor recommendation, and the scratch-generation path feeds the last 6 messages to the model.
  4. Modification vs Generation: Detects if the user is asking to modify an existing strategy or create a new one.
  5. RAG Injection: Calls enrich_market_context_for_ai().
  6. Tier Context: Informs the AI about the user's plan restrictions.
  7. Backtest Context: If a backtest_id is provided, appends backtest analytics to the prompt.
  8. LLM Call: _generate_json_response() with structured JSON output.
  9. Validation Pipeline: JSON extraction → schema validation → default injection → Pydantic validation.
  10. Return: schemas.AIChatResponse with strategy_json.

Advisor Mode

Used for strategy analysis and questions:

  1. Chat history retrieval.
  2. RAG injection with current market context.
  3. Tier context.
  4. Backtest Analytics: Parses decision traces into combination/individual foundation stats, best/worst trades.
  5. Real Trade Analytics: If available, includes live trade performance.
  6. LLM Call: _generate_text_response() for natural language output.
  7. Python Code Security Check: Scans for code injection.
  8. Return: AIChatResponse with text_response.