Secure AI Development: LLM Reference Architecture

Layer-by-Layer: What Each Component Does

Here’s how the architecture breaks down across layers, with each component handling a specific part of the request flow:

• Application layer (your LLM app — Python, Node, or TypeScript) This is where your core logic lives: prompt construction, request handling, and response processing. No changes are required here—keep using the existing OpenAI SDK and avoid building a custom client.
• SDK layer (OpenAI SDK for Python or Node)
  The SDK ensures requests follow the correct wire format (/v1/chat/completions). The only required change is updating the base_url to the Vaikora endpoint and loading the API key from your secrets manager.
• Gateway layer (Vaikora inline AI gateway)
  This is the control point. It handles detection, PII redaction, audit logging, routing, and budget enforcement before any request reaches a model. Configuration typically includes selecting a compliance preset based on data sensitivity, enabling content-free logging, and integrating SSO.
• Provider layer (LLM providers)
  This is where inference happens. You can define a primary model and one or two fallback providers in routing policies to balance latency, cost, and availability.
• Identity layer (Okta, Azure AD, Google Workspace via SAML/SCIM)
  Manages user and service identity, including authentication, group synchronization, and deprovisioning. SAML is used for SSO, while SCIM supports automated lifecycle management.
• Observability layer (SIEM tools like Splunk, Microsoft Sentinel, Elastic, or Sumo Logic)
  Captures audit and security events. The recommended approach is to ingest content-free metadata along with a SHA-256 hash-chained audit trail for compliance and forensic analysis.

Data Flow and Control Flow Through the Stack

Request data flow

A user prompt enters the application, which hands it to the OpenAI SDK. The SDK forms an HTTPS request to the gateway. The gateway authenticates the request against the identity provider (for human-initiated calls) or against the workspace API key (for service-initiated calls), runs the four detection layers in parallel, applies redaction if the policy says so, writes a content-free audit record with a SHA-256 hash of the inspected payload, and forwards the request to the upstream LLM provider selected by the routing policy. The response comes back through the same gateway, gets a response-side inspection (catching prompt-injection embedded in tool results, for example), and reaches the application as a normal OpenAI-shaped response.

Control flow

Policy decisions are deterministic where possible (block-list patterns, exact-match PII detectors, hard limits) and probabilistic where needed (semantic similarity, ML classifiers, behavioral baselines). The combination is described as deterministic policy enforcement with probabilistic risk scoring. When a request fails policy, the gateway returns a structured error to the client and records the block decision in the audit log; the upstream LLM is never called. When a primary provider fails (429, 5xx, timeout), the gateway transparently routes to the next provider in the fallback chain — every Vaikora control runs identically on the new path.

The 10-Item Secure AI Development Checklist

This is the configuration that turns the reference architecture into a deployment. Run through it once per workspace before traffic moves to production.

1. Redaction mode set per data class. synthetic for traceable testing, mask for production user data, hash for high-sensitivity fields. The default for HIPAA / PCI / GDPR workspaces is mask + reversible mapping.
2. Audit logging set to content: false. Metadata-only mode is the default; the SHA-256 hash chain still satisfies SOC 2, HIPAA, GDPR, and PCI DSS evidence requirements without storing prompts.
3. Compliance preset attached to the workspace. Choose standard, strict, permissive, hipaa, pci-dss, or gdpr based on the workload’s data class. Override individual rules from the preset; do not start from a blank policy.
4. SAML SSO enabled for human access. Console access for developers, security engineers, and auditors flows through your existing identity provider (Okta / Azure AD / Google Workspace). No local accounts.
5. SCIM provisioning enabled for automated lifecycle. Group-to-role mappings deprovision access automatically when an employee leaves; do not rely on manual offboarding.
6. Fail mode configured (fail-closed for regulated workloads). If the gateway cannot reach the policy engine, fail-closed (block the request) is the default for hipaa / pci-dss / gdpr presets. Fail-open is acceptable for permissive presets only.
7. Budget cap and rate limit set per workspace. A per-workspace daily token budget and per-key request-per-minute cap stops a runaway loop or compromised key from racking up six-figure spend.
8. Provider fallback declared (primary + 1–2 backups). Resilience is part of the security posture; a regulated app that goes down is an availability incident. Default 3-leg: OpenAI → Anthropic → Azure OpenAI.
9. SIEM ingest configured (content-free metadata + hash chain). Audit events flow into Splunk / Microsoft Sentinel / Elastic / Sumo Logic. Detection-rule library aligned with the four-layer detection model.
10. Prompt-injection coverage validated on response side, not just request. The 4-layer detection runs on tool outputs and retrieved RAG content as well as on user prompts — that is where the most consequential injections actually arrive.

Default Config Recommendations (Drop-In)

For a team adopting the reference architecture from scratch, the following defaults match the most common enterprise deployment shape. Override only what your workload actually requires.