Skip to main content

AI Gateway - Configuring LLMs

Spice provides a high-performance, OpenAI API-compatible AI Gateway optimized for managing and scaling large language models (LLMs). Additionally, Spice offers tools for Enterprise Retrieval-Augmented Generation (RAG), such as SQL query across federated datasets and an advanced search feature (see Search).

Spice also supports full OpenTelemetry observability, enabling detailed tracking of data flows and requests for full transparency and easier debugging.

Configuring Language Models​

Spice supports a variety of LLMs (see Model Components).

Core Features​

  • Custom Tools: Equip models with tools to interact with the Spice runtime.
  • System Prompts: Customize system prompts and override defaults for v1/chat/completion.

For detailed configuration and API usage, refer to the API Documentation.

Example: Configuring an OpenAI Model​

To use a language model hosted on OpenAI (or compatible), specify the openai path and model ID in from.

Example spicepod.yml:

models:
  - from: openai:gpt-4o-mini
    name: openai
    params:
      openai_api_key: ${ secrets:SPICE_OPENAI_API_KEY }

  - from: openai:llama3-groq-70b-8192-tool-use-preview
    name: groq-llama
    params:
      endpoint: https://api.groq.com/openai/v1
      openai_api_key: ${ secrets:SPICE_GROQ_API_KEY }

For details, see OpenAI (or Compatible) Language Models.