Configuration

How to Configure the Gateway

Below details how to configure Glide.

In order to start, start Glide gateway, we need to pass a config file:

glide --config /path/to/config.yaml

Configuration Structure

Glide’s configuration is a declarative YAML file with the following high level sections:

• routers - defines Glide’s routers per model modality (e.g. routers.language, routers.speech, etc). For more information on router types see the router page.
• api - defines Glide’s server configuration for specific protocol. Read more about this below
• telemetry - defines configuration for Glide’s observability. Read more about this below.

Basic Configuration

At its simplest the gateway can be configured with a single router and two models.

This example shows a failover router leveraging OpenAI and Azure OpenAI. For OpenAI, Cohere, and OctoML, only an API key needs to be provided. All other parameters, including the model, are provided by default. For Azure OpenAI, a model and base_url must be specified.

• language - This is the language API endpoint which support a /chat endpoints from model providers
• id - A unique id for your router
• strategy - The type of router can be round-robin, weighted-round-robin, least-latency, priority
• models/id - A unique ID for the model configuration

routers:
  language:
    - id: my-chat-router
      strategy: priority
      models:
        - id: primary
          openai:
            api_key: "${env:OPENAI_API_KEY}"
        - id: secondary
          azureopenai:
            api_key: "${env:AZURE_OPENAI_API_KEY}"
            model: "glide-GPT-35" # the Azure OpenAI deployment name
            base_url: "https://mydeployment.openai.azure.com/" # the name of your Azure OpenAI Resource

See an example of a more advanced configuration

Secrets

Glide supports two ways of passing sensitive information like API keys:

• via environment variables:

routers:
  language:
    - id: default
      strategy: priority
      models:
        - id: primary
          openai:
            api_key: "${env:OPENAI_API_KEY}"

• via separate files (useful in cloud setups):

routers:
  language:
    - id: default
      strategy: priority
      models:
        - id: primary
          openai:
            api_key: "${file:/paht/to/secret}" # the file should be a plain text file with the content of the secret

Dotenv files

To make it easier for you to manage secrets locally, Glide automatically attempts to load environment variables from .env files located in the directory where it was called.

You can use a separate --env flag to provide a specific path:

glide --env .env.dev --config config.yaml

API Configuration

HTTP is the only supported protocol for Glide’s API right now. The following configurations are exposed for you to tweak:

• api.http.host (default: localhost) - defines address to which Glide binds its server. Usually, it’s localhost or 0.0.0.0
• api.http.read_timeout (default: 3m) - the timeout of data reading from clients
• api.http.write_timeout (default: 3m) - the timeout of data writing to clients
• api.http.idle_timeout (default: 1m) - the free timeout of the request link for persistent keep-alive connections
• api.http.max_request_body_size (default: 4Mb) - max body size of a request

A sample of the configuration:

api:
  http:
    host: 0.0.0.0 # default: localhost
    port: 9099 # default 9099
    read_timeout: 3s
    write_timeout: 3s
    idle_timeout: 1s
    max_request_body_size: 2Mi

# everything else
# ...

Telemetry

Telemetry configs defines logging, metrics (coming soon), and traces (coming soon) generated by Glide. Here are available parameters:

• telemetry.logging.level (default: info) - minimal log level to output. Possible values: debug, info, warning, error
• telemetry.logging.encoding (default: json) - logging encoding. The json outputs structured logs useful for production setup while console is much more readable, so it fits development setups.
• telemetry.logging.disable_caller - controls adding information about place in Glide’s codebase where the log was generated
• telemetry.logging.disable_stacktrace - controls adding information about full stacktrace to the code that generated logs
• telemetry.logging.output_paths (default: [] (an empty array)) - allows to save logging streams to the specified files
• telemetry.logging.output_paths (default: (an empty map)) - key-value pairs to label each Glide’s log record.

A sample of the configuration:

telemetry:
  logging:
    level: info # available values: debug, info, warning, error
    encoding: json # available values: console, json
    disable_caller: false
    disable_stacktrace: false
    output_paths: []
    initial_fields: {}

# everything else
# ...

Advanced Configuration

Additionally, parameters can be specified in the configuration as well. Refer to the providers section for more information on parameters.

telemetry:
  logging:
    level: debug  # debug, info, warn, error, fatal
    encoding: console

api:
  http:
    host: 0.0.0.0
    port: 7685
    max_body_size: "2Mi"

routers:
  language:
    - id: openai-pool
      strategy: priority # round-robin, weighted-round-robin, priority, least-latency, priority, etc.
      models:
        - id: primary
          openai: # cohere, azureopenai, gemini, other providers we support
            model: gpt-3.5-turbo
            api_key: ${env:OPENAI_API_KEY}
            default_params:
              temperature: 0.1

        - id: secondary
          cohere:
            model: command-light
            api_key: ${env:COHERE_API_KEY}
            default_params: # set the default request params
              temperature: 0.1

    - id: latency-critical-pool
      strategy: least_latency
      models:
        - id: primary
          timeout_ms: 200
          openai:
            model: gpt-3.5-turbo
            api_key: ${env:OPENAI_API_KEY}

        - id: secondary
          timeout_ms: 200
          cohere:
            api_key: ${env:COHERE_API_KEY}

    - id: cohere-openai-ab-test
      strategy: round_robin
      models:
        - id: openai
          openai:
            api_key: ${env:OPENAI_API_KEY}
        - id: cohere
          cohere:
            api_key: ${env:COHERE_API_KEY}

    - id: weight-test
      strategy: weighted_round_robin
      models:
        - id: openai
          weight: 30
          openai:
            api_key: ${env:OPENAI_API_KEY}
        - id: cohere
          weight: 70
          octoml:
            api_key: ${env:OCTOML_API_KEY}