LLM Usage Example

Learn how to make LLM requests using BlueNexus confidential AI models

Assumptions:

1. You have already obtained an access token from your user

2. Your access token has access to the necessary scopes to perform LLM operations

Introduction

BlueNexus provides OpenAI-compatible LLM endpoints that support confidential computing environments. The service offers both regular and confidential AI models with automatic scope-based access control.

Select Your Model

Before making LLM requests, you need to choose a model. BlueNexus supports multiple providers and models based on your authorization scope.

Listing Available Models

// Import dependencies
import OpenAI from "openai";

// Initialize the client with your access token
const client = new OpenAI({
  apiKey: YourAccessToken, // Your BlueNexus access token
  baseURL: "https://api.bluenexus.ai/api/v1",
  timeout: 30000, // 30 second timeout
});

// List all available models
async function listModels() {
  const response = await client.models.list();

  console.log("Available models:");
  for (const model of response.data) {
    console.log(`- ${model.id}: ${model.owned_by}`);
  }

  return response.data;
}

// Example models you might see:
// - redpill:phala/gpt-oss-120b (Confidential)
// - redpill:phala/qwen-2.5-7b-instruct (Confidential)
// - redpill:phala/gemma-3-27b-it (Confidential)

Understanding Model Scopes

Models are categorized by scope:

• Confidential models: Run in Trusted Execution Environments (TEEs) for enhanced privacy

• Other models: Standard cloud-based models

Your access token determines which models you can use:

• llm:all - Access to all models

• llm:confidential - Access only to confidential models

• llm:other - Access only to non-confidential models

Make LLM Requests

Basic Chat Completion

For simple request-response interactions:

// Import dependencies
import OpenAI from "openai";

// Initialize client
const client = new OpenAI({
  apiKey: YourAccessToken, // Your BlueNexus access token
  baseURL: "https://api.bluenexus.ai/api/v1",
  timeout: 120000, // 2 minute timeout for LLM requests
  maxRetries: 3,
});

// Create a chat completion
const completion = await client.chat.completions.create({
  model: "redpill:phala/gpt-oss-120b",
  messages: [
    {
      role: "system",
      content: "You are a helpful AI assistant.",
    },
    {
      role: "user",
      content: "What are the benefits of confidential computing?",
    },
  ],
  temperature: 0.7,
  max_tokens: 4000,
});

// Extract the assistant's response
const assistantMessage = completion.choices[0].message.content;
console.log("Assistant:", assistantMessage);

// Access usage information
console.log("Tokens used:", {
  prompt: completion.usage?.prompt_tokens,
  completion: completion.usage?.completion_tokens,
  total: completion.usage?.total_tokens,
});

Streaming Chat Completion

For real-time responses where you want to display content as it's generated:

// Import dependencies
import OpenAI from "openai";

// Initialize client
const client = new OpenAI({
  apiKey: YourAccessToken,
  baseURL: "https://api.bluenexus.ai/api/v1",
  timeout: 120000,
  maxRetries: 3,
});

// Create a streaming chat completion
const stream = await client.chat.completions.create({
  model: "redpill:phala/gpt-oss-120b",
  messages: [
    {
      role: "user",
      content: "Explain how TEE-based AI models protect user privacy.",
    },
  ],
  temperature: 0.7,
  max_tokens: 4000,
  stream: true, // Enable streaming
});

let fullContent = "";

// Process each chunk as it arrives
for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta;

  // Stream text content in real-time
  if (delta?.content) {
    fullContent += delta.content;
    process.stdout.write(delta.content); // Display content as it arrives
  }

  // Check if streaming is complete
  if (chunk.choices[0]?.finish_reason === "stop") {
    console.log("\n\nStreaming complete!");
    break;
  }
}

Multi-Turn Conversations

Maintain conversation context by appending messages to the conversation history:

// Import dependencies
import OpenAI from "openai";
import type { ChatCompletionMessageParam } from "openai/resources/chat/completions";

// Initialize client
const client = new OpenAI({
  apiKey: YourAccessToken,
  baseURL: "https://api.bluenexus.ai/api/v1",
});

// Conversation state
const conversationHistory: ChatCompletionMessageParam[] = [
  {
    role: "system",
    content: "You are a helpful AI assistant.",
  },
];

// Helper function to add messages and get response
async function chat(userMessage: string): Promise<string> {
  // Add user message to history
  conversationHistory.push({
    role: "user",
    content: userMessage,
  });

  // Get LLM response
  const completion = await client.chat.completions.create({
    model: "redpill:phala/gpt-oss-120b",
    messages: conversationHistory,
    temperature: 0.7,
  });

  const assistantMessage = completion.choices[0].message.content || "";

  // Add assistant response to history
  conversationHistory.push({
    role: "assistant",
    content: assistantMessage,
  });

  return assistantMessage;
}

// Example multi-turn conversation
const response1 = await chat("What is a TEE?");
console.log("Assistant:", response1);

const response2 = await chat("How does it improve security?");
console.log("Assistant:", response2);

const response3 = await chat("Can you give me an example?");
console.log("Assistant:", response3);