Telemetry

Telemetry in Raikoo provides visibility into your AI operations by capturing execution data and sending it to your observability platform. This enables organizations to monitor performance, debug issues, and optimize their workflows and chat interactions using their existing monitoring infrastructure.

What is Telemetry?

Telemetry refers to the automated collection and transmission of data about your AI operations. In the context of Raikoo, telemetry captures information about workflow executions, chat interactions, AI model calls, and operation performance.

Observability matters for AI workloads because:

• Cost Management - AI operations can be expensive; telemetry helps you understand where resources are spent
• Performance Optimization - Identify slow operations and bottlenecks in your workflows
• Debugging - When things go wrong, telemetry provides the context needed to diagnose issues
• Compliance - Maintain audit trails of AI operations for regulatory requirements

Raikoo uses a "Bring Your Own Telemetry" (BYOT) approach, allowing each organization to route telemetry data to their own observability platform rather than a shared system.

What You Get

When telemetry is configured, Raikoo captures hierarchical traces across both workflow executions and chat interactions. The raikoo.execution.type attribute distinguishes between these contexts.

Workflow Execution

Workflow telemetry provides full hierarchical tracing:

flowchart TB
    subgraph "Workflow Execution"
        A["workflow.execution (root)"] --> B["thread.MyWorkflow"]
        B --> C["operation.prompt"]
        B --> D["operation.write-file"]
        B --> E["operation.run-thread"]
        C --> F["LLM Call"]
        E --> G["thread.NestedWorkflow"]
        G --> H["operation.prompt"]
    end

This hierarchy allows you to:

• Trace End-to-End - Follow a request from workflow start to completion
• Identify Bottlenecks - See which operations consume the most time
• Debug Failures - Pinpoint exactly where and why an operation failed
• Track File Operations - Monitor file creates, updates, deletes, moves, and copies
• Analyze Nested Workflows - Track execution across nested thread invocations

Chat Execution

Chat telemetry captures the conversation context and LLM interactions:

flowchart TB
    subgraph "Chat Execution"
        A["workflow.execution (root)"] --> B["LLM Call 1"]
        A --> C["LLM Call 2"]
        A --> D["LLM Call N"]
    end

Chat spans include agent configuration, tool availability, and message counts.

How It Works

Raikoo's telemetry is built on OpenTelemetry, the industry-standard framework for observability. This ensures compatibility with a wide range of monitoring platforms and tools.

Data Flow

Telemetry is configured at the organization level, meaning all projects and workflows within an organization share the same telemetry configuration. Data flows directly from Raikoo to your observability platform—Raikoo does not store or access your telemetry data.

flowchart LR
    A[Raikoo Engine] --> B[OpenTelemetry SDK]
    B --> C[Your Observability Platform]

    subgraph "Your Infrastructure"
        C --> D[Dashboards]
        C --> E[Alerts]
        C --> F[Analytics]
    end

Span Attributes Reference

All spans use a consistent raikoo.* attribute namespace.

Core Attributes

Present on all execution spans:

Attribute	Type	Description
raikoo.execution.type	string	"chat" or "workflow"
raikoo.organization.id	string	Organization ID
raikoo.organization.name	string	Organization name
raikoo.project.id	string	Project ID
raikoo.project.name	string	Project name
raikoo.user.id	string	User ID

Model Attributes

Present when model configuration is available:

Attribute	Type	Description
raikoo.model.provider_id	string	LLM provider ID (direct selector)
raikoo.model.model_id	string	Model ID (direct selector)
raikoo.model.family_id	string	Model family ID (family selector)
raikoo.model.family_name	string	Model family name (family selector)
raikoo.model.temperature	number	Temperature setting
raikoo.model.max_tokens	number	Max tokens setting

Chat-Specific Attributes

Present when raikoo.execution.type is "chat":

Attribute	Type	Description
raikoo.agent.id	string	ChatAgent ID (if configured)
raikoo.agent.name	string	ChatAgent name (if configured)
raikoo.tool.count	number	Number of tools available
raikoo.tool.names	string	Comma-separated tool names
raikoo.message.count	number	Number of messages in conversation

Workflow-Specific Attributes

Present when raikoo.execution.type is "workflow":

Attribute	Type	Description
raikoo.thread.id	string	Thread/workflow ID
raikoo.thread.name	string	Thread/workflow name
raikoo.thread.is_root	boolean	Whether this is the root thread
raikoo.iteration.id	string	Iteration path (e.g., "0.1.2")

Operation Attributes

Present on operation spans within workflows:

Attribute	Type	Description
raikoo.operation.id	string	Operation ID
raikoo.operation.name	string	Operation name
raikoo.operation.type	string	Operation type (operationId)
raikoo.operation.status	string	Final status (complete/error/cancelled)
raikoo.operation.params	string	Comma-separated operation parameter names
raikoo.thread.params	string	Comma-separated thread parameter names
raikoo.output.type	string	Output type or "none"
raikoo.output.path	string	Resolved output file path

File Operation Attributes

Track workspace file changes during workflow execution:

Attribute	Type	Description
raikoo.files.creates	number	Files created
raikoo.files.updates	number	Files updated
raikoo.files.deletes	number	Files deleted
raikoo.files.moves	number	Files moved
raikoo.files.copies	number	Files copied
raikoo.files.total_bytes	number	Total bytes written

Supported Providers

Currently, Raikoo supports:

• Azure Application Insights - Microsoft's application performance monitoring service, part of Azure Monitor

The telemetry system is designed to be extensible, with support for additional providers planned for future releases.

Privacy Controls

Privacy settings control what data is included in LLM call spans:

Option	Description
Record Inputs	Whether prompts sent to AI models are included
Record Outputs	Whether AI model responses are included

Both options are enabled by default to provide maximum visibility. Consider disabling them when:

• Handling Sensitive Data - If prompts or responses contain PII, credentials, or confidential information
• Compliance Requirements - When regulations restrict where certain data can be transmitted
• Cost Optimization - Reducing telemetry volume can lower storage costs in your observability platform

Workflow, operation, and chat spans always include:

• Timing information (start time, duration)
• Success/failure status
• Error messages (when failures occur)
• Identifiers and context attributes listed above

Best Practices

• Start with Full Telemetry - Enable both inputs and outputs initially to establish baseline patterns and understand your workflows
• Review Privacy Implications - Before moving to production, assess whether input/output recording is appropriate for your data
• Set Up Alerts - Use your observability platform's alerting capabilities to catch issues early
• Monitor Costs - Keep an eye on telemetry volume and associated storage costs
• Use Telemetry for Optimization - Regularly review performance data to identify opportunities to improve workflow efficiency
• Filter by Execution Type - Use raikoo.execution.type to separate chat and workflow analytics

References

• OpenTelemetry Documentation
• Azure Application Insights Overview
• OpenTelemetry Setup Guide - Step-by-step configuration instructions