Skip to main content

OpenTelemetry Integration

Integrate TealTiger with OpenTelemetry to export governance decisions, cost metrics, and audit events to your existing observability infrastructure.

Why integrate TealTiger with OpenTelemetry?

OpenTelemetry is the standard for observability. Integrating TealTiger gives you:
  • Unified observability - All telemetry in one place
  • Distributed tracing - Link governance decisions to requests
  • Custom dashboards - Visualize policy decisions and costs
  • Alerting - Get notified of policy violations

Quick start

Install both packages:
npm install tealtiger @opentelemetry/sdk-node
# or
pip install tealtiger opentelemetry-sdk
Configure TealTiger to export to OpenTelemetry:
import { TealTiger } from 'tealtiger';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

// Initialize OpenTelemetry
const sdk = new NodeSDK({
  serviceName: 'my-ai-agent',
  instrumentations: [getNodeAutoInstrumentations()]
});

sdk.start();

// Configure TealTiger to export to OTel
const teal = new TealTiger({
  policies: { /* ... */ },
  telemetry: {
    opentelemetry: {
      enabled: true,
      exportDecisions: true,
      exportMetrics: true,
      exportTraces: true
    }
  }
});

// Use TealTiger - decisions automatically exported
const decision = await teal.evaluate(request);

What gets exported?

TealTiger exports three types of telemetry to OpenTelemetry:

1. Traces (spans)

Every policy evaluation creates a span:
Span: tealtiger.evaluate
  - span.kind: INTERNAL
  - tealtiger.decision.action: DENY
  - tealtiger.decision.reason_codes: ["TOOL_NOT_ALLOWED"]
  - tealtiger.decision.risk_score: 85
  - tealtiger.policy.id: "security.tool_access.v1"
  - tealtiger.policy.mode: "ENFORCE"
  - tealtiger.correlation_id: "req-abc123"

2. Metrics

Cost and usage metrics:
Metrics:
  - tealtiger.requests.total (counter)
  - tealtiger.requests.denied (counter)
  - tealtiger.cost.total (counter)
  - tealtiger.cost.per_request (histogram)
  - tealtiger.tokens.input (histogram)
  - tealtiger.tokens.output (histogram)
  - tealtiger.latency (histogram)

3. Logs

Structured audit logs:
{
  "timestamp": "2026-03-06T10:30:00Z",
  "severity": "INFO",
  "body": "Policy evaluation completed",
  "attributes": {
    "tealtiger.decision.action": "DENY",
    "tealtiger.decision.reason_codes": ["TOOL_NOT_ALLOWED"],
    "tealtiger.correlation_id": "req-abc123"
  }
}

Complete example: Full observability stack

Here’s a complete example with traces, metrics, and logs:
import { TealTiger } from 'tealtiger';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';

// 1. Initialize OpenTelemetry SDK
const sdk = new NodeSDK({
  serviceName: 'ai-agent-service',
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces'
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: 'http://localhost:4318/v1/metrics'
    }),
    exportIntervalMillis: 60000  // Export every minute
  }),
  instrumentations: [getNodeAutoInstrumentations()]
});

sdk.start();

// 2. Configure TealTiger with OTel integration
const teal = new TealTiger({
  policies: {
    tools: {
      file_delete: { allowed: false },
      web_search: { allowed: true }
    },
    budget: {
      maxCostPerRequest: 0.50,
      maxCostPerDay: 100.00
    }
  },
  
  // OpenTelemetry configuration
  telemetry: {
    opentelemetry: {
      enabled: true,
      
      // Export decisions as spans
      exportDecisions: true,
      spanAttributes: {
        'service.name': 'ai-agent-service',
        'deployment.environment': process.env.NODE_ENV
      },
      
      // Export cost metrics
      exportMetrics: true,
      metricPrefix: 'tealtiger',
      
      // Export audit logs
      exportLogs: true,
      logLevel: 'INFO'
    }
  }
});

// 3. Use TealTiger with automatic OTel export
import { trace } from '@opentelemetry/api';

async function handleRequest(request: any) {
  const tracer = trace.getTracer('ai-agent');
  
  return await tracer.startActiveSpan('handle_request', async (span) => {
    try {
      // Evaluate policy (creates child span)
      const decision = await teal.evaluate({
        action: 'tool.execute',
        tool: request.tool,
        context: {
          userId: request.userId,
          environment: 'production'
        }
      });
      
      // Add decision to span
      span.setAttribute('tealtiger.decision', decision.action);
      span.setAttribute('tealtiger.risk_score', decision.risk_score);
      
      if (decision.action === 'DENY') {
        span.setStatus({ code: 2, message: 'Policy denied' });
        throw new Error(`Blocked: ${decision.reason_codes.join(', ')}`);
      }
      
      // Execute the request
      const result = await executeRequest(request);
      
      span.setStatus({ code: 1 });
      return result;
      
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: 2, message: error.message });
      throw error;
    } finally {
      span.end();
    }
  });
}

Trace attributes

TealTiger adds these attributes to spans:
AttributeTypeDescription
tealtiger.decision.actionstringALLOW, DENY, TRANSFORM, etc.
tealtiger.decision.reason_codesarrayWhy the decision was made
tealtiger.decision.risk_scorenumberRisk score (0-100)
tealtiger.policy.idstringPolicy identifier
tealtiger.policy.versionstringPolicy version
tealtiger.policy.modestringMONITOR or ENFORCE
tealtiger.correlation_idstringRequest correlation ID
tealtiger.cost.estimatednumberEstimated cost in USD
tealtiger.tokens.inputnumberInput tokens
tealtiger.tokens.outputnumberOutput tokens

Metrics exported

TealTiger exports these metrics:

Counters

  • tealtiger.requests.total - Total requests evaluated
  • tealtiger.requests.allowed - Requests allowed
  • tealtiger.requests.denied - Requests denied
  • tealtiger.requests.transformed - Requests transformed
  • tealtiger.cost.total - Total cost in USD

Histograms

  • tealtiger.cost.per_request - Cost distribution per request
  • tealtiger.tokens.input - Input token distribution
  • tealtiger.tokens.output - Output token distribution
  • tealtiger.latency - Policy evaluation latency
  • tealtiger.risk_score - Risk score distribution

Visualizing in Grafana

Create dashboards to visualize TealTiger data:

Dashboard 1: Policy decisions

# Total requests
sum(rate(tealtiger_requests_total[5m]))

# Denial rate
sum(rate(tealtiger_requests_denied[5m])) / sum(rate(tealtiger_requests_total[5m]))

# Decisions by action
sum by (action) (rate(tealtiger_requests_total[5m]))

Dashboard 2: Cost tracking

# Total cost
sum(tealtiger_cost_total)

# Cost per request (p50, p95, p99)
histogram_quantile(0.50, rate(tealtiger_cost_per_request_bucket[5m]))
histogram_quantile(0.95, rate(tealtiger_cost_per_request_bucket[5m]))
histogram_quantile(0.99, rate(tealtiger_cost_per_request_bucket[5m]))

# Cost trend
rate(tealtiger_cost_total[1h])

Dashboard 3: Risk analysis

# Average risk score
avg(tealtiger_risk_score)

# High-risk requests (score > 80)
count(tealtiger_risk_score > 80)

# Risk distribution
histogram_quantile(0.95, rate(tealtiger_risk_score_bucket[5m]))

Alerting rules

Set up alerts for policy violations:
# Alert on high denial rate
- alert: HighPolicyDenialRate
  expr: |
    sum(rate(tealtiger_requests_denied[5m])) / 
    sum(rate(tealtiger_requests_total[5m])) > 0.10
  for: 5m
  annotations:
    summary: "High policy denial rate (>10%)"

# Alert on budget exceeded
- alert: BudgetExceeded
  expr: tealtiger_cost_total > 100
  for: 1m
  annotations:
    summary: "Daily budget exceeded ($100)"

# Alert on high-risk requests
- alert: HighRiskRequests
  expr: count(tealtiger_risk_score > 90) > 10
  for: 5m
  annotations:
    summary: "Multiple high-risk requests detected"

Best practices

  1. Use correlation IDs - Link traces across services
  2. Set service name - Identify your service in traces
  3. Export to OTLP - Use the standard protocol
  4. Create dashboards - Visualize policy decisions
  5. Set up alerts - Get notified of violations
  6. Sample traces - Don’t export every trace in production

Common issues

Issue 1: Traces not appearing

Problem: Traces aren’t showing up in your backend Solution: Check OTLP endpoint configuration:
traceExporter: new OTLPTraceExporter({
  url: 'http://localhost:4318/v1/traces',  // Correct endpoint
  headers: {
    'Authorization': 'Bearer YOUR_TOKEN'  // If required
  }
})

Issue 2: High cardinality metrics

Problem: Too many unique metric combinations Solution: Limit attribute cardinality:
telemetry: {
  opentelemetry: {
    metricAttributes: {
      'tealtiger.decision.action': true,  // Low cardinality
      'tealtiger.policy.id': true,        // Low cardinality
      'tealtiger.correlation_id': false   // High cardinality - exclude
    }
  }
}

Issue 3: Performance impact

Problem: Telemetry export slowing down requests Solution: Use batch export and sampling:
span_processor = BatchSpanProcessor(
    OTLPSpanExporter(),
    max_export_batch_size=512,
    schedule_delay_millis=5000
)

Next steps