OpenTelemetry

Ende has built in support for OpenTelemetry.

OpenTelemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

- https://opentelemetry.io/

This integration enables you to monitor your Ende applications using OpenTelemetry observability tooling with instruments like logs, metrics, and traces.

Ende provides the following features:

• Exporting of collected metrics, traces, and logs to a server using the OpenTelemetry protocol.
• Automatic instrumentation of the Ende runtime with OpenTelemetry metrics, traces, and logs.
• Collection of user defined metrics, traces, and logs created through Ende.telemetry.

Quick start

To enable the OpenTelemetry integration set the environment variable OTEL_ENDE=true:

OTEL_ENDE=true ende run my_script.ts

This will automatically collect and export runtime observability data to an OpenTelemetry endpoint at localhost:4318 using Protobuf over HTTP (http/protobuf).

OTEL_ENDE=true OTEL_EXPORTER_OTLP_PROTOCOL=console ende run my_script.ts

SPAN inner span [00000000000000000000000000000001/0000000000000002] Internal 0.495ms
  parent: 0000000000000001
  scope: example-tracer
  key: value
LOG [INFO] 2025-03-14T13:47:07.235Z "hello from inner"
  scope: [email protected]
  trace: 00000000000000000000000000000001/0000000000000002

docker run --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti \

-v "$PWD"/lgtm/grafana:/data/grafana \
-v "$PWD"/lgtm/prometheus:/data/prometheus \
-v "$PWD"/lgtm/loki:/data/loki \
-e GF_PATHS_DATA=/data/grafana \
docker.io/grafana/otel-lgtm:0.8.1

You can then access the Grafana dashboard at `http://localhost:3000` with the
username `admin` and password `admin`.

This will automatically collect and export runtime observability data like console.log, traces for HTTP requests, and metrics for the Ende runtime. Learn more about auto instrumentation.

You can also create your own metrics, traces, and logs through Ende.telemetry. Learn more about user defined metrics.

Auto instrumentation

Ende automatically collects and exports some observability data to the OTLP endpoint.

This data is exported in the built-in instrumentation scope of the Ende runtime. This scope has the name ende. The version of the Ende runtime is the version of the ende instrumentation scope. (e.g. ende:2.1.4).

Traces

Ende automatically creates spans for various operations, such as:

• Incoming HTTP requests served with Ende.serve.
• Outgoing HTTP requests made with fetch.
• HTTP/2 traffic over node:http2, both client requests and incoming server requests, with trace context propagated across services.
• Ende.cron() job invocations.

Ende.serve

When you use Ende.serve to create an HTTP server, a span is created for each incoming request. The span automatically ends when response headers are sent (not when the response body is done sending).

The name of the created span is ${method}. The span kind is server.

The following attributes are automatically added to the span on creation:

• http.request.method: The HTTP method of the request.
• url.full: The full URL of the request (as would be reported by req.url).
• url.scheme: The scheme of the request URL (e.g. http or https).
• url.path: The path of the request URL.
• url.query: The query string of the request URL.

After the request is handled, the following attributes are added:

• http.response.status_code: The status code of the response.

Ende does not automatically add a http.route attribute to the span as the route is not known by the runtime, and instead is determined by the routing logic in a user's handler function.

fetch

When you use fetch to make an HTTP request, a span is created for the request. The span automatically ends when the response headers are received.

The name of the created span is ${method}. The span kind is client.

The following attributes are automatically added to the span on creation:

• http.request.method: The HTTP method of the request.
• url.full: The full URL of the request.
• url.scheme: The scheme of the request URL.
• url.path: The path of the request URL.
• url.query: The query string of the request URL.

After the response is received, the following attributes are added:

• http.status_code: The status code of the response.

Metrics

The following metrics are automatically collected and exported:

Ende.serve

http.server.request.duration

A histogram of the duration of incoming HTTP requests served with Ende.serve. The time that is measured is from when the request is received to when the response headers are sent. This does not include the time to send the response body. The unit of this metric is seconds. The histogram buckets are [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0].

This metric is recorded with the following attributes:

• http.request.method: The HTTP method of the request.
• url.scheme: The scheme of the request URL.
• network.protocol.version: The version of the HTTP protocol used for the request (e.g. 1.1 or 2).
• server.address: The address that the server is listening on.
• server.port: The port that the server is listening on.
• http.response.status_code: The status code of the response (if the request has been handled without a fatal error).
• error.type: The type of error that occurred (if the request handling was subject to an error).

http.server.active_requests

A gauge of the number of active requests being handled by Ende.serve at any given time. This is the number of requests that have been received but not yet responded to (where the response headers have not yet been sent). This metric is recorded with the following attributes:

• http.request.method: The HTTP method of the request.
• url.scheme: The scheme of the request URL.
• server.address: The address that the server is listening on.
• server.port: The port that the server is listening on.

http.server.request.body.size

A histogram of the size of the request body of incoming HTTP requests served with Ende.serve. The unit of this metric is bytes. The histogram buckets are [0, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000].

This metric is recorded with the following attributes:

• http.request.method: The HTTP method of the request.
• url.scheme: The scheme of the request URL.
• network.protocol.version: The version of the HTTP protocol used for the request (e.g. 1.1 or 2).
• server.address: The address that the server is listening on.
• server.port: The port that the server is listening on.
• http.response.status_code: The status code of the response (if the request has been handled without a fatal error).
• error.type: The type of error that occurred (if the request handling was subject to an error).

http.server.response.body.size

A histogram of the size of the response body of incoming HTTP requests served with Ende.serve. The unit of this metric is bytes. The histogram buckets are [0, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000].

This metric is recorded with the following attributes:

• http.request.method: The HTTP method of the request.
• url.scheme: The scheme of the request URL.
• network.protocol.version: The version of the HTTP protocol used for the request (e.g. 1.1 or 2).
• server.address: The address that the server is listening on.
• server.port: The port that the server is listening on.
• http.response.status_code: The status code of the response (if the request has been handled without a fatal error).
• error.type: The type of error that occurred (if the request handling was subject to an error).

Logs

The following logs are automatically collected and exported:

• Any logs created with console.* methods such as console.log and console.error.
• Any logs created by the Ende runtime, such as debug logs, Downloading logs, and similar.
• Any errors that cause the Ende runtime to exit (both from user code, and from the runtime itself).

Logs raised from JavaScript code will be exported with the relevant span context, if the log occurred inside of an active span.

console auto instrumentation can be configured using the OTEL_ENDE_CONSOLE environment variable:

• capture: Logs are emitted to stdout/stderr and are also exported with OpenTelemetry. (default)
• replace: Logs are only exported with OpenTelemetry, and not emitted to stdout/stderr.
• ignore: Logs are emitted only to stdout/stderr, and will not be exported with OpenTelemetry.

Permission audit

Ende can route the permission audit log into the OpenTelemetry exporter alongside the rest of your traces, metrics, and logs. Set ENDE_AUDIT_PERMISSIONS=otel (instead of a file path) and each permission access — allowed or denied — is emitted as an OpenTelemetry log record with these attributes:

• ende.permission.type — the permission name (read, net, env, …).
• ende.permission.value — the specific value being checked (a path, host, variable name, etc.).
• ende.permission.stack — the JavaScript stack frames at the access site, only present when ENDE_TRACE_PERMISSIONS is also set.

OTEL_ENDE=true ENDE_AUDIT_PERMISSIONS=otel ende run -A main.ts

This is useful when you already collect OpenTelemetry data: the audit lands in the same backend as your request traces, so you can correlate which request triggered which permission access. See permission audit for the full attribute set and the JSONL file-path mode.

User metrics

In addition to the automatically collected telemetry data, you can also create your own metrics and traces. Ende.telemetry provides a tracerProvider, a meterProvider and a contextManager that follow the OpenTelemetry JS API. They need no setup: all configuration of resources, exporter settings, etc. is done via environment variables.

For both traces and metrics, you need to define names for the tracer and meter respectively. If you are instrumenting a library, you should name the tracer or meter after the library (such as my-awesome-lib). If you are instrumenting an application, you should name the tracer or meter after the application (such as my-app). The version of the tracer or meter should be set to the version of the library or application.

Traces

To create a new span, first create a tracer:

const tracer = Ende.telemetry.tracerProvider.getTracer("my-app", "1.0.0");

Then, create a new span using the tracer.startActiveSpan method and pass a callback function to it. You have to manually end the span by calling the end method on the span object returned by startActiveSpan.

function myFunction() {
  return tracer.startActiveSpan("myFunction", (span) => {
    try {
      // do myFunction's work
    } catch (error) {
      span.recordException(error);
      span.setStatus({
        code: 2, // SpanStatusCode.ERROR
        message: (error as Error).message
      });
      throw error;
    } finally {
      span.end();
    }
  });
}

span.end() should be called in a finally block to ensure that the span is ended even if an error occurs. span.recordException and span.setStatus should also be called in a catch block, to record any errors that occur.

Inside of the callback function, the created span is the "active span". The "active span" will be used as the parent span for any spans created (manually, or automatically by the runtime) inside of the callback function (or any functions that are called from the callback function). Learn more about context propagation.

The startActiveSpan method returns the return value of the callback function.

Spans can have attributes added to them during their lifetime. Attributes are key value pairs that represent structured metadata about the span. Attributes can be added using the setAttribute and setAttributes methods on the span object.

span.setAttribute("key", "value");
span.setAttributes({ success: true, "bar.count": 42n, "foo.duration": 123.45 });

Values for attributes can be strings, numbers (floats), bigints (clamped to u64), booleans, or arrays of any of these types. If an attribute value is not one of these types, it will be ignored.

The name of a span can be updated using the updateName method on the span object.

span.updateName("new name");

The status of a span can be set using the setStatus method on the span object. The recordException method can be used to record an exception that occurred during the span's lifetime. recordException creates an event with the exception stack trace and name and attaches it to the span. recordException does not set the span status to ERROR, you must do that manually.

span.setStatus({
  code: 2, // SpanStatusCode.ERROR
  message: "An error occurred"
});
span.recordException(new Error("An error occurred"));

// or

span.setStatus({
  code: 1 // SpanStatusCode.OK
});

Spans can also have events and links added to them. Events are points in time that are associated with the span. Links are references to other spans.

// Add an event to the span
span.addEvent("button_clicked", {
  id: "submit-button",
  action: "submit"
});

// Add an event with a timestamp
span.addEvent("process_completed", { status: "success" }, Date.now());

Events can include optional attributes similar to spans. They are useful for marking significant moments within the span's lifetime without creating separate spans.

Spans can also be created manually with tracer.startSpan which returns a span object. This method does not set the created span as the active span, so it will not automatically be used as the parent span for any spans created later, or any console.log calls. A span can manually be set as the active span for a callback, by using the context propagation API.

Both tracer.startActiveSpan and tracer.startSpan can take an optional options bag containing any of the following properties:

• kind: The kind of the span. Can be SpanKind.CLIENT, SpanKind.SERVER, SpanKind.PRODUCER, SpanKind.CONSUMER, or SpanKind.INTERNAL. Defaults to SpanKind.INTERNAL.
• startTime A Date object representing the start time of the span, or a number representing the start time in milliseconds since the Unix epoch. If not provided, the current time will be used.
• attributes: An object containing attributes to add to the span.
• links: An array of links to add to the span.
• root: A boolean indicating whether the span should be a root span. If true, the span will not have a parent span (even if there is an active span).

After the options bag, both tracer.startActiveSpan and tracer.startSpan can also take a context object from the context propagation API.

Learn more about the full tracing API in the OpenTelemetry JS API docs.

Metrics

To create a metric, first create a meter:

const meter = Ende.telemetry.meterProvider.getMeter("my-app", "1.0.0");

Then, an instrument can be created from the meter, and used to record values:

const counter = meter.createCounter("my_counter", {
  description: "A simple counter",
  unit: "1"
});

counter.add(1);
counter.add(2);

Each recording can also have associated attributes:

counter.add(1, { color: "red" });
counter.add(2, { color: "blue" });

There are several types of instruments that can be created with a meter:

• Counter: A counter is a monotonically increasing value. Counters can only be positive. They can be used for values that are always increasing, such as the number of requests handled.

• UpDownCounter: An up-down counter is a value that can both increase and decrease. Up-down counters can be used for values that can increase and decrease, such as the number of active connections or requests in progress.

• Gauge: A gauge is a value that can be set to any value. They are used for values that do not "accumulate" over time, but rather have a specific value at any given time, such as the current temperature.

• Histogram: A histogram is a value that is recorded as a distribution of values. Histograms can be used for values that are not just a single number, but a distribution of numbers, such as the response time of a request in milliseconds. Histograms can be used to calculate percentiles, averages, and other statistics. They have a predefined set of boundaries that define the buckets that the values are placed into. By default, the boundaries are [0.0, 5.0, 10.0, 25.0, 50.0, 75.0, 100.0, 250.0, 500.0, 750.0, 1000.0, 2500.0, 5000.0, 7500.0, 10000.0].

There are also several types of observable instruments. These instruments do not have a synchronous recording method, but instead return a callback that can be called to record a value. The callback will be called when the OpenTelemetry SDK is ready to record a value, for example just before exporting.

const counter = meter.createObservableCounter("my_counter", {
  description: "A simple counter",
  unit: "1"
});
counter.addCallback((res) => {
  res.observe(1);
  // or
  res.observe(1, { color: "red" });
});

There are three types of observable instruments:

• ObservableCounter: An observable counter is a counter that can be observed asynchronously. It can be used for values that are always increasing, such as the number of requests handled.
• ObservableUpDownCounter: An observable up-down counter is a value that can both increase and decrease, and can be observed asynchronously. Up-down counters can be used for values that can increase and decrease, such as the number of active connections or requests in progress.
• ObservableGauge: An observable gauge is a value that can be set to any value, and can be observed asynchronously. They are used for values that do not "accumulate" over time, but rather have a specific value at any given time, such as the current temperature.

Learn more about the full metrics API in the OpenTelemetry JS API docs.

Practical Examples

For practical examples of implementing OpenTelemetry in Ende applications, see our tutorials:

• Basic OpenTelemetry Tutorial - A simple HTTP server with custom metrics and traces
• Distributed Tracing Tutorial - Advanced techniques for tracing across service boundaries

Context propagation

In OpenTelemetry, context propagation is the process of passing some context information (such as the current span) from one part of an application to another, without having to pass it explicitly as an argument to every function.

In Ende, context propagation is done using the rules of AsyncContext, the TC39 proposal for async context propagation. The AsyncContext API is not yet exposed to users in Ende, but it is used internally to propagate the active span and other context information across asynchronous boundaries.

A quick overview how AsyncContext propagation works:

• When a new asynchronous task is started (such as a promise, or a timer), the current context is saved.
• Then some other code can execute concurrently with the asynchronous task, in a different context.
• When the asynchronous task completes, the saved context is restored.

This means that async context propagation essentially behaves like a global variable that is scoped to the current asynchronous task, and is automatically copied to any new asynchronous tasks that are started from this current task.

Ende.telemetry.contextManager exposes this functionality to users. It works as follows:

const context = Ende.telemetry.contextManager;

// Get the currently active context
const currentContext = context.active();

// You can add create a new context with a value added to it
const newContext = currentContext.setValue("id", 1);

// The current context is not changed by calling setValue
console.log(currentContext.getValue("id")); // undefined

// You can run a function inside a new context
context.with(newContext, () => {
  // Any code in this block will run with the new context
  console.log(context.active().getValue("id")); // 1

  // The context is also available in any functions called from this block
  function myFunction() {
    return context.active().getValue("id");
  }
  console.log(myFunction()); // 1

  // And it is also available in any asynchronous callbacks scheduled from here
  setTimeout(() => {
    console.log(context.active().getValue("id")); // 1
  }, 10);
});

// Outside, the context is still the same
console.log(context.active().getValue("id")); // undefined

Learn more about the full context API in the OpenTelemetry JS API docs.

Configuration

The OpenTelemetry integration can be enabled by setting the OTEL_ENDE=true environment variable.

The endpoint and protocol for the OTLP exporter can be configured using the OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL environment variables. Supported values for OTEL_EXPORTER_OTLP_PROTOCOL are:

• http/protobuf (default): Export using Protobuf over HTTP to the configured endpoint.
• http/json: Export using JSON over HTTP to the configured endpoint.
• grpc: Export using gRPC (Protobuf over HTTP/2). Useful for collectors that only accept gRPC, such as Azure Container Apps' managed OpenTelemetry agent. The default OTLP gRPC port is 4317, so you typically also set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317.
• console: Print spans, logs, and metrics to stderr in a human-readable text format. This is useful for debugging and testing instrumentation without running an OTLP collector. When using console, no endpoint configuration is needed.

If the endpoint requires authentication, headers can be configured using the OTEL_EXPORTER_OTLP_HEADERS environment variable.

Endpoint can all be overridden individually for metrics, traces, and logs by using specific environment variables, such as:

• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
• OTEL_EXPORTER_OTLP_LOGS_ENDPOINT

For more information on headers that can be used to configure the OTLP exporter, see the OpenTelemetry website.

The resource that is associated with the telemetry data can be configured using the OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES environment variables. In addition to attributes set via the OTEL_RESOURCE_ATTRIBUTES environment variable, the following attributes are automatically set:

• service.name: If OTEL_SERVICE_NAME is not set, the value is set to <unknown_service>.
• process.runtime.name: ende
• process.runtime.version: The version of the Ende runtime.
• telemetry.sdk.name: ende-opentelemetry
• telemetry.sdk.language: ende-rust
• telemetry.sdk.version: The version of the Ende runtime, plus the version of the opentelemetry Rust crate being used by Ende, separated by a -.

Propagators can be configured using the OTEL_PROPAGATORS environment variable. The default value is tracecontext,baggage. Multiple propagators can be specified by separating them with commas. Currently supported propagators are:

• tracecontext: W3C Trace Context propagation format
• baggage: W3C Baggage propagation format

Trace sampling can be configured with the OTEL_TRACES_SAMPLER environment variable. The supported values are:

• always_on (the default): sample every trace.
• always_off: sample no traces.
• traceidratio: sample a fraction of traces based on the trace ID.
• parentbased_always_on, parentbased_always_off, parentbased_traceidratio: respect the parent span's sampling decision when there is one, and fall back to the matching root sampler otherwise.

For the ratio-based samplers, OTEL_TRACES_SAMPLER_ARG sets the sampling probability as a number between 0 and 1. It defaults to 1.0:

OTEL_ENDE=true OTEL_TRACES_SAMPLER=traceidratio OTEL_TRACES_SAMPLER_ARG=0.1 ende run -A main.ts

Metric collection frequency can be configured using the OTEL_METRIC_EXPORT_INTERVAL environment variable. The default value is 60000 milliseconds (60 seconds).

Span exporter batching can be configured using the batch span processor environment variables described in the OpenTelemetry specification.

Log exporter batching can be configured using the batch log record processor environment variables described in the OpenTelemetry specification.

Propagators

Ende supports context propagators which enable automatic propagation of trace context across process boundaries for distributed tracing, allowing you to track requests as they flow through different services.

Propagators are responsible for encoding and decoding context information (like trace and span IDs) into and from carrier formats (like HTTP headers). This enables the trace context to be maintained across service boundaries.

By default, Ende supports the following propagators:

• tracecontext: The W3C Trace Context propagation format, which is the standard way to propagate trace context via HTTP headers.
• baggage: The W3C Baggage propagation format, which allows passing key-value pairs across service boundaries.

Limitations

While the OpenTelemetry integration for Ende is in development, there are some limitations to be aware of:

• Traces only support links with no attributes.
• Metric exemplars are not supported.
• Custom log streams (e.g. logs other than console.log and console.error) are not supported.
• The supported exporters are OTLP (http/protobuf, http/json, grpc) and console. Other exporter formats are not supported.
• Metrics from observable (asynchronous) meters are not collected on process exit/crash, so the last value of metrics may not be exported. Synchronous metrics are exported on process exit/crash.
• The limits specified in the OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT, OTEL_SPAN_LINK_COUNT_LIMIT, OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT, and OTEL_LINK_ATTRIBUTE_COUNT_LIMIT environment variable are not respected for trace spans. The per-span attribute limit (OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT, falling back to OTEL_ATTRIBUTE_COUNT_LIMIT) and the per-span event limit (OTEL_SPAN_EVENT_COUNT_LIMIT) are respected, each defaulting to 128.
• The OTEL_METRIC_EXPORT_TIMEOUT environment variable is not respected.
• HTTP methods are that are not known are not normalized to _OTHER in the http.request.method span attribute as per the OpenTelemetry semantic conventions.
• The HTTP server span for Ende.serve does not have an OpenTelemetry status set, and if the handler throws (ie onError is invoked), the span will not have an error status set and the error will not be attached to the span via event.
• There is no mechanism to add a http.route attribute to the HTTP client span for fetch, or to update the span name to include the route.