No results for 'undefined'

Getting Started with OpenTelemetry in Java

Get up and running with OpenTelemetry in just a few quick steps! The setup process consists of two phases--getting OpenTelemetry installed and configured, and then validating that configuration to ensure that data is being sent as expected.

This guide explains how to install, configure, and validate your OpenTelemetry setup. The OpenTelemetry Java Agent automatically captures telemetry from your application's libraries, frameworks, and environment.

Requirements

  • Java version 8 or newer.
  • A service to add OpenTelemetry to. You can use this example application or bring your own.
  • A Lightstep account, or another OpenTelemetry backend.

Need an account? Create a free Lightstep account here.

Install and run OpenTelemetry

To make OpenTelemetry easier to install, we recommend the handy OTel-Launchers from Lightstep (us!).

Once you’ve downloaded the launcher, you can integrate it into your application either via Java code, or by running the launcher as a javaagent from the command line.

The full list of configuration options can be found in the README.

LS Note: When connecting to Lightstep, a project Access Token is required.

Java agent approach:

export LS_ACCESS_TOKEN=your-token
export LS_SERVICE_NAME=service-name
java -javaagent:path/to/lightstep-opentelemetry-javaagent-0.8.1.jar \
     -jar myapp.jar

If automatic instrumentation is all you want, then adding the agent is all you need to do. However, at some point you will most likely want to add instrumetnation to your own application code. To do this, you need to add two dependencies to your POM file.

pom.xml dependency:

<dependency>
   <groupId>io.opentelemetry</groupId>
   <artifactId>opentelemetry-extension-auto-annotations</artifactId>
   <version>0.8.0</version>
 </dependency>
 <dependency>
   <groupId>io.opentelemetry</groupId>
   <artifactId>opentelemetry-api</artifactId>
   <version>0.8.0</version>
 </dependency>

Validate installation by checking for traces

With your application running, you can now verify that you’ve installed OpenTelemetry correctly by confirming that telemetry data is being reported to your observability backend.

To do this, you need to make sure that your application is actually generating data. Applications will generally not produce traces unless they are being interacted with, and opentelemetry will often buffer data before sending it. So it may take some amount of time and interaction before your application data begins to appear in your backend..

Validate your traces in Lightstep:

  1. Trigger an action in your app that generates a web request.
  2. In Lightstep, click on the Explorer in the sidebar.
  3. Refresh your query until you see traces.
  4. View the traces and verify that important aspects of your application are captured by the trace data.

Library and framework support

OpenTelemetry automatically provides instrumentation for a large number of libraries and frameworks, right out of the box.

The full list of supported plugins can be found in the README.

OpenTracing support

The OpenTracing shim allows existing OpenTracing instrumentation to report to the OpenTelemetry SDK. OpenTracing support is not enabled by default. Instructions for enabling the shim can be found in the README.

Read more about upgrading to OpenTelemetry in our OpenTracing Migration Guide.

Troubleshooting your installation

Check that the agent was installed

If the Java agent is correctly installed, the following lines will be printed to the console:

[opentelemetry.auto.trace 2020-08-13 22:15:24:244 -0700] [main] INFO io.opentelemetry.auto.tooling.TracerInstaller - Installed span exporter: io.opentelemetry.exporters.otlp.OtlpGrpcSpanExporter
[opentelemetry.auto.trace 2020-08-13 22:15:24:250 -0700] [main] INFO io.opentelemetry.auto.tooling.PropagatorsInitializer - Added io.opentelemetry.extensions.trace.propagation.B3Propagator@7c0777b5 propagator
[opentelemetry.auto.trace 2020-08-13 22:15:24:252 -0700] [main] INFO io.opentelemetry.auto.tooling.VersionLogger - opentelemetry-javaagent - version: 0.7.0-SNAPSHOT~b659ee822

Running OpenTelemetry in debug mode

Set an environment variable to run the OpenTelemetry launcher in debug mode, where it logs details about the configuration and emitted spans:

export OTEL_LOG_LEVEL=debug

The output may be very verbose with some benign errors. Early in the console output, look for logs about the configuration and check that your access token is correct. Next, look for lines like the ones below, which are emitted when spans are emitted to Lightstep.

Span {
  attributes: {},
  links: [],
  events: [],
  status: { code: 0 },
  endTime: [ 1597810686, 885498645 ],
  _ended: true,
  _duration: [ 0, 43333 ],
  name: 'bar',
  spanContext: {
    traceId: 'eca3cc297720bd705e734f4941bca45a',
    spanId: '891016e5f8c134ad',
    traceFlags: 1,
    traceState: undefined
  },
  parentSpanId: 'cff3a2c6bfd4bbef',
  kind: 0,
  startTime: [ 1597810686, 885455312 ],
  resource: Resource { labels: [Object] },
  instrumentationLibrary: { name: 'example', version: '*' },
  _logger: ConsoleLogger {
    debug: [Function],
    info: [Function],
    warn: [Function],
    error: [Function]
  },
  _traceParams: {
    numberOfAttributesPerSpan: 32,
    numberOfLinksPerSpan: 32,
    numberOfEventsPerSpan: 128
  },
  _spanProcessor: MultiSpanProcessor { _spanProcessors: [Array] }
},

OpenTelemetry Resources

Telemetry data is indexed by service. In OpenTelemetry, services are described by resources, which are set when the OpenTelemetry SDK is initialized during program startup. We want our data to be normalized, so we can compare apples to apples. OpenTelemetry defines a schema for the keys and values which describe common service resources such as hostname, region, version, etc. These standards are called Semantic Conventions, and are defined in the OpenTelemetry Specification.

We recommend that, at minimum, the following resources be applied to every service:

AttributeDescriptionExampleRequired?
service.nameLogical name of the service.
MUST be the same for all instances of horizontally scaled services.
shoppingcartYes
service.versionThe version string of the service API or implementation as defined in Version Attributes.semver:2.0.0No
host.hostnameContains what the hostname command would return on the host machine.server1.mydomain.com,No

Java configuration

At this time, resources for Java can only be set from the command line via the OTEL_RESOURCE_ATTRIBUTES environment variable.

export OTEL_RESOURCE_ATTRIBUTES="service.name=my-app,service.version=v1.2.3"

The format is a comma-separated list of attributes, e.g. key1=val1,key2=val2. Attribute keys and values must contain only printable ASCII (codes between 32 and 126, inclusive) and less than 256 characters.

Semantic conventions

Standardizing the format of your data is a critical part of using OpenTelemetry. OpenTelemetry provides a schema for describing common resources, so that backends can easily parse and identify relevant information.

It is important to understand these conventions when writing instrumentation, in order to normalize your data and increase its utility. The semantic conventions for resources can be found in the specification.

The following types of resources are currently defined:

The Java constants for these conventions can be found here.

OpenTelemetry Spans

OpenTelemetry comes with many instrumentation plugins for libraries and frameworks. This should be enough detail to get started with tracing in production.

As great as that is, you will still want to add additional spans to your application code, in order to break down larger operations and gain more detailed insights into where your application is spending its time.

When you create a new span to measure a subcomponent, that span is added to the current trace as the child of the current span, and then becomes the current span itself.

Annotations

An easy way to wrap a method in a span is to use the @WithSpan annotation.

import io.opentelemetry.contrib.auto.annotations.WithSpan;
public class Chef {
  @WithSpan
  public void bakeCake() {}
}

Each time the annotated method is invoked, it will create a child span within the current trace, and record any thrown exceptions.

Disabling annotations

Individual annotations can be disabled at runtime via the exclude configuration or environment variables:

System propertyEnvironment variablePurpose
trace.classes.excludeTRACE_CLASSES_EXCLUDEExclude classes with the @WithSpan annotation
trace.methods.excludeTRACE_METHODS_EXCLUDEExclude methods with the @WithSpan annotation

Spans and the Tracer API

Accessing the tracer

In order to interact with traces, you must first acquire a handle to a Tracer. By convention, Tracers are named after the component they are instrumenting; usually a library, a package, or a class.

// Access to the tracer requires a name to associate the spans with
Tracer tracer =
    OpenTelemetry.getTracer("instrumentation-library-name");

Tracer tracer =
    OpenTelemetry.getTracer("instrumentation-library-name", "semver:1.0.0");

Note that there is no need to "set" a tracer by name before getting it. The getTracer method always returns a handle to the same tracing client. The name you provide is to help identify which component generated which spans, and to potentially disable tracing for individual components.

We recommend calling getTracer once per component during initialization and retaining a handle to the tracer, rather than calling getTracer repeatedly.

Accessing the current span

Ideally, when tracing application code, spans are created and managed in the application framework.

Assuming that your application framework is supported, a trace will automatically be created for each request, and your application code will already be wrapped in a span, which can be used for adding application specific attributes and events.

To access the currently active span, call getCurrentSpan:

Span span = tracer.getCurrentSpan()

Setting a new current span

Let’s demonstrate creating a new span by example. Imagine you have an automated kitchen, and you want to time how long the robot chef takes to bake a cake. The naive way to do this would be to just start a span, call your method, then end the span:

// Note that all new spans will set the current span as their parent by default.
Span cakeSpan = tracer.spanBuilder("bake-cake")
                      .startSpan();
chef.bakeCake()
cakeSpan.end()

The above example will work just fine, but with one big problem: the bakeCake method itself has no access to this new bake-cake span. That means there would be no way to add attributes and events to this span from within the bakeCake method. Even worse, getCurrentSpan would return the parent of “bake-cake,” since that span is still set as current.

What should we do instead? Replace the current span with bake-cake. To do this, call withSpan to make a closure around the bakeCake method. Within this closure, the getCurrentSpan method will now return bake-cake.

// in this scope, getCurrentSpan returns the parentSpan.
Span parentSpan = tracer.getCurrentSpan()

// Create a new span. Note that it is possible to set the parent manually.
Span cakeSpan = tracer.spanBuilder("bake-cake")
                      .setParent(parentSpan)
                      .startSpan();

// Replace the current active span by creating a scope.
try (Scope scope = tracer.withSpan(cakeSpan)) {

  // In this scope, cakeSpan is returned by getCurrentSpan
  Span span = tracer.getCurrentSpan()

  // bake your cake!
  chef.bakeCake()

} catch (Throwable t) {
    span.setStatus(Status.UNKNOWN.withDescription("Someone threw the cake!"));
} finally {
    span.end();
}

This pattern of wrapping method calls is important, because we always want application code to be able to assume that the current span is correct. Though you might now see why there are many cases where using the withSpan annotation is cleaner than calling withSpan manually.

Next steps

As you dive deeper into observing your application, you will naturally want to add additional, application-specific details to your spans. Check out how to decorate your spans with attributes and events.

When performing root cause analysis, span attributes are an important tool for pinpointing the source of performance issues. Much like how resources are used to describe your services, attributes are used to describe your spans.

Attributes in OpenTelemetry

Setting attributes

Here is an example of setting attributes to define an HTTP client request:

// This span represents the request.
Span span = tracer.spanBuilder("/project/:project-id/list") // operation named after the route
                 .setSpanKind(Span.Kind.CLIENT) // tag the span as a service boundary
                 .startSpan();

// The following attributes represent an HTTP client request
span.setAttribute(SemanticAttributes.HTTP_METHOD, "GET");
span.setAttribute(SemanticAttributes.HTTP_FLAVOR, "1.1");
span.setAttribute(SemanticAttributes.HTTP_URL, "https://example.com:8080/project/123/list/?page=2");
span.setAttribute(SemanticAttributes.NET_PEER_IP, "192.0.2.5");
span.setAttribute(SemanticAttributes.HTTP_STATUS_CODE, 200);
span.setAttribute(SemanticAttributes.HTTP_STATUS_TEXT,"OK");

// In addition to the standard attributes, custom attributes can be added as well.
span.setAttribute("list.page_number", 2);

// To avoid collisions, always namespace your attribute keys using dot notation.
span.setAttribute("project.id", 2);

// attributes can be added to a span at any time before the span is finished.
span.end()

Note that it is not possible to get attributes, only to set them.

Conventions

Spans represent specific operations in and between systems. Many operations represent well-known protocols like HTTP or database calls. Like with resources, OpenTelemetry defines a schema for the attributes which describe these common operations. These standards are called Semantic Conventions, and are defined in the OpenTelemetry Specification.

OpenTelemetry provides a schema for describing common attributes so that backends can easily parse and identify relevant information. It is important to understand these conventions when writing instrumentation, in order to normalize your data and increase its utility.

The following semantic conventions are defined for tracing:

  • General: General semantic attributes that may be used in describing different kinds of operations.
  • HTTP: Spans for HTTP client and server.
  • Database: Spans for SQL and NoSQL client calls.
  • RPC/RMI: Spans for remote procedure calls (e.g., gRPC).
  • Messaging: Spans for interaction with messaging systems (queues, publish/subscribe, etc.).
  • FaaS: Spans for Function as a Service (e.g., AWS Lambda).

Events in OpenTelemetry

The finest-grained tracing tool is the event system.

Span events are a form of structured logging. Each event has a name, a timestamp, and a set of attributes. When events are added to a span, they inherit the span's context. This additional context allows events to be searched, filtered, and grouped by trace ID and other span attributes.

Span context is one of the key differences between distributed tracing and traditional logging.

Adding events

Events are automatically timestamped when they are added to a span. Timestamps can also be set manually if the events are being added after the fact.

For example, enqueuing an item might be recorded as an event.

// Get the span
Span span = tracer.getCurrentSpan();

// Perform the action
queue.enqueue(myItem);

// Record the action
span.addEvent( “enqueued item“, {
  "item.id", myItem.ID()
	"queue.id": queue.ID(),
	"queue.length": queue.Length(),
});

Spans should be created for recording course-grained operations, and events should be created for recording fine-grained operations.

Recording exceptions

Many of the tracing conventions can apply to event attributes as well as span attributes. The most important event-specific convention is recording exceptions.

// recordException converts a Throwable into a span event.
span.recordException(new IllegalStateException());

// If the exception means the operation results in an 
// error state, update the span status.
span.setStatus(Status.INTERNAL);

Marking the span as an error is independent from recordings exceptions. To mark the entire span as an error, and have it count against error rates, set the SpanStatus to any value other than OK.

StatusCode definitions can be found in the OpenTelemetry specification. If no status code directly maps to the type of error you are recording, set the status code to UNKOWN for common errors, and INTERNAL for serious errors.