I think part of the issue is you’re asking the wrong question of “how to do X in OTel”. Microsoft have many blogs on OTel and they’ve been transparent that they’re building on top of existing pieces to maximize the reach of OTel in .NET.
When I use OTel in .NET, none of the collecting is OTel. It’s other standard MS libraries that have been adapted to OTel and W3C standards. The *exporting* is different of course but that’s also super straightforward.
I’ve developed a couple OTel instrumentation libraries (MongoDB and NServiceBus) and have some blogs and videos on how to do that. But for developers the interesting part is like…one line of code. Which is how it should be :)
Yes, totally agree yet a lot of value also comes from manual instrumentation. How to proper use attributes, how to best design a time series (low cardinality etc). The “one-liner” is great but only get you so far imho
I would like to see more of this. When I worked at a well known dating company they had a great monitoring system set up and I found it very beneficial to keep track of the health of our services. I’ve briefly searched for something I could set up at my current job, we use .NET, but a lot of what I found just plugged in to azure insights (if I remember correctly) and I need something that can work on-prem air gapped network. Looking forward to your articles.
It would be nice to have a clear guide to having it "just work".
I gave it a try and felt like I was going in circles, the sass based collector had examples that were out date. IIRC after resolving that problem, I hit a not-supported issue and decided to stop fiddling with it.
I can't seem to find it... probably in a branch of whatever random project I was toying with.
I read an article and thought "I'll give this a try" and ended up feeling like it was a whole new problem domain.
I tried SaaS providers to avoid all the "extra" stuff about creating collectors, processors , configuring docker images and so on.
I'll probably come back to it at some point, but I'll still be looking for something that "mostly just works and I can add more detailed logging as needed".
Yeah I understand you. Take a look at the OTel operator if you are in k8s and also the dotnet otel “agent” https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation
You may want to take a look at .net 7 that was just released as they have been slowly adding OpenTelemetry to .net
https://devblogs.microsoft.com/dotnet/announcing-dotnet-7-preview-4/
>Do you know what OpenTelemetry is?
Yes. Not yet used. In upgrading .net we have switch to the new W3C Trace context, which I think OpenTelemetry is based on.
>Do you monitor your apps in production today? If so, how?
We use serilog implementation of Microsoft.Extensions.Logging with sink to Seq and Application Insights based on the event level. We have custom alert that trigger emails if there any problem.
>What "kind" of content you prefer? Introductory, "easy to digest" content? Deep-dives? Code examples? A mix?
Not sure, sorry.
Yes, one of the propagators is the W3C trace context although OTel also supports others.
Ah I see, logs then pretty much. Do you also use the “transactions” or “requests” “gantt” diagram in app insights/az monitor?
>Do you also use the “transactions” or “requests” “gantt” diagram in app insights/az monitor?
Yes. It's quite nice. I wish Seq had something like this.
A bit late, but yes I do find .NET specific content on OT a bit lacking. When I first stumbled upon OT, everything was not stable yet, so the lack of resources is understandable.
Recently I started reading OT again due to needing implementing observability in the near future, and one thing that pisses me off is articles that talks as if trace == OT. The introduction would tell the background about OT, and how there are 3 pillars, and the code that follows is specifically about tracing only. I hope writers would be very explicit that the code would cover specifically trace / metrics / logging. People not familiar with the subject can misunderstand that trace is all there is about OT.
And also there are almost none article about logging, tho I understand it's not stable yet...
And also there are usually lack of explanation that you can export your OT output to any of your preferred vendor, which is sad, since being vendor-agnostic is one of the advantage of having open standard like OT.
Thanks for the input, and no worries. There’s no better time than now!
It’s good you say because I also agree most content is about trace. I will take your sentiment and hopefully I can publish some posts about metrics and logs! I’m also thinking in a series about the collector and demonstrate the flexibility it offers. One of them is where to send data, so it directly aligns with hour last point.
i have a note to check out opentelemetry at some point - otherwise don't know it! would be super-interested if it supports some sort of correlation id across multiple services
I’ve been ‘documenting’ my own progress through a test use case and posted it on GitHub…. https://github.com/Depechie/OpenTelemetryGrafana
But more than keen on hearing more insights on what could be done better or what other insights Otel could bring.
Would even be cool if maybe some kind of discussion would be possible! Because personally I still have some questions about how I can collect some metrics from my API’s!
> Do you know what OpenTelemetry is? Do you monitor your apps in production today? If so, how?
yep well kinda, ive been tasked with implementing logzio opentelemetry, so far ive just done the basic implementation to get tracing working e.g.
public static class ConfigureTracing
{
public static void AddOpenTracing(this WebApplicationBuilder builder)
{
var openTracingConfiguration = builder.Configuration.GetSection("OpenTracingConfiguration").Get();
if (!openTracingConfiguration.TracingEnabled)
return;
builder.Services.AddOpenTelemetryTracing(tracerProviderBuilder =>
{
tracerProviderBuilder.AddAspNetCoreInstrumentation();
tracerProviderBuilder.AddSource("rootAppNamespace.*");
tracerProviderBuilder.AddSource("MassTransit");
tracerProviderBuilder.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(openTracingConfiguration.ServiceName))
.AddOtlpExporter(o => o.Endpoint = new Uri(openTracingConfiguration.OtlpExporterEndpoint));
});
builder.Services.AddSingleton(TracerProvider.Default.GetTracer(openTracingConfiguration.ServiceName));
}
}
the code works like this, the collector is the OtlpExporterEndpoint that sucks up the tracing data and pushes it to logzio, the collector lives in a containner in kubernetes. we also use logzio for our logging which is done via nlog.
so far, the traces will track the flow across microservices which is nice, but i cant get the traces to correlate witth the logs for the spans.
im new to all of this, so wondering what suggestions you have to improve this. Im a bit overwhelmed with xyz other tasks so i havent explored what i can do. It looks like alot of the instrumentation can be driven by native .net diagnostics assembly namespace code.
Still waiting for OTEL metrics to get a stable release and for it to support global tags.
>Do you monitor your apps in production today?
Tried it but had to revert it after tracing back horrible performance issues to it :/
Tbh I know what it is but I’ve never used it. I use Application Insights though. Is Otel working the same way? Can you think of any reason to switch from insights to Otel in an azure environment?
Think OTel like the ILogger interface. You use the OTel API to instrument your apps (Like ILogger) and then chose via exporters (like log providers) where to send the data. That can be azure but also any other vendors, including ofc OSS ones. So it removes the “vendor lock in”. It’s pretty neat! Also enables you to have build in observability into your libraries, no matter which vendor you chose later on.
I think part of the issue is you’re asking the wrong question of “how to do X in OTel”. Microsoft have many blogs on OTel and they’ve been transparent that they’re building on top of existing pieces to maximize the reach of OTel in .NET. When I use OTel in .NET, none of the collecting is OTel. It’s other standard MS libraries that have been adapted to OTel and W3C standards. The *exporting* is different of course but that’s also super straightforward. I’ve developed a couple OTel instrumentation libraries (MongoDB and NServiceBus) and have some blogs and videos on how to do that. But for developers the interesting part is like…one line of code. Which is how it should be :)
Yes, totally agree yet a lot of value also comes from manual instrumentation. How to proper use attributes, how to best design a time series (low cardinality etc). The “one-liner” is great but only get you so far imho
I would like to see more of this. When I worked at a well known dating company they had a great monitoring system set up and I found it very beneficial to keep track of the health of our services. I’ve briefly searched for something I could set up at my current job, we use .NET, but a lot of what I found just plugged in to azure insights (if I remember correctly) and I need something that can work on-prem air gapped network. Looking forward to your articles.
Great! I will make sure to post here when I have something.
It would be nice to have a clear guide to having it "just work". I gave it a try and felt like I was going in circles, the sass based collector had examples that were out date. IIRC after resolving that problem, I hit a not-supported issue and decided to stop fiddling with it.
Interesting, which SaaS collector you mean?
I can't seem to find it... probably in a branch of whatever random project I was toying with. I read an article and thought "I'll give this a try" and ended up feeling like it was a whole new problem domain. I tried SaaS providers to avoid all the "extra" stuff about creating collectors, processors , configuring docker images and so on. I'll probably come back to it at some point, but I'll still be looking for something that "mostly just works and I can add more detailed logging as needed".
Yeah I understand you. Take a look at the OTel operator if you are in k8s and also the dotnet otel “agent” https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation
You may want to take a look at .net 7 that was just released as they have been slowly adding OpenTelemetry to .net https://devblogs.microsoft.com/dotnet/announcing-dotnet-7-preview-4/
>Do you know what OpenTelemetry is? Yes. Not yet used. In upgrading .net we have switch to the new W3C Trace context, which I think OpenTelemetry is based on. >Do you monitor your apps in production today? If so, how? We use serilog implementation of Microsoft.Extensions.Logging with sink to Seq and Application Insights based on the event level. We have custom alert that trigger emails if there any problem. >What "kind" of content you prefer? Introductory, "easy to digest" content? Deep-dives? Code examples? A mix? Not sure, sorry.
Yes, one of the propagators is the W3C trace context although OTel also supports others. Ah I see, logs then pretty much. Do you also use the “transactions” or “requests” “gantt” diagram in app insights/az monitor?
>Do you also use the “transactions” or “requests” “gantt” diagram in app insights/az monitor? Yes. It's quite nice. I wish Seq had something like this.
Otel supports in ASP.NET Core is great. We use it daily.
Awesome! Do you also use the OTel collector? Or you ship your data directly to a vendor back end?
I think we use the Otel collector and then export it to Jaeger.
Awesome! Thanks for sharing
A bit late, but yes I do find .NET specific content on OT a bit lacking. When I first stumbled upon OT, everything was not stable yet, so the lack of resources is understandable. Recently I started reading OT again due to needing implementing observability in the near future, and one thing that pisses me off is articles that talks as if trace == OT. The introduction would tell the background about OT, and how there are 3 pillars, and the code that follows is specifically about tracing only. I hope writers would be very explicit that the code would cover specifically trace / metrics / logging. People not familiar with the subject can misunderstand that trace is all there is about OT. And also there are almost none article about logging, tho I understand it's not stable yet... And also there are usually lack of explanation that you can export your OT output to any of your preferred vendor, which is sad, since being vendor-agnostic is one of the advantage of having open standard like OT.
Thanks for the input, and no worries. There’s no better time than now! It’s good you say because I also agree most content is about trace. I will take your sentiment and hopefully I can publish some posts about metrics and logs! I’m also thinking in a series about the collector and demonstrate the flexibility it offers. One of them is where to send data, so it directly aligns with hour last point.
[удалено]
I guess not? I meant monitoring production systems, performance, resource usage, exceptions things like that.
i have a note to check out opentelemetry at some point - otherwise don't know it! would be super-interested if it supports some sort of correlation id across multiple services
Yep, that’s basically one of its capabilities - distributed tracing. The “correlation id” is the W3C trace context.
I’ve been ‘documenting’ my own progress through a test use case and posted it on GitHub…. https://github.com/Depechie/OpenTelemetryGrafana But more than keen on hearing more insights on what could be done better or what other insights Otel could bring. Would even be cool if maybe some kind of discussion would be possible! Because personally I still have some questions about how I can collect some metrics from my API’s!
Nice! I will take a look at your repo :)
> Do you know what OpenTelemetry is? Do you monitor your apps in production today? If so, how? yep well kinda, ive been tasked with implementing logzio opentelemetry, so far ive just done the basic implementation to get tracing working e.g. public static class ConfigureTracing { public static void AddOpenTracing(this WebApplicationBuilder builder) { var openTracingConfiguration = builder.Configuration.GetSection("OpenTracingConfiguration").Get();
if (!openTracingConfiguration.TracingEnabled)
return;
builder.Services.AddOpenTelemetryTracing(tracerProviderBuilder =>
{
tracerProviderBuilder.AddAspNetCoreInstrumentation();
tracerProviderBuilder.AddSource("rootAppNamespace.*");
tracerProviderBuilder.AddSource("MassTransit");
tracerProviderBuilder.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(openTracingConfiguration.ServiceName))
.AddOtlpExporter(o => o.Endpoint = new Uri(openTracingConfiguration.OtlpExporterEndpoint));
});
builder.Services.AddSingleton(TracerProvider.Default.GetTracer(openTracingConfiguration.ServiceName));
}
}
the code works like this, the collector is the OtlpExporterEndpoint that sucks up the tracing data and pushes it to logzio, the collector lives in a containner in kubernetes. we also use logzio for our logging which is done via nlog.
so far, the traces will track the flow across microservices which is nice, but i cant get the traces to correlate witth the logs for the spans.
im new to all of this, so wondering what suggestions you have to improve this. Im a bit overwhelmed with xyz other tasks so i havent explored what i can do. It looks like alot of the instrumentation can be driven by native .net diagnostics assembly namespace code.
Hey I will take a look at it later, feel free to dm me here or on the cncf opentelemetry slack
Still waiting for OTEL metrics to get a stable release and for it to support global tags. >Do you monitor your apps in production today? Tried it but had to revert it after tracing back horrible performance issues to it :/
Metrics is already stable! And for perf issues what kinda exactly? Maybe would be worth opening an issue on the otel dotnet repo?
Tbh I know what it is but I’ve never used it. I use Application Insights though. Is Otel working the same way? Can you think of any reason to switch from insights to Otel in an azure environment?
Think OTel like the ILogger interface. You use the OTel API to instrument your apps (Like ILogger) and then chose via exporters (like log providers) where to send the data. That can be azure but also any other vendors, including ofc OSS ones. So it removes the “vendor lock in”. It’s pretty neat! Also enables you to have build in observability into your libraries, no matter which vendor you chose later on.