Discover How io.Insights Brings Observability to the Desktop

Hi, and welcome to interop.io's October 2025 five client forum. On this session, we're going to do an observability kickstarter guide. My name is Yordan Yordanov, and I'm the senior product manager responsible for IO Insights. Together with me as a keynote presenter, I have Velk0 Nikov, who is the implementation lead for IO Insights. I'm going to talk about why observability matters and give you the tip of the iceberg for IO Insights. Then Velco is going to talk about what to measure, how to set up your environment, and give you some practical examples. And finally, we are going to wrap up by talking about the current state and the next steps. So why does observability matter? Well, it matters because it turns system data into product insights. It tells us what's happening with our applications and platform deployments. It enables us to do proactive detection. In other terms, it allows us to gather insights and allows us to highlight things that are about to go wrong even before they go wrong and before the end users start complaining about them. It also allows for shared understanding between the different roles within the organization. We are talking about product managers, platform administrators, business stakeholders and technical folks as well. Observability also allows us to measure the impact of changes that have been deployed, whether good or bad. And finally, it allows engineers to do root cause analysis because the observability data is very rich with details. Okay. Let's talk about how we at Interop. Help you solve your observability challenge. We do that with IO Insights. It's our product, which is a configurable built in feature that publishes signals for key events based on the OpenTelemetry standard. We are following the three main concepts of the OpenTelemetry standard: metrics, traces and logs. Starting off with metrics. They help you monitor system health or in other terms, they highlight what is good or bad with your deployments. We also have traces that provide deeper diagnostics for debugging or in other words, they show where the thing that happened, happened. And finally, we have logs that capture granular events and explain what exactly happened at the lowest level of detail possible. This diagram is not a deployment manual, but is showing you how IO Insights works at the bird's eye level. Starting off with the left hand side, we have the end user's desktop that might be running IO Connect desktop or IO Connect browser. Within the platform, we have multiple applications and platform data as well, which you send to the IO Insights OpenTelemetry publisher that is running within the platforms. The publisher then sends the data to the Autel back end collector that you have deployed in your back infrastructure. And from there, the data is stored in your database of choice. Finally, the data gets queried to be visualized on the dashboards of your data visualization tool of choice. On this slide, I have prepared several examples of how you can visualize your data showing you the art of the possible. There are just a set of examples that we have built when experimenting with the early stages of IO Insights metrics implementations. As you can see, there are a lot of options. It all comes to the imagination of the team that is building your dashboards. If you don't have a team to help you build your dashboards or you're building them yourselves but they're struggling with doing so, we have a superb consultancy team that can help you out with building your dashboards. So please reach out to your sales rep to talk about how we can help you in this process. With that being said, I'm going to hand over to Velko for the deeper dive. Hi, everyone. I'm Velko Nikolov, the development lead of io.Insights. I'm here to give you some more detail into what you get with io.Insights and how to make use of it. The original title for this session was observability, why, what, and how, which is a little presumptuous on our part, seeing as we can't really come out here and tell you exactly what to measure, given that we don't know the needs of your organization. However, I hope to be able to give you some inspiration based on what capabilities HireInsights provides and to help you get started on the road to getting the benefits of observability in your company. To recap what was said earlier, IO Insights is an extension to the IO Connect products that publishes observability for the operations that IO Connect performs. Since IO Connect knows pretty much everything about what's going on with your applications, it's a natural place to listen for observability events, and the end result is that you need to do very little to start getting some useful information out of it. IO Connect knows what applications your users run and in what order. It knows what APIs are used, what back ends are being called, what errors they're from. It knows how long your web pages take to load, what elements your users interact with, how much time they spend on various screens. This means you shouldn't need to do anything in particular to start getting useful information right away. Just like KioConnect is the plug and play framework where your applications need to do the minimum necessary lifting to work together, similarly, IO Insights is a framework that allows your applications to do the minimum required in order for you to get useful telemetry right off the bat. This is what is referred to as the stock instrumentations: observability information that you start getting immediately simply by virtue of your applications being run inside the IO Connect platform. Of course, we give you the ability to publish custom observability, which gets integrated seamlessly with what's coming out by default, reuses the same configuration in all that's required is for you to code the IO Insights APIs. We talked about needing to do the minimum amount of work to get started. How small is this minimum exactly? The requirements are split into three levels, infrastructural, application, and operational. On the infrastructural level, the first thing you need is what is known as the OpenTelemetry stack, which refers to a set of backend observability components. First among these is an instance of the open source Auto Collector, which is an important intermediate backend that processes and transfers observability data from your applications to your database. Many of the possible components that can go into an auto stack are available as Docker containers, and we at Interop. Can provide you with a complete set of components to Docker setup. This means you can spin up a complete observability stack locally, so you can play with and test IO Insights without needing to deploy anything in your enterprise. The next step is to get an IO Insights enabled build to io.Connect desktop or IO Connect browser. At the time of this recording, IO Connect desktop with io.Insights is in beta, so drop us a line to receive an installer and instructions for testing. Finally, you need to configure and turn your build. io.Insights has very powerful configuration capabilities, but for testing purposes, it can be as simple as pointing it to your auto collector and enabling the kinds of telemetry you're interested in. We provide detailed documentation to get you started, and we're always happy to help you tweak your setup to help you achieve your goals. On the application level, as stated before, you don't need to do much to start benefiting from the stock instrumentations. The only requirements are that your applications are running in an IO Insights enabled version of IO Connect, and that they are using an up to date version of the IO Connect client libraries. At any rate, no code changes will be necessary unless you are interested in adding your own custom instrumentations. The time of this recording, IO Insights client libraries are available for JavaScript applications, and dot net support is under development. The operational requirements for observability refer to the ways your organization wants to make use of the gathered information, and they're not something that we can define for you. However, once you start evaluating our insights, you can start marking down which aspects are the most useful for your requirements, and our consultancy team will be happy to help you define your own approach to observability. Before we look in more detail into the data provided by IO Insights, it's important to revisit the three kinds of telemetry data mentioned earlier, also known as the three pillars of observability, namely traces, metrics, and walks. Traces are most people's favorite form of telemetry when it comes to diagnosing issues or making sense of series of events. A brief explanation to those who are new to the concept: A trace is a collection of diagnostic entries called spans, similar to log entries except that each span has a name a beginning and an end, and also spans have a logical structure between them. That is, when an application emits a span, it becomes active until it's ended. And if the app emits other spans during this time, the new ones are marked as children of the first one, and any spans created where one of the new spans are active become children of the corresponding new span. In this manner, you get a hierarchical relationship between the spans as parents and children. And because each span logically corresponds to a step of an operation, a child span corresponds to a subsequent step that is part of the parent or was caused by it. So, you can trace the execution of whatever operation was going on by walking the tree of spans based on their parent child relationship. We will see some examples of examining the flow of a given trace to reason about user system behavior a little later on. Besides the hierarchical structure of traces, the other critical point about them is that they are distributed by design. Meaning that child spanners do not need to be created by the same system that created the parent span. Once a span is created in a trace, whenever the application goes into another system, it can pass the current span ID and the new system can create spans in the same trace and itself propagate to further upstream systems. The final shape of the trace will be constructed in the tracing backend based on the parent span IDs, and we will present the entire operation as control moves between the different actor systems. For instance, in IO Connect, this can mean that you can follow the operation that started by the user clicking on a button in one application, then continued with some logic internal to the application, then invoked an IO Connect API that was exposed by another application, which in turn called into IO Manager, which code into a backend service, the control returned to the first application, and so on, so you can identify the root causes of issues, look for bottlenecks, analyze behavior, and more. More on this later. Most of the propagation necessary to accomplish all this is automatically handled by our insights, and the rest comes down to using our API to retrieve and populate the span ID when creating a new span. Also, the spans in a given trace are structured entries and can contain any number of key value pairs called tags, also known as attributes, which enrich the information for later as well as a status flag that indicates error situations that were encountered while the span was active, which later helps you figure out, for example, which application or API was involved in an operation, how large a data entity was, and anything else you need. When it comes to logging, everyone is familiar with the basic concept of logs. They are the greatest common denominator when it comes to diagnostics and analyzing systems, with each component of a system publishing its own log as a series of messages with a timestamp and a severity level. You can describe everything in a sequence of logs, and with enough logging you can get all the information you need, provided you can deal with the verbosity. However, logs are not useful without specific knowledge of the system, and there is no explicit relationship between log entries other than the fact that they come one after the other chronologically. So they can tell you that a thing failed to happen, but not necessarily what led to this outcome, nowhere to look for more information, especially when the trial ends, and you need to consider what happened in another system. Not to mention that gathering logs in multiple locations is a pain. IO Insights and OpenTelemetry can actually help with both since, a, it defines remote logging, so you can publish your logs to a central location, and b, logs published can actually be correlated to traces, meaning that if a trace was active at the time that the walk message was submitted, the hotel walk entry will include the span ID, and the back end that supports it with Grafana will give you the option to jump between the span of the trace and relevant walk entries, regardless what system they were generated by. Metrics are an overloaded term, but in our insights and OpenTelemetry in particular, a metric is a value that can be measured as a number at a given time and aggregated based on some key value pairs so you can get statistics out of them. The sequence of values over time for a given set of value pairs is called a time series and can represent, for example, the number of instances of an application that were opened, the time a workspace took to load, the memory usage for users machine over time, and so on. Aggregating time series can give information about, for example, average memory usage for multiple users, which are the most used applications for a region or at a given time of day, the number of errors per application version, and so on. Conceptually, metrics are simple and efficient. You can use them to drive dashboards, set alerts, ask questions about performance, usage, event frequency, and trends. However, using them effectively poses two challenges: defining the set of metrics you need for the things you're interested in together with the attributes, and then figuring out the query and dashboard that you need in order to actually get the answers you need. Note that metrics are usually aggregated, so in most cases you won't be looking at values for an individual user or a session, but performing queries to get statistics about percentile values for some set of users, applications, and so on, in order to detect trends. You can work with individual time series, but that's mostly for diagnosing issues. For example, you can use the memory and CPU metrics from MyerInsights to detect whether a machine was under heavy load when a user was experiencing some issue. However, dashboards, alerts, and business analytics are usually driven by aggregate statistics based on queries against multiple time series, which normally means you get a bird's eye view and lose detail into the individual scenarios that contributed to this statistic. Yet, OpenTelemetry has a nifty helper here too. Just like with traces being correlated to walks, they can also be correlated to metrics. Metrics time series can link to what's called exemplars, which are basically the idea of the active trace whenever a value is observed, meaning that with some configuration of the auto collector and an observability interface such as Grafana, whenever you plot statistics based on a metric, you can look at a given value and click through to the active trace, the chosen operation, the publishing applications performing at the time. One last thing about the integration between the three pillars of observability is that since spans in a trace have durations, this information can also logically be represented as a metric, and Bore io.Insights and OpenTelemetry support the ability to expose traces as counter and duration metrics, so as soon as you enable traces, you can use them to get statistics on how often various actions are performed and how long they take. Let us look into some examples of specific observability data you get with io.Insights. User journey. One of the most requested and interesting features of io.Insights is the user journey trace that's published by IO Connect desktop, which tracks the user navigating between applications during their session. This is how it works in the default waterfall visualization in Grafana, with each application being represented by a span that contains the name of the application that was focused and the duration representing how long it remained active. The empty sections of time represent intervals where the user was working outside IO Connect, although eventually we will start tracking the name of window in that case as well, so you know how often your users switch from your apps to Excel, for example. Using our API, you can also enhance the user journey with additional information. For example, the currently active app can add extra attributes to its span to add information about what the user was doing in the app, whether there were any errors, or to add an additional span to indicate the place in the user journey that specific workforce started or ended, which would help later analysis. Right off the bat here, you get some interesting information based on which applications users switch between, how long they spend in each one, and how long they spend inside versus outside the IO Connect platform. Here is an odd graph visualization from Grafana based on a single user journey from a day of work. Each circle is an application with the number representing how many times it was focused, and the color representing how long the user spent in it on average, with red being longer. The arrows represent switches from one application to the other, and the heavier the arrow, the more times the user switched from one to the other. Right away, you can see that there are a couple of applications in the top right corner that are often used together. It's a nice cluster of applications that are likely to be tightly coupled. This information could be useful to figure out what users are doing, where they are spending their time, where their applications are being adopted, and which combinations of applications are good candidates for integration. Here is another visualization that is based on a part of the user journey that was marked through the IO Insights API as separate workflow, and it shows aggregated data from multiple users. Users have run through the workflow five seventy times in total, and the arrows and numbers show how many times out of these five seventy they switch to a particular application, and the colors again show how long the users spent in each one. We see that all of the workforce include, and most end with the client portfolio app, and it also takes a long part of the users time, so it's candidate for optimization. There is also a very slow application called AskHimogen, which is used in conjunction with client portfolio, but is only used in one quarter of the workforce, so it's not as important to optimize. Ask Imogen also seems quite coupled with client portfolio, so that's another good question to ask based on this info. Clickstream Another trace that describes user's actions platform is the clickstream trace, which creates a separate trace for each application instance and then tracks the user's interaction with the elements of the page, which by default is limited to clicks, hence the name click stream, although you can also track any other JavaScript events. This gives you the ability to look into how the users are actually using the individual application and is useful for debugging issues, as well as optimizing workforce and getting usage analytics. Like the user journey, your applications can annotate the clickstream trace to help later analysis. Adding semantic information about what the user is doing, marking errors, and so on. Out of the box, clickstream information is useful for analyzing the usage patterns inside a particular application, so you can tell which features are used often, what are the common sequences of actions users performed that indicate possible user experience improvements, and possible pain points. However, user journey and clickstream can be correlated and used in combination, which gives you the complete set of actions a user has performed in the application and platform overview. One especially interesting approach to making use of this information is to feed it to an AI model that can analyze even large datasets, such as the sessions of a set of users over multiple days, and give you insights into user behavior and performance. We've already had some successful internal POCs that prove that AI can answer questions about how workflows can be optimized, which users are facing difficulties, frequent sources of errors, and the like. This screen shows a quick example of some of the conclusions that were reached by the AI. Complex operations. An area of observability where traces really shine is the recording and analysis of complex operations, consisting of many individual subtasks where control passes from multiple systems, such as applications, platforms, and backend. Here we have the traces generated by IO Insights whenever IO Connect starts, describing the various steps of the startup process. It is immediately useful to analyze how long each operation takes, work through optimization opportunities, startup dependencies, and you can even compare instances of this trace to see what happens differently if you change configuration or between different users or versions of the platform. With the propagation of active SPAN information, you can also see how the platform calls into IO manager and back end services, start service applications, and other complex operations performed as part of the startup process. Here is an example that shows what starting a workspace looks like as a trace. Note the error that was raised by one of these startup applications, which is immediately visible in one of the child spans of the trace. As mentioned before, IO Connect knows a lot about what's happening. The only thing IO Connect doesn't know about is the operational logic of your enterprise and your applications, which is why IO Insights is extensible, allowing you not only to publish custom telemetry, but to have it integrated with telemetry that's captured automatically. Here is an example that's coming from the demo application that's shipped with the ION Insights Beta that shows how an application can create a custom span programmatically, which gets properly connected to the tracing that's published by default whenever platform features are being used, all the way to the backend system. Further, this example showcases how easy it is to walk through the trace and find the place that an error occurred. No collecting logs, no digging through the application code, no asking the user to clarify their issue reports. Another dataset that is available from Hire Insights are the platform metrics that are published by default, such as memory and CPU usage, startup time, running application count, and so on. They are defined based on some use cases and questions that we've come up with that will hopefully be useful to you, but you might have your own ideas for making use of this information. You can see the details about the provided metrics in the IO Insights online documentation, and we're always happy to receive your feedback and suggestions. When it comes to logging, IO Insights provides you with the ability to publish your application logs as well as the platform logs for OpenTelem and Pre with just a few simple configuration tweaks. Finally, IO Insights provides rich configuration capabilities, which allow you to define different environments and control the data, the volume, and the verbosity of what you publish. This means that you have great flexibility in defining your environment based on the stage of development that you're in. So you can start as simple as running a few Docker containers on your local machine and progress all the way to scalable Kubernetes managed deployments with rich configuration that integrates with io.Manager. Okay. Thanks for that, Velko. It was very insightful. We're going to wrap up by talking about what the current state and next steps are. So this table showcases the current state and what we have planned for the midterm for IO insights. Let's first walk you through what each of the columns means. On the left hand side, we have platform telemetry, which means the data that you get published out of the box as soon as you enable OpenTelemetry data publishing. This is data that we have pre instrumented the platform with, so you don't need to do anything specific or complicated. As part of that data, we have some data that is for your applications as well, but this is not application specific data. It's just data that the platform is getting because the applications are running within it. And then on the right hand side, we have application telemetry, which in other terms is the custom application specific data that you would need to instrument your applications to publish. An example for such a custom application data is user clicks. Alright. Let's start off with our Connect desktop. Well, our connect desktop, we currently have metrics generally available. Then we have traces and locks currently still in beta. We intend to have a generally available version of traces and locks by the end of twenty twenty five. Moving on to IO Connect browser, similar to IO Connect desktop, we have metrics in production, but traces are currently underway. We are working hard to wrap up work on them and we are going to release them with the four point three release, which is currently scheduled for Q1 twenty twenty six. And finally, logs are scheduled for Q1 next year as well. Okay. Moving on to the custom application level telemetry data. We have the JavaScript library currently in beta for metrics, traces and logs, aiming to deliver the generally available version by the end of the year. We have scheduled work for the dot net library to be delivered by the end of H1 twenty twenty six. And that's all from us for now. Thank you for watching. If you have any questions, feel free to reach out.