Understanding io.Insights: Open Observability for io.Connect

Hello, everyone. Thank you for joining. My name is Martin, and, I'm a product manager at Interop. Io. I'm also the product manager overseeing our IO insights product, and I'm very happy to welcome you all to our first IO Insights webinar series. I'll spend a bit more time here just to go over our next plant series as well. So today, we have our first session, which is about understanding IO Insights. We are going to go on a more high level around what is IO insights, how our customers are using it, how you can turn it on, what data does it capture, and how everything fits together. Then on November 4, we have a more technical session that is going to be hosted by our lead engineer, Velponikov. He's going to talk about investigating traces, how you can effectively troubleshoot with them, and he's going he's going also to go in a bit more details. Then on November 18, I will again host a session that is about our public and final dashboard and how you can use this template version of a dashboard to get started with your observability within thirty minutes. And the last session on December 9, again, Velco is going to lead this technical deep dive into our advanced functionalities related to IO insights, traces metrics, logs, advanced configuration. So if you have time, feel free to join them. With that, let's, kick it off. What is IO Insights? This is, the question I want to set straight from the start. And IO Insights is the observability layer built directly into your IO Connect platform. Also, it's the ideal source of observability data because it runs underneath the platform and your applications. That means that it already has visibility over every application start, every window focus, every interrupt call and crash, and can follow requests all the way to your back end services. So as mentioned, it's built into the platform. It's part of the IO Connect desktop or browser. There is no separate tool for you to deploy. It's already there. The data it it produces is the standard auto output. This means that any auto compatible back end can read the data, and we are not providing an iWinsight server that you need to walk in towards the data that we produce is going directly towards your collector. That means we are not storing the data. The data is yours, and it does not pass through any cloud of ours. Next, I want to cover some of the IO Insights use cases. Overall, there are many many observability use cases that IO Insights can help with and address. I'm grouping them into these four groups. First, we have platform health monitoring, which is one dashboard for all of your KPIs where you can monitor everything effectively. Then we have troubleshooting use cases where you need to troubleshoot different issues. You need to debug. As we move further, we have user behavior use cases. These are use cases related to your applications and how your users are using your applications and the platform itself. And at the end, we have these emerging use cases where it's simply data for your AI systems, and this is desktop data. We have many cases where AI can help a lot with observability if only it has the right amount of data. So now I will go over each of these in a bit more details so that you can better understand some more granular use cases in each category. First is the part of health monitoring, and this is a quote that is from a customer of ours where, basically, it goes like this. Our traders' platform takes minutes to start instead of seconds. And usually what happens is that the support team or platform team hears this from the traders, but can't really say how many people it affects or when does this started. However, with IO insights, they're able to see the startup time of every user on one single dashboard and very effectively understand when the slow startup started and from where does it originate. So what is the affected users or groups? Usually, when they find a fix, they can also send an set an alert. And the next time, they will know this before they receive a ticket or an escalation from the traders. There are also many other use cases related to this platform health monitoring dashboard, which can be proactive monitoring on new versions deployment. You deploy a new version. Does it work well or not? You can see which users are... Have already are already using it and have upgraded to the new version. You can have built in alert certification for your most important KPIs, and you also can have insights into the hardware inventory your users are are running on as well. So, really, a lot of observability use cases. The next group is troubleshooting use cases. An example is we get tickets for app crashes, long coding times, but only from a fraction of users. And what usually happens is that you're not able to reproduce the problem on your own machines. However, with IO Insights, you'll be able to use filtering. You'll be able to use this observability data to investigate, troubleshoot the issue, and understand that, okay. These crashes happens only on machines with eight gigabyte of memory. Then we see in a trace that there is a slow workspace holding for all of these machines, and one specific applications... Application inside is taking a lot of time. So now you know how to fix this, and there are, as mentioned, other use cases as well. For example, what is the specific time it takes for an interrupt call from one app to the next? What are the c... What are the errors that an application can see, and so on. Next, we move into user behavior use cases, and these are use cases where we want to know the new feature that we shipped in our core application if that is being used or not. And if the application team does not know this, that means there is a gap in the observability. And this is also a use case that can be addressed with IO Insights because you can track how many users have seen this feature or have interacted with it. And in this typical scenario, as an example over here, you may not... You may see that everything... All of the traffic that end users have interacted with the feature are users from our London office. But from New York, there is almost no one except some testing occurrences, and this is a problem that you need to resolve. So a lot of user behavior use cases are also addressable using the data from my own insights. And last but not least, we have data for AI. As I mentioned, here here are some emerging use cases that we are hearing about. It's about finding repeated steps between applications through AI that are worth automating to spot unusual unusual patterns in the startup, in the crashes or errors. Or, for example, the platform can suggest the next application or action to new joiners automatically and so on. As you know, AI is very good at navigating a vast amount of data, finding patterns, so there are a lot of use cases that you can implement with it. Now what is the data that that actually makes all of this possible, and what is the data that Time Insights captures that addresses these use cases. So first, we have metrics. These are numbers measured over time. And, for example, these are platform and app startup duration, app crashes, platform errors count, CPU memory per application, or per machine, and Workspace's loading time breakdown as well. Then we have traces as a data signal, and this is basically a time tracker to one operation and the steps inside it. It's very useful for troubleshooting. As well, it captures a lot of metadata that you can also use. And last, we have, logs that are showing the exact root cause of a failure or other software related activities that are locked by your applications and platform. So all of this together produced by IO Insights is giving you the full observability that you need in order to implement and have these use cases that I mentioned in the slides before. Next, I want to go over the high level architecture of file insights just to make sure that you understand what we are providing as a product and what you need to have from your side in order to get started. So first, we have our IO Connect platforms that you are most likely using or you can be using. These are IO Connect desktop and IO Connect browser, and IO Insights is the implementation of a product that is built in inside the platforms itself. So it's already there. You just need to start using it. And once you activate it, we are going to track metrics, traces, logs out of the box directly to an hotel collector of your choice. So you just need to give us the URL, and we are going to send this data to your hotel collector. So the hotel collector is something that you need to provide and you need to set up. You most likely already have something in your infrastructure. And after that, you have your own back end that it can be any back end that is hotel compatible. Whether that would be Grafana, Datadog, Splunk, it's your choice. And most likely, you already have something running. You just need to set it up and provide us the place and direction where we need to send you this data. The usual question that we get asked is how we can get started with ION sites and turn it on. So turning it on is a very simple thing. You just need a simple config change to your system JSON. So it's this auto config change that you need to add, and you can decide what you want to get started with, whether that would be walks, traces, or metrics. You just need to set up your own endpoint where we are going to start this data once the platform is started and initialized. Next next, I want to show you, one potentially complete platform metering dashboard. This dashboard that I'm going to show you is in Grafana, and just to... For you to get a sense of what you can have at your site if you don't already have it and what you can track using the data that IOS has provide, This dashboard is also available inside, our community forum. So go there if you want to find it, and we have a specific article for it. So now let me share my other screen and present you this platform. Okay? Hopefully, you can see it. This is a demo observability dashboard that, we have built using our own data, and it's also something available for you to use as a template. I won't go into too many details right now because we will have a dedicated session for this and how you can use it. But, overall, we have different tiles at the top that can keep... That's where your KPIs... The most important KPIs can be located, and you have one look into their health and performance. Obviously, green is good, red is bad. And this actually depends on the thresholds that you have set over here, but tracking coverage up start time, average system CPU usage, memory, and so on can be a quick look into how the platform is running. Then we have different metrics around the system, average system CPU, average system memory, platform metrics related to the platform itself, a breakdown of the users per per platform version. So if you have users in many platform versions and you're waiting for them to upgrade, you'll be able to know which users these are. Then we have app metrics. So this is data related to the applications living inside your your platforms, which are the most used applications, which are... Which applications utilize most memory and CPU as well, where users spend most of their their time, in which application, and so on. Also, information about the workspaces, layouts that you are running, And, of course, we also have the logs and traces built in... Into this same place. So the the great thing is that you can have everything from a platform showing different metrics to your walks and traces inside one place, and you can utilize all three of these signals to troubleshoot or to implement different observability use cases. Okay. Next time, I will go over in more details about the dashboard. So with that, I will stop my screen sharing and open the forum for any questions. Okay. Let me check what do we have. Okay. A bunch of questions we have. I will start one by one. Do we need to change our application's code or create them in a certain way to use Sino Insights? This is a very good question. Velco, our lead engineer, is also here, so I will call him to answer this one. Velco, are you here? Hey, everyone. This is Velco, the lead engineer of Fire Insights. So on the question of whether you need to change your application code, you don't. You get a lot of telemetry out of the box. So as soon as you enable iInsights, you will start getting information to the to the extent that iInsights has agreed to your applications. So, for example, things like calling Interop APIs, making web requests, things like the clickstream and the user journey and platform impressions, such as startup and loading workspaces and so on. So all of this is available out of the box. However, there is always, additional context that applications can provide. So things like business logic and the specific timing of events that are of interest to the application and so on, which is something that you can instrument yourself. And it can enrich the data that you can... That you get in your dashboards and in your traces. So the floor is pretty high. So as soon as you start... Sorry. I'm getting a bit dark here. So as soon as you turn on insights, you start getting a lot of information, but the ceiling depends on you. Eventually, you want to add some more instrumentation yourself. Yeah. Thanks, Virgo. I think you answered this well. I will go with the next question because we have a few more. The next one is how do we control how much data is sent? We don't want to foot our systems. This is also a good one, I think. So there are many approaches to that. We give you a powerful configuration that you can basically tweak the data at several levels. The standard pro... The standard recommendation is to start with the development environment and, run your applications against a couple of developer users and see how how much data they generate with the different configurations and then trim it down to what you're interested in. So you can only publish at a certain interval, or you can filter out prices to only operations that you're interested in and so on. So you start off with basically seeing what the full set is and how much it costs you and how much your infrastructure can handle, and then you can trim it down to basically what provides the most value for you. Okay. Yeah. This is great feature to have because storage sometimes is expensive and too much data without being used, I guess, may not be what most teams we want to have. Cool. So next question is also an interesting one. Do we need a new server or specific tools to see the data? A new server or specific tools to see the data? So this depends on what your infrastructure is at the moment. If you already have some kind of observability solution, then almost certainly, iInsights can plug into that, thanks to OpenTelemetry and the OpenTelemetry collector, which has its for almost everything that you can think of. So things like Splunk, Datadoc, and Grafana, and so on. If you're using any of those, they'll work with us. If you don't have anything, well, that's, that's going to be a process. So you're going to need to evaluate and think what your prediction setup is going to look like eventually. To get started, the baseline is, for example, if you want to evaluate the data from sites so to have a development environment while you're working on setting up your infrastructure, you can always just run the example Docker setup that we provide for you. So you can basically run a Docker file and point our insights to that for the sake of evaluation. But, eventually, if you if you want to use observability in production, whether it's insights or anything else, you're going to need to have an observability stack. And, yeah, that's going to be part of your infrastructure. Yeah. Makes sense. Next question is from Levin. One of the nice to have details as part of workspace and layouts details captured will be application and workspace that are part of them. So, basically, which applications are part of a specific workspace or layout, I guess. I feel like this is already captured as a data. Maybe we just have not visualized it. Leiko, what do you think? When you're when you're talking about metrics, which is what is presented in the dashboards, workspace and layout information is mostly concerned with startup. So you will see you will see how long can a workspace takes to route and how much each step in the process took. So if you have some anomalies, you will be able to know which direction to go. And the applications which are part of this workspace or this layout will also publish their own their own metrics so you can see how they're behaving and how long they take to start up. But the connection between the workspace and the application in terms of metrics is not represented simply because metrics do not represent this kind of relationship. In the next session, we'll be looking at using traces and walks to analyze exactly this kind of complex relationship. So if you have a workspace with many applications and you're having trouble with it, you will start by looking at the metric to detect that something is wrong, and then you can drill into the trace for the operation, and then you will see all of the applications and what they were doing at that time. So that kind of interaction between our workspace and its components is captured in traces. So, yes, we do have it. It's not part of the dashboard simply because the dashboards don't don't represent this kind of relationship, and it's it's going to be covered in the crisis session. Makes sense. So we can think also how we can visualize this in case what Navin is saying that he wants to have a view of the exact applications part for Workspace even if they change regularly. I mean, it it will get visualized in your Backend. So Grafana will show you the case with the sections for which application, how long each one took, and whether there were errors in which application that happened. It simply is the kind of data that is represented as a trace, not symmetric. Yeah. Okay. Thank you, Elko. Another question we have is about sensitive data. Can we keep things like client details out of the telemetry that we are sending to the collector and back end. So when it comes to what you get out of the box in terms of metrics, stress, and works, what we publish by default is is conservative. So metrics will not contain user data. The cases will not contain user data unless you go ahead and configure them to publish the book level of information, which will include application related data that is... That may contain arbitrary information. So by default, we've been conservative. You can increase the data that you publish, but that is something that needs to be done carefully. And this is the situation, basically, with every kind of analytical data. So even when you're writing work files or returning error messages from a web service, you always need to be mindful of what you're putting in it... Inside it. So we do, we are thinking of providing additional filtering in the future, which will be a kind of a defense in-depth where we will actively look for additional information that you may add to the telemetry that might be sensitive and strip it. But if... Essentially, we... It's a best effort. No tool can always, strip everything, so it's a kind of a defense in that situation. We've done our best, but it's also... There's also always a a piece of responsibility in terms of duplication in the infrastructure teams to make sure that what they publish does not include idiomatic information. So if I start to do a lot of custom things and tracking, it's possible that some sensitive data will be sent, but not by default. And I also need to be careful of what my customizations are. Yeah. Just tweak with every kind of development. You will need to have more second slot to vet your applications and your systems to make sure that they do not leak anything. We'll do our best to make sure we help, but nobody can cover all cases. Yeah. That's expected, I think. Okay. Last question. It's a pretty interesting one. How much control over telemetry do we have? Let's say we need to add the username to the logs, traces, etcetera, but it's only available after the login screen. Can we do something like telemetry dot set attribute username John so that in future, all desktop and other stuff have it? Yes. We have... We expose the ability to enhance telemetry in many ways. So whether it's by configuration or current time when you have additional metadata that you want to use to correlate stuff, you can inject it. And so, yes, you do have multiple ways to do that. And in particular, this, this scenario with login and SSO related information, we spot that out of the box. Thank you, Virgo. Follow-up questions to that one or another questions if you have? No. So these were the questions. Pretty good ones. Thank you everyone for participating and asking about these things. So with that, I think we can conclude this session, and we're going to expect you on the next one. With Velco, it's going to be more technical and, I'm sure, probably more interesting for you. So please join us there as well. Thank you, and have a good great day.