Makes total sense.

Idk, I still think the insights belong to the deployment and not to the template. Eg. If I want to see the user activity I would like to see it per deployment and not per template, if I want to see the number of workspaces in a failed state I would do it for deployment so I can see of this is specific to a template, etc. I see value on scope by template, I just think deployment metrics give the user a better view on what is happening.

@bpmct what popular parameters are?

Like what rich parameters are most used with a template. Specifically:

• What images are users using with a template?
• What shell do users use with a template?
• What repos do use with a template?

All are rich parameters at the moment but there's no way to see an aggregate / summary of what people use

Ahhhhh I see. But this would be by template right?

Yes. Per template :)

Since this is planned to be an improvement to be made after the first release we can put some thought on this later. cc.:@bpmct Maybe this should live inside of a "insights" page inside of the template page and not in the "deployment" insights.

Make them draggable adds complexity because you would have to be able to resize the areas as well.

I like to put the insights endpoint under the insights route.

Agreed. In v1 we wrote ad hoc sql queries for the few metrics we supported. In v2, we are seemingly doing the same. Metrics are gathered and queried on a case by case basis.

It would be ideal to leverage similar apis to prometheus or another standard time series db. Maybe we can even leverage some library that sits ontop of postgres.

Whatever we come up with should be easy to expand and adapt to new requirements (adding labels for filters, changing the query to say daily vs weekly etc). The sql queries in v1 were very hard to maintain as they were long and difficult to follow and test.

The interface looks good!

• I think we will never need to have data aggregate hourly.
• It is nice to support multiple templates for aggregation but for now, in the UI, we are only going to support one template per time. If having this, aggregating multiple templates, are easy... that is awesome for extensibility in the future.
• I'm ok with your plan to move by stages 👍 we can make it experimental in the UI until we have a good enough first version
• Another thing to consider is, when apps usage is done, it probably should be merged with the usage built-in.
• How to collect app usage sounds like a mystery. Do you have any ideas on how to make that?

I think we will never need to have data aggregate hourly.

👍🏻

It is nice to support multiple templates for aggregation but for now, in the UI, we are only going to support one template per time. If having this, aggregating multiple templates, are easy... that is awesome for extensibility in the future.

Ok, this is good to know. One question comes to mind: If this was a view of deployment, would we want to be able to show app/parameter usage there as well? If yes, I think it makes sense keeping as is, but if no, then we can simplify this data to be for one template only.

I'm ok with your plan to move by stages 👍 we can make it experimental in the UI until we have a good enough first version

Sounds good 👍🏻

Another thing to consider is, when apps usage is done, it probably should be merged with the usage built-in.

I think this is sensible. Maybe we should change this from the get-go? Like this:

"usage": [            {"type":"builtin","display_name":"Visual Studio Code",// Could be omitted/let frontend decide."slug":"vscode","icon":"/icon/vscode.svg",// Could be omitted/let frontend decide."seconds":54000,            },            {"type":"app","display_name":"code-server","slug":"code-server","icon":"/icon/code.svg","seconds":10800,            }        ],

This format is conducive to introducing new data in the UI without needing frontend changes, the backend can simply add to the array and it would show up.

How to collect app usage sounds like a mystery. Do you have any ideas on how to make that?

I was thinking we could track activity on the proxied app URL (e.g./@user/workspace/apps/app-name/) with a type of debounce, as long as there's a long-running request (e.g. websocket) or any another other request in, say, 5 minutes, usage is increased. I can't say for sure how well this will work in practice, though.

Thanks for flashing out the API sketch! I reviewed it, and here is a set of questions I have:

1. user_latency: what if we have 1000 active users? Will it blow up the JSON?
2. usage_apps: same question about apps
3. usage_parameters:
   3.1. Is there a risk of leaking something sensitive?
   3.2. Should we link them with templates/template revisions?
   3.3. Some parameters are lists of strings. In the future we may want to support full objects. I'm not sure if we need to expose all these.
4. we only need this data for day or perhaps hour - I would expose only the aggregations we need

Thanks for the feedback@mtojek!

1. A good observation. I doubt it will be a problem for a while, but if we want to limit the impact we can change this response to be top_10/bottom_10 latencies, alternatively we split it to it's own endpoint so we can request with sorting and limit. Any preference@BrunoQuaresma? Latter could support pagination which seems like it'd be ideal for this data type.
2. Apps won't scale like users, there's only so many apps/templates and unique app names, so I think we'll be fine here
3. • I suppose we should follow the same rules as the dashboard here, i.e. if I can't view a users workspace settings/params, I shouldn't be able to see them in the statistics (or they'll be in the stats but[REDACTED], simpler to omit for now though)
   • If we limit this endpoint to be single template insights, then no, but if we support aggregates, we can addtemplate_version_ids: [...] as a field to the parameter.
   • We can support formatting all types of parameter values to strings, a list could become"item1, item2"
4. Definitely 👍🏻

Here is an updated proposal, based on the feedback:

Thetemplate_ids fields have been added to more sections so that it can be inferred which template(s) contributed to the data. This is mostly a "for free" field that can be removed if we feel it adds too much noise.

GET /api/v2/insights/templates?start_time=01-07-2023&end_time=08-07-2023&interval=day&templates=name1,name2

{"report": {"start_time":"2023-07-01T00:00:00.000000Z","end_time":"2023-07-08T00:00:00.000000Z","template_ids": ["uuid1","uuid2"],"active_users":22,"usage_apps": [            {"template_ids": ["uuid1","uuid2"],"type":"builtin","display_name":"Visual Studio Code","slug":"vscode","icon":"/icon/vscode.svg","seconds":54000,            },            {"template_ids": ["uuid1","uuid2"],"type":"builtin","display_name":"JetBrains","slug":"jetbrains","icon":"/icon/jetbrains.svg","seconds":900,            },            {"template_ids": ["uuid1","uuid2"],"type":"builtin","display_name":"Web Terminal","slug":"web-terminal","icon":"/icon/terminal.svg","seconds":5400,            },            {"template_ids": ["uuid1","uuid2"],"type":"builtin","display_name":"SSH","slug":"ssh","icon":"/icon/ssh.svg","seconds":10800,            },            {"template_ids": ["uuid1","uuid2"],"type":"app","display_name":"code-server","slug":"code-server","icon":"/icon/code.svg","seconds":10800,            }        ],"usage_parameters": [            {"template_ids": ["uuid1","uuid2"],"display_name":"Coder Repository Directory","name":"coder_repository_directory","values": [                    {"value":"~/coder","icon":"","count":10                    },                    {"value":"~/coder.com","icon":"","count":2                    }                ]            },            {"template_ids": ["uuid2"],"display_name":"Dotfiles URL","name":"dotfiles_url","values": [                    {"value":"~/usr/.file","icon":"","count":10                    },                    {"value":null,"icon":"","count":2                    }                ]            },            {"template_ids": ["uuid1"],"display_name":"Region","name":"region","values": [                    {"value":"Pittsburgh","icon":"/icon/flag1.svg","count":8                    },                    {"value":"Helsinki","icon":"/icon/flag1.svg","count":2                    },                    {"value":"Sydney","icon":"/icon/flag3.svg","count":1                    },                    {"value":"Sao Paulo","icon":"/icon/flag4.svg","count":1                    }                ]            }        ]    },"interval_reports": [        {"start_time":"2023-07-01T00:00:00.000000Z","end_time":"2023-07-02T00:00:00.000000Z","template_ids": ["uuid1","uuid2"],"interval":"day","active_users":19        },        {"start_time":"2023-07-02T00:00:00.000000Z","end_time":"2023-07-03T00:00:00.000000Z",...        },        {...},        {...},        {...},        {...},        {...}    ]}

User latency is it's own endpoint that supports filtering on template, this allows us to easily support pagination as needed.

GET /api/v2/insights/user-latency?start_time=01-07-2023&end_time=08-07-2023&templates=name1,name2&order_by=P50&order=asc

{"report": {"start_time":"2023-07-01T00:00:00.000000Z","end_time":"2023-07-08T00:00:00.000000Z","template_ids": ["uuid1","uuid2"],"latency": [            {"template_ids": ["uuid1"],"user_id":"fcb9f5c7-ad6d-4515-b12e-496bc04ca116","name":"John Doe","connection_latency_ms": {"P50":5.601,"P95":16.352049999999984                }            },            {"template_ids": ["uuid2"],"user_id":"aee4bef9-479f-488e-abb4-b2bce2bf9e0d","name":"Jane Doe","connection_latency_ms": {"P50":31.312,"P95":119.832                }            }        ],    }}

If yes, then we should provide some sample/starter Grafana dashboards.

link

I know about this. But I was thinking about the need and motivation to do this nativity.

Something that's not possible via Prometheus, for example, is giving the number of unique active users for a certain time-frame (something that's to be shown in the proposed insights page). Prometheus can show how many unique there are at any certain time, but if we want the count for a day we can't simply add these values.