i think this is the right place fortask_cpu_time andtask_udf_time. if i understand your intent correctly, fortask_idle_time, i think we need some additional information. we need to get the difference between (time at which current task finished) and (time at which next task starts)

I think what I was aiming for here is how long we are idlewithin a task, do you think that is possible here? Some situations I would hope to measure with it would be time spent betweenyields of a generator task, or from streaming backpressure due to hitting the max number of streaming blocks. Is that possible here?

time spent between yields of a generator task

Would this be the same as the time it takes for the generator to produce the next output? or more like the timing around here when we yield batches from the iterator?https://github.com/ray-project/ray/blob/master/python/ray/data/iterator.py#L185-L186

from streaming backpressure due to hitting the max number of streaming blocks

we have this existing metrictask_submission_backpressure_time, which keeps track of the cumulative time spent in backpressure to submit the next task. is this what you had in mind for your use case?
https://github.com/ray-project/ray/blob/master/python/ray/data/_internal/execution/interfaces/op_runtime_metrics.py#L204-L210

More like the timing around when we yield batches from the iterator.

As for thetask_submission_backpressure_time, I'm still a bit confused on how to interpret metric. It seems like this metric will increase as long as we have enough tasks submitted to our pool of workers. But that would always seem like the case given that the scheduling loop runs every millisecond or so. For a one-operator pipeline, what does an increasing or stagnatedtask_submission_backpressure_time indicate? I don't think it would mean that we're necessarily producing blocks faster than consuming them, because the running tasks themselves could just be slow, right?

I think Raul kind of mentions in thiscomment what I'd like to capture. When do we apply backpressure to running tasks? How can we measure that?

For a one-operator pipeline, what does an increasing or stagnated task_submission_backpressure_time indicate? I don't think it would mean that we're necessarily producing blocks faster than consuming them, because the running tasks themselves could just be slow, right?

Yeah, the reason could be several possible things, but the underlying cause is that there are too many tasks queued for the operator, so we don't want to add any more.

This is where we evaluate whether the op should be backpressured. I think older Ray versions may look slightly different, but essentiallybackpressure_policy.can_add_input() returns False if backpressure is to be applied. When backpressure is applied, the operator will no longer accept new inputs.

a slightly more descriptive line could be something like:

did you want to also add a chart for this, or just track the metric via the class and access it elsewhere? i didn't see one, i may have missed it

same for this, did you want to add a chart? or just track the metric via the class and access it elsewhere?

since this is a metric which is tracked for the overall execution of the entire dataset, and not related to iteration,update_execution_metrics() would be a more appropriate place to update this.

for the iteration related metrics, let's move them toclear_iteration_metrics().

this LGTM