I forgot to mention that I did not change cpu device interface yet in a similar way to cuda (initializeFiltersContext() not implemented for cpu). This would be a next step if this design will be accepted.

I've added one more commit to update CPU device interface to use filters path introduce inSingleStreamDecoder andDeviceInterface. Still a draft as#831 needs to be reviewed first.

I've rebase the PR on top of the recently landed#831 prerequisite. Change is ready for review.@scotts@NicolasHug

I pushed update to fix a linter issue which I have overlooked.

Error: Unable to download artifact(s): Artifact not found for name: pytorch_torchcodec__3.10_cu129_x86_64

This error in CI test seems unrelated to the change.

Pushed update to address another lint issue in the last commit.

Rebased on top of:

• Refactor context structs to use constructors #869

@dvrogozh, thank you for this work! I left a detailed comment where I'm trying to understand the value we're getting out of the new abstractions - please help me understand that better. At a higher-level, I wonder if we could take the following approach:

1. DeviceInterface is left unchanged; we don't add the concept of filter graph initialization to it.
2. SingleStreamDecoder is left unchanged.
3. CpuDeviceInterface is left unchanged.
4. CudaDeviceInterface is changed to add the new Cuda-specific filter graph implementation.

I'm sorry if that's the direction you were already going in - I know I pushed to see what the full generalization would look like. One quirk of the approach above is that each device may end up implementing the same patterns for their filter graph implementation. In that case, we could potentially add some member functions toDeviceInterface, but they would be private. They wouldn't be called bySingleStreamDecoder, but just guides for how best to implement filter graph support for a device.

@scotts : you have highlighted the alternate design approach to take. I think we have 2 options on a plate:

1. Keep decoder output conversions (filter graphs, sws calls, npp calls, etc.) within device interface classes
2. Define output conversions separate from device interface classes. In this case device interfaceconvertAVFrameToFrameOutput()becomes responsible only for the trivial operation - wrap AVFrame into Tensor without any conversions.

I am proposing to consider 2nd direction which gives more modular structure for the project and simplifies device interface to a set of trivial operations. I am trying to reduce complexity of writing a device interface for non-CUDA GPUs by reusing as much as possible.

In the sense of 2nd direction, this PR is the first step. As you saw, it moves ownership and management of the Filter Graph out of device interface (toSingleStreamDecoder). If we would take this direction further, I suggest to consider:

1. AbstractFilterGraph to allow multiple implementations, i.e. understand under Filter Graph any conversion not only via ffmpeg filters, but also via sws or npp or other libraries.
2. ImplementSwsGraph andNppGraph, i.e. move these out ofCpuDeviceInterface andCudaDeviceInterface
3. Above steps will leave device interface to support only trivial wrapping/copy of AVFrame into Tensor which is backend specific operation

Please, share your thoughts on the above. If you want, we can start by going with 1st direction (keep conversion in device interfaces) and continue weighing 2nd approach. I think enabling encoders might contribute to its justification as we again will consider filter operations. This time before encoding. If we will decide to go with 1st approach for now, then I will submit another PR with the focus on enabling 10-bit support in CUDA device interface.

@dvrogozh, thanks for explaining the direction you're going in! Based on where we are, I still prefer the first option: keeping all decoder output conversions within the device implementation.

I do think that in the future, we'll want better abstractions around filtergraph, swscale and NPP. That is, these are all ways to convert a raw decoded frame into our desired color space and potentially perform some native transforms. But what the exact abstractions we should build is not obvious to me right now. In particular, this PR has us in an in-between state where some stuff remains in the device implementation, and some it outside.

By pursuing option 1, I think we'll eventually get to a state where we have three device implementations: CPU and CUDA in this repo, and XPU in an Intel repo. At that point, it will be more clear to us what the patterns are and what abstractions to build.

By pursuing option 1, I think we'll eventually get to a state...

@scotts : I have submitted a PR to follow option 1. Please, help to review:

• Use cuda filters to support 10-bit videos #899

I will probably keep#853 opened and rebase it on top of#899 to continue working on option 2.