This PR is a request for comment on a design of using HTTP requests for inter-Head dependencies. It removes TrainHead usage of DataSource.

Background

TrainHead usesDataOrganizer.get_actor_infos to get facts about Actors. This can't be easily reduced to simple singular GcsClient calls, because it comes from a merge of Actor infos and Worker infos (e.g.actor["gpus"][0][processesPids"] are fromDataSource.node_physical_stats that roots to GCSGetAllResourceUsage rpc.

Proposal

Let the TrainHead depend on ActorHead by directly calling HTTP requests. The overhead should be small since they are guaranteed to live in a same node.

Scope?

We will do direct read to GCS as much as possible. For cases like this, where it's not trivial to adapt, and frequency is low, and (maybe) non critical, we can use http client.

Changes

• In our cache middleware, add support forCache-Control: no-cache.
• In ActorHead, add optional paramactor_ids to APIGET /logical/actors
• In TrainHead, call that API with no-cache

Alternative

Now, ResourceUsage are subscribed by ReportHead and written toDataSource.node_physical_stats (moving to NodeHead in#49878). To do "true isolation" we will need to define a way to get snapshot info of ResourceUsage for a certain Node, which can be a bigger amount of change.