Why this isnt alwaysLazyLocalNamespace? Is something actually allowed to initialize this with any kind of Mapping? There is some empties somewhere in the PR but it could be also then just empty LazyLocalNamespace

I wanted to haveLazyLocalNamespace as an implementation detail, as this gives us more flexibility, especially if in the future we expose theGenerateSchema class to Pydantic users (thetypes_namespace property exposesNamespaceTuple).

Hows the perf looking with this in ForwardRefs cases like in the mega kube file thing when using future annotations? So how this works by flatting the dicts vs just iterating them, perf wise? I guess thedef data is really never active when ForwardRefs are not being used by user, right?

I guess thedef data is really never active when ForwardRefs are not being used by user, right?

Correct.

Running memray on the k8s file, I get:

• ~3 GiB total allocations on main
• ~1.5GiB total allocations here
• ~1.9GiB total allocations here, with the future import.

So how this works by flatting the dicts vs just iterating them, perf wise?

Considering most of the local namespaces are really small (compared to globals), If I remember correctly the difference is marginal, at least by experimenting quickly withtimeit and some dummy examples.

This is great, I'm super glad to see we've reduced the duplicate allocations in context management that we were dealing with before this PR.

This is great, I'm super glad to see we've reduced the duplicate allocations in context management that we were dealing with before this PR.

CC@adriangb I think this is broken by this PR but maybe worth breaking. I think it's possible to fix by fiddling with the__get_pydantic_core_schema__ on the Pipeline and making sure we plumb through the namespace stuff, but maybe harder than it's worth doing this second

I think@Viicos decided this was worth breaking. As long as there's clear documentation showing the alternative path forward I'm okay with that.

Copy pasting what I added on Slack so that it's not lost:

the TL;DR is using lambdas with references to other symbols in annotations opens the door to a lot of weird behaviors.

This is the (simplified) test failing on the PR:

fromtyping_extensionsimportAnnotatedfrompydanticimportBaseModelfrompydantic.experimental.pipelineimportvalidate_as_deferredclassUser(BaseModel):family:'Annotated[list[User], validate_as_deferred(lambda: list[User])]'# The `family` annotation is successfully evaluated, but at some point during# core schema generation, the pipeline API logic is triggered and when the lambda# gets called, we end up with:# NameError: name 'User' is not defined

Onmain, when we evaluate (read: Python will call eval) the annotation for family, we use the following as globals:
{'User': __main__.User, 'Annotated': ..., 'BaseModel': ..., ...}
and locals are empty.

On this PR, we cleaned up how globals and locals were mixed up before. This means that we now use the following as globals:
{'Annotated': ..., 'BaseModel': ..., ...}
and locals:
{'User': __main__.User, ...}

And the issue comes from what could be considered as a limitation ofeval. Consider this example:

deffunc():A=intworks=lambda:list[A]fails=eval('lambda: list[A]',globals(),locals())works()# list[int]fails()# NameError: A is not defined.

Theeval limitation is that it does not have access to the non-locals of thelambda environment (which is a new scope, like with adef statement). Even thoughA is present in locals, it won't be used to resolveA and soeval will look up in theglobals instead (that's why it works onmain becauseUser was added inglobals for theeval call).

This limitation is documented in this open CPythonPR.

Any reason not to usecollections.ChainMap here instead of looping over all the items in all the dicts?

I thought about using chain maps but couldn't remember all the reasons I did not. Looking into it, we could also
ditchLazyLocalNamespace and use a chain map instead, butChainMap is a mutable mapping, and ideally we'd like to enforce immutability (at least from at the type checker level). I added a comment about this.

Regarding whetherdata should return a chain map instead, we discussed about it inthis comment.

Another place collections.ChainMap could maybe be used, maybe not necessary