Plugin#

classtensorrt_llm.plugin.PluginConfig(_explicitly_disable_gemm_plugin:bool=False)[source]#

Bases:object

The config that manages plugin-related options.

There are two option categories:* Plugin options (typically with xxx_plugin naming). These options can be assigned with:

• “float16”/”bfloat16”/”float32”/”int32”, which means the plugin is enabled with the specified precision; (Some plugins only support limited dtype, i.e., gemm_swiglu_plugin and low_latency_gemm_swiglu_plugin only supports fp8 now)

• “auto”, which means the plugin is enabled with the precision ofdtype field (thedtype field must be same to model dtype, i.e., the one in PretrainedConfig);

• None, which means the plugin is disabled.

• Other features. These options can be assigned with boolean:

  • True, which means the plugin is enabled;

  • False, which means the plugin is disabled.

Note: All the fields should use a prefix “_”; PluginConfigMeta will wrap each field as a property.This ensures the fields can only be assigned with allowed values.

to_legacy_setting()[source]#

Legacy setting means that all of the plugins and features aredisabled, this is needed for the legacybuild.py script, which will bemigrated to the centralized building scripttensorrt_llm/commands/build.py.

After the migration is done, this function may or may not be deleted.