Apache APISIX is an API Gateway, which builds upon theOpenResty reverse-proxy to offer a plugin-based architecture. The main benefit of such an architecture is that it brings structure to the configuration of routes. It's a help at scale, when managing hundreds or thousands of routes.

In this post, I'd like to describe how plugins, priority, and phases play together and what pitfalls you must be aware of.

APISIX plugin's priority

When you configure a route with multiple plugins, Apache APISIX needs to execute them in aconsistent order so that the results are the same over time. For this reason, every APISIX plugin has aharcodedpriority. You can check a plugin priority directly in the code. For example, here's the relevant code fragment for thebasic-auth plugin:

local_M={version=0.1,priority=2520,#1type='auth',name=plugin_name,schema=schema,consumer_schema=consumer_schema}

1. Priority is 2520

For documentation purposes, thedefault-config.yaml file lists the priority of all out-of-the-box plugins. Here's an excerpt:

plugins:-ldap-auth# priority: 2540-hmac-auth# priority: 2530-basic-auth# priority: 2520-jwt-auth# priority: 2510-key-auth# priority: 2500-consumer-restriction# priority: 2400-forward-auth# priority: 2002-opa# priority: 2001-authz-keycloak# priority: 2000#- error-log-logger              # priority: 1091-proxy-cache# priority: 1085-body-transformer# priority: 1080-proxy-mirror# priority: 1010-proxy-rewrite# priority: 1008-workflow# priority: 1006-api-breaker# priority: 1005-limit-conn# priority: 1003-limit-count# priority: 1002-limit-req# priority: 1001

Imagine a route configured withproxy-mirror andproxy-rewrite. Because of their respective priority,proxy-mirror would run beforeproxy-rewrite.

upstream:-id:1nodes:"oldapi:8081":1routes:-id:1uri:"/v1/hello*"upstream_id:1plugins:proxy-rewrite:#1-3regex_uri:["/v1(.*)","$1"]proxy-mirror:#2host:http://new.api:8082

1. The plugin ordering in this file is not relevant
2. proxy-mirror has priority 1010 and will execute first
3. proxy-rewrite has priority 1008 and will run second

The above setup has an issue. For example, if we calllocalhost:9080/v1/hello, APISIX will first mirror the request, then remove the/v1 prefix. Hence, the new API receives/v1/hello instead of/hello. It's possible to override the default priority to fix it:

routes:-id:1uri:"/v1/hello*"upstream_id:1plugins:proxy-rewrite:_meta:priority:1020#1regex_uri:["/v1(.*)","$1"]proxy-mirror:host:http://new.api:8082

1. Override the default priority

Now,proxy-rewrite has higher priority thanproxy-mirror: the former runs before the latter.

In this case, it works flawlessly; in others, it might not. Let's dive further.

APISIX phases

APISIX runs plugins not only according to their priorities but also through dedicated phases. Because APISIX builds upon OpenResty, which builds upon ngxinx, the phases are very similar to the phases of these two components.

nginx defines severalphases an HTTP request goes through:

1. NGX_HTTP_POST_READ_PHASE
2. NGX_HTTP_SERVER_REWRITE_PHASE
3. NGX_HTTP_FIND_CONFIG_PHASE
4. NGX_HTTP_REWRITE_PHASE
5. NGX_HTTP_POST_REWRITE_PHASE
6. NGX_HTTP_PREACCESS_PHASE
7. NGX_HTTP_ACCESS_PHASE
8. NGX_HTTP_POST_ACCESS_PHASE
9. NGX_HTTP_PRECONTENT_PHASE
10. NGX_HTTP_CONTENT_PHASE
11. NGX_HTTP_LOG_PHASE

Each phase focuses on a task,i.e.,NGX_HTTP_ACCESS_PHASE verifies that the client is authorized to make the request.

In turn, OpenResty offers similarly named phases.

From nginx documentation:

image:77d1c09e-1a37-11e6-97ef-d9767035fc3e.png[,840,761]

Finally, here are the phases of Apache APISIX:

1. rewrite
2. access
3. before_proxy
4. header_filter
5. body_filter
6. log

From Apache APISIX documentation:

We have seen two ways to order plugins: by priority and by phase. Now comes the most important rule:order by priority only works inside a phase!

For example, theredirect plugin has a priority of 900 and runs in phaserewrite; thegzip plugin has a priority of 995 and runs in phasebody_filter. Regardless of their respective priorities,redirect will happen beforegzip, becauserewrite happens beforebody_filter. Likewise, changing a priority won't "move" a plugin out of its phase.

The example above withproxy-mirror andproxy-rewrite worked because both run in therewrite phase.

The main issue is that priority is documented in theconfig-default.yaml file, while the phase is buried in the code. Worse, some plugins run across different phases. For example, let's check the proxyproxy-rewrite plugin and, more precisely, the functions definedthere:

• local function is_new_headers_conf(headers)
• local function check_set_headers(headers)
• function _M.check_schema(conf)
• function _M.rewrite(conf, ctx)

The name of therewrite() function is suspiciously similar to one of the phases above. Looking at other plugins, we see the same pattern repeat. Apache APISIX runs plugin functions with the same name as the phase.

I took the liberty of summarizing all plugins and their respective phases in a table.

Conclusion

I've detailed Apache APISIX plugin phases and priorities in this post. I've explained their relationship with one another. Icing on the cake, I documented each out-of-the-box plugin's phase(s). I hope it will prove helpful.

To go further:

• Default plugin priorities
• Apache APISIX plugin execution lifecycle
• Plugin execution code

Originally published atA Java Geek on December 10^th, 2023