Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit5539116

Browse files
committed
aoai recepes for apim
1 parente049780 commit5539116

File tree

4 files changed

+139
-23
lines changed

4 files changed

+139
-23
lines changed
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
title:Setup a local ai playground on your W11 machine
3+
date:2025-01-01 10:00
4+
tags:[Azure OpenAI ]
5+
excerpt:""
6+
7+
header:
8+
overlay_image:https://live.staticflickr.com/65535/52755090506_6cf0808a3c_h.jpg
9+
caption:"Photo credit: [**nicola since 1972**](https://www.flickr.com/photos/15216811@N06/52755090506)"
10+
---
11+
12+
The AI local playground: in this blog post I show how to configure your W11 machine
13+
to start experimenting with the most common artificial intelligence APIs
14+
using Visual Studio Code, Python, and Jupyter Notebooks.
15+
16+
Starting from a clean Windows 11, the components to install and configure are the following:
17+
18+
* Windows Subsystem for Linux: It is a compatibility layer for running Linux binary executables natively on Windows 11. WSL allows developers to use a Linux environment, including most command-line tools, utilities, and applications, directly on Windows, without the overhead of a traditional virtual machine or dual-boot setup.
19+
* Conda Python virtual environment: An environment created using the Conda package manager, which can manage dependencies and packages for Python and other languages.
20+
>A Python environment is a context in which Python code is executed. It includes the Python interpreter and a set of installed packages and dependencies.
21+
* Visual Studio Code: Visual Studio Code (VS Code) is a free, open-source code editor developed by Microsoft. It is available for Windows, macOS, and Linux.
22+
23+
once these components are installed, I will show a pattern to use for experimenting with the following APIs:
24+
25+
* Azure OpenAI service (chat, completition)
26+
27+

‎_drafts/apim/00-organize-aoai-instances-with-apim.md‎

Lines changed: 98 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -32,13 +32,13 @@ in questi post ci focalizzeremo sulla configurazione delle policy e di implement
3232
Il laboratorio su cui lavoró prevede le seguenti risorse:
3333

3434
* Azure API Management service (developer SKU) enterpriseapim
35-
* 2 x Azure OpenAI service (S0 SKU) apimaoai01 and apimaoai02
35+
* 2 x Azure OpenAI service (S0 SKU)`apimaoai01` and`apimaoai02`
3636

3737
come mostrato nello schema seguente:
3838

3939
![architecture](../../assets/post/2025/apim-aoai/00-architecture.png)
4040

41-
#01Add Azure Open AI as backend resource
41+
#Add Azure Open AI as backend resource
4242

4343
Go to API Management services >`nicold-apim` > Backends > Add
4444
* Name:`apim-001`
@@ -63,7 +63,7 @@ Go to API Management services > `nicold-apim` > Backends > Add
6363
Here the result:
6464
![backend resorces added](../../assets/post/2025/apim-aoai/01-add-backend-resources.png)
6565

66-
#02 show apim-001 endpoint as root API
66+
#Show apim-001 endpoint as root API
6767

6868
Go to Azure Portal > API Management Sevices >`nicold-apim` > API > Create form Azure Resources > Azure OpenAI Service
6969
* Azure OpenAI instance:`nicold-aoai-001`
@@ -84,7 +84,7 @@ To test this endpoint go to: API Management Services > `nicold-apim` > APIs > Al
8484
* request body:`{"messages": [{ "role": "system","content": "You are a helpful assistant."},{ "role": "user", "content": "Tell me a joke!"} ]}`
8585
* click:[SEND]
8686

87-
herea script to test this configuration:
87+
herealso a powershell script to test this configuration:
8888

8989
```
9090
$openai = @{
@@ -108,13 +108,18 @@ $headers = [ordered]@{
108108
# Send a request to generate an answer
109109
$url = "$($openai.api_base)/deployments/$($openai.name)/chat/completions?api-version=$($openai.api_version)"
110110
111-
$response = IRM -Uri $url -Headers $headers -Body $body -Method Post -ContentType 'application/json'
112-
$response.choices.message.content
111+
$response = Invoke-WebRequest -Uri $url -Headers $headers -Body $body -Method Post -ContentType 'application/json'
113112
113+
# Show response headers
114+
$response.headers
115+
$responseObj = ConvertFrom-Json $response.content
116+
117+
# Show response body
118+
$responseObj.choices.message.content
114119
```
115120

116121
#Implement throttling
117-
The following policy limits the access at 10 requests per minute. Paste the xml in: API Management Service >
122+
The following policy limits the access**at 10 requests per minute**. Paste the xml in: API Management Service >
118123
`nicold-apim` > APIs > All APIs >`/` > all operations > inbound processing > policies (code editor)
119124

120125
```
@@ -137,18 +142,102 @@ The following policy limits the access at 10 requests per minute. Paste the xml
137142
138143
```
139144

140-
to limit at 2 calls**per IP** in 60 seconds, use the following rate-limit xml:
145+
To limit at 2 calls**per IP** in 60 seconds, use the following rate-limit xml:
141146
```
142147
<rate-limit-by-key calls="2" renewal-period="60" counter-key="@(context.Request.IpAddress)" />
143148
```
144149

145-
to limit at 2 calls per API KEY in 60 seconds, use the following rate-limit xml:
150+
To limit at 2 calls per API KEY in 60 seconds, use the following rate-limit xml:
146151
```
147152
<rate-limit-by-key calls="2" renewal-period="60" counter-key="@(context.Request.Headers.GetValueOrDefault("api-key"))" />
148153
```
149154

155+
#Show in a Response header (_aoai-origin_) the host of the OpenAI API Called
156+
157+
Add the following XML in the**outbound** policy:
158+
159+
```
160+
<outbound>
161+
<base />
162+
<set-header name="aoai-origin" exists-action="override">
163+
<value>@(context.Request.Url.Host)</value>
164+
</set-header>
165+
</outbound>
166+
```
167+
168+
#Round robin calls between 2 instances of Open AI
169+
170+
Use the following**inbound** policy:
171+
172+
```
173+
<inbound>
174+
<base />
175+
<cache-lookup-value key="backend-rr" variable-name="backend-rr" />
176+
<choose>
177+
<when condition="@(!context.Variables.ContainsKey("backend-rr"))">
178+
<set-variable name="backend-rr" value="0" />
179+
<cache-store-value key="backend-rr" value="0" duration="100" />
180+
</when>
181+
</choose>
182+
<choose>
183+
<when condition="@(Convert.ToInt32(context.Variables["backend-rr"]) == 0)">
184+
<set-backend-service backend-id="apim-001" />
185+
<set-variable name="backend-rr" value="1" />
186+
<cache-store-value key="backend-rr" value="1" duration="100" />
187+
</when>
188+
<otherwise>
189+
<set-backend-service backend-id="apim-002" />
190+
<set-variable name="backend-rr" value="0" />
191+
<cache-store-value key="backend-rr" value="0" duration="100" />
192+
</otherwise>
193+
</choose>
194+
</inbound>
195+
```
196+
>💥In a round robin scenario, both open AI instances must have same deployments
197+
198+
#Fallback on a second openAI instance for 30 secs. If the first send a 4xx error
199+
200+
TODEBUG!!!
201+
202+
203+
<policies>
204+
<inbound>
205+
<base />
206+
<cache-lookup-value key="useSecondaryBackend" variable-name="useSecondaryBackend" />
207+
<choose>
208+
<when condition="@(context.Variables["useSecondaryBackend"] != null)">
209+
<set-backend-service backend-id="apim-002" />
210+
</when>
211+
<otherwise>
212+
<set-backend-service backend-id="apim-001" />
213+
</otherwise>
214+
</choose>
215+
</inbound>
216+
<backend>
217+
<base />
218+
</backend>
219+
<outbound>
220+
<base />
221+
</outbound>
222+
<on-error>
223+
<base />
224+
<set-header name="aoai-origin" exists-action="override">
225+
<value>@(context.Request.Url.Host)</value>
226+
</set-header>
227+
<choose>
228+
<when condition="@(context.Response.StatusCode >= 400 && context.Response.StatusCode < 500)">
229+
<set-variable name="useSecondaryBackend" value="true" />
230+
<cache-store-value key="useSecondaryBackend" value="true" duration="60" />
231+
</when>
232+
</choose>
233+
</on-error>
234+
</policies>
235+
236+
237+
150238

151239

240+
--------------------------
152241

153242
* Azure APIM to scale AOAI -https://github.com/Azure/aoai-apim
154243
* Manage Azure OpenAI using APIM -https://github.com/microsoft/AzureOpenAI-with-APIM
-26 Bytes
Loading

‎assets/post/2025/apim-aoai/schemas.drawio‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,37 @@
11
<mxfile host="Electron" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/26.0.16 Chrome/132.0.6834.196 Electron/34.2.0 Safari/537.36" version="26.0.16">
22
<diagram name="Page-1" id="FfDvVnpgteF9OjcdlrWo">
3-
<mxGraphModel dx="900" dy="566" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
3+
<mxGraphModel dx="1434" dy="956" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
44
<root>
55
<mxCell id="0" />
66
<mxCell id="1" parent="0" />
7-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-14" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#FCFCFC;dashed=1;arcSize=2;labelPosition=center;verticalLabelPosition=middle;align=center;verticalAlign=middle;"vertex="1"parent="1">
7+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-14" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#FCFCFC;dashed=1;arcSize=2;labelPosition=center;verticalLabelPosition=middle;align=center;verticalAlign=middle;"parent="1"vertex="1">
88
<mxGeometry x="370" y="140" width="370" height="300" as="geometry" />
99
</mxCell>
10-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-9" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;fontColor=#333333;strokeColor=#666666;arcSize=9;"vertex="1"parent="1">
10+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-9" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;fontColor=#333333;strokeColor=#666666;arcSize=9;"parent="1"vertex="1">
1111
<mxGeometry x="160" y="222" width="160" height="120" as="geometry" />
1212
</mxCell>
13-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-8" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;fontColor=#333333;strokeColor=#666666;arcSize=9;"vertex="1"parent="1">
13+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-8" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;fontColor=#333333;strokeColor=#666666;arcSize=9;"parent="1"vertex="1">
1414
<mxGeometry x="470" y="150" width="260" height="120" as="geometry" />
1515
</mxCell>
16-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-7" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;fontColor=#333333;strokeColor=#666666;arcSize=9;"vertex="1"parent="1">
16+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-7" value="" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;fontColor=#333333;strokeColor=#666666;arcSize=9;"parent="1"vertex="1">
1717
<mxGeometry x="470" y="310" width="260" height="120" as="geometry" />
1818
</mxCell>
19-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-1" value="&lt;b&gt;nicold-apim.azure-api.net&lt;/b&gt;" style="image;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/azure2/app_services/API_Management_Services.svg;"vertex="1"parent="1">
19+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-1" value="&lt;b&gt;nicold-apim.azure-api.net&lt;/b&gt;" style="image;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/azure2/app_services/API_Management_Services.svg;"parent="1"vertex="1">
2020
<mxGeometry x="210" y="260" width="54.17" height="50" as="geometry" />
2121
</mxCell>
22-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-2" value="nicold-aoai-001.openai.azure.com" style="image;aspect=fixed;html=1;points=[];align=left;fontSize=12;image=img/lib/azure2/ai_machine_learning/Azure_OpenAI.svg;labelPosition=right;verticalLabelPosition=middle;verticalAlign=middle;"vertex="1"parent="1">
22+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-2" value="nicold-aoai-001.openai.azure.com" style="image;aspect=fixed;html=1;points=[];align=left;fontSize=12;image=img/lib/azure2/ai_machine_learning/Azure_OpenAI.svg;labelPosition=right;verticalLabelPosition=middle;verticalAlign=middle;"parent="1"vertex="1">
2323
<mxGeometry x="478" y="160" width="40" height="40" as="geometry" />
2424
</mxCell>
25-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-3" value="nicold-aoai-002.openai.azure.com" style="image;aspect=fixed;html=1;points=[];align=left;fontSize=12;image=img/lib/azure2/ai_machine_learning/Azure_OpenAI.svg;labelPosition=right;verticalLabelPosition=middle;verticalAlign=middle;"vertex="1"parent="1">
25+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-3" value="nicold-aoai-002.openai.azure.com" style="image;aspect=fixed;html=1;points=[];align=left;fontSize=12;image=img/lib/azure2/ai_machine_learning/Azure_OpenAI.svg;labelPosition=right;verticalLabelPosition=middle;verticalAlign=middle;"parent="1"vertex="1">
2626
<mxGeometry x="478" y="320" width="40" height="40" as="geometry" />
2727
</mxCell>
28-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-4" value="model: &lt;b&gt;gpt4o-001&lt;/b&gt;" style="html=1;verticalLabelPosition=bottom;align=center;labelBackgroundColor=#ffffff;verticalAlign=top;strokeWidth=2;strokeColor=#0080F0;shadow=0;dashed=0;shape=mxgraph.ios7.icons.cube;"vertex="1"parent="1">
28+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-4" value="model: &lt;b&gt;gpt4o-001&lt;/b&gt;" style="html=1;verticalLabelPosition=bottom;align=center;labelBackgroundColor=#ffffff;verticalAlign=top;strokeWidth=2;strokeColor=#0080F0;shadow=0;dashed=0;shape=mxgraph.ios7.icons.cube;"parent="1"vertex="1">
2929
<mxGeometry x="570" y="210" width="30" height="30" as="geometry" />
3030
</mxCell>
31-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-6" value="model: &lt;b&gt;gpt4o-002&lt;/b&gt;" style="html=1;verticalLabelPosition=bottom;align=center;labelBackgroundColor=#ffffff;verticalAlign=top;strokeWidth=2;strokeColor=#0080F0;shadow=0;dashed=0;shape=mxgraph.ios7.icons.cube;"vertex="1"parent="1">
32-
<mxGeometry x="570" y="370" width="30" height="30" as="geometry" />
31+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-6" value="model: &lt;b&gt;gpt40-001&lt;/b&gt;&lt;br&gt;model: &lt;b&gt;gpt4o-002&lt;/b&gt;" style="html=1;verticalLabelPosition=bottom;align=center;labelBackgroundColor=#ffffff;verticalAlign=top;strokeWidth=2;strokeColor=#0080F0;shadow=0;dashed=0;shape=mxgraph.ios7.icons.cube;"parent="1"vertex="1">
32+
<mxGeometry x="570" y="357" width="30" height="30" as="geometry" />
3333
</mxCell>
34-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-10" value="" style="endArrow=classic;html=1;rounded=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;exitX=0.998;exitY=0.331;exitDx=0;exitDy=0;exitPerimeter=0;curved=0;fillColor=#dae8fc;strokeColor=#6c8ebf;"edge="1"parent="1" source="LyVmAa3Y3DXHs4rt1gu2-9" target="LyVmAa3Y3DXHs4rt1gu2-8">
34+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-10" value="" style="endArrow=classic;html=1;rounded=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;exitX=0.998;exitY=0.331;exitDx=0;exitDy=0;exitPerimeter=0;curved=0;fillColor=#dae8fc;strokeColor=#6c8ebf;" parent="1" source="LyVmAa3Y3DXHs4rt1gu2-9" target="LyVmAa3Y3DXHs4rt1gu2-8" edge="1">
3535
<mxGeometry width="50" height="50" relative="1" as="geometry">
3636
<mxPoint x="390" y="390" as="sourcePoint" />
3737
<mxPoint x="440" y="340" as="targetPoint" />
@@ -41,7 +41,7 @@
4141
</Array>
4242
</mxGeometry>
4343
</mxCell>
44-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-11" value="" style="endArrow=classic;html=1;rounded=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;exitX=1;exitY=0.75;exitDx=0;exitDy=0;curved=0;fillColor=#dae8fc;strokeColor=#6c8ebf;"edge="1"parent="1" source="LyVmAa3Y3DXHs4rt1gu2-9" target="LyVmAa3Y3DXHs4rt1gu2-7">
44+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-11" value="" style="endArrow=classic;html=1;rounded=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;exitX=1;exitY=0.75;exitDx=0;exitDy=0;curved=0;fillColor=#dae8fc;strokeColor=#6c8ebf;" parent="1" source="LyVmAa3Y3DXHs4rt1gu2-9" target="LyVmAa3Y3DXHs4rt1gu2-7" edge="1">
4545
<mxGeometry width="50" height="50" relative="1" as="geometry">
4646
<mxPoint x="332" y="250" as="sourcePoint" />
4747
<mxPoint x="480" y="210" as="targetPoint" />
@@ -51,7 +51,7 @@
5151
</Array>
5252
</mxGeometry>
5353
</mxCell>
54-
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-13" value="" style="endArrow=classic;html=1;rounded=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#dae8fc;strokeColor=#6c8ebf;"edge="1"parent="1" target="LyVmAa3Y3DXHs4rt1gu2-9">
54+
<mxCell id="LyVmAa3Y3DXHs4rt1gu2-13" value="" style="endArrow=classic;html=1;rounded=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#dae8fc;strokeColor=#6c8ebf;" parent="1" target="LyVmAa3Y3DXHs4rt1gu2-9" edge="1">
5555
<mxGeometry width="50" height="50" relative="1" as="geometry">
5656
<mxPoint x="80" y="282" as="sourcePoint" />
5757
<mxPoint x="110" y="270" as="targetPoint" />

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp