Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Commit71b3d50

Browse files
authored
Merge pull request#239 from GokuMohandas/dev
adding dotenv for credentials management
2 parents0133795 +097ffaa commit71b3d50

File tree

9 files changed

+1656
-1735
lines changed

9 files changed

+1656
-1735
lines changed

‎README.md‎

Lines changed: 15 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,17 @@ We'll start by setting up our cluster with the environment and compute configura
101101

102102
</details>
103103

104+
###Credentials
105+
```bash
106+
touch .env
107+
```
108+
```bash
109+
# Inside .env
110+
GITHUB_USERNAME="CHANGE_THIS_TO_YOUR_USERNAME"# ← CHANGE THIS
111+
```bash
112+
source .env
113+
```
114+
104115
### Git setup
105116

106117
Create a repository by following these instructions: [Create a new repository](https://github.com/new) → name it`Made-With-ML` → Toggle`Add a README file` (**very important** as this creates a`main` branch) → Click`Create repository` (scroll down)
@@ -109,7 +120,7 @@ Now we're ready to clone the repository that has all of our code:
109120
110121
```bash
111122
git clone https://github.com/GokuMohandas/Made-With-ML.git .
112-
git remote set-url origin https://github.com/GITHUB_USERNAME/Made-With-ML.git# <-- CHANGE THIS to your username
123+
git remote set-url origin https://github.com/$GITHUB_USERNAME/Made-With-ML.git # <-- CHANGE THIS to your username
113124
git checkout -b dev
114125
```
115126
@@ -317,15 +328,7 @@ python madewithml/predict.py predict \
317328
python madewithml/serve.py --run_id $RUN_ID
318329
```
319330
320-
While the application is running, we can use it via cURL, Python, etc.:
321-
322-
```bash
323-
# via cURL
324-
curl -X POST -H "Content-Type: application/json" -d '{
325-
"title":"Transfer learning with transformers",
326-
"description":"Using transformers for transfer learning on text classification tasks."
327-
}' http://127.0.0.1:8000/predict
328-
```
331+
Once the application is running, we can use it via cURL, Python, etc.:
329332
330333
```python
331334
# via Python
@@ -341,13 +344,6 @@ python madewithml/predict.py predict \
341344
ray stop # shutdown
342345
```
343346
344-
```bash
345-
export HOLDOUT_LOC="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/holdout.csv"
346-
curl -X POST -H"Content-Type: application/json" -d'{
347-
"dataset_loc": "https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/holdout.csv"
348-
}' http://127.0.0.1:8000/evaluate
349-
```
350-
351347
</details>
352348
353349
<details open>
@@ -362,15 +358,7 @@ curl -X POST -H "Content-Type: application/json" -d '{
362358
python madewithml/serve.py --run_id$RUN_ID
363359
```
364360

365-
While the application is running, we can use it via cURL, Python, etc.:
366-
367-
```bash
368-
# via cURL
369-
curl -X POST -H"Content-Type: application/json" -d'{
370-
"title": "Transfer learning with transformers",
371-
"description": "Using transformers for transfer learning on text classification tasks."
372-
}' http://127.0.0.1:8000/predict
373-
```
361+
Once the application is running, we can use it via cURL, Python, etc.:
374362

375363
```python
376364
# via Python
@@ -399,7 +387,7 @@ export RUN_ID=$(python madewithml/predict.py get-best-run-id --experiment-name $
399387
pytest --run-id=$RUN_ID tests/model --verbose --disable-warnings
400388
401389
# Coverage
402-
python3 -m pytest --cov madewithml --cov-report html
390+
python3 -m pytesttests/code--cov madewithml --cov-report html --disable-warnings
403391
```
404392

405393
## Production

‎madewithml/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
fromdotenvimportload_dotenv
2+
3+
load_dotenv()

‎madewithml/config.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
# config.py
22
importlogging
3+
importos
34
importsys
45
frompathlibimportPath
56

@@ -10,9 +11,10 @@
1011
ROOT_DIR=Path(__file__).parent.parent.absolute()
1112
LOGS_DIR=Path(ROOT_DIR,"logs")
1213
LOGS_DIR.mkdir(parents=True,exist_ok=True)
14+
EFS_DIR=Path(f"/efs/shared_storage/madewithml/{os.environ.get('GITHUB_USERNAME','')}")
1315

1416
# Config MLflow
15-
MODEL_REGISTRY=Path("/tmp/mlflow")
17+
MODEL_REGISTRY=Path(f"{EFS_DIR}/mlflow")
1618
Path(MODEL_REGISTRY).mkdir(parents=True,exist_ok=True)
1719
MLFLOW_TRACKING_URI="file://"+str(MODEL_REGISTRY.absolute())
1820
mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)

‎madewithml/predict.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ def predict(
125125
# Load components
126126
best_checkpoint=get_best_checkpoint(run_id=run_id)
127127
predictor=TorchPredictor.from_checkpoint(best_checkpoint)
128-
preprocessor=predictor.get_preprocessor()
128+
#preprocessor = predictor.get_preprocessor()
129129

130130
# Predict
131131
sample_df=pd.DataFrame([{"title":title,"description":description,"tag":"other"}])

‎madewithml/serve.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
importargparse
2+
importos
23
fromhttpimportHTTPStatus
34
fromtypingimportDict
45

@@ -75,5 +76,5 @@ async def _predict(self, request: Request) -> Dict:
7576
parser.add_argument("--run_id",help="run ID to use for serving.")
7677
parser.add_argument("--threshold",type=float,default=0.9,help="threshold for `other` class.")
7778
args=parser.parse_args()
78-
ray.init()
79+
ray.init(runtime_env={"env_vars": {"GITHUB_USERNAME":os.environ["GITHUB_USERNAME"]}})
7980
serve.run(ModelDeployment.bind(run_id=args.run_id,threshold=args.threshold))

‎madewithml/train.py‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
importdatetime
22
importjson
3+
importos
34
fromtypingimportTuple
45

56
importnumpyasnp
@@ -23,7 +24,7 @@
2324
fromtyping_extensionsimportAnnotated
2425

2526
frommadewithmlimportdata,models,utils
26-
frommadewithml.configimportMLFLOW_TRACKING_URI,logger
27+
frommadewithml.configimportEFS_DIR,MLFLOW_TRACKING_URI,logger
2728

2829
# Initialize Typer CLI app
2930
app=typer.Typer()
@@ -200,10 +201,7 @@ def train_model(
200201
)
201202

202203
# Run config
203-
run_config=RunConfig(
204-
callbacks=[mlflow_callback],
205-
checkpoint_config=checkpoint_config,
206-
)
204+
run_config=RunConfig(callbacks=[mlflow_callback],checkpoint_config=checkpoint_config,storage_path=EFS_DIR)
207205

208206
# Dataset
209207
ds=data.load_data(dataset_loc=dataset_loc,num_samples=train_loop_config["num_samples"])
@@ -252,5 +250,5 @@ def train_model(
252250
if__name__=="__main__":# pragma: no cover, application
253251
ifray.is_initialized():
254252
ray.shutdown()
255-
ray.init()
253+
ray.init(runtime_env={"env_vars": {"GITHUB_USERNAME":os.environ["GITHUB_USERNAME"]}})
256254
app()

‎madewithml/tune.py‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
importdatetime
22
importjson
3+
importos
34

45
importray
56
importtyper
@@ -19,7 +20,7 @@
1920
fromtyping_extensionsimportAnnotated
2021

2122
frommadewithmlimportdata,train,utils
22-
frommadewithml.configimportMLFLOW_TRACKING_URI,logger
23+
frommadewithml.configimportEFS_DIR,MLFLOW_TRACKING_URI,logger
2324

2425
# Initialize Typer CLI app
2526
app=typer.Typer()
@@ -117,10 +118,7 @@ def tune_models(
117118
experiment_name=experiment_name,
118119
save_artifact=True,
119120
)
120-
run_config=RunConfig(
121-
callbacks=[mlflow_callback],
122-
checkpoint_config=checkpoint_config,
123-
)
121+
run_config=RunConfig(callbacks=[mlflow_callback],checkpoint_config=checkpoint_config,storage_path=EFS_DIR)
124122

125123
# Hyperparameters to start with
126124
initial_params=json.loads(initial_params)
@@ -178,5 +176,5 @@ def tune_models(
178176
if__name__=="__main__":# pragma: no cover, application
179177
ifray.is_initialized():
180178
ray.shutdown()
181-
ray.init()
179+
ray.init(runtime_env={"env_vars": {"GITHUB_USERNAME":os.environ["GITHUB_USERNAME"]}})
182180
app()

0 commit comments

Comments
 (0)

[8]ページ先頭

©2009-2025 Movatter.jp