Training a simple neural network, with PyTorch data loading
Contents
Training a simple neural network, with PyTorch data loading#
Copyright 2018 The JAX Authors.
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License.You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an “AS IS” BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.
Let’s combine everything we showed in thequickstart to train a simple neural network. We will first specify and train a simple MLP on MNIST using JAX for the computation. We will use PyTorch’s data loading API to load images and labels (because it’s pretty great, and the world doesn’t need yet another data loading library).
Of course, you can use JAX with any API that is compatible with NumPy to make specifying the model a bit more plug-and-play. Here, just for explanatory purposes, we won’t use any neural network libraries or special APIs for building our model.
importjax.numpyasjnpfromjaximportgrad,jit,vmapfromjaximportrandom
Hyperparameters#
Let’s get a few bookkeeping items out of the way.
# A helper function to randomly initialize weights and biases# for a dense neural network layerdefrandom_layer_params(m,n,key,scale=1e-2):w_key,b_key=random.split(key)returnscale*random.normal(w_key,(n,m)),scale*random.normal(b_key,(n,))# Initialize all layers for a fully-connected neural network with sizes "sizes"definit_network_params(sizes,key):keys=random.split(key,len(sizes))return[random_layer_params(m,n,k)form,n,kinzip(sizes[:-1],sizes[1:],keys)]layer_sizes=[784,512,512,10]step_size=0.01num_epochs=8batch_size=128n_targets=10params=init_network_params(layer_sizes,random.key(0))
Auto-batching predictions#
Let us first define our prediction function. Note that we’re defining this for asingle image example. We’re going to use JAX’svmap function to automatically handle mini-batches, with no performance penalty.
fromjax.scipy.specialimportlogsumexpdefrelu(x):returnjnp.maximum(0,x)defpredict(params,image):# per-example predictionsactivations=imageforw,binparams[:-1]:outputs=jnp.dot(w,activations)+bactivations=relu(outputs)final_w,final_b=params[-1]logits=jnp.dot(final_w,activations)+final_breturnlogits-logsumexp(logits)
Let’s check that our prediction function only works on single images.
# This works on single examplesrandom_flattened_image=random.normal(random.key(1),(28*28,))preds=predict(params,random_flattened_image)print(preds.shape)
(10,)
# Doesn't work with a batchrandom_flattened_images=random.normal(random.key(1),(10,28*28))try:preds=predict(params,random_flattened_images)exceptTypeError:print('Invalid shapes!')
Invalid shapes!
# Let's upgrade it to handle batches using `vmap`# Make a batched version of the `predict` functionbatched_predict=vmap(predict,in_axes=(None,0))# `batched_predict` has the same call signature as `predict`batched_preds=batched_predict(params,random_flattened_images)print(batched_preds.shape)
(10, 10)
At this point, we have all the ingredients we need to define our neural network and train it. We’ve built an auto-batched version ofpredict, which we should be able to use in a loss function. We should be able to usegrad to take the derivative of the loss with respect to the neural network parameters. Last, we should be able to usejit to speed up everything.
Utility and loss functions#
defone_hot(x,k,dtype=jnp.float32):"""Create a one-hot encoding of x of size k."""returnjnp.array(x[:,None]==jnp.arange(k),dtype)defaccuracy(params,images,targets):target_class=jnp.argmax(targets,axis=1)predicted_class=jnp.argmax(batched_predict(params,images),axis=1)returnjnp.mean(predicted_class==target_class)defloss(params,images,targets):preds=batched_predict(params,images)return-jnp.mean(preds*targets)@jitdefupdate(params,x,y):grads=grad(loss)(params,x,y)return[(w-step_size*dw,b-step_size*db)for(w,b),(dw,db)inzip(params,grads)]
Data loading with PyTorch#
JAX is laser-focused on program transformations and accelerator-backed NumPy, so we don’t include data loading or munging in the JAX library. There are already a lot of great data loaders out there, so let’s just use them instead of reinventing anything. We’ll grab PyTorch’s data loader, and make a tiny shim to make it work with NumPy arrays.
!pipinstalltorchtorchvisionRequirement already satisfied: torch in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (2.4.1)Requirement already satisfied: torchvision in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (0.19.1)Requirement already satisfied: filelock in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from torch) (3.16.0)Requirement already satisfied: typing-extensions>=4.8.0 in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from torch) (4.12.2)Requirement already satisfied: sympy in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from torch) (1.13.2)Requirement already satisfied: networkx in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from torch) (3.3)Requirement already satisfied: jinja2 in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from torch) (3.1.4)Requirement already satisfied: fsspec in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from torch) (2024.9.0)Requirement already satisfied: setuptools in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from torch) (73.0.1)Requirement already satisfied: numpy in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from torchvision) (1.26.4)Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from torchvision) (10.4.0)Requirement already satisfied: MarkupSafe>=2.0 in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from jinja2->torch) (2.1.5)Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages (from sympy->torch) (1.3.0)
/home/m/.opt/miniforge3/envs/jax/lib/python3.12/pty.py:95: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock. pid, fd = os.forkpty()
importnumpyasnpfromjax.tree_utilimporttree_mapfromtorch.utils.dataimportDataLoader,default_collatefromtorchvision.datasetsimportMNISTdefnumpy_collate(batch):""" Collate function specifies how to combine a list of data samples into a batch. default_collate creates pytorch tensors, then tree_map converts them into numpy arrays. """returntree_map(np.asarray,default_collate(batch))defflatten_and_cast(pic):"""Convert PIL image to flat (1-dimensional) numpy array."""returnnp.ravel(np.array(pic,dtype=jnp.float32))
# Define our dataset, using torch datasetsmnist_dataset=MNIST('/tmp/mnist/',download=True,transform=flatten_and_cast)# Create pytorch data loader with custom collate functiontraining_generator=DataLoader(mnist_dataset,batch_size=batch_size,collate_fn=numpy_collate)
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gzFailed to download (trying next):HTTP Error 404: Not FoundDownloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gzDownloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to /tmp/mnist/MNIST/raw/train-images-idx3-ubyte.gzExtracting /tmp/mnist/MNIST/raw/train-images-idx3-ubyte.gz to /tmp/mnist/MNIST/rawDownloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gzFailed to download (trying next):HTTP Error 404: Not FoundDownloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gzDownloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz to /tmp/mnist/MNIST/raw/train-labels-idx1-ubyte.gzExtracting /tmp/mnist/MNIST/raw/train-labels-idx1-ubyte.gz to /tmp/mnist/MNIST/rawDownloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gzFailed to download (trying next):HTTP Error 404: Not FoundDownloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gzDownloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz to /tmp/mnist/MNIST/raw/t10k-images-idx3-ubyte.gzExtracting /tmp/mnist/MNIST/raw/t10k-images-idx3-ubyte.gz to /tmp/mnist/MNIST/rawDownloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gzFailed to download (trying next):HTTP Error 404: Not FoundDownloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gzDownloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz to /tmp/mnist/MNIST/raw/t10k-labels-idx1-ubyte.gzExtracting /tmp/mnist/MNIST/raw/t10k-labels-idx1-ubyte.gz to /tmp/mnist/MNIST/raw
100.0%100.0%100.0%100.0%
# Get the full train dataset (for checking accuracy while training)train_images=np.array(mnist_dataset.train_data).reshape(len(mnist_dataset.train_data),-1)train_labels=one_hot(np.array(mnist_dataset.train_labels),n_targets)# Get full test datasetmnist_dataset_test=MNIST('/tmp/mnist/',download=True,train=False)test_images=jnp.array(mnist_dataset_test.test_data.numpy().reshape(len(mnist_dataset_test.test_data),-1),dtype=jnp.float32)test_labels=one_hot(np.array(mnist_dataset_test.test_labels),n_targets)
/home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages/torchvision/datasets/mnist.py:76: UserWarning: train_data has been renamed data warnings.warn("train_data has been renamed data")/home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages/torchvision/datasets/mnist.py:66: UserWarning: train_labels has been renamed targets warnings.warn("train_labels has been renamed targets")/home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages/torchvision/datasets/mnist.py:81: UserWarning: test_data has been renamed data warnings.warn("test_data has been renamed data")/home/m/.opt/miniforge3/envs/jax/lib/python3.12/site-packages/torchvision/datasets/mnist.py:71: UserWarning: test_labels has been renamed targets warnings.warn("test_labels has been renamed targets")Training loop#
importtimeforepochinrange(num_epochs):start_time=time.time()forx,yintraining_generator:y=one_hot(y,n_targets)params=update(params,x,y)epoch_time=time.time()-start_timetrain_acc=accuracy(params,train_images,train_labels)test_acc=accuracy(params,test_images,test_labels)print("Epoch{} in{:0.2f} sec".format(epoch,epoch_time))print("Training set accuracy{}".format(train_acc))print("Test set accuracy{}".format(test_acc))
Epoch 0 in 5.53 secTraining set accuracy 0.9156666994094849Test set accuracy 0.9199000000953674Epoch 1 in 1.13 secTraining set accuracy 0.9370499849319458Test set accuracy 0.9383999705314636Epoch 2 in 1.12 secTraining set accuracy 0.9490833282470703Test set accuracy 0.9467999935150146Epoch 3 in 1.21 secTraining set accuracy 0.9568833708763123Test set accuracy 0.9532999992370605Epoch 4 in 1.17 secTraining set accuracy 0.9631666541099548Test set accuracy 0.9574999809265137Epoch 5 in 1.17 secTraining set accuracy 0.9675000309944153Test set accuracy 0.9615999460220337Epoch 6 in 1.11 secTraining set accuracy 0.9709500074386597Test set accuracy 0.9652999639511108Epoch 7 in 1.17 secTraining set accuracy 0.9736999869346619Test set accuracy 0.967199981212616
We’ve now used the whole of the JAX API:grad for derivatives,jit for speedups andvmap for auto-vectorization.We used NumPy to specify all of our computation, and borrowed the great data loaders from PyTorch, and ran the whole thing on the GPU.