Random Tensors and Seeding#

Speaking of the random tensor, did you notice the call totorch.manual_seed() immediately preceding it? Initializing tensors,such as a model’s learning weights, with random values is common butthere are times - especially in research settings - where you’ll wantsome assurance of the reproducibility of your results. Manually settingyour random number generator’s seed is the way to do this. Let’s lookmore closely:

torch.manual_seed(1729)random1=torch.rand(2,3)print(random1)random2=torch.rand(2,3)print(random2)torch.manual_seed(1729)random3=torch.rand(2,3)print(random3)random4=torch.rand(2,3)print(random4)

tensor([[0.3126, 0.3791, 0.3087],        [0.0736, 0.4216, 0.0691]])tensor([[0.2332, 0.4047, 0.2162],        [0.9927, 0.4128, 0.5938]])tensor([[0.3126, 0.3791, 0.3087],        [0.0736, 0.4216, 0.0691]])tensor([[0.2332, 0.4047, 0.2162],        [0.9927, 0.4128, 0.5938]])

What you should see above is thatrandom1 andrandom3 carryidentical values, as dorandom2 andrandom4. Manually settingthe RNG’s seed resets it, so that identical computations depending onrandom number should, in most settings, provide identical results.

For more information, see thePyTorch documentation onreproducibility.

Tensor Shapes#

Often, when you’re performing operations on two or more tensors, theywill need to be of the sameshape - that is, having the same number ofdimensions and the same number of cells in each dimension. For that, wehave thetorch.*_like() methods:

x=torch.empty(2,2,3)print(x.shape)print(x)empty_like_x=torch.empty_like(x)print(empty_like_x.shape)print(empty_like_x)zeros_like_x=torch.zeros_like(x)print(zeros_like_x.shape)print(zeros_like_x)ones_like_x=torch.ones_like(x)print(ones_like_x.shape)print(ones_like_x)rand_like_x=torch.rand_like(x)print(rand_like_x.shape)print(rand_like_x)

torch.Size([2, 2, 3])tensor([[[-7.4134e-14,  3.0931e-41,  0.0000e+00],         [ 1.4013e-45,  8.9683e-44,  0.0000e+00]],        [[ 1.1210e-43,  0.0000e+00, -5.4490e-14],         [ 3.0931e-41,  1.4013e-45,  0.0000e+00]]])torch.Size([2, 2, 3])tensor([[[-5.3096e-14,  3.0931e-41,  1.4013e-45],         [ 0.0000e+00,  1.4013e-45,  0.0000e+00]],        [[ 1.4013e-45,  0.0000e+00,  1.4013e-45],         [ 0.0000e+00,  1.4013e-45,  0.0000e+00]]])torch.Size([2, 2, 3])tensor([[[0., 0., 0.],         [0., 0., 0.]],        [[0., 0., 0.],         [0., 0., 0.]]])torch.Size([2, 2, 3])tensor([[[1., 1., 1.],         [1., 1., 1.]],        [[1., 1., 1.],         [1., 1., 1.]]])torch.Size([2, 2, 3])tensor([[[0.6128, 0.1519, 0.0453],         [0.5035, 0.9978, 0.3884]],        [[0.6929, 0.1703, 0.1384],         [0.4759, 0.7481, 0.0361]]])

The first new thing in the code cell above is the use of the.shapeproperty on a tensor. This property contains a list of the extent ofeach dimension of a tensor - in our case,x is a three-dimensionaltensor with shape 2 x 2 x 3.

Below that, we call the.empty_like(),.zeros_like(),.ones_like(), and.rand_like() methods. Using the.shapeproperty, we can verify that each of these methods returns a tensor ofidentical dimensionality and extent.

The last way to create a tensor that will cover is to specify its datadirectly from a PyTorch collection:

some_constants=torch.tensor([[3.1415926,2.71828],[1.61803,0.0072897]])print(some_constants)some_integers=torch.tensor((2,3,5,7,11,13,17,19))print(some_integers)more_integers=torch.tensor(((2,4,6),[3,6,9]))print(more_integers)

tensor([[3.1416, 2.7183],        [1.6180, 0.0073]])tensor([ 2,  3,  5,  7, 11, 13, 17, 19])tensor([[2, 4, 6],        [3, 6, 9]])

Usingtorch.tensor() is the most straightforward way to create atensor if you already have data in a Python tuple or list. As shownabove, nesting the collections will result in a multi-dimensionaltensor.

Tensor Data Types#

Setting the datatype of a tensor is possible a couple of ways:

a=torch.ones((2,3),dtype=torch.int16)print(a)b=torch.rand((2,3),dtype=torch.float64)*20.print(b)c=b.to(torch.int32)print(c)

tensor([[1, 1, 1],        [1, 1, 1]], dtype=torch.int16)tensor([[ 0.9956,  1.4148,  5.8364],        [11.2406, 11.2083, 11.6692]], dtype=torch.float64)tensor([[ 0,  1,  5],        [11, 11, 11]], dtype=torch.int32)

The simplest way to set the underlying data type of a tensor is with anoptional argument at creation time. In the first line of the cell above,we setdtype=torch.int16 for the tensora. When we printa,we can see that it’s full of1 rather than1. - Python’s subtlecue that this is an integer type rather than floating point.

Another thing to notice about printinga is that, unlike when weleftdtype as the default (32-bit floating point), printing thetensor also specifies itsdtype.

You may have also spotted that we went from specifying the tensor’sshape as a series of integer arguments, to grouping those arguments in atuple. This is not strictly necessary - PyTorch will take a series ofinitial, unlabeled integer arguments as a tensor shape - but when addingthe optional arguments, it can make your intent more readable.

The other way to set the datatype is with the.to() method. In thecell above, we create a random floating point tensorb in the usualway. Following that, we createc by convertingb to a 32-bitinteger with the.to() method. Note thatc contains all the samevalues asb, but truncated to integers.

For more information, see thedata types documentation.

In Brief: Tensor Broadcasting#

The exception to the same-shapes rule istensor broadcasting. Here’san example:

rand=torch.rand(2,4)doubled=rand*(torch.ones(1,4)*2)print(rand)print(doubled)

tensor([[0.6146, 0.5999, 0.5013, 0.9397],        [0.8656, 0.5207, 0.6865, 0.3614]])tensor([[1.2291, 1.1998, 1.0026, 1.8793],        [1.7312, 1.0413, 1.3730, 0.7228]])

What’s the trick here? How is it we got to multiply a 2x4 tensor by a1x4 tensor?

Broadcasting is a way to perform an operation between tensors that havesimilarities in their shapes. In the example above, the one-row,four-column tensor is multiplied byboth rows of the two-row,four-column tensor.

This is an important operation in Deep Learning. The common example ismultiplying a tensor of learning weights by abatch of input tensors,applying the operation to each instance in the batch separately, andreturning a tensor of identical shape - just like our (2, 4) * (1, 4)example above returned a tensor of shape (2, 4).

The rules for broadcasting are:

• Each tensor must have at least one dimension - no empty tensors.

• Comparing the dimension sizes of the two tensors,going from last tofirst:

  • Each dimension must be equal,or

  • One of the dimensions must be of size 1,or

  • The dimension does not exist in one of the tensors

Tensors of identical shape, of course, are trivially “broadcastable”, asyou saw earlier.

Here are some examples of situations that honor the above rules andallow broadcasting:

a=torch.ones(4,3,2)b=a*torch.rand(3,2)# 3rd & 2nd dims identical to a, dim 1 absentprint(b)c=a*torch.rand(3,1)# 3rd dim = 1, 2nd dim identical to aprint(c)d=a*torch.rand(1,2)# 3rd dim identical to a, 2nd dim = 1print(d)

tensor([[[0.6493, 0.2633],         [0.4762, 0.0548],         [0.2024, 0.5731]],        [[0.6493, 0.2633],         [0.4762, 0.0548],         [0.2024, 0.5731]],        [[0.6493, 0.2633],         [0.4762, 0.0548],         [0.2024, 0.5731]],        [[0.6493, 0.2633],         [0.4762, 0.0548],         [0.2024, 0.5731]]])tensor([[[0.7191, 0.7191],         [0.4067, 0.4067],         [0.7301, 0.7301]],        [[0.7191, 0.7191],         [0.4067, 0.4067],         [0.7301, 0.7301]],        [[0.7191, 0.7191],         [0.4067, 0.4067],         [0.7301, 0.7301]],        [[0.7191, 0.7191],         [0.4067, 0.4067],         [0.7301, 0.7301]]])tensor([[[0.6276, 0.7357],         [0.6276, 0.7357],         [0.6276, 0.7357]],        [[0.6276, 0.7357],         [0.6276, 0.7357],         [0.6276, 0.7357]],        [[0.6276, 0.7357],         [0.6276, 0.7357],         [0.6276, 0.7357]],        [[0.6276, 0.7357],         [0.6276, 0.7357],         [0.6276, 0.7357]]])

Look closely at the values of each tensor above:

• The multiplication operation that createdb wasbroadcast over every “layer” ofa.

• Forc, the operation was broadcast over every layer and row ofa - every 3-element column is identical.

• Ford, we switched it around - now everyrow is identical,across layers and columns.

For more information on broadcasting, see thePyTorchdocumentationon the topic.

Here are some examples of attempts at broadcasting that will fail:

a=torch.ones(4,3,2)b=a*torch.rand(4,3)# dimensions must match last-to-firstc=a*torch.rand(2,3)# both 3rd & 2nd dims differentd=a*torch.rand((0,))# can't broadcast with an empty tensor

More Math with Tensors#

PyTorch tensors have over three hundred operations that can be performedon them.

Here is a small sample from some of the major categories of operations:

# common functionsa=torch.rand(2,4)*2-1print('Common functions:')print(torch.abs(a))print(torch.ceil(a))print(torch.floor(a))print(torch.clamp(a,-0.5,0.5))# trigonometric functions and their inversesangles=torch.tensor([0,math.pi/4,math.pi/2,3*math.pi/4])sines=torch.sin(angles)inverses=torch.asin(sines)print('\nSine and arcsine:')print(angles)print(sines)print(inverses)# bitwise operationsprint('\nBitwise XOR:')b=torch.tensor([1,5,11])c=torch.tensor([2,7,10])print(torch.bitwise_xor(b,c))# comparisons:print('\nBroadcasted, element-wise equality comparison:')d=torch.tensor([[1.,2.],[3.,4.]])e=torch.ones(1,2)# many comparison ops support broadcasting!print(torch.eq(d,e))# returns a tensor of type bool# reductions:print('\nReduction ops:')print(torch.max(d))# returns a single-element tensorprint(torch.max(d).item())# extracts the value from the returned tensorprint(torch.mean(d))# averageprint(torch.std(d))# standard deviationprint(torch.prod(d))# product of all numbersprint(torch.unique(torch.tensor([1,2,1,2,1,2])))# filter unique elements# vector and linear algebra operationsv1=torch.tensor([1.,0.,0.])# x unit vectorv2=torch.tensor([0.,1.,0.])# y unit vectorm1=torch.rand(2,2)# random matrixm2=torch.tensor([[3.,0.],[0.,3.]])# three times identity matrixprint('\nVectors & Matrices:')print(torch.linalg.cross(v2,v1))# negative of z unit vector (v1 x v2 == -v2 x v1)print(m1)m3=torch.linalg.matmul(m1,m2)print(m3)# 3 times m1print(torch.linalg.svd(m3))# singular value decomposition

Common functions:tensor([[0.9238, 0.5724, 0.0791, 0.2629],        [0.1986, 0.4439, 0.6434, 0.4776]])tensor([[-0., -0., 1., -0.],        [-0., 1., 1., -0.]])tensor([[-1., -1.,  0., -1.],        [-1.,  0.,  0., -1.]])tensor([[-0.5000, -0.5000,  0.0791, -0.2629],        [-0.1986,  0.4439,  0.5000, -0.4776]])Sine and arcsine:tensor([0.0000, 0.7854, 1.5708, 2.3562])tensor([0.0000, 0.7071, 1.0000, 0.7071])tensor([0.0000, 0.7854, 1.5708, 0.7854])Bitwise XOR:tensor([3, 2, 1])Broadcasted, element-wise equality comparison:tensor([[ True, False],        [False, False]])Reduction ops:tensor(4.)4.0tensor(2.5000)tensor(1.2910)tensor(24.)tensor([1, 2])Vectors & Matrices:tensor([ 0.,  0., -1.])tensor([[0.7375, 0.8328],        [0.8444, 0.2941]])tensor([[2.2125, 2.4985],        [2.5332, 0.8822]])torch.return_types.linalg_svd(U=tensor([[-0.7889, -0.6145],        [-0.6145,  0.7889]]),S=tensor([4.1498, 1.0548]),Vh=tensor([[-0.7957, -0.6056],        [ 0.6056, -0.7957]]))

This is a small sample of operations. For more details and the full inventory ofmath functions, have a look at thedocumentation.For more details and the full inventory of linear algebra operations, have alook at thisdocumentation.

Altering Tensors in Place#

Most binary operations on tensors will return a third, new tensor. Whenwe sayc=a*b (wherea andb are tensors), the new tensorc will occupy a region of memory distinct from the other tensors.

There are times, though, that you may wish to alter a tensor in place -for example, if you’re doing an element-wise computation where you candiscard intermediate values. For this, most of the math functions have aversion with an appended underscore (_) that will alter a tensor inplace.

For example:

a=torch.tensor([0,math.pi/4,math.pi/2,3*math.pi/4])print('a:')print(a)print(torch.sin(a))# this operation creates a new tensor in memoryprint(a)# a has not changedb=torch.tensor([0,math.pi/4,math.pi/2,3*math.pi/4])print('\nb:')print(b)print(torch.sin_(b))# note the underscoreprint(b)# b has changed

a:tensor([0.0000, 0.7854, 1.5708, 2.3562])tensor([0.0000, 0.7071, 1.0000, 0.7071])tensor([0.0000, 0.7854, 1.5708, 2.3562])b:tensor([0.0000, 0.7854, 1.5708, 2.3562])tensor([0.0000, 0.7071, 1.0000, 0.7071])tensor([0.0000, 0.7071, 1.0000, 0.7071])

For arithmetic operations, there are functions that behave similarly:

a=torch.ones(2,2)b=torch.rand(2,2)print('Before:')print(a)print(b)print('\nAfter adding:')print(a.add_(b))print(a)print(b)print('\nAfter multiplying')print(b.mul_(b))print(b)

Before:tensor([[1., 1.],        [1., 1.]])tensor([[0.3788, 0.4567],        [0.0649, 0.6677]])After adding:tensor([[1.3788, 1.4567],        [1.0649, 1.6677]])tensor([[1.3788, 1.4567],        [1.0649, 1.6677]])tensor([[0.3788, 0.4567],        [0.0649, 0.6677]])After multiplyingtensor([[0.1435, 0.2086],        [0.0042, 0.4459]])tensor([[0.1435, 0.2086],        [0.0042, 0.4459]])

Note that these in-place arithmetic functions are methods on thetorch.Tensor object, not attached to thetorch module like manyother functions (e.g.,torch.sin()). As you can see froma.add_(b),the calling tensor is the one that gets changed inplace.

There is another option for placing the result of a computation in anexisting, allocated tensor. Many of the methods and functions we’ve seenso far - including creation methods! - have anout argument thatlets you specify a tensor to receive the output. If theout tensoris the correct shape anddtype, this can happen without a new memoryallocation:

a=torch.rand(2,2)b=torch.rand(2,2)c=torch.zeros(2,2)old_id=id(c)print(c)d=torch.matmul(a,b,out=c)print(c)# contents of c have changedassertcisd# test c & d are same object, not just containing equal valuesassertid(c)==old_id# make sure that our new c is the same object as the old onetorch.rand(2,2,out=c)# works for creation too!print(c)# c has changed againassertid(c)==old_id# still the same object!

tensor([[0., 0.],        [0., 0.]])tensor([[0.3653, 0.8699],        [0.2364, 0.3604]])tensor([[0.0776, 0.4004],        [0.9877, 0.0352]])

Changing the Number of Dimensions#

One case where you might need to change the number of dimensions ispassing a single instance of input to your model. PyTorch modelsgenerally expectbatches of input.

For example, imagine having a model that works on 3 x 226 x 226 images -a 226-pixel square with 3 color channels. When you load and transformit, you’ll get a tensor of shape(3,226,226). Your model, though,is expecting input of shape(N,3,226,226), whereN is thenumber of images in the batch. So how do you make a batch of one?

a=torch.rand(3,226,226)b=a.unsqueeze(0)print(a.shape)print(b.shape)

torch.Size([3, 226, 226])torch.Size([1, 3, 226, 226])

Theunsqueeze() method adds a dimension of extent 1.unsqueeze(0) adds it as a new zeroth dimension - now you have abatch of one!

So if that’sunsqueezing? What do we mean by squeezing? We’re takingadvantage of the fact that any dimension of extent 1does not changethe number of elements in the tensor.

c=torch.rand(1,1,1,1,1)print(c)

tensor([[[[[0.2347]]]]])

Continuing the example above, let’s say the model’s output is a20-element vector for each input. You would then expect the output tohave shape(N,20), whereN is the number of instances in theinput batch. That means that for our single-input batch, we’ll get anoutput of shape(1,20).

What if you want to do somenon-batched computation with that output -something that’s just expecting a 20-element vector?

a=torch.rand(1,20)print(a.shape)print(a)b=a.squeeze(0)print(b.shape)print(b)c=torch.rand(2,2)print(c.shape)d=c.squeeze(0)print(d.shape)

torch.Size([1, 20])tensor([[0.1899, 0.4067, 0.1519, 0.1506, 0.9585, 0.7756, 0.8973, 0.4929, 0.2367,         0.8194, 0.4509, 0.2690, 0.8381, 0.8207, 0.6818, 0.5057, 0.9335, 0.9769,         0.2792, 0.3277]])torch.Size([20])tensor([0.1899, 0.4067, 0.1519, 0.1506, 0.9585, 0.7756, 0.8973, 0.4929, 0.2367,        0.8194, 0.4509, 0.2690, 0.8381, 0.8207, 0.6818, 0.5057, 0.9335, 0.9769,        0.2792, 0.3277])torch.Size([2, 2])torch.Size([2, 2])

You can see from the shapes that our 2-dimensional tensor is now1-dimensional, and if you look closely at the output of the cell aboveyou’ll see that printinga shows an “extra” set of square brackets[] due to having an extra dimension.

You may onlysqueeze() dimensions of extent 1. See above where wetry to squeeze a dimension of size 2 inc, and get back the sameshape we started with. Calls tosqueeze() andunsqueeze() canonly act on dimensions of extent 1 because to do otherwise would changethe number of elements in the tensor.

Another place you might useunsqueeze() is to ease broadcasting.Recall the example above where we had the following code:

a=torch.ones(4,3,2)c=a*torch.rand(3,1)# 3rd dim = 1, 2nd dim identical to aprint(c)

The net effect of that was to broadcast the operation over dimensions 0and 2, causing the random, 3 x 1 tensor to be multiplied element-wise byevery 3-element column ina.

What if the random vector had just been 3-element vector? We’d lose theability to do the broadcast, because the final dimensions would notmatch up according to the broadcasting rules.unsqueeze() comes tothe rescue:

a=torch.ones(4,3,2)b=torch.rand(3)# trying to multiply a * b will give a runtime errorc=b.unsqueeze(1)# change to a 2-dimensional tensor, adding new dim at the endprint(c.shape)print(a*c)# broadcasting works again!

torch.Size([3, 1])tensor([[[0.1891, 0.1891],         [0.3952, 0.3952],         [0.9176, 0.9176]],        [[0.1891, 0.1891],         [0.3952, 0.3952],         [0.9176, 0.9176]],        [[0.1891, 0.1891],         [0.3952, 0.3952],         [0.9176, 0.9176]],        [[0.1891, 0.1891],         [0.3952, 0.3952],         [0.9176, 0.9176]]])

Thesqueeze() andunsqueeze() methods also have in-placeversions,squeeze_() andunsqueeze_():

batch_me=torch.rand(3,226,226)print(batch_me.shape)batch_me.unsqueeze_(0)print(batch_me.shape)

torch.Size([3, 226, 226])torch.Size([1, 3, 226, 226])

Sometimes you’ll want to change the shape of a tensor more radically,while still preserving the number of elements and their contents. Onecase where this happens is at the interface between a convolutionallayer of a model and a linear layer of the model - this is common inimage classification models. A convolution kernel will yield an outputtensor of shapefeatures x width x height, but the following linearlayer expects a 1-dimensional input.reshape() will do this for you,provided that the dimensions you request yield the same number ofelements as the input tensor has:

output3d=torch.rand(6,20,20)print(output3d.shape)input1d=output3d.reshape(6*20*20)print(input1d.shape)# can also call it as a method on the torch module:print(torch.reshape(output3d,(6*20*20,)).shape)

torch.Size([6, 20, 20])torch.Size([2400])torch.Size([2400])

When it can,reshape() will return aview on the tensor to bechanged - that is, a separate tensor object looking at the sameunderlying region of memory.This is important: That means any changemade to the source tensor will be reflected in the view on that tensor,unless youclone() it.

Thereare conditions, beyond the scope of this introduction, wherereshape() has to return a tensor carrying a copy of the data. Formore information, see thedocs.