Tutorial 8 - PyTorch¶
PyTorch is an open-source machine learning library developed by Meta’s AI research lab, and launched in 2016. Designed to facilitate deep learning, PyTorch offers tools for building neural networks, performing automatic differentiation, and managing large-scale computations. It is based on Python and NumPy-like tensors, enabling seamless execution on both CPUs and GPUs for efficient processing.
In addition to its research-oriented features, PyTorch also provides high-level APIs such as TorchVision for computer vision and TorchText for natural language processing. Over time, PyTorch has grown into a framework widely used in both research and production environments.
Install PyTorch¶
PyTorch can be installed with pip or conda if you are using Anaconda.
pip install torch torchvision
conda install torch torchvision
Note that Google Colab comes with PyTorch preinstalled.
[1]:
import numpy as np
import torch
[2]:
# Set seed for reproducibility
np.random.seed(seed=1234)
torch.manual_seed(1234)
[2]:
<torch._C.Generator at 0x296c426a350>
Basics¶
[3]:
# Creating a random tensor
x = torch.randn((2, 3)) # (rand(2,3) -> normal distribution)
print(f"Type: {x.type()}")
print(f"Size: {x.shape}")
print(f"Values: \n{x}")
Type: torch.FloatTensor
Size: torch.Size([2, 3])
Values:
tensor([[ 0.0461, 0.4024, -1.0115],
[ 0.2167, -0.6123, 0.5036]])
[4]:
# Zeros and ones tensor
x = torch.zeros(2, 3)
print (x)
x = torch.ones(2, 3)
print (x)
tensor([[0., 0., 0.],
[0., 0., 0.]])
tensor([[1., 1., 1.],
[1., 1., 1.]])
[5]:
# List to Tensor
x = torch.Tensor([[1, 2, 3],[4, 5, 6]])
print(f"Size: {x.shape}")
print(f"Values: \n{x}")
Size: torch.Size([2, 3])
Values:
tensor([[1., 2., 3.],
[4., 5., 6.]])
[6]:
# NumPy array to Tensor
x = torch.Tensor(np.random.rand(2, 3))
print(f"Size: {x.shape}")
print(f"Values: \n{x}")
Size: torch.Size([2, 3])
Values:
tensor([[0.1915, 0.6221, 0.4377],
[0.7854, 0.7800, 0.2726]])
[7]:
# Changing tensor type
x = torch.Tensor(3, 4)
print(f"Type: {x.type()}")
# Convert to double precision (float64)
x_double = x.to(torch.float64)
print(f"Type: {x_double.type()}")
# Convert to int32
x_int = x.to(torch.int32)
print(f"Type: {x_int.type()}")
Type: torch.FloatTensor
Type: torch.DoubleTensor
Type: torch.IntTensor
Operations¶
[8]:
# Addition
x = torch.Tensor([[2, 2, 2], [2, 2, 2]])
y = torch.Tensor([[3, 3, 3], [3, 3, 3]])
z = x + y
print(f"Size: {z.shape}")
print(f"Values: \n{z}")
Size: torch.Size([2, 3])
Values:
tensor([[5., 5., 5.],
[5., 5., 5.]])
[9]:
# Matrix multiplication
x = torch.Tensor([[2, 2, 2], [2, 2, 2]])
y = torch.Tensor([[3, 3], [3, 3], [3, 3]])
z = torch.mm(x, y)
print(f"Size: {z.shape}")
print(f"Values: \n{z}")
Size: torch.Size([2, 2])
Values:
tensor([[18., 18.],
[18., 18.]])
[10]:
# Transpose
x = torch.Tensor([[2, 2, 2], [2, 2, 2]])
print(f"Size: {x.shape}")
print(f"Values: \n{x}")
y = torch.t(x)
print(f"Size: {y.shape}")
print(f"Values: \n{y}")
Size: torch.Size([2, 3])
Values:
tensor([[2., 2., 2.],
[2., 2., 2.]])
Size: torch.Size([3, 2])
Values:
tensor([[2., 2.],
[2., 2.],
[2., 2.]])
[11]:
# Reshape
x = torch.Tensor([[2, 2, 2], [2, 2, 2]])
z = x.view(3, 2)
print(f"Size: {z.shape}")
print(f"Values: \n{z}")
Size: torch.Size([3, 2])
Values:
tensor([[2., 2.],
[2., 2.],
[2., 2.]])
[12]:
# Dimensional operations
x = torch.Tensor([[2, 2, 2], [2, 2, 2]])
print(f"Values: \n{x}")
y = torch.sum(x, dim=0) # sum over columns
print(f"Values: \n{y}")
z = torch.sum(x, dim=1) # sum over rows
print(f"Values: \n{z}")
Values:
tensor([[2., 2., 2.],
[2., 2., 2.]])
Values:
tensor([4., 4., 4.])
Values:
tensor([6., 6.])
Indexing, Slicing, and Joining¶
[13]:
x = torch.Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 10], [11, 12, 13]])
print (f"x: \n{x}")
# first row
print (f"x[0]: {x[0]}")
# first and second row, second and third columns
print (f"x[:2, 1:3]: \n{x[:2, 1:3]}")
x:
tensor([[ 1., 2., 3.],
[ 4., 5., 6.],
[ 7., 8., 10.],
[11., 12., 13.]])
x[0]: tensor([1., 2., 3.])
x[:2, 1:3]:
tensor([[2., 3.],
[5., 6.]])
[14]:
# Concatenation
x = torch.Tensor([[1, 2, 3], [4, 5, 6]])
print(f"Values: \n{x}")
y = torch.cat([x, x], dim=0) # stack by rows (dim=1 to stack by columns)
print(f"Values: \n{y}")
Values:
tensor([[1., 2., 3.],
[4., 5., 6.]])
Values:
tensor([[1., 2., 3.],
[4., 5., 6.],
[1., 2., 3.],
[4., 5., 6.]])
Gradients¶
\(x = 3\)
\(y = x^2 + 2x + 2\)
\(\frac{dy}{dx} = 2x + 2 = 2*3 + 2 = 8\)
[15]:
# Create a single-element tensor with gradient tracking
x = torch.tensor([3.0], requires_grad=True)
# Define a simple quadratic function
y = x**2 + 2*x + 1
# sum() is required to make y a scalar
y = y.sum()
# Backpropagate to calculate dy/dx = 2x+2
y.backward()
# Evaluate the derivative dy/dx for x=3, i.e., dy/dx = 2*3+2 = 8
dy_dx = x.grad
print(f"x: {x}")
print(f"Gradient: {dy_dx}")
x: tensor([3.], requires_grad=True)
Gradient: tensor([8.])
CUDA Tensors¶
[16]:
# Is CUDA available?
print (torch.cuda.is_available())
False
If False, CUDA is not available. In Google Colab, we can change it by Runtime > Change runtime type > Change Hardware accelerator to GPU.
[17]:
# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print (device)
cpu
[18]:
# Tensor is stored on the device
x = torch.rand(2,3).to(device)
print (x.is_cuda)
False