Tensor is the most basic data abstraction in pytoch. It is similar to the array in C or numpy and can have multiple dimensions. Tensors can also be used on GPU s to improve performance.
- Common methods of creating tensors are:
- torch.empty(): creates an uninitialized tensor.
>>> a = torch.empty(2,3)
>>> a
tensor([[1.0286e-38, 9.0919e-39, 8.9082e-39],
[9.2755e-39, 8.4490e-39, 9.6429e-39]])
>>> b = torch.empty(2,3)
>>> b
tensor([[1.0286e-38, 9.0919e-39, 8.9082e-39],
[9.2755e-39, 8.4490e-39, 9.6429e-39]])
>>> a[0][0] = 55. # You can assign values to elements
>>> a
tensor([[5.5000e+01, 9.0919e-39, 8.9082e-39],
[9.2755e-39, 8.4490e-39, 9.6429e-39]])
>>> a[1] = 22. # You can assign values uniformly according to a certain dimension
>>> a
tensor([[5.5000e+01, 9.0919e-39, 8.9082e-39],
[2.2000e+01, 2.2000e+01, 2.2000e+01]])
>>> a[:,1] = 44. # You can assign values uniformly according to a certain dimension
>>> a
tensor([[5.5000e+01, 4.4000e+01, 8.9082e-39],
[2.2000e+01, 4.4000e+01, 2.2000e+01]])- torch.zeros(): create tensors whose element values are all zero.
>>> a = torch.zeros((2,3,4))
>>> a
tensor([[[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.]],
[[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.]]])- torch.ones(): create tensors whose element values are all 1.
>>> b = torch.ones((3,4))
>>> b
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]])
>>> a = torch.empty((2,3))
>>> a
tensor([[-3.9696e-19, 4.5914e-41, -3.9696e-19],
[ 4.5914e-41, -3.9696e-19, 4.5914e-41]])
# The out parameter can specify that the generated tensor is output to other variables
# empty(), ones(), zeros(), etc. all have out parameters
>>> torch.ones((2,3), out=a)
tensor([[1., 1., 1.],
[1., 1., 1.]])
>>> a
tensor([[1., 1., 1.],
[1., 1., 1.]])- Linear filling tensor
>>> torch.arange(1., 5.) # It is basically consistent with the usage in numpy tensor([1., 2., 3., 4.]) >>> torch.arange(1., 10., 2) tensor([1., 3., 5., 7., 9.]) >>> torch.linspace(0,1,11) # It is basically consistent with the usage in numpy tensor([0.0000, 0.1000, 0.2000, 0.3000, 0.4000, 0.5000, 0.6000, 0.7000, 0.8000, 0.9000, 1.0000])
- Create a random tensor. It is often used to initialize the weight tensor w in deep learning.
>>> a = torch.rand(2,3) # [0, 1) uniform distribution
>>> a
tensor([[0.4202, 0.3845, 0.9435],
[0.8606, 0.9898, 0.7223]])
>>> b = torch.rand(2,3)
>>> b
tensor([[0.5876, 0.8672, 0.1454],
[0.8283, 0.6713, 0.1481]]
>>> torch.randn((2,6)) #Standard normal distribution (normal distribution with mean = 0 and std = 1)
tensor([[-0.6857, -0.5643, 1.1925, 1.9629, 0.7755, -0.0425],
[ 0.0553, 0.3286, 1.7301, -0.7192, 0.9045, -1.5953]])
>>>torch.randint(-1, 10, (1,20)) # low=-1, high=10, integer uniform distribution
tensor([[4, 3, 3, 3, 6, 4, 4, 3, 1, 0, 1, 2, 1, 8, 3, 8, 8, 8, 1, 6]])Random seeds can be set manually
>>> torch.manual_seed(317)
<torch._C.Generator object at 0x000001935EE0FE28>
>>> a = torch.rand(2,3)
>>> torch.manual_seed(317)
<torch._C.Generator object at 0x000001935EE0FE28>
>>> b = torch.rand(2,3)
>>> a
tensor([[0.9137, 0.1671, 0.9334],
[0.3699, 0.6616, 0.1785]])
>>> b
tensor([[0.9137, 0.1671, 0.9334],
[0.3699, 0.6616, 0.1785]])
>>> a == b
tensor([[True, True, True],
[True, True, True]])- torch.eye() creates a two-dimensional tensor in the form of identity matrix. The main diagonal elements are all 1 and other elements are all 0.
>>> torch.eye(4)
tensor([[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.]])- torch.xxxx_like() creates tensors based on the shape of other tensors.
>>> a = torch.ones((3,2)) # The shape parameter can also be passed into tuple or list
>>> a
tensor([[1., 1.],
[1., 1.],
[1., 1.]])
>>> torch.empty_like(a)
tensor([[1., 1.],
[1., 1.],
[1., 1.]])
>>> torch.zeros_like(a)
tensor([[0., 0.],
[0., 0.],
[0., 0.]])
>>> torch.ones_like(a)
tensor([[1., 1.],
[1., 1.],
[1., 1.]])- pytorch.tensor() manually creates a tensor for the given element
>>> torch.tensor(1) # Zero dimensional tensor (scalar)
tensor(1)
>>> torch.tensor((1,2)) # One dimensional tensor
tensor([1, 2])
>>> torch.tensor([[1,2],[3,4]]) # Two dimensional tensor
tensor([[1, 2],
[3, 4]])
>>> torch.tensor(((1,2),[3,4])) # 2-dimensional tensor, tuple and list can be mixed
tensor([[1, 2],
[3, 4]])- The data types of tensor include torch.bool torch.int8 torch.uint8 torch.int16 torch.int32 torch.int64 torch.half torch.float torch.double torch.bfloat
# You can specify the data type through the dtype parameter at creation time a = torch.ones((2, 3), dtype=torch.int16) print(a) b = torch.rand((2, 3), dtype=torch.float64) * 20. print(b) # It can also be converted c = b.to(torch.int32)
- Dimension information of tensor (size)
>>> x = torch.ones((1,2,3))
>>> x
tensor([[[1., 1., 1.],
[1., 1., 1.]]])
>>> x.shape
torch.Size([1, 2, 3])
>>> x.size()
torch.Size([1, 2, 3])
>>> x.shape == x.size()
True
>>> x.shape is x.size()
False>>> x.shape[0] 1 >>> x.shape[1] 2
>>> x = torch.tensor(5.) >>> x tensor(5.) >>> x.shape torch.Size([]) # shape of zero dimensional tensor
- reshape of tensor. Like numpy, you can change the shape of the tensor. The new tensor is just the view of the original tensor, and the two share storage space.
>>> a = torch.rand((3,4))
>>> b = a.reshape((4,3))
>>> b[0,0]=22.22
>>> a
tensor([[22.2200, 0.9620, 0.7581, 0.1273],
[ 0.5137, 0.9653, 0.1091, 0.3371],
[ 0.0306, 0.7359, 0.7835, 0.9451]])
# You can also use view ()
>>> c = a.view(2,-1) # Like numpy, the last dimension can be set to - 1 for automatic allocation.
>>> c
tensor([[22.2200, 0.9620, 0.7581, 0.1273, 0.5137, 0.9653],
[ 0.1091, 0.3371, 0.0306, 0.7359, 0.7835, 0.9451]])
>>> c[0,1]=33.33
>>> a
tensor([[2.2220e+01, 3.3330e+01, 7.5810e-01, 1.2730e-01],
[5.1369e-01, 9.6526e-01, 1.0907e-01, 3.3712e-01],
[3.0613e-02, 7.3591e-01, 7.8346e-01, 9.4511e-01]])Zero dimensional tensor to python value
- The slicing operation of tensor is the same as numpy.
>>> x
tensor([[8, 4, 4, 0, 3],
[5, 4, 9, 3, 2],
[4, 6, 0, 2, 2],
[3, 8, 3, 9, 9],
[3, 6, 7, 9, 3]])
>>> x[1:3,2:-1]
tensor([[9, 3],
[0, 2]])- Splicing of tensors
>>> a = torch.randn(3,2)
>>> b = torch.randn(3,1)
>>> torch.cat((a,b),dim=1)
tensor([[-0.4081, -0.1116, 0.3356],
[-0.6456, 0.0837, 0.4141],
[ 0.7786, 1.0901, -0.6719]])tensor and numpy arrays are converted to each other
>>> t2 = torch.from_numpy(n) >>> t2 tensor([99., 1., 1., 1., 1.]) >>> n[2]=22.22 >>> n array([99. , 1. , 22.22, 1. , 1. ], dtype=float32) >>> t2 # tensor and array share storage space tensor([99.0000, 1.0000, 22.2200, 1.0000, 1.0000])
Mathematical operation of tensor.
- Mathematical operations can be carried out between tensors with the same size. Unless broadcast is available, mathematical calculations cannot be performed between tensors of different sizes.
>>> a = torch.rand((2,3))
>>> a
tensor([[0.7301, 0.3747, 0.4656],
[0.9889, 0.8834, 0.7723]])
>>> b = torch.ones_like(a)
>>> b
tensor([[1., 1., 1.],
[1., 1., 1.]])
>>> a+b
tensor([[1.7301, 1.3747, 1.4656],
[1.9889, 1.8834, 1.7723]])
>>> a*b # Equivalent to torch mul(a,b)
tensor([[0.7301, 0.3747, 0.4656],
[0.9889, 0.8834, 0.7723]])
>>> a *= b
>>> a
tensor([[0.7301, 0.3747, 0.4656],
[0.9889, 0.8834, 0.7723]])
>>> a -= b
>>> a
tensor([[-0.2699, -0.6253, -0.5344],
[-0.0111, -0.1166, -0.2277]])- Like numpy, torch also comes with many mathematical functions, such as
>>> torch.sin(a)
tensor([[-0.2667, -0.5853, -0.5093],
[-0.0111, -0.1163, -0.2257]])
>>> torch.cos(a)
tensor([[0.9638, 0.8108, 0.8606],
[0.9999, 0.9932, 0.9742]])
>>> torch.asin(a)
tensor([[-0.2733, -0.6755, -0.5637],
[-0.0111, -0.1168, -0.2297]])
>>> torch.sinh(a)
tensor([[-0.2732, -0.6669, -0.5601],
[-0.0111, -0.1168, -0.2297]])
>>> torch.max(a)
tensor(-0.0111)
>>> a.mean()
tensor(-0.2975)
>>> a.mean(dim=1) # Dimensions can be specified
tensor([-0.4765, -0.1185])
>>> b
tensor([[3., 3., 3.],
[3., 3., 3.]])
>>> torch.std_mean(b) # Calculate b standard deviation and mean
(tensor(0.), tensor(3.))
>>> torch.svd(b) # SVD decomposition
torch.return_types.svd(
U=tensor([[-0.7071, 0.7071],
[-0.7071, -0.7071]]),
S=tensor([7.3485e+00, 4.0850e-07]),
V=tensor([[-0.5774, -0.8165],
[-0.5774, 0.4082],
[-0.5774, 0.4082]]))
>>> a = torch.randn(4,4)
>>> torch.det(a) # Calculating determinant of square matrix
tensor(0.6460)
# Wait, wait- Matrix multiplication, torch matmul()
>>> a = torch.randn(3,2)
>>> b=torch.randn(2,5)
>>> torch.matmul(a,b)
tensor([[-3.4743, -0.5266, 0.3270, -2.7448, -0.7288],
[-5.9377, -1.4233, 0.9677, -2.2013, -3.4487],
[-2.5309, -0.8615, 0.6115, 0.2738, -2.5426]])- tensor's broadcast is similar to numpy's broadcast
a
tensor([[1., 1., 1.],
[1., 1., 1.]])
b = a+2
b
tensor([[3., 3., 3.],
[3., 3., 3.]])
b**2
tensor([[9., 9., 9.],
[9., 9., 9.]])
(torch.rand(2, 2) - 0.5) * 2
tensor([[-0.1330, -0.1400],
[-0.9692, -0.5754]])- The rules for broadcasting between tensors of different shape s are as follows:
- There can be no empty tensor
- Compare the dimensions of two tensors from back to front (aligned from right to left), Each dimension is equal in size, or The size of one of the dimensions must be 1, or A dimension of one of the tensor s is missing.
a = torch.ones(4, 3, 2) b = a * torch.rand( 3, 2) # 3rd & 2nd dims identical to a, dim 1 absent print(b) c = a * torch.rand( 3, 1) # 3rd dim = 1, 2nd dim identical to a print(c) d = a * torch.rand( 1, 2) # 3rd dim identical to a, 2nd dim = 1 print(d)
- tensor can be transferred to GPU for calculation
b = a.to("cuda")
b
tensor([[ 1.2982, -0.1432, -0.9152, 0.8028],
[-0.3701, 0.2217, 0.7523, 0.7443],
[-0.1457, 0.7082, -1.1346, 0.8564],
[-0.2956, 0.1779, 0.9134, -0.0069]], device='cuda:0')
b**2
tensor([[1.6853e+00, 2.0511e-02, 8.3754e-01, 6.4442e-01],
[1.3698e-01, 4.9144e-02, 5.6598e-01, 5.5392e-01],
[2.1235e-02, 5.0161e-01, 1.2873e+00, 7.3339e-01],
[8.7371e-02, 3.1664e-02, 8.3422e-01, 4.8006e-05]], device='cuda:0')The tensor on the GPU can be switched back to the CPU
c = b.to("cpu")
c
tensor([[ 1.2982, -0.1432, -0.9152, 0.8028],
[-0.3701, 0.2217, 0.7523, 0.7443],
[-0.1457, 0.7082, -1.1346, 0.8564],
[-0.2956, 0.1779, 0.9134, -0.0069]])
c.device
device(type='cpu')tensors on different devices cannot be calculated.
a+b
Traceback (most recent call last):
File "E:\Python36\lib\code.py", line 91, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!