CodeBlanks Logo Icon CodeBlanks
Login Register

Dtype

Dtype specifies the data type of tensor elements.

Dtype

The element type determines memory, speed and accuracy:

  • float32: the default, standard for training.
  • float16 / bfloat16: half precision; twice the speed and half the memory on the GPU (mixed-precision training).
  • int64 (long): indices and class labels.
  • bool: masks.
x = torch.tensor([1.0, 2.0])
print(x.element_size())             # 4 bytes
print(x.half().element_size())      # 2 bytes
print((x > 1).dtype)                # torch.bool

Precision and overflow

Low-precision types lose small differences and overflow on large values. The largest value of float16 is 65504:

print(torch.tensor(70000.0).half().item())        # inf: overflow
print(torch.tensor(1.0001).half().item())         # 1.0: the detail is lost
print(torch.tensor(70000.0).bfloat16().item())    # 70144.0: bfloat16 has a wide range

bfloat16 has the same exponent range as float32, so it doesn't overflow but keeps fewer decimal digits; it is often used when training large models.

Practice Memory calculation

Print the size in bytes of x = torch.zeros(1000, 1000) (x.element_size() * x.numel()). Do the same calculation for x.half(). Then print torch.tensor(70000.0).half().item().

Starter code

model.py
import torch

x = torch.zeros(1000, 1000)

# buraya yaz

Sample solution

model.py
import torch

x = torch.zeros(1000, 1000)

print(x.element_size() * x.numel())
h = x.half()
print(h.element_size() * h.numel())
print(torch.tensor(70000.0).half().item())

Expected output

4000000
2000000
inf

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 Roughly how much memory does a 1-billion-parameter model take in float32?

  • 8 GB
  • 4 GB Correct answer
  • 1 GB
  • 32 GB

Explanation: 4 bytes per parameter: 10⁹ × 4 = 4·10⁹ bytes ≈ 4 GB (2 GB in float16).

2 Convert the tensor to half precision.

Question 
y = x.____()
Answer 
y = x.half()

Explanation: .half() produces float16.

3 Match the types.

Term Meaning
float32 4 bytes, default
float16 2 bytes, narrow range
bfloat16 2 bytes, wide range
bool Mask

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4