PyTorch Frontend

Convert PyTorch models to FPGA firmware with hls4ml

The PyTorch frontend enables conversion of PyTorch models to optimized FPGA firmware using FX graph tracing for comprehensive model analysis.

Conversion Function

convert_from_pytorch_model()

The primary function for converting PyTorch models to hls4ml.

python

import torch
import torch.nn as nn
import hls4ml

# Define your PyTorch model
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 64)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(64, 3)
    
    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

model = MyModel()
model.eval()  # Set to evaluation mode

# Convert to hls4ml
hls_model = hls4ml.converters.convert_from_pytorch_model(
    model,
    input_shape=(10,),  # Required: specify input shape
    output_dir='my-hls-test',
    project_name='myproject',
    backend='Vivado',
    io_type='io_parallel',
    hls_config={'Model': {'Precision': 'ap_fixed<16,6>', 'ReuseFactor': 1}}
)

Parameters

model · torch.nn.Module · required

PyTorch model instance to convert. Model should be in evaluation mode (.eval()).

output_dir · str · default: "my-hls-test"

Output directory for the generated HLS project.

project_name · str · default: "myproject"

Name of the HLS project.

backend · str · default: "Vivado"

Backend to use for HLS synthesis. Options: ‘Vivado’, ‘Vitis’, ‘Quartus’, ‘oneAPI’.

io_type · str · default: "io_parallel"

I/O implementation type. Options: ‘io_parallel’, ‘io_stream’.

hls_config · dict · required

Configuration dictionary for HLS conversion. Must include:

• InputShape: Tuple specifying input dimensions (without batch dimension)
• Model: Dictionary with Precision and ReuseFactor

part · str · default: "None"

Target FPGA part number (e.g., ‘xcvu9p-flgb2104-2-i’).

clock_period · int · default: "5"

Clock period in nanoseconds.

Complete Example

Simple MLP

import numpy as np
import torch
import torch.nn as nn
import hls4ml

# Define model
class SimpleMLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(16, 32)
        self.relu1 = nn.ReLU()
        self.fc2 = nn.Linear(32, 16)
        self.relu2 = nn.ReLU()
        self.fc3 = nn.Linear(16, 8)
        self.softmax = nn.Softmax(dim=-1)
    
    def forward(self, x):
        x = self.fc1(x)
        x = self.relu1(x)
        x = self.fc2(x)
        x = self.relu2(x)
        x = self.fc3(x)
        x = self.softmax(x)
        return x

# Create and prepare model
model = SimpleMLP()
model.eval()

# Test data
X_test = np.random.rand(1000, 16).astype(np.float32)
y_pytorch = model(torch.Tensor(X_test)).detach().numpy()

# Configure conversion
config = hls4ml.utils.config_from_pytorch_model(
    model,
    input_shape=(16,),
    granularity='name'
)
config['Model']['Precision'] = 'ap_fixed<16,6>'
config['Model']['ReuseFactor'] = 4

# Convert to hls4ml
hls_model = hls4ml.converters.convert_from_pytorch_model(
    model,
    hls_config=config,
    output_dir='pytorch_mlp_hls',
    backend='Vivado'
)

# Compile and test
hls_model.compile()
y_hls = hls_model.predict(X_test)

print(f"Accuracy match: {np.mean(np.argmax(y_pytorch, axis=1) == np.argmax(y_hls, axis=1))}")

Supported Layers

The PyTorch frontend uses FX graph tracing to extract model structure and supports:

Core Layers (torch.nn)

• Linear - Fully connected layers
• ReLU - Rectified Linear Unit
• Sigmoid - Sigmoid activation
• Tanh - Hyperbolic tangent
• LeakyReLU - Leaky ReLU with negative slope
• ELU - Exponential Linear Unit
• PReLU - Parametric ReLU
• Threshold - Thresholded activation
• Softmax - Softmax activation

Convolutional Layers

• Conv1d - 1D convolution
• Conv2d - 2D convolution
• BatchNorm1d - 1D batch normalization
• BatchNorm2d - 2D batch normalization

Pooling Layers

• MaxPool1d - 1D max pooling
• MaxPool2d - 2D max pooling
• AvgPool1d - 1D average pooling
• AvgPool2d - 2D average pooling
• AdaptiveAvgPool1d - Adaptive average pooling
• AdaptiveAvgPool2d - Adaptive average pooling

Recurrent Layers

• RNN - Simple RNN
• LSTM - Long Short-Term Memory
• GRU - Gated Recurrent Unit

Merge/Arithmetic Operations

• Add (torch.add, +) - Element-wise addition
• Sub (torch.sub, -) - Element-wise subtraction
• Mul (torch.mul, *) - Element-wise multiplication
• Concat (torch.cat) - Tensor concatenation
• MatMul (torch.matmul) - Matrix multiplication

Reshape Operations

• Flatten - Flatten input
• View - Reshape tensor
• Transpose - Transpose dimensions
• Permute - Permute dimensions

Special Operations

• Dropout - Training-only layer (skipped)
• Sequential - Container (transparent)
• Constant - Constant tensors

Functional Operations (torch.nn.functional)

Many operations from torch.nn.functional are supported:

• F.relu, F.sigmoid, F.tanh
• F.max_pool1d, F.max_pool2d
• F.avg_pool1d, F.avg_pool2d
• F.softmax

Framework-Specific Configuration

Input Shape Requirement

PyTorch conversion requires specifying the input shape explicitly (without batch dimension).

python

# Correct - without batch dimension
config = {'InputShape': (10,)}  # For 1D input
config = {'InputShape': (3, 32, 32)}  # For images (C, H, W)

# For multiple inputs
config = {'InputShape': [(10,), (5,)]}  # Two inputs

Channels First Format

PyTorch uses channels_first format by default (N, C, H, W), while hls4ml expects channels_last (N, H, W, C).

python

# hls4ml automatically handles conversion for io_parallel
hls_model = hls4ml.converters.convert_from_pytorch_model(
    model,
    hls_config=config,
    io_type='io_parallel'  # Automatic transpose layers added
)

# For io_stream, you may need to transpose manually
# Not all transpose operations are supported with io_stream

Model Evaluation Mode

Always set your model to evaluation mode before conversion to disable dropout and use fixed batch norm statistics.

python

model = MyModel()
model.eval()  # Critical!

# Or use torch.no_grad() context
with torch.no_grad():
    hls_model = hls4ml.converters.convert_from_pytorch_model(
        model,
        hls_config=config
    )

Layer Configuration by Name

python

# Get layer names from PyTorch model
for name, module in model.named_modules():
    print(f"{name}: {module.__class__.__name__}")

# Configure specific layers
config = hls4ml.utils.config_from_pytorch_model(model, input_shape=(10,))
config['LayerName']['fc1'] = {
    'Precision': 'ap_fixed<8,3>',
    'ReuseFactor': 16
}

Troubleshooting

Must specify input shape error

PyTorch conversion always requires explicit input shape:

python

# Error: Missing InputShape
hls_config = {'Model': {'Precision': 'ap_fixed<16,6>'}}

# Fix: Add InputShape
hls_config = {
    'InputShape': (10,),  # Required!
    'Model': {'Precision': 'ap_fixed<16,6>'}
}

hls_model = hls4ml.converters.convert_from_pytorch_model(
    model,
    hls_config=hls_config
)

Unsupported layer or operation

Common unsupported operations and solutions:

python

# Issue: F.linear is not supported
# Solution: Use nn.Linear instead
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(10, 5))
    
    def forward(self, x):
        # Don't use: return F.linear(x, self.weight)
        # Use instead:
        self.fc = nn.Linear(5, 10)
        return self.fc(x)

# Issue: torch.nn.functional.conv2d not supported
# Solution: Use nn.Conv2d module

Use torch.nn modules instead of functional operations where possible.

Graph tracing issues

Some dynamic operations may cause tracing issues:

python

# Problematic: Dynamic control flow
class BadModel(nn.Module):
    def forward(self, x):
        if x.sum() > 0:  # Dynamic condition
            return self.fc1(x)
        return self.fc2(x)

# Solution: Use static control flow or torch.jit.script
class GoodModel(nn.Module):
    def forward(self, x):
        # Static operations only
        return self.fc1(x) + self.fc2(x)

# Or: Pre-trace with representative input
import torch.fx
tracer = torch.fx.Tracer()
traced = tracer.trace(model)

Channels format issues

For io_stream with channels_first:

python

# Issue: Automatic transpose not supported for io_stream
hls_model = hls4ml.converters.convert_from_pytorch_model(
    model,
    hls_config=config,
    io_type='io_stream'  # May fail with conv layers
)

# Solution 1: Use io_parallel
io_type='io_parallel'

# Solution 2: Manually transpose in model
class ChannelsLastModel(nn.Module):
    def forward(self, x):
        x = x.permute(0, 2, 3, 1)  # NCHW -> NHWC
        # ... rest of forward pass
        return x

Precision mismatch

Improve PyTorch-to-HLS accuracy:

python

# 1. Use higher precision
config['Model']['Precision'] = 'ap_fixed<32,16>'

# 2. Configure specific layers
config['LayerName']['fc1'] = {'Precision': 'ap_fixed<24,12>'}

# 3. Check input data type matches
X_test = X_test.astype(np.float32)  # Match PyTorch default

# 4. Use same random seed
torch.manual_seed(42)
np.random.seed(42)

Convolution groups not supported

python

# Issue: Grouped convolutions (groups > 1)
self.conv = nn.Conv2d(32, 64, 3, groups=2)  # Not supported

# Solution: Use groups=1 (default)
self.conv = nn.Conv2d(32, 64, 3, groups=1)

# For depthwise separable, use separate layers
self.depthwise = nn.Conv2d(32, 32, 3, groups=32)  # Depthwise
self.pointwise = nn.Conv2d(32, 64, 1)  # Pointwise

Advanced Usage

Using FX Graph Tracing Directly

python

import torch
from torch.fx import symbolic_trace
from hls4ml.converters.pytorch_to_hls import parse_pytorch_model

# Trace model
model.eval()
traced = symbolic_trace(model)

# Inspect graph
for node in traced.graph.nodes:
    print(f"{node.name}: {node.op} {node.target}")

# Convert traced model
config = {
    'PytorchModel': model,
    'InputShape': (10,),
    'HLSConfig': {'Model': {'Precision': 'ap_fixed<16,6>'}}
}

layer_list, inputs, outputs = parse_pytorch_model(config, verbose=True)

Custom Layer Handlers

python

from hls4ml.converters import register_pytorch_layer_handler
from hls4ml.converters.pytorch_to_hls import pytorch_handler

@pytorch_handler('MyCustomLayer')
def parse_custom_layer(operation, layer_name, input_names, input_shapes, 
                       node, class_object, data_reader, config):
    layer = {}
    layer['class_name'] = 'CustomLayer'
    layer['name'] = layer_name
    layer['inputs'] = input_names
    
    # Extract parameters from class_object
    if class_object is not None:
        layer['custom_param'] = class_object.custom_param
    
    # Calculate output shape
    output_shape = input_shapes[0]
    
    return layer, output_shape

Handling State Dictionaries

python

# Load model with state dict
class MyModel(nn.Module):
    # ... model definition

model = MyModel()
state_dict = torch.load('model_weights.pth')
model.load_state_dict(state_dict)
model.eval()

# Convert
hls_model = hls4ml.converters.convert_from_pytorch_model(
    model,
    hls_config=config
)

PyTorch vs Keras Differences

Aspect	PyTorch	Keras
Input shape	Required in config	Optional (read from model)
Data format	channels_first (NCHW)	channels_last (NHWC)
Model mode	Must call .eval()	N/A
Tracing	Uses FX graph tracing	Direct layer iteration
Functional ops	Limited support	N/A
Dynamic graphs	Not supported	N/A

Source Code Reference

The PyTorch converter implementation can be found at:

• hls4ml/converters/pytorch_to_hls.py:440 - Main conversion function
• hls4ml/converters/pytorch/ - Layer-specific handlers
• hls4ml/converters/__init__.py:251 - API entry point
• hls4ml/utils/torch.py - Custom FX tracer

Next Steps

Keras Frontend

Compare with Keras conversion

Configuration

Learn about configuration options

Optimization

Optimize PyTorch models for FPGA

Backends

Explore FPGA backend options