ethernet - Jiri Brejcha

Full 5 Gigabit Ethernet on Raspberry Pi 5 with iocrest Realtek RTL8126 adapter

I’ve tested a number of 10 Gigabit Ethernet adapters on Raspberry Pi 5 based on the AQC107 chip. One adapter that negotiates PCIe Gen 3, achieves 5.5 Gbps speed and overheats. Another one which only works in PCIe Gen 2 mode and peaks at 3.44 Gbps. And even a full-size PCIe card made by TP-Link which negotiates PCIe Gen 2 link speed and doesn’t go beyond 3.44 Gbps either.

The Realtek RTL8126 chip we are testing today is so far the most suitable for Raspberry Pi 5. It is capable of 5 Gigabit Ethernet at full speed. TCP iperf3 throughput peaks at 4.7 Gbps. It doesn’t overheat. And it doesn’t excessively utilise the Raspberry Pi 5 CPU.

This particular one is sold under the iocrest brand. Like the other boards and adapters there is no increst branding on it and it will likely be sold under various brands. The RTL8126 chip is the key component here.

Raspberry Pi 5 with 5 Gigabit Ethernet network adapter

How did we connect it to the Pi? Via PCIe bus. We breakout the Raspberry Pi 5’s PCIe connector via Pineboards (aka Pineberry Pi) board to M.2 M-key slot. And in that slot we install the iocrest 5 Gigabit Ethernet network adapter – that’s the black M.2 module, plus a PCB with RJ-45 connector on a grey ribbon cable.

iocrest 5 GbE adapter connected to Raspberry Pi 5 via PCIe Gen 3 link

Here is how it looks from PCI device perspective.

Performance

It has no problem negotiating full duplex 5 Gigabit Ethernet and filling the interface with traffic fully.

iperf3 with default TCP settings peaks at 4.7 Gbps up and down. More parallel streams don’t improve the result any further. This is in PCIe Gen 3 mode.

Full 5 Gigabit Ethernet throughput in PCIe Gen 3 mode

Just for the record, if we downgrade PCIe bus to Gen 2 link speeds, we are talking 3.43 Gbps down and 3.31 Gbps up iperf3 TCP throughput-wise.

Throughput in downgraded PCIe bus to Gen 2 mode

Thermal footprint

Fully loaded by TCP traffic, I see temperature of 81.2° C (178° F) on the top surface of the RTL8126 chip. Yes, it is on the warmer side, but Raspberry Pi 5 SoC runs quite warm too and it is nowhere near 122° C temperatures I observed on this “hot” 10 Gigabit Ethernet adapter.

Chip temperature, installed in Intel NUC with M.2 slot

By the looks of it, there is no temperature sensor on the PHY so I can’t measure internal temperature.

CPU utilization and temperature of fully loaded adapter with TCP traffic

Linux software support

I happened to have Raspberry Pi OS with 6.8.0-rc7 kernel running on the Raspberry Pi 5. Out of the box, the adapter did not work. iocrest included driver download link pointing to this Chinese website but I am not so sure I want to use that one.

After installing driver from Realtek’s website, the adapter works just fine.

Power draw

This adapter in PCIe Gen 3 mode draws about 1.5 W in idle, and 2.1 W under full iperf3 load.

Switching the adapter to Gen 2 mode doesn’t make any power savings. I measured 0.1 W less in Gen 2 mode.

The whole setup of Raspberry Pi 5 with fan, Pineboards PCIe adapter, and this 5 GbE adapter in PCIe Gen 3 mode draws about 5.1 Watts in total under full iperf3 load.

Does it work on Windows 11?

Yes, it does. I installed one in Intel NUC 12th generation. It runs at full speed full and Gen 3 x1 mode.

Windows 11 driver (as of May 2024) downloaded automatically via Windows Update only allows this adapter to use 2.5 GbE. To unlock 5 GbE we download driver directly from Realtek’s website and we are all set.

Driver from Realtek’s website with full 5 GbE support

5 GbE full duplex with driver from Realtek’s website

With the adapter inserted in M.2 M-key slot, we won’t be able pop the NUC bottom lid back on. The adapter is just a bit too tall.

Bottom lid won’t fit with the adapter installed

Throughput also looks good. I might revisit Windows throughput testing tools at some point. But for now, I take 4.74 Gbps down and 4.42 Gbps up speeds. Increasing number of parallel streams did not improve throughput in any way.

For the record, Jumbo frames seem to be supported but I had no reason to explore this further this time.

Summary

As I mentioned towards the beginning, 5 Gigabit Ethernet based on Realtek RTL8126 chip seems to strike the perfect balance for Raspberry Pi 5. It delivers 4.7 Gbps up and down, doesn’t consume much power, and doesn’t produce excessive amount of heat.

Long-time test will tell how it actually performs but for now I am happy with what I’ve seen.

From driver perspective, I am wondering if the latest Linux kernel supports this chip natively or if I can enable the right kernel module manually.

10 Gigabit Ethernet on Raspberry Pi 5

Raspberry Pi 5 comes with PCI Express connection and a number of HATs (hardware attached on top) and Bottoms (the opposite of that) are now available for sale. That unlocks some very exciting options. Let’s see how fast can a 10 Gigabit Ethernet adapter on Raspberry Pi 5 go, shall we?

Pineberry’s HatDrive! Bottom proved to be really handy for converting Pi’s PCIe connection to M.2 M-key format. My Kalea-Informatique 10 Gigabit adapter uses exactly that, so that’s a match. Why did I choose this adapter? Very unscientifically this time – it was the first readily available and I was in a fail-fast mood :)

10 GbE adapter connected to Raspberry Pi 5

Pineberry HatDrive! Bottom board with 10 GbE network adapter

Detail of the AQC107 chip powering the network adapter

Enable PCIe port on Raspberry Pi 5

First things first. We need to enable the PCIe connector on the Pi.

sudo nano /boot/firmware/config.txt

# Enable the port
dtparam=pciex1

# Configure PCIe Gen
dtparam=pciex1_gen=2

Build custom Linux kernel and include the Aquantia driver module

Vanilla Raspberry Pi OS doesn’t include the Aquantia AQC107 kernel module. So we need to burn a micro SD card with a vanilla Raspberry Pi OS Bookworm image, boot the Pi 5 and build a customised kernel.

git clone --depth=1 --branch rpi-6.8.y https://github.com/raspberrypi/linux
cd linux/
sudo apt install flex bison aptitude -y
sudo aptitude install libssl-dev
make bcm2712_defconfig

Edit the config file:

sudo nano .config

Add these 2 lines to .config file:

CONFIG_AQTION=m
CONFIG_AQUANTIA_PHY=m

Trigger customised kernel build on the Pi. This will take some time, so bear with us, please.

sudo make -j4 Image.gz modules dtbs
sudo make modules_install
sudo cp -v arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/
sudo cp -v arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/overlays/
sudo cp -v arch/arm64/boot/dts/overlays/README /boot/firmware/overlays/
KERNEL=kernel_2712
sudo cp -v arch/arm64/boot/Image.gz /boot/firmware/$KERNEL.img
uname -a
sudo reboot

After reboot, the LED light on the network adapter should come to life and we can capture first impressions.

Temperature

First thing you will likely notice is how hot this network adapter runs. It runs at 85° Celsius in idle which is slightly worrying and you can literally burn your fingers if you are not careful. Thumbs down on the thermal design front.

Under load, surprisingly, it ‘only’ runs 0.5° warmer.

How fast can it go then?

Raspberry Pi 5 officially supports PCIe Gen 1 and Gen 2. It is not certified for Gen 3.

PCIe Gen 1 mode

In this slowest mode, I got 1.71 Gbps/1.53 Gbps iperf3 TCP results with standard iperf3 settings. No jumbo frames, no other tweaks.

PCIe Gen 2 mode

Again, with standard iperf3 settings, I measured 3.44 Gbps/3.04 Gbps TCP throughput between 2 computers both connected to 10 Gbps switch ports via 10 GbE Full Duplex.

In idle conditions, this setup draws 7.5 W, and 8.9 W under 10GbE adapter iperf3 -R load (3.45 Gbps). Using more iperf3 parallel streams (the -P parameter) did not help at all.

PCIe Gen 3 mode

The adapter supports PCIe Gen 3, but it doesn’t work with the Pi. The Pi is not certified for Gen 3, so I can’t say anything bad about this. The Ethernet adapter is not recognised in Gen 3 mode, and no interface is present in ip a. Sometimes the Pi will fail to boot.

According to dmesg, the Pi forced Gen 2 mode:

brcm-pcie 1000110000.pcie: link down
brcm-pcie 1000120000.pcie: Forcing gen 2

Forcing PCIe Gen 2 mode athough Gen 3 has been configured

I powered my Pi from M2 MacBook USB-C port. So I thought, I might be running into under-voltage issues. I tested the official Raspberry Pi 27 W (5 V * 5 A) AC power and it made no difference.

Did you upgrade Raspberry Pi 5 firmware?

Yes, I did. It is running the latest version available as of March 2024.

Low CPU utilisation

One feature I really enjoyed is the extremely low CPU utilisation under load. I saw slower 2.5 GbE adapters hammer CPU with interrupts, but that’s not the case for this NIC. AQC107 does really good job at keeping the CPU cool.

Low Raspberry Pi 5 CPU load under network load

Cable analytics

Marvell supports Cable Diagnostics feature which uses TDR to measure cable length and detect Ethernet cable for defects. Unfortunately, it doesn’t seem to be supported on the AQC107 chip.

Can you get 10 Gbps out of this adapter at all?

I am glad you asked. How does an Intel NUC with this 10 GbE adapter sound? I’ve just tested it, here you go.

Summary

The high operating temperature really makes this adapter something I can’t recommend. With maximum throughput below 3.5 Gbps, I think you would be better off choosing a 2.5 Gigabit Ethernet adapter, which runs cool and delivers 2.35 Gbps/2.35 Gbps throughput.

Have you tested any other 10 GbE adapter? Did you get better results? Did you find any 2.5 Gbps Ethernet adapter that supports Cable Diagnostics? I am all ears.

Special thanks

Thanks to Luke Jenkins for exploring and sharing the kernel build instructions. Also, thanks to the WLAN Pi team. You can buy the team a coffee using this link.

Plugable 2.5 Gigabit Ethernet to USB 3.0 Multigigabit Adapter

Plugable makes this inexpensive 2.5 Gigabit Ethernet USBC-E2500 adapter. It is based on Realtek RTL8156B chip. On Windows and macOS it works out of the box. If you want to use it on a Linux machine like WLAN Pi Pro or Raspberry Pi 4, expect some troubles along the way, but good performance when you get there.

The USB-C to USB-A adapter is allows you to use it with a MacBook (USB-C) or Raspberry Pi 4 (USB-A)

The adapter itself has a plastic shell and is very lightweight

Windows 11

When they say “update the driver using Windows Update first”, they mean it. Windows 11 will recognise the adapter and you can start using it, but the default driver distributed with Windows 11 significantly reduces this adapter’s performance.

727 Mbps down and 2.34 Gbps up with default driver

Now, let’s use Windows Update to download the latest driver.

Don’t forget to update the driver using Windows Update

As you can see, download throughput (from iperf3 server to iperf3 client) has dramatically improved.

1.78 Gbps down and 2.35 Gbps up with updated driver

Although the box suggests Jumbo frame support, Windows driver settings don’t give me any option to edit the MTU size. So, I assume Jumbo frames are not supported.

MacOS Monterey

On macOS, this adapter works out of the box with no additional driver installation required. That’s a very nice surprise. And performance is great.

Auto-negotiation worked just fine. If you want to configure speed or MTU manually, you can, but Jumbo frames are not supported on macOS either.

Linux

Now the bad news. If you are considering to use this adapter on a Linux machine, the default driver cdc_ncm is a trouble as it only supports 2.5 Gbps Half duplex. Setting Full duplex manually using ethtool command doesn’t work either.

Default driver only supports Half duplex

As you might expect, with the default driver and Half duplex, throughput is very poor.

1.22 Gbps down and 704 Mbps up with the default cdc_ncm driver on WLAN Pi Pro

On WLAN Pi Pro and Raspberry Pi 4 running 5.15 Linux Kernel I managed to fix the duplex issue by the steps listed below. But I hit new auto-negotiation issue between the Plugable adapter and Cisco Catalyst WS-C3560CX-8XPD switch. It took the adapter to eventually negotiate 2.5 Gbps Full duplex around 15 minutes of constantly flapping the interface. Forcing speed and duplex on the Plugable adapter by ethtool did not work. Certainly not ideal, and definitely worth testing before you commit to the Plugable adapter. With other multigigabit adapters, the Plugable had no negotiation issues.

1.7 Gbps down and 2.09 Gbps up with r8156 driver on WLAN Pi Pro

1.91 Gbps down and 2.06 Gbps up on Raspberry Pi 4 using the correct r8156 driver

Raspberry Pi 4 also known as WLAN Pi Community Edition

How to force Linux to use the right driver

To enable Full duplex capability, we need to tell Linux to use Realtek r8156 driver instead of the default cdc-ncm.

Download the latest driver from Realtek’s website
Unzip it and copy the 50-usb-realtek-net.rules file to your Linux machine
On the Linux machine copy this file here sudo cp 50-usb-realtek-net.rules /etc/udev/rules.d/
Reboot by sudo reboot
Verify that the adapter negotiated 2.5 Gbps Full duplex and is using the Realtek r8156 driver.