raspberrypi - Jiri Brejcha

Full 5 Gigabit Ethernet on Raspberry Pi 5 with iocrest Realtek RTL8126 adapter

I’ve tested a number of 10 Gigabit Ethernet adapters on Raspberry Pi 5 based on the AQC107 chip. One adapter that negotiates PCIe Gen 3, achieves 5.5 Gbps speed and overheats. Another one which only works in PCIe Gen 2 mode and peaks at 3.44 Gbps. And even a full-size PCIe card made by TP-Link which negotiates PCIe Gen 2 link speed and doesn’t go beyond 3.44 Gbps either.

The Realtek RTL8126 chip we are testing today is so far the most suitable for Raspberry Pi 5. It is capable of 5 Gigabit Ethernet at full speed. TCP iperf3 throughput peaks at 4.7 Gbps. It doesn’t overheat. And it doesn’t excessively utilise the Raspberry Pi 5 CPU.

This particular one is sold under the iocrest brand. Like the other boards and adapters there is no increst branding on it and it will likely be sold under various brands. The RTL8126 chip is the key component here.

Raspberry Pi 5 with 5 Gigabit Ethernet network adapter

How did we connect it to the Pi? Via PCIe bus. We breakout the Raspberry Pi 5’s PCIe connector via Pineboards (aka Pineberry Pi) board to M.2 M-key slot. And in that slot we install the iocrest 5 Gigabit Ethernet network adapter – that’s the black M.2 module, plus a PCB with RJ-45 connector on a grey ribbon cable.

iocrest 5 GbE adapter connected to Raspberry Pi 5 via PCIe Gen 3 link

Here is how it looks from PCI device perspective.

Performance

It has no problem negotiating full duplex 5 Gigabit Ethernet and filling the interface with traffic fully.

iperf3 with default TCP settings peaks at 4.7 Gbps up and down. More parallel streams don’t improve the result any further. This is in PCIe Gen 3 mode.

Full 5 Gigabit Ethernet throughput in PCIe Gen 3 mode

Just for the record, if we downgrade PCIe bus to Gen 2 link speeds, we are talking 3.43 Gbps down and 3.31 Gbps up iperf3 TCP throughput-wise.

Throughput in downgraded PCIe bus to Gen 2 mode

Thermal footprint

Fully loaded by TCP traffic, I see temperature of 81.2° C (178° F) on the top surface of the RTL8126 chip. Yes, it is on the warmer side, but Raspberry Pi 5 SoC runs quite warm too and it is nowhere near 122° C temperatures I observed on this “hot” 10 Gigabit Ethernet adapter.

Chip temperature, installed in Intel NUC with M.2 slot

By the looks of it, there is no temperature sensor on the PHY so I can’t measure internal temperature.

CPU utilization and temperature of fully loaded adapter with TCP traffic

Linux software support

I happened to have Raspberry Pi OS with 6.8.0-rc7 kernel running on the Raspberry Pi 5. Out of the box, the adapter did not work. iocrest included driver download link pointing to this Chinese website but I am not so sure I want to use that one.

After installing driver from Realtek’s website, the adapter works just fine.

Power draw

This adapter in PCIe Gen 3 mode draws about 1.5 W in idle, and 2.1 W under full iperf3 load.

Switching the adapter to Gen 2 mode doesn’t make any power savings. I measured 0.1 W less in Gen 2 mode.

The whole setup of Raspberry Pi 5 with fan, Pineboards PCIe adapter, and this 5 GbE adapter in PCIe Gen 3 mode draws about 5.1 Watts in total under full iperf3 load.

Does it work on Windows 11?

Yes, it does. I installed one in Intel NUC 12th generation. It runs at full speed full and Gen 3 x1 mode.

Windows 11 driver (as of May 2024) downloaded automatically via Windows Update only allows this adapter to use 2.5 GbE. To unlock 5 GbE we download driver directly from Realtek’s website and we are all set.

Driver from Realtek’s website with full 5 GbE support

5 GbE full duplex with driver from Realtek’s website

With the adapter inserted in M.2 M-key slot, we won’t be able pop the NUC bottom lid back on. The adapter is just a bit too tall.

Bottom lid won’t fit with the adapter installed

Throughput also looks good. I might revisit Windows throughput testing tools at some point. But for now, I take 4.74 Gbps down and 4.42 Gbps up speeds. Increasing number of parallel streams did not improve throughput in any way.

For the record, Jumbo frames seem to be supported but I had no reason to explore this further this time.

Summary

As I mentioned towards the beginning, 5 Gigabit Ethernet based on Realtek RTL8126 chip seems to strike the perfect balance for Raspberry Pi 5. It delivers 4.7 Gbps up and down, doesn’t consume much power, and doesn’t produce excessive amount of heat.

Long-time test will tell how it actually performs but for now I am happy with what I’ve seen.

From driver perspective, I am wondering if the latest Linux kernel supports this chip natively or if I can enable the right kernel module manually.

One step closer to 10 Gigabit Ethernet on Raspberry Pi 5 – it is hot

This journey started as an exploration of maximum PCIe capabilities of Raspberry Pi 5 (and hopefully Compute Module 5) platform. I am mainly interested in multi-gigabit Ethernet and Wi-Fi 7 adapters connected via the PCI Express (PCIe) x1 bus.

Last time, we got throughput of 3.44 Gbps. The adapter and the Pi hit the bottleneck of PCIe Gen 2. Unfortunately, they failed to establish PCIe Gen 3 mode.

Generic 10 GbE adapter in M.2 form factor

This time we are going to use a slightly different adapter. It is available from various sellers under different names, but they all look and work the same. I picked up one from “KALEA-INFORMATIQUE” which happened to be readily available in the UK.

Pineberry’s HatDrive! Bottom breaks out Raspberry Pi’s PCIe connection to M.2 M-key format, and that’s where this 10 Gigabit Ethernet adapter plugs into.

Build custom kernel with AQC107 support

This Ethernet adapter uses the same chip and driver the one we previously tested. Here are the steps to make compile a custom Linux kernel that supports the adapter.

Wait, why is it still not working?

We have connected everything, built a custom kernel, we can see the device, but the Ethernet interface is not coming up.

Look at this official product photo and my photo below. Spot one difference 😉

Did you notice the orientation of the white ribbon cable? The official photo got it wrong. The printed text on the cable needs to be on the top on one side, and on the bottom on the other one.

What speeds did you get in PCIe Gen 2 mode?

After correcting the orientation of the flexible cable, the interface came up, negotiated 10 Gbps full duplex.

I started throughput testing against MacBook with my trusty 10 GbE Thunderbolt adapter.

In PCIe Gen 2 mode, we got TCP throughput of 3.45 Gbps on the downlink and 3.07 Gbps in the upstream direction. Using more iperf3 parallel streams did not increase performance.

Were you able to use PCIe Gen 3 mode?

Yes! And I got 4.63 Gbps of TCP downstream and 5.5 Gbps (potentially up to 6 Gbps) upstream.

I managed to get up to 6 Gbps in the upstream direction

That’s hot news… yes 122° Celsius hot!

This adapter has a thermal problem. It comes with a heatsink, but even in idle mode it overheats.

In PCIe Gen 3 mode with iperf3 test running, we are talking 122.1° C hot! The Pineberry board was very hot and you can literally burn your fingers by touching the heatsink.

In PCIe Gen 2 mode, it ‘only’ runs at 96.4° C

Long story short. Don’t buy this adapter, unless you want to add a fan or significantly larger heatsink.

Toaster, 10 Gigabit adapter, aren’t they the same thing?

Make your own opinion based on these couple of thermal photos.

Thermal IR footprint of the Ethernet adapter

This Ethernet adapter as well as the OWC 10 GbE Thunderbolt both use the same Aquantia AQC107 (part of Marvell now) chip. It does really good job at keeping CPU utilisation low. I’ve seen much cheaper 2.5 GbE adapters that hammer CPU with interrupts until the CPU just can’t take no more.

But, compare size of the two heatsinks. Unlike this one, the OWC adapters delivers good thermal results. Don’t take me wrong, it still runs warm, but not anywhere near.

Same AQC107 chip, massive thermal mass difference

Summary

On the positive side, this is the first 10 Gigabit adapter I tested which actually worked in PCIe Gen 3 mode on Raspberry Pi 5. I got TCP throughput of up to 6.0 Gbps.

As far as I can tell, the limit of Raspberry Pi 5’s PCIe bus is around 6 Gbps if you look at it through the iperf3 TCP traffic lens. AQC107 silicon does an amazing job at keeping the Raspberry Pi’s CPU utilisation low. This helps us get as much throughput as we can from the Pi. But it produces a significant amount of heat.

The fact is that this adapter overheats. Don’t buy it unless you wish to use it with a fan or design a much larger heatsink yourself.

10 Gigabit Ethernet on Raspberry Pi 5

Raspberry Pi 5 comes with PCI Express connection and a number of HATs (hardware attached on top) and Bottoms (the opposite of that) are now available for sale. That unlocks some very exciting options. Let’s see how fast can a 10 Gigabit Ethernet adapter on Raspberry Pi 5 go, shall we?

Pineberry’s HatDrive! Bottom proved to be really handy for converting Pi’s PCIe connection to M.2 M-key format. My Kalea-Informatique 10 Gigabit adapter uses exactly that, so that’s a match. Why did I choose this adapter? Very unscientifically this time – it was the first readily available and I was in a fail-fast mood :)

10 GbE adapter connected to Raspberry Pi 5

Pineberry HatDrive! Bottom board with 10 GbE network adapter

Detail of the AQC107 chip powering the network adapter

Enable PCIe port on Raspberry Pi 5

First things first. We need to enable the PCIe connector on the Pi.

sudo nano /boot/firmware/config.txt

# Enable the port
dtparam=pciex1

# Configure PCIe Gen
dtparam=pciex1_gen=2

Build custom Linux kernel and include the Aquantia driver module

Vanilla Raspberry Pi OS doesn’t include the Aquantia AQC107 kernel module. So we need to burn a micro SD card with a vanilla Raspberry Pi OS Bookworm image, boot the Pi 5 and build a customised kernel.

git clone --depth=1 --branch rpi-6.8.y https://github.com/raspberrypi/linux
cd linux/
sudo apt install flex bison aptitude -y
sudo aptitude install libssl-dev
make bcm2712_defconfig

Edit the config file:

sudo nano .config

Add these 2 lines to .config file:

CONFIG_AQTION=m
CONFIG_AQUANTIA_PHY=m

Trigger customised kernel build on the Pi. This will take some time, so bear with us, please.

sudo make -j4 Image.gz modules dtbs
sudo make modules_install
sudo cp -v arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/
sudo cp -v arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/overlays/
sudo cp -v arch/arm64/boot/dts/overlays/README /boot/firmware/overlays/
KERNEL=kernel_2712
sudo cp -v arch/arm64/boot/Image.gz /boot/firmware/$KERNEL.img
uname -a
sudo reboot

After reboot, the LED light on the network adapter should come to life and we can capture first impressions.

Temperature

First thing you will likely notice is how hot this network adapter runs. It runs at 85° Celsius in idle which is slightly worrying and you can literally burn your fingers if you are not careful. Thumbs down on the thermal design front.

Under load, surprisingly, it ‘only’ runs 0.5° warmer.

How fast can it go then?

Raspberry Pi 5 officially supports PCIe Gen 1 and Gen 2. It is not certified for Gen 3.

PCIe Gen 1 mode

In this slowest mode, I got 1.71 Gbps/1.53 Gbps iperf3 TCP results with standard iperf3 settings. No jumbo frames, no other tweaks.

PCIe Gen 2 mode

Again, with standard iperf3 settings, I measured 3.44 Gbps/3.04 Gbps TCP throughput between 2 computers both connected to 10 Gbps switch ports via 10 GbE Full Duplex.

In idle conditions, this setup draws 7.5 W, and 8.9 W under 10GbE adapter iperf3 -R load (3.45 Gbps). Using more iperf3 parallel streams (the -P parameter) did not help at all.

PCIe Gen 3 mode

The adapter supports PCIe Gen 3, but it doesn’t work with the Pi. The Pi is not certified for Gen 3, so I can’t say anything bad about this. The Ethernet adapter is not recognised in Gen 3 mode, and no interface is present in ip a. Sometimes the Pi will fail to boot.

According to dmesg, the Pi forced Gen 2 mode:

brcm-pcie 1000110000.pcie: link down
brcm-pcie 1000120000.pcie: Forcing gen 2

Forcing PCIe Gen 2 mode athough Gen 3 has been configured

I powered my Pi from M2 MacBook USB-C port. So I thought, I might be running into under-voltage issues. I tested the official Raspberry Pi 27 W (5 V * 5 A) AC power and it made no difference.

Did you upgrade Raspberry Pi 5 firmware?

Yes, I did. It is running the latest version available as of March 2024.

Low CPU utilisation

One feature I really enjoyed is the extremely low CPU utilisation under load. I saw slower 2.5 GbE adapters hammer CPU with interrupts, but that’s not the case for this NIC. AQC107 does really good job at keeping the CPU cool.

Low Raspberry Pi 5 CPU load under network load

Cable analytics

Marvell supports Cable Diagnostics feature which uses TDR to measure cable length and detect Ethernet cable for defects. Unfortunately, it doesn’t seem to be supported on the AQC107 chip.

Can you get 10 Gbps out of this adapter at all?

I am glad you asked. How does an Intel NUC with this 10 GbE adapter sound? I’ve just tested it, here you go.

Summary

The high operating temperature really makes this adapter something I can’t recommend. With maximum throughput below 3.5 Gbps, I think you would be better off choosing a 2.5 Gigabit Ethernet adapter, which runs cool and delivers 2.35 Gbps/2.35 Gbps throughput.

Have you tested any other 10 GbE adapter? Did you get better results? Did you find any 2.5 Gbps Ethernet adapter that supports Cable Diagnostics? I am all ears.

Special thanks

Thanks to Luke Jenkins for exploring and sharing the kernel build instructions. Also, thanks to the WLAN Pi team. You can buy the team a coffee using this link.

Multigigabit Ethernet on the Raspberry Pi 4

Update: The Linux driver for this adapter does not seem to perform great. I tested a 2.5 GbE adapter based on Realtek chipset, which might work better for you. Here is my review.

I also tested this adapter on Windows 11 and macOS here.

With the first consumer Wi-Fi 6E routers already shipping, and enterprise access points being worked on, I think it is now time to up my iperf3 game. While the standard 1 Gbps adapters push around 950 Mbps of TCP traffic, the iperf3 server will sooner or later become a bottleneck for throughput measurements.

Raspberry Pi 4 (RPi4) is widely available, and there is a chance that you might already own one. So, the question is, can it support multigigabit speeds?

Although it does not have any PCI Express slot available, it does have a couple of USB 3.0 ports. I researched USB 3.0 multigigabit NBASE-T Ethernet adapters, and purchased a few. At the time of writing, Linux kernel 5.10 is the Raspberry Pi OS gold standard, and that’s what I used for all tests.

TL;DR … give me the short answer

The maximum TCP throughput Raspberry Pi 4 iperf3 server can handle with a 5 Gbps USB 3.0 Ethernet adapter. These were 90-second iperf3 tests with standard 1500-byte MTU and a single iperf3 stream.

Download (from RPi4 server to a client): 2.05 Gbps
Upload (from a client to RPi4 server): 528 Mbps

If you can enable 9000-byte Jumbo frames on all devices involved in the data path, the upload speed becomes much healthier.

Download (from RPi4 server to a client): 2.05 Gbps
Upload (from a client to RPi4 server): 1.73 Gbps

USB multigigabit adapters

There are a few available on the market. After reading a dozen of reviews, I decided to get 3 adapters from a company called Sabrent. They make adapters with nice metal cases, which helps with dealing with the heat they dissipate.

Left to right: 2.5 GE USB 3.0 Sabrent NT-S25G, 5 GE USB 3.1 Sabrent NT-SS5G, 10 GE Thunderbolt 3 Sabrent TH-S3EA

The 5 GE USB 3.1 Sabrent NT-SS5G is the only I recommend for use with the RPi4:

It uses the Marvell Aquantia AQC111U chipset
It works out of the box with Raspberry Pi OS and
It works on Windows 10 after you install the driver
It ships with USB-C and USB-A cables so you can connect it to your laptop using USB-C or RPi4 using USB-A
Unlike other brands it does not overhead or disconnect due to instability

5 GE USB 3.1 Sabrent NT-SS5G ships with both USB-A and USB-C cables

ServeTheHome team did a great job of comparing the 5 GE adapters using the same chipset as the 5 GE USB 3.1 Sabrent NT-SS5G. It came out as a clear winner:

USB 3.1 Gen1 To 5GbE Comparison Table AQC111U Based Q1 2021 — Credit and kudos to ServeTheHome

Why not use the cheaper 2.5 GE USB 3.0 Sabrent NT-S25G? Because it uses Realtek 8156 chipset, and there is no suitable Linux driver available at the time of writing.

Why not the 10 GE Thunderbolt 3 Sabrent TH-S3EA? Although it has a USB-C connector, it is not a USB adapter. It uses Thunderbolt 3 protocol, which is not supported by the RPi4.

Test setup

I use RPi4 with PoE HAT, because it has a fan on it, and I power the unit by a USB-C charger. Both multigigabit adapters involved in the test are 5-Gigabit Ethernet capable Sabrent NT-SS5G. The best part is that these work out of the box on the Raspberry Pi OS with no action required on your part.

But, if you are considering the purchase of these adapters for your Mac, please stop. After you plug the adapter in, it uses Apple’s 1 Gigabit Ethernet adapter driver, and it would only auto-negotiate 1 Gbps. To enable 2.5 and 5 Gigabit speeds, and support for Jumbo frames on Mac, you have to disable Apple System Integrity Protection (SIP) tool, and install a legacy kext Sabrent driver. I would discourage you from making these security compromises. If you are interested in a multigigabit adapter that works with macOS out of the box, tune in later and read this review (link to be added).

How to increase the MTU and enable Jumbo frames?

On the Raspberry Pi:

sudo ip link set dev eth1 mtu 9000

On the Cisco Catalyst switch running IOS:

System Preferences > Network > Adapter settings > Hardware > MTU on macOS

Conclusion

The RPi4 allows you to test download-only throughput up to 2 Gbps with standard MTU. Upload speeds are really poor and you would be better off using the built-in 1 Gibabit Ethernet adapter. You can use the RPi4 to run a few other tools, scripts, or take wall attenuation measurements.

With Jumbo frames enabled, 2 Gbps/1.7 Gbps is good enough for lab use or demonstrations. Keep in mind that you would have to enable Jumbo frames on all devices (RPi4, MacBook and the switch in my case).

The main cause of the relatively low performance is the storm of IRQ hammering the RPi4 CPU:

In a computer, an interrupt request (or IRQ) is a hardware signal sent to the processor that temporarily stops a running program and allows a special program, an interrupt handler, to run instead. Hardware interrupts are used to handle events such as receiving data from a modem or network card, key presses, or mouse movements.
Source: Wikipedia.org

If your use case requires a powerful iperf3 server, Apple’s Mac Mini with built-in 10 Gigabit Ethernet adapter would be something to consider today. It won’t be the cheapest option, but you won’t have to worry about performance or USB dongles. From what I’ve found, it uses Marvell AQC 1113 chipset and does 9.4 Gbps with 4 parallel iperf3 streams.

Disclosure

I purchased these adapters myself. No one asked me, or paid me, to write this blog post. I was as curious as you to see how the RPi4 performs when it comes to multigigabit Ethernet.