10 Gigabit Ethernet on Raspberry Pi 5

Raspberry Pi 5 comes with PCI Express connection and a number of HATs (hardware attached on top) and Bottoms (the opposite of that) are now available for sale. That unlocks some very exciting options. Let’s see how fast can a 10 Gigabit Ethernet adapter on Raspberry Pi 5 go, shall we?

Pineberry’s HatDrive! Bottom proved to be really handy for converting Pi’s PCIe connection to M.2 M-key format. My Kalea-Informatique 10 Gigabit adapter uses exactly that, so that’s a match. Why did I choose this adapter? Very unscientifically this time – it was the first readily available and I was in a fail-fast mood :)

10 GbE adapter connected to Raspberry Pi 5
Pineberry HatDrive! Bottom board with 10 GbE network adapter
Detail of the AQC107 chip powering the network adapter

Enable PCIe port on Raspberry Pi 5

First things first. We need to enable the PCIe connector on the Pi.

sudo nano /boot/firmware/config.txt

# Enable the port
dtparam=pciex1

# Configure PCIe Gen
dtparam=pciex1_gen=2
Enable PCIe and configure mode

Build custom Linux kernel and include the Aquantia driver module

Vanilla Raspberry Pi OS doesn’t include the Aquantia AQC107 kernel module. So we need to burn a micro SD card with a vanilla Raspberry Pi OS Bookworm image, boot the Pi 5 and build a customised kernel.

git clone --depth=1 --branch rpi-6.8.y https://github.com/raspberrypi/linux
cd linux/
sudo apt install flex bison aptitude -y
sudo aptitude install libssl-dev
make bcm2712_defconfig

Edit the config file:

sudo nano .config

Add these 2 lines to .config file:

CONFIG_AQTION=m
CONFIG_AQUANTIA_PHY=m

Trigger customised kernel build on the Pi. This will take some time, so bear with us, please.

sudo make -j4 Image.gz modules dtbs
sudo make modules_install
sudo cp -v arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/
sudo cp -v arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/overlays/
sudo cp -v arch/arm64/boot/dts/overlays/README /boot/firmware/overlays/
KERNEL=kernel_2712
sudo cp -v arch/arm64/boot/Image.gz /boot/firmware/$KERNEL.img
uname -a
sudo reboot

After reboot, the LED light on the network adapter should come to life and we can capture first impressions.

Adapter recognised
10 Gbps Full Duplex
lspci -v output

Temperature

First thing you will likely notice is how hot this network adapter runs. It runs at 85° Celsius in idle which is slightly worrying and you can literally burn your fingers if you are not careful. Thumbs down on the thermal design front.

High idle temperature

Under load, surprisingly, it ‘only’ runs 0.5° warmer.

High temperature under load

How fast can it go then?

Raspberry Pi 5 officially supports PCIe Gen 1 and Gen 2. It is not certified for Gen 3.

PCIe Gen 1 mode

In this slowest mode, I got 1.71 Gbps/1.53 Gbps iperf3 TCP results with standard iperf3 settings. No jumbo frames, no other tweaks.

PCIe Gen 1 throughput

PCIe Gen 2 mode

Again, with standard iperf3 settings, I measured 3.44 Gbps/3.04 Gbps TCP throughput between 2 computers both connected to 10 Gbps switch ports via 10 GbE Full Duplex.

PCIe Gen 2 throughput

In idle conditions, this setup draws 7.5 W, and 8.9 W under 10GbE adapter iperf3 -R load (3.45 Gbps). Using more iperf3 parallel streams (the -P parameter) did not help at all.

Power draw

PCIe Gen 3 mode

The adapter supports PCIe Gen 3, but it doesn’t work with the Pi. The Pi is not certified for Gen 3, so I can’t say anything bad about this. The Ethernet adapter is not recognised in Gen 3 mode, and no interface is present in ip a. Sometimes the Pi will fail to boot.

According to dmesg, the Pi forced Gen 2 mode:

brcm-pcie 1000110000.pcie: link down
brcm-pcie 1000120000.pcie: Forcing gen 2
Forcing PCIe Gen 2 mode athough Gen 3 has been configured

I powered my Pi from M2 MacBook USB-C port. So I thought, I might be running into under-voltage issues. I tested the official Raspberry Pi 27 W (5 V * 5 A) AC power and it made no difference.

Did you upgrade Raspberry Pi 5 firmware?

Yes, I did. It is running the latest version available as of March 2024.

Latest firmware installed

Low CPU utilisation

One feature I really enjoyed is the extremely low CPU utilisation under load. I saw slower 2.5 GbE adapters hammer CPU with interrupts, but that’s not the case for this NIC. AQC107 does really good job at keeping the CPU cool.

Low Raspberry Pi 5 CPU load under network load

Cable analytics

Marvell supports Cable Diagnostics feature which uses TDR to measure cable length and detect Ethernet cable for defects. Unfortunately, it doesn’t seem to be supported on the AQC107 chip.

Cable Diagnostics not supported

Can you get 10 Gbps out of this adapter at all?

I am glad you asked. How does an Intel NUC with this 10 GbE adapter sound? I’ve just tested it, here you go.

Intel NUC with 10 GbE adapter

Summary

The high operating temperature really makes this adapter something I can’t recommend. With maximum throughput below 3.5 Gbps, I think you would be better off choosing a 2.5 Gigabit Ethernet adapter, which runs cool and delivers 2.35 Gbps/2.35 Gbps throughput.

Have you tested any other 10 GbE adapter? Did you get better results? Did you find any 2.5 Gbps Ethernet adapter that supports Cable Diagnostics? I am all ears.

Special thanks

Thanks to Luke Jenkins for exploring and sharing the kernel build instructions. Also, thanks to the WLAN Pi team. You can buy the team a coffee using this link.

2.5 Gbps Ethernet on WLAN Pi M4

WLAN Pi is primarily a Wi-Fi tool, but occasionally I need an iperf server that would be able to deliver more than 1 Gbps of TCP throughput. In a controlled lab environment, I normally use PoE powered NanoPi R5S. I know the IP address of the iperf server by heart. Outside of the lab, I could really do with a WLAN Pi, its preinstalled software, display, buttons and everything it does out of the box. So the question is: “Can we add 2.5 GbE to WLAN Pi M4?”

M.2 slot to the rescue

WLAN Pi M4 doesn’t have any USB 3 ports. How do we add 2.5 Gbps Ethernet to it? If you don’t mind losing the Wi-Fi adapter in favour of 2.5 GbE mGig port, we can install this 2.5 Gbps Ethernet adapter in M4’s PCIe M.2 slot. It is based on Realtek RTL8125B chipset. I paid £17 for it including shipping to the UK.

M.2 A+E KEY 2.5G Ethernet RTL8125B PCI Express Network Adapter

It just works*

To my surprise, it just works*. Yes, I hear you, no one likes these asterisks, do you? 😉 Continue reading, it’s not the end of the story.

WLAN Pi M4 with 2.5 Gbps Ethernet
2.5 Gbps full duplex

The underwhelming default driver

Linux (and WLAN Pi image) has a driver for this adapter, but upload speeds, that is from iperf client to WLAN Pi iperf server, are very poor. We are talking 300 Mbps poor.

Poor 300 Mbps upload speed

Install Realtek’s latest driver to fix performance

Downloading, compiling and installing the latest Linux driver from Realtek’s website fixes the performance issue. We get symmetric 2.35 Gbps of TCP throughput with standard packet size.

2.35 Gbps of iperf3 TCP throughput

Installation of this driver isn’t as straightforward as it might look. I ended using vanilla Raspberry Pi OS image instead of the WLAN Pi one. Mainly because it is not easy to get the kernel headers for WLAN Pi image and we need them to be able to compile the new driver.

Summary

Yes, it is possible to achieve 2.35 Gbps symmetric TCP throughput on the WLAN Pi M4 with this adapter. But you should be aware of these facts:

  • This Ethernet adapter doesn’t fit inside WLAN Pi M4 case
  • You will have to give up the M.2 Wi-Fi adapter in favour of mGig Ethernet
  • From software perspective, the Realtek driver that ships in WLAN Pi image doesn’t unlock full performance of this adapter (iperf client pushing traffic to WLAN Pi iperf server). Installing the latest driver isn’t trivial on WLAN Pi.
  • We, WLAN Pi team, currently don’t support this setup. If you have a use case for 2.5 GbE support on the M4, please let us know.

Portable Catalyst 9136 Wi-Fi 6E demo powered by Zyxel 802.3bt power injector

I am building a portable Wi-fi 6E demo in a box solution. What do I use for that?

PoE powered FriendlyElec’s NanoPi R5S runs iperf3 server. Here a quick iperf3 performance review of this little, 2.5 GbE, and mighty Linux box.

My Catalyst 9800-CL controller is hosted on a cloud, so I don’t need any hardware for that. Finally, my Catalyst 9136 Wi-Fi 6E AP is powered by a Catalyst 3560CX 10 Gigabit Ethernet multigigabit switch.

6 GHz 2×2 MIMO setup powered by PoE+

Catalyst 9136 is Cisco’s premium AP with all the bells and whistles including hexa-radio architecture and built-in environmental sensors for smart building use cases. It requires an 802.3bt/UPOE power source to enable 6 GHz radio in full performance 4×4 MIMO mode. The switch I use supports 802.3at/PoE+, which is great, but 6 GHz radio downshifts to 2×2. And that’s where an 802.3bt power injector comes to the rescue.

Zyxel 5G PoE++ Injector

Cisco’s 5 GbE 802.11bt power injector (AIR-PWRINJ7=) is now available, and that’s my go to option for production use.

Since the Cisco injector isn’t widely available yet, I decided to test this Zyxel one. It provides 802.3bt power and allows the AP to run in full power and full 4×4 6 GHz radio mode with no compromise.

Do I like power injectors in production?

Absolutely not! Ideally you should design for 802.3bt/UPOE switches to power all your new APs via PoE.

It allows you to:

  • easily, centrally and remotely monitor how much power the APs use
  • enable/disable power on a port to bounce an AP
  • leverage redundant Platinum-rated power supplies for the AC to DC power conversion
  • manage the solution with ease – just think how difficult it is to manage more than 1 power injector, the number of AC power sockets, and what happens when someone disconnects the injector?
I still use C3650 UPOE mGig switch in my lab. Catalysts 9300 and 9400 the best choice these days.
UPOE and mGig capable C3650 providing full power to the AP

Final look

Carrying a full-size switch is not really an option for me, because small form factor is my main goal. So a power injector works best for me. But if I could I would love to use a compact 802.3bt switch.

Are you wondering if the PoE splitter connected to my iperf3 server (the little black box with 3 Ethernet interfaces) actually negotiated 2.5 Gbps Full duplex with the switch? Yes, it did. But keep in mind that the PoE splitter is technically only rated for 1 GbE. So use as short patch cable as possible and ideally CAT6.

Still few things to tidy up and perhaps I could build this into a nice Pelican case

OWC Thunderbolt 3 to 10 Gbps Ethernet Adapter – The Fastest Multigigabit Adapter For Your Mac

When it comes to the fastest copper Ethernet adapter for your Mac, you have only 2 options:

  • If your other half approves, get yourself an M1 Mac Mini with built-in 10 GbE port. It doesn’t get much better than this.
  • Or you can consider an add-on 10 Gigabit Ethernet Thunderbolt 3 adapter for your current Mac.

We will focus on the latter today.

Thunderbolt 3, not USB

While the USB-C connector might temp you to connect these adapters to a standard USB port, these adapters don’t support USB protocol. They use Thunderbolt 3 and they happen to use the same USB-C connector as USB. That’s the only thing USB and Thunderbolt have in common. Before you order one of these adapters, double-check that your computer supports Thunderbolt 3. That should be most new MacBooks, Mac Minis, Intel NUCs and similar platforms.

Which 10 GbE adapter shall I buy?

I tested two of these Thunderbolt 10 GbE adapters. One made by Sabrent, and the other by OWC. They both look alike, both perform very well, both get quite warm, and both work out of the box on macOS. Yes, no driver installation required on your part on macOS! 🎉

Mainly because of the loose Sabrent cable issue explained below, I recommend the OWC adapter. It comes with great documentation, and even the Thunderbolt cable itself is thicker, feels premium, and most likely delivers better shielding.

OWC Thunderbolt 3 10G Ethernet Adapter OWCTB3ADP10GBE

From throughput perspective, I personally tested it up to 3 Gbps down and 3.3 Gbps up using iperf3 with default settings. The limitation is on my part, I just don’t have another 10 GbE computer I could test against.

I’ve seen reports of:

  • between 7 Gbps and 8.74 Gbps uplink speeds with default iperf3 settings
  • 9.5 Gbps uplink iperf3 speeds with Jumbo frames enabled

When I reviewed 2.5 GbE and 5 GbE adapters, this setup has become my reference I ran all iperf3 tests against.

OWC connected to an M1 MacBook Pro
Thunderbolt side
Ethernet side
Raspberry Pi 4 for scale
10 Gbps Full Duplex
It supports Jumbo frames including a custom MTU setting

VLAN tagging

The OWC adapter also supports VLAN tagging. Here is my Trunk port with Native VLAN 129:

Trunk port configured on the access switch

Let’s tag all traffic with VLAN 130:

Create VLAN interface on macOS

Verify that we are indeed in VLAN 130:

VLAN 130 is being used instead of the Native VLAN 129

If you only want to use VLAN 130 (without touching the Native VLAN 129), you can disable the adapter itself. VLAN 130 virtual interface will stay up and forward traffic.

Disable the Native VLAN 129 and only use VLAN 130 for all traffic

Sabrent Thunderbolt 3 to 10 Gbps Ethernet Adapter TH-S3EA

I won’t go into the detail, but my main challenge with the Sabrent adapter was its loose Thunderbolt cable. The connection between the USB-C socket on the adapter and the USB-C connector on the Thunderbolt cable is very loose and practically pulls out just by the tension of the cable itself. It might have been just my unit, but I can’t recommend it.

Sabrent Thunderbolt 3 to 10Gbps Ethernet Adapter on the left
It almost felt like it needed some hot glue to keep the Thunderbolt cable connected

What about Windows and Linux support?

I tested the Sabrent adapter on Windows 10. It required a Sabrent driver installation and then it worked just fine. I would assume the same for the OWC.

I don’t have a Linux computer with a Thunderbolt port, so I can’t share anything on that front.

Sabrent 5 GbE Multigigabit Ethernet Adapter

Sabrent NT-SS5G is a 5 GbE USB adapter, which allows you to achieve higher throughput than 2.5 GbE adapters, and break the 2.35 Gbps barrier. It works great on Windows. If you are a macOS or Linux user, I recommend you consider other options like this instead.

The adapter itself is larger than 2.5 GbE adapters, it uses AQC111U chip, and ships with short 2 detachable USB-A and USB-C cables. USB-C port on its back connects the adapter to your computer. A metal shell protects it, serves as a heatsink, and also adds to its weight.

Windows 11

Install the driver from Sabrent’s website and you are good to go. In my tests with this Topton M6 Mini PC, I measured 2.93 Gbps down and 3.44 Gbps up with default iperf3 settings.

2.93 Gbps down and 3.44 Gbps up with default iperf3 settings

In adapter options, you can actually configure quite a few things including Jumbo frame support. Note that these are fixed values.

macOS

I can’t recommend this adapter for macOS users. It forces you to disable macOS System Integrity Protection (csrutil), otherwise it won’t work. It might be okay for a proof of concept or lab setup, but I would hesitate from using it in production.

This is how to install the driver if you were interested:

  1. Install the driver using the pkg file provided by Sabrent. It installs a Kernel Extension (kext), which drives this adapter.
  2. Enable the extension by going to System Preferences > Security & Privacy > enable the extension > Reboot.
  3. After reboot, unplug the adapter and plug it back in.
  4. It should work as long as you leave the System Integrity Protection disabled.

From throughput perspective, it saw download speeds of 3.30 Gbps, and upload of 3.45 Gbps. This was with default iperf3 settings, standard 1500-byte MTU and one stream. Great results considering that this adapter’s USB interface maximum theoretical throughput is 5 Gbps.

In my view, you might be better off buying a 2.5 GbE adapter, which can push 2.35 Gbps up and down consistently and with no driver installation needed. I tested one here. Alternatively, a 10GbE Thunderbolt Ethernet adapter is even faster choice, but more costly, and larger form factor. Or, if your other half approves, treat yourself to an M1 Mac Mini with built-in 10 GbE 😉

Linux

I tested this adapter on 64-bit Raspberry Pi OS running on Raspberry Pi 4. Although the default driver distributed in Linux Kernel 5.15 works, it doesn’t even deliver symmetric 1 Gbps.

Sabrent connected to Raspberry Pi 4
Upload speeds well below 1 Gbps
Default aqc111 driver details

Let’s download the latest driver from Sabrent’s website. Unfortunately that doesn’t seem to be able to compile for 64-bit OS. I tried compiling on 32-bit Raspberry OS, to no avail. If you have any ideas, please do let me know.

So, on Linux, a Realtek RTL8156B based 2.5 GbE adapter might be a better choice for you. Here is the one I tested.

Plugable 2.5 Gigabit Ethernet to USB 3.0 Multigigabit Adapter

Plugable makes this inexpensive 2.5 Gigabit Ethernet USBC-E2500 adapter. It is based on Realtek RTL8156B chip. On Windows and macOS it works out of the box. If you want to use it on a Linux machine like WLAN Pi Pro or Raspberry Pi 4, expect some troubles along the way, but good performance when you get there.

The USB-C to USB-A adapter is allows you to use it with a MacBook (USB-C) or Raspberry Pi 4 (USB-A)
The adapter itself has a plastic shell and is very lightweight

Windows 11

When they say “update the driver using Windows Update first”, they mean it. Windows 11 will recognise the adapter and you can start using it, but the default driver distributed with Windows 11 significantly reduces this adapter’s performance.

727 Mbps down and 2.34 Gbps up with default driver

Now, let’s use Windows Update to download the latest driver.

Don’t forget to update the driver using Windows Update

As you can see, download throughput (from iperf3 server to iperf3 client) has dramatically improved.

1.78 Gbps down and 2.35 Gbps up with updated driver

Although the box suggests Jumbo frame support, Windows driver settings don’t give me any option to edit the MTU size. So, I assume Jumbo frames are not supported.

MacOS Monterey

On macOS, this adapter works out of the box with no additional driver installation required. That’s a very nice surprise. And performance is great.

Symmetric 2.35 Gbps throughput on macOS

Auto-negotiation worked just fine. If you want to configure speed or MTU manually, you can, but Jumbo frames are not supported on macOS either.

Jumbo frames are not supported

Linux

Now the bad news. If you are considering to use this adapter on a Linux machine, the default driver cdc_ncm is a trouble as it only supports 2.5 Gbps Half duplex. Setting Full duplex manually using ethtool command doesn’t work either.

Default driver only supports Half duplex

As you might expect, with the default driver and Half duplex, throughput is very poor.

1.22 Gbps down and 704 Mbps up with the default cdc_ncm driver on WLAN Pi Pro

On WLAN Pi Pro and Raspberry Pi 4 running 5.15 Linux Kernel I managed to fix the duplex issue by the steps listed below. But I hit new auto-negotiation issue between the Plugable adapter and Cisco Catalyst WS-C3560CX-8XPD switch. It took the adapter to eventually negotiate 2.5 Gbps Full duplex around 15 minutes of constantly flapping the interface. Forcing speed and duplex on the Plugable adapter by ethtool did not work. Certainly not ideal, and definitely worth testing before you commit to the Plugable adapter. With other multigigabit adapters, the Plugable had no negotiation issues.

1.7 Gbps down and 2.09 Gbps up with r8156 driver on WLAN Pi Pro
1.91 Gbps down and 2.06 Gbps up on Raspberry Pi 4 using the correct r8156 driver
Raspberry Pi 4 also known as WLAN Pi Community Edition

How to force Linux to use the right driver

To enable Full duplex capability, we need to tell Linux to use Realtek r8156 driver instead of the default cdc-ncm.

  1. Download the latest driver from Realtek’s website
  2. Unzip it and copy the 50-usb-realtek-net.rules file to your Linux machine
  3. On the Linux machine copy this file here sudo cp 50-usb-realtek-net.rules /etc/udev/rules.d/
  4. Reboot by sudo reboot
  5. Verify that the adapter negotiated 2.5 Gbps Full duplex and is using the Realtek r8156 driver.