WLAN Pi is primarily a Wi-Fi tool, but occasionally I need an iperf server that would be able to deliver more than 1 Gbps of TCP throughput. In a controlled lab environment, I normally use PoE powered NanoPi R5S. I know the IP address of the iperf server by heart. Outside of the lab, I could really do with a WLAN Pi, its preinstalled software, display, buttons and everything it does out of the box. So the question is: “Can we add 2.5 GbE to WLAN Pi M4?”
M.2 slot to the rescue
WLAN Pi M4 doesn’t have any USB 3 ports. How do we add 2.5 Gbps Ethernet to it? If you don’t mind losing the Wi-Fi adapter in favour of 2.5 GbE mGig port, we can install this 2.5 Gbps Ethernet adapter in M4’s PCIe M.2 slot. It is based on Realtek RTL8125B chipset. I paid £17 for it including shipping to the UK.
It just works*
To my surprise, it just works*. Yes, I hear you, no one likes these asterisks, do you? 😉 Continue reading, it’s not the end of the story.
The underwhelming default driver
Linux (and WLAN Pi image) has a driver for this adapter, but upload speeds, that is from iperf client to WLAN Pi iperf server, are very poor. We are talking 300 Mbps poor.
Install Realtek’s latest driver to fix performance
Installation of this driver isn’t as straightforward as it might look. I ended using vanilla Raspberry Pi OS image instead of the WLAN Pi one. Mainly because it is not easy to get the kernel headers for WLAN Pi image and we need them to be able to compile the new driver.
Yes, it is possible to achieve 2.35 Gbps symmetric TCP throughput on the WLAN Pi M4 with this adapter. But you should be aware of these facts:
This Ethernet adapter doesn’t fit inside WLAN Pi M4 case
You will have to give up the M.2 Wi-Fi adapter in favour of mGig Ethernet
From software perspective, the Realtek driver that ships in WLAN Pi image doesn’t unlock full performance of this adapter (iperf client pushing traffic to WLAN Pi iperf server). Installing the latest driver isn’t trivial on WLAN Pi.
We, WLAN Pi team, currently don’t support this setup. If you have a use case for 2.5 GbE support on the M4, please let us know.
Sabrent NT-SS5G is a 5 GbE USB adapter, which allows you to achieve higher throughput than 2.5 GbE adapters, and break the 2.35 Gbps barrier. It works great on Windows. If you are a macOS or Linux user, I recommend you consider other options like this instead.
The adapter itself is larger than 2.5 GbE adapters, it uses AQC111U chip, and ships with short 2 detachable USB-A and USB-C cables. USB-C port on its back connects the adapter to your computer. A metal shell protects it, serves as a heatsink, and also adds to its weight.
In adapter options, you can actually configure quite a few things including Jumbo frame support. Note that these are fixed values.
I can’t recommend this adapter for macOS users. It forces you to disable macOS System Integrity Protection (csrutil), otherwise it won’t work. It might be okay for a proof of concept or lab setup, but I would hesitate from using it in production.
This is how to install the driver if you were interested:
Install the driver using the pkg file provided by Sabrent. It installs a Kernel Extension (kext), which drives this adapter.
Enable the extension by going to System Preferences > Security & Privacy > enable the extension > Reboot.
After reboot, unplug the adapter and plug it back in.
It should work as long as you leave the System Integrity Protection disabled.
From throughput perspective, it saw download speeds of 3.30 Gbps, and upload of 3.45 Gbps. This was with default iperf3 settings, standard 1500-byte MTU and one stream. Great results considering that this adapter’s USB interface maximum theoretical throughput is 5 Gbps.
In my view, you might be better off buying a 2.5 GbE adapter, which can push 2.35 Gbps up and down consistently and with no driver installation needed. I tested one here. Alternatively, a 10GbE Thunderbolt Ethernet adapter is even faster choice, but more costly, and larger form factor. Or, if your other half approves, treat yourself to an M1 Mac Mini with built-in 10 GbE 😉
I tested this adapter on 64-bit Raspberry Pi OS running on Raspberry Pi 4. Although the default driver distributed in Linux Kernel 5.15 works, it doesn’t even deliver symmetric 1 Gbps.
Let’s download the latest driver from Sabrent’s website. Unfortunately that doesn’t seem to be able to compile for 64-bit OS. I tried compiling on 32-bit Raspberry OS, to no avail. If you have any ideas, please do let me know.
Plugable makes this inexpensive 2.5 Gigabit Ethernet USBC-E2500 adapter. It is based on Realtek RTL8156B chip. On Windows and macOS it works out of the box. If you want to use it on a Linux machine like WLAN Pi Pro or Raspberry Pi 4, expect some troubles along the way, but good performance when you get there.
When they say “update the driver using Windows Update first”, they mean it. Windows 11 will recognise the adapter and you can start using it, but the default driver distributed with Windows 11 significantly reduces this adapter’s performance.
Now, let’s use Windows Update to download the latest driver.
As you can see, download throughput (from iperf3 server to iperf3 client) has dramatically improved.
Although the box suggests Jumbo frame support, Windows driver settings don’t give me any option to edit the MTU size. So, I assume Jumbo frames are not supported.
On macOS, this adapter works out of the box with no additional driver installation required. That’s a very nice surprise. And performance is great.
Auto-negotiation worked just fine. If you want to configure speed or MTU manually, you can, but Jumbo frames are not supported on macOS either.
Now the bad news. If you are considering to use this adapter on a Linux machine, the default driver cdc_ncm is a trouble as it only supports 2.5 Gbps Half duplex. Setting Full duplex manually using ethtool command doesn’t work either.
As you might expect, with the default driver and Half duplex, throughput is very poor.
On WLAN Pi Pro and Raspberry Pi 4 running 5.15 Linux Kernel I managed to fix the duplex issue by the steps listed below. But I hit new auto-negotiation issue between the Plugable adapter and Cisco Catalyst WS-C3560CX-8XPD switch. It took the adapter to eventually negotiate 2.5 Gbps Full duplex around 15 minutes of constantly flapping the interface. Forcing speed and duplex on the Plugable adapter by ethtool did not work. Certainly not ideal, and definitely worth testing before you commit to the Plugable adapter. With other multigigabit adapters, the Plugable had no negotiation issues.
How to force Linux to use the right driver
To enable Full duplex capability, we need to tell Linux to use Realtek r8156 driver instead of the default cdc-ncm.
Compared to the R5S, Topton M6 Mini PC is still portable, but about twice as large. Plastic case wraps the unit, but it is more fragile if you plan to carry it in your backpack or tool bag. There is a built-in fan which is always on. Not a big deal if you use it as an perf3 server, but little inconvenient when it runs on your desk for a longer period of time.
Topton M6 has a single onboard 2.5 Gigabit Ethernet port consistently capable of 2.35 Gbps up and down iperf3 throughput with default settings.
Now, can we make it go faster? Let’s see. We will use USB-A 5 Gigabit Ethernet capable Sabrent adapter. This can either be connected to a USB-A port or USB-C port of the Mini PC. In my tests, I have found that the USB-C port has limited throughput and only tops around 350 Mbps. When I connected the Sabrent 5GbE adapter to USB-C, it only auto negotiated 1 Gbps Full Duxplex.
Use any of the three USB-A 3.1 ports instead to avoid that limitation.
With the USB adapter, the whole setup get less portable. But it allows us to achieve 2.94 Gbps down and 3.27 Gbps up from clients perspective. Is it worth the extra spend? If you need to break the 2.35 Gbps barrier of the built-in 2.5 GbE port, this might be a workable solution for you.
Power adapter with an adapter
This Mini PC is quite strict when it comes to its power source. It requires 12V/2A USB-C PD adapter. Unfortunately, your USB-C MacBook or iPad chargers won’t work.
It draws around 7.5 Watts in idle mode.
If you happen to only use this PC in the US, happy days, as the power adapter ships with US plug. If you select UK during the ordering process, you will receive the US power adapter with UK adapter, which adds to its overall size.
My way around this is to use a standard non-USB-PD 12V/2A adapter with 5.5×2.1mm barrel jack connector, and a barrel jack to USB-C adapter. This particular “power brick” has a standard IEC C14 power cable connector, which you can find in any data centre and with the right European, UK, or Australian plug.
Simply use a USB-C cable and USB PD battery pack capable of delivering 12V/2A. No surprises there.
Powered by PoE
I prefer powering equipment using PoE over local power bricks. If you are in the same boat, you can power this Mini PC by a PoE splitter.
Please pay close attention to the splitter specs. We want the one with a barrel jack and 12V/2A. Since the Mini PC uses USB-C power connector, we will use a barrel jack 5.5×2.1mm to USB-C adapter. Here is the complete setup. Press the power button and voila!
Under the hood
Most of the components are soldered to the main board with little room for upgrades. I ordered the lowest 8GB DDR4 and 128GB NVMe spec with Windows 11 Pro OEM preinstalled (no actual Windows license included).
I was hoping for the Wi-Fi adapter to be replaceable, but it is not the case. It is Intel AX201 and soldered to the board. Good enough, just not ideal for Wi-Fi professionals. M.2 slot would be ideal.
A quick look at the bottom side of the PCB shows the NVMe drive.
If you absolutely need to break the 2.3 Gbps barrier, it can be done with the help of a USB 5 GbE adapter, but it is not very cost effective. The Mini PC cost me £186 including shipping to the UK. The Sabrent 5GbE USB adapter costs around £65.
Finally, it you need top performance, don’t care that much about small form factor, and money is no object, the latest Apple M1 Mac Mini can be configured with built-in 10 GbE.
Wi-Fi standards have developed and also WAN links are fast and reasonably priced these days. When it comes to throughput testing tools like iperf3 servers, 1 Gigabit Ethernet has become a bottleneck. A Wi-Fi 6E client can now easily generate more than 1 Gbps of traffic, but how do we measure it?
To overcome that issue, I am looking for a reasonably priced portable single-board computer, which can push more than 1 Gbps of traffic. It should be powered via USB-C, battery, or PoE powered, and should be portable to fit in my “just in case I need it” tool bag.
FriendlyElec NanoPi R5S
This little FriendlyElec NanoPi R5S single-board computer (SBC) delivers everything I mentioned above. Let’s have a look.
Dimensions and case
It comes with a well designed aluminium case, which also serves as a heatsink. The whole unit is smaller than the smallest iPhone, slightly thicker obviously. It runs silent. There is no built-in fan whatsoever.
It has two 2.5 Gigabit Ethernet interfaces (LAN1 and LAN2) and one 1 Gigabit Ethernet interface (WAN). Either of the LAN ports delivers 2.3 Gbps of actual useful iperf3 throughput with default 1500-byte MTU and single stream. I used MacBook with OWC 10 Gigabit Ethernet Thunderbolt 3 Adapter and Cisco WS-C3560CX-8XPD switch.
The R5S only draws 4 Watts in idle, and can be powered by any USB-C 5V power source. Your MacBook USB-C charger, iPad/iPhone charger, or USB-C battery pack would do. Alternatively, use a 1 Gigabit Ethernet 5V PoE splitter and PoE power the unit. In my lab with a 2 meter cable, the 1 Gigabit Ethernet PoE splitter actually allowed the R5S auto negotiate stable 2.5 Gbps connection with the switch.
FriedlyElec built and published two operating system SD card images for the R5S – Ubuntu and FriendlyWRT. I tested both, and for my use case FriendlyWRT works best. It has a network-centric and easy to use web UI, has iperf3 preinstalled, and delivers great performance.
Connect the WAN port to a network with existing DHCP server. If you are in the same subnet, simply ping FriendlyWrt.local to get the IP address of the R5S.
Then access the web UI or SSH to the unit, SSH is enabled by default. Change the root password now.
Now, this is important! To achieve maximum throughput, delete the pre-configured bridge interface br0, and configure both multigigabit eth1 (LAN1 port) and eth2 (LAN2 port) as standalone unbridged interfaces. Also, tweak IP address settings to your liking while you are there.
Make iperf3 automatically start by going to System > Startup > Local Startup and add iperf3 -s and hit the Save button.
Change CPU Governor setting to Performance. And CPU Minimum Frequency to the maximum value.
This little single-board computer absolutely deserves its space in my tool bag. For the 2 GB RAM model with case I paid $88 including shipping to the UK. Add a Micro SD card and that’s all you need to get started.
Finally, it you need top performance, don’t care that much about small form factor, and money is no object, the latest Apple M1 Mac Mini can be configured with built-in 10 GbE.
Update: The Linux driver for this adapter does not seem to perform great. I tested a 2.5 GbE adapter based on Realtek chipset, which might work better for you. Here is my review.
I also tested this adapter on Windows 11 and macOS here.
With the first consumer Wi-Fi 6E routers already shipping, and enterprise access points being worked on, I think it is now time to up my iperf3 game. While the standard 1 Gbps adapters push around 950 Mbps of TCP traffic, the iperf3 server will sooner or later become a bottleneck for throughput measurements.
Raspberry Pi 4 (RPi4) is widely available, and there is a chance that you might already own one. So, the question is, can it support multigigabit speeds?
Although it does not have any PCI Express slot available, it does have a couple of USB 3.0 ports. I researched USB 3.0 multigigabit NBASE-T Ethernet adapters, and purchased a few. At the time of writing, Linux kernel 5.10 is the Raspberry Pi OS gold standard, and that’s what I used for all tests.
TL;DR … give me the short answer
The maximum TCP throughput Raspberry Pi 4 iperf3 server can handle with a 5 Gbps USB 3.0 Ethernet adapter. These were 90-second iperf3 tests with standard 1500-byte MTU and a single iperf3 stream.
Download (from RPi4 server to a client): 2.05 Gbps
Upload (from a client to RPi4 server): 528 Mbps
If you can enable 9000-byte Jumbo frames on all devices involved in the data path, the upload speed becomes much healthier.
Download (from RPi4 server to a client): 2.05 Gbps
Upload (from a client to RPi4 server): 1.73 Gbps
USB multigigabit adapters
There are a few available on the market. After reading a dozen of reviews, I decided to get 3 adapters from a company called Sabrent. They make adapters with nice metal cases, which helps with dealing with the heat they dissipate.
Left to right: 2.5 GE USB 3.0 Sabrent NT-S25G, 5 GE USB 3.1 Sabrent NT-SS5G, 10 GE Thunderbolt 3 Sabrent TH-S3EA
The 5 GE USB 3.1 Sabrent NT-SS5G is the only I recommend for use with the RPi4:
It uses the Marvell Aquantia AQC111U chipset
It works out of the box with Raspberry Pi OS and
It works on Windows 10 after you install the driver
It ships with USB-C and USB-A cables so you can connect it to your laptop using USB-C or RPi4 using USB-A
Unlike other brands it does not overhead or disconnect due to instability
5 GE USB 3.1 Sabrent NT-SS5G ships with both USB-A and USB-C cables
ServeTheHome team did a great job of comparing the 5 GE adapters using the same chipset as the 5 GE USB 3.1 Sabrent NT-SS5G. It came out as a clear winner:
Why not use the cheaper 2.5 GE USB 3.0 Sabrent NT-S25G? Because it uses Realtek 8156 chipset, and there is no suitable Linux driver available at the time of writing.
Why not the 10 GE Thunderbolt 3 Sabrent TH-S3EA? Although it has a USB-C connector, it is not a USB adapter. It uses Thunderbolt 3 protocol, which is not supported by the RPi4.
I use RPi4 with PoE HAT, because it has a fan on it, and I power the unit by a USB-C charger. Both multigigabit adapters involved in the test are 5-Gigabit Ethernet capable Sabrent NT-SS5G. The best part is that these work out of the box on the Raspberry Pi OS with no action required on your part.
But, if you are considering the purchase of these adapters for your Mac, please stop. After you plug the adapter in, it uses Apple’s 1 Gigabit Ethernet adapter driver, and it would only auto-negotiate 1 Gbps. To enable 2.5 and 5 Gigabit speeds, and support for Jumbo frames on Mac, you have to disable Apple System Integrity Protection (SIP) tool, and install a legacy kext Sabrent driver. I would discourage you from making these security compromises. If you are interested in a multigigabit adapter that works with macOS out of the box, tune in later and read this review (link to be added).
How to increase the MTU and enable Jumbo frames?
On the Raspberry Pi:
sudo ip link set dev eth1 mtu 9000
On the Cisco Catalyst switch running IOS:
The RPi4 allows you to test download-only throughput up to 2 Gbps with standard MTU. Upload speeds are really poor and you would be better off using the built-in 1 Gibabit Ethernet adapter. You can use the RPi4 to run a few other tools, scripts, or take wall attenuation measurements.
With Jumbo frames enabled, 2 Gbps/1.7 Gbps is good enough for lab use or demonstrations. Keep in mind that you would have to enable Jumbo frames on all devices (RPi4, MacBook and the switch in my case).
The main cause of the relatively low performance is the storm of IRQ hammering the RPi4 CPU:
In a computer, an interrupt request (or IRQ) is a hardware signal sent to the processor that temporarily stops a running program and allows a special program, an interrupt handler, to run instead. Hardware interrupts are used to handle events such as receiving data from a modem or network card, key presses, or mouse movements.
If your use case requires a powerful iperf3 server, Apple’s Mac Mini with built-in 10 Gigabit Ethernet adapter would be something to consider today. It won’t be the cheapest option, but you won’t have to worry about performance or USB dongles. From what I’ve found, it uses Marvell AQC 1113 chipset and does 9.4 Gbps with 4 parallel iperf3 streams.
I purchased these adapters myself. No one asked me, or paid me, to write this blog post. I was as curious as you to see how the RPi4 performs when it comes to multigigabit Ethernet.
I am in the market of buying a new pair of power line adapters. Power line is a great alternative or complement to Ethernet and Wi-Fi. It provides low latency and jitter and is very flexible and easy to install.
The current tp-link TL-PA6010 adapters have served me well, but they are now reaching their maximum throughput. So, I decided to get a new pair of the fastest adapters on the market (Devolo Magic 2 Wi-Fi) and also a pair of the best adapters from tp-link (TL-PA9020P). These will be used to connect my home office and lab networks to my router.
Since there are multiple brands offering a variety of products with a variety of advertised speeds, I am curious to see if the more expensive adapters are worth the premium price, what real throughput they would provide and if and how much a passthrough socket improves the power line speed.
I tested my current low-end adapters and two new high-speed ones:
Throughput, ping, jitter, power and Wi-Fi tests
Power line speeds vary and depend on the distance between the two adapters, your electrical wiring and interference. Please take the numbers below as relative ones, which would allow you to compare how these adapters perform under the same conditions and in the same setup.
All throughput numbers below were TCP measurements taken by iPerf3 running on a WLAN Pi (a single-board computer with 1 Gbps Ethernet) and the client was my MacBook with 1 Gbps USB-C Ethernet adapter. There were no intermediate network devices between them:
MacBook iPerf3 client <-> PLC1 <-> PLC2 <-> WLAN Pi iPerf3 server
The average download speed (measured 5 times at each of the locations in my house) ranges from 13% to 26% of the advertised speeds and goes nowhere near them. With £16 per 100 Mbps, the cheapest adapter seems to be the best value for money, unless you need higher speed and are willing to pay for it. It also is the most power efficient.
Devolo Magic 2 proved to the be the fastest solution with 331 Mbps average download speeds, while TL-PA9020P provided slightly better upload speeds than Devolo.
Each of the parameters (i.e. Download average) consisted of five iPerf3 tests in each location and I then computed the average values:
Built-in Wi-Fi access point
Devolo Magic 2 Wi-Fi remote adapter comes with a built-in dual-band 802.11ac Wi-Fi AP (not just a repeater as some of the cheaper adapters), but it is unstable and resets the power line connection every single time I connect and generate some traffic. I used the latest firmware available in July 2020. If a built-in Wi-Fi is a must-have for you, do NOT buy this adapter. Wait until it gets fixed or look for alternatives.
This is what happens. The SSID is broadcast, a Wi-Fi client can associate to the AP, but when the iPerf test starts, the client gets disconnected and power line connection is torn down for 10 seconds or so and then re-establishes. I was able to reproduce this bug every single time and it was not just one-off random problem.
On the positive note, it supports 2.4 GHz only, 2.4 + 5 GHz or 5 GHz only modes. It does not let you change channel width on 5 GHz though and always uses 80 MHz, which may sound like a good idea in a small town, but it is a disaster in a shared building with many other access points and neighbours present.
If high-speed power line without Wi-Fi is what you are after, then the Magic 2 non-Wi-Fi model could be a good option for you.
Passthrough socket allows you to plug an electrical appliance to the power line adapter without generating the socket your adapter is plugged into unusable. Cheaper adapters usually do not provide this.
The other benefit is that adapters with passthrough socket use filters to suppress noise coming from the connected electrical appliance and this improves speed by 13% – 15%.
Pros and cons
Devolo Magic 2 Wi-Fi + Fastest average download speed + Comes with a mobile app and each unit has a management web GUI – Built-in access point resets the whole unit and Wi-Fi is not usable – It runs quite warm compared to the other two and is the largest
tp-link TL-PA9020P + Very good and symmetrical performance + Stable – No built-in Wi-Fi – Still quite expensive compared to the slower and cheaper units
tp-link TL-PA6010 (or similar) + Great value for money + Stable – Relatively low speeds – No passthrough socket, no Wi-Fi
And the winner is…
My personal preferences are very likely different from yours and that is fine. I am looking for symmetrical TCP throughput of at least 200 Mbps, ideally a passthrough socket support and all other features are nice to have.
Devolo Magic 2 Wi-Fi proves to be unstable as the built-in access point crashes the whole adapter and resets the power line connection. Its back side also becomes quite warm regardless the load.
So, I decided for tp-link TL-PA9020P. It is stable, does all I need it to do and both adapters come with 2 Ethernet ports which gives me flexibility to plug my own access point in or connect using wired Ethernet connection.
Dlouho jsem to plánoval a v posledních týdnech konečně zapracoval na rozjetí nového serveru Síťařina.cz. Všechny nové články zaměřené na počítačové sítě najdete právě tady. Nečekejte teoretická pojednání a smršť standardů a citací RFC. Síťařina vás bude zásobovat převážně praktickými články a tipy.
AKTUALIZOVÁNO: S příchodem nového roku 2011 se Síťařina spojila se servery NETGURU a Corporate ICT. Veškeré mé nové příspěvky z oblasti počítačových sítí a podnikových řešení tedy budou vycházet pod hlavičkou jednoho z těchto médií.