OpenWRT on Ubiquiti UniFi USG Pro 4

Update 2021-08-04: Add GitHub links for my fork and individual links to the patched files

Introduction

Due to certain events with Ubiquiti; someone I know asked if it was possible to run OpenWRT on the UniFi Security Gateway 3. As I only have a UniFi Security Gateway 4 Pro, I was curious how difficult it was to add support for either UniFi router. As it turns out, the USG3 was based on the EdgeRouter and the USG4 was based on the EdgeRouter Pro, and OpenWRT already has support for both. So the support is already there but how much effort is it to successfully install?

The Hardware

The UniFi Security Gateway Pro 4 is Ubiquiti’s "high-performance" Gigabit Router. The CPU is a Dual-Core 1 Ghz MIPS64 with Hardware Acceleration for Packet Processing. The System has 2GB of RAM and 4GB of Flash. The USG-Pro has four RJ45 ports and two SFP ports, which are used for the two LAN ports and two WAN port with the 2 WAN ports also usable as SFP. Also on the front faceplate is a RJ45 Console (compatible with CISCO console cables) and a USB 2.0 Port. The default OS is based on a custom Vyos known as EdgeOS. The system is very similar to the Ubiquiti’s EdgeRouter Pro.

NOTE: At this time, the SFP ports are completely non-functional.

The Boot Process

The Bootloader on the USG is u-boot 2012-04-01. For booting OpenWRT, some vital information is needed. The MMC and the Ethernet configurations are needed. The OpenWRT build system has support for the Octeon System-on-Chip, the only code needed was for the board definition, MMC (sysupgrade), and Ethernet Network configuration (for OpenWRT).

Since the USG4 has a Cisco-compatible Console Port, and using the following settings (115768, 8N1), I was able to get a uboot console:

u-boot load
U-Boot 2012.04.01 (UBNT Build Version: e221_002_01aa9) (Aug 17 2018 - 01:13:14)

Skipping PCIe port 0 BIST, in EP mode, can't tell if clocked.
Skipping PCIe port 1 BIST, reset not done. (port not configured)
BIST check passed.
UBNT_E220 r1:1, r2:14, serial #: 788A20446146
MPR 13-02102-14
Core clock: 1000 MHz, IO clock: 600 MHz, DDR clock: 533 MHz (1066 Mhz DDR)
Base DRAM address used by u-boot: 0x8f800000, size: 0x800000
DRAM: 2 GiB
Clearing DRAM...... done
Flash: 8 MiB
Net:   octeth0, octeth1, octeth2, octeth3
MMC:   Octeon MMC/SD0: 0
USB:   USB EHCI 1.00
scanning bus for devices... 1 USB Device(s) found
Type the command 'usb start' to scan for USB storage devices.

Hit any key to stop autoboot:  0
Octeon ubnt_e220#

As U-Boot is a fine bootloader for embedded devices, we can print out the partition table of the MMC:

u-boot mmc configuration
Octeon ubnt_e220# mmc part

Partition Map for MMC device 0  --   Partition Type: DOS

Partition     Start Sector     Num Sectors     Type
    1                 2048          290816       c
    2               292864         3418112      83
    3              3710976         3710976      83
Octeon ubnt_e220#
u-boot mmc filesystem format
Octeon ubnt_e220# fatinfo mmc 0:1
Interface:  MMC
  Device 0: Vendor: Man 450100 Snr 3a6ca11e Rev: 3.10 Prod: SEM04G
            Type: Removable Hard Disk
            Capacity: 3776.0 MB = 3.6 GB (7733248 x 512)
Partition 1: Filesystem: FAT16 "NO NAME    "
Octeon ubnt_e220#

The previous print outs gives us the following information:

  • /dev/mmcblk0p1 is used for kernel storage

    • the uboot only has support for a FAT16 Support to load the kernel

  • /dev/mmcblk0p2 is used for filesystem storage

Some information could be gathered by using the following command:

Octeon ubnt_e220# printenv
autoload=n
baudrate=115200
boardname=ubnt_e220
bootcmd=fatload mmc 0 $(loadaddr) vmlinux.64;bootoctlinux $(loadaddr) numcores=2 endbootargs mem=0 root=/dev/mmcblk0p2 rootdelay=10 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=)
bootdelay=0
bootloader_flash_update=bootloaderupdate
env_addr=0x1fbfe000
env_size=0x2000
ethact=octeth0
flash_base_addr=0x1f400000
flash_size=0x800000
flash_unused_addr=0x1f540000
flash_unused_size=0x6c0000
loadaddr=0x20000000
mtdparts=phys_mapped_flash:640k(boot0),640k(boot1),64k(eeprom)
nuke_env=protect off $(env_addr) +$(env_size);erase $(env_addr) +$(env_size)
numcores=2
octeon_failsafe_mode=0
octeon_ram_mode=0
serial#=788A20446146
stderr=serial
stdin=serial
stdout=serial
uboot_flash_addr=0x1f4a0000
uboot_flash_size=0xa0000
ver=U-Boot 2012.04.01 (UBNT Build Version: e221_002_01aa9) (Aug 17 2018 - 01:13:14)

Environment size: 952/8188 bytes

With some basic intelligence gathering completed, we need to build two artifacts from the OpenWRT source code. At the bottom of this post will be the patches used for support. Building OpenWRT takes a while…​

Next, we need to move the following files to a USB (found in bin/targets/octeon/generic/):

  • openwrt-octeon-ubnt_edgerouter-pro-initramfs-kernel.bin

  • openwrt-octeon-ubnt_edgerouter-pro-squashfs-sysupgrade.tar

Plug-in the USB into the front USB port and boot OpenWRT from the USB via the following commands:

Octeon ubnt_e220# usb start
(Re)start USB...
USB:   USB EHCI 1.00
scanning bus for devices... 2 USB Device(s) found
       scanning bus for storage devices... 1 Storage Device(s) found
Octeon ubnt_e220# fatload usb 0:1 0x20000000 e220/initramfs-kernel.bin
reading e220/initramfs-kernel.bin

17649584 bytes read
Octeon ubnt_e220# bootoctlinux 0 numcores=2 endbootargs mem=0
argv[2]: numcores=2
argv[3]: endbootargs
Allocating memory for ELF segment: addr: 0xffffffff81100000 (adjusted to: 0x1100000), size 0x2129788
## Loading big-endian Linux kernel with entry point: 0xffffffff8174b600 ...
Bootloader: Done loading app on coremask: 0x3
Starting cores 0x3
[    0.000000] Linux version 5.4.132 (snelson@) (gcc version 8.4.0 (OpenWrt GCC 8.4.0 r17131-479a2a90f7)) #0 SMP Mon Jul 19 12:51:22 2021
[    0.000000] Skipping L2 locking due to reduced L2 cache size
....
[    0.000000] Kernel command line: mtdparts=phys_mapped_flash:640k(boot0)ro,640k(boot1)ro,64k(eeprom)ro root=/dev/mmcblk0p2 rootfstype=squashfs,ext4 rootwait console=ttyS0,115200
[    0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 1995312K/2065572K available (6473K kernel code, 346K rwdata, 1560K rodata, 8888K init, 16677K bss, 70260K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     CONFIG_RCU_FANOUT set to non-default value of 32.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=2.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 127
[    0.000000] random: get_random_bytes called from start_kernel+0x370/0x5d0 with crng_init=0
....
[  210.978925] Bootbus flash: Setting flash for 8MB flash at 0x1f400000
[  211.081685] usb 1-1: new high-speed USB device number 2 using ehci-platform
[  211.312689] usb-storage 1-1:1.0: USB Mass Storage device detected
[  211.318985] scsi host0: usb-storage 1-1:1.0
[  212.451530] scsi 0:0:0:0: Direct-Access              Patriot Memory   PMAP PQ: 0 ANSI: 6
[  212.666019] phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0000c9
[  212.676400] Amd/Fujitsu Extended Query Table at 0x0040
[  212.681561]   Amd/Fujitsu Extended Query version 1.1.
[  212.686628] phys_mapped_flash: Swapping erase regions for top-boot CFI table.
[  212.693780] number of CFI chips: 1
[  212.697199] 3 cmdlinepart partitions found on MTD device phys_mapped_flash
[  212.704091] Creating 3 MTD partitions on "phys_mapped_flash":
[  212.709847] 0x000000000000-0x0000000a0000 : "boot0"
[  212.715227] 0x0000000a0000-0x000000140000 : "boot1"
[  212.720551] 0x000000140000-0x000000150000 : "eeprom"
[  212.726978] OF: fdt: not creating '/sys/firmware/fdt': CRC check failed
[  212.892258] sd 0:0:0:0: [sda] 30949376 512-byte logical blocks: (15.8 GB/14.8 GiB)
[  212.901369] sd 0:0:0:0: [sda] Write Protect is off
[  212.907621] sd 0:0:0:0: [sda] No Caching mode page found
[  212.912967] sd 0:0:0:0: [sda] Assuming drive cache: write through
[  212.950514]  sda: sda1
[  212.957002] sd 0:0:0:0: [sda] Attached SCSI removable disk
[  212.965586] Freeing unused kernel memory: 8888K
[  212.970146] This architecture does not have kernel memory protection.
[  212.976633] Run /init as init process
....
[  218.185554] kmodloader: done loading kernel modules from /etc/modules.d/*
[  220.444204] mmc0: new DDR MMC card at address 0001
[  220.449640] mmcblk0: mmc0:0001 SEM04G 3.69 GiB
[  220.454325] mmcblk0boot0: mmc0:0001 SEM04G partition 1 2.00 MiB
[  220.460377] mmcblk0boot1: mmc0:0001 SEM04G partition 2 2.00 MiB
[  220.466400] mmcblk0rpmb: mmc0:0001 SEM04G partition 3 2.00 MiB, chardev (252:0)
[  220.474850]  mmcblk0: p1 p2 p3
....

BusyBox v1.33.1 (2021-07-15 19:39:30 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r17160-32adbfc789
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:/#

Once you have a working OpenWRT console, we can attempt to install. Installation requires mounting a USB and running sysupgrade.

root@OpenWrt:/# mount /dev/sda1 /mnt
root@OpenWrt:/# sysupgrade /mnt/sysupgrade.tar
Image not in /tmp, copying...
Image metadata not found
Saving config files...
Commencing upgrade. Closing all shell sessions.
Watchdog handover: fd=3
- watchdog -
killall: telnetd: no process killed
Sending TERM to remaining processes ... uhttpd ntpd udhcpc odhcp6c fw3 ubusd urngd logd rpcd dnsmasq netifd odhcpd
Sending KILL to remaining processes ...
Switching to ramdisk...
Performing system upgrade...
flashing kernel to /dev/mmcblk0p1
flashing rootfs to /dev/mmcblk0p2
567+1 records in
567+1 records out
Upgrade completed
Rebooting system...
umount: can't unmount /dev: Resource busy
umount: can't unmount /tmp: Resource busy
[  332.085722] reboot: Restarting system

Conclusion

Once the USG Pro 4 is running OpenWRT, installation of additional packages should be possible. The first package installed on my USG is LuCI. Additional information is available in the below links.

Links

GitHub

Patches

OpenWRT Image Build Patch
diff --git a/target/linux/octeon/image/Makefile b/target/linux/octeon/image/Makefile
index f77159e1f9..b950e58b7f 100644
--- a/target/linux/octeon/image/Makefile
+++ b/target/linux/octeon/image/Makefile
@@ -82,4 +82,13 @@ define Device/ubnt_edgerouter-lite
 endef
 TARGET_DEVICES += ubnt_edgerouter-lite

+ERPRO_CMDLINE:=-mtdparts=phys_mapped_flash:640k(boot0)ro,640k(boot1)ro,64k(eeprom)ro root=/dev/mmcblk0p2 rootfstype=squashfs,ext4 rootwait
+define Device/ubnt_edgerouter-pro
+  DEVICE_VENDOR := Ubiquiti
+  DEVICE_MODEL := EdgeRouter Pro
+  BOARD_NAME := erpro
+  CMDLINE := $(ERPRO_CMDLINE)
+endef
+TARGET_DEVICES += ubnt_edgerouter-pro
+
 $(eval $(call BuildImage))
Network Configuration in OpenWRT Patch
diff --git a/target/linux/octeon/base-files/etc/board.d/01_network b/target/linux/octeon/base-files/etc/board.d/01_network
index 194faeaad9..eba0dc06a6 100644
--- a/target/linux/octeon/base-files/etc/board.d/01_network
+++ b/target/linux/octeon/base-files/etc/board.d/01_network
@@ -16,6 +16,11 @@ ubnt,edgerouter-4)
 ubnt,edgerouter-6p)
        ucidef_set_interfaces_lan_wan "lan1 lan2 lan3 lan4 lan5" "lan0"
        ;;
+ubnt,sgpro | \
+unifi,sgpro | \
+erpro)
+ ucidef_set_interfaces_lan_wan "eth0 eth1" "eth2 eth3"
+ ;;
 *)
        ucidef_set_interfaces_lan_wan "eth0" "eth1"
        ;;
OpenWRT config Patch
diff --git a/target/linux/octeon/base-files/lib/preinit/79_move_config b/target/linux/octeon/base-files/lib/preinit/79_move_config
index ae155a3c5c..b12c4048c7 100644
--- a/target/linux/octeon/base-files/lib/preinit/79_move_config
+++ b/target/linux/octeon/base-files/lib/preinit/79_move_config
@@ -25,6 +25,11 @@ octeon_move_config() {
                ubnt,edgerouter-6p)
                        move_config "/dev/mmcblk0p1"
                        ;;
+         ubnt,sgpro | \
+         unifi,sgpro | \
+         erpro)
+                 move_config "/dev/mmcblk0p1"
+                 ;;
        esac
 }
OpenWRT platform Patch
diff --git a/target/linux/octeon/base-files/lib/upgrade/platform.sh b/target/linux/octeon/base-files/lib/upgrade/platform.sh
index 84533d642a..8d66bc8e9f 100755
--- a/target/linux/octeon/base-files/lib/upgrade/platform.sh
+++ b/target/linux/octeon/base-files/lib/upgrade/platform.sh
@@ -33,6 +33,12 @@ platform_copy_config() {
        itus,shield-router)
                platform_copy_config_helper /dev/mmcblk1p1
                ;;
+ ubnt,sgpro | \
+ unifi,sgpro | \
+ sgpro | \
+ erpro)
+         platform_copy_config_helper /dev/mmcblk0p1
+         ;;
        ubnt,edgerouter-4|\
        ubnt,edgerouter-6p)
                platform_copy_config_helper /dev/mmcblk0p1
@@ -97,6 +103,12 @@ platform_do_upgrade() {
        itus,shield-router)
                kernel=ItusrouterImage
                ;;
+ ubnt,sgpro | \
+ unifi,sgpro | \
+ sgpro | \
+ erpro)
+         kernel=mmcblk0p1
+         ;;
        *)
                return 1
        esac
@@ -97,6 +103,12 @@ platform_do_upgrade() {
        itus,shield-router)
                kernel=ItusrouterImage
                ;;
+ ubnt,sgpro | \
+ unifi,sgpro | \
+ sgpro | \
+ erpro)
+         kernel=mmcblk0p1
+         ;;
        *)
                return 1
        esac