Introduction

Wi-Fi routers have always been an attractive target for attackers. When taken over, an attacker may gain access to a victim’s internal network or sensitive data. Additionally, there has been an ongoing trend of attackers continually incorporating new router exploits into their arsenal for use in botnets, such as the Mirai Botnet.

Consumer grade devices are especially attractive to attackers, due to many security flaws in them. Devices with lower security often contain multiple bugs that attackers can exploit easily, rendering them vulnerable targets. On the other hand, there are more secure devices that offer valuable insights and lessons to learn from.

This article gives a technical overview of vulnerabilities in routers for the awareness of security teams and developers, and provides suggestions in ways to avoid making mistakes that could result in such vulnerabilities. We will also look at past vulnerabilities affecting devices of various vendors to learn from their mistakes. Although the following content focuses on routers, the lessons learnt can be applied to other network devices as well.

Disclaimer: This article does not cover all bug classes.

Attack Surface

A router’s attack surface may be larger than one might expect. This is because there are various services running on it. Every service that receives requests from an external host, either on the local-area network (LAN) or wide-area network (WAN) interface, presents an attack surface as malformed requests may execute a vulnerable code path in the service. Below, we briefly explore the common services on routers that process external requests.

Admin Panel

The admin panel of a network device hosts a large variety of configurations that could be changed by the device owner/administrator. Every input field is an attack surface as they are processed by the web service running on the device. For example, there may be an input field for blocking traffic to/from a certain IP address. The web service may handle this request in a way that is vulnerable to command injection. In short, the admin panel presents a huge attack surface to an attacker.

One may argue that this is not a very concerning attack surface, because an attacker would need to authenticate into the admin panel first in order to access this attack surface. This is true, and therefore many CVEs start with the term “authenticated”, e.g. “authenticated command injection”, which states that authentication is needed to exploit the vulnerability. However, an attacker may find an authentication bypass or there may be some endpoints on the admin panel that do not verify if the user is authenticated. An authentication bypass can be chained with an “authenticated vulnerability”; vulnerabilities on endpoints that do not require authentication are categorized with the term “unauthenticated”, e.g. “unauthenticated buffer overflow”.

Other Services

Besides the admin panel, a router usually also runs other services that process requests for various protocols such as FTP, Telnet, Dynamic Host Configuration Protocol (DHCP) or Universal Plug and Play (UPnP). These services also present an attack surface on the router.

Some services such as DHCP or UPnP do not require authentication. Furthermore, on some devices, some services are accessible from the WAN interface, which means that a remote attacker that is not on the local network can also access these services and exploit any vulnerability on them. For services that are so accessible, it is especially important to ensure that they are secure.

Poor Configurations

First, we discuss some configuration mistakes present on some routers. The ones that we will discuss all follow the theme of access, namely

  • Access via hardcoded credentials
  • Access to services from a remote network
  • Access to root privileges by a running service or normal user

Hardcoded Credentials

The firmware of a device contains an archive of programs and configuration files that are used by the device for its operations. The same firmware is distributed to and installed on all devices of the same model. Hence, if the credentials for any service (e.g. FTP or Telnet) running on the device is hardcoded in the firmware, the same credentials will be used by every device.

Impact

An attacker who has access to the firmware, which can usually be downloaded from the vendor’s website, can extract the files and inspect them to obtain the credentials. If such services are exposed to the Internet, anyone from anywhere in the world may be able to gain a shell on the device (Telnet), access sensitive data (FTP), or manipulate the device’s settings (httpd), depending on which service is vulnerable.

In a less dangerous situation, such vulnerable services may not be exposed to the Internet, but just the LAN instead. It is less devious, but anyone in the same network as the device will be able to access these exposed services. In the case of a router, anyone who is connected to its Wi-Fi network can abuse the hardcoded credentials on the exposed services. This is arguably fine for a router in a home network, since everyone that knows the Wi-Fi password is a family member or trusted friend. However, this is precarious for routers in public spaces like cafes or restaurants .

Examples

Below, we share some examples of hardcoded credentials on routers that were reported by others in the past.

CVE-2022-46637 reports that the ProLink PLDT home fiber PRS1841 U V2 router contains hardcoded Telnet and FTP credentials. Furthermore, these services are exposed to the Internet, and vulnerable devices could be found through Shodan.

CVE-2020-29322 details that the telnet service of D-Link router DIR-880L with version 1.07 is initiated with hardcoded username and password. From the advisory, it is not conclusive whether the service is always initiated on startup, and whether the service is exposed to the WAN/Internet. However, if both were to be true, this would serve as an easy backdoor for any attacker to gain a shell on the device.

CVE-2019-14919 is about the Billion Smart Energy Router (SG600R2. Firmw v3.02.rc6) containing a hardcoded password in the login management binary used by Telnet. It is not mentioned in the writeup whether the service is exposed to WAN or LAN only. Besides this, unrelated to hardcoded credentials, a web shell that runs with root privileges is also found in the admin panel.

Suggestions

We recommend that all services that require authentication such as FTP or Telnet be disabled before an administrator sets a strong password for them through the admin panel.

As for the admin panel password, common practice is to randomly generate a password for every device in the factory, and have the password attached as a sticker to the bottom of the device or as a note in the packaging. Upon first login, the administrator would be required to change the password before the admin panel could be used.

Services Exposed to the Internet

As mentioned above, attackers can use hardcoded credentials to access services that are exposed to the Internet instead of just being accessible from the LAN. However, if there are no hardcoded credentials, is it then acceptable for services such as the admin panel or FTP server to be exposed to the Internet?

There are many services that may run on a router, such as FTP, SSH, UPnP, admin panel, VPN. Although the credentials may not be hardcoded, there could be vulnerabilities in these services that may not require authentication and may lead to RCE on the device. As such, it is still safer to not expose all services to the Internet unless necessary.

Suggestions

It is desirable that services like FTP, Telnet, or SSH not be turned on by default, but only upon the device administrator’s request through the admin panel. If an administrator would like to enable a service, he should be required to specify whether the services are to be exposed to the Internet, or just to the LAN. The device should not be exposing these services to the Internet without the administrator’s knowledge or consent. It is even more desirable for the administrator to be fully aware of such risks and willing to bear them before exposing these services to the Internet.

Services Running as root

On a PC, it is common to separate normal users from superusers (root/administrator). However, this is not the case for many consumer routers. Many of them only have the root user, and every service runs as root. On such devices, the FTP/Telnet/SSH/admin panel web services are running as root. So, when an attacker obtains RCE on a service, he also has a shell with root privileges. There is no privilege separation to prevent the whole device from being compromised when a service is exploited.

We list some examples below:

The writeups above show that a root shell could be obtained without a privilege escalation exploit, which means that there is no privilege separation.

Suggestions

Every process/service should be running as a different user, applying the principle of least privileges. For example, the httpd service runs as the web user; the upnpd service runs as the upnp user; the ftp service runs as the ftp user, and so on. Under this configuration, when a service is exploited, the attacker cannot compromise other services or the whole device without a privilege escalation exploit.

Password-less sudo?

On some of the routers that we have inspected, they do not run their admin panel web service as root but a normal user. This is good. However, to perform some system-level operations, they want to make use of shell commands which require root privileges. For example, the iptables command.

Bad Example

Consider the scenario where the admin panel supports blocking traffic to and from a certain IP address using iptables (requires superuser privileges). To achieve this, developers may be tempted to introduce a SUID binary that works like sudo but does not need a password. In other words, a SUID binary that takes a command string as argument and runs that command string as a shell command with root privileges. A simple example of this is shown below (let’s refer to it as custom_sudo in the following examples):

int main(int argc, char** argv)
{
  setuid(geteuid());
  system(argv[1]);
  return 0;
}

Then, the developers may use custom_sudo to run an iptables command as follows.

snprintf(cmd, 255, "iptables -A INPUT -s '%s' -j DROP", ip_addr_to_block);
execve("/usr/sbin/custom_sudo", { "/usr/sbin/custom_sudo", cmd, 0 }, 0);

Obviously, having such a program defeats the purpose of having different users, when all users can just use this program to run any commands as root, violating the principle of least privileges.

Suggestions

It is not recommended that operations requiring root privileges be executed by means of providing a whole command string to such a SUID program as sudo or custom_sudo. Instead, we suggest having a SUID program that only takes in the necessary values as arguments and then use them in carrying out the desired operations.

For the example above on blocking an IP address, we suggest having a SUID program called block_ip_addr that just takes in the IP address as an argument, then performs the iptables operation with the given IP address string internally. For example:

execve("/usr/sbin/block_ip_addr", { "/usr/sbin/block_ip_addr", ip_addr_to_block, 0 }, 0);

Then, the implementation of block_ip_addr can be as follows:

char* ip_addr_to_block = argv[1];
execve("/bin/iptables", { "/bin/iptables", "-A", "INPUT", "-s", ip_addr_to_block, "-j", "DROP", 0 }, 0);

If this is done this for all commands that require root privileges, the custom_sudo program is no longer necessary and can be removed entirely from the system.

However, if a developer strongly insists on using such a custom_sudo program, please at least verify that the program that will be executed is among a list of allowed programs. For example:

if (strncmp(argv[1], "/bin/iptables ", strlen("/bin/iptables "))) {
  // if strncmp returns a non-zero value, the command string does not start with the expected `iptables `
  // abort
  ...
}

Note that in the check above, /bin/iptables ends with a space character. This is to make sure that the command string indeed is running the /bin/iptables program, and not something like iptablesfake. Also, use the absolute path (e.g. /bin/iptables) and not just the program name (e.g. iptables), to prevent the invocation of the wrong program, as the program path is determined according to the PATH environment variable. For example /home/user/iptables created by an attacker may be executed instead of /bin/iptables if the PATH variable is configured as so.

Also, make sure that there are no command injection vulnerabilities that could be exploited to bypass such checks. This could be done by running the desired command with execve instead of system.

In short, when considering the implementation of operations that require superuser privileges, the principle of least privileges should be followed to ensure that the chosen implementation does not provide a user with more privileges than necessary.

Summary

All of the misconfigurations above have straightforward solutions. However, the implementation of these solutions incur extra development and testing time. For the consumers, it is desirable that all router vendors consider these improvements as “must have” and not just “good to have”.

Vulnerability Classes

In this section, we will discuss the following vulnerabilities affecting services running on routers.

  • Authentication Bypass
  • Command Injection
  • Buffer Overflow
  • Format String Bug

Command injection and buffer overflow are the two main vulnerability classes present in routers that lead to RCE. Format string vulnerabilities may also result in RCE, but are very rarely seen in the recent years.

Authentication Bypass

The attack surface of a service is greatly reduced if the service requires authentication. Typically, an unauthenticated client would only be able to send requests related to authentication, or for querying some information about the service or the device. Therefore, an authentication bypass vulnerability is valuable to attackers as it opens up the whole remaining attack surface.

Besides that, even without RCE, a non-administrator could still perform an authentication bypass to disclose and control sensitive settings on the admin panel. An authentication bypass on other services that require authentication such as FTP, SSH or Telnet could also lead to shell access or disclosure of sensitive information. Hence, authentication bypass should not be taken lightly.

Examples

In the following sub-sections, we share examples of authentication bypass bugs on routers that were reported in the past.

CVE-2021-32030: Mistake in authentication logic

CVE-2021-32030 reports an authentication bypass on the ASUS GT-AC2900 router. In short, the vulnerability arises due to a mistake in the authentication flow as simplified below:

  1. A header field asus_token should be supplied by the client for authenticating into the admin panel.
  2. asus_token is compared with the ifttt_token value retrieved from *nvram.
  3. If nvram does not contain ifttt_token, it returns an empty string, i.e. a null-byte.
  4. If asus_token is a null byte, the comparison succeeds and the client is authenticated successfully.

*nvram stands for non-volatile RAM. It is used by many routers to store configuration values that should persist over device reboot. Hence the term non-volatile, as the contents persist, unlike for normal RAM in which all its contents will be cleared when the device is turned off.

Suggestions

In this case, the vulnerability is caused by programmer error, in which an unexpected edge case input breaks the authentication logic. The relevant function should first ensure that ifttt_token is not an empty string, before comparing it with the client-supplied asus-token. To be extra careful, the developers may also add an additional check to ensure that asus-token is not an empty string, in case it may be compared with another token that is also retrieved from nvram in the future.

CVE-2020-8864: Mistake in authentication logic

CVE-2020-8864 reports an authentication bypass on the D-Link DIR-882, DIR-878 and DIR-867 routers. The exploit is the same as the one above in the ASUS router, although the implementation of authentication logic is different, as simplified below:

  1. LoginPassword is provided by the client for authentication.
  2. strncmp is used to compare the client-supplied password with the correct password, as shown below: 
    strncmp(db_password, attacker_provided_password, strlen(attacker_provided_password));
  3. If an attacker submits a login request with an empty password, strncmp will have 0 as its 3rd argument (length), and that returns 0, meaning the two strings compared are equal, which is correct because the first 0 characters of both strings are the same. As a result, authentication is successful.
Suggestions

Again, the vulnerability is caused by programmer error. The fix here is by passing strlen(db_password) as the 3rd argument, instead of strlen(attacker_provided_password).

CVE-2020-8863: Expected password value is controlled by attacker

CVE-2020-8863 reports another authentication bypass on the D-Link DIR-882, DIR-878 and DIR-867 routers.

The subsection above on CVE-2020-8864 was simplified by omitting some details about the authentication flow. Here, we describe it in detail so that we can accurately describe this authentication bypass later.

These routers use the Home Network Administration Protocol (HNAP), a SOAP-based protocol for the requests on the admin panel from the client to the web server. The authentication process is as follows:

  1. The client sends a request message and obtains an authentication challenge from the server.
  2. The server responds to the request with the values: Challenge and PublicKey.
  3. The client should combine the PublicKey with the password to create the PrivateKey. Then, use the PrivateKey and Challenge to generate a challenge response that is to be submitted as LoginPassword to the server.
  4. The server will perform the same computations, and if the LoginPassword matches, it means that the client knows the correct password, and authentication succeeds.

The description above is taken from this writeup on the ZDI blog. Check it out for more details and code examples.

In the web server binary, it is discovered that there is a code path that checks a PrivateLogin field in the login request. It is as follows:

//  If PrivateLogin != NULL && PrivateLogin  == "Username"  Then Password = Username
    if ((PrivateLogin == (char *)0x0) || (iVar1 = strncmp(PrivateLogin,"Username",8), iVar1 != 0)) {
      GetPassword(Password,0x40);  // [1]
    }   
    else {
      strncpy(Password,Username,0x40);  // [2]
    }   
    GenPrivateKey(Challenge,Password,Publickey,PrivateKey,0x80);

If the submitted PrivateLogin field contains “Username”, then the submitted Username value is used as the password ([2]) for generating the expected LoginPassword value (challenge response), instead of using the user’s actual password by calling GetPassword ([1]). To put it in simple terms, the client can control the password that is used to generate the expected challenge response, therefore bypassing authentication.

It is unclear what the purpose of the PrivateLogin field is. As HNAP is an obsolete proprietary protocol with no documentation online, it is hard for us to determine the original purpose of this field.

As a takeaway, ensure that when implementing an authentication protocol, secrets such as passwords should not be mixed with values submitted by the client.

Summary

Authentication bypass vulnerabilities on the admin panel arise from programmer mistakes, as they fail to consider edge case inputs such as empty passwords that may break the authentication logic. Besides that, there may also be flawed implementations of a protocol. Special attention should be given to reviewing the implementation of an authentication routine to catch unintended bypasses due to such mistakes.

Command Injection

Command injection is a commonly seen vulnerability in routers or other network and IoT devices. In this section, we discuss the vulnerable code pattern, reason behind the ubiquity of such vulnerability, guidelines to prevent them, and some examples of them in various routers in the past.

Root Cause

Command injection is possible because a command string that contains unsanitized user input is executed as a shell command, by means of system or popen in C, os.system in Python, os.execute or io.popen in Lua. For example,

sprintf(cmd, "ping %s", ip_addr);
system(cmd);

os.system(f"ping {ip_addr}")

In the examples above, an input IP address such as 127.0.0.1; reboot will result in a command string of ping 127.0.0.1; reboot, which escapes the intended ping command and executes arbitrary commands such as reboot.

Rationale for using shell commands

Running shell commands to perform various system-level operations is definitely not the norm in software development. For performance and compatibility reasons, in software running on personal computers, it is very rare to see system-level operations such as filesystem or network operations being carried out through running shell commands. However, this is almost ubiquitous in the world of embedded devices such as routers.

The reasons behind this phenomenon are somewhat acceptable. In routers, performance is not a concern because said operations are not very frequently performed. Compatibility is also not affected because programs only run on the vendor’s own devices. Without such factors in mind, it is tempting to find the simplest way to implement a required feature, and such simplest way may be flawed in security.

For example, look at the following function from [NETGEAR’s pufwUpgrade binary](https://research.nccgroup.com/2024/02/09/puckungfu-2-another-netgear-wan-command-injection/#:~:text=int-,saveCfuLastFwpath,-(char%20*).

int saveCfuLastFwpath(char *fwPath)
{
    char command [1024];
    memset(command, 0, 0x400);
    snprintf(command, 0x400, "rm %s", "/data/cfu_last_fwpath");
    system(command);
    // Command injection vulnerability
    snprintf(command, 0x400, "echo \"%s\" > %s", fwPath, "/data/cfu_last_fwpath");
    DBG_PRINT(DAT_0001620f, command);
    system(command);
    return 0;
}

There are proper C library functions for deleting a file (unlink), as well as for writing to a file (fopen and fwrite). But the developers instead chose to create shell command strings starting with rm and echo and run them with system. Admittedly, shell commands are easier to remember and use without needing to refer back to the C documentation.

Another such example is when a router firmware developer inserts a user-entered password into a command string to calculate the password hash using md5sum. It takes more time and effort to write C code that achieves the same goal.

To compensate for such reduced effort, at least some device vendors do sanitize their inputs before inserting them into a command string and calling system. This is good. However, it just takes a small mistake, such as forgetting to sanitize an input field, to introduce a command injection vulnerability allowing RCE on the device. A vulnerability may also be introduced due to certain manipulations performed on the command string, or some unexpected reasons due to the way the command is written. For example, CVE-2024-1837 for which we will publish the advisory soon. Found by me :)

From my limited exposure, I have noticed that the more expensive Cisco and ASUS routers do not take shortcuts by performing OS-level operations through shell commands, but instead they properly implement them with the corresponding API functions. If they were to execute external programs with user inputs, they only use safe functions such as execve that are not susceptible to command injection. With such efforts, they eradicate even the smallest possibility of command injection on the admin panel.

In the next section, we share some suggestions to prevent command injection in the event where external programs need to be executed.

Prevention

In this subsection, we share secure code design guidelines to prevent command injection. First of all, as mentioned repeatedly in the subsections above, not all operations need to be performed through shell commands, so please avoid doing so unless necessary.

Avoid system commands

The decision to run external programs using a shell command is the cause for potential command injection bugs. On top of just recommending developers to not write such code, security teams could help them avoid the usage of functions that run shell commands by raising warnings where such functions are called.

We list below such functions that should be avoided from the codebase. Note that the list is not exhaustive. Security teams should check if there are other such functions supported by the standard library or any third party libraries imported by the codebase.

  • C: system, popen
  • Python: os.system, subprocess.Popen/subprocess.call/subprocess.run with shell=True argument

Do not worry about whether such a rule may break compatibility, because it won’t. We provide the safe alternative below, which can do the same things that a shell command can do.

Run executable with argument list

There is a safe way to run external scripts or binaries, by specifying a program and providing an argument list, by using execve in C or subprocess.Popen/subprocess.run/subprocess.call (without the shell=True argument) in Python. For example,

execve("/bin/ping", { "/bin/ping", ip_addr, 0 }, 0);

subprocess.Popen(["/bin/ping", ip_addr])

This eliminates the possibility of command injection on the command directly. However, beware that this does not make any guarantees about whether there will be command injection in the target executable, for example /bin/ping as in the example above.

Note that in C, execve replaces the current running process with the target executable. That is, if execve is called with /bin/ping by the admin panel server, the whole service will be gone, replaced by ping. This is certainly not the intended behaviour. Remember to fork the process before calling execve.

However, watch out for code as in the example below. It defeats the purpose since it runs a shell command again.

sprintf(cmd, "ping %s", ip_addr);
execve("/bin/sh", { "/bin/sh", "-c", cmd, 0 }, 0);
Custom execve

In languages such as Lua, there may not be a library function such as execve for running a specific program with an argument list. In such unfortunate scenario, there is no choice but to use the system or popen equivalent that is available in this language. In Lua, that is os.execute.

To protect the developers from crafting command strings prone to command injection, the development team may create a function similar to execve that takes in an executable path and argument list, then crafts the command string with these values, and passes it to system for execution. The executable path can be concatenated together with all the arguments, but there are two important things to take note:

Wrap every argument with single quotes. This is to prevent command substitution, because in a shell command, contents within single quotes will be passed verbatim as an argument. With single quotes, any sequence of characters in the form $(...) or `...` will not be evaluated. Do not wrap the arguments in double quotes. Command substitution will still apply for contents within double quotes.

Escape the single quotes in every argument. If an argument contains single quotes, it will close the opening single quote before it, and any command substitution payload after it will be evaluated. Make sure all single quotes in every argument are escaped by prepending them with a backslash character, so that they do not close the opening single quote before the argument.

The example below demonstrates how the operations above could be implemented in Lua.

function custom_execute(executable_path, args)
    -- Escape single quotes within arguments
    local escape_single_quotes = function(str)
        return string.gsub(str, "'", "\\'")
    end

    -- Quote and escape each argument
    local quoted_args = {}
    for _, arg in ipairs(args) do
        table.insert(quoted_args, "'" .. escape_single_quotes(tostring(arg)) .. "'")
    end

    -- Concatenate executable path and quoted arguments
    local command = executable_path .. " " .. table.concat(quoted_args, " ")

    -- Execute the command using os.execute
    os.execute(command)
end

-- Example usage
local echo_path = "/bin/echo"
local args = {"hello", "world", "hey"}
custom_execute(echo_path, args)

The custom_execute function above removes the possibility of command injection, regardless of any user input that may be part of the argument list. Such a function gives developers a peace of mind when it is necessary to execute external programs, and removes the burden from them to consider any sanitization that is required for the user input.

Avoid eval in shell scripts

The suggestions above are applicable to services that receive input from a client request and use this input value in the execution of another program on the system. The protections above ensure that handlers for client requests are safe from command injection. However, they do not make any guarantees about the safety of the external program that is executed.

Consider the following example where /usr/sbin/custom_script is a shell script that is given a user input value as an argument. There is no command injection in executing the script. However, there could be command injection within the script that is being executed.

execve("/usr/sbin/custom_script", { "/usr/sbin/custom_script", user_input, 0 }, 0);

Consider the following shell script that inserts an argument ($1) into a command string and executes it in various ways.

  1. Using eval.
  2. Using $(...) (command substitution).
  3. Using `...` (command substitution).
#!/bin/sh
cmd="echo $1"

files=`eval "$cmd"`
echo $files

files=$($cmd)
echo $files

files=`$cmd`
echo $files

The output of the script above when executed with aaa;whoami as an argument is as follows.

$ ./script 'aaa;whoami'
aaa user        // command injection occured
aaa;whoami
aaa;whoami

Notice that when command substitution (2nd and 3rd example) is performed, there is no command injection. This is because the argument $1 is passed as an argument to the echo program as written in cmd. This means that the whole argument string containing aaa;whoami is passed to echo as an individual argument.

On the other hand, in the case of eval, $1 is interpolated, that is, expanded as a string and inserted into the command string, resulting in echo aaa;whoami being the command that is executed. Command injection is present in this case, as seen in the output attached above.

Therefore, avoid the usage of eval in shell scripts, to prevent any potential command injection vulnerabilities due to mishandling of arguments which may come from user input.

Actionable Steps

To summarize the suggestions above, we recommend development and security teams to impose the following rules on their codebase:

  1. Avoid dangerous functions that execute a command string directly, e.g. system, popen, eval. Only allow the execution of an external program by passing an argument list.
  2. A thorough review should be conducted to decide whether a function is safe or necessary.
  3. If an unsafe function is necessary (such as os.execute in Lua), use custom wrapper functions that call the function in a safe manner. For example, the custom_execute wrapper for os.execute in Lua shown above.

Examples

In the following sub-sections, we show examples of command injection in various routers, in implementations using different programming languages, in various services, to show that this vulnerability can manifest itself under different contexts.

In 2021, I discovered some command injection vulnerabilities in the DIR-1960/1360/2660/3060 and DIR-X1560 devices. The vulnerabilities were found in code that handles web requests from the admin panel.

Some were due to complete lack of sanitization of user input, inserting them in command strings to run programs like sendmail, smbpasswd and iptables. Such code is written as operations related to email or SMB can be rather complicated to implement, and a simpler solution would be to use the relevant programs that are already present on the system. The code for running such commands with user input had insufficient or no sanitization performed on the input values, resulting in command injection attacks being possible through the corresponding web endpoints.

Failed validation of IP range string

There was a rather interesting case of flawed input validation done before inserting a user input string into an iptables command string. The user input value is an IP address range, e.g. 123.123.123.123/24. The handler function did check if the user input follows the a.b.c.d/subnet format, but not correctly.

  1. It calls the inet_addr C function to verify that the front part (a.b.c.d) is a valid IP address.
  2. Then, it calls strtolon the part after the / (subnet) to check if it is a positive number.
  3. On first glance, this is useful because a string that starts with alphabets or symbols will result in 0 being returned.
  4. However, a string like 16 abc will let strtol return 16 which is considered valid.

As a result, a user input like 123.123.123.123/16 $(reboot) will successfully perform command injection when it is inserted into an iptables command string.

This is an example of failed validation that could be potentially caused by an incomplete understanding of how library functions such as strtol works.

Zyxel (Python)

Now, let’s look at the Zyxel NAS whose web management interface runs on Python. This is not a router, but serves as a good case study.

CVE-2023-4473 was reported by IBM X-Force regarding an OS command injection in the following form:

mail_hour = pyconf.get_conf_value(MAINTENANCE_LOG_MAIL, 'hour')
mail_minute = pyconf.get_conf_value(MAINTENANCE_LOG_MAIL, 'miniute')
cmd = '/usr/sbin/zylog_config mail 1 schedule daily hour %s minute %s' % (mail_hour, mail_minute)
os.system(cmd)

The values mail_hour and mail_minute are obtained from user input as provided in the POST request below.

curl -s -X POST \
  –data-binary 'schedulePeriod=daily&scheduleHour=0&scheduleMinute=0%60cmd60' \
  'http://10.20.17.122/cmd,/ck6fup6/zylog_main/configure_mail_syslog'
Fix

os.system should never ever be used with a string that contains user input. As suggested in the earlier section, use subprocess.Popen instead, by providing the executable path and its arguments as an argument list. Doing so removes the possibility of command injection because the command string is no longer executed as a shell command. For example:

cmd = '/usr/sbin/zylog_config mail 1 schedule daily hour %s minute %s' % (mail_hour, mail_minute)
subprocess.Popen(cmd.split(" "))

TP-Link’s routers use a fork of OpenWrt’s LuCI, a configuration interface based on Lua, as the backend for their admin panel.

Their Lua source code is compiled to bytecode and stored on the router, and the code is invoked according to the requests made by the client. Any security researcher who is interested in analyzing the admin panel backend code would need to decompile the Lua bytecode. Decompiling Lua bytecode is not as difficult a task as decompiling binaries compiled from C, as bytecode-based languages such Lua, Python, or Java contain more information that make them easier to decompile. There is an open source Lua decompiler luadec.

However, as mentioned, TP-Link uses a fork of LuCI, and they made some changes to it, including the underlying Lua compiler. According to this article Unscrambling Lua, TP-Link changed the bytecode opcodes emitted by its Lua compiler so that luadec would not work properly. One would have to reverse engineer the changes made, and apply the same changes to luadec, to have a working decompiler for TP-Link’s LuCI’s bytecode. There is another open source project luadec-tplink by superkhung which is not perfect, but works somewhat okay for reverse engineering TP-Link’s request handlers stored in bytecode form.

ZDI-23-451/CVE-2023-1389 reports an unauthenticated command injection that could be exploited by attackers on the LAN interface to gain RCE. The vulnerable endpoint is responsible for setting the admin panel’s locale, in particular through the country parameter.

POST /cgi-bin/luci/;stok=/locale?form=country HTTP/1.1
Host: <target router>
Content-Type: application/x-www-form-urlencoded

operation=write&country=$(id>/tmp/out)

An RCE can be gained from a request to change the country setting without requiring authentication.

The situation is made worse by ZDI-23-452/CVE-2023-27359 which is a race condition vulnerability in the firewall service hotplugd, that allows an attacker to access the vulnerable endpoint above through the WAN.

Suggestions

Although we do not have the code, it is very likely that the country parameter was inserted into a command string, then passed to either os.execute or io.popen, resulting in a command injection vulnerability. As recommended in the earlier section, it is best to create a wrapper for os.execute that wraps all arguments with single quotes and escapes all single quotes within them.

On this vulnerable router, LuCI may be running as root (according to the linked Tenable advisory), allowing any injected commands to be executed as root. As described in Services Running as root above, it is a poor practice to run any service with root privileges, as that would mean a compromise of the whole device once that service is exploited.

Remarks

TP-Link may obfuscate their Lua bytecode to prevent others from reverse engineering their code. This adds extra work for not only threat actors, but also security researchers in detecting vulnerabilities in their devices. It may form an illusion that these devices are secure when in reality there were just very few people who spent time inspecting these devices.

DHCP server (C)

Command injection is not limited to just the admin panel. Earlier, our team discovered a command injection vulnerability in the DHCP server of the NETGEAR RAX30 as shared in this writeup.

The vulnerable code is as follows:

int __fastcall send_lease_info(int a1, dhcpOfferedAddr *lease) 
{
// truncated...
  if ( !a1 )
  {
// truncated ...
    if ( body.hostName[0] )
    {
      strncpy((char *)HostName, body.hostName, 0x40u); // [1]
      snprintf((char *)v11, 0x102u, "%s", body.vendorid);
    }
    else
    {
      strncpy((char *)v10, "unknown", 0x40u);
      strncpy((char *)v11, "dhcpVendorid", 0x102u);
    }
    sprintf(
      command,
      "pudil -a %s %s %s %s \"%s\"",
      body.macAddr,
      body.ipAddr,
      (const char *)HostName,  // [2]
      body.option55,
      (const char *)v11);
    system(command);   // [3]
  }
//...
}

At [1], the hostName parameter of the DHCP request body is copied into HostName, then inserted into command at [2], and executed by system at [3]. There is no sanitization done on this user input.

Patch

The vulnerability was fixed by calling execve instead of system to execute the command.

Buffer Overflow

Buffer overflow is a common issue for programs written in the C programming language, and most services in routers are written in C. Buffer overflow vulnerabilities could be exploited by an attacker to gain RCE on the underlying device.

Over the years, buffer overflow has gotten increasingly difficult to exploit due to mitigations such as ASLR, PIE, RELRO, stack canary and NX. ASLR is enforced by the OS, while PIE, RELRO, stack canary and NX are protections added by the compiler by default. However, in the recent years, many routers are still observed to lack such mitigations. This could be due to the vendors’ still using very old versions of GCC (or other compilers) in their deployment process, which could be missing the ability to add said mitigations to the resulting binary.

With the mitigations listed above, attempts to exploit a buffer overflow vulnerability could be prevented most of the time, unless the vulnerability satisfies conditions that are favorable for exploitation. However, in successfully preventing the exploitation attempt, the mitigations will halt the running service because its internal state has been corrupted. This results in a DoS which is also not desirable. Hence, knowing that the mitigations do not guarantee full protection against all exploitation attempts, it is still most preferable that buffer overflow vulnerabilities are avoided through secure coding practices and design, which we will discuss in detail in this section.

Root Cause

The root cause of a buffer overflow bug is straightforward. A buffer is allocated a certain size, but data longer than that size is written into the buffer. As a result, other values in adjacent memory are corrupted with user-controlled values.

In routers, this mistake is commonly observed in the usage of functions that copy memory contents. We list examples in the code snippet below. For the examples below, suppose that websGet is a function that takes in a key and returns the corresponding value in the query string of an incoming web request of the router admin panel.

char* contents = websGet("contents");
int size = websGet("size");

char buffer[128];

strcpy(buffer, contents);
strncpy(buffer, contents, size);

sprintf(buffer, "%s", contents);
snprintf(buffer, "%s", contents);

buffer[0] = 0;
strcat(buffer, contents);
strncat(buffer, content, size);

memcpy(buffer, contents, size);

In the code snippet above, strcpy, sprintf and strcat copy the whole user-supplied contents string into buffer without restricting the length. If the length of contents exceeds the size allocated for buffer (i.e. 128), buffer overflow occurs.

The example above also assumes a scenario in which the user also specifies the size of contents to copy through strncpy, snprintf, strncat and memcpy. Similarly, if size exceeds 128, buffer overflow occurs.

Aside from functions that perform copying, code for manually copying memory contents can also be vulnerable to buffer overflow, as shown in the example below, in which the size field is user-supplied and could be greater than the size allocated for buffer.

for (int i = 0; i < size; ++i) buffer[i] = contents[i];

All of the vulnerable code examples above share a common theme. They do not restrict the size of the contents being copied/written. Implicitly, they had allowed the user (attacker) to choose the size. To prevent buffer overflow, do not use functions that do not restrict the length, e.g. strcpy, sprintf and strcat. Instead, use strncpy, snprintf, strncat or memcpy, and make sure that the length argument is not a user-controlled value.

The examples above are contrived, as they are just intended for demonstrating the insecure code patterns. In a more complex codebase, a user-submitted value may be stored and retrieved and manipulated repeatedly, at various locations in the code, before finally being used in the copying operation. Under such situations, it can be difficult to ensure that all such memory-writing operations are immune to buffer overflow. We provide suggestions below in the form of secure design and practices to systematically protect your code against buffer overflow bugs.

Prevention

Buffer overflow bugs can be considered as caused by human mistakes. The complexity of a codebase increases the likelihood of the occurrence of such mistakes. Through careful design and restrictions imposed on the codebase, we can do our best to protect developers against unintentionally introducing buffer overflow bugs into the program.

Use bounded functions for copying

The usage of memory-copying functions without a length limit such as strcpy, strcat and sprintf should not be allowed in the codebase. Instead, use the bounded alternatives such as strncpy, strncat, snprintf or memcpy. There is no imaginable scenario where the unbounded functions (e.g. strcpy) will be more useful than their bounded alternatives (e.g. strncpy).

Be mindful that when using functions like strncpy or memcpy, do not use strlen to determine the length to copy, as listed in the example below, as this is no different from just calling strcpy. The length argument should be independent of the user input, but based on the allocated size of the destination buffer instead.

char buffer[128];
// bad
strncpy(buffer, contents, strlen(contents));
// good
strncpy(buffer, contents, 128);
strncpy(buffer, contents, sizeof(buffer));
Pass buffer size as function argument

Even when developers put in conscious effort to ensure that memory is only copied into within a buffer’s bounds, there is another challenge: it is difficult to know what exactly is the size allocated for a buffer. The following example illustrates this problem.

void get_name(char* buf)
{
  char* name = websGet("name");
  memcpy(buf, name, ???);
}

void f2(char* buf) { get_name(buf); }
void f3(char* buf) { f2(buf); }

void store_input() {
  char* buf = (char*) malloc(64);
  f3(buf);
}

In the example above, store_input allocates 64 bytes for buf. Then, buf is passed through f3 then f2 to get_name which copies a user-provided string into it. In get_name, it is no longer obvious what was the size allocated for buf. The developer would need to find references to get_name, see that it is called by f2, then again find references to f2, see that it is called by f3, then finally get to store_input and learn that the allocated size is 64. If there are way more functions calling get_name, f2 or f3, this would be madness. The developer would have to make sure that the length specified in get_name is safe for all its callers.

Furthermore, any changes made to the allocation size in store_input also has to be made to get_name. If the developer modifying store_input was not aware of get_name, he will miss this out, resulting in get_name calling memcpy with the wrong length. In general, it is bad practice to have values serving the same purpose hardcoded in different locations.

One may consider having a global size constant SIZE that is used by the allocation in store_input and memcpy in get_name. This works well, if get_name is ever only given a buffer that is allocated SIZE bytes. This may be the case in the short term. However, could this still hold 2 years later if most of the development team has changed? For example, a new developer may decide to call get_name with a buffer of a smaller size, without being aware of the memcpy length.

We suggest designing a codebase that is resilient to the changes above, that is, by passing the destination buffer’s allocated size as a function argument. The following code snippet shows how this can be applied on the example above.

void get_name(char* buf, size_t size)
{
  char* name = websGet("name");
  memcpy(buf, name, size);
}

void f2(char* buf, size_t size) { get_name(buf, size); }
void f3(char* buf, size_t size) { f2(buf, size); }

void store_input() {
  size_t size = 64;
  char* buf = (char*) malloc(size);
  f3(buf, size);
}

In doing so, there is no room for any uncertainties in the size when memcpy is called by get_name. It is guaranteed to get_name that the size given to memcpy must be the allocated size for the buffer. When a developer works on the code in get_name, he does not need to check all of its callers to ensure that the size is suitable. Similarly, a developer working on store_input also does not need to worry whether f2, f3 or get_name will write out of bounds, assuming that the developers of those functions have used the provided size argument correctly.

Also, note that store_input passes the same size variable to malloc and f3, instead of hardcoding the value 64 in both function calls. This ensures that when the allocation size is changed, the change will immediately apply to both malloc and f3. Such practice removes the possibility of mistakes.

For code reviewers, potentials bugs are also easier to detect. In the original example, the reviewer would have to follow the flow from store_input to get_name to ensure that the size given to memcpy is within bounds. In a big codebase, there may be tens or hundreds of such flows to review whenever the implementation of a function such as get_name changes.

In this improved implementation, the reviewer just needs to ensure that functions like get_name correctly uses the size argument that is given to it, and ensure that functions like store_input provide the correct size.

Caveat: strncpy

There are caveats for using strncpy and strncat. We will discuss strncpy first.

Note that for strncpy, if the requested length to copy is smaller than the length of the source string, the copied string will not be null-terminated. Consider the following example.

char* hello = "HELLO WORLD";
char dest[10];
memset(dest, '\xAA', 10);
strncpy(dest, hello, 5);

// to print contents of `dest` in hex
for (int i = 0; i < 10; ++i) printf("%hhx ", dest[i]);
printf("\n");

The result of running the code above is as follows.

48 45 4c 4c 4f aa aa aa aa aa

As strncpy was given 5 as the length to copy, it correctly copies 5 characters, and does nothing more than that, such as adding a null byte to terminate the destination buffer. The string in dest is therefore not null-terminated as shown in the program output above.

The consequences of this may not be directly observed. By itself, there is no memory corruption, because nothing is read or written out of bounds. However, if a future operation that uses dest assumes that it is null-terminated, but in reality it may not be so, unexpected behaviour may occur. Referring to the example above, if strlen was applied on dest, it does not return 5 because there is no null-byte at the 6th position (i.e. right after the 5th position), even though it is expected to return 5. This discrepancy may affect subsequent operations in unpredicted ways.

Another example would be a scenario where the contents in dest were to be copied as part of the service’s response to the client. Similarly, as dest was not null-terminated, the program may copy adjacent memory contents (contents of other variables) or leftover memory contents in the buffer written by previous operations (i.e. \xaa\xaa\xaa in the example above). This constitutes an information leakage bug. Since there is no out-of-bounds memory write, there is no danger of RCE. However, sensitive values could be leaked, for example pointers to bypass ASLR, or secrets like passwords to bypass authentication.

In a more severe scenario, a buffer overflow may be possible too, although unlikely if proper precautions were already enforced to ensure that memory-copying operations do not rely on the position or presence of the null byte, as per the advice given above.

Custom strncpy

With this knowledge about strncpy and the implications, the developer should remember to insert a null byte to terminate the copied string, while also making sure not to write the null byte beyond the buffer’s bounds. For example:

char* hello = "HELLO WORLD";
char dest[5];

// bad, writing out of bounds
strncpy(dest, hello, 5);
dest[5] = 0;

// good, within bounds
strncpy(dest, hello, 4);
dest[4] = 0;

This does not look good. While the usage of strncpy was meant for safety purposes, it has introduced a new problem for developers to be wary of. Now, developers are required to remember adding a null byte, while also being careful about the position it is added (need to minus one from the buffer size). A lapse in concentration by a developer and a code reviewer will result in an off-by-one write bug, which under the right conditions could be exploitable by an attacker to gain RCE.

There is a reliable solution to this. The development team could create a custom version of strncpy that best suits their needs. For example, let’s call it my_strncpy. The custom my_strncpy could behave differently from strncpy by ensuring that a null byte is always added at the last position, that is, length minus one. The tradeoff would be that only length minus one characters are copied from the source string, but this is not a security concern. Internal documentation should then be maintained to ensure clarity of my_strncpy’s behaviour. Furthermore, usage of strncpy should be avoided since there would not be any good reason for it to be used anymore.

By doing the above, security is decoupled from the unexpected intricacies of standard library functions, so that developers and code reviewers are protected from making mistakes caused by unintended behaviour.

Caveat: strncat

The problem caused by strncat is the opposite of strncpy’s. Unlike strncpy which does not add a null byte to the end of the copied string, strncat adds a null byte after the end of the copied string. For example:

char* hello = "HELLO WORLD";
char dest[10];
memset(dest, '\xAA', 10);

dest[0] = 0;
strncat(dest, hello, 5);

// to print contents of `dest` in hex
for (int i = 0; i < 10; ++i) printf("%hhx ", dest[i]);
printf("\n");

Even though in the code above strncat was given 5 as the length to copy, it had also added a null byte at the 6th position, after the end of the copied string.

48 45 4c 4c 4f 0 aa aa aa aa

This is much more dangerous than the case of strncpy, because there is actually an out-of-bounds memory write. Consider the contrived example below:

char* hello = "HELLO WORLD HELLO WORLD";
char dest[12];
int needs_auth = 1;
dest[0] = 0;
strncat(dest, hello, 12);

Here, it depends on how the compiler allocates the variables on the stack. If needs_auth is allocated after dest, then as strncat is given length 12, it would write 12 bytes into dest, and a null byte into needs_auth which is stored after it. As a result, needs_auth would contain the value 0.

Such an off-by-one bug to write memory out of bounds may not appear as intimidating as a conventional buffer overflow, but it is still a buffer overflow nonetheless, despite just overflowing by just one null byte. The consequences can be dire if the null byte is written into a variable that is part of critical logic. For example, authentication could be bypassed as seen in the example above; or if values related to bounds checking were overwritten, a buffer overflow may occur.

Custom strncat

Here, we give a similar suggestion as we did for strncpy to protect the developers from making mistakes caused by this tricky behaviour. Again, although it is part of the developers’ responsibility to make sure the code is correct, it is not reliable to expect such intricacies to always be on their mind, as they may be very focused on implementing the feature requirements and have forgotten about the memory side effects of their code. Thus, it is very beneficial to set up a safe development environment that takes this burden off the developers’ shoulders.

Similar to my_strncpy, the development team could create a custom strncat as well, e.g. my_strncat, that works according to their needs. One possible implementation for my_strncat is to add the null byte at the last position, that is, for example if the length field is 5, the null byte will be added at the 5th position (1-indexed). This means that only length minus one characters are concatenated to the destination string. Such behaviour should then be clearly written in the team’s internal documentation to ensure there is no ambiguity. Furthermore, usage of strncat from the standard library should be avoided since there is no longer a good reason for it to be used.

Actionable Steps

To summarize the suggestions above, we advise development and security teams to impose the following rules on their codebase:

  1. Avoid unbounded memory-copying functions (strcpy, sprintf, strcat) and use bounded functions that do the same.
  2. Create and use your own custom implementation of library functions (such as strncpy or strncat) so that you have full control and understanding of their behaviour, to avoid pitfalls caused by unexpected intricacies of the library functions. Then, avoid using the library functions.
  3. Avoid hardcoding sizes for allocation and memory-copying operations. For a function that takes in a buffer pointer and writes to it, ensure that the buffer’s allocation size is also taken as an argument. This removes uncertainties about the size, for both the caller and the callee.

The suggestions above all follow the same principles. We want to change the problem from “are we using this function correctly” to “are we implementing this function correctly”. There will be significantly less room for error, as the former requires reviewing every single function and trying to catch incorrect usages such as the off-by-one examples given; whereas the latter removes the possibility of such pitfalls, and only requires reviewing the custom implementations to ensure that they are implemented correctly.

In other words, instead of worrying about “what are the possible pitfalls of using this function, is there some specific scenario that I have missed”, we can now call memory-writing functions with a piece of mind and just care about “is this function implemented correctly”.

In routers that are higher on the price range, we have observed a widespread application of the rules above in their codebase. As a result, it was more difficult to find bugs on these devices that are typically considered as low hanging fruits on cheaper routers.

Examples

The following are examples where a buffer overflow due to improper (or the lack of) bounds checks was exploitable to gain RCE due to the lack of mitigations such as ASLR, PIE, NX or canary.

These show that even in the recent years having modern mitigations, simple bugs are still present in routers and could be easily exploited.

Format String Bug

A format string vulnerability could be exploited to leak pointers, perform buffer overflow, or write to certain memory locations. It is a very powerful bug. However, security teams need not be too worried about this vulnerability class, as it is one that is very easy to prevent. We explain why this is the case below.

Prevention

Format string bugs are very easy to prevent, as it can only occur when a program passes user input directly into the format string argument of printf or its variant functions. An example of vulnerable code is as follows:

printf(username);
fprintf(fp, username);

The correct way to print a string with printf is by using the %s specifier, as shown below:

printf("%s", username);
fprintf(fp, "%s", username);

This mistake is very easy to catch. Even if a programmer actually missed it, modern C compilers will also raise a warning about it under default settings. By providing the -Werror=format-security flag to gcc, the compilation will fail with these warnings treated as errors.

$ cat fs.c
#include <stdio.h>

int main(int argc, char** argv)
{
  printf(argv[1]);
  return 0;
}

$ gcc fs.c
fs.c: In function ‘main’:
fs.c:5:3: warning: format not a string literal and no format arguments [-Wformat-security]
    5 |   printf(argv[1]);
      |   ^~

$ gcc fs.c -Werror=format-security
fs.c: In function ‘main’:
fs.c:5:3: error: format not a string literal and no format arguments [-Werror=format-security]
    5 |   printf(argv[1]);
      |   ^~~~~~
cc1: some warnings being treated as errors

As shown above, this bug is easy to catch and fix. However, some router binaries might have been compiled decades ago, and at that time developers might not have had such awareness of format string vulnerabilities, or the compiler used might not have emitted such warnings. There is a chance that some of these vulnerable code have persisted until today.

Actionable Steps

For security teams, we advise you to check your old codebases and ensure that there are no more such bugs. We recommend adding the -Werror=format-security flag to gcc in the deployment process, to catch any such bugs that persisted from the past as well as the ones that may be accidentally introduced in the future.

There is a caveat to take careful note of. The -Werror=format-security flag only works on GCC 4.3.x or newer (source). Please make sure that your GCC is of a sufficiently new version to support this flag. At the point of writing this, the latest version of GCC is 13.2. GCC 4.3.6, the latest version under 4.3.x, was released in 2011.

Example

CVE-2018-14713 reports a format string vulnerability on the ASUS RT-AC3200 router, in which the web server might have been written back in 1999 (according to the linked article). The format string vulnerability could be used to leak pointers in memory to bypass ASLR. Then, since there is no stack canary, a buffer overflow could be exploited by using ROP to call system in libc to gain RCE.

Besides the admin panel, other services on a router may be vulnerable to a format string vulnerability as well. In 2013, a format string vulnerability was discovered in the Universal Plug and Play (UPnP) service of many routers, described in great detail in the article From Zero to ZeroDay Journey: Router Hacking (WRT54GL Linksys Case). The service does not require any authentication and accepts requests from the WAN interface.

The scary part about this vulnerability is that it is found in the Broadcom UPnP stack, which is code that is reused by many other router vendors. At the end of the linked article, there is a long list of routers by different vendors that are affected by this vulnerability.

Conclusion

In this article, we have discussed the attack surface of a router as well as misconfigurations and vulnerability classes that affect exposed services. We also provided suggestions in the form of best practices and secure code design to prevent attackers from abusing these services. By designing the development environment intentionally through enforcing the usage of only secure functions, the development team can be protected from making mistakes that may have dire consequences.

There are other problems not covered in this article. Firstly, we have observed that for some vendors, many of their devices share the same codebase. However, when a vulnerability is reported for a device and remediated for that device, the fixes are not applied to the other devices which share similar code and have the same vulnerability. This is likely due to the vendors’ lack of knowledge about which devices share the same code, or if they do know, it may be because of the high cost of testing such devices.

When vendors leave vulnerabilities unpatched on similar devices while fixing them on only one device, attackers who notice a security update published for that one device can quickly analyse the patch and write exploits for the other unpatched devices. This leaves many consumers at risk of being compromised.

Another problem comes from the usage of open source projects. It is reasonable and beneficial for developers to import open source libraries to save development time. However, these open source projects may receive vulnerability reports and apply fixes from time to time. In a way, one benefit of using such open source projects is that developers do not need to fix the bugs found in them, as they can just update that library to the latest version. However, there may be challenges in doing so, either due to worries about compatibility, or not even being aware that a library needs to be updated.

To summarize, aside from fixing vulnerabilities within a device, there are also challenges in ensuring that patches are applied horizontally, that is, across all devices that are affected; as well as vertically, through applying updates performed on the upstream open source projects.