Mikrotik RB2011 OpenWRT Installation Procedure

RouterBoard RB2011 Install Tips Steak Electronics

\textbf{Mikrotik RB2011 Routerboard Install}
 1.  Overview
 2.  Procedure
   2.1.  Run DHCP server with TFTP boot enabled
   2.2.  Power off Mikrotik, hold reset, and power on
   2.3.  Backup existing Flash of Mikrotik (optional – recommended)
   2.4.  Sysupgrade the sysupgrade bin
 3.  Troubleshooting – Bad RAM ECC Errors
 4.  Pitfalls
   4.1.  Other Openwrt configs, beware!
   4.2.  Not all RB2011 have UART
 5.  References

1. Overview To install Openwrt on a Rack Mount server. One of the options is the Mikrotik RB2011 series of routers.

\textbf{EDIT: 11/28/19 Since this has been written, 19.01 RC has been released, and I would recommend people use that for the RB2011, as it should address issues with installation (SPI Flash write / RAM ECC errors covered in Troubleshooting section) which can occur on ‘some’ RB2011 models. }

\textbf{EDIT: 10/2020, Some of these models have a habit of marking their flash as bad blocks, making them unusable. For more details see openwrt rb2011 page. Also, these devices have Routerboot, which is proprietary…}

2. Procedure This paper will not be a record of my work (though it took about 3 nights to compile this info). What it will be is a guide for others, or myself to install Openwrt on this router in the future. Let’s begin.

The general process will be the following:
1.) Run dnsmasq command which will start a DHCP server with
TFTP boot enabled
2.) Power off RB2011, Hold reset button, and power on.
Continue holding reset button until you see the DNSMasq
window report it has sent the file.
3.) Login to the netboot version of Openwrt, and sysupgrade
to the 64M or 128M (huge) mikrotik image. (Depending on how many flash chips you have – 1 or 2).
Sounds simple? It should be! Millions of hours of engineering and manpower have gone into this, it better damn well be. But don’t be surprised if something unexpected occurs. I will cover some troubleshooting as well.

2.1. Run DHCP server with TFTP boot enabled I tend to make a shell command for the dhcp server command. It’s just one command, and will do all the gymnastics for you. Here we go:

#ifconfig eth0 up
sudo dnsmasq -i eth0 –dhcp-range=, \
–dhcp-boot=openwrt-ar71xx-mikrotik-vmlinux-initramfs.elf \
–enable-tftp –tftp-root=~/directory/where/file/is/ \
-d -u $USER -p0 -K –log-dhcp –bootp-dynamic
NOTE: Beware, that the latex wp converter I’m using changes two dashes to one long dash… So e.g. log dhcp has two hyphens before it.
You can omit the ifconfig, and run it manually. Essentially, you must start an interface with a static ip at network. Sometimes you will start getting connect: invalid argument errors when trying to run ping, or ssh, so when that happens, I simply
\textbf{ifconfig eth0}

Then again init the interface with \textbf{ifconfig eth0 netmask}

You should be connected to Ethernet port 1 on the Mikrotik. Now, that’s all for part 1. Let’s reset it.

2.2. Power off Mikrotik, hold reset, and power on This section is self explanatory. The only things to note are the following:

Don’t need to hold reset when it’s on, then power off. No 30/30/30 crap here. Don’t need to wait that long. Simply power off, hold reset, then power on and hold for 20-30 seconds.

You can let go of reset when the TFTP boot says it sent the router file, but remain connected to Ethernet port 1, until you see the tftp boot print out the Openwrt hostname from the netbooted router. It takes a few seconds, but it will. After that you can plug into ethernet port 2,3,4 or anything besides 1.

2.3. Backup existing Flash of Mikrotik (optional – recommended) EDIT: backup didn’t work. Don’t bother. Ignore the below section.
You can always load another firmware file from TFTP / bootloader should something go wrong.

Next, login to the Openwrt. Make sure you login from ethernet 2 or 3 (but not 1).

Here’s an intermediate step. Navigate to the /tmp directory and backup the flash configuration.
cat /proc/mtd (review this, you want to copy the last two filesystems)
(they will either be /dev/mtdblock1+2 or /dev/mtdblock5+6)
(depending on your Mikrotik version.)

You can directly dd to your remote computer from the router.
dd if=/dev/mtd6 | ssh user@host dd of=/somewhere.img

That saves some steps. And yes, the mtd6 is the same size
and checksum as the mtdblock6, I checked. But you should double
check too.

EDIT: You can’t restore the previous Mikrotik OS with these images, though. Not sure why.

mkdir /mnt/kernel /mnt/rootfs
(assuming it’s at mtd1)
mount -t jffs2 /dev/mtdblock1 /mnt/kernel
mount -t jffs2 /dev/mtdblock2 /mnt/rootfs
cd /mnt/kernel
tar czf /tmp/mikrotik-kernel.tar.gz
cd /mnt/rootfs
tar czf /tmp/mikrotik-rootfs.tar.gz

Finally, scp the two tar files to your local machine.

2.4. Sysupgrade the sysupgrade bin To install on the RB2011, you should use the sysupgrade bin from the netbooted Openwrt. You should already be logged into the Netboot openwrt. At this point, scp the sysupgrade bin to the tmp folder. (If you don’t know if it’s 64M or huge, open up the device and look at the flash chips. There are either 1 or 2. 2 is huge, obviously.). Navigate to /tmp and then run
\textbf{\#sysupgrade -filename-}

That should be it. Give it a few minutes to reboot, and if all goes well, you will have a functioning rack mount FOSS router. Good.

3. Troubleshooting – Bad RAM ECC Errors I’ve done this process on 3 different RB2011 routers. The first I used the 64M image, and it worked without issue. The 2nd, I tried the 64M image but it failed, so I repeated the netboot process, and used the large image. It worked from there.

On the 3rd router, it wouldn’t boot the image. I tried manually installing via flashing the rootfs, and vmlinuz images manually, from the netboot, but that didn’t work either. Finally, I connected a serial port (UART). [1] I connected Minicom and watched the Openwrt kernel attempt to boot. From the boot messages, I could clearly see what the problem was: Bad RAM (ECC Errors). So if the above steps don’t work, I recommend simply soldering a wire to the RX, wrapping a wire around the ground pin, and listening to the boot console.

You can always netboot again, if you flash the wrong image for some reason. The Mikrotik is good, in that the netboot functionality is easy to use. Well designed, I tip my hat to them. Just be careful not to write to the bootloader. Otherwise, there is also a labeled JTAG connector which might (possibly) work.

4. Pitfalls 4.1. Other Openwrt configs, beware! Don’t think you can upload a working configuration from a different piece of Openwrt hardware to this device. The ports in the /etc/config files may be different, and it might not boot. If you make a mistake, just netboot, then sysupgrade again. I will admit, I did this.

4.2. Not all RB2011 have UART If the ttl UART chip isn’t populated on your board, you won’t get serial. Some have it, some don’t…

5. References




EDIT: After installing this on 2/4/19, I found that some of them did not install without some changes. There is a forum thread on this, but for now you have to install from git master, for ‘some’ rb2011 models. In 19 release, the errors should be resolved, and the release file should just work. Installing from git is not difficult, and don’t be afraid. There are three settings you have to adjust in make menuconfig. the target architecture (ar71xx), the make (mikrotik) and that this is a Nand-large (if you have 2 flash) type. The rest of menuconfig can be left at defaults.

Here’s the copied and pasted instructions from https://forum.openwrt.org/t/building-and-installing-lede-image-from-source-on-mikrotik-rb433/7172/13

It covers everything, except that if you get warnings about missing dependencies, you must run a script in the openwrt directory. I believe it’s ./scripts/feeds. update -a

./scripts/feeds install -a

If you are unsure, search the forums.

I hope this can help someone,
leave me a message if there is something to correct.

something is taken from: https://wiki.openwrt.org/toh/mikrotik/rb433 18



sudo apt install build-essential git zlib1g-dev

as non privileged user

cd ~; mkdir lede-project
git clone https://github.com/lede-project/source.git 5
cd source
make menuconfig

i made only this choiche
--- Target System (Atheros AR7xxx/AR9xxx)
--- Subtarget (Mikrotik devices with NAND/NOR flash)
--- Target Profile (MikroTik RouterBoard (64 MB NAND))

exit and save .config file


if something missing check the errors
when everything is finished
sudo mkdir /tftpboot
sudo cp bin/targets/ar71xx/mikrotik/lede-ar71xx-mikrotik-vmlinux-initramfs-lzma.elf /tftpboot/boot.elf
sudo cp bin/targets/ar71xx/mikrotiklede-ar71xx-mikrotik-nand-64m-squashfs-sysupgrade.bin

  1. All you need is Ground and an RX of the router (it’s near the power plug, marked on the pcb).

Adding Email Notifications to Tripwire with SSMTP


 1.  Overview
 2.  Steps
   2.1.  Configuring Tripwire
     2.1.1.  Devuan/Debian
   2.2.  Configuring SSMTP

EDIT: This guide is obsolete for Debian 10 based distributions. For those, you should use MSMTP. See https://wiki.zoneminder.com/Email

This document is best read printed out on paper.
1. Overview Tripwire is intrusion detection software for GNU Linux \& BSD. Let’s document how to set it up on a server with SSMTP configured for email notifications. Tripwire isn’t too hard to setup, but I had to jump through a hoop for email notifications. Here I cover install, and how to get SSMTP working.
2. Steps 2.1. Configuring Tripwire First install Tripwire. This will depend on your package manager. Here is Debian/Devuan.
apt-get install tripwire mailutils ssmtp
2.1.1. Devuan/Debian Devuan will prompt you for a few things in an ncurses gui. Answer all of the defaults (yes for a site key, yes for a user key, etc…). Record your password.
I use the same password for both.

\textbf{After install:}The goal when setting this up is to init, as little as possible. Each init is about 2-3 minutes, so time can be avoided, if you know what configs you need. Let’s first edit the configs as much as possible.

when whitelisting, this is what needs to be commented out in devuan jessie/ascii for /etc/tripwire/twpol.txt

Filename: /etc/rc.boot
Filename: /root/mail
Filename: /root/Mail
Filename: /root/.xsession-errors
Filename: /root/.xauth
Filename: /root/.tcshrc
Filename: /root/.sawfish
Filename: /root/.pinerc
Filename: /root/.mc
Filename: /root/.gnome_private
Filename: /root/.gnome-desktop
Filename: /root/.gnome
Filename: /root/.esd_auth
Filename: /root/.elm
Filename: /root/.cshrc
Filename: /root/.bash_profile
Filename: /root/.bash_logout
Filename: /root/.amandahosts
Filename: /root/.addressbook.lu
Filename: /root/.addressbook
Filename: /root/.Xresources
Filename: /root/.Xauthority
Filename: /root/.ICEauthority
Filename: /proc/6136/fd/3
Filename: /proc/6136/fdinfo/3
Filename: /proc/6136/task/6136/fd/3
Filename: /proc/6136/task/6136/fdinfo/3
For proc, you simply comment out the whole directory. (you’ll see an entry in the file for /proc, put a \# before that). After twpol, we are not done. We also need to edit the email settings.

In /etc/tripwire/twcfg.txt we will change the following:
MAILPROGRAM =/root/script.sh
script.sh is just a script: (make sure it is executable with chmod +x[2])
/usr/sbin/sendmail -s youremail@domain.com

Finally, the last change we might make, will be for any special directories we want to watch. I put websites in the root at /sites/ so I will append the following to /etc/tripwire/twpol.txt
# Ruleset for Website
rulename = “Website Ruleset”,
severity= $(SIG_HI)
/sites/ -> $(SEC_CRIT);

Now we will init, type
sudo tripwire –init
sudo twadmin -m P /etc/tripwire/twpol.txt
sudo tripwire –init

to reconfigure twcfg.txt
\textbf{/usr/sbin/twadmin –create-cfgfile -S site.key /etc/tripwire/twcfg.txt}

if you get:
root@site:~# /usr/sbin/twadmin –create-cfgfile -S site.key /etc/tripwire/twcfg.txt
# Error: File could not be opened.
# Filename: /root/site.key
# No such file or directory
# Exiting…
You must cd to /etc/tripwire directory.

2.2. Configuring SSMTP

SSMTP is a program you configure once, and can reuse the configuration everywhere[3]. For starters, I’d recommend you install SSMTP according to this guide here:
This is a thorough guide that explains debugging. Some steps are superfluous (given that the instructions pertain to different software) but the general directions are sound. And afterwards sending an email is as easy as
\textbf{echo “Hello, World” \textbar mail -s “My email check” user@email.com}
This guide assumes you have configured SSMTP according to this guide correctly, tested it, and are able to mail from the command line. Once you’ve setup SSMTP once, you can reproduce this setup on other computers, simply by copying over the revaliases and ssmtp.conf of a valid configuration.

So let’s do that. Copy over revaliases, and ssmtp.conf. test the configuration from the command line using the above echo and mail. Once that works, test out tripwire.
\textbf{tripwire –test –email user@email.com}

  1. For a full walkthrough of this process see this URL:https://www.howtoforge.com/tutorial/how-to-monitor-and-detect-modified-files-using-tripwire-on-ubuntu-1604/ This process includes most, but not all of what you need to know.
  2. This script appears to be required in this setup.
  3. This is a strength of FOSS and let it remain that way.

Multiple Switches Can Slow a Network’s Internet Down

1. Overview

A question is, can multiple network switches, slow down either the network or the WAN. The answer is yes. Let’s discuss a working example today.

2. The Previous IT Guys

Today I was working at an office. They had a firewall connected to a Verizon FIOS modem. They had a gigabit [1]switch immediately downstream, and then a network cable ran about 50-75 feet to their office. Here they had another Gigabit switch, and then the cables went off to either computers or other switches. Let’s forget about what was downstream of that last switch except for the computers.

2.1. Internet Speeds

The internet speed from the two computers in the office was about 10Mbps down and roughly the same upload. However, I tested the speed directly from the IT closet, and found 100Mbps down and 100Mbps up. At this point, it was a simple case of following the signal path and seeing where things slowed down. At least, I knew that they should be getting 100/100Mbps.

2.2. The Tests

What I found is that as soon as I had two switches after the firewall [2]the speed would drop to 10/10. It was not dependent upon any brand of switch. I tried swapping from one other model that was available. The slowdown appeared to be switch independent. So in summary:

  • Firewall – computer – 100/100
  • Firewall – switch – computer – 100/100
  • Firewall – switch – 50 foot wire – computer – 100/100
  • Firewall – switch – 50 foot wire – Switch A- computer – 10/10
  • Firewall – switch – 50 foot wire – Switch B- computer – 10/10

What I did was remove the switch at the outside of the firewall. Now I had only

Firewall – 50 foot wire – Switch – Computer – 100/100

3. Additional Background

I’ve heard an IT person mention that he didn’t like multiple switches in network deployments, due to speed issues. On the other hand, I’ve setup Video Recording Networks with multiple switches all connecting serially to each other without issue. It seems that this is not a simple black and white issue. With the right hardware you may get away with it. Or if you avoid WAN access. In any case, the only way to know, is to test.

4. Conclusion

If you are going to put > 1 switches downstream of the firewall, make sure to double check the WAN speed tests from just after the firewall, to at the very last switch.

  1. Actually, all switches were gigabit, but it didn’t make a difference.
  2. which has its own internal switch

Remove Domain on Windows Server and IIS doesn’t work

I recently took down the domain from a server, and an application running on the same server’s IIS failed afterwards. One possible cause of this is due to the application running as a domain user. The solution:

In application pools of IIS manager, change domain\user to a local account.

Here, identity would have been one of Domain\user.

AutoSSH – a Reverse Proxy Alternative

This document is best read printed out on paper.
I recently added another apache server to an existing infrastructure, and
I wanted it to be accessible under a similar IP as another server. Due
to the complexity of the website, it was not possible to simply do a reverse
proxy without knowing the correct settings (e.g. X-Forwarded for). Instead,
AutoSSH was used. In the end, I accessed a new port on the existing IP.
Work Log
Ok, I’m going to get right to the configs that I used. You want the tool, you
don’t need to know all the details.
Here is the crontab script I used. I put this in /etc/crontab, so it has root
after the times. I only use /etc/crontab, as it’s easier to manage.

* * * * * root pgrep autossh > /dev/null || \

A few notes about this. Pgrep will search for autossh. If it doesn’t find it,
then it will try the next command. (—— is an OR). Put the bash script
wherever you want.
Bash Script
This script is obviously what the crontab calls.

logger ” /usr/local/bin/autosshzm script started.”
#source $HOME/.bash_profile #not needed.
source $HOME/.keychain/$HOSTNAME-sh
logger ” /usr/local/bin/autosshzm sourced.”
-L -f user@ipaddress sleep 31536000
&> /var/log/autosshzm/autosshzm.log
1#autossh -M 0 -o “ServerAliveInterval 30” -o “ServerAliveCountMax 3”
-L user@ipaddress &>
logger “auto ssh ran”

Note that the second autossh does not work, as it’s missing the sleep and
the -f command. 1 In order for this to work, you’ll also need the following

apt-get install keychain autossh

There were some more setup steps required for keychain… From stackex-

solves this in a painless way. It’s in the repos for Debian/Ubuntu:
sudo apt-get install keychain
and perhaps for many other distros (it looks like it originated
from Gentoo).
This program will start an ssh-agent if none is running, and
provide shell scripts that can be sourced and connect the current
shell to this particular ssh-agent.
For bash, with a private key named id_rsa, add the following to
your .profile:
keychain –nogui id_rsa
This will start an ssh-agent and add the id_rsa key on the first
login after reboot. If the key is passphrase-protected, it will
also ask for the passphrase. No need to use unprotected keys
anymore! For subsequent logins, it will recognize the agent
and not ask for a passphrase again.
Also, add the following as a last line of your .bashrc:
Figuring this kind of stuff out can take about an hour.
2. ~/.keychain/$HOSTNAME-sh
This will let the shell know where to reach the SSH agent managed
by keychain. Make sure that .bashrc is sourced from .profile.
However, it seems that cron jobs still don’t see this. As a
remedy, include the line above in the crontab, just before
your actual command:
* * * * * . ~/.keychain/$HOSTNAME-sh; your-actual-command

The only thing that I needed to do here was
keychain –nogui id rsa
The rest of it (notes about crontab) was not required.

What Did NOT Work
Here’s some things I tried that did not work.

  • https://github.com/obfusk/autossh-init – This init script, didn’t do
    much for me. Remember, I’m stuck with systemd in Ubuntu 19.04… 2
  • Reverse proxy with Apache – As I said, my website 3 was too complex,
    and I didn’t want to go down that rabbit hole.
  • Starting AutoSSH in rc.local. Didn’t work.

The scourge of deleting software history. Keep backwards compatibility at ALL COSTS, developers.
Some people might call it a web application. I will not.

How to Factory Reset Polycom VVX 300 VOIP Phones

This is from memory and my notes, and may be incomplete, or have errors, but regardless…

Power Cycle the phone.

Hit Cancel on Boot screen (you will see an option to cancel by hitting one of the menu buttons while it’s booting).

Press 1+3+5, hold them down. NOTE: there is also a function where you can hit 3+5+7. One of these is the right one. I think it’s 1+3+5, but if that doesn’t bring you to a password prompt, try 3+5+7.

A password prompt should appear.

The password is the mac address. That is the easy part. The mac is on the phone on a sticker. The hard part is that you must enter letters in lower case. You can’t preview the letters, and since you have only a number pad, you have to understand how these interfaces typically work.

What there is, is a small icon on the menu that lets you change from numbers to upper case letters, to lower case letter, then back to numbers. So in order to change from number to lower case, you have to hit that button on the screen. And also, if you change back to numbers, you must hit the button again. So sometimes you might have to blindly hit a few keys. E.g. cat would be 1+1+1 (for c), a short pause, then 1 (for a), then whatever t is. But you can’t see the a going from a to b to c. It’s all hidden behind hashmarks for “security”. So you are blind and must press the buttons in the correct sequence. Horrible design, but the idea must be that only IT guys will ever work with the interface. They should be used to getting the short end of the administration stick, so will not be surprised. I was not be surprised, to be honest, just disappointed.

Configuration for Verizon VOIP

After you put the password in, you can factory reset, or possibly it does a factory reset automatically. Now the VOIP phone is back to its base mode. When it reboots you can now access the admin section, with the password “456”. For Verizon’s voip, the settings are:
Go somewhere in the network menus (I didn’t note where).

Enable ZTP
Server type - HTTPS
Server address - https://tlcm.sipflash.com

Then I believe a reboot.

PrestaShop, Docker, SSL, Nginx Reverse Proxy

This is a common problem for Prestashop as viewed on the https://github.com/PrestaShop/docker and Prestashop forums. Here’s some configurations I found necessary to handle this. I was able to get it working. First off, there are a few files you need to keep in mind: The DB has two tables: ps_shop_url, ps_ssl_enabled, which must be edited manually. The nginx.conf must have the right settings (they can be the same as a wordpress reverse proxy, so start there). And it was necessary to set $_SERVER[‘HTTPS’] = ‘on’; in the prestashop_root/config/ folder under one of the defines. I used defines_custom.inc.php. I understand this is not best practice, but I got it to work. Interested users should research where it’s best to put custom edits for Prestashop. Here’s quickly some more notes:

# shopping site redirect
server {
#server tokens hides the nginx identifying itself
server_tokens off;
server_name shop.mysite.com;
listen 80;


location ^~ /.well-known {
alias /var/www/html/.well-known/;
#autoindex on;
location / {
##** nginx redirect ALL http requests to https ** ##
return 301 https://$server_name$request_uri;
#from gist mentioned below
#rewrite ^ https://$host$request_uri? permanent;}
}server {
#server tokens hides the nginx identifying itself
server_tokens off;
server_name shop.mysite.com;
#listen 80;
listen 443 ssl;
ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem;location / {
proxy_redirect off;
proxy_pass http://prestashop;#these were from wp, but not usable here, according to:
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;#this was from gist, but doesn't work.
#proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header Host $http_host;
#proxy_set_header X-Forwarded-Proto https;
#from git issues tracker
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;}}

Here is the docker-compose.yml:

version: '3'

    image: nginx:latest
    container_name: production_nginx
      - ./nginx.conf:/etc/nginx/nginx.conf
      - /etc/letsencrypt/:/etc/letsencrypt/
      - ./webroot/:/var/www/html/
#be careful with these. Port 80 on container side must match nginx listen port. the host side port, is only for the docker proxy.
      - 80:80
      - 443:443

    restart: always
#    command: [nginx-debug, '-g', 'daemon off;']

    image: mysql:5.7
      - MYSQL_ROOT_PASSWORD=xxxxxx
      - MYSQL_USER=xxxxxx
      - MYSQL_PASSWORD=xxxxxx
      - MYSQL_DATABASE=prestashop
      - ./db_data_presta:/var/lib/mysql
      - ./db_conf_presta:/etc/mysql/
    restart: always

        - db_prestashop
    image: 'prestashop/prestashop'
       - ./presta_root:/var/www/html/
      - DB_SERVER=db_prestashop
      - ADMIN_MAIL=xxxxxx
      - ADMIN_PASSWD=xxxxxx
      - VIRTUAL_HOST=https://URLHERE
      - VIRTUAL_PORT=80
      - PS_DOMAIN=https://URLHERE
      - DB_NAME=prestashop
      - DB_USER=xxxxxx
      - DB_PASSWD=xxxxxx

I don’t believe the DB_NAME or DB_USER variable is used in docker for prestashop. DB config is done at installation. Virtual host likewise, may not be necessary.

use prestashop

Also, one other change on the DB. You should ensure that your shop URL is set to be something like URL.com
You do NOT want https://URL.com in either domain or domain\_ssl in ps\_shop\_url

mysql> select * from ps_shop_url;
| id_shop_url | id_shop | domain                   | domain_ssl               | physical_uri | virtual_uri | main | active |
|           1 |       1 | shop.steaky.com | shop.steaky.com | /            |             |    1 |      1 |
1 row in set (0.00 sec)

Firejail and MuPDF permissions error: “X Error of failed request: GLXBadContext”

Was setting up Firejail and MuPDF on Funtoo 1.3…

$ firejail mupdf sample.pdf

Reading profile /etc/firejail/mupdf.profile
Reading profile /etc/firejail/disable-common.inc
Reading profile /etc/firejail/disable-devel.inc
Reading profile /etc/firejail/disable-interpreters.inc
Reading profile /etc/firejail/disable-passwdmgr.inc
Reading profile /etc/firejail/disable-programs.inc
Reading profile /etc/firejail/disable-xdg.inc
Reading profile /etc/firejail/whitelist-var-common.inc
Parent pid 31753, child pid 31754
Private /etc installed in 82.08 ms
Blacklist violations are logged to syslog
Child process initialized in 235.06 ms
libGL error: failed to open drm device: Permission denied
libGL error: failed to load driver: i965
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request: GLXBadContext
Major opcode of failed request: 150 (GLX)
Minor opcode of failed request: 6 (X_GLXIsDirect)
Serial number of failed request: 36
Current serial number in output stream: 35

Parent is shutting down, bye...

After searching around online with other MuPDF issues (issues tracker, etc. I had some other ideas.

$strace firejail mupdf sample.pdf

setresuid(-1, 0, -1) = -1 EPERM (Operation not permitted)
setresgid(-1, 0, -1) = -1 EPERM (Operation not permitted)
unlink("/run/firejail/bandwidth/31761-bandwidth") = -1 ENOENT (No such file or directory)
unlink("/run/firejail/network/31761-netmap") = -1 ENOENT (No such file or directory)
unlink("/run/firejail/name/31761") = -1 ENOENT (No such file or directory)
unlink("/run/firejail/x11/31761") = -1 ENOENT (No such file or directory)
unlink("/run/firejail/profile/31761") = -1 ENOENT (No such file or directory)
setresuid(-1, 1000, -1) = 0
setresgid(-1, 1000, -1) = 0
getppid() = 31759
setresuid(-1, 0, -1) = -1 EPERM (Operation not permitted)
setresgid(-1, 0, -1) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/proc/31759/comm", O_RDONLY) = 3
read(3, "strace\n", 4095) = 7
close(3) = 0
setresuid(-1, 1000, -1) = 0
setresgid(-1, 1000, -1) = 0
getuid() = 1000
geteuid() = 1000
getuid() = 1000
geteuid() = 1000
getuid() = 1000
geteuid() = 1000
openat(AT_FDCWD, "/etc/firejail/firejail.config", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=4395, ...}) = 0
read(3, "# This is Firejail system-wide c"..., 4096) = 4096
read(3, " third dimension is\n# color dept"..., 4096) = 299
read(3, "", 4096) = 0
close(3) = 0
getuid() = 1000
geteuid() = 1000
getuid() = 1000
geteuid() = 1000
getuid() = 1000
geteuid() = 1000
getuid() = 1000
stat("/bin/bash/", 0x7ffe9b611270) = -1 ENOTDIR (Not a directory)
stat("/bin/bash", {st_mode=S_IFREG|0755, st_size=926688, ...}) = 0
access("/bin/bash", X_OK) = 0
getuid() = 1000
geteuid() = 1000
openat(AT_FDCWD, "/home/farm/.config/firejail", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such
file or directory)
getuid() = 1000
geteuid() = 1000
fstat(3, {st_mode=S_IFDIR|0755, st_size=32768, ...}) = 0
getdents(3, /* 566 entries */, 32768) = 22808
getuid() = 1000
geteuid() = 1000
stat("/etc/firejail/mupdf.profile/", 0x7ffe9b60f0d0) = -1 ENOTDIR (Not a directory)
access("/etc/firejail/mupdf.profile", R_OK) = 0
openat(AT_FDCWD, "/etc/firejail/mupdf.profile", O_RDONLY) = 4
getpid() = 31761
setresuid(-1, 0, -1) = -1 EPERM (Operation not permitted)
setresgid(-1, 0, -1) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/run/firejail/profile/31761", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EACCES (Permission denied)
write(2, "Error: cannot create /run/fireja"..., 49Error: cannot create /run/firejail/profile/31761
) = 49
setresuid(-1, 0, -1) = -1 EPERM (Operation not permitted)
setresgid(-1, 0, -1) = -1 EPERM (Operation not permitted)
getpid() = 31761
unlink("/run/firejail/bandwidth/31761-bandwidth") = -1 ENOENT (No such file or directory)
unlink("/run/firejail/network/31761-netmap") = -1 ENOENT (No such file or directory)
unlink("/run/firejail/name/31761") = -1 ENOENT (No such file or directory)
unlink("/run/firejail/x11/31761") = -1 ENOENT (No such file or directory)
unlink("/run/firejail/profile/31761") = -1 ENOENT (No such file or directory)
exit_group(1) = ?
+++ exited with 1 +++

It looks like the error is:
write(2, “Error: cannot create /run/fireja”…, 49Error: cannot create /run/firejail/profile/31761
) = 49


In gentoo, the mupdf profile  in /etc/firejail/has

# Firejail profile for mupdf
# Description: Lightweight PDF viewer
# This file is overwritten after every install/update
# Persistent local customizations
include /etc/firejail/mupdf.local
# Persistent global definitions
include /etc/firejail/globals.local

noblacklist ${DOCUMENTS}

include /etc/firejail/disable-common.inc
include /etc/firejail/disable-devel.inc
include /etc/firejail/disable-interpreters.inc
include /etc/firejail/disable-passwdmgr.inc
include /etc/firejail/disable-programs.inc
include /etc/firejail/disable-xdg.inc

include /etc/firejail/whitelist-var-common.inc

caps.drop all
net none
protocol unix
# seccomp.keep
shell none

# private-bin mupdf,sh,tempfile,rm
private-etc fonts

# mupdf will never write anything
read-only ${HOME}

However, the private-etc fonts will cause this error. This was found by brute force troubleshooting (delete entries from profile, until it works).

Comment it out, and MuPDF should display correctly.

now you will get:

$ firejail mupdf sample.pdf
Reading profile /etc/firejail/mupdf.profile
Reading profile /etc/firejail/disable-common.inc
Reading profile /etc/firejail/disable-devel.inc
Reading profile /etc/firejail/disable-interpreters.inc
Reading profile /etc/firejail/disable-passwdmgr.inc
Reading profile /etc/firejail/disable-programs.inc
Reading profile /etc/firejail/disable-xdg.inc
Reading profile /etc/firejail/whitelist-var-common.inc
Parent pid 31763, child pid 31764
Blacklist violations are logged to syslog
Child process initialized in 130.35 ms
libGL error: failed to open drm device: Permission denied
libGL error: failed to load driver: i965

and the PDF opens.