OpenSSL HeartBleed: Your servers may still bleed after the update

If you have a Linux server you have to know that it is not enough to install the security updates to be safe. When it is not possible to reboot the server you must find the processes which are still using old libraries and terminate them manually or you will still suffer from the vulnerabilities and not knowing.

This is an example of processes using the old libssl lib (the one suffering from HeartBleed bug) after upgrading my home (Ubuntu) system to latest secure version.

The new and safe (until proved otherwise) version of OpenSSL:

# openssl version -a
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Apr 7 20:31:43 UTC 2014
platform: debian-i386

But some processes are still using the old library:

# lsof -n | grep ssl | grep -i del

php5-fpm 3155 root DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
php5-fpm 3158 www-data DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
php5-fpm 3159 www-data DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
nginx 3190 root DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
nginx 3191 www-data DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
nginx 3192 www-data DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
nginx 3194 www-data DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
nginx 3195 www-data DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
ubuntuone 4039 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
ubuntuone 4039 3684 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
gmain 4039 4040 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
ubuntuone 4039 4064 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
gdbus 4039 4067 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
ubuntuone 4039 4917 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
python 5049 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
python 5049 5050 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
python 5049 5059 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
python 5049 5067 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
python 5051 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
gdbus 5051 5054 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
gmain 5051 5055 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
dconf 5051 5056 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
python 5051 5057 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
python 5060 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
gdbus 5060 5063 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
gmain 5060 5064 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
dconf 5060 5065 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0
python 5060 5068 rndmusr DEL REG 8,1 134054 /lib/i386-linux-gnu/libssl.so.1.0.0

You can also use this command which has a better output, to list the processes that needs to be restarted:

grep -l ‘libssl.*deleted’ /proc/*/maps | tr -cd 0-9\\n | xargs -r ps u

root 20702 0.0 0.0 13792 972 ? Ss Apr10 0:00 nginx: master process /usr/sbin/nginx
www-data 20703 0.0 0.0 13972 1372 ? S Apr10 0:00 nginx: worker process
www-data 20704 0.0 0.0 13972 1288 ? S Apr10 0:00 nginx: worker process
www-data 20706 0.0 0.0 13972 1476 ? S Apr10 0:00 nginx: worker process
www-data 20707 0.0 0.0 13972 1448 ? S Apr10 0:00 nginx: worker process
root 20745 0.0 0.4 134328 9608 ? Ss Apr10 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
www-data 20748 0.0 0.8 136476 17156 ? S Apr10 0:00 php-fpm: pool www
www-data 20749 0.0 0.8 138660 18100 ? S Apr10 0:00 php-fpm: pool www
rndmusr 20858 0.0 0.2 44040 6052 ? Sl Apr10 0:00 /usr/bin/python /usr/bin/zim –ipc-server-main /tmp/zim-rndmusr/zim-server
rndmusr 20860 0.3 0.9 371332 19792 ? Sl Apr10 0:54 /usr/bin/python /usr/bin/zim –ipc-server-main /tmp/zim-rndmusr/zim-server
rndmusr 20869 0.0 0.7 368176 16092 ? Sl Apr10 0:00 /usr/bin/python /usr/bin/zim –ipc-server-main /tmp/zim-rndmusr/zim-server
www-data 22285 0.0 0.6 136288 12500 ? S Apr10 0:00 php-fpm: pool www

At this point I had to manually restart nginx and php5-fpm services and restart zim.

Just when I was about to pet myself on the shoulder, close the sites about ‘heartbleed’ and start a movie, I bumped into a post mentioning the nice ‘checkrestart’ from debian-goodies package. I installed it and to my surprise there were still more programs using old libraries.

# checkrestart
Found 4 processes using old versions of upgraded files
(4 distinct programs)
(4 distinct packages)

Of these, 1 seem to contain init scripts which can be used to restart them:
The following packages seem to have init scripts that could be used
to restart them:
openssh-server:

2539 /usr/sbin/sshd

These are the init scripts:
service ssh restart

These processes do not seem to have an associated init script to restart them:
openssh-client:

3639 /usr/bin/ssh-agent

colord:

1728 /usr/lib/colord/colord

update-notifier:

3716 /usr/bin/update-notifier

Those Windows guys asking for restart after each upgrade were up to something it seems 🙂

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s