Detecting
Loadable Kernel Modules
(LKM)
The purpose of
this paper is cover LKM basics, detecting "trojaned" LKM's and figuring out
which LKM is installed on your machine.
LKM
What is a LKM?
Loadable Kernel Modules (LKM) are files that contain dynamically loadable kernel
components. LKM's are normally used to load device drivers and other hardware
drivers. LKM's can be found on Linux, Solaris and BSD (Open, Free and Net). This
paper will focus on Linux.
Linux comes with
various tools to assist the system administrator in loading, listing and
unloading of the kernel modules. Before we cover the tools, lets look at some
important files and directories associated with Kernel Modules. The first
directory we want to look at is /lib/modules/"kernel_version". Lets look and see
what files we can find under
/lib/modules."Kernel_version":
|
block build cdrom fs ipv4 misc modules.dep modules.isapnpmap modules.pcimap modules.usbmap net pcmcia scsi usb video
|
Table 1. /lib/modules
listing
Table 1 shows us both directories and
files related to LKM's. The only ones listed that are not directories are
modules.dep, modules.isapnpmap, modules.pcimap and mosules.usbmap. Those are
actual files that list modules within the system (combined). Lets take a quick
look at the net directory (This is
not a complete listing, do to space):
|
3c59x.o 3c90x.o 82596.o 8390.o ac3200.o acenic.o arlan-proc.o |
Table 2. net directory
There are
a couple of important bits of information we can gather from this table. The
first bit is, modules are listed as .o. Why? Because they are object files that
contain the actual module itself. The second bit is, the listing is not
complete. I did not have room to put all of the modules.
The second
important directory/file we want to look at it /etc/conf.modules. Conf.modules
is the configuration file that allows the system administrator to specify a
variety of parameters that control the loading of the modules. The conf.modules
file typically looks like this:
|
alias parport_lowlevel
parport_pc |
Table
3. conf.modules file
The
conf.modules file allows the administrator to assign alias to commonly used
modules. Note: This file is not required. A system administrator can create
his/her own configuration file. A system administrator can do that by using the
modprobe –C command.
Since I mentioned
modprobe earlier, lets briefly take a look at modprobe and the other tools
available in Linux that load and unload modules. The first tool is modprobe.
Modprobe can load single and multiple modules. It uses the modules.dep file to
look up dependencies. Depmod creates a “Makefile” like dependency file. This
file is called modules.dep. The
last three commands I will cover are lsmod, insmod and rmmod. lsmod provides the
administrator a listing of all modules currently loaded in the kernel. This list
can also be found at /proc/modules. This is helpful if you want to figure out
what modules are currently running. You will see a little later that lsmod is
not always a good tool to use for the detection of rootkit LKM’s. insmod loads
modules(seems simple). Some of the “trojaned” LKM code I have seen uses insmod
-f to load the “trojaned” module. rmmod is normally used to remove modules from
the kernel. Normally rmmod will NOT remove any “trojaned” LKM. We will discuss
details of that a little bit later. Hopefully, this section provided everybody
the basics of Loadable Kernel Modules. If this was not detailed enough see http://www.kernel.org/LDP/.
LKM rootkits and kstat
Recently, there
has been a lot of press about adore. Well, the worm adore not the LKM adore. So
I decided to look into the LKM adore(everyone else has looked at the worm). BTW,
if want to download adore go to http://packetstorm.securify.com/filedesc/adore-0.34.html.
Adore is a Linux
LKM rootkit. It is easy to install and only requires a few minor adjustments
when configuring. Adore can be installed with a default configuration or the
user can make changes to some of the code. The readme file recommends that the
user change the settings for ELITE_CMD and for HIDDEN_PORT. When you run
./configure, adore asks you for a password. This is for the backdoor port
(hence, change the HIDDEN_PORT) If you want more information on the install you
will have to download it. I ran the default installation, which consisted of
running./configure and make. After running make, you will see two files (ava,
startadore). In order for adore too execute you have to run startadore. Once you
run startadore the user can then run ava. Table 4 is the output from
./ava.
|
Usage: ./ava
{h,u,r,R,i,v,U} [file, PID or dummy (for U)] h
hide file u
unhide file r
execute as root R
remove PID forever U
uninstall adore i
make PID invisible v
make PID visible |
Table 4. Ava
output
Table 4 shows us the options
that adore provides to us. I will not cover each switch and what it does only
because it does not help us detect this rootkit. Now that we have covered the
basics of adore (LKM) lets look at how we detect adore and other rootkits.
Many rootkits
hide processes, directories, files and even connections. But many of them do so
by modifying the source code of binaries such as ps, df, netstat, top and lsof.
There are a couple of ways to detect these types of rootkits (i.e.
t0rn):
1)
md5 checksums
2)
Compiling these binaries from a known good source (i.e. cd-rom, disk).
3)
Some rootkits have a default port that an administrator can focus in on.
4)
Running a program such as chkrootkit
Many of the techniques used
to detect rootkits like t0rn are not effective against LKM rootkits (chkrootkit
can detect some of them). Since LKM rootkits access the kernel, they can hide
processes, connections, directories and files without modifying the binaries.
MD5 checksums become useless because there are no files being modified. This
means the checksums will not change. Checking ports MIGHT be an option BUT that
only tells you that you have been “rooted”.
Well, how can I
detect these monsters? Easy. There is a program I highly recommend to everyone
concerned with LKM rootkits. This program is called KSTAT. You can find it at
http://s0ftpj.org/en/site.html. KSTAT
works by checking the memory (/dev/kmem) for information about the
host(including LKMs). Want to see what kstat looks like? Good. Here is the
output for kstat:
Usage: kstat [-i iff] [-P]
[-p pid] [-M] [-m addr] [-s]
-i iff may be specified as 'all'
or as name (e.g. eth0)
displays info
about the queried interface
-P displays all
processes
-p pid is the process id of the
queried task
-M displays the kernel's LKMs'
linked list
-m addr is the hex address of
the queried module
displays info
about the module to be found at addr
-s displays info about the
system calls' table
As you can see, kstat
provides a person with many options for detecting these rootkits. Lets go
through some of the options and from these options we will learn how to detect
the following LKM rootkits:
1)
knark
2)
adore
3)
rkit
BTW, if there’s any rootkits
I have omitted please let me know and I will update the paper. : ) Kstat –s
seems to be the best way to detect LKM rootkits. The other options are really
great, but –s works all the time. Here is an output from kstat
–s:
|
SysCall
Address sys_exit
0xc0117ce4 sys_fork
0xc0108ebc sys_read
0xc012604c sys_write 0xc0126110 sys_open
0xc0125c10 sys_close
0xc0125d60 sys_waitpid
0xc0117ff8 sys_creat
0xc0125ca4 sys_link
0xc012de60 sys_unlink 0xc012dc90 sys_execve
0xc0108f18 sys_chdir
0xc01254a0 sys_time
0xc01184b4 sys_mknod
0xc012d77c sys_chmod
0xc01256e4 |
Table 5. kstat
–s
Table 5 is not a complete
output from kstat –s, but it does give us an idea as too what the output looks
like. Remember –s provides us with a picture of the sys_call_table. Keep this
information fresh as we will revisit this again later.
Kstat –P is
another switch that is very effective. Kstat -P shows us all of the processes
running at that time. This includes the processes hidden by an LKM rootkit.
Table 6 is an example of
kstat –P:
|
PID PPID UID GID COMMAND 1 0 0 0
init 2 1 0 0
kflushd 3 1 0 0
kupdate 4 1 0 0
kpiod 5 1 0 0
kswapd 6 1 0 0
mdrecoveryd 241 1 1 0
portmap 256 1 0 0
lockd 257 256 0 0
rpciod 266 1 0 0
rpc.statd 280 1 0 0
apmd 331 1 0 0
syslogd |
Table 6. kstat –P
When I first ran this
program I wasn’t sure kstat could do what it said but…it proved my doubts WRONG.
Here is what I did to verify this switch. I used ava -i 241(portmap). I then ran
kstat –P. As you can see from table 6,
portmap(bold) shows up. I then ran ps-ef(Table 7)and could not find it.
Lsof also did not show it. Table 7 is the output from ps
–ef:
UID
PID PPID C STIME TTY
TIME CMD
root
1 0 0 Mar30 ?
00:00:06 init [3]
root
2 1 0 Mar30 ?
00:00:00 [kflushd]
root
3 1 0 Mar30 ?
00:00:00 [kupdate]
root
4 1 0 Mar30 ?
00:00:00 [kpiod]
root
5 1 0 Mar30 ?
00:00:00 [kswapd]
root
6 1 0 Mar30 ?
00:00:00 [mdrecoveryd]
root 256 1 0 Mar30 ?
00:00:00 [lockd]
root 257 256 0 Mar30 ?
00:00:00 [rpciod]
root 266 1 0 Mar30 ?
00:00:00 rpc.statd
root 280 1 0 Mar30 ?
00:00:00 /usr/sbin/apmd -p 10 -w 5 -W -s
root 331 1 0 Mar30 ?
00:00:00 syslogd -m 0
root 340 1 0 Mar30 ?
00:00:00 klogd
nobody 354 1 0 Mar30 ?
00:00:00 identd -e -o
nobody 357 354 0 Mar30 ?
00:00:00 identd -e -o
nobody 359 357 0 Mar30 ?
00:00:00 identd -e -o
nobody 360 357 0 Mar30 ?
00:00:00 identd -e -o
nobody 361 357 0 Mar30 ?
00:00:00 identd -e -o
daemon 372 1 0 Mar30 ?
00:00:00 /usr/sbin/atd
root 386 1 0 Mar30 ?
00:00:00 crond
root 404 1 0 Mar30 ?
00:00:00 inetd
root 418 1 0 Mar30 ?
00:00:00 lpd
root 462 1 0 Mar30 ?
00:00:00 sendmail: accepting connections
root 477 1 0 Mar30 ?
00:00:00 gpm -t ps/2
root 491 1 0 Mar30 ?
00:00:01 httpd
xfs
531 1 0 Mar30 ?
00:00:00 xfs -droppriv -daemon -port -1
root 571 1 0 Mar30 tty1 00:00:00 login --
root
root 572 1 0 Mar30 tty2 00:00:00
/sbin/mingetty tty2
root 573 1 0 Mar30 tty3 00:00:00
/sbin/mingetty tty3
root 574 1 0 Mar30 tty4 00:00:00
/sbin/mingetty tty4
root 575 1 0 Mar30 tty5 00:00:00
/sbin/mingetty tty5
root 576 1 0 Mar30 tty6 00:00:00
/sbin/mingetty tty6
nobody 4290 491 0 Apr01 ?
00:00:00 httpd
nobody 4291 491 0 Apr01 ?
00:00:00 httpd
nobody 4292 491 0 Apr01 ?
00:00:00 httpd
nobody 4293 491 0 Apr01 ? 00:00:00
httpd
nobody 4294 491 0 Apr01 ?
00:00:00 httpd
nobody 4295 491 0 Apr01 ?
00:00:00 httpd
nobody 4298 491 0 Apr01 ?
00:00:00 httpd
nobody 4299 491 0 Apr01 ?
00:00:00 httpd
root 8073 571 0 Apr02 tty1 00:00:00
-bash
root 10659 8073 0 11:24 tty1 00:00:00 ps
-ef
Table 7. ps –ef output
There are two other switches
we will go over, they are kstat –p and kstat –M. First, lets look at kstat –p.
kstat –p gives us more information about a process. In order to run kstat –p you
will need to provide a process id. For example: kstat –p 241. This checks
process 241 (portmap) and provide an output. Sometimes it can provide us with
more information about a LKM rootkit. TABLE 8 is the output from kstat –p
241:
|
Name:
portmap State: S
(sleeping) Pid:
241 Ppid: 1
(init) Uid: 1 1 1
1 Gid: 0 0 0
0 Flags: PF_FORKNOEXEC
PF_SUPERPRIV Crucial Capabilities
Check Open
Files 0 CHAR
/dev/null 1 CHAR
/dev/null 2 CHAR
/dev/null 3 0.0.0.0:111 0.0.0.0:0 4 0.0.0.0:111 0.0.0.0:0 7 FIFO
/// 8 FIFO
/// 21 CHAR
/dev/null
|
Table 8. kstat –p
output
Finally, lets look at kstat
–M. Under normal conditions an administrator can perform lsmod or more
/proc/modules and find out what modules he/she are running. When a machine has
been “rooted” you can’t trust lsmod for accurate information. Kstat –M will
catch many of the basic LKM rootkits that I have tested. Kstat –M list all of
the modules loaded. I have seen it where kstat –M did list anything for knark,
but…nothings perfect. The output from kstat –M is similar to
lsmod.
So far we have covered
a lot of material about LKM’s, rootkits and kstat. Now we are going to put all
of that together and learn how to detect some of the LKM rootkits available
today.
The first LKM rootkit
we want to look at is knark. Knark is probably the best-known LKM rootkit and
one of the best written as well. Detecting it with kstat is fairly straight up
as well. Remember Table 5? Well in order to detect a knark LKM rootkit you will
need to run kstat –s. Once ran you need to look for the
following:
|
sys_fork
0xc284652c WARNING! Should be at 0xc0108c88 sys_read
0xc2846868 WARNING! Should be at 0xc012699c sys_execve 0xc2846bb8
WARNING! Should be at 0xc0108ce4 sys_kill
0xc28465d4 WARNING! Should be at 0xc01106b4 sys_ioctl 0xc2846640
WARNING! Should be at 0xc012ff78 sys_settimeofday
0xc2846a8c WARNING! Should be at 0xc0118364 sys_clone 0xc2846580
WARNING! Should be at 0xc0108ca4 |
Table 9. knark
detection
Lets take a closer look at
this table and see what all of this mess means to us. First, we see that there
are seven (7) sys_call_table entries that have warnings. Lets look at
sys_settimeofday, here we see that currently it is making it’s home in memory at
0xc2846a8c. kstat –s tells us that this is wrong and it should be at 0xc0118364.
When knark is installed it changes
the sys_call_table and the memory locations where you can find: sys_fork,
sys_read, sys_execve, sys_kill, sys_ioctl, sys_settimeofday and sys_clone. This
is how you could detect knark. Knowing which sys_call_table entries have been
changed could help you identify the actual rootkit itself. Knark changes
seven(7) total. After running kstat –s, the next step would be to run ps-ef
along with kstat –P and compare the two. If the are any differences in the two
you can then take the correct action. Keep in mind that memory locations can
change from box to box. As I said earlier, detecting LKM rootkits with kstat
is quite simple.
Lets look at
adore. Table 10 will show us what adore changes:
|
sys_fork
0xc4051428 WARNING! Should be at 0xc0108c88 sys_write 0xc4051590
WARNING! Should be at 0xc01269b8 sys_close 0xc405163c
WARNING! Should be at 0xc01264a4 sys_kill
0xc40514d0 WARNING! Should be at 0xc011060c sys_mkdir 0xc405172c
WARNING! Should be at 0xc012e540 sys_clone 0xc405147c
WARNING! Should be at 0xc0108ca4 sys_getdents 0xc40512a4 WARNING! Should
be at 0xc013022c |
Table 10. Adore detection
Adore changes seven (7)
entries as well (just like knark). Knark and adore only change three (3) of the
same sys_call_table entries. They are sys_fork, sys_kill, sys_clone. This is a
good thing because it allows us the ability to detect each one separately.
Again, once this has been done, the administrator can then run kstat –P and
ps-ef and compare the two to figure out what processes are running and hiding.
The last rootkit we will
look at is rkit. This rootkit does not hide itself as well as the other two do.
As a matter of fact it only changes sys_setuid and can be found using kstat –M.
LKM
rootkits can make a system administrator’s life a nightmare. They are hard to
detect, but using tools like kstat and understanding what the rootkit changes
can make our life easier. Since tools like kstat are available, it would help
systems administrators if they took a “picture” of the sys_call_table after a
fresh install and any upgrades. This will help in identifying the good from the
bad. Although I have not had the
time, one could write a shell | perl script to automate this process and check
weekly, daily, every 10 minutes or whatever.
http://s0ftpj.org/en/site.html
http://members.prestige.net/tmiller12/papers/KNARK.htm
LKM
rootkits
http://packetstorm.securify.com/groups/thc/LKM_HACKING.html
Toby
Miller is a GIAC Certified Analyst and MCP. He is currently working towards his
CISSP and RHCE. Toby has contributed to 2 books, written papers for SANS,
Securityfocus, performs Risk Assessments and runs a lab for a living. For
entertainment Toby likes to analyze exploit signatures on his home network. Toby
can be reached at tmiller@va.prestige.net.