
Investigations and Forensic Analysis
-------------------------------------

Remarkably, after some ten years of working on Internet investigations,
we see people using the same tools and techniques that were used at the
time that the Internet worm made security a concern to system administrators
on the net.  Why hasn't more progress been made?

Certainly it's a difficult task.  However, we feel that it's high time
that more people knew about (and how to effectively utilize) MAC times,
the possibilities of exploring - and recovering - Unix files that were
removed or destroyed, capturing processes and their associated information
and a fair bit more.  If nothing else, we hope that when a Unix system 
has been broken into that the owner of the computer would have a chance 
of capturing (if not understanding) much of the crucial forensics data 
that is needed in order to understand what has happened on that system.

To do any real sort of analysis of a system you'd like to be able to 
examine fairly extensively the contents of - at least - its:

	o	network
	o	memory
	o	disk

In addition you'll require a host of commands that will tell you what 
sort of system it is, what the OS is, patches, startup files, etc.
While there is no software that can replace someone who knows their 
systems very well, we hope that this set of tools - The Coroner's Toolkit
(TCT) might be at least a start of what one needs to start an adequate
analysis.


The Coroner's Toolkit (TCT)
----------------------------

TCT is a collection of tools - some large, some small, some in perl, some
in C (but with complete source code provided, as with all reasonable 
software) - that are all either orientated towards gathering or analyzing 
data on a Unix system.  There is no single task or ultimate goal that they 
are directed to, but if there was a theme it'd be the reconstruction of 
the past - determining as much as possible what happened in the past 
with a static snapshot of a system.  And, as with all of our work we 
hope that we can educate and inspire people as well (but obviously that is a 
much more difficult goal to accomplish).

It has been designed primarily for people in the trenches - systems
administrators, security response teams, security investigators, etc.
However we hope that people are not only able to utilize this to 
assist in ongoing investigations but for an aid to better understand 
how systems operate and how better to defend themselves versus security
incidents in the future.

There are three major parts to TCT:

	o	grave-robber
	o	unrm & lazarus
	o	mactime

Each has its own man page as well as additional documentation.

Of course if there is activity going on on the system involved,
files can be changed, processes started and lost, etc. that will
never be noticed by TCT.  IT IS A STATIC ANALYSIS TOOL.

At the end of the data collection all data should be moved to safety.
At the very least MD5 or PGP signatures should be taken offline so
that you can validate the results in the future.  We further recommend 
that you use PGP to sign the data.  To sign with PGP2:

    		% pgp -bs foo.tar.gz

	The resulting signature file is "foo.tar.gz.sig".

	To verify, you can:

		% pgp foo.tar.gz

	Depending on how smart your PGP version is it will either 
	find foo.tar.gz.sig and foo.tar.gz, or it will ask some questions.

Obviously you must have a previously known/generated PGP key for this.


Grave Robbing and Electronic Dumpster Diving for Fun and Profit
----------------------------------------------------------------

The "grave-robber" tool is at the heart of much of TCT.  It is little
more than a data capturing tool, mostly running various shell commands 
in an attempt to gather the basic system information and to save some
of the files on the system that are necessary to analyze it.

For most investigations simply *getting* the data is half the battle.
We've found that automation is simply the only way to reliably capture
all the necessary data.

TCT attempts to capture as much data as is both useful and pragmatic
(and as time goes on it will hopefully become more comprehensive).

HOWEVER, one should note - due to the extreme complexities of modern 
systems, for the most part (except for MAC times and unallocated data)
no effort has been made to *interpret* data, simply collect it.

The grave robber is most useful when run over an entire system, capturing
as much information as possible.  Point it at the root file system if space
permits.  And, while TCT doesn't need to be run as root, running it
from an unprivileged account will prevent you from capturing files and 
processes that the account doesn't own.

The last thing the grave-robber does is create MD5 signatures of all
the output that was created during its latest run, in a file named
"data/hostname/MD5_all".  If you can't take all the output of TCT off
the system (you should always try this!) it would be a good idea to at 
least keep the MD5 output off-line so the results can be validated later.

Read the grave-robber.README file for more information about this.

*** You may wish to run the grave-robber with the -p flag first; this
*** flag skips over grabbing all the process memory, which can crash
*** a system.


Unrm & Lazarus, or Bringing the Dead to Life
---------------------------------------------

Many people think that destroyed or lost data - typically thought to 
be lost in Unix systems - can essentially never be recovered, either in 
whole or in part.  We are here to tell you something different.

It is important to note that UNRM & LAZARUS ARE NOT RUN IN THE 
GRAVE-ROBBER!  You need to run these separately.

Unrm and Lazarus work together for the forces of sloth and extreme
wasting of disk space.  Unrm is a C program that is essentially an 
electronic dumpster diving tool - it roots out all information on 
a Unix file system that is not currently allocated.

That means if you have a 10 GB disk and you're currently using 2,
unrm would generate 8 gigabytes of data.  Since UNIX tends to allocate
file system blocks for different files in different places you must
save this resulting data on a separate disk or you will destroy
(overwrite) your potential forensic data.

Lazarus takes the data from unrm (or, actually, from any data source,
such as RAM, swap space, etc.) and attempts to bring some order into
what is generally perceived of as garbage.  Perhaps it is most useful
when the HTML output option is used and a clickable map is generated
that allows browsing of the interpreted data.

Of course if someone really wants data deleted they can overwrite the
information with zeros or random garbage.  Most data loss is not
done intentionally, however, plus *CURRENTLY* intruders that attempt
to destroy data are not that careful, however (this will presumably
change, and perhaps quickly), and a significant amount of information
or data can at times be recovered.

Read the lazarus.README file for more information about these two programs.


MAC times - the Ghost in the Shell
-----------------------------------

Mtimes, atimes, and ctimes (or MAC times) are properties of a file that,
if not intentionally modified or obfuscated, combine to be one of the most
useful tools in forensic analysis.  The mtime contains the last time the 
file was modified, the atime holds the last access time (for reading or 
execution), and the ctime is the last time the file status has been 
changed (status is the meta data that is typically changed by a chown, chmod,
etc. shell command).

Unfortunately MAC times are easily modified and are extremely ephemeral -
the next time someone accesses or modifies a file in some way the prior
values are lost, so haste must be exercised when collecting this 
information (typically gotten with a stat (2v) function call - the
grave-robber program collects this on all directories it traverses).

However, if you are reasonably certain that the MAC times have not been
distorted in some way, they are perhaps the surest way to find out what
has been happening on a system.  See "mactime.README" and the mactime
man page for more information.

