Home » Groups » Software and Computing

Samhain and Beltane notes

Updated on Wed, 2017-10-11 10:23. Originally created by wbetts on 2014-07-23 12:39.
Initial thoughts prior to any configuration or use:

Several configuration options to consider right away, some coming with costs...

--enable-suidcheck :
seems like a good thing to do, but noted as an I/O-expensive option, see http://www.la-samhna.de/samhain/manual/suidchk.html for possible mitigation options and possible actions when found.  Question: does the use of "inotify" (available as an option in Samhain 3.0 and up) elminate the I/O associated with this option?

--with-kcheck=/path/to/System.map:
Detecting kernel modifications (potentially finding rootkits that wouldn't otherwise be detected).  Would like to use this, but it looks like a pain to use in practice.  Each kernel version needs configuration parameters, and the kernel module it uses has to be compiled for each kernel.  Samhain itself does not seem to provide any way to automate these steps (eg. DKMS for the kernel module would be nice).  The configuration parameters can be read from text files (System.map), so it might be possible to automate updates of the Samhain configuration.   Additional question however: one of Samhain's selling points is a "central" configuration - how to customise *that* for ~tens of kernel versions in service at any time on the order of 100 systems simultaneously?  (Note too that kernel errata come out roughly once a month, systems are rebooted mostly independently at various times.)  Going to leave that for consideration later, but will start with it enabled on the first couple of systems.

--enable-login-watch : also has options for detecting "outlier" logins, first logins by user and host, and "off-hours" logins (this last is not useful for us!).  Will try it, but perhaps only a good idea for STAR on servers with very small number of users (eg. db and web servers).

--enable-userfiles : seems harmless enough, will try it
 
--enable-xml-log --with-database=mysql :
requires a mysql server.  Note that having a database log server is necessary in order to use Beltane.  It is recommended to have the MySQL server on the same server as the yule service.

--enable-network=server or --enable-network=client : self-explanatory

--enable-base=B1,B2 : on the very first build, a key will be generated automatically (used for verifying email and logfile signatures).  It is suggested that future builds use the same key, which can be specified with B1 and B2 as seeds.

Not using the following initially:

--enable-mounts-check : doesn't seem terribly useful, plus will have variations from host to host.

--with-prelude : we don't use Prelude, and jumping into that is well out of scope at this point.

--with-gpg=/path/to/gpg : I don't understand some of the details - does the samhain client on each host do the gpg checks, in which case the key has to be distributed to root's keyring on each host?

stealthy features : the samhain executable and configuration files can be obfuscated, but it doesn't seem worthwhile at this point (to be effective, would then also have to obscure the init scripts, keep the "samhain_stealth" utility off each system except as needed (or also obfuscate it!), avoid deploying as an RPM (or do it stealthily too) and who knows what else).  Using the stealthy kernel module also seems fraught with pitfalls, so not going to try that...


Beltane:

The Beltane web interface has two versions.  The free one doesn't scale well beyond a few hosts and doesn't have several nice features (ie. multiple user logins).  The full-featured version costs 200Euros (a potential problem as a non-US entitiy, but we'll deal with that if it comes to it).  Assuming Samhain itself seems ok, will start with the free Beltane for demo purposes.

Initial set-up and configuration steps:

Since dashboard1 has never had a running mysql server, first start that, which initialises mysql's internal database and tables, then use the /usr/bin/mysql_secure_installation to secure the initial instance. (NB, no external access through iptables yet - TBD - may need to be accessed by remote samhain processes)

[root@dashboard1 samhain-3.1.1]# mysql -p -u root < ./sql_init/samhain.mysql.init

[root@dashboard1 samhain-3.1.1]# mysql -p -u root
GRANT SELECT,INSERT ON samhain.log TO 'samhain'@'localhost';
SET PASSWORD for 'samhain'@'localhost' = PASSWORD('........');
FLUSH PRIVILEGES;


[root@dashboard1 samhain-3.1.1]# ./configure --enable-network=server --enable-suidcheck --with-kcheck=/boot/System.map-2.6.32-431.17.1.el6.x86_64 --enable-login-watch --enable-userfiles --enable-xml-log --with-database=mysql
<snip out lots of typical confiugre output>
 samhain has been configured as follows:
     System binaries: /usr/local/sbin
  Configuration file: /etc/yulerc
        Manual pages: /usr/local/man
                Data: /var/lib/yule
            PID file: /var/run/yule.pid
            Log file: /var/log/yule/yule_log
            Base key: XXXXXXXXXX,YYYYYYYYYY

    Selected rc file: yulerc

[Note the Base key is obscured above, but should be used in future configuration builds with --enable-base=XXXXXXXXXX,YYYYYYYYYY]

[root@dashboard1 samhain-3.1.1]# make
<snip>
read_kcode: access /proc/kmem: No such file or directory

NOTE:  kern_head: apparently you have no /dev/kmem, and the
       samhain_kmem module is not loaded
       If you get this message, please proceed as follows:
       $ make samhain_kmem.ko
       $ sudo /sbin/insmod samhain_kmem.ko; sudo ./kern_head > sh_ks.h; sudo /sbin/rmmod samhain_kmem
       $ make

make: *** [sh_ks.h] Error 1

[root@dashboard1 samhain-3.1.1]# make samhain_kmem.ko
make[1]: Entering directory `/root/Software/samhain-3.1.1/m_comp'
make -C /lib/modules/2.6.32-431.17.1.el6.x86_64/build modules SUBDIRS=/root/Software/samhain-3.1.1/m_comp KBUILD_VERBOSE=2
make[2]: Entering directory `/usr/src/kernels/2.6.32-431.17.1.el6.x86_64'
  CC [M]  /root/Software/samhain-3.1.1/m_comp/yule_kmem.o - due to target missing
  Building modules, stage 2.
  MODPOST 1 modules - due to target is PHONY
  CC      /root/Software/samhain-3.1.1/m_comp/yule_kmem.mod.o - due to target missing
  LD [M]  /root/Software/samhain-3.1.1/m_comp/yule_kmem.ko.unsigned - due to target missing
  NO SIGN [M] /root/Software/samhain-3.1.1/m_comp/yule_kmem.ko - due to target missing
make[2]: Leaving directory `/usr/src/kernels/2.6.32-431.17.1.el6.x86_64'
make[1]: Leaving directory `/root/Software/samhain-3.1.1/m_comp'
[root@dashboard1 samhain-3.1.1]# /sbin/insmod samhain_kmem.ko
[root@dashboard1 samhain-3.1.1]# lsmod |grep sam
[root@dashboard1 samhain-3.1.1]# rmmod samhain_kmem
ERROR: Module samhain_kmem does not exist in /proc/modules
[root@dashboard1 samhain-3.1.1]# ./kern_head > sh_ks.h
[root@dashboard1 samhain-3.1.1]# ls -l sh_ks.h
-rw-r--r--. 1 root root 16017 Jul 23 19:44 sh_ks.h
[root@dashboard1 samhain-3.1.1]# less sh_ks.h
[root@dashboard1 samhain-3.1.1]# rmmod samhain_kmem
ERROR: Module samhain_kmem does not exist in /proc/modules

Huh?  Did that "work" or not?  There is no m_comp directory, but there is a freshly timestamped samhain_kmem.ko and a /proc/kmem, which I think is new (I did not check for it in advance, but there is no such thing on dbbak which until a couple of days ago was almost identical to dashboard1).

Not sure what to make of that kernel module stuff, but "make" completed after that without any apparent errors... 

Now could try "make rpm", "make rpm-light", or "make run" for distribution, but why bother, especially for the yule server side.

So install it!

[root@dashboard1 samhain-3.1.1]# make install
 cp samhain_kmem.ko /lib/modules/2.6.32-431.17.1.el6.x86_64/yule_kmem.ko
 depmod -a || /sbin/depmod -a
 /usr/bin/install -c -s -m 700 samhain_setpwd /usr/local/sbin/yule_setpwd
 ./sstrip /usr/local/sbin/yule_setpwd
 /usr/bin/install -c -s -m 700 yule /usr/local/sbin/yule
 ./sstrip /usr/local/sbin/yule
 /usr/bin/install -c -s -m 700 yulectl /usr/local/sbin/yulectl
 ./sstrip /usr/local/sbin/yulectl
/bin/sh ./mkinstalldirs /usr/local/man/man8
mkdir /usr/local/man
mkdir /usr/local/man/man8
/bin/sh ./mkinstalldirs /usr/local/man/man5
mkdir /usr/local/man/man5
  /usr/bin/install -c -m 644 ./man/samhain.8 /usr/local/man/man8/yule.8
  /usr/bin/install -c -m 644 ./man/samhainrc.5 /usr/local/man/man5/yulerc.5
gcc  -DHAVE_CONFIG_H -I. -I./include  -I/usr/include/mysql -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fwrapv -fPIC -DUNIV_LINUX -DUNIV_LINUX -O2 -D_FORTIFY_SOURCE=2 -fstack-protector-all -fPIE -Wall -W  -fno-strength-reduce -fno-omit-frame-pointer -Wno-empty-body  -DSH_WITH_SERVER -DSH_IDENT=\"daemon\" -DTRUST_MAIN -DSL_ALWAYS_TRUSTED=0 -o trustfile ./src/trustfile.c
mkdir /var/log/yule
mkdir /var/lib/yule
./samhain-install.sh --destdir= --express --verbose install-data
  cp yulerc samhainrc.pre
  mv -f samhainrc.pre samhainrc.install
  ./samhain-install.sh --install-sh  -m 600 samhainrc.install /etc/yulerc
  chown daemon /etc/yulerc
  checking whether paths are trustworthy
  configuration file /etc/yulerc ... OK
  state directory /var/run ... OK
  state directory /var/log/yule ... OK
  data directory /var/lib/yule ... OK

  -----------------------------------------------------
  The server will run as user daemon if started with
  root privileges, otherwise as the user of the parent
  process (use --enable-identity=USER to change).

  You may want to use: make install-user

    - to add the user daemon (if not existing already)
    - to chown the data            directory /var/lib/yule
    - to chown the log file        directory /var/log/yule
    - to chown the configuration   file      /etc/yulerc
  -----------------------------------------------------

  You can use 'samhain-install.sh uninstall' for uninstalling
  i.e. you might consider saving that script for future use

  Use 'make install-boot' if you want yule to start on system boot

[root@dashboard1 samhain-3.1.1]# make install-boot
./samhain-install.sh --destdir= --express --verbose install-boot
  Linux Standard Base system detected
  ./samhain-install.sh --install-sh -m 700  init/samhain.startLSB /etc/init.d/yule
  /usr/lib/lsb/install_initd /etc/init.d/yule
installing init scripts completed


OK, seems like it is done, and the yule service starts, listening on tcp port 49777, which is the default.  (The port is hard-coded, though can be changed in the code prior to building or overridden with a configuration option at build time (configure --with-port=N) - there is no configuration option after the build - that is, it cannot be changed in a configuration file).

Moving on to the client...

First, increased the inotify watch limit with  "echo 1048576 >/proc/sys/fs/inotify/max_user_watches" (default value is 8192) and set in /etc/syctl.conf (fs.inotify.max_user_watches=1048576)

Now building the client:

[root@dashboard1 samhain-3.1.1]# ./configure --enable-network=client --enable-suidcheck --with-kcheck=/boot/System.map-2.6.32-431.17.1.el6.x86_64 --enable-login-watch --enable-userfiles --enable-xml-log --with-database=mysql --enable-base=XXXXXXXXXX,YYYYYYYYYY
<snip>
 samhain has been configured as follows:
     System binaries: /usr/local/sbin
  Configuration file: /etc/samhainrc
        Manual pages: /usr/local/man
                Data: /var/lib/samhain
            PID file: /var/run/samhain.pid
            Log file: /var/log/samhain_log
            Base key: 1615134800,1885569214

    Selected rc file: samhainrc.linux

[root@dashboard1 samhain-3.1.1]# make

The make plows through and gives no indication of any error, so now to install:

[root@dashboard1 samhain-3.1.1]# make install
 cp samhain_kmem.ko /lib/modules/2.6.32-431.17.1.el6.x86_64/samhain_kmem.ko
 depmod -a || /sbin/depmod -a
 /usr/bin/install -c -s -m 700 samhain_setpwd /usr/local/sbin/samhain_setpwd
 ./sstrip /usr/local/sbin/samhain_setpwd
 /usr/bin/install -c -s -m 700 samhain /usr/local/sbin/samhain
 ./sstrip /usr/local/sbin/samhain
/bin/sh ./mkinstalldirs /usr/local/man/man8
/bin/sh ./mkinstalldirs /usr/local/man/man5
  /usr/bin/install -c -m 644 ./man/samhain.8 /usr/local/man/man8/samhain.8
  /usr/bin/install -c -m 644 ./man/samhainrc.5 /usr/local/man/man5/samhainrc.5
gcc  -DHAVE_CONFIG_H -I. -I./include  -I/usr/include/mysql -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fwrapv -fPIC -DUNIV_LINUX -DUNIV_LINUX -O2 -D_FORTIFY_SOURCE=2 -fstack-protector-all -pthread -DUSE_MALLOC_LOCK=1 -fPIE -Wall -W  -fno-strength-reduce -fno-omit-frame-pointer -Wno-empty-body  -DSH_WITH_CLIENT -DSH_IDENT=\"daemon\" -DTRUST_MAIN -DSL_ALWAYS_TRUSTED=0 -o trustfile ./src/trustfile.c
mkdir /var/lib/samhain
./samhain-install.sh --destdir= --express --verbose install-data
  cp samhainrc.linux samhainrc
  cp samhainrc samhainrc.pre
  mv -f samhainrc.pre samhainrc.install
  ./samhain-install.sh --install-sh  -m 600 samhainrc.install /etc/samhainrc
  checking whether paths are trustworthy
  configuration file /etc/samhainrc ... OK
  state directory /var/run ... OK
  state directory /var/log ... OK
  data directory /var/lib/samhain ... OK

  You can use 'samhain-install.sh uninstall' for uninstalling
  i.e. you might consider saving that script for future use

  Use 'make install-boot' if you want samhain to start on system boot

[root@dashboard1 samhain-3.1.1]# make install-boot
./samhain-install.sh --destdir= --express --verbose install-boot
  Linux Standard Base system detected
  ./samhain-install.sh --install-sh -m 700  init/samhain.startLSB /etc/init.d/samhain
  /usr/lib/lsb/install_initd /etc/init.d/samhain
installing init scripts completed

Now to look into the client configuration file, /etc/samhainrc and add a few things:
SetLogServer = dashboard1.starp.bnl.gov
SetTimeServer = daqman.starp.bnl.gov
SetMailAddress=wbetts@bnl.gov
SetMailRelay=bnl.gov


At this point, I stopped updating this blog with all the details.  Some things of note though:
  • the original samhain client configuration is at /etc/samhainrc.first.  This should be kept as a reference for changes made, which have grown a bunch since the short list above, some of which are to implement the changes below
  • Set the threshold levels for email and logging to warn and notice respecitvely.  (Info notes every check made, which is a huge number of events.)
  • set some things to be ignored - certain directories used by sFlow, cfengine, OCS-inventory and yum for instance that change frequently.
  • added database logging
  • added the communication key to the samhain executable (without which yule rejects the connection from the client) and "registered" the client in the yule configuration
  • Lesson learned: "samhain -t init" does not reinitialize an existing database, but rather *appends* a new baseline to the old one - then upon restarting samhain, it reads the *first* baseline and ignores the appended one(s).  To update the baseline, either delete the file and do an "init" or "use samain -t update"
I was then going to try getting Beltane working, at which point I discovered flaws in the above - the samhain client needs to be *built* with the option to use a baseline database (and configuration file) served by yule, rather than a local file.  This is the only way Beltane can be used to update the baseline configurations.  So there are two databases (tables really) - one for detection logs and one for the baselines - the distinction between the two was lost on me at first.  The baselines are stored as individual files on the server, while the detection logs are (in our case) stored in MySQL and text files.  Side note - if using the local database, the default file appears to be /var/lib/samhain/samhain_file and it is about 60MB.

So some additions were made to the configure step to build samhain, and the line is now:
./configure --enable-network=client --enable-suidcheck --with-kcheck=/boot/System.map-2.6.32-431.17.1.el6.x86_64 --enable-login-watch \
--enable-userfiles --enable-xml-log --with-database=mysql --enable-base=XXXXXXXXXX,YYYYYYY --with-logserver=dashboard1.starp.bnl.gov \
--with-data-file=REQ_FROM_SERVER/var/lib/samhain/samhain_file --with-config-file=REQ_FROM_SERVER/etc/samhainrc

A samhain configuration file and baseline file database have been added to /var/lib/yule as rc.dashboard1.starp.bnl.gov and file.dashboard1.starp.bnl.gov to be served by yule to the samhain client at startup.

I also tried yule's chroot option, but could not get it to work, at least not through the init script.  The service would not start, saying "Service yule is not installed" with return code "5", which the documentation translates to "Program is not installed".  I could not track down the cause.  The strange thing is that it seemed to only have a problem when using the "start" command.  Samhain itself would start fine without any options.  I think I followed the documentation and tried going through the steps several times, but it was the same result each time.


Beltane:


Created two new users (yule and beltane) and made them both members of a newly created samhain group.  Had to go back to rebuild yule to run as the yule user (as opposed to the daemon user, which it uses by default), adding "--enable-identity=yule" to the ./configure line.  Some chown/chgrp/chmod steps get yule and beltane access to the files they need.

To build beltane:

./configure --enable-mod-php --with-php-extension=php --with-user=beltane --with-php-dir=/var/www/html/beltane --enable-postinstall=suid


After some struggle, beltane is working.   The biggest issue was a misunderstanding of the expected data flow.  Based on a misinterpretation of the documentation of the log_ref database column, I thought each samhain *client* had to write directly to the database.  Turns out that Beltane, as written, will not work with entries written directly by the samhain clients.  (It is a trivial change in the sql.php code to make it work that way, however, that introduces several security concerns, so it is genuinely better to have samhain clients send the data to yule, and have yule put the records in the database on the clients' behalves.)

Another sticking point was getting both yule and beltrane to use FQDNs - it is not as obvious as it would seem for yule.  Yule has a "-q" command-line option for this, but getting it to be recognised from within the init script was surprisingly difficult (I must be missing something fundamental here.)  What I wound up doing was starting yule with a "bash -c" invocation within /etc/init.d/yule (and in the restart case).   Then there are the bunch of details of user accounts, file ownership and permissions for beltane, yule and apache (at the file system level) and beltane and samhain (in the database - note that as I deployed it there is a somewhat incongruous use of "samhain" as a database user instead of "yule").

Miscellaneous notes:

see http://www.la-samhna.de/samhain/manual/log-file-rotation.html for log file rotation suggestions  Also see the bottom of http://www.la-samhna.de/samhain/manual/filedef.html for additional log rotate information

For monitoring WHO changed a file, audit functionality can be used.  See http://www.la-samhna.de/samhain/manual/filedef.html


Question:  why does yule need a kernel module?  samhain, sure, but yule needs one too?

It is worth a mention that the samhain forum online has *very* little activity, but does get prompt replies from rainer, the samhain developer.

Groups: