System Monitoring

System monitoring refers to tools and methods for ensuring a computing platform is functioning as expected. You will need a solid understanding of the Linux material from chapters 3-7, 9, 11, 29. System monitoring will overlap with our other major course topics of networking and security. I recommend that you read the tutorial at http://www.garykessler.net/library/tcpip.html, material in class Chapter 14 of our text.

Monitoring includes at least the following areas.

System monitoring (OS)

• Tools to monitor OS system health:
  • CPU/memory/disk IO:  vmstat, top, iostat
    
  • /proc   :
    • stat :  see this for further info
    • loadavg  :  CPU load info
      
    • meminfo and cpuinfo :  useful summary of hardware
    • process specific info : /proc/[pid]/stat
  • Monitor users : fuser, netstats, iostat,
• logs
  • syslogd provides APIs for apps and kernel code to log information- :
    • Refer to this link: Syslog/log files
    • Ubuntu uses rsyslogd rather than syslogd. As with Syslog, it can provide interfaces for programs:
  • Sockets (to send messages to localhost daemon or to remote logger, need to enable in /etc/rsyslog.conf)
  • Scripts using the logger function which provides a centralized interface and management of the many possible logs on a Unix system. 
  • For example the following logs the message tagged with the  keyword ‘CPSC424MON' and the process ID,
    • logger -t CPSC424MON "Running CPSC424 SysMonitor!"
    • To watch the tail of syslog, try “watch tail /var/log/syslog”   
• Messaging :
  • To all users:
    • wall
    • talk, mesg
  • To desktop:
    • notify-send -a bash "First part: " "Second Part"
      
    • See the alert alias




• Control tools
  • Support for periodic processes (Ch 9, cron )
    • Cron is a method by which ‘jobs’ (i.e., programs or scripts) can be run asynchronously.  Cron is useful to run periodic tasks, like monitoring the health of a system.  Chapter 9 of the text provides a brief overview of cron.  The highlights are:
      • Cron is a system daemon program that provides the interface to allow us to schedule jobs and it provides the basic mechanism for running the jobs at the correct time.
    • A cron configuration file (a crontab) holds the information needed by cron.  To isolate users from accessing the actual crontab information (which is in /var/spool/cron), users  issue a ‘cron -e
    • Each user is allowed a single crontab. Let’s say we want to run the script ‘hw2.sh’ to run at 11:59 pm each night.
      • 11 56 * * * hw2.sh

     
    Chapter 11 of the text overviews  Syslog, a system event logger.  This provides a centralized interface and management of the many possible logs on a Unix system.  Our scripts can use the logger function to write log messages.  For example the following logs the message tagged with the  keyword ‘syscheck’  the process ID,   

    • logger -t CPSC424MON "Running CPSC424 SysMonitor!"
    • To watch the tail of syslog, try “watch tail /var/log/syslog”   

     

  • scheduling nice/renice
  • sysctl interface
  • SNMP - Simple Network Management Protocol: feedback on components for monitoring, control/set SNMP MIBs to change behaviors of components.

Network Monitoring -

this is an extension of System Monitoring. The area includes at least the following topics:

• OS and Network protocol support for monitoring
  • SNMP - protocol to support the collection of performance data from systems
  • /proc
• Network monitor tools
  • scripts:
  • programs/frameworks
    • Nagios
• Log files:
• Analysis tools (many of these are security oriented)
  • tcpdump, tcptrace, and other tools that analyze network traffic based on either live streams or packet trace files
  • netstats,
  • Port scanning (nmap)
  • Intrusion detection (SNORT)
  • Bro
  • Honeypots
• Control tools
  • Linux Firewall
  • tc/netem - traffic control / network emulation

Security Monitoring

Again, this is an extension of the previous monitoring items.. The area includes at least the following topics:

• OS and Network protocol support for monitoring
  • SNMP - protocol to support the collection of performance data from systems
  • /proc
• Log files
• monitor tools :
  • scripts: detect breakins, Denial of Service attacks
  • malware detection
  • IDS -
• Forensic Analysis tools
  • tcpdump, tcptrace, and other tools that analyze network traffic based on either live streams or packet trace files
  • log file analysis
  • 'dd' command , to create an image of a drive for forensic analysis (replace /dev/sda1 with the correct filesystem name)
    • dd if=/dev/sda1 of=/home/andrew/newimage.dd bs=512 conv=noerror,sync
  • foremost
  • paladin forensic (live CD)
  • Port scanning (nmap)
  • Intrusion detection (SNORT, Bro)
  • Honeypots
  • Tools used after an attack or breach :
    • forensics
• Control tools
  • Linux Firewall
  • tc/netem - traffic control / network emulation

Repairing a damaged system

Some sort of error or failure might cause an OS from properly booting. Historically with Unix, issues related to improper shutdown or reboots can cause file system errors. If the OS is able to boot in single user or some form of error/recovery mode, fsck can usually fix file system issues.If the system will not boot, you can boot on a repair CD or USB drive. Rescue disks listed below can be used (note: for a VM system, I would first try to use the VMManager program on the Host computer to repair a VM system.

• Hiren Boot CD
• FalconFour Boot
• System Rescue CD
• Trinity Rescue Kit
• Or the much simpler boot-repair-disk

Additionally, Linux distributions such the ones found below are lightweight bootable versions of Linux that contain a host of handy tools to fix common problems, recover data, transfer data, scan for viruses, manage partitions, etc.

• PuppyLinux (http://puppylinux.org/main/Overview%20and%20Getting%20Started.htm)
• Ubuntu LiveCD (https://help.ubuntu.com/community/LiveCD)
• Knoppix (http://knoppix.net/)
• ALT Linux Rescue (http://en.altlinux.org/Rescue)
• Frenzy LiveCD (http://frenzy.org.ua/eng/)

Finally, you could also try a Rescue Disc from a popular antivirus vendor, such as:

• AVG Rescue CD (http://www.avg.com/gb-en/avg-rescue-cd)
• F-Secure Rescue CD (http://www.f-secure.com/en/web/labs_global/removal-tools/-/carousel/view/142)
• Avira AntiVir Rescue System (http://www.avira.com/en/download/product/avira-antivir-rescue-system)
• Comodo Rescue Disk (https://www.comodo.com/business-security/network-protection/rescue-disk.php)

Last update: 3/2/2017