The general issues to take into consideration when troubleshooting your system are listed below. You should read them carefully before trying to solve more specific problems.
-
You should always remember where you are currently located in your terminal. Check it periodically using the
pwd
,hostname
,ifconfig
,cat /proc/vz/veinfo
commands. One and the same command executed inside a virtual machine or container and on the hardware node can lead to very different results. You can also set up thePS1
environment variable to show the full path in thebash
prompt. To do this, add these lines to/root/.bash_profile
:PS1="[\u@\h \w]$ " export PS1
-
If the hardware node slows down, use
vmstat
,ps
(ps axfw
),dmesg
,htop
(vztop
) to find out what is happening, never reboot the machine without investigation. If no thinking helps restore the normal operation, use the Alt+SysRq sequences to dump the memory (showMem
) and processes (showPc
). -
Do not run any binary or script that belongs to a container directly from the hardware node, for example, do not ever do this:
cd /vz/root/99/etc/init.d ./httpd status
Any script inside a container could have been changed to whatever the container owner chooses: it could have been trojaned, replaced to something like
rm -rf
, etc. You can use onlyprlctl exec/prlctl enter
to execute programs inside a container. -
Do not use init scripts on the hardware node. An init script may use
killall
to stop a service, which means that all similar processes will be killed in all containers. You can check/var/run/Service.pid
and kill the correspondent process explicitly. -
You must be able to detect any rootkit inside a container. It is recommended to use the
chkrootkit
package for detection (you can download the latest version from http://www.chkrootkit.org), or at least runrpm -Va|grep "S.5"
to check up if the MD5 sum has changed for any RPM file.
You can also run
nmap
, for example:# nmap -p 1-65535 192.168.0.1 Starting nmap V. 2.54BETA22 ( www.insecure.org/nmap/ ) Interesting ports on (192.168.0.1): (The 65531 ports scanned but not shown below are in state: closed) Port State Service 21/tcp open ftp 22/tcp open ssh 80/tcp open http 111/tcp open sunrpc Nmap run completed -- 1 IP address (1 host up) scanned in 169 seconds
to check if any ports are open that should normally be closed.
That could however be a problem to remove a rootkit from a container and make sure it is 100% removed. If you’re not sure, create a new container for that customer and migrate his/her sites and mail there.
-
Check the
/var/log/
directory on the hardware node to find out what is happening on the system. There are a number of log files that are maintained by the system and OpenVZ (theboot.log
,messages
, etc.), but other services and programs may also put their own log files here depending on your distribution of Linux and the services and applications that you are running. For example, there may be logs associated with running a mail server (themaillog
file), automatic tasks (thecron
file), and others. However, the first place to look into when you are troubleshooting is the/var/log/messages
log file. It contains the boot messages when the system came up as well as other status messages as the system runs. Errors with I/O, networking, and other general system errors are reported in this file. So, we recommend that you read to themessages
log file first and then proceed with the other files from the/var/log/
directory. - Subscribe to bug tracking lists. You should keep track of new public DoS tools or remote exploits for the software and install them into containers or at hardware nodes.
-
When using
iptables
, there is a simple rule for Chains usage to help protect both the hardware node and its containers:- use INPUT, OUTPUT to filter packets that come in/out the hardware node
- use FORWARD to filter packets that are designated for containers