Server Trubleshooting

Modified on Tue, 13 Feb, 2018 at 3:06 PM

Server Troubleshooting

Few daily important command

===============================

1) cal ---> Display calendar for current month
2) date ---> Display system date and time
3) date -s '<value>' ---> Change system date and time in mm/dd/yy
4) hwclock ---> Display the hardware clock
5) hwclock --hctosys ---> Set the system time from the hardware clock
6) clear ---> Clear the screen
7) ls ---> List content
8) ls -lh ---> List content in long listing format
9) cd <directory> ---> Change directory to.
10) cd <directory> ---> Change directory to...
11) cd .. ---> Change to parent directory
12) cd - ---> Change to previous directory
13) cd ---> Change to home directory
14) pwd ---> Print work (current) directory
15) pwd -P ---> Print parent working dir of this symlink dir
16) mkdir <directory> ---> Make directory
17) mkdir -p <directory> ---> Make parent directories also if it does not exist
18) uptime ---> Display for how long the system has been running
19) users ---> Display no. of users currently logged in
20) whoami ---> Display username of current user
21) who ---> Display users logged in the system with their respective terminals and time since logged in
22) who am i ---> Display current user, terminal and uptime
23) w ---> Display is details which files are open on which terminal
24) exit ---> Exit from console
25) init 6 /reboot ---> Reboot the system
26) init 0 / halt / poweroff ---> Halt the system after shutdown
27) top ---> To check the load , I/O and uptime

28) tm | grep <tracking word> --> To track word in the on going mail log

29) telnet <server IP> 25 ---> To check the connectivity on SMTP port 25 , we must see some Banner like "ESMTP ESMTP"

30) telnet <server IP> 110 ---> To check the connectivity on POP3 port 110

31) telnet <server IP> 143 ---> To check the connectivity on IMAP port 143 ,

How to check the Hard-disk Space.

======================================

"df -khT" is command to check the Had-disk space

[root@jayesh ~]# df -khT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda3 ext3 3.0G 1.2G 1.6G 43% /
/dev/sda2 ext3 4.3G 581M 3.5G 15% /var
/dev/sda1 ext3 190M 12M 169M 7% /boot
tmpfs tmpfs 282M 0 282M 0% /dev/shm

How to check IP of Email server

=====================================

"ipconfig" is command by which we can check the all IP of mail server.

[root@jayesh ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:41:E3:AB
inet addr:192.168.30.223 Bcast:192.168.255.255 Mask:255.255.0.0
inet6 addr: fe80::20c:29ff:fe41:e3ab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:831850 errors:0 dropped:0 overruns:0 frame:0
TX packets:28820 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:104188780 (99.3 MiB) TX bytes:2937909 (2.8 MiB)
Interrupt:177 Base address:0x1400

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:5620 errors:0 dropped:0 overruns:0 frame:0
TX packets:5620 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:410473 (400.8 KiB) TX bytes:410473 (400.8 KiB)

Now here in above example we have only one IP for eth0 , if we have multiple ethernet cards with IP then we can check individual by following command

[root@jayesh ~]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:0C:29:41:E3:AB
inet addr:192.168.30.223 Bcast:192.168.255.255 Mask:255.255.0.0
inet6 addr: fe80::20c:29ff:fe41:e3ab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:843223 errors:0 dropped:0 overruns:0 frame:0
TX packets:28882 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:105207150 (100.3 MiB) TX bytes:2944836 (2.8 MiB)
Interrupt:177 Base address:0x1400

If you have another ethernet card say example eth1 , then we can check by either of the below command.

[root@jayesh ~]# ifconfig eth1.

To see all ethernet cards give below command.

[root@jayesh ~]# ifconfig | less

To come back to command prompt again type "q" to

How to see Gateway of the server

=======================================

"route -n" is the command by which we can see the server Gate IP.

Example :--

[root@jayesh ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.2.234 0.0.0.0 UG 0 0 0 eth0

Now here "192.168.2.234" is the Gateway IP of server.

How to delete the gateway temporary from command line

============================================================

"route del default" is command to delete the gateway.

Example :--

[root@jayesh ~]# route del default

[root@jayesh ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

How to re-add / newly add Gateway IP by command line

=========================================================

route add default gw <IP address> dev <ethernet card> is command to add / configure the gatway.

Example :--

[root@jayesh ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

[root@jayesh ~]# route add default gw 192.168.2.234 dev eth0

How to check ethernet link is UP or not

================================================

"ethtool" is the command by which we can check ethernet card' attributes / option and Link status

Example :--

[root@jayesh ~]# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000033 (51)
drv probe ifdown ifup
Link detected: yes

The Link "detected:yes" means , the patch cord is connected to Switch / HUB / Router.

The Link "detected:no" means , the patch cord is not connected to Switch / HUB / Router.

How to check the Mail queue of the Email server.

===============================================

"mailq" and "nmqt" is the command to see the email queue.

[root@jayesh ~]# mailq
-Queue ID- -----Size-- ----Arrival Time---- -Sender/Recipient-------

1904427C014 25472 Sat Oct 6 07:57:36 sender1@netcore.co.in
(connect to lfvsfcp10127.dn.net[130.94.69.225]:25: Connection refused)
recipient@lfvsfcp10127.dn.net

A28D9374004 2728 Sat Oct 6 14:49:43 sender2@netcore.co.in
(connect to p3nlhg941.shr.prod.phx3.secureserver.net[50.63.197.30]:25: Connection timed out)
recipient2@p3nlhg941.shr.prod.phx3.secureserver.net

F397C1274049 2067 Sat Oct 6 15:41:54 sender3@netcore.co.in
2 http://support.google.com/mail/bin/answer.py?answer=6558 nu6si871202icb.104 (in reply to RCPT TO command))
recipient@mfactorfireworks.com

-- 36831 Kbytes in 3 Requests.

[root@nlb2 ~]# nmqt
3

The above command means on server "3" emails are in queue.

How to check the Internet OR LAN connectivity.

==================================================

"ping" is the command by which we can check the internet connectivity.

Example :-- ping -c 3 <ip / host name>

LAN IP ping check

[jayesh@jayesh ~]$ ping 192.168.2.234 -c 3
PING 192.168.2.234 (192.168.2.234) 56(84) bytes of data.
64 bytes from 192.168.2.234: icmp_req=1 ttl=64 time=0.315 ms
64 bytes from 192.168.2.234: icmp_req=2 ttl=64 time=0.384 ms
64 bytes from 192.168.2.234: icmp_req=3 ttl=64 time=0.312 ms

--- 192.168.2.234 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.312/0.337/0.384/0.033 ms

Internet IP ping check

[root@jayesh ~]# ping yahoo.com -c 3
PING yahoo.com (98.138.253.109) 56(84) bytes of data.
64 bytes from ir1.fp.vip.ne1.yahoo.com (98.138.253.109): icmp_seq=1 ttl=45 time=369 ms
64 bytes from ir1.fp.vip.ne1.yahoo.com (98.138.253.109): icmp_seq=2 ttl=46 time=292 ms
64 bytes from ir1.fp.vip.ne1.yahoo.com (98.138.253.109): icmp_seq=3 ttl=46 time=377 ms

--- yahoo.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 292.231/346.275/377.582/38.381 ms

How to check Load

==============================

Observing the output of things like top , uptime, w and perhaps others, you certainly pay for the line load average

Try to understand the meaning of these numbers. So, in other words, the numbers reflect the number of blocking processes in the queue for execution at a specific time interval, namely, 1 minute, 5 minutes and 15 minutes, respectively. In this case, the blocking process - a process that is waiting for resources to continue. As a rule, there is the expectation of resources such as CPU, disk I / O or network I / O subsystem.

High values of load average load average indicate that the system can not cope with the load. . When it comes to the target server that is under heavy load, it is usually helpful to fine-tune the operating system (the network subsystem, the restriction on the number of open files, and so on). High CPU can also be caused by hardware problems, such as failure of the drive.

For the diagnosis of appeal to other useful data provided output top . Line CPU (s) contains information on the distribution of CPU time.The first two values directly reflect the work of CPU processing processes:

CPU LOAD

Prolonged high (99-100%) indicators point to the CPU as a bottleneck.

Parameter wa suggests a simple, connected to I / O:

I/O wait

Above 80% is considered normal and not quite clearly shows us that the processor spends a lot of time waiting for I / O (this usually means that the failure of HDD or NIC).

SAR command :--

We can Also check the Load and I/O with SAR command , The SAR command come with sysstat-7.0.2-1.el5.rpm package. Which need to be install.

Example :-- sar <-switch> <time interval in sec> <number of time o/p require>

For checking the i/owait of server give . "-u" switch use for checking the "iowait"

[root@jayesh ~]# sar -u 2 5
Linux 2.6.18-92.el5 (Jayesh) 10/10/2012

05:30:57 PM CPU %user %nice %system %iowait %steal %idle
05:30:59 PM all 75.08 0.00 21.45 0.33   0.66 2.48
05:31:01 PM all 72.55 0.00 20.75 0.82    0.49 5.39
05:31:03 PM all 72.33 0.00 21.67 2.33 0.67 3.00
05:31:05 PM all 75.99 0.00 13.74 0.17    1.16 8.94
05:31:07 PM all 74.63 0.00 17.18 0.82 1.15 6.22
Average: all 74.12 0.00 18.96 0.89    0.82 5.21

For checking the Server Load give . "-q" switch

[root@jayesh ~]# sar -q 2 5
Linux 2.6.18-92.el (jayesh) 10/10/2012

05:35:05 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
05:35:07 PM 5 246 11.18 11.63 12.03
05:35:09 PM 5 245 11.18 11.63 12.03
05:35:11 PM 9 245 11.09 11.60 12.02
05:35:13 PM 8 245 11.09 11.60 12.02
05:35:15 PM 6 245 11.09 11.60 12.02
Average: 7 245 11.13 11.61 12.02

How to check connectivity by telnet

===============================

For checking the connectivity from Windows desktop , Go to Start ---> Run ---> cmd and then execute the below commands.

Checking the connectivity on POP3 ( 110 port )

[root@jayesh ~]# telnet pop2.netcore.co.in 110
Trying 202.162.229.40...
Connected to pop2.netcore.co.in (202.162.229.40).
Escape character is '^]'.
+OK pop2.netcore.co.in Cyrus POP3 v2.4.12-Invoca-RPM-2.4.12-2 server ready <11008234458491230169.1349936105@pop2.netcore.co.in>

Checking the connectivity on IMAP ( 143 port )

Checking the connectivity on SMTP ( port 25 )

[root@jayesh ~]# telnet smtp3.netcore.co.in 25
Trying 202.162.240.180...
Connected to smtp3.netcore.co.in (202.162.240.180).
Escape character is '^]'.
220 ESMTP ESMTP

How to check whether Email id and password is correct

=========================================================

For checking whether Email ID and passwords are correct from Windows desktop , Go to Start ---> Run ---> cmd and then execute the below commands. Enter the authenticated ID and authenticate password which you use in Outlook or in Webmail. If we are not getting output mention in blue , then either your email id or password is incorrect.

[root@jayesh ~]# telnet pop2.netcore.co.in 143
Trying 202.162.229.40...
Connected to pop2.netcore.co.in ( 202.162.229.40).
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR] tower.netcore.co.in Cyrus IMAP v2.4.12-Invoca-RPM-2.4.12-2 server ready
a login jayesh.shinde@netcore.co.in password123
a OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY X-NETSCAPE LOGINDISABLED COMPRESS=DEFLATE IDLE] User logged in SESSIONID=<pop2.netcore.co.in-8649-1349952658-1>
^]
telnet> q
Connection closed.

[root@jayesh ~]# telnet 202.162.229.40 110
Trying 202.162.229.40
Connected to port2.netcore.co.in (202.162.229.40).
Escape character is '^]'.
+OK pop2.netcore.co.in Cyrus POP3 v2.4.12-Invoca-RPM-2.4.12-2 server ready <12775447681623682887.1349952689@pop2.netcore.co.in>
user jayesh.shinde@netcore.co.in
+OK Name is a valid mailbox
pass password123
+OK Mailbox locked and ready SESSIONID=<pop2.netcore.co.in-12165-1349952689-1>
^]
telnet> q
Connection closed.

How to send email by telnet + SMTP Auth

========================================

Telnet from your computer to the mail server.
$ telnet 192.168.30.223 25
You should receive a reply like:
Trying 192.168.30.223...
Connected to 192.168.30.223 (192.168.30.223).
Escape character is '^]'.
220 ESMTP ESMTP

If the last line is made of lots of stars with an occasional 2 or 0 then you are probably going through a Cisco PIX, and none of the extended commands will work.

You will then need to declare where you are sending the email from. You can use either standard or extended SMTP commands to do this. Extended commands allow you to see what is supported by the server and to authorize yourself:

Extended SMTP
Standard SMTP
HELO netcore.co.in 250 location.exampledomain.com
`EHLO netcore.co.in` This should give you: 250-location.exampledomain.com 250-PIPELINING 250-SIZE 9999360 250-VRFY 250-ETRN 250-AUTH LOGIN PLAIN 250-AUTH=LOGIN PLAIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN If "AUTH" is listed you can login to the server. This will usually allow some things which are normally restricted, for example relaying. You will need to use your username and password in Base64 (See here for an encoder). jayesh.shinde = amF5ZXNoLnNoaW5kZUBuZXRjb3JlLmNvLmlu & password123 = cGFzc3dvcmQxMjM= `AUTH LOGIN` `334` amF5ZXNoLnNoaW5kZUBuZXRjb3JlLmNvLmlu `bmFtZQ==` `334` cGFzc3dvcmQxMjM= `U2VjcmV0` `235 2.0.0 OK Authenticated`

Dont worry too much about your local domain name although you really should use your exact fully qualified domain name as seen by the outside world the mail server has no choice but to take your word for it as of RFC822-RFC1123.

Now give your email address:
MAIL FROM: <jayesh.shinde@netcore.co.in>
The angle brackets around the name are important; some servers won't accept the name unless they are there.
Should yield:
250 2.1.0 jayesh.shinde@netcore.co.in... Sender ok

Now give the recipients address:
RCPT TO: <netcoresolutions@gmail.com>
Again, use the angle brackets.
Should yield:
250 2.1.0 netcoresolutions@gmail.com... Recipient ok

To start composing the message issue the command:
DATA Subject:-test mail 1-
then press enter twice (these are needed to conform to RFC 882)
You may now proceed to type the body of your message on as many lines as it takes. To tell the mail server that you have completed the message enter a single "." on a line by itself.
The mail server should reply with:
250 2.0.0 ???????? Message accepted for delivery

You close the connection by issuing:
QUIT
The mailserver should reply with something like:
221 2.0.0 192.168.30.223 closing connection Connection closed by foreign host.

Summary :--

telnet 192.168.30.223 25
Trying 192.168.30.223...
Connected to 192.168.30.223 (192.168.30.223).
Escape character is '^]'.
220 ESMTP ESMTP
EHLO netcore.co.in
250-location.exampledomain.com
250-PIPELINING
250-SIZE 9999360
250-VRFY
250-ETRN
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
AUTH LOGIN
334 amF5ZXNoLnNoaW5kZUBuZXRjb3JlLmNvLmlu
bmFtZQ==
  334 cGFzc3dvcmQxMjM=
U2VjcmV0
235 2.0.0 OK Authenticated
MAIL FROM: <jayesh.shinde@netcore.co.in>
250 2.1.0 jayesh.shinde@netcore.co.in... Sender ok
  RCPT TO: <netcoresolutions@gmail.com>
250 2.1.0 netcoresolutions@gmail.com... Recipient ok
  DATA
  Subject:- Test email 1
Welcome to netCORE
   .
250 2.0.0 ???????? Message accepted for delivery
QUIT
221 2.0.0 192.168.30.223 closing connection
Connection closed by foreign host.

Try to understand the meaning of these numbers. Итак, проще говоря, числа отражают число блокирующих процессов в очереди на исполнение в определенный временной интервал, а именно 1 минута, 5 минут и 15 минут, соответственно. So, in other words, the numbers reflect the number of blocking processes in the queue for execution at a specific time interval, namely, 1 minute, 5 minutes and 15 minutes, respectively. В данном случае, блокирующий процесс — это процесс, который ожидает ресурсов для продолжения работы. In this case, the blocking process - a process that is waiting for resources to continue. Как правило, происходит ожидание таких ресурсов, как центральный процессор, дисковая подсистема ввода/вывода или сетевая подсистема ввода/вывода. As a rule, there is the expectation of resources such as CPU, disk I / O or network I / O subsystem.

Высокие значения показателей High values of load average load average говорят о том, что система не справляется с нагрузкой. indicate that the system can not cope with the load. Если речь идет о целевом сервере, работающем под высокой нагрузкой, то обычно полезно провести тонкую настройку операционной системы (сетевая подсистема, ограничение на количество одновременно открытых файлов и тому подобное). When it comes to the target server that is under heavy load, it is usually helpful to fine-tune the operating system (the network subsystem, the restriction on the number of open files, and so on). Высокая загрузка также может быть вызвана аппаратными проблемами, например, выходом из строя накопителя. High CPU can also be caused by hardware problems, such as failure of the drive.

Для диагностики обратимся к другим полезным данным, предоставляемым выводом For the diagnosis of appeal to other useful data provided output top . top. Строка Line Cpu(s) Cpu (s) содержит информацию о распределении процессорного времени. contains information on the distribution of CPU time. Первые два значения непосредственно отражают работу CPU по обработке процессов: The first two values directly reflect the work of CPU processing processes: