Torrent Suite SoftwareAdministration Guide

Torrent Suite Software space on Ion Community

Administration Guide TOC

Troubleshooting

These troubleshooting suggestions apply to system level issues such as networking, disk space, and system load.

For investigations of an individual failed analysis run, see instead Handle a Failed Analysis Run .

Check Crawler and Job Server status

Access the Crawler and Jobs Server page:

• Click the Admin gear menu (near of top right of the Torrent Browser) and select Services :

The following table lists the background processes that run on Torrent Suite Software:

If these processes are not running, run information is not updated and analysis reports are not generated. If this occurs, there is no risk of data loss but the Crawler and Jobs Server processes should always be running. The Archive process only runs if archiving has been configured.

Process status is displayed in the Admin Services tab, as shown in the following figure:







If a process is not running, a Down or Offline reason is displayed in the Admin Services tab. An example is "The crawler is offline".

Queue status

Click the Queue Status link in the Active Jobs section to open a table of SGE queue activity:

Processes

Use the status command to verify that the processes are running:

             ps ax | grep py
            

This displays the running Torrent Suite Software processes (output is chopped for width restrictions):

             ionadmin@ion-torrent-server:/etc/init.d$ ps ax | grep py
 9279 ?        Sl     0:06 /usr/bin/python /opt/ion/iondb/bin/crawler.py
 9303 ?        Sl     0:02 /usr/bin/python /opt/ion/iondb/anaserve/serve.py
 9338 ?        Sl     0:02 /usr/bin/python /opt/ion/iondb/bin/ionPlugin.py
29852 ?        Sl     0:03 /usr/bin/python /opt/ion/iondb/manage.py celeryd --concurrency=4
29861 ?        S      0:00 /usr/bin/python /opt/ion/iondb/manage.py celeryd --concurrency=4
29862 ?        S      0:00 /usr/bin/python /opt/ion/iondb/manage.py celeryd --concurrency=4
29863 ?        S      0:00 /usr/bin/python /opt/ion/iondb/manage.py celeryd --concurrency=4
29864 ?        S      0:00 /usr/bin/python /opt/ion/iondb/manage.py celeryd --concurrency=4
            

Processes do not start automatically. If a process ( crawler.py , manage.py , ionPlugin.py , or serve.py ) is not running, you must manually restart it.

Restart services

Currently, there is no method to restart a process using the Torrent Browser UI. The easiest approach is to shutdown and restart the server. Before restarting the server, make sure that no Ion PGM, Ion Proton , or Ion S5 sequencers are uploading data to the server, otherwise the file transfer is interrupted.

You can also restart the processes using the scripts located in the /etc/init.d directory.For example, use the following command to restart the Crawler:

             user@svr:/etc/init.d$ sudo /etc/init.d/ionCrawler restart
Stopping crawler
Starting crawler pid = 26025
            

Use the ps ax | grep py command or the Torrent Browser UI to verify that the processes are running.

After restarting a process, it continues from the point where it was interrupted, and no more user interaction is needed.

If the processes do not continue to run after being restarted, contact your Ion Torrent representative for assistance.

Verify network connectivity and name resolution

There can be many reasons for network connectivity or name resolution to fail. Use the following procedure to try to resolve connectivity and name resolution problems:

1. Go to the admin interface Management Actions area and click View Network Settings . (See View Network Settings for details.) The Torrent Browser performs several network checks:
   
   
   
   
2. You can also verify that the Torrent Server is configured correctly by reviewing the Torrent Server deployment instructions.
   
   
3. Find the IP address of the Torrent Server as described in Verify Torrent Server IP address .

If you cannot reach the Torrent Server an IP address, you are likely to need help from the site I.T. administrator who understands how the local network is configured.

Verify Torrent Server IP address

The Torrent Server is configured out-of- the-box to automatically get an IP address from the DHCP server on the network. Unless the local I.T. administrator has specifically assigned an IP address in advance, you will not know what the current IP address is.

The Torrent Server has several Ethernet ports on the back. Make sure your site network is connected to the port labeled LAN , called eth0 in Linux terminology.The Ethernet port are identified as eth0 , eth1 , ..., for as many ports as are available. On Torrent Server, eth0 is the only port connected to your network and is configured by DHCP.

To determine the IP address assigned to eth0 , login and type: ifconfig eth0 . This displays the following output:

ionadmin@ion-torrent-server:~$ ifconfig eth0

eth0 Link encap:Ethernet HWaddr 00:1b:21:5b:bb:44

inet addr:192.168.1.123 Bcast:192.169.4.255 Mask:255.255.255.0

inet6 addr: fe80::21b:21ff:fe5b:bb44/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:209970726 errors:0 dropped:0 overruns:0 frame:0

TX packets:419252947 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:14131928595 (14.1 GB) TX bytes:607398487997 (607.3 GB)

Memory:fbea0000-fbec0000

Your IP address is the inet addr:

inet addr:192.168.1.1 Bcast:192.169.4.255 Mask:255.255.255.0

Another useful check is the line beginning with UP , which indicated the interface is active and working:

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

If the eth0 port is not available, it is possible the Ethernet cable is connected to a network, so you will not see the word UP :

BROADCAST MULTICAST MTU:1500 Metric:1

If an IP address is assigned, the interface is likely to work.If no IP address is assigned and the interface is not UP, you may need to get help from your site I.T. a dministrator.

If you are still concerned about network connectivity, you can test that different desktops are able to successfully ping the server IP address. If you are not able to ping the server from the desktops that need to access the Torrent Browser running on the server, contact your site I.T. a dministrator.

Troubleshoot and configure the time service

The Torrent Server uses the Linux Network Time Protocol (NTP)program to synchronize its time with another time server.By default, the Torrent Server is configured to synchronize its time service to a trusted time service on the Internet. This requires that the network configuration permits the NTP network protocol to connect to that time service on the Internet.

The Torrent Servercan also act as a time server for Ion PGM, Ion Proton , and Ion S5 s equencers. However, if the server is not able to synchronize with the trusted time service, it does not act as a time server for the sequencers (Torrent Server does not forward potentially incorrect information to other machines).

If the network configuration is blocking the NTP protocol from reaching the Internet, the Torrent Server and the Ion PGM, Ion Proton , and Ion S5 s equencers are not be able to synchronize time.

Your site network administrator is probably aware of this connectivity restriction, and it is likely that IT has a time server in the network.

Change the time server used by the Torrent Server

After the site I.T. a dministrator has identified the local time server, use the following procedure to update the Torrent Server to use the new time server:

1. Edit the NTP configuration file
   
   
   1. Edit /etc/ntp.conf (using the nano or vi text editor, for example).
   2. Change the line from server ntp.ubuntu.com to server <hostname> .
      
      
2. Restart the time service on the Torrent Server.
   
   
   1. Restart NTP, using the command: sudo /etc/init.d/ntp restart .

When a time server is configured, you can also set the Ion PGM, Ion Proton , and Ion S5 equipment to get time from either the Torrent Server or the I.T.-provided time server.

Check the NTP Configuration on the Torrent Server

To test the NTP configuration on the Torrent Server, enter the following command on the Torrent Server command line:

ntpq -p

On a working server, the response looks similar to the following example:

remoterefid st t when poll reach delay offset jitter

=============================================================

xxx.yyy 69.65.40.29 3 u 196 1024 377 0.175 14.235 0.9

On a server that cannot connect,the response looks similar to the following example:

remoterefid st t when poll reach delay offset jitter

=============================================================

xxx.yyy .INIT. 16 u - 64 }}{{0 0.000 0.000 0.000

Notice, for the server that cannot synchronize time, the following columns indicate an error condition:

Also, the time stamps are all 0.0000. These usually show some deviation when operating typically.

Verify file transfer

Verify that all files successfully transferred from the Ion PGM, Ion Proton , and Ion S5 s equencersto the Torrent Server. Manually transfer files, if needed, by going to the Data Management screen of the sequencer and select the Re-transfer option for any of the runs in question. You can safely re-transfer data.

Do not delete the data from the Ion PGM, Ion Proton , or Ion S5 s equencer until you are confident that the data is present on the Torrent Server, the analysis is successful, and the Analysis Report has been generated successfully.

Use QMON to manage the SGE cluster

QMON is a graphical tool to manage the SGE Cluster. With QMON you can do tasks such as the following:

• See the status of the servers in the cluster
• Check server CPU load and memory utilization
• See the status of jobs, are they running, completed, or still waiting in the queue
• Monitor these job queues:

• • tl.q Top level control jobs
  • all.q Analysis jobs
  • plugin.q Plugin jobs
  • thumbnail.q Ion Protonthumbnail analysis jobs
• Delete jobs that are currently processing or waiting in queue
• Disable servers from accepting new jobs, and re-enable them later
• And many other advanced configuration options

See Manage your SGE Cluster with QMON .

System diagnostics

System diagnostics information can help diagnose network, disk space, and system status issues. See View System Support Diagnostics .

Instrument diagnostics

Use instrument diagnostics to investigate chip and sequencing instrument issues, such as pH levels. See View Instrument Diagnostics .

Generate a Customer Support Archive

See Customer Support Archive (CSA) for instructions on generating a CSA archive and descriptions of the files that are included in an archive.

Further investigation and problem resolution

After the root cause of a major problem is identified, the following more intrusive action may be needed:

• Replace failed hard disk drive
• Downgrade software packages
• Reinstall software
• Modify config files
• Add, modify, or delete database information

Please contact your Ion Torrent representative for assistance before you attempt any of these steps.