User Tools

Site Tools


operations:documentation.ivs.bonnetransfer

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
operations:documentation.ivs.bonnetransfer [2012/11/13 23:35]
Jamie McCallum
operations:documentation.ivs.bonnetransfer [2013/07/12 06:30] (current)
Jamie McCallum
Line 1: Line 1:
 ====== Transferring Data to Bonn ====== ====== Transferring Data to Bonn ======
 +
 +**Update: Transfers directly from mounted diskpacks have significantly higher rates of failure. Staging data to RAID is strongly recommended!**
 +
 +**Update: The mark5B upgrade to SDK9.2 has adversely affected fuseMk5 performance. It is not uncommon for the program to crash, sometimes with kernel panic during copy operations. A workaround script using DIMino has been kludged together and is in ~oper/​recovery/​ but is still in development.**
 +
   * Use a VNC session to run the transfers - the hobart.phys.utas.edu.au:​1 session is normally a good option. ​   * Use a VNC session to run the transfers - the hobart.phys.utas.edu.au:​1 session is normally a good option. ​
   * Open four terminal windows on the Mark5 unit the data will be sent from   * Open four terminal windows on the Mark5 unit the data will be sent from
-  * In the first terminal, make sure that neither fuseMk5 or dimino is running <​code>​ps -ef | grep dim</​code>​ and <​code>​ps -ef | grep fuse</​code>​ and that another transfer isn't running <​code>​ps -ef | grep tsuna</​code>​. ​+  * In the first terminal, make sure that neither fuseMk5 or dimino is running <​code>​ps -ef | grep DIM</​code>​ and <​code>​ps -ef | grep fuse</​code>​ and that another transfer isn't running <​code>​ps -ef | grep tsuna</​code>​. ​
   * If fuseMk5A is running and the wrong module is mounted, dismount with <​code>​fusermount -u <​directory_name></​code>​   * If fuseMk5A is running and the wrong module is mounted, dismount with <​code>​fusermount -u <​directory_name></​code>​
-  * Then use fuseMk5 ​(or fuseMk5A for unupgraded mark5 units) ​to mount the module: <​code>​fuseMk5 --bank=0 --verbose --cachesize=128000000 /​mnt/​diskpack</​code>​ This will mount Bank A (set bank=1 for Bank B) at /​mnt/​diskpack. Wait for a message to say <​code>​.... Registering fuseMk5A to FUSE with: ....</​code>​. It can take a while. Background this process with Ctrl-z and <​code>​ bg </​code>​. ​+  * Then use fuseMk5 to mount the module: <​code>​fuseMk5 --bank=0 --verbose --cachesize=128000000 /​mnt/​diskpack</​code>​ This will mount Bank A (set bank=1 for Bank B) at /​mnt/​diskpack. Wait for a message to say <​code>​.... Registering fuseMk5A to FUSE with: ....</​code>​. It can take a while. Background this process with Ctrl-z and <​code>​ bg </​code>​. ​
   * In a second terminal, change to the mounted directory and make sure the data are there <​code>​cd /​mnt/​diskpack   * In a second terminal, change to the mounted directory and make sure the data are there <​code>​cd /​mnt/​diskpack
 ls ls
Line 13: Line 18:
 </​code>​ The [[http://​www3.mpifr-bonn.mpg.de/​cgi-bin/​showtransfers.cgi </​code>​ The [[http://​www3.mpifr-bonn.mpg.de/​cgi-bin/​showtransfers.cgi
   |Bonn web page]] shows available servers and space.   |Bonn web page]] shows available servers and space.
-(194.94.199.166 = sneezy2)+(194.94.199.163 = io10, 194.94.199.164 = io3, 194.94.199.166 = sneezy2)
  
   * Start tsunami and set up for a transfer   * Start tsunami and set up for a transfer
Line 24: Line 29:
   * in the fourth terminal window, create an empty file with a name describing the start time of the transfer, rate etc: <​code>​touch 20110525033000_r1485_Hb_Bonn_300m_52100_sneezy2_start</​code>​ This puts an entry on the [[http://​www3.mpifr-bonn.mpg.de/​cgi-bin/​showtransfers.cgi   * in the fourth terminal window, create an empty file with a name describing the start time of the transfer, rate etc: <​code>​touch 20110525033000_r1485_Hb_Bonn_300m_52100_sneezy2_start</​code>​ This puts an entry on the [[http://​www3.mpifr-bonn.mpg.de/​cgi-bin/​showtransfers.cgi
   |Bonn web page]] to say there'​s a transfer under way   |Bonn web page]] to say there'​s a transfer under way
-  * and send it to the web server <​code>​ncftpput ftp.mpifr-bonn.mpg.de /​incoming/​geodesy/​transfers ​0110525033000_r1485hb_Hb177_Bonn_300m_52100_sneezy2_start</​code>​+  * and send it to the web server <​code>​ncftpput ftp.mpifr-bonn.mpg.de /​incoming/​geodesy/​transfers ​20110525033000_r1485hb_Hb177_Bonn_300m_52100_sneezy2_start</​code>​
   * When the transfer finishes, send a stop message with <​code>​touch 20110525033000_Hb_stop   * When the transfer finishes, send a stop message with <​code>​touch 20110525033000_Hb_stop
 ncftpput ftp.mpifr-bonn.mpg.de /​incoming/​geodesy/​transfers 20110525033000_Hb177_stop</​code>​ ncftpput ftp.mpifr-bonn.mpg.de /​incoming/​geodesy/​transfers 20110525033000_Hb177_stop</​code>​
  
-It is also possible (and preferable) to export the data onto a RAID first. Currently, there are three RAIDS which can beused ​for temporary storage at Hobart - Vortex, Cornucopia ​and 131.217.63.140 ​(Currently unnamed and contains two separate RAIDS of 6 and 12 TB). These can be NFS mounted on the mk5 machine with <​code>​mount vortex.phys.utas.edu.au:/​exports/​vortex_internal/​ /​mnt/​vortex</​code>​ (as root). Once mounted the data can be copied with <​code>​cp /​mnt/​diskpack/​r1485* /​mnt/​vortex/​r1485hb/</​code>​.+It is also possible (and preferable) to export the data onto a RAID first. Currently, there are five RAIDS which can be used for temporary storage at Hobart - Vortex, Cornucopia, Jupiter ​(which contains two separate RAIDS of 6 and 12 TB) and Magilla. These can be NFS mounted on the mk5 machine with <​code>​mount vortex.phys.utas.edu.au:/​exports/​vortex_internal/​ /​mnt/​vortex</​code>​ (as root). Once mounted the data can be copied with <​code>​cp /​mnt/​diskpack/​r1485* /​mnt/​vortex/​r1485hb/</​code>​.
  
-Once fully copied, please check the data has not been corrupted in the transfer. ssh into hex0 and NFS mount the RAID if necessary. Then run <​code>​directory2filelist ​/mnt/vortex/r1485hb Mark5B-256-16-1 ​> /tmp/ok.txt</​code>​ . The output to the screen will be the list of those scans which show problems. ​It's not uncommon to see up to ~10% of the scans fail the test - these files should ​be deleted and retransferred to the RAID+Once fully copied, please check the data has not been corrupted in the transfer. ssh into the NFS host as observer ​and run <​code>​./​directory2list ​/exports/vortex_internal/r1485hb Mark5B-256-16-1</​code>​ . The output to the screen will be the list of all the scans with the start mjd/time and a summary indicating if problems ​were foundIf bad scans are found, a summary at the end will warn you, and list of bad scans will be made available.
  
-The same <​code>​directory2filelist</​code> ​procedure should be run after a transfer to Bonn completes ​to check for errors in transmission. ​+After a transfer to Bonn completes, you should run <​code>​directory2filelist ​/​data3/​r1/​hobart12/​r1485/​ Mark5B-256-16-1</​code>​ to check for errors in transmission. ​
/home/www/auscope/opswiki/data/attic/operations/documentation.ivs.bonnetransfer.1352849748.txt.gz · Last modified: 2012/11/13 23:35 by Jamie McCallum