User Tools

Site Tools


operations:aum-hb

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
operations:aum-hb [2018/07/30 06:49]
Jamie McCallum [Revised checklist]
operations:aum-hb [2019/07/06 12:02] (current)
Chin Chuan Lim
Line 30: Line 30:
 The next step is to convert the existing snp file into one which will control the flexbuff recordings appropriately. This is done via a script at present - please use the following commands <​code>​cp aum001hb.snp aum001hb.orig.snp The next step is to convert the existing snp file into one which will control the flexbuff recordings appropriately. This is done via a script at present - please use the following commands <​code>​cp aum001hb.snp aum001hb.orig.snp
 /​usr2/​sched/​snp2flexbuff.sh aum001hb.orig.snp aum001hb.snp hb</​code>​ /​usr2/​sched/​snp2flexbuff.sh aum001hb.orig.snp aum001hb.snp hb</​code>​
 +
 +Updated: The DBBC3KE (which will be recording the X-band data is now configured. "​correlation statistics"​ might be slightly lower from the DBBC than  expected (~16000000 rather than 18000000+). It hopefully won't be too  much of an issue (Lim: cited from Jamie). Use the following script instead:
 +<​code>​snp2flexbuff.aum.sh aum001hb.orig.snp aum001hb.snp hb
 +aum007.setup.dbbc3ke.py</​code>​
 +The flexbuff logs will be under 
 +<​code>/​usr2/​log/​$(lognnm).flexbuffhb.log</​code>​ (S-band recordings only, 46236) and <​code>/​usr2/​log/​$(lognnm).flexbuflyg.log</​code>​ (X-band recordings only, 46229 & 46230)
  
 The newly created aum001hb.snp file retains the timing of the source commands, scan names and times but calls a script to run the recorder and omits many of the typical procedures like preob, etc.  **NB - do not use the line number entries in the sum file!** ​ The newly created aum001hb.snp file retains the timing of the source commands, scan names and times but calls a script to run the recorder and omits many of the typical procedures like preob, etc.  **NB - do not use the line number entries in the sum file!** ​
  
-  * RF and IF paths configured/​DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on pcfshb ''/​usr2/​sched/​aum.32mhz.py''​. Running this should set up all the essential settings. ​+  * RF and IF paths configured/​DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on pcfshb ''​~/aum007.setup.py''​. Running this should set up all the essential settings ​for both the X-band recorded through the DBBC3 and S-band through the 26m backend and dbbcho
     * No RF configuration is needed for the X-band data recorded through the DBBC3, but it is necessary to set the inputs on the 26m's remote backend (e.g "the folly" or "​PalfreymansBackend"​. Go to the hobart26 VNC session (''​vncviewer newsmerd:​1''​) and go to desktop 3. Refresh the Folly display with '​r'​ and confirm that **S/X SRCP** and **S/X XRCP** are highlighted for channels 1 and 2 respectively. ​     * No RF configuration is needed for the X-band data recorded through the DBBC3, but it is necessary to set the inputs on the 26m's remote backend (e.g "the folly" or "​PalfreymansBackend"​. Go to the hobart26 VNC session (''​vncviewer newsmerd:​1''​) and go to desktop 3. Refresh the Folly display with '​r'​ and confirm that **S/X SRCP** and **S/X XRCP** are highlighted for channels 1 and 2 respectively. ​
 +    * DBBC server: If the aum007.setup.py script cannot communicate with the DBBC servers, you can restart the software running through the VNC sessions on dbbc3hb (''​DBBC#​ Control DDC_V_v122.exe''​) and dbbcho (''​DBBC2 Control DDC v105_2.exe''​ and ''​SerialServer.sem.py''​),​ respectively. ​
 +    * IF selections ​ can be checked with the ''​aum.check.sh''​ script on pcfshb. For the DBBC3, all inputs and filters should be ''​2'' ​ and all power levels should have a target level of ''​42000''​. For the dbbcho, the input should be ''​1'',​ the filter ''​2''​ and the target power level should be ''​48000''​
 +  * Maser ok: Check the Hobart26 maser as per usual
 +  * Mark5: Not used. Instead log in to the Flexbuff recorder (with ''​ssh observer@flexbuffhb''​) and confirm that the time is correct (with ''​ntpq -np''​) and that jive5ab is running (with ''​ps -ef | grep jive''​)
 +  * The clock offsets can be (partially) read through the FS. ''​maserdelay''​ reads the maser-GPS difference as usual while ''​dbbc3delay''​ logs the DBBC3-GPS delay //as gps-fmout//​. The existing ''​clkoff''​ command is measuring the dbbchb-gps difference and logging this as //​fmout-gps//​ - this is irrelevant to the Hb12 recordings but is needed for the simultaneous Ho26 experiments!. In summary - compare the outputs of the ''​maserdelay''​ and ''​dbbc3delay''​ commands, logged as ''/​maser2gps/''​ and ''/​gps-fmout/''​ respectively. You should check the dbbc-gps difference for DBBCHO using the ptzhb camera. The relevant counter is to the left of the mark4 rack at the top of rack 9. It's labeled ''​DBBCHO-TACGPS''​and is typically ~1 microsecond offset from the maserdelay. You do not need to log this value but not the difference and confirm that it is steady.
 +  * Test Recordings: To make a test recording, as ''​oper@pcfshb''​ run  ''​sxflexbuff.sh test 19''​ to carry out a 20s test recording. You can check the status of the flexbuff by tailing the log file (e.g ''​ tail -100f /​usr2/​log/​$(lognm).flexbuff.log''​). Confirm that the total number of packets is incrementing after each check and that there are no lost or out-of-order packets - these are symptoms of the flexbuff being overloaded. ​
 +  * Check for any integer second offsets in the recorded data by running ''​sxtest.sh''​ on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries (presented as ''​MJD = 58330/​01:​09:​59.91''​) should agree to within 0.01 seconds. ​
 +  * No autocorrelation script has been implemented yet. 
 +  * There is no systemp measurewment implemented yet. 
 +  * There is no pointing/​SEFD check implemented as yet
 +  * No fringe check implemented as yet.
 +
 +== Monitoring ==
 +  * Antenna checks as per usual
 +  * LO check unavailable
 +  * No autocorrelations
 +  * Check the delays with ''​maserdelay''​ and ''​dbbc3delay'',​ confirm the dbbcho time delay is stable
 +  * Check the Hb maser as usual. ​
 +  * Check the weather data is being logged & add in sky conditions as usual. ​
 +  * No Tsys checks.
 +  * On pcfshb, run ''​./​aum.check.sh''​ to check the dbbc and recorder status. There is a lot of debugging info stored here, so use a large terminal window to run the script and be aware that you will need to press space to go through the list. Included below is the full output of a check, with the relevant parameters to check commented on below.
 +<​code>​
 +###############################​
 +Checking dbbc3hb
 +###############################​
 +dbbcifc/ 2,​33,​agc,​2,​41815,​42000
 +dbbcifd/ 2,​22,​agc,​2,​42416,​42000
 +</​code>​
 +This first block is the equivalent of ''​iread''​ - check the inputs and filters are set to ''​2''​ and that the power level matches the target
 +<​code>​
 +dbbc17/ 2204.990000,​a,​32,​1,​agc,​55,​63,​14974,​15206,​0,​0
 +dbbc18/ 2244.990000,​a,​32,​1,​agc,​58,​69,​17035,​15254,​0,​0
 +dbbc19/ 2344.990000,​a,​32,​1,​agc,​79,​100,​17160,​14828,​0,​0
 +dbbc20/ 2504.990000,​a,​32,​1,​agc,​124,​108,​15562,​14913,​0,​0
 +dbbc21/ 2724.990000,​a,​32,​1,​agc,​107,​121,​16908,​15844,​0,​0
 +dbbc22/ 2844.990000,​a,​32,​1,​agc,​181,​181,​16086,​16144,​0,​0
 +dbbc23/ 2904.990000,​a,​32,​1,​agc,​188,​196,​16211,​16191,​0,​0
 +dbbc24/ 2924.990000,​a,​32,​1,​agc,​200,​202,​16298,​16046,​0,​0
 +dbbc25/ 2204.990000,​a,​32,​1,​agc,​57,​58,​13976,​17561,​0,​0
 +dbbc26/ 2244.990000,​a,​32,​1,​agc,​68,​75,​15413,​16071,​0,​0
 +dbbc27/ 2344.990000,​a,​32,​1,​agc,​90,​92,​15310,​14988,​0,​0
 +dbbc28/ 2504.990000,​a,​32,​1,​agc,​119,​100,​16736,​16911,​0,​0
 +dbbc29/ 2724.990000,​a,​32,​1,​agc,​104,​109,​16772,​14937,​0,​0
 +dbbc30/ 2844.990000,​a,​32,​1,​agc,​149,​172,​16204,​15428,​0,​0
 +dbbc31/ 2904.990000,​a,​32,​1,​agc,​183,​175,​15675,​16114,​0,​0
 +dbbc32/ 2924.990000,​a,​32,​1,​agc,​171,​209,​16013,​16102,​0,​0
 +</​code>​
 +This is the quivalent of ''​bread''​ confirm all the values are within ~1000 of the 16000 target and that the frequencies match those listed above.
 +<​code>​
 +pps_delay/ [1]: 43 ns, [2] 43 ns, [3] 39 ns, [4] 39 ns, [5] 39 ns, [6] 39 ns, [7] 0 ns, [8] 0 ns;
 +</​code>​
 +This is the translated equivalent of ''​dbbc=pps_delay'',​ separated by processing board. Check that all the listed values are either 43 or 39 ns, and are stable. ​
 +<​code>​
 +core3hstats/ ​
 +Core3H[3] Power:
 +Sampler 0: 60781106
 +Sampler 1: 61694675
 +Sampler 2: 62199248
 +Sampler 3: 62160250
 +</​code>​
 +The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use. 
 +<​code>​
 +Core3H[3] Bstat.
 +Sampler 0:
 +11:       44 0.28%
 +10:     7699 49.27%
 +01:     7846 50.21%
 +00:       34 0.22%
 +
 +Sampler 1:
 +11:       71 0.45%
 +10:     7915 50.66%
 +01:     7604 48.67%
 +00:       33 0.21%
 +
 +Sampler 2:
 +11:       48 0.31%
 +10:     7768 49.72%
 +01:     7773 49.75%
 +00:       33 0.21%
 +
 +Sampler 3:
 +11:       45 0.29%
 +10:     7700 49.28%
 +01:     7844 50.20%
 +00:       35 0.22%
 +</​code>​
 +The Bstats can be ignored at this time - they'​re displayed because it's complicated to hide them. 
 +<​code>​
 +Core3H[3] Corr.
 +Sampler 0-1: 180656919
 +Sampler 1-2: 182717105
 +Sampler 2-3: 182062361
 +</​code>​
 +All inter-sampler correlation values should be ~180000000. ​
 +<​code>​
 +core3hstats/ ​
 +Core3H[4] Power:
 +Sampler 0: 88700574
 +Sampler 1: 90430690
 +Sampler 2: 92322148
 +Sampler 3: 93681776
 +</​code>​
 +The Core3H power readings from each sampler should all agree to within ~3-5%. This is the second sampler of two, and the above readings are ok.
 +<​code>​
 +Core3H[4] Bstat.
 +Sampler 0:
 +11:      170 1.09%
 +10:     7635 48.86%
 +01:     7680 49.15%
 +00:      138 0.88%
 +
 +Sampler 1:
 +11:      226 1.45%
 +10:     7680 49.15%
 +01:     7582 48.52%
 +00:      135 0.86%
 +
 +Sampler 2:
 +11:      184 1.18%
 +10:     7638 48.88%
 +01:     7658 49.01%
 +00:      142 0.91%
 +
 +Sampler 3:
 +11:      181 1.16%
 +10:     7569 48.44%
 +01:     7733 49.49%
 +00:      140 0.90%
 +</​code>​
 +The Bstats can be ignored at this time - they'​re displayed because it's complicated to hide them. 
 +<​code>​
 +
 +Core3H[4] Corr.
 +Sampler 0-1: 186212284
 +Sampler 1-2: 190747558
 +Sampler 2-3: 187420379
 +</​code>​
 +All inter-sampler correlation values should be ~180000000. ​
 +<​code>​
 +###############################​
 +Checking dbbcho
 +###############################​
 +dbbcifa/​1,​4,​agc,​2,​47097,​48000,​1
 +dbbcifb/​1,​14,​agc,​2,​47991,​48000,​1
 +</​code>​
 +This is the equivalent of ''​iread''​. All inputs for dbbcho should be ''​1'',​ all filters ''​2''​ and all target power levels ''​48000''​
 +<​code>​
 +dbbc01/​317.990000,​a,​32,​1,​agc,​97,​89,​14803,​14952,​0,​0
 +dbbc03/​337.990000,​a,​32,​1,​agc,​108,​89,​14945,​15083,​0,​0
 +dbbc05/​357.990000,​a,​32,​1,​agc,​103,​111,​14891,​14875,​0,​0
 +dbbc07/​387.990000,​a,​32,​1,​agc,​119,​100,​12784,​14783,​0,​0
 +dbbc09/​317.990000,​b,​32,​1,​agc,​130,​119,​16077,​16249,​0,​0
 +dbbc11/​337.990000,​b,​32,​1,​agc,​139,​123,​16091,​16175,​0,​0
 +dbbc13/​357.990000,​b,​32,​1,​agc,​149,​138,​16248,​16228,​0,​0
 +dbbc15/​387.990000,​b,​32,​1,​agc,​125,​145,​14144,​16119,​0,​0
 +</​code>​
 +This is the equivalent of ''​bread''​. Note that only odd-numbered bbcs are used. As this is S-band, power levels may vary significantly so a range of 10000-25000 is acceptable (but notable - please put an entr in the handover notes)
 +<​code>​
 +pps_delay/​61971
 +</​code>​
 +The pps_delay should be stable at 61971, to within 1 count of so.
 +<​code>​
 +sysstat
 +
 +^MSystem status:
 +^M Selected input      : vsi1
 +^M Input sample rate   : 64000000 Hz
 +^M VSI input swapped ​  : no
 +^M VSI input bitmask ​  : 0x0000FFFF
 +^M VSI input width     : 16 bit
 +^M PPS count           : 78607
 +
 +^M TVG mode            : vsi-h
 +
 +^M MK5B timesync ​      : yes
 +^M VDIF timesync ​      : yes
 +^M GPS receiver ​       : installed
 +
 +^M Output ​             : started
 +^M Output 0 format ​    : vdif
 +^M Output 0 dest.      : 192.168.1.11:​46236
 +^M Output 1 format ​    : vdif
 +^M Output 1 dest.      : none
 +^M Ethernet ARPs       : off
 +
 +^M Selected VSI output : vsi1-2
 +
 +^MFiLa10G % 
 +^MFiLa10G % 
 +</​code>​
 +This last block is the Fila10G status. Please confirm that these settings match the output when performing the checks. ​
  
 +  * You can check on the status of the recordings by tailing the flexbuff log (''/​usr2/​log/​aum001hb.flexbuff.log''​ on pcfshb - the name will change when a new FS log file is started). This is written to by the recording script and running ''​tail -100f /​usr2/​log/​aum001hb.flexbuff.log''​ will show the same output as in the recording tests, e.g 
 +<​code>​
 +50
 +!runtime = 0 : stream46230 ;!evlbi? 0 : total : 782143 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
 +!runtime = 0 : stream46236 ;!evlbi? 0 : total : 782107 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
 +!runtime = 0 : stream46229 ;!evlbi? 0 : total : 782064 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
 +</​code>​
  
 +Please note that the output is "​laggy"​ and is usually delayed by ~1 minute. You can see the current status of the recorder using [[http://​ptzhb.phys.utas.edu.au/​web/​admin.html|ptzhb]] - when recording you will see the blue lights of disk activity scattered over the flexbuff (below the DBBC in rack 13). 
/home/www/auscope/opswiki/data/attic/operations/aum-hb.1532933362.txt.gz · Last modified: 2018/07/30 06:49 by Jamie McCallum