As of July 2021, the AUM experiments can be completely setup and monitored through the Field System, bypassing the need for external scripts.
Preparing the schedule files
- For mixed mode observations (AUA/AUM/AOV experiments), this can now be done from the standard slogit scripts. Please use either
schedules $exper hb ke
to download and drudg the schedues. Alternatively
slogit.auscope.sh $exper $station
will also work. When you are asked to check the prc files, please make sure that the values for bbcs033-048 are is the range of 2200-2950 MHz and bbc001-bbc012 are in the range of 300-450 MHz. These correspond to the X- and S-band frequencies, observing at dual linear polarisations.
The A/B inputs of the DBBC3 can either record S-band or the 3-7 GHz band (for VGOS). For mixed-mode (S/X) observations, we need to select S-band. This is done through one of the Internet power switches either pduhb for Hobart12 or ipske for Katherine. look for the port labelled as
RfOnForS and make sure that it is powered on - this will enable the S-band input.
Symptomatic of RfOnForS not switched on are bad delays reported on most IF's
Field system set up
- For hb12 use pcfshb, dbbc3hb and flexbuff-2hb machines using
- For ke12 use pcfske, dbbc3ke and flexbuffke
- The DBBC3 server program should be “DBBC3 Control DDC_V_v124.exe” for both sites
- Terminate any existing instances of the field system on either of the pcfs machines (e.g on pcfshb and/or pcfs-2hb)
- Start the field system on pcfske, and pcfshb (not pcfs-2hb) with
fs-aum(choose flexbuff-2hb as Hobart recorder) - this script will set up the appropriate control files and procedure libraries.
- Load the experiment procedure library and then run the following commands
proc=aum035ke setupsx setupaum
- Important - the
setupaumcommand is equivalent to fmset on the DBBC2/mark5B systems - it will force a re-synchronization and reinitialization of all DBBC3 boards and definitely introduce a delay jump. Do not run this unless you have to, such as in the event of a flexbuff crash or DBBC3 failure
- Check the status of the DBBC/mark5 with
midob- this will call many monitoring commands. Some things to look for in the output are listed below
iread- Check that the IF power levels match the targets NB - Hobart often has problems with A/B due to RFI so variations of ~20% are not unusual
bread- Confirm that all channels are in AGC mode and that the power levels are ~15000 for [ bbc017-32 @Ke ]/[ bbc033-048 @ Hb ] and between ~8000 and 30000 for bbc001-bbc012 (S-band RFI)
wthshould return temperature/pressure/humidity values at Hb. Ke data still not available unfortunately.
maserdelayshould have a stable difference (~-0.11 microsec at Katherine, ~0.34 microsec at Hb).
mk5=mode?should return the string
mk5=rtime?should report the free space in GB - check that there is enough free space for the experiment. Note - the recorded data volumes are three times those reported in the sum file due to the fixed 32 MHz bandwidths.
mk5=set_disks?should report a full set of disks (34 for flexbuffhb, 36 for flexbuff-2hb/flexbuffke). Note any missing disks in the handover notes and inform Jamie/Warren.
dbbc3=pps_delayshould report 43/39 ns for all the DBBC3 Cores.
samplersoutput contains all the details of the DBBC3 samplers and is intended for debugging purposes. The main things to check are that in each sampler block, the Power readings are relatively similar (to within a few percent) and that the Corr readings are all > 180E6 for Hobart12 and > 170E6 for Katherine. The samplers output is difficult to scan by eye, and a script that uses the multicast messages from the DBBC is preferable, see next point.
- Log in to terminals on each of pcfs-2hb and pcfs-2ke and run
~/AuscopeVLBI/dbbc3_multicast.py. The summarised output gives a green OK or red NO when things are out of range. Hobart12 often has issues with the power readings for A/B, due to RFI. (Not available on pcfshb or pcfske where we are running the fs)
- Ke requires IF's ABCD to be good (ignore EF, not used)
- Hb requires IF's ABEF to be good (ignore CDGH, not used)
- Ensure RfSwOnForS is turned ON on pduhb and ipske (see Rf Configuration above) otherwise wideband mode enabled, unsuitable for mixed mode s/x , giving bad delays.
Once the DBBC3/flexbuff is configured, please make a test recording to test for integer second offsets in the recorded data. As
Typical output is included below
#### DBBC3 time epoch #### SX (IFA) MJD = 59407/18:01:25.72 SY (IFB) MJD = 59407/18:01:25.72 XX (IFC) MJD = 59407/18:01:25.72 XY (IFD) MJD = 59407/18:01:25.72
Please confirm that all the frequency/polarisation products report the same time, and check that this matches the timestamp in the FS log for the
/disk_record/on,timetest_hb_timetest entry. If either of these is not the case, please re-try the recording script at least twice. If the script fails with errors about missing data, then it's probably necessary to restart the jive5ab server on the flexbuff (using
pkill jive5ab and
StartJ5) & rerun
setupaum from the FS.
In the event of the DBBC3 samplers going bad or losing synch, the best remedy is (unfortunately) a full power cycle. To do this, first shut down the Windows PC through the VNC interface. Then switch off the UPS powering the DBBC3 (instructions are here), wait for it to boot and then restart the server. It takes ~20-30 minutes for the system to fully configure - this usually fixes the problems for the rest of the session but isn't 100% effective.