Start the schedule with e.g.
as usual. Then send a start message to IVS.
If in the operations room, start econtrol:
Applications -> AuScope Hobart -> econtrol
Click on the boxes next to “StatusMonitor“ and “Logging and Operator Input”. This should trigger the program to connect to pcfshb and you’ll see a status window and log window. If you don't see any information updating in the status window, type [ctrl]-[shift]-e to start the server process on the field system. You can send PCFS commands and comments from the bottom of the log window. Note you won’t see the log window update until the next scheduled command is sent.
During the experiment, the following checks can be made. Please log them using the new Checklist GUI rather than the paper checklist:
Please go through the checklist as often as you like but at least once every 2 hours. Below is an explanation of the current list of items to check during an observation:
onsourceand check the antenna is tracking, which it should be provided a new source command hasn't recently been issued.
maserdelay. These values should be within 0.5 microsecond of each other and stable (i.e. similar results if you issue the commands again). The monitoring software will calculate the difference for you and should ring an alarm if the difference is not acceptable. See this entry in the common Problems section for a remedy.
mk5=mode?. The result should be
/mk5/!mode? 0 : ext : 0x55555555 : 2 : 2 ;for R1 and R4 experiments,
/mk5/!mode? 0 : ext : 0x55555555 : 4 : 2 ;for OHIG, APSG and CRF observations.
mk5=dot?. Make sure it reports a small offset (~<10ms) as the final value, that
FHG_offdepending on whether it is currently recording or not.
disk_posin econtrol should report three values - the current number of btyes recorded, bytes at start of previous scan and bytes at start of current scan. If not currently recording, the first and third values should agree. It is normal for Yarragadee
disk_posto lag its expected value due to regular stows for USN uplinks.
wthcommand, which will look like this:\\/#wx#/16.1,1007.9,58.6\\Also make a note in the log of present weather conditions (if you're at the observatory).
/usr/bin/xterm -name monit2 -e /usr2/fs/bin/monit2
You might get a
ERROR sc -13 setcl: formatter to FS time difference 0.5 seconds or greater
to fix this do a:
sy=run setcl offset
Note this error is likely to reappear regularly.
Note also that the error message
?ERROR sc -18 setcl: program is already running, try "run setcl" instead.
has been seen recently when the command is issued from a terminal window. The problem has not been seen when the command is entered into the oprin window. If you do get this error when entering the command into the oprin window, please tell Jim.
The origin of this problem is presently unknown but the FS time can get seriously out of step. To fix this, while not recording start the
fmset program from an
oper@pcfshb terminal and issue the ”+” and “-” commands, then quit from fmset (ESC). Restart fmset and the FS time should now be correct. You may need to resync the mark5B pps after this procedure.
Be sure to check that FHG=off. Sometimes if there is a power glitch while the Mark5 is still recording, it can get 'stuck' in record mode. This will need to be stopped with disk_record=off, then run fmset again.
The clkoff command measures the difference in the 1 PPS (pulse per second) signal coming from the GPS with the 1PPS from the Mark5. The Mark5 1PPS has travelled through both the DBBC and Mark5 and is a good diagnostic of a timing problem in our hardware.
There are occasionally timing glitches (clock jumps) that cause the clkoff value to change. There are several possible causes:
The easiest way to check for clock stability is to compare the clkoff and maserdelay values. The difference between these two should remain stable at around 0.3 us. The Log Monitor software calculates the difference and logs it as the “Delay difference”. If this value exceeds abs(0.5) us, an alarm is sounded (by default).
The first thing to do is not panic. If the new delay remains constant and less than abs(20) us, the correlator can handle it. Re-setting the delay introduces another clock jump which makes the correlation more difficult. So the first thing to do is in the Log Monitor:
maserdelaycommands from e-RemoteCtrl
cd /vlbobs/ivs/logs gnuplot plot 'yg_ddif.txt' u linesp
This will plot the delay difference against day number. You can use the right mouse button in the plot window to zoom in. Every time you press “Export data” the output files are refreshed and you can replot the values in gnuplot either by typing 'replot' or by pressing the “Replot” button in the plot window. Other possible useful files to plot are
yg_maser2gps.txt, the difference between the maser and GPS 1PPS, and
yg_fmout.txt, the difference between GPS and Mark5 output 1PPS.
set terminal 'wxt' 2; plot 'yg_ddif.txt'
If the delay difference is stable you don't need to do anything.
If the delay difference is more than 20 us, or gets so large that the
maserdelay values lose precision, run
fmset to get the delays back to something manageable. Make sure you are not recording while running fmset! Issuing a
halt command from e-RemoteCtrl followed by
disk_record=off is usually a safe method.
The first thing to do is try the command
in e-RemoteCtrl. Check to see if this worked by typing
maserdelay. If this doesn't fix it, proceed with the steps below.
If the delay difference is drifting (usually linearly), the DBBC probably needs reconfiguring. This can be done from e-RemoteCtrl as follows (again, best to halt the schedule and make sure you're not recording):
Monitor how things are going in the DBBC VNC session. A reconfig takes about 2 minutes. When it's completed, synchronise the dbbc:
Then in a terminal window on pcfs[hb|ke|yg], run fmset to get the clocks lined up.
Now resume observations with a
This occurs when communication with the power sensor (a USB device) in the IF rack is lost. The power sensor is required for System Temperature (Tsys) measurements. The solution is to cycle power to the sensor by unplugging it's USB connection into the Field System PC and then plugging it back in again. If you are not at the site, and cannot contact anyone on-site to fix it, you can disable the Tsys measurements as follows:
pfmed pfmed: pf,station pfmed: ed,systemp12
An editor will start. Comment out the command by putting a double-quote at the start of the line. It should then look like this:
Now exit the editor, and
Lastly, please make a note in the log that Tsys measurements have been disabled.
It is possible to remotely reset the power sensor at both Yarragadee and Hobart. You should first follow the procedure outlines above, then kill any remaining
ReadPower.sh processes running on pcfsyg/pcfshb (use
ps -ef | grep ReadPower to identify the process IDs). Become root with
su and issue the command
It will run a series of procedures to toggle the power and then try to re-establish communications. It may take two tries to get it fully working - when it is ok, you should get a blithely cheery message to this effect, and be wished good luck. When you receive this message, wait for a break in the recording and test the power sensor by running
/home/oper/systemp12rcp.sh. All being well, there should be no timeouts although the measured power is likely to be nonsensical (there will be bogus values written into the data from the previous timeouts). If it fails with timeouts, persevere with the
/etc/init.d/AgilentU2000 restart procedure. Once you have it working, repeat the pfmed process and remove the comment from the systemp12 procedure.
Currently there’s a fault that sometimes develops in the Hobart 12m drives (or the control software) that causes the antenna to stop moving in Azimuth. If this happens the station alarm should sound and the PC Field system will report an error like this:
WARNING: ONSOURCE status is SLEWING.
You will also notice that the antenna control/monitoring GUI (called HMI) on the Windows PC will show constant azimuth position, and probably the Azimuth brakes on. You can see this display by starting up a VNC session to timehb.
To fix this problem, click on “Reboot System”, then either wait for the schedule to send the antenna to the next source, or look back through the schedule and re-issue the last “source=…” command. (Note the ‘onsource’ command doesn’t seem to remedy the problem at the moment. Check the snap file for the syntax of the command & the most recent usage). The “Reboot System” button is shown here:
The above screenshot shows the antenna in a healthy state. You will see various boxes in the POWER and DRIVES STATUS areas go red when there’s a problem.
Recording continues as econtrol is a front-end viewer for the field system, so don't panic :)
When you restart econtrol from the menu it may be unable to load the telescope information (the drop-down menu boxes), and the terminal from which econtrol runs produces “Can't open interface” type errors. If this happens, in the econtrol window (the green one, not the terminal) press
Control+shift+e, and then try to open one of the drop down boxes again - this time the icon in the bottom right corner should go from red through 'connecting' to green, the information will now load, and observing can continue as normal.