The PIC in the IF rack at Yg has a problem that prevents diode switching and hence Tsys measurements. The preob procedure in station.prc has been edited to comment out systemp12. Will need re-enabling when tsys is back (Jim)
If the network connection to Yarragadee dies, the VPN could need reseting:
The weather sensor at Yarragadee is currently away for repair (humidity sensor is broken). Weather is still being logged but from a separate sensor that is not connected to the Field System. Until our sensor is repaired and returned, there is an extra procedure to go through to recover the weather information:
(added 14th Oct 2015, updated 9 Aug 2016)
find” command to the end, and paste into the command prompt.
rm /home/oper/auto_cor_test/*.epi rm /home/oper/auto_cor_test/*.png
Please note here any issues you have with the new eRC. Whenever there is a problem, it would be helpful for Alexander to know whether the time is still running in the eRC Status Monitor and what is your actual user name in the eRC (oper, oper(1),…).
During set up of R1569, mk5hb's response to ntpq -np indicated very high offsets and high jitter (25 ms). Problem persists after a reboot. Not critical at the moment but needs watching in case it gets worse.
Antenna “stuck” as elevation brakes will not release. fs, halt schedule. Need local person to switch off big yellow rotatry power switch in pedestal, open power cabinet and switch to maintenance mode. Switch power back on, but this time to hand box control. Should be able to drive by hand. Switch power back over to remote mode. EM reset orange button press twice. HMI interface should now drive telescope might need to “Reboot Central” in HMI. fs, antenna=open, antenna=operate, resume schedule. Good luck with all of that….
………… I have had calls to assist. These seem to come more often during the change from one experiment to another. I have a suspiscion that if the change over period from one experiment to the next is brief, then the problem happens. Hope that helps. It particularly happens with Austral weekend experiments, or so it seems. Regards, Brett.
…………. It played up today again from around 8am Hobart time. Jack was able to move it in maintenance mode though. At midday Hobart I was able to drive the antenna for a while in remote after doing “reset timer”. But this only lasted about 5 minutes. Better get Mark Godwin to check it. Regards, Brett. 29/8/14
If the ops2 display locks up, or has odd sized displays, etc, first try powering off the screens - use the power pointi rather than the button on the controller. A switched power board (labelled) has been added to make this easier - it's under the screens on the left hand size and (badly) labelled as “screen power”. Once the screens are power cycled, you may need to run use “ctrl-alt-f1” and then “ctrl-alt-f7” to swap to the terminal and then back to the X-windows desktop.
on pcfshb (Jim)
If “mk5=dot?” gives a time difference (string of numbers at the end: -0.003224s ;) of ~0 or ~integer steps, then it is not necessary to run fmset, as the correlator can resolve this time step. If it is some other fraction between, then fmset needs to be run with either '+' or '-' to sync.
The S/X cryogenics have been warming up regularly. Please keep an eye on them and let Brett know ASAP if you notice any problems. Things to look out for are documented on the 26m Checklist page and the Log Monitor picks up on some of them.
By default, Log Monitor will ring an alarm if any of these conditions occur:
If either of these alarms go off, please phone Brett.
When the cryogenics are recovering (or while the problem is known but not yet fixed), you can change the alarm to a beep or silence it altogether. Do this from the Log Monitor menu: “Configure → Cryo monitoring” but please remember to turn the alarm back on once the receiver is working within range again.
Lastly, there’s one other thing to monitor which Log Monitor can’t ring an alarm on at the moment. The Helium supply pressure should stay above 275 psi. If it get’s below that, Brett needs to know ASAP. This quantity should be monitored via MONICA as part of the regular 2-hourly checks.
Occasionally the 26m drives will trip out with an encoder error. This should cause the log monitor to beep but doesn't ring an alarm (yet!). Example log message:
2014.127.05:29:38.24#antcn#Error: Antenna reported error from last command (0) 2014.127.05:29:38.24#antcn#Antenna control system Local control asserted by DESK 2014.127.05:29:38.24#antcn#Check if antenna is on-source 2014.127.05:29:38.64#antcn#Antenna drives not on, switching them on 2014.127.05:29:59.84#antcn#Successfully sent new source position to antenna
The antenna should recover the next time it is sent to a source or when there's an insource command issued. However, if you notice it happening, you can get things going again sooner with the command
Something has gone wrong with the position measurement setup and the readings from 'rxp' currently do not correspond to the receiver's actual position. Brett manually positioned the platform on 15/5, and the S/X receiver should be on axis.
We've noticed at Hb that it doesn't recover properly from wind stows. Check the wind speed (average should be below ~50 km/h) and the anemometer display on time[hb|ke|yg]. If it shows the antenna NOT in a stow state but the field system or eRemoteCtrl report it's in a stow state, then it needs fixing. Recovering seems to be problematical but this seems to work if nothing else does:
(just to make sure the recorder is stopped)
(should turn the drives off, check HMI on timehb)
(This will exit the field system software)
. Then in eRemoteCtrl:
(check HMI and see if drives power up)
(schedule should start. Wait for next source command and see if antenna slews)
Target power levels are listed on the DBBC IF Target Power Levels page:
If the power levels in the dbbc are not reaching their targets, check if the automatic gain control (AGC) is reaching its limit:
The important part here are the last three numbers, 45000 here is the target power, 63 is the automatic attenuation, and 48147 is the measured power.
63 is the maximum attenuation agc can give you and our power is still too high so we need to add more attenuation to the IF chain.
The RF and IF Configuration page has with instructions on how to set attenuation manually.
If the isdx settings when drudging are wrong, for example:
ifa=1,agc,1 ifb=2,agc,4 ifc=3,agc,2 ifd=4,agc,2
Or if the response from iread looks like (for example)
2013.018.01:46:35.31/ifa/1,agc,1,1,63,28940 2013.018.01:46:35.31/ifb/4,agc,4,32000,63,32352 2013.018.01:46:35.31/ifc/2,agc,2,1,63,11389 2013.018.01:46:35.31/ifd/2,agc,2,40000,41,39524
this indicates that the DBBC target power levels are incorrectly set. To correct this, use pfmed and edit the ifdsx procedure to set the target levels directly in the current active schedule procedure file: e.g: r1593ke on pcfske
pfmed pfmed: pf,r1593ke pfmed: ed,ifdsx
ifa=1,agc,1,35000 ifb=4,agc,4,28000 ifc=2,agc,2,28000 ifd=2,agc,2,40000
If the DBBC computer hangs or crashes and the vnc session and ssh server are unreachable you may need to reboot the machine. Details for how to do this can be found on the Internet Power Switch (IPS) Page
If you are unable to convince the DBBC, Mark5, and field system to sync their clocks, first follow some basic troubleshooting. Check the maser is working correctly, the gps clock (tac32) has the correct time on the (ho/ke/yg)time vnc, that ntpq -np returns a low offset and/or reboot the dbbc/Mark 5 and run fmset.
If all this fails you may need to reconfigure the dbbc into an 8 MHz (non CRDS) mode to get it to sync and then swap it back to 4.
Symptoms: It's not possible to set a VSN code. On the camera, both modules are shown as 'active' (i.e. red light).
Cause: This happens when both modules have a blank VSN. In this case, the Mark5 thinks it is in dual-bank mode (recording to two modules simultaneously) and the Dimino software in unable to recover from this.
Solution: There is an alternative to Dimino called jive5ab which allows relabelling of VSNs when in this dual-bank mode.
operand stop Dimino:
When you have completed the operations with jive5ab, you should kill the process before restarting DIMino. You can do this either by using Ctrl-C in the window running jive5ab or with the “kill” command - check the process number with “ps -ef | grep jive5”. Either way, allow a few seconds to pass between ending jive5ab and starting DIMino to let the diskpack controller settle.
If the Katherine skycam doesn't show up on the “web cameras” webpage, it could either be that VLC (which saves the images from the camera) or the FTP script, which uploads the image) has stopped working. To reset these, VNC to
In the case of Yarragadee and Hobart: the script is ASC Uploader on the Desktop
In case of dodgy screen display at the Observatory (4 screens) open a terminal on ops1hb and type