-- DebraShepherd - 31 Aug 2007

• Downtime Log: Please list active use + passive use + unscheduled time + down time = 24 hrs (see previous page for definition of each of these types of activities).

September 30 Sunday

• CIPT -
• PSI -
• Sci - verify radiometric pointing 6-9am - Debra for Antonio Hales
• Sci - determined antenna efficiency from 5-point scans on Mercury + tipping curve. Results: T_rx(VA) = 178 K, T_rx(AEC) = 73 K, Total antenna aperture efficiency for both VA & AEC = 51% +/- 10% - Debra for Peter Napier
• Dlog - Active: xx hrs, Passive: xx hrs, Unscheduled: xx hrs, Downtime: xx hrs

September 29 Saturday

• CIPT - Regression tests: 9:30am - 4pm
• PSI -
• Sci - verify radiometric pointing 6-9am & 4-11pm (2 hrs downtime) - Debra for Antonio Hales.
• Dlog - Active: xx hrs, Passive: xx hrs, Unscheduled: xx hrs, Downtime: 2 hrs

September 28 Friday

• CIPT/Sci: User tests using new mount & control master. No arrays can be created. 10am-6pm. Software problems (down time) ~ 3 of 8 hours. - Debra
• BEND/PSI static monitoring for the rest of the day
• Dlog - Active: 5 hrs, Passive: 16 hrs, Downtime: 3 hrs (no unscheduled time).

September 27 Thurs.

• CIPT -
• CIPT - developed script to read in PSI archive data (single-valued) for plotting with CASA table tool. - Debra
• PSI/AEC: FE power cable came loose from power strip, reinstalled. - Jack
• PSI/AEC: Caught two mice overnight, will reload and go hunting again tonight. - Jack
• PSI/AEC: Hydraulic brake system inspection shows several leaks at the pump unit. Monthly inspections of fluid levels are planned. - Jack
• PSI/Vertex: Reinstalled the cap screw Juan G. removed from the subreflector. - Jack

September 26 Wed

• FO signal to both LORR's optimized: Wed Sep 26 12:42:12 2007 optimized crd to lorr va 0.5714 @ 2deg and aec 0.2637 at 95deg - Gene
• NTP issue blocked CorrGUI^? blocking 3hrs PSI scheduled time
• LORR FO power issues was attributed to defective fiber - angled connector and key assembled 180deg off in orientation by FTS supplier - Gene
• DLog: Active Use = Downtime = 3h CIPTtime =
• Troubleshot the AEC PFE power supply and bypassed the shutdown circuit so that it is always on. - Jack
• Verified that the Anritsu (BE's) signal generator has no output. - Jack
• Ralph Marson & Jeff Kern worked on getting the ALMA-4.1.3 release working. Problems fixed included
  • Sending an alarm causes the component to become unusable. Looks like a pointer is being deleted twice. This could not be reproduced on a laptop. Work around this by hacking the software to not send the alarm. This change has not been checked in to CVS.
  • The precession from 200 to current date was being done twice. Fix was checked into the branch yesterday and merged into the 4.1.3 branch today.
  • Some problems with old files in the init.py file in the INTROOT. A clean build should flush this out. Hacked this for now.
• Ralph and Jeff worked on understanding the mysterious intermittent crashes, seen with ALMA-4.1.2, when doing five-point scans. We only saw the problem once after trying for some time to reproduce it. Our guess is that there is a race condition in the maci manager as the hangup occurred at a point were a lot of object references are obtained and released. When we saw it happen there were two separate clients trying to release components and one of them immediately then trying to get new references. A fix was put in place for the client that was, unnecessarily frequently, trying to get and release references so it only does it when necessary. In addition Jeff and Ralph will put together some toy code that tests the hypothesis of a race condition in the manager.
• PSI/AEC/Vertex: Installed custom fiber cables from PRR's to WMA's. - Jack
• PSI/AEC: Installed PFE power supply.

September 25 Tue

• NTP issue blocked CorrGUI^? blocking 6hrs PSI scheduled time
• DLog: Active Use = Downtime = 6h CIPTtime =

September 24 Mon

• AEC compressor unit OFF at 07:30MDT (13:30UTC). It was ON at 06:35 when AEC was unlock for PSI test. Turned ON at 07:33MDT, GM_4K_STAGE=26K.
• AEC Front End Total Power Tipping curve tests: R. Ridgeway+E.Duvall+Napier: 07:00-> MDT
• VA shutter was checked and it works both from VA platform and from PSI. 09:20 MDT
• Problem to access VA; ACU screen was frozen (CAMError message on PSI), ACU power cycled and access was recovered, both from ACU screen as from PSI. 11:10 MDT
• On VA: Check of signals from PFE to IFP. J.P. Caram ~ 11:30MDT.
• DLog: Active Use = 06:30 - 16:30 = 10 Downtime = CIPTtime = 0

September 23 Sun

• SciOps^? started at 09:00MDT. Problems with VA: need to Clear Fault using Python-CCL because ObjExp^? did not solved the problem. My mistake, 2 spiral5pt on VA give place to a rare signal on TPD detectors!. Later, system crashed, recovered after 10 minutes. Bad weather, poor signals on Mars, Saturn. G. Duvall and S. Albanna tweak Laser Synthesizer - WMA lock trouble on both? antennas: ~ 20' around 11:00MDT. Time used by R. Laing and A. Hales to exercise, test and stress the system.
• VA Shutter not responding to PSI commands and PSI status 'does not make a sense'. Now it is open: 12:00MDT
• AEC GM_4K_stage temperature = 13K @ 20:00MDT.
• DLog: Active Use = 09:00-19:00 - Dtime= 9h30' Downtime = 30' CIPTtime = 0

September 22 Sat

• SciOps^? started at 06:00, tracking Venus with both antennas o.k. _System Crashed_ @ 06:43. Checked GAS @ VLA-CB, network connection problems. System recovered by Mauricio Zambrano (he also fixed broken VNC connection.) Resume operations @ 09:30MDT. Mount VA crashed at Spiral's call! (AEC was already running.)Resume operations @ 10:05MDT. Venus,Saturn,Mars 3 times (averaging on B baseband on AEC.) Mercury. We observed many _sudden jumps in AEC TPD Monitor BB Detectors_. Signal jumps ~ 0.01V back and forth. We record snapshots of annoying beahaviour. Stop at UTC 00:40 after crash of VA.
• At 12:30 MDT AEC Compressor unit OFF, GM_4K_Stage_Temp=150K!. Turned ON. 6 hours later GM_4K_Stage_Temp=37K.
• CIPT time from 18:40 UTC up to 23:40 UTC: Regression tests around ACS=6.0.4 build (M. Zambrano.)
• SciOps^? resumed @ 23:45 UTC; tracking+spiral on Jupiter with both antennas. End of run, 21:30MDT, 03:30UTC.
• DLog: Active Use = 43'+35'+3h45'=5h03' Downtime = 2h47' CIPTtime = 6h

September 21 Fri

• Operations Started ~06:15 {Emerson-Laing-Remijan-Hales}, system crashes (~06:35) after a number of tries with the spiral, recovered and restarted, crashes again same way (~07:20.) Anyway, foggy, no wind T=6 C. Crashes again on Mars/spiral ~ 08:55MDT (still foggy.)
• Released top CIPT at 09:19MDT. ~ 09:50, induced mount failure, this time on AEC. AZ=66d EL_1=15, EL_2=25. Mount goes into 'Delay Getting Status' & ABM002 crashed.
• Resume SciOps^? @ 16:50 MDT on AEC. Spiral5pt on Mercury, Quasar_3C273, Jupiter. Inestable signal ~ 19:10 MDT, unstable atmosphere?. Used VA to check it was not an AEC artifact. Pointed to the Moon with same objective.
• Track + Spiral5pt on Quasar 3C454.3 @ 21:20 MDT, average=e=5. Reasonable fits in both antennas.

• DLog: Active Use = 2h30' Downtime = 30' CIPTtime = (09:20 - 16:40) = 7h20'

September 20 Thu

• A final test was done in which we moved _fast_ between two positions, just using the mountPanel and we suceed in crashing the mount. So, apparently, the most recent Mount failures are not hard linked to TrackPlanetObject^? and Spiral5pt but embedded at a lower level in the mount itself.
• Precession Tests over Jupiter, use of ObjExp^? to change Epoch and check RA - DEC data in Mount. Actual position is always JD2000 no matter the epoch you used.
• Failure of VA at ~ 17:30. After successful tests in which the antenna was moved between 2 fixed positions, the mount failed when the command "TrackPlanetObject -a ALMA01 -p jupiter" was sent. System to SHUTDOWN, ACSDown, Session EXIT, revioewed and 'cleaned, then reatarted. CCC and CDP CORR containers were down; restarted. System o.k.
• Tests on VA Shutter: it works BUT does not close at EL=90degrees! (same as for AEC)
• Tests on VA Shutter, moving and checking with PSI, and later a check with OBJEXP:GET_GET_SHUTTER; when the shutter it is open, GET_GET_SHUTTER throws 1, a 0 when it is closed, and there is no intermediate (moving) state.
• Mouse chewed through the LO fiber to AEC antenna inside the antenna base, just before the Az fiber wrap. Drew spliced the fiber, and it works fine now. mouse traps were provided by Juan Pablo, and were set out inside the vault and antenna base by Juan Pablo and Drew. This prevented work on the Dynamic Intferometry.
• fiber optic cable damaged by mice in the AEC antenna.:
  
• it rained
• CIPT (Thomas) & Sylas tested the CRD hardware. When reseting the CRD counter, the TE and the 1PPS signals are not aligned. Sylas has the firmware already fixed but the unit did not have its firmware updated.
• Tested Array Time with the "defect" CRD.
• DLog: Active Use = (17:10 to 19:20 - 20) 1h50' Downtime = 20' CIPTtime = 6 hours?

September 19 Wed

• J.Kern, M. Zambrano, T. Juerges 09:30-17:15: Use just one mount per now!
• D. Emerson, R. Laing, A. Remijan 17:15-23:30=06:15 : exercise of TrackPlanetObject^?+spiral5pt with AEC only, over Jupiter, Mercury.
• _Failed at 01:18 UTC_ after a restart of spiral5pt inmediately after move from Mercury to Jupiter. Restarted, works fine.
• Track accuracy test moving a bit over TrackPlanetObject^? position using MICA-ephemerides.
• Collimation tests CAOffset, ELOffset=30 (first AZ, then EL, then back to zero) PtModel^? ~rlaing/aecBand3...
• Test of Spiral5pt on Quasar 1921-293, average parameter e=8. Scattered clouds!
• Tests on AEC Shutter, moving and checking with PSI, and later a check with OBJEXP:GET_GET_SHUTTER; when the shutter it is open, GET_GET_SHUTTER throws 1, a 2 when it is closed, and a 4 in an intermediate state (moving.)
• Tests on AEC Shutter: it works BUT does not close at EL=90degrees!
• Test tracking on an empty position + spiral5pt: still it gets some results, bad quality anyway.
• Used http://www.jgiesen.de/astro/astroJS/siderealClock/ to get LST. VLA_longitude=-107.5592659896938 degrees. It works fine.
• DLog: Active Use = 05h35' (17:15-23:30 - Downtime) Downtime = 40' CIPTtime =07h45'

September 18 Tue

• 3 hr Walsh Function test prevented by non-functioning system software -Peter
• MountController^? fails after 'spiral' calls and this in time crash the mount, the ABM and the LabView^? TPD BB signal. Apparently the procedure over CONTROL/ACC/pythonContainer crashes the other ABM. System switched to a previous build plus patch and then weather station worked fine. It was observed that Refraction Coefficients did not update automatically; it is needed to call TrackPlanetObject^? again over the same position (planet.)
• DLog: Active Use: 4 hours. Downtime 5 Hours (rough estimate: from 13:30 to 22:30 MDT)

September 17 Mon

• Downtime Log:
• Quick fix and exception handling for TrackPlanetObject
• Small network glitch caused by a loose cable caused a full software restart. (Cable is secure now)
• Artm and lo-pdu are now on separated cables, and artm1 is on a 100Mbps network.
• Juan Pablo Caram solved the e-stop problem in AEC and recovers the normal use of this antenna(15:00MDT)
• Problems with Weather Station data gathering. pyWEATHER fails after minutes of work. VLA side looks fine. Refraction coefficientes wrong under ObjExp^?.(M.Zambrano)
• Failure of '''spiral5pt.py''' linked to MountController^?. MountPanel^? goes into error mode. Cleaning system with user tools does not recover an stable system. Consistently there were failures of CONTROL/ARTM&DMC{cppContainer}. Status of those containers evolved green, red, green, finally red. Many restarts and rolling back to 'old' system builds did not helped. Pending! (J. Kern, M. Zambrano.)
• DLog: Active Use= 1 hour Downtime= 7 hours (operations started ~14:30 ended ~22:30 MDT)

September 16 Sun

• Downtime Log:

September 15 Sat

• A temporal script called getRefraction was created. It is located in ~ntroncos/getRefraction
• Applied a MountController from HEAD patch to ALMA-4_1_2_2 to fix regresion of functionality that would permit getting the refraction coeficients.
• SW switched to ALMA-4_1_2_2
• Regression test performed: see ATFSWRegressionTestResults20070915

September 14 Fri

• Downtime Log:
• Delay Testing clearly identified that the DGCK and Correlator are implementing Delay steps in the opposite direction. The Residual Delay correction is clearly wrong and causing confusing results.

September 13 Thur

• Downtime Log:
• New mount patch was applied from tag ATF-2007-09-ITER1
  • Robert Laing helped all day with testing
  • TrackObject script stopped working, so Nico T. wrote TrackPlanetObject
  • Modified Five points script from Gene and Darrel didn't work rigth away(they

moved the antenna and captured data but didn't calculate a fit). After increasing the waiting time between commanding a move, and reading data from 3 to 5 seconds it worked. It seems the new mount is slower. * Tried new methods of the MountController, most of them work OK, Ralph was informed of the problems we found. * In mountPanel: * commanded RA is wrong * no sidereal time is shown anywhere in the panel

• Tested monitor points in the FLOOG
  • GET_FREQ is now working
  • GET_PHASE_OFFSET is wrong
  • GET_MODULE_CODES is not implemented
  • Could test the control points, need to fix the code generator template to

allow control points even in an error state.

September 12 Wed

• Downtime Log:
  • Active Use: 6 hrs
  • Downtime: 3 hrs
    • 2 Hrs: Hardware issues not fully understood, may have been an interaction with the new software
    • 1 Hr: CorrGUI^? timeout recovery
  • Passive Use: 15
• Delay Tracking testing, understanding the Correlator Delay Tracking, numerous detailed issues to be investigated, JK to enter in Jira.

September 11 Tues

• Downtime Log:
  • Active Use: 13 hrs
  • Passive: 3 hrs
  • Downtime: 8 hrs 1 -> CorrGUI^? timeout recovery and corr subsystem restart recovery, 2 Hours -> Thunderstorm and recovery from power glitch, 5 hrs system SW restart failure
  • Unscheduled: 0 hrs
• CIPT today established connection between the Delay Server and the FLOOG and re-established connection with the DGCK. We undertook quantitative tests of the Fringe Tracking capability but the results were difficult to interpret. We think we have a clearly designed test for tomorrows work.
• PSI performed tip test to determine correlation efficiency - Peter
• PSI revised DG and Correlator statistics tool to work on new two station correlator - Gene
• PSI had first look at DTS-DG statistics, adjustements required but looks good overall - Gene

September 10 Mon

• Downtime log: 18 hrs active, 6 hrs passive, 1 hr storms and VA shutter trouble
• BE installed latest version CRD
• PSI fixed TE alignment at correlator caused by CRD swap - Juan Pablo
• PSI quantified IFP BB power levels at DTX face plate - Drew

September 9 Sun

• Downtime log: active use + passive use + unscheduled time + down time = 24 hrs

September 8 Sat

• Downtime log: 2 hours, half an hour restarting the system and 90 minutes with a hardware problem with AEC
• Weather - cloudy day
• Peter did some observations since 4 am. Weather was not good.
• CIPT tried old scripts on the new software at the ATF(ALMA-4_1_2-B). Some of them are not working any more. SipralSearch^? is missing a import AcsutilPy^?.ACSImport and others like SpiralFivePoints^? are not working. The ArrayTime^? was tested at ATF. It was installed as part of the intlist. MasterClock^? was also missing on these release, it went also to the intlist. Software was rolled back.- Thomas, Roberto, Mauricio
• There was a problem with the azimuth driver on the Alcatel Antenna, it was stuck while trying to power on the Azimuth driver. There is Jira ticket AECANT-17. It was fixed remotely by Gene sending an unlock to the break.
• PSI photonic took control of the antennas at 17:00

September 7 Friday

• Downtime log: 15 hrs active, 6 hrs passive, 3 hrs lost on AEC due lost TE overnight, and need to recover.
• Weather - very clear AM
• CIPT: Work on DGCK went very well connection from Simple Delay Server to DGCK was established resolved a few technical details and excersised system.
  • Peter points out that on a 30 m baseline we can transition to the 1/16 sample fine delay steps without issue.
• Sci: 5-point scans on planets 4-9am (3 hrs lost due to AEC TE problem, recovery from photonics work previous night) and 2-5pm (went very well).

September 6 Thurs

• Downtime log: 11 hrs active, 4 hrs passive, 2 hrs unscheduled, 7 hrs down (software failures: CorrGUI^? & performance issues)
• Weather - cloudy, raining
• PSI time most of the day. Stability tests and characterization of the system. - Gene, Peter
• CIPT: Support PSI, trouble-shoot and characterize software problem: NFS overload problem.
• PSI: photonics work - Sarmad
• All: Scheduling meeting to balance priorities and time next week.

September 5 Wednesday

• Downtime log: 2 hrs down (software restart problems), 7 hrs passive use, 15 hrs active use.
• Weather - clear AM, partly cloudy PM
• CIPT ATF: tracked down suspicious weather data - weather update rate for dew point temp is too slow (every 20min, other parms are OK). Drew contacted VLA folks & they agreed to update it every 5 minutes. Provided Darrel with a script to monitor the 2 weather components. Darrel requested to get the 2 refractions constants from the mount - added this information to Tzu's patch for the mount and installed it at the ATF. - Mauricio
• ATF: 6 floor tiles were replaced in the control room of the control trailer. sometime last night one broke off into many pieces. several tiles around this one were also damaged, or so loose and old they crumbled when removing the damaged ones around it. - Drew, Roberto
• SCI ATF: Fringes on Venus were observed while five point was running. Near noon, a chance observation of a quasar near venus was attempted. fringes seen, and screen captures were taken. - Peter, Darrel, Drew.
• Sci ATF: 5-point scans on planets every available time the weather was clear (9:30-midnight except for a few hours during the day and evening). - Darrel, Chris Carilli
• Tony Beasley & Gianni Raffi visited ATF ~9pm, observed 5-point on Jupiter and simultaneous fringes on CorrGUI. A good time was had by all. - Tony, Gianni, Darrel, Chris, Debra, Roberto

September 4 Tuesday

• Downtime log: 14 hrs active, 10 hrs down (5 hrs for software recovery due to OMC failure due to user error early AM, 5 hrs due to AEC subreflector not responding after power outage.)
• CIPT: DGCK commissioning - Jeff
• PSI: Some work on IFDC level characterization but not adequate time to finish. - Drew
• Sci: SD radiometric pointing on Jupiter. - Darrel

September 3 Monday (Labor Day)

• Downtime log: 22 hrs passive, 2 hrs down (interupt of data collection)
• PSI collection of monitor data for PSI and BE. - Gene

September 2 Sunday

• Downtime log: 8 hrs active, 8 hrs unscheduled, 8 hrs down (power outage recovery)
• Computing: ATF-2007-09-ITER-1 build testing. New version of HEAD - Nicolas, Bernhard.
• Computing: OPT tests - Bernhard
• Computing: DGCK Commissioning. - Ruben
• Computing: Primary and Secundary pointing model testing. - Ruben
• PSI/ATF: After storm recovery. Restarted AEC power and all functions, Vertex recovered on its own. Computing guys are here recovering systems and attempting OPT and software fixes and upgrades. So far so good. Gene will look in on the systems later today. - Jack, Manuel, Nico, Ruben, Bernard.

September 1 Saturday

• Downtime log: 5 hrs active, 5 hrs passive, 14 hrs down (major power outage)
• PSI/AEC: Fiber work: removed connectors, spliced, installed fiber in wire tray and foam. - Jack
• PSI/AEC: Installed custom length fiber cables from diplexer to PRR and LORR. - Jack
• PSI/Vertex: Installed replacement charcoal trap. - Juan G.
• PSI/AEC/Vertex: Took images of fiber upgrade for report to be issued soon. - Jack, Juan G.
• Computing: DGCK commissionning. DGCK CCL libraries implementation/tests - Ruben
• Computing: OMC DSP Holography pluggin development - Nicolas
• PSI/Vertex: Installed custom length fiber cables from diplexer to PRR and LORR. - Jack
• Computing: Nico working on central LO aux. rack ethernet power switch.
• PSI/ATF: Major storm, power outage VLA wide, no backup generator. - Jack