Goals for R2.1

1. Put in what didn't make it for R2
   (GianniRaffi) Specify please explictly R2 overdue features and R2.1 features needed to meet these Goals/Deliverables.
2. Expand and enhance the simulation capability, with the aim of being able to test and exercise the system in the absence of antenna, receiver and correlator hardware more extensively than has been possible up to now. There are two types of simulation that we should use:
   1. A small subset of hardware simulators (discussed in more detail in the Control and Correlator sections below). This should have as a by-product the development of more unit tests. These simulators should be able to produce realistic data, including noise and anomalous conditions.
   2. More sophisticated simulation of ALMA-like data streams, using either data produced by the Telescope Modeler (Offline) or by reformatting and reusing data from other observatories, e.g., Plateau de Bure. These simulations will exercise the systems downstream from Control and Correlator, but in some cases will use projects generated by the OT.
3. Run a "realistic" end-to-end system test, from the creation of a project via the OT through Scheduling, a simple simulation (details) in which Control and Correlator are coordinated, TelCal, QuickLook and Offline.
4. Deliver clean code by release date, including: standard, agreed-upon names for notification channels and events, no configuration parameters hardwired into compiled code and no use, implied or explicit, of INTROOT.

Preconditions

1. All subsystems are to use and be built with ACS 4.0

Subsystem deliverables

ACS

• Deliver any patches to ACS 4.0 that become necessary during subsystem preparation for R2.1
  (GianniRaffi) Should patch release dates be planned, given also that Archive relies on them?

Archive

Driven by the need to have the archive reliable and stable, its release cycle is currently tied to that of ACS. Therefore, for R2.1, the archive released with ACS 4.0 in November 2004 will be used. However, the first five items listed below will become available in advance of R2.1 and be distributed with the usual ACS patch releases like 4.0.1; whether the ASDM-related task will be completed by R2.1 is TBD.

1. Design a flexible data model for user management together with obsprep, executive and scheduling. Provide an implementation that satisfies the needs of these other subsystems as well as the archive-internal ones, such as access restrictions based on users or organisations etc. Archive will have a draft proposal for discussion at the December all-hands meeting.
2. Fix to streaming interface to bulk store; Performance and reliability tests on bulk store
3. Provide facility for storing, searching, and retrieving:
   1. log information of the form produced by the ACS logging system.
   2. event information of the form produced by the ACS event system. Check whether code from ACS (event browser) or Exec can be reused to optionally filter messages.
4. Enhance archive manager to follow links among XML documents, so that it can display entire observing projects.
5. Transparent rewriting of monitor store to be aligned with new bulk store again.
6. Export of ASDM representations, and support for queries on ASDM.
   Both tasks require a better definition of the ASDM; for the latter, the so-called "physical model" must be worked out, defining the maximum chunk size of ASDM storage in the archive which can still support the queries.
7. Monitor point Archive through a convenient (enough) interface.

Control

1. Provide support to the prototype system integration laboratories. This includes both supporting/improving delivered software, and also doing device-level check-out. This will take a substantial amount of time.
2. Substantially complete the list of device drivers. This is now possible now that most ICDs are believed to be in decent shape.
   • Every device driver should have a very simple (constant values type) simulator.
     • Very few device drivers could attach to a shared simulator (see more under Offline).
   • It would be nice if all device drivers came along with Abeans panel. May not be realistic.
3. Complete the Command Control API:
   • complete specification by 1 January 2005; coordinate with ObsPrep to express this specification in a way that can be used for syntax checking and parameter validation
   • for each command in the API, documentation of the low-level hardware commands that are issued as a result
   • first cut observing scripts for single field interferometry prepared by subsystems scientist and/or other SSR members by 1 February 2005 (Salome'/SSR should write scripts, agreed to by Lucas).
   • complete implementation by 1 March 2005.
   • Delivered PSI scripting functionality should be in terms of defining low-level parts of the CCL API from the bottom up.
4. Data capture
   1. (Re)define Telescope Data Model (TDM) keeping in mind GlendenningsPreviousTDMComments (mainly: minimize changes).
   2. Attach at least all simulated values and some critical calculated values to the TDM.
   3. Work with Offline to reimplement data capture to cope with new TDM.
5. TBD GUIs integrated with the Executive subsystem (Glendenning: cannot imply significant GUI development by control. Possibly include a simple Abean panel.)
6. Antenna blanking signal to correlator. Marson thinks that this is unlikely, Glendenning thinks that if we confine it to sending the blanking signal based only on the antenna positions then it should be possible.

Correlator

1. Produce output appropriate to the state of the coordinated Control-Correlator simulation object (see discussion under Offline)
2. Additional TBD
3. From Correlator subsystem feature list for R 2.0 and 2.1
   1. Delay Control
      • Subscription to geometric delay channel
      • Application of quantized delays to the correlator for the ATF baseline
      • Removal of residual delay
   2. Maintenance
      • From Maintenance.idl
        1. Limited runDiagnostics()
        2. setAntennaNumbers()
        3. Limited reset() - may be deferred to post R2.1
      • Quadrature cable tests
   3. Antenna Blanking
      • Dependent on Control which tentatively has planned to implement antenna blanking R2.1.
4. Tunable Filter Board Support
   • There are a set of tasks required to support the TFB in order to meet their CDR deadline of March 2005:
   1. Programming of TFBs for two-antenna tests
   2. Revised quantization correction
   3. Sub-band stitching
   4. Various testing support software.
5. System Integration Lab Support
   • The SI lab tests will be needing support for the prototype correlator in the coming months. The tenative schedule has first fringes at the ATF in June 2005.

Executive (JS & HS)

1. (currently under discussion:) generic display of GUI frames from the other subsystems: HLA, Exec, Control and Pipeline (Quicklook) need to formulate an interface spec for this. There must be a technical interface (e.g., based on Abeans which would allow the easy use of gadgets, and provide defined error handling and logging). Perhaps also define facade component: each subsystem's master component could become a facade that provides data to the GUI and gets updated from the other components of that subsystem.
   Probably also need to request other subsystems to provide info on foreseen number of values to be used for display, estimated mask size, etc, to be delivered well before R2.1.
   (GianniRaffi) This model could be more or less loosely coupled to Exec. One extreme is to have stand-alone GUIs with modes (master/monitor, verbose/quiet) passed as parameters to Master Component. TBD

HLA

1. Design and implement first version of Observatory Characteristics
2. Complete transition to HLA-maintained and -generated ASDM in collaboration with Offline.
   1. Issue revised ASDM document.
3. Continue to develop APDM to meet Control, Correlator, Scheduling, Pipeline requirements for ATF support.
4. Produce and publish html documentation for artifacts (code, schemas) generated from data models.
   (GianniRaffi) Was the site characteristics DB an HLA R2 deliverable?

ObsPrep

1. With HLA, ensure that APDM (and its support in the OT) is adequate for ATF use.
   (GianniRaffi) Looks like a pre-requisite, surely it should come early at least.
   (AlanBridger - 22 Oct 2004) We believe we are close, but we need feedback. Our User Test 2 will help, so would running the whole system in simulation.
2. Provide better support for entering Observing Scripts in the Command Control Language:
   1. Checking of Python syntax
   2. Assuming a satisfactory computer parseable definition of the CCL is provided by Control, provide specific CCL syntax checking (method signatures, types, values of arguments, etc.)
   3. With Control and HLA, investigate, define and implement mechanism for Observing Script to retrieve parameters from the static part of an SB.
3. With HLA, Scheduling, Control, Correlator, define items needed from ObservatoryCharacteristics database.
4. Ensure that OT-provided I/F for interactive scheduling at the ATF works the way ATF users want it to.
5. See also APDMChangesAfterR2

Offline

1. Complete DataCapture process (write of ASDM)
   (GianniRaffi) Could this be done quickly? Seems on critical path for e.g. Pipeline.
2. ASDM filler for Offline package
3. Convert to UML ASDM generation framework (pending 2005/01 review - added 2004/11 - JPM)
4. ATM library use for calibration within AIPS++
5. Simulator: ASDM generation facility based on SchedBlock parameters
6. Offline Requirements work (SS9, SS10, SS11); see http://projectoffice.aips2.nrao.edu
7. Testing Goals:
   1. Single Field
   2. Mosaic
   3. Single Dish/Interferometric Combination
8. Framework migration: ACS/AIPS++/Task parameter prototype

Control/Correlator coordinated simulation: (BrianGlendenning) I think the most that would be practical is: having a point source model of the sky and having the correlator reflect where the antennas are pointed (i.e., tie together the correlator and the ACU simulator). One could conceivably add the nutator and total power detectors, but in general even the first part may be ambitious. This simulation could be as simple as:

• Point-source model of the sky
• Apply a simple primary beam model for sources off the tracking center (ACU coupling)
• Apply the phase offsets from the phase tracking center (coupling to highish-level control entity, correlator)
  • While very simple, I believe that this could allow us to do end to end tests (e.g., of the Sci IPT desired calibrator survey) for all subsystems (e.g., TelCal could probably do pointing calibration).

[I edited out my earlier general comment on simulation as the scope is more clear now.] I just comment that we are not expecting to test TelCal engines on this simulation; they can be fairly well tested on data from other telescopes or using the off-line simulator when available.

-- RobertLucas - 18 Oct 2004

A comment from RalphMarson

1. Its unclear to me what we will achieve with the coordinated simulation discussed above. One goal is we can simulate the telescope calibration survey but then Robert says there are other ways to simulate data input to TelCal. We need to be clear about why we are doing a more coordinated simulation and one way of doing that is to state specific goals. Then we can balance the cost of simulation against the benefit.

Pipeline (JS/HS for display; AF for content), modified by DSS based on planning spreadsheets

(GianniRaffi) We seem to have lost on the way the R2 Python deliverables.

Quicklook Pipeline goals

1. Implement at least one display via the Exec (see discussion under Exec)
   • I (Lindsey) agree that the main focus for R2.1 should be on delivering a simple but useful GUI that displays a small group of telescope calibration results (some known e.g. phase rms, some TBD) and array monitoring results (TBD, probably those mostly closely related to the calibration results).
   • It is still unclear to me (Lindsey) what exactly the executive requires me to provide (besides the GUI) and how the executive plans to use the GUI. Knowing the latter use case might help a bit with the implementation. I gather that the Telcal system provided the Executive with a GUI for R2 (?), so maybe some of these details have been worked out? I am concerned about any GUI standards that would force me to use Abeans as these seem to require the use of baci properties and will probably not be appropriate for certain kinds of scientific data display in any case...
     Topic for all-hands meeting in December (JS)
2. I (Lindsey) think it would be useful to implement at least one quicklook alarm (TBD, but probably one dependent on calibration results) in order to understand the alarm facility. I gather support for alarms already exists in ACS, but it is unclear to me to how they will be implemented ALMA wide. Has this been worked out or is it premature to start worrying about alarms at this point ? In order to do this, communication with other subsystems is important. Specifically:
   • Keep up with data capture development. This is very important but likely to be time consuming as data capture will probably evolve considerably between now and 2.1.
   • Keep up with the Offline filler work which is beginning now. It is very important for quicklook to determine how it will deliver the data capture and correlator data to the Offline filler. This is likely to be time consuming as considerable testing will be required.

Science Pipeline goals:

1. With ObsPrep and HLA, define content and format of pipeline parameters sufficient for single field interferometric processing.
   • Notes on this goal: Work out and implement the heuristics parameter passing mechanism from user-> obsprep(via ObsProject)-> scheduling(via ProjectStatus)->pipeline with obsprep, scheduling, and hla. This mechanism should be kept very simple and be as transparent as possible to the actual parameter definitions. The definitions will evolve rapidly and should be kept under the control of Pipeline for the time being.
2. The most important goal for the science pipeline is to be able to do basic processing for single field interferometry in the ALMA environment. This involves work in three areas: 1) concepts / definitions, 2) heuristics, 3) infrastructure. Most of the concepts / definitions actions are clarify actions because these are things that are not yet totally clear to me ... Some I thought I understood but am no longer totally sure of ... All of them involve other parts of the system so clarification cannot be done by the pipeline alone. Concepts/Definitions clarification needed to support this goal:
   • Clarify the concept of a standard observing mode, e.g. single field interferometry, and how this concept will flow through the system. Can the standard processing mode really have a one to one relationship with the standard observing mode(s) as defined? Can user intent, e.g detect a weak point source in a single field, really be captured by a standard observing mode definition such as single field interferometry.
   • Clarify how and at what level(s), e.g. observing unit set, scheduling block, scan etc, user intent will be specified and how the observer intent information will flow through the system.
   • Clarify the relationship between the science pipeline and the calibration database. How will the science pipeline use the calibration database results, calibration results in the auxilary data, etc. How and in what format will it store its own calibration results? and flagging tables? (not really a calibration issue but ...). What is the relationship between the calibration database discussed in the past and the observatory characteristics database?. Are these the same, different, or related?
   • Clarify the meaning of an observing unit set? Is is an observing unit or a data processing unit or both? If both how do the two relate?

Heuristics goals:

1. Convert to Python scripting environment (R2 unmet deliverable).
2. Develop initial heuristics for single field interferometry with emphasis on automated flagging (both raw and calibrated data) and generating calibration solutions.

Infrastructure goals:

1. Use the filler provided by Offline to convert the ASDM into a measurement set. This is a critical step which depends on the Offline schedule.
2. ALMA/protopipeline/AIPS++ build integration. This involves matching system with different release schedules (ALMA / AIPS++) so the protopipeline code keeps functioning smoothly.

-- LindseyDavis - 20 Oct 2004

Scheduling (AF)

1. Add support for fixed-time scheduling blocks.
2. Add support for expanded U-V coverage (requires APDM and ObsPrep collaboration.
3. Add support for specification of elementary weather quality conditions.
4. The details of how the above requests are to be specified in the project definition is to be worked out with ObsPrep.
5. Send messages to the Telescope Operator and the PI via the Executive interfaces.

Software Engineering

1. Ensure that standard names following standard conventions are used by all subsystems for Notification Channels, Containers, Components and Events.

TelCal (JS & HS)

1. Prioritized items to be extracted from PlansForR2p1