THE CLUSTER MISSION CONTROL SYSTEM: A DISTRIBUTED ARCHITECTURE FOR A REAL-TIME CRITICAL AND MULTIPLE SPACECRAFT CONTROL SYSTEM (CMCS)
Erik M. Sørensen*, Gianpiero Di Girolamo* & Richard Corkill**
*European Space Operations Centre (ESOC)
Robert-Bosch Strasse 5, D-64293 Darmstadt, Germany
Fax: 49-6151-903010
E-mail: esorense@esoc.esa.de, gdigirol@esoc.esa.de
**Cray Systems GmbH
In Leuschnerpark 4, D-64347 Griesheim, Germany
Fax: 49-6155-871787
E-mail: rcorkill@esoc.esa.de
On 4 June 1996, the Cluster mission terminated prematurely due to the Ariane 5 accident. The Cluster mission was aimed at the study of small scale structures that are believed to be the fundamental in determining the behaviour of key interaction processes of cosmic plasma. The mission was to be controlled from the European Space Operations Centre (ESOC). ESOC was also in charge of the commanding of the scientific payloads on board the four Cluster spacecraft and of collecting and distributing the scientific results to the scientific community.
This paper describes the architecture of the Cluster Mission Control System (CMCS). The CMCS is a software system developed in ESOC in order to support the Cluster Mission Operations activities related to the on-line monitoring and control of the four spacecraft. Although the CMCS has not been used to support the mission operations as foreseen, it has been extensively tested in the pre-launch campaign, and validated both with the spacecraft and the simulator. This combined with the modular design makes the CMCS a potentially valuable input for the development of other control systems.
Cluster was a scientific mission funded by the Solar Terrestrial Science Programme (STSP) whose purpose was to investigate key interaction processes of cosmic plasma. The Cluster mission comprised four identical spacecraft launched into large, elliptical polar orbits around the Earth. After transfer to their operational orbits they were to fly in formation to allow scientists to measure subtle changes in the interaction between the Earth and the Sun. The four spacecraft look at how particles from the Sun interact with the Earth's magnetic field. Cluster was to observe the magnetic and electrical interactions between the Earth and the Sun, by making direct measurements of the fields and particles trapped in the Earth's magnetic field. For the first time, small-scale fluctuations in interplanetary space would have been measured between each spacecraft as they orbit the Earth.
The responsibility for the definition, design and implementation of the Cluster ground segment (described in [1]) was assigned to the European Space Operations Centre (ESOC) of the European Space Agency (ESA) located in Darmstadt, Germany. ESOC also had the responsibility for the operations of the Cluster spacecraft (both platform and payload) and for the Cluster ground segment.
The Cluster Mission Control System (CMCS) is an important part of the ESOC Operations Control Centre (OCC), the central facility responsible for the operations of the four Cluster spacecraft. The CMCS was developed under the responsibility of the Data Processing Division in the Flight and Control Systems Department at ESOC, supported by the contractors Cray Systems (UK) and Spacebel (Belgium).
The goal of this paper is to present the modular software and hardware architecture of the CMCS, concentrating on the novel aspects of the system.
Figure 1 shows the place of the CMCS in the Cluster ground and space segments. The CMCS, which is the central software component of the OCC, has a number of interfaces to systems both internal and external to the OCC. Rather than describing each of these interfaces in detail here, we provide an overview by briefly describing a typical cycle of operations during the routine phase. This description is not intended to be complete, but is aimed at providing a suitable background for the understanding of the remainder of the paper.
Each experiment on-board the spacecraft is represented by a chief scientist called the Principle Invesigator (PI). The PI is responsible for co-ordinating the inputs from his team of scientists, which may be distributed between several Home Institutes, (e.g. universities). Each PI submits a required schedule of operations for his experiment to a central facility called the Joint Science Operations Centre (JSOC), which is located at the Rutherford Appleton Laboratories in Oxfordshire UK. The JSOC merges these schedules into a single schedule for submission to the OCC.
The merged schedule of required experiment operations is processed by the Cluster Mission Planning System (CMPS) at the OCC. The CMPS takes into account ground constraints (e.g. visibility of the spacecraft from the ground stations) and spacecraft constraints (e.g. power, tape recorder capacity) in order to produce a detailed schedule of operations which is sent to the CMCS.
ESOC spacecraft operations personnel, who are responsible for the health of the spacecraft, operate the CMCS. They accept telecommanding inputs not only from the scientists (via the CMPS), but also from two other sources, viz. the Flight Operations Procedures Software (FOPS), and the Flight Dynamics System (FDS). A further interface (to the spacecraft Software Development Environment, SDE) is supported to facilitate the maintenance of the on-board software.
The CMCS interfaces to the ground stations for the uplink of telecommands and reception of telemetry. The telemetry data is of essentially two types, viz. spacecraft housekeeping data and science data. Housekeeping data is periodically transferred to the Spacecraft Evaluation System (SPEVAL) at the OCC, which is responsible for maintaining a full mission archive of housekeeping data.
The science data is routed directly from the CMCS to the Cluster Data Disposition System (CDDS) from where it is made available to the PIs. The CDDS supports two facilities for accessing science telemetry: a near real-time on-line access to the last 10 days data for small data sets; and a full data set which is dispatched daily on CD-ROMs. Therefore, as shown in Figure 1, all electronic OCC-external interfaces are via the CDDS.
The Cluster Mission Control System (CMCS) provides functionalities to support the real-time and near real-time data processing tasks that are essential to control the mission. One of the main drivers for the design of the CMCS is the unique characteristics of the Cluster Mission being the first multiple spacecraft mission controlled by ESOC. In particular, the CMCS simultaneously supports four spacecraft during the Launch and Early Orbit Phase (LEOP) and two during the routine Mission Operations Phase (MOP).
Most spacecraft and payload operations are pre-planned and executed from the on-board time tagged queue. Nominal real-time operations are limited to the acquisition of on-line and stored telemetry data of all types (housekeeping, memory dumps, science), the former generated at fixed time interval and the latter recorded during the non-coverage periods and dumped to the ground station during the real-time contact periods, and the uplink to the on-board memory of time tagged commands for the execution of all pre-planned platform and payload operations. In addition, real-time operations are also used for special situations like complex payload software maintenance and contingencies.
All the requirements related to the characteristic of this mission as defined by the users have been carefully analysed and resulted in the design of a distributed computer system, as shown in Figure 2.
The main motivations for such a decision are as follows:
The distributed architecture reduces the costs of hardware and software.
The costs of procuring and maintaining many small-sized computers (VAXstations) are significantly lower than those for a large computer capable of hosting the whole system.
Software licenses are also usually cheaper for small-sized computers. Furthermore, the distributed architecture allows to optimise the use of licences throughout the system.
In the distributed architecture, each element is independent. In general, the failure of one element does not affect the other ones. Furthermore, the global system performance is not compromised in case of heavy usage of resources by one element. This results of particular importance above all in the case of time-critical tasks (e.g. real-time commanding).
In the distributed architecture, off-line tasks are hosted on a dedicated computer. Important, resource-demanding off-line tasks (e.g. database maintenance) are totally decoupled from the actual spacecraft operations and other time-critical tasks.
3.1 CMCS FUNCTIONAL BREAK-DOWN AND COMPONENTS
The main CMCS functionalities and system components are briefly described below. All these functionalites are supported by a sophisticated and homogeneous graphical user interface.
3.1.1 GROUND STATION INTERFACE AND CONTROL (CNCTRS)
The Cluster Network Controller and Telemetry Router System (CNCTRS) performs the ground station (GS) interface and control functionalities. These include facilities for telemetry (TM) reception, telecommand (TC) transmission, range and range rate measurements, and ground station scheduling for the two unmanned MOP stations. The housekeeping (HK) TM is routed to the relevant Cluster Dedicated Control System (CDCS) for more specific processing. The Science TM are delivered to the Cluster Data Disposition System. Furthermore, the CNCTRS gathers all the TCs as they are sent from the CDCSs and forwards them to the relevant GS. The CNCTRS is also used for acquiring and storing range rate measurements done at the GS which are made available for access to the Flight Dynamics System and for transferring files to the GS (i.e. Antenna Pointing data, Backup TC and GS Schedule, the latter received from the CMPS).
The following factors have driven the design and implementation of the CNCTRS:
§ The CNCTRS co-ordinates ground station operations.
It is possible to use any of the ground stations to support any of the four spacecraft. This requires co-ordination at the OCC, which is provided by the CNCTRS.
§ The CNCTRS is generic as far as possible.
Since the ground station equipment to which the CNCTRS interfaces may be re-used on future ESOC missions, the CNCTRS has also been designed with this same goal in mind. In particular: (a) the software is table driven; (b) the design is modular, to minimise the effort required to add new interfaces; (c) the OCC-internal interfaces typically use TCP/IP and FTP protocols, since these are widely used protocols.
§ The CNCTRS makes use of software developed for past missions.
From a cost point of view, it could not be justified to develop all CNCTRS software from new, when past missions (e.g. ERS-1, EURECA) had already successfully developed software to support particular ground station interfaces.
Although the CNCTRS supports a number of interfaces which have been used before on other missions, it also supports some new interfaces. In particular, Cluster is the first mission to use the Telemanipulator Mark3+ ground station equipment (TMP3+) for telemetry data.
3.1.2 THE SPACECRAFT DEDICATED CONTROL SYSTEMS (CDCSn)
Each of the four Cluster Spacecraft Dedicated Control Systems (CDCSn, n=1,...,4) performs the functionalities relevant to the spacecraft monitoring and control and on-board software maintenance for a specific spacecraft. The CDCSn comprises three principal sub-systems.
1. Housekeeping Telemetry Processing provides the capability to process HK data, including traditional binary and analogue parameters, special Cluster event messages and Memory Dump data. The spacecraft HK and science TM are time-stamped with a measurement error less than 2 milliseconds. Parameters can be processed using calibration curves to convert them into engineering and/or functional parameter values. Computation algorithms can also be triggered during the TM processing in order to derive auxiliary values from parameters received in the TM. Then HK TM is made available to the Operational staff onto the On-Line Monitoring System where all the relevant parameters values are extracted from the frames and displayed on the Operators consoles in alphanumeric, plotted and synoptic format. Finally, the TM is filed into Short Terms History Files (covering the last 10 days of mission HK data) where it is available for retrieval (e.g. for off-line investigations and remote access to the Flight Dynamics Control team). The TM is regularly transferred to the SPEVAL system for long-term archiving; here all the spacecraft performance evaluation activities are performed.
2. Telecommand Uplinking and Verification supports the telecommand chain of the CMCS. The spacecraft operators are provided both with interactive manual command and automatic schedule uplink capabilities. Different sources for TC are supported: commands or command sequences can be directly imported from the users maintained Data Base; OBSM dump and patch command are received from the OBSM; time-tagged sequences to schedule on-board activities are received from the Mission Planning; other time-tagged sequences to perform manoeuvres are received from Flight Dynamics; and Instrument Patch Commands can be received from JSOC. Any uplinkable item is passed to the CNCTRS. The CDCSn is also responsible for performing the TC uplink and on-board execution verification via returned TM.
On-board Software Maintenance (OBSM) provides a facility for the configuration management of on-board software images received from the Software Development Environment (SDE) and memory dumps downlinked from the spacecraft. It also supports the comparison of these dumps, and the generation of patch and dump commands.
The following factors have played an important role in the design and implementation of the CDCSn:
§ The CDCSn systems are essentially independent.
There were two main reasons for implementing CDCSn on four separate machines: (a) When the CDCSn is supporting operations which are CPU intensive (e.g. playback telemetry processing) then this will not impact CDCSm's (mn) real-time critical tasks; (b) The CDCSn systems are essentially independent.
§ The CDCSn software is identical for each spacecraft.
The spacecraft number n is not hard-coded in the software, but configured via a logical name which is read at run-time. The advantages of this approach are that: (a) It facilitates the maintenance of the software; software changes only need to be made once, and not four times. (b) The system can be easily switched from one spacecraft to another.
§ The CDCSn is based on the spacecraft control software kernel (SCOS-1).
Since 1988, most of the spacecraft control systems developed at ESOC have been based on a software kernel called the Satellite Control and Operations System (SCOS-1). SCOS-1 provides a number of core functions such as the processing, filing and display of telemetry data, thereby reducing the cost of the control system development.
§ The CDCSn telecommanding software is based on the ERS-1 telecommanding software.
In the area of telecommanding, a cost-effective low risk solution has been adopted. The ERS-1 general approach and design have been re-used. This has resulted in an accelerated development, with no major problems.
In summary, the approach to the CDCSn has been to provide a low risk solution, building on ESOC's experience of previous missions. At the same time, the opportunity has been taken to make improvements where these are dictated by user requirements, or where they can be provided by recent advances in technology (e.g. in the area of the user interfaces).
3.1.3 DATABASE MANAGEMENT SYSTEM (CDBS)
The Cluster Database System (CDBS) supports all the database functions. The CDBS is an off-line system that manages a single database where the TM/TC characteristics for all four spacecraft are defined and maintained for the entire duration of the mission. It provides the facility to import the AIT Data Base from the Spacecraft Manufacturer.
The following factors have driven the design and implementation of the CDBS:
§ The CDBS must support the spacecraft control software kernel (SCOS-1).
SCOS supports the telemetry-related definitions in the database. SCOS-1, therefore, places a constraint on the design of CDBS, (as for the CDCSn). In particular, the SCOS-1 database is implemented using the ORACLE relational database running under VAX/VMS.
§ The CDBS is implemented on a machine separate from the on-line control systems.
This approach was selected because database maintenance is an off-line activity which can, at times, be very computer resource intensive. Further, it reduces licence costs.
§ The CDBS maintains a single ORACLE database containing the information for all four spacecraft.
The database design identified the need for two types of ORACLE tables: (a) those that are spacecraft specific; (b) those that are common to all spacecraft. A single source database has therefore been implemented. However, by setting up different ORACLE user accounts with different views, the source database is separated into four logically separate databases.
§ A windowing user interface was required.
Previous mission have used an user interface implemented using SQLforms. Although satisfactory for single spacecraft missions, in the case of Cluster a more advanced user interface was necessary, e.g. in order to easily copy information from one spacecraft to another. This user interface, which uses Motif, has been implemented with re-usability in mind.
CDBS provides a number of other functions which are additional to those supported on other ESOC missions. For example, an interactive facility is provided for the automatic import of the spacecraft manufacturer's database; support for the reprogramming of telemetry is provided. In conclusion, CDBS provides a good starting point for the database facilities of the control systems of the future.
3.2 HARDWARE AND SOFTWARE ENVIRONMENT
The CMCS is based on a distributed architecture comprising six prime computers, each with a dedicated standby machine. More details about the CMCS architectural design can be found in [2].
The CMCS sits on the operational LAN and is accessed by operations staff at ESOC (i.e. no remote access is allowed). The external access for exchanging data with external nodes is nevertheless possible via the CDDS. In fact, the CDDS allows secure connection between the external world (e.g. Cluster scientific community) and the ESOC protected operational LAN in support, for instance, of payload command requests. The man-machine interface runs on SUN workstations for spacecraft monitoring and control and on X-window terminals for Database maintenance, OBSM facility and CNCTRS application. Each CDCSn can support up to ten SUN workstations each having three screens and a total of nine windows per SUN workstation. For monitoring and control, telemetry can be displayed in alpha numeric format, graphical format and as synoptic mimics.
The modularity of the system along with the flexibility of each of its logical components is expected to be a valuable source of input for the development of other mission control systems. The flexible design of the system configuration has already allowed the usage of CMCS in supporting missions related activities namely tracking campaign of various satellites.
In similar way it would be possible to expand the current architecture in order to interface with protocols not foreseen in the Cluster development still with an amount of effort considerably lower than a brand new development. Moreover all the modifications required in this process would be restricted to well identified modular system components.
Needless to say, the system as it is today is ready to fully support the Phoenix mission.
The experience gained in the Cluster preparation has shown that a distributed architecture for science operations provides more flexibility and allows trade-offs to be made between the various elements of the whole system. This allows to optimise the end-to-end data return and to reduce project's costs.
On the other hand, distributed systems call for clear definitions of the interfaces among the elements. Great attention has to be paid to the definition, the review, and the configuration control of these interfaces. As many of these interfaces involve the science community, it is critical that scientific community members take actively part in their definition and review. It is also important that these interfaces are tested well in advance of the start of the mission with all parties involved.
[1] The Cluster Mission Operations, P. Ferri, M. Warhaut, ESA Bulletin no. 8
[2] Cluster MCS Architectural Design Document, ESOC, CL-ESC-RS-0310, Issue 2.4, 31/8/93