Enhanced remote astronomical archive system based on the file-level Unlimited Sliding-Window technique

Data archiving is one of the most critical issues for modern astronomical observations. With the development of a new generation of radio telescopes, the transfer and archiving of massive remote data have become urgent problems to be solved. Herein, we present a practical and robust file-level flow-control approach, called the Unlimited Sliding-Window(USW), by referring to the classic flow-control method in the TCP protocol. Based on the USW and the Next Generation Archive System(NGAS) developed for the Murchison Widefield Array telescope, we further implemented an enhanced archive system(ENGAS)using ZeroMQ middleware. The ENGAS substantially improves the transfer performance and ensures the integrity of transferred files. In the tests, the ENGAS is approximately three to twelve times faster than the NGAS and can fully utilize the bandwidth of network links. Thus, for archiving radio observation data, the ENGAS reduces the communication time, improves the bandwidth utilization, and solves the remote synchronous archiving of data from observatories such as Mingantu spectral radioheliograph. It also provides a better reference for the future construction of the Square Kilometer Array(SKA) Science Regional Center.     

Optimising NGAS for the MWA Archive

The Murchison Widefield Array (MWA) is a next-generation radio telescope, generating visibility data products continuously at about 400 MB/s. Efficiently managing and archiving this data is a challenge. The MWA Archive consists of dataflows and storage sub-systems distributed across three tiers. At its core is the open source software—the Next-Generation Archive System (NGAS)—that was initially developed in ESO. However, to meet the MWA data challenge, we have tailored and optimised NGAS to achieve high-throughput data ingestion, efficient data flow management, multi-tiered data storage and processing-aware data staging.     

The Data Flow, from Observations to the Archive: Simulating the Case of TNG

The telescopes of the new generation allow an archive to be built as a section of data management; nevertheless, careful planning is needed and data handling needs to be designed together with the control system of the telescope itself, both for space-borne and for ground-based facilities. Simulations are essential to understand how observations will be archived, and to build and test an archiving system capable of dealing efficiently with the expected data flow.The TNG (Telescopio Nazionale Galileo) will be one of the first ground-based observing facilities where archiving of both technical and scientific data will be made directly at the telescope as a natural extension of the data handling chain. The results obtained testing the prototype implementation of the archive system at the TNG with a simulated data flow will be shown.     

The GALEX public archive at MAST

The Galaxy Evolution Explorer (GALEX) was successfully launched on April 28, 2003. Over the last 7 years, the GALEX mission has expanded our understanding of the process of star formation and galaxy evolution. GALEX data, consisting of far-UV (1344-1786A) and near-UV (1771-2831A) imaging over most of the sky, and grism spectroscopy, are available to the entire astronomical community and to the general public via the Multi-mission Archive at Space Telescope (MAST). The main characteristics and access tools to the large data archive (16TB) are presented. Groups within the GALEX Science Team, and the MAST archive teams, have undertaken complementary projects to conduct all-sky searches for time-variable and transient sources within GALEX data (e.g. Welsh et al. in Astron. J. 130:823, 2005; Wheatley et al. in Astron. J. 136:259, 2008). Results and tools for variability searches developed by MAST during this process, will be made available to the general community.     

Data Archive of the Hinode Mission

All of the Hinode telemetry data are to be reformatted and archived in the DARTS system at ISAS and mirrored to data centers around the word. The archived data are distributed to users through the Internet. This paper gives an overview of the files in the archive, including the file formats. All formats are portable and have heritage from the previous missions. From the reformatted files, index information is created for faster data search. Users can perform queries based on information contained in the index. This allows for searches to return observations that conform to particular observing conditions.     

Automated identification of transiting exoplanet candidates in NASA Transiting Exoplanets Survey Satellite (TESS) data with machine learning methods

A novel artificial intelligence (AI) technique that uses machine learning (ML) methodologies combines several algorithms, which were developed by ThetaRay, Inc., is applied to NASA’s Transiting Exoplanets Survey Satellite (TESS) dataset to identify exoplanetary candidates. The AI/ML ThetaRay system is trained initially with Kepler exoplanetary data and validated with confirmed exoplanets before its application to TESS data. Existing and new features of the data, based on various observational parameters, are constructed and used in the AI/ML analysis by employing semi-supervised and unsupervised machine learning techniques. By the application of ThetaRay system to 10,803 light curves of threshold crossing events (TCEs) produced by the TESS mission, obtained from the Mikulski Archive for Space Telescopes, the algorithm yields about 50 targets for further analysis, and we uncover three new exoplanetary candidates by further manual vetting. This study demonstrates for the first time the successful application of the particular combined multiple AI/ML-based methodologies to a large astrophysical dataset for rapid automated classification of TCEs.     

The European HST science data archive

The design of the Data Management Facility (DMF), the system to handle the data from HST, has been based on the concepts of modularity and flexibility, so that changes inevitably due to occur during the long lifetime of the project could be gracefully accomodated. At the Space Telescope European Coordinating Facility (ST-ECF), new constraints have led the archive group to evolve the system, so to allow a more efficient ingesting of the data imported from the Space Telescope Science Institute (STScI), an easier browsing of the HST catalogue of observations, and a more efficient servicing of archive researchers' retrieval requests. In this paper, the European Science Data Archive of HST data is described, with particular reference to dataflow, hardware and software system structure, operations, differences with DMF, and foreseen developments.Affiliated with the Astrophysics Division, Space Science Dept., ESAThe observations of the NASA/ESA Hubble Space Telescope are obtained at the Space Telescope Science Institute, which is operated by AURA, Inc., under NASA contract NAS 5-26555.     

Astrovirtel—From Archival Research to the Astrophysical Virtual Observatory

Given the high cost of modern astronomical observing facilities it is evident that efforts must be made to optimally exploit the data in order to maximize the return on investment. This concept was first implemented on a large scale for the Hubble Space Telescope, and has since been taken over for other space borne and large ground-based facilities. The European HST Science Data Archive is located at the European Southern Observatory (ESO). It has been extended to include data from ESO telescopes and instruments, especially the Very Large Telescope (VLT) and Wide Field Imager (WFI). It was thus natural to design the archive such that queries could be extended across its full content, regardless of the origin of the data. This constituted a first step toward a virtual observatory. The Astrovirtel program, first established in 1999–2000 with funding provided by the European Commission, makes it possible for scientists to use this facility for their investigations. At the same time it allowed us to establish science requirements for archive cross queries, and to define capabilities required for VO's. Recently the European Commission decided to provide the funding for the implementation of the Astrophysical Virtual Observatory (AVO). This will include several European observatories and scientific organizations. It is being developed in close coordination with the US National Virtual Observatory.     

East Asian VLBI Network observations of active galactic nuclei jets:imaging with KaVA+Tianma+Nanshan

The East Asian Very Long Baseline Interferometry(VLBI) Network(EAVN) is a rapidly evolving international VLBI array that is currently promoted under joint efforts among China, Japan and Korea. EAVN aims at forming a joint VLBI Network by combining a large number of radio telescopes distributed over East Asian regions. After the combination of the Korean VLBI Network(KVN) and the VLBI Exploration of Radio Astrometry(VERA) into Ka VA, further expansion with the joint array in East Asia is actively promoted. Here we report the first imaging results(at 22 and 43 GHz) of bright radio sources obtained with Ka VA connected to Tianma 65-m and Nanshan 26-m Radio Telescopes in China. To test the EAVN imaging performance for different sources, we observed four active galactic nuclei(AGN) having different brightness and morphology. As a result, we confirmed that the Tianma 65-m Radio Telescope(TMRT) significantly enhances the overall array sensitivity, a factor of 4 improvement in baseline sensitivity and 2 in image dynamic range compared to the case of Ka VA only. The addition of the Nanshan 26-m Radio Telescope(NSRT) further doubled the east-west angular resolution. With the resulting high-dynamic-range,high-resolution images with EAVN(Ka VA+TMRT+NSRT), various fine-scale structures in our targets,such as the counter-jet in M87, a kink-like morphology of the 3 C 273 jet and the weak emission in other sources are successfully detected. This demonstrates the powerful capability of EAVN to study AGN jets and to achieve other science goals in general. Ongoing expansion of EAVN will further enhance the angular resolution, detection sensitivity and frequency coverage of the network.     

Weakly polarized microwave sources in active regions prior to large X-ray flares

The data set archive of the Siberian Solar Radio Telescope was used to synthesize two-dimensional images of more than 50 active solar regions where X-ray flares occurred. Reasonably bright microwave sources with a relatively low level of circular polarization were found to be present in 2/3 of cases prior (days, hours) to the flare. Such characteristics are normally inherent in sources located above the polarity inversion line of the photospheric magnetic field. This result, which has been obtained for the first time using such a representative data set, confirms the significance of these sources in the context of X-ray flare formation.     

中国频谱射电日像仪FITS-IDI文件格式研究

我国新一代中国频谱射电日像仪（Chinese Spectral Radio Heliograph,CSRH）原始观测数据采用自定义格式,在进行后续处理与共享使用时必须转换相应的格式.在分析FITS-IDI（FITS Interferometry Data Interchange）格式的基础上,结合CSRH的实际观测模式与数据产出方式,定义与设计了符合项目情况的FITS-IDI格式及字段,并对FITS-IDI文件中若干字段的值如何获取、计算进行了深入讨论.根据定义生成的FITS-IDI文件已成功导入CASA软件,并可以进行后续处理.经过对CASA测量集文件的核实,证明了数据生成的正确性.本研究有效地推进了CSRH的建设工作,也对其他射电干涉阵数据存储有一定的参考价值.     

用MapReduce框架构建虚拟天文台数据节点

MapReduce是一种大规模分布式并行处理框架,最初被用于互联网服务中的海量数据处理,并逐渐扩展到各个行业领域。目前,虚拟天文台面临着越来越多的地面及空间望远镜观测到的海量天文数据。为了提高中国虚拟天文台数据节点处理海量天文数据的能力,首次提出基于MapReduce框架构建中国虚拟天文台数据节点的方法,并以批量星表交叉认证为例描述了具体实现过程,性能评估结果证明基于MapReduce框架构建虚拟天文台数据节点,可以在性能、扩展性与成本等多方面获得收益。     

A classification of the available astrophysical data of particular Hii regions

A detailed classification of the available data on W3 is presented. Radio, optical and infrared maps of the object are shown in order of increasing angular resolution. The physical parameters and energy spectra of the various components of W3 and their sources are presented. A summary of the current theories on W3 is also included.     

The SUMER Data in the SOHO Archive

We have released an archive of all observational data of the VUV spectrometer Solar Ultraviolet Measurements of Emitted Radiation (SUMER) on SOHO that have been acquired until now. The operational phase started with ‘first light’ observations on 27 January 1996 and will end in 2014. Future data will be added to the archive when they become available. The archive consists of a set of raw data (Level 0) and a set of data that are processed and calibrated to the best knowledge we have today (Level 1). This communication describes step by step the data acquisition and processing that has been applied in an automated manner to build the archive. It summarizes the expertise and insights into the scientific use of SUMER spectra that has accumulated over the years. It also indicates possibilities for further enhancement of the data quality. With this article we intend to convey our own understanding of the instrument performance to the scientific community and to introduce the new, standard FITS-format database.     

The Astro-WISE optical image pipeline

We have designed and implemented a novel way to process wide-field astronomical data within a distributed environment of hardware resources and humanpower. The system is characterized by integration of archiving, calibration, and post-calibration analysis of data from raw, through intermediate, to final data products. It is a true integration thanks to complete linking of data lineage from the final catalogs back to the raw data. This paper describes the pipeline processing of optical wide-field astronomical data from the WFI (http://www.eso.org/lasilla/instruments/wfi/) and OmegaCAM (http://www.astro-wise.org/~omegacam/) instruments using the Astro-WISE information system (the Astro-WISE Environment or simply AWE). This information system is an environment of hardware resources and humanpower distributed over Europe. AWE is characterized by integration of archiving, data calibration, post-calibration analysis, and archiving of raw, intermediate, and final data products. The true integration enables a complete data processing cycle from the raw data up to the publication of science-ready catalogs. The advantages of this system for very large datasets are in the areas of: survey operations management, quality control, calibration analyses, and massive processing.     

Antenna data storage concept for phased array radio astronomical instruments

Low frequency Radio Astronomy instruments like LOFAR and SKA-LOW use arrays of dipole antennas for the collection of radio signals from the sky. Due to the large number of antennas involved, the total data rate produced by all the antennas is enormous. Storage of the antenna data is both economically and technologically infeasible using the current state of the art storage technology. Therefore, real-time processing of the antenna voltage data using beam forming and correlation is applied to achieve a data reduction throughout the signal chain. However, most science could equally well be performed using an archive of raw antenna voltage data coming straight from the A/D converters instead of capturing and processing the antenna data in real time over and over again. Trends on storage and computing technology make such an approach feasible on a time scale of approximately 10 years. The benefits of such a system approach are more science output and a higher flexibility with respect to the science operations. In this paper we present a radically new system concept for a radio telescope based on storage of raw antenna data. LOFAR is used as an example for such a future instrument.     

Scientific Research using the Electronic Archive of the Hubble Space Telescope

The Hubble Space Telescope (HST), a collaborative project between the US National Aeronautics and Space Administration (NASA) and the European Space Agency (ESA) has been operating since 1990. The European HST Science Data Archive, operated at the Space Telescope European Coordinating Facility(ST-ECF) in Garching, Germany, contains all data taken with the scienceinstruments of the HST. The services of the Archive include data browsing,preview, retrieval and re-calibration. Access to the Archive is providedthrough the World Wide Web, data distribution is through the Internet oron bulk media.This paper describes the data products, access to theArchive services, and the procedures for the use of the HST Science DataArchive by qualified scientists from remote locations. The Archive providestools implemented as Java Applets which can be used over the Net. For moresophisticated operations access to a suitable data analysis system isrequired; such software can be downloaded from our site.     

Deep 610-MHz Giant Metrewave Radio Telescope observations of the Spitzer extragalactic First Look Survey field – I. Observations, data analysis and source catalogue

Observations of the Spitzer extragalactic First Look Survey field taken at 610 MHz with the Giant Metrewave Radio Telescope are presented. Seven individual pointings were observed, covering a total area of ∼4 deg² with a resolution of 5.8 × 4.7 arcsec², PA 60°. The rms noise at the centre of the pointings is between 27 and 30 μJy before correction for the Giant Metrewave Radio Telescope (GMRT) primary beam. The techniques used for data reduction and production of a mosaicked image of the region are described, and the final mosaic, along with a catalogue of 3944 sources detected above ∼5σ, are presented. The survey complements existing radio and infrared data available for this region.     

The Representation of OTA Images’ Astrometric Results with WCS-SIP Coefficientstwo

Cross-matching the sources in the observed image with the star catalogs or earlier records is a necessary procedure for detecting the position or brightness variation of a specific source in the Optical Telescopes Array (OTA) survey. Therefore, knowing the celestial positions of these objects on the observed image is prerequisite. A method based on matrix manipulation is applied to converting the existing plate constants of OTA images into the World Coordinate System (WCS) and Simple Imaging Polynomial (SIP) coefficients, and recording such mapping relations on the FITS (Flexible Image Transport System) files of OTA for the convenience of use. The improvement on the astrometry of OTA resulted by this method, as well as the discussion about some problems of the OTA astrometry are also presented.     

General Architecture On-Board and Ground Segment for the Spectrum-UV Mission Real-Time Section

The general architecture of the on-board and ground segment section for the handling of the scientific real-time operations is presented. The science data flow, from the on-board scientific instrumentation to the raw archive is described, and the interactive loop and the planned ground based available facilities are outlined.