1. Introduction
Time domain astronomy is a data-intensive frontier that opens new avenues for understanding the dynamic universe. Large-scale surveys, such as the Zwicky Transient Facility (ZTF), Rubin Observatory Legacy Survey of Space and Time (LSST) and emerging projects in China including the Wide Field Survey Telescope (WFST) and SiTian, systematically detect and monitor variable sources across timescales from millisecond transients to long-period variable stars, generating large volumes of light curve data that exceed manual analysis capabilities.
Machine learning has been extensively adopted to address data analysis challenges. For example, the ALeRCE alert broker system employs a balanced random forest classifier with engineered features to classify ZTF light curves (Förster et al. Reference Förster, Cabrera-Vives and Castillo-Navarrete2021). However, supervised learning approaches require substantial labeled training datasets and these approaches remain largely task-specific, limiting their broader applicability across diverse task contexts. Recent advances in self-supervised learning represent a promising paradigm shift toward more adaptable and generalisable solutions. ASTROMER (Donoso-Oliva et al. Reference Donoso-Oliva, Becker and Protopapas2023), a self-supervised light curve embedding method, demonstrates superior accuracy over specialized LSTM networks in classification tasks under data-limited scenarios through pre-training on large unlabeled datasets followed by task-specific fine-tuning. Similarly, Pan et al. (Reference Pan, Ting and Huang2024) show that self-supervised pretraining approaches enhance stellar parameter inference with improved sample efficiency compared to supervised methods.
Motivated by the need for unified solutions to time-domain data processing challenges, we introduce FALCO, a foundation model of astronomical light curves for time-domain astronomy. Unlike specialised models targeting individual tasks, FALCO leverages self-supervised learning on diverse astronomical time-series to develop generalisable representations applicable across multiple astrophysical tasks. This paper presents the conceptual framework of FALCO and demonstrates an initial model implementation using Kepler data, establishing a foundational basis for diverse light curve analysis tasks.
2. FALCO: A foundation model approach
FALCO is designed as a foundation model to address the fundamental challenge of developing unified, scalable solutions for diverse time-domain astronomy tasks. The conceptual framework, illustrated in Fig. 1, establishes a paradigm where a pre-trained model can be adapted to multiple downstream tasks including parameter estimation, variable classification, and transient identification.
The conceptual framework of FALCO.
To build such a foundation model, there are several key problems that need to be addressed. Establishing a generalizable representation that can be used in various light curve analysis tasks is the key point. In addition, when extending to data from different surveys, the heterogeneity in telescope instrumentation and sampling frequencies introduces additional complexity to the data, demanding more sophisticated modeling approaches. Without implementing unified data preprocessing and standardization procedures, the model should inherently accommodate both irregular temporal sampling and instrumental variations across different observational systems.
Given the computational demands of training large-scale foundation models, our initial implementation uses high-quality Kepler mission data for proof-of-concept development. This implementation encompasses multi-task evaluation across representative astronomical applications, performance benchmarking against specialized task-specific approaches, thereby demonstrating FALCO’s effectiveness while providing a foundation for more comprehensive deployments.
3. Model implementation
Transformer-based models have been successfully employed to develop foundation models for time-series, exhibiting outstanding performance across a range of tasks, including forecasting, classification, and anomaly detection. The initial implementation of FALCO adopts a decoder-only transformer architecture based on GPT-2, chosen for its proven autoregressive capabilities and natural compatibility with sequential data, following the practices established by Pan et al. (Reference Pan, Ting and Huang2024) for light curves. The implementation utilises the nanoGPT frameworkFootnote † for efficient development.
The model structure follows the standard GPT-2 architecture with token and positional embeddings, followed by stacked Transformer blocks. Each block comprises multi-head self-attention, position-wise feed-forward networks, layer normalisation, and residual connections. The final Transformer block outputs are processed through a linear projection layer with softmax activation for next-token prediction. To adapt GPT-2 for continuous light curve data, we replace the discrete token embedding layer with learnable multilayer perceptron (MLP) layer that encode photometric measurements. We employ four model sizes with 85M, 308M, 708M, and 1.47B parameters, enabling systematic evaluation of model scale effects on astronomical tasks.
Pre-training lays the groundwork for foundation model. Our initial implementation of the model was trained on approximately 130,000 Kepler light curves with a cadence of approximately 29.4 minutes. The light curves undergo Median Absolute Deviation (MAD) normalisation and are segmented by quarter gaps to preserve temporal coherence. The normalised light curves are partitioned into varying sequence lengths (each containing 20, 30, 50, 100, 200 data points) for an evaluation of the input length effects on prediction performance. Individual photometric measurements are tokenised as point-wise inputs for sequence modeling. The pretraining employs next-point prediction, analogous to language modeling in LLMs but adapted for continuous photometric sequences using Huber loss instead of cross-entropy for robust regression.
4. Scientific use cases
We applied the model in three light curve analysis tasks: surface gravity inference, variable source classification and stellar flare identification. To validate the effectiveness and generalisability of FALCO on different tasks, we employ the model to encode light curve data and the output representations are used to build task-specific MLPs. In all the tasks, we found that increased model parameter scales and input sequence lengths resulted in improved performance. Subsequently, we report the performance obtained using the 1.47B-parameter FALCO model with 200-point input sequences.
4.1. Surface gravity inference
The inference of surface gravity (
$\log g$
) constitutes a fundamental task in stellar parameter determination. Estimating
$\log g$
from photometric variability capitalises on granulation and oscillation signals encoded within light curves. For evaluating FALCO on surface gravity inference task, we employ a comprehensive dataset encompassing red clump stars, red giants, and main sequence stars across the full parameter range, with corresponding
$\log g$
values derived from asteroseismic analysis (Sayeed et al. Reference Sayeed, Huber, Wheeler and Ness2021). The representation of the FALCO model demonstrates outstanding performance, achieving an overall root mean square error (RMSE) of 0.1305 dex with only 6,000 training samples, with significantly improved precision of RMSE
$<$
0.08 dex at low gravity end where log
$g < 1$
, and approximately 0.02 dex near log
$g \approx 3$
.
4.2. Variable source classification
The classification of variable sources represents an important task for large time-domain surveys, presenting challenges arising from heterogeneous variability patterns and limited labelled examples of rare astronomical classes. In this study, we evaluate the classification problem across eight distinct classes: aperiodic stars, contact binaries and rotational variables,
$\delta$
Scuti and
$\beta$
Cephei stars, eclipsing binaries,
$\gamma$
Doradus and slowly pulsating B-type stars, RR Lyrae and Cepheid variables, solar-like oscillators and non-variable objects. In particular, we employed a dataset combining a curated Kepler Q9 data developed for an ensemble meta-classifier (Audenaert et al. Reference Audenaert, Kuszlewicz and Handberg2021) with a non-variable class identified in Paunzen et al. (Reference Paunzen, Binder, Cyniburk and Duffek2024), yielding a total of 8,523 labelled light curves. The FALCO embedding, when employed with a three-layer MLP, achieved an overall classification accuracy of 95% across the eight defined classes.
4.3. Stellar flare identification
The identification of stellar flares in light-curve segments presents a highly imbalanced binary classification challenge essential to stellar activity studies. We constructed 200-point segments around flare events validated by Yang and Liu (Reference Yang and Liu2019), wherein flare-positive samples comprised fewer than 3% of the dataset, reflecting the natural rarity of these transient phenomena. A simple MLP classifier applied to FALCO representation achieves 87% precision on the held-out test set despite severe class imbalance.
5. Conclusion and Future Works
We propose FALCO (Foundation model of Astronomical Light Curves for time dOmain astronomy) to address the challenges in analysing large-scale light curves from time domain surveys. The initial version of FALCO trained with Kepler data achieves excellent performance for multiple astronomical tasks such as stellar surface gravity (
$\log g$
) inference, variable source classification and stellar flare identification, while markedly reducing the required amount of task-specific labelled training data.
The next step would be to adapt the model to handle irregularly sampled data from multiple surveys. We also plan to explore more science applications, to support comprehensive analysis of astronomical light curves, opening new pathways for understanding the dynamic universe.
