Manual labelling was performed for all quicklook images from the winter season 2019–2020 as well as for January and February 2019; quicklook images have a reduced resolution of 480 × 480 pixels and about a 6 min cadence. The manual labelling aimed to provide a ground truth for developing an automatic classification routine to detect images which contain auroral emission. This dataset contained approximately 37 000 labelled images. From this dataset, only those images that contained auroral emission and that were not dominated by clouds were further used in unsupervised learning. This became approximately 12 000 images.

To prepare the images for unsupervised learning, each image was first processed to remove features that could lead to biases that are unrelated to the auroral morphology, such as a caption indicating the camera type, observatory location, date and time. After removing the caption, images with very faint or barely detectable aurora were removed. In the quicklook images, the colour of each pixel is represented by individual intensities in red, green and blue colours. This is commonly known as the RGB colour space. However, for processing based on brightness, we transform the RGB colours into the HSV and L*a*b* colour spaces, where the brightness and actual colour content are more clearly separated.

In the HSV-colour space, the colours are expressed by the hue (H) and saturation (S), while the brightness is contained in the value (V) . If V was less than 50 out of 255 in more than 90 % of the pixels in the image, the image was discarded as too faint. This left 10 300 images containing sufficiently bright aurora.

The L*a*b*, or CIELAB colour space, aims to represent the colours as a human observer would perceive them . Here, L* denotes the lightness of the colour, while a* and b* provide the colour information. Unlike RGB and HSV, L*a*b* is a device-independent colour space based on a standard human observer and standard outdoor illumination. Similarly to earlier work (e.g. ), we clipped the L* values to the range from the 0.5th to 99.5th percentiles. To reduce the influence of the background sky conditions on the auroral classes, the median values of the a* and b* colour channels were used to provide a neutral white balance for all the images. Next, the images were cropped to 400 × 400 pixels around the center. This removes the biases due to dark corners outside the circular field of view of the all-sky images. This step also removes most of the auroral emission at the lowest elevation angles, which is beneficial as the morphology of the auroral forms near the horizon is heavily distorted by the fish-eye optics and is therefore difficult to examine.

To better take into account faint aurora, a contrast enhancement was performed on the images by equalizing the histograms of the L* channel for each image. Furthermore, a 5 × 5 median filter was used to remove single bright pixels due to stars, similarly to the 3 × 3 filter used by . Finally, the images were resized to 224 × 224 pixels and converted back to RGB, which is the expected format for most feature extractors.

As an example of the effect of the preprocessing steps outlined above, Fig.  shows two original quicklook images (left column) and the corresponding preprocessed images (right column). The images demonstrate the impact of cropping, where aurora close to the horizon is excluded, as well as the effect of the colour enhancement and brightness normalization.

The cadence of 6 min of the quicklook data favours quiet aurora over active aurora. This happens because active aurora occurs in short highly dynamic bursts, which are poorly sampled by a cadence of 6 min, while quiet aurora experiences longer lifetimes. To study the influence of this bias and to better represent the active auroral forms, a second dataset was created. In this dataset, the cadence was reduced to 24 s during selected periods of active aurora. This additional dataset is also referred to as a high-resolution dataset in this work. These selected periods were visually identified as the 19 brightest auroral displays with a large north–south extent as seen in the keograms (sample keogram in Fig. a).

In contrast to supervised learning, where we know the correct answer to a classification question and can train a classifier accordingly, in unsupervised learning we do not know the answer. The manual labelling of auroral images provided us with a dataset of unknown auroral forms. We try to learn the types of aurora in the data by using clustering algorithms. Intuitively, similar auroral shapes in the images should belong to the same cluster.

The first step is to define a numerical measure of the image content in the form of a feature vector. In the second step, the images are clustered based on their feature vectors: here, the true number of clusters is, of course, unverifiable. As clustering is often based on similarity metrics, the results also depend on how well the feature vectors represent the image content. In his Master's thesis, evaluated a number of different feature vector extractors as well as clustering approaches. The following provides a brief description of the central concepts and choices of the methodology.

Classifiers based on convolutional neural networks (CNNs) are becoming increasingly popular in image analysis. Training an artificial neural network refers to the process of optimizing all parameters within the network to minimize the classification error.

The analysis of auroral images in our dataset uses a processing chain which includes a CNN architecture followed by a smaller neural network. We start with the inclusion of high-time-resolution data as they provide a better balance between images of auroral displays during active and quiet times. We use a variation of the SimCLR feature extractor used by . This feature extractor builds on the Resnet-50 neural network architecture. Random rotations and other image transformations are often used in training image classifiers to both artificially increase the number of “different” sample images and to obtain invariance in orientation. In this study, we limit ourselves to horizontal flipping (east–west in auroral images) and random cropping (random parts of the sky with aurora) in the training phase .

We used the UMAP (uniform manifold approximation and projection for dimension reduction) method to reduce the dimensionality of the feature vectors . This step was carried out to improve clustering . In the UMAP reducer, there are two critical parameters: one impacting how much the algorithm focuses on local versus global structures (N_neighbors) and another determining the minimum allowed distance between the features in the dimensionally reduced space (min_dist). Based on the experiments by , we used the values N_neighbors=20 and min_dist=0.

Finally, a clustering algorithm was used to determine clusters of the dimensionally reduced feature vectors. Two different clustering algorithms were used: K-means and hierarchical clustering. These algorithms were used to divide images into 30 or 100 clusters. Consequently, the four clustering results (two clustering algorithms and two numbers of clusters) were fused together to balance the weaknesses and biases of the individual approaches. This also helps to identify outlier images that are connected to different auroral features, depending on the clustering algorithm that is used. The fusion was done by using modified majority voting with a co-association matrix algorithm . The original algorithm was modified to have two tuneable parameters. The first parameter is the majority threshold of the co-association matrix, above which two samples are merged in the same cluster. The second parameter is the minimal cluster size, under which a cluster is merged into a cluster of “outliers” to indicate hard-to-classify images. The values of these parameters were determined by changing them to minimize the loss function 1L=1nc∑k=0N|nk-n0k|+nencwithn0k=Nncifk<nc,otherwisen0k=0, where N is the total number of samples, nk the number of samples in cluster k and ne the number of outlier samples. Parameter nc is used to influence the number of clusters resulting from the fusion and was set to 25. The fusion resulted in 37 clusters with numbers ranging from -1 to 35, where the -1 cluster is for the outlier images not belonging to any of the other clusters.

Using a basic desktop computer (in 2022), the preprocessing, training and clustering required roughly 1 d of computer time. Predicting clusters for additional images takes less than 1 s per image. Table  summarizes the numbers of images at each stage of the data processing.

The CNN analysis was run on the quicklook images with a 6 min cadence alone as well as on the full dataset with the inclusion of higher-resolution images with a 24 s cadence. As a result, we found that the occurrence of some morphological clusters increased greatly with the increasing temporal resolution, while the occurrence rate of the other clusters was nearly unaffected.

The high sensitivity to the increased temporal resolution in the image data (at least about an 80 % change, red vertical line in Fig. ) suggests that these clusters primarily describe the morphology of active auroral displays. These are the cluster numbers 0, 13, 26, 8, 21, 7, 14 and 35. The occurrence change of the individual clusters is illustrated by Fig. . It shows the occurrence rate difference of all 37 clusters between quicklook and high-resolution data in percentage with respect to the total occurrence rate of each cluster. The clusters, for which the occurrence showed little or no change (maximum of about 20 % change, blue vertical line in Fig. ) with the increasing temporal resolution (cluster numbers 20, 4, 9, 17, 34, 22, 32, 10, 3, 18, 25, 1 and 29), are likely to contain auroral structures mainly related to quiet auroral displays. By quiet auroral displays, we refer to images of structures which are relatively simple, are dim and evolve slowly in time. A typical example of this is an auroral arc, or a multiple arc, which changes little between consecutive images with a higher temporal resolution of 24 s. A set of 10 randomly selected images from each of the individual clusters included in quiet aurora is shown in Fig. . In addition to the simple green auroral structures, daytime and afternoon overhead aurora with a notable red emission contribution belongs to this category.

Active auroral displays, in turn, refer to images with one or multiple more complex auroral structures, which evolve significantly between consecutive images in our higher-temporal-resolution dataset and have little resemblance between consecutive images at quicklook time resolution (6 min).

A random selection of images in each individual cluster included in active aurora is displayed in Fig. . These active auroral displays typically include bright emissions in vortical, rayed and small-scale structures. The threshold values for the occurrence rate changes for quiet and active aurora (∼ 20 % and ∼ 80 %, respectively) are based on visual inspection of the cluster occurrence rate histogram in Fig.  and are therefore somewhat arbitrary. Note that cluster number 1 is included in the quiet aurora, although its change rate is just over 20 %. Similarly, cluster number 8 is included in the active aurora, although its change rate is just under 80 %. These choices were made with the idea that a large percentage difference is a conservative choice as long as it results in several clusters in each extreme.

From image samples in Figs.  and it is obvious that a detailed description of the type of aurora in each individual cluster is difficult to determine. Simple properties, such as the brightness, contrast, colour, alignment or location of the aurora in the image, seem to play a role in the numerical clustering. For both the quiet and active aurora categories, the images containing the Moon have been collected in the same clusters. Any more detailed interpretation of the individual clusters will require more analysis.

An example of the time evolution of auroral all-sky image data with the detected active and quiet auroral displays is provided by Fig. a, which is a full keogram of 25 January 2019, and Fig. b, which shows the time evolution of individual images of active (red dots) and quiet (blue dots) auroral displays. The images of active and quiet aurora are most frequent in the time periods of clear skies and aurora, because our preprocessing excludes clear skies with no aurora and images with clouds. Active aurora coincides with the brightest aurora in the keograms, as expected. Figure b shows ground-magnetic data (X component) from the Svalbard magnetometer stations in Hornsund (HOR, 77.00°Glat), Longyearbyen (LYR, 78.20°Glat) and Ny-Ålesund (NAL, 78.92°Glat). These stations are part of the IMAGE magnetometer network (https://space.fmi.fi/image/www/index.php, last access: 19 April 2024) . The magnetic variations are measured and plotted at 10 s time resolution. The three periods of active aurora correspond to small (∼ 100 nT) to moderate (∼ 400 nT) magnetic deflections in the north–south component. They could all be interpreted as substorms, the first of which expands towards Svalbard from further south, while the latter two events occur more directly overhead. The visual correspondence between the periods of active aurora and the magnetic deflections during the latter two substorms appears particularly good.

To further investigate these different morphological cluster groups of “active” and “quiet” aurora, we selected time periods of active or quiet aurora that included three or more consecutive images with a maximum cadence of 6 min. For a sequence longer than three consecutive images, one single quiet-class image would be allowed within an active aurora period, or the other way round. This procedure resulted in 28 active aurora events and 222 quiet aurora events in the time frame of all our labelled data in 2019–2020.

Table  gives an overview of the typical parameters for periods of active and quiet aurora. The duration of the active aurora events varied from 60 s to 5 h, while that of the quiet events ranged from 2 min to 25 h. The median values (row 2 in the table), however, do not differ much. The active aurora events typically started at magnetic midnight (row 3), while the quiet aurora events were spread more evenly over the magnetic local time (MLT) hours, which is illustrated by Fig. . The median start time is therefore not a good indicator of quiet aurora, which experiences three occurrence maxima in 24 h: right after midnight, at about 06:00–09:00 MLT and at about 15:00–19:00 MLT.

The geomagnetic indices with coarser spatial resolution (a low number of stations, which provide the data) are not dramatically different between the periods of active and quiet aurora. These are the Dst index (row 4) and the Hp30 index (row 6). Neither of these indices with low spatial resolution includes high-latitude magnetic field data, and therefore they do not reflect the high-latitude magnetic deflections, which may have different timings compared to the corresponding deflections at lower latitudes. However, the SML electrojet index (row 5), which has good spatial coverage of magnetic data from a range of different latitudes, shows a difference of more than 100 nT between active and quiet aurora. Similarly to the low-resolution magnetic indices, the median solar wind speed (row 7) is more enhanced during active aurora than quiet aurora, but only by some tens of kilometers per second, which is not significant.

We performed a visual inspection of what the active aurora periods correspond to by using keograms (as in Fig. a). The main mission of the validation procedure was to find out what the detected events looked like, whether they corresponded to magnetic substorm activity, whether keogram inspection could reveal missed auroral activations, and, if yes, what kind of events they would be. Keograms were therefore viewed for each day from January 2019 until March 2020. We inspected all detected active aurora events as well as each keogram for any additional auroral brightening, which was not detected.

Most active aurora periods were auroral brightenings for which the bright emission extended over the full (or nearly full) field of view of the all-sky image, as illustrated by the keogram in Fig. a. Most of the active aurora periods also coincided with a small to moderate deflection of the local magnetic north–south (X) component, which indicates substorm activity. The maximum magnetic deflections for the active aurora events ranged from about 50 nT up to about 400 nT, with a median value of 100–200 nT. Cluster numbers of individual images were also investigated during the active aurora periods, but no consistent time evolution or preferential order was found among the individual active aurora clusters.

Taking the auroral breakup overhead as a benchmark, five false-positive events were identified in the visual inspection. Three of our false-positive cases were morning sector auroral events (see the lonely blue bars in Fig. ). These consisted of transient brightenings in the morning sector diffuse and pulsating aurora. Two of these events are from the shortest end of the duration spectrum containing only two consecutive images. This type of event could be filtered out simply by increasing the minimum duration of the active aurora events in order to produce a cleaner auroral breakup list. Two of the other events we call false-positive cases were clearly substorm-like aurora which extended about halfway in the sky towards the local zenith, such as the first active aurora event in Fig. . These types of events are not exactly wrongly detected but are rather a milder category compared to the rest of the cases. They may include substorms, which inherently do not take place at high latitudes but rather expand towards Svalbard from further south, where the strongest magnetic disturbances take place.

As for false negatives, i.e. any additional auroral brightenings that the active aurora periods did not cover, no events were found. Based on keograms, we visually identified some auroral brightenings that nearly extended across the full sky but that were not automatically identified as active aurora. These included (1) events where the sky was fully or partly cloudy, so that the auroral structures were obscured; (2) events where the image was contaminated by moonlight, so that the auroral brightness (and therefore the structure) was not very visible; and (3) events which were substorm-like auroral activations that did not even reach halfway toward the local zenith. Images in the first two categories would have been excluded from the data fed to the classifier due to the clouds or the moonlight. In the last category, the largest magnetic deflections occurred typically further south of Svalbard, which was confirmed by visual examination of the IMAGE magnetometer data.

Figure  shows snapshots of the ionospheric equivalent currents, derived from 10 s IMAGE magnetometer data , on 25 January 2019 around a period of active aurora in Fig. . The plots show an equivalent current pattern typical for the westward traveling surge (WTS) (e.g. ) sweeping past LYR from southeast to northwest. Two minutes before the period of active aurora (18:36:45 UT), the U-shaped head of the surge pattern was still located south and east of the LYR field of view (indicated by the black circle). The surge head appears to reach LYR at the time the active aurora started (18:38:45 UT) and to disappear around the time the active aurora ended (19:19:27 UT). A similar sequence of equivalent current dynamics (not shown) can be seen around the next interval of active aurora (Fig. ) between 20:18:14 and 20:41:04 UT. We have also examined several other active aurora events and found similar equivalent current development. This indicates that periods of active aurora may be associated with a WTS which originated earlier at a substorm onset location just east and south of LYR and which traveled westward and northward past LYR.

Figure  shows the ASC image of the surge head aurora as it comes into the ASC field of view, which corresponds to the strongest ionospheric equivalent currents at 18:59:20 UT (Fig. ). Here the image is plotted onto geographic coordinates to make it more comparable to the coordinates of the equivalent current maps (grey grid). Characteristics of the WTS aurora are the intense auroral emission and vortical structures located within the region of an upward field-aligned current (blue colour, negative external jZ). The auroral evolution leading to this included poleward expansion from the southern edge of the field of view as well as more localized brightenings propagating from east to west along the poleward boundary of the surge aurora. This evolution is in agreement with that of the equivalent currents as described above.