Ionosondes measure the time delay of vertically propagated Medium Frequency (MF) and HF radio waves (the combined MF-HF range spans from 0.3 to 30 MHz), which reflect off of the bottomside of the ionosphere. The transmitter sweeps through a subset of the MF and HF range to obtain information about the electron density at multiple altitudes. To analyze ionosonde data, the frequency-altitude output must be scaled and inverted from delay as a function of frequency to electron density as a function of altitude. This can be done using an auto-scaler program or performed manually by an experienced analyst. The data for these validation studies was collected from the Lowell GIRO Data Center (LGDC), which collects data from ionosondes around the world . The majority of the validation data in this study has been scaled using versions 4, 4.5, and 5 of an autoscaler called Automatic Real-Time Ionogram Scaler with True height (ARTIST), although some of the outputs from LGDC indicate that the scaling method used is unknown . These were likely scaled by other autoscalers used by different institutions. The ARTIST outputs do not include errors in the measurements but they do include a confidence score that can be used to determine if the profile is usable.

Using hand-scaled ionograms is generally considered preferable to using autoscaled ionograms, since the results have been verified by a knowledgeable data processor. However, it can be very time consuming and the results may not be available for a real-time operational validation. discusses the challenges and best practices for using ARTIST autoscaled parameters. They describe how to use the autoscale scores and conclude that it varies depending on parameter and is not as simple as using a cutoff value. Based off this analysis, we use a cutoff of 70 for both foF₂ and hmF₂ to use the same times for both analyses. In they discuss the foF₂ error for three sites by comparing the foF₂ to hand scaled ionosondes. They found that 95 % of automatically determined foF₂ from ARTIST 5 fell within -0.3 to 0.4 MHz of the hand scaled profiles. The hand scaled profiles also have errors associated with them that are harder to quantify. Only auto-scaled ionosondes were used in this study to replicate what would done in a real-time verification. Since the autoscale confidence score does not catch all issues in scaling, it is likely that poorly scaled ionosonde data were included in the data validation, despite efforts to ensure a clean data set. The ingested data uses a different error analysis and may also include poorly scaled data.

Ionosondes are relatively inexpensive instruments to install and operate, making them a popular source for ionospheric measurements across the globe. Figure  shows the locations of the sites used in this study for both assimilation and validation. Different time periods are used in this study, so different ionosondes are used for assimilation and validation for each run. For each model run, not every ingested site and not every validation site from the map is used. Sites that are used for assimilation and ingestion were ingested for at least one of the runs and also used to validate at least one run but for any given time period they may not have been used for both ingestion and validation. Additionally, since the assimilated data are processed differently from the validation data, the ingested data is not identical to the validation data at these sites that are used for both. When specific ionosonde sites are used for validation, they are identified by station name and location. Another complication for this validation effort is that some of these ionosondes are also used in NIMO's data assimilation. The NOAA Mirrion 2 ionosonde database includes some of these ionosondes, but not all of them. This database also includes ionosondes that are not included in the validation. Although the ingestion process does remove some of the provided data to prevent the dominance of one data set on the assimilation, this information is not available post-processing. Due to this complication, instances where ionosonde data is or is not assimilated are addressed on a case-by-case basis and the regional statistics do not make a distinction between these two possible states. Additionally, the distribution of ionosonde locations means global statistics will be biased towards mid-latitude results.

An Incoherent Scatter Radar (ISR) probes the ionosphere through very weak stochastic collective Thomson backscatter from thermal electrons in the ionosphere, electrostatically modified by the presence of ions . The very weak nature of the scatter allows full altitude profiles of plasma parameters into the topside ionosphere. The ISR systems need to be large and powerful , since the electron radar scattering cross-section is roughly 10-28 m² and the plasma in the ionosphere is at least 90–100 km away. There are only six ISR systems in the American sector, located in Alaska, Nunavut (Canada), Massachusetts, Puerto Rico, and Peru. This study uses the available ISR data, which comes from the Millstone Hill (MH), Arecibo Observatory (AO), and Jicamarca Radio Observatory (JRO) ISRs.

The MH ISR is operated as part of the Millstone Hill Geospace Facility at the Haystack Observatory, a sub-auroral/mid-latitude location (42.62° N, -71.95° W). This ISR transmits 2.5 MW maximum peak power at Ultra-High Frequencies (UHFs, ranging from 300 MHz to 1 GHz), and has a wide field of view with a 46 m fully steerable antenna and a 68 m zenith/vertical antenna. The data presented in this document were acquired with the fixed zenith antenna using an interleaved combination of uncoded and alternating code waveforms, and yields a changing altitude resolution from 4.5 km at the E region to less than 60 km at an altitude of 400 km. MH data uses a radar calibration constant to yield absolute electron density values; two different methods of obtaining electron density profiles were used for the experiments in this study. Before 2015, MH “ion line” observations, yielding full altitude plasma parameters including electron density, were calibrated against F₂ region peak electron density values from the co-located UM Lowell digisonde (station code MHJ45) at the observatory. After 2015, MH “ion line” observations were self-calibrated to simultaneous extremely weak but precise F₂ peak altitude Langmuir mode/“plasma line” observations, which by their nature produce absolute electron density referenced to fundamental physical constants.

The AO is located in Puerto Rico at (18.3° N, 66.7° W), and is considered a low-latitude site. It had the largest single dish ISR until 2019, when it ceased operations. The ISR instrument consisted of two 430 MHz radar antennas that could be used together or independently, in addition to other antennas that transmitted different frequencies used for radio astronomy, planetary, and HF studies. The Arecibo ISR had the highest sensitivity among all ISRs, with range resolution of 150 m, time resolution of the order of ms, and frequency resolution of 0.7 kHz. The data in this study uses the calibrated ion line to validate the NIMO simulations.

The JRO is located in Peru at (11.9° S, 76.9° W), under the magnetic equator, with a magnetic dip angle around 1°. The JRO ISR is an array of 18 432 crossed-dipole 50 MHz antennas, covering an area of 85 000 m², that transmits up to 4.5 MW of power. The JRO ISR dual-polarization makes it possible to measure the Faraday rotation of the incoherent backscatter waves, which allows an absolute estimate of electron density independent of any other instrumentation . The range resolution of the data is set by the radar transmitted codes . JRO uses alternate uncoded pulses that minimize the clutter at F region heights and provide a height resolution of 15 km.

TEC is measured by satellite-based transmitters paired with either ground- or space-based receivers that measure the time delay between signals at two different frequencies. The TEC provides a column integrated density measurement rather than vertically resolved electron density profiles, but the high accuracy and vast coverage of this data set make it, arguably, the most complete ionospheric specification data set currently available.

The TEC measured by distributed ground- and space-based receivers and transmitters onboard the GNSS satellite constellation provide line-of-site, or slant TEC (STEC), measurements of the density along the path between the transmitter and receiver. This path typically covers a range of latitude and longitudes, and so is frequently converted to vertical TEC (which projects the STEC vertically using assumptions about the ionospheric structure). The Massachusetts Institute of Technology (MIT) Haystack Madrigal database provides two TEC products as part of Millstone Hill Geospace Facility TEC analysis operations: the STEC and unbinned vertical TEC as well as vertical TEC binned at a resolution of 5 min, 1° latitude and 1° longitude . The sources of uncertainties and biases in the TEC processing are reported in and , and the gridded vertical TEC files report data errors for each gridded value. These errors typically range between 0.1 and 1.0 TEC Unit (TECU) (where 1 TECU = 10¹⁶ electrons m⁻²) over the validation region. Figure  shows the typical coverage of the vertical TEC data with the area used for validation marked by a green box.

As was the case with the ionosonde data, TEC was also ingested into the NIMO data assimilation. However, in this instance the assimilated TEC was obtained from a different source and does not retain any information about the receiver bias calibration, as the relative TEC (not the total slant or vertical TEC) is assimilated. Thus, the validation data set will be distinct from the assimilated data set, even if the same satellite-receiver pairs are used, especially when considering statistics such as the bias.

The JASON satellites are part of a mission to supply scientific and commercial data about sea level rise, ocean temperatures and circulation, and climate change. The JASON-3 satellite launched in 2016 and is still operational, and so covers many of the validation periods shown in Table . The validation periods in 2014 are covered by JASON-2, which was operational between 20 June 2008 and 9 October 2019. These satellites include dual-frequency altimeters, operating at 13.575 GHz (Ku-band) and 5.3 GHz (C-band), to measure the height of the ocean surface to high accuracy. The JASON satellites fly in an orbit with a 66° inclination and a 10 d repeating reference orbit, advancing approximately 2° d⁻¹.

Corrections must be applied to these measurements due to the dispersive nature of the atmosphere that results in path delay of the radar signal. The ionospheric correction, or delay, is directly proportional to the electron content along the ray path and inversely proportional to the frequency (f) squared of the signal. The difference in delay between the altimeters’ dual-frequency measurements can be used to calculate the TEC in the nadir (vertical) direction from the spacecraft at 1340 km altitude to the surface over the oceans . TEC is calculated using the following formula: 1TEC=-dR×f240.3

In Equation , TEC is the vertical TEC in the ionosphere measured in TECU, dR is the Ku-band ionospheric range correction in meters provided in the JASON-2 and JASON-3 Geophysical Data Records (GDRs) that are available at, for example, https://www.ncei.noaa.gov/data/oceans/jason3/gdr/gdr/ (last access: 7 October 2025) . The sampling rate of the JASON-2 and JASON-3 instruments is 1 Hz. However, as recommended by and the JASON-3 Handbook , the ionospheric range correction should be smoothed over 100 km or more to reduce instrument noise. To calculate the JASON-2 and JASON-3 TEC used in this study, we have averaged the measurements over 18 s, which yields approximately ∼ 1° bins. The σN precision is ∼ 1 TECU. Because both JASON-2 and JASON-3 satellites are used in this validation study, we have been careful to take the bias between the instruments into account. During the tandem period (February–October 2016), JASON-3 was inserted into the same orbit as JASON-2 and trailing by one minute. An evaluation of the TEC measurements during this time period shows that the JASON-2 instrument has a +2.6 TECU bias with respect to the JASON-3 instrument. For this study, we have subtracted 2.6 TECU from the JASON-2 measurements to account for this bias.

While JASON-3 does not provide a dense set of measurements, it does provide a direct TEC measurement over bodies of water. Altimeter data, such as this, has been used extensively to validate TEC models and other measurement techniques e.g.,.

The Coupled Ion-Neutral Dynamics Investigation (CINDI) mission operated onboard the Communication/Navigation Outage Forecasting System (C/NOFS) satellite  between 2 August 2008 and 26 November 2015. The C/NOFS satellite was launched into a low earth orbit with an inclination of 13° and an orbital period of about 97 min. Initially, the satellite orbit had a perigee of 400 km and an apogee near 860 km. The mission was placed in safety mode (not providing any data) between 5 June 2013 and 22 October 2013, before becoming operational again (at this point the C/NOFS satellite had a perigee of 388 km and apogee of 690 km). This orbit allowed the orbital plane to precess 24 h in local time over a period of 3 months.

Two instruments from the CINDI mission are used in the data validation efforts: the Retarding Potential Analyzer (RPA) and the Ion Drift Meter (IDM) . Together known as the Ion Velocity Meter (IVM), these instruments provided the 3D ion velocity, converted into magnetic coordinates using the International Geomagnetic Reference Field (IGRF) , as well as the ion density, temperature, and composition. The pysat data framework includes cleaning routines for these instruments, ensuring a high quality of data when performing validations . The orbital characteristics of the C/NOFS satellite allow CINDI observations to validate model runs in the topside, equatorial ionosphere. The data files were retrieved from the NASA Space Physics Data Facility .

The DMSP Special Sensor for Ions, Electrons, and Scintillation (SSIES), an earlier incarnation of the CINDI IVM, was flown onboard four of the DMSP satellites (F15, F16, F17, and F18) as a part of their mission is to monitor the meteorological, oceanographic, and geospace environment . The first DMSP satellite was launched in 1962, and there are currently several satellites operational and in their desired orbits. These orbits are sun-synchronous, polar orbits with altitudes near 830 km. Each satellite crosses the equatorial plane at a different Solar Local Time (SLT), allowing DMSP to validate the topside ionospheric density at a range of latitudes and local times. DMSP SSIES data (Level 1 processing) was processed by Boston College and retrieved from the CEDAR Madrigal database .

The Ionospheric CONnection Explorer (ICON) IVM suite measured the in situ ionospheric drift using RPAs and IDMs at the front (A) and rear (B) of the spacecraft. For the periods used in this validation study, only IVM-A returned observations. The IVM measures the ion density, temperature, 3D drift, and composition. The ion density, which is the density for all species, may be directly compared to the model's electron density as there are only significant populations of singly-ionized species in the topside ionosphere. The data files were retrieved from the NASA SPDF .

A standard way of visualizing the agreement between two data sets is to plot a histogram of the differences between the two data sets. If there is no systematic offset between the two data sets and any disagreement is randomly distributed, the histogram will have a classic Gaussian bell shape centered about zero. The offset of the peak, pronouncement of the tails, and symmetry of the distribution at the half-maximum point can all be easily assessed using a difference histogram.

The Bias (B) or the Mean Error (ME) is a scale-dependent bias, measured by differencing the means of two data sets, as shown in Eq. (). As previously stated, the subscript mod refers to model data and the subscript obs refers to observational data. X acts as a placeholder for the different observations used in the validation efforts, and 〈X〉 denotes the mean of the enclosed data set. B and ME are the same statistic. 2B=〈Xmod〉-〈Xobs〉

The Root Mean Squared Error (RMSE) provides a measure of the magnitude of the difference between two data sets. It is defined as: 3RMSE=〈Xmod-Xobs2〉.

The Mean Absolute Error (MAE) is another way to measure the magnitude of the difference between two data sets. Described in Eq. (), it is simply the mean of the absolute value of the difference between the modeled and observed data sets. 4MAE=〈|Xmod-Xobs|〉

The Median Absolute Error (Med. AE) is a more robust measure of the magnitude of the difference between two data sets. It is calculated in the same way the MAE is, but instead of taking the mean of the absolute value of the differences between the modeled and observational data sets, the median is calculated. This makes the result less susceptible to outliers, but it does not measure the distribution of differences between the data pairs.

The Mean Absolute Percentage Error (MAPE) is a way to measure the magnitude of the difference between two data sets. Described in Eq. (), it is simply the mean of the absolute value of the difference between the modeled and observed data sets, expressed as a percentage. 5MAPE=|Xmod-Xobs|Xobs×100%

Correlation coefficients are a way to show how strongly two sets of data are related. The Pearson correlation coefficient (r1) measures the strength of the linear correlation between two sets of data by determining the ratio between the covariance of two variables and the product of their standard deviation. Positive values indicate a strong association among the two sets, negative values indicate an anti-correlation, and values equal to zero indicate no relationship at all. The correlation of a set of data with itself gives an r1 of unity. The Pearson correlation coefficient equation between the modeled and observed data sets is calculated using: 6r1=∑iN(Xmodi-〈Xmod〉)(Xobsi-〈Xobs〉)∑iN(Xmodi-〈Xmod〉)2∑iN(Xobsi-〈Xobs〉)2. In the above equation, N is the number of samples and i is the index of an individual model-observation pairing.

Correlation and anti-correlation can be differentiated together from the absence of any correlation by using the square of the Pearson correlation coefficient (R2). R2 is simply the square of r1, and is useful for separating data sets that have a relationship from those that have no relationship at all.

Meta analysis is commonly performed to obtain a clear picture of a large number of statistics. This study uses meta analysis to clarify the comparative model performance of one particular statistic. More specifically, when evaluating the performance of one of the statistics presented above (S), the percentage of validation runs where NIMO outperforms IRI-2016 (PS) can be determined using: 7PS=∑iNci,NIMO,IRI-2016N×100%, where: 8ci,NIMO,IRI-2016=1,ifSiNIMOisbetterthanSiIRI-20160,otherwise.

In the equations above N is the number of validation runs. The qualifier “better than” is used instead of a numerical description because the numerical condition that indicates a “better” value changes. For example, the RMSE is better when the value is lower, but the B is better when its absolute value is closer to zero.

While ionosondes provide a wealth of data that can be used to validate several different aspects of the bottomside ionosphere, this study uses only the foF₂ and the hmF₂. This choice was made to ensure that the most reliable aspects of the auto-scaled ionosonde data were used in the validation process. Focusing on the critical frequency and height of the F₂ peak has the additional advantage of validating the location and strength of a major ionospheric feature.

Figure  shows the observed and modeled foF₂ for a day at two of the ionosonde sites; the top panel shows data that was included in the data assimilation (Boulder, USA) and the bottom shows data that was not assimilated (Beijing, China). Ionosonde measurements are shown as red dots, the NIMO output is marked by a blue line and triangles and the orange line and squares shows the IRI-2016 output. NIMO shows significantly more variation than IRI-2016, as it is not a climatological model. Additionally, the top panel shows a solar quiet period, while the bottom panel shows a solar active period. This illustrates how NIMO is typically more successful than IRI-2016 at capturing the critical frequency (and therefore density) at the F₂ peak, whether or not data was assimilated at this location.

However, it is more challenging to capture the correct hmF₂. Figure  shows the same time periods and stations as Fig. , but for the hmF₂. This demonstrates that the IDA-4D data assimilation process is also blindly successful, meaning that the model will attempt to drive the ionospheric state towards incorrect assimilated values and create unrealistic modeled output. At 01:30 UT the model fits a very high and unrealistic hmF₂. The error in the EDP did match the true value of the error that was by determined by looking at the ionogram. All ionosonde values, even those with low confidence scores, are plotted here to show how the IDA-4D assimilation uses the data.

The B, RMSE, and MAE (see Sect. , , and ) are used to compare ionosonde observations with NIMO and IRI-2016 outputs. The global results from all available ionosondes are shown in Tables  and and plotted in Figs.  and for the desired validation periods. These statistics are calculated for all local times, daytime hours (06:00–18:00 SLT), and nighttime hours (18:00–06:00 SLT). Although these local times do contain some overlap of sunlit and non-sunlit locations (as solar zenith angle is not accounted for), they are each dominated by either day- or night-time processes. A subset of these runs was used to evaluate the foF₂ using only ionosondes not used in the data assimilation process (see Table ).

The B shows that both NIMO and IRI-2016 overestimate foF₂, while NIMO is more likely to overestimate the hmF₂. In general, IRI-2016 tends to have a lower B than NIMO. Recall that a low bias may indicate a good performance or that a model is equally likely to overestimate as it is to underestimate for the chosen evaluation period. The interpretation that NIMO tends to overestimate the foF₂ and hmF₂, while IRI-2016 overestimates peaks and underestimates minima for these values is supported by the examples shown in Figs.  and .

Examining the other foF₂ statistics further supports this interpretation. Calculating PMAE for all validation runs and local time periods shows that NIMO performs better than IRI-2016 in 85.2 % of the cases. Similarly, PRMSE=74.1%. When considering the potential impact of removing the comparison with ionosondes used in the data assimilation, Table  shows that both NIMO and IRI-2016 have higher biases and errors for these four runs. However, the only time there is a change in relative performance between the two models is the July 2014 MAE, which is now lower for IRI-2016.

A challenge when using ionosondes for validation is that the errors from the ionosondes are not well defined, as previously discussed in Sect. . This impacts both the NIMO output, due to the ingested ionosondes, and the foF₂ and hmF₂ validation with the ionosondes. Since this NIMO validation uses many ionosonde sites over long periods of time, it is likely that poorly-behaved ionosondes are included in the analysis. This will increase the error in an unquantifiable manner, but the robustness of the foF₂ and hmF₂ parameters allows one to consider the ionosonde foF₂ as significantly more reliable than the ionosonde hmF₂.

As expected due to the challenges with obtaining the hmF₂, the validation against hmF₂ is less consistent. IRI-2016 typically models the hmF₂ better than NIMO, with PMAE=44.5% and PRMSE=25.9%. However, the difference between the MAE and RMSE values are usually less than 1 km, which is not significant.

The run where NIMO most consistently performs better than IRI-2016 is the Mar 2015 storm period. During the geomagnetic storm, NIMO more correctly estimates the increases in foF₂ and hmF₂ during the daytime and the nighttime. The nighttime differences between NIMO and IRI-2016 are more significant, though, with an improvement in the RMSE of 0.28 MHz for the foF₂ and 2 km for the hmF₂. This is unsurprising, as NIMO better equipped to handle the combination of drivers needed to correctly specify an individual geomagnetic storm than a climatological model such as IRI-2016. Fig.  shows an example for NIMO (dark blue) and IRI-2016 (light blue) at a location near the Jicamarca Radio Observatory that demonstrates NIMO's ability to capture the expected uplifting of the F₂ layer caused by the geomagnetic storm.

The observations from ISRs located at AO, JRO, and MH were available for five of the desired validation periods: two active solar times, two quiet solar times, and the geomagnetic storm period. Following the lead of the ionosondes, the ISR data used for validation is currently limited to the hmF₂ and NmF₂. The hmF₂ and NmF₂ values for the ISR data are obtained by finding the maximum density and the corresponding altitude for each electron density profile at a given time.

Because NIMO captures both sub-hourly variations in the ionosphere and long-term climatological trends, it is useful to evaluate both the standard NIMO output and the trend of the NIMO output (which removes the short-term variations). The NIMO trend was calculated for the model's hmF₂ and N_mF₂ by taking a boxcar average of the standard NIMO outputs using a full window width of 2 h. The trend results were provided at a 15 min resolution, matching the NIMO output cadence. A similar analysis was performed for the ionosonde validations, but the trend statistics were extremely close to the standard model statistics, and so are not included here in the interest of brevity.

The bias and the r1 coefficient are calculated to determine both the degree and presence of a correlation between the models and the ISR observations. For the NIMO validation runs, the model trends for hmF₂ and NmF₂ are also examined to see whether NIMO captures the main features of the hmF₂ and NmF₂ in a manner comparable to IRI-2016. An example of the ISR data and model outputs is shown in Fig. , which shows the observations (purple), NIMO results (blue), NIMO trends (green), and IRI-2016 results (yellow).

The results of the ISR hmF₂ and NmF₂ validation are shown in Figs.  and (and presented in Tables  and ), respectively. The results show that at all locations NIMO performs equally well or better than IRI-2016, when considering the standard model output and the NIMO trend. The NIMO trend usually performs slightly better than the standard NIMO output (except during the storm run), though the differences in the metrics are small. This indicates that there is room for improvement in the smaller-scale model variations, be it through data assimilation or the physics-based modeling. NIMO performs significantly better than IRI-2016 at times of higher ionospheric density (such as geomagnetic storm period), while the differences in the bias and r1 are smaller during solar quiet times. Note particularly that the performance of NIMO does not change against MH data before and after the start of 2015, when as previously mentioned the electron density profile calibration changed from ionosonde referenced to self-calibrated using the MH plasma line. The change in calibration method does not alter the seasonal or storm time trends in model performance observed by the other ISRs.

This section examines the GNSS VTEC from the Madrigal database over a validation region within the Contiguous United States (CONUS), shown with the green box in Fig. . This region was chosen for this validation because it is heavily instrumented. To facilitate analysis, the Madrigal TEC is linearly interpolated onto the NIMO TEC grid, reducing the resolution to 1° in latitude and 4° in longitude. The Madrigal data time step closest to each NIMO output time is retained for comparisons.

The Madrigal TEC is also compared to IRI-2016 TEC outputs. The IRI-2016 TEC is also linearly interpolated onto the NIMO TEC grid increasing the resolution to 1° in latitude and 4° in longitude. The Madrigal data time step closest to each IRI-2016 output time is retained for these comparisons.

The results of the Madrigal vertical TEC validation, presented in Table  and Fig. , show that NIMO typically outperforms IRI-2016, having a PB=66.67% and a PRMSE=100%. Both NIMO and IRI-2016 agree with the data better during the solar quiet periods than the solar active or storm periods.

An example of the time evolution of these statistics is included in Fig. , which shows one of the time periods where the IRI-2016 B performs better than the NIMO B. The top left panel shows the average TEC within CONUS over the duration of the October 2014 run for both NIMO and Madrigal TEC. NIMO tracks the daily variation well but overshoots the daily maxima by a little bit during the latter half of the month. The generally good agreement is supported by the resulting small B and RMSE in the top right panel.

By comparison, IRI-2016 and Madrigal average TEC are shown in the bottom left panel. IRI-2016 typically undershoots the TEC for the first half of the month but significantly overshoots the daily maxima during the second half of the month. This tendency to both overestimate and underestimate the TEC in IRI-2016 is captured by the B trending negative during the first half of the month and then positive during the second half of the month. However, these opposing biases average out when considering the period as a whole. This causes the small B for the entire run, shown in Table , and highlights why the B must be considered alongside other metrics when evaluating model performance.

The JASON-2 and JASON-3 altimeters provide direct measurements of the nadir TEC between the spacecraft (mean altitude of 1340 km) and the Earth's surface. The measurements are only available over the oceans (Fig. ), but are a good complement to land-based GPS receiver measurements. With an inclination angle of 66°, each JASON satellite slowly precesses through all local times in ∼ 2 months; thus each validation month covers roughly half of the local times, as illustrated in Fig.  for January 2021. For more details on the JASON measurements, refer to Sect.  and references therein.

Overall, the validation shows that NIMO TEC, calculated up to 1340 km in altitude, compares well with JASON TEC measurements and outperforms IRI-2016. The global statistics for each validation run are shown in Table  and by the filled circles in Figure . Seasonal and solar cycle variations in the RMSE are highlighted in Fig. .

During the solar active months in 2014, the NIMO RMSE was lower than that of IRI-2016 by 37 % and during solar quiet months the improvement is more than 40 %. Both models show a seasonal change in performance during the solar active runs that is not present during the solar quiet runs, with the solstices having lower RMSE values than the equinoxes. Additional metrics used for this analysis include the B and R2 (see Sect. , , ). The R2 shows a higher correlation between the NIMO results and the data in all instances, while the B is typically lower in the NIMO results.

Breaking the data down into the four regions shown in Fig.  permits the validation of low-latitudes (within 35° of the magnetic equator) and mid-latitudes (between 60 and 35° north and south of the magnetic equator) in two local time regions that largely encompass day (06:00–18:00 SLT) and night (18:00–06:00 SLT) times, as was done for the ionosondes. The mid-latitude results are presented in Table  and shown in Fig.  by squares and stars. To summarize across all of the validation runs, the mid-latitude daytime results show NIMO has more realistic TEC values than IRI-2016 with PR2=77.78%, PB=88.89% and PRMSE=100%. The mid-latitude nighttime results also show better agreement between NIMO and the JASON observations, with PR2=100%, PB=77.78% and PRMSE=77.78%. In the low-latitudes (presented in Table  and shown in Fig.  by upward and downward facing triangles), all of the daytime and nighttime NIMO results correlate better with the JASON observations (PR2=100%). The PRMSE also shows NIMO doing a better job representing the observations than IRI-2016 in both local time groups, with values of 100% for the daytime and 88.89 % for the nighttime. The B comparison is weaker, with NIMO outperforming IRI-2016 in the daytime (PB=66.67%), but not at night (PB=44.44%). However, this can be explained by examining the different TEC distrubutions. An example of the type of distribution where IRI-2016 performs better than NIMO is shown in Fig. . The wider typical deviation between the IRI-2016 and JASON observations has a tendency to balance out, as reflected in the performance of the R2 and RMSE statistics.

The topside ionosphere, the O⁺ dominated region above the F₂ peak, is dominated by plasma transport processes. In different latitudinal regions, the magnetic field structure leads to different types of interactions between the ionosphere, thermosphere, and magnetosphere. This validation uses in situ measurements of the plasma density from IVMs flown onboard the C/NOFS, DMSP, and ICON satellites. The data from the satellites are paired directly with the model output, reducing the number of samples available for the validation based on the model's 15 min cadence. This direct pairing was chosen to avoid introducing interpolation into the validation process and avoid considering structures with time-scales smaller than the model is capable of reproducing. For more details about these data sets, see Sect. , , and .

The results of the topside ionosphere validations, presented in Tables  and , may be summarized for all satellites and regions with NIMO outperforming IRI-2016 in the error meta analysis, but not the bias, with PB=27.9%, PRMSE=74.4%, and PMed.AE=62.8%.

A more detailed look at the topside error is presented in Fig. , which presents the seasonal variations for each spacecraft used in the model validation for the solar active and solar quiet periods. The seasonal and solar cycle variations are similar to those seen by the JASON TEC in Fig. . During the solar active year of 2014 NIMO and IRI-2016 perform better during solstices than equinoxes, with the best performance seen during July 2014. During the solar quiet years, no seasonal variation is present, and both IRI-2016 and NIMO have lower Med. AE values then they do during the solar active years.

The DMSP results were further broken down into different local time regions that were chosen to encompass times with similar physical processes, four of which contained sufficient data to be used for model validation. These four groups, and the reason behind their selection are: 02:00–06:00 SLT contains sunrise, 06:00–08:00 SLT contains the morning hours at northern mid-latitudes (for F18, the only spacecraft to use this bin), 12:00–17:00 SLT contains the afternoon density peak, and 17:00–20:00 SLT contains sunset. The summary of these results is shown in Fig. . This figure shows that NIMO most consistently performs better than IRI-2016 during the 02:00–06:00 SLT period. During the other local time periods NIMO models the topside plasma density better than IRI-2016 during the solar active runs, but not during the solar quiet runs.