Space-borne clear air lidar measurements in the presence of broken cloud

A number of proposed lidar systems, such as ESA's AEOLUS (formerly ADM) and DIAL missions (e.g. WALES) are to make use of lidar returns in clear air. However, on average, two-thirds of the globe is covered in cloud. Hence, there is a strong likelihood that data from these instruments may be contaminated by cloud. Similarly, optically thick cloud may not be penetrated by a lidar pulse, resulting in unobservable regions that are overshadowed by the cloud. To address this, it is suggested, for example, in AEOLUS, that a number of consecutive short sections of lidar data (between 1 and 3.5 km in length) be tested for cloud contamination or for overshadowing and only those that are unaffected by cloud be used to derive atmospheric profiles. The probability of obtaining profiles to near ground level using this technique is investigated both analytically and using UV airborne lidar data recorded during the CLARE'98 campaign. These data were measured in the presence of broken cloud on a number of flights over southern England over a four-day period and were chosen because the lidar used has the same wavelength, footprint and could match the along-track spacing of the proposed AEOLUS lidar.


Introduction
Currently, there are a number of proposed satellite radar and lidar instruments (such as those to be carried on the AE-OLUS, CALIPSO and EARTHCARE missions), which are to make observations either along or parallel to their subsatellite track.One of these, ESA's proposed Doppler wind lidar of the Atmospheric Explorer for Observations with Lidar in Ultraviolet from Space (AEOLUS) mission (formerly the Atmospheric Dynamics Mission) (European Space Agency, SP-1233SP- (4), 1999)), is to provide every 200 km along Correspondence to: I. Astin (iva@mail.nerc-essc.ac.uk) track an altitude profile of the component of the wind vector along the line of sight of the lidar beam.These profiles are formed by integrating the returns from a number of consecutive lidar pulses, spaced 70 metres apart along track.Hardware considerations (ESA, 1999) restrict to 14 the minimum possible number of lidar pulses to be integrated together, corresponding to a pixel of length 1 km along track, and the maximum number to 50, corresponding to 3.5 km.The lidar system is to make use of clear air (molecular) Rayleigh returns and aerosol Mie returns along 50-km sections (equating to 700 pulses).Thus, each 50-km section is made up from between 14 (3.5 km) and 50 (1.0km) of the integrated pixels.The lidar is then "switch off" for 150 km before finding another profile along the next 50-km section.However, these lidar returns are swamped by any cloud returns present in the integrated signal.This causes problems because the component of the wind will be measured at the position of the cloud.Clouds have another more important effect in that they may be sufficiently optically thick to prevent the lidar pulse from penetrating them and so prevent the lidar from making observations below them and obtaining a wind profile to near ground level.In order to get around this problem, it is proposed (ESA, 1999) that only the integrated pixels (of length 1.0 to 3.5 km) within a 50-km section uncontaminated by cloud should be used to form the profile.For this to be an effective strategy any "holes" in cloud layers must be of a length comparable to the lengths of these sections and be sufficiently close together.The probability of obtaining uncontaminated profiles is investigated in the following, both analytically, and using air-borne ultra-violet lidar data recorded during the CLARE'98 (Cloud Lidar And Radar Experiment) campaign of 1998 (Baptista et al., 2000).It may a priori be expected that integrating the minimum possible number (14) of pulses is the best choice; however, a ADM/AEOLUS type lidar in operational mode giving ∼300 mJ pulses (ESA, 1999) would need 10 or more shots to provide a meaningful wind profile in daytime measurements (ESA, 1990).Thus the use of only 14 shots is near the minimum required to obtain a wind profile.Hence, the alternate mode of 50 shot in-tegration has been chosen as the normal mode (ESA, 1999).Thus, in part, the purpose of this study is to find if there is a great difference between the performance of these two accumulation lengths.

Analytical approach
The general formula for the amount of any geophysical parameter contained within a finite transect given any distribution for the length of geophysical regions and for the gaps between such regions is derived by Astin and Di Girolamo (1999).This can be applied to each 50-km section, and to the individual sections (of length 1.0-3.5 km) making up the 50-km section, to find the probability of their cloud contamination.In cloud fields Astin and Latter (1998) showed that exponential distributions provided a good fit to the observed length distributions for cloud and "holes" within cloud.These show that if the mean length of the holes is 1/λ and the cloud free fraction is f clear , then the probability, p, of a randomly chosen interval of length L, being completely cloud free, is just (1) Hence, if f cloud is the long-term cloud fraction (= 1 − f clear ), then the probability that any randomly chosen 50-km section is contaminated with cloud is Results published on the International Satellite Cloud Climatology Project (Rossow and Shiffer, 1991) web site (http: //isccp.giss.nasa.gov/climanal1.html)show that the globally averaged mean cloud cover fraction averaged over the period June 1983 to December 1999 inclusive is 0.675.With this value the probability of a randomly chosen 50-km section (anywhere over the Globe) being contaminated with cloud is Thus, for example, with this fixed cloud fraction (of 0.675), if the mean length of the holes (1/λ) is less than 40 km, say, then this probability will always exceed 0.90.Astin and Latter (1998) shows examples (from low Earth orbit and geostationary satellite cloud images) where the mean length of such holes is comparable to this length.Hence, by integrating all 700 laser shots together (covering a 50-km length), it is possible that more than 90% of all wind profiles will be contaminated by cloud at some altitude level.The probability of contamination, as given by Eq. (3), will be reduced if the actual mean length of the holes (1/λ) is larger than 40 km but always exceed 67.5%, whatever its value.Similarly, if L is made smaller than 50 km, then the probability of contamination is reduced for a given value of 1/λ, but again the probability will always exceed 67.5%.It is proposed (ESA, 1999) in the AEOLUS mission to try to improve on this limiting probability by considering separately all the contiguous (1.0 to 3.5 km) sections that add to make each 50-km section and to include only those uncontaminated sections to form the profile.As these shorter sections are contiguous, they cannot be considered as chosen at random, which, if assumed, would considerably simplify the calculation of the probability of finding at least one uncontaminated section (as in ESA, 1998).In fact, the calculation of the probability of obtaining at least one (1.0 to 3.5 km) uncontaminated section within any 50-km interval is extremely complicated (as given in the Appendix).So a simulation program was written to evaluate numerically the probability mass function for the number of uncontaminated sections of length 1.0 or 3.5 km within a randomly chosen 50-km section.Results from such simulations are presented in Figs. 1 and 2. In Fig. 1 the cloud fraction is assumed to be 33%, as found by Stevermer (1997), for all data recorded in September 1994 using the LITE lidar instrument (Winker et al., 1996).Four different mean hole lengths are assumed.The first (540 km), given in Fig. 1a, is consistent with cloud decks observed by LITE (Stevermer, 1997), the second (36.5 km), given in Fig. 1b, is from Astin and Latter (1998), as derived from passive satellite images, and the other two (2.0 and 7.1 km), in Figs.1c and d, are chosen to give mean cloud lengths equal to the two pixel lengths (1.0 and 3.5 km).Since the cloud fraction is fixed at 33%, and since it is made up of a sequence of clouds and holes, the mean cloud length must always be almost half the mean gap length.Hence, if the mean hole length is large, then the mean cloud length is also large and if the mean hole length is short, then so is the mean cloud length.Figure 1 indicates that where the cloud fraction is near 33% the method of rejecting contaminated pixels before deriving wind profiles results in an uncontaminated wind profile at least 82% of the time if the mean hole length is less than 36.5 km.This improvement can be explained by the fact that the mean hole length is smaller than the 50-km section but larger than the 1.0-3.5 km sections.Thus, in sliding a 50-km section over a sequence of clouds and holes it can be envisaged that it is likely that the 50-km section will contain some cloud (and be contaminated) but may also contain several holes, only one of which need be sufficiently large (to exceed 1.0-3.5 km) to result in an uncontaminated wind profile.However, there is little improvement in the probability of obtaining an uncontaminated profile (at 69%) for the large mean hole length (540 km) over that achieved using a single (1.0 to 3.5 km) pixel (at just over 66%).This is because, for this case, individual clouds and holes are in general very much larger than the 50-km section of interest and so, except at the edge of clouds, the sections will mostly be either cloud filled or cloud free.Thus, all sections within a 50-km length must also be cloud free or cloud filled and so no improvement is possible.There is also evidence of a reduction in the probability of gaining an uncontaminated wind profile at the shortest mean length considered (2.0 km).This would be because, though it is likely that a large number of holes are present in any 50-km section, many holes would likely be smaller than 1.0 to 3.5 km, resulting in a contaminated profile.Figure 2   data (over Borneo) from Astin and Latter (1998) for cloud fractions of 53% (Fig. 2a) and 83% (Fig. 2b).Less improvement is gained in looking for short, uncontaminated sections if the cloud fraction is 53% or higher for these mean lengths.This appears to result from a reduction in the mean length of the holes but an increase in the mean length of clouds as the cloud fraction increases, the latter giving a reduction in the number of holes per 50-km section.Thus, there is an increase in cloud amount, increasing the probability of a 50-km section being contaminated and a reduction in the probability of finding an uncontaminated 1.0 to 3.5 km section, since there will be fewer and smaller holes within any 50-km section than for the 33% cloud fraction, negating any possible improvement.
The above describes only the probability of obtaining at least one uncontaminated 1.0-or 3.5-km section in a 50-km interval.However, it may be anticipated that the more uncontaminated sections used, the smaller the error in derived wind speed.As a first step Figs. 1 and 2 give the probability of observing more than one such interval.In all panels in both figures a binomial distribution is also plotted assuming independence between adjacent sections, with the probability of a completely clear section being given by Eq. ( 1).This gives an almost exact fit in the right-hand panel of Fig. 1d but is a very poor fit in all the other cases considered.Thus, independence between adjacent sections can only be assumed where the mean gap and cloud lengths are less than the section length.This is to be expected for an exponential field, where independence between points only occurs if they are separated by more than the sum of the two means (Astin, 1997).

DLR data
A number of data sets, most notably that from the LITE campaign (carried on the Space Shuttle), provide observations of cloud using a lidar at UV wavelengths.However, it was decided to use air-borne UV (354 nm) lidar data of the DLR Falcon research aircraft collected during the CLARE'98 campaign (Baptista et al., 2000).Though these data come only from southern England, rather than the near global (out to ±57 • latitude) LITE data, the parameters of the air-borne lidar are much closer to the AEOLUS lidar than that of LITE.For example, the DLR lidar footprint is comparable to that for the AEOLUS lidar of near 7 m at cloud level rather than LITE's 270 m.This reduces the possibility of multiple reflections within clouds.Also, the pulse repetition frequency for the LITE lidar is 10 Hz, which, given the speed of the satellite, equates to a footprint separation of 740 m, whereas the AEOLUS lidar has a PRF of 100 Hz and a footprint separation of 70 m.Hence, the AEOLUS lidar will sample the cloud field (at least before integration) at much higher resolution than LITE.However, the AEOLUS sampling is bettered by the airborne lidar due to the slower speed of the aircraft and so it will be more accurately represented by the DLR data.
Of the DLR campaign data's approximately 36 000 laser shots, each were considered from 4 separate days (13, 14, 20 and 21 October 1998).This was equal to an hour flight on each day.The original DLR data were collected at a pulse rate of 10 Hz, resulting in an approximate along-track interpulse spacing of 20 m, when compared to the speed of the aircraft (of ∼200 m/s).This is finer than that of AEOLUS (70 m) and so only data from every third pulse was used.The data were recorded over southern England by a lidar carried on an aircraft flying backwards and forwards along a straight leg, as demonstrated in Fig. 3.This figure gives the flight path, altitude and time, for 13 October 1998, overlaid on a terrain height map of the region (50.9-51.2• N and 3.2-1.0• W).Over each of the four days the aircraft covered approximately 700 km.An image of such data, as used in this study, is presented in Fig. 4, which shows data from a straight, near 50-km long section.This closely matches the length of the interval over which an AEOLUS wind profile is to be made.This was recorded over the period 12:47 to 12:51 LT on 13 October, and data from exactly 700 lidar shots are presented.This would represent "raw" AEOLUS data, constituting a 50-km section, provided no horizontal or vertical integration was used.The continuous line, at an altitude averaging ∼100 m, gives the orography below the flight path, which is flat and relatively low lying.Broken cloud layers are present, and on most occasions these are sufficiently optically thick to prevent lidar pulse penetrating to anywhere near the ground, for example, near 12:48 and after 12:49 LT.On rare occasions, for example, around 12:49:30 LT, near ground returns can be observed below cloud layers.
The vertical resolution of the DLR data is 15 m and to match the AEOLUS data this is integrated to give a near 1km vertical resolution.This is achieved by adding the data from 64 vertically contiguous 15-m altitude bins together.Similarly, data were integrated in the horizontal to give a horizontal resolution of 1 km or 3.5 km (i.e. by integrating 14 or 50 shots).
Figure 5 shows the returned lidar signal centred on the altitude of 3 km (a cloud deck) for data recorded over the hour (12:17 to 13:17 LT) for a flight on 13 October 1998.This plot is formed after vertical and horizontal integration.In this case the 1-km pixels along this 680 km transect that are contaminated with cloud can be seen to have a reflectivity of several thousand (in the arbitrary units).A histogram of these data is presented in Fig. 6 that shows 3 clear regimes, of a peak below about 800, another peak between 800 and 6000 and a third broader peak beyond 6000.It is assumed that the peak below 800 is the background, that beyond 6000 are cloud returns and in between these are clear air returns.
Of the 680 1-km long pixels recorded at the 1-km altitude level (near ground) during this hour it was found that 33.3% had a reflectivity lying between 800 and 6000 and so are assumed to be unaffected by cloud.The data were also integrated to give 3.5-km pixels, by adding together 50 consecutive pulses.It was found using the above upper and lower threshold values that the probability of obtaining an uncontaminated pixel was, at around 33%, barely less than that for 1-km integration.Similar results, Table 1, were found on the other three days, indicating that on these occasions the holes in the cloud are mostly larger than 3.5 km.The table also shows that the probability of finding one or more holes greater in length than 3.5 km in any 50-km section ranges from 55% on 20 October 1998 to near 100% on 13 October 1998.This considerably exceeds the probability that any randomly chosen 3.5 km section is cloud free.Similar results hold for a section length of 1 km.

Conclusions
ESA's AEOLUS mission is to make use of lidar returns from clear air along a 50-km transect.Results on global cloud fraction from ISCCP would indicate that at least two-thirds of such transects would contain some cloud at some altitude within the column observed by the lidar.However, the probability of contamination is also a function of the mean length of the holes within cloud layers.It is shown that as this mean decreases, the probability of contamination increases to be greater than 90%, if the mean length is less than 40 km, and only approaches a value of two-thirds for very long mean lengths.Clouds may be optically thick (Fig. 4) and may not be penetrated by a lidar pulse, resulting in unobservable regions overshadowed by the cloud.To address these two problems, it is suggested that a number of contiguous short sections of between 1.0 and 3.5 km in length, making up the 50-km section, be tested for cloud contamination or for overshadowing.Those that are unaffected by cloud would then be used to derive atmospheric wind profiles.It is shown analytically that this should be a very effective strategy if the   cloud fraction is less than a critical value, irrespective of the mean hole length.But if the cloud fraction greatly exceeds this, then the strategy should not be much more effective than checking at random just one of the short transect (of length 1.0-3.5 km), within the 50-km transect, for cloud contamination.Evidence for such a critical value is seen in the study of lidar data recorded in broken cloud during the CLARE'98 campaign, where it is shown that the strategy (Table 1) can be very effective even if the cloud fraction is up to 67% (13 and 14 October) but is much less effective (on 20 and 21 October) when the cloud fraction exceeds 70%.

A1 Two contiguous pixels
It is shown (in Eq. 1) that the probability of an individual pixel of length L being cloud free is f clear e −λL .Hence, the probability of two contiguous pixels both being cloud free is f clear e −2λL , since this is just the probability of a pixel of length 2L being cloud free.Now, if the first pixel is cloud free, then the second pixel must either be cloud free or contaminated with cloud.Thus, the probability of having a cloud-free pixel followed by a contaminated one is the difference between the probability of the first pixel being cloud free and the probability of the first and second pixels being both cloud free, that is f clear e −λL − f clear e −2λL .Given this and that the probability of the second (or indeed any individual) pixel being contaminated with cloud is 1.0 − f clear e −λL , and that a contaminated pixel must be adjacent to either a cloud free or contaminated pixel, then the probability of two consecutive pixels being contaminated, p contamination , must be ) This is the difference between the two probabilities (since the events are exclusive and exhaustive).

A2 Three contiguous pixels
The probability of three contiguous pixels all being cloud free is f clear e −3λL , as again this is the probability of a pixel of length 3L being cloud free.As they are contiguous, if the first pixel is cloud free, then it must be that the second pixel starts in a cloud free region.Thus, given that the first pixel is cloud free, the probability that it is also cloud free at the start of the third pixel is f clear + f cloud e −(λ+µ)L , where 1/µ is the mean cloud length (Astin, 1997).This is a standard result for a binary Markov process that starts in a fixed state.Hence, irrespective of the second pixel, the probability that both the first and third pixels are cloud free is f clear e −λL (f clear + f cloud e −(λ+µ)L )e −λL , which is the probability of the first pixel being cloud free multiplied by the probability that the third pixel is cloud free, given that the first is cloud free.As the probability of obtaining three contiguous cloud free pixels is f clear e −3λL , the probability of obtaining a contaminated pixel between two cloud free ones is thus f clear e −2λL (f clear + f cloud e −(λ+µ)L ) − f clear e −3λL .The probability of obtaining a clear pixel followed by two contaminated ones is thus f clear e −λL − f clear e −2λL − f clear e −2λL (f clear + f cloud e −(λ+µ)L ) + f clear e −3λL .As two contaminated pixels must be adjacent to either a cloud-free or contaminated pixel, the probability, p (contamination) , of obtaining three contiguous contaminated pixels is just the probability of obtaining two contaminated pixels (Eq.4) minus the latter probability, i.e.Trying to evaluate this for 14 or 50 consecutive pixels becomes progressively more complicated and so a simulation program was written to evaluate these probabilities numerically and examples are presented in Figs. 1 and 2.

Fig. 1 .
Fig.1.The probability mass function (squares) for the number of uncontaminated 1-km sections (left-hand panels) and 3.5 sections (righthand panels) in a 50-km interval for a cloud fraction of 33% for a mean hole length of (a) 540 km, (b) 36.5 km, (c) 3.5 km and (d) 1.0 km.Also shown is the binomial distribution that would apply if adjacent sections were assumed independent.

Fig. 2 .
Fig.2.The probability mass function (squares) for the number of uncontaminated 1-km sections (left-hand panels) and 3.5 sections (righthand panels) in a 50-km interval for a cloud fraction of (a) 53% with mean hole length of 35.3 km, and (b) 83% with mean hole length 13.4 km.Again, also shown is the binomial distribution that would apply if adjacent sections were assumed independent.

Fig. 3 .
Fig. 3.The aircraft flight path for 13 October 1998.The observing region is from 50.9-51.2• N and 3.2-1.0• W. The altitude of the aircraft (near 10 km) is also marked, and the local orography is given in green and brown to represent heights above sea level.

Fig. 4 .
Fig. 4. Atmospheric lidar backscatter at 354 nm.This shows data from 700 lidar pulses along a 50-km transect.The local orography is also plotted.There are several cloud layers present and on occasions the returns near the ground can be observed.

Fig. 5 .
Fig. 5. Integrated lidar returns over 1 km by 1 km horizontal (14 pulses) and vertical sections, at an attitude of 3 km, covering 12000 lidar pulses (equal to approximately 680 km along track or 1-h flying time).

Table 1 .
The probability of obtaining at least one pixel of mean height between 1.0 and 1.5 km that has a reflectivity in the range 800 to 6000