SITCOMTN-148

LSST Camera Electro-Optical Test Results

This is a draft.

By:

• Aaron Roodman
• Yousuke Utsumi
• Sean MacBride

Abstract

This note collects results from the LSST Camera electro-optical testing prior to installation on the TMA. We describe the CCD and Focal Plane optimization and the resulting default settings. Results from eo_pipe are shown for standard runs such as B-protocols, Dense and SuperDense PTCs, gain stability, OpSim runs of Darks, and Darks with variable delays. We also describe features such as e2v Persistence, ITL phosphorescence in coffee stains, remnant charge near Serial register following saturated images, vampire pixels, ITL dips, and others.

Electro-optical setup

Run 7 Optical modifications

For Run 7 there were a few changes with our setup on Level 3 as compared to Run 6 taken in IR2 at SLAC. One of the changes was that we did not have access to the CCOB Narrow/Thin beam. .. While the set up was on Level 3, we did not have the resources or expertise to get it setup. As such, the majority of the testing was done with the CCOB Wide Beam projector. We did obtain an additional projector, the 4k projector, part way through Run 7 that will be discussed later. With the CCOB Wide Beam, we used a cone attached to the L1 cover as well as shroud to create a dark environment as seen below.

Fig. 1 Final shroud configuration of LSSTCam in Level 3 to reduce light leaks.

Fig. 2 CCOB Wide Beam attached to the cone and shrouded.

This allowed us to operate on Level 3 with a dark current of <0.1 ADU/sec with the shutter open. The initial set up of the CCOB Wide Beam was the same as Run 6, we had a minimal ND filter (10 %) attached to a C-mount lens. One change was that the f/stop of the lens was changed from 2.6 to 1.6 (fully open). This was done to try and reduce the effect of the ‘weather’ and the ‘CMB patten’ two effects that we found in Run 6 and were found to be due to our projection setup (see [Banovetz2024]), While changing it did reduce the weather pattern, it also caused a much steeper roll off across the focal plane. The below figures show the weather pattern as compared to Run 6 and the rolloff of the light as compared to Run 6.

Fig. 3 Final shroud configuration of LSSTCam in Level 3 to reduce light leaks.

Fig. 4 CCOB Wide Beam attached to the cone and shrouded.

Fig. 5 Illumination across the focal plane from Run 7 with the diffuser () as compared to Run 6 ().

To both reduce the effect of the ‘weather’ and ‘CMB’ but retain uniform illumination across the focal plane, we installed a diffuser in the cone attached to L1. Below shows the placement of the diffuser along the cone.

Fig. 6 Diffuser installed into the light cone.

The diffuser combined with a fully open F/stop effectively reduced the incoming light by roughly 35%. Adjusting for that, we found that it severly reduced the ‘weather’ and eliminated the CMB pattern, as well as fully illuminateing the focal plane. Below shows the effect of the diffuser in regards to the weather, CMB, and the overall illumination of the focal plane.

Fig. 7 Full focal plane image showing the fractional difference in Run 6.

Fig. 8 Full focal plane image showing the fractional difference in Run 7.

Fig. 9 Illumination across the focal plane from Run 7 with the diffuser () as compared to Run 6 ().

The diffuser was installed for all B protocol and PTC runs moving forward, only being taken out for pinhole projection and when using the 4K projector.

The newest addition to the projectors used for EO testing was a 4K projector, simliar to those used in conference rooms. This projector was first tested at SLAC before coming to Chile around half through Run 7. This was used primarily as a spot projector, as the pinhole filter wasn’t operational but more importantly, this could illuminate all 3206 amplifiers instead of the 21 illuminated by the pinhole projector. Most runs included the spots and the spot fluxes were controlled by the shutter instead of any flashing (e.g. CCOB Wide). One downside that was found was that the illumination of ‘dark’ regions (regions not supposed to be illuminated) were still giving off light. This background region had structure that changed with time and could not be easily subtracted. It also caused the contrast between the spot and the background to be around 6. Changing the shape to large rectangles for crosstalk measurements increased this contrast to 30.

[Banovetz2024]

https://ui.adsabs.harvard.edu/abs/2024arXiv241113386B/abstract

Projector spots

hello world.

This section describes the spots and rectangles tested with the 4k projector

• Projector background

• Spots on many amps

• Spots on one amp

• Optical setup

Dark current and light leaks

This section describes dark current and light leaks in Run 7 testing.

Light leak mitigation with shrouding the camera body

One of the first tests we attempted with the camera was measuring dark current and sources of light leaks in the camera body. This began by finding gaps between the L1 cover and the gaskets. This was covered up by tape where we felt comfortable applying it. Below shows the gaps that we could see between L1 and its cover.

Once these were sealed, we took some intital measurements and then started to cover the camera body with shroud. We also found light leaks where the light cone attached to L1 was housed, and from the utility trunk. Below is the table of observations, their corressponding dark current, and comments on what changed.

Table 1 Table showing the different 15 second dark exposures, the different conditions, and the resulting dark current.

Exposure	Dark Current	Room Lights	Shutter	Comments
MC_C20240909_000012	0.158	Off	Closed
MC_C20240909_000018	0.158	On	Closed
MC_C20240909_000038	2.94	On	Open	Initial Covering (just the CCOB cone and around the L1 cover)
MC_C20240909_000054	1.34	On	Open	Initial Covering + Blanket over the FCS
MC_C20240909_000072	0.41	On	Open	Initial Covering + Blanket over AND under the FCS
MC_C20240909_000078	0.18	Off	Open	Initial Covering + Blanket over AND under the FCS
MC_C20240910_000031	0.033	On	Open	Initial Covering + Blanket over AND under the FCS + UT

Below is a the image of the final shroud configuration covering the camera.

Fig. 10 Final shroud configuration of LSSTCam in Level 3 to reduce light leaks.

Successful Autochanger Light Leaks masking

A dedicated dark/light leak study was performed during the Run 6 at SLAC in summer 2023 and a localized faint light source going up to ~ 0.04 e-/s/pixel was associated to the 24 V Clean of the FES auto-changer.

In the Auto-Changer this voltage is used to power some probes and all controllers. In february 2024, as AC-1 was extracted from the camera for a global maintenance, a direct investigation to localized the light source was performed without success. A light source in the AC wasn’t expected as in the AC all controllers LED have been removed, and most electronics are in “black boxes”. Still two small probes , which had LEDs that could not be removed, were initially masked by a black epoxy. As we had doubt on the quality of this masking in the IR, we applied and extra-making (aluminium black tape) on them during the Feb 2024 maintenance (on AC 1 and 2).

At the start of the Run 7 a new study of the light leak based on 900s dark exposures with the shutter open and the empty frame filter en place, showed that the AC light leaks was still present ( see left plots of AC light leak ). Following this finding, a full review of all the AC hardware powered by the 24 V dirty was performed, and a candidate was found : the coders of the 5 main motors of the AC had a partial documentation from the vendor not mentioning the presence of LED. After interaction with the vendor the presence of ~700nm LEDs incide the coders were reported. The hypothesis of ~700nm LED source has been found compatible with the observation as no AC light leaks were detected using the different filters in camera at the start of Run 7 (g,r and y filters) . A dedicated test in Paris using an AC spare coder and a precision photometric set-up allowed to identify leak in the masking of those LED in the vendor packaging. A complementary masking method based on a 3D printed part + tape + cable tie was qualified in Paris: it is masking the light leak and it is safe (all parts correctly secured ).

In November 2024, we masked all the lights in the back of level 3 clean room ( not the part with the camera) to setup a high quality dark room allowing a direct observation with a CMOS camera of the light leak on the AC2 motors coders. Also the level of darkness reached, allowed us to validate the quality of the AC coders light masking. Notice that the FES-prototype in Paris doesn’t have coder on the Online Clamps, so we had to tune/qualify directly on the AC2 at summit the masking of those coders.

For each AutoChanger (1 & 2), the 5 motors coders with vendor issue on their LED masking, have been successfully enveloped in a light tight mask.

Notice that the AC was off starting the Sep 27th at 21:15 UTC in the first part of the Run 7. For the end of Run 7 (run taken after mid-November) the AC was back On: as the AC 1 was back in camera with the new coders light mask in place, we were able to take a new series of 900s dark with AC On & off, confirming that we had no light leak left associated to the Filter Exchange System. (see right plots of AC light leak)

The left plot shows the original impact of the AC light leak on 900s dark ( AC On - AC off). On the right plot, after masking the AC LED coders, no light associated to the FES is present in 900s dark difference( FES On - FES off).

Shutter condition impact on darks

Filter condition impact on darks

Final measurements of dark current

Reverification

Baseline characterization

Background

Initial characterization studies performed on LSSTCam were used two primary acquisition sequences.

• B protocols: this acquisition sequence consists of the minimal set of camera acquisitions, including

  • Bias images

  • Dark images

  • Flat pairs - flats taken at varying flux levels

  • Stability flats - flats taken at consistent flux levels

  • Wavelength flats - flats taken in different LEDs

  • A persistence dataset - a saturated flat, followed by several darks

• PTCs (photon transfer curves): this acquisition sequence consists of a sequence of flat pairs taken at different flux levels. The flat acquisition sequence samples different flux levels at a higher density than the B protocol flat sequence, enabling a more precise estimate of flat pair metrics.

All EO camera data is processed through the calibration products and electro-optical pipelines to extract key metrics from the data run. The key camera metrics from Run 7, and their comparison to previous runs are discussed below.

The naming of the EO runs was established during initial camera integration and testing. The final SLAC IR2 run from November 2023 was named “Run 6”, while the data acquisitions from Cerro Pachon are considered “Run 7.” Additionally, individual EO acquisitions are tagged with a run identifier. This is commonly referred to a Run ID. For all SLAC runs, the run identifier was a five digit numeric code, while the Cerro Pachon runs were “E-numbers” that started with a capital E followed by a numeric code.

For comparison between Cerro Pachon EO runs and the final SLAC IR2, the following runs are used.

Run Type	SLAC IR2 Run	Cerro Pachón Run
B Protocol	13557	E1071
PTC	13591	E749

Among all of these measurements, primary concern is that the camera has maintained its performance standards between the SLAC IR2 run in November 2023 and the Cerro Pachon run in October 2024.

Stability flat metrics

Charge transfer inefficiency

CTI, or charge transfer inefficiency, measures the fraction of charge that fails to transfer from the image area to the readout register during image readout. Consequences of high CTI include loss of charge, distorted signals in the direction of the parallel register, and reduced sensitivity in low light imaging. CTI measurements are made using the EPER method [EPER], which compares the ratio of the residual charge in the overscan pixels to the total signal charge in the imaging region. In the context of LSSTCam, we measure CTI along both the serial and parallel registers.

Serial CTI

The CTI along the serial register is consistent between both Run 6 and Run 7. Both sensor types show extremely low CTI on the order of 1E-3 %, and differ on the order of ~2E-5 % for E2V sensors, and by ~4E-6 % for ITL sensors.

Parallel CTI

The CTI along the parallel register is consistent between both Run 6 and Run 7. Both sensor types show extremely low CTI on the order of 1E-5 %, and differ on the order of ~2E-7 % for E2V sensors, and by ~7E-6 % for ITL sensors.

Dark metrics

Dark current

Dark current is the small amount of electrical charge generated in the absence of light due to thermal activity within the CCD’s semiconductor material. This effect occurs when thermal energy causes electrons to be released from atoms in the CCD, mimicking the signal that light would produce. Dark current increases with temperature, so cooling the CCD is a common method to reduce it in sensitive imaging applications. Dark current introduces noise into an image, degrading its quality, particularly in low-light conditions or long exposures. In the context of LSSTCam, we measure dark current from the combined dark images across all amplifiers.

Surprisingly, dark current was significantly lowered in Run 7 compared to run 6. Possible reasons for this could be improved shrouding conditions on the camera on Cerro Pachon compared to SLAC.

Bright defects

Bright defects are localized regions or individual pixels that produce abnormally high signal levels, even in the absence of light. These defects are typically caused by imperfections in the CCD’s semiconductor material or manufacturing process. Bright defects can manifest as “hot pixelspixels with consistently high dark current), small clusters of pixels with elevated output, or as “hot columns” (pixels along the same parallel register that have high dark current). In the context of LSSTCam, we extract bright pixels from the dark current, with the threshold for a bright defect set at 5 e- / pix / s, above which the pixel is registered as a bright defect.

Reviewing the differences in bright pixels, we find consistent bright defect counts between Run 6 and Run 7. There appears to be a small excess of bright defects in Run 7.

Taking the difference of defect counts on each amplifier, and separating the amplifiers by the detector manufacturer shows a small excess of bright defects in run 7 when compared to run 6. For ITL sensors, we find 12% of the amplifiers with more bright pixels than run 6. For E2V sensors, we find 4% of the amplifiers with more bright pixels than run 6. Despite this, the number of bright defects between runs does not increase for most sensors.

Flat pair metrics

Linearity and PTC turnoff

Linearity turnoff and PTC turnoff are two closely related metrics used to characterize the upper limit of the usable signal range for accurate imaging. Linearity turnoff is the point at which LSSTCam deviates from linearity in the PTC curve. In our case, the deviation threshold is 2%. PTC turnoff refers to the high signal region of the PTC where the PTC begins to decrease noise for higher flux. This is due to blooming and saturation within the CCDs. While slightly different, both metrics provide important information about the upper limits of the dynamic range in our sensors. Linearity turnoff is measured in units of e-, while PTC turnoff is measured in ADU.

In our linearity turnoff measurements, we find close agreement between our Run 7 and Run 6 measurements. Both ITL and E2V sensors show tight agreement between results.

PTC Gain

PTC gain is the conversion factor between the number of electrons generated in the CCD’s pixels and the digital output signal. It is one of the key parameters derived from the Photon Transfer Curve, as it is the slope from where the noise is dominated by shot noise. Gain is expressed in e- / ADU, and quantifies how effective the CCD’s analog signal is digitized.

PTC gain measurements agree extremely closely across all sensors in the focal plane.

Brighter fatter a_00 coefficient

This redistribution causes the charge to “spillnto adjacent pixels, effectively broadening the point spread function (PSF). The brighter fatter effect is the most dominant source of variance in the PTC curve. The brighter-fatter effect in CCDs refers to the phenomenon where brighter pixels appear larger (or “fatterthan dimmer ones. This occurs due to electrostatic interactions within the CCD, when a pixel accumulates a high charge from incoming photons and creates an electric field that slightly repels incoming charge carriers into neighboring pixels. The brighter fatter effect can be modeled as the most dominant source of pixel-pixel correlations. Following the PTC model from [Astier], a00 describes the change of a pixel area due to its own charge content, or the relative strength of the brighter-fatter effect. Since same-charge carriers repel each other, this pixel area has to shrink as charge accumulates inside the pixel, which implies a00 < 0. In eo_pipe, an absolute value is taken of the a_00 parameter, so the measurements appear positive.

Comparing the results on the strength of the brighter fatter effect, both runs are generally comparable. A few outliers exist across the focal plane, regardless of detector type.

However, the differences in brighter fatter strength between run 6 and run 7 show that the strength of the A_00 coefficient decreased for most of our outliers, which implies an improvement in focal-plane performance

Divisadero Tearing

Divisadero tearing are large signal variations at amplifier boundaries. To quantify divisadero tearing, we measure the column signal, and compare it to the mean column signal from flat fields to quantify the amplitude of the effect, measured in a percent variation relative to the mean column signal value.

Divisadero tearing in E2V CCDs appears higher in Run 7 than Run 6. ITL sensors are very consistent between runs.

Run 7 shows a ~0.3% excess in divisadero tearing for E2V sensors, compared to an excess of ~0.1% excess in run 6 divisadero tearing for ITL sensors.

Dark defects

Dark defects are localized regions or individual pixels that produce abnormally low signal levels, even in the presence of light. These defects are typically caused by imperfections in the CCD’s semiconductor material or manufacturing process. In the context of LSSTCam, we extract dark pixels from combined flats, with the threshold for a dark defect set to a 20% deviation from flatness.

Dark pixels measures between SLAC and Cerro Pachon average ~1800 per amplifier, regardless of manufacturer. The reason for the high dark pixel counts is due to a picture-frame response near the edges of the sensors.

Due to the contamination of the edge frame response, it is difficult to extract useful information about the dark defects in the focal plane. The configuration for generating dark defects considers a border pixel region that is masked differently from the dark pixels. The default configuration has a border of zero. The largest region allowed for the picture frame region is 9 pixels, determined by LCA-19363. Due to incompatibility with the current pipelines, a direct comparison of a 9 pixel mask using run 6 data is not currently available. However, a 9 pixel mask can be applied to the Run 7 data.

Add conclusion when pipelines on E1071 are complete

Persistence

Persistence is a feature in LSSTCam where charge is trapped in the surface layer after high flux exposures [Persistence]. Persistence is described in detail in the persistence optimization section. Here we will consider the measurements taken as part of a persistence measurement task in the typical B protocol. For a persistence measurement, a high flux acquisition is taken, followed by a sequence of dark images. The persistence signal has been shown to decrease in subsequent dark images. To create a metric for persistence, one can take the difference between the residual ADU in the first dark image and the average of the residual ADU in the final dark images.

In the initial run 7 measurements, we have not changed any operating parameters of LSSTCam, so we would expect persistence to still be present in the focal plane.

Both runs show a consistent persistence signal in E2V sensors. Several outliers exist with higher persistence signal in Run 7. The outliers in these measurements are due to higher initial persistence signal measurements, resulting in an excess of ~5 ADU when comparing run 6 with run 7.

Differences from previous runs

I will add this once we have agreed upon the set of parameters important for characterization

[Persistence]

https://dmtn-276.lsst.io/

[Astier]

https://www.aanda.org/articles/aa/abs/2019/09/aa35508-19/aa35508-19.html

[EPER]

https://www.spiedigitallibrary.org/journals/Journal-of-Astronomical-Telescopes-Instruments-and-Systems/volume-7/issue-4/048002/Characterization-and-correction-of-serial-deferred-charge-in-LSST-camera/10.1117/1.JATIS.7.4.048002.full

Final Characterization

Background

For final characterization, we compared the initial Cerro Pachon runs to our final acquistions with the camera operating parameters described in the final operating parameters section.

For analysis of the initial Cerro Pachon EO run and the final Cerro Pachon EO run, we used the following runs.

Run Type	Initial Cerro Pachón Run	Final Cerro Pachón Run
B Protocol	E1071	E1071
PTC	E749	E749

Bias metrics

CTI

Bias stability

Dark metrics

Dark current

Bright defects

Stability flat metrics

Gain stability

Flat pair metrics

Linearity turnoff

PTC turnoff

Maximum observed signal

PTC Gain

Brighter fatter a_00 coefficient

Brighter-fatter correlation

Row means variance

PTC Noise

Divisadero Tearing

Dark defects

Persistence

Differences from previous runs

Camera Optimization

Persistence optimization

Leftover signal in the following dark after a blast of illumination has been observed. It is called “Persistence”. Persistence has been observed in an early prototype E2V sensor as early as 2014 ([D2014]). It was confirmed that the amplitude of the persistence decreased as the parallel swing voltage got smaller. This is consistent with the Residual Surface Image [J2001] – the excessive charges are being stuck at the surface layer. The level of persistence is about 10–20 ADU, and the decaying time constant is about 30 sec [dmtn-276].

During the EO testing in 2021, we also found the persistence made a streak toward the readout direction from the place where the bright spot located in a previous image. We call this trailing persistence.

E2V sensors have another major problem, so-called “tearing”, which is considered a consequence of the non-uniform distribution of holes. Our primary focus in the optimization was given to mitigate the tearing over years, and we have successfully eliminated the tearing by bringing the E2V voltage from the unipolar voltage (both parallel rails high and low are positive) to the bipolar voltage (the parallel high is positive, and the low is negative) following the formula [Bipolar]. However, the persistence issue still remained unchanged.

For the persistence issue, if this is the residual surface image, two approaches could be taken as discussed in [U2024]. Either 1) establishing the pinning condition where the holes make a thin layer at the front surface so that the excessive charges recombine with the holes or 2) narrowing the parallel swing so that the accumulated charges in the silicon do not get close to the surface state.

The pinning condition could be established by bringing the parallel low voltage down as low as -7V or lower. The transition voltage needs to be empirically determined. However, E2V pointed out that the measured current flow increases as the parallel low voltage goes low, which increases the risk of damaging the sensor by making a breakdown [1]. Also, the excessive charges could get recombined by the thin layer of the holes, which could disturb the linearity at the high flux end where charges start to interact with the holes.

The parallel swing determines the fullwell. Depending on whether the accumulated charges spread over the columns or interact with the surface layer, there are blooming fullwell regimes and the surface fullwell regime. The fullwell between these two regimes is considered as the optimal fullwell [J2001], where we don’t have persistence and as high dynamic range as possible. Seeing the persistence, we likely operate the sensor in the surface fullwell condition and we need to go to a narrower voltage to get the blooming fullwell or the optimal fullwell. The obvious downside is to narrow the fullwell.

The voltages are defined relative to each other. Changing the parallel swing (for example) also requires changes to all other voltages to operate the sensor properly, for example, properly reset the amplifier. The initial voltage was given in the original formula [Bipolar] but to go to the narrow voltage we had to switch to the new formula in order to satisfy constraints [PersistenceMitigationVoltage].

[S2024], set up a single sensor test-stand at UC Davis. They attempted multiple different approaches mentioned above and reported the results [DavisReport]. The summary is as follows:

• The new voltages following the rule work fine.

• Narrowing the parallel swing eliminates the persistence.

• Lowering the parallel low voltage didn’t seem to work as we expected; the going further negative voltage is probably needed.

Note that the UCD setup didn’t show up the persistence. It might be due to the characteristic of the sensor, or might be due to the difference in the electronics (the long cable between CCD and REB, for example). They need to move the parallel rails up.

[1]

We note that ITL operates at the parallel low voltage of -8V. We have observed the increased current flow. But we have the software protection so that the current flow does not go too high.

[dmtn-276]

https://dmtn-276.lsst.io

[Bipolar] (1,2)

https://github.com/lsst-camera-dh/mkconfigs/blob/master/newformula.py

[PersistenceMitigationVoltage]

https://github.com/lsst-camera-dh/e2v_voltages/blob/main/setup_e2v_v4.py

[DavisReport]

https://docs.google.com/document/d/1V4o9tzKBLnI1nlOlMFImPko8pDkD6qE7jzzk-duE-Qo/edit?tab=t.0#heading=h.frkqtvvyydkr

[J2001] (1,2)

https://www.spiedigitallibrary.org/ebooks/PM/Scientific-Charge-Coupled-Devices/eISBN-9780819480392/10.1117/3.374903

[D2014]

https://ui.adsabs.harvard.edu/abs/2014SPIE.9154E..18D/abstract

[U2024]

https://ui.adsabs.harvard.edu/abs/2024SPIE13103E..0WU/abstract

[S2024]

https://ui.adsabs.harvard.edu/abs/2024SPIE13103E..21S/abstract

Persistence optimization

Based on this test result, we decided to try out the new voltage with the narrower voltage swing on the main camera focal plane. Keeping the parallel low voltage at -6V in order to operate the sensor safely (very conservative limit), we changed the parallel swing voltage from 9.3V to 8.0V as well as all the other voltages using the new formula. We overexposed CCDs and took 20 darks after. The image shown below is the mean or median of pixel-by-pixel difference between the first and the last dark exposures, as a function of the parallel swing. As the parallel swing is lowered, the residual signal becomes small; it becomes roughly 10 times lower than the original 9.3V. Although we sampled midpoints between 8.0 and 9.3V, 8.0V appears to work the best and could be lower with the penalty of losing the full well.

Fig. 11 The remaining charges measured in every amplifier but aggregated by mean or median as a function of the parallel clock swing are shown.

The following figures display how the persistence is reduced by the voltage change. The images were processed by the standard instrumental signature removal and get assembled in the full focal-plane view. The dark exposure was taken right after the 400ke-equivalent flat exposure. The figure shows the distinct pattern of elevated signal associated with the vendor. The inner part of the focal plane is filled by e2v sensors which have the persistence signal.

Fig. 12 The first dark exposure after a 400k flat image under the parallel swing of 9.3V (Run E1110).

The next figure shows the same dark exposure but taken with the narrow parallel swing voltage of 8.0V. The distinct pattern goes away. This demonstrates the persistence in e2v sensors becomes the level of ITL’s ones.

Fig. 13 The first dark exposure after a 400k flat image under the parallel swing of 8.0V (Run E1880). The figure shows no distinct patterns from persistence in e2v sensors anymore. Note that the guider sensors were not displayed here because they were in the guider mode. Also some of residuals in ITL caused by defects disappeared because of the employment of the new sequencer file (v30).

Impact on full-well

Reduction of the full well is expected by narrowing the parallel swing voltage. This subsection explores how much reduction in the PTC turnoff is observed in the dense PTC run. Two runs are acquired with identical setting except for the CCD operating voltage (E1113 for 9.3V and E1335 for 8.0V). As the PTC turnoff is defined in ADU, it needs to be multiplied by PTC_GAIN to make a comparison. The figure below compares the PTC turnoff in electrons and their difference in ratio. The median reduction was 22% .

Fig. 14 Histograms of the PTC turn offs (left) and the ratios of differences (right) between E1113 (9.3V) vs E1335 (8.0V). The median of the reduction is 22%.

Impact on Brighter-Fatter effect

Reducing the parallel swing is expected to enhance the brighter-fatter effect (BFE), possibly in an anisotropic way. The BFE can be characterized via the evolution of the variance and covariances of flatfield exposures as a function of flux. In order to evaluate the impact of reducing the parallel voltage swing on e2v sensors, we acquired two series of flatfield exposures with the respective voltage setups and extracted the “area” coefficients the “area” coefficients (Equation (1) in [A2023]) from these two data sets. The area coefficients describe by how much a unit charge stored in a pixel wil alter the area of some other pixel (or itself). We find that reducing the parallele swing from 9.3V to 8V typically increases the area coefficients by 10% (between 5 and 19% depending on distance), and the increase is almost isotropic (along serial and parallel directions). From these measurements, we anticipate that the increase of star sizes with flux will not become more isotropic at 8V than it was at 9.3V, and hence does not introduce new threats on the measurement of the PSF ellipticity

Fig. 15 Scatter plots of area coefficients (one entry per amplifier) measured at 8V and 9.3V. The 9 subfigures correspond to separations between the source of the area distortion and its victim, with the self interaction at the bottom left. The first neighbors increase respectively by 19% in the parallel direction by 14% in the serial direction. So the BFE is slightly larger at 8V but not significantly more anistropic.

[A2023]

https://arxiv.org/pdf/2301.03274

Summary

E2V sensors had persistence. We confirmed changing the E2V CCD operating voltage greatly reduced persistence. As penalties, we observed 22% of full well reduction, and a ~10% increase of the brightter-fatter effect, essentially in an isotropic way.

Sequencer Optimization

Sequencer files have undergone evolution for both ITL and e2v versions. The latest sequencer file from Run 6 was the v26_noRG version for ITL and the regular v26 for e2v. The suffix _noRG indicates that the RG bit is not toggled during parallel transfer. This modification appears to enhance the stability of the bias structure for most ITL amplifiers.

During Run 7, several changes were implemented, as described below:

• v27 incorporated guider functionalities, including ParallelFlushG and ReadGFrame. However, the noRG change was inadvertently included. Consequently, we abandoned this version and switched to v28.

• v28 sequencer files merged v26_no_RG and v27. https://rubinobs.atlassian.net/browse/LSSTCAM-5

• v29 introduced changes to speed up the guider. https://rubinobs.atlassian.net/browse/LSSTCAM-34

• v30 primarily focused on e2v. We introduced a new approach to NopSf for e2v sensors https://github.com/lsst-camera-dh/sequencer-files/pull/17. To align timing with the ITL version, a change was made. https://github.com/lsst-camera-dh/sequencer-files/pull/18

This section describes sequencer optimization.

• CCD clear conclusions

  The complete discussion is given in Improvement of Clear CCD but here is a summary:

  • No Pocket

  We introduced the V29_NoP (No Pocket) sequencer, which is an improved clear with a serial register configuration that reduces the creation of pockets at the Image/Serial register interface. This clear shown a factor ~2 improvement in saturated image clear for e2v, and fully solved the problem for ITL, except for R01_S11 for which the result with No Pocket appends to be worse by a factor 2 than with the default clear. This ITL ccd presents for a not understood reason, a large quantities of uncleared charges(100’s of lines) after a saturated flat. This issue prevents to use the No Pocket configuration with ITL.

  • No Pocket with Serial flush

  We introduced in V29_NoPSF ( No Pocket with Serial Flush), an improved version of the No Pocket Clear sequencer, including a variable configuration of the Serial register during the clear (mimicking a serial flush), to further prevent the formation of pockets. This solution has been shown to completely prevent the presence of leftover charges after the clear of a saturated image for e2v devices.

  No Pocket with Serial flush is now the default clear method for e2v devices starting with V30. Until a solution is found for clearing saturated images in R01_S10, the initial clear method (Serial phases always up) will remain the default for ITL devices.

• Overlap conclusions

• e2v “No RG” conclusions

Thermal Optimization

hello world.

This section describes thermal optimization.

• Background

• Idle flush off & it’s stability

• impact on other parameters

Characterization & Camera stability

The final result of B protocol and PTC need to be presented here.

Final Characterization

Background

For final characterization, we compared the initial Cerro Pachon runs to our final acquistions with the camera operating parameters described in the final operating parameters section.

For analysis of the initial Cerro Pachon EO run and the final Cerro Pachon EO run, we used the following runs.

Run Type	Initial Cerro Pachón Run	Final Cerro Pachón Run
B Protocol	E1071	E1071
PTC	E749	E749

Bias metrics

CTI

Bias stability

Dark metrics

Dark current

Bright defects

Stability flat metrics

Gain stability

Flat pair metrics

Linearity turnoff

PTC turnoff

Maximum observed signal

PTC Gain

Brighter fatter a_00 coefficient

Brighter-fatter correlation

Row means variance

PTC Noise

Divisadero Tearing

Dark defects

Persistence

Differences from previous runs

Guider operation

hello world.

This section describes guider operation.

• initial guider operation

• power cycling the guiders to get to proper mode

• synchronization

• guider roi characterization

Defect stability

hello world.

This section describes defect stability.

• Bright defects

• Dark defects with picture frame

Bias stability

Bias instabilities (typically above the 1-ADU level) are observed over a significant number of sensors for both ITL and e2v CCDs. The main issues are referred as:

1. The ITL bias jumps : large variation of the column-wise structure from exposure to exposure.

2. The e2v yellow corners : residual 2D shape of the bias even after 2D-overscan correction. These residuals depend on the acquisition sequence and of the exposure time.

Both issues were observed and deeply studied in Run 6 EO data. The ITL issue is believed to be phase shifts in clocks between Readout Electronics Boards (REB) because REBs rely on the frequency converted from their natural frequency. We tried to mitigate the e2v issue by optimizing the acquisition configuration in Run 7.

For the baseline acquisition configuration (see conclusion), three relevant stability runs were recorded:

1. Run E2136: 15s darks with some very long delays throughout the run

2. Run E2236: 50 15s darks, 50 biases recorded with 30s delays between exposures

3. Run E2330: 15s and 30s darks with variables delays between exposures

To process these runs, the eo_pipe bias stability task is used: for the ISR part, a serial (‘mean_per_row’) overscan correction and a bias subtraction (computed from the corresponding B-protocol run) are applied. The final data product is the mean of the per-amplifier science image over the full set of exposures of the run. Two typical examples from Run E2136 are shown in the figures below. In the stable case, the variations are typically at the 0.1 ADU level; in the instable case, the variations go up to 4 ADUs.

Fig. 16 Stable case (R21 S21)

Fig. 17 Instable case (R23 S22)

A comparison of the results for an instable CDD is shown below for the three runs.

Fig. 18 Run E2136, R33 S02

Fig. 19 Run E2236, R33 S02

Fig. 20 Run E2330, R33 S02

In order to highlight the 2D shape differences, a 2D-overscan correction is applied. A few exposures illustrating the variations of the 2D shape for an instable CCD are shown below. The 2D shape of the image in amplifier C01 is different in the 3 cases.

Fig. 21 Bias exposure, run 1880, R33 S02

Fig. 22 15-s dark exposure, run E2136 in ‘stable’ conditions, R33 S02

Fig. 23 15-s dark exposure, run E2136 after a 3-minute delay, R33 S02

In order to quantify the number of e2v instable amplifiers, a stability metric d is defined from the eo_pipe stability task data products. More precisely, d is defined, for a given amplifier in a given run, as the difference between the 5th and 95th percentiles of the image mean over all the exposures. The distribution of d for run E2136 is shown below. Applying a threshold at 0.3, 51 amplifiers are identified as instable (see the corresponding mosaic). This corresponds to ~3% of the e2v amplifiers.

Fig. 24 Distribution of the stability metric for the e2v amplifiers in run E2136

Fig. 25 Mosaic of e2v amplifiers identified as instable (white color) in run E2136

Further studies are required in order to converge on the best mitigation strategy for the start of the LSST survey.

Gain stability

hello world.

This section describes gain stability.

• No temp variation, fixed flux

• no temp variation, variation in flux

• Temp variation, fixed flux

Sensor features

Tree rings

hello world.

This section describes tree rings.

• Tree rings without diffuser

• Tree rings with diffuser

ITL Dips

One of the phenomenon that was stuided in the later part of Run 7 was ITL dips. These were first discovered in LSST ComCam with on-sky data as bleed trails from bright stars that traversed the entire detector, jumping over the amplifier boundaries. These bleed trails are unique though in that the core of the bleed trail is actually ‘dark’ compared to the wings of the bleed trail, with a lower flux of around 2% compared to the rest of the bleed trail.

We tried to investigate if there were any ITL dips in the sensors of LSSTCam. For this study, we used spots and rectangles created by a 4K projector onto the focal plane. The spots were roughly 30 pixels across and were in every amplifier of each detector. The rectangles were only in the top right amplifier (C10). One unique feature with this spot projection was that there was a background illumination caused by the projector. This led to the spots having a signal only 6 times higher than the background and the rectangles with a signal 30 times higher than the background.

Investigating these images, we were not able to find any evidence of ITL dips. Below are the images themselves along with binned horizontal cutouts of the the amplifier below the source. These show the background pattern of the projector, but no 2% dip.

While we were not able to find evidence of the ITL dip in Run 7 data, it is still not clear if this will not be visible in LSSTCam on-sky data. The photon rate of the in-lab data was roughly XXX per second for the 15 second exposures. The stars that were seen in ComCam with the ITL dip have a magnitude of XXX corresponding to a photon rate of XXX. This is combined with a sky background of XXX as compared with the lab sensor background of XXX.

Vampire pixels

First observations

Vampire pixels were first observed in ComCam observations [need more info to properly give context] - Andy’s study on Oct. 8 - Agnes masking effort

LSSTCam vampire pixel features

The vampire pixels have distinct features, both on the individual defect level, and across the focal plane

Individual vampire features

• General size

• Radial kernel

• uniformity

Vampire features across the focal plane

• sensor type

• static or dynamic

• higher concentrations? Particularly bad sensors?

Current masking conditions

• Bright pixels

• Dark pixels

• Jim’s task

Analysis of flats

• LED effect

• Intensity effect

Analysis of darks

• Previous LED effect

• Intensity of LED effect

• dark cadence and exposure times

Current models of vampires

• Tony & Craig model

• Others?

Improved Clear

Overview

In this section we will describe the work done during Run 7 to improve the image clear prior to collect a new exposure.

The problem we wanted to address is the presence of residual charges in the first lines read for image taken just after the clear of a saturated image. These “hard to clear” charges , are associated to highly saturated flat or column(s) (or stars as observed in AuxTel or ComCam), that leave signal in the first lines of the following exposure. We have the following signature of the effect:

• in all ITL CCD (except in R01_S10 for which the effect is much more significant and that will be addressed later in this section):

  the first CCD line of an exposure read after an image with saturated overscan, is close to saturation and in most of the case there is also a small left over signal in the 2nd line read.

• in e2v CCD:

  the effect is slightly amplifier dependent, still, like in ITL, the first line read in an exposure following an exposure with saturated overscan, is close to saturation, and a significant signal is visible in the following 20-50 lines. ( see left plots of clear e2v image)

These left over electrons are not associated to what we usually call residual image or persistence. They are suspected to be associated to pockets, induced by the electric field configuration in the sensor and the field associated to saturated pixels: pocket(s) that survive to a clear, will prevent charges to be cleared. A change of the electric field (ex: a change in clocks configuration) can remove the pockets, and free the charges, allowing them to be cleared. If charges stuck in pocket(s) are not removed by a clear, we observed that an image read (ex: a bias) will fully remove them: only the first exposure taken after an image with saturated overscan is impacted. If the clocks configuration used in our standard clear is not able to flush away those charges, a standard readout of >~ 2000 lines does remove them.

The localisation of these uncleared electrons in the first lines of the CCDs, spots the interface between the image area and the serial register as the location for those pockets. For this reason we investigated changes in the field configuration of the serial register during the clear, to avoid pockets at this image-serial register interface.

New sequencers

To addresse this clear issue, we focussed on updating the serial register field as the lines are moved to it. The constrain being that the changes introduced should not significantly increase the clear execution time. It should be notice that we tried in 2021 a sequencer called “Deep Clear” ( [sequencerV23_DC] ) as a first try to address the clear issue: it added one full line flush on top of the existing one at the end of the clear. This sequencer did improve the clear, still not fully fixing the clear issue ( see Summary table).

In the Run 7, We considered on top of the default clear, 2 new configurations. The changes are in the ParallelFlush function, which move the charges from the image area to the serial register:

• the default clear (V29): In the default clear, all serial clocks are kept up as the parallel clocks move charges from the image area to the serial register ([sequencerV29]). The charges once on the serial register will hopefully flow to the ground: the serial register clocks being all up, without pixels boundary, and with its amplifier in clear state. At the end of the clear, a full flush of the serial register is done (~ the serial clocks changes to read a single line ).

• the No-pocket Clear (NoP): a clear where the serial register has the same configuration (S1 & S2 up, S3 low) when the parallel clock P1 moves the charges to the serial register than in a standard image read . Still we keept all phases up the rest of the time for a fast clear of the charges along the serial register ([sequencerV29_NoP]). The idea is that the S3 phase is not designed to be up when charges are transfered to the serail register, and is probably playing a major role in the pockets creation.

• The No-pocket with serial flush Clear (NoPSF): this sequencer is close to the NoP solution , except that during the transfered of 1 line to the serial register, the serial phases are also moved to transfer two pixels along teh serial register. The changes in electric field at the image-serial register interface are then even more representative to what a standard read will produce, and should further prevent the creation of pockets. ([sequencerV29_NoPSF]).

[sequencerV23_DC]

https:parallelgithub.com/lsst-camera-dh/sequencer-files/blob/master/run5/FP_E2V_2s_ir2_v23_DC.seq

[sequencerV29]

https:parallelgithub.com/lsst-camera-dh/sequencer-files/blob/master/Run 7/FP_E2V_2s_l3cp_v29.seq

[sequencerV29_NoP]

https:parallelgithub.com/lsst-camera-dh/sequencer-files/blob/master/Run 7/FP_E2V_2s_l3cp_v29_Nop.seq

[sequencerV29_NoPSF]

https:parallelgithub.com/lsst-camera-dh/sequencer-files/blob/master/Run 7/FP_E2V_2s_l3cp_v29_NopSf.seq

Results on standard e2v and itl CCD

Figure showing the impact of the various types of clear on a bias taken after a saturated flat for an E2V sensor.

Figure showing the impact of the various types of clear on a bias taken after a saturated flat for an ITL sensor.

In the above images, we present for 3 types of sequencer (from left to right: V29, NoP and NopSF), a zoom on the first lines of an itl or e2v amplifier (for itl R03_S11 C14 and for e2v R12_S20 C10 ) shown as a 2D lines-columns image (top plots) or as the mean signal per line for the first lines read of an amplifier (bottom plots).

As seen in see left plots of clear e2v image for an e2v CCD, a bias taken just after a saturated flat will show a residual signal in the first lines read when using the default clear (left images,clear=v29): the first line has an almost saturated signal (~ 100 kADU here), and a significant signal is seen up to the line ~50. In practice, in function of the amplifier, signal can be seen up to line 20-50. When using the NoP clear (central plots), we can already see a strong reduction of the uncleared charges in the first acquired bias after a saturated flat, still a small residual signal is visible in the first ~20 lines. The NoPSF clear (right plots) fully clear the saturated flat, and no uncleared charges are observed in the following bias.

As seen in see left plots of clear itl image for an itl CCD, a bias taken just after a saturated flat will show a residual signal in the first lines read when using the default clear (left images,clear=v29) : the first line has an almost saturated signal (~ 100 kADU here), and a significant signal is seen in the following line. Both NoP clear (central plots) and NoPSF clear (right plots) fully clear the saturated flat, and no uncleared charges are observed in the following bias.

Results on itl R01_S10

Figure showing the impact of the various types of clear on ITL R01_S10 after a saturated flat (bias after a saturated flat), from left to right: 1 standard Clear, 3 standard Clear, 5 standard Clear, 1 NoP Clear, 1 NoPSF Clear

There is one ITL sensor, R01_S10, that presents a specific and non-understood behavior:

• It has a quite low full well (2/3 of nominal)

• The 3 CCD of this REB have a gain 20% lower than all other ITL CCD?

• The images taken after a large staturation, as seen in figure clear in itl R01_S10, show a large amount of uncleared charged (with the standard clear: 4 amplifiers with ~500 lines of saturated signal!)

It apears that putting S3 low during the clear as done in NoP or NoPSF, is even worse than a standard clear. This is strange as a full frame read, which does this too, manages to clear such image. We can notice that NoPSF is ~50% better than NoP, but still worse than the standard clear, in particular for the 12 amplifiers almost correct with the standard clear.

At this stage we don’t have a correct way to clear this sensor once it collects a saturated flat, but It’s not known if a saturated star in this sensor, leaving signal in the parallele overscan, will presents the same clear issue.

Conclusion

Table 2 This table summaries the different clear methods used so far.

	Default Clear 1 Clear (seq. V29)	Multi Clear 3 Clears (seq. V29)	Multi Clear 5 Clears (seq. V29 )	Deep Clear 1 Clear (Seq. V23 DC)	No Pocket(NoP) 1 Clear (seq. V29_NoP)	No Pocket Serial Flush(NoPSF) 1 Clear ( seq. V29_NoPSF, V30 )
Clear duration	65.5 ms	196.5 ms	327.4 ms	64.69 ms	65.8 ms	67 ms
“E2V” after saturated Flat	1st line saturated signal up to line 50	No residual electrons	No residual electrons	1st line saturated signal up to line <20	signal up to line 20	No residual electrons
“ITL” after saturated Flat	1st line saturated signal up to 2nd line	No residual electrons	No residual electrons	tiny signal left in the first line	No residual electrons	No residual electrons
R01_S10 ITL “unique”	first 500 lines saturated for 4 amp. 13 amp. with signals.	first 150 lines saturated for 2 amp. 5 amp. with signals.	first 100 lines saturated for 2 amp. 2 amp. impacted	not measured	first 1000 lines saturated for 16 amp. 16 amp. with signals.	first 750 lines saturated for 16 amp. 16 amp. with signals.

Even if NoP or NoPSF are overcoming the clear issue we had with ITL sensors, the exception of R01_S10 prevented the usage of those sequencers for ITL device for the Run 7. Notice that beyond R01_S10 the numbers of line potencilly “not cleared” are small (2 first lines) in ITL device, and they correspond to a CCD area hard to use anyway (sensor edges with low efficciency). So at this stage the default clear is still our default for ITL, and further studies to overcome the problem with R01_S10 are forseen (ex: do a continuous serial flush during exposure at low rate, 10^6 pixels flush in 15s).

On the other side, after those studies in Run 7, we now have a good way to fully clear the e2v devices through the NopSF clear. The NoPSF clear grants that the first 50 lines of e2v device that had un-cleared electrons from the previous exposure, are now free of such contamination.

From now:

• for e2v, NoPSF will be the default clear method

• for ITL, the origial clear (serial phase 3 always), slightly extended in time to match the NoPSF e2v clear execution time, will stay the default method.

Phosphorescence

hello world.

This section describes phosphorescence.

• phosphorescence background

• phosphorescence on flat fields

• phosphorescence on spot projections

Conclusions

Run 7 final operating parameters

This section describes the conclusions of run 7 optimization and the operating conditions of the camera. Decisions regarding these parameters were decided based upon the results of the voltage optimization, sequencer optimization, and thermal optimization.

Voltage conditions

Table 3 Voltage conditions

Parameter	dp80 (new voltage)	dp93 (Run 5)
pclkHigh	2.0	3.3
pclkLow	-6.0	-6.0
dpclk	8.0	9.3
sclkHigh	3.55	3.9
sclkLow	-5.75	-5.4
rgHigh	5.01	6.1
rgLow	-4.99	-4.0
rd	10.5	11.6
od	22.3	23.4
og	-3.75	-3.4
gd	26.0	26.0

Sequencer conditions

Table 4 Sequencer conditions

Detector type	File name
E2V	FP_E2V_2s_l3cp_v30.seq
ITL	FP_ITL_2s_l3cp_v30.seq

• v30 sequencers are identical to the FP_ITL_2s_l3cp_v29_Noppp.seq and FP_E2V_2s_l3cp_v29_NopSf.seq. All sequencer files can be found in the github repository.

Other camera conditions

• Idle flush disabled

Record runs

This section describes run 7 record runs.

All runs use our camera operating configuration, unless otherwise noted.

Table 5 Record runs

Run Type	Run ID	Links	Notes
B protocol	E1880
	E2233		Identical to E1880. Acquired after CCS subsystem reboot
PTCs	E1886		Red LED dense. Dark interleaving between flat pairs
	E1881		Red LED dense. No dark interleaving between flat pairs
	E748		nm960 dense
	E2237		Red LED dense. Acquired after CCS subsystem reboot.
	E2016		Super dense red LED. HV Bias off for R13/Reb2. jGroups meltdown interrupted acquisitions, restarted
Long dark acquisitions	E1117
	E1116
	E1115
	E1114
	E1075
Projector acquisitions	E1558		Flat pairs, fine scan in flux from 1-100s in 1s intervals. E2V:v29_NoP, ITL:v29_NoPP
	E1553		Flat pairs, coarse scan in flux from 5-120s in 5s interval.E2V:v29_NoP, ITL:v29_NoPP
	E1586		One 100s flat exposure, spots moved to selected phosphorescent regions.E2V:v29_NoP, ITL:v29_NoPP
	E2181		Flat pairs from 2-60s in 2s intervals. Two 15s darks interleaved after flat acquisition. Rectangle on C10 amplifier.E2V:v29_NoP, ITL:v29_NoPP
	E2184		10 30s dark images to capture background pattern
OpSim runs	E1717		Long dark sequence, no filter changes
	E2330		Short dark sequence, filter changes in headers through OCS
	E1414		30 minutes OpSim run with shutter control, filter change, and realistic survey cadence
	E2328		Flats with shutter-controlled exposure
	E1657		10 hour OpSim dark run, ~50% of darks were acquired properly
Phosphorescence datasets	E2015		10 flats at 10ke- followed by 10x15s darks
	E2014		1 flat at 10ke- followed by 10x15s darks
	E2011		20 flats at 10ke- followed by 10x15s darks
	E2012		10 flats at 1ke- followed by 10x15 s darks
	E2013		10 flats at 10ke- followed by 10x15s darks. Interleaved biases with the darks
Tree ring flats	E1050
	E1052
	E1053
	E1055
	E1056
	E1021
	E1023
	E1024
	E1025
	E1026
Gain stability runs	E1955
	E2008
	E1968
	E1367
	E1362
	E756
	E1496
Persistence datasets	E1503
	E1504
	E1505
	E1506
	E2286
	E1502
	E1501
	E1500
	E1499
	E1498
	E1494
	E1493
	E1492
	E1490
	E1491
	E1489
	E1488
	E1487
	E1486
	E1485
	E1478
	E1477
	E1479
	E1483
	E1484
Guider ROI acquisitions	E1510
	E1518
	E1519
	E1508
	E1509
	E1520
	E1511
	E1521
	E1512
	E1513
	E1514
	E1517