SITCOMTN-148

LSST Camera Electro-Optical Test Results#

Abstract

This note collects results from the LSST Camera electro-optical testing prior to installation on the TMA. We describe the CCD and Focal Plane optimization and the resulting default settings. Results from eo_pipe are shown for standard runs such as B-protocols, Dense and SuperDense PTCs, gain stability, OpSim runs of Darks, and Darks with variable delays. We also describe features such as e2v Persistence, ITL phosphorescence in coffee stains, remnant charge near Serial register following saturated images, vampire pixels, ITL dips, and others.

Electro-optical setup#

Run 7 Optical modifications#

hello world.

This section describes run 7 optical changes to the CCOB, projector, etc.

  • refresh of setup with items the same as IR2 (CCOB, no narrow-beam)

  • diffuser install

  • projector

Projector spots#

hello world.

This section describes the spots and rectangles tested with the 4k projector

  • Projector background

  • Spots on many amps

  • Spots on one amp

  • Optical setup

Dark current and light leaks#

This section describes dark current and light leaks in Run 7 testing.

One of the first tests we attempted with the camera was measuring dark current and sources of light leaks in the camera body.

Light leak mitigation with shrouding the camera body#

Successful Autochanger Light Leaks masking#

A dedicated dark/light leak study was performed during the run6 at SLAC in summer 2023 and a localized faint light source going up to ~ 0.04 e-/s/pixel was associated to the 24 V Clean of the FES auto-changer.

In the Auto-Changer this voltage is used to power some probes and all controllers. In february 2024, as AC-1 was extracted from the camera for a global maintenance, a direct investigation to localized the light source was performed without success. A light source in the AC wasn’t expected as in the AC all controllers LED have been removed, and most electronics are in “black boxes”. Still two small probes , which had LEDs that could not be removed, were initially masked by a black epoxy. As we had doubt on the quality of this masking in the IR, we applied and extra-making (aluminium black tape) on them during the Feb 2024 maintenance ( on AC 1 and 2 ).

At the start of the run 7 a new study of the light leak based on 900s dark exposures with the shutter open and the empty frame filter en place, showed that the AC light leaks was still present ( see left plots of AC light leak ). Following this finding, a full review of all the AC hardware powered by the 24 V dirty was performed, and a candidate was found : the coders of the 5 main motors of the AC had a partial documentation from the vendor not mentioning the presence of LED. After interaction with the vendor the presence of ~700nm LEDs incide the coders were reported. The hypothesis of ~700nm LED source has been found compatible with the observation as no AC light leaks were detected using the different filters in camera at the start of run 7 ( g,r and y filters) . A dedicated test in Paris using an AC spare coder and a precision photometric set-up allowed to identify leak in the masking of those LED in the vendor packaging. A complementary masking method based on a 3D printed part + tape + cable tie was qualified in Paris: it is masking the light leak and it is safe (all parts correctly secured ).

In November 2024, we masked all the lights in the back of level 3 clean room ( not the part with the camera) to setup a high quality dark room allowing a direct observation with a CMOS camera of the light leak on the AC2 motors coders. Also the level of darkness reached, allowed us to validate the quality of the AC coders light masking. Notice that the FES-prototype in Paris doesn’t have coder on the Online Clamps, so we had to tune/qualify directly on the AC2 at summit the masking of those coders.

For each AutoChanger ( 1 & 2 ), the 5 motors coders with vendor issue on their LED masking, have been successfully enveloped in a light tight mask.

Notice that the AC was off starting the Sep 27th at 21:15 UTC in the first part of the run7. For the end of run7 (run taken after mid-November) the AC was back On: as the AC 1 was back in camera with the new coders light mask in place, we were able to take a new series of 900s dark with AC On & off, confirming that we had no light leak left associated to the Filter Exchange System. (see right plots of AC light leak)

*The left plot shows the original impact of the AC light leak on 900s dark ( AC On - AC off). On the right plot, after masking the AC LED coders, no light associated to the FES is present.*

The left plot shows the original impact of the AC light leak on 900s dark ( AC On - AC off). On the right plot, after masking the AC LED coders, no light associated to the FES is present in 900s dark difference( FES On - FES off).

Shutter condition impact on darks#
Filter condition impact on darks#

Final measurements of dark current#

Reverification#

Baseline characterization#

hello world.

This section describes baseline characterization and reverification

  • first B protocols and PTCS

  • comparison to IR2 metrics

  • new features in any baseline runs?

Camera Optimization#

Persistence optimization#

Leftover signal in the following dark after a blast of illumination has been observed. It is called “Persistence”. Persistence has been observed in an early prototype E2V sensor as early as 2014 ([D2014]). It was confirmed that the amplitude of the persistence decreased as the parallel swing voltage got smaller. This is consistent with the Residual Surface Image [J2001] – the excessive charges are being stuck at the surface layer. The level of persistence is about 10–20 ADU, and the decaying time constant is about 30 sec [dmtn-276].

During the EO testing in 2021, we also found the persistence made a streak toward the readout direction from the place where the bright spot located in a previous image. We call this trailing persistence.

E2V sensors have another major problem, so-called “tearing”, which is considered a consequence of the non-uniform distribution of holes. Our primary focus in the optimization was given to mitigate the tearing over years, and we have successfully eliminated the tearing by bringing the E2V voltage from the unipolar voltage (both parallel rails high and low are positive) to the bipolar voltage (the parallel high is positive, and the low is negative) following the formula [Bipolar]. However, the persistence issue still remained unchanged.

For the persistence issue, if this is the residual surface image, two approaches could be taken as discussed in [U2024]. Either 1) establishing the pinning condition where the holes make a thin layer at the front surface so that the excessive charges recombine with the holes or 2) narrowing the parallel swing so that the accumulated charges in the silicon do not get close to the surface state.

The pinning condition could be established by bringing the parallel low voltage down as low as -7V or lower. The transition voltage needs to be empirically determined. However, E2V pointed out that the measured current flow increases as the parallel low voltage goes low, which increases the risk of damaging the sensor by making a breakdown [1]. Also, the excessive charges could get recombined by the thin layer of the holes, which could disturb the linearity at the high flux end where charges start to interact with the holes.

The parallel swing determines the fullwell. Depending on whether the accumulated charges spread over the columns or interact with the surface layer, there are blooming fullwell regimes and the surface fullwell regime. The fullwell between these two regimes is considered as the optimal fullwell [J2001], where we don’t have persistence and as high dynamic range as possible. Seeing the persistence, we likely operate the sensor in the surface fullwell condition and we need to go to a narrower voltage to get the blooming fullwell or the optimal fullwell. The obvious downside is to narrow the fullwell.

The voltages are defined relative to each other. Changing the parallel swing (for example) also requires changes to all other voltages to operate the sensor properly, for example, properly reset the amplifier. The initial voltage was given in the original formula [Bipolar] but to go to the narrow voltage we had to switch to the new formula in order to satisfy constraints [PersistenceMitigationVoltage].

[S2024], set up a single sensor test-stand at UC Davis. They attempted multiple different approaches mentioned above and reported the results [DavisReport]. The summary is as follows:

  • The new voltages following the rule work fine.

  • Narrowing the parallel swing eliminates the persistence.

  • Lowering the parallel low voltage didn’t seem to work as we expected; the going further negative voltage is probably needed.

Note that the UCD setup didn’t show up the persistence. It might be due to the characteristic of the sensor, or might be due to the difference in the electronics (the long cable between CCD and REB, for example). They need to move the parallel rails up.

Persistence optimization#

Based on this test result, we decided to try out the new voltage with the narrower voltage swing on the main camera focal plane. Keeping the parallel low voltage at -6V in order to operate the sensor safely (very conservative limit), we changed the parallel swing voltage from 9.3V to 8.0V as well as all the other voltages using the new formula. We overexposed CCDs and took 20 darks after. The image shown below is the mean or median of pixel-by-pixel difference between the first and the last dark exposures, as a function of the parallel swing. As the parallel swing is lowered, the residual signal becomes small; it becomes roughly 10 times lower than the original 9.3V. Although we sampled midpoints between 8.0 and 9.3V, 8.0V appears to work the best and could be lower with the penalty of losing the full well.

_images/e2v_transient_dark_vs_dp.png

Fig. 1 The remaining charges measured in every amplifier but aggregated by mean or median as a function of the parallel clock swing are shown.#

The following figures display how the persistence is reduced by the voltage change. The images were processed by the standard instrumental signature removal and get assembled in the full focal-plane view. The dark exposure was taken right after the 400ke-equivalent flat exposure. The figure shows the distinct pattern of elevated signal associated with the vendor. The inner part of the focal plane is filled by e2v sensors which have the persistence signal.

_images/E1110dp93.png

Fig. 2 The first dark exposure after a 400k flat image under the parallel swing of 9.3V (Run E1110).#

The next figure shows the same dark exposure but taken with the narrow parallel swing voltage of 8.0V. The distinct pattern goes away. This demonstrates the persistence in e2v sensors becomes the level of ITL’s ones.

_images/E1310dp80.png

Fig. 3 The first dark exposure after a 400k flat image under the parallel swing of 8.0V (Run E1310). The figure shows no distinct patterns from persistence in e2v sensors anymore.#

Impact on full-well#

Reduction of the full well is expected by narrowing the parallel swing voltage. This subsection explores how much reduction in the PTC turnoff is observed in the dense PTC run. Two runs are acquired with identical setting except for the CCD operating voltage (E1113 for 9.3V and E1335 for 8.0V). As the PTC turnoff is defined in ADU, it needs to be multiplied by PTC_GAIN to make a comparison. The figure below compares the PTC turnoff in electrons and their difference in ratio. The median reduction was 22% .

_images/PtcTurnoffRatio.png

Fig. 4 Histograms of the PTC turn offs (left) and the ratios of differences (right) between E1113 (9.3V) vs E1335 (8.0V). The median of the reduction is 22%.#

Impact on Brighter-Fatter effect#

Yassine will put his material here.

Summary#

E2V sensors had persistence. We confirmed changing the E2V CCD operating voltage greatly reduced persistence. As penalties, we observed 22% of full well reduction, and XXXX

Sequencer Optimization#

hello world.

This section describes sequencer optimization.

  • No-pocket conclusions

  • Overlap conclusions

  • Serial flush conclusions

Thermal Optimization#

hello world.

This section describes thermal optimization.

  • Background

  • Idle flush off & it’s stability

  • impact on other parameters

Characterization & Camera stability#

The final result of B protocol and PTC need to be presented here.

B protocol result and PTC result needs to be summarized here.

Guider operation#

hello world.

This section describes guider operation.

  • initial guider operation

  • power cycling the guiders to get to proper mode

  • synchronization

  • guider roi characterization

Defect stability#

hello world.

This section describes defect stability.

  • Bright defects

  • Dark defects with picture frame

Bias stability#

hello world.

This section describes bias stability.

  • Typical bias stability runs

  • dark delay

  • dark with bias delays

Gain stability#

hello world.

This section describes gain stability.

  • No temp variation, fixed flux

  • no temp variation, variation in flux

  • Temp variation, fixed flux

Sensor features#

Tree rings#

hello world.

This section describes tree rings.

  • Tree rings without diffuser

  • Tree rings with diffuser

ITL Dips#

hello world.

This section describes ITL Dips.

Vampire pixels#

First observations#

Vampire pixels were first observed in ComCam observations [need more info to properly give context] - Andy’s study on Oct. 8 - Agnes masking effort

LSSTCam vampire pixel features#

The vampire pixels have distinct features, both on the individual defect level, and across the focal plane

Individual vampire features#
  • General size

  • Radial kernel

  • uniformity

Vampire features across the focal plane#
  • sensor type

  • static or dynamic

  • higher concentrations? Particularly bad sensors?

Current masking conditions#

  • Bright pixels

  • Dark pixels

  • Jim’s task

Analysis of flats#

  • LED effect

  • Intensity effect

Analysis of darks#

  • Previous LED effect

  • Intensity of LED effect

  • dark cadence and exposure times

Current models of vampires#

  • Tony & Craig model

  • Others?

Improved Clear#

Overview#

In this section we will describe the work done during run 7 to improve the image clear prior to collect a new exposure.

The problem we wanted to address is the presence of residual charges in the first lines read for image taken just after the clear of a saturated image. These “hard to clear” charges , are associated to highly saturated flat or column(s) ( or stars as observed in AuxTel or ComCam), that leave signal in the first lines of the following exposure. We have the following signature of the effect :

  • in all ITL CCD (except in R01_S10 for which the effect is much more significant and that will be addressed later in this section):

    the first CCD line of an exposure read after an image with saturated overscan, is close to saturation and in most of the case there is also a small left over signal in the 2nd line read.

  • in e2v CCD :

    the effect is slightly amplifier dependent, still, like in ITL, the first line read in an exposure following an exposure with saturated overscan, is close to saturation, and a significant signal is visible in the following 20-50 lines. ( see left plots of clear e2v image )

These left over electrons are not associated to what we usually call residual image or persistence. They are suspected to be associated to pockets, induced by the electric field configuration in the sensor and the field associated to saturated pixels : pocket(s) that survive to a clear, will prevent charges to be cleared. A change of the electric field (ex: a change in clocks configuration) can remove the pockets, and free the charges, allowing them to be cleared. If charges stuck in pocket(s) are not removed by a clear, we observed that an image read (ex: a bias) will fully remove them: only the first exposure taken after an image with saturated overscan is impacted. If the clocks configuration used in our standard clear is not able to flush away those charges, a standard readout of >~ 2000 lines does remove them.

The localisation of these uncleared electrons in the first lines of the CCDs, spots the interface between the image area and the serial register as the location for those pockets. For this reason we investigated changes in the field configuration of the serial register during the clear, to avoid pockets at this image-serial register interface.

New sequencers#

To addresse this clear issue, we focussed on updating the serial register field as the lines are moved to it. The constrain being that the changes introduced should not significantly increase the clear execution time. It should be notice that we tried in 2021 a sequencer called “Deep Clear” ( [sequencerV23_DC] ) as a first try to address the clear issue: it added one full line flush on top of the existing one at the end of the clear. This sequencer did improve the clear , still not fully fixing the clear issue ( see Summary table).

In the run7, We considered on top of the default clear, 2 new configurations. The changes are in the ParallelFlush function , which move the charges from the image area to the serial register :

  • the default clear (V29) : In the default clear, all serial clocks are kept up as the // clocks move charges from the image area to the serial register ( [sequencerV29] ). The charges once on the serial register will hopefully flow to the ground : the serial register clocks being all up , without pixels boundary , and with its amplifier in clear state. At the end of the clear, a full flush of the serial register is done ( ~ the serial clocks changes to read a single line ).

  • the No-pocket Clear (NoP) : a clear where the serial register has the same configuration ( S1 & S2 up , S3 low ) when the // clock P1 moves the charges to the serial register than in a standard image read . Still we keept all phases up the rest of the time for a fast clear of the charges along the serial register ( [sequencerV29_NoP] ) . The idea is that the S3 phase is not designed to be up when charges are transfered to the serail register, and is probably playing a major role in the pockets creation.

  • The No-pocket with serial flush Clear (NoPSF) : this sequencer is close to the NoP solution , except that during the transfered of 1 line to the serial register, the serial phases are also moved to transfer two pixels along teh serial register. The changes in electric field at the image-serial register interface are then even more representative to what a standard read will produce, and should further prevent the creation of pockets. ( [sequencerV29_NoPSF] ).

Results on standard e2v and itl CCD#

Figure showing the impact of the various types of clear on a bias taken after a saturated flat for an E2V sensor.

Figure showing the impact of the various types of clear on a bias taken after a saturated flat for an E2V sensor.

Figure showing the impact of the various types of clear on a bias taken after a saturated flat for an ITL sensor.

Figure showing the impact of the various types of clear on a bias taken after a saturated flat for an ITL sensor.

In the above images , we present for 3 types of sequencer ( from left to right : V29 , NoP and NopSF), a zoom on the first lines of an itl or e2v amplifier ( for itl R03_S11 C14 and for e2v R12_S20 C10 ) shown as a 2D lines-columns image ( top plots) or as the mean signal per line for the first lines read of an amplifier (bottom plots).

As seen in see left plots of clear e2v image for an e2v CCD, a bias taken just after a saturated flat will show a residual signal in the first lines read when using the default clear (left images,clear=v29 ) : the first line has an almost saturated signal ( ~ 100 kADU here), and a significant signal is seen up to the line ~50. In practice, in function of the amplifier, signal can be seen up to line 20-50. When using the NoP clear (central plots), we can already see a strong reduction of the uncleared charges in the first acquired bias after a saturated flat, still a small residual signal is visible in the first ~ 20 lines. The NoPSF clear (right plots) fully clear the saturated flat, and no uncleared charges are observed in the following bias.

As seen in see left plots of clear itl image for an itl CCD, a bias taken just after a saturated flat will show a residual signal in the first lines read when using the default clear (left images,clear=v29 ) : the first line has an almost saturated signal ( ~ 100 kADU here), and a significant signal is seen in the following line. Both NoP clear (central plots) and NoPSF clear (right plots) fully clear the saturated flat, and no uncleared charges are observed in the following bias.

Results on itl R01_S10#

Figure showing the impact of the various types of clear on ITL R01_S10.

Figure showing the impact of the various types of clear on ITL R01_S10 after a saturated flat ( bias after a saturated flat), from left to right : 1 standard Clear , 3 standard Clear , 5 standard Clear , 1 NoP Clear, 1 NoPSF Clear

There is one ITL sensor, R01_S10, that presents a specific and non-understood behavior :

  • It has a quite low full well (2/3 of nominal )

  • The 3 CCD of this REB have a gain 20% lower than all other ITL CCD ?

  • The images taken after a large staturation, as seen in figure clear in itl R01_S10, show a large amount of uncleared charged ( with the standard clear : 4 amplifiers with ~500 lines of saturated signal !)

It apears that putting S3 low during the clear as done in NoP or NoPSF , is even worse than a standard clear. This is strange as a full frame read , which does this too, manages to clear such image. We can notice that NoPSF is ~ 50% better than NoP , but still worse than the standard clear , in particular for the 12 amplifiers almost correct with the standard clear.

At this stage we don’t have a correct way to clear this sensor once it collects a saturated flat, but It’s not known if a saturated star in this sensor, leaving signal in the parallele overscan, will presents the same clear issue.

Conclusion#

Table 1 This table summaries the different clear methods used so far.#

Default Clear 1 Clear (seq. V29)

Multi Clear 3 Clears (seq. V29)

Multi Clear 5 Clears (seq. V29 )

Deep Clear 1 Clear (Seq. V23 DC)

No Pocket(NoP) 1 Clear (seq. V29_NoP)

No Pocket Serial Flush(NoPSF) 1 Clear ( seq. V29_NoPSF, V30 )

Clear duration

65.5 ms

196.5 ms

327.4 ms

64.69 ms

65.8 ms

67 ms

“E2V” after saturated Flat

1st line saturated signal up to line 50

No residual

electrons

No residual

electrons

1st line saturated signal up to line <20

signal up to line 20

No residual

electrons

“ITL” after saturated Flat

1st line saturated signal up to 2nd line

No residual

electrons

No residual

electrons

tiny signal left in the first line

No residual

electrons

No residual

electrons

R01_S10 ITL “unique”

first 500 lines saturated for 4 amp. 13 amp. with signals.

first 150 lines saturated for 2 amp. 5 amp. with signals.

first 100 lines saturated for 2 amp. 2 amp. impacted

not measured

first 1000 lines saturated for 16 amp. 16 amp. with signals.

first 750 lines saturated for 16 amp. 16 amp. with signals.

Even if NoP or NoPSF are overcoming the clear issue we had with ITL sensors, the exception of R01_S10 prevented the usage of those sequencers for ITL device for the run7. Notice that beyond R01_S10 the numbers of line potencilly “not cleared” are small (2 first lines)in ITL device, and they correspond to a CCD area hard to use anyway ( sensor edges with low efficciency). So at this stage the default clear is still our default for ITL, and further studies to overcome the problem with R01_S10 are forseen ( ex : do a continuous serial flush during exposure at low rate , 10^6 pixels flush in 15s).

On the other side , after those studies in run7, we now have a good way to fully clear the e2v devices through the NopSF clear. The NoPSF clear grants that the first 50 lines of e2v device that had un-cleared electrons from the previous exposure, are now free of such contamination.

From now :

  • for e2v, NoPSF will be the default clear method

  • for ITL, the origial clear (serial phase 3 always ), slightly extended in time to match the NoPSF e2v clear execution time , will stay the default method.

Phosphorescence#

hello world.

This section describes phosphorescence.

  • phosphorescence background

  • phosphorescence on flat fields

  • phosphorescence on spot projections

Conclusions#

Run 7 final operating parameters#

This section describes the conclusions of run 7 optimization and the operating conditions of the camera. Decisions regarding these parameters were decided based upon the results of the voltage optimization, sequencer optimization, and thermal optimization.

Voltage conditions#

Table 2 Voltage conditions#

Parameter

dp80 (new voltage)

dp93 (Run 5)

pclkHigh

2.0

3.3

pclkLow

-6.0

-6.0

dpclk

8.0

9.3

sclkHigh

3.55

3.9

sclkLow

-5.75

-5.4

rgHigh

5.01

6.1

rgLow

-4.99

-4.0

rd

10.5

11.6

od

22.3

23.4

og

-3.75

-3.4

gd

26.0

26.0

Sequencer conditions#

Table 3 Sequencer conditions#

Detector type

File name

E2V

FP_E2V_2s_l3cp_v30.seq

ITL

FP_ITL_2s_l3cp_v30.seq

  • v30 sequencers are identical to the FP_ITL_2s_l3cp_v29_Noppp.seq and FP_E2V_2s_l3cp_v29_NopSf.seq. All sequencer files can be found in the github repository.

Other camera conditions#

  • Idle flush disabled

Record runs#

This section describes run 7 record runs.

All runs use our camera operating configuration, unless otherwise noted.

Table 4 Record runs#

Run Type

Run ID

Links

Notes

B protocol

E1880

E2233

Identical to E1880. Acquired after CCS subsystem reboot

PTCs

E1886

Red LED dense. Dark interleaving between flat pairs

E1881

Red LED dense. No dark interleaving between flat pairs

E748

nm960 dense

E2237

Red LED dense. Acquired after CCS subsystem reboot.

E2016

Super dense red LED. HV Bias off for R13/Reb2. jGroups meltdown interrupted acquisitions, restarted

Long dark acquisitions

E1117

E1116

E1115

E1114

E1075

Projector acquisitions

E1558

Flat pairs, fine scan in flux from 1-100s in 1s intervals. E2V:v29_NoP, ITL:v29_NoPP

E1553

Flat pairs, coarse scan in flux from 5-120s in 5s interval.E2V:v29_NoP, ITL:v29_NoPP

E1586

One 100s flat exposure, spots moved to selected phosphorescent regions.E2V:v29_NoP, ITL:v29_NoPP

E2181

Flat pairs from 2-60s in 2s intervals. Two 15s darks interleaved after flat acquisition. Rectangle on C10 amplifier.E2V:v29_NoP, ITL:v29_NoPP

E2184

10 30s dark images to capture background pattern

OpSim runs

E1717

Long dark sequence, no filter changes

E2330

Short dark sequence, filter changes in headers through OCS

E1414

30 minutes OpSim run with shutter control, filter change, and realistic survey cadence

E2328

Flats with shutter-controlled exposure

E1657

10 hour OpSim dark run, ~50% of darks were acquired properly

Phosphorescence datasets

E2015

10 flats at 10ke- followed by 10x15s darks

E2014

1 flat at 10ke- followed by 10x15s darks

E2011

20 flats at 10ke- followed by 10x15s darks

E2012

10 flats at 1ke- followed by 10x15 s darks

E2013

10 flats at 10ke- followed by 10x15s darks. Interleaved biases with the darks

Tree ring flats

E1050

E1052

E1053

E1055

E1056

E1021

E1023

E1024

E1025

E1026

Gain stability runs

E1955

E2008

E1968

E1367

E1362

E756

E1496

Persistence datasets

E1503

E1504

E1505

E1506

E2286

E1502

E1501

E1500

E1499

E1498

E1494

E1493

E1492

E1490

E1491

E1489

E1488

E1487

E1486

E1485

E1478

E1477

E1479

E1483

E1484

Guider ROI acquisitions

E1510

E1518

E1519

E1508

E1509

E1520

E1511

E1521

E1512

E1513

E1514

E1517