Questions on medical history will target the candidates’ handedness (Edinburgh Handedness Inventory) and the presence of diseases, which are among the exclusion criteria [33-35]. Additional questions on health status will inquire about the presence of influences that could alter the brain activity of interest, such as the use of stimulants [36]. CI users will receive questions about the duration of their hearing loss, and if they have tinnitus, about the objectivity and laterality of their tinnitus [37]. The Hearing Ability Questionnaires will investigate lipreading experience and hearing-associated factors, including the Speech, Spatial, and Qualities (SSQ-12) questions [38]. The question sheet should cover the subjective assessment of hearing ability in the last 6 months.

The audiometric measurements and the fNIRS recordings will take place in an acoustic chamber (6 m × 4 m × 2 m) with a separate ventilation system and electromagnetic shielding. The broadband reverberation time is ~200 ms.

In normal hearing participants, we will assess pure tone air-conduction hearing thresholds with a clinical audiometer (GSI 61, Grason-Stadler). The findings must confirm that participants have no hidden or undetected hearing loss (Table 1). For CI users, audiograms are available from clinical routine measurements.

In all participants, we will measure the word recognition score for Freiburg monosyllabic word lists at a sound pressure level (SPL) of 65 dB [39]. Additionally, we will perform the widely used Oldenburg Sentence Test (OLSA) [40-42]. The sentences will be played with 65 dB SPL background noise, using an adaptive version of the female OLSA test [43-45]. The OLSA sentences will also be used as a stimulus during the fNIRS measurement. Speech material will be presented from a loudspeaker (Control 1 Pro) placed in front of the participants at a distance of 1 m.

During fNIRS recording, each study participant will sit in a comfortable chair with an armrest, headrest, and lumbar support (Figure 1). A desk will be placed in front of the participant with the electrical equipment. Visual stimuli will be presented through a computer screen (P2210, Dell) placed on the table at a distance of 120 cm in front of the participant. The acoustic stimuli will be played through a loudspeaker (8040B, Genelec) placed above the monitor at a distance of 130 cm from the ears. The loudspeaker will receive input from an external ASIO sound card (Scarlett 2i2, FocusRite) connected to the control laptop (XPS 13, Dell) via USB. The system will be calibrated to 65 dB SPL with the OLSA calibration noise and an acoustic analyzer (XL2, NTi Audio).

The stimulation protocol will be controlled by a custom-written script (Python 3.8.8) using Tkinter and python-vlc libraries. The script will send triggers via the serial interface to a trigger-box (MMBT-S Interface Box, NEUROSPEC AG), which converts the signals to transistor-transistor Logic (TTL) levels. The TTL-encoded signals will then be received by the fNIRS machine (FOIRE-3000, Shimadzu).

Participants will interact with the control laptop using the buttons of a mouse (WM527, Dell). The pointing function of the mouse will be disabled to ensure that participants control the experiment only by clicking and rolling. During the fNIRS measurement, the participants will be able to press an alarm button (Switchbox, Delock) positioned in a reachable distance on the table.

We will select the regions of interest (ROIs) for the placement of the optodes considering previous studies. We expect responses related to audiovisual speech comprehension in the auditory and visual cortex, more specifically in the following ROIs: superior temporal gyrus (STG), primary visual cortex (V1), and visual association cortex (V2) [28,31,46-50]. Additionally, during similar conditions, the left inferior frontal gyrus (LIFG) has been associated with effortful listening [27,51], and elevated cortical responses have been reported in the middle temporal gyrus (MTG) and middle frontal gyrus (MFG) [52]. Based on the defined ROIs, we will determine the optimal selection of EEG coordinates using the fNIRS optode's location decider (fOLD) toolbox [53]. We will further consider the position of the audio processor and receiver coil in CI participants to avoid interference with optodes.

The montage will consist of 16 sources and 16 detectors placed on the surface of the skull according to the international 10-10 system of electrode placement (Figure 2A) [54]. The source-detector pairs will result in a total of 43 channels in a multidistance setup: 3 of them are short-separation channels with a 15-mm interoptode distance, 4 are extra-long channels with a distance of 36-37 mm, and 36 are normal length channels that are approximately 30 mm apart. In a multidistance approach, shorter channels (15 mm) provide information about the interfering systemic signals in the outer cortex and longer channels (36+ mm) about brain activation in deep regions [55,56]. Practically, however, the signal-to-noise ratio may be poor in long distances, so in many cases we will not be able to use those channels. The Monte Carlo sensitivity simulation of all source-detector pairs is shown in Figure 2B and indicates a uniform sensitivity profile across the temporal, visual, and prefrontal cortical regions [57]. The sampling rate will be set to 14 Hz.

The optode holder cap will be assembled using the manufacturer's components (Holder kit, Shimadzu) and custom 3D printed parts (colored optode markers and stabilizers for different head sizes). The parts will be designed in a solid modelling software (SolidWorks 2019, Dassault Systemes) and printed using a 3D printer (Prusa i3 MK3S+, Prusa Research).

At the end of the experiment, we will digitize the position of all optodes with a depth sensing camera (Structure Sensor Pro, Occipital Inc) connected to an iPad (iPad Pro 2020, Apple Inc). The depth sensing camera will be set up for optimized scanning of dark objects with low ambient infrared light. The infrared exposure time, gain, and depth resolution will be set to the highest available settings so that the colored optode markers can be easily identified on the 3D scan.

During fNIRS recordings, we will instruct the participants to concentrate on the screen, follow the instructions, and reduce head movements. If the participants feel uncomfortable, an emergency button in front of them will be made available to stop the experiment. We will give all instructions both verbally and in writing. Before the recordings, the participants will conduct a short familiarization session with 4 example stimulations. Once the participant confirms that the task is understood, we will start the definitive recording. The functional recordings will begin with a 5-minute resting state period (Figure 3A). We will instruct the participant to sit still, close their eyes, and relax but try not to fall asleep. Then 2 stimulation sessions approximately 12 minutes each will follow. Between the 3 sessions (ie, the resting and the 2 stimulation sessions), the participants can take a break of their chosen duration.

As stimulus material, we will use a video version of the female OLSA test [40,58]. A single stimulus will consist of one sentence (eg, “Nina gives twelve red flowers”) which will be repeated 3 times. The duration of one stimulus will be 13 seconds, comparable to hemodynamic responses [59].

A single stimulation block will contain 4 different stimuli, presented in one of the following modalities in a counterbalanced order (Figure 3B): (1) speech in quiet (audio only), (2) speech in noise (audio only), (3) speech in quiet (video only, ie, lipreading), and (4) speech in quiet (audio and video). The stimulation will be followed by 20-25 seconds of nonstimulus interval, during which a white fixation point will be presented on a black screen. During the audio-only conditions, the same black screen will be displayed so that the participant will have no indication other than hearing whether the stimulation has already started or not.

At random points, participants will be asked to answer questions to ensure attention and monitor speech comprehension during the test. The questions will be displayed in the nonstimulus epoch, for which the nonstimulus interval will be shortened to 10 seconds. The questions will ask to repeat the correct name or number of the last sentence from 4 possible answers. For example, if the sentence is “Nina gives 12 red flowers,” the question is either ”How many red flowers?“ or ”Who gives 12 red flowers?“ To answer the question, the participant will have to select 1 of 4 choices: 2 randomly selected numbers/names from the OLSA sentence matrix (wrong answers), an option if the respondent is not sure of the answer (skipped answer), and the correct answer. For the previous question, a possible combination could be (1) ”Britta,” (2) “Nina,” (3) “Peter,” and (4) “I cannot decide.” The participant will select an option with the roller on the computer mouse and confirm the answer with a double click. In the previous example, the participant must select the second option (”Nina“). The questions and the answers will be randomly generated, and the position of the question within the blocks will also be randomly chosen.

The shortened nonstimulus interval of 10 seconds prior to the question window will allow us to evaluate the fNIRS responses. Therefore, the interleaved questions will not harm the overall effectiveness of the measurement. After the question is answered, the regular 20-25 second relaxation time will be applied to ensure that the brain responses return to baseline. Overall, 2 questions per modality will be asked, resulting in 8 questions throughout the entire fNIRS measurement.

Following the breaks, before the first stimulation, there will be a stimulus-free interval of 20 seconds. This will ensure homogeneity of responses, meaning that all stimuli are perceived under similar circumstances. Overall, 10 blocks will be presented, resulting in 10 responses per stimulation modality, and the total fNIRS measurement time will be around 45 minutes. At the beginning of every event (start/stop of a block, resting state, stimulation, question, answer), a trigger will be sent from the control computer to the fNIRS machine through the trigger-box and stored as an extra channel in the fNIRS raw data.

Following 5 stimulation blocks, we will ask every participant to rate their listening effort to the different stimuli, their rating of fatigue and their level of mind-wandering (Figure 3A) [60-64]. To evaluate the listening effort, we will use Adaptive Categorical Listening Effort Scaling (ACALES) [65].

All written source documents will be completed in a neat, legible manner to ensure accurate interpretation of data. For each participant, a case report form (CRF) will be maintained, including the participant number. In CRFs and other project-specific documents, participants are only identified by a unique participant number. fNIRS measurements will be stored in a closed research environment (REDCap, Vanderbilt University, Nashville, United States). The secure web application is running on a local server maintained and backed up by the University of Bern. All documents related to the study, including the CRFs will be considered as source data, and these will be stored at the measurement site in accordance with relevant standards.

Data preprocessing will be performed in MATLAB (MathWorks) using the Homer2 (v2.3) [66] and NIRS [67] toolboxes. The signal quality will be checked based on the heart rate content of the signal, using a sliding window approach [68-71]. Channels and time points with insufficient signal quality will be removed. Short channel correction will be applied to the absorbance data, using short separation regression [56,72]. The motion artifacts will be removed with a WaveletFilter module of the NIRS toolbox [67]. The signal will be bandpass filtered between 0.01 and 0.12 Hz with the BandpassFilter function from the Homer toolbox [66]. Then, the absorbance data will be converted to concentration changes of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) in mMolar*cm using the following equations, as specified by the manufacturer based on the modified Beer-Lambert law [73]:

A further correction step will be performed to reduce noise based on the principle that the concentration changes of oxygenated and deoxygenated hemoglobin should be negatively correlated [74].

We will perform the postprocessing of the scans with a 3D mesh processing tool (MeshLab) and custom-written scripts (MathWorks) [75].

We will manually select the coordinates of the optodes and anatomical landmarks with MeshLab on the obtained 3D scans. The list of coordinates will then be exported and projected into Montreal Neurological Institute (MNI) space. The MNI coordinates will be displayed on the preoperative MRI scan of every CI-user participant, and the exact source of measured hemodynamic activation will be determined. Additionally, the mean and the standard deviation of optode coordinates will be calculated and reported as quality measure for optode fittings [55,76].

Data analysis will be performed in Python using the MNE-Toolbox [77] and MNE-NIRS package [78]. Individual epochs will be subtracted from the channel data, from t=0 seconds to t=24 seconds relative to the stimulus onset. The epochs will be baseline-corrected by subtracting the mean of the signal between t =−5 seconds and t=0 seconds. Using the Glover canonical hemodynamic response function [79] a design matrix for the general linear model (GLM) will be constructed [80,81]. After GLM fitting, the regression results will be stored. Following this, temporal and spatial features will be extracted from each epoch (amplitude, area under curve, peak latency, laterality, power). The regression results and the extracted features will be weight-averaged over ROIs by taking the inverse of the standard error of the GLM fit for each channel [67]. The data will be averaged over the participants, and group-level statistics will be calculated using correlation analysis and linear mixed-effects models.

The answers from the questionnaires will be digitized, and correlation analysis will be performed to reveal relations between the measured brain activation patterns and the evaluated questionnaires. Additionally, further behavioral data will be obtained from the triggers, such as reaction time to questions across the measurement as a measure of fatigue or response accuracy for each stimulation type as a measure of speech understanding.