Near-Infrared Optical Constants and Guided-Mode Benchmarkng of High-Index MoSe2 for Nanophotonics

1. Introduction

Dense photonic integrated circuits require tight optical confinement to minimize individual device footprints [1,2]. In conventional isotropic platforms, such as silicon and silicon nitride, physical scaling is bottlenecked by evanescent field leakage, which causes severe optical crosstalk at sub-wavelength routing pitches [3]. Layered van der Waals (vdW) materials with a strong in-plane covalent bonding and weak out-of-plane vdW forces, offer a structural solution to this confinement limit [4]. VdW materials have become a central materials class in modern nanophotonics because they combine strong light–matter interaction with crystallographically defined optical anisotropy and broad compatibility with heterogeneous integration [2,3,5,6,7,8]. Within this family, transition metal dichalcogenides (TMDCs) are especially relevant for guided-wave and resonant nanophotonics because bulk and multilayer crystals can simultaneously provide a high refractive index, giant uniaxial optical birefringence, and low loss over selected spectral windows [4,9,10,11,12,13,14]. From an integrated photonics perspective, these properties are not only of fundamental interest. The refractive-index contrast and the anisotropic dielectric response directly determine modal confinement, single-mode design limitations, evanescent-field penetration, and the strength of coupling between adjacent optical channels [5,9,15,16]. As a result, reliable optical constants in the near-infrared around the telecommunication band are crucial for realistic benchmarking of TMDC waveguides, couplers, and other building blocks of photonic integrated circuits.

Molybdenum diselenide (MoSe₂) is a particularly interesting material of the high-index TMDC class [5,17]. Prior studies have established that MoSe₂ can sustain guided exciton–polariton transport in crystal flake waveguides closer to the visible and excitonic spectral range, highly-confined optical modes and high nonlinear response in epitaxially grown MoSe₂ in near-infrared range [12,18,19]. The combination of a high in-plane refractive index (n_ab > 4), low absorption, and giant optical anisotropy, birefringence Δn = n_ab − n_c ~ 1, in crystal bulk MoSe₂ enables the localization of the electromagnetic field at subwavelength scales and provides a single-mode operation in compact photonic waveguides [4,5,9,20,21]. In this regard, the design of low-loss near-infrared photonic components requires a device-oriented description that links anisotropic optical constants, experimentally observed guided modes, and engineering figures of merit such as coupling length. However, determination of the dielectric permittivity of vdW crystals is a non-trivial, since the studied samples, microcrystals, i.e., flakes, obtained by mechanical exfoliation from the original crystal, often have dimensions of up to several tens of micrometers [22,23,24]. This restricts the utilization of traditional methods such as spectroscopic ellipsometry and reflectometry for determining the optical constants over a wide spectral range. Moreover, these far-field optical methods have low sensitivity to determine the out-of-plane component of refractive index (n_c) [10,17]. In contrast, scattering-type scanning near-field optical microscopy (s-SNOM) enables local measurement of the optical response, excitation and propagation of guided modes in vdW materials with spatial resolution exceeding the diffraction limit and allows verification of the numerical values of the anisotropy of optical constants obtained independently, using spectroscopic ellipsometry [9,10,16,24,25,26].

In this work, we present a detailed characterization of the optical constants and waveguide properties of the high-index crystal MoSe₂ in the near-infrared by combining the macroscopic spectroscopic ellipsometry, s-SNOM, and electromagnetic modeling to build and test an anisotropic optical model demonstrating the applications of MoSe₂ in nanophotonics. First, we extract the in-plane optical constants of a macroscopic crystal in the 1000–2000 nm range. Next, we visualize guided TM modes in an exfoliated MoSe₂ flake and compare the measured modal dispersion with transfer-matrix calculations. Finally, we use the experimentally supported model to benchmark ridge MoSe₂ waveguides against WS₂ counterparts in the single-mode regime.

2. Materials and Methods

2.1. Material Preparation and Morphological Characterization

A commercially available ultra-flat large-area bulk MoSe₂ crystal grown by chemical vapor transport (CVT, 2D Semiconductors, Inc., USA) was used for ellipsometric characterization. The crystal composition was examined by energy-dispersive X-ray spectroscopy (EDS, Bruker QUANTAX EDX) in a scanning electron microscope setup (SEM, JEOL JSM-7001F) working in secondary electron imaging mode. The value of the acceleration voltage was 15 keV. The EDS spectrum was analyzed using the software provided by Bruker. For near-field measurements, MoSe₂ flakes were obtained by mechanical exfoliation from the same parent crystal and transferred onto a SiO₂(285nm)/Si substrate. Optical microscopy was used to identify flakes of sufficient lateral size, and atomic force microscopy (AFM, NT-MDT Spectrum Instruments) in contact mode was used to determine the flake thickness. The cantilevers employed were NSA01 tips (TipsNano), characterized by a spring constant of 5.1 N/m, a tip radius of less than 10 nm, and a resonant frequency of 150 kHz. Analysis of AFM scans was performed using Gwyddion software.

2.2. Spectroscopic Ellipsometry

The optical constants of the macroscopic MoSe₂ crystal were measured with a variable-angle spectroscopic ellipsometer (V-VASE, J.A. Woollam Co.) in the wavelength range 1000–2000 nm. A large-area crystal (~1 cm²) was required because the ellipsometer spot size was approximately ~1.5 mm. The ellipsometric parameters Ψ and Δ were recorded in reflection geometry at incidence angles of 50°, 55°, and 60° with a wavelength step of 5 nm. The data were fitted in WVASE using a parallel-sided slab model that included backside reflection. The dielectric response was parameterized with the Tauc–Lorentz dispersion model, which preserves the Kramers–Kronig consistency and has been broadly used for TMDC optical constants extraction [5,12,20,21].

2.3. Scattering-Type Scanning Near-Field Optical Microscopy

Near-field imaging of guided modes was performed with a commercial s-SNOM system (neaSNOM, Neaspec GmbH) operated in reflection mode. A tunable continuous-wave fiber-coupled laser (Agilent 81600B) with a range of 1460–1640 nm was used as the excitation source. The optical probe was a metalized AFM tip (NanoWorld, ARROW-NCPt-50) oscillating at Ω ≈ 280 kHz. A pseudo-heterodyne interferometric scheme was employed, and the detector signal was demodulated third harmonic to isolate the background-suppressed near-field amplitude and phase [27]. The excitation polarization was chosen to launch TM-polarized guided modes in the flake. To select a suitable planar waveguide thickness, the TM-mode dispersion of the MoSe₂/SiO₂/Si stack was first evaluated by the transfer-matrix method [28,29] for the measured in-plane response and the adopted out-of-plane refractive index.

The near-field fringe pattern was analyzed in the Fourier domain using line profiles extracted perpendicularly to the flake edge. In accordance with the established interpretation of guided-mode s-SNOM in anisotropic TMDCs flakes, the observed pattern is attributed to interference between the incident field and a tip-launched guided mode that is scattered at the flake boundary. The Fourier peak position yields the apparent normalized in-plane momentum n_s-SNOM. The effective mode index was then extracted as $n_{\mathrm{e}\mathrm{f}\mathrm{f}}=n_{\mathrm{s}-\mathrm{S}\mathrm{N}\mathrm{O}\mathrm{M}}+\cos \alpha ·\sin \beta$, where α is the angle between the illumination wavevector and its projection onto the sample plane, and β is the angle between the in-plane projection and the flake edge. For the present geometry, α = 45° and β = 90°.

2.4. Numerical Modeling

Ridge waveguides with MoSe₂ or WS₂ cores on a SiO₂ substrate and air cladding were analyzed using the finite-element method in COMSOL Multiphysics. The crystal optic axis was assumed to be normal to the substrate, consistent with the layered-flake geometry. To quantify coupling between neighboring waveguides, even and odd supermodes of a coupled-waveguide pair were calculated for different center-to-center spacings d. The coupling length was then obtained from [30]

$L_{\mathrm{c}}=\lambda /\left[2\left|n_{\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}}-n_{\mathrm{o}\mathrm{d}\mathrm{d}}\right|\right].$

Coupling length was optimized at core widths and heights, at which waveguide operates in the single-mode regime.

3. Results and Discussion

For the initial determination of the in-plane optical constants, spectroscopic ellipsometry was applied to the molybdenum diselenide (MoSe₂) crystal. Measurements were performed using a Woollam V-VASE ellipsometer in the wavelength range λ = 1000–2000 nm, which is important for telecommunications applications. A large-area crystal (~1 cm²), synthesized by CVT, was chosen as the MoSe₂ sample. In the experiment, the choice of a macroscopic crystal was dictated by the minimum spot size of the ellipsometer measurement beam, which is approximately ~1.5 mm in diameter. Measurements of the ellipsometric parameters Ψ and Δ were performed in reflection mode (see Methods). To determine the in-plane refractive index n_ab and extinction coefficient k_ab of the MoSe₂ crystal, a layered model including backside reflection was used. For the dielectric function, the Tauc–Lorentz oscillator dispersion model was employed in fitting the Ψ and Δ spectra, providing analytical satisfaction of the Kramers–Kronig relations, similarly to approaches used for other TMDCs, using the WVASE software [9,10,20,21]. The results of the ellipsometry data analysis for MoSe₂ are presented in Figure 1a. The in-plane refractive index n_ab varies smoothly in the range of 4.1–4.7, while the extinction coefficient k_ab is close to zero within the sensitivity of the ellipsometer, remaining well above the values typical of conventional dielectric photonic materials in the near-infrared [5,10,22]. Since the ellipsometry methodology used in this work is limited to determining optical constants within an isotropic model, to account for anisotropy for further investigation of the waveguide properties of MoSe₂, the out-of-plane refractive index n_c from reference [12] was used.

The s-SNOM technique allows for the independent verification of the optical constants of vdW crystals and confirmation of the magnitude of optical anisotropy, s-SNOM was used to measure the propagation of photonic modes in MoSe₂ [10,21]. To this end, planar waveguides were fabricated by mechanically exfoliating MoSe₂ flakes onto a SiO₂(285 nm)/Si substrate from a macroscopic crystal. Suitable MoSe₂ flakes with sufficient area and the required thickness for the propagation of the fundamental photonic TM mode in the wavelength range near 1500 nm were identified using optical microscopy and AFM. The range of thicknesses required for the propagation of a single TM mode was determined by numerical calculation using the transfer matrix method [28] and the optical constants of MoSe₂ in Figure 1a, yielding thickness values from 200 to 370 nm. This choice is important because it enables a direct comparison between measured and calculated modal dispersion without the ambiguity introduced by higher-order guided states. The AFM step profile on a chosen flake yielded a thickness of 235 ± 5 nm (see inset in Figure 1c), which falls inside the calculated single-mode interval for the planar TM mode near λ = 1.5 μm and was selected for s-SNOM measurements. In addition, the EDS spectrum in Figure 1b confirms the expected stoichiometry of the parent crystal, with measured atomic fractions of 34.7% for Mo and 65.3% for Se, corresponding closely to the 1:2 composition of MoSe₂.

Next, to study photonic modes in MoSe₂ planar waveguides, the s-SNOM metallic tip is illuminated with laser radiation in the 1460–1640 nm range in TM polarization (see Section 2.3 for more details). Scanning the tip perpendicular to the crystal edge allows mapping of interference fringes. The resulting s-SNOM amplitude and phase images recorded near the flake edge at wavelengths of 1475 and 1625 nm are shown in Figure 2a. Both wavelengths reveal a clear sequence of near-field fringes extending inward from the flake boundary, which is consistent with the excitation of a guided TM₀ mode in the planar MoSe₂ waveguide. The fringe periodicity is directly related to the mode wavelength (λ_p) and, consequently, to the effective mode index (n_eff = λ₀/λ_p, where λ₀ is the free-space wavelength). The effective refractive index of the propagating TM mode in an anisotropic planar MoSe₂ waveguide depends on both the in-plane (n_ab) and out-of-plane (n_c) refractive indices. Thus, by determining n_eff and comparing it with theoretical calculations of the dispersion curve using the transfer matrix method, it is possible to verify the anisotropic optical constants. To determine n_eff, complex fast Fourier transform of line profiles extracted from the near-field images was performed and the corresponding Fourier spectra are shown in Figure 2b. The dominant peaks (at q(k₀) ~ 1.1) assigned to the TM₀ mode were used to determine the in-plane modal momenta, while additional peaks in the Fourier spectra arise due to the propagation of the wave in air and do not affect the position of the peak corresponding to the desired TM₀ mode After applying the geometric frequency shift described in Section 2.3, the extracted effective indices at 1475 and 1625 nm were compared with transfer-matrix calculations of the energy (E~1/λ₀)-momentum (q=1/λ_p) dispersion relation of the waveguide mode for the anisotropic MoSe₂/SiO₂/Si stack. As shown in Figure 2c, the experimental effective indices points follow the calculated dispersion curve. This agreement does not by itself constitute a full independent reconstruction of the complete dielectric tensor, because the out-of-plane response was taken from literature, but it does provide an important consistency check: the measured in-plane optical constants n_ab and k_ab, the adopted n_c, and the flake thickness together describe the observed guided-mode dispersion without the need for further adjustable parameters.

Using the experimentally supported optical model, we next evaluated the prospects of using MoSe₂ in nanophotonics, by numerical modeling of the behavior of waveguides based on this material. Characteristics of MoSe₂ waveguides were compared with those of WS₂, another TMDC with a high refractive index and optical anisotropy, whose waveguiding performance has previously been shown to be better than silicon, the backbone material of modern nanophotonics [9]. First, the effective indices of the supported waveguide modes were calculated (see Methods). This allowed determination of the number of supported waveguide modes as a function of the waveguide core width (w) and height (h), as well as the construction of effective index maps for the fundamental mode, i.e., the mode with the highest index. From the nanophotonic viewpoint, the effective index is an important parameter characterizing the degree of field confinement within the waveguide core. In particular, it determines the size of resonators and modulators, and also governs radiation losses in bent waveguide sections. Furthermore, the higher the effective index, the faster the evanescent field decays in the waveguide cladding, leading to reduced crostalk between adjacent waveguides and, consequently, enabling denser integration of waveguides on a chip, thereby enhancing the performance of photonic integrated circuits [9]. In Figure 3 comparison of the effective-index maps for MoSe₂ and WS₂ rectangular waveguides show that MoSe₂ core supports modes with a higher effective index at smaller core dimensions, which is expected due to the higher refractive index of MoSe₂. On the other hand, for equal dimensions, MoSe₂ waveguides support more modes, whereas in practice a single-mode regime is important, as it allows better control of signal transmission through the waveguide. Nevertheless, in the single-mode regime, i.e., at smaller core dimensions, MoSe₂ waveguides exhibit a higher effective index than WS₂ waveguides, suggesting weaker crosstalk compared to WS₂.

To test this hypothesis, the coupling length L_c, which characterizes the mutual influence of adjacent waveguides, was calculated. After propagating this distance along the waveguide, the mode completely transfers from one waveguide to the neighboring one (see Methods for the details on crosstalk calculations). The calculation results show that for a fixed waveguide core width w, L_c increases with core height; therefore, the optimal core dimensions should lie on the cutoff curve of the second waveguide mode (the white curve in Figure 3). For a fixed center-to-center distance d = 1 μm between waveguides, the maximum achievable coupling length L_c for MoSe₂ waveguides exceeds 15 mm, which is approximately 3.5 times higher than the same value for WS₂ waveguides, which is 4.3 mm.

A more detailed investigation of the dependence of L_c on d (Figure 4) shows that the advantage of MoSe₂ over WS₂ increases as the distance between waveguides grows. At the same time, if we compare the minimum distance between waveguides at which L_c = 10 mm, which is sufficient for constructing global interconnects on a chip, then MoSe₂ (d_min = 960 nm) outperforms WS₂ (d_min = 1080 nm) by 12.5%. This enhancement is very significant and hard to achieve considering the near-exponential dependence of the coupling length on the distance between waveguides, which is caused by the exponential decay of the waveguide mode field outside the waveguide core. Moreover, there exist methods for further increasing integration density by engineering the space between waveguides, which can not only reduce the distance between waveguides but also almost completely suppress waveguide coupling [30,31].

4. Conclusions

We have evaluated bulk-like MoSe₂ as a near-infrared waveguiding material by combining spectroscopic ellipsometry, s-SNOM near-field imaging, and electromagnetic modeling. The fitted in-plane refractive index remains between 4.1 and 4.7 from 1000 to 2000 nm, while the fitted extinction coefficient is close to zero within the sensitivity of the fit. Guided TM₀ modes imaged at 1475 and 1625 nm yield effective indices consistent with transfer-matrix calculations based on the measured in-plane response, the measured flake thickness, and a literature out-of-plane index. This agreement supports the internal consistency of the anisotropic optical model. Waveguide benchmarking further showed that, in the single-mode regime, MoSe₂ provides higher effective indices and longer coupling lengths than another high-refractive-index anisotropic material WS₂. This leads to a reduction in crosstalk by approximately a factor of 3.5, demonstrating that MoSe₂ waveguides are more compact and effective, which can be used to increase the integration density and reliability of photonic circuits. These results support MoSe₂ as a high-index anisotropic material platform for compact near-infrared photonic routing.