1. INTRODUCTION
The interaction of an intense, picosecond laser pulse propagating in an underdense plasma is of fundamental interest in plasma physics, with applications to fast-ignition fusion, harmonic generation, and laser–plasma accelerator concepts (Nakajima, Reference Nakajima2000; Hora et al., Reference Hora, Miley, Flippo, Lalousis, Castillo, Yang, Malekynia and Ghoranneviss2011; Reference Hora, Miley, Ghoranneviss and Salar Elahi2013; Foldes et al., Reference Foldes, Kocsis, Racz, Szatmari and Veres2013). Stimulated Raman scattering and stimulated Brillouin scattering are two important instabilities that occur in an underdense plasma, which are still a major issues in the realization and understanding of inertial confinement fusion (ICF) because the driving energy for the implosion is provided by the incident laser and the occurrence of backscattered light could greatly reduce the laser energy absorption efficiency (Bawaaneh, Reference Bawaaneh2006; Paknezhad & Dorranian, Reference Paknezhad and Dorranian2011, Paknezhad, Reference Paknezhad2013a, Reference Paknezhadb). Hence, the control of this instability is therefore of a fundamental concern for ICF. Brillouin scattering occurs in the interaction of a picosecond laser pulse with a plasma, which involves coupling of a large-amplitude light wave to a scattered light wave and an ion acoustic wave (Baton et al., Reference Baton, Rousseaux, Mounaix, Labaune, La Fontaine, Pesme, Renard, Gary, Louis-Jacquet and Baldis1994; Froula et al., Reference Froula, Divol, Braun, Cohen, Gregori, Mackinnon and Williams2003). Stimulated Brillouin scattering in a transversely magnetized plasma involves decaying of a pump laser wave (ω 0, k 0) into an ion acoustic upper-hybrid wave (ω, k) and two scattered upshifted (anti-Stokes) and downshifted (Stokes) light waves (ω s = ω 0 ± ω, k s = k ± k 0) (Paknezhad, Reference Paknezhad2012; Reference Paknezhad2013a; Sharma et al., Reference Sharma, Bhardwaj and Sharma2012). Brillouin instability has been extensively studied in magnetized and unmagnetized plasmas for many years both theoretically and experimentally (Wang et al., Reference Wang, Lu, He, Zheng and Zhao2009; Bernhardt et al., Reference Bernhardt, Selcher, Lehmberg, Rodriguez, Thomason, Groves, Carrick and Frazer2010; Paknezhad, Reference Paknezhad2012; Reference Paknezhad2013a). In an important earlier research, Salimullah and Hassan (Reference Salimullah and Hassan1990) investigated the relativistic effects on the stimulated scattering of large-amplitude laser radiation in the presence of the self-generated magnetic field in a laser-produced plasma. They show that the extreme relativistic effects increase the growth rate of the stimulated scattering. Recently, Sharma and Singh (Reference Sharma and Singh2013) have studied stimulated Brillouin backscattering of filamented hollow Gaussian beams in a collisionless plasma. They have shown that self-focusing and Brillouin backscattering decrease for higher order of a hollow Gaussian beam.
In addition to this topic, generation of harmonic radiation is also an important issue in the study of laser–plasma instabilities (Gibbon, Reference Gibbon1997; Gupta et al., Reference Gupta, Sharma and Mahmoud2007; Jha et al., Reference Jha, Mishra, Raj and Upadhyay2007). In a study in connection with this subject related to the Brillouin instability, Singh and Tripathi (Reference Singh and Tripathi2010) have carried out a numerical analysis of resonant Brillouin-shifted third-harmonic generation of a Gaussian laser pulse in a plasma. They demonstrated that the ion acoustic wave, generated in the stimulated Brillouin backscattering process, provides an uncompensated momentum between the harmonic photon and the combining fundamental photons, leading to the resonant enhancement of the harmonic power through the coupling of the laser with the ion acoustic wave.
Brillouin instability has the largest growth rate in backward scattering in which only the scattered Stokes wave is important. Figure 1 shows the schematic and k-vector matching condition of the Brillouin backscattering process in a transversely magnetized plasma. During the Brillouin backscattering process, the density perturbation at (ω 2 = 2ω 0 − ω, k 2 = k − 2k 0), which arises from the coupling of density perturbation associated with the ion acoustic upper-hybrid wave at (ω, k) and the second-harmonic oscillatory velocity of plasma electrons at (2ω 0, 2k 0), beats with the fundamental electron velocity at (ω 0, k 0) to generate a third-harmonic nonlinear current density at (ω 3s = 3ω 0 − ω, k 3s = k − 3k 0).
(a) Schematic and (b) the k-vector diagram of the Brillouin backscattering process in a transversely magnetized plasma, where k, k0, and ks are the wave vectors of the excited ion acoustic upper-hybrid wave, pump wave, and scattered Stokes wave, respectively.

Our aim in this work is to study the effect of third-harmonic generation on the growth rate of nonlinear Brillouin backscattering instability in a mildly relativistic transversely magnetized plasma. To achieve this goal, we follow the same method that has been used for the third-harmonic Raman backscattering process in our previous work (Paknezhad, Reference Paknezhad2013c). The present paper is organized as follows: In Section 2, we formulate a nonlinear wave equation for Brillouin scattering. In Section 3, we calculate the growth rate of third-harmonic Brillouin backscattering (3HBBS) instability in a magnetized plasma. Section 4 is devoted to results and discussion, and conclusion is presented in Section 4.
2. GENERAL FORMALISM
We now analyze the behavior of an electromagnetic wave that propagates in a homogeneous and magnetized plasma with initial density n 0. A constant magnetic field B0 in the y-direction is considered in the plasma channel. Here, the pump laser pulse is assumed to be a short pulse and have a high frequency, and since the duration of the short pulse is less than the plasma period, we ignore filamentation and self-focusing effects, hence is valid when the laser spot size is much bigger than c/ω p. We consider an electromagnetic wave propagates perpendicularly to B0. For an extraordinary wave (defined also as X-mode) the electric field is perpendicular to B0, and therefore the electric field induces an electron velocity in the x–z-plane. As a result, the force v × B0 introduces another velocity term in this plane. As a wave propagates through a region in which ωp and ω c are changing, it may encounter cutoffs and resonances. A wave generally reflected at a cutoff and absorbed at a resonance. There are two cutoffs that occur when the index of refraction goes to a zero value. The cutoffs of the extraordinary wave are found by setting k 0 equal to zero in the dispersion relation of the incident electromagnetic wave. The cutoff frequencies are
${{\rm \omega} _1}={\textstyle{1 \over 2}}[ {{\rm \omega} _{\rm c}}+{\lpar 4{\rm \omega} _{\rm p}^2+{\rm \omega} _{\rm c}^2 \rpar ^{1/2}}] $ and
${{\rm \omega} _2}={\textstyle{1 \over 2}}[ - {{\rm \omega} _{\rm c}}+{\lpar 4{\rm \omega} _{\rm p}^2+{\rm \omega} _{\rm c}^2 \rpar ^{1/2}}] $, which correspond to plasmas with ω p < ω 0 and ω p > ω 0, respectively. Since we consider the interaction of a high-frequency laser pulse with an underdense plasma, in which ω p≪ω 0 and ω c≪ω 0, the cutoff points therefore do not appear in our investigation.
Consider an extraordinary laser pump wave propagating in a homogeneous underdense plasma embedded in a constant external magnetic field
${{\bf B}_0}\hat y$ (see Fig. 2). The electric vector of the radiation field propagating along the z-direction is represented by
where E 0, k 0, and ω 0 are amplitude, wave number, and frequency of the radiation field, respectively. Similarly, the vector potential of the radiation field is defined as
${\bf A}\lpar z\comma \; t\rpar ={A_0}\lpar - i\hat x+{{\rm \alpha} _0}\hat z\rpar \, {e^{i\lpar {k_0}z - {{\rm \omega} _0}t\rpar }}$. Parameter α 0, which is attributable to the extraordinary property of the incident light wave in the presence of ion motion, is given by
$${{\rm \alpha} _0}=\displaystyle{{\lpar {{\rm \omega} _{\rm c}}/{{\rm \omega} _0}\rpar \lpar {\rm \omega} _{\rm p}^2 /\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar \rpar - \lpar {\Omega _{\rm c}}/{{\rm \omega} _0}\rpar \lpar \Omega _{\rm p}^2 /\lpar {\rm \omega} _0^2 - \Omega _{\rm c}^2 \rpar \rpar } \over {1 - \lpar {\rm \omega} _{\rm p}^2 /\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar \rpar - \lpar \Omega _{\rm p}^2 /\lpar {\rm \omega} _0^2 - \Omega _{\rm c}^2 \rpar \rpar }}\comma \;$$
where Ωc is the ion cyclotron frequency and Ωp is the ion oscillation frequency in the plasma (Krall & Trivelpicec, Reference Krall and Trivelpicec1973). The laser beam polarization is assumed to remain unaltered while propagating through a transversely magnetized plasma. Following the same procedure from our previous work (Paknezhad, Reference Paknezhad2013c) and replacing α 0 with
${{\rm \beta} _0}\lpar =\lpar {{\rm \omega} _{\rm c}}/{{\rm \omega} _0}\rpar \lpar {\rm \omega} _{\rm p}^2 /\lpar {\rm \omega} _0^2 - {\rm \omega} _{{\rm UH}}^2 \rpar \rpar \rpar $ in all computations, the x-component of the nonlinear wave equation of the extraordinary wave in magnetized plasma turns out to be
$$\left({\displaystyle{{{\partial ^2}} \over {\partial {t^2}}} - {c^2}\displaystyle{{{\partial ^2}} \over {\partial {z^2}}}} \right){A_x}=\displaystyle{{2{\rm \pi} in{e^2}} \over m}{{\rm \mu} _0}{A_0}\, {e^{i\lpar {k_0}z - {{\rm \omega} _0}t\rpar }}\comma \;$$where
$$\eqalign{& {{\rm \mu} _0}\lpar {{\rm \omega} _0}\rpar = \left({\displaystyle{{{\rm \omega} _0^2 + {{\rm \alpha} _0}{{\rm \omega} _0}{{\rm \omega} _{\rm c}}} \over {{\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 }}} \right)\cr & \qquad \qquad + a_0^2 \left({\displaystyle{{{c^2}k_0^2 {{\rm \omega} _0}{M_1}} \over {4{{\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar }^4}\lpar 4{\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar }} - \displaystyle{3 \over 8}\displaystyle{{{{\rm \omega} _0}{M_2}} \over {{{\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar }^4}}}} \right)}$$with
$$\eqalign{{M_1}& =2{\rm \alpha} _0^2 {\rm \omega} _0^7+{{\rm \alpha} _0}\lpar 6{\rm \alpha} _0^2 - 25\rpar {\rm \omega} _0^4 {\rm \omega} _{\rm c}^3+\lpar 26{\rm \alpha} _0^2 - 5\rpar {\rm \omega} _0^3 {\rm \omega} _{\rm c}^4 \cr & \quad - \lpar 30{\rm \alpha} _0^2+5\rpar {\rm \omega} _0^5 {\rm \omega} _{\rm c}^2 +27{{\rm \alpha} _0}{\rm \omega} _0^2 {\rm \omega} _{\rm c}^5 - {{\rm \alpha} _0}\lpar 6{\rm \alpha} _0^2 - 5\rpar {\rm \omega} _0^6 {{\rm \omega} _{\rm c}}\cr & \quad +10{{\rm \omega} _0}{\rm \omega} _{\rm c}^6+{{\rm \alpha} _0}{\rm \omega} _{\rm c}^7 }$$and
$$\eqalign{& {M_2} = \lpar 3{\rm \alpha} _0^2 + 1\rpar \lpar {\rm \omega} _0^7 + 6{\rm \omega} _0^5 {\rm \omega} _{\rm c}^2 + {\rm \omega} _0^3 {\rm \omega} _c^4 \rpar \cr & \quad \quad + 4{{\rm \alpha} _0}\lpar {\rm \alpha} _0^2 + 3\rpar \lpar {\rm \omega} _0^6 {{\rm \omega} _{\rm c}} + {\rm \omega} _0^4 {\rm \omega} _{\rm c}^3 \rpar .}$$The electric field components (E x, E z), the magnetic field B, and the wave vector k 0 of the extraordinary pump laser wave in a transversely magnetized plasma. The electric field is elliptically polarized in a plane perpendicular to the external magnetic field B0.

Equation (1) gives the nonlinear dispersion relation of the extraordinary pump laser wave in the magnetized plasma:
$$\displaystyle{{{c^2}k_0^2 } \over {{\rm \omega} _0^2 }}=1 - {{\rm \mu} _0}\displaystyle{{{\rm \omega} _{\rm p}^2 } \over {{\rm \omega} _0^2 }}.$$By neglecting the second and fourth terms in coefficient μ 0 and then replacing it in the dispersion relation (2) and after some algebra, we recover the linear dispersion relation of the extraordinary electromagnetic wave in a non-relativistic magnetized plasma, which is known as (Krall & Trivelpicec, Reference Krall and Trivelpicec1973)
$$\eqalign{& \displaystyle{{{c^2}k_0^2 } \over {{\rm \omega} _0^2 }} = 1 - \displaystyle{{{\rm \omega} _{\rm p}^2 } \over {{\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 }} - \displaystyle{{\Omega _{\rm p}^2 } \over {{\rm \omega} _0^2 - \Omega _{\rm c}^2 }} \cr & \qquad \quad - \displaystyle{{{{\lpar \lpar {{\rm \omega} _{\rm c}}/{{\rm \omega} _0}\rpar \lpar {\rm \omega} _{\rm p}^2 /\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar \rpar - \lpar {\Omega _{\rm c}}/{{\rm \omega} _0}\rpar \lpar \Omega _{\rm p}^2 /\lpar {\rm \omega} _0^2 - \Omega _{\rm c}^2 \rpar \rpar \rpar }^2}} \over {1 - \lpar {\rm \omega} _{\rm p}^2 /\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar \rpar - \lpar \Omega _{\rm p}^2 /\lpar {\rm \omega} _0^2 - \Omega _{\rm c}^2 \rpar \rpar }}.}$$3. THIRD-HARMONIC BRILLOUIN INSTABILITY ANALISIS
As the Brillouin instability switches into the third-harmonic scattering process, the density perturbation at (2ω 0 ± ω, k ± 2k 0) couples with the electron velocity at (ω 0, k 0) to generate a third-harmonic nonlinear current density at (3ω 0 ± ω, k ± 3k 0). Therefore, the wave equation for the third-harmonic scattered waves becomes
$$\left({\displaystyle{{{\partial ^2}} \over {\partial {t^2}}} - {c^2}\displaystyle{{{\partial ^2}} \over {\partial {z^2}}}} \right){A_{x\lpar 3{{\rm \omega} _0} \pm {\rm \omega} \rpar }}=4{\rm \pi} {J_{x\lpar 3{{\rm \omega} _0} \pm {\rm \omega} \rpar }}\comma \;$$
where
${J_{x\lpar 3{{\rm \omega} _0} \pm {\rm \omega} \rpar }}$ is the third-harmonic nonlinear current density associated with the third-harmonic Brillouin scattering and is determined using the equation of continuity for density perturbation at (2ω 0 ± ω), which has the form
with
$${{{\rm \sigma} _0}=- {{{{c^2}a_0^2\, k_0^2 [\lpar {\rm \alpha} _0^2+4\rpar {\rm \omega} _0^2 {\rm \omega} _{\rm c}^2+2\lpar {\rm \alpha} _0^2 - 1\rpar {\rm \omega} _0^4+2{{\rm \alpha} _0}{\rm \omega} _0^3 {{\rm \omega} _{\rm c}}+4{{\rm \alpha} _0}{{\rm \omega} _0}{\rm \omega} _{\rm c}^3+{\rm \omega} _{\rm c}^4] } \over {4{{\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar }^2}\lpar 4{\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar }}}}.$$Here, n ω is the low-frequency density fluctuation associated with the electrostatic ion acoustic upper-hybrid wave arising in the stimulated Brillouin scattering, and a 3s and μ3s, which are defined by replacing ω0 with (3ω0 ± ω) in the definition of parameter μ0, are the normalized amplitude and nonlinear coefficient of the scattered wave, respectively. Hence, the linearized nonlinear wave equation governing the propagation of scattered Stokes and anti-Stokes waves with amplitude A 3± associated with the Brillouin scattering in magnetized plasma is obtained as
where k 3± = k ± 3k 0 and ω 3± = 3ω 0 ± ω are the wave number and frequency of the scattered Stokes/anti-Stokes waves, respectively. Combining the equation of electron's motion with the continuity equation in the direction of laser wave gives the second-coupled equation associated with the Brillouin scattering of X-mode laser wave in a magnetized plasma
$$\displaystyle{e \over m}\left({1+\displaystyle{{{\rm \omega} _{\rm c}^2 } \over {{\rm \omega} _{\rm p}^2 }}} \right)\displaystyle{{\partial {{\rm \phi} _{\rm s}}} \over {\partial z}}=\displaystyle{{{e^2}} \over {{m^2}{c^2}}}\displaystyle{\partial \over {\partial z}}[{{\rm \mu} _{3{\rm s}}}{{\bf A}_{3{\rm s}}} \cdot {{\bf A}_{3x}}+{{\rm \chi} _{3{\rm s}}}{{\bf A}_{3{\rm s}}} \cdot {{\bf A}_{3z}}] +\displaystyle{{v_{\rm T}^2 } \over {{n_0}}}\displaystyle{{\partial {n_{\rm \omega} }} \over {\partial z}}.$$ The second term on the right-hand side of Eq. (6) is due to the thermal pressure of plasma electrons, where
${v_{\rm T}}\, \lpar =\sqrt {{T_{\rm e}}/m} \rpar $ is the electron thermal velocity. In this equation, ϕs is the electrostatic potential of plasma fluctuations, and A3, A3s are the vector potentials of the third-harmonic incident and scattered waves, respectively. Also, μ3s is defined by replacing (3ω0 ± ω) with ω0 in the definition of parameter μ0, and parameter χ 3s [which is determined by replacing (3ω0 ± ω) with ω0 in the definition of parameter χ 0] is given by
$$\eqalign{& {{\rm \chi} _0} = \displaystyle{{{{\rm \alpha} _0}{\rm \omega} _0^2 + {{\rm \omega} _0}{{\rm \omega} _{\rm c}}} \over {{\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 }} + \displaystyle{{a_0^2 } \over {{{\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar }^4}}} \cr & \qquad \times \left[\displaystyle{{{c^2}k_0^2 } \over {4\lpar 4{\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar }}\lpar {M_1}{{\rm \omega} _{\rm c}} + {M_3}\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar \rpar \right. \cr & \qquad \left. - \displaystyle{3 \over 8}\lpar {M_2}{{\rm \omega} _{\rm c}} + {M_4}\lpar {\rm \omega} _0^2 - {\rm \omega} _{\rm c}^2 \rpar \rpar \right]}$$with
$$\eqalign{& {M_3} = {{\rm \alpha} _0}\lpar 3{\rm \alpha} _0^2 - 25\rpar {\rm \omega} _0^4 {\rm \omega} _{\rm c}^2 + \lpar 14{\rm \alpha} _0^2 - 9\rpar {\rm \omega} _0^3 {\rm \omega} _{\rm c}^3 - \lpar 26{\rm \alpha} _0^2 + 1\rpar {\rm \omega} _0^5 {{\rm \omega} _{\rm c}} \cr & \quad \quad - {{\rm \alpha} _0}\lpar 6{\rm \alpha} _0^2 + 3\rpar {\rm \omega} _0^6 + 15{\rm \alpha} _0^2 {\rm \omega} _{\rm c}^4 + 7{{\rm \omega} _0}{\rm \omega} _{\rm c}^5 }$$and
$$\eqalign{& {M_4} = {{\rm \alpha} _0}\lpar {\rm \alpha} _0^2 + 3\rpar {\rm \omega} _0^6 + \lpar 3{\rm \alpha} _0^2 + 1\rpar \lpar {\rm \omega} _0^3 {\rm \omega} _{\rm c}^3 + 3{\rm \omega} _0^5 {{\rm \omega} _{\rm c}}\rpar \cr & \quad \quad + {{\rm \alpha} _0}\lpar 3{\rm \alpha} _0^2 + 6{{\rm \alpha} _0} + 3\rpar {\rm \omega} _0^4 {\rm \omega} _{\rm c}^2 .}$$Now, combining the equations of ion motion and continuity gives (Paknezhad, Reference Paknezhad2013c)
where M and Z are the ion rest mass and the ion charge state respectively and Ωc (=e B0/Mc) is the ion cyclotron frequency, and we assumed that both electron and ion fluids have the same density in the plasma. By substituting ϕs from Eq. (6) into Eq. (7), we finally obtain a nonlinear equation for the low-frequency density fluctuation in plasma:
$$\eqalign{&\left[{\displaystyle{{{{\partial} ^2}} \over {{\partial} {t^2}}}+\Omega _{\rm c}^2 - c_{\rm p}^2 \displaystyle{{{\rm \omega} _{\rm p}^2 } \over {{\rm \omega} _{{\rm UH}}^2 }}\displaystyle{\partial {^2} \over {{\partial} {z^2}}}} \right]{n_{\rm \omega} } \cr & \quad=\displaystyle{{{\rm \omega} _{\rm p}^2 } \over {{\rm \omega} _{{\rm UH}}^2 }}\displaystyle{{{n_0}Z{e^2}} \over {mM{c^2}}}\displaystyle{{{\partial ^2}} \over {\partial {z^2}}}[{{\rm \mu} _{3{\rm s}}}{{\bf A}_{3{\rm s}}} \cdot {{\bf A}_{3x}}+{{\rm \chi} _{3{\rm s}}}{{\bf A}_{3{\rm s}}} \cdot {{\bf A}_{3z}}] \comma \; }$$
where c s = (ZT e/M)1/2 is the ion acoustic velocity. Here,
${{\bf A}_{3x}}={A_0}\, {e^{i[ 3\lpar {k_0}z - {{\rm \omega} _0}t\rpar ]}}$ and
${{\bf A}_{3z}}=i{{\rm \alpha} _{3{{\rm \omega} _0}}}{A_0}\, {e^{i[3\lpar {k_0}z - {{\rm \omega} _0}t\rpar }}$ are the third-harmonic vector potential components, and
${A_{3{\rm s}x}}=$
${A_{3 \pm }}{e^{i[\lpar k \pm 3{k_0}\rpar z - \lpar 3{{\rm \omega} _0} \pm {\rm \omega} \rpar t\rpar]}}$ and
${{\bf A}_{3{\rm s}z}}=i{{\rm \alpha} _{3 \pm }}{A_{3 \pm }}\, {e^{i[ \lpar k \pm 3{k_0}\rpar z - \lpar 3{{\rm \omega} _0} \pm {\rm \omega} \rpar t\rpar]}}$ are the vector potential components of the scattered Stokes and anti-Stokes waves associated with the Brillouin scattering, where
${{\rm \alpha} _{3{{\rm \omega} _0}}}$ and α 3± are respectively determined by replacing 3ω 0 and (3ω 0 ± ω) with ω 0 in the definition of α 0. Fourier transform of Eq. (8), and then combining with Eq. (5) gives the dispersion relation of the exited ion acoustic upper-hybrid wave
$$\eqalign{& {{\rm \omega} ^2} - \Omega _{\rm c}^2 - {k^2}c_{\rm s}^2 \displaystyle{{{\rm \omega} _{\rm p}^2 } \over {{\rm \omega} _{{\rm UH}}^2 }} \cr &= \displaystyle{{{c^2}{k^2}\Omega _{\rm p}^2 a_0^2 {{\rm \mu} _0}{{\rm \sigma} _0}} \over 4}\displaystyle{{{\rm \omega} _{\rm p}^2 } \over {{\rm \omega} _{{\rm UH}}^2 }} \left[\displaystyle{{\lpar {{\rm \mu} _{3 + }} + {{\rm \chi} _{3 + }}{{\rm \alpha} _{3{{\rm \omega} _0}}}{{\rm \alpha} _{3 + }}\rpar } \over {{D_{3 + }}}} \right. \cr & \left. \;\quad+ \displaystyle{{\lpar {{\rm \mu} _{3 - }} + {{\rm \chi} _{3 - }}{{\rm \alpha} _{3{{\rm \omega} _0}}}{{\rm \alpha} _{3 - }}\rpar } \over {{D_{3 - }}}} \right].}$$ Here,
${\Omega _{\rm p}}\; \lpar ={{\rm \omega} _{\rm p}}\sqrt {Zm/M} \rpar $ is the plasma ion frequency, and
${D_{3 \pm }}\; [\!\! ={\rm \omega} _{3 \pm }^2 - {c^2}k_{3 \pm }^2 - {{\rm \mu} _{3 \pm }}{\rm \omega} _p^2 ]$ is the dispersion coefficient of the third-harmonic scattered waves (Stokes/anti-Stokes) in the Brillouin scattering process.
For Brillouin instability, maximum growth rate occurs in backward scattering in which both the scattered Stocks light wave and the exited ion acoustic upper-hybrid wave are resonant modes (ω = Ω, D 3− = 0), that is, when
where
$\Omega={\lpar \Omega _{\rm c}^2+\lpar {\rm \omega} _{\rm p}^2 /{\rm \omega} _{{\rm UH}}^2 \rpar {k^2}c_{\rm s}^2 \rpar ^{1/2}}$. The frequency-matching condition for Stokes wave is ω 3− = 3ω 0 − Ω and the wave number k is determined from Eq. (10). Assuming Ωc≪ω 0, we obtain the normalized wave number of the ion acoustic upper-hybrid wave in a transversely magnetized plasma
$$\eqalign{\displaystyle{{k}\over{k_o}} & =3 \left[\displaystyle{{1 - \lpar {{\rm \omega} _{\rm p}}/{{\rm \omega} _{{\rm UH}}}\rpar \lpar {c_{\rm s}}/c\rpar {{\lpar 1 - {{\rm \mu} _0}\lpar {\rm \omega} _{\rm p}^2 /{\rm \omega} _0^2 \rpar \rpar }^{ - 1/2}}} \over {1 - \lpar {\rm \omega} _{\rm p}^2 /{\rm \omega} _{{\rm UH}}^2 \rpar \lpar c_{\rm s}^2 /{c^2}\rpar }} \right. \cr & \left.\quad + \displaystyle{{\matrix{[ {\lpar 1 - \lpar {{\rm \omega} _{\rm p}}/{{\rm \omega} _{{\rm UH}}}\rpar \lpar {c_{\rm s}}/c\rpar {{\lpar 1 - {{\rm \mu} _0}\lpar {\rm \omega} _{\rm p}^2 /{\rm \omega} _0^2 \rpar \rpar }^{ - 1/2}}\rpar }^2 \cr + \; \lpar 1 - \lpar {\rm \omega} _{\rm p}^2 /{\rm \omega} _{{\rm UH}}^2 \rpar \lpar c_{\rm s}^2 /{c^2}\rpar \rpar {{\lpar \lpar \lpar 1 + \lpar \Omega _{\rm c}^2 /9{\rm \omega} _0^2 \rpar \rpar}} \cr \lpar {{\rm \mu} _{3 - }} + {{\rm \chi} _{3 - }} {{\rm \alpha} _{3 - }} \rpar \lpar {\rm \omega} _{\rm p}^2 /9{\rm \omega} _0^2 \rpar \rpar /\lpar \lpar 1 - {{\rm \mu} _0}\lpar {\rm \omega} _{\rm p}^2 /{\rm \omega} _0^2 \rpar \rpar - 1\rpar ] ^{1/2} \cr} } \over {1 - \lpar {\rm \omega} _{\rm p}^2 /{\rm \omega} _{{\rm UH}}^2 \rpar \lpar c_{\rm s}^2 /{c^2}\rpar }} \right]}.$$By ignoring the nonlinear terms, Eq. (11) reduces to
$$\displaystyle{{{k_{\rm p}}} \over {{k_0}}} \simeq 6\, \left({1 - \displaystyle{{{{\rm \omega} _{\rm p}}} \over {{{\rm \omega} _{{\rm UH}}}}}\displaystyle{{{c_{\rm s}}} \over c}} \right){\left({1 - \displaystyle{{{\rm \omega} _{\rm p}^2 } \over {{\rm \omega} _{{\rm UH}}^2 }}\displaystyle{{c_{\rm s}^2 } \over {{c^2}}}} \right)^{ - 1}}\comma \; $$which is three times the normalized linear wave number of ion acoustic upper-hybrid wave in fundamental Brillouin backscattering in a magnetized plasma (Paknezhad, Reference Paknezhad2013a). Note that for Brillouin backscattering, only the downshifted light wave needs to be retained (Kruer, Reference Kruer1998; Paknezhad, Reference Paknezhad2012; Reference Paknezhad2013a), and assuming the weakly coupled regime (Γ≪kc s), we substitute ω = Ω + iΓNBBS in Eq. (9) to find
$$\eqalign{& 2i{\Gamma _{{\rm NBBS}}}\Omega \, [\!\!-\!\! 2i{\Gamma _{{\rm NBBS}}}\lpar {{\rm \omega} _0} - \Omega \rpar] \cr & \quad= \displaystyle{{{\rm \omega} _{\rm p}^2 } \over {{\rm \omega} _{{\rm UH}}^2 }} \displaystyle{{{c^2}{k^2}a_0^2 \Omega _{\rm p}^2 {{\rm \mu} _0}{{\rm \sigma} _0}\lpar {{\rm \mu} _{3 - }} + {{\rm \chi} _{3 - }}{{\rm \alpha} _{3{{\rm \omega} _0}}}{{\rm \alpha} _{3 - }}\rpar } \over 4}.}$$ The real part of Eq. (13) gives the nonlinear growth rate of 3HBBS instability in the backscattering of an extraordinary laser wave in a weakly coupled magnetized plasma, where ΓNBBS≪ω UH and
$a_0^2 \ll 1$,
$${\Gamma _{3{\rm HBBS}}}=\displaystyle{{ck{a_0}{\Omega _{\rm p}}} \over 4}\displaystyle{{{{\rm \omega} _{\rm p}}} \over {{{\rm \omega} _{{\rm UH}}}}}\sqrt {\displaystyle{{{{\rm \mu} _0}{{\rm \sigma} _0}\lpar {{\rm \mu} _{3 - }} - {{\rm \chi} _{3 - }}{{\rm \alpha} _{3 - }}{{\rm \alpha} _{3{{\rm \omega} _0}}}\rpar } \over {\Omega \, \lpar 3{{\rm \omega} _0} - \Omega \rpar }}} .$$In Eq. (6), parameter σ0 arises as a result of the effect of 3HBBS of laser in a mildly relativistic magnetized plasma. Now, we investigate the variation of the growth rate of 3HBBS instability on the basis of Eq. (14).
In Figure 3, the variation of normalized growth rate of 3HBBS instability Γ3HBBS/ω0 has been plotted against the normalized wave number k/k 0 for a 0 = 0.02, and two values of ωc/ω0 = 0 and ωc/ω0 = 0.1. It is demonstrated that for the given values, the starting point of k/k 0 for 3HBBS instability in the unmagnetized plasma (ωc/ω0 = 0) is around 5.8824, whereas it is 6 for the magnetized plasma (ωc/ω0 = 0.1). For both cases, as the normalized wave number decreases (by increasing plasma density), the growth rate increases and then saturates for a low value of k/k 0 (high plasma densities).
Variation of the normalized growth rate of 3HBBS instability as a function of normalized wave number k/k 0 in unmagnetized (ωc/ω0 = 0) and magnetized (ωc/ω0 = 0.1) plasmas for a 0 = 0.02, c s/c = 0.02.

Figure 4 describes the variation of normalized growth rate of 3HBBS instability Γ3HBBS/ω0 with normalized plasma frequency ωp/ω0 for ωc/ω0 = 0.05, and two values of a 0 = 0.02 and a 0 = 0.03. It is observed that an increase in the plasma frequency (plasma density) leads to an increase in the growth rate. The growth rate also increases with laser amplitude.
Variation of normalized growth rate of 3HBBS instability as a function of normalized plasma frequency for ωc/ω0 = 0.05, and c s/c = 0.02.

Figure 5 depicts the variation of normalized growth rate of 3HBBS instability as a function of normalized cyclotron frequency ωc/ω0 at a 0 = 0.02 and ωp/ω0 = 0.1. Similar to the third-harmonic Raman backscattering instability (Paknezhad, Reference Paknezhad2013c), the growth rate decreases by increasing the external magnetic field. However, in the case of third-harmonic Raman backscattering, the growth rate has a further decrease because of the absence of ion's motion.
Variation of normalized growth rate of 3HBBS instability as a function of normalized electron cyclotron frequency for a 0 = 0.02, c s/c = 0.02, and ωp/ω0 = 0.1.

4. CONCLUSION
To summarize, we have developed a theoretical formalism to calculate the growth rate of nonlinear 3HBBS instability, which is observable in plasmas with a density higher than the quarter critical density. We have demonstrated that the growth rate has a reduction in magnetized plasma in comparison with the unmagnetized plasma. This result is found to be in good agreement with that obtained by Grebogy and Liu (Reference Grebogi and Liu1980) for fundamental Raman and Brillouin backscattering instabilities in a magnetized plasma. However, the result differs from those of Salimullah et al. (1994). Additionally, the growth rate increases with plasma density and laser wave amplitude, which agrees with the results attained for fundamental Brillouin scattering in our previous work (Paknezhad, Reference Paknezhad2012); however, the growth rate of 3HBBS instability obtained in this paper is less than that of fundamental Brillouin backscattering instability. It is also less than the growth rate of third-harmonic Raman backscattering instability that has been obtained in our previous study (Paknezhad, Reference Paknezhad2013c).
Acknowledgement
This work is a part of a research plan entitled “Nonlinear study of Brilouin instability in a magnetized plasma” and was supported in part by the Islamic Azad University-Shabestar Branch under contract number 51953890818005.




