Correct option is (3) 308
\(\vec{a}=2 \hat{i}-\hat{j}+3 \hat{k}\)
\(\vec{\mathrm{b}}=3 \hat{\mathrm{i}}-5 \hat{\mathrm{j}}+\hat{\mathrm{k}}\)
\(\vec{\mathrm{a}} \times \vec{\mathrm{c}}=\vec{\mathrm{c}} \times \vec{\mathrm{b}}\)
\(\vec{\mathrm{a}} \times \vec{\mathrm{c}}+\vec{\mathrm{b}} \times \vec{\mathrm{c}}=0\)
\((\vec{a}+\vec{b}) \times \vec{c}=0\)
\(\Rightarrow \vec{\mathrm{c}}=\lambda(\vec{\mathrm{a}}+\vec{\mathrm{b}})\)
\(\vec{\mathrm{c}}=\lambda(5 \hat{\mathrm{i}}-6 \hat{\mathrm{j}}+4 \hat{\mathrm{k}})\) ....(1)
\(|\vec{\mathrm{c}}|^{2}=\lambda^{2}(25+36+16)\)
\(|\vec{\mathrm{c}}|^{2}=77 \lambda^{2}\)
\((\vec{a}+\vec{c}) \cdot(\vec{b}+\vec{c})=168\)
\(\vec{a} \cdot \vec{b}+\vec{a} \cdot \vec{c}+\vec{c} \cdot \vec{b}+|\vec{c}|^{2}=168\)
\(14+\vec{c} \cdot(\vec{a}+\vec{b})+77 \lambda^{2}=168\)
using equation (1)
\(\lambda|5 \hat{\mathrm{i}}-6 \hat{\mathrm{j}}+4 \hat{\mathrm{k}}|^{2}+77 \lambda^{2}=154\)
\(77 \lambda+77 \lambda^{2}-154=0\)
\(\lambda^{2}+\lambda-2=0\)
\(\lambda=-2,1\)
\(\therefore\) Maximum value of \(|\overrightarrow{\mathrm{c}}|^{2}\) occurs when \(\lambda=-2\)
\(|\vec{\mathrm{c}}|^{2}=77 \lambda^{2}\)
\(=77 \times 4\)
= 308
