Correct Answer - Option 2 : Is empty
From question, the vectors given are:
\(\vec a = \alpha \hat i + \hat j + 3\hat k\)
\(\vec b = 2\hat i + \hat j - \alpha \hat k\)
\(\vec c = \alpha \hat i - 2\hat j + 3\hat k\)
From question, the set has three given vectors which are coplanar.
The condition of coplanar is:
\(\left[ {\vec a\vec b\vec c} \right] = 0\)
\(\Rightarrow \left| {\begin{array}{*{20}{c}}
\alpha &1&3\\
2&1&{ - \alpha }\\
\alpha &{ - 2}&3
\end{array}} \right| = 0\)
⇒ α(3 – 2α) -1(6 + α2) + 3(-4 – α) = 0
⇒ 3α – 2α2 – 6 – α2 – 12 – 3α = 0
⇒ -3α2 = 18
⇒ α2 = -6
Which is a imaginary number.
∴ S = ϕ
Thus, the set ‘S’ is an empty set.