We know that,
the system of linear equations
`a_(1)x + b_(1)y + c_(1)z =0`
`a_(2)x + b_(2)y +c_(2)z =0`
`a_(3)x + b_(3)y + c_(3)z = 0`
has a non-trivial solution, if
`|{:(a_(1), b_(1), c_(1)), (a_(2), b_(2), c_(2)), (a_(3), b_(3), c_(3)):}| =0`
Now, if the given system of linear equations
`x + 3y +7z =0`
`-x +4y + 7z =0`
and `("sin" 3 theta)x + ("cos" 2 theta)y + 2z =0`
has non-trivial solution, then
`|{:(1, 3, 7), (-1, 4, 7), ("sin"3theta, "cos"2 theta, 2):}| =0`
`rArr 1(8-7"cos" 2 theta)-3(-2-7 "sin" 3 theta) + 7 (-"cos" 2 theta 2 theta -4"sin" 3 theta) =0`
`rArr 8-7 "cos" 2 theta + 6 +21 "sin" 3 theta - 7 "cos" 2 theta -28 "sin" 3 theta =0`
`rArr -7 "sin" 3 theta -14 "cos" 2 theta + 14 =0`
`rArr -7(3 "sin" theta -4"sin"^(3) theta)-14 (1-2 "sin"^(2)theta) + 14 =0 " " [because "sin"3 A = 3 "sin" A-4"sin"^(3)A " and cos"2A = 1-2"sin"^(2)A]`
`rArr 28 "sin"^(3) theta + 28"sin"^(2) theta -21"sin" theta-14 + 14 =0`
`rArr 7 "sin" theta [4"sin"^(2) theta + 4"sin" theta -3] =0`
`rArr "sin" theta [4"sin"^(2) theta + 6"sin" theta -2"sin" theta-3] =0`
`rArr "sin" theta [2"sin" theta (2"sin" theta + 3)-1 (2"sin" theta+ 3)]=0`
`rArr ("sin" theta) (2"sin" theta -1) (2"sin"theta +3) = 0`
Now, either `"sin" theta =0 "or" (1)/(2) " "[because "sin" theta ne - (3)/(2) " as" -1 le "sin" theta le 1]`
In given interval `(0, pi)`
`"sin"theta = (1)/(2)`
`rArr theta = (pi)/(6), (5pi)/(6) " " [because "sin" theta ne 0, theta in (0, pi)]`
Hence, 2 solution in `(0, pi)`
