Correct Answer - A
Assume that the mass of each sphere is concentrated at this centre. Then we have a system of three point masses situated at the vertices of an equilateral triangle. This force of attraction on A, due to sphere B is
`F_(1)=(GM xx M)/((2R)^(2))=(GM^(2))/(4R^(2))` along `vec(AB)` and the force on A due to sphere C is `F_(2)=(GM^(2))/(4R^(2))` along AC. Thus `F_(1)=F_(2)` in magnitude, As angule between `vec(F)_(1)` and `vec(F)_(2)` is `60^(@)`, the magnitude of the resultant force,
`F=sqrt(F_(1)^(2)+F_(2)^(2)+2F_(1)F_(2)cos 60^(@))`
`F = sqrt(F_(1)^(2)+F_(1)^(2)+2F_(1)F_(1)xx(1)/(2))=sqrt(2F_(1)^(2)+F_(1)^(2))`
`therefore F = sqrt(3F_(1))=(sqrt(3)xxGM^(2))/(4R^(2))`