Correct option is (D) \((\sqrt{(M + m)^2+ M^2})g\)

Free body diagram of pulley is shown in figure. Pulley is in equilibrium under four forces. Three forces as shown in figure and the fourth, which is equal and opposite to the resultant of these three forces, is the force applied by the clamp on the pulley (say F).
Resultant R of these three forces is
R = \((\sqrt{(M + m)^2+ M^2})g\)
Therefore, the force F is equal and opposite to R as shown in figure.
∴ F = \((\sqrt{(M + m)^2+ M^2})g\)
