Correct Answer - Ellipse having foci (a,b) and (0,0)
Let the given circles ` C_(1) and C_(2)` have centres `O_(1) and O_(2)` ad radii `r_(1) and r_(2)`, respectively.
Let the variable circle C touching `C_(1) " internally, " C_(2)` externally have a radius r and centre at O.
Now, `OO_(2)r+r+r_(2)and OO_(1)=r_(1)-r`
`rArr OO_(1)+OO_(2)=r_(1) + r_(2)`
which is greater than `O_(1)O_(2)" as " O_(1)O_(2)lt r_(1)+ r_(2)`
[`therefore C_(2) " lies inside " C_(1)`]
`rArr` Locus of O is an ellipse with foci ` O_(1) and O_(2)`.
Alternate Solution
Let equations of `C_(1) " be " x^(2)+y^(2)=r_(1)^(2) " and of " C_(2) " be " (x-a)^(2)+(y-b)^(2)=r_(2)^(2)`
Let centre C be (h,k) and radius r, they by the given condition
`sqrt((h-a)^(2)+(k-b)^(2)=r+r_(2)) and sqrt(h^(2)+k^(2)=r_(1)-r)`
`rArr sqrt((h-a)^(2)+(k-b)^(2))+sqrt(h^(2)+k^(2))=r_(1)+r_(2)`
Required locus is
`sqrt((x-a)^(2)+(y-b)^(2))+sqrt(x^(2)+y^(2))=r_(1)+r_(2)`
which represents an ellipse whose foci are (a, b) and (0, 0).