Let's give it a try.
I'm going to make two assumptions based on your screenshot:
1. The large circles A and B are touching each other
2. You know the radius (a,b,c) of each circle and want the third one (circle C) to touch both of the first two.
What I'd do is place the center of both circles A and B on the same horizontal line and choose a frame of reference such that the center of circle A is the origin, and the center of circle B is placed at coordinates (a+b, 0)
Now we are looking for the coordinates (x, y) of the center of circle C, placed above the x-axis. Which by the way is one of two solutions, as there is a symmetrical circle C' placed below the x-axis, with the coordinates (x, -y)
We know that if we traced a straight line from the center of C, it would intersect the x-axis at a 90° angle. So drawing that line creates two triangles which each have a right angle in this spot:
- one triangle on the left, where the hypothenuse goes from the center of A (0,0) to the center of C (x,y). Its length is the sum of the radii of A and C.
- one triangle on the right, where the hypothenuse goes from the center of B (a+b,0) to the center of C (x,y). Its length is the sum of the radii of B and C.
Both of these triangles share a vertical segment of length (y).
The left triangle's bottom segment has a length of (x) and the right triangle's bottom segment has a length of: (a+b) - x
We know from Pythagore that the square of the length of the hypothenuse is equal to the sum of the squares of the two sides of each triangle, so we know that:
(a+c)^2 = x^2 + y^2
(b+c)^2 = ((a+b) - x)^2 + y^2
So
y^2 = (a+c)^2 - x^2 = (b+c)^2 - ((a+b) - x)^2
(a+c)^2 - x^2 = (b+c)^2 - ((a+b) - x)^2
Develop it all
a^2 + c^2 + 2ac - x^2 = b^2 + c^2 + 2bc - ((a+b)^2 + x^2 - 2ax - 2bx)
Simplify
a^2 + 2ac = b^2 + 2bc - (a^2 + b^2 + 2ab - 2ax - 2bx)
a^2 + 2ac = 2bc - a^2 - 2ab + 2ax + 2bx
2 a^2 + 2ac - 2bc + 2ab = 2ax + 2bx
a^2 + ac + ab - bc
x = ------------------
a+b
and from there you find y, from y^2 = (a+c)^2 - x^2.
I did notice that in your screenshot A and B are of the same size, so if you knew this from the start it becomes way simpler.
x = a (of course the center of your new circle is at the vertical of the point where both circles touch, which is obvious due to the symmetry of the problem)
and
y = squareroot(c^2 + 2ac)