"For example, the program is able to determine whether each square is white or black by calculating its average color and comparing it to the black and white colors" I bet you could get away with determining the color of a corner square and assuming the rest.
Some of the square edges actually belongs to the surroundings squares because the cut is not 100% accurate and the chess piece are not always position at the center of the square but I could get around that.
The way this algorithm works is simple. I compare the RGB values for each pixel of a square to the black color and white color. Black is Red=0, Green=0, Blue=0 and white is Red=255, Green=255 and Blue=255. I add up the 3 RGB values for a pixel, if it is closer to 0 and 255*3, I consider the pixel is mostly black. I keep a counter of black pixels and another for white pixels.
Currently, the algo is just comparing the two counters. If I found most of the pixel were black, I consider it is a black square.
However, I could improve it as following: - > 90% of black pixels: empty black square or black square with black piece - > 90% of white pixels: empty white square or white square with white piece - other white: square with a piece of the opposite color.
For the 3rd case, I might need to implement your idea.