The major components of panels that are mined and processed are quartz, copper and aluminium. Various attempts to calculate how much greenhouse gas is produced in the mining of these materials and the manufacture of panels suggest that this is recovered by production of carbon-free electricity in about two years. This is regarded as a good return on investment, particularly as panels are expected to be used for well over 25 years. So from a global warming perspective they have a positive environmental impact.
However, there can be negative social and environmental impacts from the mining process, which depend upon where the minerals are sourced and how well the mines are managed. Panels made from ethically sourced materials do exist, so it is worth asking your supplier about this (even if just to get them thinking about it). Ethical Consumer rank different panel manufacturers, but you do need to subscribe to them to access the rationale behind their scoring. Recycling at the end of their lifetime is yet to be fully solved, but once well-established, this should reduce the need for new raw materials.
There are more serious social and environmental issues around the mining of the materials used in electrical batteries, which are mainly lithium, cobalt, nickel and manganese. Again, there is significant work to be done to set up systems and facilities for recycling these batteries, as well as research and development of alternative battery technology that avoids lithium in particular.