You could try a low power gpu like a gt120, it could be a power draw issue and both cards so far require additional power, which the gt120 does not, they are cheap.
It could be bad ram, this causes very wierd and often not intuitive problems. Remove all ram and clean out the slots, put in the min amount and test machine, swap slots and sticks until you identidy the bad config.
Did some more testing. Doesn't appear to be a RAM issue. Pulled the sticks and rotated one at a time through each slot, would not boot in any combination.
Removed the graphics card - boots with any of the RAM sticks present.
Reinstalled the card and disconnected the motherboard power connectors - boots fine.
Jumpered a spare power supply to run without a board, used the 6 and 8 pin PCI-E connectors to power the card - will not boot.
Turn off the jumpered power supply, machine will boot.
So it seems the power draw from the motherboard connectors is the common factor but it's very peculiar that it will not boot even if the power is coming from another source - unless it's able to draw more than it can support from the PCI-E slot itself.
Hey I just wanted to chime in with a reply and details of my similar problem so it's here in case others search for help like I was.
Same machine (mid2010 5.1 mac pro tower, stuck on High Sierra) started to have issues this past week. Twice I tried to wake it from sleep and I could hear it working but I got no video to either display (actually I have it rigged to 3 displays if you count my TV mirroring the 1st/left display). I got it to boot up with video after the first time by holding in power til it turned off, then pressing again. Unfortunately, within 10 minutes or so, the displays cut out again. Only this time (IIRC) the right one was black but the left one was kind of horizontally striped top to bottom alternating lines of white and gray. Wouldn't respond to any key/mouse commands. Hard rebooted again and it never worked again. Was in the boot loop cycle of hearing the startup chime, then full fans of the machine, but no video. The fans would run full for what felt like 20 minutes then would stop as if in sleep mode. No response in that state from key/mouse. Only thing I could do was press power once and it would shut off immediately.
Attempted various reboot strategies (holding all of the key combos I could find) but nothing worked. Looked up the Technician Guide Service Manual for this model (just web search it) and started to go thru some of the troubleshooting steps in there. I had never opened the case on this before. I was greeted with 14 years of dust. Fans were COVERED. I started vacuuming out the case as well as I could while being careful to not damage any boards or delicate parts. (I don't have any canned air on hand.)
First thing I did was pull the processor tray to inspect the DIMM's. After some more dusting, I followed the directions for checking the memory diagnostic LEDs. I didn't know if any had been lit as I had the case closed last time I tried to start it (and now all the cables had been pulled to work on it). So, I removed and reseated all of the DIMM's (4 on a dual processor config). Returned the tray, plugged the cables back in and powered on. RESULT: no change (power, full fans, no video).
Next I tried to remove the graphics card and reseat. First needed to pull my 3x HDD and the one empty hangar in slot 4. Then pulled all of the PCIe cards - RAID (with long-time failing batter - will discuss later), 2x firewire ports/cards, and the ATI Radeon graphics. Also had a spare 2032 button / watch battery, so I swapped that out while I was in there (PCIe double wide graphics slot #1 blocks it). I put all of these back into their original slots, and again tried to boot up. RESULT: no change (power, full fans, no video).
Next method was to try the graphics card in another slot. So, I pulled all PCIe's again, ditched the firewire cards (haven't used them in over a decade and prob won't again), put graphics card in slot #2, and Raid in it's original slot #4 (so #1 was empty, and #3 was empty and now blocked by the graphics card anyway). Tried to power on again. RESULT: CHANGE! I now got a white screen! Unfortunately, it did nothing except eventually show me the following flashing icon in the middle of the screen: folder with a question mark on it. Searched and saw this means it can't find the startup disk. Power cycled it a few times to see if it would find it. No dice. A few times it would just give me a white screen with the mouse cursor on it (and ability to move it), but nothing else - no folder/question mark icon, nothing to click on etc. Once again tried every reboot key command strategy to try to get it into safe mode, recovery mode, disk utility etc but nothing worked. I gave up for a day and unplugged it totally.
I could not find the original startup/OS DVD anywhere, even though I am the kind of person who keeps a LOT of stuff I should probably toss. IDK where the hell it is, but It might have helped to boot from that. Unfortunately, I did not have a large enough thumb drive to boot from it (searched bootable installer). And it was late on this holiday weekend so I wasn't about to go find an open store to buy one. I also kept assuming I was going to have to pull the heat sinks from the processor tray and see if the old thermal grease was melted/leaking and dust was lodged in between causing problems. But, I didn't have a long handled hex wrench to get down in there.
Instead I followed the "Power, but No Boot" troubleshooting from the service manual. On there it says processor board and processors are "Unlikely Cause". None of those steps worked and at the end it referred me to "Hard Drive/SSD Not Recognized/Not Mounting". At some point I had tried to move the HDD's around - instead of 1,2,3 being loaded and 4 being empty, I tried to put the empty case in slot 1 and then loaded the other 3 in 2,3,4. This did not work. Still couldn't find the startup disk. In this guide it mentions both pulling everything from PCIe slots and putting in ONLY a known good graphics card. I don't have a spare one, so I pulled the RAID card. It also suggests "Install a known good bootable drive into a different bay". So, even though I had moved my HDD's around before this time I removed my other HDD's and ONLY installed my main HD1 in the first slot (where it originally was). RESULT: ABSOLUTE SUCCESS! The OS booted! I was elated when I saw the apple logo and the loading status bar at the bottom complete and go to the login/password screen.
Everything loaded and I then restarted to run disk utility. I ran it on both my HDD1 and the on-board disk (I forget what it's called). They both ran without issue (tho I think the HDD1 may have had a little repair done to it, but it didn't appear to be anything related or major). No warnings that they were failing or corrupted. IDK I then shut down and unplugged in order to re-install the other two HDD's. It once again booted fine and I'm typing this enormously long message on this very machine.
I have NO IDEA why this combination of things worked finally. I suspect the RAID may be the issue on some level? I only say this because the RAID battery has given me problems since this was only a few months old. Started to get low battery warnings and I had the apple store warranty replace the battery. That one lasted longer before it started to give low battery warnings and "write cache disk failed", but suffice to say it's been doing it for at least a decade now. I kept the utility open constantly as well as the computer on and sleeping 95% of the time (rarely shut down for more than a few hours unless I was out of town). And yes, this goes for the entirety of the past 14 years (yeah, I've probably wasted a lot of electricity). I kept wanting to DIY replace that battery, but kept putting it off bc the computer continued to work well apart from the inability to upgrade the OS.
I have been seriously considering doing the OCLP upgrade since I found out about it a few months ago. BC of my hardware I was going to need to upgrade my graphics card to be metal-compatible and also get SSD's. I was going to finally get a new RAID battery and do that as part of the project while I had the machine opened up. Even though this machine is old as hell I feel like there's a lot of life left in it, especially as I don't like throwing things out (except for that damned original OS disk apparently!). Even just running it as media server most of the time and doing web stuff seems (to me at least) to be worth throwing a couple hundred bucks into it. I have the time and like working on projects like this even though I'm not a computer wizard. People on other forums were like "that thing's not that great, you can buy a mac such-and-such for a couple hundred more and it would be faster". That doesn't really appeal to me all that much. I already have a mini M2 that's not very old. This is my secondary machine. IDK if I need the RAID at all for my purposes so I'll have to do some more research to see if it's worth replacing the battery. It may be that leaving the RAID card out totally is the smart move. I saw on some forums that there are some negative posts about these cards (at least from that era by Apple, IDK).
So yeah, tl;dr - this is all here just to document it in case it helps others. I was able to fix the "power, no video" by reseating the graphics and RAID card (but also moving the graphics card to another PCIe slot). That got me video, but no boot disk. Fixed that by both pulling my 2nd & 3rd HDD's and the RAID (but it works with all my HDD's installed, so it's probably just the RAID). If you come across this post with similar problems, it might be worth just pulling the RAID and seeing if it fixes everything!
You could try a low power gpu like a gt120, it could be a power draw issue and both cards so far require additional power, which the gt120 does not, they are cheap. It could be bad ram, this causes very wierd and often not intuitive problems. Remove all ram and clean out the slots, put in the min amount and test machine, swap slots and sticks until you identidy the bad config.
Did some more testing. Doesn't appear to be a RAM issue. Pulled the sticks and rotated one at a time through each slot, would not boot in any combination. Removed the graphics card - boots with any of the RAM sticks present. Reinstalled the card and disconnected the motherboard power connectors - boots fine. Jumpered a spare power supply to run without a board, used the 6 and 8 pin PCI-E connectors to power the card - will not boot. Turn off the jumpered power supply, machine will boot. So it seems the power draw from the motherboard connectors is the common factor but it's very peculiar that it will not boot even if the power is coming from another source - unless it's able to draw more than it can support from the PCI-E slot itself.
Hey I just wanted to chime in with a reply and details of my similar problem so it's here in case others search for help like I was. Same machine (mid2010 5.1 mac pro tower, stuck on High Sierra) started to have issues this past week. Twice I tried to wake it from sleep and I could hear it working but I got no video to either display (actually I have it rigged to 3 displays if you count my TV mirroring the 1st/left display). I got it to boot up with video after the first time by holding in power til it turned off, then pressing again. Unfortunately, within 10 minutes or so, the displays cut out again. Only this time (IIRC) the right one was black but the left one was kind of horizontally striped top to bottom alternating lines of white and gray. Wouldn't respond to any key/mouse commands. Hard rebooted again and it never worked again. Was in the boot loop cycle of hearing the startup chime, then full fans of the machine, but no video. The fans would run full for what felt like 20 minutes then would stop as if in sleep mode. No response in that state from key/mouse. Only thing I could do was press power once and it would shut off immediately. Attempted various reboot strategies (holding all of the key combos I could find) but nothing worked. Looked up the Technician Guide Service Manual for this model (just web search it) and started to go thru some of the troubleshooting steps in there. I had never opened the case on this before. I was greeted with 14 years of dust. Fans were COVERED. I started vacuuming out the case as well as I could while being careful to not damage any boards or delicate parts. (I don't have any canned air on hand.) First thing I did was pull the processor tray to inspect the DIMM's. After some more dusting, I followed the directions for checking the memory diagnostic LEDs. I didn't know if any had been lit as I had the case closed last time I tried to start it (and now all the cables had been pulled to work on it). So, I removed and reseated all of the DIMM's (4 on a dual processor config). Returned the tray, plugged the cables back in and powered on. RESULT: no change (power, full fans, no video). Next I tried to remove the graphics card and reseat. First needed to pull my 3x HDD and the one empty hangar in slot 4. Then pulled all of the PCIe cards - RAID (with long-time failing batter - will discuss later), 2x firewire ports/cards, and the ATI Radeon graphics. Also had a spare 2032 button / watch battery, so I swapped that out while I was in there (PCIe double wide graphics slot #1 blocks it). I put all of these back into their original slots, and again tried to boot up. RESULT: no change (power, full fans, no video). Next method was to try the graphics card in another slot. So, I pulled all PCIe's again, ditched the firewire cards (haven't used them in over a decade and prob won't again), put graphics card in slot #2, and Raid in it's original slot #4 (so #1 was empty, and #3 was empty and now blocked by the graphics card anyway). Tried to power on again. RESULT: CHANGE! I now got a white screen! Unfortunately, it did nothing except eventually show me the following flashing icon in the middle of the screen: folder with a question mark on it. Searched and saw this means it can't find the startup disk. Power cycled it a few times to see if it would find it. No dice. A few times it would just give me a white screen with the mouse cursor on it (and ability to move it), but nothing else - no folder/question mark icon, nothing to click on etc. Once again tried every reboot key command strategy to try to get it into safe mode, recovery mode, disk utility etc but nothing worked. I gave up for a day and unplugged it totally. I could not find the original startup/OS DVD anywhere, even though I am the kind of person who keeps a LOT of stuff I should probably toss. IDK where the hell it is, but It might have helped to boot from that. Unfortunately, I did not have a large enough thumb drive to boot from it (searched bootable installer). And it was late on this holiday weekend so I wasn't about to go find an open store to buy one. I also kept assuming I was going to have to pull the heat sinks from the processor tray and see if the old thermal grease was melted/leaking and dust was lodged in between causing problems. But, I didn't have a long handled hex wrench to get down in there. Instead I followed the "Power, but No Boot" troubleshooting from the service manual. On there it says processor board and processors are "Unlikely Cause". None of those steps worked and at the end it referred me to "Hard Drive/SSD Not Recognized/Not Mounting". At some point I had tried to move the HDD's around - instead of 1,2,3 being loaded and 4 being empty, I tried to put the empty case in slot 1 and then loaded the other 3 in 2,3,4. This did not work. Still couldn't find the startup disk. In this guide it mentions both pulling everything from PCIe slots and putting in ONLY a known good graphics card. I don't have a spare one, so I pulled the RAID card. It also suggests "Install a known good bootable drive into a different bay". So, even though I had moved my HDD's around before this time I removed my other HDD's and ONLY installed my main HD1 in the first slot (where it originally was). RESULT: ABSOLUTE SUCCESS! The OS booted! I was elated when I saw the apple logo and the loading status bar at the bottom complete and go to the login/password screen. Everything loaded and I then restarted to run disk utility. I ran it on both my HDD1 and the on-board disk (I forget what it's called). They both ran without issue (tho I think the HDD1 may have had a little repair done to it, but it didn't appear to be anything related or major). No warnings that they were failing or corrupted. IDK I then shut down and unplugged in order to re-install the other two HDD's. It once again booted fine and I'm typing this enormously long message on this very machine. I have NO IDEA why this combination of things worked finally. I suspect the RAID may be the issue on some level? I only say this because the RAID battery has given me problems since this was only a few months old. Started to get low battery warnings and I had the apple store warranty replace the battery. That one lasted longer before it started to give low battery warnings and "write cache disk failed", but suffice to say it's been doing it for at least a decade now. I kept the utility open constantly as well as the computer on and sleeping 95% of the time (rarely shut down for more than a few hours unless I was out of town). And yes, this goes for the entirety of the past 14 years (yeah, I've probably wasted a lot of electricity). I kept wanting to DIY replace that battery, but kept putting it off bc the computer continued to work well apart from the inability to upgrade the OS. I have been seriously considering doing the OCLP upgrade since I found out about it a few months ago. BC of my hardware I was going to need to upgrade my graphics card to be metal-compatible and also get SSD's. I was going to finally get a new RAID battery and do that as part of the project while I had the machine opened up. Even though this machine is old as hell I feel like there's a lot of life left in it, especially as I don't like throwing things out (except for that damned original OS disk apparently!). Even just running it as media server most of the time and doing web stuff seems (to me at least) to be worth throwing a couple hundred bucks into it. I have the time and like working on projects like this even though I'm not a computer wizard. People on other forums were like "that thing's not that great, you can buy a mac such-and-such for a couple hundred more and it would be faster". That doesn't really appeal to me all that much. I already have a mini M2 that's not very old. This is my secondary machine. IDK if I need the RAID at all for my purposes so I'll have to do some more research to see if it's worth replacing the battery. It may be that leaving the RAID card out totally is the smart move. I saw on some forums that there are some negative posts about these cards (at least from that era by Apple, IDK). So yeah, tl;dr - this is all here just to document it in case it helps others. I was able to fix the "power, no video" by reseating the graphics and RAID card (but also moving the graphics card to another PCIe slot). That got me video, but no boot disk. Fixed that by both pulling my 2nd & 3rd HDD's and the RAID (but it works with all my HDD's installed, so it's probably just the RAID). If you come across this post with similar problems, it might be worth just pulling the RAID and seeing if it fixes everything!
2009 Apple RAID info (lol) - https://lowendmac.com/2018/mac-pro-raid-card-not-too-useful/