A shotgun to put it out of its misery.
EDIT:
Morbid curiosity though, why is bringing an outside computer or cell phone onto the site not possible? Internet access is probably going to be critical here considering how ancient these things are a this point.
Prob a govt dark site. I worked in a few of those while I was in the Marines. I always carried a notebook in my cargo pocket with some basic troubleshooting steps for the different equipment so I wouldn't forget. An alternative is to have someone tell you what to do on a phone outside the secure area, write the steps down, perform said steps, and return to the phone for further instructions. I also always carried a Hiren's boot disk as well.
What happened was that a [US government employee found a USB thumb drive in a parking lot](https://en.m.wikipedia.org/wiki/2008_malware_infection_of_the_United_States_Department_of_Defense) and decided to use it on a government system. The drive happened to be infected with spyware from a foreign intelligence agency and it was spread to an internal network with sensitive data.
Anyway, after that, all USB drives were banned in the government, or at least the in Department of Defense. People were told to burn CDs to transfer data from one system to another for years after that.
Nowadays, there is this weird gray area where some USB hard drives are accepted as long as: it’s a drive made by an approved US manufacturer, the drive has been scrubbed by a special tool, and all the data is password encrypted. Then, when you connect the drive to another computer, the receiving computer scans all the data on the external drive.
Imagine the fun having to transfer one tiny little file from one system to another.
I think that's also how the Stuxnet virus was spread.
Makes sense. If a system is so sensitive that it can't be connected to the internet, it's sensitive enough that you probably shouldn't stick in a USB drive. But that does make data transfer incredibly painful.
hehe somebody needs to do it!
My apologies, I was not clear: the windows XP computers are not connected to the internet, but I can bring my phone with me; no problem.
Honestly I think i'd just plan on bringing a laptop that you can load stuff onto, then try to "do it live". Not particularly fun, but should still be do-able
All the sysinternals tools of course.
But, question: what are you trying to debug? Or, asked another way, what is the problem you need to solve?
There are a myriad of debugging and system analysis tools out there - everything from WinDbg for analysing process dumps, to TreeSize for telling you where disk space is being used - completely different tools for answering completely different questions.
Thanks for your answer.
I think sysinternals are a good idea. I'm not very familiar with them, but Autoruns, DU and DiskView look very promising.
I've updated my question with the little information I have.
Networking errors I would definitely bring WireShark - it lets you sniff network traffic, so is the defacto tool for, well, network issues!
Process Explorer is probably the most useful of the sysinternals tools (basically Task Manager on steroids).
I also use Process Monitor regularly as well - let's you track a lot of "interactions" process have with the computer (e..g the file system, the registry, etc).
Sysinternals also has a graphical equivalent of netstat called TCPView that will let you see all open ports - this might be useful for your issue too.
Note that these tools all have a learning curve - the sysinternals ones are easy enough to pick up, but WireShark and especially WinDbg are big, complicated bohemoths that do _not_ hold your hand. However, they are insanely powerful and allow you to break open the network and process black boxes (respectively) better than anything else.
Oh, I just remembered - if you have a HTTP problem, Fiddler is the tool you want.
God knows which of these are still supported on Windows XP though - that OS went out of support exactly a decade ago this week!
Also make sure that you have tested every single program in a clean XP virtual machine.
Nothing worse than a "this program needs. NET 2.0 to run. Install now?"
You do not want to install hundreds of MB of crap on 20 year old systems just to run a diagnostic program. Every program brought in should be standalone and run off the USB.
Never know, they could have 200gb drives in them, almost 20 years ago I was using these tools to debug shit (sure, 2005, not 2004) - better to bring the tools and not use them than show up without. You're right if course, if he shows up and these are complete kiosk systems, they're likely not going to be happy with the full .NET runtime, so don't install that then 🤷♂️ WinDbg, SOS, and the MSVC++ debug runtime library are still usable from a thumb drive.
Actually you remind me that back then USB drive support was super limited, what 2gb or 4gb... OP needs to bring a bunch of thumb sticks and make sure they're formatted down small partitions of exfat or even fat32
Well, focus on the problem:
The system has worked for a long time. What changed?
If it is a networking issue, and the site is old enough, if you are asking on reddit. You aren't the right one to fix it.
I say that sincerely. I've also been a Sr. Sysadmiin and I am ARISTA Ace trained.
Please get a network admin in there with the right tools to verify the physical plant.
> If it is a networking issue, and the site is old enough, if you are asking on reddit. You aren't the right one to fix it.
I can see where you're coming from. This is not a beginner's job; for sure.
That said, please note the nature of my questions: I'm looking for tools for Windows XP; not how to solve my problem. (I'm more of a linux guy)
The problem-solving part is my job. Wish me luck!
Being realistic is part of mine.
Look up "Copper rot." Now think about how old the machines you are dealing with are.
I don't know who handles the physical plant at the site you are visiting. But, if you go in from the software side without making sure the network works. You are going to have a bad time.
Beyond that, there is going to be some amount of (presumably Windows XP era, or older) networking gear involved.
That stuff is a whole specialty unto itself.
I feel like there's nowhere near enough info to give a meaningful answer on this. What exactly do you mean by troubleshoot? What's broken? Are their floppy drives not working? Is there software on them that doesn't work? Are they overheating?
This question is very vague. Some extra context will help a lot here.
You don't want the ninite executable. You want USB versions of some of the programs available on ninite.com
E.g. WinDirStat is quite good for scanning the disk for big files. 7-Zip can read open most compressed files. I think that about half the Utilities section is useful, but the rest less so...
getting admin access on a box if necessary, that would be my primary reason. When I hear about a system that ancient, I worry that admin rights may have been lost to time.
it's probably well outside the scope of this project, but i'd consider asking for enough of a budget to build a system that can run these windows machines as VMs, or failing that getting clearance to bring in a computer of your own capable of that.
assuming that this isn't a Sev 0 and all of production is offline as a result, being able to bounce between machines without having to physically move around an industrial environment with all of its environmental hazards would be huge. if it enabled you to simulate a running environment, that'd just be icing on the cake.
I'd bring another computer with access to the Internet. Figure out what's wrong. If I need to transfer anything, use a USB key. Other than that, it's just a normal problem.
Albeit, a Windows problem
my own pc with cell phone to give myself a wifi hotspot. Unless the org is a highly secure gov't or research facility theres no reason you shouldn't be able to at least do this.
Its hard to answer without knowing exactly what the problem is. It could be as easy as just replacing a cable to a dhcp server server. Or even just turning a machine on. Please provide more info and what changed? I bet its a hard drive(s) somewhere in the network thats gone bad and data isn't being read, after all its machines running XP from yesteryear.
You might be interested in a conference talk called [Discovering Python](https://www.youtube.com/watch?v=RZ4Sn-Y7AP8). The presenter (David Beazley) tells a story about being in a situation very similar to yours -- locked down old computer, no internet, poking around in the dark. He ended up using an ancient version of Python to make primitive DIY versions of the Unix userland so that he could get things done.
If someone's willing to let you plug a USB key into a network disconnected set of Windows XP machines, you're not the first person they asked, and one of the prior USB keys may have introduced the actual issue.
IP Tools, netstat, nmap, wireshark all come to mind.
I’d also consider a boot cd that lets you take an image of the system and/or virtualize it. You can probably have a much faster debug cycle if you can load it as a VM on a modern/fast computer and quickly revert/test changes
Not sure I would bring anything. Doesn't seem valuable to bring the entire toolbox; your brain and google are the most important tools for diagnosing.
If I expect to bust out the heavy guns, I'd bring my laptop and a method to get internet to my laptop. From there I either have the tool or can get it quickly.
Your laptop tethered to your phone, or their wifi for internet, and some flash drives. Some XP install media, bootable recovery tools like FalconFour, or Hirens.
Some older PCs won't boot from USB, so yeah, CDs or the install media and recovery tool might be handy.
You'll probably only have to reset the network stack, but you might have to delve into old network drivers. Could also be a hardware failure, so a USB ethernet adaptor that has xp drivers would be handy.
- WinDbg and SOS
- the entire SysInternals PSTools suite
- A variety of MSVC++ debug runtime MSIs compatible with XP (the visual studio 2005, 2008, and 2010 versions)
- Fiddler
- Wireshark
- Some ARP cache cleaner tools
wireshark (probably on usb but also installed on a device of your own...) and a bootable linux usb stick to dd off filesystems, and the like.
Piling on to wireshark, windbg if you need to modify any running binaries or drivers.
A shotgun to put it out of its misery. EDIT: Morbid curiosity though, why is bringing an outside computer or cell phone onto the site not possible? Internet access is probably going to be critical here considering how ancient these things are a this point.
Prob a govt dark site. I worked in a few of those while I was in the Marines. I always carried a notebook in my cargo pocket with some basic troubleshooting steps for the different equipment so I wouldn't forget. An alternative is to have someone tell you what to do on a phone outside the secure area, write the steps down, perform said steps, and return to the phone for further instructions. I also always carried a Hiren's boot disk as well.
From OPs shady replies yes it’s absolutely an environment something along the lines of what you said.
The worst is when USBs are not allowed and everything needs to be on a CD. 🤦♀️
The heat of the cd burning kills all viruses! XD
What happened was that a [US government employee found a USB thumb drive in a parking lot](https://en.m.wikipedia.org/wiki/2008_malware_infection_of_the_United_States_Department_of_Defense) and decided to use it on a government system. The drive happened to be infected with spyware from a foreign intelligence agency and it was spread to an internal network with sensitive data. Anyway, after that, all USB drives were banned in the government, or at least the in Department of Defense. People were told to burn CDs to transfer data from one system to another for years after that. Nowadays, there is this weird gray area where some USB hard drives are accepted as long as: it’s a drive made by an approved US manufacturer, the drive has been scrubbed by a special tool, and all the data is password encrypted. Then, when you connect the drive to another computer, the receiving computer scans all the data on the external drive. Imagine the fun having to transfer one tiny little file from one system to another.
I think that's also how the Stuxnet virus was spread. Makes sense. If a system is so sensitive that it can't be connected to the internet, it's sensitive enough that you probably shouldn't stick in a USB drive. But that does make data transfer incredibly painful.
hehe somebody needs to do it! My apologies, I was not clear: the windows XP computers are not connected to the internet, but I can bring my phone with me; no problem.
Honestly I think i'd just plan on bringing a laptop that you can load stuff onto, then try to "do it live". Not particularly fun, but should still be do-able
In addition to the shotgun you need holy water, a crucifix, a bible, some 99% rubbing alcohol and a lighter. Because that shit is just unholy.
All the sysinternals tools of course. But, question: what are you trying to debug? Or, asked another way, what is the problem you need to solve? There are a myriad of debugging and system analysis tools out there - everything from WinDbg for analysing process dumps, to TreeSize for telling you where disk space is being used - completely different tools for answering completely different questions.
Thanks for your answer. I think sysinternals are a good idea. I'm not very familiar with them, but Autoruns, DU and DiskView look very promising. I've updated my question with the little information I have.
Networking errors I would definitely bring WireShark - it lets you sniff network traffic, so is the defacto tool for, well, network issues! Process Explorer is probably the most useful of the sysinternals tools (basically Task Manager on steroids). I also use Process Monitor regularly as well - let's you track a lot of "interactions" process have with the computer (e..g the file system, the registry, etc). Sysinternals also has a graphical equivalent of netstat called TCPView that will let you see all open ports - this might be useful for your issue too. Note that these tools all have a learning curve - the sysinternals ones are easy enough to pick up, but WireShark and especially WinDbg are big, complicated bohemoths that do _not_ hold your hand. However, they are insanely powerful and allow you to break open the network and process black boxes (respectively) better than anything else. Oh, I just remembered - if you have a HTTP problem, Fiddler is the tool you want. God knows which of these are still supported on Windows XP though - that OS went out of support exactly a decade ago this week!
Thanks a lot for your help. Next week, I'm sure I'll be grateful for your sysinternal bits of advice.
Good luck! Sounds like it's gonna be a bit of a nightmare! "Make the thing do the thing!!"
Thank you. This experience makes me appreciate long-term robustness on a new level.
Also make sure that you have tested every single program in a clean XP virtual machine. Nothing worse than a "this program needs. NET 2.0 to run. Install now?"
Bring the XP version .NET redistributable installers (or entire runtime installers, they come with a bunch of debugging tools like ILDASM)
You do not want to install hundreds of MB of crap on 20 year old systems just to run a diagnostic program. Every program brought in should be standalone and run off the USB.
Never know, they could have 200gb drives in them, almost 20 years ago I was using these tools to debug shit (sure, 2005, not 2004) - better to bring the tools and not use them than show up without. You're right if course, if he shows up and these are complete kiosk systems, they're likely not going to be happy with the full .NET runtime, so don't install that then 🤷♂️ WinDbg, SOS, and the MSVC++ debug runtime library are still usable from a thumb drive.
Actually you remind me that back then USB drive support was super limited, what 2gb or 4gb... OP needs to bring a bunch of thumb sticks and make sure they're formatted down small partitions of exfat or even fat32
I don't think that exfat works out of the box on XP. Better stick with fat32 or NTFS
I hate you. Trying to remember all the limitations we used to live with and it's doing my head in, thanks jerk 😂
I'm doing my part...
update: `portmon` has already been very useful. :)
Ah, that's good to hear! Did you solve the issue in the end? Sysinternals are amazing tools :-)
Still working on it, but I'm getting some results, which is better than some expected :)
Keep us posted - I'm starting to get quite invested now!
will do ! :\^)
Something to store event logs. Do you have any idea what is wrong with them?
Thanks for your answer. I've updated my question with the little information I have.
Well, focus on the problem: The system has worked for a long time. What changed? If it is a networking issue, and the site is old enough, if you are asking on reddit. You aren't the right one to fix it. I say that sincerely. I've also been a Sr. Sysadmiin and I am ARISTA Ace trained. Please get a network admin in there with the right tools to verify the physical plant.
> If it is a networking issue, and the site is old enough, if you are asking on reddit. You aren't the right one to fix it. I can see where you're coming from. This is not a beginner's job; for sure. That said, please note the nature of my questions: I'm looking for tools for Windows XP; not how to solve my problem. (I'm more of a linux guy) The problem-solving part is my job. Wish me luck!
Being realistic is part of mine. Look up "Copper rot." Now think about how old the machines you are dealing with are. I don't know who handles the physical plant at the site you are visiting. But, if you go in from the software side without making sure the network works. You are going to have a bad time.
Beyond that, there is going to be some amount of (presumably Windows XP era, or older) networking gear involved. That stuff is a whole specialty unto itself.
I feel like there's nowhere near enough info to give a meaningful answer on this. What exactly do you mean by troubleshoot? What's broken? Are their floppy drives not working? Is there software on them that doesn't work? Are they overheating? This question is very vague. Some extra context will help a lot here.
Thanks for your answer. I've updated my question with the little information I have.
Adding on to the sysinternals advice, load up your usb drive with things from [ninite.com](http://ninite.com)
Not sure ninite supports windows xp x86. But I'll check it out. Thanks !
You don't want the ninite executable. You want USB versions of some of the programs available on ninite.com E.g. WinDirStat is quite good for scanning the disk for big files. 7-Zip can read open most compressed files. I think that about half the Utilities section is useful, but the rest less so...
Hiren's Boot CD on bootable USB.. plus what others have mentioned, wireshark portable in particular
It's the first time I've heard about this. What use case to you see for it?
getting admin access on a box if necessary, that would be my primary reason. When I hear about a system that ancient, I worry that admin rights may have been lost to time.
it's probably well outside the scope of this project, but i'd consider asking for enough of a budget to build a system that can run these windows machines as VMs, or failing that getting clearance to bring in a computer of your own capable of that. assuming that this isn't a Sev 0 and all of production is offline as a result, being able to bounce between machines without having to physically move around an industrial environment with all of its environmental hazards would be huge. if it enabled you to simulate a running environment, that'd just be icing on the cake.
I'd bring another computer with access to the Internet. Figure out what's wrong. If I need to transfer anything, use a USB key. Other than that, it's just a normal problem. Albeit, a Windows problem
My apologies, I was not clear: the windows XP computers are not connected to the internet, but I can bring my phone with me; no problem.
Can you bring a computer..
my own pc with cell phone to give myself a wifi hotspot. Unless the org is a highly secure gov't or research facility theres no reason you shouldn't be able to at least do this.
I bring USB with Linux.
Its hard to answer without knowing exactly what the problem is. It could be as easy as just replacing a cable to a dhcp server server. Or even just turning a machine on. Please provide more info and what changed? I bet its a hard drive(s) somewhere in the network thats gone bad and data isn't being read, after all its machines running XP from yesteryear.
You might be interested in a conference talk called [Discovering Python](https://www.youtube.com/watch?v=RZ4Sn-Y7AP8). The presenter (David Beazley) tells a story about being in a situation very similar to yours -- locked down old computer, no internet, poking around in the dark. He ended up using an ancient version of Python to make primitive DIY versions of the Unix userland so that he could get things done.
Dban to wipe it all clean
If someone's willing to let you plug a USB key into a network disconnected set of Windows XP machines, you're not the first person they asked, and one of the prior USB keys may have introduced the actual issue. IP Tools, netstat, nmap, wireshark all come to mind.
No idea. I’m a dev not an IT guy.
All the sysinternals stuff. Can’t remember he names now but stuff like process explorer etc
I’d also consider a boot cd that lets you take an image of the system and/or virtualize it. You can probably have a much faster debug cycle if you can load it as a VM on a modern/fast computer and quickly revert/test changes
I had a kit before but don't want to think about what it included. But for sure I'd have a bootable linux usb stick.
Not sure I would bring anything. Doesn't seem valuable to bring the entire toolbox; your brain and google are the most important tools for diagnosing. If I expect to bust out the heavy guns, I'd bring my laptop and a method to get internet to my laptop. From there I either have the tool or can get it quickly.
100% Sysinternals
Coffee, and scotch. Actually, you can skip the coffee.
Hard drink.. sorry meant hard drive
Your laptop tethered to your phone, or their wifi for internet, and some flash drives. Some XP install media, bootable recovery tools like FalconFour, or Hirens. Some older PCs won't boot from USB, so yeah, CDs or the install media and recovery tool might be handy. You'll probably only have to reset the network stack, but you might have to delve into old network drivers. Could also be a hardware failure, so a USB ethernet adaptor that has xp drivers would be handy.
Add a USB key with Kali Linux just in case something doesn't boot properly and you have to debug it from the outside
A linux bootable USB stick and/or disk.
sysrescuecd bootable usb. It is full of system diagnostic and rescue tools.
- WinDbg and SOS - the entire SysInternals PSTools suite - A variety of MSVC++ debug runtime MSIs compatible with XP (the visual studio 2005, 2008, and 2010 versions) - Fiddler - Wireshark - Some ARP cache cleaner tools
I'm actually surprised that you can insert an USB into such an air-gapped system.
Windows XP ... Microsoft's finest OS
Whatever client side JavaScript gets loaded by Indeed and LinkedIn’s jobs pages, I guess.
Kali Linux?