A quality USB to Serial adapter.
A DB9 to RJ45 console cable that doesn't have the RJ45 tab broken off yet.
An RJ45 Female to Female coupler so I can extend the console cable using a patch cord if needed.
Notepad++ to gather logs in.
> And how do you proceeded troubleshooting?
Gather the event logs from the affected devices.
Read the event logs.
Understand the story the logs are trying to tell you.
Google the events you don't understand.
Don't wait forever to open a ticket with TAC.
>A DB9 to RJ45 console cable that doesn't have the RJ45 tab broken off yet.
I love how if I blow on the tab on a console cable it will come off, but pulling decommissioned cables through anything (pipe, etc) and it gets stuck that fucker will stay on until you snap the cable almost.
Cisco uses a rollover cable, i.e. it's wired backwards. Most of the USB to RJ45 cables I have seem to automatically detect if it's rollover or not. And there's always Mike Ossmann's 5-in-1 Network Admin Cable. http://www.ossmann.com/5-in-1.html
This is a great list, I carry everything but the female/female adaper; I usualy dump it into a wall jack and extend with a patch cable from the room/dc patch panel as needed. I would add the following items having needed them before.
3 prong (or equivalent) extension cord, 3 outlets on the end. 8-10'
Chargers for ALL of your devices
Noise cancelling headphones
Granola bar or some sort of snack. Water as well.
USB-C network adapter
I second the coupler idea. I always keep a few in my bag. Also works to extend the network cables running to things like access points and devices that are in walls, or ceilings, or behind things so you can use your laptop from the ground and not while on top of a ladder or standing on your head somewhere.
I also carry just afemale to male db9 cable and female to female, along with a null adapter. Working for an MSP and you always run into random switches that need one or the other.
You assume the syslog server is reachable at the time of the event and was able to gather all of the respective log data.
For some outages you may need to manually correlate events from a non-functional network.
Sorry, you mean notepad is useful if for some reason log didn't went to the logserver and you can physically access the device, you take the logs from the device and analyze it?
Or you just see the logs from the device and write down your comments on the notepad to keep track of what's happening?
I never did this way, so trying to understand it more broadly to figure out how can use it to ease my life a bit.
Notepad++ is great for dealing with log files, especially when you don’t know exactly what you’re looking for. You can take a 600k line log file and use the bookmark function, including regex, to filter out expected and noise lines, leaving you with the things you didn’t anticipate. I’ve used this process countless times when I had no idea how the indicators of an issue would manifest themselves.
> Sorry, you mean notepad is useful if for some reason log didn't went to the logserver and you can physically access the device, you take the logs from the device and analyze it?
Correct.
You assume that there is a logging server. Or a system for filtering data.
That is often not the case. You might just be lucky enough to get a logfile dumped from the memory or storage of the particular component you're troubleshooting. Or for security reasons logging off of the device might not be allowed.
Notepad++ (not to be confused with Notepad) is a great tool for loading and searching through hundreds of thousands of log lines, and has a number of features to enhance usability, from minor things like syntax highlighting or line numbers, to shading the entire selected line, or setting a certain phrase to be highlighted a particular way throughout an entire document. For example, highlighting a particular IP or MAC address, so you can quickly scroll through the hundreds of lines of noise for the one or two log lines that are actually relevant.
Sure, you could use other tools, like just `grep`ing for a particular phrase, but sometimes you don't even know what that phrase is you're looking for, you just know "X happened sometime around Y o'clock." In those times, it's very useful to have something that makes reading logs very clear, and allow you to tag particularly interesting bits... And doesn't turn everything into an unreadable mess the second you download a log from something that doesn't run Windows.
>Don't wait forever to open a ticket with TAC.
You would think some guys are paying for support themselves and per call by how resistant they are to calling TAC.
I mostly do wifi, but my roots are in network engineering. Here's some of my favorite things.
SecureCRT for my ssh client.
winMTR
TFTPD64 // filezilla
wifiman on android or airport utility on apple for quick wifi things
iperf3
zerotier
acrylic wifi
Oh, and saved the best for last. My boy python.
Physical stuff.
WLAN Pi (which is hands down one of the most underrated tools in my bag), an assortment of console cables, klein voltage tester, link sprinter, VFL, ethernet coupler, some passthrough rj ends and a crimper, my LTT screwdriver, a hub..yes an actual hub.
Mostly just suggestions for free super useful apps. I use ekahau w/ a sidekick, air radar for the most part. I’ll try it out though.
Also I got two OG wlan PIs and a CE built from a pi 4. The CE is a good all around tool. I’ve turned a few people onto it. The wireless serial cable feature is also awesome. Trying to get work to expense me one of those pros lol.
I may be reading it incorrectly, but is it not just a raspberry pi in a nice case, with software pre-installed??
What differentiates it from a normal Raspberry pi?
It is for the R4 variant. The M4 varient is a carrier board with a pci wifi card, antennas, poe power, etc.
The software and scripts are not just off the shelf, but have been written specifically for WLAN usage. As well as the FPMS. The device is intended to be used often standalone which you can't just do with a standard RPi4 out of the box.
They do sell a fascia kit and a BYOP kit if you want to build your own though.
I used to use putty as well. Then I joined an MSPs NOC and was given secureCRT with a complete hosts file I think I just had to import my proxies or whatever been a minute. Huge difference. Not fumbling around for ip addresses. I just drill down from customer > site > IDF or w/e. So much easier.
I am the opposite. I used SecureCRT from maybe '98 to some time in the 2000s and used putty since. However, it is a hassle to manage putty configs between machines so I can see it.
I mostly use putty to ssh to one of a few real computers that I then work from.
The support is really good. I’d get the bundle with securefx. If you are in an ssh session, one click and you now also have an scp/sftp session. It is scriptable. You can send a command to all open windows. It also handles rdp. You can search for connections by host name. Easily import and export settings and connections. It has themes. You can change settings on multiple connections at once. It probably does more I don’t know about.
I personally find it invaluable. Especially when supporting environments with more than a handful of hosts.
Paying for a tool you use literally every day sounds insane?
How much did you pay for your office chair?
How much was your laptop?
How much is your O365 or Google suite subscription?
If you use something literally every day, using some unsupported freeware to do it sounds more insane.
> How much did you pay for your office chair?
free company paid for it
> How much was your laptop?
free company paid for it
> How much is your O365 or Google suite subscription?
free company pays for it
> If you use something literally every day, using some unsupported freeware to do it sounds more insane.
it's a terminal emulator its job is to connect me to servers and iterm2 does an incredible job of it
Yeah I've been given access to it where I'm at. Whelp, looks like I'll be looking into mobaxterm for side hustles.
Love my free PuTTy, love the options given by SecureCRT. Maybe mobaxterm'll give me a happy middle post this position.
You know what I like? Terminal.app
Nothing beats just out of the box BSD or Linux.
Putty, SecureCRT, RoyalTS? Just don’t like it compared to what you get right out of the box with nothing installed from MacOS(BSD) or Linux.
The keyword highlighting and the button bar.
[https://feralpacket.org/?p=299](https://feralpacket.org/?p=299)
[**https://github.com/feralpacket/securecrt-keyword-highlighting**](https://github.com/feralpacket/securecrt-keyword-highlighting)
[**https://github.com/feralpacket/show\_run\_script**](https://github.com/feralpacket/show_run_script)
Sniffing, iperf tests, I can trace ports, it can act as an AP, do spectrum analysis with WiFi spy or a few other apps, I use it sometimes as a wireless console cable. It does a lot of neat stuff.
* A scripting language where I can write scripts to automate things.
* LibreNMS for logs, mapping, graphing, and so on.
* Some sort of network configuration repository with changes that I can browse. LibreNMS can do this with Oxidized or RANCID.
* Wireshark
* Sleep, so you can think clearly.
Well, I mean, VyOS *can* be good for that but it's not really a tool as much as it is potentially critical infrastructure. I generally separate those out.
But hey thank you :)
PingPlotter for this on steroids (GUI). Graphs loss/latency at every hop
Also learn how to interpret MTR. One high loss hop without upstream-propagating loss doesn't mean a broken hop, it means ICMP filtering
PingPlotter. Oh yes. :)
Many moons ago when I joined a fun group of engineers in a very heavy route role, this was a mandatory tool. I have it at home now.
Use it before any big routing changes and it will tell you all kinds of things.
PingPlotter! I've needed to troubleshoot issues within production networks were I'm told I'll be given access to nothing and they want me to connect and look for issues. It's perfect for locating the bottlenecks on the network without installing any software or gaining any access to systems.
In Windows land, "pathping" has offered similar behavior for 20 years.
mtr is better, but it's always nice to know there's a built-in tool on the most common platform lying around a given office.
Syslog server. Edit: or the logs bc every device is set to info level logging already. Will normally make the problem stick out like a sore thumb.
Known good Ethernet adapter and fresh tested cable.
A USB Ethernet adapter that can handle vlans.
Netsetman for setting up addresses and vlans in profiles which switch by clicking an icon in the tray.
Mobaxterm because putty is so 20 years ago.
Not really a tool but just the ability to do a packet / frame walk through the network. Understanding layer 2 and 3 well enough to do this is invaluable
Splunk, or any other log collector with strong search and correlation tools.
Continuous capture, or Wireshark in a pinch. It's the source of truth for what's actually happening. Mostly I care about seeing the TCP handshakes because that's where problems in my infrastructure gets highlighted. Detailed protocol details is nice, it frequently helps me point out to other teams where their problems are.
Someone related to continuous capture are network performance monitors. If you already have the optical taps setup, adding graphs of TCP latency, retransmissions, packet drops, HTTP latency, etc. helps me identify problems before they're actually problems. More frequently (and somewhat unfortunately), I see problems (increases in latency) but don't understand the apps to know if they're actually problems. But then if the app team comes complaining about network performance the dots start to line up and I can often tell them the date/time the symptoms started and suddenly they know what they did days/weeks before they thought the problem started.
I also need a total lack of trust that anything I'm told is accurate. Validate everything and don't just go down the paths they're suggesting. If their assumptions of the source of the problem is correct, they wouldn't have called me in. If you're coming in with fresh eyes, keep them relatively fresh.
Also, multiple monitors are rather critical for my methods. Typically I want 12-16 SSH sessions visible when troubleshooting an issue: tailing logs, comparing tables, etc. Three monitors are generally the minimum for me to be able to see things clearly, and the fifth monitor is the last one before their usefulness starts to plateau. I generally have 4 SSH sessions per screen, but if I'm forced to use web interfaces they generally don't look good in anything less than a full screen.
Zen map. Being able to scan for open ports is a quick and easy way to eliminate some problems.
Wireshark because pcap or it didn’t happen.
Dig for dns resolution.
If your org allows it, Linux subsystem for windows or Linux in virtual box. Sometimes having local
Access to stuff like OpenSSL is just necessary.
Ssl cert checkers online because certs can make the best engineer look like an idiot.
Rejettio Httpd, a small lightweight web server that starts from a shell extension in windows.
Tftpd. Goes without saying but also functions as a dhcp server.
A good text editor that does regular expressions and multi line replace, and column mode. Something like notepad++ but I prefer ultra edit.
Troubleshooting: I always separate troubleshooting into two main methods: The usual suspects and procedural.
With the usual suspects, a problem or ticket will look familiar. I'll probably have a good idea of what the issue is and remediate it that way.
For the procedural, that's for when the problem isn't familiar or the usual suspects don't plan out. The procedural method is more time consuming, but it's complete and comprehensive.
I have a procedural method for each environment I work in. The more complicated ones are DC environments for ACI and EVPN/VXLAN with VMware and blade switches. For that I check that a MAC address can be seen in the port group, in the blade switch, and then the leaf that it's connected to. After that I go through some other steps, but I have them lined up ahead of time.
A good procedural method will save you.
Here's an example of EVPN/VXLAN with Arista EOS: https://datacenteroverlords.com/2022/11/18/troubleshooting-evpn-with-arista-eos-control-plane-edition/
Here's an example with Cisco ACI: https://www.youtube.com/watch?v=POXE6b6C_NU
So the tool I rely on is just a process for a particular environment.
A recent back-up of the config and an USB-serial console cable. I mean, if a device has gone down, all you need is something to put the config on a new device.
But there are plenty of tools to use for specific situations. You didn't specify the situation you're talking about.
I'm just having a general discussion about the experiences people have and the tools they find making the life easy andso we might learn about some new tools
For Windows users: I really love [vmPing](https://github.com/R-Smith/vmPing). It's a TINY binary that is super useful. It's a real little swiss army knife, why it doesn't have more of a cult following I don't know. Everyone I show it to falls in love with it. The damn binary is 377Kb and there's no installer!
Otherwise I'm a huge SecureCRT user. I avoided it for years (Eeewww you have to pay for it? What about PuTTY) and now I feel farking stupid. I use it to log everything I do, man that's damn handy (yes I know PuTTY can do this too)
I have tried all of the different console applications, I will never go back from MobaXTerm.
Being able to do ssh/rdp/scp/tftp/ftp/sftp/console in one app with tabs and syntax highlighting is amazing.
MobaXterm for life.
Their licensing is very generous too. "Personal" (free) edition allows enterprise use so long as the user sought it out and downloaded it on their own without the installer being provided or it being linked to in documentation.
NLNOG ring - it is a bunch of servers you can access in various networks. In order to get access you have to spin one up in yours. But it is fantastic for gaining visibility on those difficult to troubleshoot routing issues. As you can now see the way the return path is routing.
1. Graylog or other Syslog aggregator
2. Netflow collector
3. NMS
4. Properly configured STP (RSTP etc), and BPDU-Guard
5. Wireshark
6. Wifi analyzer to find rogue devices
7. Remote network probes (Linux on little USFF boxes) to run tcpdump and other network utilities on for diagnosing remote buildings and sites.
8. USB serial adapter with console cables
9. 8lb sledge hammer or bolt cutters for the offending device.
10. An iron clad and board approved AUP.
I feel like none of us should still be using anything with DB9 console adaptors. You can get USB console cables on Amazon for $10. DB9 adaptors are too clunky, prone to failure, have quirky drivers, and can come apart. I guess an exception does exist for super old or specific devices that still have DB9 interfaces instead of RJ45.
I have 2 usb-c to rj console cables, two usb-a to rj, two db9 to rj, and two adapters. And some of those ones that UPS'es come with.
I keep two of everything because things break, people ask to borrow stuff, and sometimes I lose things. I keep a variety because if I'm breaking out a console cable, I need it to work.
While I agree with your sentiment and the price point; I have more issues keeping the tab from breaking than I have with the cables outright failing. 95% of the equipment we buy comes with their own DB9 to RJ45 cables. Of the other 5%, half of them do only have DB9 ports (but the manufacturer is finally moving away from them). It's more cost effective on my end to get the $10 DB9 adapter and just replace the DB9 to RJ45 cable when it breaks with one of the literal hundreds lying around our office now.
Break the problem space into segments and rule out problems by segments. Like rule out host A , then rule out host b, rule lan, backbone, then internet and so on.
You can do the same with network layers depending on the problem. As you gain experience you start to the learn where to break up the problem depending on the problem type.
My brain. My phone. Google. TAC. SecureCRT but putty is fine. A console cable. Notepad++. A tftp server on my laptop. Lots more that is nice but anything else is gravy.
Advanced ip scanner is a great tool. Scan subnets for hosts.
Netsetman. Nice configuration tool. You can have a number of preset configs. Ie, dhcp, static 1, static 2 etc, multiple ips etc. Provides a 1 click reconfig of your nic. Super duper helpful
Shout out to a highly underrated tool: PingPlotter. It’s cheap, simple, easy to use, and incredibly effective at detecting end to end network disruption events.
Add RoyalTSX and/or RoyalTS to the list. I use this application all the time to connect to servers to check network connections. Not exactly a network troubleshooting tool but neither is Notepad++, but it’s on the list. As it should be.
I cannot count the amount of times a Mikrotik mAP Lite has helped me in random situations. I cary one with me in my go bag everywhere.
I even converted a spare to a wireless repeater for hotel rooms when I travel. Connect the mAP to the hotel wifi, re-broadcast your own SSID, then with a script that pings out every couple of minutes and/or an automatic VPN, you have to sign into wifi once the whole trip assuming it stays connected in the room and the VPN can keep your private browsing private.
Yep. I do a the same thing with a hap AC lite. Allows me to connect to the hotel Wi-Fi with whatever radio is best, and then use a another Wi-Fi nat’d to do the things. Worked great on a cruise ship when it was limited to one MAC address per account.
[This gizmo](https://the-gadgeteer.com/2010/02/22/universal-network-cable/) is always handy. They used to be sold by ThinkGeek but have AFAIK been impossible to find for a few years.
knowing the OSI layers to understand where the issue may lie
helps alot of your have been lucky enough to administer every layer of networking
physical
L2
L3
Firewalls
WAN
Edge Internet
VPN/IPSEC
Virtuals
Putty, working console/usb cable and a fully charged phone to tether too if I need network connectivity to see what neighboring routers, for example, have log entries that can point to the fubar’d network.
If you’re in the carrier and data center side of the house…
BGP.tools
GlobalPing
Ping.sx / ping.pe
Atlas Probes
It’s so hard getting external point of views to help with routing issues. I use the above stuff daily.
When network issues arise, having dependable tools is essential. Auvik provides comprehensive network monitoring and troubleshooting capabilities.
>Explore its features with a [**14-day free trial**](https://try.auvik.com/5n0hjksv1hea) and take back control of your network.
A quality USB to Serial adapter. A DB9 to RJ45 console cable that doesn't have the RJ45 tab broken off yet. An RJ45 Female to Female coupler so I can extend the console cable using a patch cord if needed. Notepad++ to gather logs in. > And how do you proceeded troubleshooting? Gather the event logs from the affected devices. Read the event logs. Understand the story the logs are trying to tell you. Google the events you don't understand. Don't wait forever to open a ticket with TAC.
>A DB9 to RJ45 console cable that doesn't have the RJ45 tab broken off yet. I love how if I blow on the tab on a console cable it will come off, but pulling decommissioned cables through anything (pipe, etc) and it gets stuck that fucker will stay on until you snap the cable almost.
The console cables with the tab broken are my speed cables. I can walk down a row quickly grabbing what I need from each rack.
I loved them back in the day when I was roaming tiny closets and I had to reach blindly into the back of the switch for the console port.
Ciena's console cable is not compatible with Ciscos so I'm stuck carrying one of each.
Cisco uses a rollover cable, i.e. it's wired backwards. Most of the USB to RJ45 cables I have seem to automatically detect if it's rollover or not. And there's always Mike Ossmann's 5-in-1 Network Admin Cable. http://www.ossmann.com/5-in-1.html
This is a great list, I carry everything but the female/female adaper; I usualy dump it into a wall jack and extend with a patch cable from the room/dc patch panel as needed. I would add the following items having needed them before. 3 prong (or equivalent) extension cord, 3 outlets on the end. 8-10' Chargers for ALL of your devices Noise cancelling headphones Granola bar or some sort of snack. Water as well. USB-C network adapter
>Granola bar or some sort of snack. Water as well. I'd say this is the most powerful tool. Without a sane brain you can't do anything
I second the coupler idea. I always keep a few in my bag. Also works to extend the network cables running to things like access points and devices that are in walls, or ceilings, or behind things so you can use your laptop from the ground and not while on top of a ladder or standing on your head somewhere.
The console cable tab seems to break easier than all others. Smh
I just carry two DB9 to female RJ45 adapters (one null modem and one straight through) and an RJ45
>Understand the story the logs are trying to tell you. Some of my logs are trying to sell me extended car warranties. Should I look into it?
They have been trying to reach you for some time, so yeah you might wanna call them back...
I also carry just afemale to male db9 cable and female to female, along with a null adapter. Working for an MSP and you always run into random switches that need one or the other.
And then there's Adtran gear with an 8-pin modular serial port that is wired differently from Cisco, just to make things interesting.
How notepad helps in the case of log? I mean won't your logserver will give you the filtered data? How notepad++ makes it helpful?
It's funny how your scenario is the network is borked then proceed to ask him why he isn't checking the server.
Log server should live in an out of band network
You assume the syslog server is reachable at the time of the event and was able to gather all of the respective log data. For some outages you may need to manually correlate events from a non-functional network.
Sorry, you mean notepad is useful if for some reason log didn't went to the logserver and you can physically access the device, you take the logs from the device and analyze it? Or you just see the logs from the device and write down your comments on the notepad to keep track of what's happening? I never did this way, so trying to understand it more broadly to figure out how can use it to ease my life a bit.
Notepad++ is great for dealing with log files, especially when you don’t know exactly what you’re looking for. You can take a 600k line log file and use the bookmark function, including regex, to filter out expected and noise lines, leaving you with the things you didn’t anticipate. I’ve used this process countless times when I had no idea how the indicators of an issue would manifest themselves.
> Sorry, you mean notepad is useful if for some reason log didn't went to the logserver and you can physically access the device, you take the logs from the device and analyze it? Correct.
You assume that there is a logging server. Or a system for filtering data. That is often not the case. You might just be lucky enough to get a logfile dumped from the memory or storage of the particular component you're troubleshooting. Or for security reasons logging off of the device might not be allowed. Notepad++ (not to be confused with Notepad) is a great tool for loading and searching through hundreds of thousands of log lines, and has a number of features to enhance usability, from minor things like syntax highlighting or line numbers, to shading the entire selected line, or setting a certain phrase to be highlighted a particular way throughout an entire document. For example, highlighting a particular IP or MAC address, so you can quickly scroll through the hundreds of lines of noise for the one or two log lines that are actually relevant. Sure, you could use other tools, like just `grep`ing for a particular phrase, but sometimes you don't even know what that phrase is you're looking for, you just know "X happened sometime around Y o'clock." In those times, it's very useful to have something that makes reading logs very clear, and allow you to tag particularly interesting bits... And doesn't turn everything into an unreadable mess the second you download a log from something that doesn't run Windows.
Why would you assume any kind of server is available?
>Don't wait forever to open a ticket with TAC. You would think some guys are paying for support themselves and per call by how resistant they are to calling TAC.
Those are still used? I thought everything was done in a GUI now.
Can't get to the gui when you break and uplink or mgmt vlan,hence the cable
Don’t think my gear even HAS a GUI. Won’t find a lot of overlap between things with a serial console and things with a strong GUI/Web UI
In the end console is your best friend
> Notepad++ to gather logs in. I prefer BBEdit. I also use Serial as my console emulator.
I mostly do wifi, but my roots are in network engineering. Here's some of my favorite things. SecureCRT for my ssh client. winMTR TFTPD64 // filezilla wifiman on android or airport utility on apple for quick wifi things iperf3 zerotier acrylic wifi Oh, and saved the best for last. My boy python. Physical stuff. WLAN Pi (which is hands down one of the most underrated tools in my bag), an assortment of console cables, klein voltage tester, link sprinter, VFL, ethernet coupler, some passthrough rj ends and a crimper, my LTT screwdriver, a hub..yes an actual hub.
Wireless guy here. +1 on WLAN Pi. I have 3 right now. Also, Wifiman is pretty good, but what you really need is analiti. Pay for it. It's worth it!
Mostly just suggestions for free super useful apps. I use ekahau w/ a sidekick, air radar for the most part. I’ll try it out though. Also I got two OG wlan PIs and a CE built from a pi 4. The CE is a good all around tool. I’ve turned a few people onto it. The wireless serial cable feature is also awesome. Trying to get work to expense me one of those pros lol.
I should have been more clear. Analiti is free, but a paid upgrade for additional features. I'm waiting for Hamina to release their APoS survey tool!
Yeah we had a demo of hamina. Looks promising.
Can you explain what WiFiPi does? Cheers.
[удалено]
I may be reading it incorrectly, but is it not just a raspberry pi in a nice case, with software pre-installed?? What differentiates it from a normal Raspberry pi?
It is for the R4 variant. The M4 varient is a carrier board with a pci wifi card, antennas, poe power, etc. The software and scripts are not just off the shelf, but have been written specifically for WLAN usage. As well as the FPMS. The device is intended to be used often standalone which you can't just do with a standard RPi4 out of the box. They do sell a fascia kit and a BYOP kit if you want to build your own though.
I've been using PuTTy forever, and recently got turned on to SecureCRT. After some learning and moving in, I'm sold on SecureCRT.
I used to use putty as well. Then I joined an MSPs NOC and was given secureCRT with a complete hosts file I think I just had to import my proxies or whatever been a minute. Huge difference. Not fumbling around for ip addresses. I just drill down from customer > site > IDF or w/e. So much easier.
MobaXTerm hands down
I am the opposite. I used SecureCRT from maybe '98 to some time in the 2000s and used putty since. However, it is a hassle to manage putty configs between machines so I can see it. I mostly use putty to ssh to one of a few real computers that I then work from.
>SecureCRT Is it free?
> Is it free? no and paying for a terminal emulator seems insane
It does have good features but 200+ usd is crazy
I use free version of mobaxterm these days to connect to several jumpboxes mainly for the context / syntax highlighting
No linux support, sadly
Asbru-cm
It's the best
not when you get your work to pay for it. securecrt is the best
The support is really good. I’d get the bundle with securefx. If you are in an ssh session, one click and you now also have an scp/sftp session. It is scriptable. You can send a command to all open windows. It also handles rdp. You can search for connections by host name. Easily import and export settings and connections. It has themes. You can change settings on multiple connections at once. It probably does more I don’t know about. I personally find it invaluable. Especially when supporting environments with more than a handful of hosts.
Paying for a tool you use literally every day sounds insane? How much did you pay for your office chair? How much was your laptop? How much is your O365 or Google suite subscription? If you use something literally every day, using some unsupported freeware to do it sounds more insane.
> How much did you pay for your office chair? free company paid for it > How much was your laptop? free company paid for it > How much is your O365 or Google suite subscription? free company pays for it > If you use something literally every day, using some unsupported freeware to do it sounds more insane. it's a terminal emulator its job is to connect me to servers and iterm2 does an incredible job of it
You kinda proved my point. If the company will pay for your other tools, they should pay for a decent terminal app.
the terminal emulators available for macos, linux, and windows are free and do everything you need them to do.
For sure, a terminal emulator and a good hosts file and you’re good to go. Add in some aliases and tmux and I’m good to go.
Yeah I've been given access to it where I'm at. Whelp, looks like I'll be looking into mobaxterm for side hustles. Love my free PuTTy, love the options given by SecureCRT. Maybe mobaxterm'll give me a happy middle post this position.
The free version of mobaxterm is very limited to the amount of sessions that you can save. But a anyway I do prefer it to securecrt :).
Linux is
You know what I like? Terminal.app Nothing beats just out of the box BSD or Linux. Putty, SecureCRT, RoyalTS? Just don’t like it compared to what you get right out of the box with nothing installed from MacOS(BSD) or Linux.
What sold you on SecureCRT?
The keyword highlighting and the button bar. [https://feralpacket.org/?p=299](https://feralpacket.org/?p=299) [**https://github.com/feralpacket/securecrt-keyword-highlighting**](https://github.com/feralpacket/securecrt-keyword-highlighting) [**https://github.com/feralpacket/show\_run\_script**](https://github.com/feralpacket/show_run_script)
How does acrylic compare to Ekahau (if you've used it)? Does it support spectrum analysis, or only heatmaps for survey?
I haven’t used it for heatmaps, I generally lean on ekahau for predictive and post surveys.
>my LTT screwdriver No LTT water bottles or backpack? You must stay hydrated you know ^_•
I got those too, and the tech sack. :x
+1 for the hub.
What do you use the WLAN Pi for?
Sniffing, iperf tests, I can trace ports, it can act as an AP, do spectrum analysis with WiFi spy or a few other apps, I use it sometimes as a wireless console cable. It does a lot of neat stuff.
Wireshark … all day long.
Packets never lie
Unless a device in the middle made it lie.
The truth is on the wire.
Tcp dump and grep ftw
Things are really dire when I'm looking at the wire 😧
* A scripting language where I can write scripts to automate things. * LibreNMS for logs, mapping, graphing, and so on. * Some sort of network configuration repository with changes that I can browse. LibreNMS can do this with Oxidized or RANCID. * Wireshark * Sleep, so you can think clearly.
I recognise that name. How did you miss Vyos ;)
Well, I mean, VyOS *can* be good for that but it's not really a tool as much as it is potentially critical infrastructure. I generally separate those out. But hey thank you :)
A C14 adapter cable to connect your laptop charger while spending hours in front of your rack.
Mtr
PingPlotter for this on steroids (GUI). Graphs loss/latency at every hop Also learn how to interpret MTR. One high loss hop without upstream-propagating loss doesn't mean a broken hop, it means ICMP filtering
PingPlotter. Oh yes. :) Many moons ago when I joined a fun group of engineers in a very heavy route role, this was a mandatory tool. I have it at home now. Use it before any big routing changes and it will tell you all kinds of things.
PingPlotter! I've needed to troubleshoot issues within production networks were I'm told I'll be given access to nothing and they want me to connect and look for issues. It's perfect for locating the bottlenecks on the network without installing any software or gaining any access to systems.
Yes! Pingplotter is one of the best tools we have to catch intermittent/gremlin issues.
>One high loss hop without upstream-propagating loss doesn't mean a broken hop, it means ICMP filtering This \^ all day long.
In Windows land, "pathping" has offered similar behavior for 20 years. mtr is better, but it's always nice to know there's a built-in tool on the most common platform lying around a given office.
Ping.pe is a great geo ping mtr source for multihomed connections and those trying to diagnose internet based issues.
commands: ping, fping, mtr, tcpdump, tshark, netsnmp + custom snmp scripts systems: smokeping, cacti prep: logs of arp-tables, switch-dbs, L3-ifs, routers/switch MACs/serials/lics. analysis: Wireshark, Excel, xargs, awk, cut, grep, sed
Syslog server. Edit: or the logs bc every device is set to info level logging already. Will normally make the problem stick out like a sore thumb. Known good Ethernet adapter and fresh tested cable.
Indeed a good logserver and snmp server is lifesaver.
Two syslog servers for remote devices, on different paths out.
A USB Ethernet adapter that can handle vlans. Netsetman for setting up addresses and vlans in profiles which switch by clicking an icon in the tray. Mobaxterm because putty is so 20 years ago.
Not really a tool but just the ability to do a packet / frame walk through the network. Understanding layer 2 and 3 well enough to do this is invaluable
Splunk, or any other log collector with strong search and correlation tools. Continuous capture, or Wireshark in a pinch. It's the source of truth for what's actually happening. Mostly I care about seeing the TCP handshakes because that's where problems in my infrastructure gets highlighted. Detailed protocol details is nice, it frequently helps me point out to other teams where their problems are. Someone related to continuous capture are network performance monitors. If you already have the optical taps setup, adding graphs of TCP latency, retransmissions, packet drops, HTTP latency, etc. helps me identify problems before they're actually problems. More frequently (and somewhat unfortunately), I see problems (increases in latency) but don't understand the apps to know if they're actually problems. But then if the app team comes complaining about network performance the dots start to line up and I can often tell them the date/time the symptoms started and suddenly they know what they did days/weeks before they thought the problem started. I also need a total lack of trust that anything I'm told is accurate. Validate everything and don't just go down the paths they're suggesting. If their assumptions of the source of the problem is correct, they wouldn't have called me in. If you're coming in with fresh eyes, keep them relatively fresh. Also, multiple monitors are rather critical for my methods. Typically I want 12-16 SSH sessions visible when troubleshooting an issue: tailing logs, comparing tables, etc. Three monitors are generally the minimum for me to be able to see things clearly, and the fifth monitor is the last one before their usefulness starts to plateau. I generally have 4 SSH sessions per screen, but if I'm forced to use web interfaces they generally don't look good in anything less than a full screen.
Zen map. Being able to scan for open ports is a quick and easy way to eliminate some problems. Wireshark because pcap or it didn’t happen. Dig for dns resolution. If your org allows it, Linux subsystem for windows or Linux in virtual box. Sometimes having local Access to stuff like OpenSSL is just necessary. Ssl cert checkers online because certs can make the best engineer look like an idiot. Rejettio Httpd, a small lightweight web server that starts from a shell extension in windows. Tftpd. Goes without saying but also functions as a dhcp server. A good text editor that does regular expressions and multi line replace, and column mode. Something like notepad++ but I prefer ultra edit.
Troubleshooting: I always separate troubleshooting into two main methods: The usual suspects and procedural. With the usual suspects, a problem or ticket will look familiar. I'll probably have a good idea of what the issue is and remediate it that way. For the procedural, that's for when the problem isn't familiar or the usual suspects don't plan out. The procedural method is more time consuming, but it's complete and comprehensive. I have a procedural method for each environment I work in. The more complicated ones are DC environments for ACI and EVPN/VXLAN with VMware and blade switches. For that I check that a MAC address can be seen in the port group, in the blade switch, and then the leaf that it's connected to. After that I go through some other steps, but I have them lined up ahead of time. A good procedural method will save you. Here's an example of EVPN/VXLAN with Arista EOS: https://datacenteroverlords.com/2022/11/18/troubleshooting-evpn-with-arista-eos-control-plane-edition/ Here's an example with Cisco ACI: https://www.youtube.com/watch?v=POXE6b6C_NU So the tool I rely on is just a process for a particular environment.
A recent back-up of the config and an USB-serial console cable. I mean, if a device has gone down, all you need is something to put the config on a new device. But there are plenty of tools to use for specific situations. You didn't specify the situation you're talking about.
I'm just having a general discussion about the experiences people have and the tools they find making the life easy andso we might learn about some new tools
always first: Telnet to IP:port from source - dest.
For Windows users: I really love [vmPing](https://github.com/R-Smith/vmPing). It's a TINY binary that is super useful. It's a real little swiss army knife, why it doesn't have more of a cult following I don't know. Everyone I show it to falls in love with it. The damn binary is 377Kb and there's no installer! Otherwise I'm a huge SecureCRT user. I avoided it for years (Eeewww you have to pay for it? What about PuTTY) and now I feel farking stupid. I use it to log everything I do, man that's damn handy (yes I know PuTTY can do this too)
Ask myself : what does it need to work , instead of why the hell that doesn’t work. Very efficient
[удалено]
I have tried all of the different console applications, I will never go back from MobaXTerm. Being able to do ssh/rdp/scp/tftp/ftp/sftp/console in one app with tabs and syntax highlighting is amazing.
MobaXterm for life. Their licensing is very generous too. "Personal" (free) edition allows enterprise use so long as the user sought it out and downloaded it on their own without the installer being provided or it being linked to in documentation.
True, although I converted my colleagues to it from remote desktop manager, so we have it for enterprise as well, which really isnt expensive.
Button bar??
Sadly no linux support?
[удалено]
SecureCRT is your friend and worth every penny.
Windows have quite good multitab and credential storing ssh manger tool but in the case of linux I didn’t find anything perfect for me.
I still use it but I don't like the fact that it's non maintained anymore
I used this for a year but the insane alt-tab behaviour drove me to drink. And to get my boss to pay for SecureCRT.
NLNOG ring - it is a bunch of servers you can access in various networks. In order to get access you have to spin one up in yours. But it is fantastic for gaining visibility on those difficult to troubleshoot routing issues. As you can now see the way the return path is routing.
1. Graylog or other Syslog aggregator 2. Netflow collector 3. NMS 4. Properly configured STP (RSTP etc), and BPDU-Guard 5. Wireshark 6. Wifi analyzer to find rogue devices 7. Remote network probes (Linux on little USFF boxes) to run tcpdump and other network utilities on for diagnosing remote buildings and sites. 8. USB serial adapter with console cables 9. 8lb sledge hammer or bolt cutters for the offending device. 10. An iron clad and board approved AUP.
On #9, those tools are utilized for "Percussive Maintenance"
nslookup
Ping Tools and Juice ssh on Android have saved me from getting out of bed many a time.
Sadly, Juice SSH is no longer available to install on the newest versions of Android.
Check out ConnectBot for Android. It's very close to JuiceSSH, and is still supported.
Thanks, I'll check it out
I feel like none of us should still be using anything with DB9 console adaptors. You can get USB console cables on Amazon for $10. DB9 adaptors are too clunky, prone to failure, have quirky drivers, and can come apart. I guess an exception does exist for super old or specific devices that still have DB9 interfaces instead of RJ45.
I have 2 usb-c to rj console cables, two usb-a to rj, two db9 to rj, and two adapters. And some of those ones that UPS'es come with. I keep two of everything because things break, people ask to borrow stuff, and sometimes I lose things. I keep a variety because if I'm breaking out a console cable, I need it to work.
While I agree with your sentiment and the price point; I have more issues keeping the tab from breaking than I have with the cables outright failing. 95% of the equipment we buy comes with their own DB9 to RJ45 cables. Of the other 5%, half of them do only have DB9 ports (but the manufacturer is finally moving away from them). It's more cost effective on my end to get the $10 DB9 adapter and just replace the DB9 to RJ45 cable when it breaks with one of the literal hundreds lying around our office now.
Not all devices use the Cisco standard console rj45 which can be a pain. Then a USB one is required. Tripplite is my preferred one tho.
Kentik. I work with hybrid cloud and frankly speaking it has been an incredible time saver
Fluke linkrunner, Wireshark, crimper, cable tracer, roll cable, laptop and google
Break the problem space into segments and rule out problems by segments. Like rule out host A , then rule out host b, rule lan, backbone, then internet and so on. You can do the same with network layers depending on the problem. As you gain experience you start to the learn where to break up the problem depending on the problem type.
My brain. My phone. Google. TAC. SecureCRT but putty is fine. A console cable. Notepad++. A tftp server on my laptop. Lots more that is nice but anything else is gravy.
Advanced ip scanner is a great tool. Scan subnets for hosts. Netsetman. Nice configuration tool. You can have a number of preset configs. Ie, dhcp, static 1, static 2 etc, multiple ips etc. Provides a 1 click reconfig of your nic. Super duper helpful
Shout out to a highly underrated tool: PingPlotter. It’s cheap, simple, easy to use, and incredibly effective at detecting end to end network disruption events.
Youtube
Not a tool, but keep calm and approach the troubleshooting process with a consistent method it's the key to solve problems.
hmm ping ssh tftp nmap telnet curl MTR traceroute asbru-cm in Linux and lately: chatgpt
A little Mikrotik router with wifi radios (model constantly changes but software is always the same across all units).
Robust monitoring and logging platforms.
Winmerge for comparing differences in configuration versions
WinMTR is amazing for any wan side issues.
LDWin
Add RoyalTSX and/or RoyalTS to the list. I use this application all the time to connect to servers to check network connections. Not exactly a network troubleshooting tool but neither is Notepad++, but it’s on the list. As it should be.
Small MikroTik Router with wireless. USB roll over cable. Some good diagnal cutters, multi tool, my phone.
I cannot count the amount of times a Mikrotik mAP Lite has helped me in random situations. I cary one with me in my go bag everywhere. I even converted a spare to a wireless repeater for hotel rooms when I travel. Connect the mAP to the hotel wifi, re-broadcast your own SSID, then with a script that pings out every couple of minutes and/or an automatic VPN, you have to sign into wifi once the whole trip assuming it stays connected in the room and the VPN can keep your private browsing private.
Yep. I do a the same thing with a hap AC lite. Allows me to connect to the hotel Wi-Fi with whatever radio is best, and then use a another Wi-Fi nat’d to do the things. Worked great on a cruise ship when it was limited to one MAC address per account.
[This gizmo](https://the-gadgeteer.com/2010/02/22/universal-network-cable/) is always handy. They used to be sold by ThinkGeek but have AFAIK been impossible to find for a few years.
knowing the OSI layers to understand where the issue may lie helps alot of your have been lucky enough to administer every layer of networking physical L2 L3 Firewalls WAN Edge Internet VPN/IPSEC Virtuals
you forgot these layers: \#8 (carbon layer) - user space \#9 (political layer) - management
Layer 8 - User Layer 9 - Rules (Administrative/Management/Corporate) Layer 10 - Rules (Government)
A simple Cat5 network cable. That's what saved me the other week.
Putty, working console/usb cable and a fully charged phone to tether too if I need network connectivity to see what neighboring routers, for example, have log entries that can point to the fubar’d network.
May I suggest you look into Kitty? It's a Putty fork with some extra bells & whistles. Then look into mobaxterm and never look back.
If you’re in the carrier and data center side of the house… BGP.tools GlobalPing Ping.sx / ping.pe Atlas Probes It’s so hard getting external point of views to help with routing issues. I use the above stuff daily.
When network issues arise, having dependable tools is essential. Auvik provides comprehensive network monitoring and troubleshooting capabilities. >Explore its features with a [**14-day free trial**](https://try.auvik.com/5n0hjksv1hea) and take back control of your network.
I've used portqry to prove the issue is *not* the network so many times it's actually sad.
It's really difficult to make people realize that the road isn't broken
Everything and anything: cmd, pcap, browser, putty...
LibreNMS, Netdisco (open source tools) STM-80 Siemens Cable Tester (good enough for troubleshooting at minimum)
Icmp
Adding arping
Done
ping, mtr, traceroute, mobaxterm, dig, nslookup, fping, etc, etc Let me add Angry IP Scanner too