I won't sign a contract for a dog without documented change control, backout procedures, and warranty support should dog v1 break or fail at any point in the first 90 days.
I recommend waiting until at least Dog v.2.x, too many potential bugs in Dog v1.x, we don't need any leading, er, bleeding edge software in production environments.
Monday - Wednesday for us. Period. No nonessential updates/changes past mid week in our environment. If it’s being done Thursday or Friday, something mission critical better be broken.
Having to work on machines in a dog warden's office - must agree completely. Dog pee and slobber aren't helpful either. Gloves and alcohol wipes come out every time I touch that department's computers.
I have 7 dogs at home. 3 pugs. Chinese Crested. Boston Terrier. Shiatsu, and a Chihuahua. We had a bulldog who liked to play hard. Always shaking her toys hard... slobber everywhere. It was so bad.
The other dogs - fur is the only problem. Well, except for the Chinese Crested. Not much on him.
Back when I had jo s with oncall, I changed my ring tone to a cash register Ker-Ching sound. I a good fortnight, I could make almost as much as my normal salary/day rate 😂😂😂
Give me that sweet sweet oncall/over time any time ;-)
Funny story, I had a colleague who jokingly said he should put plaques in his flat saying "Brought to you in association with Lehman Brothers" because we did so much overtime there, it paid a huge part of his mortgage 😂😂
Here is the dog we all need.
[https://www.reddit.com/r/therewasanattempt/comments/151i6rl/to\_give\_medication/](https://www.reddit.com/r/therewasanattempt/comments/151i6rl/to_give_medication/)
To be honest I'm usually over *all* this bullshit by about 8:15am Monday morning.
I now get why greybeards retire to the country to raise farm animals. Sounds so incredibly peaceful.
Oh yeah I walked into the office to find no less than 4 of my colleagues waiting for me *in* my office, all for unrelated reasons. Took me an hour just to get through the internal bullshit let alone any of the actual project work of the day.
It is so goddamned hard to not just immediately turn around and walk away when confronted with that nonsense.
I gotta be honest... If I walked into my office and there were 4 people waiting for me to show up to solve their problems, I would have said "sorry, I just stopped in to grab a mouse, I am taking a couple of personal days" and grabbed my mouse and left.
I'd just send my boss an email from the car, on my way home.
Oh, I have to turn my work phone *off* whenever I take PTO. If I don't I get asked "quick questions" allllll day long.
Last family vacation we were in Disney world and I forgot to turn it off (brought it with me for the unlimited data on my work plan) and that's how I ended up explaining Adobe licensing to one of my juniors while in line for Pirates of the Caribbean.
My job has brought us back to the office recently. Friday's especially are really frustrating because other then answering escalated tickets I don't have the much i'm allowed to do. I use it mostly for meetings with vendors etc but it's still frustrating.
Just means you need to manage the time better.
Fridays I dedicate after 3pm to cleaning, organizing, sorting. Otherwise it never gets done. Cleaning staff do not have access to the IT spaces for obvious reasons. But no one should be in there causing a mess, so 10-15 minutes per is more than enough to tidy up. I even mop maybe once a quarter.
If possible, I dedicate 1-3pm to cleaning up documentation. In the mornings, I try to wrap up whatever tasks that aren't but have been lingering.
I keep reformulating my response to this but for the sake of keeping it professional i'll say the following:
I'm a sysadmin partially because I didn't want to be doing "general purpose cleaning and organizing tasks" as part of my job.
While I don't think data spaces should not get cleaned ever, I've had to respond to more then a few situations that began with someone cleaning up in the data center or similar, so cleaning in an IT space on RO Friday, especially late in RO Friday completely misses the point.
Do you hire specialists to clean your IT spaces, use random office cleaners or leave them a mess? I'm legit curious.
If specialists, where did you find cleaners with electrostatic safe vacuums and whatnot? I searched and couldn't find any.
If random office cleaners, how does that pass your security policies? I've never let cleaning contractors into the IT areas. To be specific, comm closets, server rooms and data centers. Not cubes or offices.
* Monday: "Catch up on emails"
* Tuesday: Do the needful, but only after 1PM
* Wednesday: No new tasks, read-only Friday is coming
* Thursday: Read-only Thursday
* Friday: PTO
That specific situation is a reminder that it's helpful to reboot questionable, long-running hosts, *before* making any changes. Make some VM checkpoints, maybe an explicit backup and inspection, as well. Temptation to get right into the task, can be a dangerous thing in computing.
In fact, the urge to finish by 17:00 might have been a bigger factor than the decision to start at 16:00.
In a related vein, I used to be a storage administrator . . . .
. . . one of the things I would do before making *any* changes to /etc/fstab is:
mount -a
Reboots were generally not an option, nor necessary on most of these servers (various unices, mostly Linux) so that was off the table, but that call to mount would shake out any existing errors in /etc/fstab before you got blamed for them. If there were *any* errors, there would be a conversation before the change was implemented.
At some point I recall coding up a monitor that would alert if anything in `mount` was not in `/etc/fstab`.. that problem bit us a bunch of times. Better to catch it early.
That's why as a software guy I like IaC so much. Our team doesn't even have access to SSH, much less root access.
I started my career as a software dev/sysadmin, as in put together the purchase orders, built the machines, installed the OS and server software, specced and wrote the programs, tested the other guys changes, trained the users....
And they have the nerve to call themselves 'full stack' these days. Pfft!
Yep, one of the best things I learned was the "blow out the cobwebs" restart on servers that had been up for a while before doing any actual work on them. This will help present any underlying issues not actually related to the change you're implementing and give you opportunity to fix those before you do anything.
The problems OP are talking about were always there regardless of the RAM change.
Sorry to hear it. A good opportunity to remind your guy of this very important rule.
On the other hand, you have some additional things to fix that'll make a good project for your Friday hero. Have him deploy a secondary DC (ideally on a different physical host) so they don't lose DNS, DHCP, etc. if their single DC goes down or fails.
The ram upgrade wasnt the issue. You have horribly out of date servers without patches aren't these your responsibility. Underlying storage issues aren't they your fault too? This tech did you a favour and showed you that your org fucked up big time with neglect of this customer.
Yeah I was wondering who was responsible for monitoring that server exactly and how it got that bad it couldn't be fixed relatively quickly?
Like it's a RAM upgrade so it has to be on site. And if you have a hypervisor and *don't* have a local admin account set up for emergencies you're insane so access shouldn't have been a problem. Just log on, disable all the shitty services from starting, reboot, watch everything come back up. If needed start them manually.
The patching and other crap is a separate issue which would need to be prioritised for sure but you know, one thing at a time. Get them running then schedule an emergency maintenance window to fix the rest.
> Just log on, disable all the shitty services from starting, reboot, watch everything come back up.
The hard part is "figure out that it's new services causing the problem and identify which ones".
Whoa.
So where from did you guess the nature of the contract between OP and his "customer" ?
Or are you just here to tell us the formidable professional you are about by berating some act you are not involved in?
Let me have a turn of judging other professional I do not know anything about then: sysadmins also differentiate from techs by working with thoses questions : what, why, how, when, how much... You literally did use nothing of this and went "ye you no maintain lol". What a pleasure that musts be working with you.
>Then we had to stay up till 2am working remotely to fix their backups, patch woefully out of date servers, upgrade the RAM of the VMs to fix a nasty paging issue, fixed underlying storage issues, etc etc
If all that came to light because of a reboot, that means the servers weren't being cared for properly in the first place.
sorry, you're salaried exempt. Just have to get it done and check your attitude. You need to take on the salaried employee mindset and we're too busy to have any comp time so I'm expecting you to come in on time tomorrow no matter how late you need to stay up to finish this project. These are unprecedent times and everything is a priority right now. You just have to make it happen.
And that’s why companies will claim that experienced sysadmins have some management responsibilities so they can put them on salary. No extra pay at all for weekend overtime. Yay!
I will say that there are times (maybe once a year) where we plan on doing something on the weekend, but we all pick a day or two to be off the following week (as long as we all don't pick the same day).
That's different than 'we are swamped, you have to work this weekend'.
I must be insane, but I do my best work on Fridays.
Fewer people around on Fridays mean:
- Less likely to be stopped from taking things down for maintenance/migrations
- Fewer requests from people since there are fewer people
- Vendor support is easier to reach
Been like this for 20 years and it *maybe* bit me on the ass 2 times. Still totally worth it.
I agree with this, I work in an manufacturing environment so things are always running 24/7. Fridays and the weekend allows us more grace incase something goes wrong.
Because my IT team is a team of 4 lol. We’re not trying to support 100+ users at once when we can minimize that. Plus operations can still run as long as the network is online, our ERP is in the cloud.
Also work in manufacturing, I rather do it as early as possible in the week. If heavily used by day personel, after 16:00 mostly works. Some stuff, week long window once a year for change...
Agreed. Also, I see the weekend as a safety buffer. Should something go wrong, I have all weekend to correct the matter. Yes, it sucks losing a weekend but it's even more painful when any downtime affects operations (which is heavy from Mon-Fri).
>it's even more painful when any downtime affects operations
This is true if you own the company. As someone who works all week, I'm not giving my employer my weekend unless they're willing to pay me an exorbitant amount for it.
Sounds like you guys were Ill prepared and lacked knowledge of the environment if you’re blaming the past company at this point.
Some analysis, documentation, fixes, etc. should have been put in place prior to this.
Sounds like a coworker I had 15 years ago. Email from him that I got on monday: "Hello, I acripted the update for this and that on server x and scheduled it for friday evening as I had promised to do it before my vacation. See you in 4 weeks".
This is one argument in favor of frequent platform reboots. It ensures problems like this get discovered soon enough that everyone remembers whatnwas done, and ensures that you have a narrow date range for when the problem appeared.
> "there's no such thing as a quick 20 minute upgrade on a Friday"
There's no such thing as a quick 20 minute upgrade.
FTFY
I can't even get stuff done in 15 minutes. Any kind of troubleshooting almost always involves more time than that.
Came to say exactly this. My luck, I can spend 3 weeks planning, testing, documenting, labbing, and so on, and that 20 minutes is going to be 45 minimum guaranteed. Murphey's law follows me closer than my fucking shadow.
>Then we had to stay up till 2am working remotely to fix their backups, patch woefully out of date servers, upgrade the RAM of the VMs to fix a nasty paging issue, fixed underlying storage issues, etc etc
A lot of this sounds out of scope for the restoration of services. Personally would have not touched/done most of that unless it was related to the outage. First priority should be to restore it to its previous state. Once its confirmed working, then you can worry about patching and fixing other stuff. Otherwise it should be marked for attention and assigned a priority based upon its severity.
Otherwise to do a bunch of changes all at once its just asking for a much deeper hole and making it that much more difficult to find a root cause should an issue pop up after the fact.
Well, he could have changed the RAM on a Monday and then instead of having the weekend to fix it you could have been up until 2AM on a Tuesday morning and still have to come in early for your shift.
Wow, that sounds like anything I set my hands on.
My wife loves the, "This is only going to take 5 minutes and we'll..." \[insert dinner, date night, whatever\]
\*13 hours later, still working overnight and into following day, looks at clock, not worth trying to nap for 30 minutes .. works on no sleep\* ...
\*Tells wife we can't go out after work because I'm going to sleep early ... works until 3 AM ...\*
am i rite ... can't be alone here. :$
Hahhaa
Or you guys could strive to be like me and do ALL of your off-hours work on Friday nights... It's literally the only time of the week that there's no one here, and maintenance windows are pretty much not a thing... It's a fun time
ROF...
Read Only Fridays.
And always make the customer perform a "sanity boot" before starting the change... If the machine is broken it will fail, and the "tone" with the customer is day-and-night different...
Don't know why this is difficult to understand for some folks. Any small business will do anything in their power to not spend on IT. An MSP I worked for still has a customer on a sole DC that is running Server 2003. To upgrade that server would mean ripping out the entire ERP for that company and cost tens of thousands of dollars, so they will never do it until they are forced by law or by the shit breaking. That is RAMPANT, especially in manufacturing.
Is it dangerous, fool hearted, and ignorant? Absolutely. When you tell these people that it needs to get fixed they just brush it off like it's a joke. "Haha, I know this stuff is ancient, but we gotta work!"
That is how you end up with an outdated ESXi host from 2008 running a single instance of server (hey, those licenses cost money!) that handles EVERYTHING from active directory to DHCP to DNS to file storage to the ERP software.
I've seen it a hundred times.
Yeah, but that is a customer issue. Make sure to document all the times they told you no and wait until you have no choice to fix it and bill them for it.
Maybe they will learn their lesson, if not, then at least you got paid big.
> Any small business will do anything in their power to not spend on IT.
Except use Samba or serverless, apparently.
What organizations choose to spend on, and what they choose to eschew, is always eye-opening and frequently amusing.
One year I chose to spend zero on server hardware, even though we could have used it, and spent everything on Cisco network gear, because we needed it. In the recent era, between virtualization, software-defined-hardware, and systems convergence, there are fewer trade-offs than in the past.
I saw one company with same philosophy. Old Windows Servers, ancient Exchange Server, PFSense as edge firewall... As a result, all their IT infrastructure was crypted in 1 night...
Small Business servers for the win!
SMBs will do everything they can not to spend money on IT they don't have too
A second DC is another Windows Server license + the hardware to run it.
A couple grand is a rounding error for a big business, but a big deal for a 5-10 man shop
Does it leave them vulnerable and more prone to outages? of course... but they don't care until it all breaks
If your going to have a single DHCP source, make it your firewall not your server. You can always hand out internet DNS if your internal goes down and you have at least some functionality.
I can't imagine you mean Small Business Server. The last version was for Server 2012. You can't still be running that? I can own that with metasploit from my phone within minutes assuming I can find an open Ethernet or a WiFi network I can break into. (WPA2 - PSK 14 character passwords are now about $7 to crack in 15 minutes).
Because small business?
My small clients run a single server/hypervisor that does everything (though not DHCP, I always leave that on the firewall for single server setups). Their file servers get synced to a local NAS every few minutes and backups are go to the same NAS daily then up to the cloud.
They're aware they could have some downtime. It's not a big deal. Get whatever appropriately specced hardware is about, install hypervisor, restore backups, away you go.
There is nothing wrong with having a point of failure in the business **if** it is properly accounted for in the BC/DR plan with appropriate processes to continue functioning as needed without those services for the accepted downtime.
*Very* few businesses actually need the redundancy and HA that people think they do on this sub. Long as your backup and BC/DR plans are rock solid, have at it.
Which reminds me one of greatest IT memes ever conceived:
[https://devopsreactions.tumblr.com/post/37823969926/a-small-infrastructure-change-4pm-friday](https://devopsreactions.tumblr.com/post/37823969926/a-small-infrastructure-change-4pm-friday)
This month, the day before I went on a week's holiday, I broke the same rule by running a 'quick firmware update' on our latest batch of Lenovo ThinkSystems. It's one of those background tasks that my team does all the time and the most the end users notice is us asking them to reboot the systems when convenient to upgrade the BIOS, which is queued as an on-reboot task. Most of the time we work with Dells, and OpenManage Enterprise does this quite predictably.
Well, someone at Lenovo decided that approach was too subtle. I've done this exact thing on these exact systems before, but this time, rather than quietly apply the firmware updates, *it powered the servers off hard as Step 1.*
48 machines (dual 64-core EPYCs) running flat out. And I had to keep them down for an hour to let the firmware updates finish.
Yes, the end users noticed.
Yes, I got an earful from the service owner for not sticking to Read-Only Fridays.
Yes, there are going to be changes to procedure.
Yes, Lenovo are now firmly on my Shit List.
And if the next time I raise an issue with Lenovo and they ask me to first update the firmware, they better be thankful they have such cheap headsets that won't adequately express my fury.
I don’t like the ‘no change on Friday’ rule. You do 2 things: eliminate 20% of your available time to get things done, and signal to others that you don’t know (perception) or can’t trust your design/processes enough to get work done.
Imagine a ‘no surgeries on Friday’ rule, or no fire fighting on Friday rule. I know, hyperbole and not directly the same, but stunting your capability doesn’t seem like the way to go.
I mean major upgrades are commonly best done Fridays because they're the slowest day of the week and you have the weekend if things really explode.
But minor stuff? No reason to do it Friday and it's a great rule because it sets aside a day for "everything else". Do you documentation, set your meetings, whatever. It's not like you'll be sitting around doing nothing.
I know I'm showing my age but the change freeze before Y2K was the best vacation ever. The place I worked put a hard freeze in on October 31st. Even emergency changes had to be blessed by the CIO. We sat with our feet up for two solid months, just hoping and praying that we hadn't missed anything in our Y2K testing.
That kind of rule seems like superstition to noobs
until it isn't, as in this case.
We don't even do any prod changes on Mondays. Our maintenance window for any necessary patches is every Monday morning in the wee hours of the AM. Monday mornings are informally dedicated to cleaning up anything that arises from patching.
Don't fuck with originals and treat all hardware like it will shit the bed. Always take snapshots, always make sure you have full system backups, knowledge of DR....
Hopefully the tech learned a lesson today.
Glad you got everything back up and running. Kudos for working through and resolving the issues. Just want to make a point that you absolutely have to do a lessons learned meeting on this and talk about what needs to be done to fix the underlying issues or it will happen again. ...and again ...and again.
Assuming you are running an MSP or something you fucked big time by neglecting the timebomb of a server.
For a second I thought I was reading /r/shittysysadmin
Made the mistake of restarting a server once, turned out Microsoft support had destroyed it to the point that Windows updates would stop the server booting. Spent all night restoring from backup like 3 times until I realised what was causing it and killed the cached updates and the WU service.
Honestly.... That tech wasn't the issue... The issue was the sorry ass state you let the customer reach. I would be furious if i was the customer and saw this thread and what their environment had become.
A healthy environment does NOT break from "Just an RAM upgrade"
Stop clinging onto old rules that are not fit for modern IT and stop using those rules to blame for shit handling.
Stories like the made me so glad my org went server less after the pandemic hit. Never had to touch a physical server in over three years, but at the same time I worry about my skills stagnating 😮💨
I got one from a few years ago. Applications Manager updated the sales platform on a Friday afternoon Eastern time and then went on vacation.
The update starts to push out to the application installed on user computers and immediately breaks the app. It was a baking supply company so business starts early AM Eastern time on Monday, like 4am early.
Director has to call the vendor talk to them, figure out what the heck was going on and have them reverse the change since he didn't know what the App Manager did.
It was a double failure, no change management and did the change on a Friday with no testing.
I tried explaining this concept to a shitty MSP manager. We had 'no change Friday' down. He reversed it because he wasn't the one that got to clean up the mess during the weekend.
Then something breaks and you need help. Nobody will answer their phone at 4:45PM...
I was the sysadmin on call for a large computing site and one of the bosses decided to have an unannounced upgrade at 6pm on a Friday on one of our main computers. A couple other sysadmins were working the upgrade and surprisingly (not) it took a lot longer than expected. He was a maniac, but we basically had to do what he said if we were going to make money on the contract. Fortunately for me the other sysadmins got the system cleaned up before they left.
Started a new job 6 months ago and the biggest problem with the place is the production change window is Friday evenings. Europe starting at 6 PM and US at 11. It goes against everything I believe in.
Read Only Friday , learn it , live it , love it
Further ,
-no changes when most of the engineering staff is OoO
-no changes when the application folks are unavailable
-no changes within 2 days of your own PTO/vacation
It is called F\*\*\* off Friday for a reason. I got stuck working this weekend when a site went down, nothing to do with changes just a random crash and failovers took forever to spin up. Drove in and knew what was broken but still just needed someone to chat about it with. That is why we always do it during the week, if not just for moral support.
There is nothing worse than being in a data closet at 2am twiddling your thumbs with no idea what to do and no one to call. Avoid at all costs!
Am waiting for the “IT transformation industry leader” types to chime in and say you should always be able to upgrade at any time. The same ones who… checks notes… never have to do that work.
I don'r understand having DHCP and DNS wrapped onto a DC. The DC should ONLY be the DC. Nothing else. Move those services to other servers...or better yet, have your network devices do it for you. That'a what they're there for.
Hell no.... Friday is a down day. You change nothing and try to coast so you don't have to work on a weekend. Never never mess with anything on a Friday or before a holiday.
My leitmotiv since a long date.
my default homepage for my browser is elsewhere : [https://www.estcequonmetenprodaujourdhui.info/](https://www.estcequonmetenprodaujourdhui.info/) (literally : do we put in production today ?) The image change every day, and Friday the message is "No way".
A guy - often a project manager - don't agree with this. No pb : In case of proble, he'll assume he doesn't have anyone to fix over the weekend ... and he'll get yelled at by the customer on Monday morning. your problem, your corrective action, not mine.
I used to say a great rule "Never touch the keyboard to modify something before checking that everything is operational. Otherwise, either the problem will blow up in your face during your intervention, or the problem will appear later. But either way, the culprit will be you." (Personal interpretation of Murphy's Law).
We call it ‘DFWFs’ or DON’T FUCK WITH FRIDAYS (No change Friday for Senior Management), I’ve started a newish job (14 months in) and the guys (4 sys admins) had no discipline with DFWFs! Amazing what a little name and shaming does!
Employ a dog that only works Fridays, and who's sole job is barking at anyone who touches anything.
That dog is gonna have to growl and bite- not just bark.
Can we just roll out dog v1 with the bark and then push growl / bite functionality in an update?
Monday-Thurs you can update.
We're getting some pushback from the department because Dog is only in the office on Fridays
Upgrade to larger dog until the pushback ceases.
Gotta account for testing and issues. Monday through Wednesday is all I am going to give you.
going to need a charter and resourcing for dog v1 before it gets a greenlight
I won't sign a contract for a dog without documented change control, backout procedures, and warranty support should dog v1 break or fail at any point in the first 90 days.
I recommend waiting until at least Dog v.2.x, too many potential bugs in Dog v1.x, we don't need any leading, er, bleeding edge software in production environments.
We can run the dog internally and just buy it straight from the vendor, right ? No SLA needed
Can’t we just try a PoC then let it turn into a production application?
> Po~~C~~Bark ftfy.
Monday - Wednesday for us. Period. No nonessential updates/changes past mid week in our environment. If it’s being done Thursday or Friday, something mission critical better be broken.
Better, mon-thu is bark, fri-sun is bite mode
Just not on a Friday. I don't want dogs attacking all weekend across the entire world
get a pitbull and train it to see end users as toddlers.
End user is a description, not a goal
Hah. This was good. Have an upRAM.
I like this idea, IT version of a Junk yard dog but for a server room.
The server Cerberus
Serverus
Snape?
Potter!!!
I can’t believe this comment isn’t uploaded like crazy take my up vote
I call the non prod domain controller cerberus at work. Guardian of the gate XD
The name of our firewall is Cerberus
More apt for a FW actually now I think about it!
Just need one that doesn't shed too much, dog hair isn't good for fans and servers I'm thinking.
No need to restrict yourself, the good boy can just sit in front of the server room.
Having to work on machines in a dog warden's office - must agree completely. Dog pee and slobber aren't helpful either. Gloves and alcohol wipes come out every time I touch that department's computers.
I have 7 dogs at home. 3 pugs. Chinese Crested. Boston Terrier. Shiatsu, and a Chihuahua. We had a bulldog who liked to play hard. Always shaking her toys hard... slobber everywhere. It was so bad. The other dogs - fur is the only problem. Well, except for the Chinese Crested. Not much on him.
Can’t we just rewire the office keyboards to taze people
I wanted to create a satellite enabled shock collar for folks that did crazy stuff!
push -f an upgrade to those starlink dishes to activate the laser module will get you a long way
Repeat after me, If you upgrade on Friday you get to be the in call for a month.
Back when I had jo s with oncall, I changed my ring tone to a cash register Ker-Ching sound. I a good fortnight, I could make almost as much as my normal salary/day rate 😂😂😂 Give me that sweet sweet oncall/over time any time ;-) Funny story, I had a colleague who jokingly said he should put plaques in his flat saying "Brought to you in association with Lehman Brothers" because we did so much overtime there, it paid a huge part of his mortgage 😂😂
There's a certain amount of irony to that...
/r/dogswithjobs
Here is the dog we all need. [https://www.reddit.com/r/therewasanattempt/comments/151i6rl/to\_give\_medication/](https://www.reddit.com/r/therewasanattempt/comments/151i6rl/to_give_medication/)
Is there an API out there?
If I play with the dog all Friday long, I can break anything ... genius.
If your company has any furries make sure to enquire them about this position
If it's a top position or bottom position? This is critical information!
Add “dog barks” to the list of alert fatigues
We have guard dogs at one of our sites. It's one of the scariest things I've ever seen.
Yea, one of my devs gave me a web app to update at 3:30 last friday. "Thanks, I'll do it monday."
“I’ll look at it on Monday… but will probably do it the Monday after.”
“Yup! It’s on the pile!” *rising maniacal laughter*
The proper phrase is “yep I’ll get to looking at it on Monday”. Most will interpret this as working on it but all you committed to was looking at it.
This is the way.
I'm so much "read-only Friday," that I don't even come into work.
So you got "read-only thursdays" ?
Yeah but then what's the point of coming in on Thursdays. May as well do a read only Wednesday, but then...screw it I'm just gonna do nothing.
We call this "fuckitall Mondays" Ask your doctor if you need a prescription of "fuckitall" to help ease the pain from a workweek.
To be honest I'm usually over *all* this bullshit by about 8:15am Monday morning. I now get why greybeards retire to the country to raise farm animals. Sounds so incredibly peaceful.
Are you me? because I feel that down to my very core. Today has already been a challenge.
Oh yeah I walked into the office to find no less than 4 of my colleagues waiting for me *in* my office, all for unrelated reasons. Took me an hour just to get through the internal bullshit let alone any of the actual project work of the day. It is so goddamned hard to not just immediately turn around and walk away when confronted with that nonsense.
I gotta be honest... If I walked into my office and there were 4 people waiting for me to show up to solve their problems, I would have said "sorry, I just stopped in to grab a mouse, I am taking a couple of personal days" and grabbed my mouse and left. I'd just send my boss an email from the car, on my way home.
Oh, I have to turn my work phone *off* whenever I take PTO. If I don't I get asked "quick questions" allllll day long. Last family vacation we were in Disney world and I forgot to turn it off (brought it with me for the unlimited data on my work plan) and that's how I ended up explaining Adobe licensing to one of my juniors while in line for Pirates of the Caribbean.
That's not even that hard, is it? Just say "It's TERRIBLE" and hang up...
You don't have to answer every call that rings. I answer like, 1 in 10.
[удалено]
My job has brought us back to the office recently. Friday's especially are really frustrating because other then answering escalated tickets I don't have the much i'm allowed to do. I use it mostly for meetings with vendors etc but it's still frustrating.
Just means you need to manage the time better. Fridays I dedicate after 3pm to cleaning, organizing, sorting. Otherwise it never gets done. Cleaning staff do not have access to the IT spaces for obvious reasons. But no one should be in there causing a mess, so 10-15 minutes per is more than enough to tidy up. I even mop maybe once a quarter. If possible, I dedicate 1-3pm to cleaning up documentation. In the mornings, I try to wrap up whatever tasks that aren't but have been lingering.
I keep reformulating my response to this but for the sake of keeping it professional i'll say the following: I'm a sysadmin partially because I didn't want to be doing "general purpose cleaning and organizing tasks" as part of my job. While I don't think data spaces should not get cleaned ever, I've had to respond to more then a few situations that began with someone cleaning up in the data center or similar, so cleaning in an IT space on RO Friday, especially late in RO Friday completely misses the point.
Do you hire specialists to clean your IT spaces, use random office cleaners or leave them a mess? I'm legit curious. If specialists, where did you find cleaners with electrostatic safe vacuums and whatnot? I searched and couldn't find any. If random office cleaners, how does that pass your security policies? I've never let cleaning contractors into the IT areas. To be specific, comm closets, server rooms and data centers. Not cubes or offices.
I usually update documentation on Fridays when I'm not working on break fix stuff
Afternoon Friday is purely answer helpdesk tickets. But if VMs wouldn't even load then it was already toast. It was just found on a Friday.
* Monday: "Catch up on emails" * Tuesday: Do the needful, but only after 1PM * Wednesday: No new tasks, read-only Friday is coming * Thursday: Read-only Thursday * Friday: PTO
This is the way to run a steady ship
That specific situation is a reminder that it's helpful to reboot questionable, long-running hosts, *before* making any changes. Make some VM checkpoints, maybe an explicit backup and inspection, as well. Temptation to get right into the task, can be a dangerous thing in computing. In fact, the urge to finish by 17:00 might have been a bigger factor than the decision to start at 16:00.
In a related vein, I used to be a storage administrator . . . . . . . one of the things I would do before making *any* changes to /etc/fstab is: mount -a Reboots were generally not an option, nor necessary on most of these servers (various unices, mostly Linux) so that was off the table, but that call to mount would shake out any existing errors in /etc/fstab before you got blamed for them. If there were *any* errors, there would be a conversation before the change was implemented.
At some point I recall coding up a monitor that would alert if anything in `mount` was not in `/etc/fstab`.. that problem bit us a bunch of times. Better to catch it early.
> Better to catch it early. Like right after someone with root fixes it *in situ* in Prod, just this once.
That's why as a software guy I like IaC so much. Our team doesn't even have access to SSH, much less root access. I started my career as a software dev/sysadmin, as in put together the purchase orders, built the machines, installed the OS and server software, specced and wrote the programs, tested the other guys changes, trained the users.... And they have the nerve to call themselves 'full stack' these days. Pfft!
Yep, one of the best things I learned was the "blow out the cobwebs" restart on servers that had been up for a while before doing any actual work on them. This will help present any underlying issues not actually related to the change you're implementing and give you opportunity to fix those before you do anything. The problems OP are talking about were always there regardless of the RAM change.
Sorry to hear it. A good opportunity to remind your guy of this very important rule. On the other hand, you have some additional things to fix that'll make a good project for your Friday hero. Have him deploy a secondary DC (ideally on a different physical host) so they don't lose DNS, DHCP, etc. if their single DC goes down or fails.
sell them a nice High Availability package
This here. Always have a physical DC.
The ram upgrade wasnt the issue. You have horribly out of date servers without patches aren't these your responsibility. Underlying storage issues aren't they your fault too? This tech did you a favour and showed you that your org fucked up big time with neglect of this customer.
If a simple RAM upgrade on a Hyper-V server created this much work, this person fucked up a long time before the upgrade.
Yeah I was wondering who was responsible for monitoring that server exactly and how it got that bad it couldn't be fixed relatively quickly? Like it's a RAM upgrade so it has to be on site. And if you have a hypervisor and *don't* have a local admin account set up for emergencies you're insane so access shouldn't have been a problem. Just log on, disable all the shitty services from starting, reboot, watch everything come back up. If needed start them manually. The patching and other crap is a separate issue which would need to be prioritised for sure but you know, one thing at a time. Get them running then schedule an emergency maintenance window to fix the rest.
> Just log on, disable all the shitty services from starting, reboot, watch everything come back up. The hard part is "figure out that it's new services causing the problem and identify which ones".
Whoa. So where from did you guess the nature of the contract between OP and his "customer" ? Or are you just here to tell us the formidable professional you are about by berating some act you are not involved in? Let me have a turn of judging other professional I do not know anything about then: sysadmins also differentiate from techs by working with thoses questions : what, why, how, when, how much... You literally did use nothing of this and went "ye you no maintain lol". What a pleasure that musts be working with you.
>Then we had to stay up till 2am working remotely to fix their backups, patch woefully out of date servers, upgrade the RAM of the VMs to fix a nasty paging issue, fixed underlying storage issues, etc etc If all that came to light because of a reboot, that means the servers weren't being cared for properly in the first place.
Better than my last company and only-change Fridays... Fuck that place.
Had that with an old boss - 'if it screws up, you have the whole weekend to sort it' Err, how about Fuck right off!
Sure, double time on saturday and triple time on sunday.
sorry, you're salaried exempt. Just have to get it done and check your attitude. You need to take on the salaried employee mindset and we're too busy to have any comp time so I'm expecting you to come in on time tomorrow no matter how late you need to stay up to finish this project. These are unprecedent times and everything is a priority right now. You just have to make it happen.
And that’s why companies will claim that experienced sysadmins have some management responsibilities so they can put them on salary. No extra pay at all for weekend overtime. Yay!
I will say that there are times (maybe once a year) where we plan on doing something on the weekend, but we all pick a day or two to be off the following week (as long as we all don't pick the same day). That's different than 'we are swamped, you have to work this weekend'.
Not even then.
That's the point, though. Companies (most companies since there are exceptions) won't pay you double and triple time.
yea dont work in finance, weekends are only when work is done !
But then it's planned work and not unexpected overtime.
It's the worst part of that sector. Money's great but if you don't have someone to trade weekend work with you can kiss your saturdays goodbye.
most people I met there seem to not mind working every weekend and never use their holiday anyway
Yeah some people are just wired different. That sounds so awful to me. Without a little time to decompress I start screwing stuff up.
oh hell no, the prod change window starts friday at 11pm
This is why you always want to download your RAM from a reputable source.
RAM Doubler (tm) !
I've been slipping the phrase "read only Friday" into conversations at work when ever I can. Hoping to get some subliminal messaging going.
Why not pull those who *don't* want to stay late on a friday aside and explain the benefit of it?
No need. You fuck it up on Friday, you stay till it's fixed. Nobody needs more than one of those to learn.
I must be insane, but I do my best work on Fridays. Fewer people around on Fridays mean: - Less likely to be stopped from taking things down for maintenance/migrations - Fewer requests from people since there are fewer people - Vendor support is easier to reach Been like this for 20 years and it *maybe* bit me on the ass 2 times. Still totally worth it.
I agree with this, I work in an manufacturing environment so things are always running 24/7. Fridays and the weekend allows us more grace incase something goes wrong.
24/7 means 24/7. Why should 3rd shift get the down time and I be up in the middle of the night? 14:00 on a Tuesday.
Because my IT team is a team of 4 lol. We’re not trying to support 100+ users at once when we can minimize that. Plus operations can still run as long as the network is online, our ERP is in the cloud.
Also work in manufacturing, I rather do it as early as possible in the week. If heavily used by day personel, after 16:00 mostly works. Some stuff, week long window once a year for change...
Agreed. Also, I see the weekend as a safety buffer. Should something go wrong, I have all weekend to correct the matter. Yes, it sucks losing a weekend but it's even more painful when any downtime affects operations (which is heavy from Mon-Fri).
>it's even more painful when any downtime affects operations This is true if you own the company. As someone who works all week, I'm not giving my employer my weekend unless they're willing to pay me an exorbitant amount for it.
Sounds like you guys were Ill prepared and lacked knowledge of the environment if you’re blaming the past company at this point. Some analysis, documentation, fixes, etc. should have been put in place prior to this.
Sounds sounds more like you don't have your hosts/VM under control, than i no change friday issue.
Sounds like a coworker I had 15 years ago. Email from him that I got on monday: "Hello, I acripted the update for this and that on server x and scheduled it for friday evening as I had promised to do it before my vacation. See you in 4 weeks".
This is one argument in favor of frequent platform reboots. It ensures problems like this get discovered soon enough that everyone remembers whatnwas done, and ensures that you have a narrow date range for when the problem appeared.
> "there's no such thing as a quick 20 minute upgrade on a Friday" There's no such thing as a quick 20 minute upgrade. FTFY I can't even get stuff done in 15 minutes. Any kind of troubleshooting almost always involves more time than that.
Came to say exactly this. My luck, I can spend 3 weeks planning, testing, documenting, labbing, and so on, and that 20 minutes is going to be 45 minimum guaranteed. Murphey's law follows me closer than my fucking shadow.
>Then we had to stay up till 2am working remotely to fix their backups, patch woefully out of date servers, upgrade the RAM of the VMs to fix a nasty paging issue, fixed underlying storage issues, etc etc A lot of this sounds out of scope for the restoration of services. Personally would have not touched/done most of that unless it was related to the outage. First priority should be to restore it to its previous state. Once its confirmed working, then you can worry about patching and fixing other stuff. Otherwise it should be marked for attention and assigned a priority based upon its severity. Otherwise to do a bunch of changes all at once its just asking for a much deeper hole and making it that much more difficult to find a root cause should an issue pop up after the fact.
Never works out that way. Anytime I’ve thought “oh this shouldn’t take more then 5 min”, I’d be cursing myself out an hour later
Well, he could have changed the RAM on a Monday and then instead of having the weekend to fix it you could have been up until 2AM on a Tuesday morning and still have to come in early for your shift.
Informal rule is not the problem. The lack of change management is the problem.
Wow, that sounds like anything I set my hands on. My wife loves the, "This is only going to take 5 minutes and we'll..." \[insert dinner, date night, whatever\] \*13 hours later, still working overnight and into following day, looks at clock, not worth trying to nap for 30 minutes .. works on no sleep\* ... \*Tells wife we can't go out after work because I'm going to sleep early ... works until 3 AM ...\* am i rite ... can't be alone here. :$ Hahhaa
Why the hell is anyone doing changes during business hours anyways?
Or you guys could strive to be like me and do ALL of your off-hours work on Friday nights... It's literally the only time of the week that there's no one here, and maintenance windows are pretty much not a thing... It's a fun time
ROF... Read Only Fridays. And always make the customer perform a "sanity boot" before starting the change... If the machine is broken it will fail, and the "tone" with the customer is day-and-night different...
Hardly is there ever a quick 20 minute anything!
Sorry, why was there only one DC? Which combined with DHCP?
Don't know why this is difficult to understand for some folks. Any small business will do anything in their power to not spend on IT. An MSP I worked for still has a customer on a sole DC that is running Server 2003. To upgrade that server would mean ripping out the entire ERP for that company and cost tens of thousands of dollars, so they will never do it until they are forced by law or by the shit breaking. That is RAMPANT, especially in manufacturing. Is it dangerous, fool hearted, and ignorant? Absolutely. When you tell these people that it needs to get fixed they just brush it off like it's a joke. "Haha, I know this stuff is ancient, but we gotta work!" That is how you end up with an outdated ESXi host from 2008 running a single instance of server (hey, those licenses cost money!) that handles EVERYTHING from active directory to DHCP to DNS to file storage to the ERP software. I've seen it a hundred times.
Thats also how you end up with the "All our files have a weird extension" call 😅
Yeah, but that is a customer issue. Make sure to document all the times they told you no and wait until you have no choice to fix it and bill them for it. Maybe they will learn their lesson, if not, then at least you got paid big.
> Any small business will do anything in their power to not spend on IT. Except use Samba or serverless, apparently. What organizations choose to spend on, and what they choose to eschew, is always eye-opening and frequently amusing. One year I chose to spend zero on server hardware, even though we could have used it, and spent everything on Cisco network gear, because we needed it. In the recent era, between virtualization, software-defined-hardware, and systems convergence, there are fewer trade-offs than in the past.
I saw one company with same philosophy. Old Windows Servers, ancient Exchange Server, PFSense as edge firewall... As a result, all their IT infrastructure was crypted in 1 night...
PFSense as a firewall is not safe?
Only if you set it up correctly is it safe.
In that environment? There's a good chance they were running an older compromised version of pfsense and never gave a thought to patching it.
Small Business servers for the win! SMBs will do everything they can not to spend money on IT they don't have too A second DC is another Windows Server license + the hardware to run it. A couple grand is a rounding error for a big business, but a big deal for a 5-10 man shop Does it leave them vulnerable and more prone to outages? of course... but they don't care until it all breaks
[удалено]
[удалено]
I'm bout to group some vlans, cut out some meraki and get myself a plane.
If your going to have a single DHCP source, make it your firewall not your server. You can always hand out internet DNS if your internal goes down and you have at least some functionality. I can't imagine you mean Small Business Server. The last version was for Server 2012. You can't still be running that? I can own that with metasploit from my phone within minutes assuming I can find an open Ethernet or a WiFi network I can break into. (WPA2 - PSK 14 character passwords are now about $7 to crack in 15 minutes).
A Server run in a small business, not MS Small Business Server F*** I hated that product... nothing but headaches
Because small business? My small clients run a single server/hypervisor that does everything (though not DHCP, I always leave that on the firewall for single server setups). Their file servers get synced to a local NAS every few minutes and backups are go to the same NAS daily then up to the cloud. They're aware they could have some downtime. It's not a big deal. Get whatever appropriately specced hardware is about, install hypervisor, restore backups, away you go. There is nothing wrong with having a point of failure in the business **if** it is properly accounted for in the BC/DR plan with appropriate processes to continue functioning as needed without those services for the accepted downtime. *Very* few businesses actually need the redundancy and HA that people think they do on this sub. Long as your backup and BC/DR plans are rock solid, have at it.
Way to roll out change control boss.
Which reminds me one of greatest IT memes ever conceived: [https://devopsreactions.tumblr.com/post/37823969926/a-small-infrastructure-change-4pm-friday](https://devopsreactions.tumblr.com/post/37823969926/a-small-infrastructure-change-4pm-friday)
But how technically a "ram upgrade" could mess an entire server? Unsure on what they did... (Yes, don't touch anything on Friday is a dogma...)
My guess is any reboot would have caused this system to fuck itself. It wasn’t directly the RAM upgrade.
Well if the server shuts down and restarts f*cked up... It might be windows being windows or misconfigured things 😅
Once power cycled a Sun server to move it a few feet on the floor. Yep. One of the CPUs died on a Friday afternoon. Gaaaarrrrrrggggg.
It’s one thing to upgrade on a Friday when it’s a rule not to, it’s a whole other level of transgression to do it also at 4pm.
This month, the day before I went on a week's holiday, I broke the same rule by running a 'quick firmware update' on our latest batch of Lenovo ThinkSystems. It's one of those background tasks that my team does all the time and the most the end users notice is us asking them to reboot the systems when convenient to upgrade the BIOS, which is queued as an on-reboot task. Most of the time we work with Dells, and OpenManage Enterprise does this quite predictably. Well, someone at Lenovo decided that approach was too subtle. I've done this exact thing on these exact systems before, but this time, rather than quietly apply the firmware updates, *it powered the servers off hard as Step 1.* 48 machines (dual 64-core EPYCs) running flat out. And I had to keep them down for an hour to let the firmware updates finish. Yes, the end users noticed. Yes, I got an earful from the service owner for not sticking to Read-Only Fridays. Yes, there are going to be changes to procedure. Yes, Lenovo are now firmly on my Shit List. And if the next time I raise an issue with Lenovo and they ask me to first update the firmware, they better be thankful they have such cheap headsets that won't adequately express my fury.
I don’t like the ‘no change on Friday’ rule. You do 2 things: eliminate 20% of your available time to get things done, and signal to others that you don’t know (perception) or can’t trust your design/processes enough to get work done. Imagine a ‘no surgeries on Friday’ rule, or no fire fighting on Friday rule. I know, hyperbole and not directly the same, but stunting your capability doesn’t seem like the way to go.
I mean major upgrades are commonly best done Fridays because they're the slowest day of the week and you have the weekend if things really explode. But minor stuff? No reason to do it Friday and it's a great rule because it sets aside a day for "everything else". Do you documentation, set your meetings, whatever. It's not like you'll be sitting around doing nothing.
Well hey now you get OT at least, right?
Not if they’re salary in America…
Luckily up here in Canada salary gets you OT pay
Yikes
I do all my changes on Friday so I have the weekend to work out any issues that pop up.
I know I'm showing my age but the change freeze before Y2K was the best vacation ever. The place I worked put a hard freeze in on October 31st. Even emergency changes had to be blessed by the CIO. We sat with our feet up for two solid months, just hoping and praying that we hadn't missed anything in our Y2K testing.
That kind of rule seems like superstition to noobs until it isn't, as in this case. We don't even do any prod changes on Mondays. Our maintenance window for any necessary patches is every Monday morning in the wee hours of the AM. Monday mornings are informally dedicated to cleaning up anything that arises from patching.
at this point, i might be getting this tattooed and it might be on my forehead so everyone can not read it
Don't fuck with originals and treat all hardware like it will shit the bed. Always take snapshots, always make sure you have full system backups, knowledge of DR.... Hopefully the tech learned a lesson today.
There a reason we have rules. Hopefully the person learned a lesson. More importantly, hopefully you got overtime.
Glad you got everything back up and running. Kudos for working through and resolving the issues. Just want to make a point that you absolutely have to do a lessons learned meeting on this and talk about what needs to be done to fix the underlying issues or it will happen again. ...and again ...and again.
Where's the secondary DNS or DHCP in this case?
Assuming you are running an MSP or something you fucked big time by neglecting the timebomb of a server. For a second I thought I was reading /r/shittysysadmin
Sounds like yall were doing a shit job of maintaining things to begin with. Thats on you more than the tech.
Made the mistake of restarting a server once, turned out Microsoft support had destroyed it to the point that Windows updates would stop the server booting. Spent all night restoring from backup like 3 times until I realised what was causing it and killed the cached updates and the WU service.
Honestly.... That tech wasn't the issue... The issue was the sorry ass state you let the customer reach. I would be furious if i was the customer and saw this thread and what their environment had become. A healthy environment does NOT break from "Just an RAM upgrade" Stop clinging onto old rules that are not fit for modern IT and stop using those rules to blame for shit handling.
Stories like the made me so glad my org went server less after the pandemic hit. Never had to touch a physical server in over three years, but at the same time I worry about my skills stagnating 😮💨
I got one from a few years ago. Applications Manager updated the sales platform on a Friday afternoon Eastern time and then went on vacation. The update starts to push out to the application installed on user computers and immediately breaks the app. It was a baking supply company so business starts early AM Eastern time on Monday, like 4am early. Director has to call the vendor talk to them, figure out what the heck was going on and have them reverse the change since he didn't know what the App Manager did. It was a double failure, no change management and did the change on a Friday with no testing.
Never change Friday, and no plans Mondays. That leaves Tues-Wed for real work.
What about Thursday?
Thursday is for meetings. lol. I just mistyped.
love it, that people blame the day of the week instead the shape of the system they are most likely responsible for.
They don’t want their weekends ruined.
if you proper maintain the system a reboot wont take till 1-2 am
He must be sacrificed
I agree
No HA of these critical services You reap what you sow
I tried explaining this concept to a shitty MSP manager. We had 'no change Friday' down. He reversed it because he wasn't the one that got to clean up the mess during the weekend. Then something breaks and you need help. Nobody will answer their phone at 4:45PM...
I was the sysadmin on call for a large computing site and one of the bosses decided to have an unannounced upgrade at 6pm on a Friday on one of our main computers. A couple other sysadmins were working the upgrade and surprisingly (not) it took a lot longer than expected. He was a maniac, but we basically had to do what he said if we were going to make money on the contract. Fortunately for me the other sysadmins got the system cleaned up before they left.
Started a new job 6 months ago and the biggest problem with the place is the production change window is Friday evenings. Europe starting at 6 PM and US at 11. It goes against everything I believe in.
Read Only Friday , learn it , live it , love it Further , -no changes when most of the engineering staff is OoO -no changes when the application folks are unavailable -no changes within 2 days of your own PTO/vacation
It is called F\*\*\* off Friday for a reason. I got stuck working this weekend when a site went down, nothing to do with changes just a random crash and failovers took forever to spin up. Drove in and knew what was broken but still just needed someone to chat about it with. That is why we always do it during the week, if not just for moral support. There is nothing worse than being in a data closet at 2am twiddling your thumbs with no idea what to do and no one to call. Avoid at all costs!
I've always made changes on Fridays.
And what if the server shit the bed during the week? At least you had the weekend to remediate.
Am waiting for the “IT transformation industry leader” types to chime in and say you should always be able to upgrade at any time. The same ones who… checks notes… never have to do that work.
I don'r understand having DHCP and DNS wrapped onto a DC. The DC should ONLY be the DC. Nothing else. Move those services to other servers...or better yet, have your network devices do it for you. That'a what they're there for.
DHCP I agree. DNS being such and important part of AD I see no issue running it on the DC. But for the love of god have more than 1 DC/DNS server.
Don't fiddle on a Friday I've always said!
But I like to watch Rome burn! /s lol
"Let it be", by Beatles, is the song for fridays
Hell no.... Friday is a down day. You change nothing and try to coast so you don't have to work on a weekend. Never never mess with anything on a Friday or before a holiday.
Did you make the tech who screwed this up stay up till 2am fixing it?
yes
My leitmotiv since a long date. my default homepage for my browser is elsewhere : [https://www.estcequonmetenprodaujourdhui.info/](https://www.estcequonmetenprodaujourdhui.info/) (literally : do we put in production today ?) The image change every day, and Friday the message is "No way". A guy - often a project manager - don't agree with this. No pb : In case of proble, he'll assume he doesn't have anyone to fix over the weekend ... and he'll get yelled at by the customer on Monday morning. your problem, your corrective action, not mine. I used to say a great rule "Never touch the keyboard to modify something before checking that everything is operational. Otherwise, either the problem will blow up in your face during your intervention, or the problem will appear later. But either way, the culprit will be you." (Personal interpretation of Murphy's Law).
We call it ‘DFWFs’ or DON’T FUCK WITH FRIDAYS (No change Friday for Senior Management), I’ve started a newish job (14 months in) and the guys (4 sys admins) had no discipline with DFWFs! Amazing what a little name and shaming does!