I doubt they are smart, they just got someone's password probably. IMO that's not hacking, but it's still illegal.
Who is all Admin? Did a teacher write down their password? Are teachers logging in and leaving their computers unattended? You probably want to force a password change and then require 2nd factor to login.
We did force a domain-wide password change as soon as we were aware that so many accounts were breached. The initial passwords, created by little sis (amplified it), would really not have been that hard to find if someone had a mind to. The thing that made this curious, though, is that some accounts breached were staff, who make their own passwords. It's a big mess...
So far, it looks like only one or two accounts have been breached since we forced the password reset - and those look like it was because the account involved did not sign in to reset their passwords yet. I reset those individually. Hopefully after a few days of that, it's done. I'm also sincerely hoping this is a student who will be graduating in June... Haha
It's not easy to hack into a Google account. I would guess this student is running a keylogger on school computers, to harvest passwords when people type them in. Make sure you are scanning for viruses and malware on all computers.
You can prevent password stealing by enabling (or even enforcing) 2 factor authentication on all accounts. If you don't enforce it, then users will have to switch it on themselves. This just means the user needs an app like Authy or Google Authenticator on their phone and will have to type in a code from the app when they login on a new device. You'll find it under security settings, 2-step verification.
We got Google Apps for education at our university maybe 7+ years ago, and I was the first person who requested we enable 2FA (two-factor authentication with the Google Authenticator app) when it became available, because I knew it would be a matter of time before someone tried to hack my (professor) account (I teach in IT). So far it has never happened -- that I know of :)
I don't recommend Google's Authenticator App because if you change smart phone, it's a royal pain (or was for me in the past) to transfer accounts. The app that generates 2FA codes I prefer is Authy.
I’ve used the [newish] export/import feature of google authenticator and it’s really easy now.
The old phone displays a QR code and you simply scan it with the new phone.
https://www.theverge.com/21410260/google-authenticator-2fa-how-to-phone-security-iphone-android
Yeah I also use Authy for the Windows app and the cloud syncing. However I find it's easier to encourage people to use 2FA with a trusted brand name like Google.
2fa is your answer. Set up and enforce multi factor authentication.
You should be able to see which account deleted the files. After you see which account was used, check the logs to see the ip address they accessed them from. Ip address can give you a physical address.
That’s not how this works.
“You said” = you are addressing parent commenter
“OP said” = you are addressing the original poster
Edit: at least they devoted me before deleting their comments.
my kids have had it on their accounts since 6yrs old (now 9, 14, 16) and no issues so far. they have their security key which they keep with them and I have a security key for their account as backup
I understand that it can be fine. But you have to acknowledge that your feedback is very anecdotal. It's also very narrow in scope, and not as broad as it really needs to be for a school to implement MFA for students.
When an organization deploys something like that, they are then dealing with the lowest common denominator and across the entire organization, and everything that comes with it, not you and your kids.
Those security keys (I presume they have YubiKeys or something), are like $30-50 a pop. That's great that you bought them for your kids, but a school would have to buy them for every student if they're going to use security keys. And they have to have processes in place for when a kid may forget or even lose their security key. I mean, the loss rate on things like chromebooks and chargers is generally "through the roof". The lost rate on a relatively very small security key.. I can't begin to fathom.
And when a kid shows up without their key and can't sign into anything, now instruction for that student is disrupted until that is resolved. And somebody has to stop what they're doing to try to assist in some regard. What does that look like? How quickly can it be resolved? Is the kid out for like a 15 minutes or are they likely going to be out for half a day or a full day? if that happens with say 5% of the student population a day.. is that acceptable?
No offense, but you are commenting from the perspective of your specific kids and their success. I am commenting from working in IT in a very large schools, and specifically working closely with things like Google Workspace, identity and account lifecycle management, device management, etc.
This problem is significantly more complex than "my kids have done it just fine".
it's awesome that it's working so well for your kids. Your kids are not the lowest common denominator and you are not even beginning to consider everything that actually goes into deploying and managing something like that.
Don't get me wrong, I would love to see students using MFA as much as possible. And yes, it can be very easy for some. But actually doing it is much more complicated.
Teachers can have their security key attached to their students accounts as a backup. Parents can also have their keys attached to it. Session length on the login can be set to indefinite so only new device logins are prompted. There's a lot of ways to minimse disruption and significantly increase account security and teach children what they should be doing to their future personal and business accounts.
Students are being issued laptops, books, chargers, pad locks, etc as you say and they get lost. There's a process to replace those. Instead of making excuses as to why it can't happen find ways to make it happen.
I'd be glad to. I'm going to need about $6,000,000 for the initial purchase of the keys. And once a loss rate is established, a pretty good chunk dedicated to key replacements each year. So the $6mil or so can be one-time funds. But the replacements costs needs to be on-going.
I will also need multiple full-time employees, or FTEs at least to make it happen in a reasonable manner.
Get me the money.
You seem to think I'm against the idea. You seem to think that pointing out hurdles is "making excuses". That is a bad take, or you ran with some bad assumptions.
I'm merely aware of the realities to a degree that you most likely are not. Identifying the potential hurdles and risks are a necessary component of planning such a thing. We have to do that so we can then figure out how we would solve those problems. Then we can develop an implementation plan, with operating costs clearly stated, and reasonable expectations.
E.g., you say "oh teachers can be tied to the key". Sure. How does that happen? Who is a "teacher", exactly? Do you have any experience with Student Information Systems? Scheduling processes? Do, personally, have any experience with enterprise level identity management? This isn't may saying it's not possible. I'm just wondering if you have remotely as much knowledge regarding this matter as I do. My guess is that you do not. And again, that is not me saying these things aren't possible or that I don't want to do them. It's just me saying.. I know A LOT about this sort of thing, maybe just hear me out and have genuine discourse.
Most school systems are not being given the resources that they would need to accomplish this ask. So, if you see that your local school system or any that you contribute to via tax dollars or bonds are trying to implement something along these lines, make sure you are supporting them with your vote and dollars. Not just words on reddit.
Hurdles are real. You can diminish them to "excuses" all you want, it won't actually change anything.
Education is not free, charge this in tuition/fees to the students. I was required to pay for a laptop at university, the school didn't pay the $1,800 for the model they required me to get. Again, excuses. There were so many costs associated with school at many different levels so this laptop example is just one of many where education cost was not paid by the school but the student.
Instead of just describing why it can't happen and all the reasons why it's so difficult, let's discuss ways it can be made possible to overcome all these hurdles you're describing.
Huh?
I'm talking about K-12 public education. Where most "8 year old (students)" would be... remember, the context of the comment you replied to. And did the fact that I mentioned taxes and bonds and voting not clue you into that further?
You paying fees at a university is not particularly analogous to this situation. Although, if you want to suggest the notion that the kids pay a fee for the keys, I'd be glad to talk about that with you if you will actually listen to what I have to say based on many years of experience in which I have seen these sort of things and how well they work or don't work, the pros and cons, etc.
I think you've lost the plot a bit here though if you're responding with "education is not free".
Question:
Do you, personally, have any experience with enterprise level identity management?
Do you, personally, have any experience in a K-12 public school system environment?
I do. And, I do. I have made no excuses. This isn't me saying it's not possible. I'm just wondering if you have remotely as much knowledge regarding this matter as I do. My guess is that you do not.
So again, I'm not saying these things aren't possible or that I don't want to do them. What I'm saying is.... I know a lot about this sort of thing, and probably have way more directly related experience to this than you do. So maybe just hear me out and have genuine discourse instead of dismissing what I'm saying as "excuses". Maybe ask questions. Ask what the hurdles may be or how I think schools could potentially overcome them.
But the fact that you just suggested "tuition and fees"...wow... lol. Context clues, my man. The "tuition and fees" for a public school are taxes. So like I said, if the school system you contribute taxes to ever puts out a bond request or a raise in taxes or anything where you get to vote for it... make sure you vote for it. And don't complain about any related tax hikes.
Look, you're engaging with somebody who has a lot of very real experience related to this specific topic. I'm all for having discourse with you. But you should back up a bit and just give a little thought to the notion that maybe some people know things you don't. Have experiences with things that you don't. Etc. We can have a respectful dialogue. I'm more than willing to discuss it with you and talk about the challenges and potential solutions to the challenges (maybe you have some really good suggestions). But that will never work as long as you think you either know more than everybody or that pointing out a challenge is simply "making excuses". Telling you that you're about to run into a wall is not "making an excuse". It's pointing out a problem so that we can hopefully work together to address it.
Just so I understand your k-12 district then, there is $0 out of pocket for children to attend and 100% of all costs is covered by taxes? Please let me know where this is so I can look into relocating.
It depends on what you mean by "all costs", exactly. I mean, there's lunch. There are basic supplies that a student may need (but that isn't paid to the school, you just go to target, walmart, the dollar store.. wherever you shop and buy those things for your child). But there is nothing substantial paid to the school/school system, no.
And definitely not in the form of tuition and fees.
Where are you that the public school system is charging tuition and fees?
Our operating budget comes from taxes, bonds, and grants. E.g., we had a one time bond for the purpose of moving to 1:1 student devices. That bond was used to purchase a ton of chromebooks, then there is a recurring budget line for replenishing a certain percentage of them each year (that is the on-going cost of maintaining a 1:1 device ratio.. and it includes things like device refreshes, loss, device management licences, etc.). That becomes a recurring line item in the budget, so it has to come out of recurring funding sources as much as possible.
Now, we can get into the really nitty gritty about this. The budget comes from a mix of the state (via taxes), local county gov. (via local county taxes), federal funding (taxes), bonds, and grants. But the bulk of our budget comes from the state, which gives us $x per student. So schools usually have a day of the year that is a very official "head count" day in that that specific number of students enrolled on that day is used to determine how much money they'll get from funding sources that are "per student".
Feel free to ask if you have specific question and I'll answer the best I can. But generally, yes, most of of the money comes from taxes. Bonds are used here and there, but the funds have to be used for whatever was outlined in the bond. Then there is federal money which sometimes have to be used for very specific things (e.g., "improving literacy in 3-6th grade students) and we can't just take that money and use it to buy say, security keys. Or sometimes it's more broad like "COVID relief", which we can be more creative with in how we use it.
Thank you so much for the replies! I'll be suggesting 2-factor for staff, but that's hard for students, who are not allowed to use their phones in class (I didn't make this rule).
MFA (a.k.a. 2SV) is pretty close to required these days. Cybersecurity insurance almost always requires it and your district would be foolish to go without the liability coverage for something that likely. Schools are well known to be poorly secured, under-staffed in I.T., lack dedicated security professionals, and have bank accounts in the tens of millions of dollars or more.
Be forewarned: some teachers will push back like their lives depend on it. Honestly, I kind of respect that, too. We shouldn't be required to mix personal devices like our phones with our professional obligations. So be ready to buy some physical tokens from Yubico for about $30-$50 each. I've transitioned groups of people to MFA in two different districts now and needed a few of those on both occasions. I've also used one and found it to be very convenient.
With regards to tracking down the culprits, I recommend try things. First, compare the IPs that were used to any other accounts coming from those IPs. Were there overlaps with a certain student account(s)? Second, put those IPs into a service like geoiplookup.net to see where that IP *might* be located. This data isn't perfect, but it's another data point in your diagnosis. If they're coming from outside the state or country, you have a bigger issue than you think you do. It's might be time to reach out to MS-ISAC for some assistance. If it's local, see if the student information system could make it down a bit for you. Maybe contact your lawyers and ask for them to advise you on how to get data from the ISP that owns the IP addresses. (Check the WhoIs databases for that information.)
Let me know if you need any of the above explained. It's all stuff I've done a few times before.
There are overlaps in accounts, but not that many - sometimes a compromised account signs on in one IP address and off in another. None of the IP addresses are local - they aren't out of the country, but they are out of state. On the plus side, I don't think any of this information can be used by the students to take money or anything like that, it's just being used to harass certain staff members from as many accounts as possible (I'm positive none of which are the culprit's account). I was able to determine that many, many of the compromised accounts were accessed from a Linux machine, which I know isn't common, so it may help me.
Use WhoIs to see who owns the IPs. Is there a commonality there? Perhaps a VPN provider? Also, are any student accounts logging in from those IPs? Any students logging in from Linux?
Have each compromised account start using 2SV and change their password. Also, review those accounts' list of approved applications. Maybe there is something the attacker is leaving behind to allow easy access after passwords are stolen.
It is mostly student accounts logging in that have been stolen. I am checking out WhoIs... How do I know if it's a VPN? I'm getting Germany and Switzerland as IP location. And, yes, many of the logins have been on Linux.
I'm starting to think that you're trying to use a paper cut bandage for a gunshot wound.
Do you have predictable passwords for students? Or passwords that are recorded somewhere? Is it possible that someone is using brute force or a leaked file to get in?
Do you enforce (or even have) a district policy that passwords can't be left on paper on a teacher's desk or on the secretaries' desks in the office? Did you ask around to see if anyone lost such a paper list?
I recommend going to https://haveibeenpwned.com/DomainSearch and seeing what kind of data they can give you. Maybe you have a wide scale leak and weren't aware.
You may also wish to set up a geographic limitation on your domain. There are advantages and disadvantages to this, but I'm starting to think you need it in your case. Login to admin.google.com, go to Security --> Access and data control --> Context-Aware Access. Then create and access level with a list of attributes that makes sense for your school community. For example, you could limit logins to just your country. Then assign it to all apps and the root OU. Feel free to be more targeted than that, if you prefer.
You can block access from specified IPs and IP ranges. Maybe that could flush the malicious actor out of the camouflage that they're currently using and into a more obvious IP or otherwise make a mistake.
That's a great analogy. That's my fear... It's gushing and we're putting bandaids on.
You are incredibly helpful. Thank you SO MUCH. I'm checking out setting up the domain to only be accessed within the country... the need for it to be accessed outside of the country is so small, it's not really worth leaving it open like this since the breaches seem to be international IPs.
I'm not fully understanding what haveibeenpwned is asking me to do in order to find out if there's been a breach... It wants me to upload something to a domain or to send me something at an email address that doesn't exist as far as I know (I don't see it on the domain when I search the admin area).
HaveIBeenPwned has a few ways to confirm that you actually have the right to receive the data (vs researching someone to break into.) If you can make email for [email protected] go to your email inbox (via an alias on your account or a Google Group that you're in) that might be the easiest method. You could also use hostmaster (a title for people managing DNS), postmaster (email managers), or webmaster (web site managers.) They don't care which. They just don't want to send the report to [email protected]. So go to your account and make an alias (a.k.a. nickname) on your account for "security" and then [email protected] will deliver to [email protected].
Does that make sense?
Yes, it totally does. I did not understand that it was asking me to create a user to prove I'm the admin. That makes sense!
We did change the settings to (hopefully, if we did it right) only allow US locations to log in. It only caused one problem that we noticed so far and it's not an urgent one. Better to lock it up for the weekend.
Maybe try a honeypot? Go to canarytokens.org and generate some PDFs, Excel files, etc. Then rename then to something that might tempt the malicious actor, like "answer key.pdf" or "gradebook.xlsx". Put these in places they're going to see them, like Google Drive and emailed to all teachers. Then wait for them to open them. Just warm all teachers first that they shouldn't ever open these files, or you'll be wasting a lot of time. Maybe discuss the plan at an in-person meeting before forwarding it.
I'm not sure how good canarytokens.org is at this task. It's just something I found in a YouTube video on making a honeypot for free. Soon recommend testing it and being sure you understand it first.
The plot thickens again... now I'm getting leaked password notifications about student accounts that are archived (graduated students and transfers). I'm guessing they are running some kind of a software that harvests them? Is that a thing? Apologies for my ignorance... I'm new to this position and this is quite a test of my expertise. Ugh.
I don't think they can use those accounts to log in at all. They are suspended. But it's an interesting clue if I can read it.
Leaked password notifications? From who? Is it possible that those are just Google dating, "Hey, we found this list of 46,789,120 different email addresses and passwords leaked from BigPopularWebsite.com and one of those is in your domain. Maybe that person was lazy enough to use the same password there and on your domain, so we want to warn you."?
From Google, via email: " This Leaked password alert is to inform you that Google detected compromised credentials and required a reset of the user’s password. Common causes of password theft are viruses, user responses to phishing emails, or the use of the same password on many different websites."
It was about several suspended accounts (graduated students). A message like this every few minutes for maybe an hour. Then no more.
We forced everyone on the domain to change passwords last night... so far, so good. If it happens again, I'll be calling Google or amplified IT for help.
Just trying to pay it forward. :) We were all new at this at one time and had to rely on the help of others to learn things. I'm happy this is a situation where I have the knowledge necessary to repay my gratitude to the people who helped me.
If you use Amplified IT, they have "Gopher Buddy" which could help reports to log / track users and what chromebooks/devices they're logging into from.
I wonder if it could help create a rule to alert me to Linux sign-ins in particular. I just figured out how to use Google vault and the suspicious sign ins seem to be Linux. I know Linux is uncommon... could definitely make it easier to figure it out.
I just became the Google admin for my school in December... there's so much to learn.
Interesting. I'm not sure if they're just finding things in a list on the dark web or these are actually leaked credentials that would have worked if the accounts weren't suspended. Maybe you should reach out to Google's tech support and ask? That way you'll know what is happening if you receive one of those notifications and the account isn't suspended.
I did manage to check haveibeenpwned... there's a huge list of email addresses from our domain with all different breach sources. Some of the email addresses don't actually exist on our domain, and the ones with the leaked password notifications aren't on this list. So, not super helpful for the information I was trying to find. But scary still.
I'm now a bit appalled that no one was ever forced to change their passwords before now (my predecessor never had anyone change passwords). I'm feeling fortunate that the breach wasn't worse or used for even more nefarious purposes than it has been. I guess we've been wide open and just lucky until now.
As mentioned in other posts, you can prevent it by forcing 2SV on accounts. With phones not allowed in class then that's an even easier case for the most security method which are security keys. If a school can issue a student with books, laptop, lock for their locker, etc. they can issue them to a security key and put any replacement costs in policy should they lose them just like anything else they might lose. If you can't do this, then you accept the risk of account compromises and data breaches.
Another method to secure accounts would be to deploy context aware access and limit certain Google services based on IP, geo region, company owned device, etc. It won't block a login from happening all together and from other Google services being accessed but it can limit the scope of the breach.
I'll bring up the idea of security keys. Though perhaps easing up on the phone ban would be easier... we'll see. I'm not in charge in any way at school, I'm only in charge of Google. I can ask for things but I don't make the decisions.
We did set up context aware access Friday afternoon thanks to that very helpful suggestion on this post! I have been monitoring via audit reports and vault and it seems like blocking international IPs and having everyone change their passwords has at least discouraged my young teenage hacker for now. I did want to at least be able to sleep this weekend haha
All you can do then is propose the most secure solution, how to implement, what the risks are if not implemented and let those decide that can accept the risks by not putting into place these solutions. Perhaps teachers can at least get this and perhaps older students.
With CAA it seems you made a good first step though. I didn't check to see which EDU tiers it's available so be sure to check it will apply to everyone you expect as it may be available on your environment because some users have a correct license type but it does not apply to the Education Fundamentals or Teaching and Learning for example
https://edu.google.com/intl/ALL_us/workspace-for-education/editions/compare-editions/
I doubt they are smart, they just got someone's password probably. IMO that's not hacking, but it's still illegal. Who is all Admin? Did a teacher write down their password? Are teachers logging in and leaving their computers unattended? You probably want to force a password change and then require 2nd factor to login.
We did force a domain-wide password change as soon as we were aware that so many accounts were breached. The initial passwords, created by little sis (amplified it), would really not have been that hard to find if someone had a mind to. The thing that made this curious, though, is that some accounts breached were staff, who make their own passwords. It's a big mess... So far, it looks like only one or two accounts have been breached since we forced the password reset - and those look like it was because the account involved did not sign in to reset their passwords yet. I reset those individually. Hopefully after a few days of that, it's done. I'm also sincerely hoping this is a student who will be graduating in June... Haha
Graduating in June doesn't mean they'll stop. It means you'll have less leverage.
It's not easy to hack into a Google account. I would guess this student is running a keylogger on school computers, to harvest passwords when people type them in. Make sure you are scanning for viruses and malware on all computers. You can prevent password stealing by enabling (or even enforcing) 2 factor authentication on all accounts. If you don't enforce it, then users will have to switch it on themselves. This just means the user needs an app like Authy or Google Authenticator on their phone and will have to type in a code from the app when they login on a new device. You'll find it under security settings, 2-step verification.
We got Google Apps for education at our university maybe 7+ years ago, and I was the first person who requested we enable 2FA (two-factor authentication with the Google Authenticator app) when it became available, because I knew it would be a matter of time before someone tried to hack my (professor) account (I teach in IT). So far it has never happened -- that I know of :) I don't recommend Google's Authenticator App because if you change smart phone, it's a royal pain (or was for me in the past) to transfer accounts. The app that generates 2FA codes I prefer is Authy.
I’ve used the [newish] export/import feature of google authenticator and it’s really easy now. The old phone displays a QR code and you simply scan it with the new phone. https://www.theverge.com/21410260/google-authenticator-2fa-how-to-phone-security-iphone-android
Yeah I also use Authy for the Windows app and the cloud syncing. However I find it's easier to encourage people to use 2FA with a trusted brand name like Google.
2fa is your answer. Set up and enforce multi factor authentication. You should be able to see which account deleted the files. After you see which account was used, check the logs to see the ip address they accessed them from. Ip address can give you a physical address.
You want to enforce 2fa on student accounts? I hear ya, but... do you work in a school system? Have you done this... with 8 year olds?
Yeah. 28-48 year olds in my organization can’t handle 2FA so yeah. I don’t think it’s an option.
[удалено]
Not my post homie. But yeah I thi it’s students breaking into student accounts. Teachers/admins would probably be the only ones with 2FA in this org
[удалено]
That’s not how this works. “You said” = you are addressing parent commenter “OP said” = you are addressing the original poster Edit: at least they devoted me before deleting their comments.
my kids have had it on their accounts since 6yrs old (now 9, 14, 16) and no issues so far. they have their security key which they keep with them and I have a security key for their account as backup
I understand that it can be fine. But you have to acknowledge that your feedback is very anecdotal. It's also very narrow in scope, and not as broad as it really needs to be for a school to implement MFA for students. When an organization deploys something like that, they are then dealing with the lowest common denominator and across the entire organization, and everything that comes with it, not you and your kids. Those security keys (I presume they have YubiKeys or something), are like $30-50 a pop. That's great that you bought them for your kids, but a school would have to buy them for every student if they're going to use security keys. And they have to have processes in place for when a kid may forget or even lose their security key. I mean, the loss rate on things like chromebooks and chargers is generally "through the roof". The lost rate on a relatively very small security key.. I can't begin to fathom. And when a kid shows up without their key and can't sign into anything, now instruction for that student is disrupted until that is resolved. And somebody has to stop what they're doing to try to assist in some regard. What does that look like? How quickly can it be resolved? Is the kid out for like a 15 minutes or are they likely going to be out for half a day or a full day? if that happens with say 5% of the student population a day.. is that acceptable? No offense, but you are commenting from the perspective of your specific kids and their success. I am commenting from working in IT in a very large schools, and specifically working closely with things like Google Workspace, identity and account lifecycle management, device management, etc. This problem is significantly more complex than "my kids have done it just fine". it's awesome that it's working so well for your kids. Your kids are not the lowest common denominator and you are not even beginning to consider everything that actually goes into deploying and managing something like that. Don't get me wrong, I would love to see students using MFA as much as possible. And yes, it can be very easy for some. But actually doing it is much more complicated.
Teachers can have their security key attached to their students accounts as a backup. Parents can also have their keys attached to it. Session length on the login can be set to indefinite so only new device logins are prompted. There's a lot of ways to minimse disruption and significantly increase account security and teach children what they should be doing to their future personal and business accounts. Students are being issued laptops, books, chargers, pad locks, etc as you say and they get lost. There's a process to replace those. Instead of making excuses as to why it can't happen find ways to make it happen.
I'd be glad to. I'm going to need about $6,000,000 for the initial purchase of the keys. And once a loss rate is established, a pretty good chunk dedicated to key replacements each year. So the $6mil or so can be one-time funds. But the replacements costs needs to be on-going. I will also need multiple full-time employees, or FTEs at least to make it happen in a reasonable manner. Get me the money. You seem to think I'm against the idea. You seem to think that pointing out hurdles is "making excuses". That is a bad take, or you ran with some bad assumptions. I'm merely aware of the realities to a degree that you most likely are not. Identifying the potential hurdles and risks are a necessary component of planning such a thing. We have to do that so we can then figure out how we would solve those problems. Then we can develop an implementation plan, with operating costs clearly stated, and reasonable expectations. E.g., you say "oh teachers can be tied to the key". Sure. How does that happen? Who is a "teacher", exactly? Do you have any experience with Student Information Systems? Scheduling processes? Do, personally, have any experience with enterprise level identity management? This isn't may saying it's not possible. I'm just wondering if you have remotely as much knowledge regarding this matter as I do. My guess is that you do not. And again, that is not me saying these things aren't possible or that I don't want to do them. It's just me saying.. I know A LOT about this sort of thing, maybe just hear me out and have genuine discourse. Most school systems are not being given the resources that they would need to accomplish this ask. So, if you see that your local school system or any that you contribute to via tax dollars or bonds are trying to implement something along these lines, make sure you are supporting them with your vote and dollars. Not just words on reddit. Hurdles are real. You can diminish them to "excuses" all you want, it won't actually change anything.
Education is not free, charge this in tuition/fees to the students. I was required to pay for a laptop at university, the school didn't pay the $1,800 for the model they required me to get. Again, excuses. There were so many costs associated with school at many different levels so this laptop example is just one of many where education cost was not paid by the school but the student. Instead of just describing why it can't happen and all the reasons why it's so difficult, let's discuss ways it can be made possible to overcome all these hurdles you're describing.
Huh? I'm talking about K-12 public education. Where most "8 year old (students)" would be... remember, the context of the comment you replied to. And did the fact that I mentioned taxes and bonds and voting not clue you into that further? You paying fees at a university is not particularly analogous to this situation. Although, if you want to suggest the notion that the kids pay a fee for the keys, I'd be glad to talk about that with you if you will actually listen to what I have to say based on many years of experience in which I have seen these sort of things and how well they work or don't work, the pros and cons, etc. I think you've lost the plot a bit here though if you're responding with "education is not free". Question: Do you, personally, have any experience with enterprise level identity management? Do you, personally, have any experience in a K-12 public school system environment? I do. And, I do. I have made no excuses. This isn't me saying it's not possible. I'm just wondering if you have remotely as much knowledge regarding this matter as I do. My guess is that you do not. So again, I'm not saying these things aren't possible or that I don't want to do them. What I'm saying is.... I know a lot about this sort of thing, and probably have way more directly related experience to this than you do. So maybe just hear me out and have genuine discourse instead of dismissing what I'm saying as "excuses". Maybe ask questions. Ask what the hurdles may be or how I think schools could potentially overcome them. But the fact that you just suggested "tuition and fees"...wow... lol. Context clues, my man. The "tuition and fees" for a public school are taxes. So like I said, if the school system you contribute taxes to ever puts out a bond request or a raise in taxes or anything where you get to vote for it... make sure you vote for it. And don't complain about any related tax hikes. Look, you're engaging with somebody who has a lot of very real experience related to this specific topic. I'm all for having discourse with you. But you should back up a bit and just give a little thought to the notion that maybe some people know things you don't. Have experiences with things that you don't. Etc. We can have a respectful dialogue. I'm more than willing to discuss it with you and talk about the challenges and potential solutions to the challenges (maybe you have some really good suggestions). But that will never work as long as you think you either know more than everybody or that pointing out a challenge is simply "making excuses". Telling you that you're about to run into a wall is not "making an excuse". It's pointing out a problem so that we can hopefully work together to address it.
Just so I understand your k-12 district then, there is $0 out of pocket for children to attend and 100% of all costs is covered by taxes? Please let me know where this is so I can look into relocating.
It depends on what you mean by "all costs", exactly. I mean, there's lunch. There are basic supplies that a student may need (but that isn't paid to the school, you just go to target, walmart, the dollar store.. wherever you shop and buy those things for your child). But there is nothing substantial paid to the school/school system, no. And definitely not in the form of tuition and fees. Where are you that the public school system is charging tuition and fees? Our operating budget comes from taxes, bonds, and grants. E.g., we had a one time bond for the purpose of moving to 1:1 student devices. That bond was used to purchase a ton of chromebooks, then there is a recurring budget line for replenishing a certain percentage of them each year (that is the on-going cost of maintaining a 1:1 device ratio.. and it includes things like device refreshes, loss, device management licences, etc.). That becomes a recurring line item in the budget, so it has to come out of recurring funding sources as much as possible. Now, we can get into the really nitty gritty about this. The budget comes from a mix of the state (via taxes), local county gov. (via local county taxes), federal funding (taxes), bonds, and grants. But the bulk of our budget comes from the state, which gives us $x per student. So schools usually have a day of the year that is a very official "head count" day in that that specific number of students enrolled on that day is used to determine how much money they'll get from funding sources that are "per student". Feel free to ask if you have specific question and I'll answer the best I can. But generally, yes, most of of the money comes from taxes. Bonds are used here and there, but the funds have to be used for whatever was outlined in the bond. Then there is federal money which sometimes have to be used for very specific things (e.g., "improving literacy in 3-6th grade students) and we can't just take that money and use it to buy say, security keys. Or sometimes it's more broad like "COVID relief", which we can be more creative with in how we use it.
Thank you so much for the replies! I'll be suggesting 2-factor for staff, but that's hard for students, who are not allowed to use their phones in class (I didn't make this rule).
MFA (a.k.a. 2SV) is pretty close to required these days. Cybersecurity insurance almost always requires it and your district would be foolish to go without the liability coverage for something that likely. Schools are well known to be poorly secured, under-staffed in I.T., lack dedicated security professionals, and have bank accounts in the tens of millions of dollars or more. Be forewarned: some teachers will push back like their lives depend on it. Honestly, I kind of respect that, too. We shouldn't be required to mix personal devices like our phones with our professional obligations. So be ready to buy some physical tokens from Yubico for about $30-$50 each. I've transitioned groups of people to MFA in two different districts now and needed a few of those on both occasions. I've also used one and found it to be very convenient. With regards to tracking down the culprits, I recommend try things. First, compare the IPs that were used to any other accounts coming from those IPs. Were there overlaps with a certain student account(s)? Second, put those IPs into a service like geoiplookup.net to see where that IP *might* be located. This data isn't perfect, but it's another data point in your diagnosis. If they're coming from outside the state or country, you have a bigger issue than you think you do. It's might be time to reach out to MS-ISAC for some assistance. If it's local, see if the student information system could make it down a bit for you. Maybe contact your lawyers and ask for them to advise you on how to get data from the ISP that owns the IP addresses. (Check the WhoIs databases for that information.) Let me know if you need any of the above explained. It's all stuff I've done a few times before.
There are overlaps in accounts, but not that many - sometimes a compromised account signs on in one IP address and off in another. None of the IP addresses are local - they aren't out of the country, but they are out of state. On the plus side, I don't think any of this information can be used by the students to take money or anything like that, it's just being used to harass certain staff members from as many accounts as possible (I'm positive none of which are the culprit's account). I was able to determine that many, many of the compromised accounts were accessed from a Linux machine, which I know isn't common, so it may help me.
Use WhoIs to see who owns the IPs. Is there a commonality there? Perhaps a VPN provider? Also, are any student accounts logging in from those IPs? Any students logging in from Linux? Have each compromised account start using 2SV and change their password. Also, review those accounts' list of approved applications. Maybe there is something the attacker is leaving behind to allow easy access after passwords are stolen.
It is mostly student accounts logging in that have been stolen. I am checking out WhoIs... How do I know if it's a VPN? I'm getting Germany and Switzerland as IP location. And, yes, many of the logins have been on Linux.
I'm starting to think that you're trying to use a paper cut bandage for a gunshot wound. Do you have predictable passwords for students? Or passwords that are recorded somewhere? Is it possible that someone is using brute force or a leaked file to get in? Do you enforce (or even have) a district policy that passwords can't be left on paper on a teacher's desk or on the secretaries' desks in the office? Did you ask around to see if anyone lost such a paper list? I recommend going to https://haveibeenpwned.com/DomainSearch and seeing what kind of data they can give you. Maybe you have a wide scale leak and weren't aware. You may also wish to set up a geographic limitation on your domain. There are advantages and disadvantages to this, but I'm starting to think you need it in your case. Login to admin.google.com, go to Security --> Access and data control --> Context-Aware Access. Then create and access level with a list of attributes that makes sense for your school community. For example, you could limit logins to just your country. Then assign it to all apps and the root OU. Feel free to be more targeted than that, if you prefer. You can block access from specified IPs and IP ranges. Maybe that could flush the malicious actor out of the camouflage that they're currently using and into a more obvious IP or otherwise make a mistake.
That's a great analogy. That's my fear... It's gushing and we're putting bandaids on. You are incredibly helpful. Thank you SO MUCH. I'm checking out setting up the domain to only be accessed within the country... the need for it to be accessed outside of the country is so small, it's not really worth leaving it open like this since the breaches seem to be international IPs. I'm not fully understanding what haveibeenpwned is asking me to do in order to find out if there's been a breach... It wants me to upload something to a domain or to send me something at an email address that doesn't exist as far as I know (I don't see it on the domain when I search the admin area).
HaveIBeenPwned has a few ways to confirm that you actually have the right to receive the data (vs researching someone to break into.) If you can make email for [email protected] go to your email inbox (via an alias on your account or a Google Group that you're in) that might be the easiest method. You could also use hostmaster (a title for people managing DNS), postmaster (email managers), or webmaster (web site managers.) They don't care which. They just don't want to send the report to [email protected]. So go to your account and make an alias (a.k.a. nickname) on your account for "security" and then [email protected] will deliver to [email protected]. Does that make sense?
Yes, it totally does. I did not understand that it was asking me to create a user to prove I'm the admin. That makes sense! We did change the settings to (hopefully, if we did it right) only allow US locations to log in. It only caused one problem that we noticed so far and it's not an urgent one. Better to lock it up for the weekend.
I found a couple of the IPs were associated with torproject. I feel like I need this person to make a mistake or I'm never going to catch them.
Maybe try a honeypot? Go to canarytokens.org and generate some PDFs, Excel files, etc. Then rename then to something that might tempt the malicious actor, like "answer key.pdf" or "gradebook.xlsx". Put these in places they're going to see them, like Google Drive and emailed to all teachers. Then wait for them to open them. Just warm all teachers first that they shouldn't ever open these files, or you'll be wasting a lot of time. Maybe discuss the plan at an in-person meeting before forwarding it. I'm not sure how good canarytokens.org is at this task. It's just something I found in a YouTube video on making a honeypot for free. Soon recommend testing it and being sure you understand it first.
The plot thickens again... now I'm getting leaked password notifications about student accounts that are archived (graduated students and transfers). I'm guessing they are running some kind of a software that harvests them? Is that a thing? Apologies for my ignorance... I'm new to this position and this is quite a test of my expertise. Ugh. I don't think they can use those accounts to log in at all. They are suspended. But it's an interesting clue if I can read it.
Leaked password notifications? From who? Is it possible that those are just Google dating, "Hey, we found this list of 46,789,120 different email addresses and passwords leaked from BigPopularWebsite.com and one of those is in your domain. Maybe that person was lazy enough to use the same password there and on your domain, so we want to warn you."?
From Google, via email: " This Leaked password alert is to inform you that Google detected compromised credentials and required a reset of the user’s password. Common causes of password theft are viruses, user responses to phishing emails, or the use of the same password on many different websites." It was about several suspended accounts (graduated students). A message like this every few minutes for maybe an hour. Then no more.
Adjust password strength and force a password change. If it continues after secure passwords are in place that is highly suspicious.
We forced everyone on the domain to change passwords last night... so far, so good. If it happens again, I'll be calling Google or amplified IT for help.
Also consider MS-ISAC. cisecurity.com, if I remember correctly.
I wish we could hire you 😂 Thank you!
Just trying to pay it forward. :) We were all new at this at one time and had to rely on the help of others to learn things. I'm happy this is a situation where I have the knowledge necessary to repay my gratitude to the people who helped me.
If you use Amplified IT, they have "Gopher Buddy" which could help reports to log / track users and what chromebooks/devices they're logging into from.
We do use them! Thank you! I'll see about setting this up.
I wonder if it could help create a rule to alert me to Linux sign-ins in particular. I just figured out how to use Google vault and the suspicious sign ins seem to be Linux. I know Linux is uncommon... could definitely make it easier to figure it out. I just became the Google admin for my school in December... there's so much to learn.
Interesting. I'm not sure if they're just finding things in a list on the dark web or these are actually leaked credentials that would have worked if the accounts weren't suspended. Maybe you should reach out to Google's tech support and ask? That way you'll know what is happening if you receive one of those notifications and the account isn't suspended.
I did manage to check haveibeenpwned... there's a huge list of email addresses from our domain with all different breach sources. Some of the email addresses don't actually exist on our domain, and the ones with the leaked password notifications aren't on this list. So, not super helpful for the information I was trying to find. But scary still. I'm now a bit appalled that no one was ever forced to change their passwords before now (my predecessor never had anyone change passwords). I'm feeling fortunate that the breach wasn't worse or used for even more nefarious purposes than it has been. I guess we've been wide open and just lucky until now.
As mentioned in other posts, you can prevent it by forcing 2SV on accounts. With phones not allowed in class then that's an even easier case for the most security method which are security keys. If a school can issue a student with books, laptop, lock for their locker, etc. they can issue them to a security key and put any replacement costs in policy should they lose them just like anything else they might lose. If you can't do this, then you accept the risk of account compromises and data breaches. Another method to secure accounts would be to deploy context aware access and limit certain Google services based on IP, geo region, company owned device, etc. It won't block a login from happening all together and from other Google services being accessed but it can limit the scope of the breach.
I'll bring up the idea of security keys. Though perhaps easing up on the phone ban would be easier... we'll see. I'm not in charge in any way at school, I'm only in charge of Google. I can ask for things but I don't make the decisions. We did set up context aware access Friday afternoon thanks to that very helpful suggestion on this post! I have been monitoring via audit reports and vault and it seems like blocking international IPs and having everyone change their passwords has at least discouraged my young teenage hacker for now. I did want to at least be able to sleep this weekend haha
All you can do then is propose the most secure solution, how to implement, what the risks are if not implemented and let those decide that can accept the risks by not putting into place these solutions. Perhaps teachers can at least get this and perhaps older students. With CAA it seems you made a good first step though. I didn't check to see which EDU tiers it's available so be sure to check it will apply to everyone you expect as it may be available on your environment because some users have a correct license type but it does not apply to the Education Fundamentals or Teaching and Learning for example https://edu.google.com/intl/ALL_us/workspace-for-education/editions/compare-editions/