|
GreenNight posted:Were the DC's patched recently? Been reading about authentication issues with the latest Windows patch. Yes. but I had pre-verified that we would not be affected by the KRB issues. This is definitely some sort of weird race condition or some other stupid poo poo whereby Windows had it set to "only use THESE credentials for delegation" and ones those had changed there was no visible way to update them.
|
#
?
Nov 29, 2022 20:00
|
|
|
|
| # ? May 28, 2024 12:40 |
|
|
Number19 posted:Yes. but I had pre-verified that we would not be affected by the KRB issues. This is definitely some sort of weird race condition or some other stupid poo poo whereby Windows had it set to "only use THESE credentials for delegation" and ones those had changed there was no visible way to update them. Did you try to remove the patches and see if that fixed it? While you pre-verified, it could have been an undocumented issue.
|
|
#
?
Nov 29, 2022 20:03
|
|
|
If it were that, it would have blown up two weeks ago and also would have been more wide spread. I had this issue happen months ago, well before the patch. I’m pretty sure that’s not the root cause here
|
|
#
?
Nov 29, 2022 20:12
|
|
|
Are your DFS namespace servers all healthy? Are the DNS zones for the namespace as they should be?
|
|
#
?
Nov 29, 2022 20:27
|
|
|
Thanks Ants posted:Are your DFS namespace servers all healthy? Are the DNS zones for the namespace as they should be? The namespace servers are healthy and DNS works right. This really seems like a case where Windows somehow got configured (or configured itself) for the computer account to access DFS via the FQDN using one explicit set of credentials and then had the computer account get stuck when those credentials were no longer valid. The very first fault I had with this was because the primary user (who's account would have been who's credentials the computer account got stuck like this) was disabled because they left the company. I can definitely fix it now at least but it really should not get stuck like this. I get why it's stuck since the computer account has no way to ask the user to re-authenticate themselves but it really should throw those credentials out and start over if they are obviously broken. We did a password reset wave not too long ago so I expect I will see many more failures as more cached credentials get "stuck" like this. e: also of note, this problem persisted through a domain leave and rejoin to a new computer account object, which seems extra absurd to me
|
|
#
?
Nov 29, 2022 20:36
|
|
|
To put the capstone on this: the cause here was domain credentials for the FQDN being saved in the computer account's credential manager. Once the password was no longer valid, the computer account was trying to use these cached credentials to access the DFS namespace and failing. By using psexec and deleting that credential, the computer is unstuck and back to working properly. How did those get set for only a few computers? Truly a mystery and not one I plan on trying to figure out right now. It really seems like Windows should not allow this to happen and should definitely fall back to not using those credentials when they don't work.
|
|
#
?
Nov 29, 2022 22:23
|
|
|
It sounds like someone's tried to script something in the past and targeted the wrong context
|
|
#
?
Nov 29, 2022 22:29
|
|
|
Thanks Ants posted:It sounds like someone's tried to script something in the past and targeted the wrong context That's about all I can guess. The list of affected users has no common thread to them, so I can't figure out what this diverse group would all have done. I think that part of the RCA is going to remain a mystery. If it pops up again I will be able to dig in by tracking recent changes.
|
|
#
?
Nov 29, 2022 22:33
|
|
|
Question on CAL license move (if I even can). A clinic I'm administering has a pair of servers with VMs on top of those servers to deliver their Terminal Services. The servers are server 2016, the VMs are 2012. The clinic is doing a software upgrade that doesn't support 2012 (of course). Instead of creating new VMs I'm planning on just moving the Terminal Services down to the 2016 boxes. I've been looking at this this and want to make sure I'm not missing anything, is it that simple to move the TS CALs to 2016? Basically my plan is as follows: 1- Back up server 1 2 - Install TS on 2016 3 - Move CALS from VM1 to Server1 4 - shut down VM1 5 - make sure TS on Server 1 works as expected. 6 - Repeat for server 2. Once that's done, I'll make sure load balancing is done properly, etc.
|
|
#
?
Dec 9, 2022 16:32
|
|
|
Make sure that your RDS CALs are at or above Server 2016. If you have Software Assurance, you can reissue the CALs as 2016 (or higher). If you don't, you may need to buy new CALs.
|
|
#
?
Dec 9, 2022 16:43
|
|
|
GreenNight posted:Were the DC's patched recently? Been reading about authentication issues with the latest Windows patch. We put the November patches in Test last week and auth broke on 9 of the 10 DC's.
|
|
#
?
Dec 9, 2022 21:21
|
|
|
the spyder posted:We put the November patches in Test last week and auth broke on 9 of the 10 DC's. Holy poo poo. You have test DC's? Awesome.
|
|
#
?
Dec 9, 2022 21:34
|
|
|
tangent! We have a child domain (I know) that cannot enroll with the parent's CA with access denied. Done certutil - setreg Policy\EditFlags +EDITF_ENABLELDAPREFERRALS and https://learn.microsoft.com/en-us/troubleshoot/windows-server/windows-security/cannot-publish-certificate-trusted-domain#scenario-1 I need some LDAPs with this child.
|
|
#
?
Dec 9, 2022 21:39
|
|
|
Potato Salad posted:Make sure that your RDS CALs are at or above Server 2016. If you have Software Assurance, you can reissue the CALs as 2016 (or higher). If you don't, you may need to buy new CALs. yay I can make them spend money. I seriously doubt the previous tech had then get Software Assurance, and the CALs are definitely not 2016. Thanks for the help.
|
|
#
?
Dec 10, 2022 01:27
|
|
|
the spyder posted:We put the November patches in Test last week and auth broke on 9 of the 10 DC's. They released an out of band replacement that is not syncing via WSUS or SCCM. I had to install manually, but it worked in place of the released patch. Only needed on your AD servers if the monthly broke kerberos as it did in your case. https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/november-2022-out-of-band-update-released-take-action/ba-p/3680144
|
|
#
?
Dec 10, 2022 01:34
|
|
|
Gothmog1065 posted:yay I can make them spend money. I seriously doubt the previous tech had then get Software Assurance, and the CALs are definitely not 2016. You can buy RDS CALs now through CSP rather than as an upfront cost
|
|
#
?
Dec 10, 2022 01:36
|
|
|
One of our gaps in DR is having a comprehensive AD recovery plan. We are geo-redundant for DCs/aad connect/okta ad agents so I’m not too concerned about losing our on-prem datacenter - seize the roles in the DCs in azure and move on with life. However, the scenarios that I mainly see the need to account for are unrecoverable AD corruption that is propagated to all DCs, or a security incident where the domain is compromised. While unlikely, the consequences are too high not to be ready to be able to recover from zero. The technical parts I’m not concerned about and the MS documentation is good for a forest recovery, but are there any other published guides by nist/etc that provide best practices for a recovery plan? The main discussion point right now is around backup retention - how long do you keep backup copies for DCs? Right now we run a 30 day daily VM backup window as a blanket policy and are looking to add a separate one for the DCs specifically. My thought is that we definitely need longer retention but I’m not sure what that should be. I can see a scenario where an attacker could be in the network and sit around on a compromised DC beyond the current 30 day window.
|
|
#
?
Dec 13, 2022 17:55
|
|
|
Mandiant once mandated a client of mine set up a recovery forest after getting positively owned by Empire Powershell attacks. I don’t remember what it was tied to specifically, but I think it has some connection to NIST standards (their website says ADRF is referenced here: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1800-25.pdf) because that’s predominantly what were working off at that point.
|
|
|
#
?
Dec 13, 2022 19:26
|
|
|
Came across an interesting conundrum while working on getting these servers updated. I have the two RDS servers with the VM's built on top with Hyper-V. The physical servers are running Server 2016, VMs are 2012 (soon to be 2016). The physical servers have matching hardware: Intel E5540 32 GB RAM (24 GB Allocated to the VMs) Intel Broadcomm 82576 Dual NIC - Hyper-V Virtual Switch on it. The basic issue is that when the VMs are running, the server is almost impossible to work on. it's laggy and starts dropping packets, reconnecting, and other things. I can ping the server from another internal machine on the same switch, and you can see it just stop responding. When I turn off the VM, this seems to completely stop. Is there some common issue, or is this, for some reason, expected behavior?
|
|
#
?
Dec 17, 2022 18:01
|
|
|
what's the storage? If you open Performance Monitor in the host ahead of time, you'll get to watch what's going on too.
|
|
#
?
Dec 18, 2022 14:34
|
|
|
My initial thought would be memory, depending on what's running on the host (security software etc). Drop the guest memory a bit and see if the problem changes at all? Also, is your vswitch configured properly?
|
|
#
?
Dec 19, 2022 07:11
|
|
|
Potato Salad posted:what's the storage? This was done prior to me and made me cry. Pair of 2TB platters in RAID 1. Luckily there's not a lot of read/write happening on the disks themselves for the most part until we shut down/start VMs. Silly Newbie posted:My initial thought would be memory, depending on what's running on the host (security software etc). Drop the guest memory a bit and see if the problem changes at all? I'll turn the memory down and see if that helps some. As for the vswitch, probably not. One of those, "It's been functioning don't touch" that was a leftover. Looking at them, considering one is a "new virtual switch" and the other is an "External switch" and the doesn't show the physical connection in the network and sharing summary -- I'd say 'no'. The second machine is MUCH worse in terms of connectivity, so I'll work on getting them to match and see if that alleviates the issues any.
|
|
#
?
Dec 19, 2022 17:58
|
|
|
What's the disk queue depth when stuff is grinding to a halt? I suspect your issue is that you're just asking way too much from a very small quantity of spinning rust.
|
|
#
?
Dec 19, 2022 18:30
|
|
|
Yikes, those disks. I'm not a big Hyper-V user, but double check to make sure there's no a setting that is forcing those VMs to use virtual memory on the host instead of real memory, because that would explain a lot. VMware would call it Memory Limit, but no idea in Hyper-V.
|
|
#
?
Dec 19, 2022 18:50
|
|
|
I am not happy with the disks, but that was done about 6 months before I took over, and the previous tech absolutely screwed this office over, and it took them a while to pay off the loan, and they're leery of getting new equipment. Either way, when in use and not doing something specific (like downloading updates, etc), the queue length stays <1. When doing downloads or something of that nature, it can jump to 10, but I wasn't seeing the severity of the packets dropping was nowhere near as bad as previous. I'm also not seeing any real correlation between the queue length and packets dropping. I'll try dropping the ram size down to 16 which should be plenty for both sides. They're using 10+ year old hardware, with 4 physical servers that were all set up in raid 1 (Probably the previous guy's version of a "backup"). I've already taken them off of their USB external drives as a "backup" and put it on a NAS server (the DB is also backed up offsite), replaced their DB (high intensity) platters with higher end server SSDs, but at this point, I think I'm just going to have to talk them into refreshing their servers. They're still using an ancient 2008 server for their fax server, which I have to talk their EMR into moving off of 
|
|
#
?
Dec 19, 2022 21:47
|
|
|
There is also a small shop thread here that might be more helpful for your scenario. Folks around here might ask you for a trigger warning on that situation. https://forums.somethingawful.com/showthread.php?threadid=3723832
|
|
#
?
Dec 19, 2022 21:51
|
|
|
Been a while since the Belkin USB hub with two sticks photo appeared in here
|
|
#
?
Dec 19, 2022 21:59
|
|
|
Gothmog1065 posted:2008 server ... EMR  Nothing you're saying is particularly unique--a lot of businesses run like this. One alternative to refreshing the server hardware would be to stuff some of their workload into the  , do you have any idea if there's any appetite for that?
Potato Salad fucked around with this message at 22:17 on Dec 19, 2022 |
|
#
?
Dec 19, 2022 22:14
|
|
|
Internet Explorer posted:There is also a small shop thread here that might be more helpful for your scenario. Folks around here might ask you for a trigger warning on that situation. I'll pop over there with other questions, thank you.
|
|
#
?
Dec 20, 2022 18:07
|
|
|
Please make sure to update us as well, this poo poo is fascinating. I'll throw my lot in with disk after memory given new information. I've never seen disk r/w times cause a problem like that, but I can totally see it with the hardware involved.
|
|
#
?
Dec 22, 2022 07:40
|
|
|
Azure AD cross tenant sync is in preview, which looks like it helps out quite a lot https://learn.microsoft.com/en-gb/azure/active-directory/multi-tenant-organizations/cross-tenant-synchronization-overview
|
|
#
?
Jan 25, 2023 10:58
|
|
|
i don't want to be asked to deal with m&a like this  I kinda like the hard line status quo that forces migrations to be one big shift rather than an old AD-style cross domain trust then little moves that are expected to be done invisibly and with no downtime
|
|
#
?
Jan 25, 2023 14:18
|
|
|
We’ve only done two acquisitions so far but that is absolutely the way to go about it. Get everyone up and running with their new login in Okta so they can get into Workday to start, then have a cut weekend where email/onedrive is migrated over, mx records switched, old aliases reapplied, etc, and Monday morning they all log in to their brand new autopilot provisioned laptop. We did a small divestiture of ~40 people a couple of weeks ago where they went to a new domain entirely and it was as easy as them giving me the olddomain/newdomain email mappings, apply some forwarding transport rules, and copy the mailboxes/onedrive to their new tenant with bittitan. They kept the laptops and so their new IT guy had to reimage them over the weekend, but that sure as hell wasn’t our problem!
|
|
#
?
Jan 25, 2023 15:57
|
|
|
So here is a issue I have been roped into. There are some computers that are joined to a particular AD OU that makes the computer into a "kiosk" type of workstation with group policy by running a script that configures auto logon with a generic domain account that has rights to update its password with the domain and store it in the registry. So whenever you turn it on or reboot it to the user it just logs into a desktop. Now they all seem to randomly fall off the domain once a week or with the trust relationship error message. My question is what I can I dig out of the computer to either show the users responsible for it or the person who wrote the script that either one or the other needs to change ie don't turn on all five of these machines each day at the same time because the generic account that is used across each pc will freak out or something is wrong with the script because it only happens to these machines joined to this particular OU? I am fairly familiar with joining/unjoining machines but I'm not sure if there is a log someplace in the event viewer that would be helpful or if I need to make sure the system never tries to do a system restore.
|
|
#
?
Feb 1, 2023 04:04
|
|
|
Hint: It's not the number of logins for each generic account. I've got 12-22 uses per generic account in some of the labs I support. We only get a handful of machines a year out of 1500-1600 falling off the domain without being offline for 3+ months. Of course, AD at enterprise scale is in perpetual closed beta since MS just can't, simply cannot, do regression testing for AD code in test environments that look anything like our production environments. So poo poo happens, it just hasn't for me in your situation.
|
|
#
?
Feb 1, 2023 06:52
|
|
|
Budget Dracula posted:So here is a issue I have been roped into. There are some computers that are joined to a particular AD OU that makes the computer into a "kiosk" type of workstation with group policy by running a script that configures auto logon with a generic domain account that has rights to update its password with the domain and store it in the registry. So whenever you turn it on or reboot it to the user it just logs into a desktop. Now they all seem to randomly fall off the domain once a week or with the trust relationship error message. My question is what I can I dig out of the computer to either show the users responsible for it or the person who wrote the script that either one or the other needs to change ie don't turn on all five of these machines each day at the same time because the generic account that is used across each pc will freak out or something is wrong with the script because it only happens to these machines joined to this particular OU? I am fairly familiar with joining/unjoining machines but I'm not sure if there is a log someplace in the event viewer that would be helpful or if I need to make sure the system never tries to do a system restore. I was going to write up a bunch of stuff, but honestly this webpage covers most of it. The computer account password is getting changed somehow and is breaking the trust relationship between the computer and the domain. Computer accounts have passwords just like user accounts, but usually all that stuff is handled behind the scenes. Most of the times I've ran into this problem is because computers were not being imaged properly (using not sysprep'd images) https://theitbros.com/fix-trust-relationship-failed-without-domain-rejoining/ I haven't done client compute stuff in over 5 years now so I may be out of the loop a bit, but thats what it was back in the day. I'm going to guess that all of these machines were imaged off the same base image (not sysprep'd), and 1 of them is doing its thing and changing the computer account password and causing the rest of them to have the trust fail. Also best practice would be to create a generic kiosk login for each one of those kiosk computers. KioskUser1-5 for KioskComputer 1-5. I'd also lock the poo poo down out of those accounts so they could only login to the 1 computer, but that's just me.
|
|
#
?
Feb 1, 2023 18:10
|
|
|
Sweet it’s helpful to see all the fixes on one page finally and I was able to query all the machines in my area to compare the busted ones against working ones. Hopefully I can come up with something other than please turn these machines on one way a time and wait til windows boots before going to the next one.
|
|
#
?
Feb 1, 2023 23:42
|
|
|
I'm working on a very rough proof of concept on the family of services now bundled together as "Azure Service Management," and I fear that I'm too early an adopter of this bundled concept because an Azure Automation PM asked me what Azure Server Management Services was
|
|
#
?
Feb 2, 2023 22:55
|
|
|
Wait Azure Automation still has PMs?
|
|
#
?
Feb 3, 2023 03:44
|
|
|
|
| # ? May 28, 2024 12:40 |
|
|
Well, I read the last three pages of the thread and even though I consider myself tech-savvy (well, that's a lie - all my boomer and Gen X co-workers think I'm some sort of computer wizard, but I know the truth) most of all of that went over my head. That's a lot of words to say this maaaaaaay not be the right thread for this sort of question; feel free to point me in the right direction! So, my day job for the last almost-decade has been Corporate Spreadsheet Jockey for Fortune 100 type places - heavy use of Teams, Sharepoint, etc. in very tightly controlled environments. I've recently picked up a side gig doing Spreadsheet Jockey work for my buddy's small business (remotely) and we're struggling to find a screenshare app that works well for us. In theory we could both pay the extra few bucks a month to upgrade our Microsoft accounts to include Teams, but right now he's mostly in a Google Workflow environment (please kill me, every one of these loving Google "apps" is a twisted mockery of life itself) and we've been using Google Meet and we're both about over it. I have a 4k monitor, he has a 1440p Ultrawide. When I share my screen I can either share a single window, or I can share my entire desktop - so my options are either: • Keep unsharing and resharing as I flip between apps • Crank my scaling to the moon so that text on my screen is so big that it can be seen FROM the moon Both options kinda suck; the first one is annoying especially when I'm just doing something quick in one app before going back to another, and the second is terrible because that means any screenshots I take for documentation are loving enormous and I have to gently caress around scaling each one of them down. Ideally I'd either like to be able to: • Only share a region of my screen and move anything and everything I'm sharing into and out of that region as needed • Share my entire screen and have him be able to zoom into a section of the screen and I do the same thing as above I'm not sure anything like that exists. Teams may be the closest as I THINK there's a "zoom in" functionality on the viewer side although I only accessed it once by accident and haven't found it again so maybe I imagined it? Other software I'm aware of: • Discord: would require a monthly fee to be usable at 4k, and I don't think it has either functionality anyway • Teamviewer: I've only ever used the free version and to my knowledge it doesn't have either functionality • Slack: I use it very lightly but never for voice/screenshare so the capabilities in this space are largely unknown to me In theory I could probably hook up a third monitor ONLY for sharing but it won't be in a very useful position, but I guess at the moment that's my worst-case scenario. Any thoughts on other software (preferably free or inexpensive) that would fit the bill for the functionality I'm looking for? I wouldn't have thought this was a difficult ask but maybe that just shows what I know 
|
|
#
?
Feb 5, 2023 06:38
|
|
