Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
mayodreams
Jul 4, 2003


Hello darkness,
my old friend
I know I mostly lament in this thread, so I thought I would share something more positive.

I have been tasked with building out our new Cisco UCS environment and I have exactly zero other Cisco experience. We had an engineer help us to set up our Fabric Interconnects and a basic template and service profile. We have had it in production for a few months, but really haven't been working with the management other than me setting up a Windows template and profile on my own. Since we have another UCS system going in shortly, I decided to try and get UCS Central working, so I deployed the OVA and got the basic setup finished. I then tackled LDAP auth and got it working late yesterday afternoon and just dropped the mic and went home. I always find LDAP integration with AD from 3rd party stuff painful, and this was particularly so. But it feels so good when it works.
:boom:

Adbot
ADBOT LOVES YOU

Super-NintendoUser
Jan 16, 2004

COWABUNGERDER COMPADRES
Soiled Meat

AreWeDrunkYet posted:

I get this for a retail environment or something, but wouldn't Friday night be exactly when you want to schedule changes for if your users are on a normal Monday-Friday schedule? If something goes wrong, you have 2 days to roll back or troubleshoot.

I used to work for a hedge fund and we'd schedule network maintenance for Friday afternoon. Problem was all the traders would play big halo matches after work, so I couldn't take the internet and network down until they were finished. It eventually became a big issue since I"d be sitting there until 8 or 9pm waiting for them finish. I had a wife and a new baby home at the time so I'd rather be there than playing halo, and plus if I get on a later local train it'd take me two hours to get home. I talked to their director about it, and he had me setup an isolated network an internet line, for the to play on.

Now, I work in an industry related to publication, and a lot of my clients are newspapers, so weekend maintenance is off limits due to the Sunday paper being the biggest seller. All my maintenance tends to be done on Monday night, which is fantastic because it doesn't screw up my weekend if it goes wrong.

mattfl
Aug 27, 2004

When I was doing RIS/PACS support we had a client call up on a Friday around 4pm saying his system was down and docs can't read images.

Come to find out he had decided a Friday afternoon was the perfect time to install windows updates on his servers. I walked over to my boss to let her know what was going on and forgot that I hadn't muted my headset and said something along the lines of, what kind of idiot system admin installs untested windows patches on a Friday afternoon?! My boss had to calm the guy down and I got a talking to on Monday for that. But really, who installs lathes on a Friday afternoon?!

Proteus Jones
Feb 28, 2013



We typically don't do changes from Friday through Sunday.

We have a lab and pre-prod environments to ensure things go smoothly. If things break, or if the change is not progressing according to schedule, we roll-back the change. Issues and problems are replicated in the labs or pre-prod for troubleshooting.

Generally, you should always have a tested back-out plan and avoid troubleshooting in a production environment. Of course there are exceptions for certain kinds of changes, but I've found having a formal change process with reviews and actual deployment testing in pre-prod make life so much easier.

Super-NintendoUser
Jan 16, 2004

COWABUNGERDER COMPADRES
Soiled Meat

flosofl posted:

Generally, you should always have a tested back-out plan and avoid troubleshooting in a production environment. Of course there are exceptions for certain kinds of changes, but I've found having a formal change process with reviews and actual deployment testing in pre-prod make life so much easier.

One thing I love about my current job is this. Every customer has a dev and qa (some have a uat) environment so every upgrade and change is tested in three environments before going to production. This causes a funny issue when there's an update that comes out during this cycle, we have to either start over or push an old version to prod.

CLAM DOWN
Feb 13, 2007




AreWeDrunkYet posted:

I get this for a retail environment or something, but wouldn't Friday night be exactly when you want to schedule changes for if your users are on a normal Monday-Friday schedule? If something goes wrong, you have 2 days to roll back or troubleshoot.

gently caress that, I'm going to enjoy a social life on Friday nights and make whatever changes I have to during business hours.

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer

AreWeDrunkYet posted:

I get this for a retail environment or something, but wouldn't Friday night be exactly when you want to schedule changes for if your users are on a normal Monday-Friday schedule? If something goes wrong, you have 2 days to roll back or troubleshoot.
Vendor support is generally unavailable or crappy on a weekend.

Proud Christian Mom
Dec 20, 2006
READING COMPREHENSION IS HARD
For 24/7 clients regular updates go Monday through Thursday night because if something really unexpected happens with third party stuff we have time to revert or at least punt a few hours until support is available. gently caress trying to find 'that one guy' on Sunday afternoon.

jaegerx
Sep 10, 2012

Maybe this post will get me on your ignore list!


CLAM DOWN posted:

gently caress that, I'm going to enjoy a social life on Friday nights and make whatever changes I have to during business hours.

Friends can wait. Servers, storage and networking cannot.

JHVH-1
Jun 28, 2002
The thing I don't like about Friday night deployments besides having to work on a loving Friday night is that you wake up Monday morning and everyone is complaining and asking why XYZ isn't working right. Not the way I want to start my week.

Ahdinko
Oct 27, 2007

WHAT A LOVELY DAY

AreWeDrunkYet posted:

I get this for a retail environment or something, but wouldn't Friday night be exactly when you want to schedule changes for if your users are on a normal Monday-Friday schedule? If something goes wrong, you have 2 days to roll back or troubleshoot.

I have learnt this from my years in IT. Changes on Sunday, or out of hours in the weekdays.

Put yourself in this completely hypothetical situtation that totally did not happen to me:

When it gets to Friday at 5:30pm, and instead of you skipping home to go down the pub, you're staying till 8PM to sort out that new firewall. But no problem because you'll miss the traffic and commuters and be in the pub for 9 so its still ok, right?
Until it doesn't work, and you're putting the old firewall back in at 2am because you're sick of looking at the new firewall not working and cant figure it out and you want to go the gently caress home.
Oh and the tube stops running at like 1am so you have to get a cab from Central London to 50 miles away in the Home Counties (costs about £200)
And then you go back in on saturday morning to sort out the issues you created when you put in the old firewall and didn't cable it right because you were sleepy as hell.

I was young and naive once. Now I do my changes on a day where I'm going to be in on the next day anyway. This way when it all goes tits up, the next morning you're sitting at your desk ready to fix it anyway, instead of wasting your weekend in the office. Also if you do happen to be up until the silly hours fixing a thing, you can email your manager at 3am and say "gently caress this It works now but I'm going to sleep for a bit, I'll be in at midday" so its like free time off work


mayodreams posted:

I know I mostly lament in this thread, so I thought I would share something more positive.

I have been tasked with building out our new Cisco UCS environment and I have exactly zero other Cisco experience. We had an engineer help us to set up our Fabric Interconnects and a basic template and service profile. We have had it in production for a few months, but really haven't been working with the management other than me setting up a Windows template and profile on my own. Since we have another UCS system going in shortly, I decided to try and get UCS Central working, so I deployed the OVA and got the basic setup finished. I then tackled LDAP auth and got it working late yesterday afternoon and just dropped the mic and went home. I always find LDAP integration with AD from 3rd party stuff painful, and this was particularly so. But it feels so good when it works.
:boom:

Well done, I setup UCS for the first time a few months ago. I've been a Cisco man for 7-8 years (although never in the server world, normally routers/switches) and I really struggled. It seems really cool and powerful but some shits just in the most unintuitive places in UCS manager. I got this shiny UCS director license sitting here though that I really need to get round to putting in.

Ahdinko fucked around with this message at 15:28 on Jul 27, 2015

mayodreams
Jul 4, 2003


Hello darkness,
my old friend
Anyone else having mail flow failures in O365? Both my inbound and outbound messages are failing due to a transport agent rule.
code:
Error Message:  Reason: [{LRT=};{LED=550 4.3.2 QUEUE.TransportAgent; message deleted by transport agent};{FDQN=};{IP=}]
Looks to me that we are no longer authoritative for our own domain. Portal checks for domain health and accepted domains look fine. Awesome way to start the week.

Proteus Jones
Feb 28, 2013



mayodreams posted:

Anyone else having mail flow failures in O365? Both my inbound and outbound messages are failing due to a transport agent rule.
code:
Error Message:  Reason: [{LRT=};{LED=550 4.3.2 QUEUE.TransportAgent; message deleted by transport agent};{FDQN=};{IP=}]
Looks to me that we are no longer authoritative for our own domain. Portal checks for domain health and accepted domains look fine. Awesome way to start the week.

I can't comment on what the mail guys are seeing, but I can confirm that neither Outlook desktop client or the Outlook web client show any new email since around 10:30 AM CDT. I have email that *is* showing up in the my "sent" folder, but nothing has shown up when I sent tests to two of my external accounts. So it looks like the server accepted the messages, but they haven't moved past the server.

I've not received any error messages as a client.

mayodreams
Jul 4, 2003


Hello darkness,
my old friend

flosofl posted:

I can't comment on what the mail guys are seeing, but I can confirm that neither Outlook desktop client or the Outlook web client show any new email since around 10:30 AM CDT. I have email that *is* showing up in the my "sent" folder, but nothing has shown up when I sent tests to two of my external accounts. So it looks like the server accepted the messages, but they haven't moved past the server.

I've not received any error messages as a client.

Yeah that is about the same time as us.

EDIT: MS just issued Incident EX28346

quote:

Current Status: Engineers have determined that the degraded portion of infrastructure is experiencing higher-than-normal CPU usage. The investigation is currently focused on isolating the processes that are consuming excessive resources so that corrective action can be taken.

User Experience: Affected users are intermittently unable to connect to the Exchange Online service when using multiple protocols including Outlook, Outlook Web App (OWA), Exchange ActiveSync (EAS), and Exchange Web Services (EWS). Affected users may also experience delays when sending or receiving email.

Customer Impact: A higher than average number of customers are reporting this issue. Analysis indicates that customers will likely have some users experiencing this issue.

Incident Start Time: Monday, July 27, 2015, at 2:27 PM UTC

Next Update by: Monday, July 27, 2015, at 5:00 PM UTC

mayodreams fucked around with this message at 18:07 on Jul 27, 2015

Super-NintendoUser
Jan 16, 2004

COWABUNGERDER COMPADRES
Soiled Meat
My O365 account is running fine, no issues. I don't manage it though.

Anecdote time.

At my last job, the guy they hired to do IT for some reason decided he wanted to migrate us all the our own in house Exchange server. We had Google Apps, and had just transitioned to O365, so there was no reason to do this. Even stranger was that were a child office controlled by an out of the country main office in Israel, and they handled all the IT. All this guy was supposed to do was deal with peoples laptop and do basic helpdesk support, But instead he sets up a demo Windows AD server, and starts messing around with Exchange, and trying to convince the NJ office managers that we need in house mail.

First off, we didn't control DNS, so there's no way to even do this, and second of all, the Israeli office already had a domain, we just weren't on it. They wanted us to start transitioning our computers to their domain, and he was supposed to be doing that. Instead he configures an Exchange server on his own, and then asks me for help to point out mail at it so we can test it.

I just look at him with :psyduck: and ask him how's he's planning on setting up the MX records, with out access to our DNS or register. He doesn't know what any of that is. I just laugh and walk out of his office.

Then later he asks me for help moving all the PCs in the office to the domain, since he's spent all day trying to get his on the domain. I notice that he's running Windows 7 Home, and can't figure out why here's no add to domain button.

bobmarleysghost
Mar 7, 2006



SIR FAT JONY IVES posted:

Then later he asks me for help moving all the PCs in the office to the domain, since he's spent all day trying to get his on the domain. I notice that he's running Windows 7 Home, and can't figure out why here's no add to domain button.

Who let him bring his own laptop to work?

3 Action Economist
May 22, 2002

Educate. Agitate. Liberate.

bobmarleysghost posted:

Who let him bring his own laptop to work?

He's in IT, he'll do whatever he wants!

rafikki
Mar 8, 2008

I see what you did there. (It's pretty easy, since ducks have a field of vision spanning 340 degrees.)

~SMcD


This should be fun:

http://arstechnica.com/security/2015/07/950-million-android-phones-can-be-hijacked-by-malicious-text-messages/

chocolateTHUNDER
Jul 19, 2008

GIVE ME ALL YOUR FREE AGENTS

ALL OF THEM

Yawn. Another day, another exploit. It'll never end.

Maneki Neko
Oct 27, 2000

What are people using for on prem hosted file transfers with 3rd parties? We've got clients using FTP/SFTP for data exchange with their clients/vendors which is goddamn support nightmare, but they also don't want to move to THE CLOUD, so my standard recommendation of "just use sharefile you idiots" won't seem to work.

I looked at the WSFTP web transfer stuff, but it's stupid expensive for what it is.

Internet Explorer
Jun 1, 2005





Not sure if it meets your requirements, but ShareFile does have the option to use your own storage. Everything else exists in the cloud, but the data itself can be kept on site.

Gothmog1065
May 14, 2009
I'm kind of excited. I'm up to :yotj: from the school I'm at, with a ~30% raise. However, my director wants to talk to me (The new place called for work references), so he might be going to try to give me a raise. It :feelsgood: to be wanted.

Super-NintendoUser
Jan 16, 2004

COWABUNGERDER COMPADRES
Soiled Meat

bobmarleysghost posted:

Who let him bring his own laptop to work?

They only bought us Dell Laptops with Windows Home. For a long time, they didn't have a domain, so it didn't really matter, I guess, but once they started to wan to push us to a domain it because a big issue.

Maneki Neko
Oct 27, 2000

Internet Explorer posted:

Not sure if it meets your requirements, but ShareFile does have the option to use your own storage. Everything else exists in the cloud, but the data itself can be kept on site.

Yeah, I talked to them about that, for a service provider that's geared towards 500+ user installs per tenant, which is probably overkill for this scenario.

Dick Trauma
Nov 30, 2007

God damn it, you've got to be kind.
We have Sharefile and we also have Box. I haven't been able to figure out why we have both.

mayodreams
Jul 4, 2003


Hello darkness,
my old friend
So Microsoft needs to stop shutting down Office 365 with lovely patches.

The damage today was no mail in or out from 9:30am till around 1pm. Messages from that period took about 3-4 hours to come in after that, so we got a flood towards the end of the day. hosed meetings the worst.

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer
so +1 for on prem email i guess. Including scheduled downtime, I think we've had about 2 hours of email being unavailable during the past year.

Thanks Ants
May 21, 2004

#essereFerrari


If ultimate reliability is your thing then I think two datacenters in diverse locations, multiple different connectivity providers, Exchange clusters run by a team of MCSEs and the ability to have maintenance windows when it suits the business is always going to win out over Exchange Online. But that's all hella expensive.

Moving email to the cloud is a business decision, if you wouldn't otherwise need a huge team of staff and all the infrastructure that you have dedicated to email then it probably makes sense to balance the savings with the increased downtime (although up until a month ago I couldn't have told you the last time Office 365 had a major issue). You could even throw something like Mimecast into the mix with the automatic failover of inboxes that it can support and come out ahead.

CloFan
Nov 6, 2004

We're looking at O365 because our Exchange server is 2003 and bossman wants to move to hosted instead of on-prem. The past couple weeks seem spotty for O365. The other alternative is Google, which he's apprehensive about for whatever reason.

Proud Christian Mom
Dec 20, 2006
READING COMPREHENSION IS HARD

adorai posted:

so +1 for on prem email i guess. Including scheduled downtime, I think we've had about 2 hours of email being unavailable during the past year.

Another grand case of businesses shooting themselves in the foot long term to save a little bit now. Glad we're not selling this poo poo platform to people.

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

go3 posted:

Another grand case of businesses shooting themselves in the foot long term to save a little bit now. Glad we're not selling this poo poo platform to people.

How serious an e-mail outage is depends on the business. For some it can mean a serious loss of productivity that can be measured in dollars cost. For others it's mostly an inconvenience. If the money saved is more than the money lost during downtime then it makes perfect sense.

I work from a 30 person company that uses O365. We could run our own email (we're a consulting company, we certainly have the ability) but taking a consultant off of billable work for even a few days a year to deal with email issues would cost more than the subscription costs for the year, and that's not accounting for costs of equipment and software to run it.

On the other hand I worked at a place that threw literally millions into email infrastructure and did a worse job of keeping it up than O365, despite having a dedicated team of six managing it. There's not perfect solution so usually the cheaper easier one wins.

lampey
Mar 27, 2012

CloFan posted:

We're looking at O365 because our Exchange server is 2003 and bossman wants to move to hosted instead of on-prem. The past couple weeks seem spotty for O365. The other alternative is Google, which he's apprehensive about for whatever reason.

We have had nearly zero downtime with apptix.net and appriver both. Office 365 is really cheap for nonprofit and education though.

Gucci Loafers
May 20, 2006

Ask yourself, do you really want to talk to pair of really nice gaudy shoes?


CloFan posted:

The other alternative is Google, which he's apprehensive about for whatever reason.

You'll pry my Outlook from my cold dead hands!

I'm a little perplexed too but for whatever reason, that's the biggest reason that keeps people from *Gmail.

*-From a Small-Medium Business Perspective

Gucci Loafers fucked around with this message at 03:09 on Jul 28, 2015

mayodreams
Jul 4, 2003


Hello darkness,
my old friend
I really can't complain because the amount of work I do to manage O365 is a fraction of what on prem would be. It is just annoying that we had a full year of no issues and everything is making GBS threads the bed right now.

I really hope that the Windows 10 rollout doesn't cause issues Wednesday.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

lampey posted:

We have had nearly zero downtime with apptix.net and appriver both. Office 365 is really cheap for nonprofit and education though.
AppRiver did have a complete outage in 2013 in excess of three consecutive days where the response from support to the impacted accounts was "you should probably look into migrating to another hosting company."

I realize that this isn't a good statistical model for future failure, but they never did really own it.

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer

Thanks Ants posted:

If ultimate reliability is your thing then I think two datacenters in diverse locations, multiple different connectivity providers, Exchange clusters run by a team of MCSEs and the ability to have maintenance windows when it suits the business is always going to win out over Exchange Online. But that's all hella expensive.
We do have two geographically diverse datacenters, but exactly zero certifications. We are very much a 7:00am to 6:00pm business though, so that's nice.

Inspector_666
Oct 7, 2003

benny with the good hair

go3 posted:

Another grand case of businesses shooting themselves in the foot long term to save a little bit now. Glad we're not selling this poo poo platform to people.

Hosted Exchange is a perfectly reasonable move. The features of O365 are pretty nice, too.

MS just can't keep the loving thing up, which is a pretty big problem.

psydude
Apr 1, 2008

I'm going to tell pre-sales that we need to implement an intelligence test for customers before we sell them on any design that utilizes routing beyond a collapsed core with a static default route.

dox
Mar 4, 2006
I think you guys are blowing it out of proportion. We probably have dozens of people here on O365 and yet only two post. Hell, we have 50+ clients on it and only two were affected.

I'm still on the side that it's a quality service with extremely limited issues. What you see here are more people's gripes with it than its success. But maybe I drank the juju sauce.... who knows.

Adbot
ADBOT LOVES YOU

The Fool
Oct 16, 2003


psydude posted:

a collapsed core

I don't know what this means.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply