Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

anthonypants posted:

I thought it might've been this guy



Oh God ilkhan....

Adbot
ADBOT LOVES YOU

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

go3 posted:

SHE NEEDS TO FILE A TICKET FIRST

You need to open a JIRA ticket before I can address the issue, mom.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
"If it was important I would've told it to you personally" - ex-Sys Admin who documented nothing.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

PBS posted:

No one uses a wiki or knowledge base?

We use Wiki combined with SVN.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

psydude posted:

F5s are far and away the most headache inducing piece of equipment, though. The funny thing is that it has nothing to do with the device and everything to do with the organization: in non-DevOps shops, the developers and infrastructure team constantly punt issues to one another every time something breaks because nobody wants to take responsibility for the box. DevOps organizations are the only ones that make them work without mountains of stupidity.

We had an F5 load balancer installed in place of our CISCO device that did round robin, and the F5 kept screwing up the load balancing on our Resin cluster, the F5 team insisted it was not due to their appliance, despite the fact that we could solve the issue in QA via a round-robin only load balance.

Finally got them to try it, and ta-da, issue disappeared.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Goon Matchmaker posted:

I'm on the unix team at work. We take care of the F5s for some reason that only management can decipher. Recently some firewalls were life-cycled in our secondary datacenter. This caused a strange problem where traffic to nodes in our primary datacenter from the secondary data center's f5s is being intermittently dropped. The other way around is fine... Network security refuses to investigate the issue as there's "no possible way it could be the firewalls." Networking won't touch it either (yes networking is separate from network security). Meanwhile I get to deal with app owners whose apps are experiencing bizarre intermittent issues. I have logs that clearly show the problem did not exist with the old firewalls and as soon as they put the new firewalls in place, blammo. Security still won't do anything but blame everyone else. It's maddening.

There was an issue they were tracking where F5s would randomly accept traffic and then never forward said traffic to the endpoint.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Goon Matchmaker posted:

F5? I don't think that's in play here. 2nd DC to 1st DC = dropped traffic. 1st DC to 2nd DC = Fine. It started immediately after they flipped everything over to the new firewalls. I think the F5's are triggering some kind of port scan protection on the new firewalls but I'm not sure.

No, no, that is exactly what happens. DC on one side can communicate properly, but anything coming BACK through the F5, the traffic gets accepted but its fails to pass it onto the internal network.

There may be a configuration that was tied to the MAC/IP tagging for the long gone firewalls, and you may have to rebuild the F5 configuration to resolve this issue. If you have vetted your Firewall rules, I'd look further at the F5 itself.

https://support.f5.com/kb/en-us/solutions/public/12000/700/sol12703.html

CommieGIR fucked around with this message at 22:49 on Jan 1, 2016

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Goon Matchmaker posted:

Given we're on 11.5.3 I don't think that applies to us.

:shrug: Don't be afraid to look into the F5 more anyways.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

RyuHimora posted:

I feel like taking a job at Best Buy or some other consumer-based computer shop rather than stop working with the skillset I love.

Don't do this. Especially not Best Buy. You really do NOT want to work at Best Buy.

Have you considered getting into Systems Administration? Datacenters are always looking for infrastructure/server janitors.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

RyuHimora posted:

My understanding is that you can't buy servers/business network gear without a repair agreement, which still leaves me buying things from a predetermined list without needing any knowledge other than the budget, and shipping it off if one little thing goes wrong. I guess the problem is I want to be very hands-on with the computers, but that's just not a skill outside of consumer gaming rigs anymore :(

Plenty of whitebox servers still getting built dude.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
Seriously, you are going to want to get into Server Infrastructure/Sys Admin stuff if you really want to get near hardware. Unfortunately, outside of some small Whitebox manufacturers, there just isn't much of a call for a hardware only guy anymore.

I've been doing Systems Admin for 10 years, and I started the same way, and I STILL love doing hardware, but its so much more fun, like Docjowles said, to mess with systems at the operations level.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Swink posted:

I'm a sysadmin with a static workload environment. My day to day job is to look after the Windows servers. Definitely pets, not cattle.

What skills would I need to hone to land a more interesting (to me) admin role in a DevOps environment? I'm aware of all the big name technologies like Chef, DSC, Vagrant etc and the big cloud vendors, the "configuration as code" idea. But I have very little practical experience with them, as zero experience in in production.

If I'm sitting across from a hiring manager, what do I need to do to convince him that I'm hireable?

Bonus question: What would I ask them about their workplace to ensure I wasn't walking into a "Devops" role that was really just another CJ role?

Learn the basics about the Software lifecycle (QA - > UAT - > Production) and understand the basic ideas behind those tools and I think they'll be willing to help you with the rest. Most DevOps positions are just looking for someone WILLING to handle these sort of things. Get some experience with Git/SVN, understand how to check out/check in code, learn about things like JIRA and Jenkins build cycles.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Swink posted:

So I need to know how to ship code, how much do I need to know about writing code? If I learn Ruby and walk into a shop that uses C#, will that be a deal-breaker?

No, not really, DevOps is more about managing the code lifecycle, and most places are just happy to have someone who is willing to learn a new code on their team, especially if they hired you do fr DevOps, not to actually be a developer.

Its like Sys Admin'ing, but instead of just handling boxes, you are helping the development team deploy/manage their code as it goes from concept to production, but there is a lot of managing boxes inbetween and helping make sure a code roll out goes flawlessly on the operations side.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Sickening posted:

Sysadmin jobs are being taken by the cloud.

Someone Sysadmins those boxes.

But really, its not entirely true. Yes, cloud is a great go to for small, maybe some medium sized companies, but most medium to large companies still require having either a datacenter they rent or own under their own control.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

I'm irony/sarcasm impaired. But you wouldn't believe how many Developers look at me funny and go 'You need Sys Admins for cloud systems?' because the freaking hype over managed services and buzzwords has been taken so seriously.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

CLAM DOWN posted:

uh isn't the cloud just magically everywhere and it runs my email?

You just click your heels together and say 'There's no place like someone else's datacenter'

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Vulture Culture posted:

If you're competent with your tech strategy you need two sysadmins and not twenty

I run five digit numbers of server instances with one part-time guy on top of myself

That's still two sys admins. For most Enterprise level companies, that's plenty.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

GreenNight posted:

Did I mention his normal network account is Enterprise Admin?

"Software slows done my computer, so I don't even want an Operating System"

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Tab8715 posted:

What's the "right" way to setup a windows domain? As far as I am aware, there isn't any built in JIT Access.

Are you recommending everyone should just have normal user accounts and only use admin accounts when needed?

Its preferable. Forcing Support users to need to escalate in order to do administrative changes helps assure that they are less likely to make mistaken changes.

That, and ensuring that they have a need to access various security roles, i.e. only giving them roles that they actually need access to daily so that if they need to make changes outside their normal boundaries they have to escalate.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Tab8715 posted:

No GUI is Server Core, correct? Does that mean absolutely everything must be done via Powershell or is there a way a connecting guest may still use RDP?

Yes, its Server Core. You can still manage the server via Server Manager, but most things will have to be done via Powershell. I want to say that RDP DOES work, but it still gives you a powershell Window unless the server is also running a Terminal Server, in which case you can get a desktop but will act as a separate machine.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

mayodreams posted:

That goes for systems teams too. I don't like giving the networking guys Domain Admin just because it's easier. If they need to manage DNS and DHCP, give them admin access for those roles.

To follow up from yesterday, all of our Service Desk people are Domain Admins, and because there is a trust with our production domains.... guess what!? I told my boss who is the CISO and he was extremely unhappy as the SD reports to him too.
I am adding delegated roles in AD to the proper OU and creating an admin server post haste.

Yeah, the idea is find someone in the support team that you trust and give them domain admin, and require other support users to get admin changes done through him in order to facilitate some sort of official privilege escalation.

That or train someone

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
So our entire network is down and we have no failover network connection to back up our point to point.

And we're a Software Development Consulting firm :smithicide:

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
I'm doing consulting for an IT Operations shop right now. Its a small company growing into a middle sized company, but they are stuck in the mom n' pop small shop mindset. They are primarily a Windows shop with a AIX core system. They have no centralized logging, and I keep pushing them for it, because they are actually filling VM drives with logs, have no proper log rotation method, and are manually dumping logs by hand.

Its a mess. I'm handling getting network monitoring going for them using a ManageEngine product. Not my favorite, but they are sold on it. But they have so many issues...and now they are planning a datacenter move in less than 6 months.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

MC Fruit Stripe posted:

They're going through what my team went through over the last few years. We exploded in growth from a mom and pop, to mid sized, to enterprise, all within a 5 year period. Go easy on them - they WANT to grow, and they realize a lot of their processes are garbage, but a lot of them were Good Enough for the time.

In my own case, there were 3 of us doing what a team of 50 now does. 3 people can't do the work of 50 people, so there was a lot of jack of all trades, put a bandaid on it, close enough is good enough going on. Now it's a whole new story. Much better. Just takes time to grow into it - not because they/we think/thought that our way was the best, it's just that you don't want to break anything that IS working while you change all the processes.

e: An example from my own growth would be patching. Our application doesn't self heal. You have to take it down and bring it up in a very specific way. It's tedious af, as the kids would say. It's tedious with 3 people, it's tedious with 50. It's tempting for a consultant to kick the door down as though he's the first person ever to hear about a patching tool, "listen children as I tell you that you can simply patch every server at once and schedule a reboot", but in our very specific use case, we need to pump the brakes because we have to take the application into account.

The biggest issue I'm running into, and I have 10 years of doing consulting with IT Operations groups and Datacenter groups, is that their processes are not documented, they have no Standards and Practices, nothing is documented. Even their infrastructure is nothing more than a drawining on a whiteboard, the only hard infrasctructure documentation they have is more than 4 years old now.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

MC Fruit Stripe posted:

Yep yep, that's another thing about it - 3 people who do the work that 50 people should do, don't have time for your silly rear end documenting. :)

The problem is they have been hiring and have expanded their headcount a lot, but are too busy putting out fires that shouldn't be fires because they don't put the things in place like Logstash or Splunk and Monitoring that will help pre-empt the fires.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
Virus on a desktop? Contain it and clean it. Its fine. I usually wipe my desktops if they get infected, but its fine.

Virus on a Server? :sherman:

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

SEKCobra posted:

I don't really see the risk. I mean I run a segregated infrastructure with no AV on most servers anyway. But somehow playing up the fact that ANY virus touched a server is ridiculous.
First of all you have to look at what kind of virus it was. And what kind of detection. My file server has AV so I know when a user has a virus and copies it on there if they have an outdated AV or whatever. Just because the AV detected a virus in a user folder doesn't mean I'm going into panic mode and revert to backups. The OP gave no details about what kind of virus or how it was detected. If his boss is even willing to take the liability, you really don't have to give much of a gently caress when it's possibly a harmless virus on a sheltered spooler at best.

In this day and age, with breaches left and right, if the infection is on the server itself and its not a file server, its better safe than sorry to just restore the system to a prior point.

I still want to know how it got infected in the first place, is it a file share?

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Sefal posted:

It is a file server. And we got hit with Derusbi malware.

http://www.novetta.com/wp-content/uploads/2014/11/Derusbi.pdf

Interesting reading.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Tab8715 posted:

People that don't use Outlook rules are weird.

Thoughts?

Company I'm consulting for hired a new Systems Engineer, and the guy didn't know how to setup rules, then proceeded to complain about the amount of email he got. I showed him how to sort and use rules, he still doesn't use them.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
I really don't like sharepoint.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
Powershell is awesome, Xen XCP is awesome for virtualization and a lot of lessons learned there carry over to VMWare, learn some basic bash/shell and get familiar with Linux file structure and commands.

Setup a Xen Hypervisor, and you can build up a virtual network with devices like pfsense, etc.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
Tech interviews where they expect you to know complex solutions off hand are just as bad as homework. But yeah, the homework assumes you have tons of time available to give back to a job you might not even get.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
"The cloud will save us"

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug
smart enough to have two ISPs, not smart enough to actually setup proper network redundancy.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

i am a moron posted:

Eh depending on the firewalls and switches involved ISP failovers are a bitch. Reasonable for a 20 person office imo

There are much better ways to do this, like Enterprise rated firewalls with dual WAN. Its absolutely insane for a 20 person office.

i am a moron posted:

You ever seen an entire office connected to the internet by ViyOS installed on a repurposed desktop computer? I’ve seen some poo poo. The firewalls capable of doing that are probably still more expensive than using a couple pfsenses or something you janked together, and to their credit they’re actually trying to do something so A for effort B for execution

pfsense supports dual/redundant want. You don't need two of them. You need better hardware.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

i am a moron posted:

I mean I don’t really know or care about pfsenses and I’d stub my toe than work on some small business firewall poo poo. But they won’t have a lot of technical skills (even with some MSP) and if it works for them, who cares

pfsense is like the epitome of what a good small business firewall is, because you just need 2-3 NICs and some old enterprise hardware and it'll run for years without intervention.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Internet Explorer posted:

Autologon, use an account that doesn't have a lot of rights and doesn't use a password you reuse anywhere else, because it is very insecure.

And also get UPSes.

This. Auto-logon is not a great idea, so use a very limited service account that has a strong password. If you are a larger company with (lol) good AD, see if you can implement LAPS and have the password change ever 15-30 days.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

CloFan posted:

We run pfSense on campus and a few small remote sites, it's a great platform. Recently put in a 10Gbps TNSR router that I'm very pleased with as well that cost 1/10th of Juniper/Cisco

I use pfsense for all my homelab routing/segmentation. Its a workhorse. Been trying out OPNSense which has the same feel, but supposed to have some more Security features.

https://teklager.se/en/pfsense-vs-opnsense/#:~:text=OPNsense%20has%20slightly%20better%20security,fewer%20releases%20and%20ZFS%20support.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Internet Explorer posted:

One of the other new guys on the team I'm on just dipped after likes month with no notice for "personal reasons." I don't blame them. It really is amazing how bad some places are run. I feel like I give up a bit of my soul every time I go to a new place and get things in order, just to repeat the process a few years later. poo poo is exhausting and I don't blame anyone for jumping ship.

My current company is that way: Fortune 150, when I was just consulting for them it felt like they really did want to change. Now that I'm full time, its the same old uphill battle to get actual things done, instead they just keep purchasing useless and overpriced tools.

They really need to get back to basics.

Adbot
ADBOT LOVES YOU

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

GreenNight posted:

They wont change until they get cryptolocked.

And even then still won't change.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply