|
The Fool posted:I don't know what this means. Thank you for providing a baseline.
|
#
?
Jul 28, 2015 05:37
|
|
|
|
| # ? May 24, 2024 20:59 |
|
|
psydude posted:Thank you for providing a baseline. Thank you for being a goon. Given the context, it's obvious you're referring to a sites core router, however, I have never seen the term "compact core" before, and was merely hoping to expand my own knowledge.
|
|
#
?
Jul 28, 2015 05:45
|
|
|
The Fool posted:Thank you for being a goon. Sorry, I'm grumpy and thought you were setting up a joke. A collapsed core is when your distribution and core switching is consolidated into a single logical switch. In other words, all VLANs terminate on a single VSS domain that contains a route (or routes) to a perimeter network, meaning that a single logical device handles all layer 3 functionality for a LAN. This vastly reduces the amount of explicit routing configuration that's necessary, but it's not a one size fits all solution; dynamic routing can be insanely complex to implement and manage, especially if you don't have trained staff on hand, but if you know what you're doing it can be a very good way to improve redundancy and efficiency in your network. psydude fucked around with this message at 05:58 on Jul 28, 2015 |
|
#
?
Jul 28, 2015 05:51
|
|
|
e, fb. I generally work with smallish SaaS data centers, but in my recollection the collapsed core generally refers to using a single (paired) l3 switch that handles a bunch of vlans and some access switches and hopefully not much else. You default, static route into an asa which is hopefully externally managed so they can't possibly gently caress it up, and then the asa terminates an sslvpn which they also can't possibly gently caress up. Basically you want to keep them from touching layer 3 as much as possible because it will eventually turn into a rats nest of static, asymmetric routes. In my experience anyway.
|
|
#
?
Jul 28, 2015 05:52
|
|
|
Reiz posted:Basically you want to keep them from touching layer 3 as much as possible because it will eventually turn into a rats nest of static, asymmetric routes. In my experience anyway. I have one customer right now that doesn't actually have a core, but rather a series of distribution layer switches that are divided into effectively two separate autonomous systems that they then try to make act like one and which are all connected via static routs, with plenty of discontiguous networks thrown in. Every single loving change at the perimeter results in routing loops. Just getting them to use EIGRP for edge connectivity was an uphill battle.
|
|
#
?
Jul 28, 2015 06:00
|
|
|
That's pretty much my current full time job, too, so I feel your pain. I'm working against 8 years of institutionalized misunderstanding of layer 3. No routing protocols, its a Web service so the admin team thinks they know what they're doing but really it's a mess of virtual machines where the VM hosts don't participate in STP, and some poor bastard from before my time was coerced into intentionally implementing asymmetric routing for about 1/2 the network. My first day on the job I checked the logs for one of the vmhost switches and found 2.5 years straight of warnings that all of the VMs on one link were mac flapping between the other link for that host. I guess that's just fine. That's what we do here -- gently caress layer 3 we're still working on layer 2. You can look at the mac table on the core switches and watch it completely change every 20 seconds. Tracing a layer 2 path is something of a probability equation in our "cloud LAN". It's really dangerous to be smart enough to make changes but not smart enough to know what you're doing. E: of course I ran into the classic "directly connected route has lower AD than configured static route and everyone is confused". Really just poor decision making all around -- I'm going to start asking to see configs before I accept employment offers. 12 rats tied together fucked around with this message at 06:16 on Jul 28, 2015 |
|
#
?
Jul 28, 2015 06:14
|
|
|
dox posted:I think you guys are blowing it out of proportion. We probably have dozens of people here on O365 and yet only two post. Hell, we have 50+ clients on it and only two were affected. You won't anyone here making the case that Exchange Online is poor service, it just so happens the topic of "On-Prem vs. Cloud E-mail" has recently become a popular subject.
|
|
#
?
Jul 28, 2015 07:09
|
|
|
dox posted:I think you guys are blowing it out of proportion. We probably have dozens of people here on O365 and yet only two post. Hell, we have 50+ clients on it and only two were affected. We use it as well for our company (paying a consultant to work on exchange internally instead of billing is more expensive than the sub). Haven;t been affected by any of the recent service degradations. Having run exchange for a small company while wearing other hats, I can confirm I have consumed the KoolAid on O365 though (generally for when you're not large enough to have a dedicated "exchange team" and GRC issues don't prevent it). Completely different subject but the Bossman is asking me to gain a basic familiarity with KVM. Anyone have a solid Idiots Guide to Not Looking Like An Idiot When Discussing KVM or some basic tutorials you would recommend to get my feet wet? Zaepho fucked around with this message at 16:17 on Jul 28, 2015 |
|
#
?
Jul 28, 2015 16:11
|
|
|
Zaepho posted:Completely different subject but the Bossman is asking me to gain a basic familiarity with KVM. Anyone have a solid Idiots Guide to Not Looking Like An Idiot When Discussing KVM or some basic tutorials you would recommend to get my feet wet? Rule #1 of Not Looking Like An Idiot When Discussing KVM: Make sure you know whether the discussion is about "Kernel-based Virtual Machines" or "Keyboard, Video and Mouse"
|
|
#
?
Jul 28, 2015 16:25
|
|
|
Dr. Arbitrary posted:Rule #1 of Not Looking Like An Idiot When Discussing KVM: Good point. And to clarify I meant Kernel-Based VMs.
|
|
#
?
Jul 28, 2015 16:33
|
|
|
I thought it was the second one. Good point.
|
|
#
?
Jul 28, 2015 17:04
|
|
|
Zaepho posted:Completely different subject but the Bossman is asking me to gain a basic familiarity with KVM. Anyone have a solid Idiots Guide to Not Looking Like An Idiot When Discussing KVM or some basic tutorials you would recommend to get my feet wet? How much experience do you have with virtualization and what do you wanna know? And what do you want to do with it? KVM isn't a product. It's a kernel module/driver which functions as an accelerator for qemu, nothing more. Anything you wanna build on top of it (ovirt/rhev, openstack, kimchi, rolling your own stuff with libvirt, whatever) is up to you. There are some valid products out there (mostly ovirt/rhev and openstack), but the use case can broadly be seen as "does the same stuff as VMware for less/free" or "build my own cloud" (though VMware has their own openstack stuff now which apparently requires NSX, and you've always been able to use the vsphere driver to back openstack's compute anyway). Those couple of sentences are a decent idiot's guide to "how do I not talk about KVM like it's VMware", but for "how can I make KVM work for me/my company", you'll have to be more specific. Disclosure: I work on projects built on top of KVM
|
|
#
?
Jul 28, 2015 17:17
|
|
|
Maneki Neko posted:What are people using for on prem hosted file transfers with 3rd parties? We've got clients using FTP/SFTP for data exchange with their clients/vendors which is goddamn support nightmare, but they also don't want to move to THE CLOUD, so my standard recommendation of "just use sharefile you idiots" won't seem to work. We're using GlobalScape's EFT product - just the SFTP portion. It's ok to support, logging is decent, but it's a little annoying to manage at times. We've also worked with ShareFile a little, but we never managed to sell it as a replacement for EFT. I think for the business, on prem storage was more desirable than cloud storage.
|
|
#
?
Jul 28, 2015 17:30
|
|
|
evol262 posted:How much experience do you have with virtualization and what do you wanna know? And what do you want to do with it? I've been working at least tangentially with Virtualization for a decade or so (VMware and Hyper-V) and am proficient in Hyper-V and the cloud stack on top of it. From the sounds of it, there's some desire for me to help a hosting customer build out some Windows 2012 R2 templates for an offering they're planning. I'm thinking there's probably not a whole lot of difference in building out templates for use on top of KVM as any other hypervisor but wan't to be familiar enough with the differences to be able to handle/figure out what needs to be done even without being an expert on the toolset. As a new monkey wrench to add into the scenario, the customer has build out a custom tool for managing things.
|
|
#
?
Jul 28, 2015 18:13
|
|
|
Zaepho posted:I've been working at least tangentially with Virtualization for a decade or so (VMware and Hyper-V) and am proficient in Hyper-V and the cloud stack on top of it. Zaepho posted:From the sounds of it, there's some desire for me to help a hosting customer build out some Windows 2012 R2 templates for an offering they're planning. I'm thinking there's probably not a whole lot of difference in building out templates for use on top of KVM as any other hypervisor but wan't to be familiar enough with the differences to be able to handle/figure out what needs to be done even without being an expert on the toolset. As a new monkey wrench to add into the scenario, the customer has build out a custom tool for managing things. This actually doesn't answer or explain anything. So, you want to build out templates. How are you realizing them? There's a huge difference between KVM and any other hypervisor, in that KVM has no tooling. That's what I was getting at in my previous post, and you totally missed it. You need to pick your own tooling and build on top of it. What kind of interface do you want? Traditional virt or cloud? Web management or CLI or a console application (which cannot be on Windows)? What kind of storage do you want to use? How are you managing it? What "things" is their custom tool managing, the guests or hosts? If the hosts, are they comfortable building support for a bunch of libvirt stuff into it?
|
|
#
?
Jul 28, 2015 18:22
|
|
|
evol262 posted:What cloud stack on top of Hyper-V? Windows Azure pack. Not cloud in the "Elastic Compute" sense, but it provides the multi Tenancy, Software Defined Networking and various other pieces to use Hyper-V in a IaaS type of offering. evol262 posted:This actually doesn't answer or explain anything. Most of what I'm being told about the need is being filtered through Microsoft so I'm sure there's quite a bit being lost in translation and a ton being left out. It might be best if I figure out what they're actually doing/needing and then work from that standpoint. What I'm gathering is that KVM is the hypervisor without any tooling whatsoever and that the tooling is build against KVM using libvirt for managing Virtual Machines? I leave that as a question since i'm not certain I'm piecing it together correctly. I'm glad I asked here since I'd rather look like an idiot here than in front of a customer.
|
|
#
?
Jul 28, 2015 18:42
|
|
|
Zaepho posted:Most of what I'm being told about the need is being filtered through Microsoft so I'm sure there's quite a bit being lost in translation and a ton being left out. It might be best if I figure out what they're actually doing/needing and then work from that standpoint. KVM is the hypervisor. It's broadly equivalent to what vmkernel does in VMware, and some levels of the hypervisor layer here (Hyper-V is more conceptually similar to Xen than KVM). Years ago, there was kqemu (this page explains it a bit). KVM is an accelerator for QEMU which uses/used the same interface to qemu, but replaced the accelerated translation with hypercalls, basically. qemu (not KVM) provides all the hardware and bits that a guest sees. qemu is required. Well, not required, since you can call KVM directly if you want to implement your own hardware platform or re-invent APIC, but all of that is way beyond this. qemu-img create can create storage, and qemu can boot from that directly. In practice, almost nobody does. libvirt is a wrapper around a bunch of different bits to conveniently manage networking (by creating virtual networks and starting dnsmasq on them), storage (so you can map iscsi, ceph, gluster, NFS, FC, or whatever storage directly to guests), and some other stuff. It does this by dumping configuration into XML which is used to build an argument string which is passed to qemu, as something like: code:code:So, almost everyone uses libvirt. For small setups, anyway. But libvirt isn't really great for large setups. It's ok. You can manage. And it can do more these days. But it doesn't do things like HA or resilient VMs. Or building from templates. Or configuring quotas easily. Or auto-balancing VMs between hosts. Because it's not really designed as a cluster management solution. You can use Openstack. Openstack manages everything separately, copies prebuilt images over, and fires them up with libvirt through Nova (nova ==openstack-compute), and Nova keeps track of them. But for HA, you still end up with yet another service (well, services -- Heat and Ceilometer) watching to see "did something go wrong? Did that VM stop? Try to restart it." It's still not a nicely built in agent. oVirt/RHEV (same thing, just upstream/downstream) manage everything through libvirt. And a local daemon reports on the state of it to a java app/webpage, as well as taking orders from there (start this VM, stop that VM, change your network config). The app/webpage (I'm just gonna call it the engine, since that's the name of the project) keeps track of HA rules and everything. This is the most similar solution to Hyper-V/VMware, and probably what you actually want. But I wouldn't call it "KVM" either. It's a KVM-based product.
|
|
#
?
Jul 28, 2015 19:32
|
|
|
evol262 posted:Effort Post Thanks for that explanation. That makes things a LOT more clear so hopefully I'll be able to understand what the customer is doing and be able to help them integrate windows guests into their existing environment. My google-fu has lead me to a lot of really confusing information as well that distilled through this explanation becomes far more clear. Still unlikely to know WTF I'm doing with the specific tools, but might be able to ask better questions from them.
|
|
#
?
Jul 28, 2015 20:03
|
|
|
psydude posted:I have one customer right now that doesn't actually have a core, but rather a series of distribution layer switches that are divided into effectively two separate autonomous systems that they then try to make act like one and which are all connected via static routs, with plenty of discontiguous networks thrown in. Every single loving change at the perimeter results in routing loops. Just getting them to use EIGRP for edge connectivity was an uphill battle. 1) 800ish person company 2) mostly branch based workforce My core network consists of two datacenters at our largest campus locations connected with metro ethernet. Each has a pair of layer 2 only cisco nexus 10G switches. These switches, paired with numerous virtual instances of VyOS form our network core. The switches at these campus locations connect back to the nexus 5k switches via vpc links, though there are a few chained switches. Despite that, the design does not qualify as distribution in any way. We have a variety of WAN providers, primarily mutltipoint to multipoint evc networks with a little bit of mpls/vpls. I use OSPF almost exclusively, except for the MPLS providers which require BGP. I have about 5 static routes defined in my network, if you do not count the static routes defined in the VPN devices so they actually use their WAN link to build the VPN tunnel. I rarely have outages or performance issues. It's very simple to diagram and I could teach someone new everything they needed to know about it to get started in about an hour. I expect that we could at least double if not triple in size without any major changes required to the design. The dumbest part of my network is that we use the second octet for the service type and the third octet for the branch, for instance 10.1.0.0/16 is for client PCs and printers, 10.3.0.0/16 is for phones. Makes access lists and firewall rules easier, but complicates routing as I cannot summarize anything. At least I can claim that I inherited that design.
|
|
#
?
Jul 28, 2015 23:26
|
|
|
Anyone working over night? I'm doing an upgrade on a client system at midnight EST. It's not difficult, since my boss did their test systems, and he's super thorough about staging files and getting all the details right. I just have to drop some files in on the inactive cluster node, switch over the cluster to the upgraded node, and then party. Of course it won't be that easy, but we'll see.
|
|
#
?
Jul 29, 2015 04:33
|
|
|
upgrades pretty much always go according to plan around here. lulz.
|
|
#
?
Jul 29, 2015 04:42
|
|
|
adorai posted:upgrades pretty much always go according to plan around here. Nice thing about this is if it goes badly, I just swing back to the other un-upgraded cluster node. But yeah, we'll see. 15 minutes until go time, I guess I can watch Ti5 until then.
|
|
#
?
Jul 29, 2015 04:43
|
|
|
Nerd. Also, I'm half-jealous. All of my overnight stuff, I had to gently caress around on the forums on 2-3g, usually in the middle of Nebraska or Wyoming. Not enough to stream video. The other half isn't jealous at all, though, because I do an 8-5 now. The latest I stayed was 5:30 when we had a company-wide outage of our POS and inventory systems.
|
|
#
?
Jul 29, 2015 04:49
|
|
|
22 Eargesplitten posted:Nerd. I'm working from home, I also get to stay home tomorrow. EDIT: Sorry, not bragging, but I've been in years of a terrible job doing overnight over weekend upgrades, migrations, phone system roll-outs. This is a much needed respite.
|
|
#
?
Jul 29, 2015 04:51
|
|
|
.
Methanar fucked around with this message at 05:33 on Aug 6, 2016 |
|
#
?
Jul 29, 2015 05:08
|
|
|
SIR FAT JONY IVES posted:Anyone working over night? I'm doing an upgrade on a client system at midnight EST. It's not difficult, since my boss did their test systems, and he's super thorough about staging files and getting all the details right. I just have to drop some files in on the inactive cluster node, switch over the cluster to the upgraded node, and then party. Would have qualified as overnight but I got home 3 minutes before midnight. Everything went as planned other than a firmware issue (bad batch of MX80s, apparently) which required a five minute call to Meraki support to resolve. I'm sure I'll wake up to a slew of emails about the internet being down when people get to the office and realize they can't access Facebook or Craigslist from work anymore.
|
|
#
?
Jul 29, 2015 05:16
|
|
|
Sheep posted:Would have qualified as overnight but I got home 3 minutes before midnight. Everything went as planned other than a firmware issue (bad batch of MX80s, apparently) which required a five minute call to Meraki support to resolve. I had a weird thing, so the core process wouldn't start, because the start script has a legacy variable in it, that I have never seen before, and my upgrade docs don't mention removing it. The client worked with my boss on the test system and knew it was a problem, and he fixed it. So that's good. Weird.
|
|
#
?
Jul 29, 2015 05:22
|
|
|
Highlight of my day: no more mysterious Thinkserver tower under a desk running Server 2008 (not even R2) with no failover or backup doing DHCP (and nothing else) that no one knew about until it lost power and nothing could get an address after rebooting! Don't even ask about CALs.
|
|
#
?
Jul 29, 2015 05:28
|
|
|
Welp, the upgrade went well. The only stupid problem is that when upgraded our software from v4 to v5, we changed the way a database object is handled. Basically, we do tag some objects with permissions that specify which users can view that object. In v4 we had a permission that sort of opt-out where you assume everyone gets everything but you specify when you can't, but in v6 we instead assume you don't have permission and you have to opt in. it's more technical than that, but basically if you update an existing database in our world from v4 to v6, like I'm doing tonight, you have to run a script that crawls the database, checks every object to see if it has that permission set, and if it does change the tag to the new style permission. The way this works is you run the script, and it says "checking" where it takes an hour to crawl the DB and figure out how many it needs to change, and then it reports back "need to fix 29384810 objects". The a separate log file just has dots: ........... ...... ........... .. . .......... And the dots don't relate to anything, it's a sort of progress bar, but if you count the dots after it finishes, they don't quite match up to the value reported early, they are off by like 5%. Sometimes it more, sometimes it's less. There's no rhyme or reason for it. Once it's done, the first log just spits out "done" and the second log just stops getting new dots. It's insane. Also it takes hours to run. Hours and hours. I just started it, and this DB is by far one of the largest we manage. Fortunately, it can be run while the system is in production, but if a user tries to access an object that is in limbo, it just doesn't show up, since they sort of implicitly now don't have access to it. If they have specifically defined permissions on it some other way, they can open it. If the script is attempting to lock and modify that object at the same time as someone happens to access it manually? The developer just said "  doesn't seem to be an issue."
|
|
#
?
Jul 29, 2015 06:05
|
|
|
Methanar posted:I want to be able to say I've done this huge list of things over the summer so I can be qualified for a big boy job when I'm finished school. Honestly you can probably just lie and say you actually did it. You've done all of the hard work already and it's super impressive (to me anyway) that you, as a junior something-or-other? understand the concept of domain functional levels, migrations, replication and especially all of the gotchas involved in splitting a server apart. You're pretty much already qualified. I've worked with people in windows sysadmin roles who've been working in IT for 5 years plus who would struggle to even explain what a domain controller is except for some amalgamation of "its important". Maybe the bar is higher in whatever non-US country you are in, but I kind of doubt it. I actually know a couple dudes right now in the US who would hire you on the spot for a Junior position if you even mentioned the phrase 'functional level' in an interview.
|
|
#
?
Jul 29, 2015 15:07
|
|
|
I assume you already have a fallback plan in case your domain upgrade fails. If not, I think this should work and might help allay some fears: Have two or more 2003 DC's. Use VMware to move all but one onto a private network. Those DC's will think that the world ended but will dutifully maintain your old database in case any survivors make through the radioactive hellscape into your DC vault. Meanwhile, you add your 2012R2 servers to the real domain. Perform your migration. Verify functionality. Kill the 2003 DC. Give it a week, then kill the remaining 2003 servers. If things don't work out, take all the screwed up DC's offline. Move the Vault DC's back onto your real network and let them rebuild society.
|
|
#
?
Jul 29, 2015 15:21
|
|
|
Never underestimate the importance of setting MAXDOP on a high core count SQL Server. We migrated to our new DB cluster on Sunday and it's a dual 18 core R630. I had forgotten to set maxdop down from it's default value of 0 (which tells SQL to decide for itself). Well, some of our long running processes were taking less time than they were on the old server, but nothing dramatic. Well, found my oversight yesterday and set MAXDOP to 8. Suddenly a job step that took 1hr 40 minutes to complete yesterday took only 11 minutes to complete today. We simply gave SQL too much rope to hang itself with before and it was going mad scientist crazy about dividing queries up. "Oh, I have 72 logical cores, let's slice this puppy up into 72 parallel tasks!" The cost of all that parallelization became greater than the cost of running the query itself and it spent more time waiting on itself than it did running the workload.
|
|
#
?
Jul 29, 2015 15:32
|
|
|
Dr. Arbitrary posted:Use VMware to move all but one onto a private network. Those DC's will think that the world ended but will dutifully maintain your old database in case any survivors make through the radioactive hellscape into your DC vault. No no no, after you verify functionality you begin the Vault-Tec testing procedures on the VMs.
|
|
#
?
Jul 29, 2015 15:33
|
|
|
Inspector_666 posted:No no no, after you verify functionality you begin the Vault-Tec testing procedures on the VMs. Whip up a Powershell script that just continually adds randomly generated GPOs and then after a few months try to add a new computer to the domain.
|
|
#
?
Jul 29, 2015 15:38
|
|
|
SIR FAT JONY IVES posted:Welp, the upgrade went well. The only stupid problem is that when upgraded our software from v4 to v5, we changed the way a database object is handled. Basically, we do tag some objects with permissions that specify which users can view that object. In v4 we had a permission that sort of opt-out where you assume everyone gets everything but you specify when you can't, but in v6 we instead assume you don't have permission and you have to opt in. it's more technical than that, but basically if you update an existing database in our world from v4 to v6, like I'm doing tonight, you have to run a script that crawls the database, checks every object to see if it has that permission set, and if it does change the tag to the new style permission. The way this works is you run the script, and it says "checking" where it takes an hour to crawl the DB and figure out how many it needs to change, and then it reports back "need to fix 29384810 objects". The a separate log file just has dots: Update on this, that no one cares about, I launched the DB fix script, and it's sitting at calculating objects. It's been sitting here for about 10 hours now. I've never seen it take longer than one hour, but this database is 10x larger than any I've fixed before. I called my engineering lead, and says to just wait it out. Oh boy, nothing like watching a process that gives no feed back and hoping it's running ok.
|
|
#
?
Jul 29, 2015 16:51
|
|
|
SIR FAT JONY IVES posted:Update on this, that no one cares about, I launched the DB fix script, and it's sitting at calculating objects. It's been sitting here for about 10 hours now. I've never seen it take longer than one hour, but this database is 10x larger than any I've fixed before. I called my engineering lead, and says to just wait it out. Oh boy, nothing like watching a process that gives no feed back and hoping it's running ok. I turned on verbose logging on all of our domain computers just so that I could quit having users come by and bitch that windows was taking forever to load. At least this way they can see what's going on in the background and it seems to make them a bit more patient.
|
|
#
?
Jul 29, 2015 18:39
|
|
|
BaseballPCHiker posted:I turned on verbose logging on all of our domain computers just so that I could quit having users come by and bitch that windows was taking forever to load. At least this way they can see what's going on in the background and it seems to make them a bit more patient. Yeah, that'd be nice, but this script is like a one-off custom binary our guys developed to fix one weird issue with our software on a sort of edge case, so I don't think they'll go back and fix he logging.
|
|
#
?
Jul 29, 2015 18:41
|
|
|
It appears no one has ever run the cleanup tool on this WSUS server. 
|
|
#
?
Jul 29, 2015 18:57
|
|
|
My 150 gig WSUS partition is basically full and I ran that tool and yay cleaned up 3 gigs... Of course I run it monthly but drat, need more bits for storage.
|
|
#
?
Jul 29, 2015 18:59
|
|
|
|
| # ? May 24, 2024 20:59 |
|
|
I was trying to run a backup on a server and ended up discovering a badmail folder with 700,000 tiny files in it
|
|
#
?
Jul 29, 2015 19:44
|
|
