|
mayodreams posted:NTP settings is a huge pet peeve of mine because of an awful experience I had at an MSP. I feel you. My last company had a clusterfuck for NTP in a business where precise timing mattered. It's not like NTP is a new technology either.
|
#
?
Aug 17, 2016 22:12
|
|
|
|
| # ? May 22, 2024 03:31 |
|
|
Trading?
|
|
#
?
Aug 18, 2016 01:17
|
|
|
I am curious too. Even in banking, 'precise' timing isn't that important. It has to be close, but 30s of drift isn't going to hurt anything.
|
|
#
?
Aug 18, 2016 01:33
|
|
|
High frequency trading is absolutely critically dependent on timing.
|
|
#
?
Aug 18, 2016 01:45
|
|
|
Potato Salad posted:High frequency trading is absolutely critically dependent on timing. And location. Relativistic statistical arbitrage quote:I. INTRODUCTION
|
|
#
?
Aug 18, 2016 01:58
|
|
|
I once couldn't get a guy on Spiceworks to understand why, while analysts and portfolio managers can work in a remote office, trading systems themselves needed to be near the exchange itself as a bar of entry to HF trading. His management was riding him for not providing an uplink to the exchange that worked faster than the speed of light would permit for the remote office they wanted to move their trading infrastructure to  What a company was doing with some fool who couldn't see that, hey, the speed of light was his limit....there are high-6-figure-salaried experts on this kind of network architecture for a reason.
|
|
#
?
Aug 18, 2016 02:29
|
|
|
Before new exchanges implemented rules (or was it the SEC...can't remember) on placing enough fiber in your uplink to impose a minimum latency to the exchange, traders were successful only when they had literally the closest office.
|
|
#
?
Aug 18, 2016 02:31
|
|
|
I'll share my NTP story that I use as an example of what not to do for interviews. We had a number of small banks as clients, and this particular one was a newer addition that we were adding some new servers for as a play to get more business. So I was sent out to the main branch to update the firmware of the ESXi hosts we had put into production like a month earlier. You might be thinking, 'Hey mayodreams, why would you do that if they just went into production?', and you would be right. I got to the site around noon, and I told the lady overseeing my access to the room it should take about 60-90 minutes for me to update the servers and things should be good to go. There were 3 servers, so I start moving VMs around so I could power one down and start the process. First server goes great, and I'm shuffling VMs to work on the second one. Then something happens and I'm locked out of the admin server I was using. I try the VMware console and it wont authenticate with my AD creds, and since we didn't believe in documentation, I can't find a local admin credential to use. I am starting to get worried. My boss and his boss (the director) were literally upstairs negotiating a 7 figure contract with this bank, and things are starting to go sideways during the day. Then I realize the problem: NTP was not configured on the last host I moved the domain controller to, and both DCs are like 5 hours off the time of the other workstations and servers. Panic is setting in. How long before workstations start locking because the time is off? I could have had a major impact at this client in the middle of the day. I am furiously calling and ping/emailing my colleagues to find the local admin creds. I finally get them, but not before the lady comes back and asks what is taking so long. I am trying to keep my cool, which I do, and I tell her that some of the patches are taking a little longer, but I am working on it. I get the creds to both the DCs and login to update the time once I fixed NTP on ESXi and things are good. My boss comes down after their pitch and tells me not worry and freak out over this. I am astonished. A couple of days later during the weekly call, I say that as a result of the issue I had, we REALLY need to get checklists for builds so things like this don't get missed an we cause issues for our clients. I then get chewed out for 45 minutes by my boss with the rest of the team on the call. Two days later I was fired on day 90 of my 90 day probationary period because 'we are professionals, so we don't need checklists' and 'documentation is pointless because it is out of date as soon as it is written'. And that ended the worst experience of my professional career.
|
|
#
?
Aug 18, 2016 02:50
|
|
|
Potato Salad posted:Before new exchanges implemented rules (or was it the SEC...can't remember) on placing enough fiber in your uplink to impose a minimum latency to the exchange, traders were successful only when they had literally the closest office. That was IEX, who put 38 miles of fiber (on spools, in a box, mounted in a rack) between them and the entry point for traders. They're in Weehawken, and their longest trip length to another exchange in New York is 320 microseconds, so they added 350 microseconds of fiber to prevent people from taking advantage of pricing differences between exchanges. Flash Boys is a bit sensationalist, but it's a good read. edit: in other exchanges it's about the closest racks in the data center, not offices.
|
|
#
?
Aug 18, 2016 03:04
|
|
|
Casino Gaming. All sorts of data is collected, and the events are expected to occur in a certain order. If you have two systems that are supposed to be talking to each other and they're synchronized to different time sources and start to drift, you end up with nonsensical orders of events, where someone cashes out a ticket, walks to an ATM, and tries to cash it out before the time associated with the ticket. I guess when I say precise, I don't mean like 1/1000th of a second, but when stuff starts to drift and is never corrected, you start to see some weird stuff.
|
|
#
?
Aug 18, 2016 03:24
|
|
|
Precise timekeeping can also be very critical for anything used in an experimental research capacity. Or real time voice. Or SCADA systems.
|
|
#
?
Aug 18, 2016 03:29
|
|
|
mayodreams posted:Two days later I was fired on day 90 of my 90 day probationary period because 'we are professionals, so we don't need checklists' and 'documentation is pointless because it is out of date as soon as it is written'. > professionals > documentation is pointless 
|
|
#
?
Aug 18, 2016 04:40
|
|
|
Dr. Arbitrary posted:I guess when I say precise, I don't mean like 1/1000th of a second, but when stuff starts to drift and is never corrected, you start to see some weird stuff.
|
|
#
?
Aug 18, 2016 04:59
|
|
|
mayodreams posted:I'll share my NTP story that I use as an example of what not to do for interviews. You basically pointed out a major leadership failure so he threw you under the bus. It sounds like its a good thing though because that sounds like a terrible place to work!
|
|
#
?
Aug 18, 2016 05:50
|
|
|
1000101 posted:You basically pointed out a major leadership failure so he threw you under the bus. It sounds like its a good thing though because that sounds like a terrible place to work! The irony is that they hired me to 'come in and clean things up' because they were losing clients. Turns out THEY were the problem! A director, a senior engineer, a manager, and me were all hired in about 6 weeks. The manager and other engineer were gone after a year and the director after 2. The core idiots and manager are still there bitching about how they can't find 'talented' people. So yeah, I learned a lot about what to ask in interviews and what to look for in lovely organizations.
|
|
#
?
Aug 18, 2016 14:10
|
|
|
adorai posted:So that's where you need a definition. to me, precise time is off by less than a second, whereas good enough to 99.9999% of the population is 30s off. I'm sure I'm not the only one who finds it somewhat satisfying when you have a room full of devices being tested and watch all their clocks tick over simultaneously. Whatever level of accuracy that requires, where events that are supposed to be simultaneous actually appear to be to a human observer, that's what I'd call precise. I assume that's somewhere in to the hundredths or maybe thousandths of a second range. Agreed that 30 seconds is more than good enough for most people, since that means at least the clocks will all be on the same minute most of the time.
|
|
#
?
Aug 18, 2016 16:30
|
|
|
Hey VM thread, I'm hoping you can help point me in a direction on where to start looking. I don't administer the updates or anything to our environment, and figuring out exactly what happened can be a challenge because of that. But hopefully someone can help anyway! About a year and a half ago we got a new host to use as a virtual data center in VMWare. We scoped it out to run about 100-200 VMs depending on size. Each VM has ~20GB disk space, 2gb RAM and 1 CPU core. We then loan this environment out to different teams so they can run testing. One of the common use cases we have is that people want 100 VMs with snapshots so they can run a test; revert; re-run the test. This worked great for about a year. Then, two weeks ago we get patched. ESXi 6 is put on our two hosts and now when we try to reboot 100 VMs they start timing out and failing. Previously it would just take a really long time (we'd do it before we left and come in the next morning and they'd all be rebooted). The IT staff working on this don't want to say they don't believe us that this used to work, but they don't. They're all new and are essentially saying that the failed state is the expected result because the ESX hypervisor will not reserve any CPU for management. So if you VMs exhaust CPU, other machines that are waiting can eventually timeout and fail. Anyone have any suggestions for where to look to even begin troubleshooting this? We did some iometer stuff and we're pegging storage, so IT is telling us there's nothing we can do and there's nothing VMWare will do if we're pegging anything. Help? edit: We rolled back to 5.5 and rolled back some of the firmware upgrades on the host. Things run better, but not where they were prior to patching. Winkle-Daddy fucked around with this message at 21:18 on Aug 18, 2016 |
|
#
?
Aug 18, 2016 21:14
|
|
|
Specs for the hosts? What storage is this running on?
|
|
#
?
Aug 18, 2016 21:20
|
|
|
Potato Salad posted:Specs for the hosts? What storage is this running on? It's some Dell server. It has ~400GB RAM and 36 cores. It's using (I know, I know) local storage. The thing is, this worked fine for >1 year. So even though we'd trample the hell out of storage throughput and stomp all over CPU, I could reboot 100-200 VMs over the course of a few hours by rebooting all of them. There's about 5.5TB of storage. Now if I try to reboot >10 VMs then they stop responding.
|
|
#
?
Aug 18, 2016 21:26
|
|
|
If you're the only tenants and it's a shitshow anyway just write a delayed loop and start that before you leave for the night.
|
|
#
?
Aug 18, 2016 21:35
|
|
|
Dr. Arbitrary posted:Casino Gaming. All sorts of data is collected, and the events are expected to occur in a certain order. I had a class on a real time app that used Xen and the instructor showed use that if the time on the host OS was behind the guest, If you pinged the host from the guest, the response would be delayed by the difference. So if the host said it was 10:01 and the guest was 10:00, the pings would wait a minute to respond. What was even weirder, if the host was ahead of the guest, you would start to get ping responses from pings that hadn't even been sent yet.
|
|
#
?
Aug 18, 2016 21:41
|
|
|
SubjectVerbObject posted:What was even weirder, if the host was ahead of the guest, you would start to get ping responses from pings that hadn't even been sent yet. My brain spontaneously twisted like five degrees to the left reading this  (Think you got your example backwards though)
|
|
#
?
Aug 18, 2016 21:47
|
|
|
evil_bunnY posted:If you're the only tenants and it's a shitshow anyway just write a delayed loop and start that before you leave for the night. That might be a work around, but would require changes to all of the automation we've worked on over the last year when this wasn't a relevant requirement. We've also got a case opened with VMWare (after spending some time arguing with people, got IT to open one). I was really just hoping that maybe someone else had seen something similar and might have some ideas for what to look at that would likely be update related. Edit: The two specific failures we see is that VMWare will see the .vmdk as corrupted or that storage is gone, and it will not longer try to boot the VM. It used to not give a poo poo and would happily sit there spinning until they all came up. Give me back my boot storm 
Winkle-Daddy fucked around with this message at 22:18 on Aug 18, 2016 |
|
#
?
Aug 18, 2016 21:56
|
|
|
adorai posted:So that's where you need a definition. to me, precise time is off by less than a second, whereas good enough to 99.9999% of the population is 30s off. When trying to troubleshoot multi-system issues, being able to accurately compare logs and packet traces and the like and be sure of the order of events is pretty useful. And that generally means sub-second accuracy.
|
|
#
?
Aug 18, 2016 22:43
|
|
|
If you have the cash, buy enough 2TB Samsung 850 PRO SSDs to handle your storage issues. 100vms on magnetic disk? What are the VM OSes? I can't imagine trying to run that kind of demand on a single spinning disk. This is coming off as incredibly seat-of-pants hobo dev.
|
|
#
?
Aug 18, 2016 23:41
|
|
|
The two vmdk errors you're seeing happen when I try to do too much with lovely storage and elements of the storage stack start timing out or silently and gracelessly crashing. 100vms per single 4tb archival spinning hard drive is way off the edge of the map.
|
|
#
?
Aug 18, 2016 23:44
|
|
|
There's a guy in another thread talking about some developers being, to be frank, uneducated but argumentative with their expectations for a barebones esxi setup and I can't help but wonder. Maybe the support ticket with vmware will find some bug or misconfiguration (that's your fault if you were the guys who did the updates then rollbacks), but I can already kinda feel the sigh of the guy who jumps on the phone with your IT team tomorrow hearing that, yep, its another shop with devs who think virtualization is magic and haven't sat down and worked out what 100vms booting simultaneously on a single high-capacity drive or small array of drives on a cheap-rear end controller means just in terms of seek time alone. "But it worked before!" Potato Salad fucked around with this message at 23:55 on Aug 18, 2016 |
|
#
?
Aug 18, 2016 23:49
|
|
|
I'm coming off harsher than I mean to. Boot storms are not inconveniences, they are problems.
|
|
#
?
Aug 18, 2016 23:59
|
|
|
You are sounding kind of like a tier 1 phone support rep right now, yeah. "Don't believe the customer, they always lie." And why you think we're running on a single spinning disk, I don't know; it's a RAID array. I'm also not sure why you think it's somehow a devs fault when an environment that worked one way for more than a year suddenly behaves differently during patch validation while taking identical steps...steps taken at least every 3 weeks... 
|
|
#
?
Aug 19, 2016 04:24
|
|
|
Like, I feel sorry for the guys who have to deal with supporting this too. That's why I came here to see if anyone had noticed any different behavior when taxing systems. Like, we can adjust to the different behavior; but to then be like "it's not different, you're just misremembering" is surreal.
|
|
#
?
Aug 19, 2016 04:26
|
|
|
Whoah, vcenter 6 appliance linked mode owns. Single pane of glass to production and dr! Now to wait for the compellent install in two weeks and it's migration time. 
|
|
#
?
Aug 19, 2016 12:29
|
|
|
Winkle-Daddy posted:And why you think we're running on a single spinning disk, I don't know; it's a RAID array. Do you happen to have the model of the raid controller? Budget and even medium-range servers tend to ship with the lowest or next-to-lowest controller available for that generation unless you ask for something better during purchasing. The H710 that most (in my experience) sales reps include in quotes for gen 8 poweredge systems is weaker in a parity array than some high-end enterprise single disks in the real world. Additionally, just because this is a raid controller doesn't mean that seek time is eliminated as an issue. Striping can increase straight sequential reads, sure, but OS booting rarely is sequential. It's random as hell, and striping really doesn't help you when a disk still has to move from sector to sector physically. What striping/mirroring scheme are you using, and what's the storage system on your guests so we can figure out influences like dedupe. Is the raid controller in write through or write back mode? Storage is complex. What are the OSes of the guests, and what build of esxi 5.5 did you roll back to? Same as before? If I am coming off like a tier 1 call center guy right now, it's because you may not have had the personal experience of seeing how deep the virt storage stack rabbit hole can go. Details are going to be important when you're maxing the system out for hours on end and expecting the storage stack not to collapse and start crashing. Potato Salad fucked around with this message at 14:26 on Aug 19, 2016 |
|
#
?
Aug 19, 2016 14:05
|
|
|
The bit about "it's not working like it was before" is invalid in a high-demand situation with respect to the capabilities of the underlying hardware. Are there pending operations that the OSes of the guests are trying to run at boot, for example? Is the guest trying to run any number of health checks on boot as they all crashed on the upgraded hypervisors before your rollback? The system as it is right now is not congruent with the way it was before. If you were using an out-of-the-box esxi5 image before and reverted back to that image, then what's the thing still around right now that hasn't reverted? Perhaps configuration of the hypervisors, but also perhaps the state of your guests. edit: aaaaand I was a Tier 1 guy at the beginning of my career for a high performance computing environment for climate modeling researchers, so I am aware of the friction that can take place between a developer and the hardware architects in a situation where the hardware is being intentionally pressed to the limit. Every single detail and element in the storage stack is absolutely essential to understand and draw out to reveal all the moving (proverbially) parts. Spending a few thousand bucks on enterprise-grade SSDs and removing parity calculations from your raid controller's workload may save you a lot of trouble in the long run if upgrades and changes are things you like to do this environment and you want those to go more smoothly. A Samsung 850 PRO at 1TB will last six years if written from empty to *full* every single day with a workload incurring a write amplification factor of three. If you mostly read with these devices and write less than 80B per day on average (which is more than likely if your disks are only 40GB in size), a $300 1TB 850 EVO would carry you through to its 5 year warranty and probably well beyond. Any chance you or your IT team have metrics on steady-state I/O of your testing and boot I/O? If not, something like PRTG is free for 100 sensors (more than enough to monitor two esxi hypervisors) That is, assuming it's not just a simple hurf-durf configuration error someone made somewhere 
Potato Salad fucked around with this message at 15:15 on Aug 19, 2016 |
|
#
?
Aug 19, 2016 14:12
|
|
|
Just mulling off the top of my head right now on what I'd do as your IT guy, I'd probably sequentially boot up, let run for a bit, then shut down the VMs in small batches and watch to see whether or not their io is atypical. Paravirtualized devices could have changed on upgrade and the OSes needed a clean reboot after updating opcode or the drivers for your underlying hardware changed with the upgrade and would also require an OS reboot or at least result in a longer reboot depending on your guest OS. There's a lot that goes on when you upgrade esxi generations. Perhaps all the guests need is a clean reboot in small batches and then they'll be good to go as before. edit: consider, for example, what guest you have and whether switching or changing the HAL is something that'll happen on boot for it after the upgrade. It may be the case that this particular bootstorm that is timing out your VMs is a particularly nasty one that needs hand holding https://en.wikipedia.org/wiki/Hardware_abstraction#In_Operating_Systems Potato Salad fucked around with this message at 14:32 on Aug 19, 2016 |
|
#
?
Aug 19, 2016 14:24
|
|
|
As your IT guy, I'd probably also get a quote for a few $300 1TB 850 EVO SSDs and suggest that the endurance of these TLC drives may well be suited for your test environment if you're writing less than 80GB/day to each one each day on average (which would be 560 GB per week per ssd, or 56GB per VM per weekly test if that's anywhere close to your use case). I also need to apologize for what has been a departure from the normally-chill tone of this thread. Potato Salad fucked around with this message at 15:23 on Aug 19, 2016 |
|
#
?
Aug 19, 2016 15:06
|
|
|
I'm working on getting some more specifics. This isn't actually my thing, but the dude working on it was getting really frustrated so I figured I'd ask around. e: To clarify, when I mean "the same thing" I mean literally the exact same thing as the VMs are re-deployed via knife vsphere from templates built by packer that do not change. The Win7 unpatched image is the one we generally use for testing this. We'll also spin up CentOS 6.7 hosts from template and then test their reboot capacity. Being unable to solve this problem isn't the worst thing in the world because it would force dev to support a better testing methodology beyond "make changes, revert to snapshot and reboot." As each time they want to do that we have to make a big thing about changing around permissions for whichever team wanted to use it. Winkle-Daddy fucked around with this message at 15:43 on Aug 19, 2016 |
|
#
?
Aug 19, 2016 15:40
|
|
|
Okay, the drive configuration is 10x1TB 10k SAS drives in a RAID10. In case I wasn't really clear about our goal, all we're really hoping for (best possibly outcome for us) is some config somewhere that tells VMWare to ignore that this host is telling us there's corrupt VMDKs and storage went away. Just chill out and try again in a bit. That's what it looked like it was doing previously...this is essentially what our ticket with VMWare is about, as well. The IT guy told us that VMWare never did anything like that and will utilize all available resources that the ESX OS sees, which also seems crazy to me, but that could very well be the case. This is my first VMWare based stack and I'm far more versed in KVM/OpenStack things. In the meantime, I worked out a script for the dev teams to use to just rolling reboot 3 VMs at a time until they're all rebooted. 'sup tier 1 support buddy? Thankfully I got to support cool clients like Pixar 'n poo poo supporting Adobe garbage.
|
|
#
?
Aug 19, 2016 16:04
|
|
|
devmd01 posted:when certain reports query a million row tables and there are 8 people running those 5-6 times each in the morning, it sends the disk latency through the loving roof because our ancient compellent can't keep up. Pretty good candidate for host-side SSD acceleration if upgrading the backend storage isn't an option. Either VHD-level acceleration with the VMware native features or pernix to accelerate hot blocks.
|
|
#
?
Aug 19, 2016 18:03
|
|
|
devmd01 posted:Whoah, vcenter 6 appliance linked mode owns. Single pane of glass to production and dr! Yeah, I just got some new FX2 chassis for the handful of standalone blades for stuff I don't want on the cluster (backup, DCs, vCenter) and they match on both sites. Going to be so nice instead of this "manually pull the vmx out of the isolated host's datastore and startup" stuff we were doing before.
|
|
#
?
Aug 19, 2016 18:09
|
|
|
|
| # ? May 22, 2024 03:31 |
|
|
So, I have this weird thing happening every once in awhile (intervals from 10 minutes to a couple days). I do lots of work in a plain old VMWare vm hosted with Workstation on my Windows 10 machine. Only while that machine is running, I get system-wide "freezes". And by freeze, I mean things stop happening. The mouse works, and keyboard input is buffered and gets dumped when the machine unfreezes, and my CPU goes to 0. This lasts from a second to maybe 20 seconds. I'm not 100% sure it's tied to VMWare, so I'm hoping someone will say "oh yeah, that's this problem X with VMWare and you have to do Y".
|
|
#
?
Aug 19, 2016 22:42
|
|
