|
NippleFloss posted:StorageTek uses SUN Common Array Management, which is also pretty bad. CAM also includes a cli that is also pretty goofy, but is better than CAM. God, CAM is liquid dog poo poo; Thankfully, our last Sun array is rolling out the door in a few weeks. If the DS4000 used the same rebadged LSI/Engenio software as some of the other LSI resellers, it was pretty inoffensive. in a well actually fucked around with this message at 23:25 on Jun 25, 2012 |
#
¿
Jun 20, 2012 19:12
|
|
|
|
| # ¿ Apr 27, 2024 18:34 |
|
|
KS posted:I'm looking to add some secondary storage for D2D backups -- about 25TB, and I don't want to take more than 2-3U and ~$30k. Not opposed to roll-your-own for this, but only if it saves a bunch of money. Performance needs to not suck. It needs to export as either NFS or iscsi. We've been pretty happy with OmniOS, and the support is much cheaper than Nexenta. Just make sure the hardware is on the Illumos (nee OpenSolaris) HCL; the Dell R720XDs H310 is not as of this writing.
|
|
#
¿
Jul 6, 2012 18:15
|
|
|
Jadus posted:I'm looking at doing this same thing, and am leaning towards something like this guy did using a SuperMicro SC847 - 36 drive chassis and FreeNAS. We've been happy with that chassis w/ OpenSolaris for a few years for light/medium workloads.
|
|
#
¿
Jul 6, 2012 19:52
|
|
|
cheese-cube posted:I've worked with V7000's of varying configurations for a while now and I agree they are brilliant devices (If you can afford them). The 9900 is a rebadged data direct networks 9900, which is two generations behind DDN's current offering. You see them in hpc and broadcast; they do some interesting things to keep those pipes full that make them less well suited for general SAN workloads.
|
|
#
¿
Jul 24, 2012 18:11
|
|
|
Rusty Kettle posted:I am trying to rebuild a RAID array that is managed by Linux using 'fdisk' and 'mdadm' and I am having problems. I am a grad student who I guess is now the IT guy for our research group, so I have little to no experience with this kind of thing. I am very nervous and google isn't helping much. Probably the Linux Questions thread would be a better choice; is it degraded but accessible or completely offline? Also, do whatever you can to not be your research group's IT guy.
|
|
#
¿
Aug 3, 2012 17:29
|
|
|
ghostinmyshell posted:I don't know where my boss keeps finding these supermicro/nexenta vendors but it's like playing Bingo asking them about their short commings and they are like, "How do you know all of this???" I'd be interested in hearing about some of these, if you wouldn't mind. Unrelated tip of the day: always have well-defined required performance numbers in your RFP and make sure they pass before you pay the vendor.
|
|
#
¿
Aug 9, 2012 16:18
|
|
|
Misogynist posted:OpenIndiana and FreeBSD are similar enough to administer that it's really not worth the hassle. If you want a Linux userland with a kernel that better supports ZFS, look into Illumian (formerly Nexenta Core). FWIW, there's been some OpenIndiana drama recently. The project lead quit: http://article.gmane.org/gmane.os.openindiana.devel/1578 There are other server-focused Illumos derivatives that are more actively developed.
|
|
#
¿
Oct 5, 2012 21:14
|
|
|
the spyder posted:I racked one of my new internal 336TB (raw) ZFS SAN's this week and realized that my field engineers are not configuring hot spares. How many controllers? How many CPUs? How many heads? What do the paths from controller to enclosure to enclosure look like? Do you have a SSD ZIL? What does your workload look like? What are your availability requirements? SAS or SATA? We've run as few as 6 disks and as many as 45 disks in vdevs.
|
|
#
¿
Nov 21, 2012 01:04
|
|
|
the spyder posted:One controller, Dual x5650's, 192GB ram, Mellanox Dual port 40Gb IB. Second cold spare racked, identical config. Two LSI HBA's, (2ext ports each) four cables ran to 8+2 w/ 2 hot spares seems reasonable; you may want to keep a few cold spares on hand. You may want to stripe each vdev across the controllers. For the ZIL, we're switching to ZeusRAM on our new servers.
|
|
#
¿
Nov 21, 2012 23:37
|
|
|
Confused_Donkey posted:Reposting this from the HP support forum as I'm running out of ideas here. http://www.ebay.com/sch/i.html?_odkw=eva+5000&_osacat=0&_from=R40&_trksid=p2045573.m570.l1313&_nkw=eva+5000+battery&_sacat=0 Or one of the secondhand enterprise vendors like: http://www.xsnet.com/
|
|
#
¿
Jan 31, 2013 05:43
|
|
|
Novo posted:I could use some information from any ZFS experts in this thread. OmniOS; http://omnios.omniti.com/ ; OSS, and support contracts are v. reasonable. ZFS versions are meaningless since Oracle and Illumos maintain two different forks. Stay away from Solaris 11; there's still a hilarious data loss bug that still isn't fixed in their free release. I think FreeBSD's ZFS implementation si pretty stable, and the Linux ports are making surprisingly decent process.
|
|
#
¿
Apr 19, 2013 00:51
|
|
|
the spyder posted:I support 1.4PB of ZFS based storage across our two sites. In the field, we have over 4PB of ZFS storage. Our older systems are all based off NexantaStor and everything in the last year off OpenIndiana- both running Napp-it 0.8h. Half is linked via Infiniband to our cluster, the other half is archival/local storage. I just ordered another 100TB of disks/chassis, one will be for consolidating our VM's local storage and completely SSD based- the other two for local backup storage. I am very eager to see what 22x240gb Intel 520 series will do performance wise. The other two are 48 3TB 7200rpm spindles. We have suffered some pretty bad disk failures thanks to the dance club downstairs, but not once lost data. (24 failed drives in 12 months.) If you have any specifics, let me know. Everything is based on Supermicro dual quad xeons, 96gb+ of ram, LSI HBA's, and Seagate/Intel drives. How's the IB support working for you? NFS over IPoIB, I assume?
|
|
#
¿
Apr 19, 2013 22:09
|
|
|
hackedaccount posted:You should really drop the OpenIndiana guys a line about this. I'm sure they would enjoy a writing a whitepaper about how their ZFS can survive a dance club. There really aren't any OpenIndiana guys anymore; the OpenSolaris community's moved to Illumos-based distributions. You can see tumbleweeds drifting through OI's mailing lists and hg repos.
|
|
#
¿
Apr 23, 2013 03:52
|
|
|
Misogynist posted:Technically, the most popular Illumos distributions are fairly single-purpose (Nexenta, SmartOS). Illumian's as much of a ghost town as OpenIndiana. The only active general-purpose platform development seems to be centered around OmniOS, which is starting to look like abandonware as well. FWIW OmniOS's being actively developed by OmniTI and it is cash-flow positive for them.
|
|
#
¿
Apr 23, 2013 04:37
|
|
|
Misogynist posted:Man, I basically lifted the "starting to look like abandonware" part from a different part of the post without noticing, sorry. What I meant is that the platform is new and the userland is pretty neglected, at least compared to what people might expect from a modern-day Linux/BSD. I do hope that some more momentum springs up behind it later; OmniTI being the Red Hat of Illumos would be a pretty cool thing for Oracle to have to fight with. NP; I think that's all pretty reasonable, and thanks for clarifying re: abandonware. I do think that the minimalism isn't as big of a deal in a NAS role if you're reasonably comfortable with ZFS and Unix.
|
|
#
¿
Apr 23, 2013 19:13
|
|
|
FISHMANPET posted:We've got a faculty member looking to buy 100-200TB, which to us is "big data." They're looking at some of those ridiculous SuperMicro servers with drives on both sides of the chasis, sold by a SuperMicro reseller we've worked with in the past (aka they would sell us a complete warrantied system). We also have a Compellent SAN, but we don't think getting a pile of trays is going to be cost effective (though it might be, we're still getting quotes). Those SuperMicro chassis are fine for that use case. What OS/platform is the reseller offering? quote:Are there any good inexpensive SANs for big data? We don't need high performance or a lot of features because this will mostly be static data, we just want the system to be manageable and expandable. Not in the SuperMicro price range.
|
|
#
¿
Jul 17, 2013 14:49
|
|
|
Nukelear v.2 posted:What is everyone using for PCIe SSD storage? We're going to be rolling out an ElasticSearch cluster and want to run it on SSD. Initially was planning to just plug some 2.5 drives into hotswap bays, but SSD cards seems to have gotten much more reasonable and then I won't have to try to use aftermarket/unsupported drives in a Dell. We're running a handful of STEC s1120s and they seem OK, but with the HGST acquisition I'm not sure where they fit into HGST's product line. Comedy answer: The 400GB flash DIMMs IBM is selling in the new X3850s.
|
|
#
¿
Feb 20, 2014 20:16
|
|
|
KillHour posted:This thread is probably my best bet because it's storage heavy, even if we aren't using SANs. Supermicro's support process is time-intensive. Good results for end users requires an engaged, active, and knowledgeable reseller. quote:For people that have done OEM work in the past, who is easiest to work with? I've done some work with Dell in the past when I worked at Ingram, and it didn't go very well. JBODs+external servers are going to be a bit less dense than a server w/ a ton of drives. How many are you planning on buying? There are other vendors that sell those form factors, but generally don't deliver a HP/Dell/IBM support experience. quote:Also, one of the biggest issues with large camera systems is disk throughput. I see systems all the time that use 6 or 8 15k SAS drives in a RAID 10 just for 24 hours of storage so that they can offload the video to the slower 7200 RPM drives at night when there's less recording happening. Milestone (the VMS we're using) actually requires a "live" array for this reason. Is there a reason not to use SSDs instead of 15k SAS for something like this? It seems less expensive, and if I use a PCIe card like this, I can even save some drive bays. Be mindful of SSD write lifecycles.
|
|
#
¿
Mar 4, 2014 19:26
|
|
|
CrazyLittle posted:Are SAS ssd's significantly better in that regard? How are the big vendors doing it? Somewhat better, not immune. Better manufacturing/testing tolerances, better controllers, more spare cells at listed capacity, overprovisioning. Dilbert As gently caress posted:To be fair they have gotten much better in the past years, and generally a mechanical drive will fail before an SSD reaches a write life. A rule of thumb is that the better eMLC SSDs can handle 10 full writes per day over 3 years. I'd be concerned if KH's live tier is sized close to his daily write size, especially if using cheaper SSDs. Failure mode depends on the drive and manufacturer. We've seen failing writes, failing reads, the drive disappearing entirely, or taking the entire SAS expander or controller offline (that was a fun one.) KillHour posted:We're very knowledgeable and more than capable of building and supporting systems. That being said, we have about 10,000 cameras out there, so we need to have reliable systems that we won't have to touch often and can be fixed quickly if they do break, or we'll drown. Also, we don't have the luxury of billing by the hour for support. Most of our customers pay for a yearly support contract, and we need to handle those quickly or we can loose a lot of money. Unless you maintain an inventory of cold spare parts or work with a third party fulfillment service you probably don't want to go with SuperMicro.
|
|
#
¿
Mar 4, 2014 21:10
|
|
|
KillHour posted:Wouldn't that mean that if I fill the drive once per day (which is pretty much what would happen), then I should expect about 30 years? I... don't see that being a problem. Also, I'd obviously use RAID 1. I just don't want to be going out there every year to swap out a drive. It's not necessarily a linear relationship and it's not a guarantee, but an average. IIRC, that's with the enterprise Intel drives. Generally speaking, an expected workload within an order of magnitude of predicted lifetime would make me nervous. Check out the SSD megathread OP for more info on lifespan, write averaging, etc. I've heard some anecdotal evidence that some consumer hardware RAID cards don't play well with SSDs, but I don't know much about that.
|
|
#
¿
Mar 5, 2014 02:00
|
|
|
Modern Pragmatist posted:Right now we're looking at 128 GB or RAM but that's purely what the hardware guys recommended. We have no problem spending more if necessary. The ARC works well, give it as much as you can cram in. quote:The setup doesn't necessarily require an SSD for L2ARC or SLOG we were just thinking this would give us a pretty big performance boost. Would it be better to use mirrored 15k spinning disks for these? Is there a more widely used option that we haven't considered? Heh, no; don't bother with 15K drives in any case. I'm going to disagree with thebigcow slightly; if you're really concerned about degraded performance in case of a SSD failure, it's better to add them as two separate devices and let ZFS balance the load across the two. Better peak, and in case of failure your write is only as bad as it would be when mirrored. Frankly, the SLOG and L2ARC failure modes we've seen are clean enough (not in Oracle Solaris (lol), but Illumos is OK) that I wouldn't be too worried, especially for a departmental server. Writes go sync (a big deal for NFS clients, mostly), and read cache falls back to ARC only. L2ARCs mostly make sense when your average working set is larger than RAM but small and nonrandom enough that the L2ARC can effectively cache warm data. My favorite SLOG device: http://www.hgst.com/solid-state-storage/enterprise-ssd/sas-ssd/zeusram-sas-ssd
|
|
#
¿
Apr 18, 2014 05:45
|
|
|
parid posted:I hear a lot about paralyzed file systems. Heh. Lustre is pretty popular. GPFS if you do business with IBM. Ceph if you're interested in emerging technologies.
|
|
#
¿
Apr 27, 2014 20:16
|
|
|
parid posted:I don't have a lot of details. I wouldn't take responsibility for a whole cluster, but I might be asked to help them with storage and I'd like to be useful if they do. Existing systems are getting long in the tooth and haven't been meeting their needs well. Its all NFS right now. I wouldn't be surprised if it was time to step up into something more specific to their use. No, block-level isn't really appropriate for HPC systems; multi-node access at block level falls over badly for more than a few nodes. Lustre and GPFS expose a POSIX-like file system to all clients. For the end user, a GPFS or Lustre installation will look like a NFS or local file system. A standard Lustre* installation would look like E-series** JBODs, connected via SAS to dedicated Lustre servers, which are exposed via the HPC interconnect*** to the main cluster processing nodes**** If you're using Lustre, you'll need a metadata server with some high IOPS disks connected to the HPC interconnect. * (or GPFS native raid) ** (48-72 3.5" 7200 RPM SATA drives and enclosure with a passive backplane and SAS expander) *** (Infiniband or 10GbE if you're lucky, GbE if you're not) **** which will either use a native client (kernel modules or FUSE) or NFS exported from a server that has the native client mounted. I would strongly suggest looking at preconfigured Lustre or GPFS appliances; conveniently, NetApp sells one: http://www.netapp.com/us/media/ds-3243.pdf but you can get them elsewhere.
|
|
#
¿
Apr 29, 2014 04:35
|
|
|
Dilbert As gently caress posted:Isn't SmartOS just a limited VAAI and Vm-open-tools ready appliance? no
|
|
#
¿
Jun 12, 2014 03:43
|
|
|
Dilbert As gently caress posted:Yeah Smart OS is based on OpenSolairs; how the gently caress do you not know the VAAI plugins or tools load in? I'd love to hear about the SmartOS VAAI plugins  , and what you mean by 'just an appliance.'
|
|
#
¿
Jun 12, 2014 04:37
|
|
|
Dilbert As gently caress posted:Okay want to live demo it? On SmartOS, I'm not going to be able to do much more than what you'll find on YouTube; it's not the Illumos distro I primarily use (or that I'd recommend) for storage. I can talk more about OmniOS; my employer has a significant amount of storage using ZFS on OmniOS that we export via NFS and Samba.
|
|
#
¿
Jun 12, 2014 05:30
|
|
|
pixaal posted:Are these actual keys? Shameful. Forums user FCKGW would like a word with you.
|
|
#
¿
Jul 16, 2014 03:07
|
|
|
feld posted:(fyi, we're building servers with 24 500GB SSDs You're doing this with a LSI RAID controller? You're going to hit its limit before you hit the capability of the SSDs.
|
|
#
¿
Jul 29, 2014 15:52
|
|
)
|
feld posted:400,000 IOPS is the limit of the raid controller in question. And you've got ~1-2M IOPS worth of SSD behind it. Which is fine if: feld posted:We don't need to go beyond the 400,000 IOPS limitation, so that's not a concern. but my original statement still stands. goobernoodles posted:Anyone know how to get IBM on the phone without waiting for a callback for SAN issue? loving waiting for a callback. Call your sales rep/account manager. (This only works if you spend enough with IBM.)
|
|
#
¿
Jul 29, 2014 17:50
|
|
|
Declustered RAID is so cool.Number19 posted:I'd even out the raid groups and have a third hot spare. 3 is way too many hot spares for ~45 disks. 1 per 30 to 60 drives is normal.
|
|
#
¿
Jul 31, 2014 03:10
|
|
|
I know we've talked about this before, but what does an entry-level Nimble CS210 or CS220G go for?
|
|
#
¿
Jul 31, 2014 04:07
|
|
|
NippleFloss posted:So this is what a storage hipster looks like? spinning rust is warmer
|
|
#
¿
Aug 6, 2014 21:15
|
|
|
the spyder posted:I figured I would toss this here: I know it's not EMC, Netapp, or Nimble, but I'm prepping 1.8PB of whitebox storage this week. Any testing you guys would like to see before it ships next week? They are very simple and designed to be nothing more then a rather large NAS. The only consumer part are the SSD's. This goes along side a whitebox processing array containing 240 cores, 2.5TB ram, and 20 Xeon Phi's. I'm going to be building another system in the near future, with E5 V3/SAS3/Intel SSD's, so I'm curious how they will compare. Linux, FreeBSD, or Illumos/Solaris?
|
|
#
¿
Sep 30, 2014 04:25
|
|
|
Misogynist posted:Not sure whether you're gay-shaming or slut-shaming your SAN or just implying that your SAN is spending company time on activities it ought not to nah just that operating near capacity brings most sans to their knees
|
|
#
¿
Oct 12, 2014 00:31
|
|
|
adorai posted:If you are going to roll your own, I'd look into openindiana. Stay the gently caress away from OpenIndiana; it is dead and it sucks. If you want a OpenSolaris derivative, use one of the Illumos distributions. Alternatively, the FreeBSD/ZFS port is pretty mature, and the ZFS on Linux port isn't bad for experimental/development work if you're comfortable with either one of those.
|
|
#
¿
Jan 30, 2015 03:51
|
|
|
thebigcow posted:OpenIndiana is an Illumos distribution, are you thinking of something else? When was the last OpenIndiana release? Almost two years ago. Tumbleweeds on the dev-list, etc.
|
|
#
¿
Jan 30, 2015 04:50
|
|
|
adorai posted:There is development on the hipster branch. https://github.com/OpenIndiana/oi-userland/graphs/contributors Unless I'm misreading this, almost all of the updates in the last 6 months are from one sysadmin in Russia.
|
|
#
¿
Jan 30, 2015 21:48
|
|
|
cheese-cube posted:Just want to quote NippleFloss because holy poo poo so many people gloss-over FC based on throughput numbers yet they don't actually understand how resilient FC is as a protocol. You could have an FC fabric with >200 unique domain IDs and still have sub-millisecond RSCN propagation times. FC is great; if you've got the budget for diamonds why go with a polished turd.
|
|
#
¿
Mar 3, 2015 21:11
|
|
|
Mr Shiny Pants posted:If you have diamonds why not go whole hog and get Infiniband? I like Infiniband; it's so weird, the per-port cost's lower than 40GbE, but the enterprise fabric management is not anywhere near what you get with FC. adorai posted:sometimes children get their hands chopped off because of the diamond trade. A small price to pay to escape bob metcalfe's abomination.
|
|
#
¿
Mar 4, 2015 01:59
|
|
|
|
| # ¿ Apr 27, 2024 18:34 |
|
|
If you have Ceph experience or are interested in Ceph and have a resume, send me a PM.
|
|
#
¿
Feb 1, 2016 23:20
|
|