|
PCIe does rule, and it's especially great how extensible it is by adding on to both the protocol level (NVMe) and physical level (CXL). One thing that's been always interesting to me with PCIe throughout the generations is how when you're on a new version of it the work and debug involved to get, say 3.0 rates going at the time will be huge, but then when it is 4.0 the previous gen speeds will just work, no big deals. Definitely an initial hiccup as the tunings, board materials and routing are worked out but then going forward, easy. So for PCIe 6 I'm really curious how this will work out because the signal frequency will be the same so the boards don't need to get fancier, however using the PAM-4 signalling introduces a funky new wrinkle. This should be a solved problem as other specifications have used PAM stuff for ages but still.. I wonder. Gen5 is still some ragged edge stuff in a lot of cases too lol
|
#
?
May 25, 2023 19:48
|
|
|
|
| # ? May 24, 2024 03:11 |
|
|
priznat posted:PCIe does rule, and it's especially great how extensible it is by adding on to both the protocol level (NVMe) and physical level (CXL). All paths lead to 
|
|
#
?
May 25, 2023 20:46
|
|
|
movax posted:All paths lead to Yup I was predicting ages ago (before nvme became the major thing it is now) that the only interfaces we will need will be ethernet, pcie and (lol) usb. RIP SATA, HDMI/Displayport/DVI, and all the other nonsense connections. SAS may soldier on but really x1 nvme spinning disks should hopefully displace them VGA and RS232 will survive for last gasp server debug and provisioning
|
|
#
?
May 25, 2023 20:53
|
|
|
priznat posted:Yup I was predicting ages ago (before nvme became the major thing it is now) that the only interfaces we will need will be ethernet, pcie and (lol) usb. Nah, HDMI is going to stick around because it's where all the DRM bullshit like HDCP needs to go. x1 nvme spinning disks seem kinda stupid unless you've got access to really cheap PCIe switch chips. With SAS you can chain a whole lotta drives together with cheap expanders and backplanes.
|
|
#
?
May 25, 2023 21:05
|
|
|
priznat posted:Yup I was predicting ages ago (before nvme became the major thing it is now) that the only interfaces we will need will be ethernet, pcie and (lol) usb. I want a better UART / better USB, still... multi-mode SPDIF to create a low bandwidth, cheap POF solution would be neat. One of my weird things is repurposing TOSLINK transmitters/receivers to do all sorts of hosed up things...
|
|
#
?
May 25, 2023 21:09
|
|
|
Expanders and HBAs are garbage products NVMe has an ever expanding feature set and moving to NVMe let’s you unify your support steps and ecosystem Also let’s you get on the hot NVMeoF craze WhyteRyce fucked around with this message at 21:20 on May 25, 2023 |
|
#
?
May 25, 2023 21:11
|
|
|
WhyteRyce posted:Expanders and HBAs are garbage products And why bother with Ethernet anymore when you can just send PCIe over fiber and toss switch chips on either end! (screams in early 2010s PLX NT bridging)
|
|
#
?
May 25, 2023 21:13
|
|
|
movax posted:And why bother with Ethernet anymore when you can just send PCIe over fiber and toss switch chips on either end! That sounds like hot sex. Is that the kind of thing that costs a ton or is it home-lab friendly?
|
|
#
?
May 25, 2023 23:50
|
|
|
redeyes posted:That sounds like hot sex. Is that the kind of thing that costs a ton or is it home-lab friendly? NVMeoF is extremely cool and you might be able to find a couple older mellanox cards for cheap you could set up with a QSFP28 copper link to go from a host to storage box PCIe switch cards would be harder to find for cheap though, usually they are built in to server motherboards or backplanes and the AIC cards that do exist are extremely limited run from Serial Cables or other type companies. Might be able to find a Gen3 or Gen4 for cheap-ish
|
|
#
?
May 26, 2023 00:33
|
|
|
WhyteRyce posted:Expanders and HBAs are garbage products 
|
|
#
?
May 26, 2023 01:41
|
|
|
I worked for a bit on a SAS controller and I agree. The things that really don’t make sense to me are the tri mode raid controllers where people are plugging nvme drives into it. Is there really any benefit? Do people really want to run RAID on nvme in tyool 2023?? This honestly seems like a scam to me for people selling enterprise hardware to people who don’t know any better.
|
|
#
?
May 26, 2023 01:46
|
|
|
priznat posted:I worked for a bit on a SAS controller and I agree. Those NVME hardware RAID controllers confuse the hell out of me too. That's got to do a horrible number on write throughput, right? A single AMD Epyc CPU has 128 PCIE 5.0 lanes, and you get 80 on a Xeon, why aren't you just using those?
|
|
#
?
May 26, 2023 02:05
|
|
|
priznat posted:I worked for a bit on a SAS controller and I agree. LSI is ok overall, especially if you stick to some popular models. Or at least that seems to be the homelab consensus. trimode still supports SAS3/6 where HDDs are a thing and you might want to raid. not cost effective but hey, it's supported, it's the premium model, it does everything. And also if someone's contractual compliance checkbox requires them to have "hardware/battery-backed RAID redundant storage" or active driver support from the vendor then well... it still seems like an idiot trap but there are a lot of idiots in control of a lot of IT spending/a lot of dumb PCI standards compliance poo poo / a lot of poo poo that lingers in SLAs for way too long in mid-size corps/etc. if people wanna buy dumb poo poo, sell it to them Paul MaudDib fucked around with this message at 02:47 on May 26, 2023 |
|
#
?
May 26, 2023 02:35
|
|
|
Wasn’t U.3 was created just so you could do NVMe/SAS/Sata tri mode bullshit so enterprise builders and customers could just recycle their same old rear end platform and chassis designs
WhyteRyce fucked around with this message at 02:49 on May 26, 2023 |
|
#
?
May 26, 2023 02:47
|
|
|
WhyteRyce posted:Wasn’t U.3 was created just so you could do NVMe/SAS/Sata tri mode bullshit so enterprise builders and customers could just recycle their same old rear end platform and chassis designs U.2 u.3 whatever happened to u.1
|
|
#
?
May 26, 2023 02:50
|
|
|
priznat posted:The things that really don’t make sense to me are the tri mode raid controllers where people are plugging nvme drives into it. Is there really any benefit? Do people really want to run RAID on nvme in tyool 2023?? This honestly seems like a scam to me for people selling enterprise hardware to people who don’t know any better. At work we'll probably end up with that until RHEL starts supporting ZFS. And even then it will probably take some convincing that ZFS is better than battery backed write cache. It probably limits them quite a bit, but I would assume NVMe behind RAID is still much faster than SAS.
|
|
#
?
May 26, 2023 06:15
|
|
|
Saukkis posted:At work we'll probably end up with that until RHEL starts supporting ZFS. And even then it will probably take some convincing that ZFS is better than battery backed write cache. It probably limits them quite a bit, but I would assume NVMe behind RAID is still much faster than SAS. Ahh ok that makes sense. Once ZFS is more adopted though.. I am really interested to see if there will be some CXL solutions with battery backed up cache at some point, it seems like a great use case/memory tier.
|
|
#
?
May 26, 2023 06:43
|
|
|
I thought Samsung already demoed a CXL dram+ssd package. I don’t recall if they were just two discrete components on the same package or if it was some kind of persistent memory solution though I’m assuming no reason why you couldn’t though. Can fit a whole lot more stuff on a CXL package than you could on a dimm WhyteRyce fucked around with this message at 07:04 on May 26, 2023 |
|
#
?
May 26, 2023 07:02
|
|
|
True, enterprise NVMe should have battery, and I think some of them actually have them, if not all. I think flash could end up in bad shape if it loses power suddenly. I think RAID-cards nowadays mostly have flash-backed write cache. A stick of RAM, flash storage and enough capacitor to write the RAM contents to flash in power outage.
|
|
#
?
May 26, 2023 07:46
|
|
|
Yeah the enterprise M.2s have big fat caps on them for the power loss, and U.2s or AICs do too. It’s nice to take it off the controller altogether and then you don’t even need a discrete controller anymore! Also WhyteRyce you jogged my memory, I have seen that Samsung E3.S CXL thing pictured around, looks interesting! I bet it does have at least enough SSD to backup the memory on it at least, the one I saw was a .mem DDR device iirc.
|
|
#
?
May 26, 2023 07:51
|
|
|
priznat posted:Yeah the enterprise M.2s have big fat caps on them for the power loss, and U.2s or AICs do too. It’s nice to take it off the controller altogether and then you don’t even need a discrete controller anymore! https://www.tomshardware.com/news/samsung-memory-semantic-cxl-ssd-brings-20x-performance-uplift quote:To address such workloads, Samsung developed its special Memory-Semantic SSD that combines a huge built-in DRAM cache, traditional NAND storage, and a PCIe Gen5 interface with the CXL technology on top. Applications can write data to the DRAM cache at DRAM speeds and with low latency enabled by the CXL.mem protocol. Then, Samsung's proprietary controller (or controllers) of the drive transfers data to NAND memory. The result is a 20 times random read/write performance uplift compared to traditional SSDs.
|
|
#
?
May 26, 2023 14:57
|
|
|
And if you didn't have to use some lovely RISC-V or ARM based microcontroller running a proprietary RTOS as firmware, it'd be even faster - because then you could do priority queueing all throughout the stack.
|
|
#
?
May 26, 2023 16:09
|
|
|
DIMM also needs to die. edit: to clarify, I want the bus interface to die, idgaf about the form factor Beef fucked around with this message at 16:24 on May 26, 2023 |
|
#
?
May 26, 2023 16:15
|
|
|
Beef posted:DIMM also needs to die. Form factor too imho, being able to have E3 modules you can just yoink out from the front of a rack would be p sweet.
|
|
#
?
May 26, 2023 16:53
|
|
|
Beef posted:DIMM also needs to die. The compression fitting that's taking over in laptops looks a little fussy: https://www.pcworld.com/article/1473126/camm-the-future-of-laptop-memory-has-arrived.html I guess that's the same bus though. How do you see things being packaged for server platforms in the future? Server requirements vary so much that it's going to be hard to keep costs rock bottom while meeting everyone's needs if we just soldered memory next to the package.
|
|
#
?
May 26, 2023 16:57
|
|
|
Twerk from Home posted:The compression fitting that's taking over in laptops looks a little fussy: https://www.pcworld.com/article/1473126/camm-the-future-of-laptop-memory-has-arrived.html My guess is that we're going to see memory being tiered into a fast tier with on-package memory and slower pooled-memory tier with CXL memory modules. Reasoning: Memory pooling and memory tiering has been talked about for ages in the server space. With CXL.mem and CXL.cache we're seeing a standard where that could happen properly. Apparently, Micron is already going in the direction of CXL memory modules: https://www.micron.com/solutions/server/cxl You could make a DRAM + CXL controller board or rack unit, slot a bunch of them on top of the rack and have each compute node use some of that pooled memory at ~ 2x latency cost. The hyperscalars are already looking into (using?) cold page compression, so why not drop the cold pages into off-board pooled memory. The budget you save on not having to plug 1TB per node could go into faster on-package stacked memory. I would say HBM, but that's too pricey for a super-wide bus that's going to be wasted on most workloads. There's room here for non-bus memory standards. CLX for memory also opens up more options for some forms of in-memory processing. The current memory bus system with it's fixed command rates, shared bandwidth etc make it really hard to do it effectively. For the problems I am working on, I would absolutely love to be able to do a merge or linear search without having to pull in all cache lines through the CPU pins.
|
|
#
?
May 26, 2023 19:07
|
|
|
The future: all nodes just have HBM and your DRAM is sitting out on the fabric on on some CXL jbod
|
|
#
?
May 27, 2023 01:31
|
|
|
https://twitter.com/VideoCardz/status/1662005677814841344?s=20
|
|
#
?
May 27, 2023 01:39
|
|
|
I'm not even going to touch latency, but bandwidth wise even 800gbit networking, which is the current cutting edge of Ethernet is only the bandwidth of 2 channels of DDR5-6400. A dual-socket Epyc server has 24 memory channels right now. I realize current DDR5 is a little slower than that, but it's improving quickly.
|
|
#
?
May 27, 2023 01:56
|
|
|
Quantity is quality
|
|
#
?
May 27, 2023 02:07
|
|
|
buying scrap low end from 3.5 years ago and putting a new IHS on them is such a good chinese company PR scam
|
|
#
?
May 27, 2023 02:10
|
|
|
Saukkis posted:At work we'll probably end up with that until RHEL starts supporting ZFS. And even then it will probably take some convincing that ZFS is better than battery backed write cache. It probably limits them quite a bit, but I would assume NVMe behind RAID is still much faster than SAS. ZFS on nvme is fine, but kinda underwhelming. A design around spinning rust doesn’t really drive a billion low latency iops well. RHEL will never officially support ZFS unless Oracle changes the license. (Yes I know it’s a yum install away, but that’s too far for corporate)
|
|
#
?
May 27, 2023 04:37
|
|
|
in a well actually posted:ZFS on nvme is fine, but kinda underwhelming. A design around spinning rust doesn’t really drive a billion low latency iops well. Yeah, or if Red Hat and Oracle make some kind of deal. But then I remember how Red Hat suddenly purged Oracle Java from their distros. So not much hope for ZFS with us, we are loathe to use anything besides vanilla kernel. We mostly use HPE ProLiant servers and we don't even use drivers from HPE, except in one server where the kernel RAID driver kept crashing for some reason. But that hasn't been without issues either, one time I updated to the latest smartpqi and the RAID ran so slow it caused an outage for the service and we had to revert to the old version. Haven't dared to upgrade again.
|
|
#
?
May 27, 2023 05:04
|
|
|
in a well actually posted:ZFS on nvme is fine, but kinda underwhelming. A design around spinning rust doesn’t really drive a billion low latency iops well. There's one (or two?) companies involved in working on optimizing ZFS for NVMe - which primarily means taking advantage of 2^16 queues with 2^16 commands per device.
|
|
#
?
May 27, 2023 16:48
|
|
|
Cygni posted:buying scrap low end from 3.5 years ago and putting a new IHS on them is such a good chinese company PR scam Its probably not a new IHS, just scrubbed and reprinted.
|
|
#
?
May 29, 2023 13:15
|
|
|
I haven't been paying much attention to MTL but some information is starting to get out. In particular there appears to be a 16-core with a build it "VPU" for AI stuff: https://videocardz.com/newz/intel-shows-off-16-core-22-thread-meteor-lake-cpu-in-generative-ai-content-demo Any ideas what's with the core configuration of 6+8+2? Why are there two e-cores on the SoC and what are they going to do? It's also pretty weird that these are likely to be launching this year with Crestmont cores but Gracemont-based CPUs are just making it to market.
|
|
#
?
May 30, 2023 17:07
|
|
|
mobby_6kl posted:I haven't been paying much attention to MTL but some information is starting to get out. In particular there appears to be a 16-core with a build it "VPU" for AI stuff: I believe the idea is you don't even have to power up the compute tile at all for low level low perf background stuff. Idle power/battery life optimization.
|
|
#
?
May 30, 2023 18:03
|
|
|
mobby_6kl posted:I haven't been paying much attention to MTL but some information is starting to get out. In particular there appears to be a 16-core with a build it "VPU" for AI stuff: You can power off everything else except the two E-cores for super low power compute.
|
|
#
?
May 30, 2023 21:01
|
|
|
This is the first set of benchmarks I've seen for the sapphire rapids max CPUs that have built-in HBM: https://www.nextplatform.com/2023/06/12/intel-pits-its-sapphire-rapids-xeon-sp-against-amd-genoa-epycs/ Unfortunately they're (a) first party benchmarks, and (b) they compare Intel's best 32 core chip to one of AMD's lesser models, avoiding the fastest clocked one and publishing the benchmarks right before AMD release their 3D v-cache Epycs. All the same, the results are really impressive for some workloads. Hopefully some trustworthy tech reviewers do a head-to-head between genoa-x and xeon max. e: AMD just released some benchmarks where they did exactly that. ee: or not, turns out the comparison was with a vanilla sapphire rapids CPU. ConanTheLibrarian fucked around with this message at 19:37 on Jun 14, 2023 |
|
#
?
Jun 13, 2023 21:39
|
|
|
|
| # ? May 24, 2024 03:11 |
|
|
https://twitter.com/VideoCardz/status/1669286326292586497?t=tZVDQ4sEkhxseusFIPMDBA&s=19
|
|
#
?
Jun 15, 2023 12:56
|
|
