|
Thanks Plorkyeran! Indeed I didn't know about some of that stuff.Xarn posted:The latter is you being bad at cmake, per project flags are easy and old. Yes I agree, I certainly won't claim (or aspire) to be good at CMake. You're probably right about per-file flags, but I definitely wanted per-executable flags. The source of this was I have some tests (separate executables) and I don't remember which switch I wanted to compile with some of them different and after a while of google searching and documentation reading I couldn't figure out how to do so.
|
#
?
Sep 29, 2021 03:47
|
|
|
|
| # ? May 17, 2024 09:34 |
|
|
Does anyone have any favorite books / references / articles / blog posts about static vs dynamic linking? I'm trying to wrap my head around a good plan for an HPC cluster that I inherited without much guidance. The most common options in the HPC space seem to be either statically linking everything, or putting a network share on LD_LIBRARY_PATH and putting all of the .sos that binaries need there. I didn't know about these approaches and have just been merrily installing all of the library dependencies across the cluster with Ansible, but I think it's time to fall more in line with norms. More broadly on the topic of static vs dynamic linking, I found this approach pretty pragmatic, to dynamically link, but ship all of your dependencies other than libc with your binary. Am I overthinking this and just static linking everything is the way to go for portable binaries of software that's built outside of a system's repository infrastructure? Some relevant thoughts about why static linking is king for scientific computing: https://ro-che.info/articles/2016-09-09-static-binaries-scientific-computing Twerk from Home fucked around with this message at 04:14 on Sep 29, 2021 |
|
#
?
Sep 29, 2021 04:06
|
|
|
Dynamic linking typically makes linking faster, which can be great for debug builds. For release builds where you're shipping all the libraries yourself, not sharing them with anything, and don't care to support people replacing your copy of library with a different one, there isn't much of a reason to dynamically link.
|
|
#
?
Sep 29, 2021 04:26
|
|
|
static link everything you have sources for and isn't limited by license (e.g. ffmpeg with lgpl).
|
|
#
?
Sep 29, 2021 07:17
|
|
|
Vino posted:Thanks Plorkyeran! Indeed I didn't know about some of that stuff. It is target_compile_options, which as far as I can tell was introduced in 3.0, aka before I even started programming.  But yeah, CMake documentation is amazingly unopinionated and does bad job steering you towards doing the right thing. Basically you have to look for tutorials/guides for "Modern CMake", and even then you run into tutorials for like 3.10, with workaround for things you can do straightforwardly in current versions. But at least "Modern CMake" steers you towards doing things through targets, which is the basic requirement not to go insane. Foxfire_ posted:"Use different warnings settings on these files" isn't an unreasonable ask warning flags are one of the few things that won't break things if they are inconsistent, but it already seems sketchy tbh. I would rather you have the same set of warning flags in a project, and disable the ones you don't care for locally than trying to pick apart a build script to see which warnings apply to which files.
|
|
#
?
Sep 29, 2021 12:02
|
|
|
Plorkyeran posted:Dynamic linking typically makes linking faster, which can be great for debug builds. For release builds where you're shipping all the libraries yourself, not sharing them with anything, and don't care to support people replacing your copy of library with a different one, there isn't much of a reason to dynamically link.
|
|
#
?
Sep 29, 2021 12:34
|
|
|
learning the win32 api at long last (just to fix a bug in a software I want to use lol) i think everyone first project is to write a function to report system library errors, here is mine. which brings me to ask, how you do test for signed integer overflow when you use Microsoft's compiler? is there anything similar to GNU's __builtin_<op>_overflow() functions? or are you just supposed to make assumptions about widths of types and write your own in which you cast to the size next larger to test the result?
|
|
#
?
Sep 29, 2021 13:14
|
|
|
Xarn posted:warning flags are one of the few things that won't break things if they are inconsistent, but it already seems sketchy tbh. I would rather you have the same set of warning flags in a project, and disable the ones you don't care for locally than trying to pick apart a build script to see which warnings apply to which files. External code (third party libraries, headers supplied by vendors) are an excellent use case for setting different warning levels in the build pipeline and not modifying the code to set #pragmas or whatever, since you probably shouldn’t be or can’t be making local modifications to code that isn’t yours. Even then though best practice would have you keeping all external code in a separate directory for which you could set the warnings rather than per file.
|
|
#
?
Sep 29, 2021 16:30
|
|
|
Xarn posted:It is target_compile_options, which as far as I can tell was introduced in 3.0, aka before I even started programming. Thank you! I'll give it a try next time I run in to this.
|
|
#
?
Sep 29, 2021 19:05
|
|
|
csammis posted:External code (third party libraries, headers supplied by vendors) are an excellent use case for setting different warning levels in the build pipeline and not modifying the code to set #pragmas or whatever, since you probably shouldn’t be or can’t be making local modifications to code that isn’t yours. That's why you pragma it away in your own code, or create wrapper over it. But really, if the problem is third party headers dragging in warnings into your own TUs, the chances are that what you want is -isystem, because afaik even MSVC bumbled their way into useful implementation of it, because that is simpler than ensuring that all your TUs that touch said header disable specific warnings.
|
|
#
?
Sep 29, 2021 19:18
|
|
|
matti posted:learning the win32 api at long last (just to fix a bug in a software I want to use lol) On Win32 (including 64 bit) the int type is always 32 bit wide. As far as I can tell, Microsoft's compiler does not have a way to detect integer overflow after the fact. Write your code so it doesn't cause overflow in the first place. (Or assume nobody's going to supply a message and format input exceeding 2 GB.) Write a function or two to do saturating addition if you really want to attempt to handle those absurd cases. At least x86 and x86_64 offer saturating arithmetic intrinsics. Personally I think it'd be better to just limit the buffers to something reasonable like 64k. Edit: I guess you can use the _addcarry_u32 function on x86/64 to do addition and check for nonzero carry to detect overflow. quote:
And it looks really dumb when a program in one language prints an error message in a different language, that it fetched from the OS. You should definitely try to request the message in the UI language your program is using rather than the user default UI language for the system. But if the requested language isn't on the system it should be falling back to one that does exist. nielsm fucked around with this message at 22:11 on Sep 29, 2021 |
|
#
?
Sep 29, 2021 22:00
|
|
|
i will take that as MSVC will not re-order my overflow checks.nielsm posted:(Or assume nobody's going to supply a message and format input exceeding 2 GB.) hmmmm matti fucked around with this message at 22:21 on Sep 29, 2021 |
|
#
?
Sep 29, 2021 22:17
|
|
|
Xarn posted:That's why you pragma it away in your own code, or create wrapper over it. matti posted:i will take that as MSVC will not re-order my overflow checks. intsafe.h is the MSVC equivalent of the gcc __builting_*overflow() functions [though looking at their implementations, they're undefined behavior/using secret knowledge about how the optimizer works and just doing the same overflow checks you would do]
|
|
#
?
Sep 29, 2021 22:39
|
|
|
Foxfire_ posted:intsafe.h is the MSVC equivalent of the gcc __builting_*overflow() functions [though looking at their implementations, they're undefined behavior/using secret knowledge about how the optimizer works and just doing the same overflow checks you would do] thanks this is the knowledge i was looking for xoxo
|
|
#
?
Sep 29, 2021 22:47
|
|
|
i ended up on the solution where i just write trivial inline assembly routines for signed integer addition when i have to do it functions have the signature <op>_<types> or <op>_<type>_<to|by>_<type> and take a jmp_buf as a third argument for handling overflow. only downside is having to declare pointers as volatile if you want to free them in the clean-up code. but it keeps any arithmetic very readable. 'unno, this is for personal stuff, i would not do this in an organization matti fucked around with this message at 17:24 on Sep 30, 2021 |
|
#
?
Sep 30, 2021 17:20
|
|
|
If this is for personal stuff, have you considered just using clang? It’s literally supported by Microsoft now.
|
|
#
?
Oct 1, 2021 04:45
|
|
|
too new, maybe in couple of years i do not typically expect anything to work matti fucked around with this message at 05:54 on Oct 1, 2021 |
|
#
?
Oct 1, 2021 05:06
|
|
|
I'm back with more CMAKE troubles. I'm trying to statically link an application that's using BLAS/LAPACK with a CMAKE build. I am intending to have this support both OpenBLAS an Intel MKL. Cmake is finding my libraries just fine with FindBLAS, like this: code:code:Edit: I worked around this by doing an incredibly un-Cmake thing and manually passing it the linker flags I want it to use, so now my build is incredibly un-portable and Intel MKL only to build a static build. I think that I really want two completely different CMakeLists.txt , one that is able to build on most machines letting Cmake find the libraries, and using the libraries it finds to build & link a dynamically linked executable, and another entirely separate build process that is extremely tightly coupled to Ubuntu 20.04 with Intel MKL, but produces a static executable that will run everywhere. Thoughts about that approach? Twerk from Home fucked around with this message at 23:01 on Oct 1, 2021 |
|
#
?
Oct 1, 2021 20:42
|
|
|
I have a question about boost::shared_mutex and boost::shared_lock. I have this class:C++ code:Is it ... ok? I'd say it isn't.
|
|
#
?
Oct 2, 2021 00:38
|
|
|
I don’t see anything in the documentation that says that boost::shared_mutex supports recursive locking, no. It’s just a reader/writer lock.
|
|
#
?
Oct 2, 2021 00:53
|
|
|
rjmccall posted:I don’t see anything in the documentation that says that boost::shared_mutex supports recursive locking, no. It’s just a reader/writer lock. That's exactly it. Nobody said "this is cool" and nobody said "don't do it". I'm just hunting for 2 days for that writer lock and I can't find it. That someone, that must be there that must hold the exclusive lock and I just can't find it. Everyone is saying they're waiting to get the lock (attaching with gdb to the deadlocked program) but nobody seems to own the drat lock. For now, I just changed the code to just return value*2 and so far so good. I'm just ... dunno, throwing my hands up in the air. Or, are you saying that if they don't specify that they support recursive locking therefore implicitly it's not cool at all?
|
|
#
?
Oct 2, 2021 01:25
|
|
|
Volguus posted:That's exactly it. Nobody said "this is cool" and nobody said "don't do it". I'm just hunting for 2 days for that writer lock and I can't find it. That someone, that must be there that must hold the exclusive lock and I just can't find it. Everyone is saying they're waiting to get the lock (attaching with gdb to the deadlocked program) but nobody seems to own the drat lock. For now, I just changed the code to just return value*2 and so far so good. I'm just ... dunno, throwing my hands up in the air. So if you recurse the readlock like you do in that example, you potentially run into a case where you go ... code:
|
|
#
?
Oct 2, 2021 01:39
|
|
|
Volguus posted:Or, are you saying that if they don't specify that they support recursive locking therefore implicitly it's not cool at all? This, yes. You should not assume a mutex supports recursive locking unless it says it does. Separately, yes, if everything acquiring a reader/writer lock is acquiring it in reader mode then the lock isn’t doing anything useful.
|
|
#
?
Oct 2, 2021 01:52
|
|
|
Twerk from Home posted:I'm back with more CMAKE troubles. The FindBLAS documentation says that it exports BLAS_LIBRARIES and BLAS_LINKER_FLAGS variables in addition to the BLAS::BLAS alias target, the latter of which excludes all the -l and -L dynamic libraries. Perhaps try to link against those rather than use the exported target?
|
|
#
?
Oct 2, 2021 02:01
|
|
|
Thank you so much. This all became so much clearer now. It's definitely what's going on, as roomforthetuna stated, the unique_locks that are happening are just waiting for the readlocks to return which they never do since that write lock is ... locking. I never used the shared/unique locks of boost since I never really understood them. It seems that the original writers of this code (there are a bunch, over many years, so it all makes sense) didn't either. Yes, removing the recursive locks should be first priority for now.
|
|
#
?
Oct 2, 2021 02:04
|
|
|
Are the boost synchronization primitives compatible with clang thread safety annotations like GUARDED_BY, LOCKS_EXCLUDED, etc.? Those things are great for preventing you from accidentally trying to double-lock.
|
|
#
?
Oct 2, 2021 02:04
|
|
|
Related question, are shared locks significantly more expensive than just doing regular exclusive locks? I mean obviously they're better for the cases where they're clearly appropriate, but if e.g. you just have two threads, one reading and one writing, is using shared_lock for the reader and unique_lock for the writer going to be a meaningful performance-suck compared to if you just used std::scoped_lock for both, or is it basically irrelevant such that using a read-write lock is worth it just in case occasionally you spin up a third thread and two reads sometimes want to happen at the same time?
|
|
#
?
Oct 2, 2021 02:10
|
|
|
Yeah, even if shared acquire/release happen to work under recursion, it’s quite likely that upgrades don’t and will deadlock if performed recursively while holding a shared lock. That would be a natural result of an implementation that e.g. just kept a reader count. Regardless, of course, you shouldn’t rely on a property that isn’t promised by the documentation.
|
|
#
?
Oct 2, 2021 02:19
|
|
|
Twerk from Home posted:Edit: Sucks, but you do what you gotta do sometimes. For what it's worth, I tried to recreate your problem on Fedora and failed. pre:[khulud@phoenix build]$ cat ../CMakeLists.txt cmake_minimum_required(VERSION 3.10) project(test) add_executable(exe main.cpp) set(CMAKE_EXE_LINKER_FLAGS -static) set(BLA_STATIC on) find_package(BLAS) target_link_libraries(exe PRIVATE BLAS::BLAS) [khulud@phoenix build]$ cmake --version cmake version 3.20.5 CMake suite maintained and supported by Kitware (kitware.com/cmake). [khulud@phoenix build]$ cmake -G Ninja .. [...] -- Found BLAS: -Wl,--start-group;/opt/intel/oneapi/mkl/2021.4.0/lib/intel64/libmkl_intel_lp64.a;/opt/intel/oneapi/mkl/2021.4.0/lib/intel64/libmkl_sequential.a;/opt/intel/oneapi/mkl/2021.4.0/lib/intel64/libmkl_core.a;-Wl,--end-group;-lpthread;-lm;-ldl -- Configuring done -- Generating done -- Build files have been written to: /tmp/wow/build [khulud@phoenix build]$ ninja [2/2] Linking CXX executable exe [khulud@phoenix build]$ ninja -t commands /usr/bin/c++ -MD -MT CMakeFiles/exe.dir/main.cpp.o -MF CMakeFiles/exe.dir/main.cpp.o.d -o CMakeFiles/exe.dir/main.cpp.o -c ../main.cpp : && /usr/bin/c++ -static CMakeFiles/exe.dir/main.cpp.o -o exe -Wl,--start-group /opt/intel/oneapi/mkl/2021.4.0/lib/intel64/libmkl_intel_lp64.a /opt/intel/oneapi/mkl/2021.4.0/lib/intel64/libmkl_sequential.a /opt/intel/oneapi/mkl/2021.4.0/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl && : [khulud@phoenix build]$ file exe exe: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=2f2cf350d5012a37de7b552295e438dd0d334552, for GNU/Linux 3.2.0, not stripped, too many notes (256) [khulud@phoenix build]$ ./exe Major version: 2021 Minor version: 0 Update version: 4 Product status: Product Build: 20210904 Platform: Intel(R) 64 architecture Processor optimization: Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors ================================================================
|
|
#
?
Oct 2, 2021 02:27
|
|
|
roomforthetuna posted:Related question, are shared locks significantly more expensive than just doing regular exclusive locks? In principle, no, they should be doing roughly the same atomic operations in roughly the same sequence as would be require for an exclusive lock, and the only cost difference is that reader-writer locks avoid blocking threads in more situations. In practice, reader-writer locks tend to be substantially less tuned; it’s not uncommon for a reader-writer lock to be implemented with something like a mutex and a condition variable behind the scenes, which will be much less efficient. Also, while I’ve heard legends of reader-writer locks with priority inversion avoidance, I don’t know of any platforms that actually do it.
|
|
#
?
Oct 2, 2021 02:34
|
|
|
Winter Stormer posted:Sucks, but you do what you gotta do sometimes. I really appreciate this! I was going to post a minimal example to pull this out from all the other libraries I'm linking hoping to get some help like this, but you went and did it for me. I'm going to fix up my CMakeLists and probably be in business. I'm a CMake novice muddling through, and what I had was more like: pre:cmake_minimum_required(VERSION 3.0)
project(main)
set(CMAKE_CXX_STANDARD 17)
set(BLA_STATIC ON)
find_package(LAPACK REQUIRED)
find_package(BLAS REQUIRED)
... other libs and all project support files
target_link_libraries(main -static)
target_link_libraries(main ${BLAS_LIBRARIES})
|
|
#
?
Oct 2, 2021 02:36
|
|
|
Twerk from Home posted:I really appreciate this! I was going to post a minimal example to pull this out from all the other libraries I'm linking hoping to get some help like this, but you went and did it for me. I'm going to fix up my CMakeLists and probably be in business. I'm a CMake novice muddling through, and what I had was more like: I futzed with it a little further just now and did manage to break my build by setting cmake_version_required < version 3.3. Since that function sets CMake policy versions, I went looking at what policies 3.3 added, and I'm pretty certain you're running into the problem fixed by https://cmake.org/cmake/help/latest/policy/CMP0060.html. Try setting your minimum to be 3.3 or better.
|
|
#
?
Oct 2, 2021 02:41
|
|
|
rjmccall posted:In principle, no, they should be doing roughly the same atomic operations in roughly the same sequence as would be require for an exclusive lock, and the only cost difference is that reader-writer locks avoid blocking threads in more situations.
|
|
#
?
Oct 2, 2021 02:45
|
|
|
If a reader writer lock isn’t at least as efficient as an exclusive lock in situations where it’s appropriate, it shouldn’t be provided. The alternative is idiotic. But we inhabit an absolutely idiotic reality, so…
|
|
#
?
Oct 2, 2021 02:53
|
|
|
This entire thing I've inherited looks like white-papers promising the silver-bullet, never delivering on it and just being taken at face value. Quite promising indeed.
|
|
#
?
Oct 2, 2021 02:59
|
|
|
Volguus posted:This entire thing I've inherited looks like white-papers promising the silver-bullet, never delivering on it and just being taken at face value. Quite promising indeed.
|
|
#
?
Oct 2, 2021 03:03
|
|
|
Winter Stormer posted:I futzed with it a little further just now and did manage to break my build by setting cmake_version_required < version 3.3. Since that function sets CMake policy versions, I went looking at what policies 3.3 added, and I'm pretty certain you're running into the problem fixed by https://cmake.org/cmake/help/latest/policy/CMP0060.html. Try setting your minimum to be 3.3 or better. I checked out the commit that had me pulling my hair out, set cmake_minimum_required to (VERSION 3.16), and everything worked. That would have saved me about 2 hours of trying things and deep diving on how CMake chooses how to interpret linker flags. Thanks!
|
|
#
?
Oct 2, 2021 03:39
|
|
|
roomforthetuna posted:Cool, thanks! So theoretically you should use reader-writer locks, but in reality you should just use exclusive locks unless you're pretty sure you have a good reason to go readwrite, because the cognitive overhead of the more complicated lock and the probable implementation quality difference are likely bigger than the benefit otherwise. Yeah, so, if you have multiple competing reader threads and a slow but rare critical section in a writer, even a lazily-implemented reader-writer lock will have the inherent advantage of letting the reader threads make progress concurrently. So if that’s the structure of your problem, reader-writer locks can be a good solution. There are four main problems with reader-writer locks:
So I see reader-writer locks as, at best, a temporary stopgap that can be introduced fairly easily and feels a lot better but usually doesn’t pay off very well and comes with a lot of drawbacks; and at worst, an attractive nuisance. rjmccall fucked around with this message at 03:50 on Oct 2, 2021 |
|
#
?
Oct 2, 2021 03:46
|
|
|
roomforthetuna posted:Are the boost synchronization primitives compatible with clang thread safety annotations like GUARDED_BY, LOCKS_EXCLUDED, etc.? Those things are great for preventing you from accidentally trying to double-lock. The clang annotations work with everything that has an appropriate API. You could theoretically even use them for checking things other than locking correctness.
|
|
#
?
Oct 2, 2021 04:50
|
|
|
|
| # ? May 17, 2024 09:34 |
|
|
Plorkyeran posted:The clang annotations work with everything that has an appropriate API. You could theoretically even use them for checking things other than locking correctness. code:Digging around, I found someone suggesting that std::mutex doesn't need wrapping now (if you compile with --stdlib=libc++), but I wasn't able to get it to work in a few minutes of attempting it. code:
|
|
#
?
Oct 2, 2021 13:46
|
|