|
Rocko Bonaparte posted:I have multiple variables packed together in a memory buffer that I'd like to assign to a struct. Are there any gotchas for this beyond some basic memory alignment stuff? What I want to assume is: How is the data laid out in this buffer? Are you trying to cast the buffer to the struct? You can pack the structs to remove the padding and force the alignment to 1 I’m assuming your architecture will handle whatever unaligned reads you are doing
|
# ? Sep 2, 2021 23:14 |
|
|
# ? Jun 8, 2024 07:25 |
|
Rocko Bonaparte posted:I have multiple variables packed together in a memory buffer that I'd like to assign to a struct. Are there any gotchas for this beyond some basic memory alignment stuff? What I want to assume is: The only gotchas I’m aware of is dealing with pointer sizes on 32- vs 64-bit and dealing with the viable pointer if you’ve got virtual functions. I’m also not sure how standardized the alignment and padding is, but I’ve implemented something along these lines for game data serialization before and I know it worked fine for the 360, PS3 and PC at the time.
|
# ? Sep 2, 2021 23:23 |
|
As mentioned before, this works fine as long as you know the struct alignment/packing. That can be adjusted in most compilers using pre:#pragma pack But I've definitely done things like that, you just need to be sure you've got it right on all platforms/compilers.
|
# ? Sep 2, 2021 23:39 |
|
I would memcpy each member into the struct from the packed byte stream, and let the compiler sort out the alignment and padding.
|
# ? Sep 2, 2021 23:49 |
|
It is undefined behavior to alias a char buffer as a struct by casting a pointer (the reverse is legal because there's a special exception for char*) It will likely produce correct behavior in many cases and gcc specifically has a switch (-fno-strict-aliasing) to make it safe in all cases by disabling the optimizations that use the no-aliasing assumption. The safe way to do it is with a memcpy(), which your compiler may or may not be smart enough to optimize out if it's equivalent to an alias
|
# ? Sep 3, 2021 00:03 |
|
Aliasing like that in C is legal but if you use a static analyzer it will probably complain. Like everyone else, I'd just use memcpy.
|
# ? Sep 3, 2021 00:10 |
|
It's not legal in C either. It's specified in section 6.5.7 in C11. Example of something that will break on it The output will change depending on whether you add -fno-strict-aliasing or not. Without it, the optimizer assumes that a float* and a int* never point to the same memory, so that it doesn't have to reload the other one after a write (the generated assembly is kind of dumb for an example, because it sees the whole program and can keep folding until the result is a constant. Look at the outputted numbers). There is a complicated exception for structs that share common prefixes that gets used by BSD sockets. I don't really understand its subtilties and don't use it Doing something like char aBigDataBuffer[4096]; SomeStruct* someStruct = (SomeStruct*)aBigDataBuffer; is more likely to happen to work, but it's violating assumptions in the optimizer even if you make all the alignment issues work out. You may end up with behavior where changes in the buffer are sometimes not reflected in the struct (because the optimizer can assume that changing a char[] doesn't invalidate previous reads of a SomeStruct's fields because they're different types) Foxfire_ fucked around with this message at 05:03 on Sep 3, 2021 |
# ? Sep 3, 2021 04:59 |
|
Thanks for the clarification. I still fall into the trap that if the compiler doesn't complain, and the assembly looks OK, it must be legal!
|
# ? Sep 3, 2021 15:56 |
|
Foxfire_ posted:Doing something like lol wait is this not actually legal/well defined? our I/O code does this all over the place
|
# ? Sep 3, 2021 20:57 |
|
Yep, that's undefined behavior. The legal way to do it is to do a memcpy() because char* pointers are permitted to alias anything (but not the reverse). A good optimizer will often eliminate the actual copy, but if you're stuck with a less fancy compiler you end up having to decide between technically wrong code (that will probably work if your optimizer isn't sophisticated enough to use those aliasing assumptions anyway) or extra copies and temporary space Example with gcc eliding a memcpy()
|
# ? Sep 3, 2021 23:00 |
|
Can someone point me to a crash course in autotools? I'm hacking away at https://github.com/samtools/htslib, experimenting with adding zstd as a compression format over libz or libdeflate, and I can't get the error message to yell at me if libzstd-dev isn't installed and available: I've added code:
|
# ? Sep 3, 2021 23:15 |
|
Foxfire_ posted:Yep, that's undefined behavior. Very interesting. I mean we do know it works, but no wonder the static analysis tools we recently implemented is so grumpy about it
|
# ? Sep 3, 2021 23:29 |
|
You're missing a $ in if test zstd_devel = missing and so are just comparing some string literals. There's also a bunch of nitpicky autotools things that that code isn't doing properly but I'm assuming you copied and pasted that from existing checks and so you should just stick with matching the existing style.
|
# ? Sep 3, 2021 23:31 |
|
Plorkyeran posted:You're missing a $ in if test zstd_devel = missing and so are just comparing some string literals. I am absolutely interested in hearing about the nitpicky autotools things!
|
# ? Sep 3, 2021 23:33 |
|
Is there a way to refer to a specific member function of a class, without referring to any specific instantiation of that function? Or, alternatively, any way to concisely do what I'm doing here without the macro:code:
|
# ? Sep 4, 2021 00:03 |
|
cheetah7071 posted:Is there a way to refer to a specific member function of a class, without referring to any specific instantiation of that function? Or, alternatively, any way to concisely do what I'm doing here without the macro: Not clear why you wouldn't be able to do what you're trying to do with a function pointer.
|
# ? Sep 4, 2021 00:06 |
|
leper khan posted:Not clear why you wouldn't be able to do what you're trying to do with a function pointer. It might be entirely possible but I couldn't figure it out when I first tried. Basically, the GridMetric class has a whole bunch of functions, of which a subset are relevant any time I run this code. I need all of my GridMetric objects to call the ones in a list I specify, which isn't known at compile time, and I want to do that concisely--i.e., ideally with one line per function at most, which I currently accomplish with the macro. I'd prefer to call a function rather than a macro, obviously, but I wasn't able to figure out how to get a function pointer to the member function in a way that all of my GridMetric objects could refer to it (which is trivial with a macro because it's just manipulating text). Maybe it's really easy and I just wasn't googling the right things to find the syntax cheetah7071 fucked around with this message at 00:20 on Sep 4, 2021 |
# ? Sep 4, 2021 00:13 |
|
Twerk from Home posted:I am absolutely interested in hearing about the nitpicky autotools things! Always use AS_IF rather than shell if. AS_IF just expands to if but it results in macro expansion happening in a different order. This almost never matters but it's easier to just always use AS_IF than to determine if it matters. There's some convoluted idioms around handling various values of --enable-foo=bar. The libcurl check in the file you linked is a decent example: if it's passed --enable-libcurl=check then it'll warn if libcurl isn't found instead of erroring. Similarly, each of the checks sets libcurl_devel to a human-readable string in the error case which is then used in the error message. It's also checking that libcurl_devel is ok rather than for the specific error string as there's multiple error cases (which you currently have wrong). You're probably missing a AC_DEFINE([HAVE_LIBZSTD]). This one is a bit less nitpicky....
|
# ? Sep 4, 2021 00:36 |
|
cheetah7071 posted:It might be entirely possible but I couldn't figure it out when I first tried. The pointer-to-member-function syntax is a pain and a half, but if I'm following you right you're looking to do something like C++ code:
Xerophyte fucked around with this message at 14:40 on Sep 4, 2021 |
# ? Sep 4, 2021 01:30 |
|
Thanks, that was exactly what I was looking for, and the messiness involved is probably why I couldn't figure it out on my own e: ::* is a spicy bit of syntax cheetah7071 fucked around with this message at 02:05 on Sep 4, 2021 |
# ? Sep 4, 2021 01:56 |
|
cheetah7071 posted:Thanks, that was exactly what I was looking for, and the messiness involved is probably why I couldn't figure it out on my own You could just get into a habit of using std::function. It is slightly less messy! C++ code:
|
# ? Sep 4, 2021 02:36 |
|
Nalin posted:You could just get into a habit of using std::function. It is slightly less messy! I actually tried something like that when I was first writing this, and it didn't work so I assumed you couldn't reference member function pointers that way. I must have just made some minor error throwing it all off.
|
# ? Sep 4, 2021 02:53 |
|
Re: This about concatenating the line number to a token: https://stackoverflow.com/questions/1597007/creating-c-macro-with-and-line-token-concatenation-with-positioning-macr code:
|
# ? Sep 5, 2021 02:04 |
|
__LINE__ is not substituted when UNIQUE is expanded because it doesn't exist yet at that point - it's only created as the result of expanding UNIQUE. Given this: code:
code:
code:
code:
|
# ? Sep 5, 2021 04:19 |
|
C macro “evaluation order” is kindof weird. When expansion identifies a use of a function-like macro, it doesn’t immediately expand macros in the arguments. Therefore, the argument token sequence passed by UNIQUE is just the raw identifier __LINE__. But within the “callee”, argument substitution does happen immediately, and that expansion normally immediately recursively expands macros in the argument. So the parameter in TOKENPASTE2 is replaced by the macro expansion of the argument token sequence, i.e. 42 or whatever, and then that becomes the argument received by TOKENPASTE. But as a special case, that immediate expansion of arguments is not done during argument substitution when the parameter was immediately adjacent to an operator like ##. So if UNIQUE used TOKENPASTE directly, the parameter in TOKENPASTE would just be replaced with __LINE__, and that’s what would be token-concatenated.
|
# ? Sep 5, 2021 11:11 |
|
Okay so going back to the question of pulling data from a char buffer, what's the best way to actually handle something like this?code:
Currently this is actually handled in the code by declaring DataStruct with a bunch of char arrays instead of the full types, and then just casting it into the full type when required, as below (with simplified operations of course, these do need to be accessed separately): code:
The options as I see it are:
Is there a serious argument to made for one approach over the other?
|
# ? Sep 8, 2021 00:02 |
|
(1) is equivalent to what you are doing now, just without the undefined behavior. It will crash if unaligned loads don't work on your architecture. (2) will work even if unaligned loads don't. I would either do (1) or something like: code:
GCC optimizer is willing to elide all of that into unaligned loads directly from the buffer (presumably it wouldn't if you did a lot of math using it and it decided the load cost was worth a copy) The casting thing you're doing now looks error prone and hard to maintain. Foxfire_ fucked around with this message at 02:03 on Sep 8, 2021 |
# ? Sep 8, 2021 01:46 |
|
I would use (2)
|
# ? Sep 8, 2021 01:58 |
|
16 members isn’t too bad. Maybe look for opportunities to have arrays of values or even substructures that you can wrap in loops. Or helper functions that advance the pointer so you can have 16 straightforward lines of code.
|
# ? Sep 8, 2021 03:04 |
|
I would leave it the way it is and tell the static analyzer to gently caress off But that's just me.
|
# ? Sep 8, 2021 18:00 |
|
Very helpful, thanks for the input all!
|
# ? Sep 8, 2021 20:36 |
|
A colleague has bumped into an interesting issue. For reasons, he needs a way to force the compiler to use the stack to pass arguments to certain functions. My first thought was something like forcing cdecl, but this is x86-64, so that doesn't work. The only way I see this can work is either : - by adding six (5 for members) garbage arguments to the function, maybe with some macro magic to make it palatable, or - working with varargs. Either way, it's messy. Anyone has a better way?
|
# ? Sep 13, 2021 13:03 |
|
Beef posted:A colleague has bumped into an interesting issue. For reasons, he needs a way to force the compiler to use the stack to pass arguments to certain functions. My first thought was something like forcing cdecl, but this is x86-64, so that doesn't work. I have to know why this is a thing, please expound
|
# ? Sep 13, 2021 13:06 |
|
Sweeper posted:I have to know why this is a thing, please expound This, and a preemptive please don't mix abis
|
# ? Sep 13, 2021 13:37 |
|
Weird research co-processor thing. He says he can pass it a function pointer and its stack, and let it do its thing. I suggested doing something like pthread_create where the function takes a struct of args, but that's apparently too labor intensive when a bunch of existing code has to be adapted. (imo, this is what happens when no compiler person gets involved)
|
# ? Sep 13, 2021 13:47 |
Write a generator for asm call thunks.
|
|
# ? Sep 13, 2021 13:49 |
|
Write what where now? Are you suggesting a compiler modification or a C/C++ function to generate the asm string? If it is the latter, that sounds like it requires compile-time information.
|
# ? Sep 13, 2021 14:18 |
|
I assume you've got some thunk functions that take in parameters, and then instead of doing the work themselves they invoke a routine in the coprocessor. And you want to avoid shuffling things around unnecessarily, so you'd prefer the caller pass all the arguments on the stack (which would give you a neat array of all the parameters instead of having some of them in registers), so you can just pass a pointer to your own stack to the coprocessor? I'd be inclined to just copy everything into an array and set whatever attributes would heavily encourage the whole thunk function to be inlined. Let the optimizer take care of putting those values in the right spots straight away instead of copying them around excessively. Also doing this explicitly instead of playing calling convention games means you can more easily do things like "put the coprocessor arguments somewhere that isn't on the thread stack, so we can do other stuff while the coprocessor is working without having to worry about corrupting its arguments".
|
# ? Sep 13, 2021 14:33 |
|
Yeah, for that I would allocate some heap space for the coprocessor to use as its stack and directly place the arguments in that memory where you want them. Sharing stacks here sounds like borrowing hard-to-debug trouble. (For one thing, what happens if you’re near the edge of the “main” stack’s allocated memory and the co-processor need more? Does it run in the right process context for the kernel to allocate more correctly? What if your main task gets scheduled away while the co-processor is running, as is likely if you’re sleeping on something waiting for the result.) Subjunctive fucked around with this message at 18:00 on Sep 13, 2021 |
# ? Sep 13, 2021 17:57 |
|
|
# ? Jun 8, 2024 07:25 |
|
Yeah, there’s no apparent reason this is tied to argument passing except it worked that way on i386 and they’re being lazy about the port. Write the “arguments” into memory and send a pointer to that to the coprocessor.
|
# ? Sep 13, 2021 18:00 |