|
Have you looked into spatial databases? I don't know much about those except that they exist, but it's possible that everything you want to do is implemented as standard queries.
|
# ? May 15, 2022 18:06 |
|
|
# ? Jun 8, 2024 06:14 |
|
cheetah7071 posted:e: this approach also has the advantage that its trivial to restart a run that failed because windows decided to update or something
|
# ? May 15, 2022 18:13 |
ultrafilter posted:Have you looked into spatial databases? I don't know much about those except that they exist, but it's possible that everything you want to do is implemented as standard queries. I think the worry is that the dataset is already large, and it needs to fit and be processed on a single regular high-end laptop, maybe with an external USB drive. That's why the data is compressed. ExcessBLarg! posted:I was going to say, instead of being too concerned about "worrying" memory conditions, maybe implement a checkpointing system so you can restart if you do run out of memory, or for any other reason like the user straight up quitting. Reminds me of the people in the Windows 10 thread crying about it forcing a reboot to install updates, while their scientific processing software is running and the multi-day job is not checkpointed or restartable.
|
|
# ? May 15, 2022 18:15 |
|
nielsm posted:I think the worry is that the dataset is already large, and it needs to fit and be processed on a single regular high-end laptop, maybe with an external USB drive. That's why the data is compressed. this, basically. Holding the entire dataset in memory--or even restructuring the entire dataset into more convenient files--is a non-starter for how the intended audience will be actually using the data ExcessBLarg! posted:I was going to say, instead of being too concerned about "worrying" memory conditions, maybe implement a checkpointing system so you can restart if you do run out of memory, or for any other reason like the user straight up quitting. that makes a lot of sense
|
# ? May 15, 2022 18:18 |
|
nielsm posted:Reminds me of the people in the Windows 10 thread crying about it forcing a reboot to install updates, while their scientific processing software is running and the multi-day job is not checkpointed or restartable. I'm in this picture lol the whole purpose of this project is to take the tiny niche of a niche of a niche that I'm an expert in and make something more user-friendly than the current options that exist. The current options are either 20 years old with command line interfaces and memory management designed for 32-bit systems that take forever to run, or an R package which is honestly excellent but requires the user to know R when a lot of them aren't too solid on what a computer is e: I know it sounds ironic for someone asking basic questions like "why is my code crashing when I manually call a destructor???" to have an ambition like this but seriously the number of people with venn diagram of knowledge needed to even make the attempt is probably <20 worldwide and I'm tired of the tools I have to work with and the inability for anybody who doesn't live in that venn diagram to get anything done without asking me for help cheetah7071 fucked around with this message at 18:24 on May 15, 2022 |
# ? May 15, 2022 18:21 |
|
I want to thank this thread for poking deeper instead of just answering my super-specific questions because while researching quantile approximation algorithms I realized that the fact that my data is numeric and I have known-ahead-of-time upper and lower bounds means I can just store all my data as a histogram instead of as the raw data and get really good estimates of basically every statistic at like 1% of the memory usage
|
# ? May 15, 2022 20:59 |
|
Also, and this is not useful in any way - quote:The desired output is a new grid laid over the same area, with a much finer resolution, where the value in each gridcell is stuff like "the mean z value of all points whose x/y fall within this cell". Basic statistics like that. The resolution of these grids is such that they easily fit in memory on an average machine, but might still be up into the tens of millions of values in a large dataset. In the normal case, where the input files are non-overlapping, the majority of gridcells (~90% of them or so) will only need data from one file. Along the edge between files, they'll need two, and the rare gridcells at the intersection will need four. Am I just missing something, or do you mean a coarser resolution?
|
# ? May 16, 2022 18:46 |
|
Computer viking posted:Also, and this is not useful in any way - No I meant finer than the resolution of the input files. The input files cover areas a few square km; I want metrics calculated every few dozen meters.
|
# ? May 16, 2022 19:38 |
|
cheetah7071 posted:No I meant finer than the resolution of the input files. The input files cover areas a few square km; I want metrics calculated every few dozen meters. Right - it sounded like you were summarizing to a lower resolution. I take it this is more akin to a 2D interpolation? If so: a) I have zero useful insight and b) that does sound sort of everything-intensive.
|
# ? May 16, 2022 21:32 |
|
Does anyone know of a project with a GNU linker script for a Cortex M system that’s run completely in RAM (alongside linux)? This would be more for a reference to make sure I’m doing things properly. Our vendor has given us a demo project with a proprietary toolchain and IDE, but switching to ARM’s would be a lot more flexible. Of course, translating their linker script directly is a no go because it’s linked with their startup code and C library which is being replaced with newlib. The other core will copy a firmware image to RAM and release the M core from reset. I think I’ll just take a standard flash linker script and omit both the .data relocation and NOLOAD from the .bss section so memory should be good to go when Reset_Handler is called. Sound right? edit: Nevermind, this system is structured the same as a flash based microcontroller, only the "flash" area is actually SRAM. I'll do it the normal way and avoid any possible headaches about rules for the different memory regions. yippee cahier fucked around with this message at 18:48 on May 23, 2022 |
# ? May 23, 2022 16:08 |
|
You should be able to just get rid of all the AT> directives so load and run addresses are the same, then change startup assembly to not do any copying or zeroing bss
|
# ? May 23, 2022 20:34 |
|
Is nix the best "package manager" in 2022? Trying not to go crazy with setting up a personal project and forgot how bad C++ was without everything being managed by bazel. Also curious about build systems 👀 it looks like zig is one of the better compilers for cross platform stuffs at the moment. I just want to gently caress around with SDL2 and WebAssembky but almost want to switch languages because of the headache sailormoon fucked around with this message at 05:41 on May 25, 2022 |
# ? May 25, 2022 05:28 |
|
nix is idiosyncratic and best practices for it are in flux, but it's really good at its job. Highly recommended for managing build environment and dependencies, if you only need Linux.
Ralith fucked around with this message at 23:15 on May 25, 2022 |
# ? May 25, 2022 22:02 |
|
I'm bumping into a truly bizarre error I have code that is working perfectly well in my old tests, which were written in the visual studio testing framework. I wrote them in that framework mostly out of inertia because that was the framework I learned when I first learned C++, because it was built in and simple. But it's kind of clunky and not OS-portable, so I finally got off my rear end and started porting my tests to Boost's framework. I'm just providing the context here to say that this is already tested code and I'm not writing new tests for new code. The truly bizarre error is that I have a bunch of constants in one of my headers like code:
code:
Are Boost's tests just like, cursed? Do constants just not work in them for some reason???? It's not even like some secret corner of the code can be redefining them before my cout, because they're declared const. I'm probably gonna just switch to Google tests because I've barely even started porting my tests over so it's a pretty low sunk cost, and I don't really care to try to deal with inscrutable nonsense on already-working code. But I'd at least be interested to hear why this bizarre behavior is even happening, if anyone knows. cheetah7071 fucked around with this message at 04:02 on May 27, 2022 |
# ? May 27, 2022 03:53 |
|
Is the thing that sees then as uninitialized memory also called from a global constructor? And is that constructor in another translation unit? Cause the order that those run is arbitrary and will change with different compilers or filesystems, but everything will work fine if it happens to be the order you intended
|
# ? May 27, 2022 03:59 |
|
Ah, that is it! I was lazily defining my test variables directly as global variables in the test .cpp because the version of Boost's macros I was using wants them to be directly passed in as macro parameters, so it's kind of inevitable you'll be defining them as global constructors. I never really use global variables except for tiny little independent constants in my actual app, so that explains why everything is working normally there. To be honest, it would be a pain in the rear end to fix this problem, which suggests looking into Google tests, anyways
|
# ? May 27, 2022 04:05 |
|
You might be able to use constexpr to force the initialization to happen before global constructors kick in.
|
# ? May 27, 2022 04:40 |
|
Yep, that looks very constexprable.
|
# ? May 27, 2022 13:03 |
|
I recently stumbled over a kinda-similar problem, where I've defined some quick & dirty non-const-globals. https://www.learncpp.com/cpp-tutorial/why-non-const-global-variables-are-evil/ explains it all. Edit: I'm not positioning this for a long-term-solution or a solution for constants. Wipfmetz fucked around with this message at 05:56 on May 30, 2022 |
# ? May 30, 2022 05:41 |
|
Google tests seem a lot easier to use than Boost tests anyways, and completely sidestep the issue The thing that caused me to want to use global variables in Boost tests was the ability to do automatic loops over test cases. Like if you were testing your addition function on a custom class, this code works: code:
I sort of got the impression from the documentation that Boost's testing library is powerful to the point of being impenetrable. I'm much happier having switched to Google tests even setting aside the issue of global variables. The functionality is fairly basic but it doesn't need to be complicated, it needs to be understandable. And the output to the console when something goes wrong is very well-formatted, which is a big bonus. Probably not news to anyone who's been using C++ longer than I have, but it's a big improvement over visual studio's native unit tests, which are kind of awful and I picked them more or less at random when I first started with the language, and then kept using them out of inertia. cheetah7071 fucked around with this message at 06:26 on May 30, 2022 |
# ? May 30, 2022 06:06 |
|
Wipfmetz posted:I recently stumbled over a kinda-similar problem, where I've defined some quick & dirty non-const-globals. code:
|
# ? May 30, 2022 12:45 |
|
I want to do something with dates and times but I don't know where to start. I want to create yearly "events" for which I can specify a month/day/time that they start. Then I need to be able to tell where those events are in relation to the current time. I might even want to do more advanced things like "the first day of spring." Which library should I be using? Can std::chrono help me do this? I can't really understand anything in there. I'm using C++17. I may just wind up manually entering timestamps for the events for the next 10 years or something. But if there's a library out there that can make this easy then I'd like to do things properly.
|
# ? May 30, 2022 19:10 |
|
baby puzzle posted:I want to do something with dates and times but I don't know where to start. Honestly, std::chrono is not going to help you unless you move to C++20 as all of the important calendar functions you want weren't implemented until then. With C++17, all you can really do is grab time points from your system clock and calculate durations between two times. The calendar functionality is what you need to create time points based on dates. It lets you construct time points based off month/day/year, the second Tuesday of a month, etc, and even has some basic functionality like figuring out what day of the week your time is on. However, it cannot do anything more advanced like finding out what day spring starts. But it could do everything else you need. But again, C++20 required. And sorry, I don't know of any date library that would help you off the top of my head. And sadly cppreference.com doesn't list any either.
|
# ? May 30, 2022 19:56 |
|
baby puzzle posted:I want to do something with dates and times but I don't know where to start. You might want to check out boost's calendar for this, I think that would work.
|
# ? May 30, 2022 20:32 |
|
The c++20 date stuff is available as a standalone library.
|
# ? May 30, 2022 20:43 |
|
You can use the C time facilities like struct tm.
|
# ? May 30, 2022 22:36 |
|
Predicting the equinoxes is complicated and involves big tables of observed solar data anyway, so you're probably best off just tabling that
|
# ? May 30, 2022 23:12 |
|
Thanks for the answers.Foxfire_ posted:Predicting the equinoxes is complicated and involves big tables of observed solar data anyway, so you're probably best off just tabling that You are right about that. I'll just google a few years worth and then wonder why the events stopped happening all of the sudden when the table runs out.
|
# ? May 31, 2022 14:04 |
|
baby puzzle posted:Thanks for the answers. I believe the traditional approach is to include a local copy of a library that completely dwarfs your own code, running a complicated set of calculations on every startup.
|
# ? May 31, 2022 21:59 |
|
baby puzzle posted:Thanks for the answers. Makes me wonder if you could get a support contract which involved doing this every two years
|
# ? May 31, 2022 22:04 |
|
Just pull in all of ICU as a dep for your 100kb library it's nbd.
|
# ? May 31, 2022 22:57 |
|
I have a little problem with a program of mine that's built with vcpkg's libraries, uses boost's ASIO and includes openssl. Two days ago Fedora updated the crypto-policies packages and among other changes they have updated /usr/share/crypto-policies/DEFAULT/opensslcnf.txt to include the following thing: pre:[openssl_init] alg_section = evp_properties [evp_properties] rh-allow-sha1-signatures = yes pre:terminate called after throwing an instance of 'boost::wrapexcept<boost::system::system_error>' what(): context: unknown option (digital envelope routines) [asio.ssl:50331817] Is there a way to tell ASIO's SSL context to ignore unknown options in the policy files? What could I do? submit a patch to openssl? vcpkg? Is it a version mismatch? Should I use just the openssl libraries installed on the OS?
|
# ? Jun 11, 2022 18:35 |
Use the OS-supplied libraries, or configure your own library so it doesn't read a config file possibly written for a different, incompatible version.
|
|
# ? Jun 11, 2022 21:59 |
|
Hmm, I was afraid of that. Ended up copying the openssl port from the root of vcpkg into my project's dir, told vcpkg that that's where I have overlays and changed it to always set no-autoload-config option. I have no idea if this improves or decreases security, but at least doesn't load potential broken (for this release) configuration files.
|
# ? Jun 12, 2022 03:41 |
|
Does anyone know of a place with actual documentation/information on generating LLVM debug information using its C++ API? I've been writing my own language and while I'm used to LLVM's poor documentation, information on generating debug info seems mostly non-existent besides its doxygen and a single, very brief section in the Kaleidoscope tutorial. I'm guessing I'll end up having to go through the LLVM sources to figure out how that stuff actually works, but figured I'd ask first. Falcorum fucked around with this message at 23:23 on Jun 13, 2022 |
# ? Jun 13, 2022 20:36 |
|
I have a question for testing a program written in C in Linux that uses system calls. I have this fully working loop that forks off a bunch of worker processes from the parent: code:
Is there any way I can inject an arbitrary failure into a system call, like saying that after x forks I want the next one to fail?
|
# ? Jun 15, 2022 14:41 |
|
BattleMaster posted:I have a question for testing a program written in C in Linux that uses system calls. Easiest thing is to have your test code set numWorkers to intmax.
|
# ? Jun 15, 2022 14:45 |
|
On Linux you could use ulimit to cap the number of processes for a user.
|
# ? Jun 15, 2022 14:50 |
|
leper khan posted:Easiest thing is to have your test code set numWorkers to intmax. Thanks! I had a bad feeling about spawning so many processes that each peg a full core but it doesn't take long to fail and it showed me that I messed up a couple of things in the kill loop Though if there are ways to make system calls arbitrarily fail that could still help me in the future.
|
# ? Jun 15, 2022 14:55 |
|
|
# ? Jun 8, 2024 06:14 |
|
You can strace a program and inject failures into specific system calls using -e inject. Not really something you can easily fit into a unit testing framework or similar, but if you're after once-off testing then it seems like exactly what you want.
|
# ? Jun 15, 2022 15:06 |