VikingofRock posted:

CommunistPancake posted:

so a monad is... mapreduce?

VikingofRock posted:

Thank you both very much for the feedback. Yeah, I thought my design was a little lock heavy, but my thinking was that each lock is only going to be held for the length of a lookup so it's not too bad. I think rjmccall your design with the std::optional is better though (although I am stuck on C++14 so I'll be using boost::optional). I'll give that a shot tomorrow.

meatpotato posted:

I'm the terrible programmer who has managed to avoid writing anything concurrent for the last four years out of fear and ignorance but finally need to learn how to do it, kind of.

Maybe you guys can give me some tips for my situation, I think it might be simple. I'm using C++11.

Right now I have this:

reader_thing is waiting for data most of the time from a hardware peripheral (reader_thing is sleeping on a select() or something underneath). When data is available reader_thing unblocks and is given a pointer to that data. After doing some things, reader_thing loops and waits for data again.

What I want to add is the following:

consumer_thing is a thing that waits around until reader_thing has its pointer to new data. After consumer_thing is unblocked it can also do things with that data from reader_thing (read only). Eventually a consumer_thing loops and blocks again waiting for reader_thing to get new data. Hopefully consumer_thing didn't take too much time doing things and miss some data from reader_thing!

rjmccall posted:

so, this is actually really important to the design. we can assume that it's undesirable for a consumer to miss some data. is it unacceptable? if it is acceptable, does the consumer at least need to be told that it's missed something?

also, how important is it to avoid copies? what about allocating memory?

meatpotato posted:

I'm the terrible programmer who has managed to avoid writing anything concurrent for the last four years out of fear and ignorance but finally need to learn how to do it, kind of.
...

meatpotato posted:

I'm honestly not sure how important it is to avoid copies, this code is running on a ~500 MHz MIPS SoC with 64 MB RAM, think budget wifi router level memory and speed

Jabor posted:

It might be conceptually easier to treat it as an exercise in concurrent reference counting and just use a counter to detect missed updates.

cinci zoo sniper posted:

jetbrains officially support rust now

rjmccall posted:

i was considering this but i couldn't quite convince myself that the use-after-"free" wasn't an insurmountable problem. normally that kind of approach is a non-starter because the memory actually is freed, which invalidates racing consumers' attempts to check the refcount. here the buffer isn't actually freed but by returning it to the buffer pool i'm not sure you can't see similar effects. at the very least, the reader must be aware when preparing a new buffer that there might be consumers with a stale handle to this buffer that just haven't yet checked its refcount, so the act of setting the refcount to 1 actually publishes the buffer even before it is written to the shared reference. i think that might end up being ok as long as you make the refcount checks ordered, but it would be very easy to disturb in ways that will badly break it, and you must literally never free a buffer

MALE SHOEGAZE posted:

I've been using the intellij rust plugin for about a month and I can confirm that it's good. definitely more powerful than the current RLS setups.

only weird thing is that it's kinda fucky about text in a way that can only be experienced. i think it might be a conflict with intellij vim but I'm not turning off vim mode to find out

Jabor posted:

yeah, in the extreme case the buffer could be recycled all the way through the pool and be repopulated with new data in between getting the reference and incrementing the refcount. not actually a problem in this specific scenario though, since it's not actually any worse than if the consumer goes into a deep sleep immediately after incrementing the refcount instead of immediately before.

not being able to free buffers sucks. if you add another layer of indirection (or if your heap supports in-place reallocs) you can still free the actual data buffer and just keep the metadata though.

i wonder if you could fix the stale pointer issue (and be able to free buffers) if you had an atomic primitive big enough for the buffer pointer plus a count of consumers? each consumer increments the refcount in the shared ref, and to release the buffer it increments a field for "released references" in the buffer metadata. when the shared reference is replaced, the reader atomically subtracts the consumer count from the released references, and the buffer is disposed of (returned to the pool, freed, whatever) when it rises from -1 to 0. essentially the bigger atomic primitive guarantees each consumer is in one of two states, either it has the pointer and has incremented the refcount, or it doesn't have the pointer and has not incremented the refcount, no divergences.

meatpotato posted:

jesus christ concurrency is truly awful

thanks to both of you for the ideas, though I still don't understand either well enough to write it...

I'll keep reading your posts over and over until it clicks

maybe you can tell me what's bad about this, which is what I wrote before you replied:

code:

#include <chrono>
#include <iostream>
#include <thread>
#include <vector>

struct Buffer {
    std::shared_ptr<std::vector<int32_t>> data;

    std::mutex lock;
    std::condition_variable new_data;

    void update(std::shared_ptr<std::vector<int32_t>> samples){
        std::unique_lock<std::mutex> l(lock);
        data = samples;
        new_data.notify_all();
    }

    std::shared_ptr<std::vector<int32_t>> fetch(){
        std::unique_lock<std::mutex> l(lock);
        new_data.wait(l);
        return data;
    }
};

std::vector<int32_t> get_data()
{
    static int32_t iteration = 0;
    return std::vector<int32_t>(2048, iteration++);
}

void prod(Buffer& buf)
{
    while (1) {
        std::this_thread::sleep_for(std::chrono::milliseconds(250));
        std::cout << "Producing data\n";
        auto data = std::make_shared<std::vector<int32_t>>(get_data());
        buf.update(data);
    }
}

void consume(int id, Buffer& buf)
{
    while (1) {
        auto data = buf.fetch();
        std::this_thread::sleep_for(std::chrono::milliseconds(10));
        std::cout << "Consumer #" << id << " read data" << '\n';
        std::cout << "data[0] = " << data->at(0) << '\n';
    }
}

int main(int argc, char* argv[])
{
    std::cout << "Hello, world!\n";

    Buffer buf;

    std::thread producer(prod, std::ref(buf));
    std::thread c1(consume, 1, std::ref(buf));
    std::thread c2(consume, 2, std::ref(buf));
    std::thread c3(consume, 3, std::ref(buf));
    std::thread c4(consume, 4, std::ref(buf));

    c4.join();
    c3.join();
    c2.join();
    c1.join();
    producer.join();

    return 0;
}

I ripped a lot of this off some blog, not sure if I actually need to use std::ref or if I'm using mutexes, locks or condition variables correctly. It seems to work but consumers don't know if they missed anything. Also, it eats more CPU than I expect from a glance at 'top' meaning I'm probably doing something wrong.

Like I said, I wrote this before you replied... I'm not trying to be the goon in the well replacing your advice with my own lovely plans

Thanks again for the help!

ratbert90 posted:

How about you stop using a text editor from the 70's?

ratbert90 posted:

How about you stop using a text editor from the 70's?

feedmegin posted:

Emacs 4 lyfe bro

meatpotato posted:

jesus christ concurrency is truly awful

thanks to both of you for the ideas, though I still don't understand either well enough to write it...

I'll keep reading your posts over and over until it clicks

maybe you can tell me what's bad about this, which is what I wrote before you replied:

code:

#include <chrono>
#include <iostream>
#include <thread>
#include <vector>

struct Buffer {
    std::shared_ptr<std::vector<int32_t>> data;

    std::mutex lock;
    std::condition_variable new_data;

    void update(std::shared_ptr<std::vector<int32_t>> samples){
        std::unique_lock<std::mutex> l(lock);
        data = samples;
        new_data.notify_all();
    }

    std::shared_ptr<std::vector<int32_t>> fetch(){
        std::unique_lock<std::mutex> l(lock);
        new_data.wait(l);
        return data;
    }
};

std::vector<int32_t> get_data()
{
    static int32_t iteration = 0;
    return std::vector<int32_t>(2048, iteration++);
}

void prod(Buffer& buf)
{
    while (1) {
        std::this_thread::sleep_for(std::chrono::milliseconds(250));
        std::cout << "Producing data\n";
        auto data = std::make_shared<std::vector<int32_t>>(get_data());
        buf.update(data);
    }
}

void consume(int id, Buffer& buf)
{
    while (1) {
        auto data = buf.fetch();
        std::this_thread::sleep_for(std::chrono::milliseconds(10));
        std::cout << "Consumer #" << id << " read data" << '\n';
        std::cout << "data[0] = " << data->at(0) << '\n';
    }
}

int main(int argc, char* argv[])
{
    std::cout << "Hello, world!\n";

    Buffer buf;

    std::thread producer(prod, std::ref(buf));
    std::thread c1(consume, 1, std::ref(buf));
    std::thread c2(consume, 2, std::ref(buf));
    std::thread c3(consume, 3, std::ref(buf));
    std::thread c4(consume, 4, std::ref(buf));

    c4.join();
    c3.join();
    c2.join();
    c1.join();
    producer.join();

    return 0;
}

I ripped a lot of this off some blog, not sure if I actually need to use std::ref or if I'm using mutexes, locks or condition variables correctly. It seems to work but consumers don't know if they missed anything. Also, it eats more CPU than I expect from a glance at 'top' meaning I'm probably doing something wrong.

Like I said, I wrote this before you replied... I'm not trying to be the goon in the well replacing your advice with my own lovely plans

Thanks again for the help!

leper khan posted:

I usually approach these things as a game. one player tries to make things blow up horribly (e.g. by pre-empting threads until very certain timings occur) and another player places locks to limit the actions of the attacker. in development, you get to be the attacker. in production, it's the OS and malicious agents.

meatpotato posted:

jesus christ concurrency is truly awful

thanks to both of you for the ideas, though I still don't understand either well enough to write it...

suffix posted:

https://deadlockempire.github.io/
this is pretty good to put the fear in you if you've ever thought oh threading isn't so hard just use a mutex

rjmccall posted:

yeah, so i was just here to have fun puzzling out a lock-free, copy-free algorithm. i think maybe you're struggling with a lot of things besides concurrency that i'm not going to have the time to explain, sorry

CRIP EATIN BREAD posted:

you're constantly reassigning the data over and over again instead of just filling a buffer. you're better to read from hardware and write to a shared buffer.

akadajet posted:

this

necrotic posted:

wrong modal editing is the best

Soricidus posted:

I too love when i type something in and then realise a couple seconds later that I wasn't in insert mode. keeps life interesting

necrotic posted:

wrong modal editing is the best