Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Poopernickel
Oct 28, 2005

electricity bad
Fun Shoe
I'm working on a system that will use RS-485 to poll various sensor data from 25 different devices. Currently, I'm planning the interface around a 115.2k serial port.

My question is: How should I manage the data coming back?

I'm intending to write a GUI with a few running plots that be be set to view any of the data in real-time (I'm planning for 4 updates per second on the GUI, and 8 updates per second coming back from each sensor). I've been thinking about how to structure the management application. Ideally, I'd like to decouple the GUI from the actual data collector (which would be some kind of python app that sent queries out the serial port and logged the responses).

But if I went this route, what would be the best way to get that data from my data collector back into the GUI? What if I also wanted it to log the data to disk (either by using a database, or by flushing once per minute or so and storing the last 24 hours worth)?

Is a SQLite database worth using for this kind of thing? If so, do I need to worry about having one program (the data collector) writing to it and deleting old records while the other program (the GUI) is trying to access it? Would I even be able to get reasonable performance out of it? I don't have a good feel for whether 100 SQL writes and maybe 500-600 SQL reads per second and is totally impractical. Thoughts?

Adbot
ADBOT LOVES YOU

Poopernickel
Oct 28, 2005

electricity bad
Fun Shoe

armorer posted:

Without knowing what kind of hardware the python program would be running on, it is tough to say for sure. That is higher throughput than I would expect to be able to get from SQLite (or mysql) though. You are looking at a lot of overhead if you try to write each measurement individually in it's own transaction. You might be able to batch write those 100 records each second, or structure the table so that a row corresponds to a second of data or something rather than a single measurement. (Also, wouldn't it be 200 writes per second? 25*8?)

I am curious why you expect to have so many reads? I honestly expect that you won't even get close to those performance numbers if you are running this on a standard issue desktop computer. I would suggest writing up a little python app that just loops and writes fake (but representative) data to your intended database target to see how the machine performs.

Basically, the project I'm working on is a big stack of DC-DC converters that step 375VDC down to various user-selectable voltages. We're planning to package 12 of them for a particular niche application, with an RS-485 uplink back to the base station.

The input and output of each converter have a little opto-isolated daughter board with a microcontroller that takes measurements of the voltage, current, and leakage current. The input-side board also sends some control signals to the DC-DC converters. I'd like to be able to monitor all of the various signals simultaneously and record the data. There's no strict requirement on how often I measure them, but I figure I can get 8-10 samples per second on all measurements in the system if I keep the RS-485 bus saturated at 115.2kbps.

I've done plenty of microcontroller firmware in the past, so that's no big deal. However, this time around I'm also in charge of the PC-side host application. And I'm trying to get my head around exactly how I should deal with all of the data (how I should record it, where the serial port interface should live, etc).

On the host PC, it's not too hard to write an application to poll the RS-485 bus fast enough. But I'm scratching my head at how I should structure the rest of the system. I want to be able to log all of the data as it comes in, and then also display some of the data on a host GUI (varied, depending on which screen is up).

My initial thought was that I could do it via SQLite, with my serial application polling data and writing to a database that the GUI could read from at its leisure. But before I go and code it all up, is this approach the 'right' one to take? Would you recommend a different topology?

Poopernickel
Oct 28, 2005

electricity bad
Fun Shoe

armorer posted:

- interesting responses -

Mocking up a little application is a great idea - especially because this is sounding more and more feasible as I dig into it. I could probably do my database operations in chunks as you describe.

I don't need to log forever, so I was planning for the database to have 4 hours worth of data, and then every four hours it could be copied into a folder that always keeps the 24 hours' worth of logs.

So here's the next question - I'm going back and forth right now in my head about how to get info into the GUI. I'm torn between designing the GUI to read from my database (which has some lag issues: when the user turns a power supply on or off, he'll have to wait to see it happen) versus an approach where the logging application spams all of its data to STDOUT, and is launched with STDOUT connected to a pipe to the GUI app.

What do you think about that approach for the GUI accepting info? Is STDOUT fast enough to be used like that? Would the approach work if somebody wanted to port the application to Windows?

Poopernickel
Oct 28, 2005

electricity bad
Fun Shoe

Otto Skorzeny posted:

What do folks here do for unit testing? I'd like to do more of it than I currently am, but there are some issues that I keep running into:

  1. I don't have enough flash/ROM for the tests to live in the main firmware along with the libraries that I or $coworkers[0] wrote and the normal application logic, so I wind up with two binaries (one with the libraries and application logic, and the other with the libraries, the tests and a thin quick+dirty test harness that runs the tests and spits out a report) and I run one then the other.

  2. The unit tests don't catch most memory-related or timing-related issues, which are a minority of bugs by number but a majority of debugging time/effort.

  3. There are some bits of code that I really want to test but I can't think of a good way to unit test - eg. keypress debouncing routines.

  4. due to the above items, I have to do functional testing anyways, which makes it a harder sell to my boss that it's worthwhile to take time to write/maintain the tests because "the functional [read: user] testing will catch the bugs".

There are also some "quis custodiat ipsos custodes" issues wrt. the test harness relying on code to dump the report to the LCD and/or an SD card and that code not being bug-free, but those are minor. It seems like there ought to be a better way to do things.

Whenever possible, I try to design my libraries so that I can compile them natively on my developer machine with GCC. Then I write unit test suites for the libraries with Unity. This doesn't work for libraries that are tightly-coupled to hardware, but can be a real lifesaver for poo poo like command parsers, communication libraries, etc.

I just finished a serial communication library that packetizes, adds error-checking, encodes, and transmits packets of data point-to-point over a UART. It works on a PC as well, because the actual UART byte transmit/receiver functions are left as function pointers. That allowed me to mock them out and write a full unit test suite for the library.

Poopernickel
Oct 28, 2005

electricity bad
Fun Shoe

qsvui posted:

Does it make sense to hit the watchdog from inside a periodic interrupt? Dynamic C (version 9) seems to do this by default and I just find this bizarre.

As a rule of thumb, kicking the watchdog from a periodic interrupt defeats the whole point of having a watchdog. I'd be pretty nervous if one of my coders did that in a project I was overseeing.

I guess having a 'smart' multi-source watchdog routine does make some sense, but only as long as you disable new interrupts during the span of time that you're servicing your watchdog. Then if your watchdog service routine freezes or crashes due to a bug, you can still get the reset.

If you do go that route though, I'd recommend a full unit-test suite for your watchdog routine.

Poopernickel
Oct 28, 2005

electricity bad
Fun Shoe
I actually do a roll-your-own malloc() in a lot of my projects. There's no free() that goes with it - I just use the allocator a bunch of times during initialization, and then never call it during normal operation.

It's a nice tradeoff because it gives most of the flexibility and power of run-time memory allocation, without the overhead of a 'real' malloc. And since I don't actually want dynamically allocated memory after initialization, I don't need a free().

code:
#include <stdint.h>
#include <stdlib.h>

enum
{
    Memory_Size = 2048,
    Word_Size = sizeof(void *),
    Word_Size_Mask = Word_Size - 1
};

static char memory[Memory_Size];
static uint32_t allocated = 0;

void allocator_reset()
{
    uint32_t x;

    for(x = 0; x < Memory_Size; x++) {
        memory[x] = 0;
    }

    allocated = 0;
}

void * allocator(size_t num_bytes)
{
    void * result;

    if ((allocated + num_bytes) >= Memory_Size) {
        return NULL;
    }

    result = (void *)(memory + allocated);
    allocated += num_bytes;

    while (allocated & Word_Size_Mask) {
        allocated++;
    }

    return result;
}

Adbot
ADBOT LOVES YOU

Poopernickel
Oct 28, 2005

electricity bad
Fun Shoe

movax posted:

I was at Xilinx HQ a few months ago, and at least some of their senior guys admitted "yeah, we kind of hosed up" when it came to making Vivado and tools play nice with VCS.

that's p lol, and also matches my experience exactly

  • Locked thread