Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
JawnV6
Jul 4, 2004

So hot ...

ElCondemn posted:

My understanding of SSDs, linux and hard drive controllers is counter to what's being described, powering off an rpi suddenly should not cause system corruption ever, at worst you'd lose the last few seconds of data being written due to caching (does the rpi even have a disk cache?)
Can I assume this list is exhaustive and you're totally ignorant of actually managing Flash devices in an embedded context with limited power? Absolutely blind to anything below the abstraction provided by a SSD? Great, we can probably help teach you something, but pretending we're in desktop land with stable rails and 1/0 being the lowest abstraction isn't going to help you here.

Have you ever had to write magic values to flash to ensure writes are actually sinking? Have you ever had to count attempts to write and ensured the error handler fit into RAM to make sure the part didn't reboot when it tried to load code from flash in a busy state? These are all real scenarios that exist with flash that are all way below most kernel drivers.

ElCondemn posted:

I was mostly just pointing out that writes don't happen how it's been described in this thread. In fact all operating systems and kernels have been aware of the limitations of flash memory for a while. For instance, the linux kernel knows to write to a new block instead of overwriting a previous block (look up wear leveling) so you're more likely to have your original data intact than is likely to lose any data at all. Of course if you power down during a write it will fail to write, but it would not cause system wide corruption or even affect any data that was previously on disk.
For all the accusations that we're blind to the complexities hidden behind an abstraction you're quite content to sit way off in kernel land and pretend this hardware Just Works. I referred to linux's method to ensure block writes appear atomic above, so it's kinda patronizing to see it replayed with less fidelity.

The video was queued up to an explanation that the SD card is coalescing writes and not actually sinking them to the nonvolatile as aggressively as you're suggesting. Did you lose the timestamp and start at 0?

Adbot
ADBOT LOVES YOU

ElCondemn
Aug 7, 2005


mod sassinator posted:

No, SD cards are not like other disk drives ('other' being spinning hard disks) at all. They might look like a hard drive as far as the operating system is concerned but the physical characteristics are completely different. Entire cells of flash have to erased at a time vs. a hard disk that can zero out specific locations. Erasing data is order of magnitudes slower for flash vs. hard drives. Random access is orders of magnitude faster for flash vs. hard drive. The video I linked to explains these differences in great detail and how SD card controllers try to deal with the impedance mismatch that filesystems and OS' have been built and optimized for years to work with that characteristics of spinning disks vs. flash storage.

You're taking what I'm saying completely out of context, I'm very aware of how flash memory differs from traditional spinning platter disks. Nothing you've said or posted has explained away the weird "when you read data it can be corrupted" idea, also the concept of a "lookup table" being lost as if it only exists in memory is weird and wrong too.

mod sassinator posted:

This is very bad advice and you are doing more harm continuing to say it. People new to the Pi absolutely should not pull the power from the Pi without shutting down the operating system. Please stop saying otherwise.

I guess we'll just have to agree to disagree.

ElCondemn
Aug 7, 2005


JawnV6 posted:

Have you ever had to write magic values to flash to ensure writes are actually sinking? Have you ever had to count attempts to write and ensured the error handler fit into RAM to make sure the part didn't reboot when it tried to load code from flash in a busy state? These are all real scenarios that exist with flash that are all way below most kernel drivers.

Most of my experience is with SSDs in enterprise equipment, where data integrity, read/write speed and caching is handled very differently than the rpi. I admit I might not know how different SSDs are compared to SD cards or how the disk controller functions exactly in the rpi. But I do know some simple facts, like the fact that pages aren't erased before a write with flash disks, especially not with modern flash controllers that don't even write to the same place unless you force them to, unlike spinning disks they have garbage collection.

JawnV6 posted:

For all the accusations that we're blind to the complexities hidden behind an abstraction you're quite content to sit way off in kernel land and pretend this hardware Just Works. I referred to linux's method to ensure block writes appear atomic above, so it's kinda patronizing to see it replayed with less fidelity.

I don't mean to sound patronizing, I just don't agree that removing power could cause your SD card to become entirely corrupted.

JawnV6 posted:

The video was queued up to an explanation that the SD card is coalescing writes and not actually sinking them to the nonvolatile as aggressively as you're suggesting. Did you lose the timestamp and start at 0?

I'm pretty sure I linked to the right place, the part where he says "flash controllers are optimized to not write to the first erase block (location of the FAT table) until *some point, it's not clear*", which is just crazy and would cause the problems you guys describe. I'm saying that's not how they work and I don't believe it's reasonable to think that's what a controller would do, it would make them wildly unreliable, especially since one of the primary and wide use cases for SD cards are cameras which almost never lose data and are powered on and off with abandon.

ElCondemn fucked around with this message at 00:03 on Jun 18, 2015

FSMC
Apr 27, 2003
I love to live this lie

ElCondemn posted:

Most of my experience is with SSDs in enterprise equipment, where data integrity, read/write speed and caching is handled very differently than the rpi. I admit I might not know how different SSDs are compared to SD cards or how the disk controller functions exactly in the rpi. But I do know some simple facts, like the fact that pages aren't erased before a write with flash disks, especially not with modern flash controllers that don't even write to the same place unless you force them to, unlike spinning disks they have garbage collection.


I don't mean to sound patronizing, I just don't agree that removing power could cause your SD card to become entirely corrupted.


I'm pretty sure I linked to the right place, the part where he says "flash controllers are optimized to not write to the first erase block (location of the FAT table) until *some point, it's not clear*", which is just crazy and would cause the problems you guys describe. I'm saying that's not how they work and I don't believe it's reasonable to think that's what a controller would do, it would make them wildly unreliable, especially since one of the primary and wide use cases for SD cards are cameras which almost never lose data and are powered on and off with abandon.

Are you loving comparing SSDs with SD cards? LMFAO assuming garbage collection on SD card.... OMG are you a loving idiot

Edit: Note that it's the people who post in the embedded programming thread that are calling you an idiot.

FSMC fucked around with this message at 00:26 on Jun 18, 2015

Hadlock
Nov 9, 2004

FSMC posted:

Edit: Note that it's the people who post in the embedded programming thread that are calling you an idiot.

:munch:

Given that you can read/write to an SD card using GPIO (albeit very slowly) I'm inclined to lean very strongly in favor of the embedded programming guys on any topic regarding SD read/write.

ElCondemn
Aug 7, 2005


FSMC posted:

Are you loving comparing SSDs with SD cards? LMFAO assuming garbage collection on SD card.... OMG are you a loving idiot

Edit: Note that it's the people who post in the embedded programming thread that are calling you an idiot.

Well, according to the video that was posted SD cards do have garbage collection.

mod sassinator
Dec 13, 2006
I came here to Kick Ass and Chew Bubblegum,
and I'm All out of Ass

ElCondemn posted:

You're taking what I'm saying completely out of context, I'm very aware of how flash memory differs from traditional spinning platter disks. Nothing you've said or posted has explained away the weird "when you read data it can be corrupted" idea, also the concept of a "lookup table" being lost as if it only exists in memory is weird and wrong too.

:confused: I don't think anyone is saying that reads are problematic on SD cards? Deletes and writes are where you can get into trouble, since the SD card controller can cache data before it writes it to flash. If power is suddenly removed then that cached data is gone forever. If that cached data is a critical part of the filesystem then you're totally hosed.

ElCondemn
Aug 7, 2005


mod sassinator posted:

:confused: I don't think anyone is saying that reads are problematic on SD cards? Deletes and writes are where you can get into trouble, since the SD card controller can cache data before it writes it to flash. If power is suddenly removed then that cached data is gone forever. If that cached data is a critical part of the filesystem then you're totally hosed.

You said...

mod sassinator posted:

If you don't do this you will very easily corrupt the entire SD card and have to reinstall the operating system. SD cards are not designed to suddenly lose power and the file system will get destroyed if power is pulled when it's moving around blocks and changing other card state.

This is not true, SD cards are definitely designed to suddenly lose power, and no the file system will not be destroyed if you do that. The risk of not issuing a shutdown command is losing the data you're writing, you're not at much risk of having your whole SD card becoming corrupted forcing you to reinstall.

I agree with your new position though, if you're writing to disk and lose power it's likely to corrupt any data that hasn't been flushed to disk yet, much like traditional disks.

edit: I'd also like to point out that there are no critical system files in linux that are loaded into memory which would be lost or corrupted if you suddenly lose power.

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE

mod sassinator posted:

:shrug: I'm not sure what else I can say--see over 37k Google hits for SD card corruption on the Pi forums: [url]https://www.google.com/search?q=raspberry+pi+sd+card+corrupt+site:https://www.raspberrypi.org[/url]

Will it always happen? No. Can it happen? Absolutely, yes. SD card firmware varies greatly--cheap cards might have very little caching and be less susceptible than more expensive cards that keep more data in RAM to be faster. There can be bugs or issues in the SD card firmware that make the issue more or less of a problem. It doesn't change the fact that it's extremely bad advice to tell a Pi user that they can safely pull the power without shutting down the Pi's OS. Anecdotes about it not happening with your card under your conditions are unfortunately just anecdotes.

This is the goddamned truth right here. Shut up and listen.

Also just because it's happening doesn't necessarily mean you notice it. Critical parts of the filesystem are disproportionately likely to be in-flight at any given time but it can also just be random data that you don't notice the corruption in.

I have personally corrupted several filesystems on a Pi by pulling the power. While I was toying around with it I would hard-cycle it because I was lazy. Later on it got accidentally switched off via powerstrips and kicked cords. It's very nearly russian roulette odds, I'd put the risk at 1:10 to 1:50 hard resets depending on circumstances (filesystem + how hard it's writing).

And again remember that ext4 has significantly increased in-flight time over ext3 - up to a minute in some common cases. Until then it only lives in memory and if you power cycle it's just gone.

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE

ElCondemn posted:

This is not true, SD cards are definitely designed to suddenly lose power, and no the file system will not be destroyed if you do that. The risk of not issuing a shutdown command is losing the data you're writing, you're not at much risk of having your whole SD card becoming corrupted forcing you to reinstall.

I agree with your new position though, if you're writing to disk and lose power it's likely to corrupt any data that hasn't been flushed to disk yet, much like traditional disks.

It depends on what it's writing though. If you're writing filesystem information when power loss occurs you can damage your ability to understand the jigsaw puzzle that is your filesystem. If you're writing a file that's smaller than the cell size (say 64kb) then you're also nuking any other files in that cell. There's lots of files that are <64kb which your system cannot boot without. Many of them are frequently written, and many even live in the same directories, which makes them extra likely to suffer collateral damage.

There are definitely ways you can clobber other stuff with collateral damage. SD cards aren't magic, and the use-case of an ext4 filesystem containing a million files with writes on heavy-usage files potentially delayed up to and beyond a full minute is much more likely to suffer damage due to data loss than a simple fat32 table full of 8MB JPEGs being sequentially written out. If you try hard enough you can even damage those. Especially (it sounds like) if you're pulling the plug rather than removing the card and letting the controller have one last attempt at flushing.

Again, to be clear: you may not cause corruption the first time you do it, or maybe not the next. Or maybe you might cause corruption and you don't notice it because it didn't render the system unbootable. But you're playing Russian Roulette, sooner or later you will lose. Just shut it down properly or buy yourself a power button if you care about the data managed by that system.

Paul MaudDib fucked around with this message at 01:26 on Jun 18, 2015

ElCondemn
Aug 7, 2005


Paul MaudDib posted:

It depends on what it's writing though. If you're writing filesystem information when power loss occurs you can damage your ability to understand the jigsaw puzzle that is your filesystem. If you're writing a file that's smaller than the cell size (say 64kb) then you're also nuking any other files in that cell.

You're right, but only If you're using FAT, it's still unlikely that this would happen with an rpi. Modern file systems have superblocks and backup blocks that prevent/protect this kind of thing, a fsck on boot will resolve most of these kinds of problems. If you're using an rpi you're almost certainly not using FAT as your system FS so the concern is very minimal, that's all.

I really don't mean to call anyone out or anything, I just think it's maybe being a bit too cautious. I wouldn't want any new rpi users to think it's a useless toy that's always in danger of being corrupted because they didn't think to put a power button on the thing.

Paul MaudDib posted:

There's lots of files that are <64kb which your system cannot boot without. Many of them are frequently written, and many even live in the same directories, which makes them extra likely to suffer collateral damage.

Inodes are not sequential, just because your file exists in the same folder logically it does not mean you're risking any other files in that folder if you write to disk. You're usually only risking what was most recently written to disk (due to wear leveling, etc.)

Paul MaudDib posted:

There are definitely ways you can clobber other stuff with collateral damage. SD cards aren't magic, and the use-case of an ext4 filesystem containing a million files with writes on heavy-usage files potentially delayed up to and beyond a full minute is much more likely to suffer damage due to data loss than a simple fat32 table full of 8MB JPEGs being sequentially written out. If you try hard enough you can even damage those. Especially (it sounds like) if you're pulling the plug rather than removing the card and letting the controller have one last attempt at flushing.

I think the biggest risk is using an SD card for anything other than a boot disk. Getting the system up and keeping everything else in memory or accessible through a network share seems like the right way to go about it.

ElCondemn fucked around with this message at 01:28 on Jun 18, 2015

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE
You might be able to make a ghetto UPS by taking one of those 20 amp-hour batteries with a USB interface if it supported charge-while-deliver. Go charger -> battery -> Pi, if the charger drops off the battery should take over. Hopefully it doesn't cut off the output during the transition.

mod sassinator
Dec 13, 2006
I came here to Kick Ass and Chew Bubblegum,
and I'm All out of Ass

ElCondemn posted:

You said...


This is not true, SD cards are definitely designed to suddenly lose power, and no the file system will not be destroyed if you do that. The risk of not issuing a shutdown command is losing the data you're writing, you're not at much risk of having your whole SD card becoming corrupted forcing you to reinstall.

I agree with your new position though, if you're writing to disk and lose power it's likely to corrupt any data that hasn't been flushed to disk yet, much like traditional disks.

edit: I'd also like to point out that there are no critical system files in linux that are loaded into memory which would be lost or corrupted if you suddenly lose power.

I'm not going to keep going back and forth anymore but I'd just like to point out my position has not changed at all. The statement I made, "SD cards are not designed to suddenly lose power and the file system will get destroyed if power is pulled when it's moving around blocks and changing other card state." is not implying corruption can happen during reads--somehow you are inventing that idea. "moving around blocks and changing other card state" is pretty clearly a write or delete.

quote:

If you're using an rpi you're almost certainly not using FAT as your system FS so the concern is very minimal, that's all.

Incorrect, if you're using a Raspberry Pi you absolutely are using a FAT filesystem somewhere. The Broadcom GPU that controls the entire bootup process of the Pi will only boot from a FAT partition on the disk. The operating system can be any filesystem (ext4 for Raspbian) but the Pi's hardware requires a FAT filesystem to bootstrap the process.

mod sassinator fucked around with this message at 02:37 on Jun 18, 2015

Fuzz1111
Mar 17, 2001

Sorry. I couldn't find anyone to make you a cool cipher-themed avatar, and the look on this guy's face cracks me the fuck up.
Anecdotally - some sdcard's are more prone to corruption than others. I've got a few pi's of different types and have never had an issue with corruption even though I yank power all the time - I'm I using patriot EP's for the RPi-B's, and sandisk extreme pro's for the B+ and 2 (the latter was chosen because it has very good random read/write).

However I've also set up a few for pi's for friends, where I've just used any old poo poo for the sdcard - a few of those have gone wrong, and it's these cards which seem particularly prone to loving themselves:


I should note that most of these pi's are running openelec which doesn't use a swap file/partition and has most things mounted readonly. I do try to properly shut down the pi's running raspbian especially if they are running a gui.

JawnV6
Jul 4, 2004

So hot ...

ElCondemn posted:

Most of my experience is with SSDs in enterprise equipment, where data integrity, read/write speed and caching is handled very differently than the rpi. I admit I might not know how different SSDs are compared to SD cards or how the disk controller functions exactly in the rpi. But I do know some simple facts, like the fact that pages aren't erased before a write with flash disks, especially not with modern flash controllers that don't even write to the same place unless you force them to, unlike spinning disks they have garbage collection.
Functionality that is handled by a dedicated IC in enterprise-grade piece of equipment doesn't necessarily appear on-die in a form factor a fraction of the size in a commodity market. It's likely the "modern flash controllers" you're familiar with are larger in size, memory, and team budget than what goes into a SD card. Give up your world of certainty, stable rails, SLA's in contracts, and face to embedded.

ElCondemn posted:

I don't mean to sound patronizing, I just don't agree that removing power could cause your SD card to become entirely corrupted.
I tend not to make such definitive statements in the face of overwhelming empirical evidence to the contrary but you're welcome to the position.

Imagine a 64k page stays hot. Another inode's updated to point in or out of that page and is successfully written. That first write, advertised to the system as complete, is delayed and power's dropped. When it's turned on, the old version of that page is successfully recovered. Oops, now our fs is in an inconsistent state and the OS can't boot. You can argue here that the SD isn't "entirely corrupted" and you're totally correct but it's that weird kind where the user's still SOL despite marginally incorrect terminology. The SD abstraction isn't providing the same guarantees as a SSD and it can provide inconsistent state and write ordering/visibility that the fs is not expecting when power is dropped.

ElCondemn posted:

I'm pretty sure I linked to the right place, the part where he says "flash controllers are optimized to not write to the first erase block (location of the FAT table) until *some point, it's not clear*", which is just crazy and would cause the problems you guys describe. I'm saying that's not how they work and I don't believe it's reasonable to think that's what a controller would do, it would make them wildly unreliable, especially since one of the primary and wide use cases for SD cards are cameras which almost never lose data and are powered on and off with abandon.
Taking a picture is a wildly different use case and read/write pattern than a running GUI OS responding to user input. Moreover, my camera will specifically wait while an image is written out, so the "on/off" use case has an explicit guard against the SD card having power yanked at the user's will.

ElCondemn posted:

Inodes are not sequential, just because your file exists in the same folder logically it does not mean you're risking any other files in that folder if you write to disk. You're usually only risking what was most recently written to disk (due to wear leveling, etc.)
It's kinda funny to see you take the knowledge that files in the same directory aren't necessarily contiguous on disk and use it to rule out two critical OS files being next to each other instead of imagining pathological cases of .ico files written on every page view sharing a sector with an inode.

Skarsnik
Oct 21, 2008

I...AM...RUUUDE!




Fuzz1111 posted:

Anecdotally - some sdcard's are more prone to corruption than others. I've got a few pi's of different types and have never had an issue with corruption even though I yank power all the time - I'm I using patriot EP's for the RPi-B's, and sandisk extreme pro's for the B+ and 2 (the latter was chosen because it has very good random read/write).

However I've also set up a few for pi's for friends, where I've just used any old poo poo for the sdcard - a few of those have gone wrong, and it's these cards which seem particularly prone to loving themselves:


I should note that most of these pi's are running openelec which doesn't use a swap file/partition and has most things mounted readonly. I do try to properly shut down the pi's running raspbian especially if they are running a gui.

This has been my experience too. I had a pi in work that was getting the power pulled all the time and would need re-imaging every couple of weeks. It had a Sandisk card

After putting a Samsung card in, it hasn't broken once in 2 years

Wasn't there a thing a few years back that said that Sandisk were the biggest target of counterfeit cards being sold online?

ElCondemn
Aug 7, 2005


JawnV6 posted:

Imagine a 64k page stays hot. Another inode's updated to point in or out of that page and is successfully written. That first write, advertised to the system as complete, is delayed and power's dropped. When it's turned on, the old version of that page is successfully recovered. Oops, now our fs is in an inconsistent state and the OS can't boot. You can argue here that the SD isn't "entirely corrupted" and you're totally correct but it's that weird kind where the user's still SOL despite marginally incorrect terminology. The SD abstraction isn't providing the same guarantees as a SSD and it can provide inconsistent state and write ordering/visibility that the fs is not expecting when power is dropped.

Taking a picture is a wildly different use case and read/write pattern than a running GUI OS responding to user input. Moreover, my camera will specifically wait while an image is written out, so the "on/off" use case has an explicit guard against the SD card having power yanked at the user's will.

It's kinda funny to see you take the knowledge that files in the same directory aren't necessarily contiguous on disk and use it to rule out two critical OS files being next to each other instead of imagining pathological cases of .ico files written on every page view sharing a sector with an inode.

Yes it's possible, I still don't think it's very likely. There are really very few files that are necessary to boot an rpi, most linux distros only require a hand full of services to boot and run an fsck. But regardless of how common I think it is, you clearly believe it's way more likely than I do, feel free to issue a shutdown to your rpi's, I will continue to do what I do. I do not fault you for being cautious.

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE

Skarsnik posted:

Wasn't there a thing a few years back that said that Sandisk were the biggest target of counterfeit cards being sold online?
Anecdotally I wonder if cheap cards have a high probability of being garbage cards from the very edge of the silicon chip high-capacity line that happen to have the first 16K cells work. It would explain the high failure rate in Rpi usage.

That said, I've had high-quality Toshiba cards bought directly from Newegg get corrupted from power-cycling with an Rpi within 1-2 dozen hard power cycles. And Toshiba makes flash memory themselves, there's no repackaging there.

There's absolutely no question of the danger here - you can listen to a guy who works in a data-center with power-backed enterprise-grade SSDs designed to provide data loss protection or you can take it from 36,000 posts on the Rpi forums or literally anyone who's ever worked intensively with a Raspberry Pi - whose advice will be don't hard power-cycle your Pi.

Paul MaudDib fucked around with this message at 07:34 on Jun 18, 2015

Amberskin
Dec 22, 2013

We come in peace! Legit!

ElCondemn posted:

Yes it's possible, I still don't think it's very likely. There are really very few files that are necessary to boot an rpi, most linux distros only require a hand full of services to boot and run an fsck. But regardless of how common I think it is, you clearly believe it's way more likely than I do, feel free to issue a shutdown to your rpi's, I will continue to do what I do. I do not fault you for being cautious.

Errr... no. If the root directory gets hosed you are toast. And by 'hosed' I mean damaged beyond fsck ability to repair it.

theperminator
Sep 16, 2009

by Smythe
Fun Shoe
It's a raspberry pi, who even gives a poo poo about fs corruption?

Use Ext4, mount your root fs with the sync option and find something else to care too much about.

G-Prime
Apr 30, 2003

Baby, when it's love,
if it's not rough it isn't fun.
I'd guess people who use it in a production environment with low power parallel tasks running on multiple boxes might have concerns about corruption. But who knows, maybe they enjoy doing rebuilds.

beuges
Jul 4, 2005
fluffy bunny butterfly broomstick
I've worked on a couple of projects using raspberry pi's in environments where they could lose power unpredictably, and I can tell you that card corruption is a very real thing. We've gotten to the point where we have / and /boot mounted read-only, and only have a little scratch partition writable for processing stuff... doing that has brought corruption issues down to pretty much zero - the only issues we have now are when the cards have physical issues. In our case, we have these things in some fairly remote places, and having to drive half a day to swap out a corrupt card is not good at all. Certain brands seem to work better than others, certain batches from the same brands seem to work better than others. The only way you can properly safeguard yourself from a corrupt card is by either shutting down cleanly each time, or have the entire operating system on a read-only filesystem so it can't really get corrupted anyways.

Badly Jester
Apr 9, 2010


Bitches!
How reliably does syncthing work on a Raspberry Pi? I'm looking to keep files for my dissertation synchronized between my laptop and my desktop, so I'll obviously keep backups, but if syncthing frequently screws up/has issues with versioning, I might have to stick with Btsync, which I'm not to fond of.

ElCondemn
Aug 7, 2005


Badly Jester posted:

How reliably does syncthing work on a Raspberry Pi? I'm looking to keep files for my dissertation synchronized between my laptop and my desktop, so I'll obviously keep backups, but if syncthing frequently screws up/has issues with versioning, I might have to stick with Btsync, which I'm not to fond of.

Checkout seafile, it works really well and I've had success running it on my rpi with USB attached storage. I tried out owncloud, syncthing and a few others and I've found that seafile can handle tons of small files way better than any of the other options (faster, more reliable, stable, etc.).

ElCondemn fucked around with this message at 22:55 on Jun 18, 2015

theperminator
Sep 16, 2009

by Smythe
Fun Shoe

G-Prime posted:

I'd guess people who use it in a production environment with low power parallel tasks running on multiple boxes might have concerns about corruption. But who knows, maybe they enjoy doing rebuilds.

I hadn't even thought about that, what kind of production loads does the rpi get used for?

Using SDCards and for your rootfs will kill them within a few years of prod use anyway, I had to manage a fleet of SDCard booting Linux servers at my last job and the SDCards all died within years and there were no sudden power loss events.

Maybe newer sdcards are better, but Linux is constantly logging all sorts of poo poo so you've always got writes going on.

evol262
Nov 30, 2010
#!/usr/bin/perl
You can and should be remote syslogging poo poo you care about and sending chatty poo poo you don't care about to tmpfs. No reason to keep anything other than /var/log/audit* and /var/log/{messages,syslog,secure,wtmp,btmp}

Also, journald makes a lot of this easier to manage.

JawnV6
Jul 4, 2004

So hot ...

ElCondemn posted:

Yes it's possible, I still don't think it's very likely. There are really very few files that are necessary to boot an rpi, most linux distros only require a hand full of services to boot and run an fsck. But regardless of how common I think it is, you clearly believe it's way more likely than I do, feel free to issue a shutdown to your rpi's, I will continue to do what I do. I do not fault you for being cautious.

I fault you for proffering lovely advice despite repeated correction based on inapplicable experience. I don't give a poo poo what you, personally, do to your devices and acting like that's what we've been talking about is so terribly misleading. Again, addressing the reams of evidence of others actually hitting this scenario rather than your personal wishes on it's likelihood would be more convincing. Were you planning on addressing the FAT correction?

Badly Jester
Apr 9, 2010


Bitches!

ElCondemn posted:

Checkout seafile, it works really well and I've had success running it on my rpi with USB attached storage. I tried out owncloud, syncthing and a few others and I've found that seafile can handle tons of small files way better than any of the other options (faster, more reliable, stable, etc.).

Thanks, I'll take a look at that then. Hopefully it'll play nice with my lovely provider-issued and -castrated router. That's the one thing BTSync did really well.

ElCondemn
Aug 7, 2005


JawnV6 posted:

I fault you for proffering lovely advice despite repeated correction based on inapplicable experience. I don't give a poo poo what you, personally, do to your devices and acting like that's what we've been talking about is so terribly misleading. Again, addressing the reams of evidence of others actually hitting this scenario rather than your personal wishes on it's likelihood would be more convincing. Were you planning on addressing the FAT correction?

I didn't want to stir things up again because it's pointless, but since you asked.

The boot partition is for the bootloader, it's only written to when you make changes to the boot loader. So even if that dude in the video was right about how the first page has some kind of "hold off on updating the FAT table until other stuff is done" thing you're not using that file system for writing, like ever.

Again, since you're not writing to that sector, you run little risk of corrupting your bootloader. I think it's much more likely, I think many people would agree, that SD cards are just not meant for the type of wear that they're being put under. Constant writes, constant reads, they're not designed for this sort of thing and they fail because they're cheap and lovely, and unplugging the power suddenly is only a small concern.

The Sweet Hereafter
Jan 11, 2010
So I have this half formed plan to combine my HDMIPi screen and my picoPSU into some sort of monster, probably including a powered USB hub and an amplifier circuit using the sound taken off the HDMI by the HDMI-Pi driver board. If I can make it work and figure out an arrangement I will probably try to 3D print parts for a case to enclose the lot.

My main question, though, is can I run all that off a picoPSU? I was planning on running a 12v line to the HDMIPi and letting that also power the Pi like it's designed to do, then taking 5v to the USB hub and 3.3v to the amplifier circuit. I could alternatively run a separate 5v to the Pi if case arrangement meant it couldn't line up with the driver board. I can't think of any reason why this shouldn't work but I am not a genius, so I figured I'd ask here.

G-Prime
Apr 30, 2003

Baby, when it's love,
if it's not rough it isn't fun.

theperminator posted:

I hadn't even thought about that, what kind of production loads does the rpi get used for?

Using SDCards and for your rootfs will kill them within a few years of prod use anyway, I had to manage a fleet of SDCard booting Linux servers at my last job and the SDCards all died within years and there were no sudden power loss events.

Maybe newer sdcards are better, but Linux is constantly logging all sorts of poo poo so you've always got writes going on.

My use case (which I'm still trying to sell up the chain to my VP, maybe on next year's budget) is to use them as workers attached to a job server for low-intensity polling tasks on our network at a moderately large US ISP. We do literal hundreds of thousands of SNMP polls and raw text screen scrapes from devices on our network daily, and do small amounts of mangling and parsing to them. Right now, we have a couple larger VMs doing various tasks related to these, but the tasks aren't doled out on the fly. They're just cronned (or in some cases manually run) on the boxes. My plan is to set up a central job server (probably two of them, set up for HA) which assigns the tasks to any free worker, with all workers able to run any given job type.

The reason this would be a good fit for us is that while collecting text and parsing it isn't really CPU intensive, doing it in parallel on a single physical CPU is a hell of a lot of context switching, and we get somewhat slower results by virtue of that. Additionally, we have some jobs that are currently built to run serial rather than parallel, which is dramatically slower. Serial's only worthwhile to us for cases where the order of the data is critical and there's no good way to sort it. In parallel, via a job server, we can just dump all results to a key/val store like memcached or redis, and collect them as needed, while any given node moves on to its next job as soon as it's free to.

That said, with a cluster of like 16 of these, it'd be a pain in my rear end to have to reimage the SD cards at any given time, especially when we're in the process of moving to a colo across town for our local DC.

Hadlock
Nov 9, 2004

Sorry to derail your SD Card beatdown ElCondemn, but I just wired up my Pi-based robot for the first time :dance:

Got the Pi (it's the A+ model, smaller, uses about half the power of the B+) wired in to the motor controller which takes in 6-16v and provides raw power output to two motors, in addition to a regulated 5v output. Wire the 5v output directly in to the 5v rail on the GPIO and everything seems to work really well. It's powered by a pair of 3.7v laptop batteries which when they discharge down to about 2.8v the 5v rail on the motor controller cuts out and the Pi sort of auto-shuts down. You can't leave it totally unattended as the motor controller would eventually suck the batteries dry below 2.6v but it's good enough for now and they're $1 ebay 18650s anyways.

The motor controller uses PWM so I am going to have to cannibalize my PWM controller from another project, but I found more 16 channel PWM controllers for $4.80 shipped on ebay, so I have two more on the way (I have two chassis, the idea is to have two working robots at some point). Chassis is based on the Shrimpbot 1.6 chassis.



I have an 18v -> 7.4v solar charging solution to keep the batteries topped off when not in use ready to go, but i'm not quite ready to wire everything in permanently yet.

Rexxed
May 1, 2010

Dis is amazing!
I gotta try dis!

Hadlock posted:

Sorry to derail your SD Card beatdown ElCondemn, but I just wired up my Pi-based robot for the first time :dance:

Got the Pi (it's the A+ model, smaller, uses about half the power of the B+) wired in to the motor controller which takes in 6-16v and provides raw power output to two motors, in addition to a regulated 5v output. Wire the 5v output directly in to the 5v rail on the GPIO and everything seems to work really well. It's powered by a pair of 3.7v laptop batteries which when they discharge down to about 2.8v the 5v rail on the motor controller cuts out and the Pi sort of auto-shuts down. You can't leave it totally unattended as the motor controller would eventually suck the batteries dry below 2.6v but it's good enough for now and they're $1 ebay 18650s anyways.

The motor controller uses PWM so I am going to have to cannibalize my PWM controller from another project, but I found more 16 channel PWM controllers for $4.80 shipped on ebay, so I have two more on the way (I have two chassis, the idea is to have two working robots at some point). Chassis is based on the Shrimpbot 1.6 chassis.



I have an 18v -> 7.4v solar charging solution to keep the batteries topped off when not in use ready to go, but i'm not quite ready to wire everything in permanently yet.

That looks really neat. I've been considering making a robot or at least an articulated arm just to mess with robotics but I can't think of a practical application that I've decided "yep I should make a robot to do this" for yet. I'm starting to think that I should just build one to learn more about making stuff and not worry about the practical application because they're just cool and I'm sure I can figure something out (besides scaring the dog).

sleepy gary
Jan 11, 2006

I have a Raspberry Pi B and an Edimax USB wifi dongle (the popular tiny one). Is there anything wrong with eliminating the need for a powered USB hub by cutting a USB extension cable open and hardwiring 5v on the Edimax side and then connecting "directly" to the Pi (with only D+ and D- intact from Pi to Edimax dongle)?

mod sassinator
Dec 13, 2006
I came here to Kick Ass and Chew Bubblegum,
and I'm All out of Ass
Try connecting the dongle directly to the Pi without a powered hub. As long as you power the Pi with a ~1-2 amp power supply it should be fine. I use my Edimax dongle directly with my Pi's without any issues.

Hadlock
Nov 9, 2004

In my first robot photo you can see that exact edimax WiFi adapter being powered directly by the Pi. Pi + Edimax draws about 170 ma at idle. You should be fine. Power requirements are really low.

sleepy gary
Jan 11, 2006

mod sassinator posted:

Try connecting the dongle directly to the Pi without a powered hub. As long as you power the Pi with a ~1-2 amp power supply it should be fine. I use my Edimax dongle directly with my Pi's without any issues.

Welp. That's working fine. Thanks, that saves me some money and hassle.

Would I be asking too much to also add a USB flash drive to the mix? This is a B rev. 2 (two usb ports, 512mb RAM)

Skarsnik
Oct 21, 2008

I...AM...RUUUDE!




DNova posted:

Welp. That's working fine. Thanks, that saves me some money and hassle.

Would I be asking too much to also add a USB flash drive to the mix? This is a B rev. 2 (two usb ports, 512mb RAM)

No it'll be fine, I do the same all the time

The Sweet Hereafter
Jan 11, 2010

DNova posted:

Welp. That's working fine. Thanks, that saves me some money and hassle.

Would I be asking too much to also add a USB flash drive to the mix? This is a B rev. 2 (two usb ports, 512mb RAM)

I've never had a problem using WiFi/USB stick/wireless keyboard & mouse dongle all at once on any 4 USB port Pi I've owned.

This might change shortly, but since nobody had anything to say about my PicoPSU plans above I'm going to start soldering and see what happens.

Adbot
ADBOT LOVES YOU

ante
Apr 9, 2005

SUNSHINE AND RAINBOWS

The Sweet Hereafter posted:

I've never had a problem using WiFi/USB stick/wireless keyboard & mouse dongle all at once on any 4 USB port Pi I've owned.

This might change shortly, but since nobody had anything to say about my PicoPSU plans above I'm going to start soldering and see what happens.

It looks like the PicoPSU is more expensive and has more bulky connectors than what you actually need.


I'd probably pick up an old 19v laptop power supply, and a bunch of Chinese DC-DC adjustable step-down modules. You can get them for ~$2 each, and then you can add your own connector. Should end up being about the same size and power capabilities.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply