Embedded Linux! Who needs security or new packages?

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Embedded Linux! Who needs security or new packages?

FlapYoJacks: Feb 12, 2009

I have been meaning to write a embedded Linux op for quite some time, but I have been too lazy or just plain forgot, but no more!

My background
I started out messing around with Linux 16 years ago with Slackware 7.1, and used it as my primary operating system ever since.
In 2010 I was unemployed and fixing cars in my garage while going to college when I was called up randomly by a friend I hadn't seen in a few years. His boss who ran a tiny engineering company (2 employees and himself) had just picked up/inherited a camera project. This particular camera ran embedded Linux off of a TI DM368 processor and they needed help getting Linux up and running on it, and I was the only "Linux guy" he knew. I was told I was only going to be needed for a few weeks. That job lasted 4 and a half years.

In those 4 and a half years I learned how to program starting with C, how to write kernel drivers, board layout, the basics of electronics, power management, project management, system architecting, and so on; basically how a product goes from concept to the shelf. During my tenure at that company I completed 3 finished projects including a Android tablet before our main client hosed over my boss and tanked his little engineering company.

6 Months later I moved halfway across the United States to Michigan to continue my career. My current project is working on a SIP phone, with my main focus on creating a incredibly flexible embedded distribution based off of buildroot with a huge focus on security.

Linux Vs RtOS
There are several reasons one should choose embedded Linux over a RtOS.

- The Linux Kernel:
The Linux kernel provides a incredibly rich feature set that gives the developer a absolutely enormous amount of support for tons of common hardware components and features, including a FULLY FEATURES AND STABLE NETWORK STACK.

- A ever growing amount of development boards.
Wand ultra cheap? Raspberry Pi being the most popular
Need Emmc? Beagle Bone black is for you!
Like Freescale (now nxp?) I like the wandboard, Boundarydevices is also great

- Feature rich SDK's
The three major ones I know of are:
Buildroot
Yocto
Ptxdist

I am a contributor to buildroot, and I am familiar as well with ptxdist. I need to someday learn yocto.

- Major distributions work on many embedded platforms as well!
Ubuntu, fedora, Gentoo, Arch, CentOS/RHEL and many more all have builds that work on many popular embedded devices.

Why does this thread exist?
I think this thread could be useful for many people getting started or even asking more advanced questions about the embedded Linux world. I will try to post any neat stuff I am doing that isn't covered under my NDA as well.

Feel free to tell me I missed something/ask me questions!

# ¿ Jul 18, 2016 15:44

Adbot: ADBOT LOVES YOU

# ¿ May 16, 2024 06:08

FlapYoJacks: Feb 12, 2009

Currently I am working on making our embedded product secure. Here are some steps I am taking to do so:

System:

SELinux: Set to enforcing.
Custom SELinux contexts for our program.
A custom kernel HAL layer for talking to the hardware.
A user space application to talk to the HAL layer.
A OTP key in the processor die that our software authenticates against, if it doesn't find it then the software won't work.
All flashing is done in house.
Root is disabled.

Web interface:
We are using nginx with the nasxi module, php7, and fcgi, all with appropriate custom SELinux contexts.
The user password is set to the company name, but the software will require the user to change the password before the product works.
Of course all passwords are hashed and salted via bcrypt (is there something better?).

Unfortunately many of our customers still want telnet, so I have to include it, but it's turned off by default and a gigantic warning will pop up on the screen telling end users that the trafic isn't encrypted and that we strongly advise people to use ssh instead.

Anything else I am missing or could do better?

# ¿ Jul 19, 2016 15:29

FlapYoJacks: Feb 12, 2009

Malcolm XML posted:

whats your threat model

no threat model = defending against nothing

My threat model isn't necessary customers, more corporate espionage than anything else. People DO and WILL pay engineers to reverse engineer this things.

Malcolm XML posted:

in actual questions, i wanna modulate an ASK 433mhz thing with pretty precise microsecond timing. does linux have a "treat this thread as real-time" like windows

http://www.embedded.com/design/operating-systems/4371651/Comparing-the-real-time-scheduling-policies-of-the-Linux-kernel-and-an-RTOS-

Round robin is what your RTOS is *probably* using.

# ¿ Sep 6, 2016 17:49

FlapYoJacks: Feb 12, 2009

yippee cahier posted:

I'm getting started, coming from an RTOS-on-microcontroller environment. I got myself a Raspberry Pi. Would a good starting path be to read up on and use buildroot, then attempt to write a kernel driver for some random SPI chip with interrupts?

I'm guessing a standard-ish project exposes the hardware to userspace in this way, then the actual application logic could be done in any old scripting language?

Just wanna learn some stuff here, feeling uneasy about getting pigeon-holed in RTOS land, especially when I'm reimplementing a wheel for months on end.

good for you! As for buildroot? https://buildroot.org/downloads/manual/manual.html

But the gist of it is:

download buildroot
make raspberrypi_defconfig
make

Then flash the output in "output/images" to a SD card and boot.

As for you wanting to access a driver via userspace? Depends on what your driver DOES.

The common way for a new guy to do so is via a gpio call in /sys/class/gpio
http://elinux.org/CI20_GPIO_LED_Blink_Tutorial

The second way is to create a /sys/ device and shove folders and files in there, although for a simple device setup I wouldn't do that.

Another way is to create a /dev/ device and talk to it that way.

# ¿ Sep 6, 2016 17:54

FlapYoJacks: Feb 12, 2009

Malcolm XML posted:

OK re security, so your model is corp espionage. What modes of attack are you looking at? Full device access? I mean that's essentially impossible to deal with unless youre gonna go real deep and blow fuses and stuff, have designed ur board to avoid leaking info in side channel attacks like differential power analysis, thermal analysis, decapping etc. If it's just SW access, then yeah u need to limit any access that's not needed by the software to the bare minimum, disable all unnecessary services, ensure authentication by all accesses, etc

OK so for the process i can just pin it to a core and set the scheduler policy so that that process is the only thing on the core. Works for me!

I'm glad you asked!

We have taken device access in consideration and have implemented a OTP key into the die of the processor which our software authenticates with. If it doesn't find that key the software itself refuses to run. The FS itself is R/O and encrypted as well. As for power analyses, not much I can do there.

Malcolm XML posted:

linux has a SPI driver and it's a device tree overlay away in raspbian, so no dicking with kernel drivers (u dont want to, its a PITA)

http://www.haifux.org/lectures/258/gpio_spi_i2c_userspace.pdf

GPIO also has a standard sysfs overlay you can tinker with.

# ¿ Sep 7, 2016 20:57

FlapYoJacks: Feb 12, 2009

Malcolm XML posted:

cool, but like, what kind of adversary we talking about, it's not exactly unknown to literally acid etch off encapsulation and count 1's and 0's in ROM (http://zacsblog.aperturelabs.com/2013/02/decapping-integrated-circuits-using.html)

i once knew a guy who did SEM/TEM scans of HDDs for forensics, and u can do similar poo poo on NAND: http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1555687

might wanna ask here for SW help beyond the standard principle of least privilege: http://forums.somethingawful.com/showthread.php?noseen=0&threadid=3500975&perpage=40&pagenumber=42#pti15

Telecom companies. The company I work for has 36 employees right now so we can't get too crazy. Our adversaries are multi billion dollar corporations though.

# ¿ Sep 8, 2016 01:19

FlapYoJacks: Feb 12, 2009

forgot to say! We got Alpha boards in on Tuesday! My former boss who I got a contract gig at my current place of work doing board layout got the Alpha boards last week, and verified all the power supplies before shipping them to us.

9 bodges total for a 1300+ piece board with 17 connectors isn't too bad!

Things done so far:

- Verified USB (used for debugging, will be removed for production)

- Verified EMMC

- Verified UART 1

Linux won't be booting on this thing until late next week at the earliest. For now it's just uboot.

# ¿ Sep 9, 2016 13:59

FlapYoJacks: Feb 12, 2009

So quick update on my companies project.

I have a Micrel KSZ8794CNX connected via RGMII to a imx6 solo.

All the data lines look good, all the clock lines look good, everything looks good.

RX works, TX doesn't. What the gently caress Micrel?

# ¿ Sep 23, 2016 01:52

FlapYoJacks: Feb 12, 2009

Minus Pants posted:

How'd you get into/learn board layout? I've had some simple low freq boards made, but I haven't found many good, practical resources for common high-speed signal routing (USB, SATA, memory, etc.). Also, what's your verification workflow like (test gear, software, lessons learned)?

I learned from my former boss who has over 30 years of experience in low level programming and board layout/electronics. He's probably the best analog guy in the United States. I suggest getting as lucky as me. :v:

# ¿ Sep 24, 2016 20:58

FlapYoJacks: Feb 12, 2009

sound_soc_read doesn't read from the sound codec as expected! :argh:

# ¿ Oct 25, 2016 23:34

FlapYoJacks: Feb 12, 2009

Storysmith posted:

I saw your post and I'm genuinely curious now, as someone with vague interest in i2s codecs for dumb nerd poo poo: what machinations do you have to do to actually read the i2c status back? What possible reason is there to have this implemented this way? Did someone miss a volatile somewhere or something, or is this a straight up intentional behavior within the driver?

Sorry for the single line rant. I will cross post my much larger rant from the coding horrors thread here:

Let me take you down a magical journey!

Here's the C file for anybody who want's to play along:
http://lxr.free-electrons.com/source/sound/soc/codecs/tas2552.c

On line 41 you have struct reg_default tas2552_reg_defs[] which is used as part of a overall larger struct called regmap_config on line 678.

tas2552_reg_defs[] should be set to DEFAULT REGISTER VALUES

On line 707 this struct is passed to devm_regmap_init_i2c, which calls regmap_get_i2c_bus (this passes because there are 2 other chips working on the same i2c line), and devm_regmap_init.

devm_regmap_init calls regmap_init which then puts the register map into sysfs. This is actually very useful for debugging.

At this point, no registers have been written, as they are supposedly DEFAULT VALUES.

So here is where things get hosed up.

The first time a write to the chip supposedly occurs is on line 595. The function snd_soc_update_bits returns 0 on success, or a negative error code otherwise. Now what happens here is this logic:

C code:

int snd_soc_component_update_bits(struct snd_soc_component *component,
	unsigned int reg, unsigned int mask, unsigned int val)
{
	bool change;
	int ret;

	if (component->regmap)
		ret = regmap_update_bits_check(component->regmap, reg, mask,
			val, &change);
	else
		ret = snd_soc_component_update_bits_legacy(component, reg,
			mask, val, &change);

	if (ret < 0)
		return ret;
	return change;
}

Oh look, if you have a regmap then regmap_update_bits_check is called, which to be fair to this function does fail and does return a negative error number, however the driver code itself does not check anywhere the return value and goes on it's merry way like nothing bad ever happened. This isn't the real horror though. Sure it's bad, but not the real travisty of the story.

The real problem is when sound_soc_read is called. This is where things get hosed up. snd_soc_read calls snd_soc_component_read which has this logic:

code:

int snd_soc_component_read(struct snd_soc_component *component,
	unsigned int reg, unsigned int *val)
{
	int ret;

	if (component->regmap)
		ret = regmap_read(component->regmap, reg, val);
	else if (component->read)
		ret = component->read(component, reg, val);
	else
		ret = -EIO;

	return ret;
}

regmap_read calls _regmap_read which has this logic in it:

code:

static int _regmap_read(struct regmap *map, unsigned int reg,
			unsigned int *val)
{
	int ret;
	void *context = _regmap_map_get_context(map);

	WARN_ON(!map->reg_read);

	if (!map->cache_bypass) {
		ret = regcache_read(map, reg, val);
		if (ret == 0)
			return 0;
	}

	if (map->cache_only)
		return -EBUSY;

	if (!regmap_readable(map, reg))
		return -EIO;

	ret = map->reg_read(context, reg, val);
	if (ret == 0) {
#ifdef LOG_DEVICE
		if (map->dev && strcmp(dev_name(map->dev), LOG_DEVICE) == 0)
			dev_info(map->dev, "%x => %x\n", reg, *val);
#endif

		trace_regmap_reg_read(map, reg, *val);

		if (!map->cache_bypass)
			regcache_write(map, reg, *val);
	}

	return ret;
}

And there's the problem. If you don't set cache_bypass as a flag (which is NOT THE DEFAULT) it reads FROM THE MODIFIED ARRAY AND NOT THE loving CHIP.

At this point you get a success on return, and it creates a perfect loving storm of making it look like i2c is working.

Three weeks. I will never EVER trust wrappers around read/writes again.

# ¿ Oct 26, 2016 19:21

FlapYoJacks: Feb 12, 2009

Buildroot-2016.11-rc1 has been released. Seems to be a pretty solid update.

# ¿ Nov 5, 2016 19:45

FlapYoJacks: Feb 12, 2009

meatpotato posted:

A little late to the show, but I also know the pain of poorly-used regmap and wrangling sound_soc drivers, WM8962 in my case.

Regmap is actually pretty dang cool, but if misused it creates major headaches. The default values array thing is kind of hacky because I wouldn't depend on them not changing between codec chip revisions, but I get that it satisfies some needs. Because of that I'm conservative about what goes into the default values array.

If you're not too worried about the power or latency costs of read/modify/write cycles when changing codec settings, just disable regmap caching entirely, which it looks like you've done by setting the cache bypass flag. The other way of disabling it is at regmap init, there's an option for a cache type and REGCAHCE_NONE is valid.

http://lxr.free-electrons.com/source/include/linux/regmap.h?v=3.2

Also, Mark Brown, the author of regmap, has answered my questions before on the alsa dev irc channel.

Also a little late to this, but yes; I agree; the original intentions of regmap are awesome and make for some great power savings techniques. The true blame is like, 10% regmap, 45% the driver, 45% me for failing to diagnose it correctly for 2 weeks.

In other news! Buildroot 2016.11 was just released, looks like a solid update! One of the best features though is that it will now use the hosts cmake if it's available. This saves me almost 10 minutes of compile time!

# ¿ Dec 1, 2016 16:21

FlapYoJacks: Feb 12, 2009

Up to 60 hours this week, completely my fault. I said that all the hardware on our new board verified and it didn't. I can talk to the camera on our new board via I2C, I can read, write, and configure it (also in uboot.)

However when I ask the drat thing for a picture it pukes, tripping the MIPI_CSI_ERR1 register of the IMX.6 with 0x3, which translates back to:

Start of Transmission Error on data lane 1 and 0 (no synchronization achieved) on the Mipi bus.

Here's the funny thing: we are using a OV5640 on our board with the exact same driver, filing system, and even schematic as the WCAM-ADPT-G; except for one thing: We added 600Ohm, 300mA Ferrite beads to the clock lines to reduce EMI, so now to test I have to remove 7 0201 beads and replace them with 0hm resistors. :suicide:

# ¿ Dec 9, 2016 15:25

FlapYoJacks: Feb 12, 2009

Popete posted:

How many MIPI lanes are you using? I had a project awhile back with two displays on an i.MX6 that used both the direct MIPI interface (2 lanes) to the i.MX and then a 4 lane MIPI interface display that was connected via the i.MX LVDS interface to a LVDS/MIPI bridge and then to the display. I never quite got the bridge interface to work but the 2 lane direct MIPI interface was fairly straightforward.

The ov5640 uses two lanes. Its probably the beads, on the plus side I am finally getting soldering tweezers!

# ¿ Dec 10, 2016 20:12

FlapYoJacks: Feb 12, 2009

So fun things are happening right now with my project:

The camera works perfectly now that the beads are replaced, however rtp streaming the camera stream glitches and misses frames repeatedly unless MTU is set to 256 AND bitrate is set to 5Mbp/s or lower.

This is unacceptable for many many reasons. Right now my friend/former boss/hardware layout guy is removing beads from the ethernet clock lines to see if that clears up the issue, then I get to TeamViewer in and run gstreamer to see if the issue is cleared up.

Fun fact: With the wandboard connected to the same camera, I can stream 1080p30@20Mb/s no problem.

Also another fun issue:
ping -s 65507 to the wandboard works fine, on our board it times out above 39kbp/s.

Timing is hard. :argh:

Edit*

This is the last hardware verification I have to do to fully verify a board with 25 connectors, and 6 other IC's that all needed to integrate and work properly with Linux.

FlapYoJacks fucked around with this message at 18:50 on Dec 28, 2016

# ¿ Dec 28, 2016 18:45

FlapYoJacks: Feb 12, 2009

I am vindicated. I just had this wonderful conversation with my friend:

quote:

Friend:
OK well there is no way I'm putting an 8-inch trace on the board

Me:
Aww come on! Why not?

Friend:
Might be easier to provide the dual SPI/I2C option

Me:
Did you try an 8inch wire?

Friend:
yes

Me:
and?

Friend:
)*&^%)*(&%^&

Me:
a hahaha
65k?

Friend:
)(*&^*&^%

Me:
AH
FREAKING
HA

Now; why is this hilarious and amazing? Becuase I debated with him for almost 2 - 12 hour days over whether or not the TXC line on the RGMII interface was the culprit to my problem. He was absolutely 100% insistent that it couldn't possibly be the issue. He almost had me convinced as well, I was about to start debugging the kernel network stack.

I am happy now. :smug:

# ¿ Dec 30, 2016 23:57

FlapYoJacks: Feb 12, 2009

I was dumb and forgot to configure my buildroot build for hard floating points. Now I can use opengl to render overlays on gstreamer! Hooray!

# ¿ Jan 16, 2017 18:02

FlapYoJacks: Feb 12, 2009

Let me tell you guys about a common thing that happens to engineers. It's called: Magic goes here.

It's a phenomenon that happens when you gloss over something on a spec sheet because another thing on the spec sheet is working.

Case in point: We have a Micrel switch on the board I am working on. I got the Switch working and all is good correct?

Wrong! The engineering specification calls for VLAN support on the other two ports, and the only way to do that is through SPI (because this chip sucks and apparently doesn't want to give access to all the registers through i2c????).

I glossed over the VLAN part because I was just happy to finally get all the timing issues fixed initially.

Don't be like me, make sure that everything works. :v:

# ¿ Mar 3, 2017 16:48

FlapYoJacks: Feb 12, 2009

I had to learn auto tools today. Porting packages with hard coded variables is never a good thing.
On the plus side, I will soon have a patch set up for the TI Davinci DM36X line or DSP's for Buildroot
that will include gstreamer1.0 codecs!

# ¿ Apr 18, 2017 22:20

FlapYoJacks: Feb 12, 2009

As an FYI: Buildroot 2017.05 was released a few days ago. I was on vacation, but there are some nice improvements.

The next release in will have gcc7 as well.

# ¿ Jun 7, 2017 16:05

FlapYoJacks: Feb 12, 2009

Just as a heads up, I was able to get libressl committed to mainline Buildroot a few days ago.

# ¿ Jul 13, 2017 11:48

FlapYoJacks: Feb 12, 2009

Poopernickel posted:

Why should I use a buildroot over a Yocto? Or vice versa?

https://www.youtube.com/watch?v=13LZ0szWSVg&t=2200s

I prefer buildroot because it's more simple. Layers loving suck (explained in the video.)
I can use yocto just as well as Buildroot, but I vastly prefer Buildroot.

# ¿ Jul 16, 2017 15:52

FlapYoJacks: Feb 12, 2009

BuildRoot 2017.08 is officially out, with GCC7 support!

# ¿ Sep 5, 2017 14:11

FlapYoJacks: Feb 12, 2009

BuildRoot 2017.11 is out!

The big prominent new feature I contributed is:
LibreSSL is now not complete poo poo!

The real OpenSSL was moved to libopenssl.
A new openssl virtual package was created.
LibreSSL and OpenSSL now both provide OpenSSL and a user can choose to select one or the other.

There are a few applications that don't fully support LibreSSL, these are:

hostapd
OpenLDAP
libevent
opusfile
FFmpeg
mosquitto
wpa_supplicant
softether (don't use this lol)
libpjsip
NTP (although you can use OpenNTPD instead)

I really really suggest people stop using OpenSSL if they can get away with it. It's a pile of poo poo.

Other things I contributed:
- SELinux packages are now updated to 2.7
- SETools is now updated to 4.0
- GStreamer packages have been updated to 1.12.3
- libpjsip: bump to 2.7.1
- snmp++: bump to v3.3.10
- Janus-gateway: bump to v0.2.5
- boost: bump to 1.65.1
- qemu: bump to 2.10.1
- glibmm: bump to 2.54.1
- libglib2: bump to 2.54.1
- sngrep: bump to v1.4.4
- Audit: bump to 2.7.8
- busybox: bump to 1.27.2

Things that are in master that didn't make it into this release:
- PostgreSQL is now at 10.1
- Refpolicy is now at 2.20170805. The main advantage is that the new refpolicy doesn't require python2.
- LibreSSL is at 2.6.3 in master and 2.5.5 in this release.

There are a ton of other small improvements as well. I would seriously suggest updating to 2017.11 if you are a BuildRoot user!

# ¿ Dec 4, 2017 16:43

Adbot: ADBOT LOVES YOU

# ¿ May 16, 2024 06:08

FlapYoJacks: Feb 12, 2009

As an anti-archive bump: BuildRoot 2018.02-LTS is out.

Support for this release is a full year with guarantees of security updates.

I only had 53 patches mainlined.

Noteworthy updates:

- Support for Meson
- Systemd is now at version 237 thanks to the above
- Gcc 7.3.0 support
- Support for Rust
- Getting ever so closer to a binary compatible build option.

Other changes can be seen here:
https://git.buildroot.net/buildroot/plain/CHANGES?id=2018.02

Edit* Next release will have gobject-introspection support. I have been working on it on and off for the last 6~ months.

# ¿ Mar 14, 2018 12:49

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Embedded Linux! Who needs security or new packages?