3665 stories
·
58 followers

Kees Cook: UEFI booting and RAID1

2 Comments

I spent some time yesterday building out a UEFI server that didn’t have on-board hardware RAID for its system drives. In these situations, I always use Linux’s md RAID1 for the root filesystem (and/or /boot). This worked well for BIOS booting since BIOS just transfers control blindly to the MBR of whatever disk it sees (modulo finding a “bootable partition” flag, etc, etc). This means that BIOS doesn’t really care what’s on the drive, it’ll hand over control to the GRUB code in the MBR.

With UEFI, the boot firmware is actually examining the GPT partition table, looking for the partition marked with the “EFI System Partition” (ESP) UUID. Then it looks for a FAT32 filesystem there, and does more things like looking at NVRAM boot entries, or just running BOOT/EFI/BOOTX64.EFI from the FAT32. Under Linux, this .EFI code is either GRUB itself, or Shim which loads GRUB.

So, if I want RAID1 for my root filesystem, that’s fine (GRUB will read md, LVM, etc), but how do I handle /boot/efi (the UEFI ESP)? In everything I found answering this question, the answer was “oh, just manually make an ESP on each drive in your RAID and copy the files around, add a separate NVRAM entry (with efibootmgr) for each drive, and you’re fine!” I did not like this one bit since it meant things could get out of sync between the copies, etc.

The current implementation of Linux’s md RAID puts metadata at the front of a partition. This solves more problems than it creates, but it means the RAID isn’t “invisible” to something that doesn’t know about the metadata. In fact, mdadm warns about this pretty loudly:

# mdadm --create /dev/md0 --level 1 --raid-disks 2 /dev/sda1 /dev/sdb1 mdadm: Note: this array has metadata at the start and may not be suitable as a boot device. If you plan to store '/boot' on this device please ensure that your boot-loader understands md/v1.x metadata, or use --metadata=0.90

Reading from the mdadm man page:

-e, --metadata= ... 1, 1.0, 1.1, 1.2 default Use the new version-1 format superblock. This has fewer restrictions. It can easily be moved between hosts with different endian-ness, and a recovery operation can be checkpointed and restarted. The different sub-versions store the superblock at different locations on the device, either at the end (for 1.0), at the start (for 1.1) or 4K from the start (for 1.2). "1" is equivalent to "1.2" (the commonly preferred 1.x format). "default" is equivalent to "1.2".

First we toss a FAT32 on the RAID (mkfs.fat -F32 /dev/md0), and looking at the results, the first 4K is entirely zeros, and file doesn’t see a filesystem:

# dd if=/dev/sda1 bs=1K count=5 status=none | hexdump -C 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001000 fc 4e 2b a9 01 00 00 00 00 00 00 00 00 00 00 00 |.N+.............| ... # file -s /dev/sda1 /dev/sda1: Linux Software RAID version 1.2 ...

So, instead, we’ll use --metadata 1.0 to put the RAID metadata at the end:

# mdadm --create /dev/md0 --level 1 --raid-disks 2 --metadata 1.0 /dev/sda1 /dev/sdb1 ... # mkfs.fat -F32 /dev/md0 # dd if=/dev/sda1 bs=1 skip=80 count=16 status=none | xxd 00000000: 2020 4641 5433 3220 2020 0e1f be77 7cac FAT32 ...w|. # file -s /dev/sda1 /dev/sda1: ... FAT (32 bit)

Now we have a visible FAT32 filesystem on the ESP. UEFI should be able to boot whatever disk hasn’t failed, and grub-install will write to the RAID mounted at /boot/efi.

However, we’re left with a new problem: on (at least) Debian and Ubuntu, grub-install attempts to run efibootmgr to record which disk UEFI should boot from. This fails, though, since it expects a single disk, not a RAID set. In fact, it returns nothing, and tries to run efibootmgr with an empty -d argument:

Installing for x86_64-efi platform. efibootmgr: option requires an argument -- 'd' ... grub-install: error: efibootmgr failed to register the boot entry: Operation not permitted. Failed: grub-install --target=x86_64-efi WARNING: Bootloader is not properly installed, system may not be bootable

Luckily my UEFI boots without NVRAM entries, and I can disable the NVRAM writing via the “Update NVRAM variables to automatically boot into Debian?” debconf prompt when running: dpkg-reconfigure -p low grub-efi-amd64

So, now my system will boot with both or either drive present, and updates from Linux to /boot/efi are visible on all RAID members at boot-time. HOWEVER there is one nasty risk with this setup: if UEFI writes anything to one of the drives (which this firmware did when it wrote out a “boot variable cache” file), it may lead to corrupted results once Linux mounts the RAID (since the member drives won’t have identical block-level copies of the FAT32 any more).

To deal with this “external write” situation, I see some solutions:

  • Make the partition read-only when not under Linux. (I don’t think this is a thing.)
  • Create higher-level knowledge of the root-filesystem RAID configuration is needed to keep a collection of filesystems manually synchronized instead of doing block-level RAID. (Seems like a lot of work and would need redesign of /boot/efi into something like /boot/efi/booted, /boot/efi/spare1, /boot/efi/spare2, etc)
  • Prefer one RAID member’s copy of /boot/efi and rebuild the RAID at every boot. If there were no external writes, there’s no issue. (Though what’s really the right way to pick the copy to prefer?)

Since mdadm has the “--update=resync” assembly option, I can actually do the latter option. This required updating /etc/mdadm/mdadm.conf to add <ignore> on the RAID’s ARRAY line to keep it from auto-starting:

ARRAY <ignore> metadata=1.0 UUID=123...

(Since it’s ignored, I’ve chosen /dev/md100 for the manual assembly below.) Then I added the noauto option to the /boot/efi entry in /etc/fstab:

/dev/md100 /boot/efi vfat noauto,defaults 0 0

And finally I added a systemd oneshot service that assembles the RAID with resync and mounts it:

[Unit] Description=Resync /boot/efi RAID DefaultDependencies=no After=local-fs.target [Service] Type=oneshot ExecStart=/sbin/mdadm -A /dev/md100 --uuid=123... --update=resync ExecStart=/bin/mount /boot/efi RemainAfterExit=yes [Install] WantedBy=sysinit.target

(And don’t forget to run “update-initramfs -u” so the initramfs has an updated copy of /dev/mdadm/mdadm.conf.)

If mdadm.conf supported an “update=” option for ARRAY lines, this would have been trivial. Looking at the source, though, that kind of change doesn’t look easy. I can dream!

And if I wanted to keep a “pristine” version of /boot/efi that UEFI couldn’t update I could rearrange things more dramatically to keep the primary RAID member as a loopback device on a file in the root filesystem (e.g. /boot/efi.img). This would make all external changes in the real ESPs disappear after resync. Something like:

# truncate --size 512M /boot/efi.img # losetup -f --show /boot/efi.img /dev/loop0 # mdadm --create /dev/md100 --level 1 --raid-disks 3 --metadata 1.0 /dev/loop0 /dev/sda1 /dev/sdb1

And at boot just rebuild it from /dev/loop0, though I’m not sure how to “prefer” that partition…

© 2018, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Read the whole story
jepler
4 days ago
reply
sort of a mess, but at least it's possible. ugh.
Earth, Sol system, Western spiral arm
Share this story
Delete
1 public comment
gcarothers
3 days ago
reply
Or why one sticks a USB UEFI boot device on to a server
Sebastopol, CA

Turn Right at the Burger King: Google Maps Begins Using Landmarks To Help With Guidance

1 Comment
Most navigation apps give you instructions based on streets or distance. But it's arguably in contrast to how people usually provide directions -- some usually point to landmarks that are easier to spot. Google sees some merit in that. The idea is that Google Maps is highlighting some landmarks and other points of interest (fast food restaurants) to help with guidance. TechCrunch reports that some users are already seeing this on Google Maps. And maybe to Google, this opens door for some business opportunities as well. Only time will tell.
Read the whole story
jepler
4 days ago
reply
it was doing this in India over a year ago... some people might like it.
Earth, Sol system, Western spiral arm
Share this story
Delete

When knitters teamed up with a neural...

2 Shares

[Make Caows and Shapcho - MeganAnn]

[Pitsilised Koekirjad Cushion Sampler Poncho - Maeve]

[Lacy 2047 - michaela112358]

I use algorithms called neural networks to write humor. What’s fun about neural networks is they learn by example - give them a bunch of some sort of data, and they’ll try to figure out rules that let them imitate it. They power corporate finances, recognize faces, translate text, and more. I, however, like to give them silly datasets. I’ve trained neural networks to generate new paint colors, new Halloween costumes, and new candy heart messages. When the problem is tough, the results are mixed (there was that one candy heart that just said HOLE).

One of the toughest problems I’ve ever tried? Knitting patterns.

I knew almost nothing about knitting when @JohannaB@wandering.shop sent me the suggestion one day. She sent me to the Ravelry knitting site, and to its adults-only, often-indecorous LSG forum, who as you will see are amazing people. (When asked how I should describe them, one wrote “don’t forget the glitter and swearing!”)

And so, we embarked upon Operation Hilarious Knitting Disaster.

The knitters helped me crowdsource a dataset of 500 knitting patterns, ranging from hats to squids to unmentionables. JC Briar exported another 4728 patterns from the site stitch-maps.com

I gave the knitting patterns to a couple of neural networks that I collectively named “SkyKnit”. Then, not knowing if they had produced anything remotely knittable, I started posting the patterns. Here’s an early example.

MrsNoddyNoddy wrote, “it’s difficult to explain why 6395, 71, 70, 77 is so asthma-inducingly funny.” (It seems that a 6000-plus stitch count is, as GloriaHanlon put it, “optimism”). 

As training progressed, and as I tried some higher-performance models, SkyKnit improved. Here’s a later example.

Even at its best, SkyKnit had problems. It would sometimes repeat rows, or leave them out entirely. It could count rows fairly reliably up to about 22, but after that would start haphazardly guessing random largish numbers. SkyKnit also had trouble counting stitches, and would confidently declare at the end of certain lines that it contained 12 stitches when it was nothing of the sort.

But the knitters began knitting them. This possibly marks one of the few times in history when a computer generated code to be executed by humans.

[Mystery lace - datasock]

[Reverss Shawl - citikas]

[Frost - Odonata]

The knitters didn’t follow SkyKnit’s directions exactly, as it turns out. For most of its patterns, doing them exactly as written would result in the pattern immediately unraveling (due to many dropped stitches), or turning into long whiplike tentacles (due to lots of leftover stitches). Or, to make the row counts match up with one another, they would have had to keep repeating the pattern until they’d reached a multiple of each row count - sometimes this was possible after a few repeats, while other times they would have had to make the pattern tens of thousands of stitches long. And other times, missing rows made the directions just plain impossible. 

So, the knitters just started fixing SkyKnit’s patterns.

Knitters are very good at debugging patterns, as it turns out. Not only are there a lot of knitters who are coders, but debugging is such a regular part of knitting that the complicated math becomes second nature. Notation is not always consistent, some patterns need to be adjusted for size, and some simply have mistakes. The knitters were used to taking these problems in stride. When working with one of SkyKnit’s patterns, GloriaHanlon wrote, “I’m trying not to fudge too much, basically working on the principle that the pattern was written by an elderly relative who doesn’t speak much English.”

Each pattern required a different debugging approach, and sometimes knitters would each produce their own very different-looking versions. Here are three versions of “Paw Not Pointed 2 Stitch 2″.

[Top - ActualJellyfishMiddle - LadyAurianBottom (sock version) - ShoelessJane]

Once, knitter MeganAnn came across a stitch that didn’t even exist (something SkyKnit called ’pbk’). So she had to improvise. “I googled it and went with the first definition I got, which was ‘place bead and knit’.” The resulting pattern is “Ribbed Rib Rib” below (note bead).

[Ribbed Rib Rib - MeganAnn]

Even debugged, the patterns were weird. Like, really, really nonhumanly weird.

“I love how organic it comes out,“ wrote Vastra. SylviaTX agreed, loving “the organic seeming randomness. Like bubbles on water or something,” 

SkyKnit’s patterns were also a pain. Michaela112358 called Row 15 of Mystery Lace (above) “a bit of a head melter”, commenting that it “lacked the rhythm that you tend to get with a normal pattern”. Maeve_ish wrote that Shetland Bird Pat “made my brain hurt so I went to bed.” ShoelessJane asked, “Okay, now who here has read Snow Crash?”

[Winder Socks (2 versions) - TotesMyName]

“I was laughing a few days ago because I was trying to math a Skyknit pattern and my brain…froze. Like, no longer could number at all. I stared blankly at my scribbles and at the screen wondering what had happened til somehow I rebooted. Yup, Skyknit crashed my brain.” - Rayn63

[Paw chain 2 - HMSChicago]

On the pattern SkyKnit called “Cherry and Acorns Twisted To”:

“Couple notes on the knitting experience, which while funny wasn’t terribly pleasurable: Because there’s no rhythm or symmetry to the pattern, I felt I was white-knuckling it through each line, really having to concentrate. There are also some stitch combinations that aren’t very comfortable to execute physically, YO, SSK in particular.

That said, I’m nearly tempted to add a bit of random AI lace to a project, perhaps as cuffs on a sweater or a short-row lace panel in part of a scarf, like Sylvia McFadden does in many of her shawl designs. As another person in the thread said, it would add a touch of spider-on-LSD.” -SarahScully

[cherry and acorns twisted to - Sarah Scully]

BridgetJ’s comments on “Butnet Scarf”:

“Four repeats in to this oddball, daintily alien-looking 8-row lace pattern, and I have, improbably, begun to internalize it and get in to a rhythm like every other lace pattern.

I still have a lingering suspicion that I’m knitting a pattern that could someday communicate to an AI that I want to play a game of Global Thermonuclear War, but I suppose at least I’ll have a scarf at the end of it?” -BridgetJ

[butnet scarf - BridgetJ]

There was also this beauty of a pattern, that SkyKnit called “Tiny Baby Whale Soto”. GloriaHanlon managed somehow to knit it and described it as “a bona fide eldritch horror. Think Slenderman meets Cthulu and you wouldn’t be far wrong.”

[Tiny Baby Whale Soto - GloriaHanlon]

Other than being a bit afraid of Tiny Baby Whale Soto, the knitters seem happy to do the bidding of SkyKnit, brain melts and all.

“I cast on for a lovely MKAL with a designer I totally trust and became immediately suspicious because the pattern made sense. All rows increase in an orderly manner. There are no “huh?” moments. There are no maths at all…it has all been done for me. I thought I would be happy, yo. Instead, I am kind of missing the brain scrambling and I keep looking for pigs and tentacles. Go figure.” - Rayn63

Check out the rest of the SkyKnit-generated patterns, and the glorious rainbow of weird test-knits at SkyKnit: The Collection and InfiKnit

There’s also a great article in the Atlantic that talks a bit more about the debugging. 

If you feel so inspired (and don’t mind the kind-hearted yet vigorous swearing), join the conversation on the LSG Ravelry SkyKnit thread - many of SkyKnit’s creations have not yet been test-knit at all, and others transform with every new knitter’s interpretation. Compare notes, commiserate, and do SkyKnit’s inscrutable bidding!

Heck yeah there is bonus material this week. Have some neural net-generated knitting & crochet titles. Some of them are mixed with metal band names for added creepiness. Enter your email here to get more like these:

Chicken Shrug
Snuggle Features
Cartube Party Filled Booties
Corm Fullenflops
Womp Mittens
Socks of Death
Tomb of Sweater
Shawl Ruins

Read the whole story
jepler
4 days ago
reply
Earth, Sol system, Western spiral arm
angelchrys
4 days ago
reply
Overland Park, KS
Share this story
Delete

Some Android Device Makers Are Lying About Security Patch Updates

1 Comment
An anonymous reader shares a report: Security patches for smartphones are extremely important because many people store personal data on their devices. Lots of Android phones out there get regularly security patches, but according to a new report, some of them are lying about the patches that they've actually gotten. According to a study by Security Research Labs, some Android phones are missing patches that they claim to have. Wired explains that SRL tested 1,200 phones from more than a dozen phone makers for every Android security patch released in 2017. The devices tested include ones from Google, Samsung, Motorola, LG, HTC, Xiaomi, OnePlus, Nokia, TCL, and ZTE. The study found that outside of Google and its Pixel phones, well-known phone makers had devices that were missing patches that they claimed to have. "We found several vendors that didn't install a single patch but changed the patch date forward by several months," says SRL founder Karsten Nohl.
Read the whole story
jepler
9 days ago
reply
geez: "The study found that outside of Google and its Pixel phones, well-known phone makers had devices that were missing patches that they claimed to have. "We found several vendors that didn't install a single patch but changed the patch date forward by several months," says SRL founder Karsten Nohl."
Earth, Sol system, Western spiral arm
Share this story
Delete

STAR WARS EP 6: The Last Laser Master

1 Comment
From: Auralnauts
Duration: 50:43

The conclusion of the epic saga that didn't used to have a coherent story but now kind of does. See below for links and additional info

Help us to keep making awesome stuff!
The Last Laser Master Album:
Bandcamp: https://bit.ly/2q7XStQ
Play: https://bit.ly/2Jnr2x6
Amazon: https://amzn.to/2uR5Dcs
Website: https://www.auralnauts.com/
Patreon: https://www.patreon.com/auralnauts
Merch: https://www.auralnauts.com/shop
Podcast: https://www.auralnauts.com/podcast
Music: https://www.auralnauts.com/music
Instagram: http://bit.ly/2qdzFjM
Twitter: https://twitter.com/Auralnauts
Facebook: https://www.facebook.com/auralnauts/

Cast Links:
Jon Bailey: https://www.youtube.com/user/jon3pnt0
Silver Letomi: https://www.youtube.com/user/SilverLetomi

Music Playlist:
You have the Power - Trevor Jones: 00:04
The Blue Danube Waltz - Johann Strauss II: 01:50
Concerto for Orchestra - Bela Bartok: 05:33
Gamin' Time - Helen Henny & Chuck E. Cheese: 06:44
Birthday Attack - Auralnauts: 08:11
Cherokee Deliverable H-Bomb - William Stromberg & John Morgan: 09:40
The Show will Begin Shortly - Auralnauts: 11:53
The Last Laser Master - Auralnauts: 12:40
Wanna Go? - Auralnauts: 16:43
Wishing Well & The Fratelli's Find Coin - Dave Grusin: 17:46
Mikey's Vision - Dave Grusin: 18:19
Tree World (We Go Hard) - Auralnauts: 20:43
Predator Main Title - Alan Silvestri: 22:17
The Rope Bridge - John Williams: 24:31
The Mine - Jerry Goldsmith: 25:28
They're Here & Skull Cave Chase - Dave Grusin: 26:33
Dip - Danny Brown: 26:47
Que Dolor - Fanfare Ciocarlia: 28:39
Ronde No. 3 - Tielman Susato: 32:17
Atmosphere Station - James Horner: 33:31
Revalation - Auralnauts: 34:43
Mantra - Noisia: 38:26
The Han Zone - Auralnauts: 39:08
Restaurant Trash - Dave Grusin: 40:56
Ronde No. 1 - Tielman Susato: 41:41
Tarawa - James Newton Howard: 45:09
Pee Break - Dave Grusin: 47:37

Read the whole story
jepler
18 days ago
reply
oh sh---, 50 minutes !?
Earth, Sol system, Western spiral arm
emdeesee
5 days ago
"Duke, you have to stop Cree--"
Share this story
Delete

Debian GNU/Linux port for RISC-V 64-bit (riscv64) in Debian infrastructure (debian-ports)

1 Comment

tl;dr: We have a new port for RISC-V, flavour riscv64 (64-bits little-endian) in Debian Ports.

And it's doing well.

Debian-Ports 2-week Graph, 2018-03-31

Debian-Ports 2-week Graph, 2018-03-31

(PS: Despite the date still in some timezones... no, this is not an April-Fools-day type of post :-) ).

Ancient History

Almost a year ago I wrote a post announcing the availability of a Debian GNU/Linux port for RISC-V 64-bit (riscv64).

At the time it was an incomplete Debian system, with several of the most important pieces missing (toolchain: gcc, glibc, binutils), and with everything kept outside the Debian infrastructure.

The main reason for that was because support for the toolchain had not been submitted upstream, and the ABI was not very stable. It was changing often and in important and disruptive ways, without even as much of a public announcement before the breakage was noticed.

Last pieces of the toolchain upstreamed, at long last

Fortunately, over the last year, support has been merged in GNU binutils first (2.28), then GCC (7), then Linux 4.15 (pre-requisite for glibc), and finally in glibc 2.27, released in the first days of February.

Support is still not complete, for example riscv32 targets have not been merged in Linux mainline (but this is not important for us, unless someone else wants to start such a port), basic drivers are still missing from mainline, the support in GDB and Qemu is becoming available only now, there are many bugs that need shaking, etc.

Bootstrapping again, 2018 edition

Despite the very recent support, the ecosystem and software support is mature enough that we could, with the only help of qemu-system-riscv64, progress through the following steps:

1. Cross-compile a base set of packages

Dates: prior work for a few years (almost since the public announcement of RISC-V, in 2014); current bootstrapping since end of Feb until ~5th of March.

Apart from previous bootstrapping attempts, during the last months prior to proper starting this one I worked on clearing the path, for example by NMUing (Debian lingo for fixing without “ack” from maintainer, no reply in weeks/months, or “ack” agreeing that I went ahead) packages with patches proposed by Helmut and porters of other architectures (e.g. many from Daniel Schepler) and without any reply for a long time, or sometimes proposing new patches or new ways to solve the problems, etc.

After that, at the end of February and along with Karsten and Aurélien, we went through a round of cross-compiling, mostly with the help of Helmut Grohne's rebootstrap, but also solving additional problems like using gcc-8 rather than gcc-7 because it failed to work properly for a few days, also pulling parts of the toolchain from experimental instead of unstable and having different versions around causing conflicts, an experimental package of glibc 2.27 still not present in unstable at the time, which rendered packages such as GNU make unable to compile without patching, etc.

We also cross-compiled a significantly higher amount of packages than those considered by rebootstrap at the moment; some of them with crude hacks, like removing build-dependencies or avoiding to build man pages when they needed to run riscv64 native code for that. Sometimes is just faster to cross-compile and have many packages ready for when they are needed needed, even if sometimes cross-built packages don't work 100% correctly or not at all.

Many of those package are not essential to run a Debian system, but are needed later in the initial steps of “native” building. For example, the compiler/toolchain itself, which the rebootstrap script doesn't attempt to provide; or GNU make or cmake, which are often not needed to run a Linux system, but are required to build a great deal of software.

As a result of this phase, we have submitted bugs (or in some cases, uploads after a period of silence), with many still pending to send or apply, to try to sort these problems out in a clean way for other ports in the future (and better rebootstrapping and cross-compilation support in Debian overall), with the use of different tools like build profiles to avoid unwanted dependencies or to break dependency cycles, or moving the generation of documentation to an arch:all package, etc (so it's done once for all architectures, and riscv64 or any other arch-builders don't have to bother about it).

2. Use this set to launch “native” systems with qemu-system-riscv64

Dates: ~5th to ~13th of March.

Having built a “native” system from cross-built packages, we could compile packages natively that were difficult to cross-build, and also to run tests for packages providing them (so one can have some certainty that the package behaves sanely), and trying to have as much a clean native system as possible.

For example, one of the first packages needed and that we built natively is Perl (which is pretty basic for any Debian systems, and that we couldn't get to cross-compile easily).

We recompiled all (or almost all) packages that we got cross-built “natively”, and by running tests when possible, we uncovered some latent bugs in the toolchain and qemu, and in some packages.

The result was a set of packages (that we named “native-1”) enough to run systems equivalent to those installed in machines acting as automatic builders for other architectures, capable of building and running native "sbuild", "schroot", etc. We had to cut some unimportant dependency or feature here and there, but those changes did not affect the packages of the next set, or only marginally.

3. Build “native-2” to import to debian-ports

Dates: ~13th to ~23rd of March.

With systems installed from “native-1”, we have built again a restricted set of packages for a very basic Debian system, to act as “seed”, with as few differences as possible with the current state of Debian unstable, with the aim to import them in the Debian-ports infrastructure as cleanly as possible.

Intentionally, we built as few packages as possible, so most of the packages would go through the standard mechanisms.

A few packages have been built with the help of build-profiles support, for example to avoid dependencies on Java's default-jdk (openjdk-9-jdk at the moment), because we do not have that one yet.

Another typical case is disabling tests, after attemping to pass them, because a few cases failed among hundreds of them passed, often due to timeouts (system emulation being quite slow, this happens very often).

But most packages have been built completely unmodified; and those who were, will be rebuilt later when their dependencies are satisfied.

4. Import the base set “native-2” as “seed” and set-up automatic systems

Dates: ~23rd to ~25th of March.

Set everything up to build the whole archive in as much of a standard and unattended way as possible.

5. Currently, solve problems like breaking dependency cycles

Dates: ~25th of March, still on it.

In the current phase, the process as a whole is not “unattended”, because we have to build packages with a reduced set of features or build-dependencies, to break cycles (e.g. cmake depending on qt for its GUI tool, qt ecosystem depending on cmake).

To get close to 100% of the archive built, there are many important dependency cycles yet to break, some packages need patches specific to the architecture, etc.

Nevertheless, thousands of packages have been built by now, most of them completely unattended and in the same way as the other architectures available in Debian.

Current Status

In last few days and weeks, status was sometimes changing by the hour. That, along with -ENOTIME, are the main reasons why this post has been delayed until now, and the news have not been publicised widely.

We just focused on getting things done :-)

But to give a general idea of where we are now:

  • Over 4100 packages (> 30% of the source-packages of the whole archive that are not arch-independent) have been built by now.
    • Since arch-independent packages can be installed in this architecture (e.g. many modules of languages like Python, documentation, etc.), about 65 to 70% of the Debian archive is available to use.
    • As explained before, most of the packages have been built completely unattended and in the same way as the other architectures available in Debian.
    • Progress of automatic builds as it happens: https://buildd.debian.org/status/architecture.php?a=riscv64&suite=sid
  • All of this work is not tested on real hardware yet.
    • Outside of FPGA implementations, the first hardware publicly/easily available will only surface in the next few weeks, only in small quantities and expensive). We hope, and in particular I have a great deal of confidence, that the software will basically work, but... yeah.
  • We find still fairly often tests that fail to pass or get locked-up in qemu, etc.
    • It's difficult to know if these are bugs in the software, compiler/toolchain, emulation, or a combination of them.
    • Still, since we got over 30% of the arch-dependent packages of a huge collection like Debian built, passing tests for most of them when available, things do not look too bad.
  • We still are breaking dependency cycles that prevent many packages to be built.
    • For example we do not have almost any software depending on complex GUI (think of KDE, GNOME, browsers), there is still no support for languages such as Java, etc.

Who

Listing people working on different areas and at different times:

  • Karsten Merker and I for a long time, participating in RISC-V mailing lists and some events
  • Karsten Merker and I cross-building and using rebootstrap, Karsten is still focusing on that area
  • Aurélien Jarno helping more actively since the preparation of pre-releases of glibc 2.27, and other parts of the tool-chain during cross-compilation
  • Aurélien Jarno and I working closely together in the last phase of cross-compilation and fine-tuning of the toolchain; then to build the native sets of packages to bring riscv64 to debian-ports, setting-up buildds, etc.
  • Additional support by Kurt Keville of the MIT RV128 team for a couple of years now
  • Bytemark, one of the big sponsors of Debian (and other FOSS projects), sponsored a machine where I did most of the work during the previous years and up to the packages built to import into Debian ports

Apart from these people/resources, we have on our side:

  • all the people working on cross-compilation and multi-arch over the years (Standing on the Shoulders of Giants, as if it were)
  • Helmut Grohne's focus and tenacity on rebootstrap over the years, fixing problems all along (already mentioned previously)
  • many maintainers adding support eagerly to their packages (sometimes not trivial at all)
  • people starting to submit patches and requests to have packages with RISC-V support ready to install, e.g. qemu-system-riscv64
  • and various people offering help and giving support along the way, too many to list here and for my poor brain to remember.

How to Use

Download packages or create your own image from: https://wiki.debian.org/RISC-V#Package_repositories

We will try to provide pre-built images or easier ways to test in the near future.

How to Help

Please refer to:

Future Work

Usual stuff for other ports, like:

  1. Be able to build most of the archive
  2. Create images to install or ready to use in qemu, etc.
  3. Make it a first-class, full-supported architecture in future Debian Stable releases
Read the whole story
jepler
22 days ago
reply
now do a graph with each architecture's "10%" at the same baseline time. I expect the riscv64 curve to hit 75% pretty quickly, say by May 15.
Earth, Sol system, Western spiral arm
Share this story
Delete
Next Page of Stories