Fixing the Sandbrige MacBook Air display initialization

We left our hero with Debian installed on the MacBook Air, but with the display getting scrambled as soon as the i915 driver loaded.

As was reported to Matthew, the problem is as simple as a lack of the right mode for the eDP panel in the machine. This mode is supposed to come from the panel EDID data, but for some reason the driver wasn’t able to query the EDID data, and so it decided to try some random panel timings it dug out of the VBT tables, which are generally supposed to be used by LVDS panels. Apple helpfully stuck valid data there, but for some other panel — one that is 1280x800 pixels instead of the 1366x768 pixel panel in the MacBook Air.

I heard rumors that some machines would get a black screen when the i915 driver loaded. I was fortunate — my machine simply displayed a 1366x768 subset of the programmed 1280x800 mode. A bit of garbage on the right side, and a few scanlines missing at the bottom. Quite workable, especially after I ran fbset -yres 768 to keep the console in the visible portion of the screen.

DDC failure

Looking through the kernel logs, the Intel driver tries to access the EDID data and times out, as if DDC is just completely broken. This is rather unexpected; the eDP spec says that the panel is required to support DDC and provide EDID. Now, we’ve seen a lot of panels which don’t quite live up to the rigorous eDP specifications, but it’s a bit surprising from Apple, who generally do VESA stuff pretty well.

We’ve heard reports about panels reporting invalid EDID data, or EDID data which didn’t actually match the panel (causing us to prefer the VBT data on LVDS machines). But I’ve not heard of an eDP panel which didn’t have anything hanging off of the DDC channel.

But X works fine?

During early debugging, I happened to start X up. Much to my surprise, X came up at the native 1366x768 mode. Digging through the kernel logs after that, I discovered that EDID was successfully fetched from the eDP panel while X started up.

At this point, I knew it was all downhill — the EDID data was present, it just wasn’t getting picked up during the early part of the driver initialization when the console mode is initialized.

eDP power management

The CPU is given complete control over the power management of the eDP panel; sequencing through various power states and waiting appropriate amounts of time when things change. Given the goal of keeping power usage as low as possible, this makes a huge amount of sense.

The eDP spec is quite clear though, without power, the panel will not respond to anything over the aux channel, and that includes EDID data. The eDP panel power hardware in the Sandybridge chip has a special mode for dealing with this requirement. If the panel is not displaying data, you can supply power for the aux channel stuff by setting a magic bit in the panel power registers.

When initializing the frame buffer, the kernel driver turns off the panel completely so that it has all of the hardware in a known state (yeah, this is not optimal, but that’s another bug). When X started, the panel was already running with the console mode.

Given the difference between these two states — EDID querying with the panel off failed, while EDID querying with the panel on worked, it seemed pretty clear that the panel power wasn’t getting managed correctly. So, it seemed pretty clear that the magic ‘power the panel’ bit wasn’t getting turned on at the right times.

Getting the power turned on.

I stuck a check inside all of the aux channel communication functions to see where things were broken. This pointed out several places missing the panel power calls. This wasn’t quite sufficient to get EDID data flowing. The remaining problem was that the code wasn’t waiting long enough after turning the panel power on before starting the aux channel communication. A few msleep calls and huzzah! EDID at boot time and the console had the right mode.

Making it faster

However, it turns out that the driver does this a lot, and the msleeps required were fairly long — the eDP panel wants a 500ms delay from turning the panel power off before you can turn it back on.

I fix this by simply delaying the panel power off until things were idle for a ‘long’ time. Now mode setting goes zipping through, and a few seconds later, the bit to force panel power on gets turned off.

Getting these bits for yourself

I’ve pushed out the code to my (temporary) kernel repository git://people.freedesktop.org/~keithp/linux in the fix-edp-vdd-power branch. I’d love to hear if you’ve tried this on either a MacBook Air or any other eDP machine from Ironlake onwards.

Installing Linux on Sandybridge MacBook Air

Matthew Garrett got a report from someone who bought a brand new Sandybridge MacBook Air and was trying to install Fedora on it. The screen went black as soon as the i915 driver tried to set the initial mode.

I bought one to try and help out. Just getting Linux installed turned out to be a minor adventure and I thought I’d write a few notes for the next person who comes along and tries to do this.

The obvious method

I downloaded a collection of Debian ISO images, live images, netboot images from testing and CD-1 of squeeze. I downloaded a Fedora 16 alpha XFCE live image. I burned all of these to actual CDs, and then stuck them in a CD drive and held the ‘c’ key while booting the machine.

Nothing worked. The CD would spin up, the screen would switch from white to a black text mode with a blinking cursor at the upper left and then then CD would stop.

rEFIt attempts

rEFIt provides a boot selection menu and various configuration bits to provide for a multi-boot environment. This is necessary to leave the OS-X install on the disk and also install Linux.

It also lets you boot from removable media, or at least that was the story. rEFIt would list removable USB flash storage devices, but it would not list CD drives, even after the CD drive spun up and did stuff for a while. This makes me wonder if the ‘c’ key technique wasn’t actually trying to boot from the CD.

However, rEFIt wouldn’t boot from a grub2-enabled USB key.

ISO on USB flash key

What rEFIt will boot is a Fedora or Debian ISO image copied directly to the USB flash device. The Ubuntu 11.04 live image and it did not work.

Fedora F16 Alpha

The live image came up and started X in frame buffer mode. However, when I went to install the image to the hard disk, it got stuck trying to discover devices. I waited about 20 minutes and it never finished, although it did chew up a lot of CPU time.

Debian Squeeze

I copied CD-1 of the 32-bit Squeeze distribution to my flash key. That booted up to text mode and went through the install. Nothing really unusual until it came to installing grub. I let it install grub to /dev/sda4, the new partition I had created for Linux. I figured installing grub to /dev/sda would be a bad idea (I think it would have actually worked just fine in retrospect).

A slight out-of-sync diversion here — I couldn’t get the 32-bit kernel to talk to any keyboard or USB networking device, so I eventually installed a 64-bit kernel and that works just fine. You can run a 32-bit user space with a 64-bit kernel without any trouble.

Making EFI boot

Naturally, I couldn’t get the machine to boot to Linux after this. rEFIt, the multi-boot tool for Mac OS X appears only interested in booting EFI images, or at least was not interested in booting my BIOS-based Grub2 installation on /dev/sda4.

I used the ‘rescue’ mode on the USB flash drive to get back to my installed Linux image and installed grub-efi-amd64. The OS X EFI version does not support 32-bit EFI code, which seems entirely reasonable to me.

What took a while was figuring out that all of the grub files need to be readable through EFI, and that the Mac EFI code can only read FAT (and HFS?) partitions.

So, the trick appears to be to stick the grub files on the Mac OS X efi partition (/dev/sda1). I created a directory over there, /EFI/grub and then mounted that as /efi and made a symlink from /boot/grub to /efi/EFI/grub. Now, grub-install sticks the files in the right place, and update-grub will even place grub.cfg in the directory where it needs to go.

At this point, rEFIt will happily find the core.efi and grub.efi files and show them in the list of possible operating systems to boot. I don’t know why there are both core.efi and grub.efi files; I’ve only ever picked ‘grub.efi’ and that has worked nicely.

Broadcom WiFi

I haven’t gotten that working yet; the b43 supported chip list says it works in kernel 3.1. I’m busy building a 3.1-rc4 kernel as I type this, so perhaps that will ‘just work’ when I boot that.

i915 KMS

The whole reason for doing the install is to fix problems with the eDP EDID discovery which is currently preventing the machine from running X. I’ll be working on that next; a quick kludge to just read the current mode from the device should get something on the screen (we already do that for LVDS panels). Beyond that, I’ll be trying to figure out whether I can make EDID actually work, or whether it’s just not available in this machine.

Somehow the kernel frame buffer ended up as 1280x800, which was quite annoying. As a temporary kludge, I ran fbset -yres 768 to at least avoid having text disappear off the bottom of the screen. Once the panel size is discovered correctly, that should be fixed.

Status

This is sure not like installing Linux on a regular PC. However, I’m reasonably hopeful that the devices in the machine will be working pretty well under Debian within the next few days.

Rebuilding the Freedesktop.org Server Complex

Ok, so it’s really only two machines at present, but we have big plans.

As Daniel described in some detail, the machine was compromised and we’ve spent the last week rebuilding it. But, I wanted to describe our current setup and what we’d like to see in the near future.

The worst effect of the breakin was to disable all of our services, from web to email and even CVS. That’s clearly not acceptable, and I think we can expect future security issues with our vast collection of random web services. This points to the usual solution — use separate boxes for separate services. Based on conversations with other people running similar sites, I think we need:

• A Web hosting machine. Running apache2 and as little else as possible. User accounts would normally have access to this machine.
• A CVS machine. User accounts would not normally have access to this machine. CVS access is via SSH only. Project admins would have shell access to this machine to fix CVS repositories.
• An Email machine. Sysadmin access only.
• A security bastion for LDAP. LDAP updates can be managed by GPG-signed emails. Again, sysadmin access only.

We’ve got three machines available at present:

• gabe, aka freedesktop.org. This is the HP server that Jim and I wrangled into our budget last year. Dual 2.8G Xeon processors with 512K caches (the 1M cache chips were not available). 3.5G of memory, six 38G scsi disks in a hardware raid array. The only “real” server we own, it’s currently serving as web, mail and cvs server.
• tycho, aka fooishbar.org. This is an older dual PIII server that I got from VA Linux Systems back in the day. It’s got a couple of reasonable disks, but no RAID setup. It’s certainly not powerful enough for either HTTP or CVS. But, it should make a fine mail server or perhaps LDAP. It needs a reinstall job, and we need to kick daniels off of it.
• kara. This is a recent addition, a dual opteron box with IDE drives. It’s really a desktop machine, and has the (current) distinct disadvantage of not being rebootable. We’re using it as the main LDAP server for now, but I think that better be temporary until we can get tycho rebuilt.

We’ve stuck in a budget request for another “real” server. If that happens we can use that for CVS. It would be nice to get another small server for mail or ldap so that we could avoid using the dual opteron box; it’s a desktop machine and isn’t likely to be as reliable as we’d like to see.

X.org is also looking into building another site; I imagine we’ll be able to piggy-back on that effort and mirror the fd.o content whereever x.org lands.

Migrating the herd

Our new host, annarchy, is destined to become the place where we provide user accounts and web hosting. User accounts currently share the same machine, gabe, as cvs and mail. Moving from this configuration should improve performance and reduce the impact of security compromises to our web server or user accounts.

I’m just copying all of the user accounts from gabe to annarchy, and then at some point (soon), I’ll flip the big switch and ‘users.freedesktop.org’ will suddenly point at the new machine. I’ll leave logins to gabe enabled until the web services are transferred, but home directory contents will disappear from gabe and reappear on annarchy. That machine has a separate partition for user accounts, which should eliminate all of the ‘fun’ we’ve occasionally had when people filled up the partition shared with CVS and the web server.

I’ve already done a preliminary rsync of user directories to annarchy, so the remaining work is mostly to switch DNS entries and move home directory content out of the way on gabe. I’d like to get that done this week, but I don’t know whether to wait for the web content to move all at the same time or not.

Affiliation with Software in the Public Interest

Freedesktop.org is all about getting things done, and most of the time that’s just a matter of finding people to do whatever work needs doing, be that development, system administration or adding documentation to one of the wikis.

Sometimes, we need to buy hardware or services with actual money, and for that we constructed an LLC last year; prompted by the generous donation from Google of funds sufficient for most of our current hardware stack. Through this experience, it became clear that some “real” organization that offered greater transparency and structure to our limited financial dealings would be helpful. However, none of us felt like we could afford the time and energy necessary to construct such an organization on our own.

Software in the Public Interest was created to hold the assets of the Debian project, and has since expanded to fulfill a similar role for many other projects, among them PostgreSQL, wxWidgets and Drupal. They provide precisely the kind of structure that freedesktop.org needs right now; focusing purely on helping manage the financial resources of an organization without being involved in the political or social structure. Their sole responsibility is to ensure that any tax-exempt donations are managed in accordance with various legal obligations, including the disposition of funds should the project cease to exist.

I think the affiliation of freedesktop.org with SPI will benefit us and make it possible to keep our project running smoothly into the future and help continue our advance towards world domination.

Moving wikis from gabe to annarchy

With Google’s generous help, freedesktop.org has three fun new machines to play with. Part of the migration process involves moving all of the wikis from gabe to annarchy and I spent the day figuring out how that will work.

One of the big features of this move is that we can (I hope) afford to run more current software on the machine used to house the web server and user home directories. For our wiki, which is running MoinMoin, that means moving from version 1.2.4 to 1.4.99 or so. Oddly, there have been some changes in how the wiki is configured.

I’ve got a small demo wiki running at http://annarchy.freedesktop.org. That shows the new version running with canned content from the MoinMoin installation. Migrating the wiki configurations to the new format is pretty easy; requiring changes to the virtual host entries used for apache and replacing the per-host configuration files with one generated from the template provided with MoinMoin.

MoinMoin also has some migration scripts to get from 1.2.4 content format to 1.4.99, I need to figure out how to use those so we can move content across. I’m hoping these will ‘just work’, and there’s a reasonable chance they will. Of course, that’s only half the battle; I’m sure there are hard-coded assumptions in many of our wikis about what host the wiki software is running on. I guess we’ll get to find that out.