New beta release for YDL's kernel for PS3

I tested out today new beta kernel released for PS3. It indeed seems that some progress was done with firmware 2.00. I can nicely see that access point scan shows better values for my WPA1+2 network. Actually it shows now new WPA2 values.

Now everything would had been nice if YDL had also updated this aspect on their kernel update... Now it tells that we have this new type of security and this version of driver doesn't actually know how to handle it correctly... Rebooting and trying to reload network again (/etc/init.d/network restart) got it associated nicely.

In bleeding edge driver there are new encryption modes that should be settable with 'iwpriv eth0 set_alg 4' for WPA2-PSK TKIP and 'iwpriv eth0 set_alg 5' for WPA2-PSK AES. But YDL's kernel driver doesn't accept values of 4 and 5... So if you are running WPA1 only network, then your setup should work fine (haven't tested it out though), if you also have WPA2 enabled then you may have some problems making connection.

New PS3 firmware 2.00 breaks wireless support on Linux

It seems that after upgrading to new Playstation 3 firmware (2.00) it seems hypervisors wireless support is incompatible with Yellow Dog Linux 5.0.2. This means that you can't get connected with wireless connection. Haven't tried with newest kernel from newest Cell Linux Distributors kit yet, but at least ydl is now broken. So if you need wireless support you'd better wait for either updated kernel package or next firmware release.

Guide to get WPA-PSK working on PS3 - YDL 5.0.2

There were some requests for easier guide on how to get WPA working on Playstation 3 for Yellow Dog Linux 5.0.2. I looked around and found some free webspace for prebuilt binaries and for modified source RPMs. Please note that you will need your YDL root account in order to complete steps below.

This guide was last updated on: 4.8.2007

1. Update your wireless tools package

Yellow Dog Linux 5.0.2 comes with wireless tools package compiled for PowerPC-32. While this may work for other systems with Playstation 3 this is a bit problematic as ioctl32 wireless interface is a bit broken. So easiest workaround for this is to compile wireless tools as PowerPC-64 binaries. You can get modified source RPMs and pre-built binaries for easier installation from my packages website. Most users just need to download wireless-tools-28-1.ppc64.rpm and install it. You can use USB Flash Memory or any other medium to transfer files.

rpm -Uvh wireless-tools-28-1.ppc64.rpm

If it complains for conflicts add --force to command line (usually don't do this, but I didn't want to update release numbers in case Yellow Dog Linux releases updated packages to replace this). Please note that this is modification of YDL's source RPM package so it is a bit older that what kernel would like to use. It will generate some warnings, but you can ignore those.

2. Test that wireless tools works

Next step is to test out that wirless tools works. Tools needed in this experiment are ifconfig, iwconfig, iwpriv, iwlist, and dhclient.

First lets test out that you can wireless module loaded correctly. For this we will use iwconfig:

iwconfig eth0

This will display something like (your details may be different):

eth0 radio off ESSID:off/any
Mode:Managed Frequency:2.462 GHz Access Point: Not-Associated
Bit Rate:54 Mb/s
Encryption key:off
Link Quality:0 Signal level:0 Noise level:0
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:0 Invalid misc:0 Missed beacon:0

Next let's make sure you have clean configuration for your wireless connection, command eth0 down and up:

ifconfig eth0 down

ifconfig eth0 up

Next you can try to search for access point to connect to using iwlist.

iwlist eth0 scanning

This will display something like (your details will be different):

eth0 Scan completed :
Cell 01 - Address: 77:88:99:AA:BB:CC
ESSID:""
Protocol:IEEE 802.11bg
Mode:Managed
Channel:11
Encryption key:on
Quality=100/100 Signal level=-42 dBm
Bit Rates:1 Mb/s; 2 Mb/s; 5.5 Mb/s; 11 Mb/s; 18 Mb/s
24 Mb/s; 36 Mb/s; 54 Mb/s; 6 Mb/s; 9 Mb/s
12 Mb/s; 48 Mb/s
Extra:bcn_int=100
Extra:ap_sec=0201

You should note last line per entry Extra:ap_sec=. This number will tell what encryption algorithm is needed for your connection. 0201 means WPA-PSK - TKIP and 0202 would mean WPA2-PSK - AES. I have not been able to test out WPA2-PSK - AES, but it might work for you. For me 0100 shows up for WPA2-PSK - AES which is incorrect as that would mean WEP encryption.

Next lest try manually connecting to Access Point. For this you will need to know your access point's ESSID, used encryption algorithm (WPA-PSK (TKIP), WPA2-PSK (AES)) and passphrase to access network.

iwpriv eth0 set_alg 2

iwconfig eth0 key s:passphrase

iwconfig eth0 essid ESSID

If you have WPA2-PSK replace set_alg 2 with set_alg 3. For WEP this would be set_alg 1. After that you can observe output of iwconfig eth0 to see if you have been associated to access point (Access Point field changes from Not-Associated to MAC address of your AP).

Next you can check out if you can get an IP address with dhclient.

dhclient eth0

If it gives you an IP address you are free to test out your internet connection with your web browser or any other tool that needs connection.

3. Setting it up automatically on boot

Now that you have Wireless connection working it would be nice that it would work automatically after being rebooted.

There are only two files to be modified.

First, lets modify /etc/sysconfig/networking/devices/ifcfg-eth0.

Set following fields to values shown below (of course change your ESSID) (other fields can be left empty):

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
ESSID=ESSID
TYPE=Wireless

Next we need to store your passphrase so it will be automatically configured.

Edit /etc/sysconfig/networking/devices/keys-eth0 to contain following:

KEY=s:passphrase
IWPRIV="set_alg 2"

Please not that those values are the same that you used during testing.

After this you can try it, first bring eth0 down and then back to up:

ifdown eth0

ifup eth0

Thats all!

How to get YDL 5.0.2 + PS3 + WPA working

First you need to recompile wireless-tools package as PowerPC64. Currently there is a bug in kernel (or feature) that does not allow ioctl32 calls to be used correctly. Therefore this causes ydl's default wireless-tools package not to work as it is compiled as PowerPC32.

To recompile it you need source RPM for wireless tools package: wireless-tools-28-1.src.rpm. Install it. It should extract some files under /usr/src/yellowdog/*/*. Unfortunely this package has problems compiling for ppc64 so this needs to be fixed first. I have made a patch for it which can be found from yellowdog-devel mailing list. Basicly you need to copy that new makefile patch file over older makefile patch file and then you can build it. After that just run "rpmbuild -bb wireless-tools.spec --target=ppc64" and it builds could of RPMs to /usr/src/yellowdog/RPMS/ppc64/. Install wireless-tools-28-1.ppc64.rpm package (rpm -ivh <pkg>). After that you can use iwconfig, iwpriv tools nicely and configure WPA.

To get it working automatically you must modify two files:
/etc/sysconfig/networking/devices/ifcfg-eth0
/etc/sysconfig/networking/devices/keys-eth0

For ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
ESSID=<your ESSID>
TYPE=Wireless

(others can be left empty)

If you don't use DHCP for some reason configure rest how you like.

For keys-eth0:

KEY=s:<your passphrase>
IWPRIV="set_alg 2"

Change "set_alg 2" to "set_alg 3" if you have WPA2-PSK (AES). I didn't get WPA2 working and I think it is hypervisor's firmware issue (as it works correcly in GameOS). This might be relative you Wireless Accesspoint in use or your PS3 hardware & firmware version. Check how it works, and tell us if you have success.

Oh. YDL's wireless kernel driver is a bit old, so AP scan might not work correctly. It is possible to compile newer driver for YDL but I won't go to details this time.

Serving mp4 files to PS3

Playstation 3's webbrowser seems to be quite picky on what it accepts to be downloadable. On net there was some hints to rename filenames as avi. But at least for me it downloaded them but at least my mp4 trailer was reported as corrupt. Perhaps there should be correct AVI container data present for that trick to work. However that made me think. Perhaps my local server was just missing correct MIME type. Googled a bit and it seems that mp4 files should be recognized as video/mp4. Reconfigured apache to recognize this and voilá! mp4 files downloads nicely from my home server. And as extension was correct PS3's video player played that mp4 trailer nicely.

Mylyn looks great!

With recent release of new Eclipse Europa there was new tools introduced or at least new to me. One of them was Mylyn, a task based workflow integration to Eclipse. I saw some web presentations with demonstration how nicely it integrated with Java work. It looked promising but there wasn't any info how it integrates with CDT or with other stuff. So I wanted to test it out. And yes, it looks like its source listing feature also works with CDT! Yay! This looks really promising and useful feature when working with large sets of code. It just needs a trac or bugzilla so it can share those filtering settings. Something to bake on my development server ;).

New Eclipse and CDT is released!

Yeay! We got new Eclipse Europa release. While checking it out I found about mylyn project. It integrates task based working to Eclipse IDE. Watched about some web presentation about it and it looks fine addition. Need to try this one later one. Now that new release CDT 4.0 is out I would propose that anyone working with C/C++ code would test it out :). It does not have great list of toolsets configured at the moment, but you can always use just makefiles. There are tons of other stuff that is new... probably takes ages to try them out :)

First wireless WPA connection with PS3 Linux!

It seems that PS3 Linux's gelic wireless driver is not designed to work with wpa_supplicant, instead it provides services (through hypervisor) to enable WPA & WPA2 security. I got nice tip from author of the original driver on how to setup it. Only problem seems to be that from that last screenshot of iwconfig, I installed new firmware upgrade (1.82)... This does not seem to report WPA2+AES correctly anymore. During debugging it, I noticed that current driver does not seem to handle association reporting correctly to Linux wireless stack so I had to add some code to report associations correctly. When I figured this out, I tried WPA+TKIP and got connection up... Now lets see if we can get fix for this WPA2+AES issue in future firmware.

I would like to remind people wanting to get WPA or WPA2 working to wait a moment and lets fix problems before getting this on mainstream. (eg. do not ask patched driver from me ;))

More WPA2 debugging

Tonight I was debugging more wireless and it seems that driver does not report that AP is using better encryption than WEP. Wpa_supplicant seems to look for WPA IE and/or RSN IE data so I need to find out where to get that or do I need to bake it somehow (or can I bake it). Having no documentation for gelic is making it a bit hard to guess, but it would seem that this AP's IE information is not available from it. Need to study this more later on. I also implemented some ioctl's so that user space can query and set those values, wpa_supplicant was giving some warnings about those. Though I am still wondering should those values be used in driver itself and not just to provide storage helper ;). Hacked wpa_supplicant to accept my AP without whining about missing WPA, it went a bit further leaved scanning mode to another but then again failed. It looks like it sets wrong encryption session settings or something like that as it didn't get it from driver on those IE fields. Need to study this more too...

WPA problems with PS3 Linux

I was getting annoyed that WPA2 didn't work out of the box in Yellow Dog Linux, so I started to diagnose what is the problem. First I setuped seperate Linux kernel module building environment so I could compile newest ps3_gelic module in isolation. After long hours of debugging I think that ps3_gelic or gelic_wireless is working correctly. The problem is either Linux Wireless Extension userspace glue or with userspace applications. After all this seems to be problem with 64 bitness and inability of the tools to handle it correctly. I think there might be some hacks on x86-64 to get it working but those are probably not present on Cell (or powerpc arch within Linux kernel).

Some thoughts about realtime buffers

One of the problems I have faced many times before is how can I avoid doing much locking in real time software. This is not an easy task, and failure to do it properly can cause obscure problems. While reading recent Dr. Dobb's Journal issue I though I could write something about on subject, when to use locking and when it can be avoided (or minimized). I will assume that reader has at least some idea why in multi threaded application there should be some locking :)

One of the principles of parallel programming is that if some resource is shared, its access should be synchronized or algorithm should be defined in a way that there are no sharing problems (like one writing to data that is being read elsewhere and incorrect result is being retrieved).

One of the problems with synchronizing access is that it usually needs some locking to be made like in following pseudo code:

read()
  lock resource X
  tmp = resource X
  unlock resource X
  return tmp

write(new value)
  lock resource X
  resource X = new value
  unlock resource X

Otherwise fine algorithm, safe to use but in some cases excessive locking can be a problem. If there are lots of read's and write's per second locking is starting to take lots of time and in some applications this is not favorable situation. Problem comes from fact that locking usually needs context switches which is usually quite expensive. One of the possible solutions for multicore (or multiple) processors is to use spinlocks - try to acquire lock multiple times before switching thread to sleep.

One approach is to use algorithms that require lesser amount of locking. Perhaps most classical algorithm is ring buffer. Caveat here is that system must support atomic reading & writing of needed data types used to store buffer status - or otherwise some locking must be implemented in here too. Though the access time can be smaller as lock is needed only for small period of time.

Algorithm proposed in Dr. Dobb's article was called Spin Buffers. Basically it has multiple buffers (minimum of 3) set up as a ring. One buffer can only be owned by one party. Algorithm works like following. When writing new data: put data to current buffer and then if next buffer is free, switch to it for future writes, mark old buffer as free. When reading data, read current read buffer until it is empty, if next buffer is free, switch to it and mark old buffer as free. Reason why this algorithm needs at least 3 buffers is that it tries to always allow reading of written data, and if reader is not fast enough, writer can always write new data to current buffer (until its full). If reading and writing is not balanced it is possible that writer does not have possibility to switch to another buffer until next write is issued. Eg. issue many writes to fill all buffers so it will have to queue items to last buffer (before read buffer), then writer decides to go to sleep for some time. During this time write buffer pointer can't be incremented as it is reserved for writer only. Now reader wakes up and reads all data until there are no free buffers to be consumed, only buffer having data is write buffer and it cannot be accessed as it is reserved for writer. This situation can be unlocked only by waking up writer to switch to next free buffer.

Using Eclipse for GRUB 2 development

As I have been using Eclipse for some time now I decided to give forthcoming Eclipse Europa release a try. Eclipse Europa is still in release candidate state (rc4) but it should be stable enough to be used. Final release of Europa should be within this month.

I am currently using Ubuntu 7.04 for my GRUB 2 development activities so I needed to download GTK+ version of the Europa SDK. I also like to use Sun's Java so I also installed several sun-java6 packages from Ubuntu's package manager. After this, rest was quite easy. Extracted Europa SDK under /opt and gave access rights to my normal user to that directory so I could install additional plug-ins.

After starting up Eclipse I installed CDT 4.0 (C/C++ Development Tools) from Europa repository. All went nicely without an issue. I noticed that they finally added this automatic mirror selection functionality to Eclipse! I have been waiting for this feature for a looooong time.

Setting up a GRUB project was quite easy. First I downloaded up to date version from grub's developer ssh. Then I copied it to under my workspace and created a C project for it. While configuring C project, I noticed that Linux version of the Eclipse doesn't have managed build support at all. Well for this project it is not issue, but would be nice addition in future. I don't really like to play with makefiles or such, sometimes they are necessary evil. After running configure for grub2 all was ready from source for development activities. From Eclipse side, first I configured project settings (seems that they have changed how CDT's preferences looks) and added grub2's include directory to include search path. Then I added make target for building 'all'. I noticed that CVS team synchronization worked out of the box as .cvs folders was present. Tried up team synchronization, needed to add couple of folders and files to ignore to make list of files clean. All seems to be OK, so I tries to build all target. Surprisingly everything seems to work nicely. I event got warnings and errors list populated correctly. Now that those GRUB2's build files would reside on own directory......

Starting up my development blog

I decided to start blogging about my open source and other development activities. In this blog I will tell what I am currently working on and possibly even give some hints and tips. You are free to give comments and perhaps you could event influence what I write here.