Garmin and OpenStreetMap

I've got a handheld GPS from Garmin - the eTrex Vista HCx, which I bought in anticipation of a trip that didn't happen, and haven't had any real reason to use it until now.

Now, I'm headed to Myanmar and could use some GPS to guide my bike adventures. Trouble is, garmin would like me to pay a lot for some pretty useless and out of date maps. Luckily, openstreetmap has come a long way and is now looking pretty good, even for Myanmar. So here, I've just figured out the rather tricky process of getting maps onto the device and getting them recognized.

Get a usable map with the information you need for the region you'll be in. The easiest option is to download something somebody else created. Have a look here, and particularly here, although the latter appears to need a >=4GB microSD card (mine is 1GB). Unfortunately none of the pre-made maps cover Myanmar, at least not in enough detail for me.

1) Download the area of OSM you need using the API. I did the following to download.
wget -O myanmar.osm http://www.overpass-api.de/api/xapi?map?bbox=93.4,15.5,98.0,23.0
This bounding box covers most parts of Myanmar that tourists are allowed to visit.
2) Next, convert it to the format needed by Garmin. For this, use mkgmap which is a java command line program available from here. For my purposes, it was necessary to provide a custom style file which specifies what features go into the garmin map and how they look. I used the cyclemap style explained here, and downloaded cyclemap.TYP to the same directory:

svn co http://svn.openstreetmap.org/applications/utils/export/garmincyclemap/network/cyclemap

3) I unzipped the mkgmap files to another subdirectory in the same location.
4) I used the cyclemap styles unmodified, although they could have been improved for my uses (they're designed for the UK). Then I simply ran:
java -jar mkgmap-r2889/mkgmap.jar --style-file=cyclemap --remove-short-arcs --style-file=cyclemap --route --net --mapname=Myanmar --description=Myanmar --gmapsupp myanmar.osm

This produced gmapsupp.img, which is the garmin-format map file I needed. (I had no need to split the osm or ask java for extra memory, but more info here.)

The next step is getting this onto the device. In theory, the easiest option should be to simply copy this file to the Garmin\ subdirectory on the microSD card using a card reader. But in my experience, the garmin device never recognized it. That leaves two other options.

1) In windows, install the Garmin USB drivers (first!). Then connect and turn on the Garming GPS using the USB cable. Under Menu->Setup->Interface and select USB Mass Storage. This will show the SD card as a drive on the computer. Copy the gmapsupp.img file to the Garmin\ directory on the device. (Note there can only be one custom map file on the device at a time and it must be in this location/filename). Eject it and power down the device.
2) Use sendfile to copy the file onto the device (this should work in linux or windows; in windows it also requires the Garmin USB drivers). After completing, eject the device and power it down.

Finally, disconnect the GPS from the computer and power on. There should be mention of OPENSTREETMAP copyrights during bootup. To confirm that the map is there, you can go to Menu->Setup->Map and scroll right to the info icon, where you should see your map selected. You should also be able to zoom out and pan to the coverage area (if you're already within the coverage area, no need to pan!). Note that it may only really be visible at certain zoom levels. You could also "Find" a city, say, or other place in your new map.

Enjoy!

Upgrading from Linux Mint 15 (Olivia) to Linux Mint 16 (Petra)

To upgrade, as opposed to the recommended method, which is to re-install with the new CD, recover the apps you've installed using mint backup, restore your data, and re-configure everything, I used the apt system, which is a bit more risky. (It may be worth burning the install disk and running it as a live CD to ensure your hardware is well-supported). Here's what I did:

1) Update the repositories to point to their new locations:

$ sudo sed -i 's/raring/saucy/' /etc/apt/sources.list.d/*.list
$ sudo sed -i 's/olivia/petra/' /etc/apt/sources.list.d/*.list

2) Update the repositories:

$ sudo apt-get update

Look for any errors. Some repositories for non-mint apps may not yet have a special version for Mint 16, but they will likely still work. If you do get errors, find the corresponding .list file and undo the above change (eg, change 's/raring/saucy/' --> 's/saucy/raring/'), on only that .list file. Re-update the repositories.

3) Quit any open apps. Then, perform the upgrade. This step will require several gigabytes of downloads, which will take a while (if it doesn't, something is wrong; abort and fix).

$ sudo apt-get dist-upgrade

Along the way, you'll be prompted whether to keep or overwrite any config files that have been changed by the maintainers. Almost always, you'll want to say Yes. It will keep your old files with an extension like .dpkg_old, in case you need to do any merging. The main case where I needed to do this was with my samba setup.

Then, just to be safe, run:

$ sudo apt-get upgrade

4) Finally, some last minute changes. Make sure your drives in fstab will still mount. I use the ufsd from Paragon (instructions here) for ntfs and hfs+ drives, and this will need to be re-compiled for the newly installed kernel after reboot. But for now, comment them out in /etc/fstab.

Also, if you had to use the default UEFI boot file, copy the new grub file to that location (you might want to back it up first):

$ sudo cp /boot/efi/EFI/linuxmint/grubx64.efi /boot/efi/EFI/boot/bootx64.efi

When you think you've sorted out all the kinks, reboot: sudo shutdown -r now

5) If you're lucky, the machine will come up and you'll be able to log on. Otherwise, another good opportunity to use that linux live disk you've got.

Install the ufsd drivers for the new kernel following the instructions linked to above. Uncomment the lines in /etc/fstab and run sudo mount -a then verify they all loaded. For some reason, under linux mint 16 (but not 15), the ufsd kernel module will not autoload - it's necessary to add the ufsd driver to /etc/modules by adding a line "ufsd" to the end -- otherwise the file system will not be recognized and boot will hang.

6) After update, I had some duplicate icons in the menu for some reason. These are easiest removed by running gksudo nemo and navigating to /usr/share/applications and removing dups, retaining those that utilize the currently running cinnamon-settings app.

Done.

Using the same ssh key between linux and mac

If you are dual booting and you want to be able to ssh in (particularly by IP, or if the host names are the same) to both operating systems, then you'll want to set up ssh to share the same server keys between operating systems.

Otherwise, after a client connects with one operating system and then tries to connect again while you're booted in the other OS, they'll get this message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.

To use the same ssh keys, copy them from one OS to the other. I chose to use the ones from linux in macos (in macos they're stored directly under /etc, in ubuntu/linux mint they're under /etc/ssh/):

 /etc/ssh $ ls -l
total 276
-rw-r--r-- 1 root root 242091 Nov  8 14:37 moduli
-rw-r--r-- 1 root root   1669 Mar 26  2013 ssh_config
-rw-r--r-- 1 root root   2489 Nov 11 12:57 sshd_config
-rw------- 1 root root    668 Nov 11 12:57 ssh_host_dsa_key
-rw-r--r-- 1 root root    600 Nov 11 12:57 ssh_host_dsa_key.pub
-rw------- 1 root root    227 Nov 11 12:57 ssh_host_ecdsa_key
-rw-r--r-- 1 root root    172 Nov 11 12:57 ssh_host_ecdsa_key.pub
-rw------- 1 root root   1679 Nov 11 12:57 ssh_host_rsa_key
-rw-r--r-- 1 root root    392 Nov 11 12:57 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root    338 Dec  2 15:52 ssh_import_id
 /etc/ssh $ sudo mkdir /macos/etc/ssh-backup
 /etc/ssh $ sudo mv /macos/etc/ssh_host* /macos/etc/ssh-backup/
 /etc/ssh $ sudo cp ssh_host* /macos/etc
 /etc/ssh $ ls -l /macos/etc/ssh*
-rw-r--r-- 1 root root    0 Nov  4 23:27 /macos/etc/ssh_config
-rw-r--r-- 1 root root    0 Nov  4 23:27 /macos/etc/sshd_config
-rw------- 1 root root  668 Dec  4 18:48 /macos/etc/ssh_host_dsa_key
-rw-r--r-- 1 root root  600 Dec  4 18:48 /macos/etc/ssh_host_dsa_key.pub
-rw------- 1 root root  227 Dec  4 18:48 /macos/etc/ssh_host_ecdsa_key
-rw-r--r-- 1 root root  172 Dec  4 18:48 /macos/etc/ssh_host_ecdsa_key.pub
-rw------- 1 root root 1679 Dec  4 18:48 /macos/etc/ssh_host_rsa_key
-rw-r--r-- 1 root root  392 Dec  4 18:48 /macos/etc/ssh_host_rsa_key.pub

/macos/etc/ssh-backup:
total 24
-rw------- 1 root root  668 Nov 11 00:23 ssh_host_dsa_key
-rw-r--r-- 1 root root  590 Nov 11 00:23 ssh_host_dsa_key.pub
-rw------- 1 root root  965 Nov 11 00:23 ssh_host_key
-rw-r--r-- 1 root root  630 Nov 11 00:23 ssh_host_key.pub
-rw------- 1 root root 1679 Nov 11 00:23 ssh_host_rsa_key
-rw-r--r-- 1 root root  382 Nov 11 00:23 ssh_host_rsa_key.pub

Voila. You can ssh into either OS without any warnings.

You may also want to set the same ~/.ssh/authorized_keys under home dirs in both your OSes, to allow password-free authentication between the client and both servers. A reminder on how to set this up --
On the client:

cd ~/.ssh

ssh-keygen -f id_rsa -C 'comment to identify client' -N '' -t rsa -b 4096 -q

This creates a private and a public key. Copy the id_rsa.pub key to the server. (eg, scp id_rsa.pub user@serverhost:/home/user/ ). On the server:

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

rm id_rsa.pub

All set. You can repeat as desired for each client user/machine. Then just copy authorized keys to the other OS and it'll work without a password in either.

Notes on Wake on Lan (WOL)

These are some notes on a current problem, even more disjointed than usual. I wanted to have Wake on Demand working on both mac and linux on the same machine, with the same ethernet port. This isn't working out well (in linux at the moment), so I'm shelving it and at some point maybe I'll get back to it.

In MacOS, I found that the intel driver fails but the atheros card works fine for WOMP and WOD, and USB keyboard.

Under linux:
To enable wake with USB keyboard, you need to add:
# enable waking from sleep from USB HID devices
# $ lsusb -d 046d:
# the above gets the usb bus and device, but not port (5 in the case below)
echo enabled > /sys/bus/usb/devices/usb3/power/wakeup
echo enabled > /sys/bus/usb/devices/usb3/3-5/power/wakeup
to /etc/rc.local

The Intel card works fine for Wake on Magic packet (but since it doesn't for the mac, I've abandoned it for the moment).

Wake on Magic Packet fails for Atheros card. This appears to be an issue in the alx drivers, which aren't being updated it seems (maybe they are?). The lights are off during suspend. There is a compounding issue in that if you suspend via dbus (which appears to be what gnome/cinnamon does), when you come back the ethernet card will not work properly, irrespective of what you put in /usr/lib/pm-utils/sleep.d/50unload_alx to not disable/re-enable the alx driver (which was itself put there to fix a third bug in alx, which was apparently causing sleep to fail altogether).  (Updating alx doesn't help). You have to run rmmod alx; modprobe alx as root to re-enable networking after waking. If instead of using dbus, you simply call pm-suspend, there are no issues with the Ethernet connection dropping upon awake.

Note that enabling the disabled wakeup options for the card and its pci bus as below make no difference:
echo enabled > /sys/bus/pci/devices/0000\:00\:1c.0/power/wakeup
echo enabled > /sys/bus/pci/devices/0000\:00\:1c.3/power/wakeup
echo enabled > /sys/bus/pci/devices/0000\:02\:00.0/power/wakeup
( lspci & lspci -tv help with these)

Here's the dbus suspend command I've used: dbus-send --system --print-reply \

    --dest="org.freedesktop.UPower" \
    /org/freedesktop/UPower \
    org.freedesktop.UPower.Suspend

*** Actually, this guy has some interesting troubleshooting and it seems pm-hibernate might work with WOL (and since I have an SSD, isn't that different from sleep). But, I tried pm-hibernate and I seem to be getting the third bug in alx mentioned above, that hibernate fails to actually go into hibernation.

The cool app SleepProxyClient, which I'd really like to get working on the machine, uses pm-suspend (at least now). If it worked, we could sleep and then have things awake when needed (eg, samba requests), by making use of the bonjour sleep proxy on my appletv. I haven't tested it with the intel card, which might just work. Here are some useful pages about this:
http://www.cnpbagwell.com/mac-os-x/bonjour-sleep-proxy
http://stuartcheshire.org/SleepProxy/
Avahi is the linux implementation of bonjour.

There is a link about autoshutdown on the sleepproxyclient page above, and this may also help in deciding when to sleep. So might this, which I found somewhere:

#!/bin/bash
hdparm -C /dev/sda | grep "active" > /dev/null
SDA=$?
hdparm -C /dev/sdb | grep "active" > /dev/null
SDB=$?
if [ $SDA -eq 1 ] && [ $SDB -eq $SDA ]
then
 echo "drives asleep" 
 LOAD=`awk {'print $1'} /proc/loadavg`
 echo $LOAD
 if [ "$LOAD" == '0.00' ]
 then
  echo "load hit threshold"
  echo "Sleeping"
  # Call sleep process here
 fi
fi

Open and Password protected shares in Samba

If you want to have more than one share for samba, with one open to anyone on the network (as a guest) and one open only to named users (in this case 'realuser', which should be an actual unix account), there are a couple of complications. The following is a config file that works (it assumes you've followed the smbguest username setup steps from this post).

[global]
  security = User
  map to guest = Bad User
  null passwords = Yes
  obey pam restrictions = Yes
  # this account must exist as unix users
  guest account = smbguest
  unix password sync = Yes
  # these two are for performance
  socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=32768
  strict allocate = Yes

[data]
  comment = data
  path = /data
  force user = smbguest
  read only = No
  create mask = 0666
  directory mask = 0777
  guest only = Yes
  guest ok = Yes

[private]
  comment = private
  path = /private
  valid users = realuser
  admin users = realuser
  read only = No
  guest ok = No
  create mask = 0666
  directory mask = 0777
  # this makes it invisible when browsing the computer
  browseable = No

You'll have to add realuser as a username accessible from samba:

sudo smbpasswd -a realuser

But it's a bit tricky because authentication is by users, not shares. So if you want to be able to access the private share (using realuser and its username), as well as the data share (that's open to guests), you'll need to log into both as realuser. Note that in windows 7 the username is \realuser (the backslash is necessary to remove the  domain).  Note that if you do this only in linux, you'll want to make sure you have the map to guest = bad user line after the security=user line under MacOS to allow access to users who have saved mapped the drives while linux was running.

P.S. - For debugging your smb.conf file, it helps to run testparm -s (which gives warnings and errors).
For debugging windows 7-related issues related to samba, see here.

Urban Cycling Survival Guide

I've been riding my bikes around cities for a decent number of years now, and that's spanned cities in four countries. It's a personal thing, but I personally couldn't manage without a bike at this point. The benefits are simply too great. I use them almost every day I'm in a city and almost anytime I'm going anywhere between a couple blocks and a few miles away. I'd highly recommend biking to almost everyone.

So people ask me somewhat hesitantly if it's safe. I usually don't tell them about the four or five times I've been hit by a car (luckily without much harm [it helps that I'm good at landing on my feet from my gymnastics days]), and fallen a few more times. The truth is that it can be dangerous, as I've learned the hard way. But it is what you make of it, and there's a lot that can be done to minimize the risk. This post is intended to teach you how to reduce the risks to a minimum.

I can also say that in most places bike safety is improving rapidly, and most of that has to do with better driver awareness, increased numbers of cyclists, and better infrastructure for bikes. That said, it really does depend on where you are. My recent frame of reference is mostly Washington, DC and Brussels, Belgium. It might be pretty different in a spread-out place like Dallas or even LA where "roads" are more like highways.

But a lot depends on how you ride. These are some tips that can help keep you in once piece:

1. Know how to drive. Seriously. Review the rules of the road for where you are. At a minimum, know who should yield, when, and where. The most dangerous cyclists (to them and others) are those who don't really. And they do exist. Don't forget to signal when you want to turn (just stick out your right or left arm [- you'll need to practice riding with one hand!]). You don't want a car to pass you as you turn left, for example. And you don't want anyone riding your tail as you slow for the turn.
2. Control everything that's in front of you. Until you see proof to the contrary, don't assume anyone in front of you knows you're there. At least not consciously. And even if consciously, they may not comprehend. In other words, drivers often misjudge the speed and maneuverability of bikes (not to mention their own). One person who hit me looked me in the eye right before they pulled out into my path and sent me flying over the hood. Anytime I approach an intersection where there's a car (or will be one soon) I am constantly on guard, keeping lots of space between us and keeping my eyes clearly on them. Don't underestimate pedestrians' abilities to jump in your path either.
3. Beware car doors. This is one of the biggest risks in the city. Usually, there are people parallel parked on the right side, you're coming along the right side of the road, and they open their driver door right into your path. Avoid this by staying far away from such cars whenever possible, or going slow enough to react. But sometimes, the passengers of people in the traffic lane will do this too -- beware especially of cabs. I had this happen to me once, (when the traffic was about to move!) and it wasn't fun. Relatedly, beware of cars parked perpendicular to the road who may reverse out into you (just enough to tell if anything is coming, you see).
4. Be ready for the "pass you and cut you off." Surprisingly often, people will pass you on the left and then make an immediate right turn into your path. The main defense is just awareness of the possibility. Drivers appear to have little understanding about how fast bikes are going. More generally, drivers will often see a bike ahead of them and think "I've got to pass this guy". But the reality in cities at least, and especially when there's traffic, is that I go as fast or faster than cars almost everywhere (the main exceptions are big uphills or long fast stretches). So this is pretty irrational. It's a quiet pleasure to permanently re-pass these cars that are in such a hurry to pass me.
5. Avoid passing stopped/slow moving cars which may turn. Especially people who seem lost. Cars can quickly decide to make a right or left turn without looking for bikes coming from behind (remember you're usually in their blind spot when they do this, so unless they've been paying attention they really have no idea you're there). So just don't be in places where this may happen. At intersections, I like to position myself to the right of and between cars (which are before and after me in the flow of traffic). Slow down to allow the car ahead of you to pass through first; the cars behind will wait for you to cross the intersection before turning. Note that passing cars on the left at a stop sign is a bad idea too; they may turn left with no signal. I made this mistake once early in my biking career and got my foot run over as I skidded to a halt to avoid it getting more of me.  Didn't even hurt, actually.
6. If you'll ride at dusk, dawn or dark, use reflectors and lights. Bikes can be surprisingly hard to see and are often unexpected at night. That means a front and back reflector, and ones on your wheels.  Something on your feet/pedals is also good, as their movement means "bike" to drivers. You can get pretty decent lights now that are the size of nickels. In this case they are for drivers to see you, not to illuminate the road. And it doesn't hurt to have a reflective vest for long trips or dreary nights. I know if you're a racer, it's not cool to have reflectors, I don't have them either on my race bikes. But don't use those bikes when it's getting dark. You can't control what's behind you, so make sure it's no surprise that you're there. Also more generally, a helmet won't hurt [But please, learn how to fit it. My goodness how many people wear them as a useless hat. It should cover your forehead and bind at the chin strap when you open your mouth.]
7. Manage road obstacles. Gutters can be very dangerous. Always beware of vertical slits in a gutter that can eat your front wheel, especially on a road bike, and send you catapulting over the handlebars (worst case -- ride over them diagonally). Railroad tracks should be crossed at a perpendicular and don't try to turn while you're crossing them. Especially when it's wet. The same applies to slick metal manhole covers. Glass shouldn't make you fall but it might be a pretty big bummer with flats so is best to avoid (good tires go a long way to avoid these, too). Be careful on off-camber turns in rain. Especially soon after it starts raining, when the road is particularly slick. And ice, of course. Bike wheels have a large diameter and can go over most obstacles. If your room for maneuver is slight, you can probably go right over it. Certainly things like water bottles and even curbs are no big deal. Don't panic and simply keep riding straight. Let your legs absorb the shock of something like a curb so as not to ding your rim (better yet, learn to hop them). (Also, having properly inflated tires [80-100 psi on road bikes, 40-65 on city bikes and 35-45 on mountain bikes] will avoid damage.)
8. Control the lane. Sometimes you need the whole lane, and it's your right to take it. Don't be shy. It's much safer to block the lane than to risk some of the stuff above. I tend to go pretty fast though, so drivers are rarely very upset about it I don't think. The slower you go the less necessary it is (you can just stop after all), but again if you think you need the lane, command it. It is your right.  If you're not willing to do this in areas like the below, then get on the sidewalk and go even slower. When should you take over the lane? When the road is too narrow for them to pass you (don't let them try). When it's too slippery for (potentially) necessary maneuverings. When there's a traffic circle coming up. When there are parked cars on the side, especially those that might have a person in them. When there's an upcoming obstacle. When there are obstacles on the shoulder, or there is no shoulder. When going around blind curves to the right (in countries that drive on the right) -- you want any vehicles approaching from behind to see you as early as possible, so stay left. (But, do be careful not to take over the lane when someone is about to pass you, or to swerve into passing cars to avoid obstacles, say. As noted, sometimes it's best to slow/stop or to ride over things). Infrequent cyclists worry about getting hit from behind. Well, it's always possible. But it's very unlikely if they know you're there and especially if you control the lane -- the statistics show these kind of accidents are extremely rare.
9. Worry about what the drivers who will pass you need to worry about. If there's something coming up that will restrict their lane space, get out of the way. They might not know about it, as cyclists often have better situational awareness. You don't need them veering into you to avoid the oncoming car, for example. Remember, they have the big metal box to protect them. You don't. (Also, metal is hard...)
10. Beware the cycle path. It sounds crazy, and it is. But, bike paths can be some of the scariest cycling at times. I can't remember all the near misses I've seen people have. Some cyclists don't appear to understand that just because there are no large moving metal boxes to hit them doesn't mean that there aren't risks. These can be the worst accidents, because people often go fast, and make unsafe passes. If you're going fast, make yourself known to those in front of you "rider back", "on the left". If you're going slow, and basically whenever you can, stay to the right. Always pass with ample space (if someone veers and snags your handlebar, you're eating turf). If there's someone in front of you and oncoming bikes in the other direction, slow down and wait behind in your lane until you can pass safely. Also, don't pace near congestion, and don't pace anyone (e.g., the stranger in front of you) who is not a predictable rider accustomed to pacing. Instead, always give space between you and the wheel in front of you.. There can be one way lanes too, as under bridges. Don't try to squeeze two in. I've seen this one happen. Hitting a bike travelling in the other direction hurts -- like dislocated shoulder/broken collarbone hurt. Finally, bikes yield to horses. You don't want to scare a horse.

Some people won't ride in cities because they don't trust their bike handling skills. Well, the only way to improve them is to bike, but start in controlled environment. I really recommend mountain biking for this. You'll learn a lot very quickly and you don't need to be a dare-devil to do it. Just find easier trails to start. But find singletrack, otherwise you won't really have to learn to maneuver. One tip that some people learn the hard way and applies to roads too: keep your weight back in the saddle when going down hills; also arms should always be bent and relaxed. Stiff arms act as a fulcrum when the unexpected occurs, bent ones act as shock absorbers. Which reminds me that bike gloves are great; I always wear them on long rides, since I like the skin on my hands and the nerves under my palm.

So that's it -- you've heard all the terrible, horrible things that can happen, but also how to try to minimize their risks. Hopefully facing up to them and understanding how to deal with them can help you get out there. Because in the end, it's fantastic.

Addendum: http://bicyclesafe.com/ does a good job detailing most of these crash scenarios and how to avoid them.

DVD/Blu ray under linux

Unfortunately from what I read, neither MacOS nor Linux can play back blu-ray disks properly. That's because it's a closed format and they are only providing licenses for players in a very convoluted (and expensive) way. So there are no licensed players outside Windows, and certainly none that are free. That leaves those of us who still want to be able to watch a movie grappling to find solutions.

Fortunately there are some. First, vlc (apt-get install vlc) appears to play DVDs fine most of the time. You can try here for setting up blu-ray. It's a matter of adding a library and some keys, but like I said, it's apparently not great.

The alternative is to save the movies to disk first, and play them from there. This is also handy if you want to have them always available for your home theater.

To rip, MakeMKV seems to work great even though it's in beta. It's free too, for now. Just follow the directions at the link, which work flawlessly for me.

Now, you can insert a dvd/blu-ray and save it directly to disk. Unfortunately they're still full size, which can be quite big. Handbrake is a great tool for transcoding it for the output device you want to use. Simply:
sudo add-apt-repository ppa:stebbins/handbrake-releases
sudo apt-get update
sudo apt-get handbrake-gtk

Then it can be run with a nice GUI as handbrake-gtk. But there is also a useful command line interface. I've modified a tool I found in the handbrake forums (written for macos) that allows transcoding a whole directory full of files in a batch. The code is below the fold.

Linux Mint / Mac OS 10.9 dual boot setup on Haswell i7-4770 GA-H87N-WIFI

Setup Objective:  Dual Boot MacOS / Linux

My new computer hardware:

• MI--008 Mini-ITX Black Case with 250w PSU
• Gigabyte GA-H87N-WIFI motherboard with 1150 slot
• Intel Core i7-4770 Quad Core 3.4Ghz processor with HD 4600 integrated graphics
• 16GB DDR3 1600 MT/s (PC3-12800) Crucial Ballistix Sport RAM
• Samsung 840EVO 250GB SSD (boot drive)
• Seagate Barracuda 2TB HDD (data drive)
• LG Blu-Ray reader / DVD-RW UH12NS30
• Gentle Typhoon 120mm cooling fan

This amounts to a pretty tiny PC with lots of computing power (but low energy usage). I do wish I could install more RAM, but 16GB is the limit. I'll use it mostly for intensive processing for my dissertation and for work [I'm a programmer]. But I may also do photo editing with the mac side and use it as a part-time HTPC under linux/xbmc.

Instructions

A) Install MacOS:

1.  Purchase and download (but don't install) MacOS from an existing Mac. Use Unibeast to setup a USB key with a bootable MacOS installation. I used OSX 10.9 (Mavericks). Also copy Multibeast 6.0 to the USB key.
2. Setup BIOS to use the optimized settings with the following modifications:
1. Disable VT-d. 
2. Extreme Memory -> Profile1.
3. Make IGFX the first option.
4. UEFI and Legacy boot.
3. Take the key to the new computer and boot. I needed to use the "-x PCIRootUID=1" (no quotes) settings to get the video mode to work. Install normally, using the whole boot drive as a partition. It can be shrunk later. Instructions here.
4. Upon reboot, again boot with the USB, this time selecting the new mac partition, again with the same settings.
5. Setup: System Preferences->Security & Privacy->Unlock; Then: allow apps downloaded from anywhere. Back arrow. Mouse->Disable scroll direction "natural".
6. Copy multibeast to the desktop and run. With my hardware, most things work straight out of the box. Select the following (de-selecting all else):
1. DSDT Free
2. Audio->Realtek ALCxxx > Without DSDT->ALC892
3. 3rd Party SATA
4. TRIM Enabler -> 10.9.0 TRIM Patch (for SSD)
5. FakeSMC v5.3.820 (Plugins and HWMonitor are nice, optional extras)
6. Network->Atheros -- Shailua's ALXEthernet v1.0.2
7. Network->Intel -- hank's AppleIntelE1000e v2.5.4d     (need both to support both ethernet ports)
8. System->Patched AppleIntelCPUPowerManagement -> OS X 10.9.0 --- THIS IS NECESSARY; otherwise boot will hang without safe mode on after loading bluetooth drivers.
9. System -> AppleRTC Patch for CMOS Reset
10. Boot Options-> Generate CPU States, GraphicsEnabler=Yes, Hibernate mode-Desktop, PCIRootID Fix (this sets PCIRootUID=1), Use KernelCache. Might want to still edit /Extra/org.chameleon.Boot.plist to add GraphicsMode to match your monitor. [App Store -> Find/Install TextWrangler; Finder->Go->Go to folder-> /Extras]
11. System Definitions -> Mac Pro ->Mac Pro 3.1 (don't know if this is optimal, but it works)
7. If you can boot now, it's a good time to setup linux (below). But first fix the drives up some.
1. Open disk utility and shrink the macos partition on the boot drive. 
2. Format the data disk using NTFS. NTFS is the best choice for a drive between windows and anything, but also between macOS and linux, supporting >4GB files and big drives. 
3. Enable read/write support for the data drive: from the terminal, run sudo vifs and add this line: /dev/disk1s1 /Volumes/data ntfs rw,auto,nobrowse. The first is the partition (make sure this is correct) and the second is the mount point (a directory which you must create). Run sudo mount -a to test that it works; browse to this location with finder. Then drag the directory into the favorites list to the left of finder (for easy access later). It will be mounted at boot up from now on.
8. At this point most things worked, except WIFI, which is a known limitation. But not audio via the HDMI cable, and the video drivers were flickering/unstable around animations. To fix both these issues, do toleda's HDMI fix for Haswell systems. Be careful to follow the directions exactly. First verify pre-requisites, then perform all steps in the "Before you start" section, referring to the Notes section below. Finally, do either the DSDT or the SSDT installation method. I did the DSDT method, and selected the -1 variant of the driver, which worked nicely.
9. Setup SAMBA sharing to match linux.
10. VNC setup

B) Linux Setup. I used Linux Mint 15 [Cinnamon 64], for which I burned an install DVD. You will also need a grub rescue DVD.

1. Boot up linux mint on the install DVD, being sure to select the UEFI DVD option (and not the P? option) from the BIOS boot menu. DO NOT INSTALL DIRECTLY FROM BOOTUP, as it will fail to properly setup grub. Instead, once Linux Mint has booted, double click the Installer program on the desktop to start installation.
2. Choose "something else" for how to install the OS. Create an ext4 partition at the end of the drive. I did not create a swap partition, preferring a file instead as it's more flexible (see below). At the bottom, choose to install Grub in /dev/sdb3 (or whatever your new ext4 partition is). If you install it in the default /dev/sdb, it will overwrite chimera and macOS won't boot.
3. Next, boot with the grub rescue DVD. Select the partition you just installed [/dev/sdb3] to load the new linux install.
4. To add linux to the UEFI boot menu, sudo cp /boot/efi/EFI/linuxmint/grubx64.edi /boot/efi/EFI/boot/bootx64.efi [efibootmgr does not work correctly for me, I strongly suspect Gigabyte's UEFI implementation is faulty]. Grub will not install automatically to the EFI either, and I can't get Grub to load chimera or the mac kernel either. I even tried creating partitions just for grub or using the data disk MBR, to no success. Basically I tried about 100 options (which means >100 reboots) and no fully satisfactory option arose for a good boot loader for both mac and linux. I settled for using the bios to choose either the boot disk MBR (to load MACOS via chimera) or to use UEFI default boot (to load Linux via grub partition boot loader). A bit ugly, but at least it works. [Update - Linux Mint 16 may have fixed UEFI support for this motherboard. I have just installed Mint 17.1, and it appears that as long as the install DVD was booted in EFI mode, mint will set up grub properly to boot. If not, you may need to run boot repair]
5. Setup swap file (I made mine 16GB, same as RAM, but it may be worth investigating the "swapspace" tool for dynamic swap space, or you may not even need one at all):
1. sudo fallocate -l 16G /var/swapfile
2. sudo mkswap /var/swapfile
3. sudo swapon /var/swapfile
4. You should see your swap listed when you run free.
5. Edit /etc/fstab to use it at reboot, adding this line: /var/swapfile  none  swap sw  0  0
6. Setup Paragon UFSD drivers. These are kernel-level drivers that permit read-write for both NTFS drives and HFS+ drives (even with journaling enabled). As such they're a major improvement over the drivers included as redistributables with linux. NTFS writing is significantly faster since it's not a user-space driver. Although it is proprietary, it's free for personal use. Fill in a little form and a few minutes later you'll get the download link via email. To install, simply:
1. Download them to ~/Downloads. 
2. cd ~/Downloads; mkdir ufsd; cd ufsd; tar -xvzf ../Paragon-147-FRE_NTFS_Linux_8.9.0_Express.tar.gz . 
3. See the README to confirm all is good. Although the website is outdated, it supports kernels up to 3.11.x as of this writing and works fine under ubuntu/mint (I have kernel 3.8.0-19). Then run sudo ./install.sh.
4. Edit fstab to use the new drivers by replacing the mount types with ufsd. I have these two lines:  /dev/sda1  /data  ufsd uid=myusername,gid=smbusers,noatime,umask=0002  0  2
   /dev/sdb2  /macos ufsd defaults 0 3
5. You may also need to clear the dirty flag on the drive by rebooting properly from Window/MacOS, or by using fsck (otherwise ufsd will give error 1000). Check dmesg | tail for more detailed errors when mounting. In Linux mint 16, (not in 15 or 17), I had to add ufsd to /etc/modules for it to load at boot.
7. Setup SAMBA shares to match macos. 
8. To play videos in vlc (and possibly other apps), it was necessary for me to choose Tools->preferences->Video->Output as OpenGL GLX or X11 Video.
9. To get HDMI audio working, I had to do as I later saw was suggested in the release notes (directions below). I also found the pavucontrol app to be helpful. You may need to disable SPDIF using the configuration tab of this tool to use HDMI.
1. sudo add-apt-repository ppa:ubuntu-audio-dev/alsa-daily
2. apt update
3. apt install oem-audio-hda-daily-dkms
10. It may be worth changing your user's UID to match that of macos, for while writing to the HFS partition. That can be done by looking it up under macos: ls -n in the home directory (mine was 501). Then from the linux console (ctrl-alt-F1), login as root and run usermod -u uid username. If like me you chose to create an encrypted home directory, also run chown -R username /home/.ecryptfs/username. Reboot.
11. Use software manager GUI to install all the wonderful free packages you want.
    
    
    Et, Voila!  That simple, in retrospect.

Edit: Also have a look at these tips I wrote for optimizing the solid state drive (SSD) performance under linux.

Same Samba share for linux and macos

I have a shared data drive that I want to use as a file server on the local network. But as I have a dual boot, I want the same share to be available on both linux and macOS, and I want it to be an open share without a password. First, to mount it under both, see my computer setup post.

1) Under linux, it's pretty straightforward. Make sure that the data disk is mounted, and you have the samba3 package.

Then create a samba user and group:
sudo groupadd smbusers 
sudo useradd -g smbusers -d /dev/null -s /bin/false smbguest

add primary user(s) to smbusers group (just in case):

sudo usermod -a username -G smbusers

and make smbusers the default gid for the mount (within /etc/fstab; I use ufsd, otherwise replace with ntfs):

/dev/sda1    /data        ufsd    uid=briggsjt,gid=smbusers,noatime,umask=
0002    0    1

Then edit /etc/smb.conf, adding:

guest account = smbguest

force user = smbguest

null passwords = yes

and add an entry for the share:

[data]
        comment = data
        path = /data
        writeable = yes
        browseable = yes
        guest ok = yes
        read only = no
        create mask = 666  
        directory mask = 777

I recommend the following for best performance:

socket options = TCP_NODELAY IPTOS_LOWDELAY

strict allocate = yes

The latter mainly helps for ext4/xfs/btrfs drives, but hey. 

Many people recommend setting SO_RCVBUF=8192 and SO_SNDBUF=8192. Here in 2013, I found these to be much slower than the linux defaults (which are 87380 and 16384 respectively (from /proc/sys/net/ipv4/tcp_[rw]mem)). Setting them slightly higher didn't seem to help either. I played with a lot of other things but nothing else much improved over defaults. I was surprised to see how much slower my wireless network was than wired. (Wired I get 40-45MB/s sustained write to the NTFS drive, wireless it's more like 4-10, with 802.11n dual band and no obstructions).

To test changes to smbd:

sudo service smbd restart

We can also optionally have our samba service advertise itself via bonjour over the network using avahi-daemon (so it will appear in Finder on Macs, etc): Simply create a file /etc/avahi/services/smb.service with the following contents:

<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
 <name replace-wildcards="yes">%h</name>
 <service>
   <type>_smb._tcp</type>
   <port>445</port>
 </service>
 <service>
   <type>_device-info._tcp</type>
   <port>0</port>
   <txt-record>model=RackMac</txt-record>
 </service>
</service-group>

And run sudo service avahi-daemon restart

2) Under MacOS:

There is a version of smbd on macos, however since 10.7 or so, it's an apple implementation and it does not allow guest access without a username. This means there's no simple way to use the same authentication as linux so it will be available no matter which OS I'm in. So instead I installed samba via macports. [Be sure to disable System Preferences->Sharing->File Sharing to avoid conflicts].

First ensure macports is installed: Install Xcode from the appstore. Install Xcode command-line tools: xcode-select --install. Download the macports .pkg installer from macports.org, and run.

Then install samba:

sudo port selfupdate

sudo port install samba3

Grab a drink and when you come back it's there.

In /opt/local/etc/samba3 there's a sample config file. Rename it to smb.conf and edit, adding a share like before:

[data]
   path = /Volumes/data
   public = yes
   browseable = yes   
   guest ok = yes     
   writable = yes     
   create mask = 666  
   directory mask = 777

Files will be written/read as the default user, and the drive must be accessible to him (you) locally.

To test it out, run:

sudo /opt/sbin/smbd -D && sudo /opt/sbin/nmbd -D

Then to launch automatically at startup, make a file /Library/LaunchDaemons/org.samba3.smbd.plist with these contents:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.samba3.smbd</string>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/local/sbin/smbd</string>
    <string>-D</string>  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

Make another identical file named /Library/LaunchDaemons/org.samba3.nmbd.plist, replacing the two 'smbd' occurrences with 'nmbd'.

Then, disable the existing macos smbd:

sudo launchctl stop com.apple.netbiosd
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.netbiosd.plist

And run yours:

sudo launchctl load /Library/LaunchDaemons/org.samba3.smbd.plist
sudo launchctl load /Library/LaunchDaemons/org.samba3.nmbd.plist
sudo launchctl start org.samba3.smbd
sudo launchctl start org.samba3.nmbd

Finally be sure to set DHCP in your router to always assign the mac address of your computer to the same IP address (you could alternatively set the same static IP in both OSes). Then simply connect to the data samba share (eg: \\192.168.1.10\data or smb://192.168.1.10/data) and it will work no matter which OS you're running! 

As a finishing touch, you can optionally set up the computer to advertise smbd as a Bonjour service. On the Mac side, this will not only make it easy to find, but more importantly to me anyway, it will enable Wake On Demand. This is a pretty cool feature that ensures that when the mac is in sleep mode, because it's not actively being used, the drive shares will still remain available whenever anyone on the network needs them, because the computer will be notified to start back up and respond to any requests that could come. In a use scenario like mine, in a home, this can save a ton of power by often being on standby, without sacrificing convenience. To work, it requires another device on the network that is always "on" and that acts as a sleep proxy. An AirPort Base Station, Time Capsule, or Apple TV will automatically perform this function, and other devices can be set up to.

Anyway, setting it up is actually a pretty simple matter of running the dns-sd command, and leaving it running while the computer is up (this assumes that samba has the same lifetime, which it should if you did as above). It can be done by creating a file /Library/LaunchDaemons/org.samba3.smbd.servicepublisher.plist with the following content:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

<dict>

<key>Label</key>

<string>org.samba3.smbd.servicepublisher</string>

<key>ProgramArguments</key>

<array>

<string>/usr/bin/dns-sd</string>

<string>-R</string>

<string>samba</string>

<string>_smb._tcp</string>

<string>.</string>

<string>445</string>

</array>

<key>RunAtLoad</key>

<true/>

</dict>

</plist>

Note that even if you used the default mac samba implementation, it appears that macs don't advertise the samba service. Adding this file will do that and so enable Wake on Demand for it. You can run dns-sd -B _services._dns-sd._udp to see all the available services on the network, which should now include smb, once you reboot.

P.S. I had to use my second ethernet port (the atheros one) in order for this to work -- it appears the MacOS intel ethernet driver won't handle Wake On LAN properly. Details are here.

Intel power gadget for linux

This is a cool tool to tell you how much power your computer is using. Unfortunately I don't find it packaged anywhere and is not pre-built by intel. Then, it did not build for me as downloaded.

I downloaded it from here: http://software.intel.com/en-us/articles/intel-power-gadget-20 and the files were dated 24 Jun 2013. Unfortunately the makefile had the libm location wrong on the command line and wouldn't link. Here's my modification:

 CFLAGS=-g

all: rapl_lib_shared rapl_lib_static power_gadget_static

rapl_lib_shared:
    gcc $(CFLAGS) -fpic -c msr.c cpuid.c rapl.c -lm
    gcc $(CFLAGS) -shared -o librapl.so msr.o cpuid.o rapl.o -lm

rapl_lib_static:
    gcc -Wl, -t $(CFLAGS) -c msr.c cpuid.c rapl.c -lm
    ar rcs librapl.a msr.o cpuid.o rapl.o

power_gadget_static:
    gcc $(CFLAGS) power_gadget.c -I. -L. -o power_gadget ./librapl.a -lm

power_gadget:
    gcc $(CFLAGS) power_gadget.c -I. -L. -lrapl -o -lm power_gadget

gprof: CFLAGS = -pg
gprof: all
    ./power_gadget -e 100 -d 60 >/dev/null 2>&1
    gprof power_gadget > power_gadget.gprof
    rm -f gmon.out
    make clean

clean:
    rm -f power_gadget librapl.so librapl.a msr.o cpuid.o rapl.o

That builds it. But do bear in mind that it's a command-line tool in linux, unlike the nice GUIs for Windows/Mac.

Addendum: Apparently the files are no longer easily found on Intel's site. Here is the source code I downloaded.