ufsd NTFS driver on mount: Fixing 'Unknown Error 1000'

I've previously mentioned using the ufsd driver for NTFS or HFS+, because it is significantly faster at writing than the default ntfs-3g driver provided with in Linux, and it supports writing for HFS+ drives even with journaling enabled. (It is available free for non-commercial use).

But what happens if you have a hard power down with an NTFS drive mounted? Or you encounter corruption for whatever reason?

UFSD may give this error upon mounting it for read-write:

mount: Unknown error 1000

This error is because the "dirty" flag is set on the drive and ufsd won't mount it read/write for fear of corrupting it. In many simple cases, you can correct errors in Linux with ntfsfix, from the ntfsprogs package, and also use ntfsfix to clear the dirty flag.

So if the volume in question is /dev/sdb1, I could do the following as root while the drive is unmounted. The first command repairs simple issues. The second clears the dirty flag:

root ~ # ntfsfix /dev/sdb1   
root ~ # ntfsfix -d /dev/sdb1

IMPORTANT NOTE: If you have extremely valuable data, especially that was being written when the failures occurred, you should be very cautious, because these commands may cause you to lose data.

A better approach may be to load Windows and run chkdsk /f  (to fix file system errors) or chkdsk /r (to detect and mark bad sectors) on the offending drive. Chkdsk is much more sophisticated at detecting and fixing errors than anything available in Linux, although it too may cause you to lose data (and some data loss may be inevitable if power goes off while writing). Here is one approach (using a bootable CD) to using chkdsk even if you don't have Windows installed or handy.

Final note, if this drive is not critical for your machine and you mount it from fstab, you can add the errors=remount-ro flag to the fstab mount line, in order to avoid hanging up your boot when things go wrong.

OpenVPN Server Setup Made Simple

For those not familiar with it, OpenVPN is probably the best and most secure VPN protocol out there at this time, with clients available for every mainstream platform. Unlike most other VPN protocols, it uses shared keys rather than passwords for security, which does make initial configuration take slightly longer.

But the real problem used to be that the process of setting up an OpenVPN server and generating appropriate keys was long, tedious, complicated, and error prone.

Those days are gone. There are now scripts that do all the hard work of setting it up and configuring users and keys for you. Answering a few simple questions, in a couple minutes you can have it up and running on your Linux server, and using all the current best practices to boot.

There are two use cases, and there are scripts for each, supporting at least Debian/Ubuntu/CentOS, which are both based upon the earlier work of Nyr:

1. A multi-user, secure OpenVPN server, with quasi-anonymous usage.
2. An OpenVPN server for personal use, supporting 3 simultaneous connections and potentially older clients.

In case one, use Angristan's script. To get it, run:

wget --no-check-certificate http://bit.ly/ovpn-install -O openvpn-install.sh

In case two, use jtbr's script, which is a lightly modified version of Angristan's.

wget --no-check-certificate http://bit.ly/openvpn-install -O openvpn-install.sh

You can click through to the link to see the extra security measures being taken by these scripts.
For either case, now simply run it as root after adding execute permissions:

chmod +x openvpn-install.sh
sudo ./openvpn-install.sh

For case two, if you want to change the number of simultaneous clients allowed, simply change the line max-clients 3 to the appropriate number before running, or remove it altogether for no limit.

The first run will set it up, and add the first client key (.ovpn file, which is placed in your home directory). Subsequent runs allow adding or removing clients or uninstalling. The generated client keys need to be (securely!) copied on to the clients' devices and imported into their OpenVPN clients. Once it's on their computer, this is usually drag and drop. For iPad/iPhone, I found the easiest option to be to use iTunes File Sharing to save to the OpenVPN Connect app. Then they can connect at will.
You're all set!

These scripts enable clients to access the internet and appear as if they are coming from the server's IP, a typical road-warrior setup; and depending upon the location of your server, to allow clients to access the server's local network. To allow VPN clients to connect to each other, or to allow access to the clients' network, then you'll need some additional routing configuration; more help here.

Especially if you're using a VPN for anonymity, you should also put some effort into ensuring your clients don't have any IP leaks (where your IP is discoverable by sites you access). A good guide to IP leaks and how to fix them is here; for a quick check, try here.

Deploying a website to Amazon S3

You've generated a great website that you now want to deploy to Amazon S3 to have hosted as a server. I assume you've set up an S3 bucket and configured it for website hosting. Now you have to transfer the files comprising your site to S3, and keep them up to date. You can do this manually with a GUI tool like CrossFTP or S3 Browser. But perhaps you have a few tweaks you'd like to make before you deploy it. Most importantly you'd like to have clients be able to make use of gzip compression when the site is served, since this can greatly speed up site loading, and also reduce your costs. And you can optimize/minify your images for websites. Finally, you need to set all the custom S3 settings you need (Cache-Control headers, Reduced redundancy storage, etc), in one step, reducing requests (and thus costs) and saving lots of time and hassle. And ideally, if you make changes to an existing site, you should only need to upload the files that have changed, again saving time and money.

Here, I present a script that can do all of this, using the convenient example of a generated Gallerific Web Album. This script takes considerably more technical skill relative to generating an album, since you will need to use a command line in the form of a bash shell to use it (which comes with MacOS, Linux, and you can get it with cygwin in Windows, as I use), but it's worth it!

To use it you'll need several tools:

• s3cmd which is used for copying to/interacting with your S3 bucket. To install:
• Linux: normally you can get the package as here.
• Cygwin: get zip from github, run python setup.py install [which requires python-setuptools and to have run easy_install dateutil]
• Mac via homebrew: brew update && brew install s3cmd
• In all cases you must then run s3cmd --configure to provide your S3 access keys.
• mozjpeg v3.0+ which is now the best tool for compressing jpegs for use on the web (available in binary form here). (Be sure that jpegtran/djpeg/cjpeg on your path is this one, and not the default libjpeg implementation.) Lossless compression rates of 60%+ are achieved on thumbnails, and 8-10% on larger images. (It is now as good or better than common online tools and better than things like jpegrescan and adept.sh by my testing.)
• gzip, or optionally zopfli, for compressing text files as a traditional web server would do (zopfli produces files that are 4-8% smaller than gzip -9. Note that although 7zip also produces smaller files than gzip, it will not work with this script because the timestamp cannot be excluded and so MD5 signatures for files always change)

If you're curious, here is the script (deploy_to_s3):

#!/bin/bash
##################################################
# Deploys a gallerific gallery to the web server
##################################################
# Usage: deploy_to_s3 [directorytodeploy]
##############CONFIG##############################
S3_BUCKET=s3://mys3bucket
REDUCED_REDUNDANCY=-rr
ADD_HOME_LINK=1 # set 'pathtohome/index.html' as applicable
#GZIP_CMD='gzip -n -9'
GZIP_CMD=zopfli
JPEG_QUALITY=80  # lossless or number (recommend 65-85)
#S3CMD_DEBUG="-v --dry-run" # can use --dry-run on s3cmd to see the effects but not execute, also can use -v for verbose
##################################################

GALLERY_DIR=$1
GALLERY_DIR=${GALLERY_DIR%/} # remove trailing slash
if [ -z "$GALLERY_DIR" ] || [ ! -d "$GALLERY_DIR" ] ; then 
  echo Invalid Parameter. USAGE: deploy_to_s3 [directory to deploy] 
  exit
fi
DEPLOY_DIR=$(mktemp -d)
if [ $JPEG_QUALITY == "lossless" ] ; then 
  JPEG_CMD='jpegtran -outfile {}.tmp {}'
  JPEG_CMD2='mv {}.tmp {}'
else
  JPEG_CMD='djpeg -outfile {}.pnm {}' 
  JPEG_CMD2="cjpeg -quality $JPEG_QUALITY -outfile {} {}.pnm"
fi

cp -Rf $GALLERY_DIR $DEPLOY_DIR

if [ $ADD_HOME_LINK -eq 1 ]; then
  echo ADDING CUSTOM HOME LINK TO INDEX.HTML ...
  sed -i '/<div class="header"[^<]*>/ a <p><a style="color: #548BBF !important;" href="pathtohome/index.html">Home</a></p>' \
      $DEPLOY_DIR/$GALLERY_DIR/index.html 
fi

# compress text files and mark them for direct download by browsers using gzip encoding 
echo COMPRESSING TEXT FILES ...
find $DEPLOY_DIR -regex '.*\.\(html\|js\|css\|xml\|svg\|txt\)' -exec $GZIP_CMD {} \; -exec mv {}.gz {} \;
echo UPLOADING COMPRESSED TEXT FILES ...
s3cmd sync $S3CMD_DEBUG --acl-public $REDUCED_REDUNDANCY --add-header="Content-Encoding":"gzip" \
              --guess-mime-type --exclude='*' --rinclude='.*\.(html|js|css|xml)' --signature-v2 $DEPLOY_DIR/$GALLERY_DIR $S3_BUCKET/
echo
echo

# re-compress jpegs using mozjpeg encoder (https://github.com/mozilla/mozjpeg, http://mozjpeg.codelove.de/binaries.html)
echo IMPROVING COMPRESSION ON JPGS ...
# Note some EXIF/comments may be lost, particularly using lossy compression
#pushd/popd necessary because mozjpeg for windows doesn't handle roots (eg /tmp)
pushd $DEPLOY_DIR/$GALLERY_DIR
find im -regex '.*\.jpg' -exec $JPEG_CMD \; -exec $JPEG_CMD2 \;
find im -regex '.*\.pnm' -exec rm -f {} \; # clean up after lossy compression
popd
echo UPLOADING IMAGES AND REMAINING FILES ...
s3cmd sync $S3CMD_DEBUG --acl-public $REDUCED_REDUNDANCY --add-header="Cache-Control":"public,max-age=86400,no-transform" --guess-mime-type \
                --signature-v2 $DEPLOY_DIR/$GALLERY_DIR $S3_BUCKET/
echo
echo

echo REMOVING DELETED FILES ON S3 SIDE CLEANING UP TEMP FILES ...
s3cmd sync $S3CMD_DEBUG --delete-removed --acl-public $DEPLOY_DIR/$GALLERY_DIR/* $S3_BUCKET/$GALLERY_DIR/
rm -rf $DEPLOY_DIR
echo
echo
echo DONE!

To just use it as is, save the file  deploy_to_s3  and make it executable (chmod +x deploy_to_s3). Next you need to configure it:

1. Set S3_BUCKET to point to your bucket
2. If you wish to use gzip, uncomment the first GZIP_CMD line and comment the other.
3. Before you run for real, I'd recommend uncommenting the S3CMD_DEBUG option to see what the effects would be but not execute them.
4. I like to add a home link at the top of my pages. to do this set ADD_HOME_LINK=1 and change the path to your homepage in the html code (<p>...</p>) in the sed command.
5. Choose the jpeg compression quality, either "lossless", or an integer typically 65-80.
6. If you don't want to use reduced redundancy storage, comment out the REDUCED_REDUNDANCY option.
7. Change what directories get image compression (currently im/ and subdirs)

To run, go to the parent of your gallerific web album's folder. Then simply execute:

deploy_to_s3 [mygallerificwebalbum]

A few notes/features:

• If you re-generate the album or make tweaks, but only some files have changed, you can re-run deploy_to_s3 and only the files that have changed will be transferred (saving time and money again).
• Only jpegs in the im directory will be compressed. Other images in gallerific web albums have already been minified. 
• For use with gallerific, I recommend using lossy compression with this script, combined with very high quality settings when saving JPGs in Lightroom. This will lead to much smaller jpegs than using a lower quality setting in Lightroom combined with lossless compression here. For example, a full size image that I find to be acceptable quality for the web at 65% quality in Lightroom was 827kb. This can be losslessly compressed to 769k. However if the image is saved at 95% quality in Lightroom (effectively lossless) and compressed at 75% using mozjpeg, it occupies only 580kb and has approximately equivalent image quality. The only downside is potentially lost EXIF/comment data with lossy compression (and your local copy will be quite large - a perhaps more reasonable compromise if that is a concern is LR quality 92, mozjpeg 80). [Note that I've learned that LR jpeg quality for the Web is not the same as that used for normal LR export; it appears to be generally lower.]
• HTTP headers are added so that browsers will recognize and get the gzip format compressed text files over the internet and use them as normal; you and users save significant time and bandwidth and the page loads faster.
• The Cache-Control HTTP header is set to allow browsers (and CDNs/proxies) to cache all non-text (html/css/js/xml) files for 24 days, dramatically speeding reducing HTTP requests and subsequent views. But be aware that if you make changes to these binary files, users with cached files may not see them unless you rename the files.

Although it does take a bit of work to set it up, using this tool should save you and users enough time to enjoy a choice beverage, which you now deserve. So do!

Stably monitoring console programs in Linux

So you want to run a program, while keeping a log file for posterity. Obviously you redirect stdout to a log file (and in this case stderr as well, while running it in the background):

myprogram someparams > myprogram.log 2>&1 &

Now you also want to monitor it while it runs to see how things are going (there is also tee for this, but then you have to keep monitoring). So use:

tail -f myprogram.log

But what if you've ssh'ed into the shell so it's possible that you may get disconnected. You don't want to have your program abort just because you lose connection, or your shell ends for whatever reason. You could use bash's disown, if you've already run the program. But if you're thinking ahead, simply:

nohup myprogram someparams > myprogram.log 2>&1 &

You probably already knew that too. But actually, that raises a problem. No nohup is buffering stdout using glibc and so if you want to monitor output, you'll be waiting a while, and only get it in chunks (or possibly even at the end of execution). To the rescue comes unbuffer, part of the expect package. (In ubuntu, install via apt-get install expect). So finally:

unbuffer nohup myprogram someparams > myprogram.log 2>&1 &

This works by tricking the kernel into thinking it's writing to console, which changes it to line buffering. But there's actually now an even better way, using stdbuf, assuming you have it.

stdbuf -oL nohup myprogram someparams > myprogram.log 2>&1 &

And now you can tail to monitor continuously without issues!

Tips for using an SSD under Linux

Solid state drives (SSDs) offer amazing performance, but do have one major limitation. There is a maximum limit to how many times it can be written to (or to think of it another way, that memory can be overwritten). Moreover, writes can be slowed when previously freed space has not been reclaimed by the SSD, because then this reclaiming needs to happen before the write can occur. TRIM commands from the operating system to the SSD allow the drive to do this in the background. Unfortunately this isn't as well integrated into the OS as under Windows.

To get all set up and optimize the life and performance of your SSD, I recommend the following steps:

1) Set your drive not to set access times on files (otherwise the inodes need to be written each time you do)

in /etc/fstab. Normally these are not needed and will save you from writing to the drive each and every time you read. To do this, add the noatime and nodiratime parameters to your SSD drive, eg:

UUID=xxxx-xxx-xxx-xxxx / ext4 noatime,nodiratime,errors=remount-ro  0  1

Note: DO NOT use the discard option, which is the original method of enabling TRIM support and which does not perform well, especially when deleting large numbers of large files (instead, use the userspace command fstrim, as below).

Finally, if you think you really need the access times, you can use the relatime setting instead of noatime -- it doesn't save as many writes, but still saves a lot (in fact this is usually now the default option).

2) To send TRIM commands to the HD, run the fstrim utility regularly to notify the SSD of newly freed space. Make a new file called fstrim in /etc/cron.daily, with the following contents:

#!/bin/sh

echo Running fstrim on SSD.

fstrim /

where the / should be the mount point of the SSD. Run chmod +x /etc/cron.daily/fstrim to make it executable. It will run automatically each day.

3) Don't use swap space on the SSD unless you need to. Using the SSD for swap space is way faster than a HDD, but swap is normally a heavy disk user, and we need to avoid unnecessary writes on an SSD. So what to do? Well, one, you might not need a swap if you have a lot of RAM and not so much demand. In my case I do need that extra memory occasionally, and it kind of does need to be SSD-fast. So what I ended up doing is telling Linux to only use the swap when absolutely necessary (eg, when it's running out of memory) -- by setting swappiness to a setting well-below the default; I chose 5, but something in the 0-10 range is probably good (0 tells linux to only use the swap when it is completely out of physical memory when it tries to allocate; higher may allow some degree of caching in swap space). To set swappiness, add this line in /etc/sysctl.conf and reboot:

# Change default swappiness (60) to reduce unnecessary writes to SSD swap

vm.swappiness = 5

4) Use ram disks for common temporary files to avoid unnecessary writes to disk. tmpfs will allocate space for ramdisks as-needed. I did this for 3 directories: /var, /var/tmp, and /var/spool, by emptying out those directories and placing the following lines in /etc/fstab:

# Use ramdisks for temp files to avoid excessive writes to SSD

tmpfs /tmp    tmpfs defaults,noatime,mode=1777 0 0

tmpfs /var/spool tmpfs defaults,noatime,mode=1777 0 0

tmpfs /var/tmp   tmpfs defaults,noatime,mode=1777 0 0

5) Change the I/O scheduler to DEADLINE for your SDD. The default is CFQ which is great for HDDs but overkill and slower for SDDs. For SDD, you don't really even need a scheduler, and could instead choose NOOP which is a simple first in first out scheduler, but DEADLINE is a little better, since for example it prioritizes reads over writes in the queue, ensuring decent read responsiveness even under heavy writes. Put the following lines (after modifying to match your drive locations sda/sdb/etc) in your /etc/rc.login so they run every bootup.

# use deadline scheduler for SDDs (and CFQ for HDDs)
echo deadline > /sys/block/sda/queue/scheduler


5) Finally, you can move caches for your commonly used programs (especially the web browser) off of the SSD. These write a lot to disk, and are already well optimized to work with a hard drive (HDD). If you also have an HDD, you can redirect the .cache directory under your home onto that HDD. Since cache is already optimized for hard drives, it shouldn't negatively impact performance unduly. (Alternatively you could do something similar for just the browser cache(s) using tmpfs, although the cache will be lost each reboot). First, close any programs that may be using the cache (or better, login at the terminal) Then run these commands to create a symbolic link:

cd /myHDDmountpt

mkdir .cache

cp -R ~/.cache .cache

cd

rm -Rf .cache

ln -s /myHDDmountpt/.cache .

Hopefully these tips will help your SSD live long and perform well. Let me know in the comments if you know any other good tricks!

Using the same ssh key between linux and mac

If you are dual booting and you want to be able to ssh in (particularly by IP, or if the host names are the same) to both operating systems, then you'll want to set up ssh to share the same server keys between operating systems.

Otherwise, after a client connects with one operating system and then tries to connect again while you're booted in the other OS, they'll get this message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.

To use the same ssh keys, copy them from one OS to the other. I chose to use the ones from linux in macos (in macos they're stored directly under /etc, in ubuntu/linux mint they're under /etc/ssh/):

 /etc/ssh $ ls -l
total 276
-rw-r--r-- 1 root root 242091 Nov  8 14:37 moduli
-rw-r--r-- 1 root root   1669 Mar 26  2013 ssh_config
-rw-r--r-- 1 root root   2489 Nov 11 12:57 sshd_config
-rw------- 1 root root    668 Nov 11 12:57 ssh_host_dsa_key
-rw-r--r-- 1 root root    600 Nov 11 12:57 ssh_host_dsa_key.pub
-rw------- 1 root root    227 Nov 11 12:57 ssh_host_ecdsa_key
-rw-r--r-- 1 root root    172 Nov 11 12:57 ssh_host_ecdsa_key.pub
-rw------- 1 root root   1679 Nov 11 12:57 ssh_host_rsa_key
-rw-r--r-- 1 root root    392 Nov 11 12:57 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root    338 Dec  2 15:52 ssh_import_id
 /etc/ssh $ sudo mkdir /macos/etc/ssh-backup
 /etc/ssh $ sudo mv /macos/etc/ssh_host* /macos/etc/ssh-backup/
 /etc/ssh $ sudo cp ssh_host* /macos/etc
 /etc/ssh $ ls -l /macos/etc/ssh*
-rw-r--r-- 1 root root    0 Nov  4 23:27 /macos/etc/ssh_config
-rw-r--r-- 1 root root    0 Nov  4 23:27 /macos/etc/sshd_config
-rw------- 1 root root  668 Dec  4 18:48 /macos/etc/ssh_host_dsa_key
-rw-r--r-- 1 root root  600 Dec  4 18:48 /macos/etc/ssh_host_dsa_key.pub
-rw------- 1 root root  227 Dec  4 18:48 /macos/etc/ssh_host_ecdsa_key
-rw-r--r-- 1 root root  172 Dec  4 18:48 /macos/etc/ssh_host_ecdsa_key.pub
-rw------- 1 root root 1679 Dec  4 18:48 /macos/etc/ssh_host_rsa_key
-rw-r--r-- 1 root root  392 Dec  4 18:48 /macos/etc/ssh_host_rsa_key.pub

/macos/etc/ssh-backup:
total 24
-rw------- 1 root root  668 Nov 11 00:23 ssh_host_dsa_key
-rw-r--r-- 1 root root  590 Nov 11 00:23 ssh_host_dsa_key.pub
-rw------- 1 root root  965 Nov 11 00:23 ssh_host_key
-rw-r--r-- 1 root root  630 Nov 11 00:23 ssh_host_key.pub
-rw------- 1 root root 1679 Nov 11 00:23 ssh_host_rsa_key
-rw-r--r-- 1 root root  382 Nov 11 00:23 ssh_host_rsa_key.pub

Voila. You can ssh into either OS without any warnings.

You may also want to set the same ~/.ssh/authorized_keys under home dirs in both your OSes, to allow password-free authentication between the client and both servers. A reminder on how to set this up --
On the client:

cd ~/.ssh

ssh-keygen -f id_rsa -C 'comment to identify client' -N '' -t rsa -b 4096 -q

This creates a private and a public key. Copy the id_rsa.pub key to the server. (eg, scp id_rsa.pub user@serverhost:/home/user/ ). On the server:

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

rm id_rsa.pub

All set. You can repeat as desired for each client user/machine. Then just copy authorized keys to the other OS and it'll work without a password in either.

Notes on Wake on Lan (WOL)

These are some notes on a current problem, even more disjointed than usual. I wanted to have Wake on Demand working on both mac and linux on the same machine, with the same ethernet port. This isn't working out well (in linux at the moment), so I'm shelving it and at some point maybe I'll get back to it.

In MacOS, I found that the intel driver fails but the atheros card works fine for WOMP and WOD, and USB keyboard.

Under linux:
To enable wake with USB keyboard, you need to add:
# enable waking from sleep from USB HID devices
# $ lsusb -d 046d:
# the above gets the usb bus and device, but not port (5 in the case below)
echo enabled > /sys/bus/usb/devices/usb3/power/wakeup
echo enabled > /sys/bus/usb/devices/usb3/3-5/power/wakeup
to /etc/rc.local

The Intel card works fine for Wake on Magic packet (but since it doesn't for the mac, I've abandoned it for the moment).

Wake on Magic Packet fails for Atheros card. This appears to be an issue in the alx drivers, which aren't being updated it seems (maybe they are?). The lights are off during suspend. There is a compounding issue in that if you suspend via dbus (which appears to be what gnome/cinnamon does), when you come back the ethernet card will not work properly, irrespective of what you put in /usr/lib/pm-utils/sleep.d/50unload_alx to not disable/re-enable the alx driver (which was itself put there to fix a third bug in alx, which was apparently causing sleep to fail altogether).  (Updating alx doesn't help). You have to run rmmod alx; modprobe alx as root to re-enable networking after waking. If instead of using dbus, you simply call pm-suspend, there are no issues with the Ethernet connection dropping upon awake.

Note that enabling the disabled wakeup options for the card and its pci bus as below make no difference:
echo enabled > /sys/bus/pci/devices/0000\:00\:1c.0/power/wakeup
echo enabled > /sys/bus/pci/devices/0000\:00\:1c.3/power/wakeup
echo enabled > /sys/bus/pci/devices/0000\:02\:00.0/power/wakeup
( lspci & lspci -tv help with these)

Here's the dbus suspend command I've used: dbus-send --system --print-reply \

    --dest="org.freedesktop.UPower" \
    /org/freedesktop/UPower \
    org.freedesktop.UPower.Suspend

*** Actually, this guy has some interesting troubleshooting and it seems pm-hibernate might work with WOL (and since I have an SSD, isn't that different from sleep). But, I tried pm-hibernate and I seem to be getting the third bug in alx mentioned above, that hibernate fails to actually go into hibernation.

The cool app SleepProxyClient, which I'd really like to get working on the machine, uses pm-suspend (at least now). If it worked, we could sleep and then have things awake when needed (eg, samba requests), by making use of the bonjour sleep proxy on my appletv. I haven't tested it with the intel card, which might just work. Here are some useful pages about this:
http://www.cnpbagwell.com/mac-os-x/bonjour-sleep-proxy
http://stuartcheshire.org/SleepProxy/
Avahi is the linux implementation of bonjour.

There is a link about autoshutdown on the sleepproxyclient page above, and this may also help in deciding when to sleep. So might this, which I found somewhere:

#!/bin/bash
hdparm -C /dev/sda | grep "active" > /dev/null
SDA=$?
hdparm -C /dev/sdb | grep "active" > /dev/null
SDB=$?
if [ $SDA -eq 1 ] && [ $SDB -eq $SDA ]
then
 echo "drives asleep" 
 LOAD=`awk {'print $1'} /proc/loadavg`
 echo $LOAD
 if [ "$LOAD" == '0.00' ]
 then
  echo "load hit threshold"
  echo "Sleeping"
  # Call sleep process here
 fi
fi

Open and Password protected shares in Samba

If you want to have more than one share for samba, with one open to anyone on the network (as a guest) and one open only to named users (in this case 'realuser', which should be an actual unix account), there are a couple of complications. The following is a config file that works (it assumes you've followed the smbguest username setup steps from this post).

[global]
  security = User
  map to guest = Bad User
  null passwords = Yes
  obey pam restrictions = Yes
  # this account must exist as unix users
  guest account = smbguest
  unix password sync = Yes
  # these two are for performance
  socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=32768
  strict allocate = Yes

[data]
  comment = data
  path = /data
  force user = smbguest
  read only = No
  create mask = 0666
  directory mask = 0777
  guest only = Yes
  guest ok = Yes

[private]
  comment = private
  path = /private
  valid users = realuser
  admin users = realuser
  read only = No
  guest ok = No
  create mask = 0666
  directory mask = 0777
  # this makes it invisible when browsing the computer
  browseable = No

You'll have to add realuser as a username accessible from samba:

sudo smbpasswd -a realuser

But it's a bit tricky because authentication is by users, not shares. So if you want to be able to access the private share (using realuser and its username), as well as the data share (that's open to guests), you'll need to log into both as realuser. Note that in windows 7 the username is \realuser (the backslash is necessary to remove the  domain).  Note that if you do this only in linux, you'll want to make sure you have the map to guest = bad user line after the security=user line under MacOS to allow access to users who have saved mapped the drives while linux was running.

P.S. - For debugging your smb.conf file, it helps to run testparm -s (which gives warnings and errors).
For debugging windows 7-related issues related to samba, see here.

DVD/Blu ray under linux

Unfortunately from what I read, neither MacOS nor Linux can play back blu-ray disks properly. That's because it's a closed format and they are only providing licenses for players in a very convoluted (and expensive) way. So there are no licensed players outside Windows, and certainly none that are free. That leaves those of us who still want to be able to watch a movie grappling to find solutions.

Fortunately there are some. First, vlc (apt-get install vlc) appears to play DVDs fine most of the time. You can try here for setting up blu-ray. It's a matter of adding a library and some keys, but like I said, it's apparently not great.

The alternative is to save the movies to disk first, and play them from there. This is also handy if you want to have them always available for your home theater.

To rip, MakeMKV seems to work great even though it's in beta. It's free too, for now. Just follow the directions at the link, which work flawlessly for me.

Now, you can insert a dvd/blu-ray and save it directly to disk. Unfortunately they're still full size, which can be quite big. Handbrake is a great tool for transcoding it for the output device you want to use. Simply:
sudo add-apt-repository ppa:stebbins/handbrake-releases
sudo apt-get update
sudo apt-get handbrake-gtk

Then it can be run with a nice GUI as handbrake-gtk. But there is also a useful command line interface. I've modified a tool I found in the handbrake forums (written for macos) that allows transcoding a whole directory full of files in a batch. The code is below the fold.

Linux Mint / Mac OS 10.9 dual boot setup on Haswell i7-4770 GA-H87N-WIFI

Setup Objective:  Dual Boot MacOS / Linux

My new computer hardware:

• MI--008 Mini-ITX Black Case with 250w PSU
• Gigabyte GA-H87N-WIFI motherboard with 1150 slot
• Intel Core i7-4770 Quad Core 3.4Ghz processor with HD 4600 integrated graphics
• 16GB DDR3 1600 MT/s (PC3-12800) Crucial Ballistix Sport RAM
• Samsung 840EVO 250GB SSD (boot drive)
• Seagate Barracuda 2TB HDD (data drive)
• LG Blu-Ray reader / DVD-RW UH12NS30
• Gentle Typhoon 120mm cooling fan

This amounts to a pretty tiny PC with lots of computing power (but low energy usage). I do wish I could install more RAM, but 16GB is the limit. I'll use it mostly for intensive processing for my dissertation and for work [I'm a programmer]. But I may also do photo editing with the mac side and use it as a part-time HTPC under linux/xbmc.

Instructions

A) Install MacOS:

1.  Purchase and download (but don't install) MacOS from an existing Mac. Use Unibeast to setup a USB key with a bootable MacOS installation. I used OSX 10.9 (Mavericks). Also copy Multibeast 6.0 to the USB key.
2. Setup BIOS to use the optimized settings with the following modifications:
1. Disable VT-d. 
2. Extreme Memory -> Profile1.
3. Make IGFX the first option.
4. UEFI and Legacy boot.
3. Take the key to the new computer and boot. I needed to use the "-x PCIRootUID=1" (no quotes) settings to get the video mode to work. Install normally, using the whole boot drive as a partition. It can be shrunk later. Instructions here.
4. Upon reboot, again boot with the USB, this time selecting the new mac partition, again with the same settings.
5. Setup: System Preferences->Security & Privacy->Unlock; Then: allow apps downloaded from anywhere. Back arrow. Mouse->Disable scroll direction "natural".
6. Copy multibeast to the desktop and run. With my hardware, most things work straight out of the box. Select the following (de-selecting all else):
1. DSDT Free
2. Audio->Realtek ALCxxx > Without DSDT->ALC892
3. 3rd Party SATA
4. TRIM Enabler -> 10.9.0 TRIM Patch (for SSD)
5. FakeSMC v5.3.820 (Plugins and HWMonitor are nice, optional extras)
6. Network->Atheros -- Shailua's ALXEthernet v1.0.2
7. Network->Intel -- hank's AppleIntelE1000e v2.5.4d     (need both to support both ethernet ports)
8. System->Patched AppleIntelCPUPowerManagement -> OS X 10.9.0 --- THIS IS NECESSARY; otherwise boot will hang without safe mode on after loading bluetooth drivers.
9. System -> AppleRTC Patch for CMOS Reset
10. Boot Options-> Generate CPU States, GraphicsEnabler=Yes, Hibernate mode-Desktop, PCIRootID Fix (this sets PCIRootUID=1), Use KernelCache. Might want to still edit /Extra/org.chameleon.Boot.plist to add GraphicsMode to match your monitor. [App Store -> Find/Install TextWrangler; Finder->Go->Go to folder-> /Extras]
11. System Definitions -> Mac Pro ->Mac Pro 3.1 (don't know if this is optimal, but it works)
7. If you can boot now, it's a good time to setup linux (below). But first fix the drives up some.
1. Open disk utility and shrink the macos partition on the boot drive. 
2. Format the data disk using NTFS. NTFS is the best choice for a drive between windows and anything, but also between macOS and linux, supporting >4GB files and big drives. 
3. Enable read/write support for the data drive: from the terminal, run sudo vifs and add this line: /dev/disk1s1 /Volumes/data ntfs rw,auto,nobrowse. The first is the partition (make sure this is correct) and the second is the mount point (a directory which you must create). Run sudo mount -a to test that it works; browse to this location with finder. Then drag the directory into the favorites list to the left of finder (for easy access later). It will be mounted at boot up from now on.
8. At this point most things worked, except WIFI, which is a known limitation. But not audio via the HDMI cable, and the video drivers were flickering/unstable around animations. To fix both these issues, do toleda's HDMI fix for Haswell systems. Be careful to follow the directions exactly. First verify pre-requisites, then perform all steps in the "Before you start" section, referring to the Notes section below. Finally, do either the DSDT or the SSDT installation method. I did the DSDT method, and selected the -1 variant of the driver, which worked nicely.
9. Setup SAMBA sharing to match linux.
10. VNC setup

B) Linux Setup. I used Linux Mint 15 [Cinnamon 64], for which I burned an install DVD. You will also need a grub rescue DVD.

1. Boot up linux mint on the install DVD, being sure to select the UEFI DVD option (and not the P? option) from the BIOS boot menu. DO NOT INSTALL DIRECTLY FROM BOOTUP, as it will fail to properly setup grub. Instead, once Linux Mint has booted, double click the Installer program on the desktop to start installation.
2. Choose "something else" for how to install the OS. Create an ext4 partition at the end of the drive. I did not create a swap partition, preferring a file instead as it's more flexible (see below). At the bottom, choose to install Grub in /dev/sdb3 (or whatever your new ext4 partition is). If you install it in the default /dev/sdb, it will overwrite chimera and macOS won't boot.
3. Next, boot with the grub rescue DVD. Select the partition you just installed [/dev/sdb3] to load the new linux install.
4. To add linux to the UEFI boot menu, sudo cp /boot/efi/EFI/linuxmint/grubx64.edi /boot/efi/EFI/boot/bootx64.efi [efibootmgr does not work correctly for me, I strongly suspect Gigabyte's UEFI implementation is faulty]. Grub will not install automatically to the EFI either, and I can't get Grub to load chimera or the mac kernel either. I even tried creating partitions just for grub or using the data disk MBR, to no success. Basically I tried about 100 options (which means >100 reboots) and no fully satisfactory option arose for a good boot loader for both mac and linux. I settled for using the bios to choose either the boot disk MBR (to load MACOS via chimera) or to use UEFI default boot (to load Linux via grub partition boot loader). A bit ugly, but at least it works. [Update - Linux Mint 16 may have fixed UEFI support for this motherboard. I have just installed Mint 17.1, and it appears that as long as the install DVD was booted in EFI mode, mint will set up grub properly to boot. If not, you may need to run boot repair]
5. Setup swap file (I made mine 16GB, same as RAM, but it may be worth investigating the "swapspace" tool for dynamic swap space, or you may not even need one at all):
1. sudo fallocate -l 16G /var/swapfile
2. sudo mkswap /var/swapfile
3. sudo swapon /var/swapfile
4. You should see your swap listed when you run free.
5. Edit /etc/fstab to use it at reboot, adding this line: /var/swapfile  none  swap sw  0  0
6. Setup Paragon UFSD drivers. These are kernel-level drivers that permit read-write for both NTFS drives and HFS+ drives (even with journaling enabled). As such they're a major improvement over the drivers included as redistributables with linux. NTFS writing is significantly faster since it's not a user-space driver. Although it is proprietary, it's free for personal use. Fill in a little form and a few minutes later you'll get the download link via email. To install, simply:
1. Download them to ~/Downloads. 
2. cd ~/Downloads; mkdir ufsd; cd ufsd; tar -xvzf ../Paragon-147-FRE_NTFS_Linux_8.9.0_Express.tar.gz . 
3. See the README to confirm all is good. Although the website is outdated, it supports kernels up to 3.11.x as of this writing and works fine under ubuntu/mint (I have kernel 3.8.0-19). Then run sudo ./install.sh.
4. Edit fstab to use the new drivers by replacing the mount types with ufsd. I have these two lines:  /dev/sda1  /data  ufsd uid=myusername,gid=smbusers,noatime,umask=0002  0  2
   /dev/sdb2  /macos ufsd defaults 0 3
5. You may also need to clear the dirty flag on the drive by rebooting properly from Window/MacOS, or by using fsck (otherwise ufsd will give error 1000). Check dmesg | tail for more detailed errors when mounting. In Linux mint 16, (not in 15 or 17), I had to add ufsd to /etc/modules for it to load at boot.
7. Setup SAMBA shares to match macos. 
8. To play videos in vlc (and possibly other apps), it was necessary for me to choose Tools->preferences->Video->Output as OpenGL GLX or X11 Video.
9. To get HDMI audio working, I had to do as I later saw was suggested in the release notes (directions below). I also found the pavucontrol app to be helpful. You may need to disable SPDIF using the configuration tab of this tool to use HDMI.
1. sudo add-apt-repository ppa:ubuntu-audio-dev/alsa-daily
2. apt update
3. apt install oem-audio-hda-daily-dkms
10. It may be worth changing your user's UID to match that of macos, for while writing to the HFS partition. That can be done by looking it up under macos: ls -n in the home directory (mine was 501). Then from the linux console (ctrl-alt-F1), login as root and run usermod -u uid username. If like me you chose to create an encrypted home directory, also run chown -R username /home/.ecryptfs/username. Reboot.
11. Use software manager GUI to install all the wonderful free packages you want.
    
    
    Et, Voila!  That simple, in retrospect.

Edit: Also have a look at these tips I wrote for optimizing the solid state drive (SSD) performance under linux.

Same Samba share for linux and macos

I have a shared data drive that I want to use as a file server on the local network. But as I have a dual boot, I want the same share to be available on both linux and macOS, and I want it to be an open share without a password. First, to mount it under both, see my computer setup post.

1) Under linux, it's pretty straightforward. Make sure that the data disk is mounted, and you have the samba3 package.

Then create a samba user and group:
sudo groupadd smbusers 
sudo useradd -g smbusers -d /dev/null -s /bin/false smbguest

add primary user(s) to smbusers group (just in case):

sudo usermod -a username -G smbusers

and make smbusers the default gid for the mount (within /etc/fstab; I use ufsd, otherwise replace with ntfs):

/dev/sda1    /data        ufsd    uid=briggsjt,gid=smbusers,noatime,umask=
0002    0    1

Then edit /etc/smb.conf, adding:

guest account = smbguest

force user = smbguest

null passwords = yes

and add an entry for the share:

[data]
        comment = data
        path = /data
        writeable = yes
        browseable = yes
        guest ok = yes
        read only = no
        create mask = 666  
        directory mask = 777

I recommend the following for best performance:

socket options = TCP_NODELAY IPTOS_LOWDELAY

strict allocate = yes

The latter mainly helps for ext4/xfs/btrfs drives, but hey. 

Many people recommend setting SO_RCVBUF=8192 and SO_SNDBUF=8192. Here in 2013, I found these to be much slower than the linux defaults (which are 87380 and 16384 respectively (from /proc/sys/net/ipv4/tcp_[rw]mem)). Setting them slightly higher didn't seem to help either. I played with a lot of other things but nothing else much improved over defaults. I was surprised to see how much slower my wireless network was than wired. (Wired I get 40-45MB/s sustained write to the NTFS drive, wireless it's more like 4-10, with 802.11n dual band and no obstructions).

To test changes to smbd:

sudo service smbd restart

We can also optionally have our samba service advertise itself via bonjour over the network using avahi-daemon (so it will appear in Finder on Macs, etc): Simply create a file /etc/avahi/services/smb.service with the following contents:

<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
 <name replace-wildcards="yes">%h</name>
 <service>
   <type>_smb._tcp</type>
   <port>445</port>
 </service>
 <service>
   <type>_device-info._tcp</type>
   <port>0</port>
   <txt-record>model=RackMac</txt-record>
 </service>
</service-group>

And run sudo service avahi-daemon restart

2) Under MacOS:

There is a version of smbd on macos, however since 10.7 or so, it's an apple implementation and it does not allow guest access without a username. This means there's no simple way to use the same authentication as linux so it will be available no matter which OS I'm in. So instead I installed samba via macports. [Be sure to disable System Preferences->Sharing->File Sharing to avoid conflicts].

First ensure macports is installed: Install Xcode from the appstore. Install Xcode command-line tools: xcode-select --install. Download the macports .pkg installer from macports.org, and run.

Then install samba:

sudo port selfupdate

sudo port install samba3

Grab a drink and when you come back it's there.

In /opt/local/etc/samba3 there's a sample config file. Rename it to smb.conf and edit, adding a share like before:

[data]
   path = /Volumes/data
   public = yes
   browseable = yes   
   guest ok = yes     
   writable = yes     
   create mask = 666  
   directory mask = 777

Files will be written/read as the default user, and the drive must be accessible to him (you) locally.

To test it out, run:

sudo /opt/sbin/smbd -D && sudo /opt/sbin/nmbd -D

Then to launch automatically at startup, make a file /Library/LaunchDaemons/org.samba3.smbd.plist with these contents:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.samba3.smbd</string>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/local/sbin/smbd</string>
    <string>-D</string>  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

Make another identical file named /Library/LaunchDaemons/org.samba3.nmbd.plist, replacing the two 'smbd' occurrences with 'nmbd'.

Then, disable the existing macos smbd:

sudo launchctl stop com.apple.netbiosd
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.netbiosd.plist

And run yours:

sudo launchctl load /Library/LaunchDaemons/org.samba3.smbd.plist
sudo launchctl load /Library/LaunchDaemons/org.samba3.nmbd.plist
sudo launchctl start org.samba3.smbd
sudo launchctl start org.samba3.nmbd

Finally be sure to set DHCP in your router to always assign the mac address of your computer to the same IP address (you could alternatively set the same static IP in both OSes). Then simply connect to the data samba share (eg: \\192.168.1.10\data or smb://192.168.1.10/data) and it will work no matter which OS you're running! 

As a finishing touch, you can optionally set up the computer to advertise smbd as a Bonjour service. On the Mac side, this will not only make it easy to find, but more importantly to me anyway, it will enable Wake On Demand. This is a pretty cool feature that ensures that when the mac is in sleep mode, because it's not actively being used, the drive shares will still remain available whenever anyone on the network needs them, because the computer will be notified to start back up and respond to any requests that could come. In a use scenario like mine, in a home, this can save a ton of power by often being on standby, without sacrificing convenience. To work, it requires another device on the network that is always "on" and that acts as a sleep proxy. An AirPort Base Station, Time Capsule, or Apple TV will automatically perform this function, and other devices can be set up to.

Anyway, setting it up is actually a pretty simple matter of running the dns-sd command, and leaving it running while the computer is up (this assumes that samba has the same lifetime, which it should if you did as above). It can be done by creating a file /Library/LaunchDaemons/org.samba3.smbd.servicepublisher.plist with the following content:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

<dict>

<key>Label</key>

<string>org.samba3.smbd.servicepublisher</string>

<key>ProgramArguments</key>

<array>

<string>/usr/bin/dns-sd</string>

<string>-R</string>

<string>samba</string>

<string>_smb._tcp</string>

<string>.</string>

<string>445</string>

</array>

<key>RunAtLoad</key>

<true/>

</dict>

</plist>

Note that even if you used the default mac samba implementation, it appears that macs don't advertise the samba service. Adding this file will do that and so enable Wake on Demand for it. You can run dns-sd -B _services._dns-sd._udp to see all the available services on the network, which should now include smb, once you reboot.

P.S. I had to use my second ethernet port (the atheros one) in order for this to work -- it appears the MacOS intel ethernet driver won't handle Wake On LAN properly. Details are here.

Intel power gadget for linux

This is a cool tool to tell you how much power your computer is using. Unfortunately I don't find it packaged anywhere and is not pre-built by intel. Then, it did not build for me as downloaded.

I downloaded it from here: http://software.intel.com/en-us/articles/intel-power-gadget-20 and the files were dated 24 Jun 2013. Unfortunately the makefile had the libm location wrong on the command line and wouldn't link. Here's my modification:

 CFLAGS=-g

all: rapl_lib_shared rapl_lib_static power_gadget_static

rapl_lib_shared:
    gcc $(CFLAGS) -fpic -c msr.c cpuid.c rapl.c -lm
    gcc $(CFLAGS) -shared -o librapl.so msr.o cpuid.o rapl.o -lm

rapl_lib_static:
    gcc -Wl, -t $(CFLAGS) -c msr.c cpuid.c rapl.c -lm
    ar rcs librapl.a msr.o cpuid.o rapl.o

power_gadget_static:
    gcc $(CFLAGS) power_gadget.c -I. -L. -o power_gadget ./librapl.a -lm

power_gadget:
    gcc $(CFLAGS) power_gadget.c -I. -L. -lrapl -o -lm power_gadget

gprof: CFLAGS = -pg
gprof: all
    ./power_gadget -e 100 -d 60 >/dev/null 2>&1
    gprof power_gadget > power_gadget.gprof
    rm -f gmon.out
    make clean

clean:
    rm -f power_gadget librapl.so librapl.a msr.o cpuid.o rapl.o

That builds it. But do bear in mind that it's a command-line tool in linux, unlike the nice GUIs for Windows/Mac.

Addendum: Apparently the files are no longer easily found on Intel's site. Here is the source code I downloaded.