"Improving" Sony Xperia Z1 Compact D5503

It was time for a new phone since my old one has unfortunately kicked the bucket. Luckily I found the Sony Z1 compact, which is totally badass and an amazing bargain given the specs. It's available on Amazon (also in white, pink, or lime green).* Really happy with my purchase so far.

Mine came carrier unlocked so any SIM card will work -- that's an absolute requirement for me as I travel a lot. On previous android phones, I would install a custom ROM (version of Android) in order to rid myself of the carrier and manufacturer bloat and add functionality. However, Sony's version of Android is streamlined and I like it so far. Plus, unlocking the bootloader (which is necessary to use a custom ROM), although easy to do, has downsides. It may void the warranty and more importantly reduce the quality of the camera and break anything requiring DRM. Plus custom ROMs may not have all the useful features Sony has, particularly power saving modes.

However, it's possible to root the phone without unlocking the bootloader. You'll have the stock ROMs and experience with a rooted phone that you can largely customize. In particular you can install the XPOSED framework to install XPrivacy, which like Privacy Guard in CyanogenMod allows controls over app permissions. It also allows me to run apps like AirAudio that require root. And at a pretty basic level, with root I can back up the phone.

My Z1 Compact was loaded with Android 4.4.4 Kitkat.
This guide will root the phone and upgrade it to Sony's stock Android 5.1.1 Lollipop, without wiping any software/settings. 

** UPDATE 2/14/2015 with the latest versions of apps and how to root if you're already on Lollipop (5.*) **
** UPDATE 12/7/2015: it is possible to upgrade to the latest firmware version 14.6.A.1.236 using this procedure (in step 7). If you already had upgraded to rooted 5.1+ you can just do step 7+ to get the latest firmware. This update includes the stagefright 2.0 fixes. (If you use Xposed be sure to get the latest version of that as well.) **

1. Download the latest flashtool software and install it. (You can get a head start by now downloading the firmware needed in step 7.1)
2. Once installed, go to flashtool's drivers subdirectory and run the drivers installation found there. Choose flashmode and flashboot, along with drivers for the Z1C. It'll likely take a few minutes.
3. My phone shipped with build number 14.4.A.0.157 of Kitkat (you can see this under Settings->About Phone). Unfortunately the "one-click" root solution doesn't work for this version or any other above 14.4.A.0.108. Instead we'll downgrade to .108 and root. If you're already at .108, skip this step. 
1. Depending upon what version of Android your phone came with:
1. If you have KitKat (Firmware 14.4.* / Android 4.4.*) then you can downgrade the kernel (only). Download the zip file from this guide and unzip the .108.ftf file.
2. If you have Lollipop (Firmware 14.6.* / Android 5.*.*) then you must install the whole .108 firmware. This can be found here. You may have to use File->Switch Advanced, and select to wipe userdata (You will lose your user data in this case!) for step 4, in order to be able to boot.
2. Put the .ftf file into the .flashTool/firmwares directory under your home directory (Eg, C:\Users\username\.flashTool\firmwares).
3. Enable developer mode by tapping 7 times on the Android Build under Settings -> About Phone. Then under Settings -> Developer options, enable USB debugging. And under Settings -> Security, enable Unknown Sources.
4. Run Flash tool; click the lightning button, select Flashmode. Select the A.0.108 kernel firmware file and hit Flash. Then follow directions to power down and attach your phone to the computer, while holding the volume down button until the flash process begins.
5. Once complete, unplug the USB cable, close flashtool, and turn on the phone.
4. Root the device using "one click" solution from here (in windows, the tool included in the zip from last step should work as well but is older). 
1. Ensure USB debugging and unknown sources remain enabled, as above.
2. Run install.bat in windows or install.sh on mac/linux.
3. Wait until it tells you to connect the device via USB, and then do that.
4. Now, wait until it tells you "Device Rooted", dismissing any prompts on your phone if they appear.
5. Now that you can, backup using online nandroid backup software (get and install it and required BusyBox from the Google Play Store). Save all backups to external SD and/or computer. This includes the TA partition that is wiped if you bootloader unlock. To just backup that critical partition (needed to allow you to restore stock at some point if necessary), there's also this tool.
6. Install dual recovery using installer. A recovery is like a custom BIOS that enables easier flashing of ROMs and recovery from errors. With this package you get two; you can open clockworkmod recovery by pressing the up arrow after the green/purple light flashes during bootup, and the TWRP recovery by pressing the down arrow. Sometimes one is needed rather than the other.
1. Download installer Z1C-lockeddualrecovery2.8.25-RELEASE.combined.zip (or newer) from here. This version can be used both to install initially and later to flash.
2. Unzip it (but keep the .zip for later) and run install.bat/install.sh. Choose (2) rooted with SuperUser. Connect phone when prompted and wait until complete. Reboot phone.
7. Upgrade to Kitkat using this guide to create a pre-rooted firmware.
1. Run flashtool to download the firmware version you want (14.6.A.0.368 or 14.6.A.1.236 either customized for your region or generic) by clicking the XF button, finding it and downloading by clicking its name on the far right, selecting to unpack automatically. This takes a while as it is big. Once complete, it will be in your .flashTool/firmwares directory.
2. Download SuperSU
3. Download PRFCreator, unzip it and run it.
1. Choose the FTF file you just downloaded in flashtool.
2. Choose the SuperSU binary you just downloaded.
3. Choose the Z1C-lockeddualrecovery2.8.25-RELEASE.combined.zip you downloaded previously for the recovery.
4. Select all of the checkboxes except "sign zip" (including "legacy mode" which is below sign zip) and click "Create". This will take a while.
5. It will appear in the PRF directory as flashable-prerooted.zip. Copy this to the phone/SD card.
4. Install the pre-rooted firmware.
1. Reboot your phone into TWRP recovery by pressing the down arrow after the green/purple light blinks during bootup.
2. Go to wipe, advanced wipe, choose dalvik cache and cache and wipe (maybe unnecessary but doesn't hurt).
3. Then return to main menu and install. Find the flashable-prerooted.zip file and install it.
4. Once again wipe dalvik and cache.
5. Then reboot to system. It will take a while as the new OS is configured and apps are optimized. Don't fret, all this optimizing means the apps will run faster later on. The NFC firmware will also be updated. 
8. Done! You now have a rooted phone with a locked bootloader running Lollipop 5.1.1!

To optionally install XPrivacy:

1. Via the browser on your phone, download and install the latest XPosed Installer app from http://bit.ly/1LodTO5 (I used 3.0-alpha4). It won't work yet.
2. Download the latest Xposed framework zip to your phone from the same link. The Z1Compact requires the 32-bit ARM version. For Android 5.1.* it's SDK22 (if you updated to 5.0.* instead, use SDK21). I used xposed-v80-sdk22-arm.zip. (SDK23 is for 6.* which likely won't be available on the Z1 compact).
3. Reboot the phone into TWRP recovery by hitting the volume down button once the green/purple LED lights up during boot. Install the xposed SDK file zip you just downloaded and reboot to System. The Xposed Framework is now installed, the next bootup will take a while.
4. On your phone, Now visit http://bit.ly/1QfkTxT and download and install the latest XPrivacy APK. Reboot the phone a second time.
5. Go into Xposed installer, under modules, enable XPrivacy. Reboot the phone.
6. XPrivacy can now be configured.

If you're a bit hesitant about all the steps, this can be a useful video to get the idea of how the tools work (though it's outdated and following a different procedure).


Thanks to: SlikToxic, WaleDac, Jamal, zxz0O0, and Muuuly who first found the components of this solution. And to all the linked app developers!

* Really appreciate making your purchase through these links as I'll get a small commission at no cost to you.

Creating Lightroom Web Gallery plugins

The basic reference is the Lightroom SDK Guide, Chapter 6. But the relevant section hasn't been updated since it was first written, and there's a lot missing. This forum could potentially be helpful but there isn't a lot about web plugins. There are also a LOT of quirks and bugs to workaround when creating lightroom galleries, and amazingly almost none of them are discussed on the net.

This is a very incomplete list of things I've encountered or learned:

• When debugging, changes to luascript (*.lrweb) only update reliably by restarting Lightroom. Errors therein typically cause the gallery to disappear from the list of galleries. Changes to the HTML template code itself are updated whenever you change any gallery setting that triggers a preview browser refresh (or by switching to Library view and back). Errors in this HTML syntax typically (but not reliably) cause a web page showing an error message to appear. Sometimes things will just inexplicably not work.
• New or modified broken web galleries and web templates will only appear after LR restart.
• If you use manifest.lrweb's AddResources to actually move the files (ie, its purpose: the source and destination differ), then most if not all the resources will not be available when rendering the preview. One workaround is to then explicitly AddResource or AddPage for each file that's actually used in the preview. But this has problems too. I wound up using AddResources with the same source and destination directories.
• It's important that your pages (referenced in e.g. AddPage() or AddGridPages())  NOT be in your source directory for AddResources. They may however both have the same destination directories.
• In Windows, Lightroom 4 appears to use an internet explorer 6 or 7 rendering engine for the internal preview web page, so sadly this needs to be a target of your gallery. Debugging it in IE 7 (or at least a later version running with F12 compatibility mode for IE 7) is helpful to get it working in the preview mode (since I haven't figured out any facilities for debugging the actual preview album within lightroom). Not sure about other versions of LR or on a Mac.
• Correction: It turns out Lightroom 4 is actually using a modified build of the webkit library used by Safari 4.0.3 (released in August 2009) - the user agent string is "Mozilla/5.0 (Windows; U; en-US) AppleWebKit/531.9 (KHTML, like Gecko) AdobeAIR/2.7.1". Which is also seriously old too, but at least you can download it and test with this old Safari version outside lightroom, which I highly recommend if you're having trouble.
• There is a maximum of one level of indirection when referencing variables in galleryinfo.lrweb. E.g., if you want variable A to depend upon the value of another (using the function() syntax) and others to depend upon variable A, it's not possible.
• Variables names like "nonCSS.text.name" are actually twice-nested lua tables. Hyphens are not allowed in such names (are you subtracting?). To workaround this, you can express it as such: nonCSS.text["hyphenated-name"].
• Typical methods of debugging the lightroom preview webpage are unavailable (alerts, console logs, inspecting the dom, etc). live_update.js provides an AgDebugPrint() function, but I have no idea how or if it's possible to see output from this call. But one thing you can do is make a "debug" div and insert things into the content of that div to be displayed on page. Primitive, but helpful - I wish I'd thought of it sooner.
• Also, lightroom puts the preview album in the temporary folder named AgWPGPreview-### (eg, C:\Users\username\AppData\Local\Temp\AgWPGPreview-21). Inspecting the files here can tip you off about what may be going wrong when viewing inside LR (note that the images are not written here, or at least not named as you'd expect).
• Of course, it's useful to export the gallery and use typical browser debugging tools as well.
• If you want to support live update, you need to set the variable supportsLiveUpdate = true in galleryinfo.lrweb. This is unmentioned in all documentation. For debugging it's best to turn off to ensure things update as mentioned above. live_update.js calls methods under the variable 'myCallback', but checking for this variable's existence fails and causes live_update to bonk, even under lightroom preview. That means we can't check, so the code will give errors outside the preview (if testing- it shouldn't be included in the export anyway).
• For every metadata item you'd like to use for images (eg, title or gps location), you need to include them in a perImageSetting. These then need to be provided in the properties, perImage field of the GUI to be actually provided (it would also work if you don't provide a properties block, but then these settings will appear in random order on the GUI). Useful ones include com.adobe.X where X can be: caption, title, location (which is actually sublocation), city, state, country, GPS, and many more (all undocumented). If you name it as perImageSetting.title, it will be available under the image's metadata.title field in LuaPages. A longer list of properties is here.
• Luascript in LuaPages can be really tough to debug. I now do one small change at a time and check for failures by reloading the preview window and checking the generated preview html. One thing to note is that there seems to be only one pass-through by the interpreter and thus functions must be defined before they are used.
• If in your LuaPages you have long sections of html/javascript/etc with no luascript elements, then you may get the error "chunk has too many syntax levels". You can cure it by placing luascript comments e.g., <% --[[ Comment here ]] %> to break up the offending section.
• When making Lightroom UI elements with titles, you can use "\n" to break up long lines in the text. But, if you use the LOC function to localize the text and use other languages, there does not appear to be any way to break up lines. Thus other languages must fit on one line.
• Variables starting with "appearance." have magic powers in that they trigger a request to update the appearance of the webpage in the UI; other variables won't trigger refresh. Lightroom sends such changes to the web page's document.liveUpdate() function to determine how much of the page needs to be regenerated. I found that the return value "invalidateOldHTML" is almost always what I needed, and so I replaced the provided function to always return this (YMMV). Variables starting with "metadata" will be treated as text and remove any potential html tags. It also updates like those with "appearance".

Please share any more that you know about and/or know how to work around. Or if you're stuck and need a hand, perhaps it's something I've encountered. Ask in the comments.

CMake / Visual Studio "the parameter is incorrect" error

If you're trying to run configure on a build project with CMake for the first time on this computer, and while trying to detect the C/C++ compiler features, it bonks inexplicably. You have Visual Studio installed correctly after all, and it builds things without problems both within the GUI and from the command line. Inspecting the logs, you see that the errors occur when CMake is trying to compile an internal file such as CMakeCXXCompilerId.cpp using the '@' feature of the compiler to provide command line parameters via file. The only reported error is from the compiler saying that "the parameter is incorrect".

The problem turned out to be caused by BeyondTrust's Trusted Desktop software (which is corporate security software). Running 'elevated' did not help. But removing Trusted Desktop altogether made the problem go away.

In my case, this occurred on Windows 7, with Visual Studio Professional 2010, 2010sp1 and 2013 Express versions, and CMake 3.3.1. But I suspect it's not very sensitive to these versions. Since I reached the end of the internet trying to figure this out, hopefully this will help someone.

(While I'm at it, I also happen to know that Trusted Desktop is incompatible with recent versions of Oracle VirtualBox. Versions since 4.3.12 have hardened security features, verifying that DLLs are what they say they are. But these features fail under Trusted Desktop, making it impossible to run VMs. The workaround is to use version 4.3.12, which predates these features.)