Installing and using opkg on recent DD-WRT routers

Sometimes you need to extend your dd-wrt router's functionality. As of current Kong builds  (at least 29000 to 34000), this is easy to do using optware and the opkg tool to install packages originally created for Open-Wrt.

But finding instructions to do so is hard. The best that can usually be found are something like this guide, which is largely out of date, and makes it sound much harder than it is. Or this which is likely also out of date and if it were to work would install a bunch of stuff you may not want or need.

All that is necessary in most cases** is:

1. Enable JFFS under Administration->Management of the web control panel. After rebooting a nonvolatile /jffs partition will be created from flash memory.
2. Telnet or ssh into your router (log in as root, with your admin user's password)
3. Now, from the command prompt: make a directory to be used as /opt (where newly installed packages will be stored): mkdir /jffs/opt; mount --bind /jffs/opt
4. Run bootstrap and say yes to install opkg. This is the well concealed secret. Bootstrap is included on Kong builds in order to install opkg for those who want it.
5. Run opkg update to update the repository of available packages
6. To be able to use opkg and installed packages after reboot, you need to add the line mount --bind /jffs/opt /opt to the startup script (under Administration->Management in the web control panel).

Now you have opkg installed and updated and can install packages. You can see the list of available packages by running opkg list and install them by running opkg install pkgname.

Happy opkging! Now you can do fancy stuff like install SSL certificates for your lighttpd web server.

---

Alternatively, rather than using the standard opkg, there's a new project called entware-ng which has more, and more updated applications to run on embedded devices.

To install this (which is in place of the normal opkg), skip step 4. Then follow the instructions here to install on DD-WRT (you could also install /opt on a USB device as explained there, but this is not necessary unless you want many apps or your router has little flash memory). I had luck finding some packages here that weren't available or working in the opkg repository.

* - Kong builds are for routers with ARM/Broadcom chipsets. I don't know about other chipsets or builds, but I wouldn't be surprised if brainslayer or others are building boostrap into their builds as well, so this approach may work for them too.

** - If your router does not have much or any available flash memory to create a jffs partition with, or you want to install many or very large packages, you will need to create a partition for use via USB, and mount it as /opt. Kong's own howto covers this (as well as some more tips on using opkg). The key is that you need a nonvolatile, writable subdirectory /opt, big enough to store what you want to install.

Cadbury creme eggs ingredients

Cadbury's UK (as bought in Ireland):
Milk chocolate: Milk solids 14% minimum. Contains vegetable fats in addition to cocoa butter. Milk chocolate egg with a soft fondant centre (47%). Ingredients: Sugar, milk, glucose syrup, cocoa butter, invert sugar syrup, dried whey (from milk), cocoa mass, vegetable fats (palm, shea), emulsifier (E442), dried egg white, flavourings, colour (paprika extract).


Cadbury's US ingredients (as produced by Hershey's under license from Cadbury):
Milk Chocolate: (sugar, milk, chocolate, cocoa butter, milk fat, nonfat milk, soy lecethin, natural and artificial flavors), sugar, corn syrup, high fructose corn syrup, contains 2% or less of: artificial color (Yellow #6), artificial flavor, calcium chloride, egg whites.

These ingredients are from Easter 2017, so after Cadbury changed from using dairy milk chocolate in the UK.

Tasting them side by side, I was a bit surprised to find a significant difference. The UK eggs were richer and creamier, with a better flavor. The US eggs just tasted too sugary by comparison, and the fondant was more translucent, like sugar syrup.

Preventing Windows OS from sleeping while your python code runs

Do you have a python script that you want to run through to completion, but might take several hours without user interaction?

Might you run on a laptop or other Windows computer that has power management enabled, so that it might go to sleep or hibernate when not being used?

If you do nothing, windows will likely sleep or hibernate before your script can complete.

The following simple piece of code can prevent this problem. When used, it will ask windows not to sleep while the script runs. (In some cases, such as when the battery is running out, Windows will ignore your request.)

class WindowsInhibitor:

    '''Prevent OS sleep/hibernate in windows; code from:

    https://github.com/h3llrais3r/Deluge-PreventSuspendPlus/blob/master/preventsuspendplus/core.py

    API documentation:

    https://msdn.microsoft.com/en-us/library/windows/desktop/aa373208(v=vs.85).aspx'''

    ES_CONTINUOUS = 0x80000000

    ES_SYSTEM_REQUIRED = 0x00000001

    def __init__(self):

        pass

    def inhibit(self):

        import ctypes

        print("Preventing Windows from going to sleep")

        ctypes.windll.kernel32.SetThreadExecutionState(

            WindowsInhibitor.ES_CONTINUOUS | \

            WindowsInhibitor.ES_SYSTEM_REQUIRED)

    def uninhibit(self):

        import ctypes

        print("Allowing Windows to go to sleep")

        ctypes.windll.kernel32.SetThreadExecutionState(

            WindowsInhibitor.ES_CONTINUOUS)

To run it, simply:

import os

osSleep = None

# in Windows, prevent the OS from sleeping while we run

if os.name == 'nt':

    osSleep = WindowsInhibitor()

    osSleep.inhibit()

# do slow stuff

if osSleep:

    osSleep.uninhibit()

It is based on code from here, which also has code for preventing suspension on Linux under GNOME and KDE, should you need that.

ufsd NTFS driver on mount: Fixing 'Unknown Error 1000'

I've previously mentioned using the ufsd driver for NTFS or HFS+, because it is significantly faster at writing than the default ntfs-3g driver provided with in Linux, and it supports writing for HFS+ drives even with journaling enabled. (It is available free for non-commercial use).

But what happens if you have a hard power down with an NTFS drive mounted? Or you encounter corruption for whatever reason?

UFSD may give this error upon mounting it for read-write:

mount: Unknown error 1000

This error is because the "dirty" flag is set on the drive and ufsd won't mount it read/write for fear of corrupting it. In many simple cases, you can correct errors in Linux with ntfsfix, from the ntfsprogs package, and also use ntfsfix to clear the dirty flag.

So if the volume in question is /dev/sdb1, I could do the following as root while the drive is unmounted. The first command repairs simple issues. The second clears the dirty flag:

root ~ # ntfsfix /dev/sdb1   
root ~ # ntfsfix -d /dev/sdb1

IMPORTANT NOTE: If you have extremely valuable data, especially that was being written when the failures occurred, you should be very cautious, because these commands may cause you to lose data.

A better approach may be to load Windows and run chkdsk /f  (to fix file system errors) or chkdsk /r (to detect and mark bad sectors) on the offending drive. Chkdsk is much more sophisticated at detecting and fixing errors than anything available in Linux, although it too may cause you to lose data (and some data loss may be inevitable if power goes off while writing). Here is one approach (using a bootable CD) to using chkdsk even if you don't have Windows installed or handy.

Final note, if this drive is not critical for your machine and you mount it from fstab, you can add the errors=remount-ro flag to the fstab mount line, in order to avoid hanging up your boot when things go wrong.

Reinvigorating an old laptop with fresh Windows 7 and a new SSHD

I've got an older laptop running Windows 7 and it was getting bogged down by cruft after more than 4 years.

Windows update literally took over a week to check for updates last time I checked (pegging one of the processors that whole time). The lazy route would be to buy a new computer, and I looked into that. But I'm not happy with my options (long story), and otherwise, this laptop is great.

So I decided it was time for a hard drive upgrade for added speed and extra space, and re-install of the OS. As a benchmark, it took 3:30 to boot before I started (ouch!), with a 7200rpm hard drive.

After installing a fresh FireCuda 2TB hybrid SSHD drive at a very reasonable price (also in 1TB capacity), it was time to build the most lean Windows I could. (In my machine installing the drive was as easy as literally unscrewing one screw and swapping them out; google for your laptop's "service manual" if you're unsure of the procedure). If you're wondering why not install Windows 10, see below.

Start with the laptop's recovery media -- reinstall as it came, with Windows 7 pre-SP1. In my case, most drivers were not installed, and neither was Internet Explorer. I needed to manually copy over a Firefox installer in order to get going. And I attached an ethernet cable for internet, since ethernet worked out of the box (unlike wifi).

One of the main keys to keeping your install fast is to install as many updates as possible in as few steps as possible. The problem I had with windows update previously was because Windows Update performs a brute-force comparison of available updates against installed updates, and this gets massively slower as the number of installed updates increases. Here is what I did to keep things slim (links are for 64-bit Windows 7):

1. Delete unnecessary and obsolete programs that came with your computer image (using uninstall tool, or removing the installers for things that you won't install).
2. Install Windows 7 Service Pack 1 (KB976932) (if your image was from pre-SP1).
3. Install the latest .NET framework (4.6.2 as of this writing). 
4. Install IE11.
5. Update Windows Update in order to install the rollups below (KB3020369)
6.  * NOTE, I did not do this, but at this point it may be wise to install the "enterprise hotfix rollup" (KB2775511) and associated fixes mentioned at the bottom of this article, in order to avoid even more updates later *
7. Install the rollup including almost all fixes to Windows from SP1 up until May 2016 (KB3125574) - this is "almost SP2"
8. Google to install the latest "Security Monthly Quality Rollup" released either this month or last. For me I installed the January 2017 rollup (KB3212646) (you could install this even if you're doing it later; windows update can take it from here).
9. Now install the latest drivers for your machine from the vendor's website. (If you have a Lenovo laptop like me, install only their System Update tool, which may also require you to install the .NET framework first, and then use it to update all of your drivers at once).
10. Install Microsoft Security Essentials (or another antivirus software)
11. Perform a few cycles of Windows Update and reboot; you'll still have maybe 40-100 security and optional updates.
12. Delete installation files and do a Disk Cleanup (as administrator) to remove backups.
13. Clone the machine from here to be able to recover more quickly next time, starting from this point. I used EASEUS TODO backup; EASEUS disk copy is another potential option.

And you're set. Reinstall the software you use, and copy your data back on. 

For comparison's sake, I improved from booting in 3:30 to booting in 45 seconds, an almost 80% reduction (this is AFTER I re-installed all similar software). Nice!

P.S. - a few less obvious things you might have to do: 1) add a custom Inbound rule to the firewall with scope of your local networks, to allow other subnets to connect to your computer. 2) allow anonymous SMB access (you may also need to add user permissions to "Anonymous login" user), and note that many apps/appliances use SMB v1, so don't disable that

* So why not Windows 10? A few reasons -- one I'm completely happy with Windows 7. Don't fix what ain't broke. Next, I don't like that with windows 10 I'm at the mercy of upgrades from microsoft that I might not want but cannot decline, and which may break things. I also don't want many of the new features. And finally, I don't like the serious lack of privacy in windows 10 -- microsoft sends lots of data from your computer all the time. It's possible to lock it down somewhat, but it's tricky and always shifting (see above updates). Thanks but no thanks. Plus I missed the free upgrade window and definitely don't want to pay.

** If you buy the same great hard drive that I did from the link above, I'll get a small commission at no charge to you. Win-win!

Adjusted ADA scores from 1947-2015

Below you will find updated Americans for Democratic Action (ADA) scores covering the period 1947 to 2015 (the latest available). They are based upon ADA scores of selected congressional vote records independently tabulated by Groseclose, Levitt, and Snyder (1999) (for 1947-1998 originally and later extended by Groseclose to 2008) and Anderson and Habel (2009) (for 1947-2007), and have been updated and reconciled by myself (2008-2015). They are adjusted using the improved procedure from Dr. Groseclose, still based upon his original paper.

The final (adjusted) data are based upon the Anderson collection (mainly because when I started work that was what I had accessible). I correct over 150 mislabeled records that do not match valid ICPSR records as maintained by Keith Poole (as augmented with preliminary records for the 114th congress). Errors were often due to changes in seats, especially to people with similar names, or changes in party of the congressmen.

For the period 1990-2008, I additionally hand-corrected all discrepancies between the Anderson and Groseclose data. In some cases, records were missing from one source or the other. In others the scores were incorrectly transcribed or identifying data were incorrect. In a few cases the source data were ambiguous - in some years a score is provided even if the member served for less than half the eligible votes and occasionally the score does not match the recorded votes. Where possible, I trust the recorded votes over the scores, and omit any congressmen ineligible or deceased for more than half the votes. I also generally counted absences the same as negative votes, even if the congressman served a partial term (since this is ADA's general practice), although for 2007-2015 I omitted anyone who missed more than 6 votes (as I learned this was the practice of Groseclose et al.). There were around 380 such discrepancies and 150+ corrections. Given this work, the period since 1990 can be trusted to be of high accuracy. Earlier data still have discrepancies, and data prior to 1972 show a large number of discrepancies, which I highlight in the excel files provided below. These seem to largely be due to different policies for how to treat votes in years before a numerical score was assigned by ADA. I leave further correction / homogenization for future work.

I calculate the scores using two base years for the adjustment -- 1980 (as used in both the above papers), and 1999 (as used as the Political Quotient [PQ] in Dr. Groseclose's book Left Turn, which he chose because empirically that base year gives the average congressman in the 2000's an adjusted score near 50). If you would prefer a different base year, the code to produce it is below as well.


Downloads:

Adjusted scores with 1980 base year (.xlsx file)
Adjusted scores with 1999 base year (.xlsx file)
Raw data and code to reproduce (including raw output files and parameters) (.zip file)

Citations: The original papers above.