Saturday, January 28, 2017

Reinvigorating an old laptop with fresh Windows 7 and a new SSHD

I've got an older laptop running Windows 7 and it was getting bogged down by cruft after more than 4 years.

Windows update literally took over a week to check for updates last time I checked (pegging one of the processors that whole time). The lazy route would be to buy a new computer, and I looked into that. But I'm not happy with my options (long story), and otherwise, this laptop is great.

So I decided it was time for a hard drive upgrade for added speed and extra space, and re-install of the OS. As a benchmark, it took 3:30 to boot before I started (ouch!), with a 7200rpm hard drive.

After installing a fresh FireCuda 2TB hybrid SSHD drive at a very reasonable price (also in 1TB capacity), it was time to build the most lean Windows I could. (In my machine installing the drive was as easy as literally unscrewing one screw and swapping them out; google for your laptop's "service manual" if you're unsure of the procedure). If you're wondering why not install Windows 10, see below.

Start with the laptop's recovery media -- reinstall as it came, with Windows 7 pre-SP1. In my case, most drivers were not installed, and neither was Internet Explorer. I needed to manually copy over a Firefox installer in order to get going. And I attached an ethernet cable for internet, since ethernet worked out of the box (unlike wifi).

One of the main keys to keeping your install fast is to install as many updates as possible in as few steps as possible. The problem I had with windows update previously was because Windows Update performs a brute-force comparison of available updates against installed updates, and this gets massively slower as the number of installed updates increases. Here is what I did to keep things slim (links are for 64-bit Windows 7):
  1. Delete unnecessary and obsolete programs that came with your computer image (using uninstall tool, or removing the installers for things that you won't install).
  2. Install Windows 7 Service Pack 1 (KB976932) (if your image was from pre-SP1).
  3. Install the latest .NET framework (4.6.2 as of this writing). 
  4. Install IE11.
  5. Update Windows Update in order to install the rollups below (KB3020369)
  6.  * NOTE, I did not do this, but at this point it may be wise to install the "enterprise hotfix rollup" (KB2775511) and associated fixes mentioned at the bottom of this article, in order to avoid even more updates later *
  7. Install the rollup including almost all fixes to Windows from SP1 up until May 2016 (KB3125574) - this is "almost SP2"
  8. Google to install the latest "Security Monthly Quality Rollup" released either this month or last. For me I installed the January 2017 rollup (KB3212646) (you could install this even if you're doing it later; windows update can take it from here).
  9. Now install the latest drivers for your machine from the vendor's website. (If you have a Lenovo laptop like me, install only their System Update tool, which may also require you to install the .NET framework first, and then use it to update all of your drivers at once).
  10. Install Microsoft Security Essentials (or another antivirus software)
  11. Perform a few cycles of Windows Update and reboot; you'll still have maybe 40-100 security and optional updates.
  12. Delete installation files and do a Disk Cleanup (as administrator) to remove backups.
  13. Clone the machine from here to be able to recover more quickly next time, starting from this point. I used EASEUS TODO backupEASEUS disk copy is another potential option.
And you're set. Reinstall the software you use, and copy your data back on. 

For comparison's sake, I improved from booting in 3:30 to booting in 45 seconds, an almost 80% reduction (this is AFTER I re-installed all similar software). Nice!


* So why not Windows 10? A few reasons -- one I'm completely happy with Windows 7. Don't fix what ain't broke. Next, I don't like that with windows 10 I'm at the mercy of upgrades from microsoft that I might not want but cannot decline, and which may break things. I also don't want many of the new features. And finally, I don't like the serious lack of privacy in windows 10 -- microsoft sends lots of data from your computer all the time. It's possible to lock it down somewhat, but it's tricky and always shifting (see above updates). Thanks but no thanks. Plus I missed the free upgrade window and definitely don't want to pay.

** If you buy the same great hard drive that I did from the link above, I'll get a small commission at no charge to you. Win-win!

Tuesday, January 10, 2017

Adjusted ADA scores from 1947-2015

Below you will find updated Americans for Democratic Action (ADA) scores covering the period 1947 to 2015 (the latest available). They are based upon ADA scores of selected congressional vote records independently tabulated by Groseclose, Levitt, and Snyder (1999) (for 1947-1998 originally and later extended by Groseclose to 2008) and Anderson and Habel (2009) (for 1947-2007), and have been updated and reconciled by myself (2008-2015). They are adjusted using the improved procedure from Dr. Groseclose, still based upon his original paper.

The final (adjusted) data are based upon the Anderson collection (mainly because when I started work that was what I had accessible). I correct over 150 mislabeled records that do not match valid ICPSR records as maintained by Keith Poole (as augmented with preliminary records for the 114th congress). Errors were often due to changes in seats, especially to people with similar names, or changes in party of the congressmen.

For the period 1990-2008, I additionally hand-corrected all discrepancies between the Anderson and Groseclose data. In some cases, records were missing from one source or the other. In others the scores were incorrectly transcribed or identifying data were incorrect. In a few cases the source data were ambiguous - in some years a score is provided even if the member served for less than half the eligible votes and occasionally the score does not match the recorded votes. Where possible, I trust the recorded votes over the scores, and omit any congressmen ineligible or deceased for more than half the votes. I also generally counted absences the same as negative votes, even if the congressman served a partial term (since this is ADA's general practice), although for 2007-2015 I omitted anyone who missed more than 6 votes (as I learned this was the practice of Groseclose et al.). There were around 380 such discrepancies and 150+ corrections. Given this work, the period since 1990 can be trusted to be of high accuracy. Earlier data still have discrepancies, and data prior to 1972 show a large number of discrepancies, which I highlight in the excel files provided below. These seem to largely be due to different policies for how to treat votes in years before a numerical score was assigned by ADA. I leave further correction / homogenization for future work.

I calculate the scores using two base years for the adjustment -- 1980 (as used in both the above papers), and 1999 (as used as the Political Quotient [PQ] in Dr. Groseclose's book Left Turn, which he chose because empirically that base year gives the average congressman in the 2000's an adjusted score near 50). If you would prefer a different base year, the code to produce it is below as well.


Downloads:

Adjusted scores with 1980 base year (.xlsx file)
Adjusted scores with 1999 base year (.xlsx file)
Raw data and code to reproduce (including raw output files and parameters) (.zip file)

Citations: The original papers above.

Wednesday, November 30, 2016

Simplified Fonts for ggplot2 in R

I struggled for a while to get fonts to work properly with ggplot2 charts in R under Windows. The solution turned out to be easier than it seemed. The "old" way was to use library("extrafonts") which would then scan your entire fonts directory each run (slowly). Then if you got that working and you wanted to export a chart to a PDF, say, you'd need to install Ghostscript and embed the fonts subsequent to generating it. Nowadays R can do it all internally, and with a bit of setup, not have to scan the fonts at all. That's thanks to the showtext and cairo libraries.

You just have to find the filename of the font you want from your fonts directory. (In windows, open control panel -> fonts, then view details and  you may need to add a "Font File name" column". If no name appears, it may be a grouping; open the grouping and do the same.) In my case I wanted to use the Perpetua font, which has the name PER_____.ttf.


install.packages("showtext") # once
install.packages("Cairo" # once
library("showtext")
library("Cairo") # for embedding fonts in PDF; may not need to be loaded here
library("ggplot2")

# add the desired font to the font database (you can add multiple)
font.add("perpetua", "PER_____.ttf")

# the following should only be necessary in windows, and often isn't documented
# for each font you add, do this, mapping the Windows name and type to a font family
# variable (Perpetua in this case) that you will refer to it as.
windowsFonts(Perpetua=windowsFont("TT Perpetua"))

# plot something
# and use perpetua font for text (by default - any text can be customized)
qplot(1:10) +
  + theme(text = element_text(family="Perpetua")) 

# save to file; using Cairo drivers to embed the fonts as needed
ggsave("mychart.eps", width=6.5, height=5.5, device=cairo_ps)
ggsave("mychart.pdf", width=6.5, height=5.5, device=cairo_pdf)

Note that you shouldn't need any special driver to save as an image file (jpg/png/...). I have encountered a few fonts that don't seem to embed correctly and I'm not sure why that is at the moment, but most fonts seem to work fine with this method; they are viewable on screen and in PDFs. This procedure should theoretically work cross-platform (except that the windowsFonts call will not be needed), which is another advantage to this method, although I have not yet tested this.

You can also use google fonts like so, so you don't even have to find one on your system:
font.add.google("Roboto", "roboto")

There is more about the showtext library here. Hope this helps you. Be sure to leave comments if you find any improvements to this method.

Thursday, September 22, 2016

Notes on using git and github

Everybody knows about github and what an amazing resource it is. This post is about using git with github.

Fork what you're interested in on github. In the following user is your username, repository is the repository you forked.
git config --global core.editor <your_favorite_editor>

You may want to modify the EOL settings to match your preferences and platform.

git clone http://github.com/user/repository.git
cd repository
git config user.name "user"
git config user.email "user@users.noreply.github.com"
git remote add upstream https://github.com/originaluser/repository.git

To integrate new changes from the upstream repo into yours:

git fetch upstream
git rebase -i upstream/master

If there are redundant commits, 'squash' them in the first "interactive" message. The -i is important; otherwise you'll get stuck with a bunch of redundant commits.

If there are any conflicts (changes to the same general area of code, even if just adjacent, or even if equivalent), you will be dropped to the command prompt commit by commit. Edit the conflicting file as it should be, removing any >>>>> or <<<<<. When satisfied, git add thefile and git rebase --continue. They will be applied in a new commit.

Similarly if you want to create a new branch for your own use:

git checkout -b branch

To switch branches at any time (without uncommitted changes), just omit the -b. If you want to instead create the branch at a previous commit, add that branch hash at the end.

Be sure to push any changes to the master branch to github before changing to other local branches. Then pull from github before rebasing.

Then you can rebase as above with the other branch. But if the other branch is pushed to github, it is dangerous to push if anyone else is using it. If you're sure they're not, you can git push --force while on that branch (if git config --global push.default simple). If there were any other users there, after pulling, they'd have to blow away their unpushed local commits with git reset --hard origin/branch.

Here is more on rebasing and merging: https://www.atlassian.com/git/tutorials/merging-vs-rebasing/workflow-walkthrough

Or you can move just one commit to the current branch by using:
git cherry-pick <hashcode>

If you need to clean things up: http://stackoverflow.com/questions/5916329/cleanup-git-master-branch-and-move-some-commit-to-new-branch

If you need to "undo" a change made on github, first pull to update your local repo. Then:
git reset --soft HEAD^, and 
git push origin +branchName (see caveats). 

About reverting, resetting etc, see: https://www.atlassian.com/git/tutorials/resetting-checking-out-and-reverting

To update an existing remote branch from a local branch (which is currently checked out):
git push origin local_branch_name:remote_branch_name
or if the branch names match, do this and it will work in the future too:
git push --set-upstream origin local_branch_name

All in all my experience is that git is vastly inferior to mercurial (hg); git is far more finicky, harder to use and more prone to ugliness, and plus mercurial has nice GUIs from TortoiseHg. All in all, git feels like an advanced patch manager that has morphed into a version control system while mercurial feels like an advanced version control system. But alas, the linux kernel uses git and thus we have github and the rest is history. But I still use mercurial whenever I have a choice.

Saturday, June 11, 2016

Detecting CSS position: sticky support

If you're using modernizr for other things, by all means use that. If you just need a simple check to see if the current browser supports position: sticky, insert this javascript code:

var positionStickySupport = function() {
 var el = document.createElement('a'),
     mStyle = el.style;
 mStyle.cssText = "position:sticky;position:-webkit-sticky;position:-ms-sticky;";
 return mStyle.position.indexOf('sticky')!==-1;
}();

positionStickySupport will be true if it's supported. You should use the following CSS:


.myElement {
 position: -webkit-sticky;
 position: -ms-sticky;
 position: sticky;
 top: 0px;
}

Of course you can set top however you want. If you're wondering, the -ms-sticky is for future-proofing. As of this writing, Microsoft plans to support the feature in Edge, but it's not clear whether they'll use a prefix or not. Safari uses -webkit. Opera has it implemented in pre-release form without a prefix, and Firefox already supports it without one.

It would have been possible to use the CSS.supports() functionality, but Safari 6-8 support position: sticky but not CSS.supports().

Here is the one-liner version:

var positionStickySupport = function() {var el=document.createElement('a'),mStyle=el.style;mStyle.cssText="position:sticky;position:-webkit-sticky;position:-ms-sticky;";return mStyle.position.indexOf('sticky')!==-1;}();

Thursday, May 12, 2016

OpenVPN Server Setup Made Simple


For those not familiar with it, OpenVPN is probably the best and most secure VPN protocol out there at this time, with clients available for every mainstream platform. Unlike most other VPN protocols, it uses shared keys rather than passwords for security, which does make initial configuration take slightly longer.

But the real problem used to be that the process of setting up an OpenVPN server and generating appropriate keys was long, tedious, complicated, and error prone.

Those days are gone. There are now scripts that do all the hard work of setting it up and configuring users and keys for you. Answering a few simple questions, in a couple minutes you can have it up and running on your Linux server, and using all the current best practices to boot.

There are two use cases, and there are scripts for each, supporting at least Debian/Ubuntu/CentOS, which are both based upon the earlier work of Nyr:
  1. A multi-user, secure OpenVPN server, with quasi-anonymous usage.
  2. An OpenVPN server for personal use, supporting 3 simultaneous connections and potentially older clients.

In case one, use Angristan's script. To get it, run:
wget --no-check-certificate http://bit.ly/ovpn-install -O openvpn-install.sh
In case two, use jtbr's script, which is a lightly modified version of Angristan's.
wget --no-check-certificate http://bit.ly/openvpn-install -O openvpn-install.sh
You can click through to the link to see the extra security measures being taken by these scripts.
For either case, now simply run it as root after adding execute permissions:
chmod +x openvpn-install.sh
sudo ./openvpn-install.sh
For case two, if you want to change the number of simultaneous clients allowed, simply change the line max-clients 3 to the appropriate number before running, or remove it altogether for no limit.
The first run will set it up, and add the first client key (.ovpn file, which is placed in your home directory). Subsequent runs allow adding or removing clients or uninstalling. The generated client keys need to be (securely!) copied on to the clients' devices and imported into their OpenVPN clients. Once it's on their computer, this is usually drag and drop. For iPad/iPhone, I found the easiest option to be to use iTunes File Sharing to save to the OpenVPN Connect app. Then they can connect at will.
You're all set!

These scripts enable clients to access the internet and appear as if they are coming from the server's IP, a typical road-warrior setup; and depending upon the location of your server, to allow clients to access the server's local network. To allow VPN clients to connect to each other, or to allow access to the clients' network, then you'll need some additional routing configurationmore help here.

Especially if you're using a VPN for anonymity, you should also put some effort into ensuring your clients don't have any IP leaks (where your IP is discoverable by sites you access). A good guide to IP leaks and how to fix them is here; for a quick check, try here.

Wednesday, May 11, 2016

Touch-enabled swipeable scroll bars

Looking for an elegant, functional scrollbar solution for the web?

If you google around, you'll find TouchSwipe and various other Javascript-based solutions. These, while impressive, are trying too hard. They all have issues, either with handling links within the scrollable area, or with stutters in the animation, or with unnatural (on iOS anyway) non-bouncy ending of the scroll. I had several issues I could never overcome, not to mention their large code size and complexity.

You want something that works on touch devices and non-touch devices and is flexible and easy. I found the solution to be CSS-based, and this is it. It works on any modern browser and IE 9+.



The key is that you need an enclosing container (scrollWrapper) with overflow hidden, and an internal container (scrollableArea) that scrolls within it, and is allowed to scroll using the native facilities.

Normally, on a desktop, using the native scroll facilities would mean that you'd get an ugly scrollbar, so you need to hide that, and this can be done in CSS. But you need to replace that functionality for non-touch users. That means adding buttons on the ends that can move the scrollable area (and could also mean enabling the mouse wheel). The buttons call javascript functionw that simply update the scroll position and allows CSS transitions to occur. On the codepen below I even include a tiny javascript plugin that enables scrolling horizontally with a mousewheel (vertically would be automatic).

The result is something that's totally natural and totally flexible, and actually much simpler than the javascript solutions.

You can see a demo of the resulting scrollbar here

The HTML structure is trivial:

<div class="scrollWrapper">
  <a class="scrollBtn prev">&lt;</a>
  <div class="scrollableArea">
    <!-- stuff to scroll here -->  
  </div>
  <a class="scrollBtn next">&gt;</a>
</div>

The key part is this simple CSS code:
div.scrollWrapper {
 width: 96%;
 height: 91px;
 margin-left: 2%;
 position: relative;
 overflow: hidden;
 white-space: nowrap;
}
div.scrollableArea {
 position: relative;
 width: 100%;
 height: 125%; /* crop scrollbar (if applicable) outside scrollWrapper, while maintaining scrollability */
 white-space: nowrap;
 overflow-x: scroll;
 overflow-y: hidden;
 -webkit-overflow-scrolling:touch;
}

Note that the height is 91 in order to fit 81px-high divs with 5px margin on top and bottom. The scrollableArea height needs to be about 20px higher than the scrollWrapper to ensure that the desktop scrollbar will be cropped off.

Sample CSS for the scroll buttons:
.scrollBtn {
  display: inline-block;
  position: absolute;
  margin-top: 5px;
  margin-bottom: 5px;
  top: 0px;
  height: 81px;
  width: 30px;
  line-height: 81px;
  text-align: center;
  vertical-align: middle;
  background-color: #444;
  opacity: 0.5;
  text-decoration: none;
  z-index: 100;
  font-size: 30px;
  font-weight: 700;
  outline: none;
  cursor: pointer;
}
.scrollBtn:hover {
  opacity: 0.7;
  transition: 0.3s;
}
.scrollBtn.prev {
  left: 0px;
}
.scrollBtn.next {
  right: 0px;
}


Finally, here's javascript to handle the buttons and (optionally) enable horizontal scrolling (using jQuery):
$('a.scrollBtn.prev').click(function(e) {
 var scroller = $(".scrollableArea");
 scroller.animate({scrollLeft: scroller.scrollLeft() - (scroller.innerWidth() - 91)});
 e.preventDefault();
});

$('a.scrollBtn.next').click(function(e) {
 var scroller = $(".scrollableArea");
 scroller.animate({scrollLeft: scroller.scrollLeft() + (scroller.innerWidth() - 91)});
 e.preventDefault();
});

/** 
  * Enable scrolling an element horizontally using up/down mousewheel events
  * (when over the element). Amount is the number of pixels to move per
  * wheel turn (default 120) 
  **/
$.fn.hScroll = function (amount) {
 amount = amount || 120;
 $(this).bind("DOMMouseScroll mousewheel", function (event) {
  var origEvent = event.originalEvent, direction = origEvent.detail ? origEvent.detail * -amount : origEvent.wheelDelta, position = $(this).scrollLeft();
  position += direction > 0 ? -amount : amount;
  $(this).scrollLeft(position);
  event.preventDefault();
 })
};
$('.scrollableArea').hScroll(70);