Patterns

Sometimes it helps to think in terms of design patterns.

10 years ago, a friend of mine offered me a book on architectural patterns by Christopher Alexander (A Pattern Language: Towns, Buildings, Construction) I remember beautifully simple description of architectural patterns in buildings such as: a place by the window (inside a house) or a "high place" to look around town, or avoid X junctions, keep only T junctions in residential areas.

How about statistical patterns?

Including R data and plots in a Latex document with knitr

knitr default typesetting:

"The chunk option out.width is set to '\\maxwidth' by default if the output format is LaTeX."

Read part of an Excel sheet into an R data.frame

Documentation of the read.xlsx function: 

read.xlsx(file, sheetIndex, sheetName=NULL, rowIndex=NULL,  startRow=NULL, endRow=NULL, colIndex=NULL,  as.data.frame=TRUE, header=TRUE, colClasses=NA,  keepFormulas=FALSE, encoding="unknown", ...)

Returns a data.frame.
Example of use:

dtf <- file="filename,<!-----" read.xlsx="">
    sheetName = sheetname,
    rowIndex = 2:10, 
    colIndex = 5:20)

Building an R package

A package can be seen as a coherent group of functions available for future projects. Building your own package enables you to reuse and share your statistical procedures. Function parameters and examples can be documented with Roxygen to facilitate digging back into the code later on. I created my second package  based on instructions from Hadley. My package structure is composed of the following folders:

• R/   contains R code
• test/  contains tests
• inst/  contains files that will be exported with the package
• docs/  contains .Rmd documents illustrating code development steps and data analysis.
• data/    contains data sets exported with the package
• data-raw/    contains raw dataset and R code to extract raw data from disk and from web resources.

Code

Create a directory containing a package skeleton

devtools::create("packagename")

RStudio has a menu build / configure build tools where devtools package functions and document generation can be linked to keyboard shortcuts:

• document CTRL + SHIFT + D 
• build and reload CTRL + SHIFT + B

devtools::load_all() or Cmd + Shift + L, reloads all code in the package.
Add packages to the list of required packages devtools::use_package("dplyr") devtools::use_package("ggplot2", "suggests")

Data

For data I followed his recommendations in r-pkgs/data.rmd devtools::use_data(mtcars) devtools::use_data_raw() # Creates a data-raw/ folder and add it to .Rbuildignore

Tests

Example of testing for the devtools package

Bash command to build and check a package

Bash command to build a package directory:

R CMD build packagename

Bash command to check a package tarball:

R CMD check packagename_version.tar.gz

 An error log (good luck for understanding it) is visible at:

packagename.Rcheck/00check.log

Generate the documentation and check for documentation specific errors

R CMD Rd2pdf tradeflows --no-clean

The --no-clean option keeps among other files a temporary Latex which can be inspected under:

packagename.Rcheck/packagename-manual.tex

Alternatively the build and check procedure can be run in RStudio as explained above.

Debian GNU-Linux installation on a HP EliteBook

A nice laptop provided by my employer, although the name "EliteBook" sounds like marketing rubbish. My colleague installed Debian 8 on the system, as a dual boot with windows 8.
See also my earlier posts tagged Debian.

Multi boot

The system is a dual boot with Windows. Debian was installed with a traditional - non UEFI - boot loader. Therfore  I have to press F9 each time I want to start Debian. This remains a minor annoyance since I usually restart only every 2 weeks or so (often when the battery has fully drained after a trip). On ordinary days, I put the system in sleep mode in the evening and it awakens in Debian again in the morning.

HP boot instruction didn't work to change the boot order.

 Language

The Debian language was set to French, change back to English.
Reconfigure locales:
As root, run :

dpkg-reconfigure locale

User privileges

as a super user

apt-get install sudo

Add a user to the super user group

adduser username sudo

Log out and log back in for this change to take effect. Now the user should be in the sudo group.

Wireless card

To use the intel non free wireless drivers iwlwifi, in the synapticpackage manager settings / repositories, add "main contrib non-free" to the sections of Jessie packages. Then install firmware-iwlwifi.
In a terminal, restart the modules

 modprobe -r iwlwifi ; modprobe iwlwifi 

Wireless network Eduroam

security: WPA & WPA2 entreprise
Authentication: Tunneled TLS
CA certificate: /etc/ssl/certs/addtrust_external_root.pem
Inner authentication: PAP
Username: user_name@institution.fr
Password: ******

Email and calendar with Evolution

I configured email. I use 2 step authentication and had to generate an application password to load events from my private Google calendar. To add a public Google calendar, I added an "on the web calendar" and inserted the ical link for that public calendar.

Under Edit / Preferences / Contacts / Automatic contacts / I selected to create an automatic contact when sending an email.

R and R studio

Since a previous post on R in Debian, I have changed Debian version from Wheezy to Jessie. Luckily, Jessie contains a recent version of R,I don't need to add the Cran repositories anymore.

Installed Rstudio.

Installed the folowing packages (at a R prompt or in RStudio):

install.packages(c("plyr", "reshape2", "ggplot2"))
install.packages(c("xtable", "markdown", "devtools"))

Install RmySQL, requires mysql development library. At the Shell:

sudo apt-get install libmysqlclient-dev

In R :

install.packages("RMySQL")

Installing R package xlsx requires the java development kit. In the system shell:

sudo apt-get install openjdk-7-jdk

sudo R CMD javareconf

Then in R:

install.packages("xlsx")

Firefox rebranded as iceweasel  + Adblock plus

Installed adblock plus.

Installed icedove to access outlook email backup

See my previous post on accessing outlook email archive from Debian.

Debian dist-upgrade from Wheezy to Jessie

I needed a couple of recent software versions (Lyx version> 2.1 and Latex), and they were not available in the wheezy-backports, therefore I decided to upgrade from Debian Wheezy, the current stable version to Debian Jessie, the testing version. Debian Install FAQ.

This page gives the configuration changes and commands to be used:

1. Edit /etc/apt/sources.list and replace all occurrences of "wheezy" with "jessie".
2. run apt-get update
3. run apt-get upgrade
4. run apt-get dist-upgrade

For several reasons the upgrade process didn't complete properly. After a system reboot, there was no networking, no graphical desktop and even after starting the Gnome desktop by hand with "startx", there were several issues with system settings, such as printer setup.

Networking

I had an issue with networking not working.
According to the debian page on network setup, changes below are not recommended (see below).

I fixed by editing/etc/network/interfaces

# Lines added from odoepner.wordpress.com
auto eth0 

allow-hotplug eth0

iface eth0 inet dhcp

Then ran as root

etc/init.d/networking start

And network was working again.

This page explains how to put this network as a managed interface again: 

But I shouldn't have done this.
According to the debian page on network setup: "Keep configuration of "/etc/network/interfaces" as simple as in the following".
auto lo
iface lo inet loopback

No desktop manager at startup

I realised that there was no desktop manager at startup. I first followed answers to this question and edited  /etc/inittab. But it is not needed, because the boot manager systemd doesn't look into that configuration file.
I spent some time reading about a systemd controversy in Debian and why it doesn't matter so much in the end.

For the moment I start the Gnome desktop from a terminal with the command:

xstart

It looks like not all package had been upgraded.
I ran as root

apt-get upgrade -f

The following packages will be upgraded:
  live-tools

[...]

Preparing to unpack .../live-tools_4.0.2-1_all.deb ...
dpkg-divert: error: rename involves overwriting `/usr/bin/uptime' with
  different file `/usr/bin/uptime.orig.procps', not allowed

From Synaptic, I did a complete removal of the package live-tools.
After that many configuration steps took place. Printer setup utility was working again. And now there only remains a bluetooth issue.

Bluetooth issue

dpkg: error processing package gnome-bluetooth (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 bluez
 bluetooth
 gnome-bluetooth

Lyx upgraded to version 2.1.2

At least Lyx was upgraded to version 2.1.2. I had to "reconfigure LyX with Tools→Reconfigure;".Then I could create documents with Lyx again. And the R integration worked as well.

Upgrade might not have been needed

Afterwards, I realised that an upgrade might not have been needed:

"If you want to install a single package in Debian, you do not need to update the whole system. It can be done with three commands by inserting the repos for Testing and Unstable in /etc/apt/sources.list, fixing the distro you (mostly) wish by setting APT::Default-Release "stable" in /etc/apt/apt.conf.d/local, then doing aptitude install packagae_name/testing -t testing"

Gandi CNAME records to map google sites

Map a google site to a custom domain.

1. Click the More Actions button and select Manage Site.
2. Click the Web Address tab.
3. Enter your custom URL in the Web Address text box, then click the Add button at the top of the page.

GANDI CNAME records to map google site.

Enter your CNAME record using the examples listed in the CNAME record values table and the guidelines below: Name: Enter your custom URL prefix (such as mail or www).
Type: Select CNAME.
Value: Enter ghs.googlehosted.com., making sure to include a trailing dot (.) at the end of the value.

Be aware it may take as long as 72 hours before DNS changes are propagated, depending on the time to live (TTL) that was configured for your records. Until records have been updated worldwide, you will still receive traffic to your old server.


You can check DNS propagation with this tool:

• https://www.whatsmydns.net/ 
•  

In my case it took a few hours before the new CNAME was propagated in the whole world. The first DNS server to be updated were in California, it's as if they use a shorter time to live for CNAME records. 10 hours later there was only one DNS srever in Malaysia that wasn't up to date. 

Data manipulation with dplyr

Dplyr is a package for data manipulation developed by Hadley Wickham and Romain Francois for the R statistical software.

• Introduction to dplyr
• A Tutorial from João Neto (dplyr.Rmd) gives examples of tools for grouped operations: 
• n(): number of observations in the current group
• n_distinct(x): count the number of unique values in x.
• first(x), last(x) and nth(x, n) - these work similarly to x[1], x[length(x)], and x[n] but give you more control of the result if the value isn’t present.
• min(), max(), mean(), sum(), sd(), median(), and IQR()

Non standard evaluation

dplyr uses non standard evaluation. To use standard evaluation a work around has to be found. See Stackoverflow question.

GNU screen for long running server processes

Use screen to keep a long process running on a server after you close the ssh session. I started a screen session with:

    screen -S sessionname

In order to find the screen session later you might want to rename it using sessionname. Or on the first screen invocation use the s flag -S sessionname
I started the R software in this screen session, started a long running process. Then detached the session with:

    CTRL-A-D

I could re-attach the session later with:

    screen -r sessionname

If the session was not detached properly, it might be necessary to detach it and re attach it:

   screen -d -r sessionname

Screen detach before leaving the ssh session. You may want to use the autodetach configuration option.

Log out of your ssh session. Log in back again later:

screen -ls # list sessions
screen -r sessionname # attach a session

There might be solutions to move an already started process to GNU screen but its not straightforward:

"You cannot do this, easily. I'd suggest making it a habit to start screen as the first thing you do after opening a console. However, for your actual problem, there's another thing you could try: after having launched your job from the terminal, background it by typing ctrl-z and then bg. After that, detach the job from it's parent shell; in bash you'd do disown -h %. After that, you can safely close the terminal and the job will continue running."

More tips in this discussion on screen.

SSH tunnel and port forwarding

• Ubuntu Remote_Port_Forwarding

• ssh -C -D 1080 laptop

• http://straightedgelinux.com/blog/howto/socks.html
• ssh -N -D 1080 klaatu@home.linuxserver.com 
• https://wiki.debian.org/SOCKS
•  Assuming your SOCKS server is running on your localhost, listening on port 20000, you could run one of the following commands:
• ssh root@mysshserver -o ProxyCommand='connect-proxy -S localhost:20000 %h %p'
• http://www.debian-administration.org/article/449/SSH_dynamic_port_forwarding_with_SOCKS

• ssh -D 1080 shell.example.org
  tsocks thunderbird

   

  Ubuntu Remote_Port_Forwarding

        ssh -C -D 1080 laptop

    http://straightedgelinux.com/blog/howto/socks.html
        ssh -N -D 1080 klaatu@home.linuxserver.com
    https://wiki.debian.org/SOCKS


http://www.debian-administration.org/article/449/SSH_dynamic_port_forwarding_with_SOCKS

        ssh -D 1080 shell.example.org
        tsocks thunderbird

R, packages and Rstudio install on Debian wheezy

See also my previous post on Debian GNU-Linux installation on a Lenovo T400.

R install

I used the Synaptic package manager to add the R repository for Debian from a nearby mirror, under : settings / repositories / other software / add.
Add this APT line:

deb http://cran.univ-paris1.fr/bin/linux/debian/ wheezy-cran3/

There was an error:

W: GPG error: http://cran.univ-paris1.fr wheezy-cran3/ Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 06F90DE5381BA480

After looking at several forums, and this stackoverflow question, I installed debian-keyring and added the key with the commands:

gpg --keyserver pgpkeys.mit.edu --recv-key 06F90DE5381BA480
gpg -a --export 06F90DE5381BA480 |sudo apt-key add -

I could then install R version 3 from the synaptic package manager.

Rstudio

I downloaded R-studio and installed it. There was a missing dependency for libjpeg62. I installed that package from Synaptic. Then ran the dpkg command to install rstudio.

dpkg -i rstudio-0.98.507-i386.deb

Tools

Then I installed Git in order to clone my R project from an online repository.

git clone  project_repository_url

Packages

Within Rstudio, I installed a few packages:

install.packages(c("plyr", "reshape2", "ggplot2"))
install.packages(c("xtable", "markdown", "devtools"))

devtools

The devtools packages requires a libcurl dev Debian package. You can install it at the shell prompt:

$ sudo apt-get install libcurl4-gnutls-dev

Back at the R prompt

install.packages("devtools")

Other dependencies might be needed, the RStudio page on devtools recommends installing the Debian package r-base-dev.

dplyr

The dplyr package required the latest version of a Rcpp package. Which was not available on my CRAN mirror. I installed it from source, (based on this message):

install.packages("Rcpp", type = "source")

install.packages("dplyr")

xlsx

The xlsx package installation complained:

configure: error: Cannot compile a simple JNI program. See config.log for details.
Make sure you have Java Development Kit installed and correctly registered in R.
If in doubt, re-run "R CMD javareconf" as root.

Required the latest version of java 7. (inspired by this post). I installed openjdk-7 from the synaptic package manager. Then ran

update-alternatives --config java  # Choose java 7 as the default

R CMD javareconf

Then

install.packages("xlsx") # worked

RMySQL

MySQL client and server are installed on my system.
While installing RMySQL, I struggled with a configuration error:

  could not find the MySQL installation include and/or library
  directories.  Manually specify the location of the MySQL
  libraries and the header files and re-run R CMD INSTALL.

This post has an answer (thanks!):

sudo apt-get install libdbd-mysql  libmysqlclient-dev

That fixes the issue!
I can connect to the database

library(RMySQL)
mychannel <- br="" dbconnect="" host="localhost" user="paul" ysql="">                       password="***", dbname="dbname")

R packages which are better installed from the Debian package manager

Some packages, such as ‘minqa’, ‘SparseM’ and ‘car’ return an error when one tries to install them from the R prompt. The can only be installed from the Debian package manager, where they have names starting with "r-cran": "r-cran-car", "r-cran-sparsem", "r-cran-minqa".

Ready to work!

Google sites fonts

• Monospace fonts
• 10 fonts : Kavoon regular
• 40 fonts: Chubby cheeks

Email backup from Microsoft Outlook to Tunderbird / Icedove under Debian

To read an outlook email archive, .pst file, on a Debian system:

1. On the windows system in Thunderbird, I imported
2. In Debian I accessed it from Icedove, (the rebranded thunderbird)

1. In Windows 

There was an error in the 3.9 GB pst file created in September 2014.

scanpst.exe

As explained on microsoft support, I tried to use the inbox repair tool: scanpst.exe It has found errors in the file I have asked it to repair the file. It copied the whole content in a backup file .bak. It created a log file and it corrected the archive. I could open it in Outllook, under file / data file management. I compacted the archive, it's now 3.7 Gb in size. I closed Outlook and put it in my online backup folder.

Thunderbird

To be able to access this mail archive under Linux.  I followed a Mozilla article which recommends to import outlook mail (including the .pst file) in Thunderbirdwith the menu:

Tools  / Import /   mail

Thunderbird can only open .pst archive files on a windows machine which has outlook installed. Then I closed Thunderbird and copied the file found under.

2. In Debian 

Icedove

I installed icedove from the synaptic package manager. I started it and closed it again. This created a directory in my home folder: .icedove, this is where I'll paste the Thunderbird backup.
Mozilla's explanation on how to restore Thunderbird backup :

1. Locate the backed up profile folder on your hard drive or backup medium (e.g., your USB-stick).
2. Open the profile folder backup (e.g., the xxxxxxxx.default backup).
3. Copy the entire contents of the profile folder backup, such as the mimeTypes.rdf file, prefs.js file, bookmarkbackups folder, etc.
4. Locate and open the new profile folder as explained above and then close icedove (if open).
5. Paste the contents of the backed up profile folder into the new profile folder, overwriting existing files of the same name.
6. Start Icedove.

Online data storage with mono /usr/lib/hubic/hubiC.exe main-loop

List of online content delivery platforms in French

I am testing Hubic, a French online storage service. I installed it on GNU-Linux, it's actually using a windows like executable through mono, the process running is the following:

mono /usr/lib/hubic/hubiC.exe main-loop

To show or sets current synchronization directory:

hubic syncdir

To print some information about account and running operations.

hubic status

To show options

hubic config

To set the option time between Synchronization

hubic config TimeBetweenSynchronization 30

Create Ubuntu unity Launcher for the wine program irfanview

Irfanview was installed using the winetricks program, which requires to load an older version 4.33 found on oldapps.com
Then I created a launcher called  "irfanview.desktop" :

[Desktop Entry]
Type=Application
Name=Irfanview Program Loader
Exec=wine /home/paul/.local/share/wineprefixes/irfanview/drive_c/Program\ Files\ \(x86\)/IrfanView/i_view32.exe
MimeType=application/x-ms-dos-executable;application/x-msi;application/x-ms-shortcut;
Icon=/home/paul/.local/share/applications/irfanview.png
NoDisplay=true
StartupNotify=true

This launcher is a .desktop file based on an askubuntu answer which starts by suggesting to copy an existing .desktop file from :

ls /usr/share/applications/*.desktop

I copied the wine launcher.
I changed the image based on this Ubuntu Handbook post.

Right click menu "open with"

It's also possible to edit context menus. I tried editing: "/home/paul/.local/share/applications/wine-extension-jpe.desktop"

I replaced the line starting with Exec by the following:

Exec=env WINEPREFIX="/home/paul/.local/share/wineprefixes/irfanview" wine start /ProgIDOpen IrfanView %f

I adapted the exec command from this message.
Now when I right click on a JPEG image, open with / Irfanview. Irfanview starts with the image open.

Associate as default application

May require to edit this file:

/usr/share/gnome/applications/defaults.list

 And this page recomments to add a mime type in the folder:

/usr/share/mimelnk/application/

Make word, pdf and html documents with markdown and pandoc

Markdown is a simple text markup language.
Pandoc is a document converter. Pandoc demo and sample command.

Pandoc commands

Convert a markdown file to PDF :

pandoc -o README.pdf README.md

The pandoc man page says: "If  the input or output format is not specified explicitly, pandoc will attempt to guess it from the extensions of the input and output filenames." That's what happens above. However "The input format can be specified using the -r/--read or -f/--from options, the output format using the -w/--write or -t/--to options."

Makefile 

This phsychologist blogs about using a makefile to create beamer presentations.
This researchers providers a make file for pandoc templates.

With this simple make file, I can create Microsoft Word, HTML and PDF documents from the same markdown file:

all: docx pdf html

docx: file.md
        pandoc -o file.docx file.md

pdf: file.md
        pandoc -o file.pdf file.md

html: deliverable.md
        pandoc -o file.html file.md

clean:
        rm -f *.html *.pdf *.docx

To create all documents type

make

To create only a docx type

make docx

To delete all created document type

make clean

Improved makefile with variable

file.pdf : file.md
    pandoc -o file.pdf file.md

%.pdf: %.md
    pandoc -o $@ $<

Guide makefiles:

"Here, we have used the percent (%) character to denote that part of the target and dependency that matches whatever the pattern is used for, and the $< is a special variable (imaging it like $(<)) that means "whatever the depencies are". Another useful variable is $@, which means "the target"."

## Makefile to generate documents based on markdown files
## Inspired by this makefile
## https://github.com/kjhealy/pandoc-templates/blob/master/examples/Makefile
##
## I should use vraibles for filenames
## Command line to converts:

## How to make this using variables?
## No space allowed in file names there could be a replacement but I didn't try
## http://www.cmcrossroads.com/article/gnu-make-meets-file-names-spaces-them

## Markdown extension (e.g. md, markdown, mdown).
MEXT = md
## All markdown files in the working directory
SRC = $(wildcard *.$(MEXT))


DOCX=$(SRC:.md=.docx)
PDFS=$(SRC:.md=.pdf)
HTML=$(SRC:.md=.html)


all: $(PDFS)  $(DOCX)
pdf:    clean $(PDFS)
docx:   clean $(DOCX)
#html:   clean $(HTML)


#scrap : scrap.md
#    pandoc -o scrap.pdf scrap.md


# Separator for these lines need to start with a hard tab, not 4 spaces!
%.pdf: %.md
    pandoc -o $@ $<

%.docx: %.md
    pandoc -o $@ $<

clean:
    rm -f *.html *.pdf *.docx

Debian GNU/Linux

Configuring the system 

Disk partitioning

Recommended Partitioning Scheme: "For new users, personal Debian boxes, home systems, and other single-user setups, a single / partition (plus swap) is probably the easiest, simplest way to go."  [...] " For multi-user systems or systems with lots of disk space, it's best to put /usr, /var, /tmp, and /home each on their own partitions separate from the / partition."

Keyboard layout

I use a laptop with a Finnish keyboard and a docking station with a French keyboard. I chose the "SHIFT+ caps lock" keys to change from one keyboard layout to the other. By default the French keyboad has a comma as decimal separator on the numeric keypad. I changed this under:

 system settings / region and language / layouts / French / Options / Numeric keypad delete key behaviour / 4 level key with dot

Installing programs

Package management with apt-get or aptitude. Aptitude is recommended. Here is a comparison in a forum.

Programming

Install a version tracking system

sudo aptitude install git

Add coloration:

git config --global color.ui true

Install the vim text editor

sudo aptitude install vim

Edit the vim configuration file:

vim ~/.vimrc

In this file add:

syntax on
set tabstop=4
set expandtab
set softtabstop=4
set shiftwidth=4
filetype indent on

au BufNewFile,BufRead *.md set filetype=txt

Non-Free (unfortunately)

Skype on Debian wiki (for 64 architecture) try alternatives Ekiga and Linphone (Linphone use with French ISP free). Load skype for Debian from the website, then:

sudo dpkg --add-architecture i386
sudo apt-get update 

sudo dpkg -i skype-install_file_you_downloaded.deb 
sudo apt-get -f install

Adobe Flash

sudo apt-get install flashplugin-nonfree
sudo update-flashplugin-nonfree --install

Super user

As a super user install the program called "sudo" :

apt-get install sudo

how to-properly-configure-sudoers-file-on-debian-wheezy
I am in the sudo group

adduser paul sudo
  The user `paul' is already a member of `sudo'.

addgroup sudo
  addgroup: The group `sudo' already exists.

But I get the  warning message:

paul is not in the sudoers file.  This incident will be reported.

It seems I have to restart the system.

 Web browser

Why Firefox has been rebranded as iceweasel.

"Debian was initially given permission to use the trademarks, and adopted the Firefox name. However, because the artwork in Firefox had a proprietary copyright license which was not compatible with the Debian Free Software Guidelines, the substituted logo had to remain."

PHP programming

Interactive shell

php -a

Resources

Online PHP manual

VIM commands

Help

• :help  -  vim help
• :help commandname - help on a particular command
• CTRL+] - jump to a highlighted topic
• CTRL+T - jump backwards

Motion

• :help left-right-motion
• j,k move up down
• h,l move left right
• b,w move previous or next word
• ctrl+b, ctrl+d move page up or page down

Undo redo

• u: undo last change (can be repeated to undo preceding commands)
• Ctrl-R: Redo changes which were undone (undo the undos). 
• Compare to '.' to repeat a previous change, at the current cursor position. Ctrl-R will redo a previously undone change, wherever the change occurred. 

Switch between navigation and editing mode

• A - move to the end of the line and switch to editing mode 
• I - switch to editing mode at the current place
• Escape - switch to navigation mode
• alt+h alt+j alt+k alt+l - switch to navigation mode and move
• alt+: - switch to navigation mode and send a command

Search and replace characters

• "/" to search, then n forward and N backwards
• "*" to search for the word under the cursor (you can also create a mapping to search for a selected text)

Vim wiki on search and replace 

• :s/foo/bar/g Find each occurrence of 'foo' (in the current line only), and replace it with 'bar'. 

• :%s/foo/bar/g Find each occurrence of 'foo' (in all lines), and replace it with 'bar'.

• %s/option value=".*"//g remove all beginnings of line. :%s/\option\n/, /g replace all end of line by comma + space. This cleans an html list of species for inclusion in a text.

Markdown

Display a list of first level header in a markdown document (found in quick markdown navigation/TOC)

:g/^# /#

Then enter the line number to jump to that line.

Line numbers

 Display line numbers

:set nu

Disable line numbers

:set nonu

Editing a whole line

• dd to delete a whole line
• yy to copy a whole line
• p to paste the copied or deleted text after the current line or 
• P to paste the copied or deleted text before the current line 

Copy, cut and paste

• Position the cursor where you want to begin cutting.
• Press v (or upper case V if you want to cut whole lines).
• Move the cursor to the end of what you want to cut.
• Press d to cut or y to copy.
• Move to where you would like to paste.
• Press P to paste before the cursor, or p to paste after. 

Indentation

Indentation replaced by spaces, add this to the ~/.vimrc file

set tabstop=4
set expandtab
set softtabstop=4
set shiftwidth=4
filetype indent on 

More details on vim indentation in the python wiki.

Multiple files and windows

• :e filename - edit another file 
• :ls         - show current buffers
• :b 2        - open buffer #2 in this window
• :b filename - open buffer #filename in this window
• :bd         - close the current buffer (! to forget changes)
• :bd filename -close a buffer by name 

Windows

• :sp[lit] filename  - split window and load another file
• :vs[plit] - same but split vertically  
• ctrl-w up arrow - move cursor up a window
• ctrl-w ctrl-w   - move cursor to another window (cycle)
• ctrl-w_         - maximize current window
• ctrl-w=         - make all equal size
• CTRL+z - suspend the process and get back to the shell
• fg - get back to vim

Vimdiff

View differences between file1 and file2 (vim documentation)

vimdiff file1 file2

spell check

Set spell check only in the local buffer:

:setlocal spell spelllang=en_gb  

 Turn spell check off

:set nospell

Mark word as correct, this creates a spell file under /home/user/.vim/spell:

zg

Mark word as incorrect

zw

Plugins for programming languages

• Vim R
• Vim on Python wiki (set softtabstop=4) or your  python development environment (set tabstop=4 and set softtabstop=4)

.vimrc

Text colour.
Add syntax highlight to your .vimrc

syntax enable

How to add a file extension to vim syntax highlight

au BufNewFile,BufRead *.dump set filetype=sql

I used it to display markdown files as text files:

au BufNewFile,BufRead *.md set filetype=txt

Debian GNU-Linux installation on a Lenovo T400

I needed an operating system more stable than Microsoft windows for my daily tasks such as: programming with the R statistical software and writing PDF reports with the Lyx document processor. I read about John MacFarlane (author of pandoc) who is using Debian with the xmonad desktop. This blogger documented why he switched from Ubuntu to Debian. The same blogger also wrote interesting posts at the electronic Frontier Fondation on privacy issues with Ubuntu: on the one side Ubuntu offers an easy way to install full Hard Drive encryption, on the other side, Ubuntu's default desktop search sends search requests over unencripted internet (I should move this additional content to another post).

15 years ago already, I had talked with a friend who was using Debian. I wonder if he still is?

I decided to give Debian a try.

Creating a bootable USB stick

• On a windows machine, I have downloaded the latest Debian live iso (I chose the .iso with the Gnome desktop). I  created a bootable USB stick with the help of Linux Live USB creator. 
• Alternatively, on a Debian machine, for another installation later I choose the Debian Net install version. Then according to Debian Instructions on how to prepare a USB stick, I copied it to the USB with the command :
• cp debian-7.6.0-amd64-netinst.iso /dev/sdb1
• sync
• Unfortunately this net install didn't work for me. Luckily, the live USB install worked fine.

Trying the live version from a USB stick

I introduced the USB key in the laptop,  pressed the "blue thinkvantage" key, entered the BIOS setup and changed the boot order. I placed the USB devices first in the list. The live version seemed to work fine, the Gnome desktop was responsive enough on that machine (Lenovo Thinkpad T400). The external screen was easy to set-up, and network access configured automatically. So I decided to install that Debian system on the hard drive.

Installation

I restarted the laptop with the USB key inserted. I chose the graphical installer. There was a small issue with the fact that the installer was looking for a CD-ROM. But no CD-ROM was available because I was running the install from a USB stick. Based on this blog post, I jumped to the command line interface and typed:

mount /dev/sdb /cdrom 

This command fixed the issue, installation could continue. I partitionned my hard drive to leave 100 GB for the windows partition and the rest (144 GB) for the new GNU-Linux system. Partitioning took a frightening amount of time. During which I thought the system as frozen. But after maybe an hour, installation carried on... I restarted the system. I am now writing this blog post from a fresh Debian system.

Hardware support

Docking station

Hotpluging usb keyboard was working on the laptop itself, but not working on the docking station. A quick fix was to disconnect and reconnect the docking station.
I have connected the docking station to an external larger screen I set this up as my main monitor under system settings / displays.

Different Keyboard layouts

My laptop has a Finnish keyboard, and I am located in France, therefore I use a French keyboard. Under system settings / region and language settings I could add the French keyboard layout. The fact that it's possible to "use the same layout for all windows" is a plus compared to windows. I regularly switch from the Finnish to the French keyboard layout. In windows, I had to change the keyboard layout for all windows one by one.

Network drive

From the Nautilus file manager, I could access windows shared drive using my company's internal network login details.
I have been trying to synchronize this folder using Unison (GUI for rsync). Setting up a local synchronization requires to mount the windows share in the file system. mount.cifs from cifs utils can mount windows shared drives.

Network printer

Under system settings /  printers, the network printer appears. But I didn't manage to connect to the Sharp MX 2600 N.

• MX 2600 N is not in the list of PDD files for sharp printers
• I found a PDD file for MX 2600N on a TU München site.

Wireless card

Show all hardware

lspci

 Network controller: Intel Corporation PRO/Wireless 5100 AGN

Install support for wifi device on Debian

apt-get install firmware-iwlwifi

Reinsert this module to access installed firmware:

# modprobe -r iwlwifi ; modprobe iwlwifi

Wifi works! :-)

Location of software packages

A mirror for France is located at http://ftp.fr.debian.org/debian/.

Install research software 

See also other software install tips in this post on Debian GNU Linux
Lyx editor for Latex

sudo aptitude install lyx

Jabref

sudo aptitude install jabref

Evolution with Microsoft Exchange plugin

My Company uses microsoft exchange on its email servers. An "exchange" plugin can be installed from the Synaptic package manager but this only works for exchange 200 and 2003. This site recommends using the mapi plugin instead which should work for exchange 2007 and 2010. As explained here, my company also uses a different server for the web mail access and the internal connection to the mail server.

To find the internal sever name, I went to the windows version of outlook 2007. Right click on mail box / properties / advanced ... / Microsoft Exchange server.

Mail loads fine now. I can send email too.

Calendar events are not displayed, even though outlook calendar events appear within the desktop calendar (The little calendar that appears when clicking on the desktop clock). This page says that I should upgrade to a more recent version of Evolution. This would require the use of Debian backports. Couldn't get backports to work for now. It said that I already have the most recent version of evolution installed.
I add this line to the 

# Backports repository
deb http://ftp.debian.org/debian wheezy-backports main contrib non-free

Ran this command

sudo apt-get update
sudo apt-get -t wheezy-backports install evolution

Got this message

evolution is already the newest version.

Well it's actually normal because evolution is currently (August 2014) not in the list of Debian backports.

R and Rstudio install on Debian wheezy 7

Instruction below kept here for historical purposes, I'll update these R specific instruction in another blog post.

I used the Synaptic package manager to add the R repository for Debian from a nearby mirror, under : settings / repositories / other software / add.
Add this APT line:

deb http://cran.univ-paris1.fr/bin/linux/debian/ wheezy-cran3/

There was an error:

W: GPG error: http://cran.univ-paris1.fr wheezy-cran3/ Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 06F90DE5381BA480

After looking at several forums, and this stackoverflow question, I installed debian-keyring and added the key with the commands:

gpg --keyserver pgpkeys.mit.edu --recv-key 06F90DE5381BA480
gpg -a --export 06F90DE5381BA480 |sudo apt-key add -

I could then install R version 3 from the synaptic package manager.

I downloaded R-studio and installed it. There was a missing dependency for libjpeg62. I installed that package from Synaptic. Then ran the dpkg command to install rstudio.

dpkg -i rstudio-0.98.507-i386.deb

Then I installed Git in order to clone my R project from an online repository.

git clone  project_repository_url

Within Rstudio, I installed a few packages:

install.packages(c("plyr", "reshape2", "ggplot2"))
install.packages(c("xtable", "markdown", "devtools"))

Ready to work!

Further reading

Wiki on installing Debian on Thinkpad laptops.