Gauss commands

Comments begin "/*" end "*/" or begin "@" end "@"

    /* Comments */
    @ Comments @

Change working directory:

    chdir
 
Load data 
The filename can be either a literal or a string. If the filename is in a string variable, then the ^ (caret) operator must precede the name of the string, as in:

    filestr = "data/filename.txt";
    loadm x = ^filestr;
 
Run a script 

    run file_name;
     
Indexing matrices
See help aptech.com.gauss.13.0/doc/LF.6-DataTypes.html
The statement

    y = x[1:3,5:8];
 
Will put the intersection of the first three rows and the 
fifth through eighth columns of x into the matrix y.

Plot

plotXY(datax[.,1], datax[.,2:cols(datax)]) 
plotXY(datay[.,1], datay[.,2:cols(datay)])

Gauss resources

Basic GAUSS workshop 2002
Aptech Tutorial, running a program file

Octave commands

I am trying to run Matlab based test statistics in GNU Octave.

Octave commands 

List variables available in memory

who %this is a comment
whos %provides class details

Change and display working directory

cd directory_name
pwd

Manipulate data structures:

x.a = 1;
x.b = [1, 2; 3, 4];
x.c = "string";

Display the value of a variable

disp(x)

Loop over a list of files

csvfiles = dir("*.csv")
for file= csvfiles'
fprintf(1,'Doing something with %s\n',file.name)
end

Creating character arrays

"In the MATLAB^® computing environment, all variables are arrays, and strings are of type char (character arrays)."

Reading data from an Excel or CSV file

The test statistics I wanted to use loads data from an Excel file but this returned the error :" 'xlsread' undefined ". Reading excel file is provided by the IO package which is not installed by default. The package is available in the Debian repository under "octave-io" , with the description "This package [...] contains functions to [...] read Excel spreadsheet (xlsread) and OpenDocument spreadsheet (odsread)." It is based on Apache POI. Load the package an try to read a file:

pkg load io;
data=xlsread('file_name.xls');

xlsread returns an error "Detected XLS interfaces: None."  This forum post recommends to load the java and windows packages as well. Those packages are not available in the Debian repositories.
I decided to convert the Excel file to csv and use csvread instead.

The script now gives the same output as on a windows machine running Matlab.

Warning: possible Matlab-style short-circuit operator 

Short-circuit boolean operators explains that:

"MATLAB has special behavior that allows the operators ‘&’ and ‘|’ to short-circuit when used in the truth expression for if and while statements. The Octave parser may be instructed to behave in the same manner, but its use is strongly discouraged." [...]

I wonder why it is strongly discouraged. 

"To obtain short-circuit behavior for logical expressions in new programs, you should always use the ‘&&’ and ‘||’ operators."

I replaced "|" by "||" in the code.

Writing test results to a file

Matlab low level file IO, explains how to use fprintf (a vectorised implementation of the c function) to write text data to a file.

Stata commands

Load csv data

cd /home/paul/ 
insheet using filename.csv

tsset and xtset for panel variables

The 2 commands are basically similar (STATA forum discussion). tsset mentions "If you tsset panelvar timevar, you do not need to xtset panelvar timevar to use the xt commands."

xtset country year

View available test results

How to access stored estimation results
 
Stata help: "to see what was returned from an estimation command", type:

ereturn list

Then display results with:

 display e(depvar)

matrix list e(b)

View the source code of a command

viewsource xtset.ado

Why should research organisations release free software?

Research organisations protect software with Intellectual Property (IP) rights. Some of these IP rights authorise the release of source code but some prevent source code release. Within the organisation, a decision maker should ask herself:

• Can the organisation pay a person or a group of person in years to come to maintain that program in the long run?

If the answer is no, read on.

Researchers frequently move to other job positions. Once a researcher has moved to another job, the software codes he/she wrote is likely to sit idle on the organisation's storage drives. When no insider knows how to modify a computer program's code, the value of that program for the organisation will depend on the possibility for outsiders to modify the code.

1. If researchers outside the organisation are not allowed to update the software, it will not be used. IP rights preventing source code modification don't have any value.
2. If on the other hand, the piece of software is released as free and open source software, researchers outside the organisations are likely to update the software once the need arises. IP rights ensure that the first creator's contribution with its organisation's affiliation remains cast in the software's stone. An acknowledgement mentioning the organisation will travel with the piece of software as long as this piece of code is useful. This is likely to attract future project contribution and funding to the host organisation.

Panel cross section dependence tests in STATA and R

STATA example 

Using the Grunfeld investment data:

        use "http://fmwww.bc.edu/ec-p/data/Greene2000/TBL15-1.dta"
        xtset firm year
        xtreg i f c,fe
        xtcsd, pesaran

Output of the xtcsd command only:

Pesaran's test of cross sectional independence =     1.098, Pr = 0.2722

R example

Using the same data: 

library(foreign) # To import STATA .dta files
grunfeld <- font="" read.data="">"http://fmwww.bc.edu/ec-p/data/Greene2000/TBL15-1.dta")

pcdtest(i ~ f + c, data=grunfeld, model = "within", effect = "individual", index = c("firm","year"))

Ouput of the pcdtest command:

    Pesaran CD test for cross-sectional dependence in panels

data:  formula
z = 1.0979, p-value = 0.2722
alternative hypothesis: cross-sectional dependence

Installing STATA on Debian GNU-LINUX

I needed to install STATA to collaborate with a colleague at work. The computer guy gave me the software on a disk, with an installation guide. Here are the commands I entered following those instructions:

Create a directory for Stata

# mkdir /usr/local/stata13
# ln -s /usr/local/stata13/ /usr/local/stata

Install Stata

# cd /usr/local/stata13
# /media/paul/Stata/install

Stata 13 installation
---------------------

  1.  uncompressing files
  2.  extracting files
  3.  setting permissions

Done.  The next step is to run the license installer.  Type:

        ./stinit

If the licensed software is Stata/IC 13, you will be able to run Stata/IC by typing
        xstata              (Run windowed version of Stata/IC)
        stata               (Run console  version of Stata/IC)

Run the license installer

./stinit

There follows some questions about user name and affiliation. "The two lines, jointly, should not be longer than 67 characters."
Then comes the message:

Stata is initialized.
You should now, as superuser, verify that you can enter Stata by typing

        # ./stata
or
    # ./xstata

I added this to my .bashrc so that stata and xstata can be used as a command directly:

 export PATH=$PATH:/usr/local/stata

Both command "stata" and "xstata" work as a normal user now.

There is an error message when running xstata:

'Failed to load module "canberra-gtk-module"'

But this was not a problem at the start.

GNOME application launcher

I added STATA to the GNOME application lancher, by typing "application" in the launcher, then "main menu", "new menu".

R to Stata

I use R most of the time for data analysis and will export csv files to STATA.
R command to export csv files:

write.csv(dtf, "filename.csv", row.names = FALSE, na = ".")

STATA command to import csv files:

insheet using "filename.csv", delimiter(",")

Big scientist

Hilary Mason:

Big data is data that cannot hold on one node.
[...] Some people spread the idea that big data will tell you what to do. [...] This is bullshit, it concerns me that this is starting to get steam outside of the tech community.

Neha Kothari

Linked In Hadoop cluster contains information on all clicks made by users. 1000 employees have access to the cluster and run queries on the data with pig. 

 Women in data science

LIbre office crash at startup

Since I opened a bizarre Excel document. Libre office crashed at startup. Deleting the user profile solves the issue:

rm -r .config/libreoffice/4/user/

Bulk resize images with a shell script and imagemagick

I like to resize images to 1000 pixels wide before sending them per email, it's large enough for most screens and I can send a dozen of pictures without cluttering my friends inbox. A pity that in 2015 there is still no easy way to do this integrated in the Gnome file manager. Maybe there is another way, or maybe I should use a proper image management program. Add a comment on how you resize images. 
 
But it's easy to create a shell script:

#!/bin/sh
mkdir -p small
for i in *.jpg; do
    echo $i;
    convert $i -resize 1000x small/$(basename $i .jpg).jpg;
done

Since I had images ending with .JPG extension in capital letters, I added a second loop:

for i in *.JPG; do
    echo $i;
    convert $i -resize 1000x small/$(basename $i .JPG).jpg;
done

Inspired by this question on stack overflow, and the mkdir complain corrected by looking at this question.

Debian GNU-LINUX communications

Ekiga

Ekiga seems to be the tool to use for audio and video calls on the Gnome desktop. Ekiga is based on a communication framework called Telepathy which enables connection by multiple clients simultaneously.  What communications network should I use?

• the original XMPP server jabber.org
• a French or German XMPP server

Skype

I use Skype to communicate with Windows users and also with some other Ubunutu-GNU/Linux users. Would it be possible to use other tools to communicate with them?

Recordmydesktop

Recordmydesktop can be started from the command line, a flag is required to use the pulse sound server:

recordmydesktop --device pulse

Stop recording with CTRL+C.
To use the GUI, the advanced / Sound / Device setting should also be changed to "pulse". To stop a recording started from the GUI, show the Gnome bottom bar, using the windows + M key, then press on the icon that appears there.

As explained in this Fedora forum, Gnome also has an own desktop recording system which can be started and stopped with CTRL + SHIFT + ALT + R.

 Webex

Some colleagues would like to use Cisco Webex. This requires the Java plugin for Firefox. Based on this blog, I installed it with:

$> apt-get install icedtea-7-plugin
This will install web browser plugin based on OpenJDK 7 and IcedTea. 

But there is no sound. Because Webex uses a proprietary 32 bit sound application, the only fix seems to be to install a 32 bit version of Firefox / Iceweasel.

Blog: Webex support on 64 bit Fedora Linux system explain that 32 bit version of has to be used:

"One means of successfully accessing WebEx from Fedora 12 x86_64 is to use a 32-bit version of Mozilla Firefox with Sun JRE and Adobe Flash 32-bit plugins.

Why 32-bit?

1. Per the JRE download site, the 64-bit version does not have support for java applets or Java Web Start (JWS is required to run another WebEx like app named Elluminate).
2. Per the WebEx System Requirements page only, 32-bit versions of Linux are supported."

Blog: Install 32 bit Firefox and Thuderbird on Debian 64 bit.
Blog: 32 bit applications on 64 bits Linux Mint

This user had an issue with i386 packages and explains how to remove them from his system. How do I remove all i386 architecture packages from my Debian installation?

The Debian multi architecture page explains how to install 32 bit programs (called the i386 architecture) on a 64 bit machine.

Patterns

Sometimes it helps to think in terms of design patterns.

10 years ago, a friend of mine offered me a book on architectural patterns by Christopher Alexander (A Pattern Language: Towns, Buildings, Construction) I remember beautifully simple description of architectural patterns in buildings such as: a place by the window (inside a house) or a "high place" to look around town, or avoid X junctions, keep only T junctions in residential areas.

How about statistical patterns?

Including R data and plots in a Latex document with knitr

knitr default typesetting:

"The chunk option out.width is set to '\\maxwidth' by default if the output format is LaTeX."

Read part of an Excel sheet into an R data.frame

Documentation of the read.xlsx function: 

read.xlsx(file, sheetIndex, sheetName=NULL, rowIndex=NULL,  startRow=NULL, endRow=NULL, colIndex=NULL,  as.data.frame=TRUE, header=TRUE, colClasses=NA,  keepFormulas=FALSE, encoding="unknown", ...)

Returns a data.frame.
Example of use:

dtf <- file="filename,<!-----" read.xlsx="">
    sheetName = sheetname,
    rowIndex = 2:10, 
    colIndex = 5:20)

Building an R package

A package can be seen as a coherent group of functions available for future projects. Building your own package enables you to reuse and share your statistical procedures. Function parameters and examples can be documented with Roxygen to facilitate digging back into the code later on. I created my second package  based on instructions from Hadley. My package structure is composed of the following folders:

• R/   contains R code
• test/  contains tests
• inst/  contains files that will be exported with the package
• docs/  contains .Rmd documents illustrating code development steps and data analysis.
• data/    contains data sets exported with the package
• data-raw/    contains raw dataset and R code to extract raw data from disk and from web resources.

Code

Create a directory containing a package skeleton

devtools::create("packagename")

RStudio has a menu build / configure build tools where devtools package functions and document generation can be linked to keyboard shortcuts:

• document CTRL + SHIFT + D 
• build and reload CTRL + SHIFT + B

devtools::load_all() or Cmd + Shift + L, reloads all code in the package.
Add packages to the list of required packages devtools::use_package("dplyr") devtools::use_package("ggplot2", "suggests")

Data

For data I followed his recommendations in r-pkgs/data.rmd devtools::use_data(mtcars) devtools::use_data_raw() # Creates a data-raw/ folder and add it to .Rbuildignore

Tests

Example of testing for the devtools package

Bash command to build and check a package

Bash command to build a package directory:

R CMD build packagename

Bash command to check a package tarball:

R CMD check packagename_version.tar.gz

 An error log (good luck for understanding it) is visible at:

packagename.Rcheck/00check.log

Generate the documentation and check for documentation specific errors

R CMD Rd2pdf tradeflows --no-clean

The --no-clean option keeps among other files a temporary Latex which can be inspected under:

packagename.Rcheck/packagename-manual.tex

Alternatively the build and check procedure can be run in RStudio as explained above.

Debian GNU-Linux installation on a HP EliteBook

A nice laptop provided by my employer, although the name "EliteBook" sounds like marketing rubbish. My colleague installed Debian 8 on the system, as a dual boot with windows 8.
See also my earlier posts tagged Debian.

Multi boot

The system is a dual boot with Windows. Debian was installed with a traditional - non UEFI - boot loader. Therfore  I have to press F9 each time I want to start Debian. This remains a minor annoyance since I usually restart only every 2 weeks or so (often when the battery has fully drained after a trip). On ordinary days, I put the system in sleep mode in the evening and it awakens in Debian again in the morning.

HP boot instruction didn't work to change the boot order.

 Language

The Debian language was set to French, change back to English.
Reconfigure locales:
As root, run :

dpkg-reconfigure locale

User privileges

as a super user

apt-get install sudo

Add a user to the super user group

adduser username sudo

Log out and log back in for this change to take effect. Now the user should be in the sudo group.

Wireless card

To use the intel non free wireless drivers iwlwifi, in the synapticpackage manager settings / repositories, add "main contrib non-free" to the sections of Jessie packages. Then install firmware-iwlwifi.
In a terminal, restart the modules

 modprobe -r iwlwifi ; modprobe iwlwifi 

Wireless network Eduroam

security: WPA & WPA2 entreprise
Authentication: Tunneled TLS
CA certificate: /etc/ssl/certs/addtrust_external_root.pem
Inner authentication: PAP
Username: user_name@institution.fr
Password: ******

Email and calendar with Evolution

I configured email. I use 2 step authentication and had to generate an application password to load events from my private Google calendar. To add a public Google calendar, I added an "on the web calendar" and inserted the ical link for that public calendar.

Under Edit / Preferences / Contacts / Automatic contacts / I selected to create an automatic contact when sending an email.

R and R studio

Since a previous post on R in Debian, I have changed Debian version from Wheezy to Jessie. Luckily, Jessie contains a recent version of R,I don't need to add the Cran repositories anymore.

Installed Rstudio.

Installed the folowing packages (at a R prompt or in RStudio):

install.packages(c("plyr", "reshape2", "ggplot2"))
install.packages(c("xtable", "markdown", "devtools"))

Install RmySQL, requires mysql development library. At the Shell:

sudo apt-get install libmysqlclient-dev

In R :

install.packages("RMySQL")

Installing R package xlsx requires the java development kit. In the system shell:

sudo apt-get install openjdk-7-jdk

sudo R CMD javareconf

Then in R:

install.packages("xlsx")

Firefox rebranded as iceweasel  + Adblock plus

Installed adblock plus.

Installed icedove to access outlook email backup

See my previous post on accessing outlook email archive from Debian.

Debian dist-upgrade from Wheezy to Jessie

I needed a couple of recent software versions (Lyx version> 2.1 and Latex), and they were not available in the wheezy-backports, therefore I decided to upgrade from Debian Wheezy, the current stable version to Debian Jessie, the testing version. Debian Install FAQ.

This page gives the configuration changes and commands to be used:

1. Edit /etc/apt/sources.list and replace all occurrences of "wheezy" with "jessie".
2. run apt-get update
3. run apt-get upgrade
4. run apt-get dist-upgrade

For several reasons the upgrade process didn't complete properly. After a system reboot, there was no networking, no graphical desktop and even after starting the Gnome desktop by hand with "startx", there were several issues with system settings, such as printer setup.

Networking

I had an issue with networking not working.
According to the debian page on network setup, changes below are not recommended (see below).

I fixed by editing/etc/network/interfaces

# Lines added from odoepner.wordpress.com
auto eth0 

allow-hotplug eth0

iface eth0 inet dhcp

Then ran as root

etc/init.d/networking start

And network was working again.

This page explains how to put this network as a managed interface again: 

But I shouldn't have done this.
According to the debian page on network setup: "Keep configuration of "/etc/network/interfaces" as simple as in the following".
auto lo
iface lo inet loopback

No desktop manager at startup

I realised that there was no desktop manager at startup. I first followed answers to this question and edited  /etc/inittab. But it is not needed, because the boot manager systemd doesn't look into that configuration file.
I spent some time reading about a systemd controversy in Debian and why it doesn't matter so much in the end.

For the moment I start the Gnome desktop from a terminal with the command:

xstart

It looks like not all package had been upgraded.
I ran as root

apt-get upgrade -f

The following packages will be upgraded:
  live-tools

[...]

Preparing to unpack .../live-tools_4.0.2-1_all.deb ...
dpkg-divert: error: rename involves overwriting `/usr/bin/uptime' with
  different file `/usr/bin/uptime.orig.procps', not allowed

From Synaptic, I did a complete removal of the package live-tools.
After that many configuration steps took place. Printer setup utility was working again. And now there only remains a bluetooth issue.

Bluetooth issue

dpkg: error processing package gnome-bluetooth (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 bluez
 bluetooth
 gnome-bluetooth

Lyx upgraded to version 2.1.2

At least Lyx was upgraded to version 2.1.2. I had to "reconfigure LyX with Tools→Reconfigure;".Then I could create documents with Lyx again. And the R integration worked as well.

Upgrade might not have been needed

Afterwards, I realised that an upgrade might not have been needed:

"If you want to install a single package in Debian, you do not need to update the whole system. It can be done with three commands by inserting the repos for Testing and Unstable in /etc/apt/sources.list, fixing the distro you (mostly) wish by setting APT::Default-Release "stable" in /etc/apt/apt.conf.d/local, then doing aptitude install packagae_name/testing -t testing"

Gandi CNAME records to map google sites

Map a google site to a custom domain.

1. Click the More Actions button and select Manage Site.
2. Click the Web Address tab.
3. Enter your custom URL in the Web Address text box, then click the Add button at the top of the page.

GANDI CNAME records to map google site.

Enter your CNAME record using the examples listed in the CNAME record values table and the guidelines below: Name: Enter your custom URL prefix (such as mail or www).
Type: Select CNAME.
Value: Enter ghs.googlehosted.com., making sure to include a trailing dot (.) at the end of the value.

Be aware it may take as long as 72 hours before DNS changes are propagated, depending on the time to live (TTL) that was configured for your records. Until records have been updated worldwide, you will still receive traffic to your old server.


You can check DNS propagation with this tool:

• https://www.whatsmydns.net/ 
•  

In my case it took a few hours before the new CNAME was propagated in the whole world. The first DNS server to be updated were in California, it's as if they use a shorter time to live for CNAME records. 10 hours later there was only one DNS srever in Malaysia that wasn't up to date. 

Data manipulation with dplyr

Dplyr is a package for data manipulation developed by Hadley Wickham and Romain Francois for the R statistical software.

• Introduction to dplyr
• A Tutorial from João Neto (dplyr.Rmd) gives examples of tools for grouped operations: 
• n(): number of observations in the current group
• n_distinct(x): count the number of unique values in x.
• first(x), last(x) and nth(x, n) - these work similarly to x[1], x[length(x)], and x[n] but give you more control of the result if the value isn’t present.
• min(), max(), mean(), sum(), sd(), median(), and IQR()

Non standard evaluation

dplyr uses non standard evaluation. To use standard evaluation a work around has to be found. See Stackoverflow question.

GNU screen for long running server processes

Use screen to keep a long process running on a server after you close the ssh session. I started a screen session with:

    screen -S sessionname

In order to find the screen session later you might want to rename it using sessionname. Or on the first screen invocation use the s flag -S sessionname
I started the R software in this screen session, started a long running process. Then detached the session with:

    CTRL-A-D

I could re-attach the session later with:

    screen -r sessionname

If the session was not detached properly, it might be necessary to detach it and re attach it:

   screen -d -r sessionname

Screen detach before leaving the ssh session. You may want to use the autodetach configuration option.

Log out of your ssh session. Log in back again later:

screen -ls # list sessions
screen -r sessionname # attach a session

There might be solutions to move an already started process to GNU screen but its not straightforward:

"You cannot do this, easily. I'd suggest making it a habit to start screen as the first thing you do after opening a console. However, for your actual problem, there's another thing you could try: after having launched your job from the terminal, background it by typing ctrl-z and then bg. After that, detach the job from it's parent shell; in bash you'd do disown -h %. After that, you can safely close the terminal and the job will continue running."

More tips in this discussion on screen.

SSH tunnel and port forwarding

• Ubuntu Remote_Port_Forwarding

• ssh -C -D 1080 laptop

• http://straightedgelinux.com/blog/howto/socks.html
• ssh -N -D 1080 klaatu@home.linuxserver.com 
• https://wiki.debian.org/SOCKS
•  Assuming your SOCKS server is running on your localhost, listening on port 20000, you could run one of the following commands:
• ssh root@mysshserver -o ProxyCommand='connect-proxy -S localhost:20000 %h %p'
• http://www.debian-administration.org/article/449/SSH_dynamic_port_forwarding_with_SOCKS

• ssh -D 1080 shell.example.org
  tsocks thunderbird

   

  Ubuntu Remote_Port_Forwarding

        ssh -C -D 1080 laptop

    http://straightedgelinux.com/blog/howto/socks.html
        ssh -N -D 1080 klaatu@home.linuxserver.com
    https://wiki.debian.org/SOCKS


http://www.debian-administration.org/article/449/SSH_dynamic_port_forwarding_with_SOCKS

        ssh -D 1080 shell.example.org
        tsocks thunderbird