Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Wednesday, May 25, 2016

Jmulti over wine on Linux

Jmulti is a Java based time series analysis software. Unfortunately the Linux version is not maintained any more. I installed wine in the hope to use Jmulti on top of wine. I downloaded the executable “jmultiVM_win-4.24.exe” which is said to contain the Java virtual machine. Then I ran “wine jmultiVM_win-4.24.exe”. Install complains about “InvokeShellLinker failed to extract icon from L"C:\\jmulti4\\jmulti.exe" “ but installation is successful nevertheless.

To start jmulti go to the newly created directory
cd ~/.wine/drive_c/jmulti4/  
then:
 wine jmulti.exe

Black screen issue

Jmulti starts but I have black areas in menu. Rmathew explains how to remove DirectX-based acceleration for Java 2D completely.  Looking for a registry key like:
HKEY_CURRENT_USER\Software\JavaSoft\Java2D\1.5.0_11
and setting the value of "DXAcceleration" to "0" fixes it.

Monday, February 08, 2016

FreeRDP

FreeRDP enables a connection to a windows machine from Debian GNU-Linux. freerdp-x11 can be installed from the Synaptic package manager. The program is called xfreerdp.


Usage example:
xfreerdp /u:**user** /p:**password** /v:**IP** /drive:data,/home/paul/R/forestproductsdemand/data-end
Connect in fullscreen mode using a stored configuration connection.rdp and the password Pwd123! :
xfreerdp connection.rdp /p:Pwd123! /f
Connect to host rdp.contoso.com with user CONTOSO\\JohnDoe and password Pwd123!
xfreerdp /u:CONTOSO\\JohnDoe /p:Pwd123! /v:rdp.contoso.com

Connect to host 192.168.1.100 on port 4489 with user JohnDoe, password Pwd123!. The screen width is set to 1366 and the height to 768. See also option /drive below.
xfreerdp /u:JohnDoe /p:Pwd123! /w:1366 /h:768 /v:192.168.1.100:

The screen shot below shows a windows session accessed from within the Gnome desktop. A folder is shared an can be visible in both the windows and Gnome file explorers.


Options

Command line options
/drive:nameofdrive,localfolder : Redirect localfolder as nameofdrive on the server machine

Window size

Pressing CTRL+ALT+ENTER will switch from windowed to full screen and vice-versa. This is sufficient for me to see the full screen on a Gnome Desktop. But there is no option for scroll bars if the distant screen is to large to appear on screen.

Multiple RDP connections to a windows machine

I'm working with a colleague on the same machine. She has long running processes on the machine and I need to use specific software. According to this Microsoft forum, multiple RDP connection is not possible on windows.

Change keyboard layout

List available keyboard layout
xfreerdp /kbd-list
I didn't manage to enter the keyboard option. I asked a question on SuperUser.

Automation for a thin client

Command script to open xfreerdp and reopen it when it closes suggests this script: 
while (true); do xfreerdp -f xxx.xxx.xxx.xxx ; sleep 2; done

Monday, January 25, 2016

A year on Debian

Inspired by other users seen on the Linux setup. I switched my work computer to Debian GNU/Linux in January 2015. I used the Jessie version which was "unstable" at the time. It became "stable" in april 2015 and I think the adjective is deserved. I'm happy.


I mostly use the R statistical software, the iceweasel web browser, the Lyx editor, the reference manager Jabref and the Evolution mail and calendar program. I learned a lot by using git (file revision system) and databases through the command line. I sometimes use vim to edit text files and I use the RStudio editor in Vim mode. I also use the Libre office Calc, Writer and Draw software.

As long as you stick with what is in the huge Debian package repository, software updates are easy. If you need programs updated very recently (in the last months), installation can become tricky. Although it mostly involves adding new software repositories or installing .deb files directly. For example I managed to install the proprietary statistical software STATA.

Friday, December 18, 2015

Using SSH keys to access remote servers and git repositories

An SSH key can be used to access a virtual private server or a remote git repository without the need to enter a password every time. By sharing your public key with the remote server, your compter is authenticated as a trusted access point.


Creating SSH keys 

In Debian GNU Linux, using the Gnome desktop, you can create a private and public SSH key pair with for example the seahorse key manager. Under File / New / Secure Shell Key.

Created keys will be visible under ~/.ssh/ the private key is called id_rsa and the public key id_rsa.pub. You should only share the public key.

At the command line, you can create keys with
ssh-keygen -t rsa -C "your_email@example.com"

Virtual Private Server

I bought a virtual private server with Debian pre-installed. A public key can be added in the file ~/.ssh/authorized_keys. When connected to the server, edit the file:
vim ~/.ssh/authorized_keys
You might need to change access permission to that file as explained in this gist.

Bitbucket

Your public key can be added to your bitbucket account under manage account / security / SSH keys. This page explains how to use the SSH protocol with Bitbucket in more details.

Github

Your public key can be added to your Github account under profile / settings / SSH key. More details on how to generate and use SSH keys for github.

Then at the top of your Github repository you should see the "clone URL". Copy the SSH URL, in the form: git@github.com:yourusername/yourrepository.git
Add it as a remote origin:
git remote add origin git@github.com:yourusername/yourrepository.git
If there was already a remote repository you might need to delete it first with git remote remove origin.



The push and set the remote repository as an upstream repository:
git push --set-upstream origin master
Subsequent push can be simply made with
git push

See also

See also my other blog posts on the bash shell commands and on git commands.

Wednesday, April 08, 2015

Ipython notebook and R

I chose to use python 3. Several of the shell commands below have a "3" suffix in Debian testing as of April 2015: ipython3, pip3.

Install programs

I installed ipython-3-notebook (in Debian Jessie) from the synaptic package manager.

In order to install the R module, I installed PIP for python 3 in the synaptic package manager. PIP is the Python Package Index, a module installation tool. Then I used pip3 to install rpy2
sudo pip3 install rpy2
There is a blog post on how to avoid using sudo to install pip modules.

Install statsmodel, a module for statistical modelling and econometrics in python. Maybe I should have installed python-statsmodels as a Debian package instead? But I it seems to be linked to python 2.x instead of python 3 (it had a dependency on python 2.7-dev). Therefore I installed statsmodels with pip3, using the --user flag mentioned above to install is as a user only module.
pip3 install --user statsmodels
The installation took several minutes on my system. It seemed to be installing a number of dependencies. Many warnings about variables defined but not used were returned but the installation kept running. The final message was:
Successfully installed statsmodels numpy scipy pandas patsy python-dateutil pytz
Cleaning up...

Starting the Ipython notebook

Move to a directory where the notebooks will be stored, start a ipython notebook kernel
cd python
ipython3 notebook

Shortcuts

See also the Ipython Notebook shortcuts. Useful shorcuts are ESCAPE to go in navigation mode, ENTER, to enter edit mode. It seems one can use vim navigation keys j and k to move up and down cells. Pressing the "d" key twice deletes a cell. CTRL+ENTER run cell in place, SHIFT+ENTER to run the cell and jump to the next one, and ALT+ENTER to run the cell and insert a new cell below. 

Run R commands in the Ipython notebook


Load an ipython extension that deals with R commands
%load_ext rpy2.ipython
 Display a standard R dataset
%R head(cars)
%R plot(cars)
Use data from the python statsmodels module based on this page.
import statsmodels.datasets as sd
data = sd.longley.load_pandas()
Print column names of the dataset
print(data.endog_name)
print(data.exog_name)
Print a dataset as an html table by simply giving its name in the cell. For example this data frame contains exogenous variables:
data.exog
Python can pass variables to R with the following command:
totemp = data.endog
gnp = data.exog['GNP']
%R -i totemp,gnp
Estimate a linear model with R
%%R
fit <- br="" gnp="" least-squares="" lm="" nbsp="" regression="" totemp="">print(fit$coefficients)  # Display the coefficients of the fit.
plot(gnp, totemp)  # Plot the data points.
abline(fit)  # And plot the linear regression.
Plot the datapoints and linear regression with the ggplot2 package
%%R
library(ggplot2)
ggplot(data = NULL, aes(x =gnp, y = totemp)) +
    geom_point() +
    geom_abline( aes(intercept=coef(fit)[1], slope=coef(fit)[2]))

Wednesday, April 01, 2015

Virtual Machine setup for development purposes


Creating a Virtual machine with Vagrant and PuPHeT.


According to those 2013 stack overflow questions, there were many reasons not to develop in a VM, unless one had to specifically develop for several OS:
But in the same year, the PhPHet developer explained why he thinks that one has to develop in a virtual machine.

Running a VM 

I followed the vagrant instructions to install a basic VM.
vagrant init hashicorp/precise32 vagrant up
"The guest machine entered an invalid state while waiting for it
to boot. " [...] "If the provider you're using has a GUI that comes with it, it is often helpful to open that and watch the machine"
I started the virtual machine in virtual box, an error message came up: 
"VT-x is disabled in the BIOS. (VERR_VMX_MSR_VMXON_DISABLED)."
Under Machine / Settings/ System / Acceleration, I disabled the Hardware virtualisation. The VM could then start. This works for 32 bits systems. Unfortunately 64 bit systems require hardware virtualisation, this means I cannot change this setting for 64 systems. I'll have to enable VT-x in the BIOS later on.

After I installed Virtual box, my mouse was rendered invisible. This may be due to the fact that the mouse was captured and that I didn't know the host capture key (default to the right Ctrl key) to free the mouse from the virtual machine's window.

Connecting to the virtual machine

Connecting from the virtual box GUI. The default user is "vagrant" and password "vagrant".

Connecting with SSH into the machine from a command prompt:
 vagrant ssh

 

Shared folder

A folder can be share with the host operating system. In virtual Box settings for the machine, under shared folder, create a machine folder and set it to auto-mount in the guest operating system.

Other tools



Messages by the vagrant creator

Tao of hashicorp
Comparing Filesystem Performance in Virtual Machines Automation Obsessed

Thursday, February 26, 2015

Installing STATA on Debian GNU-LINUX


I needed to install STATA to collaborate with a colleague at work. The computer guy gave me the software on a disk, with an installation guide. Here are the commands I entered following those instructions:

Create a directory for Stata
# mkdir /usr/local/stata13
# ln -s /usr/local/stata13/ /usr/local/stata
Install Stata
# cd /usr/local/stata13
# /media/paul/Stata/install
Stata 13 installation
---------------------

  1.  uncompressing files
  2.  extracting files
  3.  setting permissions

Done.  The next step is to run the license installer.  Type:

        ./stinit
If the licensed software is Stata/IC 13, you will be able to run Stata/IC by typing
        xstata              (Run windowed version of Stata/IC)
        stata               (Run console  version of Stata/IC)

Run the license installer
./stinit
There follows some questions about user name and affiliation. "The two lines, jointly, should not be longer than 67 characters."
Then comes the message:
Stata is initialized.
You should now, as superuser, verify that you can enter Stata by typing

        # ./stata
or
    # ./xstata

I added this to my .bashrc so that stata and xstata can be used as a command directly:
 export PATH=$PATH:/usr/local/stata

Both command "stata" and "xstata" work as a normal user now.

There is an error message when running xstata:
'Failed to load module "canberra-gtk-module"'
But this was not a problem at the start.

GNOME application launcher


I added STATA to the GNOME application lancher, by typing "application" in the launcher, then "main menu", "new menu".

R to Stata

I use R most of the time for data analysis and will export csv files to STATA.
R command to export csv files:
write.csv(dtf, "filename.csv", row.names = FALSE, na = ".")
STATA command to import csv files:
insheet using "filename.csv", delimiter(",")


Monday, February 09, 2015

Bulk resize images with a shell script and imagemagick

I like to resize images to 1000 pixels wide before sending them per email, it's large enough for most screens and I can send a dozen of pictures without cluttering my friends inbox. A pity that in 2015 there is still no easy way to do this integrated in the Gnome file manager. Maybe there is another way, or maybe I should use a proper image management program. Add a comment on how you resize images. 
 
But it's easy to create a shell script:
#!/bin/sh
mkdir -p small
for i in *.jpg; do
    echo $i;
    convert $i -resize 1000x small/$(basename $i .jpg).jpg;
done
Since I had images ending with .JPG extension in capital letters, I added a second loop:
for i in *.JPG; do
    echo $i;
    convert $i -resize 1000x small/$(basename $i .JPG).jpg;
done

Inspired by this question on stack overflow, and the mkdir complain corrected by looking at this question.

Tuesday, January 06, 2015

Debian dist-upgrade from Wheezy to Jessie

I needed a couple of recent software versions (Lyx version> 2.1 and Latex), and they were not available in the wheezy-backports, therefore I decided to upgrade from Debian Wheezy, the current stable version to Debian Jessie, the testing version. Debian Install FAQ.

This page gives the configuration changes and commands to be used:

  1. Edit /etc/apt/sources.list and replace all occurrences of "wheezy" with "jessie".
  2. run apt-get update
  3. run apt-get upgrade
  4. run apt-get dist-upgrade
For several reasons the upgrade process didn't complete properly. After a system reboot, there was no networking, no graphical desktop and even after starting the Gnome desktop by hand with "startx", there were several issues with system settings, such as printer setup.

Networking

I had an issue with networking not working.
According to the debian page on network setup, changes below are not recommended (see below).

I fixed by editing/etc/network/interfaces
# Lines added from odoepner.wordpress.com
auto eth0 
allow-hotplug eth0
iface eth0 inet dhcp

Then ran as root
etc/init.d/networking start
And network was working again.

This page explains how to put this network as a managed interface again: 

But I shouldn't have done this.
According to the debian page on network setup: "Keep configuration of "/etc/network/interfaces" as simple as in the following".
auto lo
iface lo inet loopback

No desktop manager at startup

I realised that there was no desktop manager at startup. I first followed answers to this question and edited  /etc/inittab. But it is not needed, because the boot manager systemd doesn't look into that configuration file.
I spent some time reading about a systemd controversy in Debian and why it doesn't matter so much in the end.

For the moment I start the Gnome desktop from a terminal with the command:
xstart

It looks like not all package had been upgraded.
I ran as root
apt-get upgrade -f
The following packages will be upgraded:
  live-tools
[...]
Preparing to unpack .../live-tools_4.0.2-1_all.deb ...
dpkg-divert: error: rename involves overwriting `/usr/bin/uptime' with
  different file `/usr/bin/uptime.orig.procps', not allowed
From Synaptic, I did a complete removal of the package live-tools.
After that many configuration steps took place. Printer setup utility was working again. And now there only remains a bluetooth issue.

Bluetooth issue

dpkg: error processing package gnome-bluetooth (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 bluez
 bluetooth
 gnome-bluetooth

Lyx upgraded to version 2.1.2

At least Lyx was upgraded to version 2.1.2. I had to "reconfigure LyX with Tools→Reconfigure;".Then I could create documents with Lyx again. And the R integration worked as well.

Upgrade might not have been needed

Afterwards, I realised that an upgrade might not have been needed:
"If you want to install a single package in Debian, you do not need to update the whole system. It can be done with three commands by inserting the repos for Testing and Unstable in /etc/apt/sources.list, fixing the distro you (mostly) wish by setting APT::Default-Release "stable" in /etc/apt/apt.conf.d/local, then doing aptitude install packagae_name/testing -t testing"

Thursday, November 13, 2014

GNU screen for long running server processes

Use screen to keep a long process running on a server after you close the ssh session. I started a screen session with:
    screen -S sessionname
In order to find the screen session later you might want to rename it using sessionname. Or on the first screen invocation use the s flag -S sessionname
I started the R software in this screen session, started a long running process. Then detached the session with:
    CTRL-A-D
I could re-attach the session later with:
    screen -r sessionname
If the session was not detached properly, it might be necessary to detach it and re attach it:
   screen -d -r sessionname
Screen detach before leaving the ssh session. You may want to use the autodetach configuration option.

Log out of your ssh session. Log in back again later:
screen -ls # list sessions
screen -r sessionname # attach a session

There might be solutions to move an already started process to GNU screen but its not straightforward:
"You cannot do this, easily. I'd suggest making it a habit to start screen as the first thing you do after opening a console. However, for your actual problem, there's another thing you could try: after having launched your job from the terminal, background it by typing ctrl-z and then bg. After that, detach the job from it's parent shell; in bash you'd do disown -h %. After that, you can safely close the terminal and the job will continue running."

More tips in this discussion on screen.

Thursday, November 06, 2014

SSH tunnel and port forwarding


  Ubuntu Remote_Port_Forwarding

        ssh -C -D 1080 laptop

    http://straightedgelinux.com/blog/howto/socks.html
        ssh -N -D 1080 klaatu@home.linuxserver.com
    https://wiki.debian.org/SOCKS


http://www.debian-administration.org/article/449/SSH_dynamic_port_forwarding_with_SOCKS

        ssh -D 1080 shell.example.org
        tsocks thunderbird

Thursday, October 30, 2014

R, packages and Rstudio install on Debian wheezy


See also my previous post on Debian GNU-Linux installation on a Lenovo T400.

R install

I used the Synaptic package manager to add the R repository for Debian from a nearby mirror, under : settings / repositories / other software / add.
Add this APT line:
deb http://cran.univ-paris1.fr/bin/linux/debian/ wheezy-cran3/

There was an error:
W: GPG error: http://cran.univ-paris1.fr wheezy-cran3/ Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 06F90DE5381BA480
After looking at several forums, and this stackoverflow question, I installed debian-keyring and added the key with the commands:
gpg --keyserver pgpkeys.mit.edu --recv-key 06F90DE5381BA480
gpg -a --export 06F90DE5381BA480 |sudo apt-key add -
I could then install R version 3 from the synaptic package manager.

Rstudio

I downloaded R-studio and installed it. There was a missing dependency for libjpeg62. I installed that package from Synaptic. Then ran the dpkg command to install rstudio.
dpkg -i rstudio-0.98.507-i386.deb

Tools

Then I installed Git in order to clone my R project from an online repository.
git clone  project_repository_url

Packages

Within Rstudio, I installed a few packages:
install.packages(c("plyr", "reshape2", "ggplot2"))
install.packages(c("xtable", "markdown", "devtools"))

devtools

The devtools packages requires a libcurl dev Debian package. You can install it at the shell prompt:
$ sudo apt-get install libcurl4-gnutls-dev
Back at the R prompt
install.packages("devtools")
Other dependencies might be needed, the RStudio page on devtools recommends installing the Debian package r-base-dev.

dplyr

The dplyr package required the latest version of a Rcpp package. Which was not available on my CRAN mirror. I installed it from source, (based on this message):
install.packages("Rcpp", type = "source")
install.packages("dplyr")

xlsx

The xlsx package installation complained:
configure: error: Cannot compile a simple JNI program. See config.log for details.
Make sure you have Java Development Kit installed and correctly registered in R.
If in doubt, re-run "R CMD javareconf" as root.


Required the latest version of java 7. (inspired by this post). I installed openjdk-7 from the synaptic package manager. Then ran

update-alternatives --config java  # Choose java 7 as the default
R CMD javareconf
Then
install.packages("xlsx") # worked

RMySQL

MySQL client and server are installed on my system.
While installing RMySQL, I struggled with a configuration error:
  could not find the MySQL installation include and/or library
  directories.  Manually specify the location of the MySQL
  libraries and the header files and re-run R CMD INSTALL.
This post has an answer (thanks!):
sudo apt-get install libdbd-mysql  libmysqlclient-dev
That fixes the issue!
I can connect to the database
library(RMySQL)
mychannel <- br="" dbconnect="" host="localhost" user="paul" ysql="">                       password="***", dbname="dbname")

R packages which are better installed from the Debian package manager

Some packages, such as ‘minqa’, ‘SparseM’ and ‘car’ return an error when one tries to install them from the R prompt. The can only be installed from the Debian package manager, where they have names starting with "r-cran": "r-cran-car", "r-cran-sparsem", "r-cran-minqa".

Ready to work!


Monday, September 29, 2014

Make word, pdf and html documents with markdown and pandoc

Markdown is a simple text markup language.
Pandoc is a document converter. Pandoc demo and sample command.

Pandoc commands

Convert a markdown file to PDF :
pandoc -o README.pdf README.md
The pandoc man page says: "If  the input or output format is not specified explicitly, pandoc will attempt to guess it from the extensions of the input and output filenames." That's what happens above. However "The input format can be specified using the -r/--read or -f/--from options, the output format using the -w/--write or -t/--to options."


Makefile 

This phsychologist blogs about using a makefile to create beamer presentations.
This researchers providers a make file for pandoc templates.

With this simple make file, I can create Microsoft Word, HTML and PDF documents from the same markdown file:
all: docx pdf html

docx: file.md
        pandoc -o file.docx file.md

pdf: file.md
        pandoc -o file.pdf file.md

html: deliverable.md
        pandoc -o file.html file.md

clean:
        rm -f *.html *.pdf *.docx
To create all documents type
make
To create only a docx type
make docx
To delete all created document type
make clean

Improved makefile with variable

file.pdf : file.md
    pandoc -o file.pdf file.md

%.pdf: %.md
    pandoc -o $@ $<

Guide makefiles:
"Here, we have used the percent (%) character to denote that part of the target and dependency that matches whatever the pattern is used for, and the $< is a special variable (imaging it like $(<)) that means "whatever the depencies are". Another useful variable is $@, which means "the target"."


## Makefile to generate documents based on markdown files
## Inspired by this makefile
## https://github.com/kjhealy/pandoc-templates/blob/master/examples/Makefile
##
## I should use vraibles for filenames
## Command line to converts:

## How to make this using variables?
## No space allowed in file names there could be a replacement but I didn't try
## http://www.cmcrossroads.com/article/gnu-make-meets-file-names-spaces-them

## Markdown extension (e.g. md, markdown, mdown).
MEXT = md
## All markdown files in the working directory
SRC = $(wildcard *.$(MEXT))


DOCX=$(SRC:.md=.docx)
PDFS=$(SRC:.md=.pdf)
HTML=$(SRC:.md=.html)


all: $(PDFS)  $(DOCX)
pdf:    clean $(PDFS)
docx:   clean $(DOCX)
#html:   clean $(HTML)


#scrap : scrap.md
#    pandoc -o scrap.pdf scrap.md


# Separator for these lines need to start with a hard tab, not 4 spaces!
%.pdf: %.md
    pandoc -o $@ $<

%.docx: %.md
    pandoc -o $@ $<

clean:
    rm -f *.html *.pdf *.docx

Friday, September 26, 2014

Debian GNU/Linux


Configuring the system 

Disk partitioning

Recommended Partitioning Scheme: "For new users, personal Debian boxes, home systems, and other single-user setups, a single / partition (plus swap) is probably the easiest, simplest way to go."  [...] " For multi-user systems or systems with lots of disk space, it's best to put /usr, /var, /tmp, and /home each on their own partitions separate from the / partition."

Keyboard layout

I use a laptop with a Finnish keyboard and a docking station with a French keyboard. I chose the "SHIFT+ caps lock" keys to change from one keyboard layout to the other. By default the French keyboad has a comma as decimal separator on the numeric keypad. I changed this under:
 system settings / region and language / layouts / French / Options / Numeric keypad delete key behaviour / 4 level key with dot

Installing programs

Package management with apt-get or aptitude. Aptitude is recommended. Here is a comparison in a forum.

Programming

Install a version tracking system
sudo aptitude install git
Add coloration:
git config --global color.ui true

Install the vim text editor
sudo aptitude install vim
Edit the vim configuration file:
vim ~/.vimrc
In this file add:
syntax on
set tabstop=4
set expandtab
set softtabstop=4
set shiftwidth=4
filetype indent on

au BufNewFile,BufRead *.md set filetype=txt

Non-Free (unfortunately)

Skype on Debian wiki (for 64 architecture) try alternatives Ekiga and Linphone (Linphone use with French ISP free). Load skype for Debian from the website, then:
sudo dpkg --add-architecture i386
sudo apt-get update 
sudo dpkg -i skype-install_file_you_downloaded.deb 
sudo apt-get -f install
Adobe Flash
sudo apt-get install flashplugin-nonfree
sudo update-flashplugin-nonfree --install

Super user

As a super user install the program called "sudo" :
apt-get install sudo
how to-properly-configure-sudoers-file-on-debian-wheezy
I am in the sudo group
adduser paul sudo
  The user `paul' is already a member of `sudo'.
addgroup sudo
  addgroup: The group `sudo' already exists.
But I get the  warning message:
paul is not in the sudoers file.  This incident will be reported.
It seems I have to restart the system.

 Web browser

Why Firefox has been rebranded as iceweasel.
"Debian was initially given permission to use the trademarks, and adopted the Firefox name. However, because the artwork in Firefox had a proprietary copyright license which was not compatible with the Debian Free Software Guidelines, the substituted logo had to remain."

Wednesday, July 09, 2014

VIM commands

Help

  • :help  -  vim help
  • :help commandname - help on a particular command
  • CTRL+] - jump to a highlighted topic
  • CTRL+T - jump backwards

Motion

  • :help left-right-motion
  • j,k move up down
  • h,l move left right
  • b,w move previous or next word
  • ctrl+b, ctrl+d move page up or page down

Undo redo

  • u: undo last change (can be repeated to undo preceding commands)
  • Ctrl-R: Redo changes which were undone (undo the undos). 
  • Compare to '.' to repeat a previous change, at the current cursor position. Ctrl-R will redo a previously undone change, wherever the change occurred. 

Switch between navigation and editing mode

  • A - move to the end of the line and switch to editing mode 
  • I - switch to editing mode at the current place
  • Escape - switch to navigation mode
  • alt+h alt+j alt+k alt+l - switch to navigation mode and move
  • alt+: - switch to navigation mode and send a command

Search and replace characters

Vim wiki on search and replace 
  • :s/foo/bar/g Find each occurrence of 'foo' (in the current line only), and replace it with 'bar'. 
  • :%s/foo/bar/g Find each occurrence of 'foo' (in all lines), and replace it with 'bar'.
  • %s/option value=".*"//g remove all beginnings of line. :%s/\option\n/, /g replace all end of line by comma + space. This cleans an html list of species for inclusion in a text.

Markdown

Display a list of first level header in a markdown document (found in quick markdown navigation/TOC)
:g/^# /#
Then enter the line number to jump to that line.

Line numbers

 Display line numbers
:set nu
Disable line numbers
:set nonu

Editing a whole line

  • dd to delete a whole line
  • yy to copy a whole line
  • p to paste the copied or deleted text after the current line or 
  • P to paste the copied or deleted text before the current line 

Copy, cut and paste

  • Position the cursor where you want to begin cutting.
  • Press v (or upper case V if you want to cut whole lines).
  • Move the cursor to the end of what you want to cut.
  • Press d to cut or y to copy.
  • Move to where you would like to paste.
  • Press P to paste before the cursor, or p to paste after. 

Indentation

Indentation replaced by spaces, add this to the ~/.vimrc file
set tabstop=4
set expandtab
set softtabstop=4
set shiftwidth=4
filetype indent on 
More details on vim indentation in the python wiki.

Multiple files and windows

  • :e filename - edit another file 
  • :ls         - show current buffers
  • :b 2        - open buffer #2 in this window
  • :b filename - open buffer #filename in this window
  • :bd         - close the current buffer (! to forget changes)
  • :bd filename -close a buffer by name 

Windows

  • :sp[lit] filename  - split window and load another file
  • :vs[plit] - same but split vertically  
  • ctrl-w up arrow - move cursor up a window
  • ctrl-w ctrl-w   - move cursor to another window (cycle)
  • ctrl-w_         - maximize current window
  • ctrl-w=         - make all equal size
  • CTRL+z - suspend the process and get back to the shell
  • fg - get back to vim

Vimdiff

View differences between file1 and file2 (vim documentation)
vimdiff file1 file2

spell check

Set spell check only in the local buffer:
:setlocal spell spelllang=en_gb  
 Turn spell check off
:set nospell

Mark word as correct, this creates a spell file under /home/user/.vim/spell:
zg
Mark word as incorrect
zw

Plugins for programming languages

.vimrc

Text colour.
Add syntax highlight to your .vimrc
syntax enable
How to add a file extension to vim syntax highlight
au BufNewFile,BufRead *.dump set filetype=sql
I used it to display markdown files as text files:
au BufNewFile,BufRead *.md set filetype=txt

Monday, May 26, 2014

Debian GNU-Linux installation on a Lenovo T400


I needed an operating system more stable than Microsoft windows for my daily tasks such as: programming with the R statistical software and writing PDF reports with the Lyx document processor. I read about John MacFarlane (author of pandoc) who is using Debian with the xmonad desktop. This blogger documented why he switched from Ubuntu to Debian. The same blogger also wrote interesting posts at the electronic Frontier Fondation on privacy issues with Ubuntu: on the one side Ubuntu offers an easy way to install full Hard Drive encryption, on the other side, Ubuntu's default desktop search sends search requests over unencripted internet (I should move this additional content to another post).

15 years ago already, I had talked with a friend who was using Debian. I wonder if he still is?

I decided to give Debian a try.

Creating a bootable USB stick

Trying the live version from a USB stick

I introduced the USB key in the laptop,  pressed the "blue thinkvantage" key, entered the BIOS setup and changed the boot order. I placed the USB devices first in the list. The live version seemed to work fine, the Gnome desktop was responsive enough on that machine (Lenovo Thinkpad T400). The external screen was easy to set-up, and network access configured automatically. So I decided to install that Debian system on the hard drive.

Installation

I restarted the laptop with the USB key inserted. I chose the graphical installer. There was a small issue with the fact that the installer was looking for a CD-ROM. But no CD-ROM was available because I was running the install from a USB stick. Based on this blog post, I jumped to the command line interface and typed:
mount /dev/sdb /cdrom 
This command fixed the issue, installation could continue. I partitionned my hard drive to leave 100 GB for the windows partition and the rest (144 GB) for the new GNU-Linux system. Partitioning took a frightening amount of time. During which I thought the system as frozen. But after maybe an hour, installation carried on... I restarted the system. I am now writing this blog post from a fresh Debian system.

Hardware support

Docking station

Hotpluging usb keyboard was working on the laptop itself, but not working on the docking station. A quick fix was to disconnect and reconnect the docking station.
I have connected the docking station to an external larger screen I set this up as my main monitor under system settings / displays.

Different Keyboard layouts

My laptop has a Finnish keyboard, and I am located in France, therefore I use a French keyboard. Under system settings / region and language settings I could add the French keyboard layout. The fact that it's possible to "use the same layout for all windows" is a plus compared to windows. I regularly switch from the Finnish to the French keyboard layout. In windows, I had to change the keyboard layout for all windows one by one.

Network drive

From the Nautilus file manager, I could access windows shared drive using my company's internal network login details.
I have been trying to synchronize this folder using Unison (GUI for rsync). Setting up a local synchronization requires to mount the windows share in the file system. mount.cifs from cifs utils can mount windows shared drives.

Network printer

Under system settings /  printers, the network printer appears. But I didn't manage to connect to the Sharp MX 2600 N.

Wireless card

Show all hardware
lspci
 Network controller: Intel Corporation PRO/Wireless 5100 AGN
Install support for wifi device on Debian
apt-get install firmware-iwlwifi
Reinsert this module to access installed firmware:
# modprobe -r iwlwifi ; modprobe iwlwifi
Wifi works! :-)

Location of software packages

A mirror for France is located at http://ftp.fr.debian.org/debian/.

Install research software 

See also other software install tips in this post on Debian GNU Linux
Lyx editor for Latex
sudo aptitude install lyx
Jabref
sudo aptitude install jabref

Evolution with Microsoft Exchange plugin

My Company uses microsoft exchange on its email servers. An "exchange" plugin can be installed from the Synaptic package manager but this only works for exchange 200 and 2003. This site recommends using the mapi plugin instead which should work for exchange 2007 and 2010. As explained here, my company also uses a different server for the web mail access and the internal connection to the mail server.

To find the internal sever name, I went to the windows version of outlook 2007. Right click on mail box / properties / advanced ... / Microsoft Exchange server.

Mail loads fine now. I can send email too.

Calendar events are not displayed, even though outlook calendar events appear within the desktop calendar (The little calendar that appears when clicking on the desktop clock). This page says that I should upgrade to a more recent version of Evolution. This would require the use of Debian backports. Couldn't get backports to work for now. It said that I already have the most recent version of evolution installed.
I add this line to the 
# Backports repository
deb http://ftp.debian.org/debian wheezy-backports main contrib non-free
Ran this command
sudo apt-get update
sudo apt-get -t wheezy-backports install evolution
Got this message
evolution is already the newest version.
Well it's actually normal because evolution is currently (August 2014) not in the list of Debian backports.

R and Rstudio install on Debian wheezy 7

Instruction below kept here for historical purposes, I'll update these R specific instruction in another blog post.

I used the Synaptic package manager to add the R repository for Debian from a nearby mirror, under : settings / repositories / other software / add.
Add this APT line:
deb http://cran.univ-paris1.fr/bin/linux/debian/ wheezy-cran3/

There was an error:
W: GPG error: http://cran.univ-paris1.fr wheezy-cran3/ Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 06F90DE5381BA480
After looking at several forums, and this stackoverflow question, I installed debian-keyring and added the key with the commands:
gpg --keyserver pgpkeys.mit.edu --recv-key 06F90DE5381BA480
gpg -a --export 06F90DE5381BA480 |sudo apt-key add -
I could then install R version 3 from the synaptic package manager.

I downloaded R-studio and installed it. There was a missing dependency for libjpeg62. I installed that package from Synaptic. Then ran the dpkg command to install rstudio.
dpkg -i rstudio-0.98.507-i386.deb
Then I installed Git in order to clone my R project from an online repository.
git clone  project_repository_url
Within Rstudio, I installed a few packages:
install.packages(c("plyr", "reshape2", "ggplot2"))
install.packages(c("xtable", "markdown", "devtools"))
Ready to work!

Further reading

Wiki on installing Debian on Thinkpad laptops.

Wednesday, March 26, 2014

Git config on GNU-Linux

Colors

To add coloration:
git config --global color.ui true

SSH

Connect to a remote repository without using the password (ssh key). On Bitbucket, different connection protocols have different repository URL formats. Create a private and public ssh key pair with for example the seahorse key manager. Then add the public key to bitbucket under manage account / security / SSH keys.

Moving from https to ssh url format 

My previous connection was through https. You can skip this command if you start with a fresh repository.  Rename the remote "origin" to "originhttps".
git remote rename origin originhttps
Just wanted to be able to connect through https in case the ssh connection doesn't work. This remote tracking branch can be deleted later with:
git remote remove originhttps

Tell git to use a ssh:// remote

Then tell git to use a remote (the "ssh://" protocol name is optional as shown in this how-to):
git add remote origin ssh://git@bitbucket.org/accountname/reponame.git
Fetch the new remote tracking branch into the local repository. (It will probably not load anything new but is required to set this branch as upstream later):
git fetch origin master

Set upstream branch

The commands "Git pull" and "git push" still connected to the https url (which prompts for a password). I had to set the new remote as the default for my master branch:
git branch --set-upstream-to=origin/master
List remote branches (for information)

git branch -r

The repository is now ready to fetch, merge and pull on remote origin without entering the password.


See also my post on git commands.

Monday, January 13, 2014

Msysgit makes GNU Bash available on Windows

Msysgit makes Bash available on windows. It is related to MinGW - Minimalist GNU for Windows, itself a fork of cygwin. The version I use on my system is msysgit. With this tool, some linux like command line operations can be run on windows.

 MSYS Git FAQ:
"MSys is an environment for Windows offering a Unix-type shell and a Perl interpreter. Because many parts of Git are still not builtins programmed in C, but instead shell and Perl scripts, Git for Windows needs such an environment. Therefore we ship Git for Windows with a very minimal version of MSys."
Example of commands:
Compute md5sum
md5sum filename
See also:

Thursday, January 02, 2014

GNU-Linux bash shell commands


Linux is the kernel of the operating system on top of which other programs are built. A detailed list of GNU core utilities is available under the command :
info coreutils

 Files 

determine file type and encoding
file filename
list a directory
ls
ls -R #list subdirectories recursively
ls -lh #sizes in human readable format
Find files in subdirectories of the current directory (Quotes are requited to prevent shell command expansion).
find . -name "*.pdf"
find . -mtime 0 # modified in the last 24 hours
Find files in the whole system
locate filename

File and folder compression

Decompress a file
 gunzip file.gz
How do I compress a whole directory?
tar -zcvf archive-name.tar.gz directory-name
Where
  • -z: Compress archive using gzip program
  • -c: Create archive
  • -v: Verbose i.e display progress while creating archive
  • -f: Archive File name
To extract content from the archive in the current directory
tar -zxvf archive-name.tar.gz

Rename files

For example to rename all upper-case .JPG extension into lower-case .jpg extension.
rename 's/\.JPG$/\.jpg/' *.JPG
Change file permission:
chmod a=rwx filename
chmod 777 filename 
Change file permissions recursively:
chmod 755 directoryname
Chmod instructions can be given with characters or numbers, chmod 777 or chmod a=rwx is a question of preference.
  • Some prefer 755 over 777 because giving write access to group and other users could be a security risk. 755 leaves read and execute rights to groups and other users. 755 is visible as "rwxr-xr-x" in ls -l. 
  • The default for document files on Debian seems to be chmod 644, visible as "-rw-r--r--" in ls -l.

Text files

Count the number of lines in a file
wc -l filename.txt
Count occurrences of a word in a file
grep -roh word filename.txt  | wc -w
Remove duplicated lines from a file
awk '!a[$0]++' input.txt
Search with Grep
 grep "text" file.txt
Awk tutorial, for example  filter a large file for lines that have a third field (product code) starting with 44, keep the header line:
awk -F, '$3 ~ /^44/||NR==1' nc201501.dat|less
Regexp match begining of and end of line with ^ and $.

Follow the end of a log file as it is written 
tail -f
See tab and end of line characters in a text file
cat -te filename |less

Manipulate strings in files

Replace strings
first="I love Suzy and Mary"
second="Sara"
first=${first/Suzy/$second}
Replace strings with sed

sed -i  's/pattern/replacement/g' bli.txt
sed -i  's/^.*\://g' input_file.txt # edit file in place
grep EMAIL input_file.txt |sed  's/^.*\://g' > output_file.txt
Replace strings with perl in a git repository

git grep -lz 'readcsvfromgauss'| xargs -0 perl -i'' -pE "s/readcsvfromgauss/readcsvfromgauss0/g"

PDF files

Commands based on the poppler library for PDF manipulation.
Search a text pattern in all PDF files present in a directory:
pdfgrep pattern *.pdf
 Merge multiple PDF into one:
pdfunite in-1.pdf in-2.pdf in-n.pdf out.pdf
Alternatively, pdftk can be used to merge PDF files
pdftk input1.pdf input2.pdf cat output output.pdf

Videos and audio

Install youtube-dl using pip:
 sudo pip install --upgrade youtube_dl
Download a video from youtube :
youtube-dl video_url
Download only the audio in an .mp3 format
youtube-dl --extract-audio --audio-format mp3 video_url

Users

Check your user id
id 
What group do you belong to as a user
groups
Add a user to the super users
adduser username sudo
That user needs to re-log into the shell for the change to take effect.
Add a new user
useradd username
Set a password for the new user
passwd username
Delete a user
userdel username
Show all users
getent passwd
Show all groups
getent group

System

OS release
less /etc/os-release
Disk usage
du -h
Display available space on drives
df -h
Display available RAM memory
less /proc/meminfo

Install a program
sudo apt-get install
 System name
uname -a
file /sbin/init
hostname -f
Start and quit a super user session
su
exit
Last time the system was started
last reboot 
last
Show environment variables
printenv

Job handling

List
jobs
Bring a job to the foreground
fg job_number
Run a job in the background. A command followed by an & will run in the background.

Stop a job
CTRL ^ Z
Quit a job
 CTRL ^ C
Kill a malfunctionning program:
kill process_id
Find a program id with:
ps aux
Kill a graphical program, by clicking on it:
 xkill

Users

Create a new user
adduser user_name
Temporary log in as that user
su user_name
Delete a user
userdel user_name

Secure shell

log into a remote machine
ssh user@remote_machine
Copy a local file to a file on the remote machine
scp local_file_name user@remote_machine:path_to_file/file_name
Copy a file from the remote machine to a local file
scp user@remote_machine:path_to_file/file_name  local_file_name
Copy a full directory (dmouraty) from the remote machine:
 scp -rp user@dest:/path destdirectory

Alias

alias ll="ls -lh"

Based on how can i sort du-h output by size
alias du='du -hd1 | sort -h -r'

You can place those commands in your ~/.bashrc to create a permanent alias.
bashrc:
"You may want to put all your additions into a separate file like ~/.bash_aliases, instead of adding them here directly."

.bash_profile and .bashrc

These are places where a user can turn of the system BEEP :
setterm -blength 0
.bash_profile is executed on login shell, when you login in another tty or when you access a system through ssh. .bashrc is executed on non-login shells when you open a terminal window in Gnome.

Debian Dotfiles
"Now, since bash is being invoked as a login shell (with name "-bash", a special ancient hack), it reads /etc/profile first. Then it looks in your home directory for .bash_profile, and if it finds it, it reads that."
[...] "You may have noted that .bashrc is not being read in this situation. You should therefore always have command source ~/.bashrc at the end of your .bash_profile in order to force it to be read by a login shell.  "
In .bashrc a user can set environment variables, define alias (see above).

Keyboard

bash french blogger recommended a simple shell command to change keyboard layout :
sudo loadkeys fr
fr-keyboard on Debian wiki for a more permanent system configuration and use in GUI apps. Switching between keyboads can then be done with:
setxkbmap de
setxkbmap fr

Information about the system

  • cat /proc/meminfo
  • cat /proc/cpuinfo
  • cat /etc/debian_version
  • lsb_release -a

Shortcuts

Keyboard shortcuts for bash  for example Ctrl+A to go to the beginning of a line.

Documentation

Monday, September 05, 2005

Samba on Ubuntu linux

SMB how to list shares available on a host :
smbclient -L host
Then connect to a shared folder on a host, in my case:
smbclient \\\\freebox\\Disque\ dur
Then under pcmanfm (my file browser) I can access it at the addess
smb://freebox/Disque dur



Sometimes the file browser complains that the folder is not mounted.

According to this question cifs utils should now be used to mount a smb drive:
 
sudo mount -t cifs //server_name/ samba/ -o rw
Maybe this is what my file browser does in the background when I ask it to got the the smb:// address?

Old content from my 2005 blog post :
I found some interesting samba config on this forum (link not available anymore)
I also used VNC-viewer to see the desktop from a windows machine.