Note: Reading large tables into R

Reading large tables from text files into R is possible but knowing a few tricks will make your life a lot easier and make R run a lot faster.

First, read the help page for ' read.table'. It contains many hints for how to read in large tables. Of course, help pages tend to be a little confusing so I'll try to distill the relevant details here.

The following options to 'read.table()' can affect R's ability to read large tables:

colClasses

This option takes a vector whose length is equal to the number of columns in year table. Specifying this option instead of using the default can make 'read.table' run MUCH faster, often twice as fast. In order to use this option, you have to know the class of each column in your data frame. If all of the columns are "numeric", for example, then you can just set 'colClasses = "numeric"'. If the columns are all different classes, or perhaps you just don't know, then you can have R do some of the work for you.

You can read in just a few rows of the table and then create a vector of classes from just the few rows. For example, if I have a file called "datatable.txt", I can read in the first 5 rows and determine the column classes from that:

tab5rows <- read.table("datatable.txt", header = TRUE, nrows = 5)
classes <- sapply(tab5rows, class)
tabAll <- read.table("datatable.txt", header = TRUE, colClasses = classes)

Always try to use 'colClasses', it will make a big difference.
nrows

Specifying the 'nrows' argument doesn't necessary make things go faster but it can help a lot with memory usage. R doesn't know how many rows it's going to read in so it first makes a guess, and then when it runs out of room it allocates more memory. The constant allocations can take a lot of time, and if R overestimates the amount of memory it needs, your computer might run out of memory.

Of course, you may not know how many rows your table has. The easiest way to find this out is to use the 'wc' command in Unix. So if you run 'wc datafile.txt' in Unix, then it will report to you the number of lines in the file (the first number). You can then pass this number to the 'nrows' argument of 'read.table()'.

If you can't use 'wc' for some reason, but you know that there are definitely less than, say, N rows, then you can specify 'nrows = N' and things will still be okay. A mild overestimate for 'nrows' is better than none at all.
comment.char
If your file has no comments in it (e.g. lines starting with '#'), then setting 'comment.char = ""' will sometimes make 'read.table()' run faster.

I have tested the method with command "file <- read.table("quant-norm.pm-gcbg.plier.pca-select.summary.txt", header = TRUE,skip=80, colClasses = classes,sep="\t", comment.char = "", nrows=1236087)". I found that 50% memory can be saved for these three parameters, while it will be out of memory if passby these settings. 4G RAM in Linux for test.

Simple the biological data

Cite: Scientia's Glorikian this week said that is unlikely that any one technology, be it qPCR, arrays, FISH, or second-gen sequencing, will win out in the end when it comes to molecular diagnostics.

I do agree with it and think that proteomics data should be a complement.

BP制造

Missions of April Youth?

我是不是“四月青年”？我在不在仰望星空？国家和社会对我们难道没有期望？必须清醒，但须坦然。

Where am I in spirit?

Kierkegaard's three modes of existence -- the aesthetic, the ethical, and the religious. I am in the second stage.

Søren Aabye Kierkegaard:(5 May 1813 – 11 November 1855) was a prolific 19th century Danish philosopher and theologian. Kierkegaard strongly criticized both the Hegelianism of his time, and what he saw as the empty formalities of the Danish church.

From wiki: http://en.wikipedia.org/wiki/S%C3%B8ren_Kierkegaard

Personal genomics and medicine resources

Some companies
https://www.23andme.com/
http://www.navigenics.com/
http://www.decode.com/
http://www.knome.com/home/
http://www.dnadirect.com/
http://www.pacificbiosciences.com/

http://www.hgbiochip.com
http://www.yigene.com/


Bioinfor companies for NGS
http://www.genomequest.com/

Public personal genomics
http://thepersonalgenome.com/
http://www.yourgenome.org/

A good news website of personal genomics
http://www.eyeondna.com/

Resources for paticipate medicine and predictive medicine
http://mydaughtersdna.org/
http://www.framinghamheartstudy.org/risk/hrdcoronary.html

A aggregator for pathway database
http://www.pathguide.org/

Two popular medical informatics system.
VISTA http://www.virec.research.va.gov/DataSourcesName/VISTA/VISTA.htm
the clinical health services framework implemented by the Boston-based CareGroup

From genotyping to complete genome analysis.

Kabasiji key resource

http://a.kavkiskey.com/new.html
Useful resource.

History and philosophy

Recently I like reading book of history and philosophy very much. I learned something from those stories. Human is in nature and belong to nature. We should love nature.

When will my book of Wang Yang-Ming arrive? Wang Yang-Ming was a Ming Chinese idealist Neo-Confucian philosopher, official, educationist, calligraphist and general. Please see http://en.wikipedia.org/wiki/Wang_Yangming to know him.

Bird Hang is my first English name, I like it.

When I was enrolled in my university in 1998, I played basketball with my classmates almost everyday under the sunshine. I remembered that I was very confident of my shooting at that time and they gave me a nickname "bird".

I can't fly and I am not a bird at all. It's from "Larry Bird" of Celtic in NBA. Ten years went by, I am bird now reminded by Dr. Chen. Additionally, happy to heard from B that he took part in parade of Celtic in Boston for championship. I like the air at that site.

Splice life from points to system

Life is a mystery for me. I had tried my best to understand life for about 3 decades. It is just like a cake, you should cut it and taste it. In your mouth, you will find it not only as a small point but also as a huge system.

We should respect the works of God and human nature.