python - Read in large table file but keep only small subset of rows using pandas -


i have large table file (around 2 gb) holds distance matrix indexed first column. rows like

a 0 1.2 1.3 ... b 1.2 0 3.5 ... c 1.5 0 4.5 ...

however, need keep small subset of rows. if i'm given list of indices need keep, best , fastest way read file pandas dataframe. right now, using

distance_matrix = pd.read_table("hla_distmat.txt", header = none, index_col = 0)[columns_to_keep]

to read in file, running memory issues read_table command. there faster , more memory efficient way this? thanks.

you need usecols parameter if need filter columns , skiprows filter rows, have specify column has removed list or range or np.array:

distance_matrix = pd.read_table("hla_distmat.txt",                                   header = none,                                   index_col = 0,                                   usecols=[columns_to_keep],                                  skiprows = range(10, 100))

sample: (in real data omit sep parameter, sep='\t' default in read_table)

import pandas pd import numpy np  pandas.compat import stringio  temp=u"""0;119.02;0.0 1;121.20;0.0 3;112.49;0.0 4;113.94;0.0 5;114.67;0.0 6;111.77;0.0 7;117.57;0.0 6648;0.00;420.0 6649;0.00;420.0 6650;0.00;420.0""" #after testing replace 'stringio(temp)' 'filename.csv'  columns_to_keep = [0,1]  df = pd.read_table(stringio(temp),                     sep=";",                     header=none,                    index_col=0,                     usecols=columns_to_keep,                    skiprows = range(5, 100)) print (df)         1 0         0  119.02 1  121.20 3  112.49 4  113.94 5  114.67

more general solution numpy.setdiff1d:

#if index_col = 0 need first column (0) columns_to_keep = [0,1] #for keep second, third, fifth row rows_to_keep = [1,2,4] #estimated row count or use solution http://stackoverflow.com/q/19001402/2901002 max_rows = 100  df = pd.read_table(stringio(temp),                     sep=";",                     header=none,                    index_col=0,                     usecols=columns_to_keep,                    skiprows = np.setdiff1d(np.arange(max_rows), np.array(rows_to_keep))) print (df)         1 0         1  121.20 3  112.49 5  114.67

Comments

Post a Comment

Popular posts from this blog

How to understand 2 main() functions after using uftrace to profile the C++ program? -

i trying uftrace profile following simple c++ program: #include <iostream> class { public: a() {std::cout << "a created" << std::endl;} ~a() {std::cout << "a destroyed" << std::endl;} }; int main() { a; return 0; } the profile result this: # uftrace a.out created destroyed # duration tid function 2.026 [ 4828] | __cxa_atexit(); [ 4828] | main() { [ 4828] | __static_initialization_and_destruction_0() { 89.397 [ 4828] | std::ios_base::init::init(); 0.768 [ 4828] | __cxa_atexit(); 93.029 [ 4828] | } /* __static_initialization_and_destruction_0 */ 94.425 [ 4828] | } /* main */ [ 4828] | main() { [ 4828] | a::a() { 11.104 [ 4828] | std::operator <<(); 10.825 [ 4828] | std::basic_ostream::operator <<(); 24.514 [ 4828] | } /* a::a */ [ 4828] | a::~a() { 0.978 [ 4828] | ...
Read more

c# - Update a combobox from a presenter (MVP) -

i using mvp in project, new in mvp. have 2 comboboxes. when select option in combobox, combobox should filled new data. this action in presenter. view 'view1' in presenter, , introduced combobox1 , combobox2 properties in 'view1', because need 'datasource', 'displaymember', 'valuemember' , 'refresh()' in method below. but, when using pattern, enough send property like public string combobox2 { { return combobox1.selectedvalue.tosstring(); } } into presenter not whole combobox. how can solve problem? public void onselectedindexchangedcombobox1() { if (view1.combobox1.selectedindex == -1) { return; } datatable dt = tools.getdatatable("a path"); var query = (from o in dt.asenumerable() o.field<string>("afield") == farmerview.combobox1.selectedvalue.tostring() orderby o.field<string>("anotherfield") select...
Read more

How to put a lock and transaction on table using spring 4 or above using jdbcTemplate and annotations like @Transactional? -

i trying put lock on table while writing on table , if happened in between roll-back . trying convert below code lock table test_g1 read; lock table test_g write; -- begin; start transaction; insert test_g1 values(143); insert test_g values(145); select * test_g1; select * test_g; rollback; select * test_g; unlock tables; how convert above code @transactional spring jdbctemplate code? @transactional(rollbackfor={dataaccessexception.class}) public void test(){ jdbctemplate.execute("insert test1 (id, nam) values (4, 'a')"); throw new dataaccessexception("error") { }; } here trying throw error, insert statement should rollback it's not happening . thanks edit-1 attaching code , doing in jdbcdaoimpl.java , mentioned problem comment above test() . app.java package com.cgiri.javabrains.spring4; import org.springframework.beans.factory.annotation.autowired; import org.springframework.context.applic...
Read more