Cassandra replication node failed when try to read from keyspace -

i have 3 nodes in cassandra (3.0.2) replication set. consistency level "one". @ beginning keyspaces had replication factor equal 1. changed table altering , ran "nodetool repair" on nodes. right when i'm trying select data (not on every keyspace) (select * keyspace.table):

traceback (most recent call last): file "/usr/bin/cqlsh.py", line 1258, in perform_simple_statement result = future.result() file "cassandra/cluster.py", line 3781, in cassandra.cluster.responsefuture.result (cassandra/cluster.c:73073) raise self._final_exception readfailure: error server: code=1300 [replica(s) failed execute read] message="operation failed - received 0 responses , 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'one'}

in "/var/log/cassandra/system.log" get:

warn [sharedpool-worker-2] 2017-04-07 12:46:20,036 abstracttracingawareexecutorservice.java:169 - uncaught exception on thread thread[sharedpool-worker-2,5,main]: {} java.lang.assertionerror: null @ org.apache.cassandra.db.columniterator.abstractsstableiterator$indexstate.updateblock(abstractsstableiterator.java:463) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.columniterator.sstableiterator$forwardindexedreader.computenext(sstableiterator.java:268) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.columniterator.sstableiterator$forwardreader.hasnextinternal(sstableiterator.java:158) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.columniterator.abstractsstableiterator$reader.hasnext(abstractsstableiterator.java:352) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.columniterator.abstractsstableiterator.hasnext(abstractsstableiterator.java:219) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.columniterator.sstableiterator.hasnext(sstableiterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.rows.lazilyinitializedunfilteredrowiterator.computenext(lazilyinitializedunfilteredrowiterator.java:95) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.rows.lazilyinitializedunfilteredrowiterator.computenext(lazilyinitializedunfilteredrowiterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.utils.abstractiterator.hasnext(abstractiterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.utils.mergeiterator$candidate.advance(mergeiterator.java:369) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.utils.mergeiterator$manytoone.advance(mergeiterator.java:189) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.utils.mergeiterator$manytoone.computenext(mergeiterator.java:158) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.utils.abstractiterator.hasnext(abstractiterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.rows.unfilteredrowiterators$unfilteredrowmergeiterator.computenext(unfilteredrowiterators.java:426) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.rows.unfilteredrowiterators$unfilteredrowmergeiterator.computenext(unfilteredrowiterators.java:286) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.utils.abstractiterator.hasnext(abstractiterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.rows.lazilyinitializedunfilteredrowiterator.computenext(lazilyinitializedunfilteredrowiterator.java:95) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.rows.lazilyinitializedunfilteredrowiterator.computenext(lazilyinitializedunfilteredrowiterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.utils.abstractiterator.hasnext(abstractiterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.transform.baserows.hasnext(baserows.java:108) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.rows.unfilteredrowiteratorserializer.serialize(unfilteredrowiteratorserializer.java:131) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.rows.unfilteredrowiteratorserializer.serialize(unfilteredrowiteratorserializer.java:87) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.rows.unfilteredrowiteratorserializer.serialize(unfilteredrowiteratorserializer.java:77) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.partitions.unfilteredpartitioniterators$serializer.serialize(unfilteredpartitioniterators.java:298) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.readresponse$localdataresponse.build(readresponse.java:136) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.readresponse$localdataresponse.(readresponse.java:128) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.readresponse$localdataresponse.(readresponse.java:123) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.readresponse.createdataresponse(readresponse.java:65) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.db.readcommand.createresponse(readcommand.java:289) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.service.storageproxy$localreadrunnable.runmaythrow(storageproxy.java:1721) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.service.storageproxy$droppablerunnable.run(storageproxy.java:2375) ~[apache-cassandra-3.0.2.jar:3.0.2] @ java.util.concurrent.executors$runnableadapter.call(executors.java:511) ~[na:1.8.0_121] @ org.apache.cassandra.concurrent.abstracttracingawareexecutorservice$futuretask.run(abstracttracingawareexecutorservice.java:164) ~[apache-cassandra-3.0.2.jar:3.0.2] @ org.apache.cassandra.concurrent.sepworker.run(sepworker.java:105) [apache-cassandra-3.0.2.jar:3.0.2] @ java.lang.thread.run(thread.java:745) [na:1.8.0_121] debug [sharedpool-worker-1] 2017-04-07 12:46:20,037 readcallback.java:126 - failed; received 0 of 1 responses

also get:

debug [sharedpool-worker-1] 2017-04-07 13:20:30,002 readcallback.java:126 - timed out; received 0 of 1 responses

i checked there connection between nodes on ports 9042 , 7000. changed options in "/etc/cassandra/cassandra.yml" "read_request_timeout_in_ms", "range_request_timeout_in_ms", "write_request_timeout_in_ms" oraz "truncate_request_timeout_in_ms". changed file "~/.cassandra/cqlshrc" , option "client_timeout = 3600". additonally when example "select * keyspace.table column1 = 'value' , column2 = value" get:

readtimeout: error server: code=1200 [coordinator node timed out waiting replica nodes' responses] message="operation timed out - received 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'one'}

any ideas?

this more or less comment since there won't fit comment.

it nice replication factor changed value. i'll assume 3 because it's pretty standard. again since have cluster of 3 people set rf 2. mentioned updated replication factor on table. far know replication factor setup on keyspace level.

it helpful if posted description of keyspaces error occurring.

take account select * something might pretty intensive in cluster, if have lot of data. if query in cqlsh might 10 000 again mentioned cqlsh , no application code i'm note sure 1 etc.

could please provide nodetool status make sure not running query nodes down. because first error looks this.

by second error posted stack trace looks missing sstables on disk? there chance other process has somehow manipulated sstables in way?

you changed lot of properties in cassandra.yaml reduced expected response times 50% , guess it's no wonder nodes don't have time respond... count on whole table takes more 3.6 seconds

reasoning why values changed missing.