consistency - cassandra replica exception HUnavailableException -

i have cassandra 2 datacenter pair single replication each datacenter containing single node , each datacenter located on separate physical servers on network. if 1 datacenter crashes, other 1 continue available reads , writes started java application, on 3rd server, , running ok. it's reading , writing cassandra.

next disconnected, pulled network cable, 2nd datacenter server network. expected application continue running no exceptions against 1st datacenter, not case.

the following exception started occur in application:

me.prettyprint.hector.api.exceptions.hunavailableexception: : may not enough replicas present handle consistency level.         @ me.prettyprint.cassandra.service.exceptionstranslatorimpl.translate(exceptionstranslatorimpl.java:60)         @ me.prettyprint.cassandra.service.keyspaceserviceimpl$9.execute(keyspaceserviceimpl.java:354)         @ me.prettyprint.cassandra.service.keyspaceserviceimpl$9.execute(keyspaceserviceimpl.java:343)         @ me.prettyprint.cassandra.service.operation.executeandsetresult(operation.java:101)         @ me.prettyprint.cassandra.connection.hconnectionmanager.operatewithfailover(hconnectionmanager.java:232)         @ me.prettyprint.cassandra.service.keyspaceserviceimpl.operatewithfailover(keyspaceserviceimpl.java:131)         @ me.prettyprint.cassandra.service.keyspaceserviceimpl.getsupercolumn(keyspaceserviceimpl.java:360)         @ me.prettyprint.cassandra.model.thrift.thriftsupercolumnquery$1.doinkeyspace(thriftsupercolumnquery.java:51)         @ me.prettyprint.cassandra.model.thrift.thriftsupercolumnquery$1.doinkeyspace(thriftsupercolumnquery.java:45)         @ me.prettyprint.cassandra.model.keyspaceoperationcallback.doinkeyspaceandmeasure(keyspaceoperationcallback.java:20)         @ me.prettyprint.cassandra.model.executingkeyspace.doexecute(executingkeyspace.java:85)         @ me.prettyprint.cassandra.model.thrift.thriftsupercolumnquery.execute(thriftsupercolumnquery.java:44) 

once reconnected network cable 2nd server, error stopped.

here's more details on cassandra 1.0.10

1) here's following describe cassandra on both datacenters

keyspace: advancedads: replication strategy: org.apache.cassandra.locator.networktopologystrategy durable writes: true options: [dc2:1, dc1:1] 

2) ran node tool ring against each instance

./nodetool -h 111.111.111.111 -p 11000 ring address dc rack status state load owns token 1 111.111.111.111 dc1 rac1 # <-- usup normal 1.07 gb 100.00% 0 111.111.111.222 dc2 rac1 normal 1.1 gb 0.00% 1  ./nodetool -h 111.111.111.222 ring -port 11000 address dc rack status state load owns token 1 111.111.111.111 dc1 rac1 normal 1.07 gb 100.00% 0 111.111.111.222 dc2 rac1 # <-- usup normal 1.1 gb 0.00% 1 

3) checked cassandra.yaml

the seeds 111.111.111.111, 111.111.111.222 

4) checked cassandra-topology.properties

111.111.111.111      # cassandra node ip=data center:rack      # datacenter 1     111.111.111.111=dc1:rac1 # <--      # datacenter 2     111.111.111.222=dc2:rac1      default=dc1:r1  111.111.111.222      # cassandra node ip=data center:rack      # datacenter 1     111.111.111.111=dc1:rac1      # datacenter 2     111.111.111.222=dc2:rac1 # <--      default=dc1:r1 

5) set consistencylevel local_quorum in our java application follows:

public keyspace getkeyspace(final string keyspacename, final string serveraddresses) {             keyspace ks = null;     cluster c = clustermap.get(serveraddresses);     if (c != null)     {                     configurableconsistencylevel policy = new configurableconsistencylevel();         policy.setdefaultreadconsistencylevel(consistencylevel);         policy.setdefaultwriteconsistencylevel(consistencylevel);          // create keyspace         ks = hfactory.createkeyspace(keyspacename, c, policy);     }             return ks; } 

i told configuration work, maybe i'm missing something.

thanks insight

hector known return spurious unavailable errors. native protocol java driver not have problem: https://github.com/datastax/java-driver