neo4j - Commits are slowing down -


neo4j writes slow down after few thousand records written.

two indexes exist speed query up. also, using explain know each query constant time process.

        indexorder = schema.indexfor(order)                 .on("id")                 .create();         indexshop = schema.indexfor(shop)                 .on("domain")

this query use:

 {json} log   log.order order, log.shop shop   merge (s:shop {domain:shop.domain})     on create set s=shop  merge (s)-[:scored]->(r:order {id:order.id})     on create set r=order

here how commit store db:

private void log() {     try (transaction tx = graphdb.begintx()) {         (map map : list) {             graphdb.execute(query,                     singletonmap("json", map));          }         list = new arraylist<>();         tx.success();     } }

and call above when have 1k logs.

        list.add(map);         count++;         if (count % 1000 == 0) {             log();             system.out.println(count);         }

additional info: use these config settings:

            .setconfig(graphdatabasesettings.pagecache_memory, "512m")             .setconfig(graphdatabasesettings.string_block_size, "60")             .setconfig(graphdatabasesettings.array_block_size, "300")

this system works 200k entries if done within 1 transaction runs memory issues.

so, why 1k entries/transaction approach grind halt after 5 transactions (5k entries) commited database?

how fix problem?

  1. you should merge on order node itself, allow cypher planner use :order(id) index. (besides, should merge nodes in path pattern before doing merge on path pattern anyway, avoid creating duplicate nodes in circumstances.) change, query (but not yet ideal):

    with {json} log  log.order order, log.shop shop  merge (s:shop {domain:shop.domain})   on create set s=shop merge (r:order {id:order.id})   on create set r=order merge (s)-[:scored]->(r)

  2. you should minimize number of calls execute(), each call has lot of overhead. in fact, can make single query handle entire list of 1000 items. so, can change log() code following (i assume list defined list<map<string, object>> or that):

    private void log() {     try (transaction tx = graphdb.begintx()) {         graphdb.execute(query, singletonmap("loglist", list));         list = new arraylist<map<string, object>>();         tx.success();     } }

    and here corresponding cypher query:

    unwind {loglist} log  log.order order, log.shop shop  merge (s:shop {domain:shop.domain})   on create set s=shop merge (r:order {id:order.id})   on create set r=order merge (s)-[:scored]->(r)

Comments

Post a Comment

Popular posts from this blog

c# - Update a combobox from a presenter (MVP) -

i using mvp in project, new in mvp. have 2 comboboxes. when select option in combobox, combobox should filled new data. this action in presenter. view 'view1' in presenter, , introduced combobox1 , combobox2 properties in 'view1', because need 'datasource', 'displaymember', 'valuemember' , 'refresh()' in method below. but, when using pattern, enough send property like public string combobox2 { { return combobox1.selectedvalue.tosstring(); } } into presenter not whole combobox. how can solve problem? public void onselectedindexchangedcombobox1() { if (view1.combobox1.selectedindex == -1) { return; } datatable dt = tools.getdatatable("a path"); var query = (from o in dt.asenumerable() o.field<string>("afield") == farmerview.combobox1.selectedvalue.tostring() orderby o.field<string>("anotherfield") select...
Read more

How to understand 2 main() functions after using uftrace to profile the C++ program? -

i trying uftrace profile following simple c++ program: #include <iostream> class { public: a() {std::cout << "a created" << std::endl;} ~a() {std::cout << "a destroyed" << std::endl;} }; int main() { a; return 0; } the profile result this: # uftrace a.out created destroyed # duration tid function 2.026 [ 4828] | __cxa_atexit(); [ 4828] | main() { [ 4828] | __static_initialization_and_destruction_0() { 89.397 [ 4828] | std::ios_base::init::init(); 0.768 [ 4828] | __cxa_atexit(); 93.029 [ 4828] | } /* __static_initialization_and_destruction_0 */ 94.425 [ 4828] | } /* main */ [ 4828] | main() { [ 4828] | a::a() { 11.104 [ 4828] | std::operator <<(); 10.825 [ 4828] | std::basic_ostream::operator <<(); 24.514 [ 4828] | } /* a::a */ [ 4828] | a::~a() { 0.978 [ 4828] | ...
Read more

How to put a lock and transaction on table using spring 4 or above using jdbcTemplate and annotations like @Transactional? -

i trying put lock on table while writing on table , if happened in between roll-back . trying convert below code lock table test_g1 read; lock table test_g write; -- begin; start transaction; insert test_g1 values(143); insert test_g values(145); select * test_g1; select * test_g; rollback; select * test_g; unlock tables; how convert above code @transactional spring jdbctemplate code? @transactional(rollbackfor={dataaccessexception.class}) public void test(){ jdbctemplate.execute("insert test1 (id, nam) values (4, 'a')"); throw new dataaccessexception("error") { }; } here trying throw error, insert statement should rollback it's not happening . thanks edit-1 attaching code , doing in jdbcdaoimpl.java , mentioned problem comment above test() . app.java package com.cgiri.javabrains.spring4; import org.springframework.beans.factory.annotation.autowired; import org.springframework.context.applic...
Read more