python - How to divide up webscraping workload -


i have massive webscraping project (part 1 scraping 300k+ separate data entries website). may have more of these in future , 1 data entry @ time won't suffice. have been using selenium enter data js site, , beautifulsoup parse results. have looked @ selenium grid don't believe accomplish want because i'm not trying have every instance perform same function.

i take ~300k separate data entries , split them search, example, 8+ @ time.

is option @ point (in python) setup several vms , execute python script in each? current time finish scrape 30 hours.


Comments

Post a Comment

Popular posts from this blog

'hasOwnProperty' in javascript -

i use obj.hasownproperty judge whether object has property, when replaced obj[prop] !== undefined , not normal implementation, ask, why behind method can not use it? object.hasownproperty(prop); object[prop] !== undefined; obj[prop] !== undefined wrong 2 reasons: you can explicitly set property undefined , obj[prop] = undefined; . obj.hasownproperty(prop) return true in case. obj[prop] follow prototype chain, return property that's inherited. obj.hasownproperty(prop) returns true if property exists directly in object, returns false inherited properties.
Read more

How to understand 2 main() functions after using uftrace to profile the C++ program? -

i trying uftrace profile following simple c++ program: #include <iostream> class { public: a() {std::cout << "a created" << std::endl;} ~a() {std::cout << "a destroyed" << std::endl;} }; int main() { a; return 0; } the profile result this: # uftrace a.out created destroyed # duration tid function 2.026 [ 4828] | __cxa_atexit(); [ 4828] | main() { [ 4828] | __static_initialization_and_destruction_0() { 89.397 [ 4828] | std::ios_base::init::init(); 0.768 [ 4828] | __cxa_atexit(); 93.029 [ 4828] | } /* __static_initialization_and_destruction_0 */ 94.425 [ 4828] | } /* main */ [ 4828] | main() { [ 4828] | a::a() { 11.104 [ 4828] | std::operator <<(); 10.825 [ 4828] | std::basic_ostream::operator <<(); 24.514 [ 4828] | } /* a::a */ [ 4828] | a::~a() { 0.978 [ 4828] | ...
Read more

java - How to implement an entity bound odata action in olingo v4.3 -

similar post, i'm trying implement action bound entity how create bounded action in olingo v4 (java) but 500 no error logged when metadata document. in answer above linked question, there no mention of getactionimport() implementation described in olingo tutorial, , wonder if needs modified entity-bound action? how can olingo log error causing 500? getting debug output quite easy: handler.register(new defaultdebugsupport()); where handler odatahttphandler armed this, , after examination of olingo-server-core code, discovered: you don't need actionimport @ bound actions. the action name in url needs qualified namespace, https://issues.oasis-open.org/browse/odata-812 has not been implemented. when built client url using org.apache.olingo.client.api.uri.uribuilder , wrapped entity id in single quotes, , olingo server threw exception, reverted using stringbuilder
Read more