javascript - Is there a way to speed up dryscrape eval_script() execution? -


i scraping web page using dryscrape (as need javascript rendered parts) , using eval_script() suppress javascript based error checking on page. script i'm suppressing onkeyup listener need avoid makes mandatory user select options dropdown only. eval script -

session.eval_script("$('#input_elem').removeattr('onblur onclick onkeyup');") 

now overall scraping takes longer time compared other implementation of page on same domain doesn't require javascript modifications (hence without eval_script()).

i did bit of profiling using time.time() see script slowing , indeed, taking long time on eval_script() step(s). here results -

starting access @  0.00997018814087 visited page  https://*****/***.aspx    1.30053019524 first eval script run done  5.97628307343 second eval script run done  9.61053919792 xpath 1  9.6632771492 xpath 2  9.7702870369 xpath 3  9.90402317047 xpath of button clicked  9.91756606102 button clicked  9.97191905975 second page visited  10.4508111477 loop 1 else  10.4525721073 xpath 4  10.5330061913 xpath 5  10.6111950874 xpath 6  10.6918411255 xpath 7  10.7721481323 range begins  10.8208150864 3 range ends  13.0008580685 

although when i'm looping through table elements, taking 2 seconds, 2 eval_script() steps, combined, taking 8 seconds. when in chrome dev tools console, same scripts run in instant. why dryscrape implementation taking time?

using jquery in eval_script() seems culprit. able reduce script execution time using plain javascript -

session.eval_script("document.getelementbyid('input_elem')").removeattribute('onblur'); 

i had use 2 lines 2 separate attribute removal steps on same element.

these profiling logs -

starting access @  0.0151550769806 visited page  https://*****/***.aspx    1.73412919044 first eval script run done  1.77594304085 first eval script part 2 run done  1.81522011757 second eval script run done  1.85607099533 xpath 1  1.94704914093 xpath 2  2.03846216202 xpath 3  2.13886809349 xpath of button clicked  2.26395010948 button clicked  2.27277112007 second page visited  3.30618906021 loop 1 else  3.38708400726 xpath 4  3.46828198433 xpath 5  3.54840707779 xpath 6  3.63034701347 xpath 7  3.7106590271 range begins  3.75155210495 3 range ends  5.91926407814 

even now, each step in range loop taking 0.7 seconds i'm aiming reduce further.


Comments

Popular posts from this blog

Command prompt result in label. Python 2.7 -

javascript - How do I use URL parameters to change link href on page? -

amazon web services - AWS Route53 Trying To Get Site To Resolve To www -