javascript - Is there a way to speed up dryscrape eval_script() execution? -
i scraping web page using dryscrape (as need javascript rendered parts) , using eval_script() suppress javascript based error checking on page. script i'm suppressing onkeyup
listener need avoid makes mandatory user select options dropdown only. eval script -
session.eval_script("$('#input_elem').removeattr('onblur onclick onkeyup');")
now overall scraping takes longer time compared other implementation of page on same domain doesn't require javascript modifications (hence without eval_script()).
i did bit of profiling using time.time()
see script slowing , indeed, taking long time on eval_script() step(s). here results -
starting access @ 0.00997018814087 visited page https://*****/***.aspx 1.30053019524 first eval script run done 5.97628307343 second eval script run done 9.61053919792 xpath 1 9.6632771492 xpath 2 9.7702870369 xpath 3 9.90402317047 xpath of button clicked 9.91756606102 button clicked 9.97191905975 second page visited 10.4508111477 loop 1 else 10.4525721073 xpath 4 10.5330061913 xpath 5 10.6111950874 xpath 6 10.6918411255 xpath 7 10.7721481323 range begins 10.8208150864 3 range ends 13.0008580685
although when i'm looping through table elements, taking 2 seconds, 2 eval_script() steps, combined, taking 8 seconds. when in chrome dev tools console, same scripts run in instant. why dryscrape implementation taking time?
using jquery in eval_script() seems culprit. able reduce script execution time using plain javascript -
session.eval_script("document.getelementbyid('input_elem')").removeattribute('onblur');
i had use 2 lines 2 separate attribute removal steps on same element.
these profiling logs -
starting access @ 0.0151550769806 visited page https://*****/***.aspx 1.73412919044 first eval script run done 1.77594304085 first eval script part 2 run done 1.81522011757 second eval script run done 1.85607099533 xpath 1 1.94704914093 xpath 2 2.03846216202 xpath 3 2.13886809349 xpath of button clicked 2.26395010948 button clicked 2.27277112007 second page visited 3.30618906021 loop 1 else 3.38708400726 xpath 4 3.46828198433 xpath 5 3.54840707779 xpath 6 3.63034701347 xpath 7 3.7106590271 range begins 3.75155210495 3 range ends 5.91926407814
even now, each step in range
loop taking 0.7 seconds i'm aiming reduce further.
Comments
Post a Comment