python - Clear cookies on scrapy completely instead of changing them -


i noticed blocked while scraping because of session cookie being used on many pages. there way clear cookies during crawling initial state of crawler?

facing similar situation myself. can away here, 1 idea have subclass cookiemiddleware , write method tweak jar variable directly. it's dirty, maybe worth consideration.

another option write feature request @ least have function clear cookies. can take year implement, if deemed needed @ all, don't particularly trust scrapy devs here.

just occured me, can use own cookiejar meta, , if want return clean state, use different value (something incrementing integer do).


Comments

Post a Comment

Popular posts from this blog

javascript - Confirm a form & display message if form is valid with JQuery -

i'm developing django application , use javascript in order improve website. i have form , display 2 things : confirm form before submit if form submitted, display message : 'form saved' it's first time i'm using js , need make process. this code : <form class = "form" method='post' action=''> {% csrf_token %} <br></br> {{ form.as_p}} <br></br> <button type="submit">valider</button> </form> <script> $('#form').submit(function() { var c = confirm("click ok continue?"); return c; //you can return c because true or false }); </script> and if form valid , saved : <script type="text/javascript" > $(document).on('valider', 'form.form', function(form) { var $form = $(form); $.ajax({ ...
Read more

Retrieving ETA (estimated time of arrival) with Google Distance Matrix API and public transit as transport mode -

Image
is possible retrieve eta google's distancematrix api (or alternative api). fyi, eta different duration, when using public transport. following example clarify request. when leave on friday @ 8.15 i'll able catch train @ 8.30 , arrive @ 9.00 am. google says duration 30 minutes. however, when leave on saturday @ 8.15 i'll able catch first train @ 9.00 , arrive @ 9.30 am. google still says duration 30 minutes, means ignore delay. google map supports functionality there api around giving me eta instead of duration. can see, distance-matrix' response no good. clarifying question little. eta estimated time of arrival , it's clear user wants know entire duration of transport (including waiting time public transport).
Read more

ionic framework - Meteor - Error: Failed to execute 'insertBefore' on 'Node' -

i trying make dynamic list display using meteor, idea load more elements each time bottom of page reached. the html template following : <template name="contacts"> {{#contentfor "headertitle"}} <div class="bar bar-header bar-assertive"> <h1 class="title">contacts :</h1> </div> {{/contentfor}} {{#ionview}} {{#ioncontent}} <div class="list"> {{#each contacts}} <a class="item item-avatar" href="/contacts/{{_id}}"> <img src="/img/ico_client.png"> <h2>{{full_name}}</h2> <p>{{company}}</p> </a> {{/each}} </div> {{#if moreresults}} <div id="showmoreresults" style="margin-left: 25px;"> <span class="loading">loading...</span> </div> {{/if}} {{/ioncontent}} {{/ionview}} </te...
Read more