python - Web Scraping, unable to scroll down web page using selenium web driver -


i trying extract links forum (https://www.pakwheels.com/forums/c/travel-n-tours) scrapper class stops after scrolling down once.

from bs4 import beautifulsoup  sourceurl='https://www.pakwheels.com/forums/c/travel-n-tours'  #----------------------------------source of below code:http://stackoverflow.com/questions/32391303/how-to-scroll-to-the-end-of-the-page-using-selenium-in-python--------------------# #----------------------- scrolling bottom of page ----------------------------- ----------#  selenium import webdriver import time chrome_path=r"c:\users\shani\desktop\chromedriver.exe" driver=webdriver.chrome(chrome_path) driver.get(sourceurl) updatedlenofpage = driver.execute_script("window.scrollto(0, document.body.scrollheight);var lenofpage=document.body.scrollheight;return lenofpage;") scrollcomplete=false while(scrollcomplete==false):         currentlenofpage = updatedlenofpage         updatedlenofpage = driver.execute_script("window.scrollto(0, document.body.scrollheight);var lenofpage=document.body.scrollheight;return lenofpage;")         print('scrolling down')         time.sleep(5)         if currentlenofpage==updatedlenofpage:             scrollcomplete=true time.sleep(10) pagesource=driver.page_source  # ------------------------------------- getting links ---------------------------------- # soup = beautifulsoup(pagesource, 'lxml') # print(soup)  blogurls=[] url in soup.find_all('a'):     if((url.get('href').find('/forums/t/')!=-1) , (url.get('href').find('about-the-travel-n-tours-category')==-1) , (url.get('href').find('/forums/t/topic/')==-1)):         blogurls.append(url.get('href'))         print(url.get('href'))        print(len(blogurls))

it gives following error

traceback (most recent call last):   file "d:\liclipsworkspace\nlktlib\scrapping\scrolling.py", line 32, in <module>     if((url.get('href').find('/forums/t/')!=-1) , (url.get('href').find('about-the-travel-n-tours-category')==-1) , (url.get('href').find('/forums/t/topic/')==-1)): attributeerror: 'nonetype' object has no attribute 'find'

please help

you don't need selenium, can links json response. code gets urls first 5 pages (for getting pages change last 5 264).

import requests  in range(0, 5):     r = requests.get(         'https://www.pakwheels.com/forums/c/travel-n-tours/l/latest.json?page={}'.format(i)).json()     topics = r['topic_list']['topics']     topic in topics:         print ('https://www.pakwheels.com/forums/t/{}/{}'.format(topic['slug'], topic['id']))

Comments

Post a Comment

Popular posts from this blog

How to understand 2 main() functions after using uftrace to profile the C++ program? -

i trying uftrace profile following simple c++ program: #include <iostream> class { public: a() {std::cout << "a created" << std::endl;} ~a() {std::cout << "a destroyed" << std::endl;} }; int main() { a; return 0; } the profile result this: # uftrace a.out created destroyed # duration tid function 2.026 [ 4828] | __cxa_atexit(); [ 4828] | main() { [ 4828] | __static_initialization_and_destruction_0() { 89.397 [ 4828] | std::ios_base::init::init(); 0.768 [ 4828] | __cxa_atexit(); 93.029 [ 4828] | } /* __static_initialization_and_destruction_0 */ 94.425 [ 4828] | } /* main */ [ 4828] | main() { [ 4828] | a::a() { 11.104 [ 4828] | std::operator <<(); 10.825 [ 4828] | std::basic_ostream::operator <<(); 24.514 [ 4828] | } /* a::a */ [ 4828] | a::~a() { 0.978 [ 4828] | ...
Read more

c# - Update a combobox from a presenter (MVP) -

i using mvp in project, new in mvp. have 2 comboboxes. when select option in combobox, combobox should filled new data. this action in presenter. view 'view1' in presenter, , introduced combobox1 , combobox2 properties in 'view1', because need 'datasource', 'displaymember', 'valuemember' , 'refresh()' in method below. but, when using pattern, enough send property like public string combobox2 { { return combobox1.selectedvalue.tosstring(); } } into presenter not whole combobox. how can solve problem? public void onselectedindexchangedcombobox1() { if (view1.combobox1.selectedindex == -1) { return; } datatable dt = tools.getdatatable("a path"); var query = (from o in dt.asenumerable() o.field<string>("afield") == farmerview.combobox1.selectedvalue.tostring() orderby o.field<string>("anotherfield") select...
Read more

How to put a lock and transaction on table using spring 4 or above using jdbcTemplate and annotations like @Transactional? -

i trying put lock on table while writing on table , if happened in between roll-back . trying convert below code lock table test_g1 read; lock table test_g write; -- begin; start transaction; insert test_g1 values(143); insert test_g values(145); select * test_g1; select * test_g; rollback; select * test_g; unlock tables; how convert above code @transactional spring jdbctemplate code? @transactional(rollbackfor={dataaccessexception.class}) public void test(){ jdbctemplate.execute("insert test1 (id, nam) values (4, 'a')"); throw new dataaccessexception("error") { }; } here trying throw error, insert statement should rollback it's not happening . thanks edit-1 attaching code , doing in jdbcdaoimpl.java , mentioned problem comment above test() . app.java package com.cgiri.javabrains.spring4; import org.springframework.beans.factory.annotation.autowired; import org.springframework.context.applic...
Read more