Python request.post() not working when converted to Python Scrapy Request -
i have simple post request code.
headers = { dictionary of headers } params = ( ('param1', '0'), ('param2', '5668294380'), ('param3', '8347915011'), ) response = requests.post('https://website.com', headers=headers, params=params, data=__data)
this works standalone python program.
but want in python scrapy
request(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data, method="post")
it gives me response url cannot handle post request
i tried
formrequest(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data)
it gives me same response.
i tried
request(url='https://website.com?' + urllib.urlencode(self.params),callback=self.callback_fun, headers=headers, body=__data, method="post")
but gives me 400 bad request
whats wrong scrapy? mean pure python script works in scrapy not work.
i think main issue how send params=params
using scrapy. scrapy allows send request payload via body
parameter
class scrapy.http.formrequest(url[, formdata, ...])
parameters: formdata (dict or iterable of tuples) – dictionary (or iterable of (key, value) tuples) containing html form data url-encoded , assigned body of request.
in http, if want post data, data set in request body , encoded. can encode dict self or use scrapy formrequest
:
class formrequest(request): def __init__(self, *args, **kwargs): formdata = kwargs.pop('formdata', none) if formdata , kwargs.get('method') none: kwargs['method'] = 'post' super(formrequest, self).__init__(*args, **kwargs) if formdata: items = formdata.items() if isinstance(formdata, dict) else formdata # encode dict here querystr = _urlencode(items, self.encoding) if self.method == 'post': # set message header self.headers.setdefault(b'content-type', b'application/x-www-form-urlencoded') # set message body self._set_body(querystr) else: self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr)
----------------------------update--------------
in requests code:
response = requests.post('https://website.com', headers=headers, params=params, data=__data)
it first adds parameter url post data modified url. should change url. can url by:
print(response.url)
Comments
Post a Comment