Python request.post() not working when converted to Python Scrapy Request -
i have simple post request code.
headers = { dictionary of headers } params = ( ('param1', '0'), ('param2', '5668294380'), ('param3', '8347915011'), ) response = requests.post('https://website.com', headers=headers, params=params, data=__data) this works standalone python program.
but want in python scrapy
request(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data, method="post") it gives me response url cannot handle post request
i tried
formrequest(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data) it gives me same response.
i tried
request(url='https://website.com?' + urllib.urlencode(self.params),callback=self.callback_fun, headers=headers, body=__data, method="post") but gives me 400 bad request
whats wrong scrapy? mean pure python script works in scrapy not work.
i think main issue how send params=params using scrapy. scrapy allows send request payload via body parameter
class scrapy.http.formrequest(url[, formdata, ...]) parameters: formdata (dict or iterable of tuples) – dictionary (or iterable of (key, value) tuples) containing html form data url-encoded , assigned body of request.
in http, if want post data, data set in request body , encoded. can encode dict self or use scrapy formrequest:
class formrequest(request): def __init__(self, *args, **kwargs): formdata = kwargs.pop('formdata', none) if formdata , kwargs.get('method') none: kwargs['method'] = 'post' super(formrequest, self).__init__(*args, **kwargs) if formdata: items = formdata.items() if isinstance(formdata, dict) else formdata # encode dict here querystr = _urlencode(items, self.encoding) if self.method == 'post': # set message header self.headers.setdefault(b'content-type', b'application/x-www-form-urlencoded') # set message body self._set_body(querystr) else: self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr) ----------------------------update--------------
in requests code:
response = requests.post('https://website.com', headers=headers, params=params, data=__data) it first adds parameter url post data modified url. should change url. can url by:
print(response.url)
Comments
Post a Comment