Python request.post() not working when converted to Python Scrapy Request -


i have simple post request code.

headers = {     dictionary of headers }  params = (     ('param1', '0'),     ('param2', '5668294380'),     ('param3', '8347915011'), )  response = requests.post('https://website.com', headers=headers, params=params, data=__data) 

this works standalone python program.

but want in python scrapy

request(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data, method="post") 

it gives me response url cannot handle post request

i tried

formrequest(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data) 

it gives me same response.

i tried

request(url='https://website.com?' + urllib.urlencode(self.params),callback=self.callback_fun, headers=headers, body=__data, method="post") 

but gives me 400 bad request

whats wrong scrapy? mean pure python script works in scrapy not work.

i think main issue how send params=params using scrapy. scrapy allows send request payload via body parameter

class scrapy.http.formrequest(url[, formdata, ...]) 

parameters: formdata (dict or iterable of tuples) – dictionary (or iterable of (key, value) tuples) containing html form data url-encoded , assigned body of request.

in http, if want post data, data set in request body , encoded. can encode dict self or use scrapy formrequest:

class formrequest(request):  def __init__(self, *args, **kwargs):     formdata = kwargs.pop('formdata', none)     if formdata , kwargs.get('method') none:         kwargs['method'] = 'post'      super(formrequest, self).__init__(*args, **kwargs)      if formdata:         items = formdata.items() if isinstance(formdata, dict) else formdata         # encode dict here         querystr = _urlencode(items, self.encoding)         if self.method == 'post':             # set message header             self.headers.setdefault(b'content-type', b'application/x-www-form-urlencoded')             # set message body             self._set_body(querystr)         else:             self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr) 

----------------------------update--------------

in requests code:

response = requests.post('https://website.com', headers=headers, params=params, data=__data) 

it first adds parameter url post data modified url. should change url. can url by:

print(response.url) 

Comments

Popular posts from this blog

Command prompt result in label. Python 2.7 -

javascript - How do I use URL parameters to change link href on page? -

amazon web services - AWS Route53 Trying To Get Site To Resolve To www -