Compare commits

...

3 Commits

Author SHA1 Message Date
Grant Lanham Jr 184b34b596
Merge 23bf9f9d95 into dfaf5868e2 2024-11-11 07:50:12 +01:00
Grant Lanham 23bf9f9d95 add base http params 2024-04-25 22:58:19 -04:00
Grant Lanham 2b48584bb4 Fix bing page numbering, add sc to parameter, minor refactor
Bing page numbering doesn't increase by 10 each time. The first page returns 10 results, and all pages thereafter return 14 results. This means we need to update the page numbering

Next, the 'sc' parameter, whatever it means, needs to be present in order to not return the same results.

Finally, the code to check the page had some duplicate checks, so I refactored the code in this section which is low-risk.
2024-04-23 21:03:51 -04:00
1 changed files with 12 additions and 5 deletions

View File

@ -95,17 +95,24 @@ def request(query, params):
# don't ask why it is only sometimes / its M$ and they have never been # don't ask why it is only sometimes / its M$ and they have never been
# deterministic ;) # deterministic ;)
'pq': query, 'pq': query,
# TODO: Figure out how below parameters are populated
'sc': '0-0',
"sp": "-1",
"lq": "0",
"qs": "n",
"ghsh": "0",
"ghacc": "0",
"ghpl": "",
} }
# To get correct page, arg first and this arg FORM is needed, the value PERE # To get correct page, arg first and this arg FORM is needed, the value PERE
# is on page 2, on page 3 its PERE1 and on page 4 its PERE2 .. and so forth. # is on page 2, on page 3 its PERE1 and on page 4 its PERE2 .. and so forth.
# The 'first' arg should never send on page 1. # The 'first' arg should never send on page 1.
if page > 1: if page > 1:
query_params['first'] = _page_offset(page) # see also arg FORM query_params['first'] = _page_offset(page) # see also arg FORM
if page == 2: if page == 2:
query_params['FORM'] = 'PERE' query_params['FORM'] = 'PERE'
elif page > 2: else: # page > 2:
query_params['FORM'] = 'PERE%s' % (page - 2) query_params['FORM'] = 'PERE%s' % (page - 2)
params['url'] = f'{base_url}?{urlencode(query_params)}' params['url'] = f'{base_url}?{urlencode(query_params)}'