jandieman
/
searxng
mirror of https://github.com/searxng/searxng.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
3,679 Commits 7 Branches 17 Tags 123 MiB
Commit Graph

132 Commits

Author SHA1 Message Date
Alexandre Flament 28d51fd063 [mod] timeout in log are readable (the timeouts are compare to the start_time of the request). 2016-12-30 17:37:46 +01:00
Alexandre Flament 3a8ab5880a [mod] result_container.extend sets result['engine'] = engine_name for each result 2016-12-30 17:18:02 +01:00
Alexandre Flament 0a2fde19d0 [mod] add a search_one_request_safe function wrapper to call search_one_request. All exceptions are catched in this wrapper. 2016-12-30 11:51:49 +01:00
marc 7fba3d1179 Merge branch 'master' into languages 2016-12-16 22:14:36 -06:00
Adam Tauber cf5c7d050a [fix] engine setting if no categories specified 2016-12-15 11:59:21 +01:00
marc 1b46ecd21a [fix] broken language selector 2016-12-13 21:36:40 -06:00
Adam Tauber 75aa46a6b3 [fix] remove unused variable 2016-12-10 00:15:53 +01:00
Adam Tauber 0205838f25 [enh] handle engine response crashes 2016-12-10 00:15:33 +01:00
Alexandre Flament e48f07a367 Merge branch 'master' into searchpy2 2016-12-09 23:11:45 +01:00
Adam Tauber 971ed0abd1 [enh] add quick answer functionality with an example answerer 2016-11-19 20:53:51 +01:00
Alexandre Flament 219f047bf3 Merge branch 'master' into searchpy2 2016-11-06 09:44:07 +01:00
Adam Tauber 7d26b3f8f8 [fix] suspend end time check 2016-11-06 02:53:09 +01:00
Alexandre Flament 01e2648e93 Simplify search.py, basically updated PR #518 
The timeouts in settings.yml is about the total time (not only the HTTP request but also the prepare the request and parsing the response)
It was more or less the case before since the threaded_requests function ignores the thread after the timeout even the HTTP request is ended.

New / changed stats :
* page_load_time : record the HTTP request time
* page_load_count: the number of HTTP request
* engine_time : the execution total time of an engine
* engine_time_count : the number of "engine_time" measure

The avg response times in the preferences are the engine response time (engine_load_time / engine_load_count)

To sum up :
* Search.search() filters the engines that can't process the request
* Search.search() call search_multiple_requests function
* search_multiple_requests creates one thread per engine, each thread runs the search_one_request function
* search_one_request calls the request function, make the HTTP request, calls the response function, extends the result_container
* search_multiple_requests waits for the the thread to finish (or timeout)
 2016-11-05 13:45:20 +01:00
Alexandre Flament 58a6c045c8 [mod] replace references request.request_data (GET or POST parameters) by request.form (based on merge of POST and GET parameters) 2016-11-02 14:52:22 +01:00
dalf fbb080f358 Change plugin API : 
- pre_search(request, search)
- post_search(request, search)
- on_result(request, search, result)

with
- request is the Flask request
- search a searx.Search instance
- result a searx result as usual
 2016-11-02 14:22:16 +01:00
dalf 67e11c42b9 Clean up the architecture 
Purposes :
- isolate the plugins calls
- distinction between parsing the web request and running the search (Search class). To be able to test code easily, to run searx code outside a web server, to filter the search query parameters with plugins more easily, etc...

Details :
- request.request_data contains request.form or request.args (initialize inside pre_request() function)
- Query class is renamed RawTextQuery
- SearchQuery class defines all search parameters
- get_search_query_from_webapp create a SearchQuery instance (basically the previous Search.__init__ code)
- Search class and SearchWithPlugins class takes a SearchQuery instance as class constructor parameter
- SearchWithPlugins class inherites from Search class, and run plugins
- A dedicated function search_with_plugins executes plugins to have a well define locals() (which is used by the plugins code).
- All plugins code is executed inside the try...except block (webapp.py, index function)
- advanced_search HTTP parameter value stays in webapp.py (it is only part of UI)
- multiple calls to result_container.get_ordered_results() doesn't compute the order multiple time (note : this method was call only once before)
- paging value is stored in the result_container class (compute in the extend method)
- test about engine.suspend_end_time is done during search method call (instead of __init__)
- check that the format parameter value is one of these : html, rss, json, rss (before the html value was assumed but some text formatting wasn't not done)
 2016-11-02 14:22:16 +01:00
Adam Tauber 55abf07a4f [fix] thread safe threaded_requests() function 2016-09-06 00:36:33 +02:00
Adam Tauber ffbee4bb82 [fix] collect garbage after searches to reduce memory footprint 2016-07-31 23:39:58 +02:00
Noemi Vanyi 16a8641649 advanced-search: panel is visible when checked before 2016-07-25 23:30:48 +02:00
Noemi Vanyi 93c0c49e9a add time range search with yahoo 2016-07-25 23:19:46 +02:00
Noemi Vanyi 5867e97966 more follow ups in new preferences 2016-04-09 19:49:59 +02:00
Noemi Vanyi 8c4db08443 rename blocked engines to diabled engines 2016-04-09 19:49:59 +02:00
Noemi Vanyi fe691a0988 new preferences handling 
Preferences class was introduced in order to handle user preferences. Right now
it parses cookies and the form in preferences. Also it can retrieve settings
based on the name of the setting.

ATTENTION
Please note that engine preferences are handled differently from now on. So it
introduces incompatible changes. Every user who has saved preferences should reset and
save his/her settings again.

This change was needed, because everytime a default disabled engine was
added saved user preferences would broke. Now engine setting tracking is
fixed.
 2016-04-09 01:08:44 +02:00
Adam Tauber 10947536aa [enh] suspend engines after error 
The duration is based on the number of continuous errors, but maximized in one minute
 2016-02-20 00:21:56 +01:00
Adam Tauber 786fe49fab Merge pull request #438 from Cqoicebordel/error_page_format 
Don't throw away the request if pageno is wrong
 2015-12-01 15:50:47 +01:00
Adam Tauber b6c3cb0bdd [enh][mod] result handling refactor 
Several changes has been made:
 - Parallel result merge
 - Scoring algorithm slightly changed (see result_score())
 - Proper Thread locking on global data manipulation
 2015-10-03 17:26:07 +02:00
Cqoicebordel ed1daa32ea Don't throw away the request if pageno is wrong 
Fix #424 but maybe only partially
 2015-09-18 20:44:04 +02:00
Adam Tauber 37c3ace309 [fix] add missing scheme to duplicated results too ++ revert gigablasts handling 2015-09-11 18:33:06 +02:00
Adam Tauber f094188780 [fix] display categories of the selected engines 2015-09-01 15:21:17 +02:00
Adam Tauber 8c4d9b79d5 [fix] engine selection from url 2015-09-01 15:14:02 +02:00
Adam Tauber 42ac2c5b56 [fix] check empty engine language attribute - fixes subtitleseeker 2015-08-25 22:02:18 +02:00
Thomas Pointhuber 5bffa9ca33 [fix] rewrite scheme to http if there is no one, FIX #390 2015-08-24 11:18:58 +02:00
Adam Tauber 7c9f931baf [fix] type error 2015-08-02 20:41:44 +02:00
Adam Tauber 43cd8e0c41 [enh] default settings option to safe_search - #396 2015-08-02 20:32:32 +02:00
Cqoicebordel 00668dd0a1 New feature : language per search engine ! 2015-06-03 17:16:12 +02:00
Adam Tauber b09aaf126e [mod] allow whitespaces in category/engine list 2015-05-19 11:16:57 +02:00
Adam Tauber b6de23f540 [enh] cleaner category/engine selection from GET/POST param - closes #289 2015-05-12 08:21:39 +02:00
Adam Tauber d1b12f9016 [mod] search return value 2015-03-08 23:30:31 +01:00
Adam Tauber bf09daa31d [enh] log redirects 2015-02-13 13:02:37 +01:00
dalf 5029cb4787 [fix] close #197 for all engines 2015-02-13 11:17:00 +01:00
Thomas Pointhuber e7de9674b1 [enh] add safesearch functionality 
2 = strict
1 = moderate
0 = none
 2015-02-08 21:53:37 +01:00
Adam Tauber 1de781a143 [enh] category specific engine deactivation - closes #205 2015-02-03 18:37:38 +01:00
Adam Tauber b19e681cce [enh] default disabled engines - closes #109 2015-01-31 23:11:45 +01:00
dalf d07cfd9089 [enh] use one single http connection pool : improve response time. close #100 2015-01-21 11:33:16 +01:00
Thomas Pointhuber 5bfaabeb6b [enh] add nojs support into oscar-template 2015-01-15 15:48:50 +01:00
Adam Tauber b47b8cc2c6 Merge pull request #166 from Cqoicebordel/non-exclusif-bang 
Add non exclusif bang
 2015-01-12 14:14:40 +01:00
Adam Tauber 23b4a53248 [enh] engine exception traceback 2015-01-09 04:30:55 +01:00
Adam Tauber 299a80a1eb [enh] using the logger 2015-01-09 04:13:05 +01:00
Cqoicebordel 96c4d52eef Add non exclusif bang 
Allow to perform a search while adding an engine (or a category) without adding it "officially" to the request.
'?' is used to add an engine without modifying anything else to the request. For example, you can perform a search in the 'general' category, and if you add '?tw' the result from Twitter will be added to the originals results.
 2015-01-03 02:31:23 +01:00
Cqoicebordel 5d977056f7 Flake8 and Twitter corrections 
Lots of Flake8 corrections
Maybe we should change the rule to allow lines of 120 chars. It seems more usable.

Big twitter correction : now it outputs the words in right order...
 2014-12-29 21:31:04 +01:00
Adam Tauber 1c969186bf [mod] better search request exception handling 2014-12-19 20:01:01 +01:00
Adam Tauber 8c05c00762 [fix] handle empty search requests 2014-12-19 19:40:40 +01:00
Adam Tauber 2b7f3e28e7 [mod] display timeoutted engine name 2014-12-19 13:59:41 +01:00
Adam Tauber 789e46f1c8 [fix] timeout and response parsing order 2014-12-18 12:07:20 +01:00
Adam Tauber 77c3a27f56 [mod] strict timeout handling 2014-12-18 10:11:56 +01:00
Thomas Pointhuber 70fc1002db [fix] fix kickass engine 
thanks @Cqoicebordel in #144:
https://github.com/asciimoo/searx/pull/144#issuecomment-67036903
 2014-12-15 19:37:58 +01:00
Adam Tauber d93d703703 [fix] use queues to answers/suggestions/infoboxes 2014-12-14 02:15:04 +01:00
Adam Tauber a5324d9d82 [fix] proper search timeout handling 2014-12-14 01:18:01 +01:00
Adam Tauber 8e6ae188b8 [mod] search thread names 2014-12-08 23:55:11 +01:00
Adam Tauber d959cb1c05 [enh] gevent/grequests changed to the built-in threading lib 2014-12-05 19:24:11 +01:00
Thomas Pointhuber 5a7f85ecee [fix] #121, HTTP 500 Error if res["content"] == None 2014-11-01 17:32:26 +01:00
Adam Tauber 5740cfbf1c [fix] pep8 part II. 2014-10-19 12:41:04 +02:00
dalf 599c1137bb [fix] the bang was included in the search string 2014-10-11 12:46:12 +02:00
dalf 829c758970 Merge remote-tracking branch 'upstream/master' 2014-10-01 22:25:11 +02:00
Dalf 0a71525ab6 [enh] add infoboxes and answers (clean up) 2014-10-01 22:17:35 +02:00
Thomas Pointhuber 62d1a70c84 using Query class for parsing of search query 2014-10-01 17:57:53 +02:00
Dalf 6bfd566353 [enh] add infoboxes and answers 2014-09-28 16:51:41 +02:00
Dalf e39d9fe542 update comment 2014-09-22 23:39:21 +02:00
Dalf 6b058962e1 [fix] when two results are merged, really use the content with more text 2014-09-22 22:55:51 +02:00
Adam Tauber bd2db71fa6 Merge branch 'comments' of https://github.com/pointhi/searx 
Conflicts:
	searx/search.py
 2014-09-22 21:40:40 +02:00
Dalf 5d1610d87a [enh] stick results from the same category and template 2014-09-14 14:39:22 +02:00
Thomas Pointhuber 22da73b8bb little update update comments in search.py 2014-09-13 18:44:11 +02:00
Thomas Pointhuber c9bab0e833 add comments to search.py 
* add comments
* add licence-header
 2014-09-13 18:25:25 +02:00
Adam Tauber b0ba367a1a [enh][mod] search refactor 2014-07-07 13:59:27 +02:00
asciimoo c1d7d30b8e [mod] len() removed from conditions 2014-02-11 13:13:51 +01:00
asciimoo 9ede9b955d [fix] language query string comparison 2014-02-09 11:35:52 +01:00
asciimoo 7c031a3ea6 [enh] better search language selection from query 2014-02-09 11:12:58 +01:00
asciimoo d11a873d40 [enh] chainable query modifiers 2014-02-09 01:23:01 +01:00
asciimoo a641971a6f [enh] language selection from query 2014-02-09 01:22:30 +01:00
asciimoo 7f45e8a562 [mod] query parsing refactor vol.1 2014-02-09 01:07:18 +01:00
asciimoo 9eb31c0186 [fix] unicode and pep8 fixes 2014-02-07 02:45:12 +01:00
asciimoo 0540ea9ee2 [enh][mod] code refactor ++ search engine filtering 2014-02-07 01:19:07 +01:00