重构项目目录结构

.github/workflows/test.yml

@@ -32,4 +32,4 @@ jobs:
     - name: Test with example
       run: |
         pip install coverage
-        coverage run oneforall/example.py
+        coverage run test/example.py

.travis.yml

@@ -47,7 +47,7 @@ install:
   - pip3 install -r requirements.txt
 
 script:
- - coverage run oneforall/example.py
+ - coverage run test/example.py
 
 after_success:
  - codecov

Dockerfile

@@ -10,7 +10,7 @@ RUN git clone https://github.com/blechschmidt/massdns
 WORKDIR /massdns
 RUN make 
 ADD . /OneForAll/
-RUN mv /massdns/bin/massdns /OneForAll/oneforall/thirdparty/massdns/massdns_linux_x86_64
+RUN mv /massdns/bin/massdns /OneForAll/thirdparty/massdns/massdns_linux_x86_64
-WORKDIR /OneForAll/oneforall
+WORKDIR /OneForAll/
 
 ENTRYPOINT ["/bin/ash"]

README.en.md

@@ -50,7 +50,7 @@ At present, OneForAll is still under development, there must be a lot of problem
 * **Support subdmain verification**，default to enable subdomain verification, automatically resolve subdomain DNS, automatically request subdomain to obtain title and banner, and comprehensively determine subdomain survival.
 * **Support subdomain takeover**，By default, subdomain takeover risk checking is enabled. Automatic subdomain takeover is supported (only Github, remains to be improved at present), and batch inspection is supported.
 * **Powerful processing feature**，The found subdomain results support automatic removal, automatic DNS parsing, HTTP request detection, automatic filtering of valid subdomains, and expansion of Banner information for subdomains. The final supported export formats are `rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`.
-* **Very fast**，[collection module](https://github.com/shmilylty/OneForAll/tree/master/oneforall//collect.py) uses multithreaded calls, [blasting module](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps under the default configuration, and DNS parsing and HTTP requests use asynchronous multiprogramming in subdomain verification. Multithreaded check [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takeover.py) risk.
+* **Very fast**，[collection module](https://github.com/shmilylty/OneForAll/tree/master/collect.py) uses multithreaded calls, [blasting module](https://github.com/shmilylty/OneForAll/tree/master/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps under the default configuration, and DNS parsing and HTTP requests use asynchronous multiprogramming in subdomain verification. Multithreaded check [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/takeover.py) risk.
 * **Good experience**，Each module has a progress bar, and the results of each module are saved asynchronously.
 
 If you have any other great ideas, please let me know!😎
@@ -95,7 +95,6 @@ pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages
    cd OneForAll/
    python -m pip install -U pip setuptools wheel
    pip3 install -r requirements.txt
-   cd oneforall/
    python oneforall.py --help
 ```
 ​      For other system platforms, please refer to [dependency installation](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md). If you find that compiling a dependent library fails during the installation dependencies, Refer to the solution in the [troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md) documentation, if not resolved, welcome feedback.
@@ -130,7 +129,6 @@ docker run -it oneforall
 
 1. If you are installing dependencies through pip3, run the example using the following command: 
     ```bash
-    cd oneforall/
     python3 oneforall.py --target example.com run
     ```
 
@@ -138,7 +136,6 @@ docker run -it oneforall
 
 2. If you install dependencies through pipenv, run the example using the following command: 
    ```bash
-   cd oneforall/
    pipenv run python oneforall.py --target example.com run
    ```
 
@@ -166,11 +163,11 @@ A table like `example_com_now_result` stores the collection results of the curre
 
 **🤔Help**
 
-The command line parameters only provide some common parameters. For more detailed parameter configuration, please see [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) if you think Some parameters are frequently used in the command interface or missing parameters. Feedback is welcome. For well-known reasons, if you want to use some of the wall's collection interface, please go to [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) to configure the proxy, some collection Modules need to provide APIs (most of which are freely available for registered accounts). If you need to use them, please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py) to configure the API. Information, if not used, please ignore the error message. (For detailed modules, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
+The command line parameters only provide some common parameters. For more detailed parameter configuration, please see [config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py) if you think Some parameters are frequently used in the command interface or missing parameters. Feedback is welcome. For well-known reasons, if you want to use some of the wall's collection interface, please go to [config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py) to configure the proxy, some collection Modules need to provide APIs (most of which are freely available for registered accounts). If you need to use them, please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/api.py) to configure the API. Information, if not used, please ignore the error message. (For detailed modules, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
 
 The OneForAll command line interface is based on [Fire](https://github.com/google/python-fire/). For more advanced usage of Fire, please refer to [using the Fire CLI](https://github.com/google/Python-fire/blob/master/docs/using-cli.md), if you have any doubts during the use, please feel free to give me feedback.
 
-[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py) is the main program entry, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py) and [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py) and other modules, in order to facilitate the sub-field blasting, brute.py is isolated independently, in order to facilitate the subdomain takeover risk check independently takeover.py, in order to facilitate the database export independently dbexport.py, these modules can be run separately, and the parameters accepted are more abundant, if you want to use these modules separately, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.en.md).
+[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py) is the main program entry, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py) and [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py) and other modules, in order to facilitate the sub-field blasting, brute.py is isolated independently, in order to facilitate the subdomain takeover risk check independently takeover.py, in order to facilitate the database export independently dbexport.py, these modules can be run separately, and the parameters accepted are more abundant, if you want to use these modules separately, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.en.md).
 
 ❗ Note: When you encounter some problems or doubts during use, please use [Issues](https://github.com/shmilylty/OneForAll/issues) to search for answers. Also see [Q&troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&troubleshooting.md).
 
@@ -242,41 +239,7 @@ The OneForAll command line interface is based on [Fire](https://github.com/googl
    ```
 
 ## 🌲Directory structure
-
+For the description of the project's directory structure, please refer to [directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md).
-```bash
-D:.
-|
-+---.github
-+---docs
-|       collection_modules.md collection module description
-+---images
-\---oneforall
-    |   brute.py   Asynchronous multi-process multi-correlation subdomain blasting module, can be run separately
-    |   api.py        API configuration of some collection modules
-    |   collect.py    Upper layer call of each collection module
-    |   config.py     Configuration file
-    |   dbexport.py   Database export module, can be run separately
-    |   domains.txt   List of domain names to be blasted
-    |   oneforall.py  OneForAll main entrance, can be run separately
-    |   __init__.py
-    |
-    +---common Common call module
-    +---data   Store some of the required data
-    |       next_subdomains.txt     Next level subdomain dictionary
-    |       public_suffix_list.dat  Top-level domain name suffix
-    |       srv_names.json          Common SRV record prefix name
-    |       subdomains.txt          Common Dictionary of subdomain blasting
-    |
-    \---modules 
-        +---certificates     Using certificate transparency to collect subdomain
-        +---check            Using conventional methods to collect subdomain
-        +---crawl            Using web crawler files to collect subdomain
-        +---datasets         Using DNS datasets to collect subdomain
-        +---dnsquery         Using DNS query to collect subdomain
-        +---intelligence     Using threat intelligence platform to collect subdomain
-        \---search           Using search engine to collect subdomain
-
-```
 
 Description of the source of the subdomain dictionary::
 

README.md

@@ -50,7 +50,7 @@
 * **支持子域验证**，默认开启子域验证，自动解析子域DNS，自动请求子域获取title和banner，并综合判断子域存活情况。
 * **支持子域接管**，默认开启子域接管风险检查，支持子域自动接管（目前只有Github，有待完善），支持批量检查。
 * **处理功能强大**，发现的子域结果支持自动去除，自动DNS解析，HTTP请求探测，自动筛选出有效子域，拓展子域的Banner信息，最终支持的导出格式有`rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`。
-* **速度极快**，[收集模块](https://github.com/shmilylty/OneForAll/tree/master/oneforall//collect.py)使用多线程调用，[爆破模块](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py)使用[massdns](https://github.com/blechschmidt/massdns)，默认配置下速度最少能达到10000pps，子域验证中DNS解析和HTTP请求使用异步多协程，多线程检查[子域接管](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takeover.py)风险。
+* **速度极快**，[收集模块](https://github.com/shmilylty/OneForAll/tree/master/collect.py)使用多线程调用，[爆破模块](https://github.com/shmilylty/OneForAll/tree/master/brute.py)使用[massdns](https://github.com/blechschmidt/massdns)，默认配置下速度最少能达到10000pps，子域验证中DNS解析和HTTP请求使用异步多协程，多线程检查[子域接管](https://github.com/shmilylty/OneForAll/tree/master/takeover.py)风险。
 * **体验良好**，日志和终端输出全使用中文，各模块都有进度条，异步保存各模块结果。
 
 如果你有其他很棒的想法请务必告诉我！😎
@@ -94,7 +94,6 @@ pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python38\site-packages
    cd OneForAll/
    python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
    pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
-   cd oneforall/
    python oneforall.py --help
 ```
    其他系统平台的请参考[依赖安装](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md)，如果在安装依赖过程中发现编译某个依赖库失败时可以参考[troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md)中解决方法，如果还没有解决欢迎加群反馈。
@@ -122,7 +121,7 @@ docker run -it oneforall
 
 ```shell
 docker build -t oneforall .
-docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
+docker run -it --rm -v ~/results:/OneForAll/results oneforall
 ```
 结果会输出在本地目录`~/results`
 
@@ -130,7 +129,6 @@ docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
 
 1. 如果你是通过pip3安装的依赖则使用以下命令运行示例：   
     ```bash
-    cd oneforall/
     python3 oneforall.py --target example.com run
     ```
 
@@ -138,7 +136,6 @@ docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
 
 2. 如果你通过pipenv安装的依赖则使用以下命令运行示例：
    ```bash
-   cd oneforall/
    pipenv run python oneforall.py --target example.com run
    ```
 
@@ -166,11 +163,11 @@ docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
 
 **🤔使用帮助**
 
-命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）
+命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）
 
 OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实现，有关Fire更高级使用方法请参阅[使用Fire CLI](https://github.com/google/python-fire/blob/master/docs/using-cli.md)。
 
-[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py)是主程序入口，oneforall.py可以调用[brute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py)，[takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py)及[dbexport.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py)等模块，为了方便进行子域爆破独立出了brute.py，为了方便进行子域接管风险检查独立出了takerover.py，为了方便数据库导出独立出了dbexport.py，这些模块都可以单独运行，并且所接受参数要更丰富一点，如果要单独使用这些模块请参考[使用帮助](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.md)
+[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py)是主程序入口，oneforall.py可以调用[brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py)，[takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py)及[dbexport.py](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py)等模块，为了方便进行子域爆破独立出了brute.py，为了方便进行子域接管风险检查独立出了takerover.py，为了方便数据库导出独立出了dbexport.py，这些模块都可以单独运行，并且所接受参数要更丰富一点，如果要单独使用这些模块请参考[使用帮助](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.md)
 
 ❗注意：当你在使用过程中遇到一些问题或者疑惑时，请先到[Issues](https://github.com/shmilylty/OneForAll/issues)里使用搜索找找答案，还可以参阅[常见问题与回答](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&A.md)。
 
@@ -237,41 +234,7 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
    ```
 
 ## 🌲目录结构
-
+项目的目录结构说明请参阅[directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md)。
-```bash
-D:.
-|
-+---.github
-+---docs
-|       collection_modules.md 收集模块说明
-+---images
-\---oneforall
-    |   brute.py      子域爆破模块，可以单独运行
-    |   api.py        一些收集模块的API配置
-    |   collect.py    各个收集模块上层调用
-    |   config.py     配置文件
-    |   dbexport.py   数据库导出模块，可以单独运行
-    |   domains.txt   要批量爆破的域名列表
-    |   oneforall.py  OneForAll主入口，可以单独运行
-    |   __init__.py
-    |
-    +---common 公共调用模块
-    +---data   存放一些所需数据
-    |       next_subdomains.txt     下一层子域字典
-    |       public_suffix_list.dat  顶级域名后缀 
-    |       srv_names.json          常见SRV记录前缀名
-    |       subdomains.txt          子域爆破常见字典
-    |
-    \---modules 
-        +---certificates     利用证书透明度收集子域模块
-        +---check            常规检查收集子域模块
-        +---crawl            利用网上爬虫档案收集子域模块
-        +---datasets         利用DNS数据集收集子域模块
-        +---dnsquery         利用DNS查询收集子域模块
-        +---intelligence     利用威胁情报平台数据收集子域模块
-        \---search           利用搜索引擎发现子域模块
-
-```
 
 关于子域字典来源的说明：
 1. 开源子域收集工具中的部分高频子域名字字典。

brute.py

@@ -19,11 +19,11 @@ import tenacity
 from dns.exception import Timeout
 from dns.resolver import NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers
 
-import config
 import dbexport
 from common import utils
+from config import setting
 from common.module import Module
-from config import logger
+from config.log import logger
 
 
 @tenacity.retry(stop=tenacity.stop_after_attempt(3))
@@ -236,21 +236,21 @@ def collect_wildcard_record(domain, authoritative_ns):
 
 
 def get_nameservers_path(enable_wildcard, ns_ip_list):
-    path = config.brute_nameservers_path
+    path = setting.brute_nameservers_path
     if not enable_wildcard:
         return path
     if not ns_ip_list:
         return path
-    path = config.authoritative_dns_path
+    path = setting.authoritative_dns_path
     ns_data = '\n'.join(ns_ip_list)
     utils.save_data(path, ns_data)
     return path
 
 
 def check_dict():
-    if not config.enable_check_dict:
+    if not setting.enable_check_dict:
         return
-    sec = config.check_time
+    sec = setting.check_time
     logger.log('ALERT', f'你有{sec}秒时间检查爆破配置是否正确')
     logger.log('ALERT', f'退出爆破请使用`Ctrl+C`')
     try:
@@ -393,13 +393,13 @@ def check_ip_times(times):
     :param times: 子域IP出现次数
     :return: 判断结果
     """
-    if times > config.ip_appear_maximum:
+    if times > setting.ip_appear_maximum:
         return True
     return False
 
 
 def is_valid_subdomain(ip, ttl, times, wc_ips, wc_ttl):
-    ip_blacklist = config.brute_ip_blacklist
+    ip_blacklist = setting.brute_ip_blacklist
     if ip in ip_blacklist:  # 解析ip在黑名单ip则为非法子域
         return 0, 'IP blacklist'
     if all([wc_ips, wc_ttl]):  # 有泛解析记录才进行对比
@@ -418,9 +418,9 @@ def save_brute_dict(dict_path, dict_set):
 
 
 def delete_file(dict_path, output_paths):
-    if config.delete_generated_dict:
+    if setting.delete_generated_dict:
         dict_path.unlink()
-    if config.delete_massdns_result:
+    if setting.delete_massdns_result:
         for output_path in output_paths:
             output_path.unlink()
 
@@ -447,10 +447,10 @@ class Brute(Module):
     :param int process:      爆破进程数(默认1)
     :param int concurrent:   并发爆破数量(默认10000)
     :param bool word:        是否使用word模式进行爆破(默认False)
-    :param str wordlist:     word模式爆破使用的字典路径(默认使用config.py配置)
+    :param str wordlist:     word模式爆破使用的字典路径(默认使用setting.py配置)
     :param bool recursive:   是否使用递归进行爆破(默认False)
     :param int depth:        递归爆破的深度(默认2)
-    :param str nextlist:     递归爆破所使用的字典路径(默认使用config.py配置)
+    :param str nextlist:     递归爆破所使用的字典路径(默认使用setting.py配置)
     :param bool fuzz:        是否使用fuzz模式进行爆破(默认False)
     :param str place:        指定爆破位置(开启fuzz模式时必需指定此参数)
     :param str rule:         指定fuzz模式爆破使用的正则规则(开启fuzz模式时必需指定此参数)
@@ -469,15 +469,15 @@ class Brute(Module):
         self.source = 'Brute'
         self.target = target
         self.process_num = process or utils.get_process_num()
-        self.concurrent_num = concurrent or config.brute_concurrent_num
+        self.concurrent_num = concurrent or setting.brute_concurrent_num
         self.word = word
-        self.wordlist = wordlist or config.brute_wordlist_path
+        self.wordlist = wordlist or setting.brute_wordlist_path
-        self.recursive_brute = recursive or config.enable_recursive_brute
+        self.recursive_brute = recursive or setting.enable_recursive_brute
-        self.recursive_depth = depth or config.brute_recursive_depth
+        self.recursive_depth = depth or setting.brute_recursive_depth
-        self.recursive_nextlist = nextlist or config.recursive_nextlist_path
+        self.recursive_nextlist = nextlist or setting.recursive_nextlist_path
-        self.fuzz = fuzz or config.enable_fuzz
+        self.fuzz = fuzz or setting.enable_fuzz
-        self.place = place or config.fuzz_place
+        self.place = place or setting.fuzz_place
-        self.rule = rule or config.fuzz_rule
+        self.rule = rule or setting.fuzz_rule
         self.export = export
         self.alive = alive
         self.format = format
@@ -487,8 +487,8 @@ class Brute(Module):
         self.domain = str()  # 当前正在进行爆破的域名
         self.ips_times = dict()  # IP集合出现次数
         self.enable_wildcard = False  # 当前域名是否使用泛解析
-        self.wildcard_check = config.enable_wildcard_check
+        self.wildcard_check = setting.enable_wildcard_check
-        self.wildcard_deal = config.enable_wildcard_deal
+        self.wildcard_deal = setting.enable_wildcard_deal
         self.check_env = True
 
     def gen_brute_dict(self, domain):
@@ -546,8 +546,8 @@ class Brute(Module):
     def main(self, domain):
         start = time.time()
         logger.log('INFOR', f'正在爆破域名{domain}')
-        massdns_dir = config.third_party_dir.joinpath('massdns')
+        massdns_dir = setting.third_party_dir.joinpath('massdns')
-        result_dir = config.result_save_dir
+        result_dir = setting.result_save_dir
         temp_dir = result_dir.joinpath('temp')
         utils.check_dir(temp_dir)
         massdns_path = utils.get_massdns_path(massdns_dir)
@@ -635,7 +635,7 @@ class Brute(Module):
             logger.log('INFOR', f'结束执行{self.source}模块爆破域名{self.domain}')
             if not self.path:
                 name = f'{self.domain}_brute_result.{self.format}'
-                self.path = config.result_save_dir.joinpath(name)
+                self.path = setting.result_save_dir.joinpath(name)
             # 数据库导出
             if self.export:
                 dbexport.export(self.domain,

collect.py

@@ -1,95 +1,95 @@
-import time
+import time
-import threading
+import threading
-import importlib
+import importlib
-
+
-import config
+import dbexport
-import dbexport
+from config.log import logger
-from config import logger
+from config import setting
-
+
-
+
-class Collect(object):
+class Collect(object):
-    """
+    """
-    收集子域名类
+    收集子域名类
-    """
+    """
-    def __init__(self, domain, export=True):
+    def __init__(self, domain, export=True):
-        self.domain = domain
+        self.domain = domain
-        self.elapse = 0.0
+        self.elapse = 0.0
-        self.modules = []
+        self.modules = []
-        self.collect_funcs = []
+        self.collect_funcs = []
-        self.path = None
+        self.path = None
-        self.export = export
+        self.export = export
-        self.format = 'csv'
+        self.format = 'csv'
-
+
-    def get_mod(self):
+    def get_mod(self):
-        """
+        """
-        获取要运行的模块
+        获取要运行的模块
-        """
+        """
-        if config.enable_all_module:
+        if setting.enable_all_module:
-            # modules = ['brute', 'certificates', 'crawl',
+            # modules = ['brute', 'certificates', 'crawl',
-            # 'datasets', 'intelligence', 'search']
+            # 'datasets', 'intelligence', 'search']
-            # crawl模块还有点问题
+            # crawl模块还有点问题
-            modules = ['certificates', 'check', 'datasets',
+            modules = ['certificates', 'check', 'datasets',
-                       'dnsquery', 'intelligence', 'search']
+                       'dnsquery', 'intelligence', 'search']
-            # modules = ['intelligence']  # crawl模块还有点问题
+            # modules = ['intelligence']  # crawl模块还有点问题
-            for module in modules:
+            for module in modules:
-                module_path = config.module_dir.joinpath(module)
+                module_path = setting.module_dir.joinpath(module)
-                for path in module_path.rglob('*.py'):
+                for path in module_path.rglob('*.py'):
-                    # 需要导入的类
+                    # 需要导入的类
-                    import_module = ('modules.' + module, path.stem)
+                    import_module = ('modules.' + module, path.stem)
-                    self.modules.append(import_module)
+                    self.modules.append(import_module)
-        else:
+        else:
-            self.modules = config.enable_partial_module
+            self.modules = setting.enable_partial_module
-
+
-    def import_func(self):
+    def import_func(self):
-        """
+        """
-        导入脚本的do函数
+        导入脚本的do函数
-        """
+        """
-        for package, name in self.modules:
+        for package, name in self.modules:
-            import_object = importlib.import_module('.' + name, package)
+            import_object = importlib.import_module('.' + name, package)
-            func = getattr(import_object, 'do')
+            func = getattr(import_object, 'do')
-            self.collect_funcs.append([func, name])
+            self.collect_funcs.append([func, name])
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类运行入口
+        类运行入口
-        """
+        """
-        start = time.time()
+        start = time.time()
-        logger.log('INFOR', f'开始收集{self.domain}的子域')
+        logger.log('INFOR', f'开始收集{self.domain}的子域')
-        self.get_mod()
+        self.get_mod()
-        self.import_func()
+        self.import_func()
-
+
-        threads = []
+        threads = []
-        # 创建多个子域收集线程
+        # 创建多个子域收集线程
-        for collect_func in self.collect_funcs:
+        for collect_func in self.collect_funcs:
-            func_obj, func_name = collect_func
+            func_obj, func_name = collect_func
-            thread = threading.Thread(target=func_obj,
+            thread = threading.Thread(target=func_obj,
-                                      name=func_name,
+                                      name=func_name,
-                                      args=(self.domain,),
+                                      args=(self.domain,),
-                                      daemon=True)
+                                      daemon=True)
-            threads.append(thread)
+            threads.append(thread)
-        # 启动所有线程
+        # 启动所有线程
-        for thread in threads:
+        for thread in threads:
-            thread.start()
+            thread.start()
-        # 等待所有线程完成
+        # 等待所有线程完成
-        for thread in threads:
+        for thread in threads:
-            # 挨个线程判断超时 最坏情况主线程阻塞时间=线程数*module_thread_timeout
+            # 挨个线程判断超时 最坏情况主线程阻塞时间=线程数*module_thread_timeout
-            # 超时线程将脱离主线程 由于创建线程时已添加守护属于 所有超时线程会随着主线程结束
+            # 超时线程将脱离主线程 由于创建线程时已添加守护属于 所有超时线程会随着主线程结束
-            thread.join(config.module_thread_timeout)
+            thread.join(setting.module_thread_timeout)
-
+
-        for thread in threads:
+        for thread in threads:
-            if thread.is_alive():
+            if thread.is_alive():
-                logger.log('ALERT', f'{thread.name}模块线程发生超时')
+                logger.log('ALERT', f'{thread.name}模块线程发生超时')
-
+
-        # 数据库导出
+        # 数据库导出
-        if self.export:
+        if self.export:
-            if not self.path:
+            if not self.path:
-                name = f'{self.domain}.{self.format}'
+                name = f'{self.domain}.{self.format}'
-                self.path = config.result_save_dir.joinpath(name)
+                self.path = setting.result_save_dir.joinpath(name)
-            dbexport.export(self.domain, path=self.path, format=self.format)
+            dbexport.export(self.domain, path=self.path, format=self.format)
-        end = time.time()
+        end = time.time()
-        self.elapse = round(end - start, 1)
+        self.elapse = round(end - start, 1)
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    collect = Collect('example.com')
+    collect = Collect('example.com')
-    collect.run()
+    collect.run()

common/__init__.py

@@ -1 +1 @@
-# coding=utf-8
+# coding=utf-8

common/crawl.py

@@ -1,9 +1,9 @@
-from .module import Module
+from .module import Module
-
+
-
+
-class Crawl(Module):
+class Crawl(Module):
-    """
+    """
-    爬虫基类
+    爬虫基类
-    """
+    """
-    def __init__(self):
+    def __init__(self):
-        Module.__init__(self)
+        Module.__init__(self)

common/database.py

@@ -1,250 +1,250 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python3
-# coding=utf-8
+# coding=utf-8
-
+
-"""
+"""
-SQLite数据库初始化和操作
+SQLite数据库初始化和操作
-"""
+"""
-
+
-import records
+import records
-
+
-import config
+from records import Connection
-from records import Connection
+from config.log import logger
-from config import logger
+from config import setting
-
+
-
+
-class Database(object):
+class Database(object):
-    def __init__(self, db_path=None):
+    def __init__(self, db_path=None):
-        self.conn = self.get_conn(db_path)
+        self.conn = self.get_conn(db_path)
-
+
-    @staticmethod
+    @staticmethod
-    def get_conn(db_path):
+    def get_conn(db_path):
-        """
+        """
-        获取数据库对象
+        获取数据库对象
-
+
-        :param db_path: 数据库连接或路径
+        :param db_path: 数据库连接或路径
-        :return: SQLite数据库
+        :return: SQLite数据库
-        """
+        """
-        logger.log('TRACE', f'正在获取数据库连接')
+        logger.log('TRACE', f'正在获取数据库连接')
-        if isinstance(db_path, Connection):
+        if isinstance(db_path, Connection):
-            return db_path
+            return db_path
-        protocol = 'sqlite:///'
+        protocol = 'sqlite:///'
-        if not db_path:  # 数据库路径为空连接默认数据库
+        if not db_path:  # 数据库路径为空连接默认数据库
-            db_path = f'{protocol}{config.result_save_dir}/result.sqlite3'
+            db_path = f'{protocol}{setting.result_save_dir}/result.sqlite3'
-        else:
+        else:
-            db_path = protocol + db_path
+            db_path = protocol + db_path
-        db = records.Database(db_path)  # 不存在数据库时会新建一个数据库
+        db = records.Database(db_path)  # 不存在数据库时会新建一个数据库
-        logger.log('TRACE', f'使用数据库: {db_path}')
+        logger.log('TRACE', f'使用数据库: {db_path}')
-        return db.get_connection()
+        return db.get_connection()
-
+
-    def query(self, sql):
+    def query(self, sql):
-        try:
+        try:
-            results = self.conn.query(sql)
+            results = self.conn.query(sql)
-        except Exception as e:
+        except Exception as e:
-            logger.log('ERROR', e.args)
+            logger.log('ERROR', e.args)
-        else:
+        else:
-            return results
+            return results
-
+
-    def create_table(self, table_name):
+    def create_table(self, table_name):
-        """
+        """
-        创建表结构
+        创建表结构
-
+
-        :param str table_name: 要创建的表名
+        :param str table_name: 要创建的表名
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        if self.exist_table(table_name):
+        if self.exist_table(table_name):
-            logger.log('TRACE', f'已经存在{table_name}表')
+            logger.log('TRACE', f'已经存在{table_name}表')
-            return
+            return
-        logger.log('TRACE', f'正在创建{table_name}表')
+        logger.log('TRACE', f'正在创建{table_name}表')
-        self.query(f'create table "{table_name}" ('
+        self.query(f'create table "{table_name}" ('
-                   f'id integer primary key,'
+                   f'id integer primary key,'
-                   f'type text,'
+                   f'type text,'
-                   f'alive int,'
+                   f'alive int,'
-                   f'request int,'
+                   f'request int,'
-                   f'resolve int,'
+                   f'resolve int,'
-                   f'new int,'
+                   f'new int,'
-                   f'url text,'
+                   f'url text,'
-                   f'subdomain text,'
+                   f'subdomain text,'
-                   f'port int,'
+                   f'port int,'
-                   f'level int,'
+                   f'level int,'
-                   f'cname text,'
+                   f'cname text,'
-                   f'content text,'
+                   f'content text,'
-                   f'public int,'
+                   f'public int,'
-                   f'status int,'
+                   f'status int,'
-                   f'reason text,'
+                   f'reason text,'
-                   f'title text,'
+                   f'title text,'
-                   f'banner text,'
+                   f'banner text,'
-                   f'header text,'
+                   f'header text,'
-                   f'response text,'
+                   f'response text,'
-                   f'times text,'
+                   f'times text,'
-                   f'ttl text,'
+                   f'ttl text,'
-                   f'resolver text,'
+                   f'resolver text,'
-                   f'module text,'
+                   f'module text,'
-                   f'source text,'
+                   f'source text,'
-                   f'elapse float,'
+                   f'elapse float,'
-                   f'find int,'
+                   f'find int,'
-                   f'brute int,'
+                   f'brute int,'
-                   f'valid int)')
+                   f'valid int)')
-
+
-    def save_db(self, table_name, results, module_name=None):
+    def save_db(self, table_name, results, module_name=None):
-        """
+        """
-        将各模块结果存入数据库
+        将各模块结果存入数据库
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        :param list results: 结果列表
+        :param list results: 结果列表
-        :param str module_name: 模块名
+        :param str module_name: 模块名
-        """
+        """
-        logger.log('TRACE', f'正在将{module_name}模块发现{table_name}的子域'
+        logger.log('TRACE', f'正在将{module_name}模块发现{table_name}的子域'
-                            '结果存入数据库')
+                            '结果存入数据库')
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        if results:
+        if results:
-            try:
+            try:
-                self.conn.bulk_query(
+                self.conn.bulk_query(
-                    f'insert into "{table_name}" ('
+                    f'insert into "{table_name}" ('
-                    f'id, type, alive, resolve, request, new, url, subdomain,'
+                    f'id, type, alive, resolve, request, new, url, subdomain,'
-                    f'port, level, cname, content, public, status, reason,'
+                    f'port, level, cname, content, public, status, reason,'
-                    f'title, banner, header, response, times, ttl, resolver,'
+                    f'title, banner, header, response, times, ttl, resolver,'
-                    f'module, source, elapse, find, brute, valid) '
+                    f'module, source, elapse, find, brute, valid) '
-                    f'values (:id, :type, :alive, :resolve, :request, :new,'
+                    f'values (:id, :type, :alive, :resolve, :request, :new,'
-                    f':url, :subdomain, :port, :level, :cname, :content,'
+                    f':url, :subdomain, :port, :level, :cname, :content,'
-                    f':public, :status, :reason, :title, :banner, :header,'
+                    f':public, :status, :reason, :title, :banner, :header,'
-                    f':response, :times, :ttl, :resolver, :module, :source,'
+                    f':response, :times, :ttl, :resolver, :module, :source,'
-                    f':elapse, :find, :brute, :valid)', results)
+                    f':elapse, :find, :brute, :valid)', results)
-            except Exception as e:
+            except Exception as e:
-                logger.log('ERROR', e)
+                logger.log('ERROR', e)
-
+
-    def exist_table(self, table_name):
+    def exist_table(self, table_name):
-        """
+        """
-        判断是否存在某表
+        判断是否存在某表
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        :return: 是否存在某表
+        :return: 是否存在某表
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        logger.log('TRACE', f'正在查询是否存在{table_name}表')
+        logger.log('TRACE', f'正在查询是否存在{table_name}表')
-        results = self.query(f'select count() from sqlite_master '
+        results = self.query(f'select count() from sqlite_master '
-                             f'where type = "table" and '
+                             f'where type = "table" and '
-                             f'name = "{table_name}"')
+                             f'name = "{table_name}"')
-        if results.scalar() == 0:
+        if results.scalar() == 0:
-            return False
+            return False
-        else:
+        else:
-            return True
+            return True
-
+
-    def copy_table(self, table_name, bak_table_name):
+    def copy_table(self, table_name, bak_table_name):
-        """
+        """
-        复制表创建备份
+        复制表创建备份
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        :param str bak_table_name: 新表名
+        :param str bak_table_name: 新表名
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        bak_table_name = bak_table_name.replace('.', '_')
+        bak_table_name = bak_table_name.replace('.', '_')
-        logger.log('TRACE', f'正在将{table_name}表复制到{bak_table_name}新表')
+        logger.log('TRACE', f'正在将{table_name}表复制到{bak_table_name}新表')
-        self.query(f'drop table if exists "{bak_table_name}"')
+        self.query(f'drop table if exists "{bak_table_name}"')
-        self.query(f'create table "{bak_table_name}" '
+        self.query(f'create table "{bak_table_name}" '
-                   f'as select * from "{table_name}"')
+                   f'as select * from "{table_name}"')
-
+
-    def clear_table(self, table_name):
+    def clear_table(self, table_name):
-        """
+        """
-        清空表中数据
+        清空表中数据
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        logger.log('TRACE', f'正在清空{table_name}表中的数据')
+        logger.log('TRACE', f'正在清空{table_name}表中的数据')
-        self.query(f'delete from "{table_name}"')
+        self.query(f'delete from "{table_name}"')
-
+
-    def drop_table(self, table_name):
+    def drop_table(self, table_name):
-        """
+        """
-        删除表
+        删除表
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        logger.log('TRACE', f'正在删除{table_name}表')
+        logger.log('TRACE', f'正在删除{table_name}表')
-        self.query(f'drop table if exists "{table_name}"')
+        self.query(f'drop table if exists "{table_name}"')
-
+
-    def rename_table(self, table_name, new_table_name):
+    def rename_table(self, table_name, new_table_name):
-        """
+        """
-        重命名表名
+        重命名表名
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        :param str new_table_name: 新表名
+        :param str new_table_name: 新表名
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        new_table_name = new_table_name.replace('.', '_')
+        new_table_name = new_table_name.replace('.', '_')
-        logger.log('TRACE', f'正在将{table_name}表重命名为{table_name}表')
+        logger.log('TRACE', f'正在将{table_name}表重命名为{table_name}表')
-        self.query(f'alter table "{table_name}" '
+        self.query(f'alter table "{table_name}" '
-                   f'rename to "{new_table_name}"')
+                   f'rename to "{new_table_name}"')
-
+
-    def deduplicate_subdomain(self, table_name):
+    def deduplicate_subdomain(self, table_name):
-        """
+        """
-        去重表中的子域
+        去重表中的子域
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        logger.log('TRACE', f'正在去重{table_name}表中的子域')
+        logger.log('TRACE', f'正在去重{table_name}表中的子域')
-        self.query(f'delete from "{table_name}" where '
+        self.query(f'delete from "{table_name}" where '
-                   f'id not in (select min(id) '
+                   f'id not in (select min(id) '
-                   f'from "{table_name}" group by subdomain)')
+                   f'from "{table_name}" group by subdomain)')
-
+
-    def remove_invalid(self, table_name):
+    def remove_invalid(self, table_name):
-        """
+        """
-        去除表中的空值或无效子域
+        去除表中的空值或无效子域
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        logger.log('TRACE', f'正在去除{table_name}表中的无效子域')
+        logger.log('TRACE', f'正在去除{table_name}表中的无效子域')
-        self.query(f'delete from "{table_name}" where '
+        self.query(f'delete from "{table_name}" where '
-                   f'subdomain is null or resolve == 0')
+                   f'subdomain is null or resolve == 0')
-
+
-    def deal_table(self, deal_table_name, backup_table_name):
+    def deal_table(self, deal_table_name, backup_table_name):
-        """
+        """
-        收集任务完成时对表进行处理
+        收集任务完成时对表进行处理
-
+
-        :param str deal_table_name: 待处理的表名
+        :param str deal_table_name: 待处理的表名
-        :param str backup_table_name: 备份的表名
+        :param str backup_table_name: 备份的表名
-        """
+        """
-        self.copy_table(deal_table_name, backup_table_name)
+        self.copy_table(deal_table_name, backup_table_name)
-        self.remove_invalid(deal_table_name)
+        self.remove_invalid(deal_table_name)
-        self.deduplicate_subdomain(deal_table_name)
+        self.deduplicate_subdomain(deal_table_name)
-
+
-    def get_data(self, table_name):
+    def get_data(self, table_name):
-        """
+        """
-        获取表中的所有数据
+        获取表中的所有数据
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        logger.log('TRACE', f'获取{table_name}表中的所有数据')
+        logger.log('TRACE', f'获取{table_name}表中的所有数据')
-        return self.query(f'select * from "{table_name}"')
+        return self.query(f'select * from "{table_name}"')
-
+
-    def export_data(self, table_name, alive, limit):
+    def export_data(self, table_name, alive, limit):
-        """
+        """
-        获取表中的部分数据
+        获取表中的部分数据
-
+
-        :param str table_name: 表名
+        :param str table_name: 表名
-        :param any alive: 存活
+        :param any alive: 存活
-        :param str limit: 限制字段
+        :param str limit: 限制字段
-        """
+        """
-        table_name = table_name.replace('.', '_')
+        table_name = table_name.replace('.', '_')
-        query = f'select id, type, new, alive, request, resolve, url, ' \
+        query = f'select id, type, new, alive, request, resolve, url, ' \
-                f'subdomain, level, cname, content, public, port, status, ' \
+                f'subdomain, level, cname, content, public, port, status, ' \
-                f'reason, title, banner, times, ttl, resolver, module, ' \
+                f'reason, title, banner, times, ttl, resolver, module, ' \
-                f'source, elapse, find, brute, valid from "{table_name}"'
+                f'source, elapse, find, brute, valid from "{table_name}"'
-        if alive and limit:
+        if alive and limit:
-            if limit in ['resolve', 'request']:
+            if limit in ['resolve', 'request']:
-                where = f' where {limit} = 1'
+                where = f' where {limit} = 1'
-                query += where
+                query += where
-        elif alive:
+        elif alive:
-            where = f' where alive = 1'
+            where = f' where alive = 1'
-            query += where
+            query += where
-        logger.log('TRACE', f'获取{table_name}表中的数据')
+        logger.log('TRACE', f'获取{table_name}表中的数据')
-        return self.query(query)
+        return self.query(query)
-
+
-    def close(self):
+    def close(self):
-        """
+        """
-        关闭数据库连接
+        关闭数据库连接
-        """
+        """
-        self.conn.close()
+        self.conn.close()

common/domain.py

@@ -1,64 +1,64 @@
-import re
+import re
-import tldextract
+import tldextract
-import config
+from config import setting
-
+
-
+
-class Domain(object):
+class Domain(object):
-    """
+    """
-    域名处理类
+    域名处理类
-
+
-    :param str string: 传入的字符串
+    :param str string: 传入的字符串
-    """
+    """
-    def __init__(self, string):
+    def __init__(self, string):
-        self.string = str(string)
+        self.string = str(string)
-        self.regexp = r'\b((?=[a-z0-9-]{1,63}\.)(xn--)?[a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,63}\b'
+        self.regexp = r'\b((?=[a-z0-9-]{1,63}\.)(xn--)?[a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,63}\b'
-        self.domain = None
+        self.domain = None
-
+
-    def match(self):
+    def match(self):
-        """
+        """
-        域名匹配
+        域名匹配
-
+
-        :return: 匹配结果
+        :return: 匹配结果
-        """
+        """
-        result = re.search(self.regexp, self.string, re.I)
+        result = re.search(self.regexp, self.string, re.I)
-        if result:
+        if result:
-            return result.group()
+            return result.group()
-        else:
+        else:
-            return None
+            return None
-
+
-    def extract(self):
+    def extract(self):
-        """
+        """
-        域名导出
+        域名导出
-
+
-        >>> d = Domain('www.example.com')
+        >>> d = Domain('www.example.com')
-        <domain.Domain object>
+        <domain.Domain object>
-        >>> d.extract()
+        >>> d.extract()
-        ExtractResult(subdomain='www', domain='example', suffix='com')
+        ExtractResult(subdomain='www', domain='example', suffix='com')
-
+
-        :return: 导出结果
+        :return: 导出结果
-        """
+        """
-        data_storage_dir = config.data_storage_dir
+        data_storage_dir = setting.data_storage_dir
-        extract_cache_file = data_storage_dir.joinpath('public_suffix_list.dat')
+        extract_cache_file = data_storage_dir.joinpath('public_suffix_list.dat')
-        tldext = tldextract.TLDExtract(extract_cache_file)
+        tldext = tldextract.TLDExtract(extract_cache_file)
-        result = self.match()
+        result = self.match()
-        if result:
+        if result:
-            return tldext(result)
+            return tldext(result)
-        else:
+        else:
-            return None
+            return None
-
+
-    def registered(self):
+    def registered(self):
-        """
+        """
-        获取注册域名
+        获取注册域名
-
+
-        >>> d = Domain('www.example.com')
+        >>> d = Domain('www.example.com')
-        <domain.Domain object>
+        <domain.Domain object>
-        >>> d.registered()
+        >>> d.registered()
-        example.com
+        example.com
-
+
-        :return: 注册域名
+        :return: 注册域名
-        """
+        """
-        result = self.extract()
+        result = self.extract()
-        if result:
+        if result:
-            return result.registered_domain
+            return result.registered_domain
-        else:
+        else:
-            return None
+            return None

common/module.py

@@ -1,359 +1,359 @@
-# coding=utf-8
+# coding=utf-8
-"""
+"""
-模块基类
+模块基类
-"""
+"""
-
+
-import json
+import json
-import re
+import re
-import threading
+import threading
-import time
+import time
-
+
-import requests
+import requests
-import config
+from config.log import logger
-from config import logger
+from config import setting
-from . import utils
+from . import utils
-from .domain import Domain
+from .domain import Domain
-from common.database import Database
+from common.database import Database
-
+
-lock = threading.Lock()
+lock = threading.Lock()
-
+
-
+
-class Module(object):
+class Module(object):
-    def __init__(self):
+    def __init__(self):
-        self.module = 'Module'
+        self.module = 'Module'
-        self.source = 'BaseModule'
+        self.source = 'BaseModule'
-        self.cookie = None
+        self.cookie = None
-        self.header = dict()
+        self.header = dict()
-        self.proxy = None
+        self.proxy = None
-        self.delay = config.request_delay  # 请求睡眠时延
+        self.delay = setting.request_delay  # 请求睡眠时延
-        self.timeout = config.request_timeout  # 请求超时时间
+        self.timeout = setting.request_timeout  # 请求超时时间
-        self.verify = config.request_verify  # 请求SSL验证
+        self.verify = setting.request_verify  # 请求SSL验证
-        self.domain = str()  # 当前进行子域名收集的主域
+        self.domain = str()  # 当前进行子域名收集的主域
-        self.type = 'A'  # 对主域进行子域收集时利用的DNS记录查询类型(默认利用A记录)
+        self.type = 'A'  # 对主域进行子域收集时利用的DNS记录查询类型(默认利用A记录)
-        self.subdomains = set()  # 存放发现的子域
+        self.subdomains = set()  # 存放发现的子域
-        self.records = dict()  # 存放子域解析记录
+        self.records = dict()  # 存放子域解析记录
-        self.results = list()  # 存放模块结果
+        self.results = list()  # 存放模块结果
-        self.start = time.time()  # 模块开始执行时间
+        self.start = time.time()  # 模块开始执行时间
-        self.end = None  # 模块结束执行时间
+        self.end = None  # 模块结束执行时间
-        self.elapse = None  # 模块执行耗时
+        self.elapse = None  # 模块执行耗时
-
+
-    def check(self, *apis):
+    def check(self, *apis):
-        """
+        """
-        简单检查是否配置了api信息
+        简单检查是否配置了api信息
-
+
-        :param apis: api信息元组
+        :param apis: api信息元组
-        :return: 检查结果
+        :return: 检查结果
-        """
+        """
-        if not all(apis):
+        if not all(apis):
-            logger.log('ALERT', f'{self.source}模块没有配置API跳过执行')
+            logger.log('ALERT', f'{self.source}模块没有配置API跳过执行')
-            return False
+            return False
-        return True
+        return True
-
+
-    def begin(self):
+    def begin(self):
-        """
+        """
-        输出模块开始信息
+        输出模块开始信息
-        """
+        """
-        logger.log('DEBUG', f'开始执行{self.source}模块收集{self.domain}的子域')
+        logger.log('DEBUG', f'开始执行{self.source}模块收集{self.domain}的子域')
-
+
-    def finish(self):
+    def finish(self):
-        """
+        """
-        输出模块结束信息
+        输出模块结束信息
-        """
+        """
-        self.end = time.time()
+        self.end = time.time()
-        self.elapse = round(self.end - self.start, 1)
+        self.elapse = round(self.end - self.start, 1)
-        logger.log('DEBUG', f'结束执行{self.source}模块收集{self.domain}的子域')
+        logger.log('DEBUG', f'结束执行{self.source}模块收集{self.domain}的子域')
-        logger.log('INFOR', f'{self.source}模块耗时{self.elapse}秒发现子域'
+        logger.log('INFOR', f'{self.source}模块耗时{self.elapse}秒发现子域'
-                            f'{len(self.subdomains)}个')
+                            f'{len(self.subdomains)}个')
-        logger.log('DEBUG', f'{self.source}模块发现{self.domain}的子域\n'
+        logger.log('DEBUG', f'{self.source}模块发现{self.domain}的子域\n'
-                            f'{self.subdomains}')
+                            f'{self.subdomains}')
-
+
-    def head(self, url, params=None, check=True, **kwargs):
+    def head(self, url, params=None, check=True, **kwargs):
-        """
+        """
-        自定义head请求
+        自定义head请求
-
+
-        :param str url: 请求地址
+        :param str url: 请求地址
-        :param dict params: 请求参数
+        :param dict params: 请求参数
-        :param bool check: 检查响应
+        :param bool check: 检查响应
-        :param kwargs: 其他参数
+        :param kwargs: 其他参数
-        :return: requests响应对象
+        :return: requests响应对象
-        """
+        """
-        try:
+        try:
-            resp = requests.head(url,
+            resp = requests.head(url,
-                                 params=params,
+                                 params=params,
-                                 cookies=self.cookie,
+                                 cookies=self.cookie,
-                                 headers=self.header,
+                                 headers=self.header,
-                                 proxies=self.proxy,
+                                 proxies=self.proxy,
-                                 timeout=self.timeout,
+                                 timeout=self.timeout,
-                                 verify=self.verify,
+                                 verify=self.verify,
-                                 **kwargs)
+                                 **kwargs)
-        except Exception as e:
+        except Exception as e:
-            logger.log('ERROR', e.args)
+            logger.log('ERROR', e.args)
-            return None
+            return None
-        if not check:
+        if not check:
-            return resp
+            return resp
-        if utils.check_response('HEAD', resp):
+        if utils.check_response('HEAD', resp):
-            return resp
+            return resp
-        return None
+        return None
-
+
-    def get(self, url, params=None, check=True, **kwargs):
+    def get(self, url, params=None, check=True, **kwargs):
-        """
+        """
-        自定义get请求
+        自定义get请求
-
+
-        :param str url: 请求地址
+        :param str url: 请求地址
-        :param dict params: 请求参数
+        :param dict params: 请求参数
-        :param bool check: 检查响应
+        :param bool check: 检查响应
-        :param kwargs: 其他参数
+        :param kwargs: 其他参数
-        :return: requests响应对象
+        :return: requests响应对象
-        """
+        """
-        try:
+        try:
-            resp = requests.get(url,
+            resp = requests.get(url,
-                                params=params,
+                                params=params,
-                                cookies=self.cookie,
+                                cookies=self.cookie,
-                                headers=self.header,
+                                headers=self.header,
-                                proxies=self.proxy,
+                                proxies=self.proxy,
-                                timeout=self.timeout,
+                                timeout=self.timeout,
-                                verify=self.verify,
+                                verify=self.verify,
-                                **kwargs)
+                                **kwargs)
-        except Exception as e:
+        except Exception as e:
-            logger.log('ERROR', e.args)
+            logger.log('ERROR', e.args)
-            return None
+            return None
-        if not check:
+        if not check:
-            return resp
+            return resp
-        if utils.check_response('GET', resp):
+        if utils.check_response('GET', resp):
-            return resp
+            return resp
-        return None
+        return None
-
+
-    def post(self, url, data=None, check=True, **kwargs):
+    def post(self, url, data=None, check=True, **kwargs):
-        """
+        """
-        自定义post请求
+        自定义post请求
-
+
-        :param str url: 请求地址
+        :param str url: 请求地址
-        :param dict data: 请求数据
+        :param dict data: 请求数据
-        :param bool check: 检查响应
+        :param bool check: 检查响应
-        :param kwargs: 其他参数
+        :param kwargs: 其他参数
-        :return: requests响应对象
+        :return: requests响应对象
-        """
+        """
-        try:
+        try:
-            resp = requests.post(url,
+            resp = requests.post(url,
-                                 data=data,
+                                 data=data,
-                                 cookies=self.cookie,
+                                 cookies=self.cookie,
-                                 headers=self.header,
+                                 headers=self.header,
-                                 proxies=self.proxy,
+                                 proxies=self.proxy,
-                                 timeout=self.timeout,
+                                 timeout=self.timeout,
-                                 verify=self.verify,
+                                 verify=self.verify,
-                                 **kwargs)
+                                 **kwargs)
-        except Exception as e:
+        except Exception as e:
-            logger.log('ERROR', e.args)
+            logger.log('ERROR', e.args)
-            return None
+            return None
-        if not check:
+        if not check:
-            return resp
+            return resp
-        if utils.check_response('POST', resp):
+        if utils.check_response('POST', resp):
-            return resp
+            return resp
-        return None
+        return None
-
+
-    def get_header(self):
+    def get_header(self):
-        """
+        """
-        获取请求头
+        获取请求头
-
+
-        :return: 请求头
+        :return: 请求头
-        """
+        """
-        # logger.log('DEBUG', f'获取请求头')
+        # logger.log('DEBUG', f'获取请求头')
-        if config.enable_fake_header:
+        if setting.enable_fake_header:
-            return utils.gen_fake_header()
+            return utils.gen_fake_header()
-        else:
+        else:
-            return self.header
+            return self.header
-
+
-    def get_proxy(self, module):
+    def get_proxy(self, module):
-        """
+        """
-        获取代理
+        获取代理
-
+
-        :param str module: 模块名
+        :param str module: 模块名
-        :return: 代理字典
+        :return: 代理字典
-        """
+        """
-        if not config.enable_proxy:
+        if not setting.enable_proxy:
-            logger.log('TRACE', f'所有模块不使用代理')
+            logger.log('TRACE', f'所有模块不使用代理')
-            return self.proxy
+            return self.proxy
-        if config.proxy_all_module:
+        if setting.proxy_all_module:
-            logger.log('TRACE', f'{module}模块使用代理')
+            logger.log('TRACE', f'{module}模块使用代理')
-            return utils.get_random_proxy()
+            return utils.get_random_proxy()
-        if module in config.proxy_partial_module:
+        if module in setting.proxy_partial_module:
-            logger.log('TRACE', f'{module}模块使用代理')
+            logger.log('TRACE', f'{module}模块使用代理')
-            return utils.get_random_proxy()
+            return utils.get_random_proxy()
-        else:
+        else:
-            logger.log('TRACE', f'{module}模块不使用代理')
+            logger.log('TRACE', f'{module}模块不使用代理')
-            return self.proxy
+            return self.proxy
-
+
-    @staticmethod
+    @staticmethod
-    def match(domain, html, distinct=True):
+    def match(domain, html, distinct=True):
-        """
+        """
-        正则匹配出子域
+        正则匹配出子域
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str html: 要匹配的html响应体
+        :param str html: 要匹配的html响应体
-        :param bool distinct: 匹配结果去除
+        :param bool distinct: 匹配结果去除
-        :return: 匹配出的子域集合或列表
+        :return: 匹配出的子域集合或列表
-        :rtype: set or list
+        :rtype: set or list
-        """
+        """
-        logger.log('TRACE', f'正则匹配响应体中的子域')
+        logger.log('TRACE', f'正则匹配响应体中的子域')
-        regexp = r'(?:\>|\"|\'|\=|\,)(?:http\:\/\/|https\:\/\/)?' \
+        regexp = r'(?:\>|\"|\'|\=|\,)(?:http\:\/\/|https\:\/\/)?' \
-                 r'(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.){0,}' \
+                 r'(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.){0,}' \
-                 + domain.replace('.', r'\.')
+                 + domain.replace('.', r'\.')
-        result = re.findall(regexp, html, re.I)
+        result = re.findall(regexp, html, re.I)
-        if not result:
+        if not result:
-            return set()
+            return set()
-        regexp = r'(?:http://|https://)'
+        regexp = r'(?:http://|https://)'
-        deal = map(lambda s: re.sub(regexp, '', s[1:].lower()), result)
+        deal = map(lambda s: re.sub(regexp, '', s[1:].lower()), result)
-        if distinct:
+        if distinct:
-            return set(deal)
+            return set(deal)
-        else:
+        else:
-            return list(deal)
+            return list(deal)
-
+
-    @staticmethod
+    @staticmethod
-    def register(domain):
+    def register(domain):
-        """
+        """
-        获取注册域名
+        获取注册域名
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :return: 注册域名
+        :return: 注册域名
-        """
+        """
-        return Domain(domain).registered()
+        return Domain(domain).registered()
-
+
-    def save_json(self):
+    def save_json(self):
-        """
+        """
-        将各模块结果保存为json文件
+        将各模块结果保存为json文件
-
+
-        :return： 是否保存成功
+        :return： 是否保存成功
-        """
+        """
-        if not config.save_module_result:
+        if not setting.save_module_result:
-            return False
+            return False
-        logger.log('TRACE', f'将{self.source}模块发现的子域结果保存为json文件')
+        logger.log('TRACE', f'将{self.source}模块发现的子域结果保存为json文件')
-        path = config.result_save_dir.joinpath(self.domain, self.module)
+        path = setting.result_save_dir.joinpath(self.domain, self.module)
-        path.mkdir(parents=True, exist_ok=True)
+        path.mkdir(parents=True, exist_ok=True)
-        name = self.source + '.json'
+        name = self.source + '.json'
-        path = path.joinpath(name)
+        path = path.joinpath(name)
-        with open(path, mode='w', encoding='utf-8', errors='ignore') as file:
+        with open(path, mode='w', encoding='utf-8', errors='ignore') as file:
-            result = {'domain': self.domain,
+            result = {'domain': self.domain,
-                      'name': self.module,
+                      'name': self.module,
-                      'source': self.source,
+                      'source': self.source,
-                      'elapse': self.elapse,
+                      'elapse': self.elapse,
-                      'find': len(self.subdomains),
+                      'find': len(self.subdomains),
-                      'subdomains': list(self.subdomains),
+                      'subdomains': list(self.subdomains),
-                      'records': self.records}
+                      'records': self.records}
-            json.dump(result, file, ensure_ascii=False, indent=4)
+            json.dump(result, file, ensure_ascii=False, indent=4)
-        return True
+        return True
-
+
-    def gen_record(self, subdomains, record):
+    def gen_record(self, subdomains, record):
-        """
+        """
-        生成记录字典
+        生成记录字典
-        """
+        """
-        item = dict()
+        item = dict()
-        item['content'] = record
+        item['content'] = record
-        for subdomain in subdomains:
+        for subdomain in subdomains:
-            self.records[subdomain] = item
+            self.records[subdomain] = item
-
+
-    def gen_result(self, find=0, brute=None, valid=0):
+    def gen_result(self, find=0, brute=None, valid=0):
-        """
+        """
-        生成结果
+        生成结果
-        """
+        """
-        logger.log('DEBUG', f'正在生成最终结果')
+        logger.log('DEBUG', f'正在生成最终结果')
-        if not len(self.subdomains):  # 该模块一个子域都没有发现的情况
+        if not len(self.subdomains):  # 该模块一个子域都没有发现的情况
-            logger.log('DEBUG', f'{self.source}模块收集结果为空')
+            logger.log('DEBUG', f'{self.source}模块收集结果为空')
-            result = {'id': None,
+            result = {'id': None,
-                      'type': self.type,
+                      'type': self.type,
-                      'alive': None,
+                      'alive': None,
-                      'request': None,
+                      'request': None,
-                      'resolve': None,
+                      'resolve': None,
-                      'new': None,
+                      'new': None,
-                      'url': None,
+                      'url': None,
-                      'subdomain': None,
+                      'subdomain': None,
-                      'level': None,
+                      'level': None,
-                      'cname': None,
+                      'cname': None,
-                      'content': None,
+                      'content': None,
-                      'public': None,
+                      'public': None,
-                      'port': None,
+                      'port': None,
-                      'status': None,
+                      'status': None,
-                      'reason': None,
+                      'reason': None,
-                      'title': None,
+                      'title': None,
-                      'banner': None,
+                      'banner': None,
-                      'header': None,
+                      'header': None,
-                      'response': None,
+                      'response': None,
-                      'times': None,
+                      'times': None,
-                      'ttl': None,
+                      'ttl': None,
-                      'resolver': None,
+                      'resolver': None,
-                      'module': self.module,
+                      'module': self.module,
-                      'source': self.source,
+                      'source': self.source,
-                      'elapse': self.elapse,
+                      'elapse': self.elapse,
-                      'find': find,
+                      'find': find,
-                      'brute': brute,
+                      'brute': brute,
-                      'valid': valid}
+                      'valid': valid}
-            self.results.append(result)
+            self.results.append(result)
-        else:
+        else:
-            for subdomain in self.subdomains:
+            for subdomain in self.subdomains:
-                url = 'http://' + subdomain
+                url = 'http://' + subdomain
-                level = subdomain.count('.') - self.domain.count('.')
+                level = subdomain.count('.') - self.domain.count('.')
-                record = self.records.get(subdomain)
+                record = self.records.get(subdomain)
-                if record is None:
+                if record is None:
-                    record = dict()
+                    record = dict()
-                resolve = record.get('resolve')
+                resolve = record.get('resolve')
-                request = record.get('request')
+                request = record.get('request')
-                alive = record.get('alive')
+                alive = record.get('alive')
-                if self.type != 'A':  # 不是利用的DNS记录的A记录查询子域默认都有效
+                if self.type != 'A':  # 不是利用的DNS记录的A记录查询子域默认都有效
-                    resolve = 1
+                    resolve = 1
-                    request = 1
+                    request = 1
-                    alive = 1
+                    alive = 1
-                reason = record.get('reason')
+                reason = record.get('reason')
-                resolver = record.get('resolver')
+                resolver = record.get('resolver')
-                cname = record.get('cname')
+                cname = record.get('cname')
-                content = record.get('content')
+                content = record.get('content')
-                times = record.get('times')
+                times = record.get('times')
-                ttl = record.get('ttl')
+                ttl = record.get('ttl')
-                public = record.get('public')
+                public = record.get('public')
-                if isinstance(cname, list):
+                if isinstance(cname, list):
-                    cname = ','.join(cname)
+                    cname = ','.join(cname)
-                    content = ','.join(content)
+                    content = ','.join(content)
-                    times = ','.join([str(num) for num in times])
+                    times = ','.join([str(num) for num in times])
-                    ttl = ','.join([str(num) for num in ttl])
+                    ttl = ','.join([str(num) for num in ttl])
-                    public = ','.join([str(num) for num in public])
+                    public = ','.join([str(num) for num in public])
-                result = {'id': None,
+                result = {'id': None,
-                          'type': self.type,
+                          'type': self.type,
-                          'alive': alive,
+                          'alive': alive,
-                          'request': request,
+                          'request': request,
-                          'resolve': resolve,
+                          'resolve': resolve,
-                          'new': None,
+                          'new': None,
-                          'url': url,
+                          'url': url,
-                          'subdomain': subdomain,
+                          'subdomain': subdomain,
-                          'level': level,
+                          'level': level,
-                          'cname': cname,
+                          'cname': cname,
-                          'content': content,
+                          'content': content,
-                          'public': public,
+                          'public': public,
-                          'port': 80,
+                          'port': 80,
-                          'status': None,
+                          'status': None,
-                          'reason': reason,
+                          'reason': reason,
-                          'title': None,
+                          'title': None,
-                          'banner': None,
+                          'banner': None,
-                          'header': None,
+                          'header': None,
-                          'response': None,
+                          'response': None,
-                          'times': times,
+                          'times': times,
-                          'ttl': ttl,
+                          'ttl': ttl,
-                          'resolver': resolver,
+                          'resolver': resolver,
-                          'module': self.module,
+                          'module': self.module,
-                          'source': self.source,
+                          'source': self.source,
-                          'elapse': self.elapse,
+                          'elapse': self.elapse,
-                          'find': find,
+                          'find': find,
-                          'brute': brute,
+                          'brute': brute,
-                          'valid': valid,
+                          'valid': valid,
-                          }
+                          }
-                self.results.append(result)
+                self.results.append(result)
-
+
-    def save_db(self):
+    def save_db(self):
-        """
+        """
-        将模块结果存入数据库中
+        将模块结果存入数据库中
-
+
-        """
+        """
-        logger.log('DEBUG', f'正在将结果存入到数据库')
+        logger.log('DEBUG', f'正在将结果存入到数据库')
-        lock.acquire()
+        lock.acquire()
-        db = Database()
+        db = Database()
-        db.create_table(self.domain)
+        db.create_table(self.domain)
-        db.save_db(self.domain, self.results, self.source)
+        db.save_db(self.domain, self.results, self.source)
-        db.close()
+        db.close()
-        lock.release()
+        lock.release()

common/query.py

@@ -1,9 +1,9 @@
-from .module import Module
+from .module import Module
-
+
-
+
-class Query(Module):
+class Query(Module):
-    """
+    """
-    查询基类
+    查询基类
-    """
+    """
-    def __init__(self):
+    def __init__(self):
-        Module.__init__(self)
+        Module.__init__(self)

common/request.py

@@ -1,274 +1,274 @@
-import asyncio
+import asyncio
-import functools
+import functools
-
+
-import aiohttp
+import aiohttp
-import tqdm
+import tqdm
-from aiohttp import ClientSession
+from aiohttp import ClientSession
-from bs4 import BeautifulSoup
+from bs4 import BeautifulSoup
-
+
-import config
+from common import utils
-from common import utils
+from config.log import logger
-from config import logger
+from config import setting
-from common.database import Database
+from common.database import Database
-
+
-
+
-def get_limit_conn():
+def get_limit_conn():
-    limit_open_conn = config.limit_open_conn
+    limit_open_conn = setting.limit_open_conn
-    if limit_open_conn is None:  # 默认情况
+    if limit_open_conn is None:  # 默认情况
-        limit_open_conn = utils.get_semaphore()
+        limit_open_conn = utils.get_semaphore()
-    elif not isinstance(limit_open_conn, int):  # 如果传入不是数字的情况
+    elif not isinstance(limit_open_conn, int):  # 如果传入不是数字的情况
-        limit_open_conn = utils.get_semaphore()
+        limit_open_conn = utils.get_semaphore()
-    return limit_open_conn
+    return limit_open_conn
-
+
-
+
-def get_ports(port):
+def get_ports(port):
-    logger.log('DEBUG', f'正在获取请求端口范围')
+    logger.log('DEBUG', f'正在获取请求端口范围')
-    ports = set()
+    ports = set()
-    if isinstance(port, (set, list, tuple)):
+    if isinstance(port, (set, list, tuple)):
-        ports = port
+        ports = port
-    elif isinstance(port, int):
+    elif isinstance(port, int):
-        if 0 <= port <= 65535:
+        if 0 <= port <= 65535:
-            ports = {port}
+            ports = {port}
-    elif port in {'default', 'small', 'large'}:
+    elif port in {'default', 'small', 'large'}:
-        logger.log('DEBUG', f'请求{port}等端口范围')
+        logger.log('DEBUG', f'请求{port}等端口范围')
-        ports = config.ports.get(port)
+        ports = setting.ports.get(port)
-    if not ports:  # 意外情况
+    if not ports:  # 意外情况
-        logger.log('ERROR', f'指定请求端口范围有误')
+        logger.log('ERROR', f'指定请求端口范围有误')
-        ports = {80}
+        ports = {80}
-    logger.log('INFOR', f'请求端口范围：{ports}')
+    logger.log('INFOR', f'请求端口范围：{ports}')
-    return set(ports)
+    return set(ports)
-
+
-
+
-def gen_req_data(data, ports):
+def gen_req_data(data, ports):
-    logger.log('INFOR', f'正在生成请求地址')
+    logger.log('INFOR', f'正在生成请求地址')
-    new_data = []
+    new_data = []
-    for data in data:
+    for data in data:
-        resolve = data.get('resolve')
+        resolve = data.get('resolve')
-        # 解析失败(0)的子域不进行http请求探测
+        # 解析失败(0)的子域不进行http请求探测
-        if resolve == 0:
+        if resolve == 0:
-            continue
+            continue
-        subdomain = data.get('subdomain')
+        subdomain = data.get('subdomain')
-        for port in ports:
+        for port in ports:
-            if str(port).endswith('443'):
+            if str(port).endswith('443'):
-                url = f'https://{subdomain}:{port}'
+                url = f'https://{subdomain}:{port}'
-                if port == 443:
+                if port == 443:
-                    url = f'https://{subdomain}'
+                    url = f'https://{subdomain}'
-                data['id'] = None
+                data['id'] = None
-                data['url'] = url
+                data['url'] = url
-                data['port'] = port
+                data['port'] = port
-                new_data.append(data)
+                new_data.append(data)
-                data = dict(data)  # 需要生成一个新的字典对象
+                data = dict(data)  # 需要生成一个新的字典对象
-            else:
+            else:
-                url = f'http://{subdomain}:{port}'
+                url = f'http://{subdomain}:{port}'
-                if port == 80:
+                if port == 80:
-                    url = f'http://{subdomain}'
+                    url = f'http://{subdomain}'
-                data['id'] = None
+                data['id'] = None
-                data['url'] = url
+                data['url'] = url
-                data['port'] = port
+                data['port'] = port
-                new_data.append(data)
+                new_data.append(data)
-                data = dict(data)  # 需要生成一个新的字典对象
+                data = dict(data)  # 需要生成一个新的字典对象
-    return new_data
+    return new_data
-
+
-
+
-async def fetch(session, url):
+async def fetch(session, url):
-    """
+    """
-    请求
+    请求
-
+
-    :param session: session对象
+    :param session: session对象
-    :param str url: url地址
+    :param str url: url地址
-    :return: 响应对象和响应文本
+    :return: 响应对象和响应文本
-    """
+    """
-    method = config.request_method.upper()
+    method = setting.request_method.upper()
-    timeout = aiohttp.ClientTimeout(total=None,
+    timeout = aiohttp.ClientTimeout(total=None,
-                                    connect=None,
+                                    connect=None,
-                                    sock_read=config.sockread_timeout,
+                                    sock_read=setting.sockread_timeout,
-                                    sock_connect=config.sockconn_timeout)
+                                    sock_connect=setting.sockconn_timeout)
-    try:
+    try:
-        if method == 'HEAD':
+        if method == 'HEAD':
-            async with session.head(url,
+            async with session.head(url,
-                                    ssl=config.verify_ssl,
+                                    ssl=setting.verify_ssl,
-                                    allow_redirects=config.allow_redirects,
+                                    allow_redirects=setting.allow_redirects,
-                                    timeout=timeout,
+                                    timeout=timeout,
-                                    proxy=config.aiohttp_proxy) as resp:
+                                    proxy=setting.aiohttp_proxy) as resp:
-                text = await resp.text()
+                text = await resp.text()
-        else:
+        else:
-            async with session.get(url,
+            async with session.get(url,
-                                   ssl=config.verify_ssl,
+                                   ssl=setting.verify_ssl,
-                                   allow_redirects=config.allow_redirects,
+                                   allow_redirects=setting.allow_redirects,
-                                   timeout=timeout,
+                                   timeout=timeout,
-                                   proxy=config.aiohttp_proxy) as resp:
+                                   proxy=setting.aiohttp_proxy) as resp:
-
+
-                try:
+                try:
-                    # 先尝试用utf-8解码
+                    # 先尝试用utf-8解码
-                    text = await resp.text(encoding='utf-8', errors='strict')
+                    text = await resp.text(encoding='utf-8', errors='strict')
-                except UnicodeError:
+                except UnicodeError:
-                    try:
+                    try:
-                        # 再尝试用gb18030解码
+                        # 再尝试用gb18030解码
-                        text = await resp.text(encoding='gb18030',
+                        text = await resp.text(encoding='gb18030',
-                                               errors='strict')
+                                               errors='strict')
-                    except UnicodeError:
+                    except UnicodeError:
-                        # 最后尝试自动解码
+                        # 最后尝试自动解码
-                        text = await resp.text(encoding=None,
+                        text = await resp.text(encoding=None,
-                                               errors='ignore')
+                                               errors='ignore')
-        return resp, text
+        return resp, text
-    except Exception as e:
+    except Exception as e:
-        return e
+        return e
-
+
-
+
-def get_title(markup):
+def get_title(markup):
-    """
+    """
-    获取标题
+    获取标题
-
+
-    :param markup: html标签
+    :param markup: html标签
-    :return: 标题
+    :return: 标题
-    """
+    """
-    soup = BeautifulSoup(markup, 'html.parser')
+    soup = BeautifulSoup(markup, 'html.parser')
-
+
-    title = soup.title
+    title = soup.title
-    if title:
+    if title:
-        return title.text
+        return title.text
-
+
-    h1 = soup.h1
+    h1 = soup.h1
-    if h1:
+    if h1:
-        return h1.text
+        return h1.text
-
+
-    h2 = soup.h2
+    h2 = soup.h2
-    if h2:
+    if h2:
-        return h2.text
+        return h2.text
-
+
-    h3 = soup.h3
+    h3 = soup.h3
-    if h2:
+    if h2:
-        return h3.text
+        return h3.text
-
+
-    desc = soup.find('meta', attrs={'name': 'description'})
+    desc = soup.find('meta', attrs={'name': 'description'})
-    if desc:
+    if desc:
-        return desc['content']
+        return desc['content']
-
+
-    word = soup.find('meta', attrs={'name': 'keywords'})
+    word = soup.find('meta', attrs={'name': 'keywords'})
-    if word:
+    if word:
-        return word['content']
+        return word['content']
-
+
-    text = soup.text
+    text = soup.text
-    if len(text) <= 200:
+    if len(text) <= 200:
-        return text
+        return text
-
+
-    return 'None'
+    return 'None'
-
+
-
+
-def request_callback(future, index, datas):
+def request_callback(future, index, datas):
-    result = future.result()
+    result = future.result()
-    if isinstance(result, BaseException):
+    if isinstance(result, BaseException):
-        logger.log('TRACE', result.args)
+        logger.log('TRACE', result.args)
-        name = utils.get_classname(result)
+        name = utils.get_classname(result)
-        datas[index]['reason'] = name + ' ' + str(result)
+        datas[index]['reason'] = name + ' ' + str(result)
-        datas[index]['request'] = 0
+        datas[index]['request'] = 0
-        datas[index]['alive'] = 0
+        datas[index]['alive'] = 0
-    elif isinstance(result, tuple):
+    elif isinstance(result, tuple):
-        resp, text = result
+        resp, text = result
-        datas[index]['reason'] = resp.reason
+        datas[index]['reason'] = resp.reason
-        datas[index]['status'] = resp.status
+        datas[index]['status'] = resp.status
-        if resp.status == 400 or resp.status >= 500:
+        if resp.status == 400 or resp.status >= 500:
-            datas[index]['request'] = 0
+            datas[index]['request'] = 0
-            datas[index]['alive'] = 0
+            datas[index]['alive'] = 0
-        else:
+        else:
-            datas[index]['request'] = 1
+            datas[index]['request'] = 1
-            datas[index]['alive'] = 1
+            datas[index]['alive'] = 1
-            headers = resp.headers
+            headers = resp.headers
-            datas[index]['banner'] = utils.get_sample_banner(headers)
+            datas[index]['banner'] = utils.get_sample_banner(headers)
-            datas[index]['header'] = str(dict(headers))[1:-1]
+            datas[index]['header'] = str(dict(headers))[1:-1]
-            if isinstance(text, str):
+            if isinstance(text, str):
-                title = get_title(text).strip()
+                title = get_title(text).strip()
-                datas[index]['title'] = utils.remove_invalid_string(title)
+                datas[index]['title'] = utils.remove_invalid_string(title)
-                datas[index]['response'] = utils.remove_invalid_string(text)
+                datas[index]['response'] = utils.remove_invalid_string(text)
-
+
-
+
-def get_connector():
+def get_connector():
-    limit_open_conn = get_limit_conn()
+    limit_open_conn = get_limit_conn()
-    return aiohttp.TCPConnector(ttl_dns_cache=300,
+    return aiohttp.TCPConnector(ttl_dns_cache=300,
-                                ssl=config.verify_ssl,
+                                ssl=setting.verify_ssl,
-                                limit=limit_open_conn,
+                                limit=limit_open_conn,
-                                limit_per_host=config.limit_per_host)
+                                limit_per_host=setting.limit_per_host)
-
+
-
+
-def get_header():
+def get_header():
-    header = None
+    header = None
-    if config.fake_header:
+    if setting.fake_header:
-        header = utils.gen_fake_header()
+        header = utils.gen_fake_header()
-    return header
+    return header
-
+
-
+
-async def bulk_request(data, port):
+async def bulk_request(data, port):
-    ports = get_ports(port)
+    ports = get_ports(port)
-    no_req_data = utils.get_filtered_data(data)
+    no_req_data = utils.get_filtered_data(data)
-    to_req_data = gen_req_data(data, ports)
+    to_req_data = gen_req_data(data, ports)
-    method = config.request_method
+    method = setting.request_method
-    logger.log('INFOR', f'请求使用{method}方法')
+    logger.log('INFOR', f'请求使用{method}方法')
-    logger.log('INFOR', f'正在进行异步子域请求')
+    logger.log('INFOR', f'正在进行异步子域请求')
-    connector = get_connector()
+    connector = get_connector()
-    header = get_header()
+    header = get_header()
-    async with ClientSession(connector=connector, headers=header) as session:
+    async with ClientSession(connector=connector, headers=header) as session:
-        tasks = []
+        tasks = []
-        for i, data in enumerate(to_req_data):
+        for i, data in enumerate(to_req_data):
-            url = data.get('url')
+            url = data.get('url')
-            task = asyncio.ensure_future(fetch(session, url))
+            task = asyncio.ensure_future(fetch(session, url))
-            task.add_done_callback(functools.partial(request_callback,
+            task.add_done_callback(functools.partial(request_callback,
-                                                     index=i,
+                                                     index=i,
-                                                     datas=to_req_data))
+                                                     datas=to_req_data))
-            tasks.append(task)
+            tasks.append(task)
-        # 任务列表里有任务不空时才进行解析
+        # 任务列表里有任务不空时才进行解析
-        if tasks:
+        if tasks:
-            # 等待所有task完成 错误聚合到结果列表里
+            # 等待所有task完成 错误聚合到结果列表里
-            futures = asyncio.as_completed(tasks)
+            futures = asyncio.as_completed(tasks)
-            for future in tqdm.tqdm(futures,
+            for future in tqdm.tqdm(futures,
-                                    total=len(tasks),
+                                    total=len(tasks),
-                                    desc='Request Progress',
+                                    desc='Request Progress',
-                                    ncols=80):
+                                    ncols=80):
-                await future
+                await future
-    return to_req_data + no_req_data
+    return to_req_data + no_req_data
-
+
-
+
-def set_loop_policy():
+def set_loop_policy():
-    try:
+    try:
-        import uvloop
+        import uvloop
-    except ImportError:
+    except ImportError:
-        pass
+        pass
-    else:
+    else:
-        asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
+        asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
-
+
-
+
-def run_request(domain, data, port):
+def run_request(domain, data, port):
-    """
+    """
-    调用子域请求入口函数
+    调用子域请求入口函数
-
+
-    :param str domain: 待请求的主域
+    :param str domain: 待请求的主域
-    :param list data: 待请求的子域数据
+    :param list data: 待请求的子域数据
-    :param str port: 待请求的端口范围
+    :param str port: 待请求的端口范围
-    :return: 请求后得到的结果列表
+    :return: 请求后得到的结果列表
-    :rtype: list
+    :rtype: list
-    """
+    """
-    logger.log('INFOR', f'开始执行子域请求模块')
+    logger.log('INFOR', f'开始执行子域请求模块')
-    set_loop_policy()
+    set_loop_policy()
-    loop = asyncio.get_event_loop()
+    loop = asyncio.get_event_loop()
-    asyncio.set_event_loop(loop)
+    asyncio.set_event_loop(loop)
-    data = utils.set_id_none(data)
+    data = utils.set_id_none(data)
-    request_coroutine = bulk_request(data, port)
+    request_coroutine = bulk_request(data, port)
-    data = loop.run_until_complete(request_coroutine)
+    data = loop.run_until_complete(request_coroutine)
-    # 在关闭事件循环前加入一小段延迟让底层连接得到关闭的缓冲时间
+    # 在关闭事件循环前加入一小段延迟让底层连接得到关闭的缓冲时间
-    loop.run_until_complete(asyncio.sleep(0.25))
+    loop.run_until_complete(asyncio.sleep(0.25))
-    count = utils.count_alive(data)
+    count = utils.count_alive(data)
-    logger.log('INFOR', f'经验证{domain}存活子域{count}个')
+    logger.log('INFOR', f'经验证{domain}存活子域{count}个')
-    return data
+    return data
-
+
-
+
-def save_data(name, data):
+def save_data(name, data):
-    """
+    """
-    保存请求结果到数据库
+    保存请求结果到数据库
-
+
-    :param str name: 保存表名
+    :param str name: 保存表名
-    :param list data: 待保存的数据
+    :param list data: 待保存的数据
-    """
+    """
-    db = Database()
+    db = Database()
-    db.drop_table(name)
+    db.drop_table(name)
-    db.create_table(name)
+    db.create_table(name)
-    db.save_db(name, data, 'request')
+    db.save_db(name, data, 'request')
-    db.close()
+    db.close()

common/resolve.py

@@ -1,169 +1,169 @@
-import gc
+import gc
-import json
+import json
-
+
-import config
+from config.log import logger
-from config import logger
+from config import setting
-from common import utils
+from common import utils
-from common.database import Database
+from common.database import Database
-
+
-
+
-def filter_subdomain(data):
+def filter_subdomain(data):
-    """
+    """
-    过滤出无解析内容的子域到新的子域列表
+    过滤出无解析内容的子域到新的子域列表
-
+
-    :param list data: 待过滤的数据列表
+    :param list data: 待过滤的数据列表
-    :return: 符合条件的子域列表
+    :return: 符合条件的子域列表
-    """
+    """
-    logger.log('DEBUG', f'正在过滤出待解析的子域')
+    logger.log('DEBUG', f'正在过滤出待解析的子域')
-    subdomains = []
+    subdomains = []
-    for data in data:
+    for data in data:
-        if not data.get('content'):
+        if not data.get('content'):
-            subdomain = data.get('subdomain')
+            subdomain = data.get('subdomain')
-            subdomains.append(subdomain)
+            subdomains.append(subdomain)
-    return subdomains
+    return subdomains
-
+
-
+
-def update_data(data, records):
+def update_data(data, records):
-    """
+    """
-    更新解析结果
+    更新解析结果
-
+
-    :param list data: 待更新的数据列表
+    :param list data: 待更新的数据列表
-    :param dict records: 解析结果字典
+    :param dict records: 解析结果字典
-    :return: 更新后的数据列表
+    :return: 更新后的数据列表
-    """
+    """
-    logger.log('DEBUG', f'正在更新解析结果')
+    logger.log('DEBUG', f'正在更新解析结果')
-    if not records:
+    if not records:
-        logger.log('ERROR', f'无有效解析结果')
+        logger.log('ERROR', f'无有效解析结果')
-        return data
+        return data
-    for index, items in enumerate(data):
+    for index, items in enumerate(data):
-        if not items.get('content'):
+        if not items.get('content'):
-            subdomain = items.get('subdomain')
+            subdomain = items.get('subdomain')
-            record = records.get(subdomain)
+            record = records.get(subdomain)
-            items.update(record)
+            items.update(record)
-            data[index] = items
+            data[index] = items
-    return data
+    return data
-
+
-
+
-def save_data(name, data):
+def save_data(name, data):
-    """
+    """
-    保存解析结果到数据库
+    保存解析结果到数据库
-
+
-    :param str name: 保存表名
+    :param str name: 保存表名
-    :param list data: 待保存的数据
+    :param list data: 待保存的数据
-    """
+    """
-    logger.log('INFOR', f'正在保存解析结果')
+    logger.log('INFOR', f'正在保存解析结果')
-    db = Database()
+    db = Database()
-    db.drop_table(name)
+    db.drop_table(name)
-    db.create_table(name)
+    db.create_table(name)
-    db.save_db(name, data, 'resolve')
+    db.save_db(name, data, 'resolve')
-    db.close()
+    db.close()
-
+
-
+
-def save_subdomains(save_path, subdomain_list):
+def save_subdomains(save_path, subdomain_list):
-    logger.log('DEBUG', f'正在保存待解析的子域')
+    logger.log('DEBUG', f'正在保存待解析的子域')
-    subdomain_data = '\n'.join(subdomain_list)
+    subdomain_data = '\n'.join(subdomain_list)
-    if not utils.save_data(save_path, subdomain_data):
+    if not utils.save_data(save_path, subdomain_data):
-        logger.log('FATAL', '保存待解析的子域出错')
+        logger.log('FATAL', '保存待解析的子域出错')
-        exit(1)
+        exit(1)
-
+
-
+
-def deal_output(output_path):
+def deal_output(output_path):
-    logger.log('INFOR', f'正在处理解析结果')
+    logger.log('INFOR', f'正在处理解析结果')
-    records = dict()  # 用来记录所有域名解析数据
+    records = dict()  # 用来记录所有域名解析数据
-    with open(output_path) as fd:
+    with open(output_path) as fd:
-        for line in fd:
+        for line in fd:
-            line = line.strip()
+            line = line.strip()
-            try:
+            try:
-                items = json.loads(line)
+                items = json.loads(line)
-            except Exception as e:
+            except Exception as e:
-                logger.log('ERROR', e.args)
+                logger.log('ERROR', e.args)
-                logger.log('ERROR', f'解析行{line}出错跳过解析该行')
+                logger.log('ERROR', f'解析行{line}出错跳过解析该行')
-                continue
+                continue
-            record = dict()
+            record = dict()
-            record['resolver'] = items.get('resolver')
+            record['resolver'] = items.get('resolver')
-            qname = items.get('name')[:-1]  # 去出最右边的`.`点号
+            qname = items.get('name')[:-1]  # 去出最右边的`.`点号
-            status = items.get('status')
+            status = items.get('status')
-            if status != 'NOERROR':
+            if status != 'NOERROR':
-                record['alive'] = 0
+                record['alive'] = 0
-                record['resolve'] = 0
+                record['resolve'] = 0
-                record['reason'] = status
+                record['reason'] = status
-                records[qname] = record
+                records[qname] = record
-                continue
+                continue
-            data = items.get('data')
+            data = items.get('data')
-            if 'answers' not in data:
+            if 'answers' not in data:
-                record['alive'] = 0
+                record['alive'] = 0
-                record['resolve'] = 0
+                record['resolve'] = 0
-                record['reason'] = 'NOANSWER'
+                record['reason'] = 'NOANSWER'
-                records[qname] = record
+                records[qname] = record
-                continue
+                continue
-            flag = False
+            flag = False
-            cname = list()
+            cname = list()
-            ips = list()
+            ips = list()
-            public = list()
+            public = list()
-            ttls = list()
+            ttls = list()
-            answers = data.get('answers')
+            answers = data.get('answers')
-            for answer in answers:
+            for answer in answers:
-                if answer.get('type') == 'A':
+                if answer.get('type') == 'A':
-                    flag = True
+                    flag = True
-                    cname.append(answer.get('name')[:-1])  # 去出最右边的`.`点号
+                    cname.append(answer.get('name')[:-1])  # 去出最右边的`.`点号
-                    ip = answer.get('data')
+                    ip = answer.get('data')
-                    ips.append(ip)
+                    ips.append(ip)
-                    ttl = answer.get('ttl')
+                    ttl = answer.get('ttl')
-                    ttls.append(str(ttl))
+                    ttls.append(str(ttl))
-                    is_public = utils.ip_is_public(ip)
+                    is_public = utils.ip_is_public(ip)
-                    public.append(str(is_public))
+                    public.append(str(is_public))
-                    record['resolve'] = 1
+                    record['resolve'] = 1
-                    record['reason'] = status
+                    record['reason'] = status
-                    record['cname'] = ','.join(cname)
+                    record['cname'] = ','.join(cname)
-                    record['content'] = ','.join(ips)
+                    record['content'] = ','.join(ips)
-                    record['public'] = ','.join(public)
+                    record['public'] = ','.join(public)
-                    record['ttl'] = ','.join(ttls)
+                    record['ttl'] = ','.join(ttls)
-                    records[qname] = record
+                    records[qname] = record
-            if not flag:
+            if not flag:
-                record['alive'] = 0
+                record['alive'] = 0
-                record['resolve'] = 0
+                record['resolve'] = 0
-                record['reason'] = 'NOARECORD'
+                record['reason'] = 'NOARECORD'
-                records[qname] = record
+                records[qname] = record
-    return records
+    return records
-
+
-
+
-def run_resolve(domain, data):
+def run_resolve(domain, data):
-    """
+    """
-    调用子域解析入口函数
+    调用子域解析入口函数
-
+
-    :param str domain: 待解析的主域
+    :param str domain: 待解析的主域
-    :param list data: 待解析的子域数据列表
+    :param list data: 待解析的子域数据列表
-    :return: 解析得到的结果列表
+    :return: 解析得到的结果列表
-    :rtype: list
+    :rtype: list
-    """
+    """
-    logger.log('INFOR', f'开始解析{domain}的子域')
+    logger.log('INFOR', f'开始解析{domain}的子域')
-    subdomains = filter_subdomain(data)
+    subdomains = filter_subdomain(data)
-    if not subdomains:
+    if not subdomains:
-        return data
+        return data
-
+
-    massdns_dir = config.third_party_dir.joinpath('massdns')
+    massdns_dir = setting.third_party_dir.joinpath('massdns')
-    result_dir = config.result_save_dir
+    result_dir = setting.result_save_dir
-    temp_dir = result_dir.joinpath('temp')
+    temp_dir = result_dir.joinpath('temp')
-    utils.check_dir(temp_dir)
+    utils.check_dir(temp_dir)
-    massdns_path = utils.get_massdns_path(massdns_dir)
+    massdns_path = utils.get_massdns_path(massdns_dir)
-    timestring = utils.get_timestring()
+    timestring = utils.get_timestring()
-
+
-    save_name = f'collected_subdomains_{domain}_{timestring}.txt'
+    save_name = f'collected_subdomains_{domain}_{timestring}.txt'
-    save_path = temp_dir.joinpath(save_name)
+    save_path = temp_dir.joinpath(save_name)
-    save_subdomains(save_path, subdomains)
+    save_subdomains(save_path, subdomains)
-    del subdomains
+    del subdomains
-    gc.collect()
+    gc.collect()
-
+
-    output_name = f'resolved_result_{domain}_{timestring}.json'
+    output_name = f'resolved_result_{domain}_{timestring}.json'
-    output_path = temp_dir.joinpath(output_name)
+    output_path = temp_dir.joinpath(output_name)
-    log_path = result_dir.joinpath('massdns.log')
+    log_path = result_dir.joinpath('massdns.log')
-
+
-    ns_path = config.brute_nameservers_path
+    ns_path = setting.brute_nameservers_path
-
+
-    utils.call_massdns(massdns_path, save_path, ns_path,
+    utils.call_massdns(massdns_path, save_path, ns_path,
-                       output_path, log_path, quiet_mode=True)
+                       output_path, log_path, quiet_mode=True)
-
+
-    records = deal_output(output_path)
+    records = deal_output(output_path)
-    data = update_data(data, records)
+    data = update_data(data, records)
-    logger.log('INFOR', f'结束解析{domain}的子域')
+    logger.log('INFOR', f'结束解析{domain}的子域')
-    return data
+    return data

common/search.py

@@ -1,54 +1,54 @@
-import config
+from config import setting
-from .module import Module
+from .module import Module
-from . import utils
+from . import utils
-
+
-
+
-class Search(Module):
+class Search(Module):
-    """
+    """
-    搜索基类
+    搜索基类
-    """
+    """
-    def __init__(self):
+    def __init__(self):
-        Module.__init__(self)
+        Module.__init__(self)
-        self.page_num = 0  # 要显示搜索起始条数
+        self.page_num = 0  # 要显示搜索起始条数
-        self.per_page_num = 50  # 每页显示搜索条数
+        self.per_page_num = 50  # 每页显示搜索条数
-        self.recursive_search = config.enable_recursive_search
+        self.recursive_search = setting.enable_recursive_search
-        self.recursive_times = config.search_recursive_times
+        self.recursive_times = setting.search_recursive_times
-
+
-    @staticmethod
+    @staticmethod
-    def filter(domain, subdomain):
+    def filter(domain, subdomain):
-        """
+        """
-        生成搜索过滤语句
+        生成搜索过滤语句
-        使用搜索引擎支持的-site:语法过滤掉搜索页面较多的子域以发现新域
+        使用搜索引擎支持的-site:语法过滤掉搜索页面较多的子域以发现新域
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param set subdomain: 子域名集合
+        :param set subdomain: 子域名集合
-        :return: 过滤语句
+        :return: 过滤语句
-        :rtype: str
+        :rtype: str
-        """
+        """
-        statements_list = []
+        statements_list = []
-        subdomains_temp = set(map(lambda x: x + '.' + domain,
+        subdomains_temp = set(map(lambda x: x + '.' + domain,
-                                  config.subdomains_common))
+                                  setting.subdomains_common))
-        subdomains_temp = list(subdomain.intersection(subdomains_temp))
+        subdomains_temp = list(subdomain.intersection(subdomains_temp))
-        for i in range(0, len(subdomains_temp), 2):  # 同时排除2个子域
+        for i in range(0, len(subdomains_temp), 2):  # 同时排除2个子域
-            statements_list.append(''.join(set(map(lambda s: ' -site:' + s,
+            statements_list.append(''.join(set(map(lambda s: ' -site:' + s,
-                                                   subdomains_temp[i:i + 2]))))
+                                                   subdomains_temp[i:i + 2]))))
-        return statements_list
+        return statements_list
-
+
-    def match_location(self, domain, url):
+    def match_location(self, domain, url):
-        """
+        """
-        匹配跳转之后的url
+        匹配跳转之后的url
-        针对部分搜索引擎(如百度搜索)搜索展示url时有显示不全的情况
+        针对部分搜索引擎(如百度搜索)搜索展示url时有显示不全的情况
-        此函数会向每条结果的链接发送head请求获取响应头的location值并做子域匹配
+        此函数会向每条结果的链接发送head请求获取响应头的location值并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str url: 展示结果的url链接
+        :param str url: 展示结果的url链接
-        :return: 匹配的子域
+        :return: 匹配的子域
-        :rtype set
+        :rtype set
-        """
+        """
-        resp = self.head(url, check=False, allow_redirects=False)
+        resp = self.head(url, check=False, allow_redirects=False)
-        if not resp:
+        if not resp:
-            return set()
+            return set()
-        location = resp.headers.get('location')
+        location = resp.headers.get('location')
-        if not location:
+        if not location:
-            return set()
+            return set()
-        return set(utils.match_subdomain(domain, location))
+        return set(utils.match_subdomain(domain, location))

config/log.py

@@ -0,0 +1,35 @@
+import sys
+import pathlib
+
+from loguru import logger
+
+# 路径设置
+relative_directory = pathlib.Path(__file__).parent.parent  # OneForAll代码相对路径
+result_save_dir = relative_directory.joinpath('results')  # 结果保存目录
+log_path = result_save_dir.joinpath('oneforall.log')  # OneForAll日志保存路径
+
+# 日志配置
+# 终端日志输出格式
+stdout_fmt = '<cyan>{time:HH:mm:ss,SSS}</cyan> ' \
+             '[<level>{level: <5}</level>] ' \
+             '<blue>{module}</blue>:<cyan>{line}</cyan> - ' \
+             '<level>{message}</level>'
+# 日志文件记录格式
+logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
+              '[<level>{level: <5}</level>] ' \
+              '<cyan>{process.name}({process.id})</cyan>:' \
+              '<cyan>{thread.name: <18}({thread.id: <5})</cyan> | ' \
+              '<blue>{module}</blue>.<blue>{function}</blue>:' \
+              '<blue>{line}</blue> - <level>{message}</level>'
+
+logger.remove()
+logger.level(name='TRACE', no=5, color='<cyan><bold>', icon='✏️')
+logger.level(name='DEBUG', no=10, color='<blue><bold>', icon='🐞 ')
+logger.level(name='INFOR', no=20, color='<green><bold>', icon='ℹ️')
+logger.level(name='ALERT', no=30, color='<yellow><bold>', icon='⚠️')
+logger.level(name='ERROR', no=40, color='<red><bold>', icon='❌️')
+logger.level(name='FATAL', no=50, color='<RED><bold>', icon='☠️')
+
+logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)
+logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True,
+           encoding='utf-8')

config/setting.py

@@ -2,15 +2,12 @@
 """
 OneForAll配置
 """
-import os
-import sys
-import pathlib
 
+import pathlib
 import urllib3
-from loguru import logger
 
 # 路径设置
-relative_directory = pathlib.Path(__file__).parent  # OneForAll代码相对路径
+relative_directory = pathlib.Path(__file__).parent.parent  # OneForAll代码相对路径
 module_dir = relative_directory.joinpath('modules')  # OneForAll模块目录
 third_party_dir = relative_directory.joinpath('thirdparty')  # 三方工具目录
 data_storage_dir = relative_directory.joinpath('data')  # 数据存放目录
@@ -158,33 +155,3 @@ subdomains_common = {'i', 'w', 'm', 'en', 'us', 'zh', 'w3', 'app', 'bbs',
                      'help', 'live', 'mall', 'blogs', 'files', 'forum',
                      'store', 'mobile'}
 
-# 日志配置
-# 终端日志输出格式
-stdout_fmt = '<cyan>{time:HH:mm:ss,SSS}</cyan> ' \
-             '[<level>{level: <5}</level>] ' \
-             '<blue>{module}</blue>:<cyan>{line}</cyan> - ' \
-             '<level>{message}</level>'
-# 日志文件记录格式
-logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
-              '[<level>{level: <5}</level>] ' \
-              '<cyan>{process.name}({process.id})</cyan>:' \
-              '<cyan>{thread.name: <18}({thread.id: <5})</cyan> | ' \
-              '<blue>{module}</blue>.<blue>{function}</blue>:' \
-              '<blue>{line}</blue> - <level>{message}</level>'
-
-log_path = result_save_dir.joinpath('oneforall.log')
-
-logger.remove()
-logger.level(name='TRACE', no=5, color='<cyan><bold>', icon='✏️')
-logger.level(name='DEBUG', no=10, color='<blue><bold>', icon='🐞 ')
-logger.level(name='INFOR', no=20, color='<green><bold>', icon='ℹ️')
-logger.level(name='ALERT', no=30, color='<yellow><bold>', icon='⚠️')
-logger.level(name='ERROR', no=40, color='<red><bold>', icon='❌️')
-logger.level(name='FATAL', no=50, color='<RED><bold>', icon='☠️')
-
-if not os.environ.get('PYTHONIOENCODING'):  # 设置编码
-    os.environ['PYTHONIOENCODING'] = 'utf-8'
-
-logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)
-logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True,
-           encoding='utf-8')

data/subnames.txt

@@ -2391,7 +2391,7 @@ hadoop
 dataanalytics
 cloudera
 hortonworks
-spark	
+spark
 databricks
 browserstack
 npm
@@ -2399,7 +2399,7 @@ tomcat
 cloudbees
 hashicorp
 postgres
-enterprisedb	
+enterprisedb
 confluent
 datastax
 mindquarry
@@ -654765,7 +654765,7 @@ _ftp
 _nfs
 _pptp
 _ntp
-_snmp	
+_snmp
 _xmpp-server
 _jabber
 a.auth-ns
@@ -818945,27 +818945,27 @@ xqual
 xstudio
 zentrack
 Se0
-ad 
+ad
-db 
+db
-db01 
+db01
-db02 
+db02
-db1 
+db1
-db2 
+db2
-dhcp 
+dhcp
-id 
+id
-int 
+int
-intl 
+intl
-intranet 
+intranet
-ldap 
+ldap
-mom 
+mom
-pop 
+pop
-rtr		
+rtr
-sec 
+sec
-securid 
+securid
-smtp 
+smtp
-ups 
+ups
-webmail 
+webmail
-www-int 
+www-int
 netdisk
 zhuanlan
 vlike

dbexport.py

@@ -1,57 +1,57 @@
-#!/usr/bin/python3
+#!/usr/bin/python3
-# coding=utf-8
+# coding=utf-8
-
+
-"""
+"""
-OneForAll数据库导出模块
+OneForAll数据库导出模块
-
+
-:copyright: Copyright (c) 2019, Jing Ling. All rights reserved.
+:copyright: Copyright (c) 2019, Jing Ling. All rights reserved.
-:license: GNU General Public License v3.0, see LICENSE for more details.
+:license: GNU General Public License v3.0, see LICENSE for more details.
-"""
+"""
-
+
-import fire
+import fire
-
+
-from common import utils
+from common import utils
-from common.database import Database
+from common.database import Database
-from config import logger
+from config.log import logger
-
+
-
+
-def export(table, db=None, alive=False, limit=None, path=None, format='csv', show=False):
+def export(table, db=None, alive=False, limit=None, path=None, format='csv', show=False):
-    """
+    """
-    OneForAll数据库导出模块
+    OneForAll数据库导出模块
-
+
-    Example:
+    Example:
-        python3 dbexport.py --table name --format csv --dir= ./result.csv
+        python3 dbexport.py --table name --format csv --dir= ./result.csv
-        python3 dbexport.py --db result.db --table name --show False
+        python3 dbexport.py --db result.db --table name --show False
-
+
-    Note:
+    Note:
-        参数alive可选值True，False分别表示导出存活，全部子域结果
+        参数alive可选值True，False分别表示导出存活，全部子域结果
-        参数format可选格式有'txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
+        参数format可选格式有'txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
-                          'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
+                          'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
-        参数path默认None使用OneForAll结果目录自动生成路径
+        参数path默认None使用OneForAll结果目录自动生成路径
-
+
-    :param str table:   要导出的表
+    :param str table:   要导出的表
-    :param str db:      要导出的数据库路径(默认为results/result.sqlite3)
+    :param str db:      要导出的数据库路径(默认为results/result.sqlite3)
-    :param bool alive:  只导出存活的子域结果(默认False)
+    :param bool alive:  只导出存活的子域结果(默认False)
-    :param str limit:   导出限制条件(默认None)
+    :param str limit:   导出限制条件(默认None)
-    :param str format:  导出文件格式(默认csv)
+    :param str format:  导出文件格式(默认csv)
-    :param str path:    导出文件路径(默认None)
+    :param str path:    导出文件路径(默认None)
-    :param bool show:   终端显示导出数据(默认False)
+    :param bool show:   终端显示导出数据(默认False)
-    """
+    """
-
+
-    database = Database(db)
+    database = Database(db)
-    rows = database.export_data(table, alive, limit)
+    rows = database.export_data(table, alive, limit)
-    format = utils.check_format(format, len(rows))
+    format = utils.check_format(format, len(rows))
-    path = utils.check_path(path, table, format)
+    path = utils.check_path(path, table, format)
-    if show:
+    if show:
-        print(rows.dataset)
+        print(rows.dataset)
-    data = rows.export(format)
+    data = rows.export(format)
-    database.close()
+    database.close()
-    utils.save_data(path, data)
+    utils.save_data(path, data)
-    logger.log('INFOR', f'{table}主域的子域结果 {path}')
+    logger.log('INFOR', f'{table}主域的子域结果 {path}')
-    data_dict = rows.as_dict()
+    data_dict = rows.as_dict()
-    return data_dict
+    return data_dict
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    fire.Fire(export)
+    fire.Fire(export)
-    # save('example_com_last', format='txt')
+    # save('example_com_last', format='txt')

docs/directory_structure.md

@@ -0,0 +1,178 @@
+```
+D:.
+|   .gitignore
+|   .travis.yml
+|   brute.py        可以单独运行的子域爆破模块
+|   collect.py      各个收集模块上层调用
+|   dbexport.py     可以单独运行的数据库导出模块
+|   Dockerfile
+|   LICENSE
+|   oneforall.py    OneForAll主入口
+|   Pipfile
+|   Pipfile.lock
+|   README.en.md
+|   README.md
+|   requirements.txt
+|   takeover.py     可以单独运行的子域接口风险检查模块
+|   _config.yml
+|
++---.github
+|   +---ISSUE_TEMPLATE
+|   |       bug_report.md
+|   |       bug_report_zh.md
+|   |       custom.md
+|   |       feature_request.md
+|   |
+|   \---workflows
+|           test.yml
+|
+|
++---common 公共调用模块
+|       crawl.py
+|       database.py
+|       domain.py
+|       lookup.py
+|       module.py
+|       query.py
+|       request.py
+|       resolve.py
+|       search.py
+|       utils.py
+|       __init__.py
+|
++---config 配置目录
+|       api.py      部分收集模块的API配置文件
+|       log.py      日志模块配置文件
+|       setting.py  OneForAll主要配置文件
+|
++---data 存放一些所需数据
+|       authoritative_dns.txt   临时存放开启了泛解析域名的权威DNS名称服务器IP地址
+|       big_subnames.txt        子域爆破超大字典
+|       cn_nameservers.txt      中国主流名称服务器IP地址
+|       fingerprints.json       检查子域接管风险的指纹
+|       nameservers.txt         全球主流名称服务器IP地址
+|       next_subnames.txt       下一层子域字典
+|       public_suffix_list.dat  顶级域名后缀
+|       srv_prefixes.json       常见SRV记录前缀名
+|       subnames.txt            子域爆破常见字典
+|
++---docs 有关文档
+|       changes.md
+|       collection_modules.md
+|       contributors.md
+|       installation_dependency.md
+|       todo.md
+|       troubleshooting.md
+|       usage_example.svg
+|       usage_help.en.md
+|       usage_help.md
+|
++---images
+|       Database.png
+|       Donate.png
+|       Result.png
+|
++---modules
+|   +---autotake 自动接管模块
+|   |       github.py
+|   |
+|   +---certificates 利用证书透明度收集子域模块
+|   |       censys_api.py
+|   |       certspotter.py
+|   |       crtsh.py
+|   |       entrust.py
+|   |       google.py
+|   |       spyse_api.py
+|   |
+|   +---check 常规检查收集子域模块
+|   |       axfr.py
+|   |       cdx.py
+|   |       cert.py
+|   |       csp.py
+|   |       robots.py
+|   |       sitemap.py
+|   |
+|   +---crawl 利用网上爬虫档案收集子域模块
+|   |       archivecrawl.py
+|   |       commoncrawl.py
+|   |
+|   +---datasets 利用DNS数据集收集子域模块
+|   |       binaryedge_api.py
+|   |       bufferover.py
+|   |       cebaidu.py
+|   |       chinaz.py
+|   |       chinaz_api.py
+|   |       circl_api.py
+|   |       dnsdb_api.py
+|   |       dnsdumpster.py
+|   |       hackertarget.py
+|   |       ip138.py
+|   |       ipv4info_api.py
+|   |       netcraft.py
+|   |       passivedns_api.py
+|   |       ptrarchive.py
+|   |       qianxun.py
+|   |       rapiddns.py
+|   |       riddler.py
+|   |       robtex.py
+|   |       securitytrails_api.py
+|   |       sitedossier.py
+|   |       threatcrowd.py
+|   |       wzpc.py
+|   |       ximcx.py
+|   |
+|   +---dnsquery 利用DNS查询收集子域模块
+|   |       mx.py
+|   |       ns.py
+|   |       soa.py
+|   |       srv.py
+|   |       txt.py
+|   |
+|   +---intelligence 利用威胁情报平台数据收集子域模块
+|   |       alienvault.py
+|   |       riskiq_api.py
+|   |       threatbook_api.py
+|   |       threatminer.py
+|   |       virustotal.py
+|   |       virustotal_api.py
+|   |
+|   \---search 利用搜索引擎发现子域模块
+|           ask.py
+|           baidu.py
+|           bing.py
+|           bing_api.py
+|           exalead.py
+|           fofa_api.py
+|           gitee.py
+|           github_api.py
+|           google.py
+|           google_api.py
+|           shodan_api.py
+|           so.py
+|           sogou.py
+|           yahoo.py
+|           yandex.py
+|           zoomeye_api.py
+|
++---results 结果目录
++---test 测试目录
+|       example.py
+|
+\---thirdparty 存放要调用的三方工具
+    \---massdns
+        |   LICENSE
+        |   massdns_darwin_x86_64
+        |   massdns_linux_i686
+        |   massdns_linux_x86_64
+        |   README.md
+        |
+        \---windows
+            +---x64
+            |       cygwin1.dll
+            |       massdns_windows_amd64.exe
+            |
+            \---x86
+                    cyggcc_s-1.dll
+                    cygwin1.dll
+                    massdns_windows_i686.exe
+```

docs/installation_dependency.md

@@ -12,7 +12,6 @@ cd OneForAll/
 python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
 pip3 install pipenv -i https://mirrors.aliyun.com/pypi/simple/
 pipenv install --python 3.8
-cd oneforall
 pipenv run python oneforall.py --help
 ```
 
@@ -21,7 +20,6 @@ pipenv run python oneforall.py --help
 cd OneForAll/
 python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
 pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
-cd oneforall/
 python oneforall.py --help
 ```
 ## Linux系统
@@ -49,7 +47,6 @@ cd OneForAll/
 sudo python3.8 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
 sudo apt install pipenv
 sudo pipenv install --python 3.8
-cd oneforall
 pipenv run python3 oneforall.py --help
 ```
 
@@ -60,7 +57,6 @@ sudo apt install python3-pip
 sudo python3.8 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
 sudo pip3 install uvloop -i https://mirrors.aliyun.com/pypi/simple/
 sudo pip3 install --ignore-installed -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
-cd oneforall/
 python3.8 oneforall.py --help
 ```
 ## Darwin系统
@@ -71,7 +67,6 @@ cd OneForAll/
 python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
 pip3 install pipenv -i https://mirrors.aliyun.com/pypi/simple/
 pipenv install --python 3.8
-cd oneforall
 pipenv run python3 oneforall.py --help
 ```
 
@@ -81,6 +76,5 @@ cd OneForAll/
 python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
 pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
 pip3 install uvloop -i https://mirrors.aliyun.com/pypi/simple/
-cd oneforall/
 python3 oneforall.py --help
 ```

docs/usage_help.en.md

@@ -82,7 +82,6 @@ The OneForAll command line interface is based on [Fire](https://github.com/googl
 
    - Third, considering the blasting efficiency, there is no HTTP response volume similarity comparison and response volume content judgment,  this function has not been implemented yet, and will be implemented if necessary.
 
-   After not rigorous testing, in the 16-core CPU, using 16 processes 64 coroutines, 100M network bandwidth environment, run two million dictionaries about 10 minutes to run, about 3333 subdomains per second.
 
    ```bash
    python aiobrute.py --help

docs/usage_help.md

@@ -67,9 +67,10 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
 
    关于泛解析问题处理程序首先会访问一个随机的子域判断是否泛解析，如果使用了泛解析则是通过以下判断处理：
    - 一是主要是与泛解析的IP集合和TTL值做对比，可以参考[这篇文章](http://sh3ll.me/archives/201704041222.txt)。
+   
    - 二是多次解析到同一IP集合次数（默认设置为10，可以在config.py设置大小）
+   
    - 考虑爆破效率问题目前还没有加上HTTP响应体相似度对比和响应体内容判断
-   经过测试在16核心的CPU，使用16进程64协程，100M带宽的环境下，设置任务分割为50000，跑两百万字典大概10分钟左右跑完，大概3333个子域每秒。
 
    ```bash
    python aiobrute.py --help

modules/autotake/github.py

@@ -8,7 +8,7 @@ github自动接管
 import json
 import base64
 import requests
-import api
+from config import api
 
 HEADERS = {
     "Accept": "application/json, text/javascript, */*; q=0.01",

modules/certificates/censys_api.py

@@ -1,72 +1,72 @@
-import api
+from config import api
-from common.query import Query
+from common.query import Query
-from config import logger
+from config.log import logger
-
+
-
+
-class CensysAPI(Query):
+class CensysAPI(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Certificate'
+        self.module = 'Certificate'
-        self.source = "CensysAPIQuery"
+        self.source = "CensysAPIQuery"
-        self.addr = 'https://www.censys.io/api/v1/search/certificates'
+        self.addr = 'https://www.censys.io/api/v1/search/certificates'
-        self.id = api.censys_api_id
+        self.id = api.censys_api_id
-        self.secret = api.censys_api_secret
+        self.secret = api.censys_api_secret
-        self.delay = 3.0  # Censys 接口查询速率限制 最快2.5秒查1次
+        self.delay = 3.0  # Censys 接口查询速率限制 最快2.5秒查1次
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        data = {
+        data = {
-            'query': f'parsed.names: {self.domain}',
+            'query': f'parsed.names: {self.domain}',
-            'page': 1,
+            'page': 1,
-            'fields': ['parsed.subject_dn', 'parsed.names'],
+            'fields': ['parsed.subject_dn', 'parsed.names'],
-            'flatten': True}
+            'flatten': True}
-        resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
+        resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
-        if not resp:
+        if not resp:
-            return
+            return
-        json = resp.json()
+        json = resp.json()
-        status = json.get('status')
+        status = json.get('status')
-        if status != 'ok':
+        if status != 'ok':
-            logger.log('ALERT', status)
+            logger.log('ALERT', status)
-            return
+            return
-        subdomains = self.match(self.domain, str(json))
+        subdomains = self.match(self.domain, str(json))
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-        pages = json.get('metadata').get('pages')
+        pages = json.get('metadata').get('pages')
-        for page in range(2, pages + 1):
+        for page in range(2, pages + 1):
-            data['page'] = page
+            data['page'] = page
-            resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
+            resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(self.domain, str(resp.json()))
+            subdomains = self.match(self.domain, str(resp.json()))
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.id, self.secret):
+        if not self.check(self.id, self.secret):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = CensysAPI(domain)
+    query = CensysAPI(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/certificates/certspotter.py

@@ -1,53 +1,53 @@
-from common import utils
+from common import utils
-from common.query import Query
+from common.query import Query
-
+
-
+
-class CertSpotter(Query):
+class CertSpotter(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Certificate'
+        self.module = 'Certificate'
-        self.source = 'CertSpotterQuery'
+        self.source = 'CertSpotterQuery'
-        self.addr = 'https://api.certspotter.com/v1/issuances'
+        self.addr = 'https://api.certspotter.com/v1/issuances'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'domain': self.domain,
+        params = {'domain': self.domain,
-                  'include_subdomains': 'true',
+                  'include_subdomains': 'true',
-                  'expand': 'dns_names'}
+                  'expand': 'dns_names'}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = utils.match_subdomain(self.domain, str(resp.json()))
+        subdomains = utils.match_subdomain(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-
+
-    """
+    """
-    query = CertSpotter(domain)
+    query = CertSpotter(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/certificates/crtsh.py

@@ -1,49 +1,49 @@
-from common import utils
+from common import utils
-from common.query import Query
+from common.query import Query
-
+
-
+
-class Crtsh(Query):
+class Crtsh(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Certificate'
+        self.module = 'Certificate'
-        self.source = 'CrtshQuery'
+        self.source = 'CrtshQuery'
-        self.addr = 'https://crt.sh/'
+        self.addr = 'https://crt.sh/'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'q': f'%.{self.domain}', 'output': 'json'}
+        params = {'q': f'%.{self.domain}', 'output': 'json'}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = utils.match_subdomain(self.domain, str(resp.json()))
+        subdomains = utils.match_subdomain(self.domain, str(resp.json()))
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = Crtsh(domain)
+    query = Crtsh(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/certificates/entrust.py

@@ -1,51 +1,51 @@
-from common import utils
+from common import utils
-from common.query import Query
+from common.query import Query
-
+
-
+
-class Entrust(Query):
+class Entrust(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Certificate'
+        self.module = 'Certificate'
-        self.source = 'EntrustQuery'
+        self.source = 'EntrustQuery'
-        self.addr = 'https://ctsearch.entrust.com/api/v1/certificates'
+        self.addr = 'https://ctsearch.entrust.com/api/v1/certificates'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'fields': 'subjectDN',
+        params = {'fields': 'subjectDN',
-                  'domain': self.domain,
+                  'domain': self.domain,
-                  'includeExpired': 'true'}
+                  'includeExpired': 'true'}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = utils.match_subdomain(self.domain, str(resp.json()))
+        subdomains = utils.match_subdomain(self.domain, str(resp.json()))
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = Entrust(domain)
+    query = Entrust(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/certificates/google.py

@@ -1,53 +1,53 @@
-from common import utils
+from common import utils
-from common.query import Query
+from common.query import Query
-
+
-
+
-class Google(Query):
+class Google(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Certificate'
+        self.module = 'Certificate'
-        self.source = 'GoogleQuery'
+        self.source = 'GoogleQuery'
-        self.addr = 'https://transparencyreport.google.com/' \
+        self.addr = 'https://transparencyreport.google.com/' \
-                    'transparencyreport/api/v3/httpsreport/ct/certsearch'
+                    'transparencyreport/api/v3/httpsreport/ct/certsearch'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'include_expired': 'true',
+        params = {'include_expired': 'true',
-                  'include_subdomains': 'true',
+                  'include_subdomains': 'true',
-                  'domain': self.domain}
+                  'domain': self.domain}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = utils.match_subdomain(self.domain, resp.text)
+        subdomains = utils.match_subdomain(self.domain, resp.text)
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = Google(domain)
+    query = Google(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/certificates/spyse_api.py

@@ -1,4 +1,4 @@
-import api
+from config import api
 from common import utils
 from common.query import Query
 

modules/check/axfr.py

@@ -1,97 +1,97 @@
-"""
+"""
-查询域名的NS记录(域名服务器记录，记录该域名由哪台域名服务器解析)，检查查出的域名服务器是
+查询域名的NS记录(域名服务器记录，记录该域名由哪台域名服务器解析)，检查查出的域名服务器是
-否开启DNS域传送，如果开启且没做访问控制和身份验证便加以利用获取域名的所有记录。
+否开启DNS域传送，如果开启且没做访问控制和身份验证便加以利用获取域名的所有记录。
-
+
-DNS域传送(DNS zone transfer)指的是一台备用域名服务器使用来自主域名服务器的数据刷新自己
+DNS域传送(DNS zone transfer)指的是一台备用域名服务器使用来自主域名服务器的数据刷新自己
-的域数据库，目的是为了做冗余备份，防止主域名服务器出现故障时 dns 解析不可用。
+的域数据库，目的是为了做冗余备份，防止主域名服务器出现故障时 dns 解析不可用。
-当主服务器开启DNS域传送同时又对来请求的备用服务器未作访问控制和身份验证便可以利用此漏洞获
+当主服务器开启DNS域传送同时又对来请求的备用服务器未作访问控制和身份验证便可以利用此漏洞获
-取某个域的所有记录。
+取某个域的所有记录。
-"""
+"""
-import dns.resolver
+import dns.resolver
-import dns.zone
+import dns.zone
-
+
-from common import utils
+from common import utils
-from common.module import Module
+from common.module import Module
-from config import logger
+from config.log import logger
-
+
-
+
-class CheckAXFR(Module):
+class CheckAXFR(Module):
-    """
+    """
-    DNS域传送漏洞检查类
+    DNS域传送漏洞检查类
-    """
+    """
-    def __init__(self, domain: str):
+    def __init__(self, domain: str):
-        Module.__init__(self)
+        Module.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Check'
+        self.module = 'Check'
-        self.source = 'AXFRCheck'
+        self.source = 'AXFRCheck'
-        self.results = []
+        self.results = []
-
+
-    def axfr(self, server):
+    def axfr(self, server):
-        """
+        """
-        执行域传送
+        执行域传送
-
+
-        :param server: 域名服务器
+        :param server: 域名服务器
-        """
+        """
-        logger.log('DEBUG', f'尝试对{self.domain}的域名服务器{server}进行域传送')
+        logger.log('DEBUG', f'尝试对{self.domain}的域名服务器{server}进行域传送')
-        try:
+        try:
-            xfr = dns.query.xfr(where=server, zone=self.domain,
+            xfr = dns.query.xfr(where=server, zone=self.domain,
-                                timeout=5.0, lifetime=10.0)
+                                timeout=5.0, lifetime=10.0)
-            zone = dns.zone.from_xfr(xfr)
+            zone = dns.zone.from_xfr(xfr)
-        except Exception as e:
+        except Exception as e:
-            logger.log('DEBUG', e.args)
+            logger.log('DEBUG', e.args)
-            logger.log('DEBUG', f'对{self.domain}的域名服务器{server}进行域传送失败')
+            logger.log('DEBUG', f'对{self.domain}的域名服务器{server}进行域传送失败')
-            return
+            return
-        names = zone.nodes.keys()
+        names = zone.nodes.keys()
-        for name in names:
+        for name in names:
-            full_domain = str(name) + '.' + self.domain
+            full_domain = str(name) + '.' + self.domain
-            subdomain = utils.match_subdomain(self.domain, full_domain)
+            subdomain = utils.match_subdomain(self.domain, full_domain)
-            self.subdomains = self.subdomains.union(subdomain)
+            self.subdomains = self.subdomains.union(subdomain)
-            record = zone[name].to_text(name)
+            record = zone[name].to_text(name)
-            self.results.append(record)
+            self.results.append(record)
-        if self.results:
+        if self.results:
-            logger.log('DEBUG', f'发现{self.domain}在{server}上的域传送记录')
+            logger.log('DEBUG', f'发现{self.domain}在{server}上的域传送记录')
-            logger.log('DEBUG', '\n'.join(self.results))
+            logger.log('DEBUG', '\n'.join(self.results))
-            self.results = []
+            self.results = []
-
+
-    def check(self):
+    def check(self):
-        """
+        """
-        正则匹配响应头中的内容安全策略字段以发现子域名
+        正则匹配响应头中的内容安全策略字段以发现子域名
-        """
+        """
-        resolver = utils.dns_resolver()
+        resolver = utils.dns_resolver()
-        try:
+        try:
-            answers = resolver.query(self.domain, "NS")
+            answers = resolver.query(self.domain, "NS")
-        except Exception as e:
+        except Exception as e:
-            logger.log('ERROR', e.args)
+            logger.log('ERROR', e.args)
-            return
+            return
-        nsservers = [str(answer) for answer in answers]
+        nsservers = [str(answer) for answer in answers]
-        if not len(nsservers):
+        if not len(nsservers):
-            logger.log('ALERT', f'没有找到{self.domain}的NS域名服务器记录')
+            logger.log('ALERT', f'没有找到{self.domain}的NS域名服务器记录')
-            return
+            return
-        for nsserver in nsservers:
+        for nsserver in nsservers:
-            self.axfr(nsserver)
+            self.axfr(nsserver)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.check()
+        self.check()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    check = CheckAXFR(domain)
+    check = CheckAXFR(domain)
-    check.run()
+    check.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('ZoneTransfer.me')
+    do('ZoneTransfer.me')
-    # do('example.com')
+    # do('example.com')

modules/check/cdx.py

@@ -1,60 +1,60 @@
-"""
+"""
-检查crossdomain.xml文件收集子域名
+检查crossdomain.xml文件收集子域名
-"""
+"""
-
+
-from common.module import Module
+from common.module import Module
-from common import utils
+from common import utils
-
+
-
+
-class CheckCDX(Module):
+class CheckCDX(Module):
-    """
+    """
-    检查crossdomain.xml文件收集子域名
+    检查crossdomain.xml文件收集子域名
-    """
+    """
-    def __init__(self, domain: str):
+    def __init__(self, domain: str):
-        Module.__init__(self)
+        Module.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Check'
+        self.module = 'Check'
-        self.source = "CrossDomainXml"
+        self.source = "CrossDomainXml"
-
+
-    def check(self):
+    def check(self):
-        """
+        """
-        检查crossdomain.xml收集子域名
+        检查crossdomain.xml收集子域名
-        """
+        """
-        urls = [f'http://{self.domain}/crossdomain.xml',
+        urls = [f'http://{self.domain}/crossdomain.xml',
-                f'https://{self.domain}/crossdomain.xml',
+                f'https://{self.domain}/crossdomain.xml',
-                f'http://www.{self.domain}/crossdomain.xml',
+                f'http://www.{self.domain}/crossdomain.xml',
-                f'https://www.{self.domain}/crossdomain.xml']
+                f'https://www.{self.domain}/crossdomain.xml']
-        for url in urls:
+        for url in urls:
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            response = self.get(url, check=False)
+            response = self.get(url, check=False)
-            if not response:
+            if not response:
-                return
+                return
-            if response and len(response.content):
+            if response and len(response.content):
-                self.subdomains = utils.match_subdomain(self.domain,
+                self.subdomains = utils.match_subdomain(self.domain,
-                                                        response.text)
+                                                        response.text)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.check()
+        self.check()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param domain: 域名
+    :param domain: 域名
-    """
+    """
-    check = CheckCDX(domain)
+    check = CheckCDX(domain)
-    check.run()
+    check.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/check/cert.py

@@ -1,61 +1,61 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python3
-
+
-"""
+"""
-检查域名证书收集子域名
+检查域名证书收集子域名
-"""
+"""
-import socket
+import socket
-import ssl
+import ssl
-
+
-from common import utils
+from common import utils
-from common.module import Module
+from common.module import Module
-from config import logger
+from config.log import logger
-
+
-
+
-class CheckCert(Module):
+class CheckCert(Module):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Module.__init__(self)
+        Module.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.port = 443  # ssl port
+        self.port = 443  # ssl port
-        self.module = 'Check'
+        self.module = 'Check'
-        self.source = 'CertInfo'
+        self.source = 'CertInfo'
-
+
-    def check(self):
+    def check(self):
-        """
+        """
-        获取域名证书并匹配证书中的子域名
+        获取域名证书并匹配证书中的子域名
-        """
+        """
-        try:
+        try:
-            ctx = ssl.create_default_context()
+            ctx = ssl.create_default_context()
-            sock = ctx.wrap_socket(socket.socket(),
+            sock = ctx.wrap_socket(socket.socket(),
-                                   server_hostname=self.domain)
+                                   server_hostname=self.domain)
-            sock.connect((self.domain, self.port))
+            sock.connect((self.domain, self.port))
-            cert_dict = sock.getpeercert()
+            cert_dict = sock.getpeercert()
-        except Exception as e:
+        except Exception as e:
-            logger.log('DEBUG', e.args)
+            logger.log('DEBUG', e.args)
-            return
+            return
-        subdomains = utils.match_subdomain(self.domain, str(cert_dict))
+        subdomains = utils.match_subdomain(self.domain, str(cert_dict))
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.check()
+        self.check()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    check = CheckCert(domain)
+    check = CheckCert(domain)
-    check.run()
+    check.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/check/csp.py

@@ -1,82 +1,82 @@
-"""
+"""
-检查内容安全策略收集子域名收集子域名
+检查内容安全策略收集子域名收集子域名
-"""
+"""
-import requests
+import requests
-
+
-from common import utils
+from common import utils
-from common.module import Module
+from common.module import Module
-from config import logger
+from config.log import logger
-
+
-
+
-class CheckCSP(Module):
+class CheckCSP(Module):
-    """
+    """
-    检查内容安全策略收集子域名
+    检查内容安全策略收集子域名
-    """
+    """
-    def __init__(self, domain, header):
+    def __init__(self, domain, header):
-        Module.__init__(self)
+        Module.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Check'
+        self.module = 'Check'
-        self.source = 'ContentSecurityPolicy'
+        self.source = 'ContentSecurityPolicy'
-        self.csp_header = header
+        self.csp_header = header
-
+
-    def grab_header(self):
+    def grab_header(self):
-        """
+        """
-        抓取请求头
+        抓取请求头
-
+
-        :return: 请求头
+        :return: 请求头
-        """
+        """
-        csp_header = dict()
+        csp_header = dict()
-        urls = [f'http://{self.domain}',
+        urls = [f'http://{self.domain}',
-                f'https://{self.domain}',
+                f'https://{self.domain}',
-                f'http://www.{self.domain}',
+                f'http://www.{self.domain}',
-                f'https://www.{self.domain}']
+                f'https://www.{self.domain}']
-        for url in urls:
+        for url in urls:
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            response = self.get(url, check=False)
+            response = self.get(url, check=False)
-            if response:
+            if response:
-                csp_header = response.headers
+                csp_header = response.headers
-                break
+                break
-        return csp_header
+        return csp_header
-
+
-    def check(self):
+    def check(self):
-        """
+        """
-        正则匹配响应头中的内容安全策略字段以发现子域名
+        正则匹配响应头中的内容安全策略字段以发现子域名
-        """
+        """
-        if not self.csp_header:
+        if not self.csp_header:
-            self.csp_header = self.grab_header()
+            self.csp_header = self.grab_header()
-        csp = self.header.get('Content-Security-Policy')
+        csp = self.header.get('Content-Security-Policy')
-        if not self.csp_header:
+        if not self.csp_header:
-            logger.log('DEBUG', f'获取{self.domain}域的请求头失败')
+            logger.log('DEBUG', f'获取{self.domain}域的请求头失败')
-            return
+            return
-        if not csp:
+        if not csp:
-            logger.log('DEBUG', f'{self.domain}域的响应头不存在内容安全策略字段')
+            logger.log('DEBUG', f'{self.domain}域的响应头不存在内容安全策略字段')
-            return
+            return
-        self.subdomains = utils.match_subdomain(self.domain, csp)
+        self.subdomains = utils.match_subdomain(self.domain, csp)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.check()
+        self.check()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain, header=None):  # 统一入口名字 方便多线程调用
+def do(domain, header=None):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    :param dict or None header: 响应头
+    :param dict or None header: 响应头
-    """
+    """
-    check = CheckCSP(domain, header)
+    check = CheckCSP(domain, header)
-    check.run()
+    check.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    resp = requests.get('https://content-security-policy.com/')
+    resp = requests.get('https://content-security-policy.com/')
-    do('google-analytics.com', resp.headers)
+    do('google-analytics.com', resp.headers)

modules/crawl/archivecrawl.py

@@ -1,61 +1,61 @@
-import cdx_toolkit
+import cdx_toolkit
-from common.crawl import Crawl
+from common.crawl import Crawl
-from config import logger
+from config.log import logger
-
+
-
+
-class ArchiveCrawl(Crawl):
+class ArchiveCrawl(Crawl):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Crawl.__init__(self)
+        Crawl.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Crawl'
+        self.module = 'Crawl'
-        self.source = 'ArchiveCrawl'
+        self.source = 'ArchiveCrawl'
-
+
-    def crawl(self, domain, limit):
+    def crawl(self, domain, limit):
-        """
+        """
-
+
-        :param domain:
+        :param domain:
-        :param limit:
+        :param limit:
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        cdx = cdx_toolkit.CDXFetcher(source='ia')
+        cdx = cdx_toolkit.CDXFetcher(source='ia')
-        url = f'*.{domain}/*'
+        url = f'*.{domain}/*'
-        size = cdx.get_size_estimate(url)
+        size = cdx.get_size_estimate(url)
-        logger.log('DEBUG', f'{url} ArchiveCrawl size estimate {size}')
+        logger.log('DEBUG', f'{url} ArchiveCrawl size estimate {size}')
-
+
-        for resp in cdx.iter(url, limit=limit):
+        for resp in cdx.iter(url, limit=limit):
-            if resp.data.get('status') not in ['301', '302']:
+            if resp.data.get('status') not in ['301', '302']:
-                url = resp.data.get('url')
+                url = resp.data.get('url')
-                subdomains = self.match(self.register(domain),
+                subdomains = self.match(self.register(domain),
-                                             url + resp.text)
+                                             url + resp.text)
-                # 合并搜索子域名搜索结果
+                # 合并搜索子域名搜索结果
-                self.subdomains = self.subdomains.union(subdomains)
+                self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.crawl(self.domain, 50)
+        self.crawl(self.domain, 50)
-        # 爬取已发现的子域以发现新的子域
+        # 爬取已发现的子域以发现新的子域
-        for subdomain in self.subdomains:
+        for subdomain in self.subdomains:
-            if subdomain != self.domain:
+            if subdomain != self.domain:
-                self.crawl(subdomain, 10)
+                self.crawl(subdomain, 10)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    crawl = ArchiveCrawl(domain)
+    crawl = ArchiveCrawl(domain)
-    crawl.run()
+    crawl.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/crawl/commoncrawl.py

@@ -1,61 +1,60 @@
-import cdx_toolkit
+import cdx_toolkit
-from tqdm import tqdm
+from tqdm import tqdm
-
+
-from common.crawl import Crawl
+from common.crawl import Crawl
-from config import logger
+
-
+
-
+class CommonCrawl(Crawl):
-class CommonCrawl(Crawl):
+    def __init__(self, domain):
-    def __init__(self, domain):
+        Crawl.__init__(self)
-        Crawl.__init__(self)
+        self.domain = domain
-        self.domain = domain
+        self.module = 'Crawl'
-        self.module = 'Crawl'
+        self.source = 'CommonCrawl'
-        self.source = 'CommonCrawl'
+
-
+    def crawl(self, domain, limit):
-    def crawl(self, domain, limit):
+        """
-        """
+
-
+        :param domain:
-        :param domain:
+        :param limit:
-        :param limit:
+        """
-        """
+        self.header = self.get_header()
-        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
-        self.proxy = self.get_proxy(self.source)
+        cdx = cdx_toolkit.CDXFetcher()
-        cdx = cdx_toolkit.CDXFetcher()
+        url = f'*.{domain}/*'
-        url = f'*.{domain}/*'
+        size = cdx.get_size_estimate(url)
-        size = cdx.get_size_estimate(url)
+        print(url, 'CommonCrawl size estimate', size)
-        print(url, 'CommonCrawl size estimate', size)
+
-
+        for resp in tqdm(cdx.iter(url, limit=limit), total=limit):
-        for resp in tqdm(cdx.iter(url, limit=limit), total=limit):
+            if resp.data.get('status') not in ['301', '302']:
-            if resp.data.get('status') not in ['301', '302']:
+                subdomains = self.match(self.register(domain), resp.text)
-                subdomains = self.match(self.register(domain), resp.text)
+                # 合并搜索子域名搜索结果
-                # 合并搜索子域名搜索结果
+                self.subdomains = self.subdomains.union(subdomains)
-                self.subdomains = self.subdomains.union(subdomains)
+
-
+    def run(self):
-    def run(self):
+        """
-        """
+        类执行入口
-        类执行入口
+        """
-        """
+        self.begin()
-        self.begin()
+        self.crawl(self.domain, 50)
-        self.crawl(self.domain, 50)
+        # 爬取已发现的子域以发现新的子域
-        # 爬取已发现的子域以发现新的子域
+        for subdomain in self.subdomains:
-        for subdomain in self.subdomains:
+            if subdomain != self.domain:
-            if subdomain != self.domain:
+                self.crawl(subdomain, 10)
-                self.crawl(subdomain, 10)
+        self.finish()
-        self.finish()
+        self.save_json()
-        self.save_json()
+        self.gen_result()
-        self.gen_result()
+        self.save_db()
-        self.save_db()
+
-
+
-
+def do(domain):  # 统一入口名字 方便多线程调用
-def do(domain):  # 统一入口名字 方便多线程调用
+    """
-    """
+    类统一调用入口
-    类统一调用入口
+
-
+    :param str domain: 域名
-    :param str domain: 域名
+    """
-    """
+    crawl = CommonCrawl(domain)
-    crawl = CommonCrawl(domain)
+    crawl.run()
-    crawl.run()
+
-
+
-
+if __name__ == '__main__':
-if __name__ == '__main__':
+    do('example.com')
-    do('example.com')

modules/datasets/binaryedge_api.py

@@ -1,53 +1,53 @@
-import api
+from config import api
-from common.query import Query
+from common.query import Query
-
+
-
+
-class BinaryEdgeAPI(Query):
+class BinaryEdgeAPI(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'BinaryEdgeAPIQuery'
+        self.source = 'BinaryEdgeAPIQuery'
-        self.addr = 'https://api.binaryedge.io/v2/query/domains/subdomain/'
+        self.addr = 'https://api.binaryedge.io/v2/query/domains/subdomain/'
-        self.api = api.binaryedge_api
+        self.api = api.binaryedge_api
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.header.update({'X-Key': self.api})
+        self.header.update({'X-Key': self.api})
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        url = self.addr + self.domain
+        url = self.addr + self.domain
-        resp = self.get(url)
+        resp = self.get(url)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = self.match(self.domain, str(resp.json()))
+        subdomains = self.match(self.domain, str(resp.json()))
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.api):
+        if not self.check(self.api):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = BinaryEdgeAPI(domain)
+    query = BinaryEdgeAPI(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/bufferover.py

@@ -1,57 +1,57 @@
-import cloudscraper
+import cloudscraper
-from common.query import Query
+from common.query import Query
-from config import logger
+from config.log import logger
-
+
-
+
-class BufferOver(Query):
+class BufferOver(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'BufferOverQuery'
+        self.source = 'BufferOverQuery'
-        self.addr = 'https://dns.bufferover.run/dns?q='
+        self.addr = 'https://dns.bufferover.run/dns?q='
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        # 绕过cloudFlare验证
+        # 绕过cloudFlare验证
-        scraper = cloudscraper.create_scraper()
+        scraper = cloudscraper.create_scraper()
-        scraper.interpreter = 'js2py'
+        scraper.interpreter = 'js2py'
-        scraper.proxies = self.get_proxy(self.source)
+        scraper.proxies = self.get_proxy(self.source)
-        url = self.addr + self.domain
+        url = self.addr + self.domain
-        try:
+        try:
-            resp = scraper.get(url, timeout=self.timeout)
+            resp = scraper.get(url, timeout=self.timeout)
-        except Exception as e:
+        except Exception as e:
-            logger.log('ERROR', e.args)
+            logger.log('ERROR', e.args)
-            return
+            return
-        if resp.status_code != 200:
+        if resp.status_code != 200:
-            return
+            return
-        subdomains = self.match(self.domain, str(resp.json()))
+        subdomains = self.match(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = BufferOver(domain)
+    query = BufferOver(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/chinaz.py

@@ -1,49 +1,49 @@
-from common.query import Query
+from common.query import Query
-
+
-
+
-class Chinaz(Query):
+class Chinaz(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'ChinazQuery'
+        self.source = 'ChinazQuery'
-        self.addr = 'https://alexa.chinaz.com/'
+        self.addr = 'https://alexa.chinaz.com/'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        self.addr = self.addr + self.domain
+        self.addr = self.addr + self.domain
-        resp = self.get(self.addr)
+        resp = self.get(self.addr)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = self.match(self.domain, resp.text)
+        subdomains = self.match(self.domain, resp.text)
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = Chinaz(domain)
+    query = Chinaz(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/chinaz_api.py

@@ -1,53 +1,53 @@
-import api
+from config import api
-from common.query import Query
+from common.query import Query
-
+
-
+
-class ChinazAPI(Query):
+class ChinazAPI(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'ChinazAPIQuery'
+        self.source = 'ChinazAPIQuery'
-        self.addr = 'https://apidata.chinaz.com/CallAPI/Alexa'
+        self.addr = 'https://apidata.chinaz.com/CallAPI/Alexa'
-        self.api = api.chinaz_api
+        self.api = api.chinaz_api
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'key': self.api, 'domainName': self.domain}
+        params = {'key': self.api, 'domainName': self.domain}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = self.match(self.domain, str(resp.json()))
+        subdomains = self.match(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.api):
+        if not self.check(self.api):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = ChinazAPI(domain)
+    query = ChinazAPI(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/circl_api.py

@@ -1,53 +1,53 @@
-import api
+from config import api
-from common.query import Query
+from common.query import Query
-
+
-
+
-class CirclAPI(Query):
+class CirclAPI(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'CirclAPIQuery'
+        self.source = 'CirclAPIQuery'
-        self.addr = 'https://www.circl.lu/pdns/query/'
+        self.addr = 'https://www.circl.lu/pdns/query/'
-        self.user = api.circl_api_username
+        self.user = api.circl_api_username
-        self.pwd = api.circl_api_password
+        self.pwd = api.circl_api_password
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        resp = self.get(self.addr + self.domain, auth=(self.user, self.pwd))
+        resp = self.get(self.addr + self.domain, auth=(self.user, self.pwd))
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = self.match(self.domain, str(resp.json()))
+        subdomains = self.match(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.user, self.pwd):
+        if not self.check(self.user, self.pwd):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = CirclAPI(domain)
+    query = CirclAPI(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/dnsdb_api.py

@@ -1,56 +1,56 @@
-import api
+from config import api
-from common import utils
+from common import utils
-from common.query import Query
+from common.query import Query
-
+
-
+
-class DNSdbAPI(Query):
+class DNSdbAPI(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'DNSdbAPIQuery'
+        self.source = 'DNSdbAPIQuery'
-        self.addr = 'https://api.dnsdb.info/lookup/rrset/name/'
+        self.addr = 'https://api.dnsdb.info/lookup/rrset/name/'
-        self.api = api.dnsdb_api_key
+        self.api = api.dnsdb_api_key
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.header.update({'X-API-Key': self.api})
+        self.header.update({'X-API-Key': self.api})
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        url = f'{self.addr}*.{self.domain}'
+        url = f'{self.addr}*.{self.domain}'
-        resp = self.get(url)
+        resp = self.get(url)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = utils.match_subdomain(self.domain, resp.text)
+        subdomains = utils.match_subdomain(self.domain, resp.text)
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.api):
+        if not self.check(self.api):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-
+
-    """
+    """
-    query = DNSdbAPI(domain)
+    query = DNSdbAPI(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/dnsdumpster.py

@@ -1,58 +1,58 @@
-from common import utils
+from common import utils
-from common.query import Query
+from common.query import Query
-
+
-
+
-class DNSdumpster(Query):
+class DNSdumpster(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = "DNSdumpsterQuery"
+        self.source = "DNSdumpsterQuery"
-        self.addr = 'https://dnsdumpster.com/'
+        self.addr = 'https://dnsdumpster.com/'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.header.update({'Referer': 'https://dnsdumpster.com'})
+        self.header.update({'Referer': 'https://dnsdumpster.com'})
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        resp = self.get(self.addr)
+        resp = self.get(self.addr)
-        if not resp:
+        if not resp:
-            return
+            return
-        self.cookie = resp.cookies
+        self.cookie = resp.cookies
-        data = {'csrfmiddlewaretoken': self.cookie.get('csrftoken'),
+        data = {'csrfmiddlewaretoken': self.cookie.get('csrftoken'),
-                'targetip': self.domain}
+                'targetip': self.domain}
-        resp = self.post(self.addr, data)
+        resp = self.post(self.addr, data)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = utils.match_subdomain(self.domain, resp.text)
+        subdomains = utils.match_subdomain(self.domain, resp.text)
-        if subdomains:
+        if subdomains:
-            # 合并搜索子域名搜索结果
+            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = DNSdumpster(domain)
+    query = DNSdumpster(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-
+
-    do('example.com')
+    do('example.com')

modules/datasets/hackertarget.py

@@ -1,52 +1,52 @@
-from common import utils
+from common import utils
-from common.query import Query
+from common.query import Query
-
+
-
+
-class HackerTarget(Query):
+class HackerTarget(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = "HackerTargetQuery"
+        self.source = "HackerTargetQuery"
-        self.addr = 'https://api.hackertarget.com/hostsearch/'
+        self.addr = 'https://api.hackertarget.com/hostsearch/'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'q': self.domain}
+        params = {'q': self.domain}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        if resp.status_code == 200:
+        if resp.status_code == 200:
-            subdomains = utils.match_subdomain(self.domain, resp.text)
+            subdomains = utils.match_subdomain(self.domain, resp.text)
-            if subdomains:
+            if subdomains:
-                # 合并搜索子域名搜索结果
+                # 合并搜索子域名搜索结果
-                self.subdomains = self.subdomains.union(subdomains)
+                self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = HackerTarget(domain)
+    query = HackerTarget(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/ipv4info_api.py

@@ -1,6 +1,6 @@
-import api
+from config import api
 from common.query import Query
-from config import logger
+from config.log import logger
 
 
 class IPv4InfoAPI(Query):

modules/datasets/netcraft.py

@@ -1,83 +1,83 @@
-import hashlib
+import hashlib
-import re
+import re
-import time
+import time
-from urllib import parse
+from urllib import parse
-
+
-from common.query import Query
+from common.query import Query
-
+
-
+
-class NetCraft(Query):
+class NetCraft(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'NetCraftQuery'
+        self.source = 'NetCraftQuery'
-        self.init = 'https://searchdns.netcraft.com/'
+        self.init = 'https://searchdns.netcraft.com/'
-        self.addr = 'https://searchdns.netcraft.com/?restriction=site+contains'
+        self.addr = 'https://searchdns.netcraft.com/?restriction=site+contains'
-        self.page_num = 1
+        self.page_num = 1
-        self.per_page_num = 20
+        self.per_page_num = 20
-
+
-    def bypass_verification(self):
+    def bypass_verification(self):
-        """
+        """
-        绕过NetCraft的JS验证
+        绕过NetCraft的JS验证
-        """
+        """
-        self.header = self.get_header()  # Netcraft会检查User-Agent
+        self.header = self.get_header()  # Netcraft会检查User-Agent
-        resp = self.get(self.init)
+        resp = self.get(self.init)
-        if not resp:
+        if not resp:
-            return False
+            return False
-        self.cookie = resp.cookies
+        self.cookie = resp.cookies
-        cookie_value = self.cookie['netcraft_js_verification_challenge']
+        cookie_value = self.cookie['netcraft_js_verification_challenge']
-        cookie_encode = parse.unquote(cookie_value).encode('utf-8')
+        cookie_encode = parse.unquote(cookie_value).encode('utf-8')
-        verify_taken = hashlib.sha1(cookie_encode).hexdigest()
+        verify_taken = hashlib.sha1(cookie_encode).hexdigest()
-        self.cookie['netcraft_js_verification_response'] = verify_taken
+        self.cookie['netcraft_js_verification_response'] = verify_taken
-        return True
+        return True
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        if not self.bypass_verification():
+        if not self.bypass_verification():
-            return
+            return
-        last = ''
+        last = ''
-        while True:
+        while True:
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            params = {'host': '*.' + self.domain, 'from': self.page_num}
+            params = {'host': '*.' + self.domain, 'from': self.page_num}
-            resp = self.get(self.addr + last, params)
+            resp = self.get(self.addr + last, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(self.domain, resp.text)
+            subdomains = self.match(self.domain, resp.text)
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
+                break
-            # 合并搜索子域名搜索结果
+            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            if 'Next page' not in resp.text:  # 搜索页面没有出现下一页时停止搜索
+            if 'Next page' not in resp.text:  # 搜索页面没有出现下一页时停止搜索
-                break
+                break
-            last = re.search(r'&last=.*' + self.domain, resp.text).group(0)
+            last = re.search(r'&last=.*' + self.domain, resp.text).group(0)
-            self.page_num += self.per_page_num
+            self.page_num += self.per_page_num
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = NetCraft(domain)
+    query = NetCraft(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/passivedns_api.py

@@ -1,54 +1,54 @@
-import api
+from config import api
-from common.query import Query
+from common.query import Query
-
+
-
+
-class PassiveDnsAPI(Query):
+class PassiveDnsAPI(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'PassiveDnsQuery'
+        self.source = 'PassiveDnsQuery'
-        self.addr = api.passivedns_api_addr or 'http://api.passivedns.cn'
+        self.addr = api.passivedns_api_addr or 'http://api.passivedns.cn'
-        self.token = api.passivedns_api_token
+        self.token = api.passivedns_api_token
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.header.update({'X-AuthToken': self.token})
+        self.header.update({'X-AuthToken': self.token})
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        url = self.addr + '/flint/rrset/*.' + self.domain
+        url = self.addr + '/flint/rrset/*.' + self.domain
-        resp = self.get(url)
+        resp = self.get(url)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = self.match(self.domain, str(resp.json()))
+        subdomains = self.match(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.addr):
+        if not self.check(self.addr):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = PassiveDnsAPI(domain)
+    query = PassiveDnsAPI(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/ptrarchive.py

@@ -1,56 +1,56 @@
-import random
+import random
-
+
-from common import utils
+from common import utils
-from common.query import Query
+from common.query import Query
-
+
-
+
-class PTRArchive(Query):
+class PTRArchive(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = "PTRArchiveQuery"
+        self.source = "PTRArchiveQuery"
-        self.addr = 'http://ptrarchive.com/tools/search4.htm'
+        self.addr = 'http://ptrarchive.com/tools/search4.htm'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        # 绕过主页前端JS验证
+        # 绕过主页前端JS验证
-        self.cookie = {'pa_id': str(random.randint(0, 1000000000))}
+        self.cookie = {'pa_id': str(random.randint(0, 1000000000))}
-        params = {'label': self.domain, 'date': 'ALL'}
+        params = {'label': self.domain, 'date': 'ALL'}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        if resp.status_code == 200:
+        if resp.status_code == 200:
-            subdomains = utils.match_subdomain(self.domain, resp.text)
+            subdomains = utils.match_subdomain(self.domain, resp.text)
-            if subdomains:
+            if subdomains:
-                # 合并搜索子域名搜索结果
+                # 合并搜索子域名搜索结果
-                self.subdomains = self.subdomains.union(subdomains)
+                self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = PTRArchive(domain)
+    query = PTRArchive(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/riddler.py

@@ -1,49 +1,49 @@
-from common.query import Query
+from common.query import Query
-
+
-
+
-class Riddler(Query):
+class Riddler(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'RiddlerQuery'
+        self.source = 'RiddlerQuery'
-        self.addr = 'https://riddler.io/search'
+        self.addr = 'https://riddler.io/search'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'q': 'pld:' + self.domain}
+        params = {'q': 'pld:' + self.domain}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = self.match(self.domain, resp.text)
+        subdomains = self.match(self.domain, resp.text)
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = Riddler(domain)
+    query = Riddler(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/robtex.py

@@ -1,63 +1,63 @@
-import json
+import json
-import time
+import time
-
+
-from common.query import Query
+from common.query import Query
-
+
-
+
-class Robtex(Query):
+class Robtex(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = "RobtexQuery"
+        self.source = "RobtexQuery"
-        self.addr = 'https://freeapi.robtex.com/pdns/'
+        self.addr = 'https://freeapi.robtex.com/pdns/'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        url = self.addr + 'forward/' + self.domain
+        url = self.addr + 'forward/' + self.domain
-        resp = self.get(url)
+        resp = self.get(url)
-        if not resp:
+        if not resp:
-            return
+            return
-        text_list = resp.text.splitlines()
+        text_list = resp.text.splitlines()
-        text_json = list(map(lambda x: json.loads(x), text_list))
+        text_json = list(map(lambda x: json.loads(x), text_list))
-        for record in text_json:
+        for record in text_json:
-            if record.get('rrtype') in ['A', 'AAAA']:
+            if record.get('rrtype') in ['A', 'AAAA']:
-                time.sleep(self.delay)  # Robtex有查询频率限制
+                time.sleep(self.delay)  # Robtex有查询频率限制
-                ip = record.get('rrdata')
+                ip = record.get('rrdata')
-                url = self.addr + 'reverse/' + ip
+                url = self.addr + 'reverse/' + ip
-                resp = self.get(url)
+                resp = self.get(url)
-                if not resp:
+                if not resp:
-                    return
+                    return
-                subdomains = self.match(self.domain, resp.text)
+                subdomains = self.match(self.domain, resp.text)
-                if subdomains:
+                if subdomains:
-                    # 合并搜索子域名搜索结果
+                    # 合并搜索子域名搜索结果
-                    self.subdomains = self.subdomains.union(subdomains)
+                    self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = Robtex(domain)
+    query = Robtex(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/securitytrails_api.py

@@ -1,57 +1,57 @@
-import api
+from config import api
-from common.query import Query
+from common.query import Query
-
+
-
+
-class SecurityTrailsAPI(Query):
+class SecurityTrailsAPI(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'SecurityTrailsAPIQuery'
+        self.source = 'SecurityTrailsAPIQuery'
-        self.addr = 'https://api.securitytrails.com/v1/domain/'
+        self.addr = 'https://api.securitytrails.com/v1/domain/'
-        self.api = api.securitytrails_api
+        self.api = api.securitytrails_api
-        self.delay = 2  # SecurityTrails查询时延至少2秒
+        self.delay = 2  # SecurityTrails查询时延至少2秒
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'apikey': self.api}
+        params = {'apikey': self.api}
-        url = f'{self.addr}{self.domain}/subdomains'
+        url = f'{self.addr}{self.domain}/subdomains'
-        resp = self.get(url, params)
+        resp = self.get(url, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        prefixs = resp.json()['subdomains']
+        prefixs = resp.json()['subdomains']
-        subdomains = [f'{prefix}.{self.domain}' for prefix in prefixs]
+        subdomains = [f'{prefix}.{self.domain}' for prefix in prefixs]
-        if subdomains:
+        if subdomains:
-            # 合并搜索子域名搜索结果
+            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.api):
+        if not self.check(self.api):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = SecurityTrailsAPI(domain)
+    query = SecurityTrailsAPI(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/sitedossier.py

@@ -1,60 +1,60 @@
-
+
-from common.query import Query
+from common.query import Query
-
+
-
+
-class SiteDossier(Query):
+class SiteDossier(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Dataset'
+        self.module = 'Dataset'
-        self.source = 'SiteDossierQuery'
+        self.source = 'SiteDossierQuery'
-        self.addr = 'http://www.sitedossier.com/parentdomain/'
+        self.addr = 'http://www.sitedossier.com/parentdomain/'
-        self.delay = 2
+        self.delay = 2
-        self.page_num = 1
+        self.page_num = 1
-        self.per_page_num = 100
+        self.per_page_num = 100
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        while True:
+        while True:
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            url = f'{self.addr}{self.domain}/{self.page_num}'
+            url = f'{self.addr}{self.domain}/{self.page_num}'
-            resp = self.get(url)
+            resp = self.get(url)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(self.domain, resp.text)
+            subdomains = self.match(self.domain, resp.text)
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
+                break
-            # 合并搜索子域名搜索结果
+            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            # 搜索页面没有出现下一页时停止搜索
+            # 搜索页面没有出现下一页时停止搜索
-            if 'Show next 100 items' not in resp.text:
+            if 'Show next 100 items' not in resp.text:
-                break
+                break
-            self.page_num += self.per_page_num
+            self.page_num += self.per_page_num
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = SiteDossier(domain)
+    query = SiteDossier(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/datasets/threatcrowd.py

@@ -1,7 +1,7 @@
 import cloudscraper
 
 from common.query import Query
-from config import logger
+from config.log import logger
 
 
 class ThreatCrowd(Query):

modules/datasets/wzpc.py

@@ -1,5 +1,5 @@
 import time
-from config import logger
+from config.log import logger
 from common.query import Query
 
 

modules/dnsquery/srv.py

@@ -1,95 +1,95 @@
-"""
+"""
-通过枚举域名常见的SRV记录并做查询来发现子域
+通过枚举域名常见的SRV记录并做查询来发现子域
-"""
+"""
-
+
-import json
+import json
-import queue
+import queue
-import threading
+import threading
-
+
-from common import utils
+from common import utils
-from common.module import Module
+from common.module import Module
-from config import data_storage_dir, logger
+from config.setting import data_storage_dir
-
+
-
+
-class BruteSRV(Module):
+class BruteSRV(Module):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Module.__init__(self)
+        Module.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'dnsquery'
+        self.module = 'dnsquery'
-        self.source = "BruteSRV"
+        self.source = "BruteSRV"
-        self.type = 'SRV'  # 利用的DNS记录的SRV记录查询子域
+        self.type = 'SRV'  # 利用的DNS记录的SRV记录查询子域
-        self.thread_num = 10
+        self.thread_num = 10
-        self.names_que = queue.Queue()
+        self.names_que = queue.Queue()
-        self.answers_que = queue.Queue()
+        self.answers_que = queue.Queue()
-
+
-    def gen_names(self):
+    def gen_names(self):
-        path = data_storage_dir.joinpath('srv_prefixes.json')
+        path = data_storage_dir.joinpath('srv_prefixes.json')
-        with open(path, encoding='utf-8', errors='ignore') as file:
+        with open(path, encoding='utf-8', errors='ignore') as file:
-            prefixes = json.load(file)
+            prefixes = json.load(file)
-        names = map(lambda prefix: prefix + self.domain, prefixes)
+        names = map(lambda prefix: prefix + self.domain, prefixes)
-
+
-        for name in names:
+        for name in names:
-            self.names_que.put(name)
+            self.names_que.put(name)
-
+
-    def brute(self):
+    def brute(self):
-        """
+        """
-        枚举域名的SRV记录
+        枚举域名的SRV记录
-        """
+        """
-        self.gen_names()
+        self.gen_names()
-
+
-        for i in range(self.thread_num):
+        for i in range(self.thread_num):
-            thread = BruteThread(self.names_que, self.answers_que)
+            thread = BruteThread(self.names_que, self.answers_que)
-            thread.daemon = True
+            thread.daemon = True
-            thread.start()
+            thread.start()
-
+
-        self.names_que.join()
+        self.names_que.join()
-
+
-        while not self.answers_que.empty():
+        while not self.answers_que.empty():
-            answer = self.answers_que.get()
+            answer = self.answers_que.get()
-            if answer is None:
+            if answer is None:
-                continue
+                continue
-            for item in answer:
+            for item in answer:
-                record = str(item)
+                record = str(item)
-                subdomains = utils.match_subdomain(self.domain, record)
+                subdomains = utils.match_subdomain(self.domain, record)
-                self.subdomains = self.subdomains.union(subdomains)
+                self.subdomains = self.subdomains.union(subdomains)
-                self.gen_record(subdomains, record)
+                self.gen_record(subdomains, record)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.brute()
+        self.brute()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-class BruteThread(threading.Thread):
+class BruteThread(threading.Thread):
-    def __init__(self, names_que, answers_que):
+    def __init__(self, names_que, answers_que):
-        threading.Thread.__init__(self)
+        threading.Thread.__init__(self)
-        self.names_que = names_que
+        self.names_que = names_que
-        self.answers_que = answers_que
+        self.answers_que = answers_que
-
+
-    def run(self):
+    def run(self):
-        while True:
+        while True:
-            name = self.names_que.get()
+            name = self.names_que.get()
-            answer = utils.dns_query(name, 'SRV')
+            answer = utils.dns_query(name, 'SRV')
-            self.answers_que.put(answer)
+            self.answers_que.put(answer)
-            self.names_que.task_done()
+            self.names_que.task_done()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    brute = BruteSRV(domain)
+    brute = BruteSRV(domain)
-    brute.run()
+    brute.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('zonetransfer.me')
+    do('zonetransfer.me')
-    # do('example.com')
+    # do('example.com')

modules/intelligence/riskiq_api.py

@@ -1,4 +1,4 @@
-import api
+from config import api
 from common.query import Query
 
 

modules/intelligence/threatbook_api.py

@@ -1,4 +1,4 @@
-import api
+from config import api
 from common.query import Query
 
 

modules/intelligence/threatminer.py

@@ -1,50 +1,50 @@
-from common.query import Query
+from common.query import Query
-
+
-
+
-class ThreatMiner(Query):
+class ThreatMiner(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Intelligence'
+        self.module = 'Intelligence'
-        self.source = 'ThreatMinerQuery'
+        self.source = 'ThreatMinerQuery'
-        self.addr = 'https://www.threatminer.org/getData.php'
+        self.addr = 'https://www.threatminer.org/getData.php'
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'e': 'subdomains_container',
+        params = {'e': 'subdomains_container',
-                  'q': self.domain, 't': 0, 'rt': 10}
+                  'q': self.domain, 't': 0, 'rt': 10}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        subdomains = self.match(self.domain, resp.text)
+        subdomains = self.match(self.domain, resp.text)
-        # 合并搜索子域名搜索结果
+        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = ThreatMiner(domain)
+    query = ThreatMiner(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/intelligence/virustotal.py

@@ -1,71 +1,71 @@
-from common.query import Query
+from common.query import Query
-
+
-'''
+'''
-最多查询100条
+最多查询100条
-'''
+'''
-
+
-
+
-class VirusTotal(Query):
+class VirusTotal(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.source = 'VirusTotalQuery'
+        self.source = 'VirusTotalQuery'
-        self.module = 'Intelligence'
+        self.module = 'Intelligence'
-        self.addr = 'https://www.virustotal.com/ui/domains/{}/subdomains'
+        self.addr = 'https://www.virustotal.com/ui/domains/{}/subdomains'
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        next_cursor = ''
+        next_cursor = ''
-        while True:
+        while True:
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.header.update({'Referer': 'https://www.virustotal.com/',
+            self.header.update({'Referer': 'https://www.virustotal.com/',
-                                'TE': 'Trailers'})
+                                'TE': 'Trailers'})
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            params = {'limit': '40', 'cursor': next_cursor}
+            params = {'limit': '40', 'cursor': next_cursor}
-            resp = self.get(url=self.addr.format(self.domain), params=params)
+            resp = self.get(url=self.addr.format(self.domain), params=params)
-            if not resp:
+            if not resp:
-                return
+                return
-            data = resp.json()
+            data = resp.json()
-            subdomains = set()
+            subdomains = set()
-            datas = data.get('data')
+            datas = data.get('data')
-
+
-            if datas:
+            if datas:
-                for data in datas:
+                for data in datas:
-                    subdomain = data.get('id')
+                    subdomain = data.get('id')
-                    if subdomain:
+                    if subdomain:
-                        subdomains.add(subdomain)
+                        subdomains.add(subdomain)
-            else:
+            else:
-                break
+                break
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            meta = data.get('meta')
+            meta = data.get('meta')
-            if meta:
+            if meta:
-                next_cursor = meta.get('cursor')
+                next_cursor = meta.get('cursor')
-            else:
+            else:
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = VirusTotal(domain)
+    query = VirusTotal(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/intelligence/virustotal_api.py

@@ -1,55 +1,55 @@
-import api
+from config import api
-from common.query import Query
+from common.query import Query
-
+
-
+
-class VirusTotalAPI(Query):
+class VirusTotalAPI(Query):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Query.__init__(self)
+        Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Intelligence'
+        self.module = 'Intelligence'
-        self.source = 'VirusTotalAPIQuery'
+        self.source = 'VirusTotalAPIQuery'
-        self.addr = 'https://www.virustotal.com/vtapi/v2/domain/report'
+        self.addr = 'https://www.virustotal.com/vtapi/v2/domain/report'
-        self.key = api.virustotal_api_key
+        self.key = api.virustotal_api_key
-
+
-    def query(self):
+    def query(self):
-        """
+        """
-        向接口查询子域并做子域匹配
+        向接口查询子域并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        params = {'apikey': self.key, 'domain': self.domain}
+        params = {'apikey': self.key, 'domain': self.domain}
-        resp = self.get(self.addr, params)
+        resp = self.get(self.addr, params)
-        if not resp:
+        if not resp:
-            return
+            return
-        json = resp.json()
+        json = resp.json()
-        data = json.get('subdomains')
+        data = json.get('subdomains')
-        if data:
+        if data:
-            subdomains = set(data)
+            subdomains = set(data)
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.key):
+        if not self.check(self.key):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.query()
+        self.query()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    query = VirusTotalAPI(domain)
+    query = VirusTotalAPI(domain)
-    query.run()
+    query.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/ask.py

@@ -1,81 +1,81 @@
-import time
+import time
-from common.search import Search
+from common.search import Search
-
+
-
+
-class Ask(Search):
+class Ask(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'AskSearch'
+        self.source = 'AskSearch'
-        self.addr = 'https://www.search.ask.com/web'
+        self.addr = 'https://www.search.ask.com/web'
-        self.limit_num = 200  # 限制搜索条数
+        self.limit_num = 200  # 限制搜索条数
-        self.per_page_num = 10  # 默认每页显示10页
+        self.per_page_num = 10  # 默认每页显示10页
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        self.page_num = 1
+        self.page_num = 1
-        while True:
+        while True:
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            query = 'site:.' + domain + filtered_subdomain
+            query = 'site:.' + domain + filtered_subdomain
-            params = {'q': query, 'page': self.page_num}
+            params = {'q': query, 'page': self.page_num}
-            resp = self.get(self.addr, params)
+            resp = self.get(self.addr, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(domain, resp.text)
+            subdomains = self.match(domain, resp.text)
-            if not subdomains:
+            if not subdomains:
-                break
+                break
-            if not full_search:
+            if not full_search:
-                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
+                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            self.page_num += 1
+            self.page_num += 1
-            if '>Next<' not in resp.text:
+            if '>Next<' not in resp.text:
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = Ask(domain)
+    search = Ask(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/baidu.py

@@ -1,109 +1,109 @@
-import time
+import time
-from bs4 import BeautifulSoup
+from bs4 import BeautifulSoup
-from common.search import Search
+from common.search import Search
-
+
-
+
-class Baidu(Search):
+class Baidu(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'BaiduSearch'
+        self.source = 'BaiduSearch'
-        self.init = 'https://www.baidu.com/'
+        self.init = 'https://www.baidu.com/'
-        self.addr = 'https://www.baidu.com/s'
+        self.addr = 'https://www.baidu.com/s'
-        self.domain = domain
+        self.domain = domain
-        self.limit_num = 750  # 限制搜索条数
+        self.limit_num = 750  # 限制搜索条数
-
+
-    def redirect_match(self, domain, html):
+    def redirect_match(self, domain, html):
-        """
+        """
-        获取跳转地址并传递地址进行跳转head请求
+        获取跳转地址并传递地址进行跳转head请求
-
+
-        :param domain: 域名
+        :param domain: 域名
-        :param html: 响应体
+        :param html: 响应体
-        :return: 子域
+        :return: 子域
-        """
+        """
-        bs = BeautifulSoup(html, 'html.parser')
+        bs = BeautifulSoup(html, 'html.parser')
-        subdomains_all = set()
+        subdomains_all = set()
-        # 获取搜索结果中所有的跳转URL地址
+        # 获取搜索结果中所有的跳转URL地址
-        for find_res in bs.find_all('a', {'class': 'c-showurl'}):
+        for find_res in bs.find_all('a', {'class': 'c-showurl'}):
-            url = find_res.get('href')
+            url = find_res.get('href')
-            subdomains = self.match_location(domain, url)
+            subdomains = self.match_location(domain, url)
-            subdomains_all = subdomains_all.union(subdomains)
+            subdomains_all = subdomains_all.union(subdomains)
-        return subdomains_all
+        return subdomains_all
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        self.page_num = 0  # 二次搜索重新置0
+        self.page_num = 0  # 二次搜索重新置0
-        while True:
+        while True:
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            query = 'site:.' + domain + filtered_subdomain
+            query = 'site:.' + domain + filtered_subdomain
-            params = {'wd': query,
+            params = {'wd': query,
-                      'pn': self.page_num,
+                      'pn': self.page_num,
-                      'rn': self.per_page_num}
+                      'rn': self.per_page_num}
-            resp = self.get(self.addr, params)
+            resp = self.get(self.addr, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            if len(domain) > 12:  # 解决百度搜索结果中域名过长会显示不全的问题
+            if len(domain) > 12:  # 解决百度搜索结果中域名过长会显示不全的问题
-                # 获取百度跳转URL响应头的Location字段获取直链
+                # 获取百度跳转URL响应头的Location字段获取直链
-                subdomains = self.redirect_match(domain, resp.text)
+                subdomains = self.redirect_match(domain, resp.text)
-            else:
+            else:
-                subdomains = self.match(domain, resp.text)
+                subdomains = self.match(domain, resp.text)
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
+                break
-            if not full_search:
+            if not full_search:
-                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
+                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            # 合并搜索子域名搜索结果
+            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            self.page_num += self.per_page_num
+            self.page_num += self.per_page_num
-            # 搜索页面没有出现下一页时停止搜索
+            # 搜索页面没有出现下一页时停止搜索
-            if '&pn={next_pn}&'.format(next_pn=self.page_num) not in resp.text:
+            if '&pn={next_pn}&'.format(next_pn=self.page_num) not in resp.text:
-                break
+                break
-            if self.page_num >= self.limit_num:  # 搜索条数限制
+            if self.page_num >= self.limit_num:  # 搜索条数限制
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = Baidu(domain)
+    search = Baidu(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('huayunshuzi.com')
+    do('huayunshuzi.com')

modules/search/bing.py

@@ -1,92 +1,92 @@
-import time
+import time
-from common.search import Search
+from common.search import Search
-
+
-
+
-class Bing(Search):
+class Bing(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'BingSearch'
+        self.source = 'BingSearch'
-        self.init = 'https://www.bing.com/'
+        self.init = 'https://www.bing.com/'
-        self.addr = 'https://www.bing.com/search'
+        self.addr = 'https://www.bing.com/search'
-        self.limit_num = 1000  # 限制搜索条数
+        self.limit_num = 1000  # 限制搜索条数
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        self.page_num = 0  # 二次搜索重新置0
+        self.page_num = 0  # 二次搜索重新置0
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        resp = self.get(self.init)
+        resp = self.get(self.init)
-        if not resp:
+        if not resp:
-            return
+            return
-        self.cookie = resp.cookies  # 获取cookie bing在搜索时需要带上cookie
+        self.cookie = resp.cookies  # 获取cookie bing在搜索时需要带上cookie
-        while True:
+        while True:
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            query = 'site:.' + domain + filtered_subdomain
+            query = 'site:.' + domain + filtered_subdomain
-            params = {'q': query, 'first': self.page_num,
+            params = {'q': query, 'first': self.page_num,
-                      'count': self.per_page_num}
+                      'count': self.per_page_num}
-            resp = self.get(self.addr, params)
+            resp = self.get(self.addr, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(domain, resp.text)
+            subdomains = self.match(domain, resp.text)
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
+                break
-            if not full_search:
+            if not full_search:
-                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
+                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            # 合并搜索子域名搜索结果
+            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            # 搜索页面没有出现下一页时停止搜索
+            # 搜索页面没有出现下一页时停止搜索
-            if '<div class="sw_next">' not in resp.text:
+            if '<div class="sw_next">' not in resp.text:
-                break
+                break
-            self.page_num += self.per_page_num
+            self.page_num += self.per_page_num
-            if self.page_num >= self.limit_num:  # 搜索条数限制
+            if self.page_num >= self.limit_num:  # 搜索条数限制
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-
+
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = Bing(domain)
+    search = Bing(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/bing_api.py

@@ -1,92 +1,92 @@
-import time
+import time
-import api
+from config import api
-from common.search import Search
+from common.search import Search
-
+
-
+
-class BingAPI(Search):
+class BingAPI(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'BingAPISearch'
+        self.source = 'BingAPISearch'
-        self.addr = 'https://api.cognitive.microsoft.com/' \
+        self.addr = 'https://api.cognitive.microsoft.com/' \
-                    'bing/v7.0/search'
+                    'bing/v7.0/search'
-        self.id = api.bing_api_id
+        self.id = api.bing_api_id
-        self.key = api.bing_api_key
+        self.key = api.bing_api_key
-        self.limit_num = 1000  # 必应同一个搜索关键词限制搜索条数
+        self.limit_num = 1000  # 必应同一个搜索关键词限制搜索条数
-        self.delay = 1  # 必应自定义搜索限制时延1秒
+        self.delay = 1  # 必应自定义搜索限制时延1秒
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        self.page_num = 0  # 二次搜索重新置0
+        self.page_num = 0  # 二次搜索重新置0
-        while True:
+        while True:
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.header = {'Ocp-Apim-Subscription-Key': self.key}
+            self.header = {'Ocp-Apim-Subscription-Key': self.key}
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            query = 'site:.' + domain + filtered_subdomain
+            query = 'site:.' + domain + filtered_subdomain
-            params = {'q': query, 'safesearch': 'Off',
+            params = {'q': query, 'safesearch': 'Off',
-                      'count': self.per_page_num,
+                      'count': self.per_page_num,
-                      'offset': self.page_num}
+                      'offset': self.page_num}
-            resp = self.get(self.addr, params)
+            resp = self.get(self.addr, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(domain, str(resp.json()))
+            subdomains = self.match(domain, str(resp.json()))
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
+                break
-            if not full_search:
+            if not full_search:
-                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
+                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            # 合并搜索子域名搜索结果
+            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            self.page_num += self.per_page_num
+            self.page_num += self.per_page_num
-            if self.page_num >= self.limit_num:  # 搜索条数限制
+            if self.page_num >= self.limit_num:  # 搜索条数限制
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.id, self.key):
+        if not self.check(self.id, self.key):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-
+
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = BingAPI(domain)
+    search = BingAPI(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/exalead.py

@@ -1,87 +1,87 @@
-import random
+import random
-import time
+import time
-from common.search import Search
+from common.search import Search
-
+
-
+
-class Exalead(Search):
+class Exalead(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = "ExaleadSearch"
+        self.source = "ExaleadSearch"
-        self.addr = "http://www.exalead.com/search/web/results/"
+        self.addr = "http://www.exalead.com/search/web/results/"
-        self.per_page_num = 30
+        self.per_page_num = 30
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        self.page_num = 0
+        self.page_num = 0
-        while True:
+        while True:
-            self.delay = random.randint(1, 5)
+            self.delay = random.randint(1, 5)
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            query = 'site:.' + domain + filtered_subdomain
+            query = 'site:.' + domain + filtered_subdomain
-            params = {'q': query, 'elements_per_page': '30',
+            params = {'q': query, 'elements_per_page': '30',
-                      "start_index": self.page_num}
+                      "start_index": self.page_num}
-            resp = self.get(url=self.addr, params=params)
+            resp = self.get(url=self.addr, params=params)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(domain, resp.text)
+            subdomains = self.match(domain, resp.text)
-            if not subdomains:
+            if not subdomains:
-                break
+                break
-            if not full_search:
+            if not full_search:
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            self.page_num += self.per_page_num
+            self.page_num += self.per_page_num
-            if self.page_num > 1999:
+            if self.page_num > 1999:
-                break
+                break
-            if 'title="Go to the next page"' not in resp.text:
+            if 'title="Go to the next page"' not in resp.text:
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-
+
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-
+
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            statement = statement.replace('-site', 'and -site')
+            statement = statement.replace('-site', 'and -site')
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = Exalead(domain)
+    search = Exalead(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/fofa_api.py

@@ -1,7 +1,7 @@
 import base64
 import time
 
-import api
+from config import api
 from common.search import Search
 
 

modules/search/gitee.py

@@ -1,7 +1,7 @@
 import time
 from bs4 import BeautifulSoup
 from common.search import Search
-from config import logger
+from config.log import logger
 
 
 class Gitee(Search):

modules/search/github_api.py

@@ -1,9 +1,8 @@
 import requests
-import api
+from config import api
-import json
 from common.utils import match_subdomain
 from common.search import Search
-from config import logger
+from config.log import logger
 
 
 class GithubAPI(Search):

modules/search/google.py

@@ -1,94 +1,94 @@
-import random
+import random
-import time
+import time
-from common.search import Search
+from common.search import Search
-
+
-
+
-class Google(Search):
+class Google(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'GoogleSearch'
+        self.source = 'GoogleSearch'
-        self.init = 'https://www.google.com/'
+        self.init = 'https://www.google.com/'
-        self.addr = 'https://www.google.com/search'
+        self.addr = 'https://www.google.com/search'
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        page_num = 1
+        page_num = 1
-        per_page_num = 50
+        per_page_num = 50
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.header.update({'User-Agent': 'Googlebot',
+        self.header.update({'User-Agent': 'Googlebot',
-                            'Referer': 'https://www.google.com'})
+                            'Referer': 'https://www.google.com'})
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        resp = self.get(self.init)
+        resp = self.get(self.init)
-        if not resp:
+        if not resp:
-            return
+            return
-        self.cookie = resp.cookies
+        self.cookie = resp.cookies
-        while True:
+        while True:
-            self.delay = random.randint(1, 5)
+            self.delay = random.randint(1, 5)
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            word = 'site:.' + domain + filtered_subdomain
+            word = 'site:.' + domain + filtered_subdomain
-            payload = {'q': word, 'start': page_num, 'num': per_page_num,
+            payload = {'q': word, 'start': page_num, 'num': per_page_num,
-                       'filter': '0', 'btnG': 'Search', 'gbv': '1', 'hl': 'en'}
+                       'filter': '0', 'btnG': 'Search', 'gbv': '1', 'hl': 'en'}
-            resp = self.get(url=self.addr, params=payload)
+            resp = self.get(url=self.addr, params=payload)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(domain, resp.text)
+            subdomains = self.match(domain, resp.text)
-            if not subdomains:
+            if not subdomains:
-                break
+                break
-            if not full_search:
+            if not full_search:
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            page_num += per_page_num
+            page_num += per_page_num
-            if 'start=' + str(page_num) not in resp.text:
+            if 'start=' + str(page_num) not in resp.text:
-                break
+                break
-            if '302 Moved' in resp.text:
+            if '302 Moved' in resp.text:
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-
+
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-
+
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = Google(domain)
+    search = Google(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/google_api.py

@@ -1,88 +1,88 @@
-import time
+import time
-import api
+from config import api
-from common.search import Search
+from common.search import Search
-
+
-
+
-class GoogleAPI(Search):
+class GoogleAPI(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'GoogleAPISearch'
+        self.source = 'GoogleAPISearch'
-        self.addr = 'https://www.googleapis.com/customsearch/v1'
+        self.addr = 'https://www.googleapis.com/customsearch/v1'
-        self.delay = 1
+        self.delay = 1
-        self.key = api.google_api_key
+        self.key = api.google_api_key
-        self.cx = api.google_api_cx
+        self.cx = api.google_api_cx
-        self.per_page_num = 10  # 每次只能请求10个结果
+        self.per_page_num = 10  # 每次只能请求10个结果
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        self.page_num = 1
+        self.page_num = 1
-        while True:
+        while True:
-            word = 'site:.' + domain + filtered_subdomain
+            word = 'site:.' + domain + filtered_subdomain
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            params = {'key': self.key, 'cx': self.cx,
+            params = {'key': self.key, 'cx': self.cx,
-                      'q': word, 'fields': 'items/link',
+                      'q': word, 'fields': 'items/link',
-                      'start': self.page_num, 'num': self.per_page_num}
+                      'start': self.page_num, 'num': self.per_page_num}
-            resp = self.get(self.addr, params)
+            resp = self.get(self.addr, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(domain, str(resp.json()))
+            subdomains = self.match(domain, str(resp.json()))
-            if not subdomains:
+            if not subdomains:
-                break
+                break
-            if not full_search:
+            if not full_search:
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            self.page_num += self.per_page_num
+            self.page_num += self.per_page_num
-            if self.page_num > 100:  # 免费的API只能查询前100条结果
+            if self.page_num > 100:  # 免费的API只能查询前100条结果
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.cx, self.key):
+        if not self.check(self.cx, self.key):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-
+
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = GoogleAPI(domain)
+    search = GoogleAPI(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/shodan_api.py

@@ -1,60 +1,60 @@
-import api
+from config import api
-from common.search import Search
+from common.search import Search
-
+
-
+
-class ShodanAPI(Search):
+class ShodanAPI(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = self.register(domain)
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'ShodanAPISearch'
+        self.source = 'ShodanAPISearch'
-        self.addr = 'https://api.shodan.io/shodan/host/search'
+        self.addr = 'https://api.shodan.io/shodan/host/search'
-        self.key = api.shodan_api_key
+        self.key = api.shodan_api_key
-
+
-    def search(self):
+    def search(self):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-        """
+        """
-        self.header = self.get_header()
+        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
+        self.proxy = self.get_proxy(self.source)
-        query = 'hostname:.' + self.domain
+        query = 'hostname:.' + self.domain
-        page = 1
+        page = 1
-        while True:
+        while True:
-            params = {'key': self.key, 'page': page, 'query': query,
+            params = {'key': self.key, 'page': page, 'query': query,
-                      'minify': True, 'facets': {'hostnames'}}
+                      'minify': True, 'facets': {'hostnames'}}
-            resp = self.get(self.addr, params)
+            resp = self.get(self.addr, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(self.domain, resp.text)
+            subdomains = self.match(self.domain, resp.text)
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
+                break
-            if subdomains:
+            if subdomains:
-                self.subdomains = self.subdomains.union(subdomains)
+                self.subdomains = self.subdomains.union(subdomains)
-            page += 1
+            page += 1
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.key):
+        if not self.check(self.key):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.search()
+        self.search()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = ShodanAPI(domain)
+    search = ShodanAPI(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/so.py

@@ -1,87 +1,87 @@
-import time
+import time
-
+
-from common.search import Search
+from common.search import Search
-
+
-
+
-class So(Search):
+class So(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'SoSearch'
+        self.source = 'SoSearch'
-        self.addr = 'https://www.so.com/s'
+        self.addr = 'https://www.so.com/s'
-        self.limit_num = 640  # 限制搜索条数
+        self.limit_num = 640  # 限制搜索条数
-        self.per_page_num = 10  # 默认每页显示10页
+        self.per_page_num = 10  # 默认每页显示10页
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        page_num = 1
+        page_num = 1
-        while True:
+        while True:
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            word = 'site:.' + domain + filtered_subdomain
+            word = 'site:.' + domain + filtered_subdomain
-            payload = {'q': word, 'pn': page_num}
+            payload = {'q': word, 'pn': page_num}
-            resp = self.get(url=self.addr, params=payload)
+            resp = self.get(url=self.addr, params=payload)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(domain, resp.text)
+            subdomains = self.match(domain, resp.text)
-            if not subdomains:
+            if not subdomains:
-                break
+                break
-            if not full_search:
+            if not full_search:
-                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
+                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            page_num += 1
+            page_num += 1
-            # 搜索页面没有出现下一页时停止搜索
+            # 搜索页面没有出现下一页时停止搜索
-            if '<a id="snext"' not in resp.text:
+            if '<a id="snext"' not in resp.text:
-                break
+                break
-            # 搜索条数限制
+            # 搜索条数限制
-            if self.page_num * self.per_page_num >= self.limit_num:
+            if self.page_num * self.per_page_num >= self.limit_num:
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-
+
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = So(domain)
+    search = So(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/sogou.py

@@ -1,85 +1,85 @@
-from common.search import Search
+from common.search import Search
-
+
-
+
-class Sogou(Search):
+class Sogou(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'SogouSearch'
+        self.source = 'SogouSearch'
-        self.addr = 'https://www.sogou.com/web'
+        self.addr = 'https://www.sogou.com/web'
-        self.limit_num = 1000  # 限制搜索条数
+        self.limit_num = 1000  # 限制搜索条数
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        self.page_num = 1
+        self.page_num = 1
-        while True:
+        while True:
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            word = 'site:.' + domain + filtered_subdomain
+            word = 'site:.' + domain + filtered_subdomain
-            payload = {'query': word, 'page': self.page_num,
+            payload = {'query': word, 'page': self.page_num,
-                       "num": self.per_page_num}
+                       "num": self.per_page_num}
-            resp = self.get(self.addr, payload)
+            resp = self.get(self.addr, payload)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(domain, resp.text)
+            subdomains = self.match(domain, resp.text)
-            if not subdomains:
+            if not subdomains:
-                break
+                break
-            if not full_search:
+            if not full_search:
-                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
+                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            self.page_num += 1
+            self.page_num += 1
-            # 搜索页面没有出现下一页时停止搜索
+            # 搜索页面没有出现下一页时停止搜索
-            if '<a id="sogou_next"' not in resp.text:
+            if '<a id="sogou_next"' not in resp.text:
-                break
+                break
-            # 搜索条数限制
+            # 搜索条数限制
-            if self.page_num * self.per_page_num >= self.limit_num:
+            if self.page_num * self.per_page_num >= self.limit_num:
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-
+
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-
+
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = Sogou(domain)
+    search = Sogou(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/yahoo.py

@@ -1,92 +1,92 @@
-import time
+import time
-from common.search import Search
+from common.search import Search
-
+
-
+
-class Yahoo(Search):
+class Yahoo(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'YahooSearch'
+        self.source = 'YahooSearch'
-        self.init = 'https://search.yahoo.com/'
+        self.init = 'https://search.yahoo.com/'
-        self.addr = 'https://search.yahoo.com/search'
+        self.addr = 'https://search.yahoo.com/search'
-        self.limit_num = 1000  # Yahoo限制搜索条数
+        self.limit_num = 1000  # Yahoo限制搜索条数
-        self.delay = 2
+        self.delay = 2
-        self.per_page_num = 30  # Yahoo每次搜索最大条数
+        self.per_page_num = 30  # Yahoo每次搜索最大条数
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        resp = self.get(self.init)
+        resp = self.get(self.init)
-        if not resp:
+        if not resp:
-            return
+            return
-        self.cookie = resp.cookies  # 获取cookie Yahoo在搜索时需要带上cookie
+        self.cookie = resp.cookies  # 获取cookie Yahoo在搜索时需要带上cookie
-        while True:
+        while True:
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            query = 'site:.' + domain + filtered_subdomain
+            query = 'site:.' + domain + filtered_subdomain
-            params = {'p': query, 'b': self.page_num, 'pz': self.per_page_num}
+            params = {'p': query, 'b': self.page_num, 'pz': self.per_page_num}
-            resp = self.get(self.addr, params)
+            resp = self.get(self.addr, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            text = resp.text.replace('<b>', '').replace('</b>', '')
+            text = resp.text.replace('<b>', '').replace('</b>', '')
-            subdomains = self.match(domain, text)
+            subdomains = self.match(domain, text)
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
+                break
-            if not full_search:
+            if not full_search:
-                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
+                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            # 合并搜索子域名搜索结果
+            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            if '>Next</a>' not in resp.text:  # 搜索页面没有出现下一页时停止搜索
+            if '>Next</a>' not in resp.text:  # 搜索页面没有出现下一页时停止搜索
-                break
+                break
-            self.page_num += self.per_page_num
+            self.page_num += self.per_page_num
-            if self.page_num >= self.limit_num:  # 搜索条数限制
+            if self.page_num >= self.limit_num:  # 搜索条数限制
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-
+
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-
+
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = Yahoo(domain)
+    search = Yahoo(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/yandex.py

@@ -1,92 +1,92 @@
-import time
+import time
-from common.search import Search
+from common.search import Search
-
+
-
+
-class Yandex(Search):
+class Yandex(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'YandexSearch'
+        self.source = 'YandexSearch'
-        self.init = 'https://yandex.com/'
+        self.init = 'https://yandex.com/'
-        self.addr = 'https://yandex.com/search'
+        self.addr = 'https://yandex.com/search'
-        self.limit_num = 1000  # 限制搜索条数
+        self.limit_num = 1000  # 限制搜索条数
-        self.delay = 5
+        self.delay = 5
-
+
-    def search(self, domain, filtered_subdomain='', full_search=False):
+    def search(self, domain, filtered_subdomain='', full_search=False):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-
+
-        :param str domain: 域名
+        :param str domain: 域名
-        :param str filtered_subdomain: 过滤的子域
+        :param str filtered_subdomain: 过滤的子域
-        :param bool full_search: 全量搜索
+        :param bool full_search: 全量搜索
-        """
+        """
-        self.page_num = 0  # 二次搜索重新置0
+        self.page_num = 0  # 二次搜索重新置0
-        resp = self.get(self.init)
+        resp = self.get(self.init)
-        if not resp:
+        if not resp:
-            return
+            return
-        self.cookie = resp.cookies  # 获取cookie
+        self.cookie = resp.cookies  # 获取cookie
-        while True:
+        while True:
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            query = 'site:.' + domain + filtered_subdomain
+            query = 'site:.' + domain + filtered_subdomain
-            params = {'text': query, 'p': self.page_num,
+            params = {'text': query, 'p': self.page_num,
-                      'numdoc': self.per_page_num}
+                      'numdoc': self.per_page_num}
-            resp = self.get(self.addr, params)
+            resp = self.get(self.addr, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(domain, resp.text)
+            subdomains = self.match(domain, resp.text)
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
+                break
-            if not full_search:
+            if not full_search:
-                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
+                # 搜索中发现搜索出的结果有完全重复的结果就停止搜索
-                if subdomains.issubset(self.subdomains):
+                if subdomains.issubset(self.subdomains):
-                    break
+                    break
-            # 合并搜索子域名搜索结果
+            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            if '>next</a>' not in resp.text:  # 搜索页面没有出现下一页时停止搜索
+            if '>next</a>' not in resp.text:  # 搜索页面没有出现下一页时停止搜索
-                break
+                break
-            self.page_num += 1
+            self.page_num += 1
-            if self.page_num >= self.limit_num:  # 搜索条数限制
+            if self.page_num >= self.limit_num:  # 搜索条数限制
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        self.begin()
+        self.begin()
-
+
-        self.search(self.domain, full_search=True)
+        self.search(self.domain, full_search=True)
-
+
-        # 排除同一子域搜索结果过多的子域以发现新的子域
+        # 排除同一子域搜索结果过多的子域以发现新的子域
-        for statement in self.filter(self.domain, self.subdomains):
+        for statement in self.filter(self.domain, self.subdomains):
-            self.search(self.domain, filtered_subdomain=statement)
+            self.search(self.domain, filtered_subdomain=statement)
-
+
-        # 递归搜索下一层的子域
+        # 递归搜索下一层的子域
-        if self.recursive_search:
+        if self.recursive_search:
-            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+            # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
-            for layer_num in range(1, self.recursive_times):
+            for layer_num in range(1, self.recursive_times):
-                for subdomain in self.subdomains:
+                for subdomain in self.subdomains:
-                    # 进行下一层子域搜索的限制条件
+                    # 进行下一层子域搜索的限制条件
-                    count = subdomain.count('.') - self.domain.count('.')
+                    count = subdomain.count('.') - self.domain.count('.')
-                    if count == layer_num:
+                    if count == layer_num:
-                        self.search(subdomain)
+                        self.search(subdomain)
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = Yandex(domain)
+    search = Yandex(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('example.com')
+    do('example.com')

modules/search/zoomeye_api.py

@@ -1,86 +1,86 @@
-import time
+import time
-import api
+from config import api
-from common.search import Search
+from common.search import Search
-from config import logger
+from config.log import logger
-
+
-
+
-class ZoomEyeAPI(Search):
+class ZoomEyeAPI(Search):
-    def __init__(self, domain):
+    def __init__(self, domain):
-        Search.__init__(self)
+        Search.__init__(self)
-        self.domain = domain
+        self.domain = domain
-        self.module = 'Search'
+        self.module = 'Search'
-        self.source = 'ZoomEyeAPISearch'
+        self.source = 'ZoomEyeAPISearch'
-        self.addr = 'https://api.zoomeye.org/web/search'
+        self.addr = 'https://api.zoomeye.org/web/search'
-        self.delay = 2
+        self.delay = 2
-        self.user = api.zoomeye_api_usermail
+        self.user = api.zoomeye_api_usermail
-        self.pwd = api.zoomeye_api_password
+        self.pwd = api.zoomeye_api_password
-
+
-    def login(self):
+    def login(self):
-        """
+        """
-        登陆获取查询taken
+        登陆获取查询taken
-        """
+        """
-        url = 'https://api.zoomeye.org/user/login'
+        url = 'https://api.zoomeye.org/user/login'
-        data = {'username': self.user, 'password': self.pwd}
+        data = {'username': self.user, 'password': self.pwd}
-        resp = self.post(url=url, json=data)
+        resp = self.post(url=url, json=data)
-        if not resp:
+        if not resp:
-            logger.log('FATAL', f'登录失败无法获取{self.source}的访问token')
+            logger.log('FATAL', f'登录失败无法获取{self.source}的访问token')
-            exit(1)
+            exit(1)
-        data = resp.json()
+        data = resp.json()
-        if resp.status_code == 200:
+        if resp.status_code == 200:
-            logger.log('DEBUG', f'{self.source}模块登录成功')
+            logger.log('DEBUG', f'{self.source}模块登录成功')
-            return data.get('access_token')
+            return data.get('access_token')
-        else:
+        else:
-            logger.log('ALERT', data.get('message'))
+            logger.log('ALERT', data.get('message'))
-            exit(1)
+            exit(1)
-
+
-    def search(self):
+    def search(self):
-        """
+        """
-        发送搜索请求并做子域匹配
+        发送搜索请求并做子域匹配
-        """
+        """
-        page_num = 1
+        page_num = 1
-        access_token = self.login()
+        access_token = self.login()
-        while True:
+        while True:
-            time.sleep(self.delay)
+            time.sleep(self.delay)
-            self.header = self.get_header()
+            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
+            self.proxy = self.get_proxy(self.source)
-            self.header.update({'Authorization': 'JWT ' + access_token})
+            self.header.update({'Authorization': 'JWT ' + access_token})
-            params = {'query': 'hostname:' + self.domain, 'page': page_num}
+            params = {'query': 'hostname:' + self.domain, 'page': page_num}
-            resp = self.get(self.addr, params)
+            resp = self.get(self.addr, params)
-            if not resp:
+            if not resp:
-                return
+                return
-            subdomains = self.match(self.domain, resp.text)
+            subdomains = self.match(self.domain, resp.text)
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
+                break
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.subdomains.union(subdomains)
-            page_num += 1
+            page_num += 1
-            if page_num > 500:
+            if page_num > 500:
-                break
+                break
-            if resp.status_code == 403:
+            if resp.status_code == 403:
-                break
+                break
-
+
-    def run(self):
+    def run(self):
-        """
+        """
-        类执行入口
+        类执行入口
-        """
+        """
-        if not self.check(self.user, self.pwd):
+        if not self.check(self.user, self.pwd):
-            return
+            return
-        self.begin()
+        self.begin()
-        self.search()
+        self.search()
-        self.finish()
+        self.finish()
-        self.save_json()
+        self.save_json()
-        self.gen_result()
+        self.gen_result()
-        self.save_db()
+        self.save_db()
-
+
-
+
-def do(domain):  # 统一入口名字 方便多线程调用
+def do(domain):  # 统一入口名字 方便多线程调用
-    """
+    """
-    类统一调用入口
+    类统一调用入口
-
+
-    :param str domain: 域名
+    :param str domain: 域名
-    """
+    """
-    search = ZoomEyeAPI(domain)
+    search = ZoomEyeAPI(domain)
-    search.run()
+    search.run()
-
+
-
+
-if __name__ == '__main__':
+if __name__ == '__main__':
-    do('mi.com')
+    do('mi.com')