v0.2.0

默认收集线程超时为3分钟
测试暂时不请求子域
2026-08-26 12:57:50 +08:00 · 2020-04-27 02:06:12 +08:00 · 2020-04-27 01:56:27 +08:00 · 2020-04-27 00:12:55 +08:00 · 2020-04-27 00:08:52 +08:00 · 2020-04-26 23:46:59 +08:00
122 changed files with 5603833 additions and 15977 deletions
@@ -1,35 +0,0 @@
---
-name: 提交Bug
-about: "请务必按照模板提交Bug\U0001F64F"
-title: 请填写BUG标题
-labels: bug
-assignees: shmilylty
-
---
-
-**是否使用了最新代码**
-是或否（如果不是的话尝试克隆最新的代码再跑一下）
-
-**Bug描述**
-清晰而简洁的Bug描述
-
-**如何复现**
-复现步骤（可不写）
-复现命令
-
-**预期结果**
-清晰而简洁的预期结果描述（可不写）
-
-**实际结果**
-清晰而简洁的实际结果描述（如出现什么错误）
-
-**屏幕截图**
-
-
-**运行环境**
- - 系统：[例如Windows 10 x64]
- - Python版本：[例如3.7.1]
-
-
-**报错文本**
-复制完整的报错文本
@@ -0,0 +1,39 @@
+---
+name: 请使用这个中文模板提交Bug
+about: "请务必按照模板提交Bug\U0001F64F"
+title: 请填写BUG标题
+labels: bug
+assignees: shmilylty
+
+---
+
+**是否使用了最新代码**
+是或否（如果不是的话尝试克隆最新的代码再跑一下！）
+
+**Bug描述**
+清晰而简洁的Bug描述（必写）
+
+**运行环境**
+ - 系统：[例如Windows 10 x64]（必写）
+ - Python版本：[例如3.7.1]（必写）
+ - OneForAll版本：[例如0.0.6]（必写）
+ 
+**如何复现**
+复现步骤（选写）
+
+复现命令（必写）
+
+**报错文本**
+复制完整的报错文本（必写）
+
+**预期结果**
+清晰而简洁的预期结果描述（选写，如正常情况应该是怎么样的）
+
+**实际结果**
+清晰而简洁的实际结果描述（选写，如出现什么错误）
+
+**屏幕截图**
+完整OneForAll执行流程截图（建议上传）
+
+**日志上传**
+上传oneforall.log日志文件（复杂问题建议上传）
@@ -0,0 +1,34 @@
+name: OneForAll test
+
+on: [push]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v1
+    - name: Set up Python 3.8
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.8
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+    - name: Lint with flake8
+      run: |
+        pip install flake8
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+#     - name: Test with pytest
+#       run: |
+#       pip install pytest
+#       pytest
+    - name: Test with example
+      run: |
+        pip install coverage
+        coverage run oneforall/example.py
@@ -19,13 +19,16 @@ matrix:
      os: osx
      osx_image: xcode10.2  # Python 3.7 running on macOS 10.14.3
      language: shell       # 'language: python' is an error on Travis CI macOS
-    - name: "Python 3.7 on Windows"
+    - name: "Python 3.8 on Windows"
      os: windows           # Windows 10.0.17134 N/A Build 17134
      language: shell
      before_install:
        - choco install python
        - python -m pip install --upgrade pip
-      env: PATH=/c/Python37:/c/Python37/Scripts:$PATH
+        - chcp.com 65001
+      env: 
+        - PATH=/c/Python38:/c/Python38/Scripts:$PATH
+        - PYTHONIOENCODING=UTF-8

 install:
  - pip3 install -U pip
@@ -1,45 +0,0 @@
-# 更新日志
-OneForAll的所有值得注意的更改都将记录在此文件中。
-
-OneForAll的更新日志格式基于[Keep a Changelog](https://keepachangelog.com/zh-CN/1.0.0/)。
-
-OneForAll遵守[语义化版本格式](https://semver.org/)。
-
-## Unreleased
-## [0.0.5](https://github.com/shmilylty/oneforall/releases/tag/v0.0.5) - 2019-08-19
- 修复一些已知Bugs
- 优化各子域收集接口并添加新的子域收集接口
- 添加子域DNS解析和子域HTTP探测进度条
- 添加子域接管风险检查模块及其使用说明
- 更新OneForAll依赖
-
-## [0.0.4](https://github.com/shmilylty/oneforall/releases/tag/v0.0.4) - 2019-08-11
-### 修复
- 修复一些已知Bugs
-
-## [0.0.3](https://github.com/shmilylty/oneforall/releases/tag/v0.0.3) - 2019-08-08
-### 修改
- 代码PEP8格式化
-### 修改
- 修改一些已知Bugs
-
-## [0.0.2](https://github.com/shmilylty/oneforall/releases/tag/v0.0.2) - 2019-08-04
-### 新增
- 新增有关文档
-### 修改
- 修改有关日志输出格式和信息
-### 修复
- 升级fire库版本解决运行报错问题
-### 移除
- 移除brotlipy依赖
-
-
-## [0.0.1](https://github.com/shmilylty/oneforall/releases/tag/v0.0.1) - 2019-08-02
-### 新增
- 新增检查crossdomain.xml收集子域功能
- 新增检查域名证书收集子域功能
- 新增检查内容安全策略头收集子域功能
- 新增域传送利用功能
- 新增子域收集功能（搜索引擎，DNS数据集，证书透明度，网上爬虫档案）
- 新增子域爆破功能
- 新增数据库导出功能
@@ -1,51 +0,0 @@
-# 参与者公约
-
-## 我们的承诺
-
-为建设开放友好的环境，我们贡献者和维护者承诺：不论年龄、体型、身体健全与否、民族、性征、性别认同与表征、经验水平、教育程度、社会地位、国籍、相貌、种族、信仰、性取向，我们项目和社区的参与者皆免于骚扰。
-
-## 我们的准则
-
-有助于创造积极环境的行为包括但不限于：
-
-* 措辞友好且包容
-* 尊重不同的观点和经验
-* 耐心接受有益批评
-* 关注对社区最有利的事情
-* 与社区其他成员友善相处
-
-参与者不应采取的行为包括但不限于：
-
-* 发布与性有关的言论或图像、不受欢迎地献殷勤
-* 捣乱/煽动/造谣行为、侮辱/贬损的评论、人身及政治攻击
-* 公开或私下骚扰
-* 未经明确授权便发布他人的资料，如住址、电子邮箱等
-* 其他有理由认定为违反职业操守的不当行为
-
-## 我们的义务
-
-项目维护者有义务诠释何谓“妥当行为”，并妥善公正地纠正已发生的不当行为。
-
-项目维护者有权利和义务去删除、编辑、拒绝违背本行为标准的评论（comments）、提交（commits）、代码、wiki 编辑、问题（issues）等贡献；项目维护者可暂时或永久地封禁任何他们认为行为不当、威胁、冒犯、有害的参与者。
-
-## 适用范围
-
-本行为标准适用于本项目。当有人代表本项目或本社区时，本标准亦适用于此人所处的公共平台。
-
-代表本项目或本社区的情形包括但不限于：使用项目的官方电子邮件、通过官方媒体账号发布消息、作为指定代表参与在线或线下活动等。
-
-代表本项目的行为可由项目维护者进一步定义及解释。
-
-## 贯彻落实
-
-可以致信[admin@hackfun.org]，向项目团队举报滥用、骚扰及不当行为。
-
-维护团队将审议并调查全部投诉，妥善地予以必要的回应。项目团队有义务保密举报者信息。具体执行方针或将另行发布。
-
-未切实遵守或执行本行为标准的项目维护人员，经项目负责人或其他成员决议，可能被暂时或永久地剥夺参与本项目的资格。
-
-## 来源
-
-本行为标准改编自[参与者公约](https://www.contributor-covenant.org)的1.4版本，可[在此](https://www.contributor-covenant.org/zh-cn/version/1/4/code-of-conduct.html)查阅。
-
-You can view the Contributor Covenant Translations [here](https://www.contributor-covenant.org/translations).
@@ -1,42 +0,0 @@
-# Contribution Guidelines
-
-Before opening any issues or proposing any pull requests and to get the greatest chance of helpful responses, please observe the following notes.
-
-## Good Bug Reports
-
-Please be aware of the following things when filing bug reports:
-
-1. Avoid raising duplicate issues. *Please* use the GitHub issue search feature
-   to check whether your bug report or feature request has been mentioned in
-   the past. Duplicate bug reports and feature requests are a huge maintenance
-   burden on the limited resources of the project. If it is clear from your
-   report that you would have struggled to find the original, that's ok, but
-   if searching for a selection of words in your issue title would have found
-   the duplicate then the issue will likely be closed extremely abruptly.
-2. When filing bug reports about exceptions or tracebacks, please include the
-   *complete* traceback. Partial tracebacks, or just the exception text, are
-   not helpful. Issues that do not contain complete tracebacks may be closed
-   without warning.
-3. Make sure you provide a suitable amount of information to work with. This
-   means you should provide:
-
-   - Guidance on **how to reproduce the issue**. Ideally, this should be a
-     *small* code sample that can be run immediately by the maintainers.
-     Failing that, let us know what you're doing, how often it happens, what
-     environment you're using, etc. Be thorough: it prevents us needing to ask
-     further questions.
-   - Tell us **what you expected to happen**. When we run your example code,
-     what are we expecting to happen? What does "success" look like for your
-     code?
-   - Tell us **what actually happens**. It's not helpful for you to say "it
-     doesn't work" or "it fails". Tell us *how* it fails: do you get an
-     exception? A hang? A non-200 status code? How was the actual result
-     different from your expected result?
-   - Tell us **what version of OneForAll you're using**, and
-     **how you installed it**. Different versions of Requests behave
-     differently and have different bugs, and some distributors of Requests
-     ship patches on top of the code we supply.
-
-   If you do not provide all of these things, it will take us much longer to
-   fix your problem. If we ask you to clarify these and you never respond, we
-   will close your issue without fixing it.
@@ -1,11 +0,0 @@
-# OneForAll贡献者
-
-* **[Jing Ling](https://github.com/shmilylty)**
-  * 核心开发
-
-* **[Black Star](https://github.com/blackstar24)**
-  * 模块贡献
-
-* [**iceMatcha**](https://github.com/iceMatcha)
-  * bug调试
-
@@ -0,0 +1,15 @@
+FROM python:3.8-alpine3.10
+MAINTAINER milktea@vmoe.info
+
+RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
+RUN apk update && apk --no-cache add git build-base libffi-dev libxml2-dev libxslt-dev libressl-dev
+ADD requirements.txt /requirements.txt
+RUN pip install -r /requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
+RUN git clone https://github.com/blechschmidt/massdns
+WORKDIR /massdns
+RUN make 
+ADD . /OneForAll/
+RUN mv /massdns/bin/massdns /OneForAll/oneforall/thirdparty/massdns/massdns_linux_x86_64
+WORKDIR /OneForAll/oneforall
+
+ENTRYPOINT ["/bin/ash"]
@@ -1 +0,0 @@
-
@@ -6,7 +6,6 @@ verify_ssl = true
 [dev-packages]

 [packages]
-aiodns = "*"
 tqdm = "*"
 aiomultiprocess = "*"
 loguru = "*"
@@ -18,12 +17,13 @@ exrex = "*"
 aiohttp = "*"
 fire = "*"
 bs4 = "*"
-cchardet = "*"
-lxml = "*"
 pysocks = "*"
 cloudscraper = "*"
 js2py = "*"
 tablib = "*"
+brotli = "*"
+psutil = "*"
+tenacity = "*"

 [requires]
-python_version = "3.7"
+python_version = "3.8"
@@ -1,11 +1,11 @@
 {
    "_meta": {
        "hash": {
-            "sha256": "0e5757dad136b3c0900ba481fad3b90e1b346b8cf8974e94676890f603f2a452"
+            "sha256": "d61abc7e1e2904735bf756b2efa14cce255f158d0f961a5ef00f42f775e1fb43"
        },
        "pipfile-spec": 6,
        "requires": {
-            "python_version": "3.7"
+            "python_version": "3.8"
        },
        "sources": [
            {
@@ -16,55 +16,30 @@
        ]
    },
    "default": {
-        "aiodns": {
-            "hashes": [
-                "sha256:815fdef4607474295d68da46978a54481dd1e7be153c7d60f9e72773cd38d77d",
-                "sha256:aaa5ac584f40fe778013df0aa6544bf157799bd3f608364b451840ed2c8688de"
-            ],
-            "index": "pypi",
-            "version": "==2.0.0"
-        },
        "aiohttp": {
            "hashes": [
-                "sha256:00d198585474299c9c3b4f1d5de1a576cc230d562abc5e4a0e81d71a20a6ca55",
-                "sha256:0155af66de8c21b8dba4992aaeeabf55503caefae00067a3b1139f86d0ec50ed",
-                "sha256:09654a9eca62d1bd6d64aa44db2498f60a5c1e0ac4750953fdd79d5c88955e10",
-                "sha256:199f1d106e2b44b6dacdf6f9245493c7d716b01d0b7fbe1959318ba4dc64d1f5",
-                "sha256:296f30dedc9f4b9e7a301e5cc963012264112d78a1d3094cd83ef148fdf33ca1",
-                "sha256:368ed312550bd663ce84dc4b032a962fcb3c7cae099dbbd48663afc305e3b939",
-                "sha256:40d7ea570b88db017c51392349cf99b7aefaaddd19d2c78368aeb0bddde9d390",
-                "sha256:629102a193162e37102c50713e2e31dc9a2fe7ac5e481da83e5bb3c0cee700aa",
-                "sha256:6d5ec9b8948c3d957e75ea14d41e9330e1ac3fed24ec53766c780f82805140dc",
-                "sha256:87331d1d6810214085a50749160196391a712a13336cd02ce1c3ea3d05bcf8d5",
-                "sha256:9a02a04bbe581c8605ac423ba3a74999ec9d8bce7ae37977a3d38680f5780b6d",
-                "sha256:9c4c83f4fa1938377da32bc2d59379025ceeee8e24b89f72fcbccd8ca22dc9bf",
-                "sha256:9cddaff94c0135ee627213ac6ca6d05724bfe6e7a356e5e09ec57bd3249510f6",
-                "sha256:a25237abf327530d9561ef751eef9511ab56fd9431023ca6f4803f1994104d72",
-                "sha256:a5cbd7157b0e383738b8e29d6e556fde8726823dae0e348952a61742b21aeb12",
-                "sha256:a97a516e02b726e089cffcde2eea0d3258450389bbac48cbe89e0f0b6e7b0366",
-                "sha256:acc89b29b5f4e2332d65cd1b7d10c609a75b88ef8925d487a611ca788432dfa4",
-                "sha256:b05bd85cc99b06740aad3629c2585bda7b83bd86e080b44ba47faf905fdf1300",
-                "sha256:c2bec436a2b5dafe5eaeb297c03711074d46b6eb236d002c13c42f25c4a8ce9d",
-                "sha256:cc619d974c8c11fe84527e4b5e1c07238799a8c29ea1c1285149170524ba9303",
-                "sha256:d4392defd4648badaa42b3e101080ae3313e8f4787cb517efd3f5b8157eaefd6",
-                "sha256:e1c3c582ee11af7f63a34a46f0448fca58e59889396ffdae1f482085061a2889"
+                "sha256:1e984191d1ec186881ffaed4581092ba04f7c61582a177b187d3a2f07ed9719e",
+                "sha256:259ab809ff0727d0e834ac5e8a283dc5e3e0ecc30c4d80b3cd17a4139ce1f326",
+                "sha256:2f4d1a4fdce595c947162333353d4a44952a724fba9ca3205a3df99a33d1307a",
+                "sha256:32e5f3b7e511aa850829fbe5aa32eb455e5534eaa4b1ce93231d00e2f76e5654",
+                "sha256:344c780466b73095a72c616fac5ea9c4665add7fc129f285fbdbca3cccf4612a",
+                "sha256:460bd4237d2dbecc3b5ed57e122992f60188afe46e7319116da5eb8a9dfedba4",
+                "sha256:4c6efd824d44ae697814a2a85604d8e992b875462c6655da161ff18fd4f29f17",
+                "sha256:50aaad128e6ac62e7bf7bd1f0c0a24bc968a0c0590a726d5a955af193544bcec",
+                "sha256:6206a135d072f88da3e71cc501c59d5abffa9d0bb43269a6dcd28d66bfafdbdd",
+                "sha256:65f31b622af739a802ca6fd1a3076fd0ae523f8485c52924a89561ba10c49b48",
+                "sha256:ae55bac364c405caa23a4f2d6cfecc6a0daada500274ffca4a9230e7129eac59",
+                "sha256:b778ce0c909a2653741cb4b1ac7015b5c130ab9c897611df43ae6a58523cb965"
            ],
            "index": "pypi",
-            "version": "==3.5.4"
+            "version": "==3.6.2"
        },
        "aiomultiprocess": {
            "hashes": [
-                "sha256:fd9b616d5145ac2b01f315725277231ffad7e56eb6675885598f6fe861ef9fa3"
+                "sha256:341d53af9b7fedf6425d9c09636d029035d63eecaa54caaff7354145a969c65e"
            ],
            "index": "pypi",
-            "version": "==0.6.0"
-        },
-        "asn1crypto": {
-            "hashes": [
-                "sha256:2f1adbb7546ed199e3c90ef23ec95c5cf3585bac7d11fb7eb562a3fe89c64e87",
-                "sha256:9d5c20441baf0cb60a4ac34cc447c6c189024b6b4c6cd7877034f4965c464e49"
-            ],
-            "version": "==0.24.0"
+            "version": "==0.7.0"
        },
        "async-timeout": {
            "hashes": [
@@ -75,25 +50,18 @@
        },
        "attrs": {
            "hashes": [
-                "sha256:69c0dbf2ed392de1cb5ec704444b08a5ef81680a61cb899dc08127123af36a79",
-                "sha256:f0b870f674851ecbfbbbd364d6b5cbdff9dcedbc7f3f5e18a6891057f21fe399"
+                "sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
+                "sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
            ],
-            "version": "==19.1.0"
-        },
-        "backports.csv": {
-            "hashes": [
-                "sha256:1277dfff73130b2e106bf3dd347adb3c5f6c4340882289d88f31240da92cbd6d",
-                "sha256:21f6e09bab589e6c1f877edbc40277b65e626262a86e69a70137db714eaac5ce"
-            ],
-            "version": "==1.0.7"
+            "version": "==19.3.0"
        },
        "beautifulsoup4": {
            "hashes": [
-                "sha256:05668158c7b85b791c5abde53e50265e16f98ad601c402ba44d70f96c4159612",
-                "sha256:25288c9e176f354bf277c0a10aa96c782a6a18a17122dba2e8cec4a97e03343b",
-                "sha256:f040590be10520f2ea4c2ae8c3dae441c7cfff5308ec9d58a0ec0c1b8f81d469"
+                "sha256:594ca51a10d2b3443cbac41214e12dbb2a1cd57e1a7344659849e2e20ba6a8d8",
+                "sha256:a4bbe77fd30670455c5296242967a123ec28c37e9702a8a81bd2f20a4baf0368",
+                "sha256:d4e96ac9b0c3a6d3f0caae2e4124e6055c5dcafde8e2f831ff194c104f0775a0"
            ],
-            "version": "==4.8.0"
+            "version": "==4.9.0"
        },
        "brotli": {
            "hashes": [
@@ -103,19 +71,23 @@
                "sha256:1e1aa9c4d1558889f42749c8baf846007953bfd32c8209230cf1cd1f5ef33495",
                "sha256:2f2f4f78f29ac4a45d15b3d9fc3fd9705e0ad313a44b129f6e1d0c6916bad0e2",
                "sha256:3269f6de1dd150fd0cce1c158b61ff5ac06d627fd3ae9c6ea03aed26fbbff7ea",
+                "sha256:3f4a1f6240916c7984c7f2542786710f622992508dafee0b1714e6d340fb9ffd",
                "sha256:50dd9ad2a2bb12da4e9002a438672d182f98e546e99952de80280a1e1729664f",
                "sha256:5519a4b01b1a4f965083cbfa2ef2b9774c5a5f352341c47b50776ad109423d72",
                "sha256:5eb27722d320370315971c427eb8aa7cc0791f2a458840d357ac653bd0ad3a14",
                "sha256:5f06b4d5b6f58e5b5c220c2f23cad034dc5efa51b01fde2351ced1605bd980e2",
+                "sha256:71ceee286ea7ec613f1c36f1c6181864a6ca24ebb55e371276f33d6af8742834",
                "sha256:72848d25a5f9e736db4af4512e0c3feecc094d57d241f8f1ae959115a2c39756",
                "sha256:743001bca75f4a6b4454be3510feca46f9d61a0c782a9bc2bc684bdb245e279e",
+                "sha256:7ac98c71a15648fd11bc1f32608b6110e396121280790082e32b9a3109048bc6",
                "sha256:9d1c2dd27a1083fefd05b1b2f8df4a6bc2aaa6c21dd82cd41c8ae5e7c23a87f8",
                "sha256:a13ce9b419fe9f277c63f700efb0e444331509d1881b5610d2ba7e9080606967",
                "sha256:a19ef0952b9d2803df88dff07f45a6c92d5676afb9b8d69cf32232d684036d11",
+                "sha256:a44d41e7a80cc832f04f4c1577a67bd311e28a6e40702a258590d8949a204d9d",
                "sha256:ad766ca8b8c1419b71a22756b45264f45725c86133dc80a7cbe30b6b78c75620",
                "sha256:ad7963f261988ee0883816b6b9f206f11461c9b3cb5cfbca0c9ab5adc406d395",
+                "sha256:af0451e23016631a2f52925a10d738ac4a0f794ac315c30380b22efc0c90cbc6",
                "sha256:c16201060c5a3f8742e3deae759014251ac92f382f82bc2a41dc079ff18c3f24",
-                "sha256:c43b202f65891861a9a336984a103de25de235f756de69e32db893156f767013",
                "sha256:c675c6cce4295cb1a692f3de7416aacace7314e064b94bc86e93aceefce7fd3e",
                "sha256:d17cec0b992b1434f5f9df9986563605a4d1b1acd5574c87fc2ac014bcbd3316",
                "sha256:dc91f6129953861a73d9a65c52a8dd682b561a9ebaf65283541645cab6489917",
@@ -124,8 +96,10 @@
                "sha256:f775b07026af2b1b0b5a8b05e41571cdcf3a315a67df265d60af301656a5425b",
                "sha256:f969ec7f56ba9636679e69ca07fba548312ccaca37412ee823c7f413541ad7e0",
                "sha256:f9dc52cd70907aafb99a773b66b156f2f995c7a0d284397c487c8b71ddbef2f9",
+                "sha256:f9ee88bb52352588ceb811d045b5c9bb1dc38927bc150fd156244f60ff3f59f1",
                "sha256:fc7212e36ebeb81aebf7949c92897b622490d7c0e333a479c0395591e7994600"
            ],
+            "index": "pypi",
            "version": "==1.0.7"
        },
        "bs4": {
@@ -135,74 +109,12 @@
            "index": "pypi",
            "version": "==0.0.1"
        },
-        "cchardet": {
-            "hashes": [
-                "sha256:079aa02a14072874d943a671ba778a9def5b0e3cedc2ac9f59308526cfb31472",
-                "sha256:3e048a21688dcb4c797f40c8deb3600887bcaf435620256fd8becd4252012750",
-                "sha256:41fced7a6f05ef859fe3eac89fc2120aca3cbbfd2b6c803bed3ee4bf02956903",
-                "sha256:440903d5dca3d326f4b841e7fa760b6af1be4f950ead1a6ff77b76eaa46f0cd3",
-                "sha256:50170f346527c5df4d3cb94648ca187c666e61c0db6e510b984e867c44709d8b",
-                "sha256:6c55a6e7bc7337671c9f1ad90746c0efb2b2979ff4305c7ca1d7d381f05174c1",
-                "sha256:7f581ea172b252034f745dfd49733966b73b73907bdef0b47ad5f2008b797d54",
-                "sha256:80f7b087198827e60c81574c321b12f89188eae626ae1567d66808928be42f88",
-                "sha256:8ba753ff73ca2f3554999a0e027eab9450f6ffdb7e92e1b4e13b52be89995349",
-                "sha256:9ad8f61d6d1ca37bd4b954ad92d461ea4f58d0dc413b0790a5abed7c09e54996",
-                "sha256:a35bd23cedbaa87cc9300af1dd10bb03fda41894045fbca7bfdf1d350b813f25",
-                "sha256:a8feb9a7def2310e18c27e485a21a38669abe8c2e36b93c6ce1a1363495d4cdf",
-                "sha256:aa9dd4cee8a5210a6d0a7b263b98dc50637e00401fc4a5ad3ce2dbef54fdfa02",
-                "sha256:ab9858a0673262e467619df91f425cfef0590dcf5deef5c0c7945e9dc4dbd7d8",
-                "sha256:b09a488bbb35be95f82845e3c4312be9025e8377975b027eee67e0b39445e070",
-                "sha256:b2893d558761b3534cddf5a49ba8d77df3d8f964d7b14680b925f4a85fc13476",
-                "sha256:b5a8f9b229a30cd2432572d15e169483bc47c24418772ff58d0585050631c2fd",
-                "sha256:bded54eeccd5f810bc69e076b3d9a35819a92e5e0559ad274b9ae9061b1b881d",
-                "sha256:cbc206061e69561af6e4cba11f99abd928346c6b5bcdc83eb32ae40e9fc23a5f",
-                "sha256:cc9745e0400da4cfb49f075e7819f22473b66443f953427058fee2c7b9547cc0",
-                "sha256:db30bf3825702c07fc55a290d41663fd8151f870642a15667bbabf81fff21e0b",
-                "sha256:eeeb1b95bb5851dda93ee522860a0e6066d47921cb1d540cb778346e37e5a524",
-                "sha256:f1c3919fb71ac5da3aeee42c5b731c99dcd2beed71db7fdc28ca993c173f0402"
-            ],
-            "index": "pypi",
-            "version": "==2.1.4"
-        },
        "certifi": {
            "hashes": [
-                "sha256:046832c04d4e752f37383b628bc601a7ea7211496b4638f6514d0e5b9acc4939",
-                "sha256:945e3ba63a0b9f577b1395204e13c3a231f9bc0223888be653286534e5873695"
+                "sha256:1d987a998c75633c40847cc966fcf5904906c920a7f17ef374f5aa4282abd304",
+                "sha256:51fcb31174be6e6664c5f69e3e1691a2d72a1a12e90f872cbdb1567eb47b6519"
            ],
-            "version": "==2019.6.16"
-        },
-        "cffi": {
-            "hashes": [
-                "sha256:041c81822e9f84b1d9c401182e174996f0bae9991f33725d059b771744290774",
-                "sha256:046ef9a22f5d3eed06334d01b1e836977eeef500d9b78e9ef693f9380ad0b83d",
-                "sha256:066bc4c7895c91812eff46f4b1c285220947d4aa46fa0a2651ff85f2afae9c90",
-                "sha256:066c7ff148ae33040c01058662d6752fd73fbc8e64787229ea8498c7d7f4041b",
-                "sha256:2444d0c61f03dcd26dbf7600cf64354376ee579acad77aef459e34efcb438c63",
-                "sha256:300832850b8f7967e278870c5d51e3819b9aad8f0a2c8dbe39ab11f119237f45",
-                "sha256:34c77afe85b6b9e967bd8154e3855e847b70ca42043db6ad17f26899a3df1b25",
-                "sha256:46de5fa00f7ac09f020729148ff632819649b3e05a007d286242c4882f7b1dc3",
-                "sha256:4aa8ee7ba27c472d429b980c51e714a24f47ca296d53f4d7868075b175866f4b",
-                "sha256:4d0004eb4351e35ed950c14c11e734182591465a33e960a4ab5e8d4f04d72647",
-                "sha256:4e3d3f31a1e202b0f5a35ba3bc4eb41e2fc2b11c1eff38b362de710bcffb5016",
-                "sha256:50bec6d35e6b1aaeb17f7c4e2b9374ebf95a8975d57863546fa83e8d31bdb8c4",
-                "sha256:55cad9a6df1e2a1d62063f79d0881a414a906a6962bc160ac968cc03ed3efcfb",
-                "sha256:5662ad4e4e84f1eaa8efce5da695c5d2e229c563f9d5ce5b0113f71321bcf753",
-                "sha256:59b4dc008f98fc6ee2bb4fd7fc786a8d70000d058c2bbe2698275bc53a8d3fa7",
-                "sha256:73e1ffefe05e4ccd7bcea61af76f36077b914f92b76f95ccf00b0c1b9186f3f9",
-                "sha256:a1f0fd46eba2d71ce1589f7e50a9e2ffaeb739fb2c11e8192aa2b45d5f6cc41f",
-                "sha256:a2e85dc204556657661051ff4bab75a84e968669765c8a2cd425918699c3d0e8",
-                "sha256:a5457d47dfff24882a21492e5815f891c0ca35fefae8aa742c6c263dac16ef1f",
-                "sha256:a8dccd61d52a8dae4a825cdbb7735da530179fea472903eb871a5513b5abbfdc",
-                "sha256:ae61af521ed676cf16ae94f30fe202781a38d7178b6b4ab622e4eec8cefaff42",
-                "sha256:b012a5edb48288f77a63dba0840c92d0504aa215612da4541b7b42d849bc83a3",
-                "sha256:d2c5cfa536227f57f97c92ac30c8109688ace8fa4ac086d19d0af47d134e2909",
-                "sha256:d42b5796e20aacc9d15e66befb7a345454eef794fdb0737d1af593447c6c8f45",
-                "sha256:dee54f5d30d775f525894d67b1495625dd9322945e7fee00731952e0368ff42d",
-                "sha256:e070535507bd6aa07124258171be2ee8dfc19119c28ca94c9dfb7efd23564512",
-                "sha256:e1ff2748c84d97b065cc95429814cdba39bcbd77c9c85c89344b317dc0d9cbff",
-                "sha256:ed851c75d1e0e043cbf5ca9a8e1b13c4c90f3fbd863dacb01c0808e2b5204201"
-            ],
-            "version": "==1.12.3"
+            "version": "==2020.4.5.1"
        },
        "chardet": {
            "hashes": [
@@ -213,47 +125,19 @@
        },
        "cloudscraper": {
            "hashes": [
-                "sha256:7080a4c9ea6fe244422551172eb28a1d6c042da4c9d56d15ff319dbbac2ab6d7",
-                "sha256:79d04cd3c7d782035b404b5093a0cdbacd219cb883134017f4f7ba703f12a93c"
+                "sha256:478d3cf2044d6b70a8e4f436afff4a8f74cb730cc34841cc9411c80d41d5e9e3",
+                "sha256:50a131a9bfc1969b01d48eecb4ecc1acd92ee992be31c0bee4f1f31ac3cbae8f"
            ],
            "index": "pypi",
-            "version": "==1.1.40"
+            "version": "==1.2.34"
        },
        "colorama": {
            "hashes": [
-                "sha256:05eed71e2e327246ad6b38c540c4a3117230b19679b875190486ddd2d721422d",
-                "sha256:f8ac84de7840f5b9c4e3347b3c1eaa50f7e49c2b07596221daec5edaabbd7c48"
+                "sha256:7d73d2a99753107a36ac6b455ee49046802e59d9d076ef8e47b61499fa29afff",
+                "sha256:e96da0d330793e2cb9485e9ddfd918d456036c7149416295932478192f4436a1"
            ],
            "markers": "sys_platform == 'win32'",
-            "version": "==0.4.1"
-        },
-        "cryptography": {
-            "hashes": [
-                "sha256:24b61e5fcb506424d3ec4e18bca995833839bf13c59fc43e530e488f28d46b8c",
-                "sha256:25dd1581a183e9e7a806fe0543f485103232f940fcfc301db65e630512cce643",
-                "sha256:3452bba7c21c69f2df772762be0066c7ed5dc65df494a1d53a58b683a83e1216",
-                "sha256:41a0be220dd1ed9e998f5891948306eb8c812b512dc398e5a01846d855050799",
-                "sha256:5751d8a11b956fbfa314f6553d186b94aa70fdb03d8a4d4f1c82dcacf0cbe28a",
-                "sha256:5f61c7d749048fa6e3322258b4263463bfccefecb0dd731b6561cb617a1d9bb9",
-                "sha256:72e24c521fa2106f19623a3851e9f89ddfdeb9ac63871c7643790f872a305dfc",
-                "sha256:7b97ae6ef5cba2e3bb14256625423413d5ce8d1abb91d4f29b6d1a081da765f8",
-                "sha256:961e886d8a3590fd2c723cf07be14e2a91cf53c25f02435c04d39e90780e3b53",
-                "sha256:96d8473848e984184b6728e2c9d391482008646276c3ff084a1bd89e15ff53a1",
-                "sha256:ae536da50c7ad1e002c3eee101871d93abdc90d9c5f651818450a0d3af718609",
-                "sha256:b0db0cecf396033abb4a93c95d1602f268b3a68bb0a9cc06a7cff587bb9a7292",
-                "sha256:cfee9164954c186b191b91d4193989ca994703b2fff406f71cf454a2d3c7327e",
-                "sha256:e6347742ac8f35ded4a46ff835c60e68c22a536a8ae5c4422966d06946b6d4c6",
-                "sha256:f27d93f0139a3c056172ebb5d4f9056e770fdf0206c2f422ff2ebbad142e09ed",
-                "sha256:f57b76e46a58b63d1c6375017f4564a28f19a5ca912691fd2e4261b3414b618d"
-            ],
-            "version": "==2.7"
-        },
-        "defusedxml": {
-            "hashes": [
-                "sha256:6687150770438374ab581bb7a1b327a847dd9c5749e396102de3fad4e8a3ef93",
-                "sha256:f684034d135af4c6cbb949b8a4d2ed61634515257a67299e5f940fbaa34377f5"
-            ],
-            "version": "==0.6.0"
+            "version": "==0.4.3"
        },
        "dnspython": {
            "hashes": [
@@ -284,17 +168,17 @@
        },
        "fire": {
            "hashes": [
-                "sha256:6865fefc6981a713d2ce56a2a2c92c56c729269f74a6cddd6f4b94d16ae084c9"
+                "sha256:9736a16227c3d469e5d2d296bce5b4d8fa8d7851e953bda327a455fc2994307f"
            ],
            "index": "pypi",
-            "version": "==0.2.1"
+            "version": "==0.3.1"
        },
        "idna": {
            "hashes": [
-                "sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407",
-                "sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c"
+                "sha256:7588d1c14ae4c77d74036e8c22ff447b26d0fde8f007354fd48a7814db15b7cb",
+                "sha256:a068a21ceac8a4d63dbfd964670474107f541babbd2250d61922f029858365fa"
            ],
-            "version": "==2.8"
+            "version": "==2.9"
        },
        "jdcal": {
            "hashes": [
@@ -305,87 +189,41 @@
        },
        "js2py": {
            "hashes": [
-                "sha256:6e5628abfff2fb4051e8e77a353e44831f474e2ceb865278271897f7f326aeb6",
-                "sha256:bf87cb4432944470f11fed9c1cb8d0312dd505e7b867362f55102f24379ab94f"
+                "sha256:7568b33f6bd15ee8ab1f3655928ed05481a6236ad2acca4598703ae501571661",
+                "sha256:f78e76c88942585973129a0a10e7054b5ea1602d761d4c0ccfa0593bc819f0bb"
            ],
            "index": "pypi",
-            "version": "==0.66"
+            "version": "==0.70"
        },
        "loguru": {
            "hashes": [
-                "sha256:b6fad0d7aed357b5c147edcc6982606b933754338950b72d8123f48c150c5a4f",
-                "sha256:e3138bfdee5f57481a2a6e078714be20f8c71ab1ff3f07f8fb1cfa25191fed2a"
+                "sha256:074b3caa6748452c1e4f2b302093c94b65d5a4c5a4d7743636b4121e06437b0e",
+                "sha256:a6101fd435ac89ba5205a105a26a6ede9e4ddbb4408a6e167852efca47806d11"
            ],
            "index": "pypi",
-            "version": "==0.3.2"
-        },
-        "lxml": {
-            "hashes": [
-                "sha256:02ca7bf899da57084041bb0f6095333e4d239948ad3169443f454add9f4e9cb4",
-                "sha256:096b82c5e0ea27ce9138bcbb205313343ee66a6e132f25c5ed67e2c8d960a1bc",
-                "sha256:0a920ff98cf1aac310470c644bc23b326402d3ef667ddafecb024e1713d485f1",
-                "sha256:17cae1730a782858a6e2758fd20dd0ef7567916c47757b694a06ffafdec20046",
-                "sha256:17e3950add54c882e032527795c625929613adbd2ce5162b94667334458b5a36",
-                "sha256:1f4f214337f6ee5825bf90a65d04d70aab05526c08191ab888cb5149501923c5",
-                "sha256:2e8f77db25b0a96af679e64ff9bf9dddb27d379c9900c3272f3041c4d1327c9d",
-                "sha256:4dffd405390a45ecb95ab5ab1c1b847553c18b0ef8ed01e10c1c8b1a76452916",
-                "sha256:6b899931a5648862c7b88c795eddff7588fb585e81cecce20f8d9da16eff96e0",
-                "sha256:726c17f3e0d7a7200718c9a890ccfeab391c9133e363a577a44717c85c71db27",
-                "sha256:760c12276fee05c36f95f8040180abc7fbebb9e5011447a97cdc289b5d6ab6fc",
-                "sha256:796685d3969815a633827c818863ee199440696b0961e200b011d79b9394bbe7",
-                "sha256:891fe897b49abb7db470c55664b198b1095e4943b9f82b7dcab317a19116cd38",
-                "sha256:a471628e20f03dcdfde00770eeaf9c77811f0c331c8805219ca7b87ac17576c5",
-                "sha256:a63b4fd3e2cabdcc9d918ed280bdde3e8e9641e04f3c59a2a3109644a07b9832",
-                "sha256:b0b84408d4eabc6de9dd1e1e0bc63e7731e890c0b378a62443e5741cfd0ae90a",
-                "sha256:be78485e5d5f3684e875dab60f40cddace2f5b2a8f7fede412358ab3214c3a6f",
-                "sha256:c27eaed872185f047bb7f7da2d21a7d8913457678c9a100a50db6da890bc28b9",
-                "sha256:c81cb40bff373ab7a7446d6bbca0190bccc5be3448b47b51d729e37799bb5692",
-                "sha256:d11874b3c33ee441059464711cd365b89fa1a9cf19ae75b0c189b01fbf735b84",
-                "sha256:e9c028b5897901361d81a4718d1db217b716424a0283afe9d6735fe0caf70f79",
-                "sha256:fe489d486cd00b739be826e8c1be188ddb74c7a1ca784d93d06fda882a6a1681"
-            ],
-            "index": "pypi",
-            "version": "==4.4.1"
+            "version": "==0.4.1"
        },
        "multidict": {
            "hashes": [
-                "sha256:024b8129695a952ebd93373e45b5d341dbb87c17ce49637b34000093f243dd4f",
-                "sha256:041e9442b11409be5e4fc8b6a97e4bcead758ab1e11768d1e69160bdde18acc3",
-                "sha256:045b4dd0e5f6121e6f314d81759abd2c257db4634260abcfe0d3f7083c4908ef",
-                "sha256:047c0a04e382ef8bd74b0de01407e8d8632d7d1b4db6f2561106af812a68741b",
-                "sha256:068167c2d7bbeebd359665ac4fff756be5ffac9cda02375b5c5a7c4777038e73",
-                "sha256:148ff60e0fffa2f5fad2eb25aae7bef23d8f3b8bdaf947a65cdbe84a978092bc",
-                "sha256:1d1c77013a259971a72ddaa83b9f42c80a93ff12df6a4723be99d858fa30bee3",
-                "sha256:1d48bc124a6b7a55006d97917f695effa9725d05abe8ee78fd60d6588b8344cd",
-                "sha256:31dfa2fc323097f8ad7acd41aa38d7c614dd1960ac6681745b6da124093dc351",
-                "sha256:34f82db7f80c49f38b032c5abb605c458bac997a6c3142e0d6c130be6fb2b941",
-                "sha256:3d5dd8e5998fb4ace04789d1d008e2bb532de501218519d70bb672c4c5a2fc5d",
-                "sha256:4a6ae52bd3ee41ee0f3acf4c60ceb3f44e0e3bc52ab7da1c2b2aa6703363a3d1",
-                "sha256:4b02a3b2a2f01d0490dd39321c74273fed0568568ea0e7ea23e02bd1fb10a10b",
-                "sha256:4b843f8e1dd6a3195679d9838eb4670222e8b8d01bc36c9894d6c3538316fa0a",
-                "sha256:5de53a28f40ef3c4fd57aeab6b590c2c663de87a5af76136ced519923d3efbb3",
-                "sha256:61b2b33ede821b94fa99ce0b09c9ece049c7067a33b279f343adfe35108a4ea7",
-                "sha256:6a3a9b0f45fd75dc05d8e93dc21b18fc1670135ec9544d1ad4acbcf6b86781d0",
-                "sha256:76ad8e4c69dadbb31bad17c16baee61c0d1a4a73bed2590b741b2e1a46d3edd0",
-                "sha256:7ba19b777dc00194d1b473180d4ca89a054dd18de27d0ee2e42a103ec9b7d014",
-                "sha256:7c1b7eab7a49aa96f3db1f716f0113a8a2e93c7375dd3d5d21c4941f1405c9c5",
-                "sha256:7fc0eee3046041387cbace9314926aa48b681202f8897f8bff3809967a049036",
-                "sha256:8ccd1c5fff1aa1427100ce188557fc31f1e0a383ad8ec42c559aabd4ff08802d",
-                "sha256:8e08dd76de80539d613654915a2f5196dbccc67448df291e69a88712ea21e24a",
-                "sha256:c18498c50c59263841862ea0501da9f2b3659c00db54abfbf823a80787fde8ce",
-                "sha256:c49db89d602c24928e68c0d510f4fcf8989d77defd01c973d6cbe27e684833b1",
-                "sha256:ce20044d0317649ddbb4e54dab3c1bcc7483c78c27d3f58ab3d0c7e6bc60d26a",
-                "sha256:d1071414dd06ca2eafa90c85a079169bfeb0e5f57fd0b45d44c092546fcd6fd9",
-                "sha256:d3be11ac43ab1a3e979dac80843b42226d5d3cccd3986f2e03152720a4297cd7",
-                "sha256:db603a1c235d110c860d5f39988ebc8218ee028f07a7cbc056ba6424372ca31b"
+                "sha256:317f96bc0950d249e96d8d29ab556d01dd38888fbe68324f46fd834b430169f1",
+                "sha256:42f56542166040b4474c0c608ed051732033cd821126493cf25b6c276df7dd35",
+                "sha256:4b7df040fb5fe826d689204f9b544af469593fb3ff3a069a6ad3409f742f5928",
+                "sha256:544fae9261232a97102e27a926019100a9db75bec7b37feedd74b3aa82f29969",
+                "sha256:620b37c3fea181dab09267cd5a84b0f23fa043beb8bc50d8474dd9694de1fa6e",
+                "sha256:6e6fef114741c4d7ca46da8449038ec8b1e880bbe68674c01ceeb1ac8a648e78",
+                "sha256:7774e9f6c9af3f12f296131453f7b81dabb7ebdb948483362f5afcaac8a826f1",
+                "sha256:85cb26c38c96f76b7ff38b86c9d560dea10cf3459bb5f4caf72fc1bb932c7136",
+                "sha256:a326f4240123a2ac66bb163eeba99578e9d63a8654a59f4688a79198f9aa10f8",
+                "sha256:ae402f43604e3b2bc41e8ea8b8526c7fa7139ed76b0d64fc48e28125925275b2",
+                "sha256:aee283c49601fa4c13adc64c09c978838a7e812f85377ae130a24d7198c0331e",
+                "sha256:b51249fdd2923739cd3efc95a3d6c363b67bbf779208e9f37fd5e68540d1a4d4",
+                "sha256:bb519becc46275c594410c6c28a8a0adc66fe24fef154a9addea54c1adb006f5",
+                "sha256:c2c37185fb0af79d5c117b8d2764f4321eeb12ba8c141a95d0aa8c2c1d0a11dd",
+                "sha256:dc561313279f9d05a3d0ffa89cd15ae477528ea37aa9795c4654588a3287a9ab",
+                "sha256:e439c9a10a95cb32abd708bb8be83b2134fa93790a4fb0535ca36db3dda94d20",
+                "sha256:fc3b4adc2ee8474cb3cd2a155305d5f8eda0a9c91320f83e55748e1fcb68f8e3"
            ],
-            "version": "==4.5.2"
-        },
-        "odfpy": {
-            "hashes": [
-                "sha256:596021f0519623ca8717331951c95e3b8d7b21e86edc7efe8cb650a0d0f59a2b"
-            ],
-            "version": "==1.4.0"
+            "version": "==4.7.5"
        },
        "openpyxl": {
            "hashes": [
@@ -393,29 +231,22 @@
            ],
            "version": "==2.4.11"
        },
-        "pycares": {
+        "psutil": {
            "hashes": [
-                "sha256:2ca080db265ea238dc45f997f94effb62b979a617569889e265c26a839ed6305",
-                "sha256:6f79c6afb6ce603009db2042fddc2e348ad093ece9784cbe2daa809499871a23",
-                "sha256:70918d06eb0603016d37092a5f2c0228509eb4e6c5a3faacb4184f6ab7be7650",
-                "sha256:755187d28d24a9ea63aa2b4c0638be31d65fbf7f0ce16d41261b9f8cb55a1b99",
-                "sha256:7baa4b1f2146eb8423ff8303ebde3a20fb444a60db761fba0430d104fe35ddbf",
-                "sha256:90b27d4df86395f465a171386bc341098d6d47b65944df46518814ae298f6cc6",
-                "sha256:9e090dd6b2afa65cb51c133883b2bf2240fd0f717b130b0048714b33fb0f47ce",
-                "sha256:a11b7d63c3718775f6e805d6464cb10943780395ab042c7e5a0a7a9f612735dd",
-                "sha256:b253f5dcaa0ac7076b79388a3ac80dd8f3bd979108f813baade40d3a9b8bf0bd",
-                "sha256:c7f4f65e44ba35e35ad3febc844270665bba21cfb0fb7d749434e705b556e087",
-                "sha256:cdb342e6a254f035bd976d95807a2184038fc088d957a5104dcaab8be602c093",
-                "sha256:cf08e164f8bfb83b9fe633feb56f2754fae6baefcea663593794fa0518f8f98c",
-                "sha256:df9bc694cf03673878ea8ce674082c5acd134991d64d6c306d4bd61c0c1df98f"
+                "sha256:1413f4158eb50e110777c4f15d7c759521703bd6beb58926f1d562da40180058",
+                "sha256:298af2f14b635c3c7118fd9183843f4e73e681bb6f01e12284d4d70d48a60953",
+                "sha256:60b86f327c198561f101a92be1995f9ae0399736b6eced8f24af41ec64fb88d4",
+                "sha256:685ec16ca14d079455892f25bd124df26ff9137664af445563c1bd36629b5e0e",
+                "sha256:73f35ab66c6c7a9ce82ba44b1e9b1050be2a80cd4dcc3352cc108656b115c74f",
+                "sha256:75e22717d4dbc7ca529ec5063000b2b294fc9a367f9c9ede1f65846c7955fd38",
+                "sha256:a02f4ac50d4a23253b68233b07e7cdb567bd025b982d5cf0ee78296990c22d9e",
+                "sha256:d008ddc00c6906ec80040d26dc2d3e3962109e40ad07fd8a12d0284ce5e0e4f8",
+                "sha256:d84029b190c8a66a946e28b4d3934d2ca1528ec94764b180f7d6ea57b0e75e26",
+                "sha256:e2d0c5b07c6fe5a87fa27b7855017edb0d52ee73b71e6ee368fae268605cc3f5",
+                "sha256:f344ca230dd8e8d5eee16827596f1c22ec0876127c28e800d7ae20ed44c4b310"
            ],
-            "version": "==3.0.0"
-        },
-        "pycparser": {
-            "hashes": [
-                "sha256:a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3"
-            ],
-            "version": "==2.19"
+            "index": "pypi",
+            "version": "==5.7.0"
        },
        "pyjsparser": {
            "hashes": [
@@ -424,46 +255,21 @@
            ],
            "version": "==2.7.1"
        },
-        "pyopenssl": {
-            "hashes": [
-                "sha256:aeca66338f6de19d1aa46ed634c3b9ae519a64b458f8468aec688e7e3c20f200",
-                "sha256:c727930ad54b10fc157015014b666f2d8b41f70c0d03e83ab67624fd3dd5d1e6"
-            ],
-            "version": "==19.0.0"
-        },
        "pysocks": {
            "hashes": [
-                "sha256:15d38914b60dbcb231d276f64882a20435c049450160e953ca7d313d1405f16f",
-                "sha256:32238918ac0f19e9fd870a8692ac9bd14f5e8752b3c62624cda5851424642210",
-                "sha256:d9031ea45fdfacbe59a99273e9f0448ddb33c1580fe3831c1b09557c5718977c"
+                "sha256:08e69f092cc6dbe92a0fdd16eeb9b9ffbc13cadfe5ca4c7bd92ffb078b293299",
+                "sha256:2725bd0a9925919b9b51739eea5f9e2bae91e83288108a9ad338b2e3a4435ee5",
+                "sha256:3f8804571ebe159c380ac6de37643bb4685970655d3bba243530d6558b799aa0"
            ],
            "index": "pypi",
-            "version": "==1.7.0"
+            "version": "==1.7.1"
        },
        "pytz": {
            "hashes": [
-                "sha256:26c0b32e437e54a18161324a2fca3c4b9846b74a8dccddd843113109e1116b32",
-                "sha256:c894d57500a4cd2d5c71114aaab77dbab5eabd9022308ce5ac9bb93a60a6f0c7"
+                "sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d",
+                "sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"
            ],
-            "version": "==2019.2"
-        },
-        "pyyaml": {
-            "hashes": [
-                "sha256:0113bc0ec2ad727182326b61326afa3d1d8280ae1122493553fd6f4397f33df9",
-                "sha256:01adf0b6c6f61bd11af6e10ca52b7d4057dd0be0343eb9283c878cf3af56aee4",
-                "sha256:5124373960b0b3f4aa7df1707e63e9f109b5263eca5976c66e08b1c552d4eaf8",
-                "sha256:5ca4f10adbddae56d824b2c09668e91219bb178a1eee1faa56af6f99f11bf696",
-                "sha256:7907be34ffa3c5a32b60b95f4d95ea25361c951383a894fec31be7252b2b6f34",
-                "sha256:7ec9b2a4ed5cad025c2278a1e6a19c011c80a3caaac804fd2d329e9cc2c287c9",
-                "sha256:87ae4c829bb25b9fe99cf71fbb2140c448f534e24c998cc60f39ae4f94396a73",
-                "sha256:9de9919becc9cc2ff03637872a440195ac4241c80536632fffeb6a1e25a74299",
-                "sha256:a5a85b10e450c66b49f98846937e8cfca1db3127a9d5d1e31ca45c3d0bef4c5b",
-                "sha256:b0997827b4f6a7c286c01c5f60384d218dca4ed7d9efa945c3e1aa623d5709ae",
-                "sha256:b631ef96d3222e62861443cc89d6563ba3eeb816eeb96b2629345ab795e53681",
-                "sha256:bf47c0607522fdbca6c9e817a6e81b08491de50f3766a7a0e6a5be7905961b41",
-                "sha256:f81025eddd0327c7d4cfe9b62cf33190e1e736cc6e97502b3ec425f574b3e7a8"
-            ],
-            "version": "==5.1.2"
+            "version": "==2019.3"
        },
        "records": {
            "hashes": [
@@ -475,11 +281,11 @@
        },
        "requests": {
            "hashes": [
-                "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
-                "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
+                "sha256:43999036bfa82904b6af1d99e4882b560e5e2c68e5c4b0aa03b655f3d7d73fee",
+                "sha256:b3f43d496c6daba4493e7c431722aeb7dbc6288f52a6e04e7b6023b0247817e6"
            ],
            "index": "pypi",
-            "version": "==2.22.0"
+            "version": "==2.23.0"
        },
        "requests-file": {
            "hashes": [
@@ -497,32 +303,58 @@
        },
        "six": {
            "hashes": [
-                "sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
-                "sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
+                "sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a",
+                "sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c"
            ],
-            "version": "==1.12.0"
+            "version": "==1.14.0"
        },
        "soupsieve": {
            "hashes": [
-                "sha256:8662843366b8d8779dec4e2f921bebec9afd856a5ff2e82cd419acc5054a1a92",
-                "sha256:a5a6166b4767725fd52ae55fee8c8b6137d9a51e9f1edea461a062a759160118"
+                "sha256:e914534802d7ffd233242b785229d5ba0766a7f487385e3f714446a07bf540ae",
+                "sha256:fcd71e08c0aee99aca1b73f45478549ee7e7fc006d51b37bec9e9def7dc22b69"
            ],
-            "version": "==1.9.3"
+            "version": "==2.0"
        },
        "sqlalchemy": {
            "hashes": [
-                "sha256:0459bf0ea6478f3e904de074d65769a11d74cdc34438ab3159250c96d089aef0"
+                "sha256:083e383a1dca8384d0ea6378bd182d83c600ed4ff4ec8247d3b2442cf70db1ad",
+                "sha256:0a690a6486658d03cc6a73536d46e796b6570ac1f8a7ec133f9e28c448b69828",
+                "sha256:114b6ace30001f056e944cebd46daef38fdb41ebb98f5e5940241a03ed6cad43",
+                "sha256:128f6179325f7597a46403dde0bf148478f868df44841348dfc8d158e00db1f9",
+                "sha256:13d48cd8b925b6893a4e59b2dfb3e59a5204fd8c98289aad353af78bd214db49",
+                "sha256:211a1ce7e825f7142121144bac76f53ac28b12172716a710f4bf3eab477e730b",
+                "sha256:2dc57ee80b76813759cccd1a7affedf9c4dbe5b065a91fb6092c9d8151d66078",
+                "sha256:3e625e283eecc15aee5b1ef77203bfb542563fa4a9aa622c7643c7b55438ff49",
+                "sha256:43078c7ec0457387c79b8d52fff90a7ad352ca4c7aa841c366238c3e2cf52fdf",
+                "sha256:5b1bf3c2c2dca738235ce08079783ef04f1a7fc5b21cf24adaae77f2da4e73c3",
+                "sha256:6056b671aeda3fc451382e52ab8a753c0d5f66ef2a5ccc8fa5ba7abd20988b4d",
+                "sha256:68d78cf4a9dfade2e6cf57c4be19f7b82ed66e67dacf93b32bb390c9bed12749",
+                "sha256:7025c639ce7e170db845e94006cf5f404e243e6fc00d6c86fa19e8ad8d411880",
+                "sha256:7224e126c00b8178dfd227bc337ba5e754b197a3867d33b9f30dc0208f773d70",
+                "sha256:7d98e0785c4cd7ae30b4a451416db71f5724a1839025544b4edbd92e00b91f0f",
+                "sha256:8d8c21e9d4efef01351bf28513648ceb988031be4159745a7ad1b3e28c8ff68a",
+                "sha256:bbb545da054e6297242a1bb1ba88e7a8ffb679f518258d66798ec712b82e4e07",
+                "sha256:d00b393f05dbd4ecd65c989b7f5a81110eae4baea7a6a4cdd94c20a908d1456e",
+                "sha256:e18752cecaef61031252ca72031d4d6247b3212ebb84748fc5d1a0d2029c23ea"
            ],
            "markers": "python_version >= '3.0'",
-            "version": "==1.3.7"
+            "version": "==1.3.16"
        },
        "tablib": {
            "hashes": [
-                "sha256:0f88a9cebdaa1a2cc29ae57387082ee81015d1149ecd34e48a8c8d3b4dd21670",
-                "sha256:5f33c079b07eb10cf9c4b4696add2ecf32c89db7729240546ecdcd5c92f67e13"
+                "sha256:4d1909aa3ff1c85ba97ad16176c0aeec33c8e894dc7ea6f10f2dd44701e99ba7",
+                "sha256:80f6c3453431cedf1125f23d16b3d96b92b426495714ebf0b4dede1fa75b447d"
            ],
            "index": "pypi",
-            "version": "==0.13.0"
+            "version": "==1.1.0"
+        },
+        "tenacity": {
+            "hashes": [
+                "sha256:f7bcbf5bb53875cfc38f61f596a88b1c994af32420f120c4409542a683ad613b",
+                "sha256:fb01d8ef2474eed422d8314a566c9e391cccb6e50cf4585022add6cb5cda66c8"
+            ],
+            "index": "pypi",
+            "version": "==6.1.0"
        },
        "termcolor": {
            "hashes": [
@@ -532,19 +364,19 @@
        },
        "tldextract": {
            "hashes": [
-                "sha256:2c1c5d9d454f79734b4f3da0d603856dd9f820753410a3e9abf0a0c9fde33e97",
-                "sha256:b72bef6013de67c7fa181250bc2c2e089a994d259c09ca95a9771f2f97e29ed1"
+                "sha256:16b2f7e81d89c2a5a914d25bdbddd3932c31a6b510db886c3ce0764a195c0ee7",
+                "sha256:9aa21a1f7827df4209e242ec4fc2293af5940ec730cde46ea80f66ed97bfc808"
            ],
            "index": "pypi",
-            "version": "==2.2.1"
+            "version": "==2.2.2"
        },
        "tqdm": {
            "hashes": [
-                "sha256:438d6a735167099d75e5fd9a55175c6727c4dbba345ae406b2886c2728fe3e80",
-                "sha256:ebc205051d79b49989140f5f6c73ec23fce5f590cbc4d9cd6e4c47f168fa0f10"
+                "sha256:00339634a22c10a7a22476ee946bbde2dbe48d042ded784e4d88e0236eca5d81",
+                "sha256:ea9e3fd6bd9a37e8783d75bfc4c1faf3c6813da6bd1c3e776488b41ec683af94"
            ],
            "index": "pypi",
-            "version": "==4.34.0"
+            "version": "==4.45.0"
        },
        "tzlocal": {
            "hashes": [
@@ -555,10 +387,10 @@
        },
        "urllib3": {
            "hashes": [
-                "sha256:b246607a25ac80bedac05c6f282e3cdaf3afb65420fd024ac94435cabe6e18d1",
-                "sha256:dbe59173209418ae49d485b87d1681aefa36252ee85884c31346debd19463232"
+                "sha256:3018294ebefce6572a474f0604c2021e33b3fd8006ecd11d62107a5d2a963527",
+                "sha256:88206b0eb87e6d677d424843ac5209e3fb9d0190d0ee169599165ec25e9d9115"
            ],
-            "version": "==1.25.3"
+            "version": "==1.25.9"
        },
        "win32-setctime": {
            "hashes": [
@@ -568,35 +400,27 @@
            "markers": "sys_platform == 'win32'",
            "version": "==1.0.1"
        },
-        "xlrd": {
-            "hashes": [
-                "sha256:546eb36cee8db40c3eaa46c351e67ffee6eeb5fa2650b71bc4c758a29a1b29b2",
-                "sha256:e551fb498759fa3a5384a94ccd4c3c02eb7c00ea424426e212ac0c57be9dfbde"
-            ],
-            "version": "==1.2.0"
-        },
-        "xlwt": {
-            "hashes": [
-                "sha256:a082260524678ba48a297d922cc385f58278b8aa68741596a87de01a9c628b2e",
-                "sha256:c59912717a9b28f1a3c2a98fd60741014b06b043936dcecbc113eaaada156c88"
-            ],
-            "version": "==1.3.0"
-        },
        "yarl": {
            "hashes": [
-                "sha256:024ecdc12bc02b321bc66b41327f930d1c2c543fa9a561b39861da9388ba7aa9",
-                "sha256:2f3010703295fbe1aec51023740871e64bb9664c789cba5a6bdf404e93f7568f",
-                "sha256:3890ab952d508523ef4881457c4099056546593fa05e93da84c7250516e632eb",
-                "sha256:3e2724eb9af5dc41648e5bb304fcf4891adc33258c6e14e2a7414ea32541e320",
-                "sha256:5badb97dd0abf26623a9982cd448ff12cb39b8e4c94032ccdedf22ce01a64842",
-                "sha256:73f447d11b530d860ca1e6b582f947688286ad16ca42256413083d13f260b7a0",
-                "sha256:7ab825726f2940c16d92aaec7d204cfc34ac26c0040da727cf8ba87255a33829",
-                "sha256:b25de84a8c20540531526dfbb0e2d2b648c13fd5dd126728c496d7c3fea33310",
-                "sha256:c6e341f5a6562af74ba55205dbd56d248daf1b5748ec48a0200ba227bb9e33f4",
-                "sha256:c9bb7c249c4432cd47e75af3864bc02d26c9594f49c82e2a28624417f0ae63b8",
-                "sha256:e060906c0c585565c718d1c3841747b61c5439af2211e185f6739a9412dfbde1"
+                "sha256:0c2ab325d33f1b824734b3ef51d4d54a54e0e7a23d13b86974507602334c2cce",
+                "sha256:0ca2f395591bbd85ddd50a82eb1fde9c1066fafe888c5c7cc1d810cf03fd3cc6",
+                "sha256:2098a4b4b9d75ee352807a95cdf5f10180db903bc5b7270715c6bbe2551f64ce",
+                "sha256:25e66e5e2007c7a39541ca13b559cd8ebc2ad8fe00ea94a2aad28a9b1e44e5ae",
+                "sha256:26d7c90cb04dee1665282a5d1a998defc1a9e012fdca0f33396f81508f49696d",
+                "sha256:308b98b0c8cd1dfef1a0311dc5e38ae8f9b58349226aa0533f15a16717ad702f",
+                "sha256:3ce3d4f7c6b69c4e4f0704b32eca8123b9c58ae91af740481aa57d7857b5e41b",
+                "sha256:58cd9c469eced558cd81aa3f484b2924e8897049e06889e8ff2510435b7ef74b",
+                "sha256:5b10eb0e7f044cf0b035112446b26a3a2946bca9d7d7edb5e54a2ad2f6652abb",
+                "sha256:6faa19d3824c21bcbfdfce5171e193c8b4ddafdf0ac3f129ccf0cdfcb083e462",
+                "sha256:944494be42fa630134bf907714d40207e646fd5a94423c90d5b514f7b0713fea",
+                "sha256:a161de7e50224e8e3de6e184707476b5a989037dcb24292b391a3d66ff158e70",
+                "sha256:a4844ebb2be14768f7994f2017f70aca39d658a96c786211be5ddbe1c68794c1",
+                "sha256:c2b509ac3d4b988ae8769901c66345425e361d518aecbe4acbfc2567e416626a",
+                "sha256:c9959d49a77b0e07559e579f38b2f3711c2b8716b8410b320bf9713013215a1b",
+                "sha256:d8cdee92bc930d8b09d8bd2043cedd544d9c8bd7436a77678dd602467a993080",
+                "sha256:e15199cdb423316e15f108f51249e44eb156ae5dba232cb73be555324a1d49c2"
            ],
-            "version": "==1.3.0"
+            "version": "==1.4.2"
        }
    },
    "develop": {}
@@ -0,0 +1,355 @@
+# OneForAll
+
+[![Build Status](https://travis-ci.org/shmilylty/OneForAll.svg?branch=master)](https://travis-ci.org/shmilylty/OneForAll)
+[![codecov](https://codecov.io/gh/shmilylty/OneForAll/branch/master/graph/badge.svg)](https://codecov.io/gh/shmilylty/OneForAll)
+[![Maintainability](https://api.codeclimate.com/v1/badges/1287668a6b4c72af683e/maintainability)](https://codeclimate.com/github/shmilylty/OneForAll/maintainability)
+[![License](https://img.shields.io/github/license/shmilylty/OneForAll)](https://github.com/shmilylty/OneForAll/tree/master/LICENSE)
+[![python](https://img.shields.io/badge/python-3.8-blue)](https://github.com/shmilylty/OneForAll/tree/master/)
+[![python](https://img.shields.io/badge/release-v0.2.0-brightgreen)](https://github.com/shmilylty/OneForAll/releases)
+
+👊**OneForAll is a powerful subdomain collection tool**  📝[中文文档](https://github.com/shmilylty/OneForAll/tree/master/README.md)
+
+![Example](./docs/usage_example.svg)
+
+## 🎉Project profile
+
+Project home：[https://shmilylty.github.io/OneForAll/](https://shmilylty.github.io/OneForAll/)
+
+Project address ：[https://github.com/shmilylty/OneForAll](https://github.com/shmilylty/OneForAll)
+
+The importance of information collection in penetration testing is self-evident. Subdomain collection is an essential and very important part of information collection. At present, there are many open source tools for subdomain collection on the Internet, but there are always some of the following problems:
+
+* **Not powerful enough**，there are not enough interfaces to collect subdomains automatically, and there are no functions such as automatic subdomain resolve, verification, FUZZ and information expansion.
+* **Not friendly enough**，although the command line module is more convenient, but when there are a lot of optional parameters and the operation to be implemented is complex, using command line mode is a bit unfriendly. If there is a good interaction, With a highly operable front end, the experience will be much better.
+
+* **Lack of maintenance**，Many tools have not been updated once in years, what issues and PR are, do not exist.
+
+* **Efficiency issues**，do not take advantage of multi-process, multi-threading and asynchronous cooperation technology, the speed is slow.
+
+In order to solve the above pain points, As its name suggests, I want OneForAll to be a collection of 100 strong, powerful and fast subdomains that collect the ultimate artifact 🔨.
+
+At present, OneForAll is still under development, there must be a lot of problems and areas for improvement. Welcome to submit [Issues](https://github.com/shmilylty/OneForAll/issues) or [PR](https://github.com/shmilylty/OneForAll/pulls)，If you like, give it to a little star ✨，At present, there is a special QQ group for OneForAll communication and feedback: 👨‍👨‍👦‍👦：:[**824414244**](//shang.qq.com/wpa/qunwpa?idkey=125d3689b60445cdbb11e4ddff38036b7f6f2abbf4f7957df5dddba81aa90771)，You can also [tweet](https://twitter.com/shmilylty) to me .
+
+## 👍Features
+
+* **Powerful collection capability**，For more information, please see [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md).
+  1. Collect subdomains using certificate transparency (there are currently 6 modules: `censys_api`，`certspotter`，`crtsh`，`entrust`，`google`，`spyse_api`）
+
+  2. General check collection subdomains (there are currently 4 modules: domain transfer vulnerability exploitation`axfr`, cross-domain policy file `cdx`, HTTPS certificate `cert`, content security policy `csp`, robots file `robots`, and sitemap file `sitemap`. Check NSEC record, NSEC3 record and other modules will be added later).
+
+  3.  Collect subdomains using web crawler files (there are currently 2 modules: `archirawl`, `commoncrawl`, which is still being debugged and needs to be added and improved). 
+
+  4.  Collect subdomains using DNS datasets (there are currently 23 modules: `binaryedge_api`, `bufferover`, `cebaidu`, `chinaz`, `chinaz_api`, `circl_api`, `dnsdb_api`, `dnsdumpster`, `hackertarget`, `ip138`, `ipv4info_api`, `netcraft`, `passivedns_api`, `ptrarchive`, `qianxun`, `rapiddns`, `riddler`, `robtex`, `securitytrails_api`, `sitedossier`, `threatcrowd`, `wzpc`, `ximcx`）
+
+  5.  Collect subdomains using DNS queries (There are currently 5 modules: collecting subdomains `srv` by enumerating common SRV records and making queries, and collecting subdomains by querying MX,NS,SOA,TXT records in DNS records of domain names). 
+
+  6.  Collect subdomains using threat intelligence platform data (there are currently 6 modules: `alienvault`, `riskiq_ api`, `threatbook_ api`, `threatkeeper `, `virustotal`, `virustotal_ api`, which need to be added and improved).
+
+  7.  Use search engines to discover subdomains (there are currently 18 modules: `ask`, `baidu`, `bing`, `bing_api`, `duckduckgo`, `exalead`, `fofa_api`, `gitee`, `github`, `github_api`, `google`, `google_api`, `shodan_api`, `so`, `sogou`, `yahoo`, `yandex`, `zoomeye_api`), except for special search engines in the search module. General search engines support automatic exclusion of search, full search, recursive search. 
+* **Support subdomain blasting**，This module has both conventional dictionary blasting and custom fuzz mode. It supports batch blasting and recursive blasting, and automatically judges pan-parsing and processing.
+* **Support subdmain verification**，default to enable subdomain verification, automatically resolve subdomain DNS, automatically request subdomain to obtain title and banner, and comprehensively determine subdomain survival.
+* **Support subdomain takeover**，By default, subdomain takeover risk checking is enabled. Automatic subdomain takeover is supported (only Github, remains to be improved at present), and batch inspection is supported.
+* **Powerful processing feature**，The found subdomain results support automatic removal, automatic DNS parsing, HTTP request detection, automatic filtering of valid subdomains, and expansion of Banner information for subdomains. The final supported export formats are `rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`.
+* **Very fast**，[collection module](https://github.com/shmilylty/OneForAll/tree/master/oneforall//collect.py) uses multithreaded calls, [blasting module](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps under the default configuration, and DNS parsing and HTTP requests use asynchronous multiprogramming in subdomain verification. Multithreaded check [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takeover.py) risk.
+* **Good experience**，Each module has a progress bar, and the results of each module are saved asynchronously.
+
+If you have any other great ideas, please let me know!😎
+
+## 🚀Start Guide
+
+📢 Please take a moment to read this document to help you quickly get familiar with OneForAll!
+
+**🐍Installation requirements**
+
+OneForAll is developed and tested based on [Python 3.8.0](https://www.python.org/downloads/release/python-380/). Please use a stable release higher than Python 3.8.0, other versions may have some problems (Windows platform must use Python 3.8.0 or later). For more information on installing the Python environment, please see [Python 3 installation Guide](https://pythonguidecn.readthedocs.io/zh/latest/starting/installation.html#python-3) . Run the following command to check the Python and pip3 versions:
+```bash
+python -V
+pip3 -V
+```
+If you see the following output, there is no problem with the Python environment:
+```bash
+Python 3.8.0
+pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages\pip (python 3.8)
+```
+
+**✔Installation steps (Git version)**
+
+1. **Download**
+   
+   This project has been mirrored in [Gitee](https://gitee.com/shmilylty/OneForAll.git). If you are in China, it is recommended that you use Gitee for cloning, which is faster:
+
+   ```bash
+   git clone https://gitee.com/shmilylty/OneForAll.git
+   ```
+   or：
+   ```bash
+   git clone https://github.com/shmilylty/OneForAll.git
+   ```
+
+2. **Installation**
+
+   Since the project is under development and will continue to be updated iteratively, `git clone` is used to clone the latest code repository during download, which is also convenient for subsequent updates. Downloading from Releases is not recommended because the update of the version in Releases is slow and inconvenient.
+   
+   You can install OneForAll dependencies via pip3 (if you are familiar with [pipenv](https://docs.pipenv.org/en/latest/), then it is recommended that you use [pipenv install dependencies](https://github.com/shmilylty/OneForAll/tree/master/docs/Installation_dependency.md), the following is an example of using **pip3** to install dependencies under **Windows system**: (Note: If your Python3 is installed in the system Program Files In the directory, such as: `C:\Program Files\Python38`, then run the command prompt cmd as an administrator to execute the following command!)
+```bash
+   cd OneForAll/
+   python -m pip install -U pip setuptools wheel
+   pip3 install -r requirements.txt
+   cd oneforall/
+   python oneforall.py --help
+```
+      For other system platforms, please refer to [dependency installation](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md). If you find that compiling a dependent library fails during the installation dependencies, Refer to the solution in the [troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md) documentation, if not resolved, welcome feedback.
+
+3. **Update**
+
+   ❗Note: If you have cloned the project before, please **back** up your own modified files (such as **config.py**) to the location outside the project before updating, then execute the following command **update** project:
+
+   ```bash
+   git fetch --all
+   git reset --hard origin/master
+   git pull
+   ```
+
+**✔Installation steps (Docker version)**
+
+Method 1: directly pull the deployed image (not updated in time)
+
+```shell
+docker pull tardis07/oneforall
+docker run -it oneforall
+```
+
+Method 2: build from `Dockerfile` (same as git version)
+
+```shell
+docker build -t oneforall .
+docker run -it oneforall
+```
+
+**✨Demonstration**
+
+1. If you are installing dependencies through pip3, run the example using the following command: 
+    ```bash
+    cd oneforall/
+    python3 oneforall.py --target example.com run
+    ```
+
+    ![Example](./docs/usage_example.svg)
+
+2. If you install dependencies through pipenv, run the example using the following command: 
+   ```bash
+   cd oneforall/
+   pipenv run python oneforall.py --target example.com run
+   ```
+
+**🧐Description**. 
+
+Let's take the command `python3 oneforall.py-- target example.com run` as an example. After the default parameters are executed normally, OneForAll will generate the corresponding results in the results directory: 
+
+![Result](./images/Result.png)
+
+`example.com.csv` is the result of subdomain collection under each primary domain. 
+
+`all_subdomain_result_1583034493.csv` is the summary result of the sub-domain collected by OneForAll each time, including `example.com.csv`. It is convenient to obtain all the results in batch collection scenarios. 
+
+`result.sqlite3` is the SQLite3 result database that stores the sub-domain collected by OneForAll each time you run SQLite3. The database structure is shown below: 
+
+![Database](./images/Database.png)
+
+A table like `example_com_origin_result` stores the initial subdomain collection results of each module. 
+
+A table like `example_com_resolve_result` stores the results of resolving subdomains.
+
+A table like `example_com_last_result` stores the results of the last sub-domain collection (it needs to be collected more than twice before it is generated). 
+
+A table like `example_com_now_result` stores the collection results of the current subdomain. In general, you just need to pay attention to this table.
+
+**🤔Help**
+
+The command line parameters only provide some common parameters. For more detailed parameter configuration, please see [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) if you think Some parameters are frequently used in the command interface or missing parameters. Feedback is welcome. For well-known reasons, if you want to use some of the wall's collection interface, please go to [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) to configure the proxy, some collection Modules need to provide APIs (most of which are freely available for registered accounts). If you need to use them, please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py) to configure the API. Information, if not used, please ignore the error message. (For detailed modules, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
+
+The OneForAll command line interface is based on [Fire](https://github.com/google/python-fire/). For more advanced usage of Fire, please refer to [using the Fire CLI](https://github.com/google/Python-fire/blob/master/docs/using-cli.md), if you have any doubts during the use, please feel free to give me feedback.
+
+[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py) is the main program entry, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py) and [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py) and other modules, in order to facilitate the sub-field blasting, brute.py is isolated independently, in order to facilitate the subdomain takeover risk check independently takeover.py, in order to facilitate the database export independently dbexport.py, these modules can be run separately, and the parameters accepted are more abundant, if you want to use these modules separately, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.en.md).
+
+❗ Note: When you encounter some problems or doubts during use, please use [Issues](https://github.com/shmilylty/OneForAll/issues) to search for answers. Also see [Q&troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&troubleshooting.md).
+
+**oneforall.py help**
+
+   The following help information may not be up to date. You can use `python oneforall.py --help` to get the latest help information.
+
+   ```bash
+   python oneforall.py --help
+   ```
+   ```bash
+   NAME
+       oneforall.py - OneForAll help Information
+   
+   SYNOPSIS
+       oneforall.py --target=TARGET <flags>
+   
+   DESCRIPTION
+       OneForAll is a powerful subdomain collection tool
+   
+       Example:
+           python3 oneforall.py version
+           python3 oneforall.py --target example.com run
+           python3 oneforall.py --target ./domains.txt run
+           python3 oneforall.py --target example.com --valid None run
+           python3 oneforall.py --target example.com --brute True run
+           python3 oneforall.py --target example.com --port small run
+           python3 oneforall.py --target example.com --format csv run
+           python3 oneforall.py --target example.com --dns False run
+           python3 oneforall.py --target example.com --req False run
+           python3 oneforall.py --target example.com --takeover False run
+           python3 oneforall.py --target example.com --show True run
+   
+       Note:
+           Parameter valid optional value 1, 0, none indicates that the export is 
+           valid, invalid, and all subdomains, respectively.
+           Parameter port have optional values 'default' 'small', 'large',
+           See config.py configuration for details.
+           Parameter format have optional values 'rst', 'csv', 'tsv', 'json', 
+           'yaml', 'html', 'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'.
+           If the parameter path is None, the appropriate file is generated in the 
+           project result directory based on the format parameter and the domain 
+           name.
+           Parameter path default None uses the OneForAll result directory generation path
+   
+   ARGUMENTS
+       TARGET
+           Single domain name or file path for one domain name per line (required)
+   
+   FLAGS
+       --brute=BRUTE
+           Use blasting module (default False)
+       --dns=DNS
+           DNS resolve subdomain (default True)
+       --req=REQ
+           HTTP request subdomain (default True)
+       --port=PORT
+           Port range for request authentication (default 80 port)
+       --valid=VALID
+           Export validity of subdomains (default None)
+       --format=FORMAT
+           Export format (default xls)
+       --path=PATH
+           Export path (default None)
+       --takeover=TAKEOVER
+           Check subdomain takeover (default False)
+       --show=SHOW
+           Terminal display exported data (default False)
+   ```
+
+## 🌲Directory structure
+
+```bash
+D:.
+|
+---.github
+---docs
+|       collection_modules.md collection module description
+---images
+\---oneforall
+    |   brute.py   Asynchronous multi-process multi-correlation subdomain blasting module, can be run separately
+    |   api.py        API configuration of some collection modules
+    |   collect.py    Upper layer call of each collection module
+    |   config.py     Configuration file
+    |   dbexport.py   Database export module, can be run separately
+    |   domains.txt   List of domain names to be blasted
+    |   oneforall.py  OneForAll main entrance, can be run separately
+    |   __init__.py
+    |
+    +---common Common call module
+    +---data   Store some of the required data
+    |       next_subdomains.txt     Next level subdomain dictionary
+    |       public_suffix_list.dat  Top-level domain name suffix
+    |       srv_names.json          Common SRV record prefix name
+    |       subdomains.txt          Common Dictionary of subdomain blasting
+    |
+    \---modules 
+        +---certificates     Using certificate transparency to collect subdomain
+        +---check            Using conventional methods to collect subdomain
+        +---crawl            Using web crawler files to collect subdomain
+        +---datasets         Using DNS datasets to collect subdomain
+        +---dnsquery         Using DNS query to collect subdomain
+        +---intelligence     Using threat intelligence platform to collect subdomain
+        \---search           Using search engine to collect subdomain
+
+```
+
+Description of the source of the subdomain dictionary::
+
+1. Some high frequency subdomain name dictionary in open source subdomain collection tool.
+2. List of the most popular subdomains published by relevant service providers online.
+ * [DNSPod](https://github.com/DNSPod/oh-my-free-data)
+3. Online research results by security researchers on common subdomains throughout the network.
+ * [the_most_popular_subdomains_on_the_internet](https://bitquark.co.uk/blog/2016/02/29/the_most_popular_subdomains_on_the_internet)
+ * [The most popular subdomains on the internet (2017 edition)](https://medium.com/@cmeister2/the-most-popular-subdomains-on-the-internet-2017-edition-a6b9c8a20fd8)
+4. Common business naming rules:
+ * single letter, single letter + single number, double letter, double letter + single number, double letter + double number, three letters, four letters. 
+ * single-digit, double-digit, triple-digit. 
+5. The names of tools and software that are common in companies or in DevOps. 
+6. Common Chinese words Pinyin and common English words.
+7. Optimize sorting and dirty data removal from the dictionary obtained above.
+8. You are very welcome to contribute a better dictionary.
+
+## 👏Framework used
+
+* [aiodns](https://github.com/saghul/aiodns) - aiodns provides a simple way for doing asynchronous DNS resolutions using [pycares](https://github.com/saghul/pycares). 
+* [aiohttp](https://github.com/aio-libs/aiohttp) -  Asynchronous HTTP client/server framework for asyncio and Python 
+* [aiomultiprocess](https://github.com/jreese/aiomultiprocess) -  Take a modern Python codebase to the next level of performance. (Multiprocessing and asyncio combine to implement asynchronous multi-process multi-coroutine)
+* [beautifulsoup4](https://pypi.org/project/beautifulsoup4/) -  Beautiful Soup is a library that makes it easy to scrape information from web pages. 
+* [fire](https://github.com/google/python-fire) -  Python Fire is a library for automatically generating command line interfaces (CLIs) from absolutely any Python object. 
+* [loguru](https://github.com/Delgan/loguru) -  Loguru is a library which aims to bring enjoyable logging in Python. 
+* [massdns](https://github.com/blechschmidt/massdns) - A high-performance DNS stub resolver
+* [records](https://github.com/kennethreitz/records) -  Records is a very simple, but powerful, library for making raw SQL queries to most relational databases.
+* [requests](https://github.com/psf/requests) -  A simple, yet elegant HTTP library. 
+* [tqdm](https://github.com/tqdm/tqdm) -  A Fast, Extensible Progress Bar for Python and CLI 
+
+Thanks to these great excellent Python libraries!
+
+## 🙏Contribution
+
+Very warmly welcome all ace to improve the project together!
+
+## ⌛Follow-up plan
+
+- [ ] Continuous optimization and improvement of each module
+- [x] Subdomain monitoring (marking each newly discovered subdomain)
+- [ ] Subdomain collection crawler implementation (including collection of subdomains from static resource files such as JS)
+- [ ] Implementation of front-end interface for powerful interaction (tentative: front-end: Element + back-end: Flask)
+
+For more details, see [todo.md](https://github.com/shmilylty/OneForAll/tree/master/docs/todo.md).
+
+## 🔖Version control
+
+The project uses [SemVer](https://semver.org/) language version format for version management), and you can view the available version in [Releases](https://github.com/shmilylty/OneForAll/releases), You can check [changes.md](https://github.com/shmilylty/OneForAll/tree/master/docs/changes.md)) for historical changes.
+
+## 👨‍💻Contributors
+
+* **[Jing Ling](https://github.com/shmilylty)**
+  * Core development
+
+You can see all the developers involved in the project in [contributors.md](https://github.com/shmilylty/OneForAll/tree/master/docs/contributors.md).
+
+## 📄License
+
+The project has signed a GPL-3.0 license, for more information, please see [LICENSE](https://github.com/shmilylty/OneForAll/LICENSE).
+
+## 😘Acknowledgement
+
+Thanks to the various subdomain collection projects of online open source!
+
+Thanks ace of [A-Team](https://github.com/QAX-A-Team) for their enthusiastic and unselfish answers!
+
+## 📜Disclaimer
+
+This tool is limited to legally authorized enterprise security construction. In the process of using this tool, you should ensure that all your actions comply with local laws and regulations and have obtained sufficient authorization.
+If you have any illegal behavior in the process of using this tool, you are responsible for all consequences, and all authors and all contributors of this tool do not assume any legal and joint responsibility.
+Unless you have fully read, fully understood and accepted all the terms of this Agreement, please do not install and use this tool.
+Your use or any other express or implied representation of you to this Agreement is deemed to have been read and agreed to be bound by this Agreement.
+
+## 💖Stargazers over time
+
+[![Stargazers over time](https://starchart.cc/shmilylty/OneForAll.svg)](https://starchart.cc/shmilylty/OneForAll)
@@ -3,16 +3,20 @@
 [![Build Status](https://travis-ci.org/shmilylty/OneForAll.svg?branch=master)](https://travis-ci.org/shmilylty/OneForAll)
 [![codecov](https://codecov.io/gh/shmilylty/OneForAll/branch/master/graph/badge.svg)](https://codecov.io/gh/shmilylty/OneForAll)
 [![Maintainability](https://api.codeclimate.com/v1/badges/1287668a6b4c72af683e/maintainability)](https://codeclimate.com/github/shmilylty/OneForAll/maintainability)
-[![License](https://img.shields.io/github/license/shmilylty/OneForAll)](./LICENSE)
-[![python](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8-blue)](./)
-[![python](https://img.shields.io/badge/release-v0.0.5-brightgreen)](https://github.com/shmilylty/OneForAll/releases)
+[![License](https://img.shields.io/github/license/shmilylty/OneForAll)](https://github.com/shmilylty/OneForAll/tree/master/LICENSE)
+[![python](https://img.shields.io/badge/python-3.8-blue)](https://github.com/shmilylty/OneForAll/tree/master/)
+[![python](https://img.shields.io/badge/release-v0.2.0-brightgreen)](https://github.com/shmilylty/OneForAll/releases)

-👊**OneForAll是一款功能强大的子域收集工具**  📝[English Document](./README.en.md)
+👊**OneForAll是一款功能强大的子域收集工具**  📝[English Document](https://github.com/shmilylty/OneForAll/tree/master/README.en.md)

-![](./images/All_Might.jpg)
+![Example](./docs/usage_example.svg)

 ## 🎉项目简介

+项目主页：[https://shmilylty.github.io/OneForAll/](https://shmilylty.github.io/OneForAll/)
+
+项目地址：[https://github.com/shmilylty/OneForAll](https://github.com/shmilylty/OneForAll)
+
 在渗透测试中信息收集的重要性不言而喻，子域收集是信息收集中必不可少且非常重要的一环，目前网上也开源了许多子域收集的工具，但是总是存在以下部分问题：

 * **不够强大**，子域收集的接口不够多，不能做到对批量子域自动收集，没有自动子域解析，验证，FUZZ以及信息拓展等功能。
@@ -22,51 +26,57 @@

 * **效率问题**，没有利用多进程，多线程以及异步协程技术，速度较慢。

-为了解决以上痛点，此项目应用而生，OneForAll一词是来自我喜欢的一部日漫《[我的英雄学院](https://manhua.fzdm.com/131/)》，它是一种通过一代代的传承不断变强的潜力无穷的顶级个性，目前[番剧](https://www.bilibili.com/bangumi/media/md7452/)也更新到了第三季了，欢迎大佬们入坑😄。正如其名，我希望OneForAll是一款集百家之长，功能强大的全面快速子域收集终极神器🔨。
+为了解决以上痛点，此项目应用而生，正如其名，我希望OneForAll是一款集百家之长，功能强大的全面快速子域收集终极神器🔨。

-目前OneForAll还在开发中，肯定有不少问题和需要改进的地方，欢迎大佬们提交[Issues](https://github.com/shmilylty/OneForAll/issues)和[PR](https://github.com/shmilylty/OneForAll/pulls)，用着还行给个小星星✨吧，目前有一个专门用于OneForAll交流和反馈QQ群👨‍👨‍👦‍👦：:[**824414244**](//shang.qq.com/wpa/qunwpa?idkey=125d3689b60445cdbb11e4ddff38036b7f6f2abbf4f7957df5dddba81aa90771)，也可以给我发邮件📧[admin@hackfun.org]。
+目前OneForAll还在开发中，肯定有不少问题和需要改进的地方，欢迎大佬们提交[Issues](https://github.com/shmilylty/OneForAll/issues)和[PR](https://github.com/shmilylty/OneForAll/pulls)，用着还行给个小星星✨吧，目前有一个专门用于OneForAll交流和反馈QQ群👨‍👨‍👦‍👦：:[**824414244**](//shang.qq.com/wpa/qunwpa?idkey=125d3689b60445cdbb11e4ddff38036b7f6f2abbf4f7957df5dddba81aa90771)（加群验证：我的英雄学院）。

 ## 👍功能特性

-* **收集能力强大**，详细模块请阅读[收集模块说明](./docs/collection_modules.md)。
-  1. 利用证书透明度收集子域（目前有6个模块：`censys_api`，`certdb_api`，`certspotter`，`crtsh`，`entrust`，`google`）
+* **收集能力强大**，详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)。
+  1. 利用证书透明度收集子域（目前有6个模块：`censys_api`，`certspotter`，`crtsh`，`entrust`，`google`，`spyse_api`）

  2. 常规检查收集子域（目前有4个模块：域传送漏洞利用`axfr`，检查跨域策略文件`cdx`，检查HTTPS证书`cert`，检查内容安全策略`csp`，检查robots文件`robots`，检查sitemap文件`sitemap`，后续会添加检查NSEC记录，NSEC3记录等模块）

  3. 利用网上爬虫档案收集子域（目前有2个模块：`archivecrawl`，`commoncrawl`，此模块还在调试，该模块还有待添加和完善）

-  4. 利用DNS数据集收集子域（目前有17个模块：`binaryedge_api`, `circl_api`, `hackertarget`, `riddler`, `bufferover`, `dnsdb`, `ipv4info`, `robtex`, `chinaz`, `dnsdb_api`, `netcraft`, `securitytrails_api`, `chinaz_api`, `dnsdumpster`, `ptrarchive`, `sitedossier`,`threatcrowd`）
+  4. 利用DNS数据集收集子域（目前有23个模块：`binaryedge_api`, `bufferover`, `cebaidu`, `chinaz`, `chinaz_api`, `circl_api`, `dnsdb_api`, `dnsdumpster`, `hackertarget`, `ip138`, `ipv4info_api`, `netcraft`, `passivedns_api`, `ptrarchive`, `qianxun`, `rapiddns`, `riddler`, `robtex`, `securitytrails_api`, `sitedossier`, `threatcrowd`, `wzpc`, `ximcx`）

-  5. 利用DNS查询收集子域（目前有1个模块：通过枚举常见的SRV记录并做查询来收集子域`srv`，该模块还有待添加和完善）
+  5. 利用DNS查询收集子域（目前有5个模块：通过枚举常见的SRV记录并做查询来收集子域`srv`，以及通过查询域名的DNS记录中的MX,NS,SOA,TXT记录来收集子域）

-  6. 利用威胁情报平台数据收集子域（目前有5个模块：`riskiq_api`，`threatbook_api`，`threatminer`，`virustotal`，`virustotal_api`该模块还有待添加和完善）
+  6. 利用威胁情报平台数据收集子域（目前有6个模块：`alienvault`, `riskiq_api`，`threatbook_api`，`threatminer`，`virustotal`，`virustotal_api`该模块还有待添加和完善）

-  7. 利用搜索引擎发现子域（目前有15个模块：`ask`, `bing_api`, `fofa_api`, `shodan_api`, `yahoo`, `baidu`, `duckduckgo`, `google`, `so`, `yandex`, `bing`, `exalead`, `google_api`, `sogou`, `zoomeye_api`），在搜索模块中除特殊搜索引擎，通用的搜索引擎都支持自动排除搜索，全量搜索，递归搜索。
+  7. 利用搜索引擎发现子域（目前有18个模块：`ask`, `baidu`, `bing`, `bing_api`, `duckduckgo`, `exalead`, `fofa_api`, `gitee`, `github`, `github_api`, `google`, `google_api`, `shodan_api`, `so`, `sogou`, `yahoo`, `yandex`, `zoomeye_api`），在搜索模块中除特殊搜索引擎，通用的搜索引擎都支持自动排除搜索，全量搜索，递归搜索。
+* **支持子域爆破**，该模块有常规的字典爆破，也有自定义的fuzz模式，支持批量爆破和递归爆破，自动判断泛解析并处理。
+* **支持子域验证**，默认开启子域验证，自动解析子域DNS，自动请求子域获取title和banner，并综合判断子域存活情况。
+* **支持子域接管**，默认开启子域接管风险检查，支持子域自动接管（目前只有Github，有待完善），支持批量检查。
+* **处理功能强大**，发现的子域结果支持自动去除，自动DNS解析，HTTP请求探测，自动筛选出有效子域，拓展子域的Banner信息，最终支持的导出格式有`rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`。
+* **速度极快**，[收集模块](https://github.com/shmilylty/OneForAll/tree/master/oneforall//collect.py)使用多线程调用，[爆破模块](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py)使用[massdns](https://github.com/blechschmidt/massdns)，默认配置下速度最少能达到10000pps，子域验证中DNS解析和HTTP请求使用异步多协程，多线程检查[子域接管](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takeover.py)风险。
+* **体验良好**，日志和终端输出全使用中文，各模块都有进度条，异步保存各模块结果。

-* **处理功能强大**，发现的子域结果支持自动去除，自动DNS解析，HTTP请求探测，自动移除无效子域，拓展子域的Banner信息，最终支持的导出格式有`csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`。
-
-* **速度极速**，[收集模块](./oneforall/collect.py)使用多线程调用，[爆破模块](./oneforall/aiobrute.py)使用异步多进程多协程，DNS解析和HTTP请求使用异步多协程。
+如果你有其他很棒的想法请务必告诉我！😎

 ## 🚀上手指南

-📢由于该项目**处于开发中**，会不断进行更新迭代，下载使用最好**克隆**最新项目，请务必花一点时间阅读此文档，有助于你快速熟悉OneForAll！
+📢 请务必花一点时间阅读此文档，有助于你快速熟悉OneForAll！

 **🐍安装要求**

-OneForAll是基于CPython开发的，所以你需要Python环境才能运行，如果你的系统还没有Pythin环境你可以参考[Python 3 安装指南](https://pythonguidecn.readthedocs.io/zh/latest/starting/installation.html#python-3)，理论上Python 3.6，3.7和3.8都可以正常运行OneForAll，**但是**许多测试都是在Python 3.7上进行的，所以**推荐**你使用**Python 3.7**版本运行OneForAll。运行以下命令检查Python和pip3版本：
+OneForAll基于[Python 3.8.0]( https://www.python.org/downloads/release/python-380/ )开发和测试，请使用高于Python 3.8.0的稳定发行版本，其他版本可能会出现一些问题（Windows平台必须使用3.8.0以上版本），安装Python环境可以参考[Python 3 安装指南](https://pythonguidecn.readthedocs.io/zh/latest/starting/installation.html#python-3)。运行以下命令检查Python和pip3版本：
 ```bash
 python -V
 pip3 -V
 ```
 如果你看到以下类似输出便说明Python环境没有问题：
 ```bash
-Python 3.7.4
-pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages\pip (python 3.7)
+Python 3.8.0
+pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python38\site-packages\pip (python 3.8)
 ```

-**✔安装步骤**
+**✔安装步骤（git 版）**

 1. **下载**
+
+   由于该项目**处于开发中**，会不断进行更新迭代，下载时使用`git clone`**克隆**最新代码仓库，也方便后续的更新，不推荐从Releases下载，因为Releases里版本更新缓慢，也不方便更新，
   本项目已经在[码云](https://gitee.com/shmilylty/OneForAll.git)(Gitee)镜像了一份，国内推荐使用码云进行克隆比较快：

   ```bash
@@ -78,19 +88,19 @@ pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages
   ```

 2. **安装**
-   首先运行以下命令
-   你可以通过pip3安装OneForAll的依赖（如果你熟悉[pipenv](https://docs.pipenv.org/en/latest/)，那么推荐你使用[pipenv安装依赖]((./docs/Installation_dependency.md))），以下为**Windows系统**下使用**pip3**安装依赖的示例：（注意：如果你的Python3安装在系统Program Files目录下，如：`C:\Program Files\Python37`，那么请以管理员身份运行命令提示符cmd执行以下命令！）

-   ```bash
+   你可以通过pip3安装OneForAll的依赖（如果你熟悉[pipenv](https://docs.pipenv.org/en/latest/)，那么推荐你使用[pipenv安装依赖]((https://github.com/shmilylty/OneForAll/tree/master/docs/Installation_dependency.md))），以下为**Windows系统**下使用**pip3**安装依赖的示例：（注意：如果你的Python3安装在系统Program Files目录下，如：`C:\Program Files\Python38`，那么请以管理员身份运行命令提示符cmd执行以下命令！）
+```bash
   cd OneForAll/
-   python -m pip install --user  -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
-   pip3 install --user -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
+   python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
+   pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
   cd oneforall/
   python oneforall.py --help
-   ```
-其他系统平台的请参考[依赖安装](./docs/installation_dependency.md)，如果在安装依赖过程中发现编译某个依赖库失败时可以参考[编译失败解决方法](./docs/building_fail_solution.md)，如果还没有解决欢迎加群反馈。
+```
+   其他系统平台的请参考[依赖安装](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md)，如果在安装依赖过程中发现编译某个依赖库失败时可以参考[troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md)中解决方法，如果还没有解决欢迎加群反馈。

 3. **更新**
+
   ❗注意：如果你之前已经克隆了项目运行之前请**备份**自己修改过的文件到项目外的地方（如**config.py**），然后执行以下命令**更新**项目：

   ```bash
@@ -99,6 +109,23 @@ pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages
   git pull
   ```

+**✔安装步骤（docker 版）**
+
+方法一：直接拉取部署好的镜像（更新不及时）
+
+```shell
+docker pull tardis07/oneforall
+docker run -it oneforall
+```
+
+方法二：从 `Dockerfile` 中构建（同git版）
+
+```shell
+docker build -t oneforall .
+docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
+```
+结果会输出在本地目录`~/results`
+
 **✨使用演示**

 1. 如果你是通过pip3安装的依赖则使用以下命令运行示例：   
@@ -115,159 +142,100 @@ pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages
   pipenv run python oneforall.py --target example.com run
   ```

+**🧐结果说明**
+
+我们以`python3 oneforall.py --target example.com run`命令为例，OneForAll在默认参数正常执行完毕会在results目录生成相应结果：
+
+![Result](./images/Result.png)
+
+`example.com.csv`是每个主域下的子域收集结果。
+
+`all_subdomain_result_1583034493.csv`是每次运行OneForAll收集到子域的汇总结果，包含`example.com.csv`，方便在批量收集场景中获取全部结果。
+
+`result.sqlite3`是存放每次运行OneForAll收集到子域的SQLite3结果数据库，其数据库结构如下图：
+
+![Database](./images/Database.png)
+
+其中类似`example_com_origin_result`表存放每个模块最初子域收集结果。
+
+其中类似`example_com_resolve_result`表存放对子域进行解析后的结果。
+
+其中类似`example_com_last_result`表存放上一次子域收集结果（需要收集两次以上才会生成）。
+
+其中类似`example_com_now_result`表存放现在子域收集结果，一般情况关注这张表就可以了。
+
 **🤔使用帮助**

-命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](./oneforall/config.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](./oneforall/config.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[config.py](./oneforall/config.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](./docs/collection_modules.md)）
+命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）

-OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实现，有关Fire更高级使用方法请参阅[使用Fire CLI](https://github.com/google/python-fire/blob/master/docs/using-cli.md)，有任何使用疑惑欢迎加群交流。
+OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实现，有关Fire更高级使用方法请参阅[使用Fire CLI](https://github.com/google/python-fire/blob/master/docs/using-cli.md)。

-oneforall.py是主程序入口，oneforall.py里有调用aiobrute.py和dbexport.py，为了方便进行子域爆破和数据库导出独立出了aiobrute.py和dbexport.py，这两个文件可以单独运行，并且所接受参数要更丰富一点。
+[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py)是主程序入口，oneforall.py可以调用[brute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py)，[takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py)及[dbexport.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py)等模块，为了方便进行子域爆破独立出了brute.py，为了方便进行子域接管风险检查独立出了takerover.py，为了方便数据库导出独立出了dbexport.py，这些模块都可以单独运行，并且所接受参数要更丰富一点，如果要单独使用这些模块请参考[使用帮助](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.md)

-❗注意：当你在使用过程中遇到一些问题或者疑惑时，请先到[Issues](https://github.com/shmilylty/OneForAll/issues)里使用搜索找找答案，还可以参阅[常见问题与回答](./docs/Q&A.md)。
+❗注意：当你在使用过程中遇到一些问题或者疑惑时，请先到[Issues](https://github.com/shmilylty/OneForAll/issues)里使用搜索找找答案，还可以参阅[常见问题与回答](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&A.md)。

-1. **oneforall.py使用帮助**
+**oneforall.py使用帮助**
+    
+   以下帮助信息可能不是最新的，你可以使用`python oneforall.py --help`获取最新的帮助信息。

   ```bash
   python oneforall.py --help
   ```
   ```bash
-   NAME
-       oneforall.py - OneForAll是一款功能强大的子域收集工具
-   
-   SYNOPSIS
-       oneforall.py --target=TARGET <flags>
-   
-   DESCRIPTION
-       Version: 0.0.5
-       Project: https://git.io/fjHT1
-   
-       Example:
-           python3 oneforall.py --target example.com run
-           python3 oneforall.py --target ./domains.txt run
-           python3 oneforall.py --target example.com --brute True run
-           python3 oneforall.py --target example.com --verify False run
-           python3 oneforall.py --target example.com --valid None run
-           python3 oneforall.py --target example.com --port medium run
-           python3 oneforall.py --target example.com --format csv run
-           python3 oneforall.py --target example.com --show True run
-   
-       Note:
-           参数valid可选值1，0，None分别表示导出有效，无效，全部子域
-           参数verify为True会尝试解析和请求子域并根据结果给子域有效性打上标签
-           参数port可选值有'small', 'medium', 'large', 'xlarge'，详见config.py配置
-           参数format可选格式有'txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
-                             'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
-           参数path为None会根据format参数和域名名称在项目结果目录生成相应文件
-   
-   ARGUMENTS
-       TARGET
-           单个域名或者每行一个域名的文件路径(必需参数)
-   
-   FLAGS
-       --brute=BRUTE
-           使用爆破模块(默认False)
-       --verify=VERIFY
-           验证子域有效性(默认True)
-       --port=PORT
-           请求验证的端口范围(默认medium)
-       --valid=VALID
-           导出子域的有效性(默认1)
-       --path=PATH
-           导出路径(默认None)
-       --format=FORMAT
-           导出格式(默认xlsx)
-       --show=SHOW
-           终端显示导出数据(默认False)
+    NAME
+        oneforall.py - OneForAll帮助信息
+
+    SYNOPSIS
+        oneforall.py COMMAND | --target=TARGET <flags>
+    
+    DESCRIPTION
+        OneForAll是一款功能强大的子域收集工具
+
+        Example:
+            python3 oneforall.py version
+            python3 oneforall.py --target example.com run
+            python3 oneforall.py --target ./domains.txt run
+            python3 oneforall.py --target example.com --valid None run
+            python3 oneforall.py --target example.com --brute True run
+            python3 oneforall.py --target example.com --port small run
+            python3 oneforall.py --target example.com --format csv run
+            python3 oneforall.py --target example.com --dns False run
+            python3 oneforall.py --target example.com --req False run
+            python3 oneforall.py --target example.com --takeover False run
+            python3 oneforall.py --target example.com --show True run
+    
+        Note:
+            参数valid可选值1，0，None分别表示导出有效，无效，全部子域
+            参数port可选值有'default', 'small', 'large', 详见config.py配置
+            参数format可选格式有'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
+                              'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
+            参数path默认None使用OneForAll结果目录生成路径
+
+    ARGUMENTS
+        TARGET
+            单个域名或者每行一个域名的文件路径(必需参数)
+    
+    FLAGS
+        --brute=BRUTE
+            使用爆破模块(默认False)
+        --dns=DNS
+            DNS解析子域(默认True)
+        --req=REQ
+            HTTP请求子域(默认True)
+        --port=PORT
+            请求验证子域的端口范围(默认只探测80端口)
+        --valid=VALID
+            导出子域的有效性(默认None)
+        --format=FORMAT
+            导出文件格式(默认csv)
+        --path=PATH
+            导出文件路径(默认None)
+        --takeover=TAKEOVER
+            检查子域接管(默认False)
+        --show=SHOW
+            终端显示导出数据(默认False)
   ```

-2. **aiobrute.py使用帮助**
-
-   关于泛解析问题处理程序首先会访问一个随机的子域判断是否泛解析，如果使用了泛解析则是通过以下判断处理：
-   - 一是主要是与泛解析的IP集合和TTL值做对比，可以参考[这篇文章](http://sh3ll.me/archives/201704041222.txt)。
-   - 二是多次解析到同一IP集合次数（默认设置为10，可以在config.py设置大小）
-   - 考虑爆破效率问题目前还没有加上HTTP响应体相似度对比和响应体内容判断
-   经过测试在16核心的CPU，使用16进程64协程，100M带宽的环境下，设置任务分割为50000，跑两百万字典大概10分钟左右跑完，大概3333个子域每秒。
-
-   ```bash
-   python aiobrute.py --help
-   ```
-
-   ```bash
-   NAME
-       aiobrute.py - OneForAll多进程多协程异步子域爆破模块
-   
-   SYNOPSIS
-       aiobrute.py --target=TARGET <flags>
-   
-   DESCRIPTION
-       Example：
-           python3 aiobrute.py --target example.com run
-           python3 aiobrute.py --target ./domains.txt run
-           python3 aiobrute.py --target example.com --process 4 --coroutine 64 run
-           python3 aiobrute.py --target example.com --wordlist subdomains.txt run
-           python3 aiobrute.py --target example.com --recursive True --depth 2 run
-           python3 aiobrute.py --target m.{fuzz}.a.bz --fuzz True --rule [a-z] run
-   
-       Note:
-           参数segment的设置受CPU性能，网络带宽，运营商限制等问题影响，默认设置500个子域为任务组，
-           当你觉得你的环境不受以上因素影响，当前爆破速度较慢，那么强烈建议根据字典大小调整大小：
-           十万字典建议设置为5000，百万字典设置为50000
-           参数valid可选值1，0，None，分别表示导出有效，无效，全部子域
-           参数format可选格式有'txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
-                             'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
-           参数path为None会根据format参数和域名名称在项目结果目录生成相应文件
-   
-   ARGUMENTS
-       TARGET
-           单个域名或者每行一个域名的文件路径
-   
-   FLAGS
-       --process=PROCESS
-           爆破的进程数(默认CPU核心数)
-       --coroutine=COROUTINE
-           每个爆破进程下的协程数(默认64)
-       --wordlist=WORDLIST
-           指定爆破所使用的字典路径(默认使用config.py配置)
-       --segment=SEGMENT
-           爆破任务分割(默认500)
-       --recursive=RECURSIVE
-           是否使用递归爆破(默认False)
-       --depth=DEPTH
-           递归爆破的深度(默认2)
-       --namelist=NAMELIST
-           指定递归爆破所使用的字典路径(默认使用config.py配置)
-       --fuzz=FUZZ
-           是否使用fuzz模式进行爆破(默认False，开启须指定fuzz正则规则)
-       --rule=RULE
-           fuzz模式使用的正则规则(默认使用config.py配置)
-       --export=EXPORT
-           是否导出爆破结果(默认True)
-       --valid=VALID
-           导出子域的有效性(默认None)
-       --format=FORMAT
-           导出格式(默认xlsx)
-       --path=PATH
-           导出路径(默认None)
-       --show=SHOW
-           终端显示导出数据(默认False)
-   ```
-
-3. 其他模块使用请参考[使用帮助](./docs/using_help.md)
-
-## 👏主要框架
-
-* [aiodns](https://github.com/saghul/aiodns) - 简单DNS异步解析库。
-* [aiohttp](https://github.com/aio-libs/aiohttp) - 异步http客户端/服务器框架
-* [aiomultiprocess](https://github.com/jreese/aiomultiprocess) - 将Python代码提升到更高的性能水平(multiprocessing和asyncio结合，实现异步多进程多协程)
-* [beautifulsoup4](https://pypi.org/project/beautifulsoup4/) - 可以轻松从HTML或XML文件中提取数据的Python库
-* [fire](https://github.com/google/python-fire) - Python Fire是一个纯粹根据任何Python对象自动生成命令行界面（CLI）的库
-* [loguru](https://github.com/Delgan/loguru) - 旨在带来愉快的日志记录Python库
-* [records](https://github.com/kennethreitz/records) - Records是一个非常简单但功能强大的库，用于对大多数关系数据库进行最原始SQL查询。
-* [requests](https://github.com/psf/requests) - Requests 唯一的一个非转基因的 Python HTTP 库，人类可以安全享用。
-* [tqdm](https://github.com/tqdm/tqdm) - 适用于Python和CLI的快速，可扩展的进度条库
-
-感谢这些伟大优秀的Python库！
-
 ## 🌲目录结构

 ```bash
@@ -278,7 +246,8 @@ D:.
 |       collection_modules.md 收集模块说明
 +---images
 \---oneforall
-    |   aiobrute.py   异步多进程多协程子域爆破模块，可以单独运行
+    |   brute.py      子域爆破模块，可以单独运行
+    |   api.py        一些收集模块的API配置
    |   collect.py    各个收集模块上层调用
    |   config.py     配置文件
    |   dbexport.py   数据库导出模块，可以单独运行
@@ -304,39 +273,69 @@ D:.

 ```

+关于子域字典来源的说明：
+1. 开源子域收集工具中的部分高频子域名字字典。
+2. 网上有关服务商公布的最流行子域列表。
+ * [DNSPod](https://github.com/DNSPod/oh-my-free-data)
+3. 网上有关安全研究人员关于对全网常见子域的研究结果。
+ * [the_most_popular_subdomains_on_the_internet](https://bitquark.co.uk/blog/2016/02/29/the_most_popular_subdomains_on_the_internet)
+ * [The most popular subdomains on the internet (2017 edition)](https://medium.com/@cmeister2/the-most-popular-subdomains-on-the-internet-2017-edition-a6b9c8a20fd8)
+4. 常见业务命名规律：
+ * 单字母、单字母+单数字、双字母、双字母+单数字、双字母+双数字、三字母、四字母；
+ * 单数字、双数字、三数字；
+5. 在公司或者说在DevOps中常见的工具和软件名称。
+6. 常见中文单词拼音和常见英文单词。
+7. 从以上获取的字典做优化排序以及脏数据去除处理。
+8. 非常欢迎你贡献更好的字典。
+ 
+## 👏用到框架
+
+* [aiodns](https://github.com/saghul/aiodns) - 简单DNS异步解析库。
+* [aiohttp](https://github.com/aio-libs/aiohttp) - 异步http客户端/服务器框架
+* [aiomultiprocess](https://github.com/jreese/aiomultiprocess) - 将Python代码提升到更高的性能水平(multiprocessing和asyncio结合，实现异步多进程多协程)
+* [beautifulsoup4](https://pypi.org/project/beautifulsoup4/) - 可以轻松从HTML或XML文件中提取数据的Python库
+* [fire](https://github.com/google/python-fire) - Python Fire是一个纯粹根据任何Python对象自动生成命令行界面（CLI）的库
+* [loguru](https://github.com/Delgan/loguru) - 旨在带来愉快的日志记录Python库
+* [massdns](https://github.com/blechschmidt/massdns) - 高性能的DNS解析器
+* [records](https://github.com/kennethreitz/records) - Records是一个非常简单但功能强大的库，用于对大多数关系数据库进行最原始SQL查询。
+* [requests](https://github.com/psf/requests) - Requests 唯一的一个非转基因的 Python HTTP 库，人类可以安全享用。
+* [tqdm](https://github.com/tqdm/tqdm) - 适用于Python和CLI的快速，可扩展的进度条库
+
+感谢这些伟大优秀的Python库！
+
 ## 🙏贡献

 非常热烈欢迎各位大佬一起完善本项目！

 ## ⌛后续计划

- [ ] 子域收集模块优化
- [ ] 子域接管功能实现
- [ ] 子域收集爬虫实现
- [ ] 操作强大交互人性的前端界面实现
+- [ ] 各模块持续优化和完善
+- [x] 子域监控（标记每次新发现的子域）
+- [ ] 子域收集爬虫实现（包括从JS等静态资源文件中收集子域）
+- [ ] 操作强大交互人性的前端界面实现（暂定：前端：Element + 后端：Flask）

-更多详细信息请阅读[TODO.md](./TODO.md)。
+更多详细信息请阅读[todo.md](https://github.com/shmilylty/OneForAll/tree/master/docs/todo.md)。

 ## 🔖版本控制

-该项目使用[SemVer](https://semver.org/)语言化版本格式进行版本管理，你可以在[Releases](https://github.com/shmilylty/OneForAll/releases)查看可用版本。
+该项目使用[SemVer](https://semver.org/)语言化版本格式进行版本管理，你可以在[Releases](https://github.com/shmilylty/OneForAll/releases)查看可用版本，你可以查阅[changes.md](https://github.com/shmilylty/OneForAll/tree/master/docs/changes.md)了解历史变更情况。

-## 👨‍💻作者
+## 👨‍💻贡献者

 * **[Jing Ling](https://github.com/shmilylty)**
  * 核心开发

-* **[Black Star](https://github.com/blackstar24)**
-  * 模块贡献
+你可以在[contributors.md](https://github.com/shmilylty/OneForAll/tree/master/docs/contributors.md)中查看所有参与该项目的开发者。

-* [**iceMatcha**](https://github.com/iceMatcha)
-  * bug调试
+## ☕赞赏

-*你也可以在[CONTRIBUTORS.md](./CONTRIBUTORS.md)中参看所有参与该项目的开发者。*
+如果你觉得这个项目帮助到了你，你可以打赏一杯咖啡以资鼓励:)
+
+![](https://raw.githubusercontent.com/shmilylty/OneForAll/master/images/Donate.png)

 ## 📄版权

-该项目签署了GPL-3.0授权许可，详情请参阅[LICENSE.md](./LICENSE.md)。
+该项目签署了GPL-3.0授权许可，详情请参阅[LICENSE](https://github.com/shmilylty/OneForAll/LICENSE)。

 ## 😘鸣谢

@@ -344,6 +343,13 @@ D:.

 感谢[A-Team](https://github.com/QAX-A-Team)大哥们热情无私的问题解答！

-## 📜免责声明 ##
+## 📜免责声明

-本工具仅限于安全研究与教学使用，用户使用本工具所造成的所有后果，由用户承担全部法律及连带责任，本项目所有作者和贡献者不承担任何法律及连带责任。
+本工具仅限于合法授权的企业安全建设，在使用本工具过程中，您应确保自己所有行为符合当地的法律法规，并且已经取得了足够的授权。
+如您在使用本工具的过程中存在任何非法行为，您需自行承担所有后果，本工具所有作者和所有贡献者不承担任何法律及连带责任。
+除非您已充分阅读、完全理解并接受本协议所有条款，否则，请您不要安装并使用本工具。
+您的使用行为或者您以其他任何明示或者默示方式表示接受本协议的，即视为您已阅读并同意本协议的约束。
+
+## 💖Star趋势
+
+[![Stargazers over time](https://starchart.cc/shmilylty/OneForAll.svg)](https://starchart.cc/shmilylty/OneForAll)
@@ -1,9 +0,0 @@
-# OneForAll后续开发计划
-
-## 下一步计划
-
- [ ] 子域收集模块优化
- [ ] 子域接管功能实现
- [ ] 子域收集爬虫实现
- [ ] 操作强大交互人性的前端界面实现
-
@@ -1,64 +0,0 @@
-# 安装依赖
-
-你可以通过pip3和pipenv两种方法安装OneForAll的依赖（如果你熟悉[pipenv](https://docs.pipenv.org/en/latest/)，那么推荐使用你使用pipenv）：
-
-* **Windows系统**（注意：如果你的Python3安装在系统Program Files目录下，如：`C:\Program Files\Python37`，那么请以管理员身份运行命令提示符cmd执行以下命令！）
-
-    1. 使用pipenv
-
-    ```bash
-    cd OneForAll/
-    python -m pip install --user -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
-    pip3 install --user pipenv -i https://mirrors.aliyun.com/pypi/simple/
-    pipenv install --user --python 3.7
-    cd oneforall
-    pipenv run python oneforall.py --help
-    ```
-
-    2. 使用pip3
-
-    ```bash
-    cd OneForAll/
-    python -m pip install --user  -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
-    pip3 install --user -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
-    cd oneforall/
-    python oneforall.py --help
-    ```
-* **Linux系统**
-
-    1. 使用pipenv
-    ```bash
-    cd OneForAll/
-    python3 -m pip install --user -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
-    sudo pip3 install --user pipenv -i https://mirrors.aliyun.com/pypi/simple/
-    sudo pipenv install --user --python 3.7
-    cd oneforall
-    pipenv run python3 oneforall.py --help
-    ```
-    2. 使用pip3
-    ```bash
-    cd OneForAll/
-    python3 -m pip install --user -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
-    pip3 install --user -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
-    cd oneforall/
-    python3 oneforall.py --help
-    ```
-* **Darwin系统**
-
-    1. 使用pipenv
-    ```bash
-    cd OneForAll/
-    python3 -m pip install --user -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
-    pip3 install --user pipenv -i https://mirrors.aliyun.com/pypi/simple/
-    pipenv install --user --python 3.7
-    cd oneforall
-    pipenv run python3 oneforall.py --help
-    ```
-    2. 使用pip3
-    ```bash
-    cd OneForAll/
-    python3 -m pip install --user -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
-    pip3 install --user -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
-    cd oneforall/
-    python3 oneforall.py --help
-    ```
@@ -1,12 +0,0 @@
-# 常见问题与回答
-
-## 使用问题
-
-1. 为什么运行OneForAll之后最终结果为空？
-
-   有几种可能性：第一可能目标域名没有子域。第二由于OneForAll默认会自动验证子域，在导出是只会有效子域，所以存在导出时没有有效子域的情况，你可以在运行OneForAll使用--valid=None指定导出所有发现的子域，你也可以使用--verify=False指定不验证子域的有效性。
-
-2. 安装依赖时出现以下类似报错
-   Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
-   
-   安装依赖时尝试加上--ignore-installed参数
@@ -1,13 +0,0 @@
-如果在安装依赖过程遇到编译某个依赖库失败时可以尝试以下方法：
-
-1. 到提供编译好的whl文件的第三方平台，找到对应库手动下载安装。第三方平台平台有： 
- * [https://www.lfd.uci.edu/~gohlke/pythonlibs](https://www.lfd.uci.edu/~gohlke/pythonlibs)
- * [https://pythonwheels.com/](https://pythonwheels.com/)
-
-选择好对应版本执行以下命令手动安装。举个例子，当编译pycares时失败时，找到[https://www.lfd.uci.edu/~gohlke/pythonlibs/#pycares](https://www.lfd.uci.edu/~gohlke/pythonlibs/#pycares)，由于我的系统是Windows 10 64位，使用的Python 3.7便下载`pycares‑3.0.0‑cp37‑cp37m‑win_amd64.whl`（一般来说下载最新版本的），然后手动安装：
-
-```bash
-pip3 install pycares‑3.0.0‑cp37‑cp37m‑win_amd64.whl
-```
-
-2. 到库的项目地址issues和wiki等找找有没有解决方法，如果没有就给他们提issues发邮件😜。
@@ -0,0 +1,98 @@
+# 更新日志
+OneForAll的所有值得注意的更改都将记录在此文件中。
+
+OneForAll的更新日志格式基于[Keep a Changelog](https://keepachangelog.com/zh-CN/1.0.0/)。
+
+OneForAll遵守[语义化版本格式](https://semver.org/)。
+
+# Unreleased
+
+# Released
+## [0.2.0](https://github.com/shmilylty/oneforall/releases/tag/v0.2.0) - 2020-04-27
+- 重构子域爆破和解析模块 改用massdns 一般情况下可以达到10000pps 速度非常快
+- 优化泛解析处理
+- 优化部分子域收集模块
+- 新增了爆破字典
+- 新增了github_api和rapiddns收集模块
+- 修复了一些bug
+- 更新了文档
+
+## [0.1.0](https://github.com/shmilylty/oneforall/releases/tag/v0.1.0) - 2020-03-02
+- 重构OneForAll入口
+- 添加1个新的子域收集模块
+- 添加了查询类型type和子域层数level两个结果字段
+- 调整项目结构
+- 优化个别收集模块
+- 优化子域爆破字典
+- 优化响应体解码处理
+- 更新说明文档
+- 修复已知bug
+
+## [0.0.9](https://github.com/shmilylty/oneforall/releases/tag/v0.0.9) - 2020-02-20
+- 重构子域解析模块
+- 添加4个新的子域收集模块
+- 添加了Docker部署
+- 优化配置参数和收集模块
+- 优化子域爆破字典和默认参数
+- 优化响应体解码处理
+- 更新说明文档
+- 修复已知bug
+
+## [0.0.8](https://github.com/shmilylty/oneforall/releases/tag/v0.0.8) - 2019-10-30
+- 添加新子域监控功能
+- 优化子域爆破字典和默认参数
+- 修复端口重复问题
+- 移除aiodns依赖
+
+## [0.0.7](https://github.com/shmilylty/oneforall/releases/tag/v0.0.7) - 2019-10-18
+- 修复一些已知问题
+- 添加百度云观测接口
+- 添加添加英文Readme文档
+- 更新有关文档
+- 优化标题获取
+- 更新依赖
+
+## [0.0.6](https://github.com/shmilylty/oneforall/releases/tag/v0.0.6) - 2019-08-27
+- 修复一些已知问题
+- 添加PassiveDNS查询和Github子域搜索模块
+- 优化FoFa和BufferOver收集模块
+- 更新有关文档
+- 更新依赖
+
+## [0.0.5](https://github.com/shmilylty/oneforall/releases/tag/v0.0.5) - 2019-08-19
+- 修复一些已知Bugs
+- 优化各子域收集接口并添加新的子域收集接口
+- 添加子域DNS解析和子域HTTP探测进度条
+- 添加子域接管风险检查模块及其使用说明
+- 更新OneForAll依赖
+
+## [0.0.4](https://github.com/shmilylty/oneforall/releases/tag/v0.0.4) - 2019-08-11
+### 修复
+- 修复一些已知Bugs
+
+## [0.0.3](https://github.com/shmilylty/oneforall/releases/tag/v0.0.3) - 2019-08-08
+### 修改
+- 代码PEP8格式化
+### 修改
+- 修改一些已知Bugs
+
+## [0.0.2](https://github.com/shmilylty/oneforall/releases/tag/v0.0.2) - 2019-08-04
+### 新增
+- 新增有关文档
+### 修改
+- 修改有关日志输出格式和信息
+### 修复
+- 升级fire库版本解决运行报错问题
+### 移除
+- 移除brotlipy依赖
+
+
+## [0.0.1](https://github.com/shmilylty/oneforall/releases/tag/v0.0.1) - 2019-08-02
+### 新增
+- 新增检查crossdomain.xml收集子域功能
+- 新增检查域名证书收集子域功能
+- 新增检查内容安全策略头收集子域功能
+- 新增域传送利用功能
+- 新增子域收集功能（搜索引擎，DNS数据集，证书透明度，网上爬虫档案）
+- 新增子域爆破功能
+- 新增数据库导出功能
@@ -1,15 +1,15 @@
 #  收集模块说明 #

-如果要使用通过API收集子域的模块请先到[config.py](../oneforall/config.py)配置相关信息，大多平台的API都是可以注册账号免费获取的。
+如果要使用通过API收集子域的模块请先到[api.py](../oneforall/api.py)配置相关信息，大多平台的API都是可以注册账号免费获取的。

-如果你指定使用某些模块可以在[config.py](../oneforall/config.py)中设置：
+如果你指定使用某些模块可以在[api.py](../oneforall/api.py)中设置：

 ```python
 enable_all_module = False  # 不开启所有模块
 enable_partial_module = [('modules.search', 'ask')，('modules.search', 'baidu')]  # 只使用ask和baidu搜索引擎收集子域
 ```

-如果你指定使用某些模块使用代理可以在[config.py](../oneforall/config.py)中设置：
+如果你指定使用某些模块使用代理可以在[api.py](../oneforall/api.py)中设置：

 ```python
 enable_proxy = True  # 使用代理
@@ -23,12 +23,12 @@ proxy_partial_module = ['GoogleQuery', 'AskSearch']  # 只代理GoogleQuery和As
    
      | 模块名称    | 是否需要代理 | 是否需要API | 其他说明                                           |
      | ----------- | ------------ | ----------- | -------------------------------------------------- |
-      | censys_api  | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
-      | certdb_api  | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
+      | censys_api  | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
      | certspotter | 否           | 否          |                                                    |
      | crtsh       | 否           | 否          |                                                    |
      | entrust     | 否           | 否          |                                                    |
      | google      | 是           | 否          |                                                    |
+      | spyse_api   | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |


  2. 常规检查收集子域（目前有4个模块：域传送漏洞利用`axfr`，检查跨域策略文件`cdx`，检查HTTPS证书`cert`，检查内容安全策略`csp`，后续会添加检查NSEC记录，NSEC3记录等模块）
@@ -48,59 +48,66 @@ proxy_partial_module = ['GoogleQuery', 'AskSearch']  # 只代理GoogleQuery和As
      | archivecrawl | 否           | 否          |          |
      | commoncrawl  | 否           | 否          |          |

-  4. 利用DNS数据集收集子域（目前有17个模块：`binaryedge_api`, `circl_api`, `hackertarget`, `riddler`, `bufferover`, `dnsdb`, `ipv4info`, `robtex`, `chinaz`, `dnsdb_api`, `netcraft`, `securitytrails_api`, `chinaz_api`, `dnsdumpster`, `ptrarchive`, `sitedossier`,`threatcrowd`）
+  4. 利用DNS数据集收集子域（目前有22个模块：`cebaidu`, `binaryedge_api`, `circl_api`, `hackertarget`, `riddler`, `bufferover`, `dnsdb`, `ipv4info`, `robtex`, `chinaz`, `dnsdb_api`, `netcraft`, `securitytrails_api`, `chinaz_api`, `dnsdumpster`, `passivedns_api`,  `ptrarchive`, `sitedossier`,`threatcrowd`）

      | 模块名称           | 是否需要代理 | 是否需要API | 其他说明                                           |
      | ------------------ | ------------ | ----------- | -------------------------------------------------- |
-      | binaryedge_api     | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
+      | binaryedge_api     | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
      | bufferover         | 否           | 否          |                                                    |
+      | cebaidu            | 否           | 否          |                                                    |
      | chinaz             | 否           | 否          |                                                    |
-      | chinaz_api         | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
-      | circl_api          | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
+      | chinaz_api         | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
+      | circl_api          | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
      | dnsdb              | 否           | 否          |                                                    |
-      | dnsdb_api          | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
+      | dnsdb_api          | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
      | dnsdumpster        | 否           | 否          |                                                    |
      | hackertarget       | 否           | 否          |                                                    |
-      | ipv4info           | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
+      | ip138              | 否           | 否          |                                                    |
+      | ipv4info           | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
      | netcraft           | 否           | 否          |                                                    |
-      | ptrarchive         | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
-      | riddler            | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
+      | passivedns_api     | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
+      | ptrarchive         | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
+      | riddler            | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
      | robtex             | 否           | 否          |                                                    |
-      | securitytrails_api | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
+      | securitytrails_api | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py) |
      | sitedossier        | 否           | 否          |                                                    |
      | threatcrowd        | 否           | 否          |                                                    |
+      | ximcx              | 否           | 否          |                                                    |
  5. 利用DNS查询收集子域（目前有1个模块：通过枚举常见的SRV记录并做查询来收集子域`srv`，该模块还有待添加和完善）

      | 模块名称 | 是否需要代理 | 是否需要API | 其他说明                      |
      | -------- | ------------ | ----------- | ----------------------------- |
      | srv      | 否           | 否          | 枚举域名常见的SRV记录发现子域 |
-  6. 利用威胁平台数据收集子域（目前有5个模块：`riskiq_api`，`threatbook_api`，`threatminer`，`virustotal`，`virustotal_api`该模块还有待添加和完善）
+  6. 利用威胁平台数据收集子域（目前有6个模块：`riskiq_api`，`threatbook_api`，`threatminer`，`virustotal`，`virustotal_api`该模块还有待添加和完善）

      | 模块名称       | 是否需要代理 | 是否需要API | 其他说明                                           |
-      | -------------- | ------------ | ----------- | -------------------------------------------------- |
-      | riskiq_api     | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
-      | threatbook_api | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
+      | -------------- | ------------ | ----------- | ------------------------------------------------- |
+      | alienvault     | 否           | 否          |                                                    |
+      | riskiq_api     | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py)         |
+      | threatbook_api | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py)         |
      | threatminer    | 否           | 否          |                                                    |
      | virustotal     | 否           | 否          |                                                    |
-      | virustotal_api | 否           | 是          | API使用和申请见[config.py](../oneforall/config.py) |
-  7. 利用搜索引擎发现子域（目前有15个模块：`ask`, `bing_api`, `fofa_api`, `shodan_api`, `yahoo`, `baidu`, `duckduckgo`, `google`, `so`, `yandex`, `bing`, `exalead`, `google_api`, `sogou`, `zoomeye_api`）
+      | virustotal_api | 否           | 是          | API使用和申请见[api.py](../oneforall/api.py)         |
+  7. 利用搜索引擎发现子域（目前有16个模块：`ask`, `bing_api`, `fofa_api`, `shodan_api`, `yahoo`, `baidu`, `duckduckgo`, `github`, `google`, `so`, `yandex`, `bing`, `exalead`, `google_api`, `sogou`, `zoomeye_api`）

     除特殊搜索引擎，通用的搜索引擎都支持自动排除搜索，全量搜索，递归搜索。

-     | 模块        | 是否需要代理           | 是否需要API | 其他说明                                           |
-     | ----------- | ---------------------- | ----------- | -------------------------------------------------- |
-     | ask         | 是                     | 否          |                                                    |
-     | baidu       | 否                     | 否          |                                                    |
-     | bing        | 否                     | 否          |                                                    |
-     | bing_api    | 否                     | 是          | API使用和申请见[config.py](../oneforall/config.py) |
-     | duckduckgo  | 是                     | 否          |                                                    |
-     | exalead     | 否，最好使用国外代理。 | 否          |                                                    |
-     | fofa_api    | 否                     | 是          | API使用和申请见[config.py](../oneforall/config.py) |
-     | google      | 是                     | 否          |                                                    |
-     | google_api  | 是                     | 是          | API使用和申请见[config.py](../oneforall/config.py) |
-     | shodan_api  | 否，最好使用国外代理。 | 是          | API使用和申请见[config.py](../oneforall/config.py) |
-     | so          | 否                     | 否          |                                                    |
-     | sogou       | 否                     | 否          |                                                    |
-     | yahoo       | 是                     | 否          |                                                    |
-     | yandex      | 是                     | 否          |                                                    |
-     | zoomeye_api | 否                     | 是          | API使用和申请见[config.py](../oneforall/config.py) |
+     | 模块         | 是否需要代理            | 是否需要API  | 其他说明                                                        |
+     | ----------- | ---------------------- | -----------| ----------------------------------------------------------- |
+     | ask         | 是                     | 否          |                                                             |
+     | baidu       | 否                     | 否          |                                                             |
+     | bing        | 否                     | 否          |                                                             |
+     | bing_api    | 否                     | 是          | API使用和申请见[api.py](../oneforall/api.py)                  |
+     | duckduckgo  | 是                     | 否          |                                                             |
+     | exalead     | 否，最好使用国外代理。    | 否          |                                                             |
+     | fofa_api    | 否                     | 是          | API使用和申请见[api.py](../oneforall/api.py)                  |
+     | gitee       | 否                     | 否          |                                                             |
+     | github      | 否                     | 否          | 在[api.py](../oneforall/api.py)设置Github邮件名和密码。        |
+     | google      | 是                     | 否          |                                                             |
+     | google_api  | 是                     | 是          | API使用和申请见[api.py](../oneforall/api.py)                  |
+     | shodan_api  | 否，最好使用国外代理。    | 是          | API使用和申请见[api.py](../oneforall/api.py)                  |
+     | so          | 否                     | 否          |                                                             |
+     | sogou       | 否                     | 否          |                                                             |
+     | yahoo       | 是                     | 否          |                                                             |
+     | yandex      | 是                     | 否          |                                                             |
+     | zoomeye_api | 否                     | 是          | API使用和申请见[api.py](../oneforall/api.py)                  |
@@ -0,0 +1,14 @@
+# OneForAll贡献者
+
+* **[Jing Ling](https://github.com/shmilylty)**
+  * 核心开发
+
+* **[Black Star](https://github.com/blackstar24)**，**[Echocipher](https://github.com/Echocipher)**
+  * 模块贡献
+
+* **[iceMatcha](https://github.com/iceMatcha)**，**[mikuKeeper](https://github.com/mikuKeeper)**
+  * 工具测试
+
+* **Anyone**
+  * 工具反馈
+
@@ -0,0 +1,84 @@
+# 安装依赖
+
+你可以通过pip3和pipenv两种方法安装OneForAll的依赖（熟悉哪种就用哪种）：
+
+## Windows系统
+
+注意：如果你的Python3安装在系统Program Files目录下，如：`C:\Program Files\Python38`，那么请以管理员身份运行命令提示符cmd执行以下命令！
+
+1. 使用pipenv安装依赖
+```bash
+cd OneForAll/
+python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
+pip3 install pipenv -i https://mirrors.aliyun.com/pypi/simple/
+pipenv install --python 3.8
+cd oneforall
+pipenv run python oneforall.py --help
+```
+
+2. 使用pip3安装依赖
+```bash
+cd OneForAll/
+python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
+pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
+cd oneforall/
+python oneforall.py --help
+```
+## Linux系统
+
+* **Ubuntu/Debian系统(包括kali)**
+安装git
+```bash
+sudo apt update
+sudo apt install git
+```
+
+克隆OneForAll项目
+```bash
+git clone https://gitee.com/shmilylty/OneForAll.git
+```
+
+安装python及开发依赖
+```bash
+sudo apt install python3.8 python3.8-dev python3-testresources
+```
+接下来你可以使用以下一种方式安装OneForAll的Python库依赖:
+1. 使用pipenv安装依赖
+```bash
+cd OneForAll/
+sudo python3.8 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
+sudo apt install pipenv
+sudo pipenv install --python 3.8
+cd oneforall
+pipenv run python3 oneforall.py --help
+```
+
+2. 使用pip3安装依赖
+```bash
+cd OneForAll/
+sudo apt install python3-pip
+sudo python3.8 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
+sudo pip3 install --ignore-installed -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
+cd oneforall/
+python3.8 oneforall.py --help
+```
+## Darwin系统
+
+1. 使用pipenv安装依赖
+```bash
+cd OneForAll/
+python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
+pip3 install pipenv -i https://mirrors.aliyun.com/pypi/simple/
+pipenv install --python 3.8
+cd oneforall
+pipenv run python3 oneforall.py --help
+```
+
+2. 使用pip3安装依赖
+```bash
+cd OneForAll/
+python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
+pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
+cd oneforall/
+python3 oneforall.py --help
+```
@@ -0,0 +1,9 @@
+# OneForAll后续开发计划
+
+## 下一步计划
+
+- [ ] 各模块支持优化和完善
+- [x] 子域监控（标记每次新发现的子域）
+- [ ] 子域收集爬虫实现（包括从JS等静态资源文件中收集子域）
+- [ ] 操作强大交互人性的前端界面实现（暂定：Element+Flask）
+
@@ -0,0 +1,40 @@
+# 常见问题与回答
+
+## 依赖问题
+
+**Q: 在安装依赖过程遇到编译某个依赖库失败，怎么解决？**
+
+A: 可以尝试以下方法:
+
+1. 到提供编译好的whl文件的第三方平台，找到对应库手动下载安装。第三方平台平台有： 
+ * [https://www.lfd.uci.edu/~gohlke/pythonlibs](https://www.lfd.uci.edu/~gohlke/pythonlibs)
+ * [https://pythonwheels.com/](https://pythonwheels.com/)
+
+选择好对应版本执行以下命令手动安装。举个例子，当编译 brotlipy 时失败时，找到[https://www.lfd.uci.edu/~gohlke/pythonlibs/#brotlipy](https://www.lfd.uci.edu/~gohlke/pythonlibs/#brotlipy)，由于我的系统是Windows 10 64位，使用的Python 3.8便下载`brotlipy‑0.7.0‑cp38‑cp38‑win_amd64.whl`（一般来说下载最新版本的），然后手动安装：
+
+```bash
+pip3 install brotlipy‑0.7.0‑cp38‑cp38‑win_amd64.whl
+```
+
+2. 到库的项目地址issues和wiki等找找有没有解决方法，如果没有就给他们提issues发邮件😜。
+
+## 使用问题
+
+**Q: 为什么运行OneForAll之后最终结果为空？**
+
+*A: 有几种可能性：第一可能目标域名没有子域。第二由于OneForAll默认会自动验证子域，在导出是只会有效子域，所以存在导出时没有有效子域的情况，你可以在运行OneForAll使用--valid=None指定导出所有发现的子域，你也可以使用--verify=False指定不验证子域的有效性。*
+
+**Q: 安装依赖时出现以下类似报错，怎么解决？**
+   Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
+
+*A: 安装依赖时尝试加上--ignore-installed参数*
+
+**Q:使用Excel打开结果的csv中有中文乱码，怎么解决？**
+
+*A:由于Excel打开文件需要带BOM头识别编码，而默认生成的结果csv文件是UTF-8编码不是UTF-8-BOM，所以会使用Excel打开会出现中文乱码*，有以下解决办法：
+
+1.在一开始运行OneForAll，设置format参数为xls或者xlsx格式。
+
+2.使用NotePad++类似工具转化编码格式为UTF-8-BOM。
+
+3.使用Excel软件导入结果csv文件中的数据。
@@ -0,0 +1,148 @@
+**🤔Help**
+
+The command line parameters only provide some common parameters. For more detailed parameter configuration, please see [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) if you think Some parameters are frequently used in the command interface or missing parameters. Feedback is welcome. For well-known reasons, if you want to use some of the wall's collection interface, please go to [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) to configure the proxy, some collection Modules need to provide APIs (most of which are freely available for registered accounts). If you need to use them, please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py) to configure the API. Information, if not used, please ignore the error message. (For detailed modules, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
+
+The OneForAll command line interface is based on [Fire](https://github.com/google/python-fire/). For more advanced usage of Fire, please refer to [using the Fire CLI](https://github.com/google/Python-fire/blob/master/docs/using-cli.md), if you have any doubts during the use, please feel free to give me feedback.
+
+[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py) is the main program entry, and oneforall.py can call [aiobrute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/aiobrute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py) and [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py) and other modules, in order to facilitate the sub-field blasting, aiobrute.py is isolated independently, in order to facilitate the subdomain takeover risk check independently takeover.py, in order to facilitate the database export independently dbexport.py, these modules can be run separately, and the parameters accepted are more abundant.
+
+❗ Note: When you encounter some problems or doubts during use, please use [Issues](https://github.com/shmilylty/OneForAll/issues) to search for answers. Also see [Q&A](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&A.md).
+
+1. **oneforall.py help**
+
+   ```bash
+   python oneforall.py --help
+   ```
+   ```bash
+   NAME
+       oneforall.py - OneForAll help Information
+   
+   SYNOPSIS
+       oneforall.py --target=TARGET <flags>
+   
+   DESCRIPTION
+       OneForAll is a powerful subdomain collection tool
+   
+       Example:
+           python3 oneforall.py version
+           python3 oneforall.py --target example.com run
+           python3 oneforall.py --target ./domains.txt run
+           python3 oneforall.py --target example.com --valid None run
+           python3 oneforall.py --target example.com --brute True run
+           python3 oneforall.py --target example.com --port small run
+           python3 oneforall.py --target example.com --format csv run
+           python3 oneforall.py --target example.com --dns False run
+           python3 oneforall.py --target example.com --req False run
+           python3 oneforall.py --target example.com --takeover False run
+           python3 oneforall.py --target example.com --show True run
+   
+       Note:
+           Parameter valid optional value 1, 0, none indicates that the export is 
+           valid, invalid, and all subdomains, respectively.
+           Parameter port have optional values 'default' 'small', 'large',
+           See config.py configuration for details.
+           Parameter format have optional values 'txt', 'rst', 'csv', 'tsv', 'json', 
+           'yaml', 'html', 'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'.
+           If the parameter path is None, the appropriate file is generated in the 
+           project result directory based on the format parameter and the domain 
+           name.
+           Parameter path default None uses the OneForAll result directory generation path
+   
+   ARGUMENTS
+       TARGET
+           Single domain name or file path for one domain name per line (required)
+   
+   FLAGS
+       --brute=BRUTE
+           Use blasting module (default False)
+       --dns=DNS
+           DNS resolve subdomain (default True)
+       --req=REQ
+           HTTP request subdomain (default True)
+       --port=PORT
+           Port range for request authentication (default 80 port)
+       --valid=VALID
+           Export validity of subdomains (default None)
+       --format=FORMAT
+           Export format (default xls)
+       --path=PATH
+           Export path (default None)
+       --takeover=TAKEOVER
+           Check subdomain takeover (default False)
+       --show=SHOW
+           Terminal display exported data (default False)
+   ```
+   
+2. **aiobrute.py help**
+
+   With regard to the handling of the universal parsing problem, first of all, OneForAll accesses a random subdomain to determine whether universal parsing is used, and if universal parsing is used, it is handled by the following judgment: 
+   - First, it is mainly compared with the pan-parsed IP set and TTL values, see [this article](http://sh3ll.me/archives/201704041222.txt).
+
+   - Second, the number of times to resolve to the same IP collection multiple times (the default is 10, which can be set to size in config.py).
+
+   - Third, considering the blasting efficiency, there is no HTTP response volume similarity comparison and response volume content judgment,  this function has not been implemented yet, and will be implemented if necessary.
+
+   After not rigorous testing, in the 16-core CPU, using 16 processes 64 coroutines, 100M network bandwidth environment, run two million dictionaries about 10 minutes to run, about 3333 subdomains per second.
+
+   ```bash
+   python aiobrute.py --help
+   ```
+
+   ```bash
+   NAME
+       aiobrute.py - OneForAll multi-process multi-correlation asynchronous subdomain blasting module
+   
+   SYNOPSIS
+       aiobrute.py --target=TARGET <flags>
+   
+   DESCRIPTION
+       Example：
+           python3 aiobrute.py --target example.com run
+           python3 aiobrute.py --target ./domains.txt run
+           python3 aiobrute.py --target example.com --process 4 --coroutine 64 run
+           python3 aiobrute.py --target example.com --wordlist subdomains.txt run
+           python3 aiobrute.py --target example.com --recursive True --depth 2 run
+           python3 aiobrute.py --target m.{fuzz}.a.bz --fuzz True --rule [a-z] run
+   
+       Note:
+           Parameter valid optional value 1, 0, none indicates that the export is 
+           valid, invalid, and all subdomains, respectively.
+           
+           Parameter format have optional values 'txt', 'rst', 'csv', 'tsv', 'json', 
+           'yaml', 'html', 'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'.
+           If the parameter path is None, the appropriate file is generated in the 
+           project result directory based on the format parameter and the domain 
+           name.
+   
+   ARGUMENTS
+       TARGET
+           Single domain name or file path for one domain name per line (required)
+   
+   FLAGS
+       --process=PROCESS
+           Number of processes blasted (default CPU core count)
+       --coroutine=COROUTINE
+           Number of coroutines per blasting process (default 1024)
+       --wordlist=WORDLIST
+           Specify the dictionary path used for blasting (config.py is used by default)
+       --recursive=RECURSIVE
+           Whether to use recursive blasting (default False)
+       --depth=DEPTH
+           Depth of recursive blasting (default 2)
+       --namelist=NAMELIST
+           Specifies the dictionary path used by recursive blasting (configured by default using config.py)
+       --fuzz=FUZZ
+           Whether to use the fuzz mode for blasting (default False, you must specify the fuzz regular rule)
+       --rule=RULE
+           Regular rules used by fuzz mode (configured by default using config.py)
+       --export=EXPORT
+           Whether to export the blast result (default True)
+       --valid=VALID
+           Export validity of subdomains (default None)
+       --format=FORMAT
+           Export format (default xls)
+       --path=PATH
+           Export path (default None)
+       --show=SHOW
+           Terminal display exported data (default False)
+   ```
@@ -1,6 +1,12 @@
 # 使用帮助

-oneforall.py是主程序入口，oneforall.py里有调用aiobrute.py和dbexport.py，为了方便进行子域爆破和数据库导出独立出了aiobrute.py和dbexport.py，这两个文件可以单独运行，并且所接受参数要更丰富一点。
+命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）
+
+OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实现，有关Fire更高级使用方法请参阅[使用Fire CLI](https://github.com/google/python-fire/blob/master/docs/using-cli.md)。
+
+[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py)是主程序入口，oneforall.py可以调用[aiobrute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/aiobrute.py)，[takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py)及[dbexport.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py)等模块，为了方便进行子域爆破独立出了aiobrute.py，为了方便进行子域接管风险检查独立出了takerover.py，为了方便数据库导出独立出了dbexport.py，这些模块都可以单独运行，并且所接受参数要更丰富一点。
+
+❗注意：当你在使用过程中遇到一些问题或者疑惑时，请先到[Issues](https://github.com/shmilylty/OneForAll/issues)里使用搜索找找答案，还可以参阅[常见问题与回答](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&A.md)。

 1. oneforall.py使用帮助

@@ -1,337 +0,0 @@
-#!/usr/bin/python3
-# coding=utf-8
-
-"""
-OneForAll多进程多协程异步子域爆破模块
-
-:copyright: Copyright (c) 2019, Jing Ling. All rights reserved.
-:license: GNU General Public License v3.0, see LICENSE for more details.
-"""
-
-import asyncio
-import queue
-import secrets
-import signal
-import time
-
-import aiomultiprocess as aiomp
-import exrex
-import fire
-import tqdm
-
-import config
-import dbexport
-from common import resolve, utils
-from common.module import Module
-from common.database import Database
-from config import logger
-
-
-def init_worker():
-    signal.signal(signal.SIGINT, signal.SIG_IGN)
-
-
-def detect_wildcard(domain):
-    """
-    探测域名是否使用泛解析
-
-    :param str domain: 域名
-    :return: 如果没有使用泛解析返回False 反之返回泛解析的IP集合和ttl整型值
-    """
-    logger.log('INFOR', f'正在探测{domain}是否使用泛解析')
-    token = secrets.token_hex(16)
-    random_subdomain = f'{token}.{domain}'
-    try:
-        answers = resolve.dns_query_a(random_subdomain)
-    # 如果查询随机域名A记录出错 说明不存在随机子域的A记录 即没有开启泛解析
-    except Exception as e:
-        logger.log('DEBUG', e)
-        logger.log('INFOR', f'{domain}没有使用泛解析')
-        return False, None, None
-    ttl = answers.ttl
-    ips = {item.address for item in answers}
-    logger.log('ALERT', f'{domain}使用了泛解析')
-    logger.log('ALERT', f'{random_subdomain} 解析到IP: {ips} TTL: {ttl}')
-    return True, ips, ttl
-
-
-def wildcard_by_compare(ips, ttl, wildcard_ips, wildcard_ttl):
-    """
-    通过与泛解析返回的IP集合和判TTL值进行对比判断发现的子域是否是泛解析子域
-
-    :param set ips: 子域A记录查询出的IP集合
-    :param int ttl: 子域A记录查询出的TTL整型值
-    :param set wildcard_ips: 泛解析的IP集合
-    :param int wildcard_ttl: 泛解析的TTL整型值
-    :return: 判断结果
-    :rtype bool
-    """
-    # 参考：http://sh3ll.me/archives/201704041222.txt
-    if not ips.issubset(wildcard_ips):
-        return False
-    if ttl != wildcard_ttl and ttl % 60 == 0 and wildcard_ttl % 60 == 0:
-        return False
-    return True
-
-
-def wildcard_by_times(ips, ips_times):
-    """
-    对ips出现次数进行判断泛解析
-
-    :param set ips: 发现子域的IP集合
-    :param ips_times: 子域IP集合出现次数统计字典
-    :return: 判断结果
-    """
-    times = ips_times.get(str(ips))
-    if times > config.ips_appear_maximum:
-        return True
-    return False
-
-
-def gen_fuzz_domains(domain, rule):
-    domains = set()
-    if '{fuzz}' not in domain:
-        logger.log('FATAL', f'没有指定fuzz位置')
-        return domains
-    if not rule:
-        logger.log('FATAL', f'没有指定fuzz规则')
-        return domains
-    fuzz_count = exrex.count(rule)
-    if fuzz_count > 2000000:
-        logger.log('FATAL', f'fuzz规则范围太大：{fuzz_count} > 2000000')
-        return domains
-    logger.log('INFOR', f'fuzz字典大小：{fuzz_count}')
-    for i in range(3):
-        random_domain = domain.replace('{fuzz}', exrex.getone(rule))
-        logger.log('ALERT', f'请注意检查随机生成的{random_domain}是否正确')
-    logger.log('ALERT', f'你有10秒检查时间退出使用`CTRL+C`')
-    try:
-        time.sleep(5)
-    except KeyboardInterrupt:
-        logger.log('INFOR', '爆破终止')
-        exit(0)
-    parts = domain.split('{fuzz}')
-    for fuzz in exrex.generate(rule):
-        fuzz_domain = parts[0] + fuzz + parts[1]
-        domains.add(fuzz_domain)
-    return domains
-
-
-def gen_brute_domains(domain, path):
-    domains = set()
-    with open(path) as file:
-        for line in file:
-            brute_domain = line.strip() + '.' + domain
-            domains.add(brute_domain)
-    logger.log('INFOR', f'爆破字典大小：{len(domains)}')
-    return domains
-
-
-class AIOBrute(Module):
-    """
-    OneForAll多进程多协程异步子域爆破模块
-
-    Example：
-        python3 aiobrute.py --target subdomain.com run
-        python3 aiobrute.py --target ./subdomains.txt run
-        python3 aiobrute.py --target example.com --process 4 --coroutine 64 run
-        python3 aiobrute.py --target example.com --wordlist subnames.txt run
-        python3 aiobrute.py --target example.com --recursive True --depth 2 run
-        python3 aiobrute.py --target m.{fuzz}.a.bz --fuzz True --rule [a-z] run
-
-    Note:
-        参数segment的设置受CPU性能，网络带宽，运营商限制等限制，默认500个子域为任务组，
-        当你的环境不受以上因素影响，当前爆破速度较慢，那么强烈建议根据字典大小调整大小：
-        十万字典建议设置为5000，百万字典设置为50000
-        参数valid可选值1，0，None，分别表示导出有效，无效，全部子域
-        参数format可选格式有'txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
-                          'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
-        参数path为None会根据format参数和域名名称在项目结果目录生成相应文件
-
-    :param str target:       单个域名或者每行一个域名的文件路径
-    :param int process:      爆破的进程数(默认CPU核心数)
-    :param int coroutine:    每个爆破进程下的协程数(默认64)
-    :param str wordlist:     指定爆破所使用的字典路径(默认使用config.py配置)
-    :param int segment:      爆破任务分割(默认500)
-    :param bool recursive:   是否使用递归爆破(默认False)
-    :param int depth:        递归爆破的深度(默认2)
-    :param str namelist:     指定递归爆破所使用的字典路径(默认使用config.py配置)
-    :param bool fuzz:        是否使用fuzz模式进行爆破(默认False，开启须指定fuzz正则规则)
-    :param str rule:         fuzz模式使用的正则规则(默认使用config.py配置)
-    :param bool export:      是否导出爆破结果(默认True)
-    :param int valid:        导出子域的有效性(默认None)
-    :param str format:       导出格式(默认xls)
-    :param str path:         导出路径(默认None)
-    :param bool show:        终端显示导出数据(默认False)
-    """
-
-    def __init__(self, target, process=None, coroutine=64, wordlist=None,
-                 segment=500, recursive=False, depth=2, namelist=None,
-                 fuzz=False, rule=None, export=True, valid=None, format='xls',
-                 path=None, show=False):
-        Module.__init__(self)
-        self.domains = set()
-        self.domain = str()
-        self.module = 'Brute'
-        self.source = 'AIOBrute'
-        self.target = target
-        self.process = process or config.brute_process_num
-        self.coroutine = coroutine or config.brute_coroutine_num
-        self.wordlist = wordlist or config.brute_wordlist_path
-        self.segment = segment or config.brute_task_segment
-        self.recursive_brute = recursive or config.enable_recursive_brute
-        self.recursive_depth = depth or config.brute_recursive_depth
-        self.recursive_namelist = namelist or config.recursive_namelist_path
-        self.fuzz = fuzz or config.enable_fuzz
-        self.rule = rule or config.fuzz_rule
-        self.export = export
-        self.valid = valid
-        self.format = format
-        self.path = path
-        self.show = show
-        self.nameservers = config.resolver_nameservers
-        self.ips_times = dict()  # IP集合出现次数
-        self.enable_wildcard = False  # 当前域名是否使用泛解析
-        self.wildcard_ips = set()  # 泛解析IP集合
-        self.wildcard_ttl = int()  # 泛解析TTL整型值
-
-    def gen_tasks(self, domain):
-        # 如果domain不是self.subdomain，而是self.domain的子域 生成递归爆破字典
-        if self.domain != domain:
-            logger.log('INFOR', f'使用{self.recursive_namelist}字典')
-            domains = gen_brute_domains(domain, self.recursive_namelist)
-        elif self.fuzz and self.rule:  # 开启fuzz模式并指定了fuzz正则规则
-            logger.log('INFOR', f'正在生成{domain}的fuzz字典')
-            domains = gen_fuzz_domains(domain, self.rule)
-        else:
-            logger.log('INFOR', f'使用{self.wordlist}字典')
-            domains = gen_brute_domains(domain, self.wordlist)
-        domains = list(domains)
-        return utils.split_list(domains, self.segment)  # 分割任务组
-
-    def deal_results(self, results):
-        for result in results:
-            if isinstance(result, Exception):
-                # logger.log('DEBUG', f'爆破{subdomain}时出错 {str(answers)}')
-                continue
-            if isinstance(result, tuple):
-                subdomain, answers = result
-                if not answers:
-                    continue
-                ips = {record.host for record in answers}
-                # 取值 如果是首次出现的IP集合 出现次数先赋值0
-                value = self.ips_times.setdefault(str(ips), 0)
-                self.ips_times[str(ips)] = value + 1
-                ttl = answers[0].ttl
-                if self.enable_wildcard:
-                    if wildcard_by_compare(ips,
-                                           ttl,
-                                           self.wildcard_ips,
-                                           self.wildcard_ttl):
-                        continue
-                if wildcard_by_times(ips, self.ips_times):
-                    continue
-                logger.log('INFOR', f'发现{self.domain}的子域: {subdomain} '
-                                    f'解析IP: {ips} TTL: {ttl}')
-                self.subdomains.add(subdomain)
-                self.records[subdomain] = str(ips)
-
-    async def main(self, domain, rx_queue):
-        if not self.fuzz:  # fuzz模式不探测域名是否使用泛解析
-            self.enable_wildcard, self.wildcard_ips, self.wildcard_ttl \
-                = detect_wildcard(domain)
-        tasks = self.gen_tasks(domain)
-        logger.log('INFOR', f'正在爆破{domain}的域名')
-        for task in tqdm.tqdm(tasks,
-                              desc='Progress',
-                              smoothing=1.0,
-                              ncols=True):
-            async with aiomp.Pool(processes=self.process,
-                                  initializer=init_worker,
-                                  childconcurrency=self.coroutine) as pool:
-                try:
-                    results = await pool.map(resolve.aiodns_query_a, task)
-                except KeyboardInterrupt:
-                    logger.log('ALERT', '爆破终止正在退出')
-                    pool.terminate()  # 关闭pool，结束工作进程，不在处理未完成的任务。
-                    self.save_json()
-                    self.gen_result()
-                    rx_queue.put(self.results)
-                    return
-                self.deal_results(results)
-                self.save_json()
-                self.gen_result()
-                rx_queue.put(self.results)
-
-    def run(self, rx_queue=None):
-        self.domains = utils.get_domains(self.target)
-        while self.domains:
-            self.domain = self.domains.pop()
-            start = time.time()
-            db = Database()
-            db.create_table(self.domain)
-            if not rx_queue:
-                rx_queue = queue.Queue()
-            logger.log('INFOR', f'开始执行{self.source}模块爆破域名{self.domain}')
-            logger.log('INFOR', f'使用{self.process}进程乘{self.coroutine}协程')
-            # fuzz模式不使用递归爆破
-            if self.recursive_brute and not self.fuzz:
-                logger.log('INFOR', f'开始递归爆破{self.domain}的第1层子域')
-            loop = asyncio.get_event_loop()
-            asyncio.set_event_loop(loop)
-            loop.run_until_complete(self.main(self.domain, rx_queue))
-
-            # 递归爆破下一层的子域
-            # fuzz模式不使用递归爆破
-            if self.recursive_brute and not self.fuzz:
-                for layer_num in range(1, self.recursive_depth):
-                    # 之前已经做过1层子域爆破 当前实际递归层数是layer+1
-                    logger.log('INFOR', f'开始递归爆破{self.domain}的'
-                                        f'第{layer_num + 1}层子域')
-                    for subdomain in self.subdomains.copy():
-                        # 进行下一层子域爆破的限制条件
-                        if subdomain.count('.') - self.domain.count('.') \
-                                == layer_num:
-                            loop.run_until_complete(self.main(subdomain,
-                                                              rx_queue))
-            # 队列不空就一直取数据存数据库
-            while not rx_queue.empty():
-                source, results = rx_queue.get()
-                # 将结果存入数据库中
-                db.save_db(self.domain, results, source)
-
-            end = time.time()
-            self.elapsed = round(end - start, 1)
-            logger.log('INFOR', f'结束执行{self.source}模块爆破域名{self.domain}')
-            length = len(self.subdomains)
-            logger.log('INFOR', f'{self.source}模块耗时{self.elapsed}秒'
-                                f'发现{self.domain}的域名{length}个')
-            logger.log('DEBUG', f'{self.source}模块发现{self.domain}的域名:\n'
-                                f'{self.subdomains}')
-            # 数据库导出
-            if self.export:
-                if not self.path:
-                    name = f'{self.domain}_brute.{self.format}'
-                    self.path = config.result_save_path.joinpath(name)
-                dbexport.export(self.domain,
-                                valid=self.valid,
-                                dpath=self.path,
-                                format=self.format,
-                                show=self.show)
-
-
-def do(domain, result):  # 统一入口名字 方便多线程调用
-    """
-    类统一调用入口
-
-    :param str domain: 域名
-    :param result: 结果集队列
-    """
-    brute = AIOBrute(domain)
-    brute.run(result)
-
-
-if __name__ == '__main__':
-    fire.Fire(AIOBrute)
-    # result_queue = queue.Queue()
-    # do('example.com', result_queue)
@@ -0,0 +1,71 @@
+# 模块API配置
+# Censys可以免费注册获取API：https://censys.io/api
+censys_api_id = ''
+censys_api_secret = ''
+
+# Binaryedge可以免费注册获取API：https://app.binaryedge.io/account/api
+# 免费的API有效期只有1个月，到期之后可以再次生成，每月可以查询250次。
+binaryedge_api = ''
+
+# Chinaz可以免费注册获取API：http://api.chinaz.com/ApiDetails/Alexa
+chinaz_api = ''
+
+# Bing可以免费注册获取API：https://azure.microsoft.com/zh-cn/services/
+# cognitive-services/bing-web-search-api/#web-json
+bing_api_id = ''
+bing_api_key = ''
+
+# SecurityTrails可以免费注册获取API：https://securitytrails.com/corp/api
+securitytrails_api = ''
+
+# https://fofa.so/api
+fofa_api_email = ''  # fofa用户邮箱
+fofa_api_key = ''  # fofa用户key
+
+# Google可以免费注册获取API:
+# https://developers.google.com/custom-search/v1/overview
+# 免费的API只能查询前100条结果
+google_api_key = ''  # Google API搜索key
+google_api_cx = ''  # Google API搜索cx
+
+# https://api.passivetotal.org/api/docs/
+riskiq_api_username = ''
+riskiq_api_key = ''
+
+# Shodan可以免费注册获取API: https://account.shodan.io/register
+# 免费的API限速1秒查询1次
+shodan_api_key = ''
+# ThreatBook API 查询子域名需要收费 https://x.threatbook.cn/nodev4/vb4/myAPI
+threatbook_api_key = ''
+
+# VirusTotal可以免费注册获取API: https://developers.virustotal.com/reference
+virustotal_api_key = ''
+
+# https://www.zoomeye.org/doc?channel=api
+zoomeye_api_usermail = ''
+zoomeye_api_password = ''
+
+# Spyse可以免费注册获取API: https://spyse.com/
+spyse_api_token = ''
+
+# https://www.circl.lu/services/passive-dns/
+circl_api_username = ''
+circl_api_password = ''
+
+# https://www.dnsdb.info/
+dnsdb_api_key = ''
+
+# ipv4info可以免费注册获取API: http://ipv4info.com/tools/api/
+# 免费的API有效期只有2天，到期之后可以再次生成，每天可以查询50次。
+ipv4info_api_key = ''
+
+# https://github.com/360netlab/flint
+# passivedns_api_addr默认空使用http://api.passivedns.cn
+# passivedns_api_token可为空
+passivedns_api_addr = ''
+passivedns_api_token = ''
+
+# Github Token可以访问https://github.com/settings/tokens生成,user为Github用户名
+# 用于子域接管和子域收集
+github_api_user = ''
+github_api_token = ''
@@ -0,0 +1,636 @@
+#!/usr/bin/python3
+# coding=utf-8
+
+"""
+OneForAll子域爆破模块
+
+:copyright: Copyright (c) 2019, Jing Ling. All rights reserved.
+:license: GNU General Public License v3.0, see LICENSE for more details.
+"""
+import gc
+import json
+import time
+import random
+import secrets
+
+import exrex
+import fire
+import tenacity
+from tenacity import RetryError
+from dns.exception import Timeout
+from dns.resolver import NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers
+
+import config
+import dbexport
+from common import utils
+from common.module import Module
+from config import logger
+
+
+@tenacity.retry(reraise=True, stop=tenacity.stop_after_attempt(3))
+def do_query_a(domain, resolver):
+    try:
+        answer = resolver.query(domain, 'A')
+    # 如果查询随机域名A记录时抛出Timeout异常则重新探测
+    except Timeout as e:
+        logger.log('ALERT', f'探测超时重新探测中')
+        logger.log('DEBUG', e.args)
+        raise tenacity.TryAgain
+    # 如果查询随机域名A记录时抛出NXDOMAIN异常
+    # 则说明不存在随机子域的A记录 即没有开启泛解析
+    except (NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers) as e:
+        logger.log('DEBUG', e.args)
+        logger.log('INFOR', f'{domain}没有使用泛解析')
+        return False
+    except Exception as e:
+        logger.log('ALERT', f'探测{domain}是否使用泛解析出错')
+        logger.log('FATAL', e.args)
+        exit(1)
+    else:
+        if answer.rrset is None:
+            logger.log('ALERT', f'结果无记录重新探测中')
+            raise tenacity.TryAgain
+        ttl = answer.ttl
+        name = answer.name
+        ips = {item.address for item in answer}
+        logger.log('ALERT', f'{domain}使用了泛解析')
+        logger.log('ALERT', f'{domain} 解析到域名: {name} '
+                            f'IP: {ips} TTL: {ttl}')
+        return True
+
+
+def detect_wildcard(domain, authoritative_ns):
+    """
+    探测域名是否使用泛解析
+
+    :param str domain: 域名
+    :param list authoritative_ns: 权威DNS
+    :return: 是否使用泛解析
+    """
+    logger.log('INFOR', f'正在探测{domain}是否使用泛解析')
+    token = secrets.token_hex(4)
+    random_subdomain = f'{token}.{domain}'
+    resolver = utils.dns_resolver()
+    resolver.nameservers = authoritative_ns
+    resolver.rotate = True
+    resolver.cache = None
+    try:
+        wildcard = do_query_a(random_subdomain, resolver)
+    except Timeout as e:
+        logger.log('DEBUG', e.args)
+        logger.log('ALERT', f'多次探测超时暂且认为{domain}没有使用泛解析')
+        return False
+    else:
+        return wildcard
+
+
+def gen_fuzz_subdomains(expression, rule):
+    """
+    生成基于fuzz模式的爆破子域
+
+    :param str expression: 子域域名生成表达式
+    :param str rule: 生成子域所需的正则规则
+    :return: 用于爆破的子域
+    """
+    subdomains = list()
+    fuzz_count = exrex.count(rule)
+    if fuzz_count > 10000000:
+        logger.log('ALERT', f'请注意该规则生成的字典太大：{fuzz_count} > 10000000')
+    logger.log('DEBUG', f'fuzz模式下生成的字典大小：{fuzz_count}')
+    for fuzz_string in exrex.generate(rule):
+        fuzz_string = fuzz_string.lower()
+        if not fuzz_string.isalnum():
+            continue
+        fuzz_domain = expression.replace('*', fuzz_string)
+        subdomains.append(fuzz_domain)
+    random_domain = random.choice(subdomains)
+    logger.log('ALERT', f'请注意检查基于fuzz模式生成的{random_domain}是否正确')
+    return subdomains
+
+
+def gen_word_subdomains(expression, path):
+    """
+    生成基于word模式的爆破子域
+
+    :param str expression: 子域域名生成表达式
+    :param str path: 字典路径
+    :return: 用于爆破的子域
+    """
+    subdomains = list()
+    with open(path, encoding='utf-8', errors='ignore') as fd:
+        for line in fd:
+            word = line.strip().lower()
+            if not word.isalnum():
+                continue
+            if word.endswith('.'):
+                word = word[:-1]
+            subdomain = expression.replace('*', word)
+            subdomains.append(subdomain)
+    random_domain = random.choice(subdomains)
+    logger.log('DEBUG', f'fuzz模式下生成的字典大小：{len(subdomains)}')
+    logger.log('ALERT', f'请注意检查基于word模式生成的{random_domain}是否正确')
+    return subdomains
+
+
+def query_domain_ns_a(ns_list):
+    logger.log('INFOR', f'正在查询权威DNS名称服务器{ns_list}的A记录')
+    if not isinstance(ns_list, list):
+        return list()
+    ns_ip_list = []
+    resolver = utils.dns_resolver()
+    for ns in ns_list:
+        try:
+            answer = resolver.query(ns, 'A')
+        except Exception as e:
+            logger.log('ERROR', e.args)
+            logger.log('ERROR', f'查询权威DNS名称服务器{ns}的A记录出错')
+            continue
+        if answer:
+            for item in answer:
+                ns_ip_list.append(item.address)
+    logger.log('INFOR', f'权威DNS名称服务器对应A记录 {ns_ip_list}')
+    return ns_ip_list
+
+
+def query_domain_ns(domain):
+    logger.log('INFOR', f'正在查询{domain}的NS记录')
+    domain = utils.get_maindomain(domain)
+    resolver = utils.dns_resolver()
+    try:
+        answer = resolver.query(domain, 'NS')
+    except Exception as e:
+        logger.log('ERROR', e.args)
+        logger.log('ERROR', f'查询{domain}的NS记录出错')
+        return list()
+    ns = [item.to_text() for item in answer]
+    logger.log('INFOR', f'{domain}的权威DNS名称服务器 {ns}')
+    return ns
+
+
+@tenacity.retry(reraise=True, stop=tenacity.stop_after_attempt(2))
+def get_wildcard_record(domain, resolver):
+    logger.log('INFOR', f'查询{domain}在权威DNS名称服务器的泛解析记录')
+    try:
+        answer = resolver.query(domain, 'A')
+    # 如果查询随机域名A记录时抛出Timeout异常则重新查询
+    except Timeout as e:
+        logger.log('ALERT', f'查询超时重新查询中')
+        logger.log('DEBUG', e.args)
+        raise tenacity.TryAgain
+    except (NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers) as e:
+        logger.log('DEBUG', e.args)
+        logger.log('INFOR', f'{domain}在权威DNS名称服务器上没有A记录')
+        return None, None
+    except Exception as e:
+        logger.log('ERROR', e.args)
+        logger.log('ERROR', f'查询{domain}在权威DNS名称服务器泛解析记录出错')
+        exit(1)
+    else:
+        if answer.rrset is None:
+            logger.log('DEBUG', f'查询结果无记录')
+            return None, None
+        name = answer.name
+        ip = {item.address for item in answer}
+        ttl = answer.ttl
+        logger.log('INFOR', f'{domain} 在权威DNS上解析到域名: {name} '
+                            f'IP: {ip} TTL: {ttl}')
+        return ip, ttl
+
+
+def collect_wildcard_record(domain, authoritative_ns):
+    logger.log('INFOR', f'正在收集{domain}的泛解析记录')
+    if not authoritative_ns:
+        return list(), int()
+    resolver = utils.dns_resolver()
+    resolver.nameservers = authoritative_ns
+    resolver.rotate = True
+    resolver.cache = None
+    ips = set()
+    ips_stat = dict()
+    while True:
+        token = secrets.token_hex(4)
+        random_subdomain = f'{token}.{domain}'
+        try:
+            ip, ttl = get_wildcard_record(random_subdomain, resolver)
+        except Timeout as e:
+            logger.log('DEBUG', e.args)
+            logger.log('ALERT', f'多次查询超时将尝试查询新的随机子域')
+            continue
+        if ip is None:
+            continue
+        ips = ips.union(ip)
+        # 统计每个泛解析IP出现次数
+        for addr in ip:
+            count = ips_stat.setdefault(addr, 0)
+            ips_stat[addr] = count + 1
+        # 筛选出出现次数2次以上的IP地址
+        addrs = list()
+        for addr, times in ips_stat.items():
+            if times >= 2:
+                addrs.append(addr)
+        # 大部分的IP地址出现次数大于2次停止收集泛解析IP记录
+        if len(addrs) / len(ips) >= 0.8:
+            break
+    logger.log('DEBUG', f'收集到{domain}的泛解析记录\n{ips}\n{ttl}')
+    return ips, ttl
+
+
+def get_nameservers_path(enable_wildcard, ns_ip_list):
+    path = config.brute_nameservers_path
+    if not enable_wildcard:
+        return path
+    if not ns_ip_list:
+        return path
+    path = config.authoritative_dns_path
+    ns_data = '\n'.join(ns_ip_list)
+    utils.save_data(path, ns_data)
+    return path
+
+
+def check_dict():
+    if not config.enable_check_dict:
+        return
+    sec = config.check_time
+    logger.log('ALERT', f'你有{sec}秒时间检查爆破配置是否正确')
+    logger.log('ALERT', f'退出爆破请使用`Ctrl+C`')
+    try:
+        time.sleep(sec)
+    except KeyboardInterrupt:
+        logger.log('INFOR', '爆破配置有误退出爆破')
+        exit(0)
+
+
+def gen_records(items, records, subdomains, ip_times, wc_ips, wc_ttl):
+    qname = items.get('name')[:-1]  # 去出最右边的`.`点号
+    reason = items.get('status')
+    resolver = items.get('resolver')
+    data = items.get('data')
+    answers = data.get('answers')
+    record = dict()
+    cname = list()
+    ips = list()
+    public = list()
+    times = list()
+    ttls = list()
+    is_valid_flags = list()
+    have_a_record = False
+    for answer in answers:
+        if answer.get('type') != 'A':
+            logger.log('TRACE', f'查询{qname}返回的应答没有A记录\n{answer}')
+            continue
+        logger.log('TRACE', f'查询{qname}返回的应答具有A记录\n{answer}')
+        have_a_record = True
+        ttl = answer.get('ttl')
+        ttls.append(ttl)
+        cname.append(answer.get('name')[:-1])  # 去出最右边的`.`点号
+        ip = answer.get('data')
+        ips.append(ip)
+        public.append(utils.ip_is_public(ip))
+        num = ip_times.get(ip)
+        times.append(num)
+        isvalid, reason = is_valid_subdomain(ip, ttl, num, wc_ips, wc_ttl)
+        logger.log('TRACE', f'{ip}是否有效:{isvalid} 原因:{reason}')
+        is_valid_flags.append(isvalid)
+    if not have_a_record:
+        logger.log('TRACE', f'查询{qname}返回的所有应答都中没有A记录{answers}')
+    # 为了优化内存 只添加有A记录且通过判断的子域到记录中
+    if have_a_record and all(is_valid_flags):
+        record['resolve'] = 1
+        record['reason'] = reason
+        record['ttl'] = ttls
+        record['cname'] = cname
+        record['content'] = ips
+        record['public'] = public
+        record['times'] = times
+        record['resolver'] = resolver
+        records[qname] = record
+        subdomains.append(qname)
+    return records, subdomains
+
+
+def stat_ip_times(result_path):
+    logger.log('INFOR', f'正在统计IP次数')
+    times = dict()
+    with open(result_path) as fd:
+        for line in fd:
+            line = line.strip()
+            try:
+                items = json.loads(line)
+            except Exception as e:
+                logger.log('ERROR', e.args)
+                logger.log('ERROR', f'解析行{line}出错跳过解析该行')
+                continue
+            status = items.get('status')
+            if status != 'NOERROR':
+                continue
+            data = items.get('data')
+            if 'answers' not in data:
+                continue
+            answers = data.get('answers')
+            for answer in answers:
+                if answer.get('type') == 'A':
+                    ip = answer.get('data')
+                    # 取值 如果是首次出现的IP集合 出现次数先赋值0
+                    value = times.setdefault(ip, 0)
+                    times[ip] = value + 1
+    return times
+
+
+def deal_output(output_path, ip_times, wildcard_ips, wildcard_ttl):
+    logger.log('INFOR', f'正在处理解析结果')
+    records = dict()  # 用来记录所有域名解析数据
+    subdomains = list()  # 用来保存所有通过有效性检查的子域
+    with open(output_path) as fd:
+        for line in fd:
+            line = line.strip()
+            try:
+                items = json.loads(line)
+            except Exception as e:
+                logger.log('ERROR', e.args)
+                logger.log('ERROR', f'解析行{line}出错跳过解析该行')
+                continue
+            qname = items.get('name')[:-1]  # 去出最右边的`.`点号
+            status = items.get('status')
+            if status != 'NOERROR':
+                logger.log('TRACE', f'处理{line}时发现{qname}查询结果状态{status}')
+                continue
+            data = items.get('data')
+            if 'answers' not in data:
+                logger.log('TRACE', f'处理{line}时发现{qname}返回的结果无应答')
+                continue
+            records, subdomains = gen_records(items, records, subdomains,
+                                              ip_times, wildcard_ips,
+                                              wildcard_ttl)
+    return records, subdomains
+
+
+def check_by_compare(ip, ttl, wc_ips, wc_ttl):
+    """
+    通过与泛解析返回的IP集合和返回的TTL值进行对比判断发现的子域是否是泛解析子域
+
+    :param set ip: 子域A记录查询出的IP
+    :param int ttl: 子域A记录查询出的TTL
+    :param set wc_ips: 泛解析的IP集合
+    :param int wc_ttl: 泛解析的TTL
+    :return: 判断结果
+    """
+    # 参考：http://sh3ll.me/archives/201704041222.txt
+    if ip not in wc_ips:
+        return False  # 子域IP不在泛解析IP集合则不是泛解析
+    if ttl != wc_ttl and ttl % 60 == 0 and wc_ttl % 60 == 0:
+        return False
+    return True
+
+
+def check_ip_times(times):
+    """
+    根据ip出现次数判断是否为泛解析
+
+    :param times: 子域IP出现次数
+    :return: 判断结果
+    """
+    if times > config.ip_appear_maximum:
+        return True
+    return False
+
+
+def is_valid_subdomain(ip, ttl, times, wc_ips, wc_ttl):
+    ip_blacklist = config.brute_ip_blacklist
+    if ip in ip_blacklist:  # 解析ip在黑名单ip则为非法子域
+        return 0, 'IP blacklist'
+    if all([wc_ips, wc_ttl]):  # 有泛解析记录才进行对比
+        if check_by_compare(ip, ttl, wc_ips, wc_ttl):
+            return 0, 'IP wildcard'
+    if check_ip_times(times):
+        return 0, 'IP exceeded'
+    return 1, 'OK'
+
+
+def save_brute_dict(dict_path, dict_set):
+    dict_data = '\n'.join(dict_set)
+    if not utils.save_data(dict_path, dict_data):
+        logger.log('FATAL', '保存生成的字典出错')
+        exit(1)
+
+
+def delete_file(dict_path, output_path):
+    if config.delete_generated_dict:
+        dict_path.unlink()
+    if config.delete_massdns_result:
+        output_path.unlink()
+
+
+class Brute(Module):
+    """
+    OneForAll子域爆破模块
+
+    Example：
+        brute.py --target domain.com --word True run
+        brute.py --target ./domains.txt --word True run
+        brute.py --target domain.com --word True --process 1 run
+        brute.py --target domain.com --word True --wordlist subnames.txt run
+        brute.py --target domain.com --word True --recursive True --depth 2 run
+        brute.py --target d.com --fuzz True --place m.*.d.com --rule '[a-z]' run
+
+    Note:
+        参数alive可选值True，False分别表示导出存活，全部子域结果
+        参数format可选格式有'txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
+                          'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
+        参数path默认None使用OneForAll结果目录自动生成路径
+
+    :param str target:       单个域名或者每行一个域名的文件路径
+    :param int process:      爆破进程数(默认1)
+    :param int concurrent:   并发爆破数量(默认10000)
+    :param bool word:        是否使用word模式进行爆破(默认False)
+    :param str wordlist:     word模式爆破使用的字典路径(默认使用config.py配置)
+    :param bool recursive:   是否使用递归进行爆破(默认False)
+    :param int depth:        递归爆破的深度(默认2)
+    :param str nextlist:     递归爆破所使用的字典路径(默认使用config.py配置)
+    :param bool fuzz:        是否使用fuzz模式进行爆破(默认False)
+    :param str place:        指定爆破位置(开启fuzz模式时必需指定此参数)
+    :param str rule:         指定fuzz模式爆破使用的正则规则(开启fuzz模式时必需指定此参数)
+    :param bool export:      是否导出爆破结果(默认True)
+    :param bool alive:       只导出存活的子域结果(默认True)
+    :param str format:       结果导出格式(默认csv)
+    :param str path:         结果导出路径(默认None)
+    """
+
+    def __init__(self, target, process=None, concurrent=None, word=False,
+                 wordlist=None, recursive=False, depth=None, nextlist=None,
+                 fuzz=False, place=None, rule=None, export=True, alive=True,
+                 format='csv', path=None):
+        Module.__init__(self)
+        self.module = 'Brute'
+        self.source = 'Brute'
+        self.target = target
+        self.process_num = process or utils.get_process_num()
+        self.concurrent_num = concurrent or config.brute_concurrent_num
+        self.word = word
+        self.wordlist = wordlist or config.brute_wordlist_path
+        self.recursive_brute = recursive or config.enable_recursive_brute
+        self.recursive_depth = depth or config.brute_recursive_depth
+        self.recursive_nextlist = nextlist or config.recursive_nextlist_path
+        self.fuzz = fuzz or config.enable_fuzz
+        self.place = place or config.fuzz_place
+        self.rule = rule or config.fuzz_rule
+        self.export = export
+        self.alive = alive
+        self.format = format
+        self.path = path
+        self.bulk = False  # 是否是批量爆破场景
+        self.domains = list()  # 待爆破的所有域名集合
+        self.domain = str()  # 当前正在进行爆破的域名
+        self.ips_times = dict()  # IP集合出现次数
+        self.enable_wildcard = False  # 当前域名是否使用泛解析
+        self.wildcard_check = config.enable_wildcard_check
+        self.wildcard_deal = config.enable_wildcard_deal
+        self.check_env = True
+
+    def gen_brute_dict(self, domain):
+        logger.log('INFOR', f'正在为{domain}生成爆破字典')
+        dict_set = set()
+        # 如果domain不是self.subdomain 而是self.domain的子域则生成递归爆破字典
+        if self.place is None:
+            self.place = '*.' + domain
+        wordlist = self.wordlist
+        main_domain = self.register(domain)
+        if domain != main_domain:
+            wordlist = self.recursive_nextlist
+        if self.word:
+            word_subdomains = gen_word_subdomains(self.place, wordlist)
+            # set可以合并list
+            dict_set = dict_set.union(word_subdomains)
+        if self.fuzz:
+            fuzz_subdomains = gen_fuzz_subdomains(self.place, self.rule)
+            dict_set = dict_set.union(fuzz_subdomains)
+        # logger.log('INFOR', f'正在去重爆破字典')
+        # dict_set = utils.uniq_dict_list(dict_set)
+        count = len(dict_set)
+        logger.log('INFOR', f'生成的爆破字典大小为{count}')
+        if count > 10000000:
+            logger.log('ALERT', f'注意生成的爆破字典太大：{count} > 10000000')
+        return dict_set
+
+    def check_brute_params(self):
+        if not (self.word or self.fuzz):
+            logger.log('FATAL', f'请至少指定一种爆破模式')
+            exit(1)
+        if len(self.domains) > 1:
+            self.bulk = True
+        if self.fuzz:
+            if self.place is None or self.rule is None:
+                logger.log('FATAL', f'没有指定fuzz位置或规则')
+                exit(1)
+            if self.bulk:
+                logger.log('FATAL', f'批量爆破的场景下不能使用fuzz模式')
+                exit(1)
+            if self.recursive_brute:
+                logger.log('FATAL', f'使用fuzz模式下不能使用递归爆破')
+                exit(1)
+            fuzz_count = self.place.count('*')
+            if fuzz_count < 1:
+                logger.log('FATAL', f'没有指定fuzz位置')
+                exit(1)
+            if fuzz_count > 1:
+                logger.log('FATAL', f'只能指定1个fuzz位置')
+                exit(1)
+            if self.domain not in self.place:
+                logger.log('FATAL', f'指定fuzz的域名有误')
+                exit(1)
+
+    def main(self, domain):
+        start = time.time()
+        logger.log('INFOR', f'正在爆破域名{domain}')
+        massdns_dir = config.third_party_dir.joinpath('massdns')
+        result_dir = config.result_save_dir
+        temp_dir = result_dir.joinpath('temp')
+        utils.check_dir(temp_dir)
+        massdns_path = utils.get_massdns_path(massdns_dir)
+        timestring = utils.get_timestring()
+
+        wildcard_ips = list()  # 泛解析IP列表
+        wildcard_ttl = int()  # 泛解析TTL整型值
+        ns_list = query_domain_ns(self.domain)
+        ns_ip_list = query_domain_ns_a(ns_list)  # DNS权威名称服务器对应A记录列表
+        self.enable_wildcard = detect_wildcard(domain, ns_ip_list)
+
+        if self.enable_wildcard:
+            wildcard_ips, wildcard_ttl = collect_wildcard_record(domain,
+                                                                 ns_ip_list)
+        ns_path = get_nameservers_path(self.enable_wildcard, ns_ip_list)
+
+        dict_set = self.gen_brute_dict(domain)
+        dict_len = len(dict_set)
+
+        dict_name = f'generated_subdomains_{domain}_{timestring}.txt'
+        dict_path = temp_dir.joinpath(dict_name)
+        save_brute_dict(dict_path, dict_set)
+        del dict_set
+        gc.collect()
+
+        output_name = f'resolved_result_{domain}_{timestring}.json'
+        output_path = temp_dir.joinpath(output_name)
+        log_path = result_dir.joinpath('massdns.log')
+        check_dict()
+
+        utils.call_massdns(massdns_path, dict_path, ns_path, output_path,
+                           log_path, process_num=self.process_num,
+                           concurrent_num=self.concurrent_num)
+
+        ip_times = stat_ip_times(output_path)
+        self.records, self.subdomains = deal_output(output_path, ip_times,
+                                                    wildcard_ips, wildcard_ttl)
+        delete_file(dict_path, output_path)
+        end = time.time()
+        self.elapse = round(end - start, 1)
+        logger.log('INFOR', f'{self.source}模块耗时{self.elapse}秒'
+                            f'发现{domain}的子域{len(self.subdomains)}个')
+        logger.log('DEBUG', f'{self.source}模块发现{domain}的子域:\n'
+                            f'{self.subdomains}')
+        self.gen_result(brute=dict_len, valid=len(self.subdomains))
+        self.save_db()
+        return self.subdomains
+
+    def run(self):
+        logger.log('INFOR', f'开始执行{self.source}模块')
+        if self.check_env:
+            utils.check_env()
+        self.domains = utils.get_domains(self.target)
+        all_subdomains = list()
+        for self.domain in self.domains:
+            self.check_brute_params()
+            if self.recursive_brute:
+                logger.log('INFOR', f'开始递归爆破{self.domain}的第1层子域')
+            valid_subdomains = self.main(self.domain)
+            all_subdomains.extend(valid_subdomains)
+
+            # 递归爆破下一层的子域
+            # fuzz模式不使用递归爆破
+            if self.recursive_brute:
+                for layer_num in range(1, self.recursive_depth):
+                    # 之前已经做过1层子域爆破 当前实际递归层数是layer+1
+                    logger.log('INFOR', f'开始递归爆破{self.domain}的'
+                                        f'第{layer_num + 1}层子域')
+                    for subdomain in all_subdomains:
+                        self.place = '*.' + subdomain
+                        # 进行下一层子域爆破的限制条件
+                        num = subdomain.count('.') - self.domain.count('.')
+                        if num == layer_num:
+                            valid_subdomains = self.main(subdomain)
+                            all_subdomains.extend(valid_subdomains)
+
+            logger.log('INFOR', f'结束执行{self.source}模块爆破域名{self.domain}')
+            if not self.path:
+                name = f'{self.domain}_brute_result.{self.format}'
+                self.path = config.result_save_dir.joinpath(name)
+            # 数据库导出
+            if self.export:
+                dbexport.export(self.domain,
+                                alive=self.alive,
+                                limit='resolve',
+                                path=self.path,
+                                format=self.format)
+
+
+if __name__ == '__main__':
+    fire.Fire(Brute)
@@ -1,6 +1,7 @@
 import time
 import threading
 import importlib
+
 import config
 import dbexport
 from config import logger
@@ -12,12 +13,12 @@ class Collect(object):
    """
    def __init__(self, domain, export=True):
        self.domain = domain
-        self.elapsed = 0.0
+        self.elapse = 0.0
        self.modules = []
-        self.collect_func = []
+        self.collect_funcs = []
        self.path = None
        self.export = export
-        self.format = 'xls'
+        self.format = 'csv'

    def get_mod(self):
        """
@@ -31,7 +32,7 @@ class Collect(object):
                       'dnsquery', 'intelligence', 'search']
            # modules = ['intelligence']  # crawl模块还有点问题
            for module in modules:
-                module_path = config.oneforall_module_path.joinpath(module)
+                module_path = config.module_dir.joinpath(module)
                for path in module_path.rglob('*.py'):
                    # 需要导入的类
                    import_module = ('modules.' + module, path.stem)
@@ -44,8 +45,9 @@ class Collect(object):
        导入脚本的do函数
        """
        for package, name in self.modules:
-            import_object = importlib.import_module('.'+name, package)
-            self.collect_func.append(getattr(import_object, 'do'))
+            import_object = importlib.import_module('.' + name, package)
+            func = getattr(import_object, 'do')
+            self.collect_funcs.append([func, name])

    def run(self):
        """
@@ -58,8 +60,10 @@ class Collect(object):

        threads = []
        # 创建多个子域收集线程
-        for collect_func in self.collect_func:
-            thread = threading.Thread(target=collect_func,
+        for collect_func in self.collect_funcs:
+            func_obj, func_name = collect_func
+            thread = threading.Thread(target=func_obj,
+                                      name=func_name,
                                      args=(self.domain,),
                                      daemon=True)
            threads.append(thread)
@@ -68,18 +72,24 @@ class Collect(object):
            thread.start()
        # 等待所有线程完成
        for thread in threads:
-            thread.join()
+            # 挨个线程判断超时 最坏情况主线程阻塞时间=线程数*module_thread_timeout
+            # 超时线程将脱离主线程 由于创建线程时已添加守护属于 所有超时线程会随着主线程结束
+            thread.join(config.module_thread_timeout)
+
+        for thread in threads:
+            if thread.is_alive():
+                logger.log('ALERT', f'{thread.name}模块线程发生超时')

        # 数据库导出
        if self.export:
            if not self.path:
                name = f'{self.domain}.{self.format}'
-                self.path = config.result_save_path.joinpath(name)
-            dbexport.export(self.domain, dpath=self.path, format=self.format)
+                self.path = config.result_save_dir.joinpath(name)
+            dbexport.export(self.domain, path=self.path, format=self.format)
        end = time.time()
-        self.elapsed = round(end - start, 1)
+        self.elapse = round(end - start, 1)


 if __name__ == '__main__':
-    a = Collect('example.com')
-    a.run()
+    collect = Collect('example.com')
+    collect.run()
@@ -1,10 +1,9 @@
-# coding=utf-8
 from .module import Module


 class Crawl(Module):
    """
-
+    爬虫基类
    """
    def __init__(self):
        Module.__init__(self)
@@ -6,6 +6,7 @@ SQLite数据库初始化和操作
 """

 import records
+
 import config
 from records import Connection
 from config import logger
@@ -23,44 +24,66 @@ class Database(object):
        :param db_path: 数据库连接或路径
        :return: SQLite数据库
        """
-        logger.log('DEBUG', f'正在获取数据库连接')
+        logger.log('TRACE', f'正在获取数据库连接')
        if isinstance(db_path, Connection):
            return db_path
        protocol = 'sqlite:///'
        if not db_path:  # 数据库路径为空连接默认数据库
-            db_path = f'{protocol}{config.result_save_path}/result.sqlite3'
+            db_path = f'{protocol}{config.result_save_dir}/result.sqlite3'
        else:
            db_path = protocol + db_path
        db = records.Database(db_path)  # 不存在数据库时会新建一个数据库
-        logger.log('DEBUG', f'使用数据库: {db_path}')
+        logger.log('TRACE', f'使用数据库: {db_path}')
        return db.get_connection()

+    def query(self, sql):
+        try:
+            results = self.conn.query(sql)
+        except Exception as e:
+            logger.log('ERROR', e.args)
+        else:
+            return results
+
    def create_table(self, table_name):
        """
-        初始化数据库
+        创建表结构

        :param str table_name: 要创建的表名
        """
        table_name = table_name.replace('.', '_')
-        logger.log('DEBUG', f'正在创建{table_name}表')
-        try:
-            self.conn.query(f'create table if not exists "{table_name}" ('
-                            f'id integer primary key,'
-                            f'url text,'
-                            f'subdomain text,'
-                            f'port int,'
-                            f'ips text,'
-                            f'status int,'
-                            f'reason text,'
-                            f'valid int,'
-                            f'title text,'
-                            f'banner text,'
-                            f'module text,'
-                            f'source text,'
-                            f'elapsed float,'
-                            f'count int)')
-        except Exception as e:
-            logger.log('ERROR', e)
+        if self.exist_table(table_name):
+            logger.log('TRACE', f'已经存在{table_name}表')
+            return
+        logger.log('TRACE', f'正在创建{table_name}表')
+        self.query(f'create table "{table_name}" ('
+                   f'id integer primary key,'
+                   f'type text,'
+                   f'alive int,'
+                   f'request int,'
+                   f'resolve int,'
+                   f'new int,'
+                   f'url text,'
+                   f'subdomain text,'
+                   f'port int,'
+                   f'level int,'
+                   f'cname text,'
+                   f'content text,'
+                   f'public int,'
+                   f'status int,'
+                   f'reason text,'
+                   f'title text,'
+                   f'banner text,'
+                   f'header text,'
+                   f'response text,'
+                   f'times text,'
+                   f'ttl text,'
+                   f'resolver text,'
+                   f'module text,'
+                   f'source text,'
+                   f'elapse float,'
+                   f'find int,'
+                   f'brute int,'
+                   f'valid int)')

    def save_db(self, table_name, results, module_name=None):
        """
@@ -70,21 +93,42 @@ class Database(object):
        :param list results: 结果列表
        :param str module_name: 模块名
        """
-        logger.log('DEBUG', f'正在将{module_name}模块发现{table_name}的子域结果存入数据库')
+        logger.log('TRACE', f'正在将{module_name}模块发现{table_name}的子域'
+                            '结果存入数据库')
        table_name = table_name.replace('.', '_')
        if results:
            try:
                self.conn.bulk_query(
                    f'insert into "{table_name}" ('
-                    f'id, url, subdomain, port, ips, status, reason, valid,'
-                    f'title, banner, module, source, elapsed, count)'
-                    f'values (:id, :url, :subdomain, :port, :ips, :status,'
-                    f':reason, :valid, :title, :banner, :module, :source,'
-                    f':elapsed, :count)',
-                    results)
+                    f'id, type, alive, resolve, request, new, url, subdomain,'
+                    f'port, level, cname, content, public, status, reason,'
+                    f'title, banner, header, response, times, ttl, resolver,'
+                    f'module, source, elapse, find, brute, valid) '
+                    f'values (:id, :type, :alive, :resolve, :request, :new,'
+                    f':url, :subdomain, :port, :level, :cname, :content,'
+                    f':public, :status, :reason, :title, :banner, :header,'
+                    f':response, :times, :ttl, :resolver, :module, :source,'
+                    f':elapse, :find, :brute, :valid)', results)
            except Exception as e:
                logger.log('ERROR', e)

+    def exist_table(self, table_name):
+        """
+        判断是否存在某表
+
+        :param str table_name: 表名
+        :return: 是否存在某表
+        """
+        table_name = table_name.replace('.', '_')
+        logger.log('TRACE', f'正在查询是否存在{table_name}表')
+        results = self.query(f'select count() from sqlite_master '
+                             f'where type = "table" and '
+                             f'name = "{table_name}"')
+        if results.scalar() == 0:
+            return False
+        else:
+            return True
+
    def copy_table(self, table_name, bak_table_name):
        """
        复制表创建备份
@@ -94,13 +138,10 @@ class Database(object):
        """
        table_name = table_name.replace('.', '_')
        bak_table_name = bak_table_name.replace('.', '_')
-        logger.log('DEBUG', f'正在将{table_name}表复制到{bak_table_name}新表')
-        try:
-            self.conn.query(f'drop table if exists "{bak_table_name}"')
-            self.conn.query(f'create table "{bak_table_name}" '
-                            f'as select * from "{table_name}"')
-        except Exception as e:
-            logger.log('ERROR', e)
+        logger.log('TRACE', f'正在将{table_name}表复制到{bak_table_name}新表')
+        self.query(f'drop table if exists "{bak_table_name}"')
+        self.query(f'create table "{bak_table_name}" '
+                   f'as select * from "{table_name}"')

    def clear_table(self, table_name):
        """
@@ -109,11 +150,8 @@ class Database(object):
        :param str table_name: 表名
        """
        table_name = table_name.replace('.', '_')
-        logger.log('DEBUG', f'正在清空{table_name}表中的数据')
-        try:
-            self.conn.query(f'delete from "{table_name}"')
-        except Exception as e:
-            logger.log('ERROR', e)
+        logger.log('TRACE', f'正在清空{table_name}表中的数据')
+        self.query(f'delete from "{table_name}"')

    def drop_table(self, table_name):
        """
@@ -122,27 +160,21 @@ class Database(object):
        :param str table_name: 表名
        """
        table_name = table_name.replace('.', '_')
-        logger.log('DEBUG', f'正在删除{table_name}表')
-        try:
-            self.conn.query(f'drop table if exists "{table_name}"')
-        except Exception as e:
-            logger.log('ERROR', e)
+        logger.log('TRACE', f'正在删除{table_name}表')
+        self.query(f'drop table if exists "{table_name}"')

    def rename_table(self, table_name, new_table_name):
        """
-        复制表创建备份
+        重命名表名

        :param str table_name: 表名
        :param str new_table_name: 新表名
        """
        table_name = table_name.replace('.', '_')
        new_table_name = new_table_name.replace('.', '_')
-        logger.log('DEBUG', f'正在将{table_name}表重命名为{table_name}表')
-        try:
-            self.conn.query(f'alter table "{table_name}" '
-                            f'rename to "{new_table_name}"')
-        except Exception as e:
-            logger.log('ERROR', e)
+        logger.log('TRACE', f'正在将{table_name}表重命名为{table_name}表')
+        self.query(f'alter table "{table_name}" '
+                   f'rename to "{new_table_name}"')

    def deduplicate_subdomain(self, table_name):
        """
@@ -151,13 +183,10 @@ class Database(object):
        :param str table_name: 表名
        """
        table_name = table_name.replace('.', '_')
-        logger.log('DEBUG', f'正在去重{table_name}表中的子域')
-        try:
-            self.conn.query(
-                f'delete from "{table_name}" where id not in (select min(id) '
-                f'from "{table_name}" group by subdomain)')
-        except Exception as e:
-            logger.log('ERROR', e)
+        logger.log('TRACE', f'正在去重{table_name}表中的子域')
+        self.query(f'delete from "{table_name}" where '
+                   f'id not in (select min(id) '
+                   f'from "{table_name}" group by subdomain)')

    def remove_invalid(self, table_name):
        """
@@ -166,13 +195,20 @@ class Database(object):
        :param str table_name: 表名
        """
        table_name = table_name.replace('.', '_')
-        logger.log('DEBUG', f'正在去除{table_name}表中的无效子域')
-        try:
-            self.conn.query(
-                f'delete from "{table_name}" where '
-                f'subdomain is null or valid == 0')
-        except Exception as e:
-            logger.log('ERROR', e)
+        logger.log('TRACE', f'正在去除{table_name}表中的无效子域')
+        self.query(f'delete from "{table_name}" where '
+                   f'subdomain is null or resolve == 0')
+
+    def deal_table(self, deal_table_name, backup_table_name):
+        """
+        收集任务完成时对表进行处理
+
+        :param str deal_table_name: 待处理的表名
+        :param str backup_table_name: 备份的表名
+        """
+        self.copy_table(deal_table_name, backup_table_name)
+        self.remove_invalid(deal_table_name)
+        self.deduplicate_subdomain(deal_table_name)

    def get_data(self, table_name):
        """
@@ -181,30 +217,34 @@ class Database(object):
        :param str table_name: 表名
        """
        table_name = table_name.replace('.', '_')
-        logger.log('DEBUG', f'获取{table_name}表中的所有数据')
-        try:
-            rows = self.conn.query(f'select * from "{table_name}"')
-        except Exception as e:
-            logger.log('ERROR', e)
-        else:
-            return rows
+        logger.log('TRACE', f'获取{table_name}表中的所有数据')
+        return self.query(f'select * from "{table_name}"')

-    def get_subdomain(self, table_name, valid):
+    def export_data(self, table_name, alive, limit):
        """
-        获取表中的子域数据
+        获取表中的部分数据

        :param str table_name: 表名
-        :param int valid: 是否有效
+        :param any alive: 存活
+        :param str limit: 限制字段
        """
        table_name = table_name.replace('.', '_')
-        logger.log('DEBUG', f'获取{table_name}表中的所有数据')
-        try:
-            rows = self.conn.query(
-                f'select * from "{table_name}" where valid = {valid}')
-        except Exception as e:
-            logger.log('ERROR', e)
-        else:
-            return rows
+        query = f'select id, type, new, alive, request, resolve, url, ' \
+                f'subdomain, level, cname, content, public, port, status, ' \
+                f'reason, title, banner, times, ttl, resolver, module, ' \
+                f'source, elapse, find, brute, valid from "{table_name}"'
+        if alive and limit:
+            if limit in ['resolve', 'request']:
+                where = f' where {limit} = 1'
+                query += where
+        elif alive:
+            where = f' where alive = 1'
+            query += where
+        logger.log('TRACE', f'获取{table_name}表中的数据')
+        return self.query(query)

    def close(self):
+        """
+        关闭数据库连接
+        """
        self.conn.close()
@@ -1,4 +1,3 @@
-# coding=utf-8
 import re
 import tldextract
 import config
@@ -31,14 +30,15 @@ class Domain(object):
        """
        域名导出

-        >>>d = Domain('www.example.com')
+        >>> d = Domain('www.example.com')
        <domain.Domain object>
-        >>>d.extract()
+        >>> d.extract()
        ExtractResult(subdomain='www', domain='example', suffix='com')

        :return: 导出结果
        """
-        extract_cache_file = config.data_storage_path.joinpath('public_suffix_list.dat')
+        data_storage_dir = config.data_storage_dir
+        extract_cache_file = data_storage_dir.joinpath('public_suffix_list.dat')
        tldext = tldextract.TLDExtract(extract_cache_file)
        result = self.match()
        if result:
@@ -50,9 +50,9 @@ class Domain(object):
        """
        获取注册域名

-        >>>d = Domain('www.example.com')
+        >>> d = Domain('www.example.com')
        <domain.Domain object>
-        >>>d.registered()
+        >>> d.registered()
        example.com

        :return: 注册域名
@@ -0,0 +1,25 @@
+from .module import Module
+from common import utils
+
+
+class Lookup(Module):
+    """
+    DNS查询基类
+    """
+    def __init__(self):
+        Module.__init__(self)
+
+    def query(self):
+        """
+        查询域名的TXT记录
+        :return: 查询结果
+        """
+        answer = utils.dns_query(self.domain, self.type)
+        if answer is None:
+            return None
+        for item in answer:
+            record = item.to_text()
+            subdomains = utils.match_subdomain(self.domain, record)
+            self.subdomains = self.subdomains.union(subdomains)
+            self.gen_record(subdomains, record)
+        return self.subdomains
@@ -15,7 +15,6 @@ from . import utils
 from .domain import Domain
 from common.database import Database

-
 lock = threading.Lock()


@@ -29,22 +28,24 @@ class Module(object):
        self.delay = config.request_delay  # 请求睡眠时延
        self.timeout = config.request_timeout  # 请求超时时间
        self.verify = config.request_verify  # 请求SSL验证
-        self.domain = ''  # 要进行子域名收集的域名
+        self.domain = str()  # 当前进行子域名收集的主域
+        self.type = 'A'  # 对主域进行子域收集时利用的DNS记录查询类型(默认利用A记录)
        self.subdomains = set()  # 存放发现的子域
        self.records = dict()  # 存放子域解析记录
        self.results = list()  # 存放模块结果
        self.start = time.time()  # 模块开始执行时间
-        self.end = None
-        self.elapsed = None  # 模块执行耗时
+        self.end = None  # 模块结束执行时间
+        self.elapse = None  # 模块执行耗时

    def check(self, *apis):
        """
        简单检查是否配置了api信息
+
        :param apis: api信息元组
        :return: 检查结果
        """
        if not all(apis):
-            logger.log('ALERT', f'{self.source}模块API配置有误跳过执行')
+            logger.log('ALERT', f'{self.source}模块没有配置API跳过执行')
            return False
        return True

@@ -59,12 +60,40 @@ class Module(object):
        输出模块结束信息
        """
        self.end = time.time()
-        self.elapsed = round(self.end - self.start, 1)
+        self.elapse = round(self.end - self.start, 1)
        logger.log('DEBUG', f'结束执行{self.source}模块收集{self.domain}的子域')
-        logger.log('INFOR', f'{self.source}模块耗时{self.elapsed}秒发现子域'
-                   f'{len(self.subdomains)}个')
+        logger.log('INFOR', f'{self.source}模块耗时{self.elapse}秒发现子域'
+                            f'{len(self.subdomains)}个')
        logger.log('DEBUG', f'{self.source}模块发现{self.domain}的子域\n'
-                   f'{self.subdomains}')
+                            f'{self.subdomains}')
+
+    def head(self, url, params=None, check=True, **kwargs):
+        """
+        自定义head请求
+
+        :param str url: 请求地址
+        :param dict params: 请求参数
+        :param bool check: 检查响应
+        :param kwargs: 其他参数
+        :return: requests响应对象
+        """
+        try:
+            resp = requests.head(url,
+                                 params=params,
+                                 cookies=self.cookie,
+                                 headers=self.header,
+                                 proxies=self.proxy,
+                                 timeout=self.timeout,
+                                 verify=self.verify,
+                                 **kwargs)
+        except Exception as e:
+            logger.log('ERROR', e.args)
+            return None
+        if not check:
+            return resp
+        if utils.check_response('HEAD', resp):
+            return resp
+        return None

    def get(self, url, params=None, check=True, **kwargs):
        """
@@ -86,7 +115,7 @@ class Module(object):
                                verify=self.verify,
                                **kwargs)
        except Exception as e:
-            logger.log('ERROR', e)
+            logger.log('ERROR', e.args)
            return None
        if not check:
            return resp
@@ -114,11 +143,11 @@ class Module(object):
                                 verify=self.verify,
                                 **kwargs)
        except Exception as e:
-            logger.log('ERROR', e)
+            logger.log('ERROR', e.args)
            return None
        if not check:
            return resp
-        if utils.check_response('GET', resp):
+        if utils.check_response('POST', resp):
            return resp
        return None

@@ -142,16 +171,16 @@ class Module(object):
        :return: 代理字典
        """
        if not config.enable_proxy:
-            logger.log('DEBUG', f'所有模块不使用代理')
+            logger.log('TRACE', f'所有模块不使用代理')
            return self.proxy
        if config.proxy_all_module:
-            logger.log('DEBUG', f'{module}模块使用代理')
+            logger.log('TRACE', f'{module}模块使用代理')
            return utils.get_random_proxy()
        if module in config.proxy_partial_module:
-            logger.log('DEBUG', f'{module}模块使用代理')
+            logger.log('TRACE', f'{module}模块使用代理')
            return utils.get_random_proxy()
        else:
-            logger.log('DEBUG', f'{module}模块不使用代理')
+            logger.log('TRACE', f'{module}模块不使用代理')
            return self.proxy

    @staticmethod
@@ -165,7 +194,7 @@ class Module(object):
        :return: 匹配出的子域集合或列表
        :rtype: set or list
        """
-        logger.log('DEBUG', f'正则匹配响应体中的子域')
+        logger.log('TRACE', f'正则匹配响应体中的子域')
        regexp = r'(?:\>|\"|\'|\=|\,)(?:http\:\/\/|https\:\/\/)?' \
                 r'(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.){0,}' \
                 + domain.replace('.', r'\.')
@@ -192,69 +221,138 @@ class Module(object):
    def save_json(self):
        """
        将各模块结果保存为json文件
-        """
-        logger.log('DEBUG', f'将{self.source}模块发现的子域结果保存为json文件')
-        if config.save_module_result:
-            dpath = config.result_save_path.joinpath(self.domain, self.module)
-            dpath.mkdir(parents=True, exist_ok=True)
-            name = self.source + '.json'
-            path = dpath.joinpath(name)
-            with open(path, mode='w', encoding='utf-8') as file:
-                result = {'domain': self.domain,
-                          'name': self.module,
-                          'source': self.source,
-                          'elapsed': self.elapsed,
-                          'count': len(self.subdomains),
-                          'subdomains': list(self.subdomains),
-                          'records': self.records}
-                json.dump(result, file, ensure_ascii=False, indent=4)

-    def gen_result(self):
-        results = list()
-        if not len(self.subdomains):  # 一个子域都没有发现的情况
+        :return： 是否保存成功
+        """
+        if not config.save_module_result:
+            return False
+        logger.log('TRACE', f'将{self.source}模块发现的子域结果保存为json文件')
+        path = config.result_save_dir.joinpath(self.domain, self.module)
+        path.mkdir(parents=True, exist_ok=True)
+        name = self.source + '.json'
+        path = path.joinpath(name)
+        with open(path, mode='w', encoding='utf-8', errors='ignore') as file:
+            result = {'domain': self.domain,
+                      'name': self.module,
+                      'source': self.source,
+                      'elapse': self.elapse,
+                      'find': len(self.subdomains),
+                      'subdomains': list(self.subdomains),
+                      'records': self.records}
+            json.dump(result, file, ensure_ascii=False, indent=4)
+        return True
+
+    def gen_record(self, subdomains, record):
+        """
+        生成记录字典
+        """
+        item = dict()
+        item['content'] = record
+        for subdomain in subdomains:
+            self.records[subdomain] = item
+
+    def gen_result(self, find=0, brute=None, valid=0):
+        """
+        生成结果
+        """
+        logger.log('DEBUG', f'正在生成最终结果')
+        if not len(self.subdomains):  # 该模块一个子域都没有发现的情况
            result = {'id': None,
+                      'type': self.type,
+                      'alive': None,
+                      'request': None,
+                      'resolve': None,
+                      'new': None,
                      'url': None,
                      'subdomain': None,
+                      'level': None,
+                      'cname': None,
+                      'content': None,
+                      'public': None,
                      'port': None,
-                      'ips': None,
                      'status': None,
                      'reason': None,
-                      'valid': None,
                      'title': None,
                      'banner': None,
+                      'header': None,
+                      'response': None,
+                      'times': None,
+                      'ttl': None,
+                      'resolver': None,
                      'module': self.module,
                      'source': self.source,
-                      'elapsed': self.elapsed,
-                      'count': 0}
-            results.append(result)
-            self.results = (self.source, results)
+                      'elapse': self.elapse,
+                      'find': find,
+                      'brute': brute,
+                      'valid': valid}
+            self.results.append(result)
        else:
            for subdomain in self.subdomains:
                url = 'http://' + subdomain
-                ips = self.records.get(subdomain)
+                level = subdomain.count('.') - self.domain.count('.')
+                record = self.records.get(subdomain)
+                if record is None:
+                    record = dict()
+                resolve = record.get('resolve')
+                request = record.get('request')
+                alive = record.get('alive')
+                if self.type != 'A':  # 不是利用的DNS记录的A记录查询子域默认都有效
+                    resolve = 1
+                    request = 1
+                    alive = 1
+                reason = record.get('reason')
+                resolver = record.get('resolver')
+                cname = record.get('cname')
+                content = record.get('content')
+                times = record.get('times')
+                ttl = record.get('ttl')
+                public = record.get('public')
+                if isinstance(cname, list):
+                    cname = ','.join(cname)
+                    content = ','.join(content)
+                    times = ','.join([str(num) for num in times])
+                    ttl = ','.join([str(num) for num in ttl])
+                    public = ','.join([str(num) for num in public])
                result = {'id': None,
+                          'type': self.type,
+                          'alive': alive,
+                          'request': request,
+                          'resolve': resolve,
+                          'new': None,
                          'url': url,
                          'subdomain': subdomain,
-                          'port': None,
-                          'ips': ips,
+                          'level': level,
+                          'cname': cname,
+                          'content': content,
+                          'public': public,
+                          'port': 80,
                          'status': None,
-                          'reason': None,
-                          'valid': None,
+                          'reason': reason,
                          'title': None,
                          'banner': None,
+                          'header': None,
+                          'response': None,
+                          'times': times,
+                          'ttl': ttl,
+                          'resolver': resolver,
                          'module': self.module,
                          'source': self.source,
-                          'elapsed': self.elapsed,
-                          'count': len(self.subdomains)}
-                results.append(result)
-            self.results = (self.source, results)
+                          'elapse': self.elapse,
+                          'find': find,
+                          'brute': brute,
+                          'valid': valid,
+                          }
+                self.results.append(result)

    def save_db(self):
+        """
+        将模块结果存入数据库中
+
+        """
+        logger.log('DEBUG', f'正在将结果存入到数据库')
        lock.acquire()
        db = Database()
        db.create_table(self.domain)
-        source, results = self.results
-        # 将结果存入数据库中
-        db.save_db(self.domain, results, source)
+        db.save_db(self.domain, self.results, self.source)
        db.close()
        lock.release()
@@ -1,4 +1,3 @@
-# coding=utf-8
 from .module import Module


@@ -1,159 +1,264 @@
-# coding=utf-8
-
 import asyncio
 import functools

 import aiohttp
 import tqdm
 from aiohttp import ClientSession
-from aiohttp.resolver import AsyncResolver
 from bs4 import BeautifulSoup
+
 import config
 from common import utils
 from config import logger
+from common.database import Database


-def get_ports(port):
-    logger.log('INFOR', f'正在获取请求端口范围')
-    ports = set()
-    if isinstance(port, set):
-        ports = port
-    elif isinstance(port, str):
-        if port not in {'small', 'medium', 'large', 'xlarge'}:
-            logger.log('ERROR', f'不存在{port}等端口范围')
-            port = 'medium'
-        ports = config.ports.get(port)
-        logger.log('INFOR', f'使用{port}等端口范围')
-    if not ports:  # 意外情况 ports_range为空使用使用中等端口范围
-        logger.log('ALERT', f'使用medium等端口范围')
-        ports = config.ports.get('medium')
-    return ports
-
-
-def gen_new_datas(datas, ports):
-    logger.log('INFOR', f'正在生成请求地址')
-    new_datas = []
-    protocols = ['http://']
-    for data in datas:
-        valid = data.get('valid')
-        if valid is None:  # 子域有效性未知的才进行http请求探测
-            subdomain = data.get('subdomain')
-            for port in ports:
-                for protocol in protocols:
-                    if port == 443:
-                        url = f'https://{subdomain}:{port}'
-                    elif port == 8443:
-                        url = f'https://{subdomain}:{port}'
-                    else:
-                        url = f'{protocol}{subdomain}:{port}'
-                    data['id'] = None
-                    data['url'] = url
-                    data['port'] = port
-                    new_datas.append(data)
-                    data = dict(data)  # 需要生成一个新的字典对象
-    return new_datas
-
-
-async def fetch(session, url, semaphore):
-    """
-    请求
-
-    :param session: session对象
-    :param url: url地址
-    :param semaphore: 并发信号量
-    :return: 响应对象和响应文本
-    """
-    timeout = aiohttp.ClientTimeout(total=config.get_timeout)
-    async with semaphore:
-        async with session.get(url,
-                               ssl=config.verify_ssl,
-                               allow_redirects=config.get_redirects,
-                               timeout=timeout,
-                               proxy=config.get_proxy) as resp:
-
-            try:
-                text = await resp.text(encoding='gb2312')  # 先尝试用fb2312解码
-            except UnicodeDecodeError:
-                text = await resp.text()
-        return resp, text
-
-
-def request_callback(future, index, datas):
-    try:
-        result = future.result()
-    except Exception as e:
-        logger.log('DEBUG', e.args)
-        datas[index]['reason'] = str(e.args)
-        datas[index]['valid'] = 0
-    else:
-        resp, text = result
-        datas[index]['reason'] = resp.reason
-        datas[index]['status'] = resp.status
-        if resp.status >= 500:
-            datas[index]['valid'] = 0
-        else:
-            datas[index]['valid'] = 1
-            headers = resp.headers
-            banner = str({'Server': headers.get('Server'),
-                          'Via': headers.get('Via'),
-                          'X-Powered-By': headers.get('X-Powered-By')})
-            datas[index]['banner'] = banner
-            soup = BeautifulSoup(text, 'lxml')
-            title = soup.title
-            desc = soup.find('meta', attrs={'name': 'description'})
-            head = soup.head
-            if title:
-                datas[index]['title'] = title.text.strip()
-            elif desc:
-                datas[index]['title'] = desc['content'].strip()
-            elif head:
-                datas[index]['title'] = head.text.strip()
-            elif len(text) <= 200:
-                datas[index]['title'] = text.strip()
-
-
-async def bulk_get_request(datas, port):
-    ports = get_ports(port)
-    new_datas = gen_new_datas(datas, ports)
-    logger.log('INFOR', f'正在异步进行子域的GET请求')
-
+def get_limit_conn():
    limit_open_conn = config.limit_open_conn
    if limit_open_conn is None:  # 默认情况
        limit_open_conn = utils.get_semaphore()
    elif not isinstance(limit_open_conn, int):  # 如果传入不是数字的情况
        limit_open_conn = utils.get_semaphore()
-    # 使用异步域名解析器 自定义域名服务器
-    resolver = AsyncResolver(nameservers=config.resolver_nameservers)
-    conn = aiohttp.TCPConnector(ssl=config.verify_ssl,
-                                limit=limit_open_conn,
-                                limit_per_host=config.limit_per_host,
-                                resolver=resolver)
+    return limit_open_conn

-    semaphore = asyncio.Semaphore(limit_open_conn)
+
+def get_ports(port):
+    logger.log('DEBUG', f'正在获取请求端口范围')
+    ports = set()
+    if isinstance(port, (set, list, tuple)):
+        ports = port
+    elif isinstance(port, int):
+        if 0 <= port <= 65535:
+            ports = {port}
+    elif port in {'default', 'small', 'large'}:
+        logger.log('DEBUG', f'请求{port}等端口范围')
+        ports = config.ports.get(port)
+    if not ports:  # 意外情况
+        logger.log('ERROR', f'指定请求端口范围有误')
+        ports = {80}
+    logger.log('INFOR', f'请求端口范围：{ports}')
+    return set(ports)
+
+
+def gen_req_data(data, ports):
+    logger.log('INFOR', f'正在生成请求地址')
+    new_data = []
+    for data in data:
+        resolve = data.get('resolve')
+        # 解析失败(0)的子域不进行http请求探测
+        if resolve == 0:
+            continue
+        subdomain = data.get('subdomain')
+        for port in ports:
+            if str(port).endswith('443'):
+                url = f'https://{subdomain}:{port}'
+                if port == 443:
+                    url = f'https://{subdomain}'
+                data['id'] = None
+                data['url'] = url
+                data['port'] = port
+                new_data.append(data)
+                data = dict(data)  # 需要生成一个新的字典对象
+            else:
+                url = f'http://{subdomain}:{port}'
+                if port == 80:
+                    url = f'http://{subdomain}'
+                data['id'] = None
+                data['url'] = url
+                data['port'] = port
+                new_data.append(data)
+                data = dict(data)  # 需要生成一个新的字典对象
+    return new_data
+
+
+async def fetch(session, url):
+    """
+    请求
+
+    :param session: session对象
+    :param str url: url地址
+    :return: 响应对象和响应文本
+    """
+    method = config.request_method.upper()
+    timeout = aiohttp.ClientTimeout(total=None,
+                                    connect=None,
+                                    sock_read=config.sockread_timeout,
+                                    sock_connect=config.sockconn_timeout)
+    try:
+        if method == 'HEAD':
+            async with session.head(url,
+                                    ssl=config.verify_ssl,
+                                    allow_redirects=config.allow_redirects,
+                                    timeout=timeout,
+                                    proxy=config.aiohttp_proxy) as resp:
+                text = await resp.text()
+        else:
+            async with session.get(url,
+                                   ssl=config.verify_ssl,
+                                   allow_redirects=config.allow_redirects,
+                                   timeout=timeout,
+                                   proxy=config.aiohttp_proxy) as resp:
+
+                try:
+                    # 先尝试用utf-8解码
+                    text = await resp.text(encoding='utf-8', errors='strict')
+                except UnicodeError:
+                    try:
+                        # 再尝试用gb18030解码
+                        text = await resp.text(encoding='gb18030',
+                                               errors='strict')
+                    except UnicodeError:
+                        # 最后尝试自动解码
+                        text = await resp.text(encoding=None,
+                                               errors='ignore')
+        return resp, text
+    except Exception as e:
+        return e
+
+
+def get_title(markup):
+    """
+    获取标题
+
+    :param markup: html标签
+    :return: 标题
+    """
+    soup = BeautifulSoup(markup, 'html.parser')
+
+    title = soup.title
+    if title:
+        return title.text
+
+    h1 = soup.h1
+    if h1:
+        return h1.text
+
+    h2 = soup.h2
+    if h2:
+        return h2.text
+
+    h3 = soup.h3
+    if h2:
+        return h3.text
+
+    desc = soup.find('meta', attrs={'name': 'description'})
+    if desc:
+        return desc['content']
+
+    word = soup.find('meta', attrs={'name': 'keywords'})
+    if word:
+        return word['content']
+
+    text = soup.text
+    if len(text) <= 200:
+        return text
+
+    return 'None'
+
+
+def request_callback(future, index, datas):
+    result = future.result()
+    if isinstance(result, BaseException):
+        logger.log('TRACE', result.args)
+        name = utils.get_classname(result)
+        datas[index]['reason'] = name + ' ' + str(result)
+        datas[index]['request'] = 0
+        datas[index]['alive'] = 0
+    elif isinstance(result, tuple):
+        resp, text = result
+        datas[index]['reason'] = resp.reason
+        datas[index]['status'] = resp.status
+        if resp.status == 400 or resp.status >= 500:
+            datas[index]['request'] = 0
+            datas[index]['alive'] = 0
+        else:
+            datas[index]['request'] = 1
+            datas[index]['alive'] = 1
+            headers = resp.headers
+            datas[index]['banner'] = utils.get_sample_banner(headers)
+            datas[index]['header'] = str(dict(headers))[1:-1]
+            if isinstance(text, str):
+                title = get_title(text).strip()
+                datas[index]['title'] = utils.remove_invalid_string(title)
+                datas[index]['response'] = utils.remove_invalid_string(text)
+
+
+def get_connector():
+    limit_open_conn = get_limit_conn()
+    return aiohttp.TCPConnector(ttl_dns_cache=300,
+                                ssl=config.verify_ssl,
+                                limit=limit_open_conn,
+                                limit_per_host=config.limit_per_host)
+
+
+def get_header():
    header = None
    if config.fake_header:
        header = utils.gen_fake_header()
-    async with ClientSession(connector=conn, headers=header) as session:
+    return header
+
+
+async def bulk_request(data, port):
+    ports = get_ports(port)
+    no_req_data = utils.get_filtered_data(data)
+    to_req_data = gen_req_data(data, ports)
+    method = config.request_method
+    logger.log('INFOR', f'请求使用{method}方法')
+    logger.log('INFOR', f'正在进行异步子域请求')
+    connector = get_connector()
+    header = get_header()
+    async with ClientSession(connector=connector, headers=header) as session:
        tasks = []
-        for i, data in enumerate(new_datas):
+        for i, data in enumerate(to_req_data):
            url = data.get('url')
-            task = asyncio.ensure_future(fetch(session, url, semaphore))
+            task = asyncio.ensure_future(fetch(session, url))
            task.add_done_callback(functools.partial(request_callback,
                                                     index=i,
-                                                     datas=new_datas))
+                                                     datas=to_req_data))
            tasks.append(task)
-        if tasks:  # 任务列表里有任务不空时才进行解析
+        # 任务列表里有任务不空时才进行解析
+        if tasks:
            # 等待所有task完成 错误聚合到结果列表里
            futures = asyncio.as_completed(tasks)
            for future in tqdm.tqdm(futures,
                                    total=len(tasks),
-                                    desc='Progress',
-                                    smoothing=1.0,
-                                    ncols=True):
-                try:
-                    await future
-                except:
-                    pass
+                                    desc='Request Progress',
+                                    ncols=80):
+                await future
+    return to_req_data + no_req_data

-    logger.log('INFOR', f'完成异步进行子域的GET请求')
-    return new_datas
+
+def run_request(domain, data, port):
+    """
+    调用子域请求入口函数
+
+    :param str domain: 待请求的主域
+    :param list data: 待请求的子域数据
+    :param str port: 待请求的端口范围
+    :return: 请求后得到的结果列表
+    :rtype: list
+    """
+    logger.log('INFOR', f'开始执行子域请求模块')
+    loop = asyncio.get_event_loop()
+    asyncio.set_event_loop(loop)
+    data = utils.set_id_none(data)
+    request_coroutine = bulk_request(data, port)
+    data = loop.run_until_complete(request_coroutine)
+    # 在关闭事件循环前加入一小段延迟让底层连接得到关闭的缓冲时间
+    loop.run_until_complete(asyncio.sleep(0.25))
+    count = utils.count_alive(data)
+    logger.log('INFOR', f'经验证{domain}存活子域{count}个')
+    return data
+
+
+def save_data(name, data):
+    """
+    保存请求结果到数据库
+
+    :param str name: 保存表名
+    :param list data: 待保存的数据
+    """
+    db = Database()
+    db.drop_table(name)
+    db.create_table(name)
+    db.save_db(name, data, 'request')
+    db.close()
@@ -1,116 +1,166 @@
-# coding=utf-8
-import asyncio
-import functools
-
-import dns.resolver
-import aiodns
-import tqdm
+import gc
+import json

 import config
-from common import utils
 from config import logger
+from common import utils
+from common.database import Database


-def dns_resolver():
+def filter_subdomain(data):
    """
-    dns解析器
+    过滤出无解析内容的子域到新的子域列表
+
+    :param list data: 待过滤的数据列表
+    :return: 符合条件的子域列表
    """
-    resolver = dns.resolver.Resolver()
-    resolver.nameservers = config.resolver_nameservers
-    resolver.timeout = config.resolver_timeout
-    resolver.lifetime = config.resolver_lifetime
-    return resolver
-
-
-def dns_query_a(hostname):
-    """
-    查询A记录
-
-    :param str hostname: 主机名
-    :return: 查询结果
-    """
-    resolver = dns_resolver()
-    return resolver.query(hostname, 'A')
-
-
-def aiodns_resolver():
-    """
-    异步dns解析器
-    """
-    return aiodns.DNSResolver(nameservers=config.resolver_nameservers,
-                              timeout=config.resolver_timeout)
-
-
-async def aiodns_query_a(hostname, semaphore=None):
-    """
-    异步查询A记录
-
-    :param str hostname: 主机名
-    :param semaphore: 并发查询数量
-    :return: 主机名或查询结果或查询异常
-    """
-    if semaphore is None:
-        resolver = aiodns_resolver()
-        answers = await resolver.query(hostname, 'A')
-        return hostname, answers
-    else:
-        async with semaphore:
-            resolver = aiodns_resolver()
-            answers = await resolver.query(hostname, 'A')
-            return hostname, answers
-
-
-def resolve_callback(future, index, datas):
-    """
-    解析结果回调处理
-    :param future: future对象
-    :param index: 下标
-    :param datas: 结果集
-    """
-    try:
-        result = future.result()
-    except Exception as e:
-        datas[index]['ips'] = str(e.args)
-        datas[index]['valid'] = 0
-    else:
-        if isinstance(result, tuple):
-            _, answers = result
-            if answers:
-                ips = {record.host for record in answers}
-                datas[index]['ips'] = str(ips)
-            else:
-                datas[index]['ips'] = 'No answers'
-
-
-async def bulk_query_a(datas):
-    """
-    批量查询A记录
-
-    :param datas: 待查的数据集
-    :return: 查询过得到的结果集
-    """
-    logger.log('INFOR', '正在异步查询子域的A记录')
-    tasks = []
-    semaphore = asyncio.Semaphore(config.limit_resolve_conn)
-    for i, data in enumerate(datas):
-        if not data.get('ips'):
+    logger.log('DEBUG', f'正在过滤出待解析的子域')
+    subdomains = []
+    for data in data:
+        if not data.get('content'):
            subdomain = data.get('subdomain')
-            task = asyncio.ensure_future(aiodns_query_a(subdomain, semaphore))
-            task.add_done_callback(functools.partial(resolve_callback,
-                                                     index=i,
-                                                     datas=datas))  # 回调
-            tasks.append(task)
-    if tasks:  # 任务列表里有任务不空时才进行解析
-        futures = asyncio.as_completed(tasks)
-        for future in tqdm.tqdm(futures,
-                                total=len(tasks),
-                                desc='Progress',
-                                smoothing=1.0,
-                                ncols=True):
+            subdomains.append(subdomain)
+    return subdomains
+
+
+def update_data(data, records):
+    """
+    更新解析结果
+
+    :param list data: 待更新的数据列表
+    :param dict records: 解析结果字典
+    :return: 更新后的数据列表
+    """
+    logger.log('DEBUG', f'正在更新解析结果')
+    for index, items in enumerate(data):
+        if not items.get('content'):
+            subdomain = items.get('subdomain')
+            record = records.get(subdomain)
+            items.update(record)
+            data[index] = items
+    return data
+
+
+def save_data(name, data):
+    """
+    保存解析结果到数据库
+
+    :param str name: 保存表名
+    :param list data: 待保存的数据
+    """
+    logger.log('INFOR', f'正在保存解析结果')
+    db = Database()
+    db.drop_table(name)
+    db.create_table(name)
+    db.save_db(name, data, 'resolve')
+    db.close()
+
+
+def save_subdomains(save_path, subdomain_list):
+    logger.log('DEBUG', f'正在保存待解析的子域')
+    subdomain_data = '\n'.join(subdomain_list)
+    if not utils.save_data(save_path, subdomain_data):
+        logger.log('FATAL', '保存待解析的子域出错')
+        exit(1)
+
+
+def deal_output(output_path):
+    logger.log('INFOR', f'正在处理解析结果')
+    records = dict()  # 用来记录所有域名解析数据
+    with open(output_path) as fd:
+        for line in fd:
+            line = line.strip()
            try:
-                await future
-            except:
-                pass
-        # await asyncio.wait(tasks)  # 等待所有task完成
-    logger.log('INFOR', '完成异步查询子域的A记录')
-    return datas
+                items = json.loads(line)
+            except Exception as e:
+                logger.log('ERROR', e.args)
+                logger.log('ERROR', f'解析行{line}出错跳过解析该行')
+                continue
+            record = dict()
+            record['resolver'] = items.get('resolver')
+            qname = items.get('name')[:-1]  # 去出最右边的`.`点号
+            status = items.get('status')
+            if status != 'NOERROR':
+                record['alive'] = 0
+                record['resolve'] = 0
+                record['reason'] = status
+                records[qname] = record
+                continue
+            data = items.get('data')
+            if 'answers' not in data:
+                record['alive'] = 0
+                record['resolve'] = 0
+                record['reason'] = 'NOANSWER'
+                records[qname] = record
+                continue
+            flag = False
+            cname = list()
+            ips = list()
+            public = list()
+            ttls = list()
+            answers = data.get('answers')
+            for answer in answers:
+                if answer.get('type') == 'A':
+                    flag = True
+                    cname.append(answer.get('name')[:-1])  # 去出最右边的`.`点号
+                    ip = answer.get('data')
+                    ips.append(ip)
+                    ttl = answer.get('ttl')
+                    ttls.append(str(ttl))
+                    is_public = utils.ip_is_public(ip)
+                    public.append(str(is_public))
+                    record['resolve'] = 1
+                    record['reason'] = status
+                    record['cname'] = ','.join(cname)
+                    record['content'] = ','.join(ips)
+                    record['public'] = ','.join(public)
+                    record['ttl'] = ','.join(ttls)
+                    records[qname] = record
+            if not flag:
+                record['alive'] = 0
+                record['resolve'] = 0
+                record['reason'] = 'NOARECORD'
+                records[qname] = record
+        return records
+
+
+def run_resolve(domain, data):
+    """
+    调用子域解析入口函数
+
+    :param str domain: 待解析的主域
+    :param list data: 待解析的子域数据列表
+    :return: 解析得到的结果列表
+    :rtype: list
+    """
+    logger.log('INFOR', f'开始解析{domain}的子域')
+    subdomains = filter_subdomain(data)
+    if not subdomains:
+        return data
+
+    massdns_dir = config.third_party_dir.joinpath('massdns')
+    result_dir = config.result_save_dir
+    temp_dir = result_dir.joinpath('temp')
+    utils.check_dir(temp_dir)
+    massdns_path = utils.get_massdns_path(massdns_dir)
+    timestring = utils.get_timestring()
+
+    save_name = f'collected_subdomains_{domain}_{timestring}.txt'
+    save_path = temp_dir.joinpath(save_name)
+    save_subdomains(save_path, subdomains)
+    del subdomains
+    gc.collect()
+
+    output_name = f'resolved_result_{domain}_{timestring}.json'
+    output_path = temp_dir.joinpath(output_name)
+    log_path = result_dir.joinpath('massdns.log')
+
+    ns_path = config.brute_nameservers_path
+
+    utils.call_massdns(massdns_path, save_path, ns_path,
+                       output_path, log_path, quiet_mode=True)
+
+    records = deal_output(output_path)
+    data = update_data(data, records)
+    logger.log('INFOR', f'结束解析{domain}的子域')
+    return data
@@ -1,5 +1,3 @@
-# coding=utf-8
-import requests
 import config
 from .module import Module
 from . import utils
@@ -47,7 +45,10 @@ class Search(Module):
        :return: 匹配的子域
        :rtype set
        """
-        resp = requests.head(url, headers=self.header, proxies=self.proxy,
-                             timeout=self.timeout, allow_redirects=False)
+        resp = self.head(url, check=False, allow_redirects=False)
+        if not resp:
+            return set()
        location = resp.headers.get('location')
+        if not location:
+            return set()
        return set(utils.match_subdomain(domain, location))
@@ -1,10 +1,21 @@
-# coding=utf-8
+import os
 import re
+import sys
+import time
 import random
-import ipaddress
 import platform
-import config
+import subprocess
+from ipaddress import IPv4Address, ip_address
+from stat import S_IXUSR
+
+import psutil
+import tenacity
+import requests
 from pathlib import Path
+from records import Record, RecordCollection
+from dns.resolver import Resolver
+
+import config
 from common.domain import Domain
 from config import logger

@@ -48,7 +59,7 @@ def gen_random_ip():
    生成随机的点分十进制的IP字符串
    """
    while True:
-        ip = ipaddress.IPv4Address(random.randint(0, 2 ** 32 - 1))
+        ip = IPv4Address(random.randint(0, 2 ** 32 - 1))
        if ip.is_global:
            return ip.exploded

@@ -65,7 +76,7 @@ def gen_fake_header():
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7',
        'Cache-Control': 'max-age=0',
-        'Connection': 'keep-alive',
+        'Connection': 'close',
        'DNT': '1',
        'Referer': 'https://www.google.com/',
        'Upgrade-Insecure-Requests': '1',
@@ -117,15 +128,23 @@ def get_domains(target):
        domains = target
    elif isinstance(target, str):
        path = Path(target)
-        if path.is_file():
-            with open(target) as file:
+        if path.exists() and path.is_file():
+            with open(target, encoding='utf-8', errors='ignore') as file:
                for line in file:
-                    domain = Domain(line.strip()).match()
+                    line = line.lower().strip()
+                    domain = Domain(line).match()
                    if domain:
                        domains.append(domain)
-        elif Domain(target).match():
-            domains = [target]
-    logger.log('INFOR', f'获取到{len(domains)}个域名')
+        else:
+            target = target.lower().strip()
+            domain = Domain(target).match()
+            if domain:
+                domains.append(domain)
+    count = len(domains)
+    if count == 0:
+        logger.log('FATAL', f'获取到{count}个域名')
+        exit(1)
+    logger.log('INFOR', f'获取到{count}个域名')
    return domains


@@ -137,63 +156,99 @@ def get_semaphore():
    """
    system = platform.system()
    if system == 'Windows':
-        return 300
+        return 800
    elif system == 'Linux':
        return 800
    elif system == 'Darwin':
        return 800


-def check_dpath(dpath):
-    """
-    检查目录路径
+def check_dir(dir_path):
+    if not dir_path.exists():
+        logger.log('INFOR', f'不存在{dir_path}目录将会新建')
+        dir_path.mkdir(parents=True, exist_ok=True)

-    :param dpath: 传入的目录路径
-    :return: 目录路径
+
+def check_path(path, name, format):
    """
-    if isinstance(dpath, str):
-        dpath = Path(dpath)
+    检查结果输出目录路径
+
+    :param path: 保存路径
+    :param name: 导出名字
+    :param format: 保存格式
+    :return: 保存路径
+    """
+    filename = f'{name}.{format}'
+    default_path = config.result_save_dir.joinpath(filename)
+    if isinstance(path, str):
+        path = repr(path).replace('\\', '/')  # 将路径中的反斜杠替换为正斜杠
+        path = path.replace('\'', '')  # 去除多余的转义
    else:
-        dpath = config.result_save_path
-    if not dpath.is_dir():
-        logger.log('FATAL', f'{dpath}不是目录')
-    if not dpath.exists():
-        logger.log('ALERT', f'不存在{dpath}将会新建此目录')
-        dpath.mkdir(parents=True, exist_ok=True)
-    return dpath
+        path = default_path
+    path = Path(path)
+    if not path.suffix:  # 输入是目录的情况
+        path = path.joinpath(filename)
+    parent_dir = path.parent
+    if not parent_dir.exists():
+        logger.log('ALERT', f'不存在{parent_dir}目录将会新建')
+        parent_dir.mkdir(parents=True, exist_ok=True)
+    if path.exists():
+        logger.log('ALERT', f'存在{path}文件将会覆盖')
+    return path


-def check_format(format):
+def check_format(format, count):
    """
    检查导出格式

    :param format: 传入的导出格式
+    :param count: 数量
    :return: 导出格式
    """
-    formats = ['txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
+    formats = ['rst', 'csv', 'tsv', 'json', 'yaml', 'html',
               'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods']
+    if format == 'xls' and count > 65000:
+        logger.log('ALERT', 'xls文件限制为最多65000行')
+        logger.log('ALERT', '使用xlsx格式导出')
+        return 'xlsx'
    if format in formats:
        return format
    else:
        logger.log('ALERT', f'不支持{format}格式导出')
        logger.log('ALERT', '默认使用csv格式导出')
-        return 'xls'
+        return 'csv'


-def save_data(fpath, data):
+def save_data(path, data):
+    """
+    保存数据到文件
+
+    :param path: 保存路径
+    :param data: 待存数据
+    :return: 保存成功与否
+    """
    try:
-        with open(fpath, 'w', encoding="utf-8", newline='') as file:
+        with open(path, 'w', encoding="utf-8",
+                  errors='ignore', newline='') as file:
            file.write(data)
-            logger.log('ALERT', fpath)
+            return True
    except TypeError:
-        with open(fpath, 'wb') as file:
+        with open(path, 'wb') as file:
            file.write(data)
-            logger.log('ALERT', fpath)
+            return True
    except Exception as e:
-        logger.log('ERROR', e)
+        logger.log('ERROR', e.args)
+        return False


 def check_response(method, resp):
+    """
+    检查响应 输出非正常响应返回json的信息
+
+    :param method: 请求方法
+    :param resp: 响应体
+    :return: 是否正常响应
+    """
    if resp.status_code == 200 and resp.content:
        return True
    logger.log('ALERT', f'{method} {resp.url} {resp.status_code} - '
@@ -207,3 +262,318 @@ def check_response(method, resp):
        else:
            logger.log('ALERT', msg)
    return False
+
+
+def mark_subdomain(old_data, now_data):
+    """
+    标记新增子域并返回新的数据集
+
+    :param list old_data: 之前子域数据
+    :param list now_data: 现在子域数据
+    :return: 标记后的的子域数据
+    :rtype: list
+    """
+    # 第一次收集子域的情况
+    mark_data = now_data.copy()
+    if not old_data:
+        for index, item in enumerate(mark_data):
+            item['new'] = 1
+            mark_data[index] = item
+        return mark_data
+    # 非第一次收集子域的情况
+    old_subdomains = {item.get('subdomain') for item in old_data}
+    for index, item in enumerate(mark_data):
+        subdomain = item.get('subdomain')
+        if subdomain in old_subdomains:
+            item['new'] = 0
+        else:
+            item['new'] = 1
+        mark_data[index] = item
+    return mark_data
+
+
+def remove_invalid_string(string):
+    # Excel文件中单元格值不能直接存储以下非法字符
+    return re.sub(r'[\000-\010]|[\013-\014]|[\016-\037]', r'', string)
+
+
+def check_value(values):
+    if not isinstance(values, dict):
+        return values
+    for key, value in values.items():
+        if value is None:
+            continue
+        if isinstance(value, str) and len(value) > 32767:
+            # Excel文件中单元格值长度不能超过32767
+            values[key] = value[:32767]
+    return values
+
+
+def export_all(format, path, datas):
+    """
+    将所有结果数据导出到一个文件
+
+    :param str format: 导出文件格式
+    :param str path: 导出文件路径
+    :param list datas: 待导出的结果数据
+    """
+    format = check_format(format, len(datas))
+    timestamp = get_timestring()
+    name = f'all_subdomain_result_{timestamp}'
+    path = check_path(path, name, format)
+    logger.log('INFOR', f'所有主域的子域结果 {path}')
+    row_list = list()
+    for row in datas:
+        if 'header' in row:
+            row.pop('header')
+        if 'response' in row:
+            row.pop('response')
+        keys = row.keys()
+        values = row.values()
+        if format in {'xls', 'xlsx'}:
+            values = check_value(values)
+        row_list.append(Record(keys, values))
+    rows = RecordCollection(iter(row_list))
+    content = rows.export(format)
+    save_data(path, content)
+
+
+def dns_resolver():
+    """
+    dns解析器
+    """
+    resolver = Resolver()
+    resolver.nameservers = config.resolver_nameservers
+    resolver.timeout = config.resolver_timeout
+    resolver.lifetime = config.resolver_lifetime
+    return resolver
+
+
+def dns_query(qname, qtype):
+    """
+    查询域名DNS记录
+
+    :param str qname: 待查域名
+    :param str qtype: 查询类型
+    :return: 查询结果
+    """
+    logger.log('TRACE', f'尝试查询{qname}的{qtype}记录')
+    resolver = dns_resolver()
+    try:
+        answer = resolver.query(qname, qtype)
+    except Exception as e:
+        logger.log('TRACE', e.args)
+        logger.log('TRACE', f'查询{qname}的{qtype}记录失败')
+        return None
+    else:
+        logger.log('TRACE', f'查询{qname}的{qtype}记录成功')
+        return answer
+
+
+def get_timestamp():
+    return int(time.time())
+
+
+def get_timestring():
+    return time.strftime('%Y%m%d_%H%M%S', time.localtime(time.time()))
+
+
+def get_classname(classobj):
+    return classobj.__class__.__name__
+
+
+def python_version():
+    return sys.version
+
+
+def count_alive(data):
+    return len(list(filter(lambda item: item.get('alive') == 1, data)))
+
+
+def get_subdomains(data):
+    return set(map(lambda item: item.get('subdomain'), data))
+
+
+def set_id_none(data):
+    new_data = []
+    for item in data:
+        item['id'] = None
+        new_data.append(item)
+    return new_data
+
+
+def get_filtered_data(data):
+    filtered_data = []
+    for item in data:
+        valid = item.get('resolve')
+        if valid == 0:
+            filtered_data.append(item)
+    return filtered_data
+
+
+def get_sample_banner(headers):
+    temp_list = []
+    server = headers.get('Server')
+    if server:
+        temp_list.append(server)
+    via = headers.get('Via')
+    if via:
+        temp_list.append(via)
+    power = headers.get('X-Powered-By')
+    if power:
+        temp_list.append(power)
+    banner = ','.join(temp_list)
+    return banner
+
+
+def check_ip_public(ip_list):
+    for ip_str in ip_list:
+        ip = ip_address(ip_str)
+        if not ip.is_global:
+            return 0
+    return 1
+
+
+def ip_is_public(ip_str):
+    ip = ip_address(ip_str)
+    if not ip.is_global:
+        return 0
+    return 1
+
+
+def get_process_num():
+    process_num = config.brute_process_num
+    if isinstance(process_num, int):
+        return min(os.cpu_count(), process_num)
+    else:
+        return 1
+
+
+def get_coroutine_num():
+    coroutine_num = config.resolve_coroutine_num
+    if isinstance(coroutine_num, int):
+        return max(64, coroutine_num)
+    elif coroutine_num is None:
+        mem = psutil.virtual_memory()
+        total_mem = mem.total
+        g_size = 1024 * 1024 * 1024
+        if total_mem <= 1 * g_size:
+            return 64
+        elif total_mem <= 2 * g_size:
+            return 128
+        elif total_mem <= 4 * g_size:
+            return 256
+        elif total_mem <= 8 * g_size:
+            return 512
+        elif total_mem <= 16 * g_size:
+            return 1024
+        else:
+            return 2048
+    else:
+        return 64
+
+
+def uniq_dict_list(dict_list):
+    return list(filter(lambda name: dict_list.count(name) == 1, dict_list))
+
+
+def delete_file(*paths):
+    for path in paths:
+        try:
+            path.unlink()
+        except Exception as e:
+            logger.log('ERROR', e.args)
+
+
+@tenacity.retry(stop=tenacity.stop_after_attempt(2))
+def check_net():
+    logger.log('INFOR', '正在检查网络环境')
+    url = 'http://www.example.com/'
+    logger.log('INFOR', f'访问地址 {url}')
+    try:
+        rsp = requests.get(url)
+    except Exception as e:
+        logger.log('ERROR', e.args)
+        logger.log('ALERT', '访问外网出错 重新检查中')
+        raise tenacity.TryAgain
+    if rsp.status_code != 200:
+        logger.log('ALERT', f'{rsp.request.method} {rsp.request.url} '
+                            f'{rsp.status_code} {rsp.reason}')
+        logger.log('ALERT', '不能正常访问外网 重新检查中')
+        raise tenacity.TryAgain
+    logger.log('INFOR', '能正常访问外网')
+
+
+def check_pre():
+    logger.log('INFOR', '正在检查依赖环境')
+    system = platform.system()
+    implementation = platform.python_implementation()
+    version = platform.python_version()
+    if implementation != 'CPython':
+        logger.log('FATAL', f'OneForAll只在CPython下测试通过')
+        exit(1)
+    if version < '3.6':
+        logger.log('FATAL', 'OneForAll需要Python 3.6以上版本')
+        exit(1)
+    if system == 'Windows' and implementation == 'CPython':
+        if version < '3.8':
+            logger.log('FATAL', 'OneForAll在Windows系统运行时需要Python 3.8以上版本')
+            exit(1)
+
+
+def check_env():
+    logger.log('INFOR', '正在检查运行环境')
+    try:
+        check_net()
+    except Exception as e:
+        logger.log('DEBUG', e.args)
+        logger.log('FATAL', '不能正常访问外网')
+        exit(1)
+    check_pre()
+
+
+def get_maindomain(domain):
+    return Domain(domain).registered()
+
+
+def call_massdns(massdns_path, dict_path, ns_path, output_path, log_path,
+                 query_type='A', process_num=1, concurrent_num=10000,
+                 quiet_mode=False):
+    logger.log('INFOR', f'开始执行massdns')
+    quiet = ''
+    if quiet_mode:
+        quiet = '--quiet'
+    status_format = config.brute_status_format
+    socket_num = config.brute_socket_num
+    resolve_num = config.brute_resolve_num
+    cmd = f'{massdns_path} {quiet} --status-format {status_format} ' \
+          f'--processes {process_num} --socket-count {socket_num} ' \
+          f'--hashmap-size {concurrent_num} --resolvers {ns_path} ' \
+          f'--resolve-count {resolve_num} --type {query_type} ' \
+          f'--flush --output J --outfile {output_path} ' \
+          f'--error-log {log_path} {dict_path}'
+    logger.log('INFOR', f'执行命令 {cmd}')
+    subprocess.run(args=cmd, shell=True)
+    logger.log('INFOR', f'结束执行massdns')
+
+
+def get_massdns_path(massdns_dir):
+    path = config.brute_massdns_path
+    if path:
+        return path
+    system = platform.system().lower()
+    machine = platform.machine().lower()
+    name = f'massdns_{system}_{machine}'
+    if system == 'windows':
+        name = name + '.exe'
+        if machine == 'amd64':
+            massdns_dir = massdns_dir.joinpath('windows', 'x64')
+        else:
+            massdns_dir = massdns_dir.joinpath('windows', 'x84')
+    path = massdns_dir.joinpath(name)
+    path.chmod(S_IXUSR)
+    if not path.exists():
+        logger.log('FATAL', '暂无该系统平台及架构的massdns')
+        logger.log('INFOR', '请尝试自行编译massdns并在配置里指定路径')
+        exit(0)
+    return path
@@ -1,210 +1,190 @@
-# coding=utf-8
-"""
-OneForAll配置
-"""
-import os
-import sys
-import pathlib
-from loguru import logger
-
-# 路径设置
-oneforall_relpath = pathlib.Path(__file__).parent  # oneforall代码相对路径
-oneforall_abspath = oneforall_relpath.resolve()  # oneforall代码绝对路径
-oneforall_module_path = oneforall_relpath.joinpath('modules')  # oneforall模块目录
-data_storage_path = oneforall_relpath.joinpath('data')  # 数据存放目录
-result_save_path = oneforall_relpath.joinpath('results')  # 结果保存目录
-
-# 模块设置
-save_module_result = True  # 保存模块中各脚本结果(默认True)
-enable_all_module = True  # 启用所有模块(默认True)
-enable_partial_module = []  # 启用部分模块 必须禁用enable_all_module才能生效
-# 只使用ask和baidu搜索引擎收集子域
-# enable_partial_module = [('modules.search', 'ask')
-#                          ('modules.search', 'baidu')]
-
-
-# 爆破模块设置
-enable_brute_module = False  # 使用爆破模块(默认禁用)
-enable_verify_subdomain = True  # 验证子域有效性(默认True)
-enable_wildcard_check = True  # 开启泛解析检测 会去掉泛解析的子域
-# 爆破时使用的进程数(根据系统中CPU数量情况设置 不宜大于CPU数量 默认为系统中的CPU数量)
-brute_process_num = os.cpu_count()
-brute_coroutine_num = 128  # 爆破时每个进程下的协程数(不宜大于1000)
-# 爆破所使用的字典路径 默认data/subdomains.txt
-brute_wordlist_path = data_storage_path.joinpath('subnames.txt')
-brute_task_segment = 500
-# 参数segment的设置受CPU性能，网络带宽，运营商限制等限制，默认500个子域为一任务组，
-# 当你觉得你的环境不受以上因素影响，当前爆破速度较慢，那么强烈建议根据字典大小调整大小：
-# 十万字典建议设置为5000，百万字典设置为50000
-enable_recursive_brute = False  # 是否使用递归爆破(默认禁用)
-brute_recursive_depth = 2  # 递归爆破深度(默认2层)
-# 爆破下一层子域所使用的字典路径 默认data/next_subdomains.txt
-recursive_namelist_path = data_storage_path.joinpath('next_subnames.txt')
-enable_fuzz = False  # 是否使用fuzz模式枚举域名
-fuzz_rule = ''  # fuzz域名的正则 示例：[a-z][0-9] 第一位是字母 第二位是数字
-ips_appear_maximum = 10  # 同一IP集合出现次数超过10认为是泛解析
-
-# 代理设置
-enable_proxy = False  # 是否使用代理(全局开关)
-proxy_all_module = False  # 代理所有模块
-proxy_partial_module = ['GoogleQuery', 'AskSearch', 'DuckDuckGoSearch',
-                        'GoogleAPISearch', 'GoogleSearch', 'YahooSearch',
-                        'YandexSearch', 'CrossDomainXml',
-                        'ContentSecurityPolicy']  # 代理自定义的模块
-proxy_pool = [{'http': 'http://127.0.0.1:1080',
-               'https': 'https://127.0.0.1:1080'}]  # 代理池
-# proxy_pool = [{'http': 'socks5://127.0.0.1:10808',
-#                'https': 'socks5://127.0.0.1:10808'}]  # 代理池
-
-
-# 网络请求设置
-enable_fake_header = True  # 启用伪造请求头
-request_delay = 1  # 请求时延
-request_timeout = 30  # 请求超时
-request_verify = True  # 请求SSL验证
-
-# 搜索模块设置
-enable_recursive_search = False  # 递归搜索子域
-search_recursive_times = 2  # 递归搜索层数
-
-# DNS解析设置
-resolver_nameservers = [
-    '119.29.29.29', '182.254.116.116',  # DNSPod
-    '180.76.76.76',  # Baidu DNS
-    '223.5.5.5', '223.6.6.6',  # AliDNS
-    '114.114.114.114', '114.114.115.115'  # 114DNS
-    # '8.8.8.8', '8.8.4.4',  # Google DNS
-    # '1.0.0.1', '1.1.1.1'  # CloudFlare DNS
-    # '208.67.222.222', '208.67.220.220'  # OpenDNS
-]  # 指定查询的DNS域名服务器
-resolver_timeout = 5.0  # 解析超时时间
-resolver_lifetime = 30.0  # 解析存活时间
-limit_resolve_conn = 50  # 限制同一时间解析的数量(默认50)
-
-# http探测设置
-small_ports = {80, 443}
-medium_ports = {80, 443, 8000, 8080, 8443}  # 默认使用
-large_ports = {80, 81, 443, 591, 2082, 2087, 2095, 2096, 3000, 8000, 8001,
-               8008, 8080, 8083, 8443, 8834, 8888}
-xlarge_ports = {80, 81, 300, 443, 591, 593, 832, 981, 1010, 1311, 2082,
-                2087, 2095, 2096, 2480, 3000, 3128, 3333, 4243, 4567, 4711,
-                4712, 4993, 5000, 5104, 5108, 5800, 6543, 7000, 7396, 7474,
-                8000, 8001, 8008, 8014, 8042, 8069, 8080, 8081, 8088, 8090,
-                8091, 8118, 8123, 8172, 8222, 8243, 8280, 8281, 8333, 8443,
-                8500, 8834, 8880, 8888, 8983, 9000, 9043, 9060, 9080, 9090,
-                9091, 9200, 9443, 9800, 9981, 12443, 16080, 18091, 18092,
-                20720, 28017}
-ports = {'small': small_ports, 'medium': medium_ports,
-         'large': large_ports, 'xlarge': xlarge_ports}
-verify_ssl = False
-# aiohttp 支持 HTTP/HTTPS形式的代理
-get_proxy = None  # proxy="http://user:pass@some.proxy.com"
-get_timeout = 120  # http请求探测总超时时间 None或者0则表示不检测超时
-get_redirects = True  # 允许请求跳转
-fake_header = True  # 使用伪造请求头
-# 限制同一时间打开的连接数(默认None，根据系统不同设置，Windows系统400 其他系统800)
-limit_open_conn = None
-# 限制同一时间在同一个端点((host, port, is_ssl) 3者都一样的情况)打开的连接数
-limit_per_host = 0  # 默认0表示不限制
-
-
-# 模块API配置
-# Censys可以免费注册获取API：https://censys.io/api
-censys_api_id = ''
-censys_api_secret = ''
-
-# Binaryedge可以免费注册获取API：https://app.binaryedge.io/account/api
-# 免费的API有效期只有1个月，到期之后可以再次生成，每月可以查询250次。
-binaryedge_api = ''
-
-# Chinaz可以免费注册获取API：http://api.chinaz.com/ApiDetails/Alexa
-chinaz_api = ''
-
-# Bing可以免费注册获取API：https://azure.microsoft.com/zh-cn/services/
-# cognitive-services/bing-web-search-api/#web-json
-bing_api_id = ''
-bing_api_key = ''
-
-# SecurityTrails可以免费注册获取API：https://securitytrails.com/corp/api
-securitytrails_api = ''
-
-# https://fofa.so/api
-fofa_api_email = ''  # fofa用户邮箱
-fofa_api_key = ''  # fofa用户key
-
-# Google可以免费注册获取API:
-# https://developers.google.com/custom-search/v1/overview
-# 免费的API只能查询前100条结果
-google_api_key = ''  # Google API搜索key
-google_api_cx = ''  # Google API搜索cx
-
-# https://api.passivetotal.org/api/docs/
-riskiq_api_username = ''
-riskiq_api_key = ''
-
-# Shodan可以免费注册获取API: https://account.shodan.io/register
-# 免费的API限速1秒查询1次
-shodan_api_key = ''
-# ThreatBook API 查询子域名需要收费 https://x.threatbook.cn/nodev4/vb4/myAPI
-threatbook_api_key = ''
-
-# VirusTotal可以免费注册获取API: https://developers.virustotal.com/reference
-virustotal_api_key = ''
-
-# https://www.zoomeye.org/doc?channel=api
-zoomeye_api_username = ''
-zoomeye_api_password = ''
-
-# Certdb可以免费注册获取API: https://spyse.com/
-certdb_api_token = ''
-
-# https://www.circl.lu/services/passive-dns/
-circl_api_username = ''
-circl_api_password = ''
-
-# https://www.dnsdb.info/
-dnsdb_api_key = ''
-
-# ipv4info可以免费注册获取API: http://ipv4info.com/tools/api/
-# 免费的API有效期只有2天，到期之后可以再次生成，每天可以查询50次。
-ipv4info_api_key = ''
-
-# Github Token可以访问https://github.com/settings/tokens生成,user为Github用户名
-github_api_user = ''
-github_api_token = ''
-
-subdomains_common = {'i', 'w', 'm', 'en', 'us', 'zh', 'w3', 'app', 'bbs',
-                     'web', 'www', 'job', 'docs', 'news', 'blog', 'data',
-                     'help', 'live', 'mall', 'blogs', 'files', 'forum',
-                     'store', 'mobile'}
-
-# 日志配置
-# 终端日志输出格式
-stdout_fmt = '<cyan>{time:HH:mm:ss,SSS}</cyan> ' \
-          '[<level>{level: <5}</level>] ' \
-          '<blue>{module}</blue>:<cyan>{line}</cyan> - ' \
-          '<level>{message}</level>'
-# 日志文件记录格式
-logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
-          '[<level>{level: <5}</level>] ' \
-          '<cyan>{process.name}({process.id})</cyan>:' \
-          '<cyan>{thread.name: <10}({thread.id: <5})</cyan> | ' \
-          '<blue>{module}</blue>.<blue>{function}</blue>:' \
-          '<blue>{line}</blue> - <level>{message}</level>'
-
-log_path = result_save_path.joinpath('oneforall.log')
-
-logger.remove()
-logger.level(name='TRACE', no=5, color='<cyan><bold>', icon='✏️')
-logger.level(name='DEBUG', no=10, color='<blue><bold>', icon='🐞 ')
-logger.level(name='INFOR', no=20, color='<green><bold>', icon='ℹ️')
-logger.level(name='ALERT', no=30, color='<yellow><bold>', icon='⚠️')
-logger.level(name='ERROR', no=40, color='<red><bold>', icon='❌️')
-logger.level(name='FATAL', no=50, color='<RED><bold>', icon='☠️')
-
-if not os.environ.get('PYTHONIOENCODING'):  # 设置编码
-    os.environ['PYTHONIOENCODING'] = 'utf-8'
-
-logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)
-logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True,
-           encoding='utf-8')
+# coding=utf-8
+"""
+OneForAll配置
+"""
+import os
+import sys
+import pathlib
+
+import urllib3
+from loguru import logger
+
+# 路径设置
+relative_directory = pathlib.Path(__file__).parent  # OneForAll代码相对路径
+module_dir = relative_directory.joinpath('modules')  # OneForAll模块目录
+third_party_dir = relative_directory.joinpath('thirdparty')  # 三方工具目录
+data_storage_dir = relative_directory.joinpath('data')  # 数据存放目录
+result_save_dir = relative_directory.joinpath('results')  # 结果保存目录
+
+
+# OneForAll入口参数设置
+enable_dns_resolve = True  # 使用DNS解析子域(默认True)
+enable_http_request = True  # 使用HTTP请求子域(默认True)
+enable_takeover_check = False  # 开启子域接管风险检查(默认False)
+# 参数port可选值有'default', 'small', 'large'
+http_request_port = 'default'  # HTTP请求子域(默认'default'，探测80端口)
+# 参数alive可选值True，False分别表示导出存活，全部子域结果
+result_export_alive = False  # 只导出存活的子域结果(默认False)
+# 参数format可选格式有'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
+# 'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
+result_save_format = 'csv'  # 子域结果保存文件格式(默认csv)
+# 参数path默认None使用OneForAll结果目录自动生成路径
+result_save_path = None  # 子域结果保存文件路径(默认None)
+
+
+# 收集模块设置
+save_module_result = False  # 保存各模块发现结果为json文件(默认False)
+enable_all_module = True  # 启用所有模块(默认True)
+enable_partial_module = []  # 启用部分模块 必须禁用enable_all_module才能生效
+# 只使用ask和baidu搜索引擎收集子域的示例
+# enable_partial_module = [('modules.search', 'ask')
+#                          ('modules.search', 'baidu')]
+module_thread_timeout = 180.0  # 每个收集模块线程超时时间(默认3分钟)
+
+# 爆破模块设置
+enable_brute_module = False  # 使用爆破模块(默认False)
+enable_wildcard_check = True  # 开启泛解析检测(默认True)
+enable_wildcard_deal = True  # 开启泛解析处理(默认True)
+brute_massdns_path = None  # 默认None自动选择 如需填写请填写绝对路径
+brute_status_format = 'ansi'  # 爆破时状态输出格式（默认asni，可选json）
+# 爆破时使用的进程数(根据计算机中CPU数量情况设置 不宜大于逻辑CPU个数)
+brute_process_num = 1  # 默认1
+brute_concurrent_num = 10000  # 并发查询数量(默认10000)
+brute_socket_num = 1  # 爆破时每个进程下的socket数量
+brute_resolve_num = 50  # 解析失败时尝试换名称服务器重查次数
+# 爆破所使用的字典路径 默认data/subdomains.txt
+brute_wordlist_path = data_storage_dir.joinpath('subnames.txt')
+brute_nameservers_path = data_storage_dir.joinpath('cn_nameservers.txt')
+# 域名的权威DNS名称服务器的保存路径 当域名开启了泛解析时会使用该名称服务器来进行A记录查询
+authoritative_dns_path = data_storage_dir.joinpath('authoritative_dns.txt')
+enable_recursive_brute = False  # 是否使用递归爆破(默认False)
+brute_recursive_depth = 2  # 递归爆破深度(默认2层)
+# 爆破下一层子域所使用的字典路径 默认data/next_subdomains.txt
+recursive_nextlist_path = data_storage_dir.joinpath('next_subnames.txt')
+enable_check_dict = False  # 是否开启字典配置检查提示(默认False)
+delete_generated_dict = True  # 是否删除爆破时临时生成的字典(默认True)
+# 是否删除爆破时massdns输出的解析结果 (默认True)
+#  massdns输出的结果中包含更详细解析结果
+#  注意: 当爆破的字典较大或使用递归爆破或目标域名存在泛解析时生成的文件可能会很大
+delete_massdns_result = True
+only_save_valid = True  # 是否在处理爆破结果时只存入解析成功的子域
+check_time = 10  # 检查字典配置停留时间(默认10秒)
+enable_fuzz = False  # 是否使用fuzz模式枚举域名
+fuzz_place = None  # 指定爆破的位置 指定的位置用`@`表示 示例：www.@.example.com
+fuzz_rule = None  # fuzz域名的正则 示例：'[a-z][0-9]' 表示第一位是字母 第二位是数字
+brute_ip_blacklist = {'0.0.0.0', '0.0.0.1'}  # IP黑名单 子域解析到IP黑名单则标记为非法子域
+ip_appear_maximum = 100  # 多个子域解析到同一IP次数超过100次则标记为非法(泛解析)子域
+
+# 代理设置
+enable_proxy = False  # 是否使用代理(全局开关)
+proxy_all_module = False  # 代理所有模块
+proxy_partial_module = ['GoogleQuery', 'AskSearch', 'DuckDuckGoSearch',
+                        'GoogleAPISearch', 'GoogleSearch', 'YahooSearch',
+                        'YandexSearch', 'CrossDomainXml',
+                        'ContentSecurityPolicy']  # 代理自定义的模块
+proxy_pool = [{'http': 'http://127.0.0.1:1080',
+               'https': 'https://127.0.0.1:1080'}]  # 代理池
+# proxy_pool = [{'http': 'socks5h://127.0.0.1:10808',
+#                'https': 'socks5h://127.0.0.1:10808'}]  # 代理池
+
+
+# 网络请求设置
+enable_fake_header = True  # 启用伪造请求头
+request_delay = 1  # 请求时延
+request_timeout = 60  # 请求超时
+request_verify = False  # 请求SSL验证
+# 禁用安全警告信息
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+
+# 搜索模块设置
+enable_recursive_search = False  # 递归搜索子域
+search_recursive_times = 2  # 递归搜索层数
+
+# DNS解析设置
+resolve_coroutine_num = 64
+resolver_nameservers = [
+    '223.5.5.5',  # AliDNS
+    '119.29.29.29',  # DNSPod
+    '114.114.114.114',  # 114DNS
+    '8.8.8.8',   # Google DNS
+    '1.1.1.1'  # CloudFlare DNS
+]  # 指定查询的DNS域名服务器
+resolver_timeout = 5.0  # 解析超时时间
+resolver_lifetime = 60.0  # 解析存活时间
+limit_resolve_conn = 500  # 限制同一时间解析的数量(默认500)
+
+# 请求端口探测设置
+# 你可以在端口列表添加自定义端口
+default_ports = [80]  # 默认使用
+small_ports = [80, 443, 8000, 8080, 8443]
+# 注意：建议大厂的域名尽量不使用大端口范围，因为大厂的子域太多，加上使用大端口范围会导致生成的
+# 请求上十万，百万，千万级，可能会导致内存不足程序奔溃，另外这样级别的请求量等待时间也是漫长的。
+# OneForAll不是一个端口扫描工具，如果要扫端口建议使用nmap,zmap之类的工具。
+large_ports = [80, 81, 280, 300, 443, 591, 593, 832, 888, 901, 981, 1010, 1080,
+               1100, 1241, 1311, 1352, 1434, 1521, 1527, 1582, 1583, 1944, 2082,
+               2082, 2086, 2087, 2095, 2096, 2222, 2301, 2480, 3000, 3128, 3333,
+               4000, 4001, 4002, 4100, 4125, 4243, 4443, 4444, 4567, 4711, 4712,
+               4848, 4849, 4993, 5000, 5104, 5108, 5432, 5555, 5800, 5801, 5802,
+               5984, 5985, 5986, 6082, 6225, 6346, 6347, 6443, 6480, 6543, 6789,
+               7000, 7001, 7002, 7396, 7474, 7674, 7675, 7777, 7778, 8000, 8001,
+               8002, 8003, 8004, 8005, 8006, 8008, 8009, 8010, 8014, 8042, 8069,
+               8075, 8080, 8081, 8083, 8088, 8090, 8091, 8092, 8093, 8016, 8118,
+               8123, 8172, 8181, 8200, 8222, 8243, 8280, 8281, 8333, 8384, 8403,
+               8443, 8500, 8530, 8531, 8800, 8806, 8834, 8880, 8887, 8888, 8910,
+               8983, 8989, 8990, 8991, 9000, 9043, 9060, 9080, 9090, 9091, 9200,
+               9294, 9295, 9443, 9444, 9800, 9981, 9988, 9990, 9999, 10000,
+               10880, 11371, 12043, 12046, 12443, 15672, 16225, 16080, 18091,
+               18092, 20000, 20720, 24465, 28017, 28080, 30821, 43110, 61600]
+ports = {'default': default_ports, 'small': small_ports, 'large': large_ports}
+
+# aiohttp有关配置
+verify_ssl = False
+# aiohttp 支持 HTTP/HTTPS形式的代理
+aiohttp_proxy = None  # proxy="http://user:pass@some.proxy.com"
+allow_redirects = True  # 允许请求跳转
+fake_header = True  # 使用伪造请求头
+# 为了保证请求质量 请谨慎更改以下设置
+# request_method只能是HEAD或GET,HEAD请求方法更快，但是不能获取响应体并提取从中提取
+request_method = 'GET'  # 使用请求方法，默认GET
+sockread_timeout = 10  # 每个请求socket读取超时时间，默认5秒
+sockconn_timeout = 10  # 每个请求socket连接超时时间，默认5秒
+# 限制同一时间打开的连接总数
+limit_open_conn = 100  # 默认100
+# 限制同一时间在同一个端点((host, port, is_ssl) 3者都一样的情况)打开的连接数
+limit_per_host = 10  # 0表示不限制,默认10
+
+subdomains_common = {'i', 'w', 'm', 'en', 'us', 'zh', 'w3', 'app', 'bbs',
+                     'web', 'www', 'job', 'docs', 'news', 'blog', 'data',
+                     'help', 'live', 'mall', 'blogs', 'files', 'forum',
+                     'store', 'mobile'}
+
+# 日志配置
+# 终端日志输出格式
+stdout_fmt = '<cyan>{time:HH:mm:ss,SSS}</cyan> ' \
+             '[<level>{level: <5}</level>] ' \
+             '<blue>{module}</blue>:<cyan>{line}</cyan> - ' \
+             '<level>{message}</level>'
+# 日志文件记录格式
+logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
+              '[<level>{level: <5}</level>] ' \
+              '<cyan>{process.name}({process.id})</cyan>:' \
+              '<cyan>{thread.name: <18}({thread.id: <5})</cyan> | ' \
+              '<blue>{module}</blue>.<blue>{function}</blue>:' \
+              '<blue>{line}</blue> - <level>{message}</level>'
+
+log_path = result_save_dir.joinpath('oneforall.log')
+
+logger.remove()
+logger.level(name='TRACE', no=5, color='<cyan><bold>', icon='✏️')
+logger.level(name='DEBUG', no=10, color='<blue><bold>', icon='🐞 ')
+logger.level(name='INFOR', no=20, color='<green><bold>', icon='ℹ️')
+logger.level(name='ALERT', no=30, color='<yellow><bold>', icon='⚠️')
+logger.level(name='ERROR', no=40, color='<red><bold>', icon='❌️')
+logger.level(name='FATAL', no=50, color='<RED><bold>', icon='☠️')
+
+if not os.environ.get('PYTHONIOENCODING'):  # 设置编码
+    os.environ['PYTHONIOENCODING'] = 'utf-8'
+
+logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)
+logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True,
+           encoding='utf-8')
@@ -0,0 +1,7 @@
+223.5.5.5
+223.6.6.6
+114.114.114.114
+114.114.115.115
+180.76.76.76
+119.29.29.29
+182.254.116.116
@@ -1,102 +1,102 @@
-[
-"_afpovertcp._tcp.",
-"_aix._tcp.",
-"_autodiscover._tcp.",
-"_caldav._tcp.",
-"_certificates._tcp.",
-"_client._smtp.",
-"_cmp._tcp.",
-"_crls._tcp.",
-"_crl._tcp.",
-"_finger._tcp.",
-"_ftp._tcp.",
-"_gc._tcp.",
-"_h323be._tcp.",
-"_h323be._udp.",
-"_h323cs._tcp.",
-"_h323cs._udp.",
-"_h323ls._tcp.",
-"_h323ls._udp.",
-"_h323rs._tcp.",
-"_hkps._tcp.",
-"_hkp._tcp.",
-"_http._tcp.",
-"_iax.udp.",
-"_imaps._tcp.",
-"_imap._tcp.",
-"_jabber-client._tcp.",
-"_jabber-client._udp.",
-"_jabber._tcp.",
-"_jabber._udp.",
-"_kerberos-adm._tcp.",
-"_kerberos._tcp.",
-"_kerberos._tcp.dc._msdcs.",
-"_kerberos._udp.",
-"_kpasswd._tcp.",
-"_kpasswd._udp.",
-"_ldap._tcp.",
-"_ldap._tcp.dc._msdcs.",
-"_ldap._tcp.gc._msdcs.",
-"_ldap._tcp.pdc._msdcs.",
-"_msdcs.",
-"_mysqlsrv._tcp.",
-"_nntp._tcp.",
-"_ntp._udp.",
-"_ocsp._tcp.",
-"_pgpkeys._tcp.",
-"_pgprevokations._tcp.",
-"_PKIXREP._tcp.",
-"_pop3s._tcp.",
-"_pop3._tcp.",
-"_sipfederationtls._tcp.",
-"_sipinternal._tcp.",
-"_sipinternaltls._tcp.",
-"_sips._tcp.",
-"_sip._tcp.",
-"_sip._tls.",
-"_sip._udp.",
-"_smtp._tcp.",
-"_ssh._tcp.",
-"_stun._tcp.",
-"_stun._udp.",
-"_svcp._tcp.",
-"_tcp.",
-"_telnet._tcp.",
-"_test._tcp.",
-"_tls.",
-"_udp.",
-"_vlmcs._tcp.",
-"_vlmcs._udp.",
-"_whois._tcp.",
-"_wpad._tcp.",
-"_xmpp-client._tcp.",
-"_xmpp-client._udp.",
-"_xmpp-server._tcp.",
-"_xmpp-server._udp.",
-"_https._tcp.",
-"_imap.tcp.",
-"_kerberos.tcp.dc._msdcs.",
-"_ldap._tcp.ForestDNSZones.",
-"_submission._tcp.",
-"_caldavs._tcp.",
-"_carddav._tcp.",
-"_carddavs._tcp.",
-"_x-puppet._tcp.",
-"_x-puppet-ca._tcp.",
-"_domainkey.",
-"_pkixrep._tcp.",
-"_cisco-phone-http.",
-"_cisco-phone-tftp.",
-"_cisco-uds._tcp.",
-"_ciscowtp._tcp.",
-"_collab-edge._tls.",
-"_cuplogin._tcp.",
-"_client._smtp._tcp.",
-"_sftp._tcp.",
-"_h323rs._udp.",
-"_sql._tcp.",
-"_sip._tcp.internal.",
-"_snmp._udp.",
-"_rdp._tcp.",
-"_xmpp-server._udp."
-]
+[
+"_afpovertcp._tcp.",
+"_aix._tcp.",
+"_autodiscover._tcp.",
+"_caldav._tcp.",
+"_certificates._tcp.",
+"_client._smtp.",
+"_cmp._tcp.",
+"_crls._tcp.",
+"_crl._tcp.",
+"_finger._tcp.",
+"_ftp._tcp.",
+"_gc._tcp.",
+"_h323be._tcp.",
+"_h323be._udp.",
+"_h323cs._tcp.",
+"_h323cs._udp.",
+"_h323ls._tcp.",
+"_h323ls._udp.",
+"_h323rs._tcp.",
+"_hkps._tcp.",
+"_hkp._tcp.",
+"_http._tcp.",
+"_iax.udp.",
+"_imaps._tcp.",
+"_imap._tcp.",
+"_jabber-client._tcp.",
+"_jabber-client._udp.",
+"_jabber._tcp.",
+"_jabber._udp.",
+"_kerberos-adm._tcp.",
+"_kerberos._tcp.",
+"_kerberos._tcp.dc._msdcs.",
+"_kerberos._udp.",
+"_kpasswd._tcp.",
+"_kpasswd._udp.",
+"_ldap._tcp.",
+"_ldap._tcp.dc._msdcs.",
+"_ldap._tcp.gc._msdcs.",
+"_ldap._tcp.pdc._msdcs.",
+"_msdcs.",
+"_mysqlsrv._tcp.",
+"_nntp._tcp.",
+"_ntp._udp.",
+"_ocsp._tcp.",
+"_pgpkeys._tcp.",
+"_pgprevokations._tcp.",
+"_PKIXREP._tcp.",
+"_pop3s._tcp.",
+"_pop3._tcp.",
+"_sipfederationtls._tcp.",
+"_sipinternal._tcp.",
+"_sipinternaltls._tcp.",
+"_sips._tcp.",
+"_sip._tcp.",
+"_sip._tls.",
+"_sip._udp.",
+"_smtp._tcp.",
+"_ssh._tcp.",
+"_stun._tcp.",
+"_stun._udp.",
+"_svcp._tcp.",
+"_tcp.",
+"_telnet._tcp.",
+"_test._tcp.",
+"_tls.",
+"_udp.",
+"_vlmcs._tcp.",
+"_vlmcs._udp.",
+"_whois._tcp.",
+"_wpad._tcp.",
+"_xmpp-client._tcp.",
+"_xmpp-client._udp.",
+"_xmpp-server._tcp.",
+"_xmpp-server._udp.",
+"_https._tcp.",
+"_imap.tcp.",
+"_kerberos.tcp.dc._msdcs.",
+"_ldap._tcp.ForestDNSZones.",
+"_submission._tcp.",
+"_caldavs._tcp.",
+"_carddav._tcp.",
+"_carddavs._tcp.",
+"_x-puppet._tcp.",
+"_x-puppet-ca._tcp.",
+"_domainkey.",
+"_pkixrep._tcp.",
+"_cisco-phone-http.",
+"_cisco-phone-tftp.",
+"_cisco-uds._tcp.",
+"_ciscowtp._tcp.",
+"_collab-edge._tls.",
+"_cuplogin._tcp.",
+"_client._smtp._tcp.",
+"_sftp._tcp.",
+"_h323rs._udp.",
+"_sql._tcp.",
+"_sip._tcp.internal.",
+"_snmp._udp.",
+"_rdp._tcp.",
+"_xmpp-server._udp."
+]
@@ -9,11 +9,13 @@ OneForAll数据库导出模块
 """

 import fire
+
 from common import utils
 from common.database import Database
+from config import logger


-def export(table, db=None, valid=None, dpath=None, format='xls', show=False):
+def export(table, db=None, alive=False, limit=None, path=None, format='csv', show=False):
    """
    OneForAll数据库导出模块

@@ -22,36 +24,32 @@ def export(table, db=None, valid=None, dpath=None, format='xls', show=False):
        python3 dbexport.py --db result.db --table name --show False

    Note:
-        参数port可选值有'small', 'medium', 'large', 'xlarge'，详见config.py配置
+        参数alive可选值True，False分别表示导出存活，全部子域结果
        参数format可选格式有'txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
                          'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
-        参数dpath为None默认使用OneForAll结果目录
+        参数path默认None使用OneForAll结果目录自动生成路径

    :param str table:   要导出的表
    :param str db:      要导出的数据库路径(默认为results/result.sqlite3)
-    :param int valid:   导出子域的有效性(默认None)
-    :param str format:  导出格式(默认xls)
-    :param str dpath:    导出目录(默认None)
+    :param bool alive:  只导出存活的子域结果(默认False)
+    :param str limit:   导出限制条件(默认None)
+    :param str format:  导出文件格式(默认csv)
+    :param str path:    导出文件路径(默认None)
    :param bool show:   终端显示导出数据(默认False)
    """
-    format = utils.check_format(format)
-    dpath = utils.check_dpath(dpath)
+
    database = Database(db)
-    if valid is None:
-        rows = database.get_data(table)
-    elif isinstance(valid, int):
-        rows = database.get_subdomain(table, valid)
-    else:
-        rows = database.get_data(table)  # 意外情况导出全部子域
+    rows = database.export_data(table, alive, limit)
+    format = utils.check_format(format, len(rows))
+    path = utils.check_path(path, table, format)
    if show:
        print(rows.dataset)
-    if format == 'txt':
-        data = str(rows.dataset)
-    else:
-        data = rows.export(format)
+    data = rows.export(format)
    database.close()
-    fpath = dpath.joinpath(f'{table}.{format}')
-    utils.save_data(fpath, data)
+    utils.save_data(path, data)
+    logger.log('INFOR', f'{table}主域的子域结果 {path}')
+    data_dict = rows.as_dict()
+    return data_dict


 if __name__ == '__main__':
@@ -7,5 +7,10 @@

 import oneforall

-test = oneforall.OneForAll(target='example.com')
-test.run()
+if __name__ == '__main__':
+    test = oneforall.OneForAll(target='github.com')
+    test.brute = True
+    test.req = False
+    test.takeover = True
+    test.run()
+    result = test.datas
@@ -8,7 +8,7 @@ github自动接管
 import json
 import base64
 import requests
-import config
+import api

 HEADERS = {
    "Accept": "application/json, text/javascript, */*; q=0.01",
@@ -21,8 +21,8 @@ def github_takeover(url):
    # 读取config配置文件
    repo_name = url
    print('[*]正在读取配置文件...')
-    user = config.github_api_user
-    token = config.github_api_token
+    user = api.github_api_user
+    token = api.github_api_token
    CHECK_HEADERS = {
        "Authorization": 'token ' + token,
        "Accept": "application/vnd.github.switcheroo-preview+json"
@@ -1,5 +1,4 @@
-import time
-import config
+import api
 from common.query import Query
 from config import logger

@@ -11,8 +10,8 @@ class CensysAPI(Query):
        self.module = 'Certificate'
        self.source = "CensysAPIQuery"
        self.addr = 'https://www.censys.io/api/v1/search/certificates'
-        self.id = config.censys_api_id
-        self.secret = config.censys_api_secret
+        self.id = api.censys_api_id
+        self.secret = api.censys_api_secret
        self.delay = 3.0  # Censys 接口查询速率限制 最快2.5秒查1次

    def query(self):
@@ -24,21 +23,20 @@ class CensysAPI(Query):
        data = {
            'query': f'parsed.names: {self.domain}',
            'page': 1,
-            'fields': ['parsed.subject_dn'],
+            'fields': ['parsed.subject_dn', 'parsed.names'],
            'flatten': True}
        resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
        if not resp:
            return
-        data = resp.json()
-        status = data.get('status')
+        json = resp.json()
+        status = json.get('status')
        if status != 'ok':
            logger.log('ALERT', status)
            return
-        subdomains = self.match(self.domain, str(data))
+        subdomains = self.match(self.domain, str(json))
        self.subdomains = self.subdomains.union(subdomains)
-        pages = data.get('metadata').get('pages')
+        pages = json.get('metadata').get('pages')
        for page in range(2, pages + 1):
-            time.sleep(self.delay)
            data['page'] = page
            resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
            if not resp:
@@ -1,4 +1,3 @@
-import time
 from common import utils
 from common.query import Query

@@ -15,7 +14,6 @@ class CertSpotter(Query):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        params = {'domain': self.domain,
@@ -1,4 +1,3 @@
-import time
 from common import utils
 from common.query import Query

@@ -15,7 +14,6 @@ class Crtsh(Query):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        params = {'q': f'%.{self.domain}', 'output': 'json'}
@@ -1,4 +1,3 @@
-import time
 from common import utils
 from common.query import Query

@@ -15,7 +14,6 @@ class Entrust(Query):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        params = {'fields': 'subjectDN',
@@ -1,4 +1,3 @@
-import time
 from common import utils
 from common.query import Query

@@ -16,7 +15,6 @@ class Google(Query):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        params = {'include_expired': 'true',
@@ -1,67 +1,65 @@
-import time
-import config
-from common import utils
-from common.query import Query
-
-
-class CertDBAPI(Query):
-    def __init__(self, domain):
-        Query.__init__(self)
-        self.domain = domain
-        self.module = 'Certificate'
-        self.source = 'CertDBAPIQuery'
-        self.addr = 'https://api.spyse.com/v1/subdomains'
-        self.token = config.certdb_api_token
-
-    def query(self):
-        """
-        向接口查询子域并做子域匹配
-        """
-        page_num = 1
-        while True:
-            time.sleep(self.delay)
-            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
-            params = {'domain': self.domain,
-                      'api_token': self.token,
-                      'page': page_num}
-            resp = self.get(self.addr, params)
-            if not resp:
-                return
-            json = resp.json()
-            subdomains = utils.match_subdomain(self.domain, str(json))
-            if not subdomains:  # 搜索没有发现子域名则停止搜索
-                break
-            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
-            page_num += 1
-            # 默认每次查询最多返回30条 当前条数小于30条说明已经查完
-            if json.get('count') < 30:
-                break
-
-    def run(self):
-        """
-        类执行入口
-        """
-        if not self.check(self.token):
-            return
-        self.begin()
-        self.query()
-        self.finish()
-        self.save_json()
-        self.gen_result()
-        self.save_db()
-
-
-def do(domain):  # 统一入口名字 方便多线程调用
-    """
-    类统一调用入口
-
-    :param str domain: 域名
-    """
-    query = CertDBAPI(domain)
-    query.run()
-
-
-if __name__ == '__main__':
-    do('example.com')
+import api
+from common import utils
+from common.query import Query
+
+
+class SpyseAPI(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = domain
+        self.module = 'Certificate'
+        self.source = 'CertDBAPIQuery'
+        self.addr = 'https://api.spyse.com/v1/subdomains'
+        self.token = api.spyse_api_token
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        page_num = 1
+        while True:
+            self.header = self.get_header()
+            self.proxy = self.get_proxy(self.source)
+            params = {'domain': self.domain,
+                      'api_token': self.token,
+                      'page': page_num}
+            resp = self.get(self.addr, params)
+            if not resp:
+                return
+            json = resp.json()
+            subdomains = utils.match_subdomain(self.domain, str(json))
+            if not subdomains:  # 搜索没有发现子域名则停止搜索
+                break
+            # 合并搜索子域名搜索结果
+            self.subdomains = self.subdomains.union(subdomains)
+            page_num += 1
+            # 默认每次查询最多返回30条 当前条数小于30条说明已经查完
+            if json.get('count') < 30:
+                break
+
+    def run(self):
+        """
+        类执行入口
+        """
+        if not self.check(self.token):
+            return
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = SpyseAPI(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('example.com')
@@ -10,7 +10,7 @@ DNS域传送(DNS zone transfer)指的是一台备用域名服务器使用来自
 import dns.resolver
 import dns.zone

-from common import resolve, utils
+from common import utils
 from common.module import Module
 from config import logger

@@ -34,10 +34,11 @@ class CheckAXFR(Module):
        """
        logger.log('DEBUG', f'尝试对{self.domain}的域名服务器{server}进行域传送')
        try:
-            xfr = dns.query.xfr(server, self.domain, timeout=30.0)
+            xfr = dns.query.xfr(where=server, zone=self.domain,
+                                timeout=5.0, lifetime=10.0)
            zone = dns.zone.from_xfr(xfr)
        except Exception as e:
-            logger.log('DEBUG', str(e))
+            logger.log('DEBUG', e.args)
            logger.log('DEBUG', f'对{self.domain}的域名服务器{server}进行域传送失败')
            return
        names = zone.nodes.keys()
@@ -56,11 +57,11 @@ class CheckAXFR(Module):
        """
        正则匹配响应头中的内容安全策略字段以发现子域名
        """
-        resolver = resolve.dns_resolver()
+        resolver = utils.dns_resolver()
        try:
            answers = resolver.query(self.domain, "NS")
        except Exception as e:
-            logger.log('ERROR', e)
+            logger.log('ERROR', e.args)
            return
        nsservers = [str(answer) for answer in answers]
        if not len(nsservers):
@@ -93,4 +94,4 @@ def do(domain):  # 统一入口名字 方便多线程调用

 if __name__ == '__main__':
    do('ZoneTransfer.me')
-    do('example.com')
+    # do('example.com')
@@ -24,16 +24,15 @@ class CheckCDX(Module):
                f'https://{self.domain}/crossdomain.xml',
                f'http://www.{self.domain}/crossdomain.xml',
                f'https://www.{self.domain}/crossdomain.xml']
-        response = None
        for url in urls:
            self.header = self.get_header()
            self.proxy = self.get_proxy(self.source)
-            response = self.get(url)
-            if response:
-                break
-        if not response:
-            return
-        self.subdomains = utils.match_subdomain(self.domain, response.text)
+            response = self.get(url, check=False)
+            if not response:
+                return
+            if response and len(response.content):
+                self.subdomains = utils.match_subdomain(self.domain,
+                                                        response.text)

    def run(self):
        """
@@ -23,13 +23,14 @@ class CheckCert(Module):
        """
        获取域名证书并匹配证书中的子域名
        """
-        ctx = ssl.create_default_context()
-        sock = ctx.wrap_socket(socket.socket(), server_hostname=self.domain)
        try:
+            ctx = ssl.create_default_context()
+            sock = ctx.wrap_socket(socket.socket(),
+                                   server_hostname=self.domain)
            sock.connect((self.domain, self.port))
            cert_dict = sock.getpeercert()
        except Exception as e:
-            logger.log('ERROR', e)
+            logger.log('DEBUG', e.args)
            return
        subdomains = utils.match_subdomain(self.domain, str(cert_dict))
        self.subdomains = self.subdomains.union(subdomains)
@@ -12,38 +12,46 @@ class CheckCSP(Module):
    """
    检查内容安全策略收集子域名
    """
-
    def __init__(self, domain, header):
        Module.__init__(self)
        self.domain = self.register(domain)
        self.module = 'Check'
        self.source = 'ContentSecurityPolicy'
-        self.header = header
+        self.csp_header = header
+
+    def grab_header(self):
+        """
+        抓取请求头
+
+        :return: 请求头
+        """
+        csp_header = dict()
+        urls = [f'http://{self.domain}',
+                f'https://{self.domain}',
+                f'http://www.{self.domain}',
+                f'https://www.{self.domain}']
+        for url in urls:
+            self.header = self.get_header()
+            self.proxy = self.get_proxy(self.source)
+            response = self.get(url, check=False)
+            if response:
+                csp_header = response.headers
+                break
+        return csp_header

    def check(self):
        """
        正则匹配响应头中的内容安全策略字段以发现子域名
        """
-        if not self.header:
-            urls = [f'http://{self.domain}',
-                    f'https://{self.domain}',
-                    f'http://www.{self.domain}',
-                    f'https://www.{self.domain}']
-            response = None
-            for url in urls:
-                self.header = self.get_header()
-                self.proxy = self.get_proxy(self.source)
-                response = self.get(url)
-                if response:
-                    break
-            if not response:
-                return
-            self.header = response.headers
+        if not self.csp_header:
+            self.csp_header = self.grab_header()
        csp = self.header.get('Content-Security-Policy')
+        if not self.csp_header:
+            logger.log('DEBUG', f'获取{self.domain}域的请求头失败')
+            return
        if not csp:
            logger.log('DEBUG', f'{self.domain}域的响应头不存在内容安全策略字段')
            return
-        logger.log('DEBUG', f'{self.domain}域的响应头存在内容安全策略字段')
        self.subdomains = utils.match_subdomain(self.domain, csp)

    def run(self):
@@ -1,18 +1,14 @@
 """
 检查内容安全策略收集子域名收集子域名
 """
-import requests
-
 from common.module import Module
 from common import utils
-from config import logger


 class CheckRobots(Module):
    """
    检查robots.txt收集子域名
    """
-
    def __init__(self, domain):
        Module.__init__(self)
        self.domain = self.register(domain)
@@ -27,16 +23,15 @@ class CheckRobots(Module):
                f'https://{self.domain}/robots.txt',
                f'http://www.{self.domain}/robots.txt',
                f'https://www.{self.domain}/robots.txt']
-        response = None
        for url in urls:
            self.header = self.get_header()
            self.proxy = self.get_proxy(self.source)
-            response = self.get(url, allow_redirects=False)
-            if response:
-                break
-        if not response:
-            return
-        self.subdomains = utils.match_subdomain(self.domain, response.text)
+            response = self.get(url, check=False, allow_redirects=False)
+            if not response:
+                return
+            if response and len(response.content):
+                self.subdomains = utils.match_subdomain(self.domain,
+                                                        response.text)

    def run(self):
        """
@@ -1,18 +1,14 @@
 """
 检查内容安全策略收集子域名收集子域名
 """
-import requests
-
 from common.module import Module
 from common import utils
-from config import logger


 class CheckRobots(Module):
    """
    检查sitemap收集子域名
    """
-
    def __init__(self, domain):
        Module.__init__(self)
        self.domain = self.register(domain)
@@ -39,17 +35,17 @@ class CheckRobots(Module):
                f'https://{self.domain}/sitemap_index.xml',
                f'http://www.{self.domain}/sitemap_index.xml',
                f'https://www.{self.domain}/sitemap_index.xml']
-        response = None
        for url in urls:
            self.header = self.get_header()
            self.proxy = self.get_proxy(self.source)
            self.timeout = 10
-            response = self.get(url, allow_redirects=False)
-            if response:
-                break
-        if not response:
-            return
-        self.subdomains = utils.match_subdomain(self.domain, response.text)
+            response = self.get(url, check=False, allow_redirects=False)
+            if not response:
+                return
+            if response and len(response.content):
+                self.subdomains = utils.match_subdomain(self.domain,
+                                                        response.text)
+

    def run(self):
        """
@@ -1,5 +1,4 @@
-import time
-import config
+import api
 from common.query import Query


@@ -10,13 +9,12 @@ class BinaryEdgeAPI(Query):
        self.module = 'Dataset'
        self.source = 'BinaryEdgeAPIQuery'
        self.addr = 'https://api.binaryedge.io/v2/query/domains/subdomain/'
-        self.api = config.binaryedge_api
+        self.api = api.binaryedge_api

    def query(self):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.header.update({'X-Key': self.api})
        self.proxy = self.get_proxy(self.source)
@@ -1,5 +1,6 @@
-import time
+import cloudscraper
 from common.query import Query
+from config import logger


 class BufferOver(Query):
@@ -8,20 +9,25 @@ class BufferOver(Query):
        self.domain = self.register(domain)
        self.module = 'Dataset'
        self.source = 'BufferOverQuery'
-        self.addr = 'https://dns.bufferover.run/dns'
+        self.addr = 'https://dns.bufferover.run/dns?q='

    def query(self):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
-        self.header = self.get_header()
-        self.proxy = self.get_proxy(self.source)
-        params = {'q': self.domain}
-        resp = self.get(self.addr, params)
-        if not resp:
+        # 绕过cloudFlare验证
+        scraper = cloudscraper.create_scraper()
+        scraper.interpreter = 'js2py'
+        scraper.proxies = self.get_proxy(self.source)
+        url = self.addr + self.domain
+        try:
+            resp = scraper.get(url, timeout=self.timeout)
+        except Exception as e:
+            logger.log('ERROR', e.args)
            return
-        subdomains = self.match(self.domain, resp.text)
+        if resp.status_code != 200:
+            return
+        subdomains = self.match(self.domain, str(resp.json()))
        # 合并搜索子域名搜索结果
        self.subdomains = self.subdomains.union(subdomains)

@@ -0,0 +1,49 @@
+from common.query import Query
+
+
+class CeBaidu(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Dataset'
+        self.source = 'CeBaiduQuery'
+        self.addr = 'https://ce.baidu.com/index/getRelatedSites'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+        params = {'site_address': self.domain}
+        resp = self.get(self.addr, params)
+        if not resp:
+            return
+        subdomains = self.match(self.domain, str(resp.json()))
+        # 合并搜索子域名搜索结果
+        self.subdomains = self.subdomains.union(subdomains)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = CeBaidu(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('example.com')
@@ -1,4 +1,3 @@
-import time
 from common.query import Query


@@ -14,7 +13,6 @@ class Chinaz(Query):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        self.addr = self.addr + self.domain
@@ -1,5 +1,4 @@
-import time
-import config
+import api
 from common.query import Query


@@ -10,13 +9,12 @@ class ChinazAPI(Query):
        self.module = 'Dataset'
        self.source = 'ChinazAPIQuery'
        self.addr = 'https://apidata.chinaz.com/CallAPI/Alexa'
-        self.api = config.chinaz_api
+        self.api = api.chinaz_api

    def query(self):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        params = {'key': self.api, 'domainName': self.domain}
@@ -1,5 +1,4 @@
-import time
-import config
+import api
 from common.query import Query


@@ -10,14 +9,13 @@ class CirclAPI(Query):
        self.module = 'Dataset'
        self.source = 'CirclAPIQuery'
        self.addr = 'https://www.circl.lu/pdns/query/'
-        self.user = config.circl_api_username
-        self.pwd = config.circl_api_password
+        self.user = api.circl_api_username
+        self.pwd = api.circl_api_password

    def query(self):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        resp = self.get(self.addr + self.domain, auth=(self.user, self.pwd))
@@ -1,92 +0,0 @@
-import time
-import random
-import cloudscraper
-from bs4 import BeautifulSoup
-from common.query import Query
-from config import logger
-
-
-class DNSdb(Query):
-    def __init__(self, domain):
-        Query.__init__(self)
-        self.domain = self.register(domain)
-        self.module = 'Dataset'
-        self.source = 'DNSdbQuery'
-        self.addr = 'http://www.dnsdb.org/'
-        self.url = f'{self.addr}{self.domain}/'
-
-    def get_tokens(self):
-        """
-        绕过cloudFlare验证并获取taken
-
-        :return: 绕过失败返回None 成功返回tokens
-        """
-        scraper = cloudscraper.create_scraper()
-        scraper.interpreter = 'js2py'
-        scraper.proxies = self.get_proxy(self.source)
-        scraper.timeout = 10
-        try:
-            tokens = scraper.get_tokens(self.url)
-        except Exception as e:
-            logger.log('ERROR', e.args)
-            return None
-        if len(tokens) != 2:
-            return None
-        return tokens
-
-    def query(self):
-        """
-        向接口查询子域并做子域匹配
-        """
-        tokens = self.get_tokens()
-        if not tokens:
-            logger.log('ALERT', f'{self.source}模块绕过cloudFlare检查失败')
-            return False
-        self.cookie = tokens[0]
-        self.header = {'User-Agent': tokens[1]}
-        self.timeout = 10
-        resp = self.get(self.url)
-        if not resp:
-            return
-        if 'index' in resp.text:
-            soup = BeautifulSoup(resp.text, features='lxml')
-            base = self.addr+self.domain
-            urls = list(map(lambda a: base + '/' + a.get('href'),
-                            soup.find_all('a')))
-            urls = urls[:-1]  # idn域名暂时不考虑
-            for url in urls:
-                resp = self.get(url)
-                if not resp:
-                    return
-                subdomains = self.match(self.domain, resp.text)
-                # 合并搜索子域名搜索结果
-                self.subdomains = self.subdomains.union(subdomains)
-        else:
-            subdomains = self.match(self.domain, resp.text)
-            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
-
-    def run(self):
-        """
-        类执行入口
-        """
-        self.begin()
-        self.query()
-        self.finish()
-        self.save_json()
-        self.gen_result()
-        self.save_db()
-
-
-def do(domain):  # 统一入口名字 方便多线程调用
-    """
-    类统一调用入口
-
-    :param str domain: 域名
-    """
-    query = DNSdb(domain)
-    query.run()
-
-
-if __name__ == '__main__':
-    do('example.com')
@@ -1,5 +1,4 @@
-import time
-import config
+import api
 from common import utils
 from common.query import Query

@@ -11,13 +10,12 @@ class DNSdbAPI(Query):
        self.module = 'Dataset'
        self.source = 'DNSdbAPIQuery'
        self.addr = 'https://api.dnsdb.info/lookup/rrset/name/'
-        self.api = config.dnsdb_api_key
+        self.api = api.dnsdb_api_key

    def query(self):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.header.update({'X-API-Key': self.api})
        self.proxy = self.get_proxy(self.source)
@@ -25,10 +23,9 @@ class DNSdbAPI(Query):
        resp = self.get(url)
        if not resp:
            return
-        if resp.status_code == 200:
-            subdomains = utils.match_subdomain(self.domain, resp.text)
-            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+        subdomains = utils.match_subdomain(self.domain, resp.text)
+        # 合并搜索子域名搜索结果
+        self.subdomains = self.subdomains.union(subdomains)

    def run(self):
        """
@@ -1,4 +1,3 @@
-import time
 from common import utils
 from common.query import Query

@@ -15,7 +14,6 @@ class DNSdumpster(Query):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.header.update({'Referer': 'https://dnsdumpster.com'})
        self.proxy = self.get_proxy(self.source)
@@ -0,0 +1,49 @@
+from common.query import Query
+
+
+class IP138(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Dataset'
+        self.source = 'IP138Query'
+        self.addr = 'https://site.ip138.com/{domain}/domain.htm'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+        self.addr = self.addr.format(domain=self.domain)
+        resp = self.get(self.addr)
+        if not resp:
+            return
+        subdomains = self.match(self.domain, resp.text)
+        # 合并搜索子域名搜索结果
+        self.subdomains = self.subdomains.union(subdomains)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = IP138(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('example.com')
@@ -1,68 +1,75 @@
-import config
-from common.query import Query
-
-
-class IPv4InfoAPI(Query):
-    def __init__(self, domain):
-        Query.__init__(self)
-        self.domain = self.register(domain)
-        self.module = 'Dataset'
-        self.source = 'IPv4InfoAPIQuery'
-        self.addr = ' http://ipv4info.com/api_v1/'
-        self.api = config.ipv4info_api_key
-
-    def query(self):
-        """
-        向接口查询子域并做子域匹配
-        """
-        page = 0
-        while True:
-            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
-            params = {'type': 'SUBDOMAINS', 'key': self.api,
-                      'value': self.domain, 'page': page}
-            resp = self.get(self.addr, params)
-            if not resp:
-                return
-            if resp.status_code != 200:
-                break  # 请求不正常通常网络是有问题，不再继续请求下去
-            data = resp.json()
-            subdomains = self.match(self.domain, str(data))
-            if not subdomains:
-                break
-            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
-            # 不直接使用subdomains是因为可能里面会出现不符合标准的子域名
-            subdomains = data.get('Subdomains')
-            if subdomains:
-                # ipv4info子域查询接口每次最多返回300个 用来判断是否还有下一页
-                if len(subdomains) < 300:
-                    break
-            page += 1
-            if page >= 50:  # ipv4info子域查询接口最多允许查询50页
-                break
-
-    def run(self):
-        """
-        类执行入口
-        """
-        self.begin()
-        self.query()
-        self.finish()
-        self.save_json()
-        self.gen_result()
-        self.save_db()
-
-
-def do(domain):  # 统一入口名字 方便多线程调用
-    """
-    类统一调用入口
-
-    :param str domain: 域名
-    """
-    query = IPv4InfoAPI(domain)
-    query.run()
-
-
-if __name__ == '__main__':
-    do('example.com')
+import api
+from common.query import Query
+from config import logger
+
+
+class IPv4InfoAPI(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Dataset'
+        self.source = 'IPv4InfoAPIQuery'
+        self.addr = ' http://ipv4info.com/api_v1/'
+        self.api = api.ipv4info_api_key
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        page = 0
+        while True:
+            self.header = self.get_header()
+            self.proxy = self.get_proxy(self.source)
+            params = {'type': 'SUBDOMAINS', 'key': self.api,
+                      'value': self.domain, 'page': page}
+            resp = self.get(self.addr, params)
+            if not resp:
+                return
+            if resp.status_code != 200:
+                break  # 请求不正常通常网络是有问题，不再继续请求下去
+            try:
+                json = resp.json()
+            except Exception as e:
+                logger.log('DEBUG', e.args)
+                break
+            subdomains = self.match(self.domain, str(json))
+            if not subdomains:
+                break
+            # 合并搜索子域名搜索结果
+            self.subdomains = self.subdomains.union(subdomains)
+            # 不直接使用subdomains是因为可能里面会出现不符合标准的子域名
+            subdomains = json.get('Subdomains')
+            if subdomains:
+                # ipv4info子域查询接口每次最多返回300个 用来判断是否还有下一页
+                if len(subdomains) < 300:
+                    break
+            page += 1
+            if page >= 50:  # ipv4info子域查询接口最多允许查询50页
+                break
+
+    def run(self):
+        """
+        类执行入口
+        """
+        if not self.check(self.api):
+            return
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = IPv4InfoAPI(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('example.com')
@@ -24,12 +24,13 @@ class NetCraft(Query):
        self.header = self.get_header()  # Netcraft会检查User-Agent
        resp = self.get(self.init)
        if not resp:
-            return None
+            return False
        self.cookie = resp.cookies
        cookie_value = self.cookie['netcraft_js_verification_challenge']
        cookie_encode = parse.unquote(cookie_value).encode('utf-8')
        verify_taken = hashlib.sha1(cookie_encode).hexdigest()
        self.cookie['netcraft_js_verification_response'] = verify_taken
+        return True

    def query(self):
        """
@@ -0,0 +1,54 @@
+import api
+from common.query import Query
+
+
+class PassiveDnsAPI(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Dataset'
+        self.source = 'PassiveDnsQuery'
+        self.addr = api.passivedns_api_addr or 'http://api.passivedns.cn'
+        self.token = api.passivedns_api_token
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.header.update({'X-AuthToken': self.token})
+        self.proxy = self.get_proxy(self.source)
+        url = self.addr + '/flint/rrset/*.' + self.domain
+        resp = self.get(url)
+        if not resp:
+            return
+        subdomains = self.match(self.domain, str(resp.json()))
+        # 合并搜索子域名搜索结果
+        self.subdomains = self.subdomains.union(subdomains)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        if not self.check(self.addr):
+            return
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = PassiveDnsAPI(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('example.com')
@@ -0,0 +1,61 @@
+from common.query import Query
+
+
+class QianXun(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = domain
+        self.module = 'Query'
+        self.source = 'QianXunQuery'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+
+        num = 1
+        while True:
+            data = {'ecmsfrom': '',
+                    'show': '',
+                    'num': '',
+                    'classid': '0',
+                    'keywords': self.domain}
+            url = f'https://www.dnsscan.cn/dns.html?' \
+                  f'keywords={self.domain}&page={num}'
+            resp = self.post(url, data)
+            if not resp:
+                break
+            subdomains = self.match(self.domain, resp.text)
+            self.subdomains = self.subdomains.union(subdomains)
+            if '<div id="page" class="pagelist">' not in resp.text:
+                break
+            if '<li class="disabled"><span>&raquo;</span></li>' in resp.text:
+                break
+            num += 1
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = QianXun(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('example.com')
@@ -0,0 +1,50 @@
+from common import utils
+from common.query import Query
+
+
+class RapidDNS(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Dataset'
+        self.source = 'RapidDNSQuery'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+        url = f'http://rapiddns.io/subdomain/{self.domain}'
+        params = {'full': '1'}
+        resp = self.get(url, params)
+        if not resp:
+            return
+        subdomains = utils.match_subdomain(self.domain, resp.text)
+        # 合并搜索子域名搜索结果
+        self.subdomains = self.subdomains.union(subdomains)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = RapidDNS(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('example.com')
@@ -1,4 +1,3 @@
-import time
 from common.query import Query


@@ -14,7 +13,6 @@ class Riddler(Query):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        params = {'q': 'pld:' + self.domain}
@@ -1,5 +1,4 @@
-import time
-import config
+import api
 from common.query import Query


@@ -8,16 +7,15 @@ class SecurityTrailsAPI(Query):
        Query.__init__(self)
        self.domain = self.register(domain)
        self.module = 'Dataset'
-        self.source = 'SecurityTrailsQuery'
+        self.source = 'SecurityTrailsAPIQuery'
        self.addr = 'https://api.securitytrails.com/v1/domain/'
-        self.api = config.securitytrails_api
+        self.api = api.securitytrails_api
        self.delay = 2  # SecurityTrails查询时延至少2秒

    def query(self):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        params = {'apikey': self.api}
@@ -1,4 +1,3 @@
-import time

 from common.query import Query

@@ -19,7 +18,6 @@ class SiteDossier(Query):
        向接口查询子域并做子域匹配
        """
        while True:
-            time.sleep(self.delay)
            self.header = self.get_header()
            self.proxy = self.get_proxy(self.source)
            url = f'{self.addr}{self.domain}/{self.page_num}'
@@ -8,6 +8,7 @@ class ThreatCrowd(Query):
    def __init__(self, domain):
        Query.__init__(self)
        self.domain = self.register(domain)
+        self.module = 'Dataset'
        self.source = 'ThreatCrowdQuery'
        self.addr = 'https://www.threatcrowd.org/searchApi' \
                    '/v2/domain/report?domain='
@@ -23,7 +24,7 @@ class ThreatCrowd(Query):
        except Exception as e:
            logger.log('ERROR', e.args)
            return
-        if not resp:
+        if resp.status_code != 200:
            return
        subdomains = self.match(self.domain, str(resp.json()))
        # 合并搜索子域名搜索结果
@@ -52,4 +53,4 @@ def do(domain):  # 统一入口名字 方便多线程调用


 if __name__ == '__main__':
-    do('example.com')
+    do('mi.com')
@@ -0,0 +1,66 @@
+import time
+from config import logger
+from common.query import Query
+
+
+class WZPCQuery(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Dataset'
+        self.source = 'WZPCQuery'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+
+        base_addr = 'http://114.55.181.28/check_web/' \
+                    'databaseInfo_mainSearch.action'
+        page_num = 1
+        while True:
+            time.sleep(self.delay)
+            self.header = self.get_header()
+            self.proxy = self.get_proxy(self.source)
+            params = {'isSearch': 'true', 'searchType': 'url',
+                      'term': self.domain, 'pageNo': page_num}
+            try:
+                resp = self.get(base_addr, params)
+            except Exception as e:
+                logger.log('ERROR', e.args)
+                break
+            if not resp:
+                break
+            subdomains = self.match(self.domain, resp.text)
+            self.subdomains = self.subdomains.union(subdomains)
+            if not subdomains:
+                break
+            if page_num > 10:
+                break
+            page_num += 1
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = WZPCQuery(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('sc.gov.cn')
+    do('bkzy.org')
@@ -0,0 +1,50 @@
+from common.query import Query
+
+
+class Ximcx(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Dataset'
+        self.source = 'XimcxQuery'
+        self.addr = 'http://sbd.ximcx.cn/DomainServlet'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+        data = {'domain': self.domain}
+        resp = self.post(self.addr, data=data)
+        if not resp:
+            return
+        json = resp.json()
+        subdomains = self.match(self.domain, str(json))
+        # 合并搜索子域名搜索结果
+        self.subdomains = self.subdomains.union(subdomains)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = Ximcx(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('example.com')
@@ -0,0 +1,35 @@
+from common.lookup import Lookup
+
+
+class QueryMX(Lookup):
+    def __init__(self, domain):
+        Lookup.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'dnsquery'
+        self.source = "QueryMX"
+        self.type = 'MX'  # 利用的DNS记录的MX记录收集子域
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    brute = QueryMX(domain)
+    brute.run()
+
+
+if __name__ == '__main__':
+    do('cuit.edu.cn')
@@ -0,0 +1,35 @@
+from common.lookup import Lookup
+
+
+class QueryNS(Lookup):
+    def __init__(self, domain):
+        Lookup.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'dnsquery'
+        self.source = "QueryNS"
+        self.type = 'NS'  # 利用的DNS记录的NS记录收集子域
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    brute = QueryNS(domain)
+    brute.run()
+
+
+if __name__ == '__main__':
+    do('cuit.edu.cn')
@@ -0,0 +1,35 @@
+from common.lookup import Lookup
+
+
+class QuerySOA(Lookup):
+    def __init__(self, domain):
+        Lookup.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'dnsquery'
+        self.source = "QuerySOA"
+        self.type = 'SOA'  # 利用的DNS记录的SOA记录收集子域
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    brute = QuerySOA(domain)
+    brute.run()
+
+
+if __name__ == '__main__':
+    do('cuit.edu.cn')
@@ -1,72 +1,58 @@
-#!/usr/bin/env python3
-
 """
 通过枚举域名常见的SRV记录并做查询来发现子域
 """

-import asyncio
 import json
-
-import aiodns
+import queue
+import threading

 from common import utils
 from common.module import Module
-from config import data_storage_path, logger, resolver_nameservers
+from config import data_storage_dir, logger


 class BruteSRV(Module):
-    def __init__(self, domain: str):
+    def __init__(self, domain):
        Module.__init__(self)
        self.domain = self.register(domain)
        self.module = 'dnsquery'
        self.source = "BruteSRV"
-        self.loop = asyncio.new_event_loop()
-        self.nameservers = resolver_nameservers
-        self.resolver = aiodns.DNSResolver(self.nameservers, self.loop)
+        self.type = 'SRV'  # 利用的DNS记录的SRV记录查询子域
+        self.thread_num = 10
+        self.names_que = queue.Queue()
+        self.answers_que = queue.Queue()

-    async def query(self, name):
-        """
-        查询域名的SRV记录
-        :param str name: SRV记录
-        :return: 查询结果
-        """
-        logger.log('TRACE', f'尝试查询{name}的SRV记录')
-        try:
-            answers = await self.resolver.query(name, 'SRV')
-        except Exception as e:
-            logger.log('TRACE', e)
-            logger.log('TRACE', f'查询{name}的SRV记录失败')
-            return None
-        else:
-            logger.log('TRACE', f'查询{name}的SRV记录成功')
-            return answers
+    def gen_names(self):
+        path = data_storage_dir.joinpath('srv_prefixes.json')
+        with open(path, encoding='utf-8', errors='ignore') as file:
+            prefixes = json.load(file)
+        names = map(lambda prefix: prefix + self.domain, prefixes)
+
+        for name in names:
+            self.names_que.put(name)

    def brute(self):
        """
        枚举域名的SRV记录
        """
-        names_path = data_storage_path.joinpath('srv_names.json')
-        with open(names_path) as fp:
-            names_dict = json.load(fp)
-        query_map = map(lambda name: name + self.domain, names_dict)
+        self.gen_names()

-        tasks = []
-        for query in query_map:
-            tasks.append(self.query(query))
-        task_group = asyncio.gather(*tasks, loop=self.loop)
-        self.loop.run_until_complete(asyncio.gather(task_group))
-        self.loop.close()
-        results = task_group.result()
-        for result in results:
-            if result:
-                for answer in result:
-                    subdomains = utils.match_subdomain(self.domain, answer.host)
-                    if subdomains:
-                        self.subdomains = self.subdomains.union(subdomains)
-                    else:
-                        logger.log('DEBUG', f'{answer.host}不是{self.domain}的子域')
-        if not len(self.subdomains):
-            logger.log('DEBUG', f'没有找到{self.domain}的SRV记录')
+        for i in range(self.thread_num):
+            thread = BruteThread(self.names_que, self.answers_que)
+            thread.daemon = True
+            thread.start()
+
+        self.names_que.join()
+
+        while not self.answers_que.empty():
+            answer = self.answers_que.get()
+            if answer is None:
+                continue
+            for item in answer:
+                record = str(item)
+                subdomains = utils.match_subdomain(self.domain, record)
+                self.subdomains = self.subdomains.union(subdomains)
+                self.gen_record(subdomains, record)

    def run(self):
        """
@@ -80,6 +66,20 @@ class BruteSRV(Module):
        self.save_db()


+class BruteThread(threading.Thread):
+    def __init__(self, names_que, answers_que):
+        threading.Thread.__init__(self)
+        self.names_que = names_que
+        self.answers_que = answers_que
+
+    def run(self):
+        while True:
+            name = self.names_que.get()
+            answer = utils.dns_query(name, 'SRV')
+            self.answers_que.put(answer)
+            self.names_que.task_done()
+
+
 def do(domain):  # 统一入口名字 方便多线程调用
    """
    类统一调用入口
@@ -91,5 +91,5 @@ def do(domain):  # 统一入口名字 方便多线程调用


 if __name__ == '__main__':
-
-    do('example.com')
+    do('zonetransfer.me')
+    # do('example.com')
@@ -0,0 +1,35 @@
+from common.lookup import Lookup
+
+
+class QueryTXT(Lookup):
+    def __init__(self, domain):
+        Lookup.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'dnsquery'
+        self.source = "QueryTXT"
+        self.type = 'TXT'  # 利用的DNS记录的TXT记录收集子域
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    brute = QueryTXT(domain)
+    brute.run()
+
+
+if __name__ == '__main__':
+    do('cuit.edu.cn')
@@ -0,0 +1,58 @@
+from common.query import Query
+
+
+class AlienVault(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Intelligence'
+        self.source = 'AlienVaultQuery'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+
+        base = 'https://otx.alienvault.com/api/v1/indicators/domain'
+        dns = f'{base}/{self.domain}/passive_dns'
+        resp = self.get(dns)
+        if not resp:
+            return
+        json = resp.json()
+        subdomains = self.match(self.domain, str(json))
+        self.subdomains = self.subdomains.union(subdomains)
+
+        url = f'{base}/{self.domain}/url_list'
+        resp = self.get(url)
+        if not resp:
+            return
+        json = resp.json()
+        subdomains = self.match(self.domain, str(json))
+        self.subdomains = self.subdomains.union(subdomains)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = AlienVault(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    do('example.com')
@@ -1,4 +1,4 @@
-import config
+import api
 from common.query import Query


@@ -9,8 +9,8 @@ class RiskIQ(Query):
        self.module = 'Intelligence'
        self.source = 'RiskIQAPIQuery'
        self.addr = 'https://api.passivetotal.org/v2/enrichment/subdomains'
-        self.user = config.riskiq_api_username
-        self.key = config.riskiq_api_key
+        self.user = api.riskiq_api_username
+        self.key = api.riskiq_api_key

    def query(self):
        """
@@ -1,4 +1,4 @@
-import config
+import api
 from common.query import Query


@@ -9,7 +9,7 @@ class ThreatBookAPI(Query):
        self.module = 'Intelligence'
        self.source = 'ThreatBookAPIQuery'
        self.addr = 'https://x.threatbook.cn/api/v1/domain/query'
-        self.key = config.threatbook_api_key
+        self.key = api.threatbook_api_key

    def query(self):
        """
@@ -1,5 +1,3 @@
-import time
-
 from common.query import Query


@@ -15,7 +13,6 @@ class ThreatMiner(Query):
        """
        向接口查询子域并做子域匹配
        """
-        time.sleep(self.delay)
        self.header = self.get_header()
        self.proxy = self.get_proxy(self.source)
        params = {'e': 'subdomains_container',
@@ -1,5 +1,3 @@
-import time
-
 from common.query import Query

 '''
@@ -21,7 +19,6 @@ class VirusTotal(Query):
        """
        next_cursor = ''
        while True:
-            time.sleep(self.delay)
            self.header = self.get_header()
            self.header.update({'Referer': 'https://www.virustotal.com/',
                                'TE': 'Trailers'})
@@ -1,4 +1,4 @@
-import config
+import api
 from common.query import Query


@@ -9,7 +9,7 @@ class VirusTotalAPI(Query):
        self.module = 'Intelligence'
        self.source = 'VirusTotalAPIQuery'
        self.addr = 'https://www.virustotal.com/vtapi/v2/domain/report'
-        self.key = config.virustotal_api_key
+        self.key = api.virustotal_api_key

    def query(self):
        """
@@ -21,9 +21,11 @@ class VirusTotalAPI(Query):
        resp = self.get(self.addr, params)
        if not resp:
            return
-        data = resp.json()
-        subdomains = set(data.get('subdomains'))
-        self.subdomains = self.subdomains.union(subdomains)
+        json = resp.json()
+        data = json.get('subdomains')
+        if data:
+            subdomains = set(data)
+            self.subdomains = self.subdomains.union(subdomains)

    def run(self):
        """
@@ -15,18 +15,19 @@ class Baidu(Search):

    def redirect_match(self, domain, html):
        """
+        获取跳转地址并传递地址进行跳转head请求

-        :param domain:
-        :param html:
-        :return:
+        :param domain: 域名
+        :param html: 响应体
+        :return: 子域
        """
-        bs = BeautifulSoup(html, features='lxml')
+        bs = BeautifulSoup(html, 'html.parser')
        subdomains_all = set()
        # 获取搜索结果中所有的跳转URL地址
        for find_res in bs.find_all('a', {'class': 'c-showurl'}):
            url = find_res.get('href')
-            subdomain = self.match_location(domain, url)
-            subdomains_all = subdomains_all.union(subdomain)
+            subdomains = self.match_location(domain, url)
+            subdomains_all = subdomains_all.union(subdomains)
        return subdomains_all

    def search(self, domain, filtered_subdomain='', full_search=False):
@@ -43,7 +44,8 @@ class Baidu(Search):
            self.header = self.get_header()
            self.proxy = self.get_proxy(self.source)
            query = 'site:' + domain + filtered_subdomain
-            params = {'wd': query, 'pn': self.page_num,
+            params = {'wd': query,
+                      'pn': self.page_num,
                      'rn': self.per_page_num}
            resp = self.get(self.addr, params)
            if not resp:
@@ -73,9 +75,7 @@ class Baidu(Search):
        类执行入口
        """
        self.begin()
-
        self.search(self.domain, full_search=True)
-
        # 排除同一子域搜索结果过多的子域以发现新的子域
        for statement in self.filter(self.domain, self.subdomains):
            self.search(self.domain, filtered_subdomain=statement)
@@ -106,4 +106,4 @@ def do(domain):  # 统一入口名字 方便多线程调用


 if __name__ == '__main__':
-    do('example.com')
+    do('huayunshuzi.com')
@@ -46,7 +46,7 @@ class Bing(Search):
            # 合并搜索子域名搜索结果
            self.subdomains = self.subdomains.union(subdomains)
            # 搜索页面没有出现下一页时停止搜索
-            if '<div class="sw_next>' not in resp.text:
+            if '<div class="sw_next">' not in resp.text:
                break
            self.page_num += self.per_page_num
            if self.page_num >= self.limit_num:  # 搜索条数限制
--- a/Show More
+++ b/Show More